Tritium: One Weight Format for Ternary Inference
on Commodity CPUs and Multiplier-Free Hardware

Introduction

Autoregressive decoding at batch size 1 is memory-bound. Throughput is bounded by $$\text{tok/s} \approx \frac{\text{effective memory bandwidth}}{\text{bytes touched per token}}$$ because every weight is read once per token and almost no arithmetic is reused. The standard response is to shrink the weights, which is what four-bit and eight-bit post-training quantisation do.

Ternary models attack a different term. BitNet b1.58 [1] trains with weights constrained to $\{-1, 0, +1\}$, so the inner product $$y_r = \sum_c W_{rc}\, x_c, \qquad W_{rc} \in \{-1,0,+1\}$$ contains no multiplication at all: $$y_r = \sum_{c\,:\,W_{rc}=+1} x_c \;-\; \sum_{c\,:\,W_{rc}=-1} x_c .$$ Each weight participates additively, subtractively, or not at all. We refer to this as select-accumulate. On the 2B4T checkpoint 42.19% of weights are exactly zero, so nearly half of the terms are skips.

This is well known. What is less examined is that the runtimes which execute these models store weights in layouts chosen for a CPU, and a datapath with no multipliers in it would have to rewrite those bytes before it could consume them. The arithmetic is hardware-friendly; the encoding is not.

Tritium [4] is an attempt to remove that seam. Its contributions are:

1. A bit-plane weight layout (Section 3) whose 16-byte unit is simultaneously one SIMD inner-loop iteration and one hardware cycle’s weight input, at exactly 2.000 bits per weight.

2. A multiplier-free hardware core (Section 5) that consumes that layout directly from a memory-mapped file, verified bit-exact against a software reference and asserted free of multiplier cells by synthesis on every build.

3. An exactness-based verification methodology (Section 7) in which three independent implementations must produce identical integers. We give a concrete defect this caught that no tolerance-based test would have.

4. An evaluation that includes its own negative results (Section 8), including a comparison in which Tritium is 2.09$\times$ slower than the state of the art, and the measurement errors we made and corrected while producing it.

Background

The model

All measurements use microsoft/bitnet-b1.58-2B-4T, a 2.4B-parameter decoder-only transformer: 30 layers, hidden size 2560, FFN width 6912, 20 attention heads with 5 key/value heads (4:1 grouped-query attention), vocabulary 128256, squared-ReLU activation, tied input and output embeddings. Ternary weights appear in all seven projections per layer; norms, embeddings and the language-model head are dense.

Where the bytes are

Table 1 gives the per-token traffic. The result is not what a ternary-focused design anticipates: the dense tied head moves 2.52$\times$ more data than every ternary projection combined. Any runtime that vectorises the ternary kernel and leaves the head slow has not moved end-to-end throughput.

The .trit v1 layout

A ternary weight has three states, so two bits is the natural budget under any encoding. The question is how those two bits are arranged.

Interleaved codes versus bit planes

The obvious encoding assigns a 2-bit code per weight (00$=0$, 01$=+1$, 10$=-1$), packing four weights per byte. Extracting the sign of lane $k$ then requires a shift by $2k$ and a mask, per weight. This defeats vectorisation: there is no SIMD instruction whose operand is “every other pair of bits, sign-extended”. An earlier version of this system used that encoding and ran a branch per weight at 0.16 tokens/s.

Tritium instead stores weights as bit planes. A beat is 16 bytes carrying 64 consecutive columns of one row as two 64-bit masks:

bytes 0..8    pos   bit k set => W[r][c+k] = +1
bytes 8..16   neg   bit k set => W[r][c+k] = -1

Neither bit set encodes zero. Both bits set is invalid and is rejected at load and flagged by a sticky error in hardware. Defining it as an error rather than assigning it a value is deliberate: a kernel that accumulates the two planes separately would compute $+x-x=0$, while a lane that tests pos first would compute $+x$. Making it illegal is what prevents the software and hardware paths from disagreeing on a file neither should have accepted.

Under this layout each of the two sums in Equation 3 is a mask applied to a vector of activations. On AVX-512 the mask is literally a k register, and the operation is two masked loads and two vpdpbusd. The decision “does this weight participate, and with which sign” costs no branch, no table lookup, and no per-weight arithmetic.

The property that matters

The bit count is identical to the interleaved encoding. What changes is that one beat is simultaneously one iteration of the CPU kernel’s inner loop and one cycle’s weight input to the hardware core. A memory-mapped .trit file is not a container that the runtime decodes into some other layout; it is the layout. Nothing between the page cache and the accumulator rewrites a byte, on either target.

This is the design commitment the rest of the system is organised around. The CPU and the hardware are not two implementations of one idea kept in correspondence by discipline. They consume the same bytes.

Invariants

Two invariants are checked on demand and are load-bearing for both targets.

P1: planes are disjoint. $\texttt{pos} \wedge \texttt{neg} = 0$ for every beat.

P2: padding is clear. When cols is not a multiple of 64, the final beat of each row covers columns that do not exist. Those bits must be zero in both planes. With $$\texttt{tail} = \begin{cases} \texttt{UINT64\_MAX} & \texttt{cols} \bmod 64 = 0\\ (\texttt{UINT64\_C(1)} \ll (\texttt{cols} \bmod 64)) - 1 & \text{otherwise} \end{cases}$$ every row’s last beat satisfies $(\texttt{pos} \mid \texttt{neg}) \wedge \sim\!\texttt{tail} = 0$.

P2 is what makes zero padding exact rather than conventional: a padded column contributes nothing, so a kernel may process whole 64-column blocks unconditionally and never needs a scalar tail loop. Both invariants are cheap to establish at write time and $O(\text{payload})$ to verify.

At column counts that are multiples of 64, which every projection in this model class satisfies, the layout costs exactly 2.000 bits per weight.

CPU kernels

Kernels exist for AVX-512 VNNI, AVX-512BW, AVX2, NEON, NEON with the dotprod extension, and a portable scalar path. All read the planes in place. Because the accumulators are i32 and integer addition is exact and order-independent, every kernel is bit-identical to the portable one. That is what permits the differential tests of Section 7 to demand equality rather than a tolerance, and why changing kernels can never move a logit.

Kernel selection is resolved once by runtime feature detection. An override pins a kernel by name and errors if the CPU cannot run it rather than falling back, so a benchmark can never silently measure a path other than the one it names, and a CI runner without AVX-512 fails loudly instead of re-testing the scalar path under a different label.

On popcount

Population count is the natural primitive when activations are also 1-bit. Here weights are 1.58-bit but activations are int8, so a popcount of a weight mask only counts participating terms and cannot recover their sum. The production kernels mask activation bytes and accumulate the two planes separately.

The bit-serial popcount formulation is nonetheless implemented, because it is the formulation that maps directly onto the hardware adder tree. It measures 22$\times$ slower than the AVX-512 path at int8 activations (Table 2). It is retained as a third independent implementation for differential testing, and because it becomes the correct kernel if activations ever drop below 8 bits.

The hardware core

trit_matvec.sv is a streaming ternary matrix-vector core. Weights arrive as two 64-bit planes per beat, matching the .trit v1 payload byte for byte. Activations are preloaded as int8. Each of 64 lanes muxes $\{+x, -x, 0\}$ into a combinational adder tree feeding a 32-bit accumulator.

Two details are worth stating because they are where naive implementations diverge from software. First, lanes sign-extend to the accumulator width before negating: $x = -128$ has no positive counterpart in eight bits, and any implementation that negates in the int8 domain produces a different answer on that input. The golden vector set includes an extremes case that feeds $-128$ deliberately, and it must never be weakened. Second, the four lane states are expressed as a unique case rather than a nested conditional, which tells synthesis they are mutually exclusive; written as an if-chain the module synthesises to 48.4k cells instead of 33.7k.

Yosys generic synthesis reports 33,659 cells at 64 lanes, of which approximately 4.1k flip-flops are the flattened activation memory (block RAM on any real device). The build asserts zero $mul and zero $macc cells on every run. Norm folding (Section 6) additionally removes the reciprocal square root and the divide from the element datapath.

Hardware in the loop

The core is exposed to the runtime as an ordinary backend behind the same trait as the CPU kernels, driven through a C shim over the Verilated model. Because v1 stores planes in the core’s beat order, mapped model bytes stream in directly with no repacking between the file and the device under test.

The full 2B4T model decodes end to end through this path, producing byte-identical output to the CPU backend, at 24.4 seconds per token under simulation. That figure is a simulator speed, not a device speed, and no FPGA performance is claimed anywhere in this paper.

Numerics

Three evaluation modes form a ladder, each a strictly more exact evaluation of the same model, selected automatically to the best the architecture supports.

Reference. Textbook. RMSNorm materialised, activations quantised per matvec.

Folded. The per-element $1/\text{rms}$ divide leaves the datapath entirely. Absmax quantisation codes are invariant under a uniform positive scale, so the codes may be computed from $x \odot g$ directly and the rms survives as a single per-token scalar folded into the output scale. This is a numerics result and a synthesis result at once: it is why the hardware needs no divider.

IntMlp. The squared-ReLU stage never exists in floating point. With $g = a_g S_g$ and $u = a_u S_u$ from the gate and up projections and uniform positive scales, $\mathrm{relu}(g)^2 u = t K$ for integer $t = \mathrm{relu}(a_g)^2 a_u$ with $|t| < 2^{55}$, exact in i64, and uniform $K = S_g^2 S_u$.

Both upper rungs were validated against the real checkpoint before being made the default. Measured activation ranges motivated the design: the residual stream reaches $\sim$138k and the squared-ReLU stage $\sim$1.5$\times10^{10}$, so int16 is not viable for those stages and the wide accumulators are not conservatism.

Verification

Three independent implementations of the same arithmetic must agree:

• The oracle. A deliberately naive scalar implementation that expands bit planes to one i8 per weight. Its job is to be evidently correct, not fast.

• The runtime. Zero-copy, vectorised, threaded.

• The hardware core, under Verilator.

Agreement on the integer path is demanded exactly. Lane order, thread count and the hardware’s beat-serial accumulation must all produce identical bits. This is achievable by construction rather than by tuning, because integer addition is exact and order-independent; an implementation that cannot meet it is wrong, not approximate. Only the floating-point tail (attention, the head) admits reassociation.

Why exactness, concretely

Tolerance-based testing fails on this workload because the failure mode is silent and delayed. A wrong logit does not crash; it perturbs a probability, and the divergence appears many tokens later as a different word.

The runtime once folded two scale factors into a single constant and multiplied once, where the reference multiplies by each in turn. Floating-point multiplication is not associative, so $a(bc)$ and $(ab)c$ round differently. The observed consequence: cosine similarity 1.000000 for four token positions, 0.999624 by position seven, and at token sixteen the generated text read “known” where the reference read “the”. No tolerance that admits legitimate reassociation in the attention tail would have flagged the first four positions, and no human reading either the code or the output would have found it.

Gates

Results are recorded in a frozen gate document. Correctness gates are invariants: they may not move at all. Resource gates are targets and are expected to move, but only accompanied by a written argument naming the change and its before/after. Table 3 lists the measured values.

Defects this methodology found

Beyond the scale-folding bug, two are worth reporting because both were latent in code that a test matrix claimed to cover.

The NEON kernel accumulated into i16 and flushed every 64 beats, under a comment reasoning about 64 accumulation steps. Each beat issues four pairwise-add-accumulate instructions, so the interval was 256 steps against a safe bound of 128; the per-step bound was also understated, since two lanes of $-128$ pairwise-add to $-256$. The worst case, a 6912-column row of $-1$ weights against $-128$ activations, wrapped silently and produced a wrong row. It had never executed: the aarch64 CI job had failed at a linter since it was added and never reached an assertion, while the matrix advertised ARM coverage. A job that cannot reach its assertions is not coverage.

Separately, streaming detokenisation routed each token through a decode call that terminates in a lossy UTF-8 conversion. On this vocabulary U+1F600 splits across token 76460 (bytes f0 9f 98) and token 222 (byte 80); decoding each alone returns the replacement character, so every emoji, flag and multi-byte sequence emitted by the streaming API was corrupted while a one-shot decode of the same sequence was correct.

Evaluation

Method

All measurements are on an AMD Ryzen 9 8945HX (Zen 4, AVX-512 VNNI, 32 threads, DDR5), greedy decoding, 32 tokens, four-prompt suite, page cache pre-warmed, median of three invocations. Raw data is committed to the repository.

Three methodological points, each of which changed a published number:

Median, not best. Reporting our best run against a baseline’s median tilts every comparison. All runtimes are now reported as medians.

Pre-warm. The model file lives on a mount whose cold reads are slow. A cold measurement read 11.25 tokens/s where three consecutive warm ones read 13.5–13.9. An earlier version of Table 5 claimed Tritium was fastest single-threaded; that was an artifact of our model being warm in page cache while freshly downloaded baseline models were not.

A measured roofline. Achieved bandwidth is divided by a streaming-read probe run on the same machine in the same invocation, best of five passes. A single-pass probe under-reported by 10–15%, which inflated the reported fraction of roofline from 54% to 62%.

Absolute performance

For reference, the pre-format-change system decoded at 0.16 tokens/s with a peak resident set of 6.10 GiB.

Comparison

Table 5 is the central negative result. All three runtimes land within 11% of each other at one thread. Only Tritium fails to scale: 1.15$\times$ from one thread to eight, against 2.54$\times$ for bitnet.cpp and 1.78$\times$ for llama.cpp, ending 2.09$\times$ behind.

Part of the remaining difference is byte traffic. bitnet.cpp stores its embedding table in f16 where Tritium stores f32, moving approximately 1178 MB per token against 1834 MB.

Diagnosis: bytes are not time

An instrumented decode contradicts the byte analysis of Table 1. The head is 42% of wall time despite being 72% of the bytes, because the two halves run at opposite ends of the machine’s efficiency range: 48.8 GB/s for the head against 13.9 GB/s for the ternary path.

The head is already at roughly 92% of this machine’s measured ceiling, and no kernel work will move it: forcing the dense kernel to scalar, AVX2 and AVX-512 in turn gives 28.8, 28.1 and 27.1 ms respectively. The ternary path is the opposite. It is compute-bound in the kernel at approximately 15.8 GB/s per core, barely above what it achieves from DRAM, which means it would scale with cores if the runtime could use them.

Why it does not scale

A decode step issues 210 ternary matvecs whose largest is 4.4 MB. Each currently pays a work-stealing fork/join. Lowering the threshold at which matvecs parallelise makes matters dramatically worse, not better (Table 7, row rayon-global): even two threads is 1.8$\times$ slower than not parallelising at all.

Table 7 rules out the cheap explanations. A dedicated pool sized to the requested thread count helps substantially and is still worse than serial execution. A broadcast primitive helps more and is still worse. Only a persistent worker pool on an atomic spin barrier, the mechanism ggml uses, beats serial execution, at 2.9$\times$ on the layer path. End to end, in a back-to-back comparison, that prototype lifts scaling from 1.15$\times$ to 1.79$\times$ and decode by 1.59$\times$ at eight threads. It is a prototype: its lifetime handling is not sound and it is not merged.

Limitations

We state these because several are the first thing a reader should check.

• Throughput. 2.09$\times$ behind bitnet.cpp at eight threads on the identical checkpoint. The cause is diagnosed and a fix is prototyped but not shipped.

• No energy measurement. Joules per token is the metric this architecture exists to improve and we have never reported one, because no machine in our loop exposes a counter. The field is left empty rather than estimated.

• No silicon. The hardware core is simulation-first and not timing-closed. The single-cycle 64-term reduction and 64 parallel activation reads are fine under Verilator and unproven on a device. Activation memory is capped at 8192 columns.

• No ARM timing. The NEON kernels are verified bit-exact on aarch64 hardware in CI and have never been timed. No ARM performance figure exists.

• Scope. Batch size 1, no batched prefill, context capped at 2048 with an f32 preallocated KV cache.

Related work

bitnet.cpp [2] is the reference ternary runtime and the comparison point in Table 5. It is faster than Tritium on CPU today. It targets CPUs, and its i2_s tensor type is specific to that fork; mainline llama.cpp [3] cannot load it. Ternary tensor types exist in mainline (TQ1_0, TQ2_0) but its HuggingFace converter has no BitNet entry.

The distinction we draw is not throughput but target. These are fast CPU runtimes whose weight layouts serve a CPU. Tritium’s layout is a hardware interface that a CPU also happens to consume efficiently, which is what makes the path in Section 5 available without a repacking step.

Future work

In evidence order. Parallel scaling is first because it is worth more than everything else combined: a sound persistent thread pool, per Table 7. Second, storing the head in bfloat16. The values are already bfloat16, widened from the checkpoint during conversion and never subsequently refined, so narrowing them is lossless with respect to the source and worth approximately 1.26$\times$ with peak resident set falling from 1846 MB to roughly 1190 MB. Third, an energy measurement on a machine with a counter, which converts the central claim of this architecture from an argument into a number. Then ARM throughput, and board bring-up.

Conclusion

Ternary weights make the multiplier optional, but only if the weights are stored in a form a multiplier-free datapath can consume. Tritium’s contribution is a layout in which one 16-byte unit is both a SIMD inner-loop iteration and a hardware cycle, so a memory-mapped model file is the interface to both targets rather than an input to be repacked for either.

We do not claim throughput leadership; on CPU we are measurably behind, for a diagnosed and prototyped reason. We claim that the arithmetic is proven correct against an independent reference, that the hardware core is proven bit-exact against that same reference and free of multipliers by construction, and that this is the order in which a silicon architecture should be de-risked: the expensive irreversible step last, not first.