Ternary inference / build in public
Language models that run on select-accumulate.
Tritium is a ternary (1.58-bit) inference stack whose weight format is consumed byte-for-byte by both a commodity CPU kernel and a multiplier-free hardware core. The model file that runs on a laptop today is the beat stream for silicon next.
No multiplier. Two bits per weight. 42.19% of weights are exactly zero — measured, tritc verify.
AMD Ryzen 9 8945HX, BitNet b1.58 2B4T, greedy. Reproducible with tritd bench; frozen reference: docs/06-GATES.md. Quality vs the HF reference: mean logit cosine 0.9991, top-1 100%.
The bottleneck is the multiplier.
4-bit and 8-bit quantization on edge companion computers — drones, robots, edge IPCs — bottlenecks on the memory bus and dissipates heat doing continuous floating-point matrix multiplication. BitNet-style 1.58-bit models put every weight in {-1, 0, +1}, which turns the inner loop from multiply-accumulate into select-accumulate.
No multiplier. Two bits per weight on the wire. Tritium is the runtime that cashes that check on hardware people already own, and keeps a synthesizable RTL path open for hardware they don't yet.
| Metric | Before the pivot | Now |
|---|---|---|
| Decode | 0.16 tok/s | 14.89 tok/s |
| Time to first token | ~33 s | 470 ms |
| Peak RSS | 6.10 GiB | 1.80 GiB |
| Achieved bandwidth | ~0.3 GB/s | 27.3 GB/s |
A memory-mapped .trit is the beat stream.
Ternary weights are stored as bit planes: each 16-byte beat carries 64 columns of one row as {u64 pos, u64 neg}. Both masks for a column block arrive together — which is what the SIMD kernel wants and what the RTL core consumes directly. One beat is exactly one hardware cycle's weight input; nothing between the page cache and the accumulator rewrites a byte.
Three implementations, one arithmetic.
The oracle, the vectorized runtime, and the hardware must agree exactly — integer accumulators, no tolerance. Lane order, thread count and the RTL's beat-serial accumulation all produce identical bits.
| Crate | What it is | Status |
|---|---|---|
| crates/tritc | Converter: HF BitNet checkpoint to packed .trit v1. Folds norms, quantizes, verifies. | working |
| crates/trit-core | Format, transformer, KV cache, RoPE, sampler, tokenizer traits. No unsafe outside one mmap. | working |
| crates/trit-cpu | Bit-sliced SIMD kernels: AVX-512 VNNI / AVX-512BW / AVX2 / NEON / portable scalar. | verified on x86 and aarch64 in CI |
| crates/tritd | Host daemon and C runtime: run, serve, bench, info, plus a C ABI. | working |
| crates/trit-rtl | Hardware-in-the-loop backend over the Verilated core. | working |
| crates/tritsim | Independent golden reference. The oracle every other path is diffed against. | working |
| rtl/ | Multiplier-free SystemVerilog tritcore + Verilator testbenches. | 64 lanes, simulation-first |
| benches/ | Comparative harness vs llama.cpp / bitnet.cpp baselines. | both baselines measured |
Only Tritium fails to scale.
All three runtimes are within 11% of each other at one thread. From one thread to eight, bitnet.cpp gains 2.54x and llama.cpp 1.78x; Tritium gains 1.15x and then stops. The kernels are not the problem — per core we are ordinary. The runtime's per-matvec fork/join overhead is the whole gap, and it is measured.
The other part of the gap is bytes: bitnet.cpp stores its embedding table as f16 where Tritium stores f32. Both are on the roadmap, in that order.
| threads | tritium | llama.cpp Q4_K_M | bitnet.cpp I2_S |
|---|---|---|---|
| 1 | 13.28 | 13.91 | 12.54 |
| 2 | 14.61 | 20.01 | 19.64 |
| 4 | 15.23 | 25.54 | 27.29 |
| 8 | 15.23 | 24.72 | 31.90 |
| 16 | 14.98 | 22.84 | 30.51 |
What it does not do.
- Batch size 1. No batched prefill, no continuous batching.
- Context capped at 2048; the KV cache is f32 and preallocated (315 MB).
- Performance is measured on x86-64 only. The NEON kernels match the reference exactly on real aarch64 CI hardware, but no ARM timing has been taken and no ARM performance number is claimed.
- No FPGA silicon yet. The RTL is simulation-first: fine under Verilator, not yet timing-closed on a board.
- Tritium does not scale across cores. Best single-thread number of the three runtimes measured, worst aggregate one. Closing this is the top item on the roadmap.
- Energy per token is reported only where a real counter exists. It is never estimated.
Misses get published rather than buried. This list shrinks by measurement, not by rewording.
The primitive was proved on an FPGA.
Tritium is the direct continuation of ternoise — ternary compute on FPGA, built in public. The denoiser proved the primitive; tritium scales the same primitive (ternary MAC to add/sub tree) to transformer inference, and now runs it on commodity silicon as well.