04Benchmarks

Every number, with its method attached.

Same host (AMD Ryzen 9 8945HX, Zen 4, AVX-512 VNNI), greedy decoding, 32 tokens, median of three invocations per baseline. Tritium decodes the four-prompt suite; the baselines generate from an empty context, because llama-bench takes no prompt file. Both are steady-state batch-1 decode.

bitnet.cpp runs the identical checkpoint, so that column is the honest comparison. llama.cpp runs Qwen2.5-3B Q4_K_M because mainline cannot load the i2_s type — a different model at a different quality point. This page should not be read as "ternary beats 4-bit".

Median of three, 32 tokens, greedy. Tritium leads to four threads, then declines where bitnet.cpp keeps climbing.

Raw rows


2026-08-25	bitnet.cpp	I2_S	1	—	13.61	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-25	bitnet.cpp	I2_S	2	—	19.10	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-25	bitnet.cpp	I2_S	4	—	26.72	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-25	bitnet.cpp	I2_S	8	—	32.49	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-25	bitnet.cpp	I2_S	16	—	31.21	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-25	llama.cpp	Q4_K_M	1	—	15.40	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-25	llama.cpp	Q4_K_M	2	—	22.06	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-25	llama.cpp	Q4_K_M	4	—	27.83	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-25	llama.cpp	Q4_K_M	8	—	26.18	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-25	llama.cpp	Q4_K_M	16	—	23.88	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-25	tritium	ternary-1.58	1	avx512vnni	19.40	345.9	22.85	47.4	1220	ok
2026-08-25	tritium	ternary-1.58	2	avx512vnni	26.73	243.1	31.48	54.0	1220	ok
2026-08-25	tritium	ternary-1.58	4	avx512vnni	27.90	238.8	32.85	57.4	1220	ok
2026-08-25	tritium	ternary-1.58	8	avx512vnni	25.93	252.3	30.53	58.0	1219	ok
2026-08-25	tritium	ternary-1.58	16	avx512vnni	20.72	309.4	24.40	49.0	1218	ok
2026-08-23	bitnet.cpp	I2_S	1	—	12.54	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-23	bitnet.cpp	I2_S	2	—	19.64	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-23	bitnet.cpp	I2_S	4	—	27.29	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-23	bitnet.cpp	I2_S	8	—	31.90	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-23	bitnet.cpp	I2_S	16	—	30.51	—	—	—	—	ok	1187 MB file; same checkpoint; f16 token_embd; median of 3
2026-08-23	llama.cpp	Q4_K_M	1	—	13.91	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-23	llama.cpp	Q4_K_M	2	—	20.01	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-23	llama.cpp	Q4_K_M	4	—	25.54	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-23	llama.cpp	Q4_K_M	8	—	24.72	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-23	llama.cpp	Q4_K_M	16	—	22.84	—	—	—	—	ok	2104 MB file; different model and quality point; median of 3
2026-08-23	tritium	ternary-1.58	1	avx512vnni	13.28	499.3	24.36	51.1	1846	ok
2026-08-23	tritium	ternary-1.58	2	avx512vnni	14.61	462.7	26.80	56.3	1846	ok
2026-08-23	tritium	ternary-1.58	4	avx512vnni	15.23	446.8	27.93	53.9	1846	ok
2026-08-23	tritium	ternary-1.58	8	avx512vnni	15.23	445.8	27.93	55.8	1845	ok
2026-08-23	tritium	ternary-1.58	16	avx512vnni	14.98	454.5	27.49	57.4	1846	ok
2026-08-22	bitnet.cpp	I2_S	4	—	—	—	—	—	—	unavailable	no bitnet-cli on PATH; set BITNET_CPP_BIN
2026-08-22	llama.cpp	Q4_K_M	4	—	—	—	—	—	—	unavailable	no llama-bench or llama-cli on PATH; set LLAMA_CPP_BIN
2026-08-22	tritium	ternary-1.58	4	avx512vnni	15.01	454.2	27.54	61.6	2334	ok

Unavailable rows are kept: a baseline that was not installed on that date is a fact about the run, not a gap in the table. Hover a truncated note for the full text.

Reproduce every number on this page

export LLAMA_CPP_BIN=/path/to/llama.cpp/build/bin/llama-bench
export LLAMA_GGUF=/path/to/qwen2.5-3b-instruct-q4_k_m.gguf
export BITNET_CPP_BIN=/path/to/bitnet.cpp/build/bin/llama-bench
export BITNET_GGUF=/path/to/ggml-model-i2_s.gguf
for T in 1 2 4 8 16; do
    benches/run.sh --model models/bitnet-2b4t.trit --tokens 32 --threads "$T"
done

The defaults differ from these parameters, so reproduce with exactly this invocation. Every performance claim ships with a script; a claim without one does not appear here.