Regression gates

The frozen reference for the hardware-agnostic pivot. Every number here was measured on the pre-pivot tree at commit c3351d1 (v0 .trit, scalar kernel, single-threaded) before any pivot code landed.

Rule: nothing in the pivot may change a number in this file without an explicit, argued entry in the "Sanctioned changes" section below.

Environment of record

Other hosts will differ on timing and RSS. The correctness gates are host-independent and must hold everywhere.

G1 — Unit and integration tests

cargo test --workspace

All pass. trit-core 12, tritsim lib 19, tiny_model.rs 6.

G2 — Real-checkpoint logit agreement (the headline gate)

tritsim compare --model models/bitnet-2b4t.trit --dump logits.json

6 positions: mean cosine 0.9991, top1 match 100.0%

Last position, top-3 both sides:

ours   [(12366, 18.34601), (539, 14.126393), (264, 13.815544)]
theirs [(12366, 18.5),     (539, 14.25),     (264, 13.75)]

Single-position variant (--dump bos_dump.json): 1 positions: mean cosine 0.9977, top1 match 100.0% — identical under TRITSIM_INT_MLP=1.

CLI acceptance thresholds (crates/tritsim/src/main.rs): cosine >= 0.98, top-1 >= 0.90. The measured values sit far above them; the gate is the measured value, not the threshold.

G3 — Greedy text, byte-exact

tritsim run --model models/bitnet-2b4t.trit \
            --tokenizer models/bitnet-2b4t/tokenizer.json \
            --prompt "The capital of France is" --steps 16

 Paris. Paris is a city in the north of France, and it is the

Must reproduce byte-for-byte in Reference, Folded and IntMlp modes, and on every backend (cpu, rtl) and every CPU kernel.

G4 — RTL bit-exactness

make -C rtl lint          # clean
make -C rtl test          # all sets, exact i32 equality

SET vectors/exact_64x64:      64 rows OK
SET vectors/padded_5x100:      5 rows OK
SET vectors/extremes_4x128:    4 rows OK
SET vectors/zeros_3x64:        3 rows OK
SET vectors/wide_2x6912:       2 rows OK
SET vectors/model_k_proj_l0:   8 rows OK
SET <invalid-code>:            err raised OK

extremes_4x128 feeds x = -128 deliberately. Any CPU kernel that negates in the i8 domain fails here. This set must never be weakened.

G5 — Multiplier-free synthesis

make -C rtl synth

Yosys asserts zero $mul and zero $macc cells. Baseline cell count 33,659 (MAX_COLS chparam'd to 512; $_DFFE_PP_ 4096 is the flattened activation memory). Generic synthesis, no device target.

G6 — Hardware-in-the-loop parity

cargo test -p tritsim --features rtl --release

random_shapes_match_cpu_exactly compares the Verilated core against the CPU kernel on 8 shapes (1x64, 3x64, 2x100, 5x129, 7x640, 2x2560, 2x6912, 16x61) and requires exact equality. All pass.

G7 — Resource baseline (the numbers the pivot exists to move)

These are targets to improve, not invariants. G1-G6 are invariants.

G8 — trit-core against the tritsim oracle

Two independent implementations of the same model must agree.

cargo test -p tritsim --release --test cross_implementation              # tiny fixtures
cargo test -p tritsim --release --test cross_implementation -- --ignored # real checkpoint

On the real checkpoint, 8 positions, all three numerics rungs: cosine 1.000000 and identical top-1 at every position.

This gate found a real bug. The production path had precomputed w_scale * x_scale into a single constant and multiplied once, where the reference multiplies twice: acc as f32 * w_scale * x_scale. f32 multiplication is not associative, so acc * (w * x) and (acc * w) * x round differently. The drift was invisible at short contexts (cosine 1.000000 for four positions), grew with position to 0.999624 by position 7, and changed the 16th generated token from "the" to "known". Ordering the multiplications to match the reference restored cosine 1.000000 everywhere.

Post-pivot results

The pivot's purpose was to move the G7 numbers without moving G1-G6. Measured on the environment of record with models/bitnet-2b4t.trit (v1):

Kernel throughput on a 2560x6912 tensor, planes resident in L3:

Threading is measured, not assumed. A decode step issues 210 ternary matvecs whose largest is 4.4 MB, and splitting each across the machine costs more in fork/join than it saves: 14.13 tok/s at one thread against 2.40 tok/s at 32 before PARALLEL_MIN_BYTES was introduced. It now peaks at 4 threads (14.89) and stays flat to 32 (14.34).

G1-G6 are unchanged throughout. G2 still reports 0.9991/100%, G3 is still byte-identical in every numerics mode on both implementations, and the real checkpoint still decodes byte-identically through the Verilated RTL core -- 24.4 s/token in simulation, via tritd --backend rtl.

Sanctioned changes

Numbers here move only with an entry below, naming the phase, the reason, and the before/after.

The one change already anticipated: RoPE currently computes theta.powf(-2.0 * i / head_dim) in f32 (crates/tritsim/src/math.rs), giving angle errors up to ~1e-3 rad at position 2048. Moving both implementations onto a shared f64 inv_freq/rope_angle is expected to shift G2 slightly — in the direction of the reference, since the reference is more accurate. That gets its own isolated phase and its own row above.