ADR-0026: Ternary (1.58-bit) as the distillation target format for the ≤8GB serving artifact

Reconciliation (read first)

This ADR adds no new subsystem. ADR-0024 already commits the program to "train frontier → distill to the ≤8GB local serving artifact" (Phase 2) and leaves D7 (distillation retention threshold) open. This ADR names the format and method of that distillation target — ternary/1.58-bit (W1.58A8) — and the gate that accepts it. It does not start a training run, does not change the frontier objective (ADR-0024), and does not swap the serving loop (ADR-0021). It is the concrete answer to "distilled into what, verified how."

Context

The distill target is bounded by the 8 GB serving box (RTX 3070). MEASURED constraint ([[bigger-model-does-not-fit-8gb]]): a 14B-Q4 offloads ~34% to CPU and runs ~9× slower — Q4 caps the box at ~7B. Ternary is ~1.6 bits/weight vs Q4's ~4, so the same VRAM holds ~2–2.5× more parameters. The strategic point is not "shrink Ouro-1.4B" — it is: ternary lets theGB box serve a model 2–2.5× larger than Q4 allows (a 14B that didn't fit at Q4 plausibly fits ternary). That directly serves ADR-0024's "distill a bigger frontier base down to the local artifact."

I researched the four caveats I had flagged as open (via the !research arXiv corpus + web, 2026-07-07). All four resolve into concrete decisions below; the two I got wrong last cycle are corrected loudly:

• CORRECTION— "ternary needs QAT-from-scratch / can't cheaply distill FP→ternary." Refined.

BitDistill (arXiv:2510.13998, Microsoft) fine-tunes an off-the-shelf FP model (Qwen) to 1.58-bit with FP-comparable accuracy, 10× memory / 2.65× faster CPU, no from-scratch pretrain. Naive QAT-to-ternary is unstable and scales badly (its Fig 1: FP-gap grows 13.9→15.3 from 0.6B→4B) — my instability worry was right, but a distillation recipe fixes it.

• CORRECTION— "ternary compute win is CPU-only." Wrong. Microsoft BitNet now ships a

W1.58A8 CUDA kernel (4 ternary packed →int8, dp4a); spbitnet provides 1.58-bit + 2:4 sparse CUDA kernels for consumer Ampere (RTX 3060/3070-class). **Both serving targets (L4,

1. have GPU ternary kernels.** The compute win is reachable, not hypothetical.

Decision

Adopt ternary W1.58A8 as the canonical format of the ≤8GB serving artifact, produced by a BitDistill-style QAT-distillation from the FP frontier/teacher, accepted by the existing Σ_θ gate, and served as a layer-level kernel swap inside the custom Ouro loop. The pipeline:

S0 · Fork = QAT-distill (default). Three options ranked: (a) BitDistill QAT-distill — default, recovers FP accuracy; (b) PTQ ternary (CAT-Q arXiv:2606.26650 / TWLA arXiv:2606.13054) — cheap fallback if QAT compute is unavailable, at quality risk; (c) native-ternary pretrain — only if the frontier is trained ternary from token one (defer; owned by ADR-0024 Phase 3, not here).

S1 · FP teacher unchanged. The ADR-0024 GRPO/L4 run stays FP16; keep master weights. Ternary touches nothing on the train side — the teacher must be full-precision to distill from.

S2 · SubLN refit. Insert BitNet's SubLN sub-layer norms into the student before quantizing (required for low-bit optimization stability, BitDistill §2). Student init'd from FP weights; the norms train fresh.

S3 · Continual-pretrain warm-up. A short QAT-mode continued-pretraining pass on general corpus before task distill — BitDistill's crucial step; skipping it is why naive QAT scales poorly. Weighted heavier here than in the task-specific paper, because our artifact must stay general (not a single downstream task).

S4 · Attention distillation. MiniLM-style multi-head attention-relation distillation, ternary student ← FP teacher, on domain data. Quantizer = per-tensor absmean → {−1,0,+1} weights, 8-bit activations (per-token absmax/absmean), i.e. W1.58A8 (BitDistill Eq 1–3).

S4a · Activation-outlier mitigation (worked caveat). W1.58A8 assumes activations survive INT8; heavy-tailed layers break it. Decision: apply Hadamard/rotation smoothing (ITQ3_S arXiv:2603.27914) as the default pre-smoothing, with OffQ offsetting (arXiv:2606.07116) or Bit-by-Bit outlier-channel-splitting (arXiv:2604.07888) as escalations for any layer that still fails the per-layer range check measured during S3. TWLA (arXiv:2606.13054) is the PTQ-path fix if S0 falls back to (b).

S5 · Accept via the existing Σθ gate (worked caveat = measurement). The ternary student is a weight update → run it through the Σθ Model-Update Acceptance Gate (ADR-0025) against the FP teacher on fresh holdout. Accept iff degradation is within the D7 budget on the marks already in docs/BENCHMARKS.md: HumanEval / MBPP pass@1 as the capability floor, perplexity as the smoke signal, and the honesty marks (HaluEval / AbstentionBench) so ternarization can't silently trade away ADR-0024's defining property. No new gate; no vendor number.

S6 · Serve = kernel swap inside the custom loop (worked caveat = hardest constraint). ADR-0021 keeps Ouro's weight-tied recurrent transformers loop (Q-exit + output_hidden_states for the ADR-0017 probe). Therefore ternary ships as a layer-level swap — replace nn.Linear in the recurrent block with ternary-kernel-backed ops (BitNet W1.58A8 CUDA / spbitnet on Ampere) inside ouro_serve.py, preserving the loop, Q-exit, and mid-layer hidden-state hooks. We do NOT adopt bitnet.cpp's runtime (llama.cpp-derived: fixed-depth, no Q-exit, no hidden-state API) — that would violate ADR-0021. The kernel is a matmul, not a serving engine; the swap is feasible precisely because it is a torch-op replacement, not a runtime port.

Program invariants (inherited): evidence-classed claims; GPU-hour anchors, no invented prices; Σ_θ kill-gate on the artifact; honesty bound to external marks the model can't control; one loop, no sprawl; operator authority over the gate.

Consequences

• Positive: unlocks a 2–2.5× larger served model on theGB box than Q4 (the real payoff);

reuses the entire train (ADR-0024), gate (ADR-0025), and serving (ADR-0021) stack — the only new code is the S2–S4 distill recipe + the S6 layer swap; GPU kernels exist for both targets; ternarization is gated by the same honest holdout as any weight update.

• Negative / risks: kernel↔custom-loop integration is the real work — BitNet/spbitnet

kernels are validated inside stock runtimes, not inside a weight-tied recurrent loop with output_hidden_states; the swap must be proven to keep Q-exit + probe hooks intact (S6 is the load-bearing risk, not the math). QAT-distill costs L4 hours (S3+S4). Ternary still carries a residual quality gap vs FP (HGF arXiv:2602.05269 reports the naive 20–25% perplexity hit the recipe must close); S5 is the honest stop if it doesn't. spbitnet's 2:4-sparse path adds a second quantization axis (Sparse-BitNet arXiv:2603.05168) — opt-in, not baseline.

Alternatives considered

• Stay on Q4/Q4_K_M (status quo). Rejected as the target format: Q4 caps theGB box at ~7B

(MEASURED); it forecloses the "serve a bigger distilled frontier" payoff that motivates ADR-0024 Phase 2. Remains the safe fallback if S5 never passes.

• PTQ-only ternary (CAT-Q/TWLA), skip QAT-distill. Rejected as default (quality risk on a

general artifact), retained as the S0 fallback when QAT compute is unavailable.

• Native ternary pretraining from token one. Deferred — belongs to ADR-0024 Phase(train the

frontier ternary), not to the distill-target decision; only worth it once the frontier is committed to ternary end-to-end.

• Binary (1-bit). Rejected — losing the 0 state removes feature-selection/sparsity and costs

far more accuracy than 1.58-bit recovers (BWTA arXiv:2604.03957; the extra state is the point).

• Port serving to bitnet.cpp for a turnkey ternary runtime. Rejected — fixed-depth, no Q-exit,

no mid-layer hidden states; violates ADR-0021 and ADR-0017. Kernel swap, not runtime port.

Evidence

Amendment A1 (2026-07-27) — Accepted

Status: Accepted (Alex Place, in-session, 2026-07-27 — "approve the adr", closing #3002). Drafted by an agent per ADR-0001 and approved by the operator. The amendment is additive: it does not alter the format choice, the recipe, or the accept gate.

The gap

This ADR is scoped entirely to the 8 GB serving box, and that scoping is correct — but the organisation is now targeting thousands of users inside two years, and anGB box cannot serve thousands of concurrent users. As written, the ternary artifact has exactly one deployment shape, and it is the one that does not scale.

Proposed addition — two deployment shapes, one artifact

Same weights, same endpoint, same accept gate — this is a packaging and hosting decision, not a second model. It rides the tier split already proposed in ADR-0018, which is the right home for the routing question ("which shape serves this request?").

One addition to the accept gate

The Σ_θ gate currently accepts on quality. For the product case that motivates owning a model at all, quality is not sufficient: the thing traders need and cannot buy is a calibrated confidence — a stated "how sure am I" that tracks being right rather than tracking writing style. This repo has already measured that failure once (the gloss trap: [[v1-10-white-box-honesty-design]], where a probe scored AUROC 1.0 on glossed text and ≈chance de-glossed).

Proposed: the accept gate must include a calibration check, not only a quality check — a ternary artifact that keeps its benchmark score but loses its calibration has lost the specific capability the local shape exists to deliver, and must not pass.