BENCHMARKS — External Marks Registry

Status legend

Registry

Per-benchmark detail

HumanEval — ✅ Run

• Source: OpenAI human-eval (164 problems, pass@1, sandboxed unit tests). Public.
• Two layers we measure: the raw model (eval_humaneval_ouro.py via in-process generate()) and the whole chat product (eval_humaneval_chat.py, which drives POST /api/dream/chat/stream exactly like the browser — provider routing, local-model adapter, loop-reasoner). One extractor + one sandbox shared; never fabricates a score (humaneval_runner.py returns measured:false if the package is missing).
• Provider-parametric: --provider ollama|anthropic|"" → local↔cloud parity on one execution-grounded mark.
• Run it: python scripts/eval_humaneval_chat.py --provider ollama --limit 10 (server + local model up), --full for all 164.
• Public SOTA (re-validated 2026-07-17, llm-stats): ⚠️ the live board has gone stale — the flagships (Claude Sonnet 4.5, DeepSeek R1) have largely stopped self-reporting HumanEval, so llm-stats now tops at OSS MiniCPM-SALA 95.1% > Kimi K294.5% > Claude 3.5 Sonnet 93.7% (amodel, at #3);models, field avg 0.812. Our previously-recorded 97.6/97.4 (Sonnet 4.5 / R1) no longer appear there and survive only in a pricepertoken cache (HTTP 403, unverifiable this run). Saturated + superseded — read our 0.427 as absolute 1.4B capability; the board itself now points to SWE-bench / LiveCodeBench to differentiate. (Re-checked 2026-08-06: still saturated — where flagships self-report, frontier is 96–98% (Sonnet 4.5 97.6 / R1 97.4, codesota / llm-stats); ~53-model field avg ~90. HumanEval stays a regression tripwire for us, not a differentiator.)
• Loop-value experiment (#2178, 2026-07-06 — CORRECTED at n=50, #2210, 2026-07-10): the original n=10 verdict "depth adds no value" (0.90→0.80→0.80 at depths 1/2/4) was one problem of noise. Re-run at n=50 exec-graded (rows ouro-depth{1,2,4}-n50-2210): pass@1 0.10 → 0.22 → 0.22 at depths 1/2/4. The honest verdict is "depth helps up to ~2 loops, then plateaus" — depth 1→2 doubles pass@1 (suggestive, p=0.10 at n=50), depthresearch/2026-07-06-loop-value-experiment-2178.md== depthresearch/2026-07-06-loop-value-experiment-2178.mdexactly (no gain pastresearch/2026-07-06-loop-value-experiment-2178.mdloops, matching the ~2-loop-optimal literature). Latency ~123 s/problem; high no-parse rate (27–35/50) depresses absolute pass@1 but is ~constant across depths. NOT the flat NEGATIVE the n=10 run implied, and NOT evidence that depth>2 helps. Full writeup: research/2026-07-10-depth-value-n50.md; original: research/2026-07-06-loop-value-experiment-2178.md.
• Own-data distill sweep (#2729, mookman, 2026-07-24 — raw-Ouro layer, full-164 greedy, ouro-g-*/ouro-base-humaneval rows): 8-config QLoRA sweep on the local 4080, tripwire off + held-out val best-checkpoint selection. Raw Ouro-1.4B base (no adapter) = 7.3% (12/164, no-parse 111). Pure OpenCodeInstruct distill (2000 rows) DEGRADES the base — every 2-epoch recipe (lr 2e-5→1e-4, rank 8/16) lands 3.7–5.5% with no-parse climbing to ~138 (format-following collapse);combined-ep2epochs recovers to ~parity (7.9%). Mixing in a 2600-row replay set (→ combined 4600) reverses it into a real gain: combined-ep1 (lr 5e-5, r16,ouro-base-humanevalepoch) = 11.0% (18/164), +3.7pp over base, no-parse drops to 125; combined-ep2 = 9.8%. So the coding-distill signal does lift the owned model, but only replay-balanced (pure distill collapses the distribution). ⚠️ Single-seed n=164 — the +6-problem margin is above single-problem noise but not yet a locked headline; a multi-seed confirm is running (the --seed arg landed in #2942). Distinct from the ouro-peak-full-ck600 0.427 peak, which used a different (ck600 Σ₀) adapter, not this distill adapter.
• Fresh re-read (2026-08-06) + a provenance footgun fixed (#3127, merged): the newest raw-Ouro row is a full-recurrence smoke this window (diag-fullrec, n=20, 2026-08-01) at 5% / 17-of-20 no-parse — but that is not a new capability number: it measured the coding-regressed Σ₀ sigma0-adapters/final tune, because eval_humaneval_ouro.py defaults --adapter to ouro-sigma0-adapters/final. A bare run therefore silently benchmarks the honesty/Σ₀ tune (which trades coding for calibration), not raw capability — pass --adapter explicitly (base/None for capability ≈ 7.3%, checkpoint-600 for the 0.427 reference). Compounding it, checkpoint_id used to collapse every adapter to <model>@final (sigma0-adapters/final and the distill <tag>/final both stamped @final), so the ledger couldn't tell which adapter produced a row — #3127 fixed the collapse: rows now stamp distinctly (@ouro-sigma0-adapters-final, @lr5e5-r16-final, …), and unique leaf names like checkpoint-600 pass through unchanged. No headline moved — the real marks are unchanged: chat product 0.829 (Qwen2.5-Coder, full-164) and raw-Ouro reference 0.427 (ck600, full-164). The @final-labelled ledger rows (ouro-depth*-n50-2210, diag-fullrec) are all this regressed tune, not the model's capability.

HumanEval (chat product) — layer note

• The full-chat layer cell is now filled: qwen25coder-served-he164-fix2194 (2026-07-07) drove allproblems through POST /api/dream/chat/stream and scored 0.829 — routing + serving deliver the engine's capability (the earlier 0.72 was the #2194 extraction bug, not a product gap). Re-run this row whenever the serving path changes; the CI gate enforces it for serving-file diffs.
• Served-checkpoint delta (mookman, 2026-07-08): full-164 chat scored 0.701 on lantern-sigma0-coder vs 0.829 on qwen2.5-coder:latest (rows humaneval-mookman-20260708-full, qwen25coder-served-he164-fix2194) — the served checkpoint choice is worth ~13 pts; confirm which coder we actually ship before quoting a single chat-product number.

MBPP — ✅ Run

• Source: Mostly-Basic-Python-Problems (Google). We run an exec-and-assert subset (data/eval/mbpp-basic.jsonl).
• Run it: python scripts/eval_coding.py --label ouro-fast --model lantern-sigma0-coder.
• Public SOTA (re-validated 2026-07-17): both reproduce exactly — o4-mini 94.9% (codesota); top open-weight Sarvam-30B 92.7% (llm-stats/mbpp,models, field avg 0.70). (Note: the mbpp-pass@1 slug now renders only/mbppmodel — use /mbpp.) Like HumanEval, saturated — our subset accuracy is an absolute read, not a ranked gap.

SWE-bench Lite — ✅ Run (first graded number) — now a MILESTONE GATE, not a progress dial

• Role (repositioned 2026-07-08): single-shot SWE-bench is a floor-pinned metric for our stack — the ICLR-2024 paper (arXiv:2310.06770, Table 5) measures Lite+BM25 single-shot at ClaudeOpus 4.33%, Claude3.00%, GPT-4-turbo 2.67%, and a task-fine-tuned 7B (SWE-Llama) 1.33%. At those rates 0/n is the modal outcome for any affordable n, so repeated single-shot runs carry no signal — do not re-run them. Run this mark only against agentic-loop prediction sets (swe_agent_loop.py propose→apply→test→retry), where published scaffolds have real dynamic range; the event that matters is binary: the first nonzero resolved%, which certifies the Act/Verify loop works end-to-end on foreign repos. Week-to-week model progress belongs on marks with gradient at our rung (HumanEval/MBPP).
• Source: princeton-nlp/SWE-bench_Lite (plain) and princeton-nlp/SWE-bench_Lite_bm25_13K (BM25-retrieved text prompt). Grading is execution-graded and delegated to the official swebench harness (Docker or Modal/WSL) — we never report a resolved% we didn't measure.
• First graded run (2026-07-08, #2246): the 2026-07-07 alex-lane --direct predictions (Qwen2.5-Coder single-shot on the BM25-13K prompt, 5/5 well-formed patches) graded by swebench.harness.run_evaluation on-box: resolved 0/5 (0%) —data/eval/leaderboard.jsonlpatches applied cleanly but failed the FAIL_TO_PASS tests,swebench.harness.run_evaluationfailed to apply (astropy__astropy-14995, hallucinated hunk context). Report: data/eval/swebench/swebench-alex-predict-direct.keystone-1783465949.json; ledger row in data/eval/leaderboard.jsonl (run_id keystone-1783465949). Well-formed diffs are not correct fixes — patch-shape metrics overstate capability; only execution grading counts.
• Grading is unblocked on the dev box: WSL2 Ubuntu ships docker 29.1.3 + swebench 4.1.0. Recipe: wsl -d Ubuntu --cd /mnt/c/dev/lantern-os -- python3 -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path <preds.jsonl> --run_id <id> --max_workers 1 (~17 min /instances first-build; keep max_workers=1 — WSL is RAM-capped at 3GB).
• Single-shot vs agentic: single-shot blind patches scoreswe_agent_loop.py(now measured, above; earlier informal pilot said the same). The agentic loop (swe_agent_loop.py: propose → apply → run repo tests → feed failing test back → retry) closes that seam; swe_agentic_run.py runs it live (ollama propose on host, grade one instance in WSL).
• Run it: python scripts/eval_swebench_chat.py --provider ollama --limit--dataset princeton-nlp/SWE-bench_Lite_bm25_13K --grade (needs Docker). Predict-only without --grade, grade later.
• Next: grade an agentic-loop prediction set — the only configuration worth grading again (single-shot 0% is the measured baseline; the paper's Tablecontext-length ablation shows resolve rates fall as context grows — 13k→27k→50k: Claude1.96→1.87→1.22, SWE-Llama 7b 0.70→0.31→0.00 — so a bigger-context single-shot re-run is not a lever either).
• Public SOTA (re-validated 2026-07-17, llm-stats / benchlm / swebench.com): leaders unchanged — Claude Mythos95.5%, Claude Fable95.0%, Claude Opus 4.8 88.6% resolved (Verified; now 104 models tracked, field avg ~70%); best non-Anthropic Gemini 3.1 Pro / DeepSeek-V4 ~80.6%, SWE-bench Lite model-centric leader Claude Opus 4.6 62.7%. **The unsaturated, differentiating coding mark. Our first local number (Lite, n=5 single-shot) is 0%** — the expected floor (the ICLR paper's fine-tuned 7B single-shot BM25 cleared <1%); next lever is grading the agentic loop, not prompts.

LongMemEval — ✅ Run

• Source: xiaowu0162/LongMemEval (longmemeval_s / _m / _oracle). Measures whether the gold evidence turn is retrieved in top-k. We benchmark the canonical Python MemoryEngine (src/csf/memory_engine.py) in both keyword and multi-signal modes.
• Why it exists: MemOS publishes LongMemEval/PersonaMem numbers (+40% over OpenAI memory) and we had none — so we couldn't honestly claim our retrieval is good.
• Run it: LONGMEMEVAL_PATH=/path/to/longmemeval_s.json python experiments/longmemeval_harness.py --k--limit 200. Offline self-test (no env var) runs on a baked-in synthetic fixture.
• First real run (2026-07-01, longmemeval_s,scored, k=5): multi-signal recall@5 0.709 / MRR 0.486; keyword recall@5 0.222 / MRR 0.098 (+0.487 recall for multi-signal). Ledger: data/longmemeval/runs.jsonl.
• Next: raise multi-signal recall@5 toward MemOS-class numbers.
• Product-path row (#2111): scripts/eval_longmemeval_js.js scores the live JS chat retriever (csf-memory.js::searchConversation, keyword, windows the lastturns) — the retriever a real chat turn actually uses, not the Python engine. Measured on longmemeval_s (n=500, k=5): session-level recall@5 0.884 / MRR 0.80, 0/500 gold outside the window. ⚠️ Not the same metric as the Python row above — this is session-level (was any gold-session turn returned in top-k) on the product path, vs the Python engine's turn-level recall; the two are complementary, not comparable. Run: node scripts/eval_longmemeval_js.js --k 5 (add --semantic to also score the nomic-embed rerank when the embeddings service is up).
• Public SOTA (re-validated 2026-07-17, mem0 / OMEGA / ByteRover / MemPalace): top production memory systems report end-to-end QA accuracy ~92–97% on LongMemEval-S — MemPalace 96.6% (raw; its 100%-hybrid is a disputed reranking artifact), OMEGA 95.4% (task-averaged), ByteRover 92.2–92.8%, plus fresh entrants Hindsight 91.4% / HonCho 90.4%. Podium unchanged since Jul 7; the moving part is credibility — every figure is vendor-self-reported under a different protocol, and Mem0's 93%+ self-report collapses to 31.8% under a standardized protocol. ⚠️ Correction: the "OpenAI Memory 52.9%" previously filed here is a LoCoMo number, not LongMemEval-S — OpenAI doesn't run this benchmark; removed. Not comparable to our 0.774 recall@5 — theirs is answer accuracy on the full 500-item set; ours is retrieval recall@5. To rank, run the full retrieve→answer→LLM-judge pipeline. LongMemEval-V2 (arXiv:2605.12493, ~115M-token contexts,Qs) is far harder — the paper's own best method scores ~58–72% vs ~95% on V1 — track as the successor.

HaluEval-QA — 🟡 Partial

• Source: RUCAIBox/HaluEval (qa_data.json) — factual QA where each item ships the gold knowledge passage + right_answer + a known hallucinated_answer. Subset in data/eval/halueval-qa-subset.jsonl (n=40).
• Why it exists: the ADR-0017 surprise-gated-decoding accept gate — "the ADR is not Accepted until these numbers exist" (#1941). It measures whether injecting grounding when a model is uncertain reduces hallucination.
• A/B (grounding is the intervention's mechanism): answer each question (A) with NO grounding vs (B) with the gold knowledge injected. Deterministic grading (normalized containment of the gold answer — no LLM judge). Run it: python experiments/halueval_ab.py (needs egress + OPENAI_API_KEY).
• First real run (GPT-4o-mini, n=40): hallucination 55% → 20% with grounding, 64% relative reduction — clears the ≥20% gate. data/eval/halueval_ab_results.json.
• Honest scope: this is the mechanism with gold retrieval (an upper bound). The production controller (apps/lantern-garage/lib/surprise-intervene.js) must also detect when to ground (surprise gate) and retrieve good evidence (CSF/web/tool arms), so its real-world reduction will be lower. Next: wire the live controller (SURPRISE_CANARY=1 vs SURPRISE_INTERVENE=1) end-to-end on this set and record the full-controller row.
• Local closed-book rows (2026-07-05, data/eval/halueval-local/): Ouro-1.4B base accuracy 0.05 (hallucination 0.95); + honesty-balanced adapter 0.40 (hallucination 0.60), n=40, deterministic contains-gold. This is the local lane's closed-book floor, not the grounded product.
• Gate face-off (2026-07-05, data/eval/halueval_gates_compare_results.json): by detection AUROC, council_delta 0.909 > logprob 0.861 > self-consistency 0.851 — but logprob keeps the best routing edge (0.0585 vs council's 0.0457): rank ≠ route. Ouro hidden-state canaries (ouro_canary_vs_logprob_results.json): unsupervised AUROC ≤ 0.661 with negative routing edge — the current canaries detect weakly and don't route at all.
• Public detection SOTA (re-validated 2026-07-17): supervised hidden-state probing reaches 98.4–98.6% AUROC on HaluEval with 7B open models (MultiHaluDet, ACL 2026, arXiv:2605.24919), now nudged to ~0.998 by a plain quantized linear probe (arXiv:2606.02628) — ceiling saturation. **⚠️ But that number is largely a teacher-forced artifact: PARALLAX (arXiv:2605.17028) shows a text-similarity baseline with no model internals hits 0.98 on HaluEval, and on live-generated (non-teacher-forced) responses a top probe collapses 0.96 → 0.62 — right where our unsupervised Ouro canary (≤0.661) already sits; on genuinely hard benches (HaluBench, RAGTruth) supervised probes reach only ~0.91. So the "98.6 vs 0.66" gap is mostly benchmark construction, not capability. A clean held-out probe on Ouro hidden states is still the catch-up run, but the honest target on realistic text is ~0.9 (hard-bench), not 0.986**.

SimpleQA-Verified — 📋 Planned

• Source: Google DeepMind, arXiv:2509.07968 (Sept 2025) — a 1,000-prompt cleaned/reconciled subset of OpenAI's SimpleQA (de-duplicated, topic-balanced, label-corrected). Dataset + grader on Kaggle. Public.
• What it measures: short-form parametric factuality on long-tail facts, scored on two axes — accuracy and attempt-rate — combined as F1, so a model is rewarded for hedging ("I don't know") over confabulating. This is the same honesty axis as our own golden mark's confabulation-rate, on an externally-owned dataset.
• ⚠️ Live board moved (re-validated 2026-07-17): the Kaggle/DeepMind leaderboard now tops at Gemini 3.1 Pro Preview 77.5 > GeminiPro 72.1 > Gemini 3.5 Flash 70.4 (all three clear the paper's old 55.6); top open-weight Qwen235B 53.7, o3 52.3, Claude Opus 4.8 44.5. Gemini 2.5 Pro and GPT-5 have dropped out of the top 10. So the "~55 is the ceiling" read below is stale — the ceiling is now ~77. The paper's original Tableis kept for reference:
• Paper's original Table(Google DeepMind, Sept 2025):

• Relation to our golden mark (why they're complementary, NOT comparable): our 159-fact golden set is our own mark (not in this registry) and tests the honesty axis on famous facts + famous negatives — easy knowledge, hard honesty. SimpleQA-Verified tests it on obscure long-tail facts — hard knowledge. A 0.95 on our set does not map to the frontier ceiling here (which moved from ~0.55 in the paper to ~0.77 F1 by Jul 2026); the difficulty distributions are opposite. What transfers is the metric philosophy: DeepMind's separate Attempted/Hedged/F1 columns independently validate the split behind our confab-rate-vs-golden-score — Claude Opusis the clearest case (19.2% raw accuracy but 54.1% accuracy-given-attempted, because it hedges 64.5% instead of guessing; raw accuracy conflates honesty with knowledge, F1 separates them).
• To run it (next): pull the Kaggle dataset, add a --simpleqa-verified path to experiments/sigma0_live_bench.py with the official F1/attempted/hedged grader (needs an LLM judge for answer-matching), record our model's row here.

Planned reference targets — 📋

These are public marks tracked in data/benchmarks/ as capability targets; no harness yet. Promote a row to ✅/🟡 the moment a harness produces a measured result.

• SWE-bench Verified — extend eval_swebench_chat.py --dataset princeton-nlp/SWE-bench_Verified.
• PersonaMem — natural companion to LongMemEval; same MemoryEngine harness shape.
• ARC-AGI · Humanity's Last Exam · SuperARC · OSWorld — reasoning/agency frontier marks; see data/benchmarks/agi-capability-matrix.json.

How to maintain this list

This file is a living registry — keep it honest, not aspirational. When anything below changes, edit the table in the same PR:

1. New external benchmark gets a harness → add a row (status 🟡 or ✅), link the harness, name the loop stage it strengthens (Observe/Remember/Reason/Act/Verify/Converge — the feature gate).
2. A planned mark produces its first measured row → flip 📋 → 🟡/✅ and paste the evidence (ledger + number).
3. A new measured run lands → it goes to the ledger (leaderboard.jsonl / runs.jsonl), and you refresh the "Latest evidence" cell. Don't put run history here — the ledgers are append-only; this file holds only the current headline + a pointer.
4. Never write a number you didn't measure. A row with no evidence stays 📋. The CI gate (eval-leaderboard-gate.yml) already enforces this for the serving path: no serving change ships without a fresh leaderboard row.

Rule of thumb: if an outside party defines the dataset and the grading, it belongs here. If we define it, it doesn't.