CSF Format Specification (canonical, consolidated)

1. Version lineage

Use the canonical core for everything. import csf; csf.pack(...) / csf.compress(...). The removed symbolic "compressors" were lossy and had no decoder — never a real format. Legacy on-disk archives open read-only through csf.legacy.

2. CSF-Pack (v0.8) — arbitrary-file container

2.1 Binary layout

[Magic        4 bytes : b"CSF\x00"]

[Version      2 bytes : major, minor = 0, 8]

[Flags        2 bytes : bit0 = blobs zlib-compressed]

[ManifestLen  4 bytes : uint32 BE]

[Manifest     N bytes : UTF-8 JSON]

[Blob region  M bytes : concatenated (optionally compressed) file bytes]

[Footer      40 bytes : sha256(all preceding bytes) (32) + total size uint64 BE (8)]

2.2 Manifest JSON

{

  "format": "csf-pack", "version": "0.8", "created_at": 1750000000.0,

  "compressed": true, "file_count": 3,

  "files": [

    {"path": "src/a.txt", "size": 1050, "csize": 60,

     "sha256": "…", "offset": 0, "compressed": true,

     "description": "one-line gloss of what this file is",

     "metadata": {"loop_stage": "Remember", "verdict": "grounded"}}

  ]

}

• path — POSIX-relative arc path (directory structure preserved on unpack).
• size/csize — original / stored byte length; offset is relative to the blob region.
• sha256 — digest of the original bytes; verified on unpack.
• description (optional) — human-readable summary of the member (Σ₀ gloss).
• metadata (optional) — JSON-serialisable dict of grounding (purpose, loop

stage, verdict, confidence, source). Both are omitted when absent, so annotating is fully backward compatible — un-annotated archives are byte-identical to before the fields existed, and older readers ignore the extra keys.

Attach them at pack time with annotations={arc_path: {"description": ..., "metadata": ...}} (a bare string value is shorthand for description-only), and read them back with csf.list_archive, csf.file_annotation(archive, path), or csf.annotations(archive) (the searchable grounding index of every annotated member). Existing archives can be regenerated with descriptions via scripts/annotate_csf_archive.py.

2.3 Integrity & safety

• Footer digest (sha256 of everything before the footer) is verified before

the manifest is parsed — any tampering fails with a clean integrity error.

• Per-file sha256 is verified on extraction.
• Path traversal (.., absolute paths) is rejected on unpack (_safe_join).

2.4 API

import csf



# archive (files or in-memory blobs); per-file SHA-256 + footer integrity

csf.pack(["mydir", "file.bin"], "out.csf")          # default codec = zstd-19+LDM

csf.pack(["mydir"], "out.csf", codec="zstd", use_dict=True)  # shared dict, keeps random access

csf.list_archive("out.csf")                          # manifest (no extract)

csf.unpack("out.csf", "dest_dir")                    # -> [written paths]

data = csf.read_file("out.csf", "file.bin")          # verified single member



# per-file grounding (Σ₀): description + metadata, retrievable without extract

csf.pack(["mydir"], "out.csf",

         annotations={"mydir/a.py": {"description": "…", "metadata": {"loop_stage": "Act"}}})

csf.file_annotation("out.csf", "mydir/a.py")         # {"description": …, "metadata": …}

csf.annotations("out.csf")                            # {arc_path: {description, metadata}} index



# lightweight single-blob stream (1-byte codec header, no manifest)

blob = csf.compress(b"...")                           # -> bytes

raw  = csf.decompress(blob)

Codec is per-file and self-describing (zstd | zlib | store | omni); a missing codec reads as zlib, so pre-codec archives still extract byte-for-byte. The opt-in omni codec is the best-fit / max-ratio tier (see §2.7.1).

2.5 CLI

python -m csf.csf_pack pack <paths...> -o out.csf [--no-compress]

python -m csf.csf_pack unpack out.csf -d <dest_dir>

python -m csf.csf_pack list out.csf

Tests: tests/test_csf_pack.py (round-trip ×2, list, tamper-detection, traversal).

2.6 App routes

Wired into the server alongside the legacy tesseract pack (routes/csf.js):

POST /api/csf/pack    { paths: ["docs","README.md"], out: "data/exports/bundle.csf", compress?: true }

POST /api/csf/unpack  { archive: "data/exports/bundle.csf", dest: "data/exports/out" }

Both constrain paths to within repoRoot (no traversal) on top of the module's own guards.

2.7 Benchmark — does it work better?

python scripts/csf_pack_benchmark.py onreal repo files (2.8 MB):

Verdict (honest): CSF-Pack is size-competitive with zip (within ~1%) and strictly safer — cryptographic per-file + whole-archive integrity and path-traversal protection that zip/tar don't provide by default. tar.gz compresses better (solid stream) but offers no per-file integrity. Against the legacy symbolic CSF it's a categorical upgrade — that format can't hold arbitrary file bytes at all. Use CSF-Pack when integrity + safety matter; it's the format for general bundling.

2.7.1 CSF-Omni — the opt-in best-fit codec

The table above is the archive-vs-zip picture (text+code, ~30% spread). On the append-only memory log the codec choice dominates. CSF already ships zstd-19 + LDM by default, which compresses aMB JSONL memory log 362× (vs zlib's 14× before #835). CSF-Omni (src/csf/omni.py) is a new opt-in codec (codec="omni") that goes one better: it runs the whole panel per blob (store · zlib · bz2 · lzma · zstd · brotli + a byte transform), round-trip-verifies each, and keeps the smallest behind a 7-byte self-describing, CRC-checked header — deterministically (same input → same bytes).

Measured against the shipped zstd-19, every codec round-trip-verified lossless (experiments/csf_compression_benchmark.py; full write-up: CSF Compression Benchmark — Review v3 (PDF)):

CSF-Omni beats every other codec (zlib/zstd/lzma/bz2) on every single stream and is the only configuration that is best-or-tied everywhere. On the multi-file archive the picture is narrower: codec="omni" reaches 3.06× on the 340-file corpus, but master's existing zstd + shared dictionary (use_dict=True) already reaches 3.00× — so omni's archive edge is only +2 %, at ~7× the encode cost (omni ~31 s vs zstd+dict ~4 s vs plain zstd ~1 s). The dictionary recovers the cross-file redundancy that per-file omni cannot, so omni's durable win is on single streams, not archives.

Honest framing (Σ₀). CSF-Omni is the upper envelope, not a new entropy coder: on these corpora brotli is the frontier, so Omni matches it (payload-identical, +7-byte header) — it does not beat brotli's raw bytes, and no library available here (PPMd, paq) does. Its value is guaranteed best-in-field selection on any input plus integrity (the CRC catches corruption that zstd's default frame returns silently) and a portable stdlib-only mode. Trade-off: the panel sweep makes the omni archiver ~31 s forfiles, so zstd stays the default (hot paths) and omni is the opt-in max-ratio tier for cold/archival single-stream blobs; decode is fast.

Adversarially verified. A six-agent fleet stress-tested it (fuzz round-trip — 3,817 checks · envelope · backward-compat · code review · decode-safety · honesty audit). Two defects were found and fixed: a corrupt brotli payload could decode to silently-wrong bytes (→ CRC-32 + a ValueError decode contract), and a docstring over-claim (→ reworded to the envelope framing above). CSF tests pass.

2.7.2 Beyond the envelope — beating zstd-19 (theorized, tracked)

Omni is the upper envelope of off-the-shelf byte codecs; to go past it you must model structure or statistics LZ cannot express. Four grounded techniques are theorized in research/2026-06-29-csf-beating-zstd.md:

The Σ₀ collapse certificate (§ external) is load-bearing for #3: "deeper recurrence → fewer bits" holds only inside the grounded, non-collapsed regime; the NIS/anisotropy canary supplies the measured depth-exit. CSF-Col (#1593) is the recommended first build — see § 2.7.3 for the shipped result.

2.7.3 CSF-Col — shipped (transform id 2)

src/csf/col_transform.py is a lossless invertible byte→byte transform (registered as Omni transform id 2) that transposes flat-ish JSONL records from row-major to column-major before the entropy backend, so like-typed fields (timestamps, confidence, the near-constant reasoner/verified) form long compressible runs. Values are captured as raw source substrings (no JSON re-serialization → byte-exact), and forward() self-checks its own round-trip and raises NotApplicable otherwise — so Omni (which re-verifies and keeps the strict min) auto-selects it only on JSONL where it actually wins, and fast-skips everything else.

Measured, all round-trip-verified lossless (col+brotli selected by Omni vs the prior best baseline):

Honest framing. The win is real but modest on these corpora because they are dominated by large free-text fields (hypothesis/result) that don't columnarize — the gain comes from the small structured fields, and it grows with schema homogeneity (largest on the append-only deltas stream). It does not reach the 2–3× that schema-rich (mostly-typed-field) NDJSON sees in the literature. Because Omni selects per input, CSF-Col never regresses: on text-dominated or tiny blobs the framing overhead loses and it simply isn't picked. Tests: tests/test_csf_col.py (13, incl. 1.5k-case fuzz).

2.8 Per-user profile pack (one file per user, KB-grounded)

src/csf/profile_pack.py compacts all of one user's CSF data into a single file — data/profiles/<user>.csf (CSF-Pack v0.8) — and grounds it in the base Knowledge Center.

Archive contents:

• user/… — the user's cube (data/cubes/<user>.private), deltas, indexes,

dreamer notebooks, csf_memory, profile json.

• knowledge/index.jsonl — the embedded base KB grounding index (so the file

is self-contained and grounded), plus its .meta.json.

• _profile.json — sources, user file count, and the grounding reference (KB sha256 + section count).

python -m csf.profile_pack pack <user>            # -> data/profiles/<user>.csf

python -m csf.profile_pack info  <archive>        # embedded _profile.json

python -m csf.profile_pack unpack <archive> -d <dest>

Routes: POST /api/csf/profile/pack {user} · GET /api/csf/profile/info?user=<id>. User profile archives are gitignored (user data — privacy). Tests: tests/test_csf_profile.py.

2.9 Base Knowledge Center grounding + cheaper routing

• KB index — scripts/build_knowledge_index.py turns the Knowledge Center

source docs into data/knowledge/index.jsonl (one record per doc section with heading path + snippet). This is the base grounding corpus for "better LLM grounding."

• Cheaper deterministic / near routing —

lib/knowledge-router.js answers from the KB index before paying for an LLM:

1. deterministic — exact heading match → that section verbatim ($0)
2. near — TF-IDF nearest section above threshold → grounded answer ($0)
3. miss — caller falls through to the provider chain

Route: GET|POST /api/knowledge/query { q } → { tier, hit, source, text, score }.

Rebuild the KB index after editing core docs: python scripts/build_knowledge_index.py.

3. Retired formats & the read-only bridge

The v2 consolidation removed every legacy writer. Nothing in the codebase produces these formats anymore; existing archives open read-only through csf.legacy.

csf.legacy.open(path) sniffs and dispatches: modern CSF\0 → core; zlib blob → inflate; anything else → CsfLegacyError.

3.1 Still-live v0.7 lattice (v07/) — kept

quantum_dust, qutrit_delta, convergence_engine, plus the binary container csf_file.py (used by the Status-Cube store) and the lossless primitives in classical_compressor.py (SymbolicDictionary, sparse CSR). These are the storage face of the 3¹² lattice (§6), not a compression format.

4. Code map

5. Consolidation pointers (previously scattered)

• docs/CSF-Whitepaper-v0.3.pdf — original whitepaper
• docs/PHASE-1-CSF-BACKEND.md — backend phase notes
• caad/README.md, caad/dollhouse-csf-upgrade.md — CADD layer
• CSF-IMAGE-TRAINING.md — image-LoRA training over CSF
• csf/ingest/ — CSF ingest docs are the memory/task queue, not format specs

This spec is the authoritative format reference; the above remain for history.

6. The 3¹² lattice (storage face of the singularity)

The v0.7 symbolic engine is not a standalone compressor — it is the storage face of a single 3**12 = 531,441-cell balanced-ternary lattice that the project also reasons over geometrically (the "Tesseract"). The two are one object; the full argument and grounding live in TESSERACT-CSF-SINGULARITY.md. Bridge facts:

Why base-3, not base-2: ternary is the most economical integer radix (optimum is e, nearest integer 3), and balanced ternary {-1,0,+1} gives symmetric arithmetic — the same substrate as BitNet b1.58's ternary weights (arXiv:2402.17764). The "no change is free" dust optimisation is the storage-side twin of BitNet's ~66 % zero-weight sparsity. Citations and falsifiable experiments: see the singularity doc §5–6.