docs/adr/0029-ibkr-streaming-challenger-duel.md

ADR-0029: The IBKR streaming Challenger — a return-seeking duel against the Champion on accurate intraday data

  • Status: Proposed (awaiting Alex's approval)
  • Date: 2026-07-19
  • Loop stage: Act (streaming risk reaction + paper execution) + Verify (duel measurement)
  • Related: ADR-0019 (IBKR CPAPI via local gateway), ADR-0020 (order placement, gated, dry-by-default — still Proposed), ADR-0022/0027 (per-user broker auth), ADR-0028 (Sharpe mandate gate — unchanged by this ADR), apps/lantern-garage/lib/brake-monitor.js, apps/lantern-garage/lib/champion-book.js, apps/lantern-garage/lib/kalshi-adaptive-poll.js, experiments/DEEP_HISTORY_RESEARCH_LOG.md, PR #2744 (the monthly Leap report, where the duel scoreboard will publish)

Context

The operator asked for a real-time / streaming trader on IBKR designed to beat the Champion in returns, using more accurate historical data.

The incumbent ("the Champion"): the 8-ETF shrunk-tangency book (SPY/QQQ/IWM/EFA/TLT/GLD/XMMO/SPMO) × the streaming brake (35% vol-target × 6-mo trend gate × −30% dd-taper, gross 0–2×), marked on a $25k paper book and deployed dry on Alpaca paper. Walk-forward 2000→2026-07: $91,537 on $8,380 paid in, 12.6%/yr, Sharpe 0.65, maxDD −25.5% (run 2026-07-17).

Σ₀ discipline first: "beats the Champion" is a hypothesis, not a design property. This ADR designs a challenger plus the duel that would prove or refute it. Every expected gain below is pre-registered as an estimate, to be measured, with a fold path if it fails.

Measured facts this design stands on (receipts):

  1. Reaction lag is a real, measured return leak. The hourly brake beat the daily brake

$51,355 vs $47,077 over 2.5y at identical turnover (Sharpe 1.32 vs 1.18, maxDD −19.8% vs −22.2%), and de-levered 71 hours earlier on the worst crash day in the window (experiments/brake_intraday_evidence.json). Faster, accurate reaction has already paid once; the challenger buys the residual minute-scale slice.

  1. The current model's frictions are cruder than reality — in both directions.

brake-monitor.js charges a flat 3% T-bill proxy (BRAKE_TBILL_RATE default) and a flat T-bill+150bp funding spread; realbill ETFs yield materially more than 3%, and IBKR publishes an actual tiered margin schedule. Cash-yield and funding accuracy are mechanical return deltas — the truth may help or hurt, and either way the Champion's book gets truer.

  1. Accurate history is buyable; free intraday is not trustworthy. Yahoo intraday equity bars

have measured corrupt wicks (we clamp per-side vs 4× median). Full-history, split/dividend-correct minute aggregates exist (Polygon, back to ~2003, ~$29/mo); survivorship-free daily exists (Sharadar, ~$30/mo). The Champion was tuned on Yahoo daily adjclose — fine for its ETF universe, blind at intraday horizon.

  1. Retired paths stay retired unless new evidence appears. Single-stock 12-1 momentum on

survivorship-clean data measured Sharpe 0.60 ≈ SPY, below the Champion's 0.66 — not an upgrade (2026-07 sweep). Gross above 2× was rejected by the deep-history study (worse Sharpe, 3× the trades, margin risk; iters 1–2). Neither returns here.

Decision (proposed)

Build the Challenger: the Champion's allocation core, unchanged, plus exactly three upgrades, run on IBKR paper, in a formal duel.

1. True streaming signal plane (Act)

  • Market data from IBKR CPAPI via the local gateway (ADR-0019): websocket streaming (or 5s REST

polling fallback through the existing ibkr-cpapi.js client) for theETFs.

  • Evaluation cadence goes send-on-delta: reuse kalshi-adaptive-poll.js#createScheduler

(β/σ² control-engineering cadence, floor 5s, cap 60s) instead of the fixed 60s clock — evaluate when prices actually move, sleep when they don't.

  • Intraday realized-vol estimator: gap-aware EWMA over minute returns blended with the 20d daily

window (the brake's tuned thresholds keep their daily calibration; the estimator just stops being a day late). Trend gate and dd-taper are unchanged — they are the validated core.

  • If IBKR market data is unavailable, the Challenger halts and marks the gap — it never falls

back silently to Yahoo intraday (known wick corruption).

2. Accurate frictions, both directions (Act + Verify)

New pure module funding-model.js:

  • Borrow side: IBKR's real tiered margin schedule, pulled live and cached, charged in both

backtest and live marking (replaces flat +150bp).

  • Cash side: de-levered cash modeled as swept to a bill ETF (BIL/SGOV) at its real yield

(replaces flat 3%).

  • Fills: slippage model calibrated from the Challenger's own paper fills once live (starts at

the backtest's 2bp assumption, updated monthly).

3. Accurate history for tuning and verdicts (Verify)

  • Walk-forward substrate: minute-level, dividend/split-correct bars 2004→present (Polygon flat

files). Tune ONLY on 2004–2015; validate 2016–2026; no peeking.

  • Survivorship cross-check: Sharadar daily (optional until any single-name work returns — the

8-ETF universe has no delisting exposure, stated honestly).

  • Seam reconciliation: IBKR's own historical bars must reconcile Polygon's last 6–12 months so

the backtest's data equals the live pipeline's data at the boundary.

  • The Champion keeps its own production pipeline (Yahoo daily) in the duel — each book fights with

the data it actually runs on.

Non-goals: no new alpha sleeve; no gross > 2.0×; no change to turnover character (the band and ~monthly rebalance cadence stay — streaming changes risk reaction speed, not trading frequency; day-trade-pattern behavior is explicitly out); no live capital (Phase C below is unchanged ADR-0028).

The duel protocol (Verify)

Phase What Win condition
A — backtest duel Minute walk-forward 2004→2026, both premises ($25k book; $2k+$20 DCA), Challenger vs Champion-as-is Higher final value AND maxDD ≤ Champion's AND block-bootstrap ΔCAGR/ΔSharpe CI excludesdeep_history_significance.pyAND survives TC 2–20bp and data-vintage sensitivity (pattern: deep_history_significance.py / deep_history_tcost.py)
B — live paper duel ≥months side-by-side: Challenger on IBKR paper (DU account) vs Champion on Alpaca paper; daily marks to data/trading/duel/duel-ledger.jsonl Phase-A ranking reproduced in sign; scoreboard published monthly in the Leap report
C — real dollars Unchanged ADR-0028: live meets_ci vs the 0.79 lifetime-Buffett bar, protected paths, a human clicks Nothing in this ADR loosens any gate

Pre-registered expected sizes (estimates, not results): reaction-lag residual 0–1%/yr (the hourly step already captured ≈ +1.5%/yr relative in its window; minute-scale marginal gain may be small); cash-sweep accuracy ≈ +0.4–0.5%/yr at the Champion's historical average cash fraction; funding accuracy sign unknown until the real schedule is pulled. Net target: +1–3%/yr at ≤ equal drawdown.

Honest exits (the fold path): if Phase A shows < +0.5%/yr expected lift, or Phase B contradicts Phase A's sign, the Challenger folds: the funding/cash-sweep corrections are adopted into the Champion (they are model-accuracy fixes, not strategy changes), the streaming cadence is retired as measured-negative, and the data subscriptions are cancelled. A negative verdict is a publishable result, not a failure of the project.

Architecture (extend, don't sprawl)


ibkr-stream.js (new, thin)        ws/5s ticks for 8 ETFs via ADR-0019 gateway

        │

brake-monitor.js (extended)       injectable tick source + send-on-delta scheduler

        │                         (kalshi-adaptive-poll.createScheduler) + funding-model.js

        ▼

challenger-book.js (new)          mirrors champion-book.js: SAME targetWeights /

        │                         computeRebalance imports; IBKR adapter conforming to

        │                         alpaca-adapter's interface over ibkr-cpapi.js;

        │                         PAPER-ONLY hard refuse; ADR-0020 guard untouched;

        │                         CHALLENGER_ARM env mirrors CHAMPION_ARM (default dry)

        ▼

data/trading/duel/duel-ledger.jsonl   daily marks, both books, append-only (gate-scoreable)



experiments/challenger_minute_walkforward.py   Phase A harness (train 2004-15 / validate 2016-26)

Feature-gate check: improves Act (reaction accuracy, execution realism) and Verify (truer frictions, truer data, a scored duel). No new memory systems, no new agents, no parallel strategy engine — the allocation math is imported from the Champion, not forked.

Costs

Polygon minute history ~$29/mo + Sharadar ~$30/mo (deferrable) + IBKR paper $0 (live market-data subscriptions only if/when Phase C ever arrives). ≤ ~$60/mo during the duel, cancel-on-fold.

Risks

  • IBKR gateway fragility. Local-gateway session drops; local owner creds were stale as of

2026-07-14 (lst_signature_mismatch — only Alex can refresh). Mitigation: REST 5s fallback, halt-and-mark on data loss, duel clock pauses rather than fabricates.

  • Intraday overfitting. Mitigation: three upgrades only, tuned thresholds inherited from the

daily-validated brake, train/validate split, deflated-Sharpe + bootstrap CIs, pre-registered targets written here before any run.

  • Marginal-gain illusion. Minute-over-hourly may add ~nothing; the fold path treats that as a

clean verdict and still banks the friction-accuracy fixes.

  • Two data planes, one seam. Polygon-vs-IBKR reconciliation is a hard requirement, not a nice-

to-have; unreconciled seams void Phase A.

Phase A results — free-data variant (run 2026-07-19)

Operator directive: free data only, run now. The free slice of upgrade #2 was executed: experiments/challenger_free_rates.py swaps the engine's flat 3% cash / 3%+150bp borrow for the real daily 13-week T-bill yield (^IRX, Yahoo, free, 7,161 obs 1998→2026: mean 2.10%, min 0.00%, max 6.22%, 3.71% today) — signals, weights, band, premise byte-identical (engine-equivalence check: flat-3% series reproduces the Champion to $0.0000). Results (experiments/challenger_free.json, panel through 2026-07-17):

Book Final Sharpe maxDD Funding paid Verdict
Champion (flat 3%) $91,702 0.648 −25.5% (modeled flat) baseline
Challenger (real IRX, +150bp) $102,294 0.667 −25.2% $17,250 +11.6% final, shallower DD
Sensitivity: IBKR-Lite spread (+250bp) $90,072 0.641 −25.3% $19,615 win erased
  • Attribution: the flat model overcharged the ZIRP era — the challenger's equity gap runs

−7.5% (2009, real rates were high 2000-07) → +18.3% (2022, borrowing at ~1.65% real vs 4.5% modeled) → +11.6% final (2022-26 hikes give some back). Exactly the accurate-history story.

  • Pre-registered significance test: paired block-bootstrap Δreturn **+0.34%/yr, 95% CI

[−0.11%, +0.74%], P(>0) = 93.2% — positive but does not exclude zero, so under this ADR's own win conditions the duel is NOT declared won on returns. However, this delta is a deterministic model-accuracy correction** (true rates vs a wrong constant), not a mined edge — the fold-path rule applies: adopt the friction fix into the Champion regardless.

  • Spread sensitivity is decisive for live: the entire gain requires Pro-tier (+150bp-class)

margin pricing; Lite-class (+250bp) erases it. Broker/tier choice is a return decision.

  • Immediate adoptable fixes (pending approval, money-path): (1) the walk-forward and the

published Champion numbers should carry real-IRX carry (truer record: ~$102.3k, Sharpe 0.667); (2) brake-monitor.js should mark with the live bill yield (^IRX 3.71% today) instead of BRAKE_TBILL_RATE default 3% — both are Verify-stage truth fixes, not strategy changes.

  • Still open (paid or IBKR-gated): minute-accurate history, streaming cadence beyond 60s, and

real fills — the remaining Phase A upgrades this run could not touch for free.

Phase A results — cadence duel: real-time vs the hourly brake (run 2026-07-19)

Operator follow-up: if the Contender could brake in real time, compare it to the Champion's hourly brake. Free method (experiments/challenger_realtime_brake.py): the hourly-brake experiment's exact panel and engine (Yahoo hourly,trading days 2023-08→2026-07, $25k book, identical monthly direction), extended with OHLC lows so a realtime variant can act inside the hour — when the trend gate (or dd-taper) crosses on the linear path to the bar's clamped low, it cuts at the trigger price (5bp fill haircut; re-levering waits for bar closes — defensive asymmetry). Fake-wick guard:close repairs + 3,118 low clamps at 4× the rolling median move. Results (experiments/challenger_realtime.json, sim 2024-03→2026-07):

Variant Final Sharpe (Lo CI) maxDD Intra-bar cuts
daily brake $46,663 1.16 [−0.11, 2.43] −22.2%
hourly brake (Champion) $50,902 1.30 [0.03, 2.57] −19.8%
realtime brake (Contender) $51,543 1.33 [0.06, 2.60] −19.8% 1 (0 whipsaw, median λ 0.80)
static 2× (control) $57,410 1.41 −22.8%
  • The marginal-cadence curve is now measured end to end: daily→hourly = +$4,239;

hourly→realtime = +$641 (≈ +0.5%/yr on this window). Diminishing returns, exactly as this ADR pre-registered ("expect small, 0–1%/yr").

  • The whole realtime edge came from ONE event — a single mid-hour trend-gate crossing where

the realtime book de-levered at λ=0.80 (~12 minutes before the hourly close), whipsaw-free. The paired bootstrap says so honestly: +0.07%/yr, 95% CI [+0.00, +0.21], P(>0)=63% — a real mechanism, a fragile sample (n=1 trigger in 2.4y). Fill-haircut sensitivity 0/5/10bp barely moves it ($51,598/$51,543/$51,489).

  • Bull-window caveat, stated plainly: static 2× (no brake at all) won this window on final

dollars — 2024–26 was mostly up, and brakes are risk-protection that trails strong bulls (the deep-history/Faber result). Cadence conclusions here are about crash-response mechanism, not expected returns.

  • Package interaction: on this high-rate window the real-IRX carry costs the levered book

(hourly+IRX $49,490 < hourly-flat $50,902; real bills averaged 4.25% > the 3% model) — the mirror of the 26-year result where truth paid +11.6%. Accurate frictions cut both ways; over the full history they net positive.

  • Verdict for the build: the streaming Contender's real-time cadence is worth building only

as the cheap extension it is (the monitor already ticks at 60s; send-on-delta makes it event-driven for free) — its measured value is one good crash-day cut per couple of years, not a return engine. The return engine remains the friction-accuracy fixes and the brake itself.

Consequences

  • (+) The Champion's own book gets truer funding/cash numbers regardless of who wins.
  • (+) The IBKR execution path (ADR-0019/0020) gets exercised end-to-end on paper, de-risking any

future approval of ADR-0020.

  • (+) The monthly Leap report gains a live duel scoreboard — a user-visible, honest race.
  • (−) ~$60/mo data spend for the duel's duration; one more long-running process on the fleet host.
  • (−) If approved, ~3 modules +experiment harness of new code to maintain (mitigated by reuse of

the Champion's math and the existing scheduler/client).