Trust backstop · design
Accuracy Harness: design (routes times dates)
What this is. A build-ready spec for the trust backstop of the WiFi Odds Chrome extension: a harness that verifies the odds the extension shows are actually accurate. Spec only, nothing here is built yet. It extends the existing test/ harnesses; it does not replace them.
Status: design. Read-only pass over extension/ and test/ on 2026-07-30. No repo was modified.
One sentence. Two layers: (1) does the extension display what the tracker returned, correctly, across a route times date matrix, and (2) do the tracker's published probabilities match what actually flew, with the second layer honestly gated on data the tracker does not yet expose.
0. Where this fits (and what already exists)
The extension's predictions do not come from wifiodds.com. They come from @martinamps' Starlink trackers (unitedstarlinktracker.com, alaskastarlinktracker.com), keyed on flight number and route. wifiodds united/data.json is only the equipped-tail roster. So the system under test is the tracker, cross-checked against our roster where the two can be joined.
Three harnesses, one of which is this proposal:
| Harness | File | Answers | Determinism | Gate? |
|---|
| Phase 1: API matrix | test/phase1-api-matrix.mjs | Do the tracker's endpoints agree with each other and with our roster? What would the panel render? | Live tracker | Report only |
| Phase 2: browser E2E | test/phase2-e2e.mjs | Does the content script render what a fixed tracker response says? | Fully mocked | Release gate (exit 1) |
| Accuracy harness (this doc) | test/phase3-accuracy.mjs (new) | Layer A: display truth across routes times dates. Layer B: are the published probabilities calibrated against realized outcomes? | Layer A deterministic; Layer B replay of recorded snapshots | Layer A gates; Layer B reports |
The gap this fills: Phase 2 proves the display is correct for a response we invented. Phase 1 proves the tracker's surfaces are internally consistent right now. Neither one ever asks the only question a user actually cares about: when the extension said "68%," did a Starlink plane show up 68% of the time? That is a calibration/backtest question, and it needs realized outcomes over time, which is exactly what the data contract's Ask 3 (post-departure outcome backfill) would provide. Until then, the harness records predictions now so the backtest becomes computable the day outcomes exist, and measures everything that is checkable today.
1. Goal and the two layers
Layer A: DISPLAY accuracy (buildable today)
The extension shows what the tracker returned, correctly, across routes times dates.
This is a superset of Phase 2, widened from a handful of hand-authored cases to a route times date matrix driven by real (recorded, then replayed) tracker responses. It asserts, per cell:
- the badge's
%, obs, and confidence match what the tracker's predict-flight / predict_route_starlink actually returned (via the shared parsers in lib/tracker.mjs);
- the four empty-state strings are chosen correctly (the Codex-approved state model: "No direct-flight Starlink history yet. Connection estimate below." vs "…for this route yet." vs "Direct-flight history unavailable right now." vs a rendered list);
- a
✓ confirmed-tail mark appears only when the searched date is within about 3 days and a firm tail was named, never on a far date;
- the two surfaces agree: the united.com panel number and the Google Flights / Navan chip number for the same flight are equal (endpoint parity, already proven at a point in time in Phase 1, here asserted per matrix cell).
Layer B: PREDICTION accuracy (calibration / backtest)
The tracker's published probabilities match realized outcomes.
For a prediction of p made at horizon h for (flight, date), we later learn the realized outcome: did the aircraft that actually operated it have Starlink. Aggregated over many predictions, p should be calibrated: of all flights predicted about 60%, close to 60% should have flown Starlink. This layer is blocked on outcome data (see section 6) and therefore reports, never gates. What is computable today is a partial, near-horizon version using the tracker's own check_flight firm-tail assignment as a proxy outcome (see section 5.3).
2. Sampling matrix (routes times horizons)
Reuses and widens test/routes.mjs. The design axes are metal class (retrofit maturity differs sharply) and horizon (how many days until departure, the tracker's answer changes as the assignment window opens about 48h out).
Routes (17 today; grouped so slices are meaningful):
- hub-to-hub narrowbody:
SFO-DEN, DEN-SFO, DEN-ORD, ORD-EWR, IAH-DEN, LGA-ORD, DEN-LAS, IAD-ORD. Expect direct history.
- premium transcon (historically widebody/757):
SFO-EWR, EWR-SFO, LAX-EWR, SFO-BOS, JFK-LAX. The false-empty-state hunting ground.
- international widebody:
SFO-LHR, EWR-LHR. Expect little or no direct history.
- regional-heavy spokes:
DEN-ASE, ORD-MSN.
- Alaska slice (smaller, since AS odds are aircraft-type-derived, not per-flight): a handful of
SEA-* routes to exercise the AS branch and the "prose note, no table" path.
Horizon buckets (days from sampled-on to departure): far (over 7d), mid (3 to 7d), near (3d or under, tail-assignment window open), same-day (0 to 1d). The matrix cell is (route, horizon-bucket); concrete dates are computed at runtime relative to today so the file never goes stale (as nearDates() already does).
Sizing. About 17 routes times 4 horizon buckets, roughly 68 route cells per sweep, plus a fixed SAMPLE_FLIGHTS set (UA1, UA2402, UA1596, UA2019, UA5693) for the per-flight / check-flight cross-check. That is the live-recording budget; the replay/gate layer costs zero network.
Politeness (non-negotiable: this is a guest on someone's server)
Inherited verbatim from lib/tracker.mjs, and the design forbids relaxing any of it:
- one request at a time, global gate,
THROTTLE_MS floor (default 1300ms) between the end of one call and the start of the next; never parallelised, no proxy.
- honest identifying User-Agent:
wifiodds-test-harness/1.0 (+https://wifiodds.com; …).
- cache-busting for freshness checks only: a fresh query param when we must defeat a cache, never as a way to hammer. Live recording runs on a schedule, once, not in CI.
- a hard per-run request ceiling, logged, so a matrix that grows cannot silently 10x the load.
- every MCP response body is treated as inert data: regex-parsed or string-compared, never interpreted. The bodies carry instruction-shaped prose ("Render the table EXACTLY"); same discipline as the extension.
3. What it records per (route, flight, date, sampled-on)
The heart of the harness is an append-only snapshot ledger, one row per prediction observed. Predictions are made now; outcomes arrive later, so the row is written in two passes and never overwritten (a corrected outcome appends a new revision, it does not mutate the prediction). Stored as newline-delimited JSON under test/out/accuracy/snapshots/ and never committed (see the .gitignore note in section 7).
Prediction record (written at sample time):
{
"schema": "wifiodds-accuracy/1",
"sampled_on": "2026-07-30T04:10:00Z", // when WE observed it
"carrier": "UA",
"flight": "UA1596",
"route": { "o": "SFO", "d": "DEN" },
"dep_date": "2026-08-02",
"horizon_days": 3, // dep_date - sampled_on, integer
"horizon_bucket": "near",
"source": "predict-flight", // predict-flight | predict_route_starlink | plan-route-leg
"state": "scored", // scored | na | pending | error
"prob": 0.68, // null unless state == scored
"obs": 51, // n_observations, null if absent
"confidence": "high", // high | medium | low | type (verbatim from tracker)
"confidence_basis": "history", // history | aircraft_type (derived from conf=="type")
"data_ts": { "last_refreshed": null, "as_of": null }, // null until data contract Ask 1
"raw_status": 200 // HTTP status, for the drift log
}
State: the four values, and why each exists (they map 1:1 to bugs already shipped once):
scored: a real probability. Trust and display it.na: the tracker genuinely has no history for this flight number. Safe to cache as n/a. Distinct from a false zero (see section 4, display-integrity).pending: a near-date whose tail assignment has not published yet (about 48h out). Showing this as a fresh n/a is the "stale-as-fresh" bug.error: a 429/500/garbled 200. Must never be recorded as na, or the backtest inherits a false negative and the display shows a false n/a. This mirrors the extension's PREDICT_ERR sentinel and directOk flag: the harness records the state the extension had to infer, so a later data-contract state field can replace the inference.
Outcome record (appended later, once realized truth is known):
{
"schema": "wifiodds-accuracy-outcome/1",
"flight": "UA1596",
"dep_date": "2026-08-02",
"resolved_on": "2026-08-04T00:00:00Z",
"outcome_source": "check_flight_firm_tail", // proxy today; "operated_backfill" once Ask 3 lands
"actual_tail": "N73275",
"actual_type": "B739",
"wifi_state": "starlink", // starlink | non_starlink | unknown
"roster_agreement": true, // did our equipped roster agree? null if tail unknown
"outcome_confidence": "firm" // firm (published tail) | operated (post-departure truth)
}
The join key is (flight, dep_date). A prediction row and its outcome row are matched on that pair; a prediction with no matching outcome is pending in the backtest and excluded from the denominator: unknown is not zero, per the project doctrine. An outcome with wifi_state: "unknown" is likewise excluded, never assumed non-Starlink.
3.1 The proxy outcome available today
check_flight returns a firm tail for a near date (published about 48h out). That tail, joined to united/data.json, yields a yes/no Starlink verdict, a realized-enough outcome for near-horizon flights. This gives a limited backtest today: "for flights the tracker rated p a few days out, did the firm-assigned tail turn out to be Starlink?" It is honest but narrow: it scores the near-horizon assignment, not the far-horizon prediction, and it inherits any staleness in our roster. The full backtest (any horizon, post-departure truth) needs Ask 3.
4. Metrics
Two families: prediction-accuracy metrics (Layer B, mostly outcome-gated) and display-integrity metrics (Layer A, computable today).
4.1 Prediction-accuracy metrics (Layer B)
- Reliability diagram / calibration curve. Bin predictions into deciles of
p (0 to 10%, 10 to 20%, and so on). For each bin plot mean predicted p against observed Starlink frequency. Perfect calibration is the diagonal. Rendered as a table plus an SVG the audit report can embed.
- Brier score. Mean squared error of
p vs the 0/1 outcome, over all resolved scored predictions. One headline number, lower is better; also reported per slice.
- Coverage of "confident" predictions. Of predictions the tracker labelled
confidence: high (and separately basis: history vs aircraft_type), what fraction resolved correctly at a decision threshold (for example p ≥ 50% → predict Starlink). Surfaces whether "high confidence" is earned. type-derived odds are reported as their own cohort: they answer a different question (aircraft-type base rate, not this flight's history) and must never be pooled with history-based odds.
- Error rate, sliced. Misclassification at the threshold, sliced by: route, horizon bucket (near vs far, the key question is whether far-horizon odds are worse), and fleet-rollout era (bucket by month, so improving retrofit coverage shows as drift over time, not noise).
Every metric carries its denominator and its exclusions in the report: how many predictions resolved, how many are still pending, how many unknown. A calibration curve computed on 40 resolved points is labelled as such and never presented as if it were 4,000. This is the "publish the floor, not the blend" rule applied to the harness's own output: a metric nobody can source yet is written as "not enough resolved outcomes," never as a confident zero.
4.2 Display-integrity metrics (Layer A, today)
These check the rendering, and each maps to a real incident:
- No false zeros. A
0% badge is legitimate only when the tracker returned a real probability: 0 with observations (a genuine "this flight never gets Starlink, 51 obs"). A 0 or n/a synthesized from an error state, or from an unpublished count, is a false zero: the harness flags any badge whose 0/n/a is not backed by a scored or na state.
- No stale-as-fresh. Once the data contract exposes
last_refreshed/as_of, a value older than the display TTL rendered without a staleness cue is flagged. Today (no timestamps) the proxy is the 6h cache TTL: a badge served from a cache entry older than TTL that is not force-refreshed is flagged.
- No absence-from-failure. The
error → "unavailable" copy must appear on a failed direct-history fetch; the harness fails the cell if an error state renders as "No direct-flight Starlink history…" (the exact state-model bug from the 29 Jul findings).
- Surface parity. united.com panel
% equals GF/Navan chip % for the same flight, per cell.
5. Drift / canary behavior: fail closed
The harness must fail closed: when it can no longer trust that it is reading the tracker correctly, it stops and shouts, rather than emitting a clean-looking metric computed on garbage. Three drift classes, each with its own canary:
- Schema drift (tracker). The shared parsers in
lib/tracker.mjs mirror extension/bg.js. If predict-flight renames probability, or predict_route_starlink's table format changes, the parser yields null/zero rows. Canary: a fixed known-good probe (SAMPLE_FLIGHTS on a route with reliable history) that must parse to a scored state with a plausible shape. If the probe comes back unparseable or all-na, the run aborts with SCHEMA-DRIFT and records no snapshots: recording a sweep of false nas would poison the ledger permanently.
- Row-recognition / route-extraction drift (extension). Layer A drives the real content script. If united.com/Navan/GF markup assumptions break (rows not found, route context mis-extracted) the panel renders nothing or the wrong route. Canary: the deterministic positive-control cell (a
SFO-DEN-style fixture with a known badge) must still render its known badge, exactly as Phase 2's negative-control (E2E_NEG) already proves a seeded regression makes the gate exit 1. If the positive control fails, the gate fails.
- Parser drift between harness and extension.
lib/tracker.mjs is a hand-copy of bg.js's regexes; they can silently diverge. Canary: a small golden-fixture test that feeds a frozen set of real tracker response bodies through both the harness parser and (in the browser layer) the extension, and asserts identical decoded output. If bg.js changes a regex and the mirror does not, this fails, the same "a rule that lives where the runtime does not read it is not a rule" doctrine, applied to the parser copy.
How it signals. Two channels, matching the project's existing shape:
- Report:
test/out/accuracy/report.md (human) and accuracy-findings.json (machine), severity-ranked like Phase 1 (HIGH/MEDIUM/LOW/INFO). A drift is HIGH.
- Exit code: Layer A (display plus canaries) participates in the release gate and exits 1 on any failed check, drift included. Layer B (calibration) reports and never gates, because a calibration metric moving is a finding to investigate, not a reason to block a ship, and because it depends on data we do not yet control. A red drift canary, though, blocks: better no metric than a false one.
Fail-closed is literal: on any drift canary trip, the live-recording pass writes zero snapshots and the ledger is left exactly as it was. Same doctrine as ship.sh and reconcileUnited(): a process that corrupts the record unattended is worse than one that does nothing and says so.
6. Dependencies on the @martinamps data contract
Grounded in martinamps-data-contract-proposal.md (the three asks: explicit states, a changed-since delta feed, post-departure outcome backfill). What each unblocks:
| Harness capability | Needs | Buildable today? |
|---|
| Layer A display checks, all four states | nothing (state is inferred, as the extension does) | Yes |
| Snapshot ledger of predictions | nothing | Yes |
| Endpoint / surface parity per cell | nothing | Yes |
| Schema/row/parser drift canaries | nothing | Yes |
| Near-horizon proxy backtest (firm tail vs roster) | check_flight + united/data.json (both exist) | Yes, but narrow |
Trustworthy state (no client inference) | Ask 1: versioned record with explicit PUBLISHED / NO_HISTORY / PENDING / UPSTREAM_ERROR | No |
| Stale-as-fresh metric (real timestamps) | Ask 1: last_refreshed vs as_of | No (proxied by cache TTL today) |
confidence_basis without string-matching "type" | Ask 1: structured basis: history | aircraft_type | No (derived today) |
| Poll the tracker a fraction as often | Ask 2: changed-since / delta feed | No |
| Full calibration / Brier / reliability at any horizon | Ask 3: post-departure outcome backfill (actual tail + type + WiFi state after departure) plus a queryable sample window | No: the core of Layer B is blocked here |
The honest headline: the reliability diagram, Brier score, and any-horizon error rate, the metrics that make this a trust backstop rather than a display test, are blocked on Ask 3. What ships today is the machinery around them: the sampling, the append-only ledger, the display integrity, the drift canaries, and a narrow near-horizon proxy backtest. The day outcome backfill exists, Layer B lights up against a ledger that has been accumulating real predictions the whole time, which is the entire reason to start recording snapshots now rather than waiting.
Note also the edge-cache / proxy question (the sibling proxy note): if an edge cache is ever placed in front of the tracker, the harness must record whether a snapshot came from origin or edge, because a cached edge response can be stale in a way curl cannot see, the same "verify by response body, and the edge can rewrite what the repo serves" lesson.
7. Shape of the build
A node script plus fixtures, living under test/, consistent with the two existing phases.
test/
phase3-accuracy.mjs # entry: --record (live, scheduled) | --replay | --backtest | --gate
routes.mjs # EXTEND: add horizon buckets + Alaska slice (exists today)
lib/
tracker.mjs # reuse: polite client + mirrored parsers (exists)
roster.mjs # reuse: roster join (exists)
ledger.mjs # NEW: append-only snapshot read/write (NDJSON), two-pass join
calibration.mjs # NEW: binning, Brier, reliability-diagram SVG (pure, deterministic)
canaries.mjs # NEW: schema / parser golden-fixture checks
fixtures/
accuracy/ # NEW: frozen real response bodies for replay + golden parser tests
out/
accuracy/
snapshots/*.ndjson # the ledger (gitignored; may hold many days of predictions)
report.md # human
accuracy-findings.json # machine, severity-ranked
reliability.svg # calibration curve for the audit report
Modes (one script, four verbs):
--record: the only networked mode. Sweeps the matrix once against the live tracker, politely, and appends prediction snapshots. Runs the drift canaries first and writes nothing if they trip. Scheduled, not in CI (for example a daily wave), because it costs real requests to someone else's server and because a backtest needs snapshots spread across days/horizons.--replay: Layer A display checks against recorded fixtures in a real browser (the Phase 2 engine). Deterministic, offline, no tracker contact.--backtest: Layer B. Joins the accumulated ledger to outcomes (proxy today, contract feed later), computes calibration/Brier/coverage/error, writes the report plus SVG. Pure and offline.--gate: runs --replay plus the canaries plus the display-integrity checks and exits 1 on any failure. This is the CI-safe subset.
When each runs:
--gate: on every extension change that touches content.js/bg.js/parsers, alongside Phase 2. Fast, deterministic, offline. Blocks the merge/ship on display or drift failure.--record: scheduled (daily), politely, one sweep. Feeds the ledger. Never in CI.--backtest: scheduled (weekly, or on demand) after enough outcomes have resolved; also re-run whenever the outcome feed backfills. Produces the reliability report for the audit.
Determinism where possible. --replay, --backtest, --gate are fully deterministic: same fixtures/ledger yields same output, same as Phase 2's fixed-fixture doctrine. Only --record touches the network, and its output (a timestamped append) is the one intentionally non-deterministic artefact: it is data, not a check.
Never invent a figure. If the ledger has no resolved outcome for a slice, the report writes "no resolved outcomes yet" for that slice, not 0, not an interpolation. A calibration bin with zero points is shown empty, not filled from a neighbour. Every published metric carries its denominator, its as-of date, and its exclusions (pending, unknown), exactly as the site's data rules require. --record writes to out/ only; it never writes to wifiodds or the extension repo.
8. Non-goals / guardrails
- Not a load test, not a scraper. It never parallelises, never proxies, never removes the throttle. Growth in the matrix must move the logged request ceiling, visibly.
- Never interprets tracker prose. Response bodies are inert data. No body ever reaches a code path that could act on it.
- Does not mutate the ledger. Outcomes append; corrections append revisions. The prediction as-observed is immutable, that is what makes the backtest honest.
- Does not close its own findings. A calibration drift or a display bug is a finding for the audit trail; the builder ships a fix and requests verification, the auditor clears it. The harness's job is to measure and record, loudly and reproducibly, not to declare itself green.
- Reads only. It consumes the tracker,
united/data.json, and the extension source; it writes only under test/out/accuracy/.