The routing model

Shunt makes one model choice per session, on the first turn, before any tokens are spent. This page describes what that decision actually reads, how it is computed, why it is shaped this way, and the places where it does nothing useful. Its sibling is Error detection & auto-escalation, which acts after a verified outcome exists; this one acts before there is any evidence about the task at hand.

This page describes an opt-in. The shipped default is router.strategy: session_cascade, which starts every session on the cheapest healthy model and lets verified failure climb it. It never runs this page's rule: no embedding, no neighbour lookup, no candidate scoring, no per-task model choice. To get the routing model you set router.strategy: knn_semantic_cascade, and a default install does not. Two things follow that are easy to miss: exploration is inert under the default (it perturbs a kNN pick that is never made), and shunt doctor treats a missing embedding-weights cache as a warning rather than a failure, because nothing needs it. Why the default is the cheaper of two equal-quality points is in Results.

The rule is nonparametric: embed the task, look up the nearest past tasks whose outcomes you already verified, and take the cheapest model that cleared your quality bar on them. There is no training step and no learned weights. Everything it uses is in the outcome store on your disk.

The shape of it

flowchart TD
  IN["Input: the first turn's messages"] --> P1

  subgraph PRE["Pre-processing"]
    P1["Keep user + tool text, drop system;<br/>most recent turn first"]
    P1 --> P2["Clip to max_chars from the head"]
    P2 --> P3["fastembed ONNX encode → one 768-dim vector"]
  end

  P3 --> FP{"Corpus fingerprint<br/>still matches?"}
  FP -->|no| CS1["Cold-start model<br/>stale_embedding_space"]
  FP -->|yes| WARM{"Enough effective<br/>verified outcomes?"}
  WARM -->|no| CS2["Cold-start model<br/>cold_start"]
  WARM -->|yes| M1

  subgraph MODEL["Model: kNN over verified outcomes"]
    M1["HNSW cosine query → up to k labelled neighbours"]
    M1 --> M2["Weight each: confidence × (1 − distance)"]
    M2 --> M3["Per model: weighted success rate,<br/>weighted cost, sample count"]
    M3 --> M4{"Exploration on<br/>and inside budget?"}
    M4 -->|yes| M5["Thompson draw per model;<br/>cheapest clearing the bar;<br/>gate blocks an unbanked downshift"]
    M4 -->|no| M6{"Any model clears the bar<br/>with enough samples?"}
    M6 -->|yes| M7["Cheapest eligible<br/>cheapest_above_threshold"]
    M6 -->|no| M8["Cheapest model with no history<br/>exploration_untested"]
  end

  CS1 --> PO1
  CS2 --> PO1
  M5 --> PO1
  M7 --> PO1
  M8 --> PO1

  subgraph POST["Post-processing"]
    PO1["Apply an auto-escalation directive, if one is pending"]
    PO1 --> PO2["Lock the model onto the session"]
    PO2 --> PO3["Stamp provenance: neighbours, rule, propensity, decision index"]
  end

  PO3 --> OUT["Output: one model for the whole session<br/>X-Shunt-Decision · shunt explain"]

What it reads

Only the task-bearing text of the first turn, and only in one form: a vector.

• Roles. Content from user and tool messages. The system prompt is dropped on purpose — a coding agent's system prompt is many times longer than the clip window, and leaving it in meant every session embedded to nearly the same vector. A body with no user or tool message falls back to the flat wire-order text rather than embedding an empty string.
• Order. Most recent turn first. The clip below cuts from the head, so wire order would have thrown the task away and kept the preamble.
• Clip. The text is truncated to max_chars (packaged default 4000) before encoding. Attention is quadratic in length, so an unbounded prompt is an unbounded allocation; the cap is what keeps a long agent prompt from taking the router down. The full prompt still goes upstream untouched — the clip affects the routing signal only.
• Encoder. fastembed, CPU-only, running the model named in embedding.yaml. The shipped default is jina-code (768 dimensions); arctic is the other bundled option.
• Fingerprint. The active model, its dimension, and max_chars form a corpus fingerprint. Change any of them and the stored vectors live in a different space, so their distances mean nothing. Shunt notices, refuses to consult neighbours at all, and routes as if cold under the reason stale_embedding_space until you run shunt reindex. It does not quietly compare across spaces.

How it decides

Cold start comes first

Until there are enough verified outcomes, kNN has nothing to search, so the router routes to a fixed cheap model — deepseek-v4-flash (the cheapest live model), falling back through glm-5.2, then any healthy model in the pool, if the primary is unhealthy.

Cold start ends on effective sample size, not row count. Each outcome's weight folds in its verification confidence, and the gate uses the Kish effective size nₑ = (Σw)² / Σw² rather than counting rows. Two thresholds end it: enough Tier-2 (verified) outcomes, or enough labelled outcomes of any tier. A store full of low-confidence labels therefore stays in cold start longer than a raw count would suggest, which is the intended behaviour — the router waits for evidence it can trust, not volume.

Cold-start sessions are still embedded and still recorded. That is the point of them: they are how the corpus gets built.

On a fresh deployment the store is empty, so the router stays on the fixed cheap model until enough verified outcomes accumulate. You can warm it deliberately: python -m benchmark.routing.seed_live (from a checkout) loads the benchmark's measured — never imputed — outcome cells into the store, so the index is warm at inference. Seeded sessions use ids bench:... and the reason benchmark_seed; they are non-policy, and the router re-fits from live verified outcomes as they accumulate.

Re-seeding is incremental and skippable. Each seeded row stores a per-cell content_hash (task, model, outcome, cost, and text); on re-import only cells whose hash changed are written — updated in place, never re-embedded when unchanged. The warm-start bundle commits the measured cells via git LFS at benchmark/routing/data/seed/: one .npz per embedder fingerprint, with a plain manifest.json beside it (fresh clones need git lfs pull). Build it with make seed-bundle; make check-seed-bundle proves it current against the committed results.csv and challenges.json. Re-running seed_live skips re-importing while the stored marker's digests (results + challenges) still match — --force forces a full re-import.

Be clear about what seeding does not buy: the embedding→outcome signal on the benchmark corpus measured at chance (see the embedding-signal figure), so seeding is a start-warm convenience, not evidence the rule routes your workload.

The neighbourhood

Once warm, the embedding goes to an HNSW index (hnswlib, cosine space) which returns the k nearest embedded sessions (policy.k, default 20). Sessions without a recorded outcome are dropped, so a query can legitimately come back with fewer than k neighbours — sometimes with none.

Each surviving neighbour carries the model that ran it, its verified pass/fail, its real cost, the verification confidence of its label, and its distance. Each gets a weight:

weight = verification_confidence × max(0, 1 − distance)

so a far neighbour or a weakly-verified one counts for less, and a neighbour beyond distance 1 counts for nothing. There is no separate calibration model; this product is the whole of it.

Neighbours are then grouped by model, and each group yields a weighted success rate and a weighted cost. A neighbour whose real cost is unknown surfaces as infinite cost, which is deliberate: unknown must never sort as cheapest.

The selection rule

A model is eligible when its weighted success rate is at least success_rate_threshold (shipped 0.6) and its group holds at least min_samples neighbours (shipped 3). Among eligible models, the cheapest wins — cheapest_above_threshold. That single line is the product thesis: hold a quality bar, minimise cost under it.

When nothing is eligible, the rule walks the price-ranked pool from cheapest upward and returns the first model that has no history in this neighbourhood — exploration_untested. Read that carefully, because it is the rule's most consequential branch: on a thin or empty neighbourhood it returns the cheapest model, so an under-informed router behaves like always_cheap while reporting a reason that sounds like a considered choice. Only the debug log distinguishes the two. If every model in the pool has already been tried and none qualified, the cheapest is returned as safe_fallback — cost-minimal, not a quality bet: the strongest was in the tried set and failed the same bar, so escalating to it was a heuristic, and deliberate escalation on verified failure is the auto-escalation ladder's job (see escalation).

Exploration

Exploration ships on. When it is enabled and the cumulative exploration budget has room, a cost-aware Thompson layer runs before the selection rule.

Each model gets a Beta posterior built from the weighted neighbourhood counts on top of a prior. The prior is empirical-Bayes: seeded from that model's global offline success rate, with its pseudo-count strength capped (prior_strength_cap) so a long global history regularises a sparse local neighbourhood instead of swamping it. One rate is drawn per model; the cheapest draw clearing the same threshold wins, or the highest draw if none clears it.

Two rails sit on top. A conservative gate compares the sampled pick against the deterministic greedy pick and blocks an exploratory downshift to a cheaper model unless earlier successes have banked enough slack — otherwise the choice reverts to greedy as conservative_fallback. An exploration budget caps cumulative exploratory spend as a fraction of what exploiting would have cost. Every decision feeds the budget's denominator, exploit choices included.

The propensity of the realised choice is estimated by Monte Carlo and logged, so the policy can be evaluated off-policy later. That resampling snapshots and restores the RNG state, so turning the logging knob up cannot change which model your next request gets.

What happens to the decision

• It is locked to the session. The chosen model is written onto the session and reused for every later turn. The router is never consulted again for that session. This is the cache-safety spine: no mid-conversation model switch, so no silent full-price re-read of a cached context.
• A pending escalation may override it. If auto-escalation is enabled and a directive is due, it is applied here, at the boundary — raising the reasoning arm on the same model, or stepping a rank. See escalation.
• Provenance is stamped. The neighbours consulted, the rule that fired, the propensity, the per-model scores, and the decision index all land on the session row. shunt explain <session_id> reads them back, and the X-Shunt-Decision response header names the model and reason.
• The outcome comes back later. At session close the verifier records a verified outcome; only sessions carrying one join the index. Learning is batch: the index is rebuilt every refit.every_n_outcomes captures, so a single new outcome does not move routing until the next re-fit or restart.

Session identity

Sessions are keyed on the tool's conversation id when the tool presents one. opencode sends X-Session-Id on every request — fresh on a new conversation, stable when one is resumed, and a fork carries x-parent-session-id. A new conversation id means a new session and a fresh routing decision on the new task. A resumed conversation reuses the model previously locked for it (see session_resume / fork_resume). Tools that send no conversation id (Claude Code, aider, plain clients) still group live traffic by (source_ip, user_agent) — one client, one open session — but they can now be resumed. Each session also records a digest of its conversation's opening prompt: the first system block and first user block, with the resolved repo bound in as a one-way digest. The working directory, date and git state are normalised out of the system block only — the one the host rewrites on every launch. The user's own turn is hashed as sent, because the client replays it verbatim on a resume and because normalising it would erase the file paths and identifiers that tell one task apart from another. Appending turns does not move the digest, so when such a conversation comes back after a restart it finds its own prior decision and re-serves that model (prefix_resume).

That digest is stable across a resume but not unique to a conversation — a new conversation opening with the same question, in the same repo, from the same client hashes the same. Two rules narrow that. The router consults it only when the request replays a conversation (two or more turns on the wire); an opening request carries a single turn and is always routed on its own merits, however familiar its question. And when the stored rows carrying one digest disagree on the model they served, it refuses to resolve and routes fresh — a cold route is cheaper than attributing one conversation's outcome to another.

That second rule is a partial guard, not a guarantee, and it is worth knowing where it stops: two conversations that collided on one digest and were both served the same model expose no disagreement to refuse on, which under the shipped session_cascade preset is the common case, since every cold session starts on the same cheapest model. The protection that carries the weight is the replay rule above plus the digest's own discrimination; the disagreement check is a backstop for the rest.

The reason tokens

Every decision names its rule. These are the values you will see in the header and in shunt explain:

Reason	What happened
cold_start	Not enough effective verified outcomes yet — fixed cheap model
stale_embedding_space	The corpus was embedded by a different embedder; neighbours refused until shunt reindex
cheapest_above_threshold	A model cleared the success bar with enough samples, and was cheapest
exploration_untested	Nothing cleared the bar; the cheapest model with no local history was picked
safe_fallback	Nothing cleared the bar and every model had been tried — cheapest model
exploration	Thompson sampling diverged from the greedy pick
exploration_exploit	Thompson sampling agreed with the greedy pick
conservative_fallback	An exploratory downshift was blocked for lack of banked slack
auto_escalation	A pending escalation directive overrode the base pick
escalation_floor	This task had already escalated to a higher rank, so the base pick was lifted back to it
session_resume	A resumed conversation reused the model locked for it (persisted across restarts) — non-policy, no selection propensity
fork_resume	A conversation resumed via a fork reused the parent conversation's model — non-policy, no selection propensity
prefix_resume	A conversation with no session id was matched to its own prior turns by its opening-prompt digest, and reused the model locked for it — non-policy, no selection propensity
benchmark_seed	A session seeded from the benchmark's measured outcomes — non-policy, never a learned choice
always_cheap / always_frontier	A fixed strategy is configured; no embedding, no query. Both are pinned controls: a verified failure never moves them
session_cascade	The cheap-first cascade preset picked its base model; unlike the two above, a verified failure can raise it a rung later (see the escalation reasons)

Under knn_semantic_cascade — the opt-in routing strategy, not the default — the base pick reports one of the kNN tokens above (cheapest_above_threshold, exploration_untested, safe_fallback, cold_start) — the ladder is a later decision, and it shows up as auto_escalation and escalation_floor on subsequent sessions. Every decision also carries the config id that produced it as strategy_id in its provenance, so an analysis can separate cascade sessions from the bare selection rule without reinterpreting these tokens.

Why it is built this way

• Nonparametric, so it works from the first labelled session. There is no model to train and no minimum dataset before the mechanism functions at all — it degrades to cold start instead of failing.
• Inspectable. Every decision can be traced to the specific past sessions that produced it. A learned scalar cannot be argued with; a neighbour list can.
• Cheapest-above-a-bar states the goal directly. The alternative — optimising a reward that trades quality against cost — needs an exchange rate between them that nobody can defend. A threshold you set is honest about being your choice.
• One decision per session, because the cache is per-session. Routing per turn would break the provider's prompt cache and cost more than any routing gain.
• Price order is the only ranking available at cold start. Capability order needs measurement you do not have yet; list price is always known.

Limitations

Read these before trusting a routing decision.

• The core signal is unproven on coding work. Ranking hard tasks from easy ones off a prompt embedding is the assumption the whole rule rests on, and on agentic coding it has not cleared our viability bar. See Results. The proxy, the cache-safety guarantee, and the escalation path do not depend on it; the kNN decision does.
• A thin neighbourhood is indistinguishable from a confident one. The exploration_untested branch returns the cheapest model, and nothing in the response says the neighbourhood was empty. Check shunt explain before concluding the router "decided" anything.
• Rank is price, not capability. The pool is ordered by list price, which is not a capability ordering — stepping up a rank can lower your pass rate. The measured per-model table is in Results.
• It sees a clipped, system-free slice of the task. A task described mainly in a system prompt is invisible to it, and a long task is truncated. The truncation rate is carried on each neighbour but the rule does not otherwise compensate for it.
• It cannot react to a task that changes mid-session. The model is locked. If your session drifts from a rename into a redesign, routing does not follow — that is what escalation is for, and escalation only acts at the next boundary.
• Learning is batch and global. One corpus, one index, no per-project or per-user models. A new outcome changes nothing until the next re-fit.
• Exploration costs money and quality while it runs. It is on by default so the router can gather evidence; if you want decisions on current beliefs only, turn it off.
• SWE tasks only. Both the corpus and the verified label assume a repository with a test suite. Nothing stops you routing other work through Shunt, but the decision has no evidence behind it and no verifier to grade it afterwards.
• This rule is off unless you turn it on, and on this corpus turning it on cost money. The default session_cascade never embeds. Selecting knn_semantic_cascade opens the same ladder on the kNN pick and, on the offline corpus, reached the same pass rate for $9.78 more cache-aware — see the shipped default, measured for the number and for the reason it may understate the routing model in live use.
• The cascade numbers are an offline replay from a fresh tree. Live, an escalated rung inherits the cheap rung's half-finished work and the whole prior conversation. What that does to quality is untested and its direction is unknown — the divergence, stated once.

Pros and cons at a glance

Why use this model. It is the cheapest thing that could plausibly work: no training step, no embedding server, no learned weights — a lookup plus a threshold, and it degrades gracefully to a fixed cheap model while the evidence is thin. On the routine ~80% of tasks a cheap model handles anyway, it spends almost nothing on prediction. The rationale is above and the limits below; the table is the summary.

Pros	Cons
Works from the first labelled session — degrades to cold start, never fails	The core signal (embedding → difficulty) is unproven on coding work
Every decision traces to concrete past sessions	A thin neighbourhood looks like a confident one — check shunt explain
Threshold states your cost/quality trade directly	Rank is price, not capability — stepping up can lower pass rate
Cache-safe by construction: one decision per session	Sees a clipped, system-prompt-free slice of the task
CPU-only embeddings; runs on any laptop	Cannot react to a task that changes mid-session
Batch, global, inspectable learning	Exploration costs money and quality while it runs
	SWE tasks only today
	Off by default — knn_semantic_cascade is opt-in, and it cost more than the cheap start here

Configure it

Strategy, k, the success threshold, min_samples, exploration, the live model list, the embedding model, and the re-fit cadence are all in Configuration → Tune the router. If this page's rule is not what you want — and on agentic coding its core signal is unproven, above — you are already there: session_cascade is the default, and it skips the prediction, starts on the cheapest model, and lets verified failure walk the ladder. It costs no embedding at all.

To turn this page's rule on, set router.strategy: knn_semantic_cascade: the same ladder, opened on the kNN pick instead of on the cheapest model. It was spelled knn, then knn_cascade, before the second rename, and those values still work with a warning — they still mean kNN routing, so they resolve to knn_semantic_cascade and never to the new default. Note what is not on offer: the kNN rule without the ladder. The shipped values are in src/shunt/config/router.yaml and src/shunt/config/embedding.yaml.

Difficulty routing (judge-labelled kNN)

The embedding-based kNN above is the semantic family: it embeds the task text and finds semantically similar tasks. This page's own evidence (embedding_signal.png) shows that embedding carries almost no difficulty signal (LOO R² −0.04, inside its null), so a second family routes on the thing the embedding is missing: a difficulty label, produced by an LLM judge (gpt-5.6-terra, $1/$6 list — probed at the same label quality as the claude-sonnet-5 anchor, LOO R² +0.027 vs +0.029). The committed per-task labels live in benchmark/routing/data/judge_difficulty.json (derived by derive_judge_difficulty.py from the gitignored probe artifacts). The three benchmark rows are knn_difficulty (single-shot control), knn_difficulty_cascade (difficulty pick + the session ladder) and difficulty_band_cascade (the "just the label + escalation" rule: same-difficulty-band members vote, cheapest in-band model whose pass rate clears the bar opens the ladder). Each row's cost includes the MEASURED per-task judge bill (judge_label_cost in strategy_summary.csv), because a judge call is a real cost of running that strategy. The measured bill is the provider-reported usage cost recorded by the probe at 2026-08-26 (~$0.0016/task, ~1.6× the $1/$6 list estimate) — it is the honest number the figures plot, and it is the reason the "cheaper than the anchor" framing rests on the list price, not on a measured cross-model comparison.

Measured verdict — none cleared the inference bar, so none is wired into the product. The pre-registered gate was: match session_cascade's pass rate within 5pp at lower model+judge cost. On the 184-task corpus:

row	pass rate	cache-aware cost	verdict
session_cascade (shipped)	96.74%	$28.71	baseline
knn_difficulty_cascade	96.74%	$29.01	equal quality, more expensive
difficulty_band_cascade	96.74%	$29.92	equal quality, more expensive
knn_difficulty (control)	75.54%	$1.80	== always-cheap; never escalates

The difficulty cascades' model cost is session_cascade's plus the judge bill: knn_difficulty_cascade's is exactly that ($28.71 + $0.30) — its difficulty pick opened on the cheapest rung on all 184 tasks, so it replays session_cascade's ladder verbatim — while difficulty_band_cascade's band pick opens above rung 0 on 12 tasks (picking qwen3.7-plus), which is the ~$0.90 more of model cost. The difficulty pick bought nothing on this corpus either way — the judge bill is pure addition. On the completed scoring basis — the house basis every frontier row uses, disclosed in this section's figure limits — the single-shot row collapses to always-cheap: deepseek-v4-flash's neighbour pass rate clears the 0.6 eligibility bar in every completed-matrix difficulty neighbourhood, so the rule's cheapest-eligible pick is deepseek on all 184 scored tasks. That collapse is a property of the completed scoring basis, not a corpus invariant: on the raw 200-task matrix the same pick routes 28 tasks to kimi-k3, and an oracle label (the task's own outcome) would move 45 of the 184 picks off deepseek — the oracle pick still lands on deepseek for 139 of 184 tasks — so a strong-enough label WOULD shift the routing, and the measured +0.027 LOO label is simply not strong enough. The falsification stands on both bases: on the raw 200-task matrix the difficulty cascade run loses quality against session_cascade (164 passes vs 170) at similar cost, so neither basis changes the verdict. This is the predict-then-cascade finding restated on a second axis: on this corpus the verification ladder dominates prediction — whatever the pick, the ladder reaches the frontier on verified failure, so a better pick cannot lower the bill at equal quality. For the same reason there is no step-count win for the kNN-difficulty cascade (its path is byte-identical to session_cascade's); the band cascade opens higher on its 12 tasks, so it escalates marginally more, not less.

What moving a difficulty row to inference would require (nothing blocks it mechanically; the value does). A judge provider and key; one judge call per task at the task boundary (label the first turn before the first routing decision — a cache-safe single decision per session, ~$0.0016 and the latency of one small completion); a difficulty index over the labelled outcome history (rebuild like the semantic index, from verified outcomes, so a label never votes without a measurement); and a cold-start fallback to session_cascade when no judge is configured or the index is empty. None of it is worth building while the measured result says the pick changes nothing. The revisit condition is genuinely per family: for the single-shot and band rules — the only ones whose pick can move — the judge label would need to approach the human tag's same-pipeline LOO R² (≈0.09, roughly the "~0.1" ceiling; current terra is +0.027); for the session-cascade rows no label quality helps, because the ladder reaches the frontier on verified failure regardless of where the pick opens.

Figures

Each figure below is the PNG committed under docs/assets/figures/routing/. The image carries its claim, its sample size and — where a reader could be actively misled — one red line. The rest is here.

How much of this is imputed, in one number. Several figures below disclose the monotone-imputation share. They now count the same grid — the completed matrix is 201 tasks by the 7 enabled models, 1407 cells — so the number is 410 of the 1375 completed cells (29.8%), of which 398 are filled pass=True; 32 cells are neither measured nor safely fillable and are excluded. One number on this page is not a disagreement with it:

• The coverage banner under the cost/quality figures says 24% of frontier cells are imputed. That is the strongest capability band alone: 93 imputed against 302 real in that band's 395 completions (7 unknown). It is the highest band, because the strongest band's model was measured on roughly two thirds of the corpus and the monotone ladder only ever fills upward, so the filling lands there.

Read evidence_basis.png for the per-band and per-strategy split; it is the authority the other figures point at.

0 of 4 reasoning knobs demonstrably fired — the flat contrasts are unmeasured

0 of 4 reasoning knobs demonstrably fired (ratio ≥ 1.15× output tokens) · all 4 pooled, including the 4 that never fired: +1.58pp on 190 pairs

Caveat. 4 of 4 rows show no manipulation — their flat contrasts are unmeasured, not null. Reading. Left: the manipulation check. For each (model, low arm, high arm) pair, the ratio of mean output tokens on the tasks where BOTH arms ran. A knob that was turned moves this well above 1; the red line is where nothing changed. Right: the paired pass-rate difference for the same pairs, with a 95% paired interval. A row whose knob did not fire is greyed and carries the label — its flat difference measures nothing.

What to look for. Do not read the right panel for a row that is grey on the left. The only row that supports any claim about reasoning effort is the one whose knob demonstrably moved.

Terms. manipulation check — evidence the treatment was applied at all, measured before any outcome. Output tokens are what a reasoning-effort setting mechanically controls. paired contrast — computed only on tasks where BOTH arms ran and NEITHER was censored, so a resource-limit stop cannot masquerade as a capability failure.

Notes. Arm coverage is sparse by design (p(arm|model) sampling), so a pair below the minimum count is greyed for sample size as well as for manipulation. deepseek-v4-flash high→max: output-token ratio 0.95x on n=42 pairs — never fired — not a null; paired Δ -4.8pp [-14.0, +4.5], cost Δ \$0.0000 glm-5.2 nothink→think: output-token ratio 0.93x on n=8 pairs — never fired — not a null; paired Δ +12.5pp [-10.4, +35.4], cost Δ \$0.0037 deepseek-v4-flash nothink→high: output-token ratio 0.92x on n=113 pairs — never fired — not a null; paired Δ +2.7pp [-3.6, +8.9], cost Δ \$-0.0012 deepseek-v4-flash nothink→max: output-token ratio 0.78x on n=27 pairs — never fired — not a null; paired Δ +3.7pp [-12.5, +19.9], cost Δ \$-0.0034

Limits. The ratio is over co-measured tasks only. A knob could fire on tasks neither arm shares, and this check would not see it.

The modelled cache saving is now the same size as the whole list-to-bill discount

4 models, 758 priced rows (channel mirrors merged by canonical identity) · cache-read price measured for 4/4 models and input share for 4/4; hit rate assumed at 90% · mean billed share 0.18 of list price vs a modelled 0.19 — residual -0.14 to +0.08 per model

Caveat. Agreement in size is not calibration: blue mixes every discount, green models caching alone. Reading. Left: per model, the share of the list-price bill that was actually charged. The blue bar is MEASURED — every row's real_cost over its estimated_cost in results.csv, which is the whole gap between list price and the invoice. The green diamond is the share the registry's cache-read price predicts would survive, priced at the corpus's measured input/output token mix; the faint grey rule beneath each bar sweeps the one input nothing here measures, the cache hit rate, from 100% (left cap) to 50% (right cap). The right-hand column repeats each row as billed → modelled with its row count. Lower is cheaper. Right: the switch tax as MEASURED — per scored strategy, the dollars its cache-aware bill came in under its naive bill, which is what repeating a model inside one task actually banked on this corpus.

What to look for. Read the DISTANCE between the blue bar and the green diamond. It is now small on every model, so caching alone is a large enough effect to account for essentially the whole discount. Read that as a SIZE agreement, not as a validation: the bar mixes every reason the invoice differs from list price, and two quantities landing on the same number cannot separate them. Then read the right panel for who banks any of it: only a strategy that RETRIES inside one task can, because the discount is scoped per task, so every single-shot policy banks exactly zero.

Terms. billed share — sum(real_cost) / sum(estimated_cost) over every measured rep-0 row for that CANONICAL identity, folding every channel mirror of the same weights into one model. 1.0 means the invoice matched list price. It mixes EVERY reason the two differ — caching, negotiated rates, provider-side discounts — not caching alone. registry prediction — 1 - input_share x hit_rate x (1 - cache_read_price / input_price), the per-model cache economics benchmark.runner.kill_gate now costs with. input share — the share of a model's spend that is INPUT tokens, from the measured in_tok / out_tok mix in results.csv priced at registry rates. This corpus is input-dominated, which is why the modelled saving is large. switch tax — naive cost minus cache-aware cost. A model switch forfeits the cached prefix, so the next turn is billed at full input price; the difference is what NOT switching banked. Scoped per task — one task is one session, so only a within-task repeat can bank it.

Notes. The cache-read prices come from the shipped registry (src/shunt/config/models.yaml), so a provider price change moves this figure rather than silently invalidating it. deepseek-v4-flash: billed 0.060 (n=360), registry predicts 0.141 [discount measured, input share 0.974 measured], hit-rate band 0.046–0.523 deepseek-v4-pro: billed 0.274 (n=201), registry predicts 0.139 [discount measured, input share 0.977 measured], hit-rate band 0.043–0.521 glm-5.2: billed 0.189 (n=87), registry predicts 0.233 [discount measured, input share 0.946 measured], hit-rate band 0.148–0.574 kimi-k3: billed 0.207 (n=110), registry predicts 0.262 [discount measured, input share 0.911 measured], hit-rate band 0.180–0.590

Limits. The hit rate is still assumed: no run in this corpus records a per-turn cache-hit ratio, so only the DISCOUNT and the INPUT SHARE are measured. The grey whisker is the whole range that assumption can move the green marker over. The blue bar and the markers are different quantities drawn on one axis deliberately. They now land close together, and that is NOT a calibration result: the bar also contains negotiated rates and provider-side discounts, so agreement in magnitude cannot attribute the discount to caching. The right panel is measured, but it is a COST DECOMPOSITION of runs that already happened, not an experiment: no live session in this corpus switched model mid-conversation, because the shipped router cannot. It says what repeating banked, never what switching would have cost in a world where the router could switch. The right panel inherits the left panel's assumed hit rate: the cache-aware column re-bills a repeat at the registry cache-read price under the same 90% assumption, so the DOLLARS it reports move with that assumption even though the repeats are counted.

The split count is a LOWER bound on contestable tasks — unsampled cells can only add

201 tasks x 7 columns, 747 of 1407 cells sampled (53.1%) · coverage 21/201 (glm-5.2) to 200/201 (deepseek-v4-pro) · contestable tasks in [61, 188] of 201 — 61 split is a FLOOR, not a ceiling Reading. Left: every task (row) against every measured (model, arm) column. Green is a pass, red a fail, grey never sampled — grey is NOT a failure. Middle: how many tasks each column was actually run on, so the uneven denominators are visible. Right: the task census. A split task — at least one pass and at least one fail — can already be won or lost by a routing decision. A task solved by every SAMPLED column has not been shown uncontestable, only untested: each one still has unsampled columns, and one failing column is enough to move it into the split slice.

What to look for. Read the right panel as a FLOOR, not a ceiling. The split count is what is contestable on the evidence in hand; the solved-by-all slice sits above it as tasks no column has yet been seen to fail, so the true count lies somewhere between the split count and split-plus-solved-by-all. Then read the middle panel before comparing any two columns anywhere else in this set — two columns are only fairly compared on the rows where both are non-grey, which is a different denominator from the pooled pass rates.

Terms. split task — at least one column passed and at least one failed sampling density — sampled cells over total cells. The matrix is sparse BY DESIGN — p(arm|model) sampling — so a low density is a budget decision, not missing data. contestable floor — the measured split count. It can only rise as unsampled cells are filled: a split task stays split, while a solved-by-all task joins it as soon as any unsampled column fails.

Notes. Columns are ordered by model price, then by within-model reasoning rank, so the grid reads left-to-right as cheap-to-expensive. The upper end of the interval counts the solved-by-all slice only. Solved-by-none rows are under-sampled too, but they could join only by an unsampled column PASSING, which nothing measured here supports, and a task no model solves is not value a router can capture. solved by all: 127; split: 61; solved by none: 12; no column sampled: 1 (the four slices sum to 201) all 127 of 127 solved-by-all tasks still have unsampled columns (1 to 6 of 7, median 4) — any one of them becomes contestable if an unsampled column fails the 12 solved-by-none tasks are under-sampled too (1 to 5 columns), but could join only by an unsampled column PASSING, so they are excluded from the interval

Limits. The grid is drawn from the RAW measured cache, not the imputed matrix, so a grey cell is genuinely unmeasured rather than filled. Every other figure in this set that quotes a pass rate scores the imputed matrix instead. At this sampling density 'solved by every sampled column' is NOT 'solved by every column', so the contestable count is an interval rather than a number. Only filling the grey cells pins it down.

The live frontier at cache-aware cost — cost is the uncertain axis

12 strategies over 181 scored tasks · Session-Cascade \$23.40 at 97.2% vs baseline \$94.37 at 95.0% (25% of the bill, cache-aware) · cost is the wide axis: Session-Cascade's naive total spans \$17.86–\$40.36 (95% bootstrap) · area under the live frontier 0.935

Caveat. scored on 181 coverage-selected tasks (dropped are harder); 8 of 12 are not selectable as router.strategy. Reading. THREE PANELS, THREE MAGNIFICATIONS, ONE PLANE. Panel A is the whole set — every strategy that carries a cost, over the full cost range, at full label, interval and hull treatment. Panel B redraws the outlined box on A at the same full width; panel C redraws the outlined box on B. Each box is joined to the panel that magnifies it by two dashed lines running from its lower corners to that panel's upper corners, so the A-to-B-to-C descent is drawn rather than asserted. The three x axes are three DIFFERENT scales, each ticked in round dollars across its own window, and only the bottom panel carries the axis label. In all three, x is total dollars spent over the scored task set on a log axis, because the strategies span two decades, and y is pass rate in percent. NEITHER BOX IS WRITTEN DOWN. B's is the DETAIL WINDOW, derived from the rows every time the figure is drawn: every strategy whose pass rate clears the best measured row's own Wilson lower bound, plus the fixed-frontier baseline and the oracle bound, widened by the room the names need. C's is the group whose MARKERS overlap inside B — measured on the rendered canvas, not chosen — widened to contain the full length of every context bracket it draws: the crowd's own, and any bracket-bearing neighbour the widened window pulls inside, so no dashed rule runs off the panel's right edge. EVERY PANEL NAMES EVERY STRATEGY ITS OWN WINDOW HOLDS, so a name repeats down the stack at each scale — that is what a magnification is, one point named twice, not two measurements. A strategy outside the detail window is therefore named on A, on the plane, at its own cost and pass rate: it is not part of the comparison B is for, and it is not a footnote either. The ONE exception is the group panel C magnifies. Above C its markers are a single blob, so a name printed there would land on a mark nobody can resolve, and those names appear only on C. Any name that could be placed on no panel at all would be listed in the notes; today there is none. THE FIGURE DEGRADES ON BOTH AXES, and says which case it is in. With nothing outside the detail window there is nothing for B to add, and the figure is A plus the magnified panel. With no overlapping markers there is nothing to magnify, and the figure is A plus the detail panel. With neither, it is the single plane it has always been. The layout note states how many levels the figure has. WHERE THE INTERVALS LIVE. Panels A and B carry them; panel C carries none. The vertical rule through each marker is its 95% Wilson interval on the pass rate, drawn BEHIND the markers and faintly: several strategies share one pass rate here, so their intervals are literally the same rule drawn several times, and at full weight they read as a picket fence of separate measurements. The thin horizontal line running from a marker ends at an open tick at the NAIVE per-call sum, so the gap between the two cost models is a drawn distance; the capped bar at that tick is the 95% bootstrap interval on the naive total, drawn on the naive statistic because that is the one it was computed for. Both cost marks appear on LIVE strategies only. At C's scale every one of these runs off the panel, so C leaves them behind and the panel caption says so. IN EVERY PANEL the marker sits at CACHE-AWARE cost — what a deployment is billed once a repeat of the same model banks its cached prefix. Marker SHAPE carries what a strategy is: circles and the orange diamond can run in production today, blue squares are blocked — no router.strategy value names them — an X is a control that must never ship, and a star is a bound that is unreachable by design. Marker FILL splits the blue squares: a SOLID square is blocked and nothing equivalent runs today, while a HOLLOW one would mark a mechanism that ships under another config surface with only its NAME blocked. NO ROW IS HOLLOW TODAY — the one that was, Session-Cascade, is now live as router.strategy: session_cascade, and every remaining blue square is solid, i.e. genuinely unrunnable. Only the live points enter the Pareto test and the shaded mixture region — a frontier anchored on a strategy the router rejects at boot describes an operating point nobody can buy. HOW A NAME IS PLACED SAYS WHAT WAS CROWDED. A name printed beside its marker had room there. A name lifted onto a level above or below the plane, on a vertical leader down to its own marker, belongs to a group whose NAMES could not all be printed where they sit — the leader is vertical and lands on the marker it names, so it cannot be misread onto a neighbour. Panel C labels its whole crowd that way, because at that scale nothing else can. Inside panel C, a DEPLOYABLE escalating strategy carries a context-transfer cost model, drawn as a dashed rule hanging under the marker it belongs to and joined to it by a thin line: the MARKER is what the benchmark measures, a fresh context on every rung, which is not a setting anyone can select; the shaded segment is context_transfer: summary, a band rather than a tick because how far a summariser compresses is not a constant; and the tick at the right end is context_transfer: full, which is what ships today. That rule asserts NO pass rate — it is horizontal, and its height carries no meaning at all. A deployable escalating row panel C does not contain carries no bracket on the canvas, and the notes name it and publish its numbers. The key is drawn ONCE for the whole figure, below the bottom panel, from every class the figure actually carries.

What to look for. A router earns its existence only by sitting ABOVE the shaded mixture region — below or on it, the same cost-quality point is reachable by flipping a weighted coin between two fixed policies.

Terms. live — the router may be configured with this strategy today — the set is derived from the product's own LIVE_STRATEGIES, not restated here. live Pareto — no other LIVE strategy is both cheaper and at least as good, on CACHE-AWARE cost. Bounds, controls and blocked strategies are excluded — none of them is a setting an operator can choose. This is the same cost model strategy_summary.csv's Pareto column uses, so the plane and the table cannot disagree. cache-aware cost — a repeat of the same model on a consecutive attempt is billed at the provider's cache-read rate rather than at full input price. Cascades re-serve one model by construction, so this is exactly where the two cost models part company. context transfer — what a deployment is billed once the model an escalation moves to is resent the conversation. The router config names the two settings the panel labels: summary resends a summarised prefix, full resends all of it, and full is the shipped default. The resent prefix is a cache MISS by construction — a new model receiving a prefix it has never seen — so it is charged at the full input rate, never at the cache-read rate. Context size is estimated as t = 2 x in_tok / calls. It is a cost model, it asserts no pass rate, and it rests on the token-complete subset named in the notes. detail window — the region of the plane panel B redraws at full width, and the box drawn on panel A. DERIVED from the rows every time the figure is drawn — the strategies whose pass rate clears the best measured row's own Wilson lower bound, unioned with the fixed-frontier baseline and the oracle bound so the two reference points can never fall out of the picture, then widened by measured label extents. Never a written-down box: costs move on every re-run, and a fixed one would end up pointing at empty canvas or cropping the comparison it exists for, with nothing failing to say so. magnified window — the region panel C redraws at full width, and the box drawn on panel B. Same points, same numbers, no extra data — and no intervals, which stay in the panel above. Which region it covers is decided from the RENDERED figure, not written down: it is the group whose MARKERS overlap on the parent panel, which is the one crowd no label placement can fix. A group whose names merely collide over separable markers is stacked on levels where it sits and earns no panel. The window is the overlapping group's own bounding box widened to contain everything panel C draws, including the full length of every context bracket it draws on: the crowd's own, and any bracket-bearing neighbour the widened window pulls inside. level — one full-width panel, and one step of magnification. The figure has three when something falls outside the detail window AND a group of markers overlaps inside it, two when only one of those holds, and one when neither does. The count is derived from the rows and the rendered geometry, and the layout note states it. mixture region — the upper convex hull of the LIVE points. Any point under it is reachable by probabilistically mixing two fixed policies. blocked — no router.strategy value names it, and on this corpus that now means genuinely unrunnable: the two remaining blocked cascades verify INSIDE one task, which breaks the one-decision-per-session cache-safety spine and is excluded by design rather than pending. They are excluded from the frontier, because the frontier ranks settings an operator can choose, and they are kept because they price what session cadence costs. Each row's blocker and path to live are in benchmark/routing/strategy_class.py.

Notes. The y axis is clipped to the data range AND the intervals drawn on it — the floor clears the lowest Wilson cap on the panel, never only the lowest marker, so no whisker is cropped — not to 0-100, and each panel is clipped to its own window. Every axis label carries the range it shows; the alternative was a figure on which every strategy is the same flat line. Each panel below the first shows one REGION of the panel above it, and the region is derived rather than chosen: the layout note records the windows it landed on, and the panel above always carries every strategy the window leaves out — named there, on the plane. A strategy is never dropped; the worst that happens is that it is compared at a coarser scale than the crowd below it. The marker and the hull use the same cache-aware cost column strategy_summary.csv decides Pareto on, so the plane and the table cannot rank strategies differently. Oracle: \$14.75 cache-aware / \$14.75 naive, 97.24% (n=181, bound — unreachable by design) Price-Cascade: \$22.27 cache-aware / \$22.27 naive, 97.24% (n=181, blocked — no router.strategy names it) kNN-semantic-cascade (within-task): \$25.47 cache-aware / \$25.47 naive, 97.24% (n=181, blocked — no router.strategy names it) Session-Cascade: \$23.40 cache-aware / \$28.21 naive, 97.24% (n=181, live, on the frontier) kNN-difficulty-cascade: \$23.70 cache-aware / \$28.51 naive, 97.24% (n=181, blocked — no router.strategy names it) Difficulty-Band-cascade: \$23.70 cache-aware / \$28.51 naive, 97.24% (n=181, blocked — no router.strategy names it) kNN-semantic-cascade: \$26.88 cache-aware / \$31.66 naive, 97.24% (n=181, live, but dominated) Always-Frontier: \$94.37 cache-aware / \$94.37 naive, 95.03% (n=181, live, but dominated) kNN-semantic: \$5.70 cache-aware / \$5.70 naive, 76.80% (n=181, control — never shippable) Always-Cheap: \$1.48 cache-aware / \$1.48 naive, 75.14% (n=181, live, on the frontier) kNN-difficulty: \$1.77 cache-aware / \$1.77 naive, 75.14% (n=181, control — never shippable) kNN-semantic-tier: \$7.84 cache-aware / \$7.84 naive, 66.30% (n=181, blocked — no router.strategy names it) context transfer on Session-Cascade: \$23.40 as the benchmark measures it (a fresh context per rung — not a config value), \$23.73–\$24.39 at context_transfer: summary, \$26.68 at context_transfer: full (the shipped default) — a cost model over measured tokens and registry input prices, computed on the token-complete subset (n=166), asserting no pass rate context transfer on kNN-semantic-cascade: \$26.88 as the benchmark measures it (a fresh context per rung — not a config value), \$27.20–\$27.84 at context_transfer: summary, \$30.08 at context_transfer: full (the shipped default) — a cost model over measured tokens and registry input prices, computed on the token-complete subset (n=168), asserting no pass rate frontier vs strategy_summary.csv Pareto: drawn-only Session-Cascade (both on cache-aware cost; a difference here is the live-only filter) selection: scored on 181/200 tasks selected by coverage; deepseek-v4-flash passes 73.7% here vs 26.3% on the 19 dropped (+47.4pp) — difficulty-biased, not a random sample equal-coverage via monotone imputation — 24% of frontier cells imputed (every strategy scored on n=181). Monotonicity holds on 88% of 200 multi-observed task(s) (measured, not assumed). NEARLY every imputed cell is filled pass=True at a median measured price (the monotone ladder has a fail branch, and 12 of 410 filled cells took it), for the router as well as for the baseline — see evidence_basis.png for how much of each strategy's number that is, and kill_gate.png's measured-only row for what survives when the projection is removed. layout: THREE levels of magnification, stacked full width — panel A carries every strategy over the full cost range. panel B redraws panel A's box at full width (\$6.54–\$134.15 at 89.3–100.0%), which is the detail window — every strategy whose pass rate clears the best measured row's own Wilson lower bound, plus the fixed-frontier baseline and the oracle bound, never written down. panel C redraws panel B's box at full width (\$20.96–\$31.95 at 97.0–97.5%), which is the group whose markers overlap in the panel above, measured on the rendered canvas, never written down. the pass-rate Wilson interval and the two cost marks are drawn in panel(s) A, B, and panel C carries neither — at that scale they run off the panel — carrying the context-transfer brackets instead. every panel names every strategy its own window holds, so a name repeats down the stack at each scale; the exception is the group panel C magnifies, whose markers are one blob above it and which is therefore named only there. layout: 2 strategies within \$1.48–\$1.77 at 75.1–75.1% have separable markers but names too wide to print beside them, so the names are stacked on levels, each on a vertical leader to its own marker — no second copy of the points is drawn layout: no panel could print Difficulty-Band-cascade — their markers are drawn and their numbers are in the per-strategy note rows above

Limits. A non-live point's number is still a real measurement — it is the CONCLUSION that is limited: no router.strategy setting reproduces it, so it may not anchor the frontier or a headline. It does NOT follow that the underlying capability is unavailable — a blocked row may measure a mechanism that ships in a different layer — and the per-strategy blocker in benchmark/routing/strategy_class.py says which case it is. A cascade point prices the LADDER's cost, not its per-rung quality. The shipped ladder's cheap intermediate rungs are measured separately against the base model on this same corpus, and are null or net-harmful there — see ladder_rungs.png. The cache-aware x position rests on an ASSUMED cache hit rate; only the per-model discount and input share are measured — see cache_economics.png for the range that assumption spans. The difficulty rows' x position includes the MEASURED per-task judge label cost (gpt-5.6-terra, ~$0.0016/task measured — folded into both cost columns and published as judge_label_cost in strategy_summary.csv). A judge call is one per task and never cached, so it is identical under both cost models; every other row carries no judge bill at all. The horizontal interval belongs to the NAIVE total and is not transplanted onto the cache-aware marker. The cache-aware ratio's own 90% bootstrap CI is published by the kill gate (cache cost is scoped per task, so a whole-task resample preserves it) — not transplanted onto this plot. Pass rates are scored on the coverage-completed matrix, whose imputed cells are all pass=True — see evidence_basis.png for how much of each strategy's number that is. The scored set is chosen by coverage, not at random: the collector runs the expensive tier only on the discriminating slice, so both axes describe a difficulty-biased sample. The subtitle carries the measured gap. The dashed context bracket is a COST MODEL, not a measurement. It re-prices the marker when the context an attempt ends holding is resent to the model escalation moves to — a cache MISS by construction, so it is charged at full input rate. It asserts NO pass rate: the bracket is horizontal because nothing here measures what carrying context does to quality. The canvas labels the bracket in CONFIG vocabulary; the cost model underneath is parameterised by alpha, the share of the context an attempt ends holding that is resent. The mapping is exact: context_transfer: summary is the alpha 0.1-0.3 band (a band, because a summariser's compression ratio is not a constant and one tick would assert a precision this model does not have), context_transfer: full is alpha = 1.0, and the marker itself is alpha = 0 — a fresh context on every rung, which is what the OFFLINE benchmark replays and what live inference never does. alpha = 0 is deliberately not offered as a config value: none is not a context_transfer setting, so the marker is the offline/live divergence made visible rather than a third option. A bracket is drawn on the DEPLOYABLE escalating strategies only. The summary table also carries alpha columns for the two within-task cascades, and they are deliberately not drawn: the model prices a SESSION-BOUNDARY handoff, and those two rows are blocked precisely because they retry inside one task, so they have no boundary to hand off at. A strategy that never escalates carries nothing and correctly shows no bracket at all. The bracket's context size is estimated as t = 2 x in_tok / calls, which assumes the prefix grows LINEARLY across a task's calls. Tool output and file reads do not arrive at a constant rate, so the error is one-sided in an unknown direction, and the bracket is an ordering of magnitudes rather than a quotable dollar amount. The bracket is computed on the token-complete subset — the tasks where every attempt on the realized path landed on a measured, token-bearing cell — which is strictly smaller than the scored set, because an imputed cell carries no token columns at all. What transfers to the plotted marker is the dimensionless surcharge FACTOR, not the subset's own dollars, and that transfer ASSUMES the subset carries context per dollar the way the scored set does. The subset is not a random sample of it — it is the tasks the collector happened to measure on every rung this strategy walked — so if those tasks escalate differently from the rest, the bracket is biased in the direction of that difference and nothing here corrects it. The subset size is published as the n in the bracket note row.

Shunt's default matches the frontier baseline's quality at a quarter of the bill

3 strategies + 1 bound of 12 scored strategies, 181 scored tasks · Session-Cascade \$23.40 at 97.2% vs Always-Frontier \$94.37 at 95.0% — 25% of the bill · cache-aware cost on a LINEAR axis; the full twelve-strategy plane is cost_quality_frontier.png

Caveat. Oracle is a hindsight bound, not a setting anyone can buy — and the scored tasks are coverage-selected, not random. Reading. One plane, four points, no legend — each point is named where it sits. Left to right is the total dollars a strategy spent over the whole scored task set, on a LINEAR axis; bottom to top is the share of tasks it passed. Cheap is left, good is up, so the best place to be is the top-left corner. The blue point is what Shunt routes with by default. The orange point is the baseline it has to beat: send everything to the strongest enabled frontier model. The grey circle at the far left is the opposite extreme — send everything to the cheapest model. The grey STAR is a bound, not a product: it is what a router that already knew each task's outcome would have paid, and no configuration reproduces it. The measuring bar across the bottom is the figure's whole point: it spans the horizontal distance between the blue point and the orange one, which is the money the router does not spend.

What to look for. Read the measuring bar, then check the two heights it connects. The blue point sits far to the LEFT of the orange one, at the same height or above it — cheaper for the same work is the claim, and a router that landed below and to the right of the baseline would have failed.

Terms. pass rate — share of the scored tasks a strategy solved cache-aware cost — what a deployment is billed once a repeat of the same model on a consecutive attempt is charged at the provider's cache-read rate rather than full input price hindsight bound — the price of choosing correctly with the answers already known — a floor on what any router could cost, never a setting an operator can select

Notes. Oracle: \$14.75 cache-aware, 97.24% passed, n=181 (hindsight bound — no router.strategy value reproduces it) Session-Cascade: \$23.40 cache-aware, 97.24% passed, n=181 (the shipped router.strategy default) Always-Frontier: \$94.37 cache-aware, 95.03% passed, n=181 (the baseline the kill gate is measured against) Always-Cheap: \$1.48 cache-aware, 75.14% passed, n=181 (the cheap floor) The four rows are read from the derived strategy summary at render time, so this figure and cost_quality_frontier.png cannot quote different numbers for one strategy. No interval is drawn. This figure states an ordering, not a precision — the intervals, the mixture region and the eight strategies left out are in cost_quality_frontier.png.

Limits. This is FOUR of the strategies the benchmark scores. The full plane — every strategy, its interval, the mixture region a router has to clear, and which rows are not selectable at all — is cost_quality_frontier.png, and this figure asserts nothing the parent does not. The y axis starts above zero and is labelled with the range it shows: the four points span about twenty points of pass rate, which on a 0-100 axis is a flat line. The cache-aware x position rests on an ASSUMED cache hit rate; only the per-model discount and input share are measured — see cache_economics.png for the range that assumption spans. Pass rates are scored on the coverage-completed matrix, whose imputed cells are all pass=True — see evidence_basis.png for how much of each strategy's number that is. The scored set is chosen by coverage, not at random: the collector runs the expensive tier only on the discriminating slice, so both axes describe a difficulty-biased sample.

Real problem text carries no routable signal; a human difficulty tag does

181 tasks, 200 permutations per null, k=20 · embedding R² -0.059 vs control +0.126, null 95% [-0.110, +0.005] · task-identity ceiling η²=0.564

Caveat. Falsified, not untested: the control clears the null on this same pipeline and n. Reading. Left: routing pass-rate against k. The solid blue line holds each task OUT of its own neighbour index (what a deployed router can do), the dashed orange line lets the task see itself (pure memorisation), the green line is the best single always-one-model policy that needs no router at all, and the grey band is the same rule on outcome-shuffled data. Middle: leave-one-out R-squared predicting each task's solve rate, for the embedded problem statement and for the human difficulty tag, each against its own shuffled null, with the task-identity variance ceiling marked. Right: same-repo minus cross-repo routing advantage, at both repo-size cutoffs.

What to look for. Read the middle panel first and read it as a pair. The control must clear its null or the instrument proves nothing; it does. The embedding must then clear its own null to support embed-and-kNN routing; it does not. That pairing is what turns a null result into a falsification rather than a coverage gap.

Terms. leave-one-task-out — the task being routed is removed from the neighbour index memorisation reference — the same rule with the task left IN its own index shuffled-outcome null — outcome rows reassigned to tasks at random, preserving each model's own pass rate and breaking only the text->outcome link variance ceiling — eta-squared of task identity over the (task, model) pass matrix: the most any per-task predictor could explain. positive control — the same pipeline with the task's human difficulty label as the similarity — a control on the MEASUREMENT, not a routing proposal.

Notes. The corpus embeds the real SWE-bench problem statement (median 1185 chars). The 106-char identifier label the earlier figures encoded is kept as a contrast row so the change in the input is visible, not asserted. ADMISSIBLE: positive control +0.9875 clears chance band (>+0.6001) AND destroyed-signal null +0.5188 is at chance (+0.5000±0.1001) — the instrument recovers a real signal and does not manufacture one from noise. NULL RESULT: the leave-one-out routing pass rate at k=2 is 0.7845, INSIDE the shuffled-outcome null band [0.7735, 0.8453] (null mean 0.8065, z=-1.29, 200 permutations) NULL RESULT: the embedded problem statement leave-one-out R² is -0.0593, INSIDE the shuffled-outcome null band [-0.1098, 0.0047] (null mean -0.0543, z=-0.16, 200 permutations) the human difficulty tag (positive control) leave-one-out R² is 0.1259, above the shuffled-outcome null band [-0.1142, 0.0063] (z=+5.58, 200 permutations) the diagonal advantage over 10 repos with ≥8 tasks is 0.0636, above the shuffled-outcome null band [-0.0163, 0.0278] (z=+5.02, 200 permutations) NULL RESULT: the diagonal advantage over 8 repos with ≥16 tasks is 0.0000, INSIDE the shuffled-outcome null band [-0.0174, 0.0284] (null mean 0.0014, z=-0.14, 200 permutations)

Limits. Pass labels come from the coverage-completed matrix, in which every imputed cell is filled pass=True, so all series including the null sit above what measurement alone supports. The COMPARISON between them is the readable part, not the level. One workload (SWE-bench-style tasks over a dozen repositories). Transfer to a different task distribution is not evidence this figure can give. 396/1267 scored cells (31.3%) are monotone-IMPUTED, not measured, and 390/396 of them are filled pass=True — imputation here is near-exclusively pass-filling (the ladder's fail branch fires rarely), so it almost never adds a failure. Every rate on this figure is biased UPWARD by that fill.

A third of the evidence is filled in, and nearly every filled cell is a pass

410 of 1375 completed cells (30%) are imputed, 32 unknown · worst band 3: 59% imputed · imputation is overwhelmingly pass-only (398 of 410 filled cells)

Caveat. Band 3 holds more imputed cells (225) than real ones (156). Reading. Left: per strategy, the share of scored DOLLARS billed to measured cells against projected ones. Middle: the same split on PASSES — the channel that decides every quality claim in this set. Right: per capability band, real against imputed against still-unknown cells; the bands are ordered weakest to strongest.

What to look for. Look for a strategy whose orange share is large on the PASS panel: its pass rate is that far from a measurement. Then look at the right panel for a band where orange exceeds blue — every comparison that crosses that band rests more on the imputer than on the benchmark.

Terms. imputed cell — filled by the monotone ladder: a model at least as strong as one that passed is credited with a pass, and a model no stronger than one that failed with a fail. The pass branch dominates; the subtitle counts how far. capability band — models grouped by derived capability rank, weakest band first. A band with more imputed than real cells is carried by the imputer. unknown — a cell neither measured nor safely fillable — excluded from scoring.

Notes. The dollar split is PATH-AWARE: a cascade that probed a projected cell on its way to a measured pick counts as projected, so the measured bar is measured end to end. Always-Cheap: \$1.42 measured + \$0.05 projected; 126 measured passes + 10 projected Always-Frontier: \$52.19 measured + \$42.18 projected; 85 measured passes + 87 projected Difficulty-Band-cascade: \$22.88 measured + \$5.63 projected; 161 measured passes + 15 projected Oracle: \$14.09 measured + \$0.65 projected; 163 measured passes + 13 projected Price-Cascade: \$17.94 measured + \$4.33 projected; 161 measured passes + 15 projected Session-Cascade: \$22.61 measured + \$5.60 projected; 161 measured passes + 15 projected kNN-difficulty: \$1.42 measured + \$0.05 projected; 126 measured passes + 10 projected kNN-difficulty-cascade: \$22.88 measured + \$5.63 projected; 161 measured passes + 15 projected kNN-semantic: \$5.66 measured + \$0.04 projected; 131 measured passes + 8 projected kNN-semantic-cascade: \$26.07 measured + \$5.59 projected; 163 measured passes + 13 projected kNN-semantic-cascade (within-task): \$23.26 measured + \$2.21 projected; 165 measured passes + 11 projected kNN-semantic-tier: \$6.95 measured + \$0.89 projected; 112 measured passes + 8 projected band 1: 317 real / 82 imputed / 3 unknown band 2: 190 real / 10 imputed / 1 unknown band 3: 156 real / 225 imputed / 21 unknown band 4: 302 real / 93 imputed / 7 unknown

Limits. Imputation is directional. Nothing here corrects the bias; it states its size so a reader can discount the pass rates by it.

Exploration costs more, buys no pass rate, and its learning benefit is unmeasurable

offline Direct-Method replay of the shipped policy — recorded outcomes, no live calls · dense slice 171 tasks × 3 models (deepseek-v4-flash, deepseek-v4-pro) (1 benchmark-only model(s) (not named, outside the inference-valid pool)), 171 scored by both arms, 20 seeds · exploration bills 1.68× the exploit-only run (worst seed 1.89×) · 95% percentile-bootstrap CIs over tasks

Caveat. Static matrix: an exploratory pull can never inform a later decision — cost only, learning benefit pinned to zero. Reading. A: the cost/quality plane. Each arm is one point — mean cost per task on x, pass rate on y — with 95% bootstrap intervals on both axes; the arrow runs from the exploit-only arm to the exploring one, and the box states the PAIRED difference, which is what this slice has the power to resolve. B: where the budget went. The orange curve is the running share of decisions that were exploratory as the replay proceeds, the dotted line is the router's own confidence-weighted explore counter at the end of the run, and the dashed line is the configured cap it is measured against.

What to look for. Read the boxed paired difference in A, not the two overlapping marginal intervals: look for the cost delta and whether the pass delta clears zero. Then read B for whether the policy is spending its budget at all — a counter far under its cap means the measured overhead is not the overhead of a saturated budget.

Terms. exploration — occasionally routing to a non-preferred model to learn its outcome exploit-only — the same shipped policy with exploration switched off paired difference — per-task gap between the arms, so shared task noise cancels explore_budget_frac — cap on the router's own confidence-weighted explore counter — neighbourhood costs, not realized spend, so it is not comparable to the measured spend ratio

Notes. The replay is EXACT, not simulated: on a fully dense sub-grid the recorded outcome is looked up and no request is sent. The realized spend ratio exceeding the cap is expected, not a bug — the cap counts the router's confidence-weighted neighbourhood costs, not realized spend. Unscorable cells are skipped and counted, never guessed. Intervals are percentile-bootstrap over tasks rather than Wilson: the exploring arm's per-task pass is a mean over stochastic seeds, not a Bernoulli count. Direct-Method replay on the fully-dense slice: 513 measured cells (full matrix 68.7% dense) Cells skipped as unscorable: 0 baseline, 0.0/seed exploration Realized explore/exploit SPEND 0.409 (worst seed 0.584); the router's own counter reached 0.359 of its 0.4 cap

Limits. The outcome matrix is static, so an exploratory pull can never improve a later decision: this measures exploration's COST with its learning benefit set to zero, the pessimistic half of the ledger — not a verdict on whether exploration pays. The dense slice is found greedily, not optimally, and comes from a single workload. How much of the corpus exploration left un-probed is NOT drawn: the replay report carries aggregate decisions, not the per-(task, model) probe record that question needs. The two marginal pass-rate CIs overlap ([69%, 82%] vs [70%, 82%]) — at 171 paired tasks only the paired difference separates the arms. NO FRONTIER ARM IN THIS SLICE: the dense sub-grid covers only deepseek-v4-flash, deepseek-v4-pro, plus 1 benchmark-only model(s) (not named, outside the inference-valid pool) — the priciest model here is \$2.25/Mtok against \$18.00 across all enabled models (kimi-k3, glm-5.2, 2 benchmark-only model(s) (not named, outside the inference-valid pool) are absent). The exploration overhead measured here is between CHEAP models and is a LOWER BOUND on the shipped policy's, where an exploratory pull can land on the frontier model. NOT IN THE LIVE POOL: 1 of the 3 dense-slice models are benchmark-only model(s) (not named, outside the inference-valid pool), measured here for mechanism but never shortlisted by the shipped router. The exploration overhead is between these slice models; a valid-only companion over the inference-valid subset is not drawn.

The pre-registered arm misses the 5pp bar on every basis; the shipped default clears it

paired Tango score at the pre-registered δ=5pp · n=181/94/20/181 · arms disagree on 39 of 181 on the widest basis: MDE ±8.6pp there, ±5.8pp at 10% discordance

Caveat. 1 of 1 clearing the bar rests on 4 discordant pairs; 3 of 4 rows are WORSE by more than the margin. Reading. Left: one row per evidence basis. The dot is the paired pass-rate difference (the kNN selection rule minus fixed-frontier) in percentage points, the whisker its 95% paired interval, and the dashed red line the pre-registered non-inferiority margin of -5pp. A row is green only when the Tango score test rejects H0 at that margin, red when the router is proven WORSE by more than the margin, grey when the data cannot tell. Right: the same tasks' total spend, baseline dot to router dot; a leftward arrow is a saving.

What to look for. Read both panels together, in that order. The left panel is the gate: a saving on the right is only admissible once the left one is green. On the pre-registered rows it is not — that arm's quality deficit is several times the margin and the whisker excludes it on every basis, so the spend reduction beside it is bought at a loss that was pre-registered as unacceptable rather than at equal quality. The bottom row is a different arm and a different verdict: the shipped default clears the bar, at about 6.1 times the pre-registered arm's bill and still under half the baseline's. It was not pre-registered, so read it as an observation, not as the gate being met.

Terms. the two router rows — The kNN-semantic row is the selection rule with the escalation ladder removed — the pre-registered verdict arm, and not a value router.strategy accepts. The Session-Cascade row is what a default install runs: one decision per session, cheapest-first, with the ladder on top, published without pre-registration. non-inferiority — H0: router quality <= baseline - delta, tested by the Tango score statistic on the discordant pairs. Rejecting it is positive evidence of equivalence; an overlapping confidence interval is not. evidence basis — Which tasks enter. completed includes monotone-imputed cells; measured keeps only tasks where neither arm billed a projected cell; gate sample is the subset benchmark.runner.kill_gate itself scores at its default N. MDE — The smallest true difference the design detects at 80% power, one-sided. For a paired test it is driven by the DISCORDANT rate, not by n alone, so it is quoted both at the observed discordance and at a reference 10% discordance.

Notes. The margin is read from benchmark.yaml:collect.noninferiority_margin, so the bar on the canvas is the one that was pre-registered rather than one chosen after seeing the result. The pre-registration named the kNN selection rule as the verdict arm, and it is kept there: repointing it after seeing the data would rewrite the registered test. But router.strategy defaults to session_cascade, so the shipped default is drawn beside it on its own row, labelled NOT pre-registered. The gap is a pre-existing defect the rename exposed, not one it created — the pre-registered arm adjudicates a configuration no operator can select. pre-registered kNN-semantic · completed (imputed): Δ=-18.2pp [-24.5, -12.0], inferior, b=3 c=36, router \$5.70 vs baseline \$94.37, MDE ±8.6pp (±5.8pp at 10% discordance) pre-registered kNN-semantic · measured only: Δ=-20.2pp [-29.8, -10.6], inferior, b=3 c=22, router \$3.74 vs baseline \$52.19, MDE ±13.2pp (±8.1pp at 10% discordance) pre-registered kNN-semantic · gate sample (N=20): Δ=-20.0pp [-37.5, -2.5], inferior, b=0 c=4, router \$0.11 vs baseline \$9.13, MDE ±24.9pp (±17.6pp at 10% discordance) Session-Cascade — shipped default, NOT pre-registered: Δ=+2.2pp [+0.1, +4.4], non_inferior, b=4 c=0, router \$28.21 vs baseline \$94.37, MDE ±2.7pp (±5.8pp at 10% discordance) equal-coverage via monotone imputation — 24% of frontier cells imputed (every strategy scored on n=181). Monotonicity holds on 88% of 200 multi-observed task(s) (measured, not assumed). NEARLY every imputed cell is filled pass=True at a median measured price (the monotone ladder has a fail branch, and 12 of 410 filled cells took it), for the router as well as for the baseline — see evidence_basis.png for how much of each strategy's number that is, and kill_gate.png's measured-only row for what survives when the projection is removed.

Limits. The cost panel is TOTAL SPEND over the scored task set, at naive prices. The gate's real criterion is cache-aware cost, which the gate bootstraps per task — cache cost is scoped per task (one task is one session), so a whole-task resample preserves within-task adjacency — and publishes as a 90% CI in the tracked verdict artifact. See cache_economics.png for how far the assumed hit rate moves that ratio.

The router's one input does not predict outcomes; a human difficulty tag does

181 tasks x 7 models, k=20, leave-one-out · Brier skill -0.055 vs null 95% [-0.096, -0.012] · human-tag positive control +0.078

Caveat. Falsified, not untested: the control fires on this same pipeline while the embedding sits inside the null. Reading. Left: the reliability diagram. x is the weighted neighbourhood success rate the shipped rule computes for a (task, model) pair; y is how often that pair actually passed. A calibrated predictor tracks the dashed diagonal. Bars carry 95% Wilson intervals and the count in each bin. The red line is the shipped 0.6 eligibility threshold. Middle: how those scores are distributed, so the threshold's position is visible geometry rather than a claim. Right: Brier skill score against the marginal pass rate — above zero means the neighbourhood rate beats simply knowing how often each model passes — with the shuffled-outcome null band and the human-difficulty-tag positive control on the same axis.

What to look for. Look at the right panel first. The positive control must sit above the null band, or the instrument proves nothing either way. It does. Then look at where the observed bar sits: inside the band is a falsification, not a coverage gap.

Terms. weighted success rate — sum(similarity x outcome) / sum(similarity) over the k nearest OTHER tasks, the quantity SelectionRule thresholds at 0.6. Brier skill score — 1 - Brier(neighbour rate) / Brier(per-model base rate). Zero means the neighbourhood adds nothing over the model's marginal pass rate. positive control — The same pipeline run with the task's human difficulty label (easy/medium/hard) as the similarity, so a task's neighbours are the tasks a human called equally hard. It is a control on the MEASUREMENT, not a routing proposal.

Notes. Every rate is leave-one-out: a task is never its own neighbour, so a task cannot predict itself. ADMISSIBLE: positive control +0.9875 clears chance band (>+0.6001) AND destroyed-signal null +0.5188 is at chance (+0.5000±0.1001) — the instrument recovers a real signal and does not manufacture one from noise. bin [0.2,0.4): predicted 0.372, observed 1.000 (n=3) bin [0.4,0.6): predicted 0.537, observed 0.647 (n=139) bin [0.6,0.8): predicted 0.712, observed 0.755 (n=465) bin [0.8,1.0): predicted 0.897, observed 0.862 (n=645)

Limits. The neighbour weight is similarity only. The shipped rule also multiplies by each neighbour's verification confidence, which is 1.0 for every cell in this corpus, so the two coincide here and could diverge on live traffic. 396/1267 scored cells (31.3%) are monotone-IMPUTED, not measured, and 390/396 of them are filled pass=True — imputation here is near-exclusively pass-filling (the ladder's fail branch fires rarely), so it almost never adds a failure. Every rate on this figure is biased UPWARD by that fill.

The evidence-backed pool, and the rung the price-ranked ladder still skips

paired on the overlap of scored default-arm runs · exact paired-exchangeability null · base deepseek-v4-flash · rank_shortlist=3 visits 2 of 3 live targets · visited: deepseek-v4-pro (+0.153), glm-5.2 (+0.155) · skipped: kimi-k3 (+0.236) · not live — benchmark-only, outside the inference-valid pool (never served): qwen3.7-plus, gpt-5-mini, kimi-k2.5

Caveat. Observational overlap per pair, not a ladder replay: no logged session walked these rungs in sequence. Reading. Left: for each candidate escalation target, the paired difference in resolve rate against the cheap base model, computed only on challenges where BOTH models have a scored default-arm outcome. The dot is the point estimate, the dark whisker the paired percentile bootstrap over challenges, and the pale whisker behind it the exact paired-exchangeability null band, so a dot inside the pale band is indistinguishable from chance. Rows are ordered by list price, cheapest at the bottom, which is the same order the ladder ranks by. Right: the same rows against the SHIPPED LIVE POOL (read from src/shunt/config/router.yaml's models list) — a filled marker is a rung the ladder actually visits, a hollow one a live rung the shortlist jump skips, and a hollow square is a benchmark target the shipped router no longer routes to (it stays measured, never served). The visit sequence is drawn as a stepped path and the shortlist's jump as a single long arrow.

What to look for. Read panel A first and ignore the ladder: only two targets' intervals clear zero on the helpful side, and one of them is the most expensive rung measured. Then read panel B on the same rows: the shipped pool no longer holds the flat-to-harmful rungs (they are drawn NOT-LIVE), so the ladder's bought rungs are now the ones the evidence supports — and the remaining defect is visible on the canvas: the price-ranked walk can still jump over a net-helpful rung when a pricier frontier model's slot falls inside the shortlist. A row whose dark interval overlaps its own pale null band is unmeasured at this n, not shown to be neutral.

Terms. helps — base failed the challenge, target resolved it hurts — base resolved the challenge, target failed it delta — target resolve rate minus base resolve rate on the shared challenges == (helps - hurts) / n exact null — the two-sided paired randomization test, in closed form — no Monte Carlo, no seed rung — a model the ladder can step to; the shortlist walks the cheapest ranks one at a time and then jumps to the top rank not live — a benchmark target absent from router.yaml's models: list — benchmark-only and outside the inference-valid pool, measured for evidence, never chosen for live inference

Notes. deepseek-v4-pro at 3.1x base: n=190, helps 33, hurts 4, delta +0.1526 [+0.0947, +0.2105], exact null [-0.0579, +0.0579], p 1.1e-06, NET-HELPFUL qwen3.7-plus at 3.8x base: n=87, helps 6, hurts 3, delta +0.0345 [-0.0345, +0.1034], exact null [-0.0575, +0.0575], p 0.51, INDISTINGUISHABLE gpt-5-mini at 5.4x base: n=190, helps 4, hurts 36, delta -0.1684 [-0.2263, -0.1105], exact null [-0.0632, +0.0632], p < 1e-06, NET-HARMFUL kimi-k2.5 at 8.6x base: n=121, helps 8, hurts 10, delta -0.0165 [-0.0826, +0.0496], exact null [-0.0661, +0.0661], p 0.81, INDISTINGUISHABLE glm-5.2 at 13.8x base: n=84, helps 14, hurts 1, delta +0.1548 [+0.0714, +0.2381], exact null [-0.0833, +0.0833], p 0.00098, NET-HELPFUL kimi-k3 at 42.9x base: n=110, helps 29, hurts 3, delta +0.2364 [+0.1455, +0.3273], exact null [-0.1091, +0.1091], p 2.6e-06, NET-HELPFUL the shortlist jumps over kimi-k3, a target whose interval clears zero on this corpus

Limits. Overlap only: each row is scored on the challenges both models were run on, and those sets differ by row, so the rows are not scored on one common set and their deltas are not directly comparable to each other. Coverage is opportunistic, not assigned: which challenges each model was run on was not randomized, so a target measured on an easier overlap looks better for free. Default reasoning arm only. A rung the ladder reaches at a raised effort arm is not this row. This measures TARGETS, not the ladder: a real ladder pays for a rung only after a verified recurrence, so the cost of a harmful rung is not the whole of its price. One base, one corpus. A rung that is net-harmful here is net-harmful on this corpus's task mix, which is SWE-bench-derived and not your workload. The live pool's price order — and therefore which rung the shortlist jump skips — depends on frontier rows whose prices are research-estimated, not live Requesty listings.

What the bound's quality costs, and which of those prices you may actually pay

7 of 12 strategies reach 97.24% ± 1pp · cheapest live \$28.21 · cheapest blocked \$22.27 · bound \$14.75 · blocked strategies hold 44% of the live-to-bound headroom · cost as billed when each run happened

Caveat. 44% of the headroom sits behind a blocker, so it is a to-do, not a measured saving. Reading. Left: every strategy that reaches the bound's pass rate within one percentage point, as a DOT at its total spend on a log axis, cheapest at the bottom, coloured by class. The axis is logarithmic, so only the dot's POSITION carries the price; the grey rule behind it is a reading guide and its length means nothing. A GREEN dot is a price you can pay today — router.strategy names it. The blue bracket is the span between the cheapest LIVE way to buy that quality and the cheapest BLOCKED one — engineering work, not physics. The red bracket is the span from there down to the bound, which no strategy of any class can cross. The subtitle carries how the two divide, because that split moves with the data and this title deliberately does not claim it. Right: how many strategies each class contributes and the best pass rate it reaches, with the reason that class is kept in the corpus.

What to look for. Read the CHEAPEST GREEN dot first — that is what this quality actually costs a deployment, and if there is no green dot in the band the subtitle says so instead of pricing an empty set. Then read the two brackets against each other. A large blue span and a small red one means the shipped router's deficit is a backlog item; the reverse means the corpus has been squeezed and the remaining distance is a property of the models, not of the routing. Neither bracket is a result you can deploy — the whole point of separating the classes is that only the green dots are purchasable.

Terms. bound — a strategy that reads the query task's own realised outcome. Unreachable BY DESIGN; it exists to say how much is left, never to be shipped. blocked — no router.strategy value names it, with the reason and a path to live recorded in benchmark/routing/strategy_class.py. A costed to-do, not a result — but the to-do is sometimes only the NAME, not the mechanism. control — exists so the other numbers mean something — a strategy the measurement is compared against, which must never ship. at the bound's quality — pass rate within 1.0pp of the best bound's. A cost comparison across the band is therefore an equal-quality comparison to within that tolerance.

Notes. This axis is TOTAL SPEND over the shared scored task set, at NAIVE prices — the raw sum of what each attempt was billed, repriced only when the subtitle says so. cost_quality_frontier.png ranks on the CACHE-AWARE total instead, which prices a repeat-model discount the naive sum does not. The two are different cost models, so a span read off this figure is NOT comparable with one read off that one: a cascade that re-hits one model is cheaper there than it is here, and the gap is the discount, not a different strategy. Oracle: \$14.75, 97.24% (bound) Price-Cascade: \$22.27, 97.24% (blocked) kNN-semantic-cascade (within-task): \$25.47, 97.24% (blocked) Session-Cascade: \$28.21, 97.24% (live) kNN-difficulty-cascade: \$28.51, 97.24% (blocked) Difficulty-Band-cascade: \$28.51, 97.24% (blocked) kNN-semantic-cascade: \$31.66, 97.24% (live)

Limits. The blue bracket is what the BLOCKED strategies measured here would buy IF their blockers were removed, and the blockers are not one kind of thing: some are structural (cache-safety, an offline-fit input) and the live mechanism replacing them may land nowhere near this span, while another is only that no router.strategy value names a mechanism that already ships in a different layer. Read each blocker in benchmark/routing/strategy_class.py before treating this span as unbuilt work. Only strategies inside the quality band appear on the left panel. A cheap strategy that gives up quality is not a smaller version of this gap — read the frontier figure for that trade. The bound reads realised outcomes on the SAME corpus it is measured on, so it is a ceiling for this task set, not a general one.

The saving is a cheaper tariff, not a better prediction

price 157.1% + interaction -52.0% = 105.2% mechanism, volume -5.2% (63 both-pass tasks) · regret quoted at gamma=0.1; ordering IDENTICAL across gamma 0.001-0.33

Caveat. kNN-difficulty, kNN-semantic-tier carry MORE regret than always-cheapest. Reading. Left: the cost saving of the router against fixed-frontier, split by Oaxaca-Blinder into a price effect (cheaper tokens), a volume effect (fewer tokens) and their interaction, over the tasks where BOTH arms passed. Middle: cumulative regret against the hindsight oracle, lower is better, with 95% bootstrap intervals where the summary carries them; bar colour is the strategy's class — green runs live today, blue is blocked, orange is a control that must never ship, grey is a bound no router can reach — and a red outline marks a bar a fixed always-cheapest policy already beats. Right: the same ranking recomputed across three orders of magnitude of the cost/quality exchange rate, coloured the same way.

What to look for. Read the left panel for what routing is actually doing — if price dominates, the value is in the price list, and a fixed cheap policy captures most of it without any prediction. Then read the right panel: a flat set of lines means the middle panel's ordering does not depend on the exchange rate nobody can defend.

Terms. price effect — the saving from billing the SAME token volume at a cheaper model's rate. volume effect — the saving from producing FEWER tokens at the same rate. regret — reward the hindsight oracle collected that this strategy did not; reward is 1 for a pass, 0 for a fail, minus gamma x cost in dollars. arm oracle — hindsight over the reasoning ARM as well as the model — the ceiling for reasoning-effort routing given the arms actually sampled.

Notes. The decomposition is computed only over tasks where both arms were measured (scorable — never a coverage-gap, censored, or imputed fill) and both passed, so it is a cost comparison at genuinely equal quality on those tasks. Always-Cheap: regret 38.6727 Always-Frontier: regret 11.9620 Arm-bandit: regret 27.5229 Arm-oracle: regret 0.1057 Difficulty-Band-cascade: regret 1.3757 Oracle: regret 0.0000 Price-Cascade: regret 0.7520 Session-Cascade: regret 1.3461 kNN-difficulty: regret 38.7023 kNN-difficulty-cascade: regret 1.3757 kNN-semantic: regret 36.0955 kNN-semantic-cascade: regret 1.6908 kNN-semantic-cascade (within-task): regret 1.0722 kNN-semantic-tier: regret 55.3095 Every strategy holds the same rank at every gamma on the grid, so the ladder's ordering is a statement about quality-at-cost and not about the exchange rate.

Limits. The price decomposition treats a cheaper model as a cheaper way to get the SAME outcome. Measured per-rung, the cheap intermediate targets do not deliver the base model's outcome on this corpus, so the price term is an upper bound on what cheapness buys — see ladder_rungs.png. The bandit is an illustrative inline learner drawn for this figure only, not a shipped routing strategy. It shows that a naive learner loses here; it does not show that every learner would. The arm series exist only where more than one arm per model was sampled; the coverage is sparse by design.

The frontier is not one set — it changes with the axis you buy on

4 configurable strategies over 5 dimensions, 12 rows drawn · best row holds 4 of 5 — no strategy holds all five · the shipped default (Session-Cascade) holds 2 of 5 · no row excluded for a missing value

Caveat. Only configurable strategies enter a frontier; bounds, controls and blocked rows are drawn, never ranked. Reading. FIVE PANELS, ONE PLANE EACH, AND A MATRIX. Panels A-E share the same y axis — pass rate over the scored corpus — and differ only in what x costs: dollars (log, because the strategies span two decades), provider calls, the p95 session tail, output tokens, and the coefficient of variation of per-task cost. Panels B and D are PER TASK and are counted only over the paths that carry real counts, because an imputed cell records no calls and no tokens; panel C is counted on every scored task. THE TWO DOLLAR AXES ARE NOT ONE COST MODEL: panel A is the cache-aware total, panel E is the dispersion of the NAIVE per-task cost, and the panel labels say which is which. The green line in each panel joins that panel's Pareto frontier, computed on that panel's own two axes. MARKER SHAPE CARRIES CLASS, never colour alone: a circle can be configured today, a blue square is blocked, an X is a control that must never ship, a star is a bound unreachable by design. Only circles enter a frontier, so a hindsight oracle is drawn beside the strategies it bounds without ever being ranked as one of them. Only the configurable rows are named on the plane; the rest are read off their shape and the key. FILL, not hue, carries membership: a marker is filled only when it is on that panel's frontier, and only a circle can be — every other mark is an outline, which is also what keeps a frontier circle visible where two rows land on the same point. PANEL F IS THE FINDING. One row per configurable strategy, one column per dimension, and a filled cell where that strategy is on that dimension's frontier. Read a ROW to see how much of a strategy's optimality survives changing the currency; read a COLUMN to see who you would buy if that column were the only thing you cared about. The count at the right of each row is how many of the five it holds. THE DOTTED RULE across A-E is the pass rate most of the drawn rows share EXACTLY; it is named in panel C's right margin and explained in the notes, and along it the panels rank on x rather than trading quality for it.

What to look for. Look for a row of panel F that is filled all the way across — a strategy that is optimal whatever you are buying. There is none. Then read the row of whichever column you actually pay in, because that, and not the cost panel alone, is the frontier your deployment sits on.

Terms. frontier (per dimension) — no other LIVE strategy is at least as good on BOTH that panel's axes and strictly better on one. Computed per panel by the same routine the summary's Pareto column uses, over that panel's axis pair rather than a fixed one. live — the router may be configured with this strategy today — derived from the product's own LIVE_STRATEGIES, never restated in the benchmark. session tail (p95) — the 95th percentile of sessions spent per task. A cascade buys its pass rate by re-attempting, and this is what the worst tasks pay for that. cost CV — standard deviation of per-task cost divided by the mean. Low means a predictable bill; high means the total is carried by a few expensive tasks. excluded — a row missing a value on a panel's axis is dropped from THAT panel's frontier rather than read as zero — a missing value coerced to zero is un-dominated by construction, which certifies 'measured nothing' as optimal.

Notes. Every panel ranks the CONFIGURABLE rows only. Bounds, controls and blocked rows are drawn at their measured position and never enter a frontier, because a frontier anchored on a point no operator can select describes an operating point nobody can buy. Calls, output tokens and the session tail are the countable quantities latency is made of, not a measured latency. Nothing in this corpus times a request. Rows can land on EXACTLY the same point on these axes — a blocked cascade shares the shipped default's call count, session tail and output-token total — so a marker may carry more than one strategy. Every name is lifted onto a level above its own marker on a vertical leader, which is what keeps a shared point from being read as one row. TIE: 7 of the 12 drawn rows sit at exactly 97.24% (176 passes of 181) — the hindsight bound stops here too, so the misses are tasks no model in the matrix solved. Along this line the panels RANK on x; they do not trade quality for it. billed cost: frontier = Always-Cheap, Session-Cascade provider calls: frontier = Always-Frontier, kNN-semantic-cascade session tail (p95): frontier = Always-Frontier, Session-Cascade, kNN-semantic-cascade output tokens: frontier = Always-Cheap, Always-Frontier, kNN-semantic-cascade cost CV: frontier = Always-Frontier, kNN-semantic-cascade Always-Frontier: on 4 of 5 frontiers kNN-semantic-cascade: on 4 of 5 frontiers Always-Cheap: on 2 of 5 frontiers Session-Cascade: on 2 of 5 frontiers counted paths for panels B and D — Always-Cheap: 171 of 181 scored task(s) counted paths for panels B and D — Always-Frontier: 94 of 181 scored task(s) counted paths for panels B and D — Difficulty-Band-cascade: 166 of 181 scored task(s) counted paths for panels B and D — Oracle: 168 of 181 scored task(s) counted paths for panels B and D — Price-Cascade: 166 of 181 scored task(s) counted paths for panels B and D — Session-Cascade: 166 of 181 scored task(s) counted paths for panels B and D — kNN-difficulty: 171 of 181 scored task(s) counted paths for panels B and D — kNN-difficulty-cascade: 166 of 181 scored task(s) counted paths for panels B and D — kNN-semantic: 173 of 181 scored task(s) counted paths for panels B and D — kNN-semantic-cascade: 168 of 181 scored task(s) counted paths for panels B and D — kNN-semantic-cascade (within-task): 170 of 181 scored task(s) counted paths for panels B and D — kNN-semantic-tier: 173 of 181 scored task(s)

Limits. TWO COST MODELS SHARE THIS CANVAS. Panel A's dollars are the cache-aware total (TotalCost_cacheaware, the column cost_quality_frontier.png ranks on); panel E's CV is the dispersion of the NAIVE per-task cost (TotalCost), because the cache-aware discount is published as a row total and there is no per-task cache-aware series to take a CV of. The panel labels carry the difference; a reader must not read A and E as two views of one bill. The y axis is the same imputation-biased pass rate every figure in this set uses: every filled cell is a pass, so all five frontiers sit on quality numbers biased upward by the share evidence_basis.png publishes. The dollar axis rests on an ASSUMED cache hit rate, as cost_quality_frontier.png's does. THE OTHER FOUR ARE COUNTED, NOT MODELLED — but two of them are counted over a SUBSET, not over the corpus: an imputed cell records no calls and no tokens at all, so panels B and D are per-task rates over each row's counted paths only, and the note below gives how much of each row that is. Panel C is counted on every scored task, because a session count comes from the strategy's own ladder rather than from a cell. Nothing here is scaled up to a corpus total: an unrun cell contributes nothing, never a zero. The five dimensions are NOT independent, and this is the size of it — strongest Spearman rho over the 12 drawn rows: provider calls/session tail (p95) +0.924; session tail (p95)/output tokens +0.924; provider calls/output tokens +0.907. A cascade that re-attempts spends more calls, more output tokens and more sessions at once, so those axes move together. provider calls, cost CV select the IDENTICAL front (Always-Frontier, kNN-semantic-cascade). The figure claims only that MEMBERSHIP differs across them, which is a statement about the ordering, not a claim that the axes measure five separate things.

Only the measured live pool is inference-valid; the rest are named, not dropped

canonical weights identity · provider is a label, never part of the name · criteria: live, triage pass (KEEP/EXCEPTION), capability measured, coverage, paid channel · 4 inference-valid of 34 canonical identities named by committed evidence (20 paid, 14 free); 27 evidenced (≥1 measured cell in either channel) · coverage floor K=20 measured default-arm cells (declared default arm, sole-arm fallback)

Caveat. invalid = not selected on committed evidence, not that the model is weak Reading. Panel A: one row per CANONICAL weights identity the committed evidence names (paid and free channels merged on model_version, never on a provider-prefixed listing id), one column per criterion — channel (paid or free), the serving provider(s), selected for inference (in the live pool), triage pass (KEEP or EXCEPTION), capability rank measured rather than a price prior, coverage at or above the K default-arm cell floor (the cell count is printed), and a paid channel. The last column is DISTINCT verified challenges covered (not the same quantity as the default-arm cell count in the coverage column: a challenge can contribute several cells), shaded by how much of the corpus the model holds. A green cell passes, a red one fails; a free-only row's triage and capability cells are NOT APPLICABLE and read 'n/a' on a neutral cell, because a collection-only identity is never triaged or capability-ranked, while its paid-channel cell stays 'no' — the fact that puts it outside the pool. The subtitle separates the roster ('canonical identities named') from 'evidenced' — at least one measured cell in either channel — because the roster also names slots no committed measurement covers. Panel B counts the identities by why they are out, keeping the free channel, the collection-only promo probes, the unmeasured live slots and the triage DROPs distinct.

What to look for. Read the valid rows first — they are the only models the other routing figures show. Then read panel B to see that the excluded models are excluded for DIFFERENT reasons: a dominated benchmark model, an unmeasured frontier slot, and a collection-only probe are not the same kind of absence and must not be collapsed into one.

Terms. canonical identity — the weights slug (model_version), so the same weights served by several providers are ONE row and a provider prefix or -free marker is never a name inference-valid — in the packaged live pool AND triage pass (KEEP or EXCEPTION) AND capability rank measured AND coverage >= K measured default-arm cells (declared default arm, sole-arm fallback) AND a paid channel evidenced — at least one measured cell in either channel (paid or free). NARROWER than the roster: the roster also names live slots and collection-only listings with no committed measurement, so 34 identities named is not 27 evidenced capability measured — the derived rank clears the confidence gate (K cells, CI width W, two qualifying peers); otherwise the model sits at its price-implied slot as a price prior collection-only — a row the benchmark may collect but never enable or route: the free overlay channel or a priced -explabs promo probe

Notes. deepseek-v4-flash: VALID — inference-valid: live, triage pass, capability measured, coverage OK, paid (channel paid, providers deepseek, explabs, triage KEEP, capability measured, 219 cells, 201/500 verified challenges) deepseek-v4-pro: VALID — inference-valid: live, triage pass, capability measured, coverage OK, paid (channel paid, providers deepseek, explabs, triage KEEP, capability measured, 201 cells, 201/500 verified challenges) glm-5.2: VALID — inference-valid: live, triage pass, capability measured, coverage OK, paid (channel paid, providers requesty, triage KEEP, capability measured, 84 cells, 77/500 verified challenges) kimi-k3: VALID — inference-valid: live, triage pass, capability measured, coverage OK, paid (channel paid, providers explabs, requesty, triage KEEP, capability measured, 117 cells, 103/500 verified challenges) claude-fable-5: invalid — triage UNMEASURED-EXCEPTION — slot does not clear the frontier (channel paid, providers requesty, triage UNMEASURED-EXCEPTION, capability not-ranked, 0 cells, 0/500 verified challenges) claude-fable-5.1: invalid — collection-only promo probe — never enabled or routed (channel paid, providers explabs, triage not-triaged, capability not-ranked, 22 cells, 22/500 verified challenges) claude-opus-4-6: invalid — collection-only promo probe — never enabled or routed (channel paid, providers requesty, triage not-triaged, capability not-ranked, 0 cells, 0/500 verified challenges) claude-opus-4-8: invalid — triage UNMEASURED-EXCEPTION — slot does not clear the frontier (channel paid, providers requesty, triage UNMEASURED-EXCEPTION, capability not-ranked, 0 cells, 0/500 verified challenges) claude-sonnet-5: invalid — collection-only promo probe — never enabled or routed (channel paid, providers requesty, triage not-triaged, capability not-ranked, 0 cells, 0/500 verified challenges) gemini-3.1-pro: invalid — triage UNMEASURED-EXCEPTION — slot does not clear the frontier (channel paid, providers requesty, triage UNMEASURED-EXCEPTION, capability not-ranked, 0 cells, 0/500 verified challenges) glm-5.3: invalid — collection-only promo probe — never enabled or routed (channel paid, providers explabs, triage not-triaged, capability not-ranked, 13 cells, 13/500 verified challenges) glm-5.3-flash: invalid — collection-only promo probe — never enabled or routed (channel paid, providers openrouter, triage not-triaged, capability not-ranked, 41 cells, 41/500 verified challenges) gpt-5-mini: invalid — not in the live pool — benchmark-only, triage DROP (channel paid, providers requesty, triage DROP, capability measured, 200 cells, 200/500 verified challenges) gpt-5.6-luna: invalid — collection-only promo probe — never enabled or routed (channel paid, providers explabs, triage not-triaged, capability not-ranked, 2 cells, 2/500 verified challenges) gpt-5.6-sol: invalid — triage UNMEASURED-EXCEPTION — slot does not clear the frontier (channel paid, providers requesty, triage UNMEASURED-EXCEPTION, capability not-ranked, 0 cells, 0/500 verified challenges) gpt-5.6-terra: invalid — collection-only promo probe — never enabled or routed (channel paid, providers requesty, triage not-triaged, capability not-ranked, 0 cells, 0/500 verified challenges) gpt-6-astra: invalid — collection-only promo probe — never enabled or routed (channel paid, providers explabs, triage not-triaged, capability not-ranked, 4 cells, 4/500 verified challenges) kimi-k2.5: invalid — not in the live pool — benchmark-only, triage DROP (channel paid, providers requesty, triage DROP, capability measured, 121 cells, 121/500 verified challenges) qwen3.7-plus: invalid — not in the live pool — benchmark-only, triage DROP (channel paid, providers requesty, triage DROP, capability measured, 87 cells, 87/500 verified challenges) qwen3.8-27b: invalid — collection-only promo probe — never enabled or routed (channel paid, providers explabs, triage not-triaged, capability not-ranked, 42 cells, 42/500 verified challenges) deepseek-v4.1-flash: invalid — collection-only free channel — never enabled or routed (channel free, providers explabs, triage not-triaged, capability not-ranked, 20 cells, 20/500 verified challenges) gemini-flash-lite-latest: invalid — collection-only free channel — never enabled or routed (channel free, providers google_ai_studio, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) glm-4.7-flash: invalid — collection-only free channel — never enabled or routed (channel free, providers cloudflare_workers_ai, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) granite-4.0-h-micro: invalid — collection-only free channel — never enabled or routed (channel free, providers cloudflare_workers_ai, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) laguna-s-2.1: invalid — collection-only free channel — never enabled or routed (channel free, providers kilo_gateway, triage not-triaged, capability not-ranked, 10 cells, 10/500 verified challenges) laguna-xs-2.1: invalid — collection-only free channel — never enabled or routed (channel free, providers kilo_gateway, triage not-triaged, capability not-ranked, 10 cells, 10/500 verified challenges) ling-3-0-flash-fin: invalid — collection-only free channel — never enabled or routed (channel free, providers openrouter, triage not-triaged, capability not-ranked, 2 cells, 2/500 verified challenges) llama-3.2-1b-instruct: invalid — collection-only free channel — never enabled or routed (channel free, providers cloudflare_workers_ai, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) llama-4-scout-17b-16e-instruct: invalid — collection-only free channel — never enabled or routed (channel free, providers cloudflare_workers_ai, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) mistral-small-3.1-24b-instruct: invalid — collection-only free channel — never enabled or routed (channel free, providers cloudflare_workers_ai, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) muse-glimmer-30b: invalid — collection-only free channel — never enabled or routed (channel free, providers requesty, triage not-triaged, capability not-ranked, 1 cells, 1/500 verified challenges) nemotron-3-nano-omni-30b-a3b-reasoning: invalid — collection-only free channel — never enabled or routed (channel free, providers requesty, triage not-triaged, capability not-ranked, 3 cells, 3/500 verified challenges) nemotron-3.5-lightning: invalid — collection-only free channel — never enabled or routed (channel free, providers kilo_gateway, requesty, triage not-triaged, capability not-ranked, 15 cells, 13/500 verified challenges) step-3.7-flash: invalid — collection-only free channel — never enabled or routed (channel free, providers kilo_gateway, triage not-triaged, capability not-ranked, 20 cells, 20/500 verified challenges)

Limits. The criteria are order-independent only in the panel; the written reason reports the FIRST one a row fails, so a model that fails several shows only the first. 'Evidenced' has ONE definition here: at least one measured cell in either channel. So the 34-identity roster holds 27 evidenced identities, and the two counts differ on purpose — the roster also names live slots and collection-only listings with no committed measurement. The coverage cell count is the measured cells for the DECLARED default arm, falling back to a sole cached arm (the definition the universe's evidenced count reads). model_grid.png counts ONLY the strict declared arm, so its cell count can be smaller — this is why glm-5.2 shows 84 here and 21 there, and both labels say which definition they use. Free-channel coverage counts a handful of challenges and is not comparable with the paid corpus's; the two channels were collected under different campaigns. A free-only (collection-only) identity is never triaged or capability-ranked, so its triage and capability cells read 'n/a' rather than 'no'; the free channel — not a failed criterion — is what puts it outside the inference pool. A live slot with no committed measurement is flagged inference-invalid here, but it is a COVERAGE GAP, not evidence that the model is dominated — collect it and the verdict can change. This panel is the per-kind exclusion breakdown, not a second cumulative funnel. model_relevance.png carries the funnel (named to evidenced to paid to valid) and the pass-rate-vs-coverage plane; here panel B keeps each exclusion KIND distinct.

Which models clear the evidence bar — and why the rest fall short

canonical weights identity · hue = channel (paid/free) · marker form = validity status · no hue-per-model · 34 named · 27 evidenced (13 paid) · 20 paid channel · 14 free · 4 inference-valid · coverage floor K=20 measured default-arm cells · 500 verified challenges

Caveat. a pass rate left of the K rule is not yet evidence; wide intervals are thin coverage Reading. Panel A: one point per EVIDENCED canonical weights identity — x is its measured default-arm cell count on a log axis, y its measured pass rate with a 95% Wilson interval. The dashed vertical rule is the K-cell floor; a point to its left has too little measured evidence to clear the bar however high its rate. Hue is the CHANNEL (blue paid, orange free) — never a hue per model — and the marker FORM is the validity status: a star is inference-valid, and each other form is a first-failing criterion or an insufficient sample. Inference-valid and near-miss models are direct-labelled; a name the placement ladder cannot seat without overprinting is listed in the notes rather than drawn on a neighbour, and two markers at the same measured coordinate are separated with a short leader back to the true point. Panel B shows four milestones — named, evidenced, paid-channel, inference-valid — and panel B′ counts, per criterion, how many named identities fail it (the categories overlap, so they do not sum to the milestones). The paid-channel bar counts ALL named identities with a paid channel, not only the evidenced ones, so it is a milestone and not a nested filter: the paid subset of the evidenced set is smaller, and the notes state it.

What to look for. Read the 4 starred points as the models the router may actually serve, then read the points at and above K that are NOT valid: those are the near-misses, models with enough measured evidence that a policy change (a channel, a triage verdict) could admit. A high pass rate far left of the K rule is a promising but under-measured model, not a rejected one.

Terms. canonical identity — the weights slug (model_version), so the same weights served by several providers are ONE point and a provider prefix or -free marker is never a name inference-valid — in the live pool AND triage pass (KEEP/EXCEPTION) AND capability rank measured AND coverage >= K measured default-arm cells AND a paid channel evidenced — a canonical identity with at least one measured default-arm cell in either channel (paid or free); a named identity with no measured cell is a coverage gap, not a weak model near-miss — an evidenced identity at or above the K cell floor that is not inference-valid — enough measured evidence to be a candidate, kept out by a criterion rather than by a thin sample measured pass rate — passes over measured default-arm cells in either channel; the whisker is a 95% Wilson interval drawn symmetrically about the estimate, half-width = the larger Wilson arm, so it reads as uncertainty rather than as a bar from zero

Notes. The status taxonomy is READ from benchmark.routing.model_validity: hue is the channel and marker form the first-failing criterion or the evidence floor, so no reader has to decode a model from a colour. panel B milestones: 34 named, 27 evidenced, 20 with a paid channel (of all named), 4 inference-valid; 13 of the evidenced identities are paid deepseek-v4-flash: paid, status valid — 70.8% on n=219, $0.0080/task; 219 cells, 201/500 challenges; inference-valid: live, triage pass, capability measured, coverage OK, paid deepseek-v4-pro: paid, status valid — 85.1% on n=201, $0.1376/task; 201 cells, 201/500 challenges; inference-valid: live, triage pass, capability measured, coverage OK, paid glm-5.2: paid, status valid — 57.1% on n=84, $0.3429/task; 84 cells, 77/500 challenges; inference-valid: live, triage pass, capability measured, coverage OK, paid kimi-k3: paid, status valid — 85.5% on n=117, $0.5159/task; 117 cells, 103/500 challenges; inference-valid: live, triage pass, capability measured, coverage OK, paid claude-fable-5: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; triage UNMEASURED-EXCEPTION — slot does not clear the frontier claude-fable-5.1: paid, status invalid:live — 90.9% on n=22, $0.0726/task; 22 cells, 22/500 challenges; collection-only promo probe — never enabled or routed claude-opus-4-6: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; collection-only promo probe — never enabled or routed claude-opus-4-8: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; triage UNMEASURED-EXCEPTION — slot does not clear the frontier claude-sonnet-5: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; collection-only promo probe — never enabled or routed gemini-3.1-pro: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; triage UNMEASURED-EXCEPTION — slot does not clear the frontier glm-5.3: paid, status insufficient — 100.0% on n=13, $0.3542/task; 13 cells, 13/500 challenges; collection-only promo probe — never enabled or routed glm-5.3-flash: paid, status invalid:live — 95.1% on n=41, $0.0000/task; 41 cells, 41/500 challenges; collection-only promo probe — never enabled or routed gpt-5-mini: paid, status invalid:live — 54.5% on n=200, $0.0285/task; 200 cells, 200/500 challenges; not in the live pool — benchmark-only, triage DROP gpt-5.6-luna: paid, status insufficient — 100.0% on n=2, $0.0063/task; 2 cells, 2/500 challenges; collection-only promo probe — never enabled or routed gpt-5.6-sol: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; triage UNMEASURED-EXCEPTION — slot does not clear the frontier gpt-5.6-terra: paid, status no-evidence — no measured outcome; 0 cells, 0/500 challenges; collection-only promo probe — never enabled or routed gpt-6-astra: paid, status insufficient — 100.0% on n=4, $0.2294/task; 4 cells, 4/500 challenges; collection-only promo probe — never enabled or routed kimi-k2.5: paid, status invalid:live — 49.6% on n=121, $0.1249/task; 121 cells, 121/500 challenges; not in the live pool — benchmark-only, triage DROP qwen3.7-plus: paid, status invalid:live — 43.7% on n=87, $0.0801/task; 87 cells, 87/500 challenges; not in the live pool — benchmark-only, triage DROP qwen3.8-27b: paid, status invalid:live — 52.4% on n=42, $0.0034/task; 42 cells, 42/500 challenges; collection-only promo probe — never enabled or routed deepseek-v4.1-flash: free, status free-only — 95.0% on n=20, $0.0000/task; 20 cells, 20/500 challenges; collection-only free channel — never enabled or routed gemini-flash-lite-latest: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed glm-4.7-flash: free, status free-only — 100.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed granite-4.0-h-micro: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed laguna-s-2.1: free, status free-only — 80.0% on n=10, $0.0000/task; 10 cells, 10/500 challenges; collection-only free channel — never enabled or routed laguna-xs-2.1: free, status free-only — 80.0% on n=10, $0.0000/task; 10 cells, 10/500 challenges; collection-only free channel — never enabled or routed ling-3-0-flash-fin: free, status free-only — 100.0% on n=2, $0.0000/task; 2 cells, 2/500 challenges; collection-only free channel — never enabled or routed llama-3.2-1b-instruct: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed llama-4-scout-17b-16e-instruct: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed mistral-small-3.1-24b-instruct: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed muse-glimmer-30b: free, status free-only — 0.0% on n=1, $0.0000/task; 1 cells, 1/500 challenges; collection-only free channel — never enabled or routed nemotron-3-nano-omni-30b-a3b-reasoning: free, status free-only — 33.3% on n=3, $0.0000/task; 3 cells, 3/500 challenges; collection-only free channel — never enabled or routed nemotron-3.5-lightning: free, status free-only — 20.0% on n=15, $0.0000/task; 15 cells, 13/500 challenges; collection-only free channel — never enabled or routed step-3.7-flash: free, status free-only — 60.0% on n=20, $0.0000/task; 20 cells, 20/500 challenges; collection-only free channel — never enabled or routed

Limits. The x axis counts measured cells, not comparable task sets: models were not run on identical tasks, so two points at the same height are not a paired comparison. Free-channel points were collected under a different campaign from the paid corpus's, so their coverage is not comparable cell-for-cell and their intervals are wide from thin coverage, not from measured weakness. A named identity with no measured cell cannot be placed on panel A at all; panel B's first milestone accounts for it as the drop from named to evidenced. Panel B's paid-channel milestone is the count of named identities with a paid channel (20); it is NOT the paid subset of the 27 evidenced identities, which is smaller (13). The milestones are separate filters, not a nested chain.

Every rung: what it costs, what it weighs, what it delivers

one measured arm per model, nothing imputed · Wilson 95% intervals · 6 models: the 4 inference-valid models' measured default-arm cells of results.csv, plus 2 out-of-corpus rung(s) (†) · 30 excluded model(s) are named and explained in the model-validity figure — not drawn here · 1 at $0 (local) · 5 priced · n per model 20–200, unpaired task sets · blend = 99% input / 1% output, the corpus's own 517,584,265:7,428,975 token split

Caveat. Panel A's x axis is a list price per TOKEN, never a bill per solved task. Reading. Panel A: each model at its blended token price (x, log — with a separate column at the left for locally-served rungs, which have no per-token list price at all and so cannot sit on a log axis; that column says nothing about what such a rung costs to run, which is UNDEFINED here and stated in the row's note) against its measured pass rate (y), with Wilson 95% whiskers. Marker area follows the square root of the active parameter count, the marker edge says whether the weights are hosted or local, and hue is the coarse total-size band. Panel B: one row per model, a hollow mark at total parameters and a filled mark at active parameters — the rule joining them IS the mixture-of-experts sparsity gap. A row whose name carries a dagger was measured outside this corpus under a different harness; its note below states which, and its height is not comparable cell-for-cell with the rows beside it.

What to look for. Look for a rung that sits high and left in panel A — cheap per token and measured to resolve tasks. Then check panel B: if the high-and-left rungs are all long rules (very sparse), the ladder's cheap end is buying compute efficiency rather than size, and a small DENSE rung is not a substitute for one.

Terms. blended $/Mtok — list input and output prices mixed at the corpus's own measured input:output token ratio, the same ratio for every model active parameters — what one token decodes through — a COMPUTE claim. All of a mixture's total parameters must still be resident to serve it UNDISCLOSED — the vendor publishes no parameter count. No estimate is substituted † (dagger) — measured outside this corpus, on a different harness and a task draw that is neither paired with this corpus nor independent of it — the row's own note states the overlap. Plotted on the same axes, never pooled with the corpus rows

Notes. deepseek-v4-flash: 68.9% on n=190 · $0.1420 (blended $ per 1M tokens (log) — mix stated above) · 284B total / 13B active deepseek-v4-pro: 85.0% on n=200 · $0.4412 (blended $ per 1M tokens (log) — mix stated above) · size UNDISCLOSED glm-5.2: 76.2% on n=21 · $1.4425 (blended $ per 1M tokens (log) — mix stated above) · 753B total / 40B active granite4.2-8b-q8-local †: 20.0% on n=20 · $0 (local) · 9B total / 9B active kimi-k3: 84.5% on n=110 · $3.1698 (blended $ per 1M tokens (log) — mix stated above) · 2800B total / 104B active qwen3.5-35b-a3b †: 69.4% on n=36 · $0.1522 (blended $ per 1M tokens (log) — mix stated above) · 35B total / 3B active drawn at a fixed reference marker because no parameter count is published: deepseek-v4-pro † granite4.2-8b-q8-local: measured on mini-swe-agent, temperature 0, llama-server -c 32768 --no-context-shift --jinja, Q8_0 KV, 3600 s wall over SWE-bench Verified, a 20-instance draw fixed before the run; 7 of these 20 instances are also among the 200 in results.csv — partial overlap, so neither a paired nor an independent comparison, and all 7 are censored cells here; no dollar cost is drawn or stated: nothing is billed per token on a locally served rung, the run spent $0 on hosted APIs, and the real resource cost was 17.7 GPU-hours on one RTX 3060 12 GB — a different quantity in a different unit, deliberately not converted into dollars; 16 of 20 cells were censored (11 wall-limit, 5 context exhaustion at a 32768-token window that is a 12 GB VRAM ceiling, not the model's 131072-token limit), so the plotted height is the CONSERVATIVE all-recorded denominator — the same convention the corpus rows use — and it is a LOWER BOUND, not a measured rate: the assumption-free bounds are [20%, 100%], the censoring is INFORMATIVE (censored cells peaked higher and ran longer than resolved ones), and the model-attributable 4 of 4 is upward-biased and must not be read off this marker; verdict ceiling UNPROVEN (pass rate; 16 of 20 cells censored, true rate bounded only to [20%, 100%]) — NOT a corpus row, and not comparable cell-for-cell with the rows beside it † qwen3.5-35b-a3b: measured on mini-swe-agent, temperature 0, 262k context, $0.55 per-cell cap over SWE-bench Verified, random.Random(seed).sample(sorted(all_specs), 20) over all 500, seeds [0, 1] pooled; 13 of these 40 instances are also among the 200 in results.csv — partial overlap, so neither a paired nor an independent comparison; $0.2153 per instance as billed, averaged over all 40 cells run (the n here is the model-attributable subset of those); 2 cell(s) censored; verdict ceiling SIGNAL (2 seeds) — NOT a corpus row, and not comparable cell-for-cell with the rows beside it

Limits. Pass rates come from an uneven sample: models were not run on identical task sets, so a rate difference across two rungs is not a paired comparison. Every row counts a censored cell in its denominator and not in its passes, so a row whose run censored heavily is a LOWER BOUND on that rung's rate, not an estimate of it. Where that matters the row's own note gives the censored count and the assumption-free bounds; read the marker as the floor it is. Panel A's x axis is a LIST PRICE at one token mix, not a measured bill: it answers what a token costs, never what a solved task costs. A locally served rung has no such price at all and sits in the category column, which is not a claim that running it is free — its cost is UNDEFINED and its note says so. The $0 column and the log region are not one ruler. The gap between them is a break, and no distance across it is meaningful. Hue is a coarse size band, not a capability measurement — panel B carries the exact counts, and a band is not evidence that its members behave alike. A DAGGERED row (†) was measured outside this corpus, under a different harness. Its height is not comparable cell-for-cell with the rest of the panel — read it as a separate measurement plotted on the same axes, never as one more corpus row. Its note below states the harness, the sample, how far that sample overlaps this corpus, and the verdict ceiling.

The kNN selection rule's errors go both ways — it loses tasks, not just money

175 decidable decisions, 5 tasks no model solved · 1 picks outside the inference-valid pool (not drawn) · 117 exact / 22 over-provisioned / 36 under-provisioned

Caveat. 36 task(s) were lost to under-provisioning — those are quality, not cost. Reading. Left: rows are the model the router chose, columns the cheapest model that actually solved the task. The diagonal is an exact hit. BELOW it the router paid for a model it did not need; above it the router under-provisioned and the task was lost. Right: the same decisions as an error budget — exact, over-provisioned, under-provisioned, and the tasks no model solved, which no decision could have won.

What to look for. Read the two error columns against each other. Over-provisioning is the bill for guessing high and costs only money; under-provisioning costs a task that some dearer model would have solved, and no threshold recovers it after the fact. The rule plotted here is a single-shot kNN prediction with no verify-and-escalate step, so both are reachable — an earlier draft of this figure read the empty under-provisioned column of a CASCADE as a property of the router itself.

Terms. cheapest sufficient — the cheapest measured model that passed this task — the router's correct answer. Undefined when no model passed. over-provisioned — the chosen model was dearer than the cheapest that would have passed. under-provisioned — the chosen model failed a task some dearer model solved. outside the inference-valid pool — the router's pick is a benchmark-only or collection-only model, so it has no axis on the pool-only grid. Counted in the decision denominator, never drawn.

Notes. Both axes are in price order and rows are the CHOSEN model, so a cell below the diagonal is over-provisioning by construction rather than by convention. The grid is scoped to the inference-VALID models — the live pool clear of the coverage/triage/capability floor. Benchmark-only and collection-only models are named on model_validity.png and invalid_models.png; a choice outside this pool is not drawn, but it is counted and its count is printed in the subtitle. exact-hit rate 66.9% over the decidable set; over-provisioning is 12.6% denominator: 181 scored decisions = 175 decidable + 5 unwinnable + 1 outside the inference-valid pool. The outside-pool picks are counted but not drawn on the pool-only grid. the budget bar's base is 180 outcomes (175 decidable + 5 unwinnable), so its exact segment reads 65.0%; the exact-hit rate over the decidable set alone is 66.9%. The 1 outside-pool picks enter neither base.

Limits. Cheapest-sufficient is read off the coverage-completed matrix, so a task whose cheap cell was imputed pass=True yields a cheaper 'correct answer' than measurement alone supports — the over-provisioning count is an upper bound.

A narrow mid-k band beats the two-policy mixture; the shipped setting does not

n=181 tasks · 5-fold outer CV · 275 cells (11 log-spaced k) · selected k=20, thresh=0.8, min_samples=1 -> 86.2% at \$18.67 out of fold · best gain over the mixture line +9.9 pp at k=80, thresh=0.8 · shipped k=20, thresh=0.6 -> 75.7% at \$3.66 (91% on deepseek-v4-flash)

Caveat. 31% of scored cells are imputed, near-all pass-filled; the trace beats the mixture by at most 9.9 pp. Reading. Three panels over ONE sweep. A: each point is one k on the log-spaced grid, placed at the total cost and pass rate its best threshold achieves out of fold; the dashed line is the straight mixture of the two fixed policies (send a fraction of tasks to the cheapest model and the rest to the frontier one). B: the same grid coloured by WHAT each (k, threshold) combination allocates — categorical, because the interesting fact is the regime, not the share. C: the selected configuration scored in sample beside the nested out-of-fold score of the same selection procedure.

What to look for. In A, a router is only worth building if its trace sits ABOVE the dashed mixture line — anything on or below it is reproducible by flipping a weighted coin between two fixed policies, with no embeddings, no index and no k. In B, look for how much of the grid is mixed at all: the degenerate bands are the sweep reporting a fixed policy's number under a routing label. In C, read the gap: it is how much of the in-sample optimum is selection optimism rather than skill.

Terms. out-of-fold — scored on a fold whose tasks were absent from BOTH the neighbour index and the selection that picked the configuration mixture line — the cost/quality reachable by splitting tasks between two fixed policies k — how many nearest tasks the router consults before choosing a model success_rate_thresh — neighbour pass-rate below which the router escalates off the cheap model min_samples — min MEASURED neighbours before the router trusts them cost at equal quality — cheapest cell whose pass rate clears the best cell's 95% Wilson lower bound — the selection rule, replacing reward-argmax

Notes. Neighbourhoods use the real shipped jina embedder — the same Embedder the router runs, never a TF-IDF proxy. The k grid is log-spaced. A uniform grid spends nearly all of its cells inside one regime and reports the same number on half of them. READ PANEL B BEFORE PANEL A's WINNER. The selected cell sits at k=20 against n=181 tasks — the neighbourhood is the 20 nearest tasks, and panel B classifies the cell as mixed allocation. REWARD-ARGMAX IS DEGENERATE: maximising reward (passes - gamma x cost, gamma=0.1) picks k=80, thresh=0.8, which routes 0% of tasks to kimi-k3 using 2 distinct model(s). At this gamma one extra pass is worth 10 USD against a suite costing a few dollars, so cost is nearly a no-op and the argmax escalates everything. OUTER-LOOP CV: for each of 5 folds the configuration is chosen on the other folds — which are also the only tasks its neighbour index may hold — and scored on the fold left out. The per-fold picks were fold 0: k=12/t=0.7/m=1, fold 1: k=20/t=0.7/m=1, fold 2: k=5/t=0.7/m=5, fold 3: k=80/t=0.8/m=1, fold 4: k=5/t=0.8/m=4. Panel A's trace takes, per k, the cheapest cell whose out-of-fold pass rate clears the best cell's 95% Wilson lower bound. The two fixed policies are scored on the same corpus: always-cheapest (deepseek-v4-flash) 75.1% at \$1.48, always-frontier (kimi-k3) 95.0% at \$94.37. The k grid actually swept runs 2 to 174 (11 values), clipped to a corpus of 181 tasks. Reward is driven by success_rate_thresh (η²=0.65); k barely moves it (η²=0.04) — pick k for stability. min_samples: η²=0.01 — negligible effect

Limits. Folds are grouped by REPOSITORY — an out-of-fold task never shares a repo with a task in its own neighbour index — so the held-out number is transfer to an unseen codebase, not an estimate inflated by same-repo siblings (see embedding_signal.png's cross-repo panel). The S regime was added on 2026-09-05 and is not a new measurement: those cells were always single-model, and _regime simply could not see it. Every earlier render of this panel drew them green, as mixed allocation. 396/1267 cells (31.3%) in THE MATRIX THIS SWEEP SCORES (181 tasks x 7 ranked models — not the corpus-wide count in evidence_basis.png) are monotone-IMPUTED rather than measured, and the imputation is near-exclusively pass-filling, so it can almost never add a failure. The neighbourhood VOTE excludes those imputed cells, but the pass rates on this grid still read those synthetic passes — every quality number here is biased up Cost is model-price dependent — the selected cell moves when model prices move.

The kNN selection rule sends most of every difficulty bucket to the cheapest model

181 scored tasks (19 incomplete challenges excluded); 5 solved by no enabled model · 3 capability bands populated · 1 of panel B's picks fall outside the inference-valid pool (drawn as one grey segment, not named) · hardest bucket (0 solvers) mostly deepseek-v4-flash, easiest (4 solvers) mostly deepseek-v4-flash Reading. Left: how many tasks each capability band is the cheapest sufficient answer for, weakest band on the left, plus the tasks no enabled model solved. Right: for each count of solving models — the corpus's own difficulty measure — the share of tasks the kNN selection rule sent to each model, as stacked bars with the task count above.

What to look for. Compare the stacks across the right panel's buckets. The rule plotted here is kNN: it predicts ONCE from the neighbourhood and does not escalate, so a stack that barely moves from the hardest bucket to the easiest means the prediction is barely conditioning on difficulty at all. Read embedding_signal.png for why — the input it predicts from carries almost no routable signal.

Terms. capability band — models grouped by derived capability rank; a task's band is the weakest band containing a model that solved it. solving models — how many enabled models solved the task. Zero means unwinnable, all means free.

Notes. Bands and solving-model counts are read off the coverage-completed matrix, the same matrix every strategy is scored on. band 2: 136 tasks band 3: 5 tasks band 4: 35 tasks 0 solvers: {'deepseek-v4-flash': 3, 'deepseek-v4-pro': 2} 1 solvers: {'deepseek-v4-flash': 10} 2 solvers: {'deepseek-v4-flash': 12, 'deepseek-v4-pro': 2, 'outside inference-valid pool (not named)': 1} 3 solvers: {'deepseek-v4-flash': 17, 'deepseek-v4-pro': 1} 4 solvers: {'deepseek-v4-flash': 111, 'deepseek-v4-pro': 22} Panels A and B share one denominator: tasks with at least one inference-valid cell. A task the completion dropped (no cells at all) is excluded from both, never counted as unwinnable.

Limits. An imputed cell is always a pass, so a task's band is a LOWER bound on the capability it truly needs and the solving-model count is an upper bound. The right panel is NOT circular for the rule plotted — kNN decides before any outcome for this task exists — but it is not independent either: the neighbours it reads and the solving-model count it is plotted against come from one matrix. 'No enabled model solved it' counts the inference-VALID benchmark models at their DEFAULT arms (benchmark-only and collection-only models are excluded, because the live router cannot pick them). complementarity.png counts every sampled (model, arm) column instead, so its solved-by-none figure is smaller — a different denominator, not a disagreement. Picks naming a model outside the inference-valid pool are counted in the denominator but aggregated into one grey segment and NOT named; the individual models are named on model_validity.png and invalid_models.png.

Every model's evidence on one grid — free and paid, valid and not

one row per canonical weights identity · verified challenges as columns · 34 canonical models: 20 paid, 14 free · 4 inference-valid, 30 outside the pool · 34 named · 27 evidenced (≥1 measured outcome) · 23 invalid-with-outcome · 500 verified challenges

Caveat. Coverage is an evidence status, not a quality score — a covered cell can fail. Reading. One row per canonical weights identity the committed evidence names, one column per verified challenge, in the corpus's own sorted order. A blue cell is a paid channel's measured default-arm cell, an orange cell a free one, a pale cell no measured outcome. Vertical rules mark repository boundaries, and each row label carries its covered/total count. Rows are ordered inference-valid first (marked ★, above the green rule), then the paid collection, then the free collection.

What to look for. Read that coverage is ragged and channel-specific. The four starred rows above the green rule are the only models the main routing figures draw; every row below it is still measured and still named, and a row of pale cells says no committed outcome exists rather than that the model is weak.

Terms. canonical identity — the registry version slug, so the same weights served under several channel listings are ONE row and a provider prefix or -free marker is never a name measured cell — a committed default-arm outcome for one (challenge, model) pair; imputed cells are not counted here evidenced — a canonical identity with at least one committed measured default-arm outcome (Performance.n > 0); a named identity with no measured outcome is drawn as an empty band and is not a claim that the model is weak

Notes. Cells are measured outcomes only. The completed matrix used by the strategy figures adds imputed cells, which this canvas deliberately excludes. deepseek-v4-flash: 201/500 challenges, 219 measured cells, paid, VALID deepseek-v4-pro: 201/500 challenges, 201 measured cells, paid, VALID glm-5.2: 77/500 challenges, 84 measured cells, paid, VALID kimi-k3: 103/500 challenges, 117 measured cells, paid, VALID claude-fable-5: 0/500 challenges, 0 measured cells, paid, triage UNMEASURED-EXCEPTION — slot does not clear the frontier claude-fable-5.1: 22/500 challenges, 22 measured cells, paid, collection-only promo probe — never enabled or routed claude-opus-4-6: 0/500 challenges, 0 measured cells, paid, collection-only promo probe — never enabled or routed claude-opus-4-8: 0/500 challenges, 0 measured cells, paid, triage UNMEASURED-EXCEPTION — slot does not clear the frontier claude-sonnet-5: 0/500 challenges, 0 measured cells, paid, collection-only promo probe — never enabled or routed gemini-3.1-pro: 0/500 challenges, 0 measured cells, paid, triage UNMEASURED-EXCEPTION — slot does not clear the frontier glm-5.3: 13/500 challenges, 13 measured cells, paid, collection-only promo probe — never enabled or routed glm-5.3-flash: 41/500 challenges, 41 measured cells, paid, collection-only promo probe — never enabled or routed gpt-5-mini: 200/500 challenges, 200 measured cells, paid, not in the live pool — benchmark-only, triage DROP gpt-5.6-luna: 2/500 challenges, 2 measured cells, paid, collection-only promo probe — never enabled or routed gpt-5.6-sol: 0/500 challenges, 0 measured cells, paid, triage UNMEASURED-EXCEPTION — slot does not clear the frontier gpt-5.6-terra: 0/500 challenges, 0 measured cells, paid, collection-only promo probe — never enabled or routed gpt-6-astra: 4/500 challenges, 4 measured cells, paid, collection-only promo probe — never enabled or routed kimi-k2.5: 121/500 challenges, 121 measured cells, paid, not in the live pool — benchmark-only, triage DROP qwen3.7-plus: 87/500 challenges, 87 measured cells, paid, not in the live pool — benchmark-only, triage DROP qwen3.8-27b: 42/500 challenges, 42 measured cells, paid, collection-only promo probe — never enabled or routed deepseek-v4.1-flash: 20/500 challenges, 20 measured cells, free, collection-only free channel — never enabled or routed gemini-flash-lite-latest: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed glm-4.7-flash: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed granite-4.0-h-micro: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed laguna-s-2.1: 10/500 challenges, 10 measured cells, free, collection-only free channel — never enabled or routed laguna-xs-2.1: 10/500 challenges, 10 measured cells, free, collection-only free channel — never enabled or routed ling-3-0-flash-fin: 2/500 challenges, 2 measured cells, free, collection-only free channel — never enabled or routed llama-3.2-1b-instruct: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed llama-4-scout-17b-16e-instruct: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed mistral-small-3.1-24b-instruct: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed muse-glimmer-30b: 1/500 challenges, 1 measured cells, free, collection-only free channel — never enabled or routed nemotron-3-nano-omni-30b-a3b-reasoning: 3/500 challenges, 3 measured cells, free, collection-only free channel — never enabled or routed nemotron-3.5-lightning: 13/500 challenges, 15 measured cells, free, collection-only free channel — never enabled or routed step-3.7-flash: 20/500 challenges, 20 measured cells, free, collection-only free channel — never enabled or routed

Limits. Paid and free corpora were collected under different campaigns, so a free row's coverage is not comparable cell-for-cell with a paid row's. A row with no committed cell is drawn as an empty band and is not evidence that the model is weak.

What every free and paid model costs and delivers

measured default-arm outcomes only · blue paid, orange free · ★ inference-valid · 27 evidenced models (13 paid, 14 free) · 4 inference-valid marked ★ · 34 named · 27 evidenced (≥1 measured outcome) · 23 invalid-with-outcome

Caveat. Panel A draws dots, not bars — a log axis reads position. Free $0 and paid sub-floor rows are separate hatched markers. Reading. Panel A: per canonical identity, the mean measured cost per task on a log axis. Each priced row is a dot at its cost — a dot carries position, the one thing a log axis reads honestly, where a bar's length would encode log(cost) and misstate every ratio. A $0.0000 free row (a genuinely free channel) and a <$0.000001 paid row (a paid mean that rounds below the axis floor) are distinct hatched stubs in a fixed column at the left, not bars and not each other. Panel B: the same identity's measured pass rate with a 95% Wilson interval, same row order, so cost and quality read across one line. Blue is a paid channel, orange a free one, and a star marks the inference-valid models.

What to look for. Find the identities that are both cheap and high-pass: the free channel's rows sit far left, but their intervals are wide because their coverage is thin. The four starred rows are the ones the router may actually serve.

Terms. measured cost — the mean real_cost over that identity's committed default-arm cells — what was billed, not list price pass rate — passes over measured default-arm cells, with a 95% Wilson interval evidenced — a canonical identity with at least one committed measured default-arm outcome (Performance.n > 0); a named identity with no measured outcome is drawn as an empty band and is not a claim that the model is weak

Notes. Rows with no committed cell are absent: this canvas is a measurement, and a model with nothing measured has no point to draw. deepseek-v4-flash: 70.8% on n=219, $0.0080/task, paid, VALID deepseek-v4-pro: 85.1% on n=201, $0.1376/task, paid, VALID glm-5.2: 57.1% on n=84, $0.3429/task, paid, VALID kimi-k3: 85.5% on n=117, $0.5159/task, paid, VALID claude-fable-5.1: 90.9% on n=22, $0.0726/task, paid, collection-only promo probe — never enabled or routed glm-5.3: 100.0% on n=13, $0.3542/task, paid, collection-only promo probe — never enabled or routed glm-5.3-flash: 95.1% on n=41, $0.0000/task, paid, collection-only promo probe — never enabled or routed gpt-5-mini: 54.5% on n=200, $0.0285/task, paid, not in the live pool — benchmark-only, triage DROP gpt-5.6-luna: 100.0% on n=2, $0.0063/task, paid, collection-only promo probe — never enabled or routed gpt-6-astra: 100.0% on n=4, $0.2294/task, paid, collection-only promo probe — never enabled or routed kimi-k2.5: 49.6% on n=121, $0.1249/task, paid, not in the live pool — benchmark-only, triage DROP qwen3.7-plus: 43.7% on n=87, $0.0801/task, paid, not in the live pool — benchmark-only, triage DROP qwen3.8-27b: 52.4% on n=42, $0.0034/task, paid, collection-only promo probe — never enabled or routed deepseek-v4.1-flash: 95.0% on n=20, $0.0000/task, free, collection-only free channel — never enabled or routed gemini-flash-lite-latest: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed glm-4.7-flash: 100.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed granite-4.0-h-micro: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed laguna-s-2.1: 80.0% on n=10, $0.0000/task, free, collection-only free channel — never enabled or routed laguna-xs-2.1: 80.0% on n=10, $0.0000/task, free, collection-only free channel — never enabled or routed ling-3-0-flash-fin: 100.0% on n=2, $0.0000/task, free, collection-only free channel — never enabled or routed llama-3.2-1b-instruct: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed llama-4-scout-17b-16e-instruct: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed mistral-small-3.1-24b-instruct: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed muse-glimmer-30b: 0.0% on n=1, $0.0000/task, free, collection-only free channel — never enabled or routed nemotron-3-nano-omni-30b-a3b-reasoning: 33.3% on n=3, $0.0000/task, free, collection-only free channel — never enabled or routed nemotron-3.5-lightning: 20.0% on n=15, $0.0000/task, free, collection-only free channel — never enabled or routed step-3.7-flash: 60.0% on n=20, $0.0000/task, free, collection-only free channel — never enabled or routed

Limits. Free-channel rows are cheap by price and thin by coverage; a wide interval is a coverage gap, not a quality estimate. Mean cost is over measured cells only and is not the list price the strategy figures rank on.

Outside the inference pool: what the other measured models deliver

inference-invalid on committed evidence · not selected for routing, not erased · 23 inference-invalid evidenced models (9 paid, 14 free) · 34 named · 27 evidenced (≥1 measured outcome) · 23 invalid-with-outcome · the 4 inference-valid models are on the model-validity figure

Caveat. Invalid means not selected on this evidence — not that the model is weak. Reading. Every evidenced model that fails at least one inference criterion, on two axes. Panel A: verified-challenge coverage, with the measured-cell count printed; a row covering nothing draws a hatched zero stub, not a missing bar. Panel B: measured pass rate with a 95% Wilson interval, the mean billed cost per task printed beside it. A row with fewer than the provisional cell floor is drawn hatched and faded in both panels, so a thin sample is never read as a full measurement. Blue is a paid channel, orange a free one. Each row carries its first failing reason in the notes below.

What to look for. Read this page as the boundary of the inference pool: these models have real measured outcomes, some of them on hundreds of cells, and the reason each is outside is printed rather than implied. A dominated benchmark model and a collection-only probe are not the same kind of absence.

Terms. inference-invalid — fails at least one of: live pool, triage KEEP, capability rank measured, coverage >= K cells, a paid channel. The first failing reason is named per row. evidenced — a canonical identity with at least one committed measured default-arm outcome (Performance.n > 0); a named identity with no measured outcome is drawn as an empty band and is not a claim that the model is weak

Notes. The predicate and the first-failing reason are not restated here — they are read from benchmark.routing.model_validity, the same source the model-validity figure draws. claude-fable-5.1: collection-only promo probe — never enabled or routed — 22/500 challenges, 22 cells, paid, capability not-ranked, triage not-triaged glm-5.3: collection-only promo probe — never enabled or routed — 13/500 challenges, 13 cells, paid, capability not-ranked, triage not-triaged glm-5.3-flash: collection-only promo probe — never enabled or routed — 41/500 challenges, 41 cells, paid, capability not-ranked, triage not-triaged gpt-5-mini: not in the live pool — benchmark-only, triage DROP — 200/500 challenges, 200 cells, paid, capability measured, triage DROP gpt-5.6-luna: collection-only promo probe — never enabled or routed — 2/500 challenges, 2 cells, paid, capability not-ranked, triage not-triaged gpt-6-astra: collection-only promo probe — never enabled or routed — 4/500 challenges, 4 cells, paid, capability not-ranked, triage not-triaged kimi-k2.5: not in the live pool — benchmark-only, triage DROP — 121/500 challenges, 121 cells, paid, capability measured, triage DROP qwen3.7-plus: not in the live pool — benchmark-only, triage DROP — 87/500 challenges, 87 cells, paid, capability measured, triage DROP qwen3.8-27b: collection-only promo probe — never enabled or routed — 42/500 challenges, 42 cells, paid, capability not-ranked, triage not-triaged deepseek-v4.1-flash: collection-only free channel — never enabled or routed — 20/500 challenges, 20 cells, free, capability not-ranked, triage not-triaged gemini-flash-lite-latest: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged glm-4.7-flash: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged granite-4.0-h-micro: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged laguna-s-2.1: collection-only free channel — never enabled or routed — 10/500 challenges, 10 cells, free, capability not-ranked, triage not-triaged laguna-xs-2.1: collection-only free channel — never enabled or routed — 10/500 challenges, 10 cells, free, capability not-ranked, triage not-triaged ling-3-0-flash-fin: collection-only free channel — never enabled or routed — 2/500 challenges, 2 cells, free, capability not-ranked, triage not-triaged llama-3.2-1b-instruct: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged llama-4-scout-17b-16e-instruct: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged mistral-small-3.1-24b-instruct: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged muse-glimmer-30b: collection-only free channel — never enabled or routed — 1/500 challenges, 1 cells, free, capability not-ranked, triage not-triaged nemotron-3-nano-omni-30b-a3b-reasoning: collection-only free channel — never enabled or routed — 3/500 challenges, 3 cells, free, capability not-ranked, triage not-triaged nemotron-3.5-lightning: collection-only free channel — never enabled or routed — 13/500 challenges, 15 cells, free, capability not-ranked, triage not-triaged step-3.7-flash: collection-only free channel — never enabled or routed — 20/500 challenges, 20 cells, free, capability not-ranked, triage not-triaged

Limits. A live slot with no committed measurement is flagged invalid here but is a coverage gap, not evidence of domination; collect it and the verdict can change. Free-channel rows were collected under a different campaign and their intervals are wide from thin coverage, not from measured weakness.