PaperlayerView exact revision on arXiv

Source-linked AI summary

CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Daren Zha

arXiv:2610.20668v2cs.CRcs.CLcs.LG

TL;DR

Budgeted language-model cascades are difficult to audit when hidden targets or future outputs can influence routing despite unchanged aggregate summaries. CascadeLedger combines trace invariants with counterfactual checks of router behavior. The audit detects hidden-read faults and can change corrupted quality–resource comparisons after recomputation.

  • Problem

    A final answer, call histogram, and cost summary can remain unchanged even when routing consults a target or future output before it is observable.

    Source: §1 · p. 1

  • Method

    CascadeLedger freezes candidate outputs before routing and combines execution invariants with CPNI tests that vary hidden state while holding the visible prefix and policy randomness fixed.

    Source: §1 · p. 2

  • Results

    The declared fault population yields 0.9982 hidden-read recall and a 0.0000 clean false-positive rate, while recomputation changes 784 of 2,304 corrupted comparisons.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings distinguish aggregate performance, serialized trace validity, and behavioral information integrity under the trusted recording and binding contract.

    Source: §5 · p. 9

    Audit coverage is limited to tested families, prefixes, and couplings, and producer authenticity relies on the trusted recording contract.

    Source: §3.2 · p. 4, §4.4 · p. 9

Abstract

from arXiv · show

We introduce CascadeLedger, a counterfactual execution-audit framework for budgeted language-model cascades. The protocol enforces call-before-read semantics, binds successful reads to service and cache identities, records availability and component-wise budget transitions, and freezes the event trace before any target or task scorer is joined. The central object is aggregate-equivalent trace validity: two executions can share a final answer, declared resource projection, and service histogram while differing in whether their routing decisions used only legitimately observable information. We complement single-trace invariants with counterfactual prefix non-interference (CPNI), which couples a visible prefix and policy randomness while changing only a declared hidden family. The counterfactual influence frontier (CIF) localizes the earliest violating prefix and a minimal hidden witness; projection-preserving adversarial mutation (PPAM) searches for invalid traces that remain acceptable to a non-counterfactual checker. Across frozen-cache, independently generated, and serving evaluations, the two-layer audit reports a clean false-positive rate of 0.0000, detects 0.9983 of serialized faults and 0.9982 of unlogged hidden-read faults, and changes 0.3403 of corrupted quality-resource conclusions after deterministic recomputation. The online recorder and offline CPNI audit have separate overhead measurements. Learned routing and semantic verification provide downstream compatibility studies under the same execution contract.

1 INTRODUCTION

CascadeLedger audits budgeted language-model cascades by restricting routing to legitimately observed information and checking both serialized traces and counterfactual behavior. Its introduction also reports routing comparisons and tools for locating hidden-information violations and testing whether invalid traces change quality–resource conclusions.

1 INTRODUCTION

  • CascadeLedger freezes candidate outputs before routing, limits policies to query and successful-call observations, and joins targets only after trace freeze.Source: §1 · p. 1
  • On the primary 128-record cache, fixed and sequential routing both reach 0.3203 cache–target proxy match, while sequential routing costs more; learned gating reaches 41/128 matches versus 42/128 sequentially.The learned gate costs 1.609375 additional declared units.Source: §1 · p. 1
  • The 64/128 and 256/1,319 operating-point gates show why thresholds must travel with dataset partition and cache provenance.Source: §1 · p. 2
  • The independent task and serving protocol tests the contract across train, calibration, and held-out splits using a common scorer and measures quality, coverage, failures, token cost, and latency.Source: §1 · p. 2
  • CPNI tests whether an unrevealed target or future service output changes a routing action before its reveal boundary.Source: §1 · p. 2
  • CIF extends CPNI by identifying the earliest violating prefix and a minimal hidden-family witness under aggregate-only indistinguishability.Source: §1 · p. 2
  • The two-layer audit combines paid-read, ordering, binding, and budget invariants with counterfactual checks; PPAM searches for invalid traces that preserve a declared quality–resource projection.The authors recompute matched policy comparisons after rejecting invalid traces to test whether the reported conclusion changes.Source: §1 · p. 2

2 RELATED WORK

The related work situates CascadeLedger among conditional-computation routing, verification, sequence-level control, and agent-tool evaluation. Its stated distinction is an auditable frozen-candidate layer and a test of policy-visible information integrity, not only inspectable execution records.

2 RELATED WORK

  • Prior work frames cascades as conditional computation and connects routing to quality–cost trade-offs, reject-option coverage, verifier-guided reasoning, sequence-level control, and agent-tool environments.The cited examples include FrugalGPT and RouteLLM (Chen et al., 2023; Ong et al., 2025), and ReAct and Toolformer (Yao et al., 2023; Schick et al., 2023).Source: §2 · p. 2
  • CascadeLedger adds a frozen-candidate replay layer that makes routing events inspectable before an independent task scorer is attached.Source: §2 · p. 2
  • It records event history connecting route decisions to separately versioned scoring and service interfaces.Source: §2 · p. 2
  • Compared with routing work on deployment, contextual-bandit budgets, broad model evaluation, and adaptive cascades, CascadeLedger emphasizes provenance for confidence, verifier outputs, and service costs while keeping semantic scoring separate.The related-work passage names Cascadia (Jiang et al., 2025), PILOT (Panda et al., 2025), and RouterEval (Huang et al., 2025).Source: §2 · pp. 2–3
  • CPNI tests whether changing a still-hidden field changes the next action even when the serialized trace and aggregate projection remain well formed.It holds the visible prefix and policy randomness fixed while intervening on unrevealed state.Source: §2 · p. 3

3 METHOD

CascadeLedger audits both whether recorded events obey the execution contract and whether routing actions depend only on information legitimately visible at each prefix. Its protocol separates online routing and trace recording from post-trace scoring, then tests the audit through paired interventions and declared measurements.

3.1 PROBLEM AND PROTOCOL

  • The trace semantics make each routing request a pre-read decision and update observations, budgets, and successful-call paths only after an available, affordable call.Unavailable or unaffordable outcomes leave those state components unchanged; oracle matching follows trace freeze.Source: §3.1 · p. 4

3.2 COUNTERFACTUAL PREFIX NON-INTERFERENCE AUDIT

  • CPNI pairs executions with identical visible prefixes and policy randomness, changing one hidden family to test whether a pre-reveal routing action changes.The finite intervention suite covers declared hidden families, prefixes, seeds, and interventions; its pass result applies to that suite.Source: §3.2 · p. 4
  • Counterfactual influence is reported across hidden-family and prefix pairs, with a minimal witness recovered by hierarchical delta-debugging.The witness is inclusion-minimal under the search order, not necessarily minimum-cardinality.Source: §3.2 · p. 4

3.4 AUDIT GUARANTEES

  • An auditor that sees only a declared aggregate projection must assign the same result to valid and invalid traces with equal projections.Aggregate-equivalent traces can still differ in action order, cache binding, failure semantics, budget transitions, or hidden information flow.Source: §3.3 · p. 4, §3.4 · p. 5

3.5 AUDIT CONSEQUENCE

  • The audit consequence rate measures whether accepting a corrupted trace changes a predeclared scientific conclusion after recomputation from its bound manifest.Mutation recall measures detection, while audit consequence rate measures the effect on the conclusion.Source: §3.5 · p. 5

3.6 ROUTING AND MEASUREMENT OBJECTIVES

  • Task quality, declared resource cost, live latency, availability events, budget events, and mutation recall are measured separately with their specified evaluation units.Task scores and latency share a request denominator but use separate scoring and timing interfaces.Source: §3.6 · p. 5, Table 1 · p. 5

3.7 SEQUENTIAL ROUTE AND FALLBACK

  • A sequential route calls the small service first, then uses a paid verifier signal to stop or permit a large call, subject to availability and remaining budget.If the small-service confidence is below 0.75, it calls the format verifier; a passing verifier stops with the small candidate, while a failing one permits a large call.Source: §3.7 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate CascadeLedger’s audit coverage and routing outcomes across held-out task scoring, frozen-cache stress tests, and serving shifts. They report strong fault detection alongside distinct quality–resource trade-offs and scope boundaries.

4.2 MAIN RESULTS

  • 0.9983 of serialized faults and 0.9982 of unlogged hidden reads are detected at a 0.0000 clean false-positive rate; recomputation changes 784 of 2,304 corrupted comparisons.Without CPNI, the ledger detects 0.0000 of unlogged hidden reads; adding CPNI raises hidden-read recall while preserving serialized-fault recall.Source: §4.2 · p. 7
  • The learned router scores +0.0152 over sequential routing (paired interval [+0.0061, +0.0244]) while using fewer calls, tokens, and p50/p95 milliseconds on held-out requests.The study compares five policies under the same scorer, candidate manifests, and request denominator; Figure 2 places learned routing above and left of fixed, sequential, and budget-conditioned policies.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 2 · p. 8

4.3 MECHANISM RESULTS AND AUDIT

  • On the 128-record primary cache, fixed and sequential cascades match 41/128 targets (0.3203), but both fall to 0.2109 under large-service unavailability or the tight budget.The canonical verifier rejects all 89 selected candidates with zero marginal proxy decision value, while the independent semantic-label protocol improves GSM8K/Qwen task score from 0.7043 to 0.7230 and lowers declared units from 13.74 to 11.35.Source: §4.3 · p. 8

4.4 ANALYSIS AND ABLATIONS

  • Across five serving-shift conditions, H4 reports positive rank and linear associations, with R2 from 0.768 to 0.863 and shift residuals from −1.4 to +2.6 ms.Joint interventions raise interaction-only recall from 0.0000 to 0.9929, and the complete audit recovers 0.9976–0.9980 of projection-preserving adversarial traces that pass non-counterfactual checks; coverage is indexed by tested families, prefixes, and coupling, with producer authenticity relying on the trusted recording contract.Source: §4.4 · p. 9

5 CONCLUSION

CascadeLedger links cascade decisions to paid observations, resource transitions, and scored answers, while auditing both serialized traces and hidden-information influence. Its audits detect aggregate-preserving faults and separate performance, trace validity, and behavioral information integrity under a trusted recording and binding contract.

5 CONCLUSION

  • CascadeLedger connects each cascade decision to its paid observations, resource transitions, and scored answer.The frozen-cache diagnostic exposes redundant verifier calls, while private and public audits detect aggregate-preserving faults in reads, ordering, cache binding, and budgets.Source: §5 · p. 9
  • The complete audit rejects 0.9976–0.9980 of PPAM traces that pass non-counterfactual checks, with hidden-read recall of 0.9982 and clean false-positive rate of 0.0000.CIF recovers the exact earliest violating prefix for 0.9750 of injected traces, and recomputation changes 784 of 2,304 corrupted comparisons.Source: §5 · p. 9
  • The findings distinguish aggregate performance, serialized trace validity, and behavioral information integrity under the trusted recording and binding contract.CPNI tests unlogged influence through coupled pre-reveal actions; CIF localizes the earliest violating prefix and a minimal hidden witness.Source: §5 · p. 9

AI-USE STATEMENT

The authors disclose using generative AI tools for language polishing and reference summaries, but not for generating results, datasets, mathematical claims, proofs, or research conclusions.

AI-USE STATEMENT

  • Generative AI tools polished language and summarized references, but did not generate experimental results, synthetic datasets, mathematical claims, proofs, or research-conclusion decisions.The authors state that they designed and verified the methodology, experimental setup, analysis, and interpretation, and take responsibility for the final content.Source: AI-USE STATEMENT · p. 9
  • The authors report that other required disclosure tasks not mentioned in the statement are not applicable.Source: AI-USE STATEMENT · p. 9
  • The authors take full responsibility for the final content, including text, claims, analyses, and artifacts produced with AI assistance.Source: AI-USE STATEMENT · p. 9

ETHICS STATEMENT

The work audits routing decisions on public benchmark questions using openly released models and reports no personal data collection or human subjects beyond answer-correctness labeling. Its implementation appendix specifies the replay contract, event semantics, and separate data partitions and provenance.

ETHICS STATEMENT

  • The audit uses GSM8K and ARC-Challenge questions with openly released Qwen and Llama models, and collects no personal data or human subjects beyond benchmark answer-correctness labeling.Annotators saw only questions, candidate answers, and reference answers; the public replay bundle contains target-free traces without private caches or model weights.Source: ETHICS STATEMENT · p. 10

A IMPLEMENTATION AND AUDIT DETAILS

A.1 NOTATION AND TASK CONTRACT

  • Each replay record carries its query, cached service observations, post-trace answer, finite event sequence, and returned candidate.The online route state tracks observations already paid for, service availability, remaining budget, and the successful-call path.Source: §A.1 · p. 13

A.2 FULL REPLAY ALGORITHM AND EVENT SEMANTICS

  • The deterministic replay checks availability and budget before reading a cached service output, records unavailable or budget-blocked calls, and freezes the trace before joining the answer to the target.The checker validates event order, path alignment, cache binding, cost reconciliation, paid-read consistency, and budget conservation before oracle fields are available.Source: §A.2 · pp. 13–14

A.3 DATA PROVENANCE AND SPLIT LEDGER

  • Primary, diagnostic, and supplemental artifacts use separate record counts, prompts, model assets, and threshold decisions.The appendix retains source manifests and digests for the distinct partitions; cached candidate construction records prompt and model-version digests and generation metadata.Source: §A.3 · p. 15, Table 9 · p. 15

A.4 IMPLEMENTATION DETAILS

The implementation details define replayable event and budget accounting, then test whether event-level and counterfactual checks detect faults hidden by aggregate summaries. The extensions also localize violations and measure how accepting invalid traces can change conclusions.

A.4 IMPLEMENTATION DETAILS

  • Replaying frozen traces checks hashes, event ordering, budget transitions, and paid-call bindings before the oracle is joined; failed calls remain explicit events but do not enter the successful-call path.The frozen manifest and event sequence support deterministic reconstruction, while task correctness is assessed separately.Source: §A.4 · pp. 15–16
  • Across 2,048 clean traces, the private checker found no invariant violations; it rejected all five aggregate-preserving mutations, while the aggregate-only projection rejected none.The five mutations included target and future-output leakage, action reordering, budget corruption, and cache-ID swapping.Source: §A.4 · p. 16

A.5 MUTATION EXTENSIONS AND AUDIT COVERAGE

  • The private campaign also tests retry accounting, failure labels, resource bindings, and future-queue observations; the aggregate-only projection detects none of the first five mutation families.Each family retains its own clean and mutated denominator.Source: §A.5 · p. 17, Table 12 · p. 17

A.6 ERROR TAXONOMY AND CASE ANALYSIS

  • Route outcomes are grouped as small-only completion, verifier stop, large escalation, availability fallback, or budget fallback, with task errors and serving events reported within categories.The route analysis reports semantic errors at 370/1,319 requests per seed and timeout/retry events at 17/1,319 (1.29%).Source: §A.6 · p. 17, Table 13 · p. 18

A.7 ADDITIONAL ANALYSIS

  • The held-out learned router improves task score over sequential routing by 0.0152 while reducing calls and tokens; the frozen-cache diagnostic instead measures exact matching, route depth, and declared resources.The independent task protocol uses normalized task scores, tokens, money, and measured latency.Source: §A.7 · p. 19

B.1 EVALUATION QUESTIONS AND COUNTERFACTUAL REPLAY

  • Paired counterfactual replay holds the visible prefix and policy randomness fixed while intervening on hidden fields; marginal tests miss interaction-only influence, which joint interventions detect at 0.9929 recall.With joint interventions, delayed and stochastic influence reach 0.9951 and 0.9746 recall; clean divergence is 0.0000 for every configuration.Source: §B.1 · p. 19, §B.4 · p. 20

B.5 COUNTERFACTUAL INFLUENCE LOCALIZATION

  • CIF identifies the family for 0.9881 of traces and the exact earliest violating prefix for 0.9750, with 7.96 paired replays per trace pooled across declared families.Exact inclusion-minimal witness recovery is 0.9687; paired replay reduces search by 61.8% against exhaustive search.Source: §B.5 · pp. 20–21, Table 19 · p. 21

B.6 PROJECTION-PRESERVING ADVERSARIAL MUTATION

  • PPAM retains invalid traces that preserve a declared quality–resource projection and pass the strongest non-counterfactual checker; the full audit recovers 0.9980, 0.9979, and 0.9976 under Φ1, Φ2, and Φ3.As projections become richer, retained traces fall from 1,012 to 823 and search success from 0.9414 to 0.8037; accepting them changes 0.3656 to 0.3220 of predeclared conclusions.Source: §B.6 · p. 21, Table 20 · p. 21

B.7 COMMON-RANDOMNESS ABLATION

These extensions test how audit coverage behaves under randomness coupling, longer cascades, component removal, and distinct evaluation protocols. They also report intervention-coverage costs and examine learned routing under a separate cache audit.

B.7 COMMON-RANDOMNESS ABLATION

  • Coupling policy draws reduces clean-pair divergence from 0.0786 to 0.0000 and replay variance from 0.0734 to 0.0013, while hidden-fault recall stays at 0.9980–0.9982.The shared-randomness configuration supplies the coupling used by the primary CPNI protocol.Source: §B.7 · pp. 21–22, Table 21 · p. 22

B.8 LONGER-HORIZON CASCADE TRANSFER

  • With six paid decision stages instead of three, serialized recall changes from 0.9983 to 0.9974, hidden recall from 0.9982 to 0.9968, and exact-prefix localization from 0.9746 to 0.9613.Both recalls use the same event semantics on the same 1,319 held-out requests.Source: §B.8 · p. 22, Table 22 · p. 22

B.9 AUDIT COMPONENT ABLATION

  • Removing a structural invariant lowers its corresponding fault-column recall to 0.3047–0.7124; removing CPNI eliminates unlogged-read and joint detection, while removing joint interventions eliminates interaction-only detection.The ablation holds traces, fault seeds, and intervention prefixes fixed.Source: §B.9 · p. 22, Table 23 · p. 22

B.10 AUDIT CONSEQUENCE RATE / B.11 INDEPENDENT AND NATURAL FAULTS

  • The audit-consequence protocol recomputes rejected traces from their bound cache and configuration after first accepting the corrupted trace; independent and natural fault sets retain separate denominators and root-cause labels.Natural faults are selected through a predeclared review, and checker-independent reviewers assign labels before checker outcomes are exposed.Source: §B.10 · pp. 22–23, §B.11 · p. 23, Table 25 · p. 23

B.12 AUDIT OVERHEAD / B.13 AUDIT COST AND INTERVENTION COVERAGE

  • Full prefix coverage reaches 0.9983 recall at 10.72 ms per trace and 8.62 replays; adding CIF leaves recall unchanged and adds 2.76 ms and 0.34 replays per trace for localization.Source: §B.13 · pp. 23–24

B.14 SEMANTIC-LABEL PROTOCOL

  • Semantic-verifier labels cover exact correctness, partial correctness, invalid output, and abstention; three annotators achieve Fleiss’ κ = 0.846, with blinded adjudication for cases lacking a majority.Labels are frozen before confusion matrices and routing outcomes are computed and enter only through the post-trace scorer.Source: §B.14 · p. 24

B.15 ROUTER AND THRESHOLD SENSITIVITY / B.16 REPRODUCIBILITY CHECKLIST

  • The router-threshold study selects thresholds on calibration data before held-out routing, while the stated threat model trusts the recorder, cache producer, and manifest binding.The paper limits completed experiments to frozen Qwen candidates, exact-string scoring, and specified mutations; independent attestation is needed to authenticate a recorder that changes both observations and digests.Source: §B.15 · p. 24, §B.16 · p. 25

B.17 FIVE-FEATURE LEARNED-GATE CACHE AUDIT

  • The five-feature learned gate matches the sequential proxy-match rate at 0.3203 but spends 15.91 ledger units; its joint-shift cost rises to 21.86 units.The separate runner reports 0.9922 route failures under large unavailability and tight budget; these are not canonical event rates.Source: §B.17 · p. 25, Table 28 · p. 25

C TASK AND SERVING EVALUATION PROTOCOL

The protocol fixes independent splits, seeds, and paired comparisons, then evaluates learned routing against sequential routing across serving conditions. Learned routing’s paired task-score intervals remain above zero and the −0.05 non-inferiority margin, including under serving shifts.

C TASK AND SERVING EVALUATION PROTOCOL

  • The protocol fixes training, calibration, and held-out splits, using independent generation seeds and shared candidate manifests and scorers within each seed.Thresholds are selected on calibration, and paired comparisons use the same queries.Source: C TASK AND SERVING EVALUATION PROTOCOL · p. 25, Table 29 · p. 26
  • Under large-service unavailability and tight budgets, p50/p95 latency ranges narrow to 40.9–63.6/71.5–121.8 ms and 40.7–61.9/70.8–118.6 ms, respectively.Coverage and abstention are reported separately.Source: C TASK AND SERVING EVALUATION PROTOCOL · p. 26, Table 30 · p. 26
  • Every learned-minus-sequential paired task-score interval lies above zero and the −0.05 quality margin; the narrowest lower bound is +0.0032 under cross-cache conditions.The widest gains occur under large-service unavailability and tight budgets, where learned routing avoids escalations sequential routing cannot complete.Source: C TASK AND SERVING EVALUATION PROTOCOL · p. 26, Figure 5 · p. 26

C.1 EVALUATION QUESTIONS

  • Serving comparisons track quality alongside coverage, failure events, token cost, and request latency, with fixed held-out request counts and a common scorer.The evaluation questions distinguish paired task-score quality, clean and mutated trace validity, and request-level cost calibration.Source: §C.1 · pp. 26–27, Table 31 · p. 27

D EVALUATION PROTOCOL AND COMPLETE RESOURCE RESULTS

The evaluation fixes data, models, scoring rules, and resource budgets before comparing policies, with calibration thresholds applied unchanged to held-out requests. Learned routing uses fewer tokens and calls while scoring higher than sequential routing on held-out requests; the serving and frozen-cache analyses separately track fallback behavior and shifted costs.

D.1 SPLIT AND BASELINE CONTRACTS

  • The evaluation fixes partitions, generation seeds, model versions, scoring rules, and matched budgets before policy comparisons.Thresholds selected on calibration are applied unchanged to held-out requests, with provenance and pairing preserved.Source: D EVALUATION PROTOCOL AND COMPLETE… · p. 27, §D.1 · p. 27, Table 32 · p. 27
  • The five policies share declared budgets, model versions, request mix, and scorer; learned routing preserves the paid-observation boundary used by the controls.The fixed cascade isolates direct escalation, while sequential routing adds a paid verifier decision and budget conditioning exposes remaining resources.Source: §D.1 · p. 27, Table 33 · p. 28

D.2 RESOURCE USE ACROSS PARTITIONS

  • On held-out requests, learned routing uses 427 tokens and 1.642 calls versus 518 tokens and 2.096 calls for sequential routing, while task score is 0.7195 versus 0.7043.The resource and task tables use the same split and policy comparison.Source: §D.2 · p. 28, Table 34 · p. 28

D.3 SERVING COVERAGE AND FALLBACK BEHAVIOR

  • Coverage and event-failure rates use distinct denominators, so an unavailable or over-budget escalation can coexist with a returned small-model answer.Paired task score indicates whether that fallback remains useful.Source: §D.3 · p. 28, Table 35 · p. 29

E ADDITIONAL RESULTS AND ANALYSIS

E.1 COMPLETE FROZEN-CACHE SHIFT LEDGER

  • Under the joint price shift, the candidate set and proxy match remain unchanged while declared ledger units increase.This comparison isolates the declared-cost change from candidate changes.Source: §E.1 · p. 29

E.2 THRESHOLD TRANSFER AND PAIRED UNCERTAINTY

The section tests threshold transfer, independently generated task outcomes, serving behavior, and audit generalization under distinct manifests and paired protocols. The audit detects independently generated faults without clean false positives, while task-score, semantic-verifier, and cost-calibration results provide downstream evidence across splits and conditions.

E.2 THRESHOLD TRANSFER AND PAIRED UNCERTAINTY

  • The separate threshold sweep finds all 60 fixed-versus-sequential quality deltas across four shifts are zero; three τa = 0.95 pairs yield 39 verifier-only stops per shift.Those pairs save cost under baseline and price shifts while retaining blocked-large events under outage and tight budgets.Source: §E.2 · p. 30

E.3 INDEPENDENT TASK-SCORING RESULT SURFACE

  • The learned-minus-sequential task-score interval is positive on train, calibration, and held-out partitions, with fewer learned-router calls on all three splits.Each comparison shares its generation manifest and scorer within a seed.Source: §E.3 · p. 30, Table 38 · p. 31

E.4 LIVE SERVING AND COST CALIBRATION

  • Serving measurements bind runtime to fixed request mix, concurrency, model versions, retry rules, and availability; robustness comparisons pair query identifiers and seeds across transfer axes.Observed reversals are analyzed against changed model, scorer, or service conditions before pooling.Source: §E.4 · p. 31, Table 39 · p. 31, §E.5 · p. 32

E.6 DECISION-VALUE AND FAULT-CONSEQUENCE EXPERIMENT

  • Aggregate-only summaries preserve misleading rankings for every injected fault family, while ledger rejection exposes recomputed rankings and quality/cost deltas.Clean, faulty, and audited rankings distinguish fault detection from a decision-changing effect.Source: §E.6 · p. 32, Table 41 · p. 32

E.7 INDEPENDENT FAULT-FAMILY GENERALIZATION

  • Recall is at least 0.9961 for every independently generated fault family, with zero clean false positives.The checker was frozen before an independent generator defined held-out fault families.Source: §E.7 · p. 33, Table 42 · p. 33

E.8 SEMANTIC VERIFIER AND CROSS-DOMAIN TRANSFER

  • Across all four domain/family pairs, semantic-verifier routing has the same direction of benefit; on GSM8K with Qwen, it combines higher task score with lower declared units and p95 latency than format control.The study uses independently labeled candidates across GSM8K, ARC-Challenge, and two model families, with labels joined only after trace freezing.Source: §E.8 · p. 33, Table 43 · p. 33, Table 44 · p. 34

E.9 SERVING EXECUTION ENVIRONMENT AND COST CALIBRATION

  • Declared cost predicts latency positively across five shifts: Spearman correlation is 0.861–0.921, Pearson correlation is 0.876–0.929, and R2 is 0.768–0.863.The largest shift residual is +2.6 ms.Source: §E.9 · p. 34, Table 46 · p. 34

E.10 ROUTER FITTING AND PAIRED UNCERTAINTY

  • The independent router is fitted on training states, its operating threshold is selected on calibration, and held-out traces are frozen before target joins.The uncertainty analysis separates query, router-seed, and producer variation; the held-out learned-minus-sequential interval remains positive under hierarchical resampling.Source: §E.10 · pp. 34–35, Table 48 · p. 35

F ARTIFACT AND REPRODUCIBILITY PACKAGE

The reproducibility package binds artifacts and comparisons to their manifests, identifiers, and scorer versions. Its replay sequence checks trace validity before scoring, while analysis guidance keeps quality, resource, route, and timing measurements distinct.

F ARTIFACT AND REPRODUCIBILITY PACKAGE

F.1 REPRODUCTION AND AGGREGATION SEQUENCE

  • Reproduction fixes candidate bindings and policy configurations, replays paid observations, checks the trace, and only then applies the scorer to the returned answer.A successful replay retains both the final answer and the event path that produced it.Source: §F.1 · p. 36, Table 50 · p. 36

F.3 ROUTE-LEVEL ANALYSIS AND AGGREGATION

  • Quality–cost comparisons use paired policy-level requests, timing comparisons use service clocks, and route-category analyses condition on the first stopping or fallback event.Verifier-stop and large-escalation rows select different query subsets, so correctness is interpreted with category membership and request counts.Source: §F.3 · p. 37, Table 51 · p. 37

F.4 CONSISTENCY OF TASK AND SERVICE SUMMARIES

  • Coverage and abstention partition requests, but service-event counts can include multiple events per request; skipped large calls can affect failure statistics while a small answer remains covered.Request latency is computed from request-to-response intervals, distinct from median queue and service times.Source: §F.4 · p. 37, Table 52 · p. 37

G MEASUREMENT INTERPRETATION GUIDE

  • The claim-to-measurement ledger binds scientific claims to proxies, structural audits, or task and serving results, preventing cost or latency fields from standing in for quality.Each comparison retains its request set, manifest, and scorer identity.Source: G MEASUREMENT INTERPRETATION GUIDE · p. 38, Table 53 · p. 38

G.1 PAIRED COMPARISONS AND DENOMINATOR HANDLING

  • Paired learned-minus-sequential comparisons use the same query and seed; held-out comparisons retain separate generation seeds and compare 1,319 requests per seed.Query clusters keep repeated seed outcomes together during resampling, and tokens, money, and calls remain distinct resource measures.Source: §G.1 · p. 38

Related papers

Loading 2610.20668v2…