PaperlayerView exact revision on arXiv

Source-linked AI summary

CopRoute: Dependence-Aware Conformal Routing for Language Models

Siva Rajesh Kasa, Santhosh Kumar Kasa, Karan Gupta

arXiv:2610.13276v1cs.LG

TL;DR

Existing risk-controlled routers bound all-miss risk but ignore dependence among model failures, while dependence-aware methods lack that certificate. CopRoute orders models by incremental coverage and calibrates a cutoff; it achieves the fewest certified calls on most evaluated datasets at matched risk.

  • Problem

    Existing risk-controlled policies score models in isolation and ignore dependence among failures, although models may fail on the same queries.

    Source: §1 · p. 2

  • Method

    CopRoute orders models by the new queries they answer, selects a per-query depth, and calibrates the smallest cutoff meeting the target risk.

    Source: §1 · p. 2

  • Results

    CopRoute has the fewest certified calls on nine of ten datasets at risk 0.05 and eight of ten at 0.10, with up to 79% fewer models than RACER-G.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The tally order has a tight approximation guarantee on design data, while separate split calibration certifies all-miss risk for deployed sets.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation uses recorded correctness labels on ten datasets and pools up to 500 models, prices calls on one benchmark, and leaves latency and response-dependent aggregation to deployment studies.

    Source: §6 · p. 10

Abstract

from arXiv · show

Set-based language model routing aims to increase the coverage of the event where at least one model answers a query correctly while controlling the number of inference calls made. Existing risk-controlled routing policies bound the all-miss risk (the probability that every model in the selected set fails) by calibrating a single conformal threshold and applying it to a nested family of model sets. Because each model is scored in isolation, these policies ignore the dependence structure of model failures. Models that share training data or architecture tend to fail on the same queries, so two individually strong models can be jointly redundant while a weaker but complementary model covers the queries that the others miss. To account for this dependence, which existing risk-controlled policies ignore, we propose CopRoute, which orders models by how many new queries each one answers correctly beyond those already selected, decides per query how far down that order to go, and then picks the smallest cutoff that still meets the target risk. We show that CopRoute's ordering attains, on the design data, the best guarantee any efficient fixed-ordering algorithm can give for the average number of calls to the first correct model, and that a separate calibration step certifies the all-miss risk of the deployed sets. On RouterBench and RouterEval, CopRoute attains the fewest certified calls on nine of ten datasets at a target risk of 0.05 and on eight of ten at 0.10, making up to 79 percent fewer calls (median 24 percent) than the state-of-the-art risk-controlled policy at the same certified all-miss risk.

1 INTRODUCTION

Existing risk-controlled routers certify all-miss risk but score models in isolation, while dependence-aware methods lack that certificate. CopRoute combines dependence-aware selection with calibrated risk control.

1 INTRODUCTION

  • Models with similar training data or architectures may fail on the same queries, making individually strong models jointly redundant and complementary models valuable.Existing risk-controlled methods ignore this dependence, while dependence-aware set methods lack distribution-free all-miss guarantees.Source: §1 · p. 2
  • CopRoute has the fewest certified calls on nine of ten datasets at risk 0.05 and eight of ten at 0.10, with median 24% and up to 79% fewer models than RACER-G.It also uses fewer calls than FlexRouter and BaRP in 45 of 48 pool comparisons.Source: §1 · p. 2
  • CopRoute orders models by the new queries they answer, chooses a per-query stopping depth, and calibrates the smallest cutoff meeting the target risk.Its policies reuse the router a system already runs.Source: §1 · p. 2
  • The paper also reports 46.6% lower end-to-end inference cost than RACER-G at equal certified risk, without lowering final-answer accuracy.Certified call counts change least among compared policies when the router is swapped or degraded.Source: §1 · p. 2

2 PRELIMINARIES ON THE α-VOR PROBLEM AND NOTATION

The α-VOR problem minimizes expected calls subject to a bound on the probability that the selected models all miss. The section defines the notation and nested cutoff sets used to express this objective.

2 PRELIMINARIES ON THE α-VOR PROBLEM AND NOTATION

  • α-VOR minimizes the expected number of called models while requiring the probability of an all-miss event to be at most the target α.The target is attainable only if the full model pool's miss probability is at most α.Source: §2 · p. 3
  • For a query, coverage is the probability that at least one model in a proposed set is correct, while all-miss probability is the probability that every model is wrong.The corresponding events are complements.Source: §2 · p. 3
  • An ordering ranks models, and a depth-k prefix contains the first k models; a sizing rule assigns that prefix an estimated residual all-miss probability.The residual is required to be nonincreasing with depth.Source: §2 · p. 3
  • A cutoff selects a nested sublevel set, and calibration chooses the smallest cutoff that passes the finite-sample rule or returns NON-CERTIFIED if none passes.Tied scores enter together, and the smallest passing cutoff selects no more models than another passing cutoff in the same family.Source: §2 · pp. 3–4

3 DEPENDENCE-AWARE ROUTING

CopRoute uses a fixed order based on incremental design-set coverage, then sizes each query's model set using router estimates and, optionally, observed dependence. A final calibrated cutoff controls all-miss risk.

3 DEPENDENCE-AWARE ROUTING

  • The tally order repeatedly adds the model that answers the most design queries missed by the current prefix.This ordering is fixed rather than query-specific.Source: §3 · p. 4
  • The stopping rules estimate residual all-miss probability along that order, using router estimates for the current query.The independence rule multiplies estimated miss probabilities; the dependence rule adjusts this product using observed co-failure on the design set.Source: §3 · p. 4
  • The dependence ratio compares observed prefix co-failure with its independence prediction; values below one indicate complementary failures and values above one indicate excess co-failure.A minimum over preceding depths keeps the estimated all-miss probability from increasing as models are added.Source: §3 · p. 4
  • CopRoute-Ind uses the independence rule, while CopRoute-Dep selects a support threshold on validation and can reproduce the independence rule when that threshold is zero.The paper reports CopRoute-Dep as an ablation of the stopping rule, not as a second main policy.Source: §3 · p. 4
  • The order and stopping rule are frozen before calibration, and the smallest passing global cutoff is deployed; if none passes, the procedure returns NON-CERTIFIED.Dependence estimates from deep prefixes can be noisy because few design queries miss an entire prefix, and this noise can increase calls.Source: §3 · p. 5

4 THEORY

The theory gives a tight approximation guarantee for the fixed tally order on design data and establishes distribution-free risk certification for a frozen nested family. Within that family, the smallest certified cutoff minimizes calls.

4.1 WHY FIXED TALLY IS A NEAR-OPTIMAL ORDERING

  • The fixed tally order achieves the best possible polynomial-time approximation factor of four for average first-correct position on the realized design matrix.No polynomial-time algorithm can guarantee a factor below 4 unless P = NP; the guarantee does not transfer to population or test-time calls.Source: §4.1 · p. 5

4.2 WHY THE FINAL ROUTING SETS ARE VALID AND EFFICIENT

  • Split calibration controls all-miss risk for a frozen nested family when the calibration queries and one future query are exchangeable and the same map is applied to each.The guarantee concerns the joint event that a certificate is issued and the certified set misses, not the miss rate conditional on certificate issuance.Source: §4.2 · p. 6
  • Among certified cutoffs in a frozen family, the smallest cutoff selects a subset of every other certified set and therefore calls the fewest models.This is an exact within-family result, not an optimality claim across different orderings or stopping rules.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments compare CopRoute with risk-controlled routing baselines at matched target risk across RouterBench and RouterEval. They also assess risk sweeps, component choices, transfer, router choice, and cost and final-answer accuracy.

Setup

  • The evaluation uses RouterBench’s 11-model pool and six RouterEval benchmarks with pools of 100, 200, 300, and 500 models; results use 20 redraws.GSM8K is excluded because it has a single distinct prompt, and MBPP because the full pool misses 13.4 percent of queries, making neither target certifiable.Source: §5 · pp. 6–7

Certified calls at matched risk

  • CopRoute has the fewest certified calls on 9 of 10 datasets at α = 0.05 and 8 of 10 at α = 0.10.At α = 0.10, MuSR needs 12.7 calls versus 44.6 for RACER-G; BBH-navigate needs 1.9 versus 6.4 for FlexRouter.Source: §5 · p. 7, Table 1 · p. 7

Risk sweep

  • CopRoute’s curve is lowest across the target-risk sweep on MuSR, BBH-navigate, and IFEval, with its advantage widening as the target loosens.Certification is infeasible when the full-pool miss rate exceeds the calibration feasibility condition, which depends on calibration-set size and full-pool misses.Source: Figure 1 · p. 7, §5 · p. 8

Transfer across datasets

  • On RouterBench, transfers cost at most 2.1 extra calls, while transfers into MuSR cost 24 to 43; averaged out of domain, CopRoute remains lowest-call on RouterBench and RouterEval at α = 0.05.Recalibrating on the deployment dataset restores the guarantee in every transfer cell.Source: §5 · p. 8, Figure 2 · p. 8

Changing the router

  • Replacing KNN with RouterDC moves CopRoute by at most 0.6 calls on any dataset, and CopRoute beats RACER-G on all ten datasets at α = 0.10 using RouterDC scores.At α = 0.10, the median absolute change for CopRoute is 0.2 calls.Source: §5 · p. 8

Alternative estimators and stopping rules

  • No alternative order improves on tally in aggregate; the closest Bernoulli-mixture order uses 0.36 to 0.42 more calls, while seeded random uses about 7 more.Dependence-corrected stopping changes nothing on 12 of 20 cells; replacing tally with design-marginal ordering costs up to 21 calls on BBH-navigate and 27 on MuSR.Source: §5 · pp. 8–9, Table 2 · p. 9

Cost and final accuracy

  • At α = 0.10 on RouterBench MMLU, CopRoute costs USD 1.90 per thousand queries versus USD 3.55 for RACER-G, with a 0.79-point accuracy gain.This is a 46.6 percent cost reduction at the same certified risk; final-answer accuracy depends on the answer selector and called set.Source: §5 · p. 9, Table 3 · p. 9

6 LIMITATIONS AND CONCLUSION

CopRoute’s co-failure ordering and separate calibration yield fewer certified calls on most evaluated datasets, while the certificate and fitted policy have defined scope boundaries. The section situates this approach among routing, set-selection, and dependence-aware methods and states when individual-accuracy ordering is optimal.

6 LIMITATIONS AND CONCLUSION

  • Nine of ten datasets at α = 0.05 and eight of ten at α = 0.10 had the fewest certified calls; CopRoute also led in 29 of 36 larger-pool cells.The summary attributes the gain to co-failure ordering; replacing it with design-marginal ordering costs up to 27 calls, while swapping routers changes calls by at most 0.6.Source: §6 · p. 9
  • The certificate bounds average all-miss risk, not per-query risk or the correctness of an aggregator’s returned answer; the fitted order and cutoff are specific to the deployment pool and distribution.The dependence-corrected stopping rule uses design queries that miss an entire prefix, which become few at deep prefixes; evaluation also prices calls on one benchmark.Source: §6 · pp. 9–10, ETHICS STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Most prior routers select one model or follow a prescribed sequence, while set-valued risk-controlled methods calibrate families built from individual-model scores.Complementary-set methods address redundancy or coverage, but the cited FlexRouter objective does not provide a distribution-free all-miss certificate; aggregation methods instead combine responses already obtained.Source: A EXTENDED RELATED WORK · pp. 16–17

C.1 FORMAL JOINT TARGET AND INDEPENDENCE BENCHMARK

  • For binary correctness, the correctness subcopula is identified only on the attainable marginal grid; a full copula extension away from that grid is generally nonunique.The subcopula separates per-model accuracies from the co-failure structure.Source: A EXTENDED RELATED WORK · p. 17, §C.1 · p. 20
  • Under conditional independence and equal call costs, ordering by individual accuracy minimizes all-miss probability at every set size; under dependence, the best set need not contain the individually best models.Independence is sufficient but not necessary for individual-accuracy ordering to remain optimal.Source: §C.1 · p. 20

C.2 ORDERING PROOF

The ordering proof casts the design tally as greedy minimum-sum set cover, yielding the best polynomial-time approximation factor for a fixed ordering unless P = NP. Separate split calibration certifies the all-miss risk of the frozen family, with a sharp finite-sample rank correction.

C.2 ORDERING PROOF

  • 4×: the tally order is greedy minimum-sum set cover, and no polynomial-time fixed-ordering algorithm guarantees 4−ε on arbitrary correctness matrices unless P = NP.The hardness follows by reducing minimum-sum set cover to the correctness-matrix problem.Source: §C.2 · p. 21
  • On design data, the same tally ordering has both a per-depth coverage guarantee and a whole-order guarantee, which compare different selection objectives.The per-depth comparison is against the best set at that depth; the whole-order comparison is against the best complete ordering.Source: §C.2 · p. 21

C.3 CALIBRATION PROOF

  • Once the routing map is frozen, split calibration certifies all-miss risk; the smallest certified cutoff selects no more models than any other certified cutoff in the family.This pointwise set inclusion also preserves the inequality for any call cost nondecreasing under set inclusion.Source: §C.3 · p. 22

C.3.1 EXACT RANK FORM AND SHARPNESS

  • Below α = 1/(n + 1), no depth can be certified; otherwise attainable deterministic levels are multiples of 1/(n + 1), and the additive correction is sharp.Accepting one more calibration miss than the rule allows can exceed the target risk under an exchangeable law.Source: §C.3.1 · p. 23

D.1 ORACLE AND SCOPE RESULTS

The appendix distinguishes cardinality-optimal sets from a single usable order, then bounds when the design-data ordering guarantee transfers and what structured-estimator consistency establishes. Its results are conditional: oracle solutions need not nest, population transfer can become uninformative at large pools, and the parametric consistency result does not cover misspecification or the tally order.

D.1 ORACLE AND SCOPE RESULTS

  • An optimizer at size k + 1 need not contain any optimizer at size k, so independently optimal sets across sizes do not automatically form one routing order.The passage identifies these cardinality-specific optima as useful lower bounds, not a nested chain.Source: §D.1 · p. 24
  • The best singleton and best pair can be non-nested: model 1 is the unique best singleton, while {2, 3} is the unique best pair.The example uses three models and four population atoms.Source: §D.1 · p. 24
  • For one target α, an existence result permits a target-specific nested order to attain the smallest qualifying prefix, but another target may require another order.It does not construct an estimated policy or a chain attaining every q*_k.Source: §D.1 · p. 24
  • The cost-aware greedy order is a heuristic, not a cost minimizer, and calibration certifies its all-miss risk without supplying a cost-approximation ratio.The family must be selected without final-calibration labels and frozen before certification.Source: §D.1 · p. 25

D.2 POPULATION TRANSFER OF THE ORDERING GUARANTEE

  • With iid design queries, the tally order has a population-transfer bound with an additive sampling term, but that term is vacuous for the largest pools.The bound is informative at eleven-model pools but reaches tens of calls at 100 models and is comparable to pool size at 500; redraws measure generalization there.Source: §D.2 · p. 25
  • The transfer guarantee concerns mean first-success depth, not certified calls at a fixed target; the latter remain empirical, while test-time risk is covered by Theorem 2.A mean-competitive order can still reach the target level late.Source: §D.2 · p. 26

D.3 STRUCTURED-ESTIMATOR THEORY

  • For fixed m, a correctly specified parametric joint model fitted consistently recovers all-miss probabilities for every set in the limit, but this result gives no guarantee under misspecification.It does not apply to the tally order or establish a small-sample advantage; decision convergence also requires separation from ties or vanishing near-tie probability.Source: §D.3 · pp. 26–27

E IMPLEMENTATION DETAILS

The implementation fits a fixed order on design data, chooses stopping and other settings on validation data, and reserves final calibration for selecting the global cutoff. The appendix also specifies alternative dependence estimators and the certification protocol used to compare them.

E IMPLEMENTATION DETAILS

  • The design split is separate from router training, validation, final calibration, and test, and the score map and routing choices are frozen before final calibration selects only the global cutoff.Held-out calibration labels do not select the order or dependence objects.Source: E IMPLEMENTATION DETAILS · pp. 27, 29
  • CopRoute-Ind stops using the independence residual, while CopRoute-Dep adjusts it with a design-set dependence ratio and a support threshold selected from {0, 5, 20, 50}.The cumulative minimum keeps the estimated residual from increasing with depth; h = 0 recovers independence-based stopping.Source: E IMPLEMENTATION DETAILS · pp. 27–28
  • The main CopRoute policies use a fixed order with query-specific selected depth; alternative estimators include pairwise, correlation, latent-threshold, mixture, and shrinkage constructions.The latent-threshold family includes Gaussian, Student-t, and low-rank factor variants; the low-rank form addresses the inability to form a full correlation matrix for 3,811 models from a few hundred design labels.Source: E IMPLEMENTATION DETAILS · p. 28, F ESTIMATOR DEFINITIONS FOR THE… · pp. 29–30
  • The fixed tally order repeatedly adds the model that is wrong least often on design queries missed by the current prefix, then resolves ties by design accuracy and model index.If a prefix covers every design query, remaining models are appended by descending design accuracy and then index.Source: E IMPLEMENTATION DETAILS · p. 27, F ESTIMATOR DEFINITIONS FOR THE… · p. 29

F ESTIMATOR DEFINITIONS FOR THE COPROUTE FAMILY

  • The lineage, accuracy-weighted random, and uniform-random controls differ in whether they use model-family prefixes, design accuracy, or no model information.The lineage control uses name-prefix families as a proxy for shared failure structure.Source: F ESTIMATOR DEFINITIONS FOR THE… · p. 30

G PROTOCOL DETAILS AND CLAIM BOUNDARIES

  • The primary certification loss is the standard all-miss loss, and a threshold passes when (c + 1)/(n + 1) ≤ α; if none passes, the draw is NON-CERTIFIED.Efficiency summaries use certified redraws and report certification rates; structural infeasibility and finite-calibration non-certification are reported separately.Source: G PROTOCOL DETAILS AND CLAIM BOUNDARIES · p. 30
  • The synthetic conditional-independence diagnostic finds pooled failure correlation of 0.152 but residual correlation of 0.0008, while all three compared estimators use 2.0 calls.Their risks are 0.0656, 0.0657, and 0.0657, respectively, so pooled association alone does not indicate exploitable residual dependence.Source: G PROTOCOL DETAILS AND CLAIM BOUNDARIES · p. 31

H DETAILED CALIBRATED RESULTS

The detailed RouterEval results cover 200-, 300-, and 500-model pools, while MATH-intermediate-algebra is excluded from the main comparison because no policy certifies at its main targets. Its full-pool calibration miss rates explain that infeasibility.

H DETAILED CALIBRATED RESULTS

  • MATH-intermediate-algebra is excluded from the main comparison because no policy certifies at any pool for the main targets.At pools 100, 200, 300, and 500, the full pool misses 40.4, 34.6, 25.4, and 22.1 percent of calibration queries, respectively.Source: H DETAILED CALIBRATED RESULTS · p. 31
  • At looser targets, MATH-intermediate-algebra certifies only on some redraws: at pool 500 for α ≥ 0.15 and at pool 300 for α = 0.20.Those are the settings where the full-pool miss rate falls below the target.Source: H DETAILED CALIBRATED RESULTS · p. 31
  • At α = 0.02 on the hardest pool-100 benchmarks, certification falls for every policy; on the few certified redraws, dependence-corrected stopping reduces calls on MATH-algebra and IFEval.Calls fall from 55.1 to 39.2 on MATH-algebra and from 78.4 to 61.4 on IFEval, but one and three redraws provide no interval for these differences.Source: H DETAILED CALIBRATED RESULTS · p. 31

I COMPLETE ESTIMATOR GRIDS FOR THE COPROUTE FAMILY

Across the estimator comparisons, the tally order is the strongest aggregate default, while dependence-corrected stopping adds little consistently and can increase calls. Relative-efficiency results show CopRoute leading most RouterEval axes, with FlexRouter leading the remainder.

I COMPLETE ESTIMATOR GRIDS FOR THE COPROUTE FAMILY

  • No alternative order improves on the tally order in aggregate under either stopping rule; the roughly seven-call gap to seeded random does not isolate co-failure information.Random ordering also discards marginal quality, and it beats tally in some GPQA and BBH-navigate aggregates.Source: I COMPLETE ESTIMATOR GRIDS FOR THE… · pp. 31–33, Table 8 · p. 34
  • CopRoute is on the rim of Figure 4 on 29 of 36 axes, with FlexRouter on the remaining seven.The figure compares relative call efficiency across six RouterEval datasets, 200-, 300-, and 500-model pools, and target risks.Source: Figure 4 · p. 32
  • The Bernoulli mixture and Student-t copula beat tally at 100 models but lose at larger pools, while correlation-based orders win some cells and lose by tens of calls elsewhere.The larger-pool design budget supplies fewer observations per prefix, where the tally count is reported as more reliable.Source: I COMPLETE ESTIMATOR GRIDS FOR THE… · pp. 32–33, Table 8 · p. 34
  • The dependence-corrected stopping rule does not consistently improve calls: across 187 certified cells it ties CopRoute-Ind on 104, lowers calls on 21, and raises them on 62.The correction’s gains are small or based on a single redraw, while six differences against it reach 17 calls; replacing the tally order with the design-marginal order costs up to 21 calls on BBH-navigate and 27 on MuSR.Source: I COMPLETE ESTIMATOR GRIDS FOR THE… · p. 33, Table 7 · p. 33
  • Alternative-order wins concentrate on GPQA and BBH-navigate, while tally is strongest or tied elsewhere; error correlation gains 20 calls on MuSR and 11 on GPQA but loses 97 on MATH-algebra.Table 9 presents dataset-wise extra certified calls relative to tally, weighted across each dataset’s pools and targets.Source: I COMPLETE ESTIMATOR GRIDS FOR THE… · p. 33, Table 9 · p. 34

J TRANSFER ACROSS DATASETS FOR ALL POLICIES

The transfer analysis compares policies after fitting on one dataset and recalibrating on another; supplementary native learned-baseline metrics are separate from the common set-routing comparison.

J TRANSFER ACROSS DATASETS FOR ALL POLICIES

  • RouterEval transfer excludes source-target pairs from the same benchmark family for BBH-boolean and BBH-navigate, and omits MATH-intermediate-algebra because no policy certifies there.Source and target share a model pool, so transfers are only within RouterBench or within RouterEval.Source: J TRANSFER ACROSS DATASETS FOR ALL… · p. 34
  • CopRoute-Ind has the lowest mean calls on RouterBench at both targets and on RouterEval at α = 0.05, and is within 1.1 calls of FlexRouter at RouterEval α = 0.10.Reusing a source-calibrated cutoff raises realized risk on RouterEval by 0.12 to 0.22, so each policy requires target recalibration.Source: J TRANSFER ACROSS DATASETS FOR ALL… · p. 35, Table 10 · p. 35, Table 11 · p. 35

K SUPPLEMENTARY NATIVE LEARNED-BASELINE METRICS

  • Native learned-baseline metrics are supplementary characterization; the common comparison evaluates learned baselines under the same calibrated standard-loss rule as other policies.The native BaRP run omits GSM8K because its prompt field has at most one distinct prompt, and keeps HellaSwag and MBPP as held-out targets.Source: K SUPPLEMENTARY NATIVE… · p. 35

L COST, ANSWER SELECTION, AND FURTHER ANALYSES

The section examines answer cost, cost-aware stopping, comparisons with cascades, and further analyses of ordering, pool shifts, router degradation, and dependence structure.

DISTANCE OF THE TALLY ORDER FROM THE BEST FIXED ORDER

  • The tally order is within 0.02 calls of the exact design-objective optimum across RouterBench redraws, consistent with Theorem 1’s factor-four guarantee being loose on these instances.Exhaustive dynamic programming gives an exact optimum; beam-search and local-swap comparisons include zero in their bootstrap intervals.Source: L COST, ANSWER SELECTION, AND FURTHER… · p. 37

POOL COMPOSITION SHIFT

  • Removing one model preserves only 3.4 percent of frozen certificates, while refitting and recalibrating certifies 63.1 percent of redraws at a mean premium of 1.07 calls.Adding one model leaves every frozen certificate valid; deployment-pool changes call for refitting and recalibration.Source: L COST, ANSWER SELECTION, AND FURTHER… · p. 37

SENSITIVITY TO ROUTER DEGRADATION

  • On RouterBench MMLU at α = 0.10, CopRoute-Ind’s certified calls rise by at most +0.38 under router transformations, while realized risk stays within sampling error of target.Under strongest IFEval noise, exact independence rises by +0.59 calls and RACER-G by +1.13.Source: L COST, ANSWER SELECTION, AND FURTHER… · p. 37

COST AND END-TO-END ACCURACY

  • At α = 0.10, CopRoute-Ind answers 80.97 percent of queries at USD 1.90 per thousand, versus 80.18 percent at USD 3.55 for RACER-G, a 46.6 percent cost reduction.Final-answer accuracy depends on the answer selector and called set; the strongest single model misses 18.6 percent of queries, above both risk targets.Source: L COST, ANSWER SELECTION, AND FURTHER… · p. 35, Table 15 · p. 38

SYNTHETIC MECHANISM MAP

  • In synthetic regimes with residual dependence, winning orderings track the dependence form: pairwise count wins 14 of 15 cells for redundant families, while only full-joint estimators win in the higher-order regime.With complementary specialists, query-local ordering wins from the smallest budget; neither dependence-aware ordering wins in the two no-residual-dependence controls.Source: L COST, ANSWER SELECTION, AND FURTHER… · pp. 38–39

COST-WEIGHTED STOPPING

CASCADE ROUTING WITHOUT RESPONSE SIGNALS

  • The pre-response cascade exceeds the risk target in seven of eight cells, including 0.350 against a matched 0.110 on MATH-algebra at α = 0.10.Without response signals, the cascade reduces to a router and provides no distribution-free certificate; cascades and CopRoute optimize different objectives.Source: L COST, ANSWER SELECTION, AND FURTHER… · pp. 36–37, Table 17 · p. 39

M LIMITATIONS

The limitations concern what the certificate and theory guarantee, dependence on the deployment setting, tail estimation, and the scope of the experimental evidence.

WHAT THE CERTIFICATE COVERS

  • The certificate controls average deployment-distribution risk that all called models miss, not per-query risk or the correctness of a downstream aggregated answer.At tight targets on hard benchmarks, no policy certifies when the full pool misses too many calibration queries.Source: M LIMITATIONS · p. 39

DEPENDENCE ON THE DEPLOYMENT POOL AND DISTRIBUTION

  • The order is fitted per benchmark and pool, and the cutoff per deployment; changing the pool or query distribution requires refitting and recalibration.Removing a model requires both steps, while adding one leaves frozen certificates valid; refitting and recalibration after a pool change cost about one call on average.Source: M LIMITATIONS · p. 40, Table 18 · p. 40

DEPENDENCE-CORRECTED STOPPING

  • CopRoute-Dep’s tail correction is noisy because design queries missed by the entire prefix become rare where correction matters, so CopRoute-Ind remains the default.A structured tail estimator or a design set several times larger may support an improved stopping rule.Source: M LIMITATIONS · p. 40

SCOPE OF THE THEORY

  • Theorem 1 guarantees the design mean position of the first correct model under a fixed order, not certified calls at a fixed target; deployment calls are empirical findings.The number of certified calls depends on a quantile of first-success depth rather than its mean.Source: M LIMITATIONS · p. 40

EVALUATION ON RECORDED LABELS

  • Experiments use recorded correctness labels rather than new model queries, leaving latency, response-dependent cascades, and most answer aggregation outside the presented evidence.The cost case study prices calls on one benchmark using a public price sheet.Source: M LIMITATIONS · pp. 40–41

N REPRODUCIBILITY

The release provides implementations, split and selection logic, hyperparameter grids, seed manifests, certification status, and scripts to regenerate reported tables and figures.

N REPRODUCIBILITY

  • The release includes aggregated main-comparison results and regeneration scripts, but omits per-redraw artifacts for transfer, pool-shift, router-degradation, stopping-rule, and synthetic analyses because of their size.The omitted tables can be reproduced by rerunning the corresponding scripts on public benchmark data.Source: N REPRODUCIBILITY · p. 41

Related papers

Loading 2610.13276v1…