PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

Josef Chen

arXiv:2606.27288v1cs.AIcs.LG

TL;DR

Multi-model orchestration lacks a reliable measure of its attainable gain because pairwise error correlation misses shared failures. The paper formalizes and measures the all-models-wrong rate, finding that routing gains are largely unrealized while matched-quality, low-correlation ensembles benefit from failure-mode diversity.

  • Problem

    Pairwise error correlation is widely used to assess model diversity, but it does not identify the joint failures that cap orchestration accuracy.

  • Method

    The paper treats orchestration as allocation over a correlated model pool and uses the all-models-wrong rate β as a realizability ceiling and pre-deployment certificate.

  • Results

    Oracle routing gains were positive but learned routing realized almost none, while matched-quality lower-correlation combinations helped and naive fusion could hurt.

  • Takeaways & Limitations

    On the evaluated pool, orchestration value comes from failure-mode heterogeneity rather than adding models, with β providing the relevant ceiling for achievable gain.

  • Takeaways & Limitations

    The empirical conclusions are restricted to this pool and verifiable tasks, and the router captures approximately zero of the oracle gain there.

Abstract

from arXiv · show

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.

1 Introduction

The paper argues that β, the rate at which all models fail on the same query, is the fundamental ceiling on routing, voting, and cascading, whereas pairwise error correlation ρ cannot reveal it. It develops a finite-sample certificate and measures the resulting gap across 67 models from 21 provider families, while adding supporting economic analyses.

  • Motivation: Production orchestration increasingly allocates token and dollar budgets across heterogeneous, correlated models rather than selecting a single best model.Routing mediates cost, latency, reliability, capability, spend, governance, and provider risk.
  • Core contribution: The orchestration ceiling is 1−β: no router, vote, or cascade can exceed it, and oracle gain equals Pr[single-best wrong]−β.A Clopper–Pearson bound on β converts one graded query sample into a certificate of the largest possible gain.
  • Core contribution: Pairwise ρ cannot identify β; under tail dependence, estimating β from ρ is downward-biased, with bias diverging as pool size grows.The bias is attributed to a common-mode atom rather than tail dependence alone.
  • Empirical scope: 67 models from 21 provider families show positive oracle routing gain, yet a learned router realizes almost none of it.The study directly measures the β/ρ gap and its growth with pool size, identifying ceiling-bound and realizability-bound regimes across domains.
  • Supporting analyses: Supporting analyses treat budget-constrained routing as priced assignment, derive cost-aware diversification limits, and establish cascade calibration boundaries.These results specialize standard tools and are deferred to the appendix, alongside an observational option value of breadth under churn.

2 Related Work

Prior work studies routing, cascades, ensembling, fusion, and correlated errors, while classical ensemble theory supplies the Oracle ceiling. This paper extends that foundation by showing the ceiling applies to member-answer selection policies and by certifying it without labels.

  • Routing and cascades: Learned routers select one model per query, cascades escalate from cheap to strong models, and confidence-based deferral can be sub-optimal when downstream errors are unmodeled.Dekoninck et al. unify routing and cascades and prove optimality results; Jitkrittum et al. characterize optimal two-model deferral.
  • Ensembling and fusion: Ensembling research includes rank-and-fuse, layered aggregation, sampling-and-voting, and Self-MoA, whose advantages reflect a quality–diversity trade-off.Classical theory cautions that diversity does not guarantee accuracy gains and that heterogeneous mixing helps only when members have similar quality.
  • Accuracy ceiling: 1−β is the classical Kuncheva Oracle-combiner ceiling, but this paper applies it to every selection policy producing member answers, including debate and self-consistency.The ceiling predates LLMs and presumes labels; the paper’s extension covers broader member-answer policies.
  • Accuracy ceiling: A labels-free, finite-sample certificate converts the ceiling into a practical bound, unlike the Oracle combiner, which requires labels.The paper identifies this certificate as a distinct contribution rather than claiming novelty for the ceiling itself.
  • Error correlation and inference economics: Pairwise error correlation rises with accuracy and shared provider but cannot identify β for m ≥3; β, rather than ρ, bounds selection-policy headroom.Prior work measured bivariate agree-when-wrong statistics on multiple-choice tasks and stopped at qualitative implications.

3 Problem Formulation

The formulation models queries by latent types and assigns models with type-dependent quality and per-query prices. It evaluates routing, fusion, and cascading under dollars-per-correct or budget-constrained quality objectives, with economic results deferred to an appendix.

  • Problem formulation: Queries have latent types, while each model has type-specific quality q_i(t), price c_i, and average quality q̄_i.A routing policy maps types to distributions over the model pool.
  • Problem formulation: The framework considers routing, fusion, and cascading, optimizing dollars-per-correct or quality subject to a budget.The objective is made explicit in each section.
  • Economic scaffolding: The appendix casts orchestration as allocation, including priced assignment under budgets and correlation-dependent diversification limits for cost-aware fusion.These economic tools are standard scaffolding and do not determine the empirical results.

4 Experimental Setup

The pre-registered experiments evaluated 15 models across nine provider families on five benchmarks with programmatic grading and frozen model snapshots. A market-scale measurement expanded the pool to 67 models across 21 provider families, with costs metered per call.

  • Model pool: 15 models across 9 provider families formed the pre-registered experimental pool.The pool included frontier, mid, and cheap model tiers, with exact dated snapshots and prices frozen in the registry.
  • Benchmarks: 5 benchmarks covered a saturated mix of GSM8K, MMLU, ARC-Challenge, and MATH-500 plus the harder MMLU-Pro set.Each dataset contributed 100–200 queries.
  • Market-scale measurement: 67 models across 21 provider families supported the market-scale realizability measurement.This expansion used the live OpenRouter catalog, spanning current frontier models to small open-weights.
  • Evaluation and costs: Programmatic grading used exact-match arithmetic, multiple-choice and boxed-letter extraction, and boxed/integer answer matching without an LLM judge.Costs were metered per call against the OpenRouter account usage endpoint.
  • Evaluation and costs: ≈$47 covered the core pillar experiments, ≈$111 the market-scale realizability and two-regime measurement, and ≈$110 the code and open-ended GPQA experiments.The costs were itemized per run in Appendix C.

5 Results

Combining models is limited by the pool’s all-wrong tail: every member-answer policy has accuracy ceiling 1 − β, while pairwise-calibrated models substantially underprice β as pools grow. Empirically, routing gains are near zero despite positive oracle gains, because query prompts provide little signal about which model will be correct.

  • Routing results: 0.044 and 0.120 are positive oracle gains in saturated and hard regimes, but learned routers capture essentially none of the available gain.On the mix, TF-IDF+domain logistic reaches 0.906 versus single-best 0.901, capturing 9% of G with 95% CI [−0.67, 0.50].
  • Co-failure ceiling: 1 − β is the exact accuracy ceiling for routers, weighted votes, and cascades whose outputs are member answers; the maximum gain is Δceil = (1 − β) − asb.The ceiling is attained by the per-query oracle, and the certified bound can provide a $0 pre-deployment test when overhead exceeds it.
  • Market-scale tail: 0.052 is the observed all-wrong rate on 330 fully covered MATH-500 queries, versus 0.023 under the full-Σ Gaussian copula, with k = 17 events.The empirical rate has Clopper–Pearson interval [0.030, 0.081], while the Gaussian copula underprices the tail by 2.25×.
  • Pool-size effect: 2.5 is the median tetrachoric underpricing ratio at k = 67, rising monotonically from 1.0 at k = 2 across random pool subsets.The 5–95% band at k = 67 is [2.1, 2.7], indicating pool size rather than model composition drives the excess.
  • Cross-task replication: β > 0, Pearson calibration artifacts, and full-Σ residuals recur across open-ended math and execution-graded code but vanish on multiple-choice tasks.The results identify a cross-domain open-ended-versus-multiple-choice split rather than a math-specific effect.

6 Discussion

The discussion frames model combination as a two-timescale allocation problem: static assignment within a release epoch and option value across frontier releases. Its empirical lesson is that failure-mode heterogeneity, rather than model count, drives gains, while realizable routing offers little improvement on the 2026 frontier.

  • Two-timescale allocation: Within a release epoch, the buyer solves a budget-priced assignment capped by the realizability ceiling; across epochs, frontier releases create an option value for breadth.The decomposition separates static allocation from cross-epoch renewal and real-options dynamics.
  • Routing: Routing value is a first-moment selection effect, but the 2026 frontier shows small oracle gains and near-zero realizable routing gains.A held-out learned router captures ∼9% of G, with a confidence interval spanning zero, and barely exceeds single-best.
  • Fusion and heterogeneity: Naive fusion is a net liability when today’s best models agree, whereas matched-quality ensembles gain from lower error correlation.The discussion identifies failure-mode heterogeneity—not model count—as the operative lever.
  • Matched-quality comparison: From information-fair k=3 onward, the ρ=0.42 heterogeneous ensemble beats ρ=0.80 Self-MoA in the matched 6-model band.Under distinct-draw aggregation, the query-bootstrap 95% CI excludes zero, supporting diversification in this regime.

7 Limitations

The study’s limitations concern grading validity, benchmark saturation, changing model prices, and non-optimal cascade verification. Code results remain uncertain because they rely on only k=5 events in an 18-model pool, motivating larger replications and stronger evaluation protocols.

  • Evaluation and assumptions: Programmatic grading covers verifiable tasks, can mildly penalize verbose models through extraction heuristics, and open-ended evaluation would reintroduce judge bias.Saturated benchmarks also inflate ρ, while the hard regime mitigates but does not eliminate this effect.
  • Evaluation and assumptions: Static-price assumptions conflict with model churn, so the associated claims are restricted to within-release-epoch validity.The limitation reflects tension between the static assumptions in App. A.1–A.3 and churn documented in App. E.
  • Cascade limitations: The cascade results do not establish the optimal policy: they use a naive confidence cascade and a verifier scoring only the cheap model, a provably dominated choice.The optimal deferral rule conditions on both models, whereas the reported single-model AUC verifier is practical but weaker.
  • Replication priorities: k=5 events under a strict-but-not-official judge on an 18-model pool leave the code magnitude and its point ratio materially uncertain.The proposed sharpening is an official hidden-test replication at scale on the full pool, with logged prompts for an in-domain router.
  • Replication priorities: The remaining submission priorities include a tight-ratio code replication, ≥3 seeds, and held-out model selection.These safeguards are listed alongside the need for stronger code evaluation and router validation.

8 Conclusion … A.2 The cost-aware diversification limit

The paper frames orchestration as allocation over correlated, costly models, with achievable gains governed by failure-mode dispersion rather than model count or pairwise correlation alone. Its cost-aware analysis shows that diversification has a finite, correlation- and cost-dependent limit, while the empirical scope remains concentrated on verifiable tasks.

  • 8 Conclusion: Orchestration gains depend on failure-mode dispersion and market churn, not simply peak capability or model count, especially when models increasingly fail alike.Pairwise error correlation is blind to the joint failures that determine the ceiling, and the paper cautions that generalization beyond verifiable benchmarks remains open.
  • A Economic scaffolding: routing, diversification, and cascades: The economic scaffolding treats orchestration as allocation over a correlated, priced, and changing model pool.The appendix presents this framework as standard economic scaffolding rather than the basis for the paper’s empirical claims.
  • A.1 Routing as priced assignment: A routing oracle exceeds the best single model exactly when no model is uniformly best across query types, although learned-router performance is reduced by routing regret.The oracle gain is G := Vo − maxi q̄i, and a learned router beats the single best model when its expected routing regret R is below G.
  • A.1 Routing as priced assignment: Under a dollar budget, optimal routing assigns each query to the model maximizing qi(t)−λBci, where λB is the budget’s shadow price.The value function is nondecreasing, concave, and piecewise linear, with λB equal to its derivative wherever differentiable.
  • A.2 The cost-aware diversification limit: Under the single-factor probit, equal-weight fusion and majority voting have nonzero error floors determined by correlation, limiting the benefit of adding models.Fusion variance approaches ρσ2, while the majority-vote floor approaches 0 as ρ→0 and α as ρ→1.
  • A.2 The cost-aware diversification limit: The largest cost-aware ensemble size decreases with inter-model correlation and per-model cost, with ρ→1 implying k*→0.The optimal cardinality is the nearest feasible integer to k*+1, based on the local sensitivity λ of correctness to fused-estimate variance.
  • A.2 The cost-aware diversification limit: The cost-aware rule is valid only when λ>0; if λ<0, reducing variance worsens expected correctness and the diversification framing fails.The paper identifies λ as an operating-point Jacobian rather than a free risk-aversion parameter and verifies the sign condition numerically.
  • A.2 The cost-aware diversification limit: For block-structured errors, query-conditional correlation and the covariance matrix Σ replace global equicorrelation, making equal weights generally suboptimal.Minimum-variance weights are w ∝Σ^-1 1, and the remaining floor is the undiversifiable common-factor component of Σ.

A.3 Cascade calibration economics

Cascade calibration can reduce strong-model calls only when deferred queries are selectively difficult and the verifier has sufficient discrimination. Dominance over strong-model-only deployment additionally requires a positive deferred-tail accuracy edge, not calibration alone.

  • Cascade economics: C(β) = cL + βcH and Q(β) = aL + β[aH(τ) −1 + w(β)] describe cascade cost and quality as functions of escalation budget.The cascade escalates to H when the cheap model’s confidence falls below τ, with β = Pr[s < τ].
  • Calibration advantage: βcas/βmix →1 −aL/aH under perfect calibration, so calibrated cascades need only a fraction 1 −aL/aH as many strong-model calls as random mixing at fixed quality floor.This call reduction is independent of price.
  • Verifier quality: βcas(q) decreases with verifier AUC, and a critical AUC∗ exists below which no threshold achieves q ∈(aL, aH] at lower dollars-per-correct than H alone, given cL/cH < q/aH.The result applies within a one-parameter family of verifiers ordered by AUC.
  • Dominance condition: Cascade dominance over H-only requires both AUC(s) > AUC∗ and aH(τ) > aL(τ) on the deferred tail.If H is adversely weak where L defers, calibration alone is insufficient.

B Proofs

The proofs establish that selection policies face an accuracy ceiling of 1 − β, while pairwise error statistics cannot determine β. They also certify this ceiling from finite samples and characterize when routing, voting, model addition, and escalation can improve performance.

  • Model addition: Adding the (k + 1)-th model is worthwhile iff k(k + 1) ≤ R, where R = λσ²(1 − ρ)⁄c, so the paying ensemble size shrinks as ρ approaches 1.The marginal variance reduction is ΔV(k) = σ²(1 − ρ)⁄[k(k + 1)].
  • Budgeted routing: The budget-constrained optimal policy chooses models maximizing qi(t) − λBci at each query, with mixing only to exhaust the budget.Its value is nondecreasing, concave, and piecewise linear in the budget, becoming the per-query oracle once the budget is slack.
  • Co-failure ceiling: 1 − β is the maximum accuracy of any policy that outputs one member answer, attained by the per-query oracle.On the all-wrong event, every member answer is wrong, so selection cannot succeed.
  • Finite-sample certificate: With probability at least 1 − δ, the Clopper–Pearson lower limit βlo certifies every selection policy’s gain over single-best accuracy is at most (1 − βlo) − asb.A union bound can replace asb with an upper confidence bound, with total error at most δ + δ′.
  • Tail dependence: β(m)⁄βsf(m) → ∞ as the number of models m grows when common-shock dependence has β∞ = π > 0 but the matched Gaussian copula has βsf∞ = 0.The Gaussian copula’s conditional error probability is strictly below one, producing zero lower-tail dependence.
  • Pairwise non-identification: β is not determined by pairwise error laws: identical one- and two-dimensional marginals can yield β = 0 or β = 1⁄4.The construction uses exchangeable error triples with the same pairwise Pearson and tetrachoric correlations.

C Reproducibility · D The market-scale model pool

The paper releases the data, code, registries, graders, and logged run details needed to reproduce its analyses and costs. Its market-scale pool comprises 67 chat/instruct models across 21 families, with dated live prices and reasoning variants excluded.

  • C Reproducibility: Released outcome matrices, model registries, programmatic graders, and analysis scripts enable end-to-end regeneration of every reported number.The release includes graded correctness, token counts, metered costs, dated snapshots, and live prices.
  • C Reproducibility: ≈$47 covers core pillar experiments, while ≈$111 covers the market-scale measurement runs.Costs are summed from logged per-cell charges metered through the OpenRouter account usage endpoint.
  • C Reproducibility: β = 0.052, 2.5× tetrachoric underpricing, k=17, and n=330 are the headline MATH-500 reproducibility figures.These figures are computed on the common-coverage subset of all 67 models.
  • C Reproducibility: GPQA uses a 52-model complete-coverage subset, while the default ≥0.95-coverage filter yields a smaller valid subset with the same underpricing order of magnitude.The paper also provides scripts for truncation control and cascade-collapse analysis.
  • D The market-scale model pool: 67 models from 21 families form the market pool, using live OpenRouter prices from snapshot 2026-06-19.All models are chat/instruct systems; pure reasoning or thinking variants are excluded for clean finite-token programmatic grading.
  • D The market-scale model pool: The market-pool tables organize each model by family and tier and report input and output prices in dollars per million tokens.The pricing columns are labeled $/Mtok in and $/Mtok out.

E Optionality under churn (secondary)

Under model churn, broad access has additive option value that increases with arrival intensity and capability dispersion, while rising error correlation erodes diversification gains. Observational timeline evidence suggests large cost-per-correct improvements and task-dependent accuracy benefits, but comparative-statics assumptions remain limiting.

  • Optionality under churn: Broad access captures positive capability gaps across frontier releases, while self-hosting earns a premium but incurs switching cost.Releases arrive as a Poisson process with churn rate ν; relative capability gaps follow Γ = µ + ηZ.
  • Optionality under churn: The build-versus-route threshold is δ∗=rK+vν/(r+ν)Eg, with total value m(level)+VR when absolute capability is separable.Under separability, option value rises with both churn intensity ν and capability dispersion η.
  • Optionality under churn: As error correlation approaches 1, diversification gains shrink; with declining common error, orchestration value G and VR contract.The paper verifies the direction of rising correlation, not a unit-correlation limit.
  • Optionality under churn: 14–15×: best achievable dollars-per-correct dropped across an 18-model generational release pool, using metered rather than graded cost.The timeline ran from Claude-3-Haiku in March 2024 to Gemini-3.1-Pro in February 2026.
  • Optionality under churn: +0.33 accuracy on hard MMLU-Pro versus +0.01 on saturated GSM8K shows regime-dependent broad-access value by 2026.This is an observational single-path study; the grading-independent cost-churn result is described as the robust finding.

F The third domain: execution-graded competitive code · G Content-controlled format test: open-ended GPQA (LLM-judge panel)

Execution-graded competitive code reproduces the co-failure signature in a disjoint open-ended domain, while a content-controlled GPQA test shows that changing only answer format reopens the co-failure tail. These results support a format-linked regime split, with code magnitudes remaining uncertain because only five all-wrong events were observed.

  • F The third domain: execution-graded competitive code: Competitive programming tests whether co-failure extends beyond mathematics using open-ended generation, programmatic grading, and a disjoint task family.Problems were drawn from Codeforces-rated contests and screened so accepted Python-3 references passed the grader, reducing grader-artifact concerns.
  • F The third domain: execution-graded competitive code: 0.079 β occurred across 63 code problems, with 5 all-wrong cases and a Clopper-Pearson interval of [0.026, 0.176].Mean accuracy was 0.45, indicating a genuinely difficult benchmark.
  • F The third domain: execution-graded competitive code: 3.1× underpricing remained after tetrachoric calibration, while the full Σ Gaussian copula retained a 1.7× residual.The naive Pearson calibration instead implied a spurious 17× underpricing, reproducing the common-mode pattern seen in mathematics.
  • F The third domain: execution-graded competitive code: The code result establishes β > 0 across domains, but its precise magnitude remains uncertain because k=5 is a small event base and the ratio ranges from 1.5 to 6.2.The grader uses private and generated stress tests under a Python-fair time limit but is not the official contest grader.
  • G Content-controlled format test: open-ended GPQA (LLM-judge panel): The format test holds GPQA-Diamond content fixed, strips answer options, and asks the same 18 frontier models for free-response answers.This isolates answer format from the content confound between open-ended math/code and multiple-choice science.
  • G Content-controlled format test: open-ended GPQA (LLM-judge panel): 0.127 β emerged in open-ended GPQA, with 10 all-wrong cases and CP[0.062, 0.220], versus β ≈0 for identical multiple-choice questions.On the matched 11-model comparison, mean accuracy fell 0.66→0.51 and best-model accuracy fell 0.91→0.77; judge agreement was κ 0.73–0.92.
  • G Content-controlled format test: open-ended GPQA (LLM-judge panel): Five LLM judges use majority voting to assess answer equivalence, excluding each judge from grading its own model’s answer and breaking ties as incorrect.Inter-judge reliability was quantified rather than assumed.
Loading 2606.27288v1…