PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictable performance is suboptimal, and what it means for model routing

Guanhua Zhang, Moritz Hardt

arXiv:2610.31428v1cs.LGcs.AI

TL;DR

Performance prediction can guide model routing, but its value relative to a model’s self-confidence is the central question. The paper analyzes that comparison through recalibration and bounds, then finds larger prediction and routing advantages for more suboptimal models, with advantages shrinking after finetuning.

  • Problem

    Performance prediction can help routing decide when to trust an answer or call a stronger model, raising the question of what makes errors predictable relative to self-confidence.

    Source: §1 · p. 1

  • Method

    The paper compares predictors with self-confidence, uses recalibration to transfer correctness-log-loss gains to answer loss, and bounds predictor advantage by suboptimality plus added capacity.

    Source: §1 · pp. 1–2, §5 · p. 12

  • Results

    More suboptimal models have larger prediction and routing advantages over confidence, and finetuning reduces both advantages.

    Source: §5 · p. 12

  • Takeaways & Limitations

    When self-confidence is unavailable, predictive value can come from recovering it, exploiting suboptimality, or adding capacity; improving the model can reduce the additional value of learned prediction and routing.

    Source: §5 · p. 12

    The cascade bound controls additional weak-model errors captured; bounding full routing advantage also requires a condition on strong-model errors.

    Source: §5 · p. 13

Abstract

from arXiv · show

Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.

1 Introduction

The paper studies performance prediction against a model’s self-confidence and connects a predictor’s advantage to model improvement. It develops bounds for prediction and cascading, then tests how these advantages vary with model suboptimality and finetuning.

1 Introduction

  • The paper frames performance prediction as estimating correctness to help routing systems decide when to trust an answer or call a stronger model.The central question is what makes a model’s errors predictable and what this reveals about the model.Source: §1 · p. 1

1.1 Our Contributions

  • A predictor’s reduction in correctness log loss over self-confidence transfers exactly to an equal reduction in the model’s answer log loss through recalibration.The analysis assumes a scaling floor at each parameter budget; excess predictability is bounded by model suboptimality plus the predictor’s capacity gain.Source: §1.1 · p. 2
  • Predictors have larger advantages over self-confidence on more suboptimal models, and finetuning reduces excess predictability in 55 of 59 retained model-task pairs.The 55-of-59 result holds despite training a new predictor for each finetuned model.Source: §1.1 · pp. 2–3, Figure 1 · p. 2
  • For two-model cascades, additional weak-model errors captured are bounded by weak-model suboptimality and the capacity gain from the router’s added parameters.The cascade accuracy gain subtracts additional strong-model errors encountered; the same bound applies to that gain under the stated condition on those errors.Source: §1.1 · p. 3
  • Models with greater empirical suboptimality tend to have larger routing advantages on all four tasks, and this advantage shrinks after finetuning in 53 of 59 retained pairs.Routing is compared with confidence routing at equal escalation rates, with advantage averaged over escalation rates.Source: §1.1 · p. 3

2 Related Work

Related work studies performance prediction, routing and cascading, confidence-based decisions, and calibration. This paper positions its contribution as explaining predictor gains over self-confidence and bounding those gains under stated assumptions.

Performance prediction

  • Performance predictors estimate system success from model outputs, references, follow-up queries, activations, or benchmark-level information; this paper studies individual-input predictability against self-confidence.Related work also includes predictors trained to assess correctness or continuous quality.Source: §2 · p. 3

Model routing and cascading

  • Routing balances cost and performance by choosing models per query, while cascades use a cheaper model first and escalate based on its output.Prior approaches use preference data, benchmark results, reward scores, query difficulty, confidence, or answer verification.Source: §2 · pp. 3–4

Confidence and selective prediction; Calibration and loss minimization

  • Confidence supports selective prediction and deferral, while calibration and loss-minimization work studies how correcting predictable probability errors can reduce loss.This paper connects those ideas by showing an equal reduction in answer log loss for any correctness-log-loss reduction over self-confidence, bounded by suboptimality and added-parameter gains.Source: §2 · p. 4

3 Predictable Performance is Suboptimal

The section formalizes performance prediction against a model’s self-confidence and shows how prediction gains relate to model improvement. It then describes experiments testing these relationships across models, tasks, and finetuning stages.

3.1 Problem Definition

  • A performance predictor estimates whether the model’s chosen answer is correct, with the model’s own probability for that answer serving as the self-confidence baseline.The predictor is evaluated by correctness log loss, and its improvement over self-confidence is called excess predictability.Source: §3.1 · p. 5

3.2 Any Effective Performance Predictor Improves the Model

  • Recalibrating a model with an effective performance predictor lowers answer loss by exactly the predictor’s excess predictability over self-confidence.The recalibration changes the probability assigned to the chosen answer while preserving the relative probabilities of alternatives.Source: §3.2 · p. 5

3.3 Predictable Performance is Suboptimal

  • Excess predictability is bounded by model suboptimality plus the loss-floor gain from the predictor’s added parameter capacity.With no added parameters, positive excess predictability implies that the model is suboptimal.Source: §3.3 · p. 6
  • Models with the same maximum excess predictability can differ in suboptimality because they assign different probabilities to unchosen answers.The section also notes that, under diminishing returns, a predictor much smaller than the model has only a small capacity correction to the bound.Source: §3.3 · p. 6

3.4 Experiments

  • Experiments test whether more suboptimal models have greater excess predictability and whether task-specific finetuning reduces that advantage.The study evaluates 16 models across four benchmarks and uses the best observed answer loss among similar-size models as a proxy for the loss floor.Source: §3.4 · p. 6, §3.4.1 · p. 7

3.4.1 Setup

  • The setup uses five-shot evaluation and separately trained 4B predictors, with LoRA finetuning on gold answers or reasoning traces depending on the task.Models whose development accuracy did not improve after finetuning were excluded; retained pairs numbered 16, 16, 14, and 13 across the four tasks.Source: §3.4.1 · p. 7

Appendix B.4 gives the architecture and training details.

Across four tasks, more suboptimal models tend to have greater excess predictability, while finetuning generally reduces it. Predictor capacity and an individual finetuning trajectory provide additional context for these patterns.

3.4.2 Results

  • Excess predictability falls in 55 of 59 retained model-task pairs after finetuning, while answer loss falls in all 59.The task-specific excess-predictability counts are 14/16 on WinoGrande, 16/16 on CommonsenseQA, 12/14 on GSM8K, and 13/13 on MMLU-Pro.Source: Figure 2 · p. 8, §3.4.2 · p. 8
  • Suboptimality and excess predictability are positively correlated across tasks, with Pearson correlations of 0.98 on WinoGrande and CommonsenseQA, 0.93 on GSM8K, and 0.65 on MMLU-Pro.The association is weaker on MMLU-Pro, possibly because models can differ in answer loss without differing in excess predictability.Source: §3.4.2 · pp. 7–8
  • Excess predictability during Llama-3.1-8B finetuning on CommonsenseQA falls from 0.107 to 0.004 nats.Figure 3 shows answer and correctness losses during finetuning, with shading indicating excess predictability and predictor loss averaged over three seeds.Source: §3.4.2 · p. 8, Figure 3 · p. 8
  • Larger predictors generally capture more excess predictability, but mean excess predictability falls after finetuning at all three tested predictor sizes.The 0.6B predictor adds little beyond Platt-calibrated self-confidence, while 4B and 8B predictors generally improve on it.Source: §3.4.2 · p. 9

4 Suboptimality Bounds Cascading

The cascade analysis bounds how much a learned router can improve on confidence routing, relating that advantage to weak-model suboptimality and added predictor capacity. Across four tasks, more suboptimal models have larger routing advantages, while finetuning generally narrows them.

4.1 Cascading Setup

  • At a fixed escalation rate, cascade accuracy depends on additional weak-model errors captured minus additional strong-model errors encountered.Thus, capturing more weak-model errors does not necessarily improve cascade accuracy if the stronger model also errs on those queries.Source: §4 · p. 9, §4.1 · p. 9

4.2 From Performance Prediction to Cascade Bounds

  • A router that captures additional weak-model errors can be converted into a performance predictor with lower correctness log loss than self-confidence.This holds when the router's additional captured-error term is positive; recalibration shifts confidence according to the difference between routing decisions.Source: §4.2 · p. 10
  • The additional weak errors captured are bounded by weak-model suboptimality plus the capacity added by the routing score.The same bound applies to cascade accuracy gains when the additional strong-error term is nonnegative; it is nonnegative on average across escalation rates for most evaluated models.Source: §4.2 · p. 10

4.3.1 Setup

  • Experiments compare a 4B predictor with weak-model self-confidence for routing to Qwen3.5-122B-A10B at equal escalation rates.The 4B predictor supplies error scores, and routing advantage is the average cascade-accuracy gain across escalation rates.Source: §4.3.1 · p. 11

4.3.2 Results

  • Routing advantage correlates with empirical suboptimality across four tasks (Pearson r = 0.66–0.91), and finetuning reduces it in 53 of 59 model-task pairs.For Llama-3.1-8B on CommonsenseQA, finetuning also brings confidence routing closer to predictor routing.Source: Figure 4 · p. 10, Figure 5 · p. 11, §4.3.2 · p. 12
  • After finetuning, average routing advantage falls from 1.38 to 0.18 percentage points on WinoGrande and from 1.09 to 0.18 on CommonsenseQA.It also falls from 0.52 to 0.20 on GSM8K and from 0.94 to 0.07 on MMLU-Pro.Source: Figure 5 · p. 11, §4.3.2 · p. 12

5 Conclusion

  • The paper concludes that improving a model can reduce the need for a separate predictor, but the cascade bound controls captured weak-model errors rather than full routing advantage without a condition on strong-model errors.The full advantage also depends on errors made by the stronger model on escalated queries.Source: §5 · pp. 12–13

A.2 No Improvement Without a Better Component

  • Recalibration improves correctness probabilities while preserving the distribution over alternative answers; further improvement requires better correctness prediction, better alternative-answer prediction, or both.Under diminishing returns, a much smaller predictor's advantage is bounded essentially by the model's suboptimality when its added capacity correction is small.Source: §A.2 · p. 22, §A.3 · p. 23

A.4 Suboptimality Does Not Determine Excess Predictability

Lower suboptimality need not reduce excess predictability, while routing can improve prediction beyond confidence under conditions on weak- and strong-model errors. The cascade analysis bounds gains from capturing additional weak-model errors, but full routing advantage also depends on strong-model errors.

A.4 Suboptimality Does Not Determine Excess Predictability

  • Models can reduce suboptimality by lowering alternative-answer loss without changing their chosen answer or self-confidence, leaving maximum excess predictability unchanged.This separation can grow with the number of answer alternatives; with two answers, the source is absent.Source: §A.4 · p. 24

A.5 Routing Induces Performance Prediction

  • At a fixed escalation rate, capturing additional weak-model errors can yield a logit correction that improves the weak model’s correctness loss.If E[uϵ1(X)] > 0, some λ > 0 gives LZ(gλ, f1) < LZ(f1).Source: §A.5 · pp. 25–26

A.6 Proofs for Section 4

  • Theorem 2 bounds excess predictability by recalibrating the weak model with the routing score, escalation rate, and correction strength, then applying the loss-floor bound.The recalibrated model uses at most P1 + Q + 2 parameters.Source: §A.6 · p. 27

A.7 Cascade Advantage over Confidence Routing

  • When a learned router encounters at least as many strong-model errors as confidence routing, its cascade advantage is bounded by its additional weak-error capture.The cascade accuracy gain equals E[uϵ1(X)] − E[uϵ2(X)], so strong-model errors must also be accounted for.Source: §A.7 · p. 28

B.1 Models and Data Splits

The appendix describes the evaluated model releases, data splits, and answer-scoring and loss conventions. The splits distinguish predictor fitting and selection from testing, while the scoring procedures vary by task.

B.1 Models and Data Splits

  • The evaluation uses 16 releases from eight model families, with a release and its selected task-specific checkpoint forming a model pair.The task data include held-out development questions and test sets; MMLU-Pro is split into teacher-generation, development, and test partitions.Source: Table 1 · p. 29, §B.1 · p. 29
  • The 85/15 development partition for predictor fitting and selection is shared across model versions and predictor sizes.Demonstrations are excluded from the other splits, and MMLU-Pro metrics pool subjects.Source: §B.1 · p. 29

B.2 Answer Scoring and Losses

  • Option-scoring tasks use answer-option likelihoods or filled-context continuation scores, while generation tasks estimate answer probabilities from sampled outputs.Generation uses 128 samples per question; the most frequent successfully extracted answer supplies self-confidence, and unsampled gold answers receive the reserved unseen-answer probability.Source: §B.2 · p. 30
  • All losses are measured in nats, and empirical suboptimality uses reference answer losses computed over retained released and finetuned models.Binary cross-entropy clips probabilities to [10−6, 1 − 10−6].Source: §B.2 · p. 30, Table 3 · p. 30

B.3 Task-Specific Finetuning and Model Retention

Task-specific finetuning uses gold answers for option-scoring tasks and accepted teacher solutions for generation tasks, with checkpoints selected on development performance. Model pairs are retained based on development accuracy, and a trajectory experiment tracks predictor performance during training.

B.3 Task-Specific Finetuning and Model Retention

  • Finetuning uses LoRA on gold answers for option-scoring tasks and teacher-generated reasoning traces for generation tasks, selecting checkpoints using development performance.Training uses three seeds, and generation-task checkpoint selection uses greedy accuracy; reported results use 128-sample majority answers.Source: §B.3 · p. 31, Table 4 · p. 31
  • Pairs are retained only when finetuning increases development accuracy, leaving 16, 16, 14, and 13 pairs across the four tasks.The rule uses no test labels; Table 5 marks excluded pairs whose finetuned accuracy does not exceed the released model’s.Source: §B.3 · p. 31, Table 5 · p. 32
  • The trajectory experiment reruns Llama-3.1-8B base on CommonsenseQA and evaluates checkpoints every 100 steps through step 600.Each later checkpoint uses a new Platt map and the mean loss of three newly trained 4B predictors.Source: §B.3 · p. 31

B.4 Performance Predictor Training

The performance predictor combines question and answer information with the model’s self-confidence. It is initialized from Platt calibration, then trained and selected using held-out development data.

B.4 Performance Predictor Training

  • Predictor inputs include the question, final answer, answer distribution, and—on generation tasks—a sampled reasoning trace ending in the majority answer.Inputs exclude demonstrations, chat-template tokens, and gold answers.Source: §B.4 · p. 32
  • A Qwen3-Embedding-4B representation feeds a linear head that corrects the self-confidence logit.The predictor applies a sigmoid to the learned scalar terms and linear-head output.Source: §B.4 · p. 32
  • Platt calibration initializes the confidence-logit terms, after which the adapters, head, and scalars are trained and selected by correctness loss.Selection uses the predictor-selection split across ten seeds and all epochs, including epoch zero.Source: §B.4 · p. 32

B.5 Cascade Evaluation

Cascade accuracy is evaluated across escalation rates, and the additional analyses examine how predictor gains relate to suboptimality within released and finetuned model groups. The released-model associations persist within that group, while finetuned-model routing associations remain positive across all four tasks.

B.5 Cascade Evaluation

  • Cascade accuracy is measured by escalating the top-ranked questions at each rate, integrating accuracy over rates, and reporting mean routing advantage in percentage points.For WinoGrande and CommonsenseQA, the strong model is finetuned and selected by development answer loss; on GSM8K and MMLU-Pro, it generates 128 samples per question.Source: §B.5 · p. 33

C.1 Correlation Estimates and Uncertainty

  • Correlation estimates use Pearson’s r and Spearman’s rs with one-sided positive-association tests, assuming independent observations and no multiple-comparison adjustment.The appendix reports within-stage correlations for 16, 16, 14, and 13 models across the four tasks.Source: §C.1 · p. 34, Table 7 · p. 36

C.2 Released and Finetuned Models Separately

  • Among released models, excess-predictability correlations with suboptimality are 1.00, 0.98, 0.89, and 0.49, while routing-advantage correlations are 0.67, 0.43, 0.91, and 0.81 across the four tasks.Among finetuned models, routing-advantage correlations remain positive at 0.79, 0.81, 0.94, and 0.64; excess-predictability correlations are 0.14, 0.65, 0.99, and 0.32.Source: §C.2 · p. 35, Figure 7 · p. 35, Figure 8 · p. 36
  • Finetuned models on WinoGrande lie within 0.22 nats of the best observed loss, and 14 of 16 have slightly negative excess predictability.The weak finetuned WinoGrande correlation reflects this narrow suboptimality range; on MMLU-Pro, both groups show weaker associations.Source: §C.2 · p. 35, Figure 7 · p. 35, Figure 8 · p. 36

C.3 Predictor Capacity

The capacity analysis compares 0.6B, 4B, and 8B predictors against self-confidence and Platt calibration. Larger predictors generally gain more over self-confidence for released models, while gains shrink after finetuning and associations with suboptimality persist at 4B and 8B.

C.3 Predictor Capacity

  • After finetuning, mean gains over self-confidence decrease on every task at all three predictor sizes, while larger predictors generally achieve greater gains for released models.Increasing predictor size provides less benefit after finetuning.Source: §C.3 · p. 37, Figure 9 · p. 37
  • The 0.6B predictor adds little beyond Platt calibration, whereas 4B and 8B predictors generally achieve lower correctness log loss than Platt-calibrated self-confidence.For finetuned WinoGrande models, no predictor size improves on Platt calibration on average.Source: §C.3 · p. 37, Figure 9 · p. 37
  • At 8B, excess predictability and routing advantage remain positively correlated with suboptimality on every task, but 0.6B correlations are negative on MMLU-Pro for excess predictability and on WinoGrande and MMLU-Pro for routing advantage.On those tasks, the 0.6B predictor adds little beyond calibration.Source: §C.3 · p. 38, Figure 10 · p. 38, Figure 12 · p. 39
  • Finetuning reduces mean excess predictability on every task at all three predictor sizes, and larger suboptimality reductions accompany larger excess-predictability reductions.The latter correlations are at least 0.99, 0.98, 0.73, and 0.48 across the four tasks.Source: Figure 9 · p. 37, §C.3 · p. 38

C.4 Cascade Accuracy Across Tasks

The figure extends the CommonsenseQA cascade example to four tasks using Llama-3.1-8B base before and after finetuning. It compares predictor-based routing with self-confidence routing at matched escalation rates.

C.4 How Cascade Accuracy Varies Across Tasks

  • Across four tasks, the figure plots cascade accuracy for Llama-3.1-8B base paired with Qwen3.5-122B-A10B, before and after finetuning.The strong model is finetuned on WinoGrande and CommonsenseQA and released on the other two tasks.Source: Figure 14 · p. 41, §C.4 · p. 41
  • Solid curves show the selected 4B predictor, dashed curves show self-confidence routing, and dotted lines show random routing.Orange denotes the released weak model and blue its finetuned version.Source: Figure 14 · p. 41
  • The shaded gap compares predictor and self-confidence routing at the same escalation rate, showing their accuracy difference across escalation rates.The figure’s gap measures the benefit of choosing different questions at a matched escalation rate.Source: Figure 14 · p. 41, §C.4 · p. 41

C.5 How Strong-Model Errors Affect Routing Gains

The routing advantage separates additional weak-model errors captured from additional strong-model errors encountered. Strong-model errors often offset gains from capturing more weak errors, and the weak-error contribution tracks suboptimality more closely than total routing advantage.

C.5 How Strong-Model Errors Affect Routing Gains

  • Routing advantage is the weak-error term minus the strong-error term, up to rounding, with both terms measured as differences from self-confidence routing.The weak-error term counts the extra weak-model errors escalated by predictor routing; the strong-error term counts the extra strong-model errors encountered.Source: Table 8 · p. 42, §C.5 · p. 42
  • In 91 of 118 retained model versions, strong errors offset some gain from capturing additional weak errors; on finetuned MMLU-Pro, 0.94 points of weak-error gain minus 0.87 points of strong-error loss leaves 0.07 points.The strong model is 87.6% accurate on MMLU-Pro; on GSM8K it is 97.9% accurate, so strong errors have less effect.Source: §C.5 · p. 42, Table 8 · p. 42
  • Suboptimality correlates more strongly with the weak-error term than with routing advantage on all four tasks: 0.80, 0.72, 0.91, and 0.86 versus 0.79, 0.66, 0.91, and 0.82.Avoiding strong errors can also increase routing advantage, but the largest observed contribution is 0.12 percentage points.Source: §C.5 · p. 42

Related papers

Loading 2610.31428v1…