Disagreement-Guided Weak-to-Strong Routing

1 Introduction

• The paper asks how to route weak-model responses to stronger assistance without costly ground-truth correctness labels, especially in open-ended and agentic tasks.Correctness may depend on environment state, user preferences, downstream effects, or whole trajectories.Source: §1 · p. 2
• The routing value is the difference between the strong model’s correctness likelihood and the weak response’s correctness likelihood.This reframes routing as estimating whether stronger assistance is likely to improve the outcome, rather than only whether the weak response is wrong.Source: §1 · p. 2
• Corrected disagreement approximates that value using weak–strong disagreement minus strong–strong disagreement, both measured from model-generated responses.The self-disagreement term accounts for uncertainty within the strong model.Source: §1 · p. 2
• The approximation becomes more accurate as the strong model approaches the ground-truth distribution, and is exact when the distributions coincide.The disagreement signals can be learned from scalable model-generated data without ground-truth supervision.Source: §1 · p. 3
• Optimal routing jointly accounts for allocating weak-model inference, selecting among sampled responses, and deciding when to query a stronger model.The policy identifies population quantities governing when to continue locally and when to seek assistance.Source: §1 · p. 3
• Across agentic environments and single-turn reasoning tasks, learned disagreement signals outperform label-free baselines.The agentic evaluation covers multi-step environments where step-level correctness is difficult to define or supervise.Source: §1 · p. 3

Learning to defer and supervised routing

• Learning-to-defer and selective prediction study when a model should abstain or seek help; LLM cascades apply this question to escalation from weak to strong models.Many cascade methods train quality scorers, process verifiers, or token-level confidence predictors using ground-truth correctness labels [1] [15] [38].Source: §2 · p. 3

Label-free routing signals

• Label-free alternatives use internal uncertainty or hidden-state probes from one generation, or measure agreement across multiple generations [10] [15] [24], [46].The paper positions its single-pass, disagreement-trained probe against these baselines.Source: §2 · p. 3

Step-level routing in agentic tasks

• Agentic routing has used process-level verifiers, trajectory rewards, or cost-aware decision policies, but these can require step-level labels or episode-success evaluations [17] [20] [41], [21].The paper contrasts these approaches with predicting offline action disagreement to route actions at each step.Source: §2 · p. 4

3 Fundamentals of Weak-to-Strong Routing

• The framework jointly optimizes weak-response sampling, selection, and strong-model routing to maximize final-response accuracy under separate inference budgets.The sampling policy represents repeated weak-model responses, such as best-of-n; the analysis treats responses as discrete up to semantic equivalence.Source: §3 · p. 4
• The population quantities q_p(y) and t_p denote the correctness probabilities of a candidate response and a strong-model response, respectively.The theorem characterizes the optimal policy for a fixed weak-response sampling policy.Source: §3 · p. 5
• The optimal selection rule returns the sampled weak response with the highest correctness probability, while routing occurs when the strong model’s expected gain exceeds a global budget threshold.The gain compares the strong model’s correctness probability with that of the best sampled weak response; the threshold controls the routing budget.Source: §3 · p. 5
• With one weak response, selection is trivial and the routing gain reduces to the strong response’s correctness probability minus the weak response’s correctness probability.This is the one-shot characterization associated with Jitkrittum et al. [19] and Mozannar & Sontag [33].Source: §3 · p. 5

4.1 Deriving Corrected Disagreement

• Corrected disagreement approximates the optimal routing gain by replacing the unknown ground-truth distribution with the strong model’s predictive distribution.Its score combines disagreement between the strong model and sampled weak responses with a correction for the strong model’s self-disagreement.Source: §4 · p. 6, §4.1 · p. 6
• The approximation gap shrinks as the strong model’s predictive distribution approaches the ground-truth distribution, and the score equals the optimal routing gain when they coincide.The population comparison uses the one-shot setting and estimates score probabilities with Monte Carlo sampling.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Population Comparison of Routing Scores

• Corrected disagreement consistently outperforms raw weak–strong disagreement across budgets; when escalation budgets are B ≤ 0.4, both disagreement signals outperform the weak-model correctness baseline.The population comparison reports routing accuracy and its gap from optimal routing across escalation budgets.Source: Figure 1 · p. 7, §4.2 · pp. 7–8

5 Finite-Sample Routing Algorithms

• Given fixed sampling and selection policies, the router queries the strong model when its score exceeds a threshold; the threshold can be calibrated to a desired strong-model budget.The history contains the prompt and sampled weak responses, and the selection policy chooses one weak response to return if the router does not escalate.Source: §5 · p. 8
• The routing score approximates the optimal accuracy gain with corrected disagreement, which uses model disagreement instead of ground-truth supervision.The score balances weak–strong disagreement against strong-model self-disagreement.Source: §5 · p. 8
• Two predictors learn the disagreement terms from model-generated responses, and their outputs are combined into the corrected-disagreement score.At test time, the score requires only the prompt and sampled weak responses, not the strong model or ground-truth labels.Source: §5 · pp. 8–9
• Weak–strong disagreement estimates how often strong responses differ from a weak response, while strong-model self-disagreement estimates disagreement between strong responses.The response-equivalence indicator is used offline to construct training targets and is not required at test time.Source: §5 · p. 9

6.1 Agentic Environments: Routing Without Ground Truth

• Agentic routing is performed per action because step-level correctness is difficult to define; observable transition equivalence supplies an objective disagreement signal for offline data collection.The experiments use BFCL Multi-Turn and WebShop, with K = 10 samples per step for training targets and multi-sample baselines.Source: §6.1 · p. 10
• Both disagreement signals outperformed the best label-free baseline in agentic environments; corrected disagreement significantly improved on raw disagreement in WebShop, but not significantly in BFCL.The figure reports results averaged over m=5 runs.Source: §6.1 · p. 10, Figure 2 · p. 11

6.2 Controlled Ablations on Single-Turn Benchmarks

• Single-turn benchmarks provide gold answers for comparison with a supervised proxy reward, while evaluation fixes sampling and selection across baselines and measures task success against routing budget.The benchmarks span Math-500, Spider, and AceCode, and the plotted comparison includes one-shot (N = 1) and uniform multi-sample (N = 8) regimes.Source: §6 · p. 10, §6.2 · p. 11
• Corrected disagreement beat the supervised proxy reward and raw disagreement across all three single-turn benchmarks; multi-sample consistency narrowed the gap only on MATH and broke down on Spider and AceCode.The figure compares one-shot (N = 1) and multi-sample (N = 8) routing.Source: §6.2 · p. 12, Figure 3 · p. 12

7 Conclusion

• Corrected disagreement balances cross-model disagreement against strong-model uncertainty and consistently outperforms label-free baselines across agentic environments and reasoning benchmarks.The paper presents it as a scalable learning signal collected from model-generated responses without ground-truth labels.Source: §7 · p. 12

A.1 Extended Related Works

• The related-work framing distinguishes response-conditioned cascades from prompt-based routers, which decide before observing a weak response, and correctness- or reward-supervised approaches.The paper studies a two-model cascade in which the weak model generates candidate responses before the system decides whether to invoke the stronger model.Source: §A.1 · pp. 17–18
• Consistency-based methods often require multiple generations and semantic clustering, while ensemble routers may assume multiple candidate models; this paper uses a single weak–strong cascade and can leave the strong model black-box.The related-work discussion also distinguishes the approach from predictors of uncertainty or truthfulness because it targets optimal cascade routing gain.Source: §A.1 · p. 18

A.1.1 Population estimates of routing signals across varying weak-strong model pairs

• Corrected disagreement generally improves on raw disagreement and closely approximates optimal routing when the strong model is accurate, with its advantage over weak-model correctness most apparent at limited routing budgets.At very high escalation budgets, weak-model correctness can slightly surpass corrected disagreement because it avoids routing rare cases where the weak model is correct but the strong model fails.Source: §A.1.1 · p. 19

• Supplementary tables report routing results for Math-500, Spider, and AceCode, with additional baselines, budgets, and N ∈ {1, 4, 8, 16} weak-model samples per prompt.The Math-500 table reports accuracy at different routing budgets for Phi-3.5-mini-instruct → GPT-4o-mini.Source: §A.2 · p. 19, Table 1 · p. 20
• On Math-500, self-consistency and semantic entropy approach corrected disagreement as budgets and sample counts rise, slightly surpassing it at 50% budget for N > 1.These consistency-based scores are less effective on Spider and AceCode, where grouping equivalent responses is more difficult.Source: §A.2 · p. 19
• Corrected disagreement generally matches or improves on raw weak–strong disagreement across all three datasets.Source: §A.2 · p. 20

A.3 Ablations on Selection Policy

• The main experiments hold the selection policy constant across routing methods to isolate routing-score effects; this ablation instead varies the weak-response selection policy.Source: §A.3 · p. 20
• For AceCode with N > 1, judge-based clusters do not improve kept-program accuracy, so gains come from improved routing signals rather than a better plurality answer.Source: §A.3 · p. 22
• The optimal selection rule chooses the sampled weak response most likely to be correct, but learning that probability directly requires correctness labels.Source: §A.3 · p. 23
• Compared selection rules include majority vote, lowest weak-model token entropy, lowest predicted weak–strong disagreement, and a supervised correctness probe.The correctness probe is trained with ground-truth correctness labels.Source: §A.3 · p. 23
• On Math-500 with N = 8, majority vote achieves the highest accuracy at every reported budget across all four routing scores.Disagreement-based selection performs similarly to the supervised correctness probe, and both improve on random selection and token entropy.Source: §A.3 · p. 23
• Figures 8 and 9 extend selection-policy comparisons to Spider and AceCode, varying routing budget and weak-response sample count while fixing corrected disagreement as the routing score.The experiments assess additional candidates where identifying equivalent responses is more difficult.Source: §A.3 · p. 23

A.4 Additional Results for the Agentic Experiments

• Table 4 adds the token-entropy baseline omitted from the main text and reports mean task score alongside full-trajectory success.Source: §A.4 · p. 23

A.5 Probe Representations and Training

• The weak model remains frozen while probes learn empirical targets from offline generations using weak-model hidden states.The self-disagreement probe uses the final prompt-token state; the weak–strong probe uses mean response-token states, with trajectory history and proposed action serving as prompt and response in agentic settings.Source: §A.5 · pp. 23–24
• Table 4 compares realized routing budget, end-to-end success rate, and average task score across label-free baselines on 200 WebShop evaluation tasks.Source: Table 4 · p. 24
• Binary cross-entropy trains the disagreement probes on empirical soft targets, while an optional proxy-reward probe uses correctness targets when ground-truth answers are available.At inference, routing scores require only weak-model representations; the strong model is queried only when the router escalates.Source: §A.5 · p. 24

A.6 On-Policy Data Collection for Agentic Probes

• On-policy data collection addresses distribution mismatch because strong-model actions can steer agents into states absent from purely weak-model trajectories.Across iterative phases, the routed policy visits states, labels them, and trains later probes on the aggregated data.Source: §A.6 · p. 24
• Each agentic step samples K = 10 actions from each model, records weak-model representations, and computes empirical disagreement targets using task-specific transition agreement.The weak candidate-selection policy is fixed to execute the first sampled action.Source: §A.6 · p. 25
• After the first phase, fixed probes map corrected disagreement to a routing probability, from which collection samples whether to execute the strong action or first weak sample.The corrected disagreement score lies in [−1, 1], and the routing probability lies in [0, 1].Source: §A.6 · p. 25

B Proof of Theorem 3.1

• With prices for weak samples and strong-model queries, choose each prompt’s weak-sample count to maximize expected value net of sampling cost.Suitable prices recover the constrained optimum, with boundary randomization when necessary.Source: B Proof of Theorem 3.1 · pp. 26, 28, 29
• For finite K, the resulting sampling, selection, and routing policies jointly maximize accuracy under the stated weak-sampling and strong-query budgets.The theorem allows boundary randomization when needed.Source: B Proof of Theorem 3.1 · pp. 27, 29
• For fixed sampling, select the weak response with the highest correctness probability and route histories whose strong-model improvement exceeds a threshold.Allocate the strong-model budget to histories with the largest improvements; randomize at the threshold if needed.Source: B Proof of Theorem 3.1 · pp. 27–28
• Expected gains from additional weak samples are nonincreasing, so sampling can stop once the marginal gain falls below its price.This follows because the current best value cannot decrease while each fresh weak response has the same distribution.Source: B Proof of Theorem 3.1 · p. 29