PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation

Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Caiming Xiong, Chien-Sheng Wu

arXiv:2604.16830v1cs.LGcs.AI

TL;DR

OPD improves capability but systematically produces overconfident models because teachers use privileged context unavailable at deployment. CaOPD estimates confidence from student rollouts, replaces teacher-conditioned confidence targets, and retains the distillation pipeline. Across scales, domains, and distribution shifts, it reports calibrated confidence without the capability tax associated with RL-based reward shaping.

  • Problem

    OPD’s capability gains expose a calibration gap because teacher-conditioned supervision uses privileged context unavailable to deployed models.

  • Method

    CaOPD estimates deployment-time success from student rollouts, replaces self-reported confidence with this empirical target, and distills the revised response.

  • Results

    CaOPD collapses the Overconfidence Gap across scales, domains, and distributional shifts without the capability tax typical of RL-based reward shaping.

  • Takeaways & Limitations

    Capability distillation does not imply calibrated confidence, so confidence must be treated as an essential post-training objective.

  • Takeaways & Limitations

    CaOPD requires sufficient base-model in-context learning capacity, and verbalized confidence can occasionally fail to follow the required format at test time.

Abstract

from arXiv · show

On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration: while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias. To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training. Code: https://github.com/SalesforceAIResearch/CaOPD

1 Introduction

OPD transfers capability effectively but creates systematic overconfidence because teacher supervision uses privileged context unavailable at deployment. CaOPD decouples capability imitation from confidence supervision to improve calibration without sacrificing capability.

  • Motivation: Modern LLMs exhibit severe overconfidence, and scaling capability does not resolve this calibration failure.This pattern spans proprietary and scaled open-weight models.
  • Problem: OPD transfers high-quality capability but destabilizes confidence by conditioning teachers on privileged feedback, demonstrations, or ground-truth solutions.The resulting trajectories are success-filtered and unavailable to the deployed student.
  • Problem: Privileged teacher context creates an information mismatch: the teacher operates open-book while the deployed student reports confidence closed-book.Distillation can therefore transfer capability without safely transferring confidence.
  • Method: CaOPD estimates confidence from student rollouts, replaces the implicit confidence target, and preserves the teacher trajectory for capability cloning.Under SDPO, existing rollouts provide the empirical target without extra rollout cost.
  • Contributions: CaOPD achieves Pareto-optimal calibration across OPD regimes and domains while generalizing under distribution shifts and continual learning.The framework is presented as avoiding the capability tax associated with RL-based calibration.

2 Problem Formulation and Preliminaries

The paper formulates OPD as self-distillation from a privileged teacher to a deployment-conditioned student, with confidence included in the generated trajectory. Its central blind spot is that reverse-KL training makes the student imitate teacher certainty at confidence positions.

  • 2.1 On-Policy Distillation and Self-Distillation: Each generation is partitioned into a reasoning trajectory a and a trailing confidence segment c.The confidence instruction asks the model to append an explicit confidence statement after reasoning.
  • 2.1 On-Policy Distillation and Self-Distillation: The student policy generates from input x alone, while the teacher policy additionally conditions on privileged context z.Privileged context may include demonstrations, environment feedback, or ground-truth solutions.
  • 2.2 The Ideal Calibration Objective: Deployment success probability µ(x) is defined by marginalizing the deployed model’s own generation distribution over reasoning trajectories.The verification function R(x, a) indicates whether a trajectory yields a correct answer.
  • 2.2 The Ideal Calibration Objective: A calibrated confidence segment should have an expected parsed value equal to the deployment success probability µ(x).Thus µ(x) is the ground-truth target for confidence alignment.
  • 2.3 The OPD Objective and Its Theoretical Blind Spot: On-policy self-distillation minimizes token-level reverse KL between student and privileged-teacher distributions along a student-generated trajectory.Both policies score the same trajectory under their respective conditioning information.
  • 2.3 The OPD Objective and Its Theoretical Blind Spot: Because the loss covers confidence tokens, OPD transfers reasoning capability while also forcing the student to imitate the teacher’s near-extreme certainty.This entangles capability transfer with confidence imitation.

3 Why Does Capability Distillation Exacerbate Overconfidence?

OPD-induced overconfidence follows from information asymmetry: privileged context sharpens teacher distributions and helpful context selects unusually successful trajectories. The resulting confidence target is unavailable and upward-biased for deployment.

  • Information Asymmetry: The student lacks privileged evidence z while minimizing reverse KL against the teacher conditioned on z, so it sharpens logits to mimic deterministic-like behavior.At deployment, this produces confidence without the evidence that supported the teacher’s certainty.
  • Theoretical Consequence: The three propositions establish that OPD pursues a target that is unidentifiable from deployment input, artificially sharpened, and systematically optimistic.The paper states that full proofs and extended discussions appear in Appendix A.
  • Non-Identifiability: If privileged context contains information about correctness beyond x, teacher-conditioned success is not recoverable as a deterministic function of x alone.The paper formalizes this as non-identifiability under positive conditional mutual information.
  • Entropy Collapse: Privileged context that provides actionable information about reasoning trajectories lowers expected teacher entropy relative to conditioning on accessible input alone.Reverse-KL distillation then penalizes the student for expressing natural uncertainty given x.
  • Optimism Bias: When training contexts are selected from successful or high-quality outcomes, the distilled confidence target becomes upward-biased relative to deployment success.The bias is strict where helpful context raises expected teacher correctness above the student’s marginal capability.

4 Calibration-Aware On-Policy Distillation (CaOPD)

CaOPD anchors confidence to an empirical success rate estimated from student rollouts, then replaces confidence in both the completion and teacher context before unchanged reverse-KL distillation. This preserves capability cloning while calibrating confidence.

  • 4.1 Constructing the Student-Grounded Target: CaOPD decouples confidence supervision from privileged teacher trajectories by targeting the student’s deployment-time success probability µ(x).The target must rely solely on the student’s accessible information state.
  • 4.1 Constructing the Student-Grounded Target: For each input, CaOPD samples K student trajectories, verifies their answers, and computes the empirical success rate ˆµ(x).This Monte Carlo estimate is intended to be an unbiased, execution-based confidence target.
  • 4.1 Constructing the Student-Grounded Target: Under SDPO, CaOPD reuses existing student rollouts and adds only lightweight verifier evaluation, while the separate distillation trajectory incurs no extra cost.The method therefore amortizes test-time confidence estimation within the training loop.
  • 4.2 Target Replacement and the Decoupled Objective: Target replacement overwrites the confidence segment c with ˆµ(x), producing a revised trajectory ˜y.The reasoning portion remains unchanged.
  • 4.2 Target Replacement and the Decoupled Objective: The privileged teacher context z is likewise revised by replacing its original confidence score with ˆµ(x), yielding ˜z.Both student and teacher then score the revised completion under their respective conditioning.
  • 4.2 Target Replacement and the Decoupled Objective: Capability cloning is preserved at reasoning positions, while confidence positions are trained against the student-grounded target rather than teacher certainty.The objective explicitly separates capability cloning from confidence calibration.
  • 4.3 Optimization: CaOPD keeps the standard distillation pipeline and reverse-KL machinery unchanged, avoiding reward modification and additional optimization stages.This contrasts with RL-based calibration methods that inject scoring-rule penalties into rewards.

5 Experiments

Experiments evaluate CaOPD across domains, models, calibration metrics, and post-training baselines. Results show that it improves calibration while preserving capability, and remains robust under distribution shifts, continual training, and model scaling.

  • Experiment Setup: Experiments cover Science Q&A and Tool Use using Qwen3-8B, Olmo-3-7B-Instruct, and Qwen3 models from 0.6B to 32B.Capability is measured by Accuracy; calibration uses ECE, Brier Score, and the Overconfidence Gap.
  • Overconfidence Under OPD: Standard OPD worsens overconfidence, driving mean confidence toward saturation and expanding the Overconfidence Gap across most settings.For Qwen3-8B on Tool Use, mean confidence reaches 0.996; base Qwen3-8B has a +58.7% Overconfidence Gap on Science Q&A.
  • Calibration and Discrimination: On Qwen3-8B Tool Use, CaOPD raises SPR from 0.085 under SDFT to 0.555 while reducing ECE and Brier Score and maintaining or improving accuracy.SPR measures whether confidence ranks correct answers above mistakes.
  • Capability Trade-offs: RL-based calibration reduces capability, whereas CaOPD improves calibration while increasing Tool Use accuracy from 66.2% under SDPO to 70.9%.RLCR and CAR reduce Science Q&A accuracy to 65.8% and 61.6%, compared with 74.5% for GRPO and 80.6% for SDPO.
  • Robustness and Scaling: CaOPD preserves accuracy while calibrating confidence across optimization, out-of-distribution transfer, continual training, and model scales.It matches SDPO’s accuracy trajectory, reduces OOD ECE to 0.358, prevents calibration forgetting, and pushes the Reliability and SPR Pareto frontier upward as Qwen3 scales.

6 Related Work

Related work centers on capability-focused on-policy and self-distillation methods, calibration through reward shaping, and test-time sampling for hallucination detection. These strands motivate separating capability transfer from reliable uncertainty signaling.

  • On-Policy Distillation for Post-Training: OPD and self-distillation bootstrap reasoning capabilities using privileged context to generate high-quality teacher trajectories.Examples include verifier feedback, expert demonstrations, ground-truth solutions, and user interactions.
  • On-Policy Distillation for Post-Training: Existing OPD frameworks are overwhelmingly capability-centric and mathematically ignore the information asymmetry between teacher supervision and deployment.
  • Confidence Calibration and Reward Shaping: Calibration research commonly uses reinforcement learning with explicit reward shaping to address overconfidence.RLCR and related methods exemplify this approach.
  • Test-Time Amortization and Hallucination: Test-time multi-sampling is widely used to detect and mitigate hallucinations caused by models behaving like aggressive test-takers.

7 Conclusion

The paper identifies a Scaling Law of Miscalibration in which standard OPD traps models in an illusion of certainty. CaOPD decouples capability distillation from confidence supervision and targets calibrated confidence without the capability tax associated with RL-based shaping.

  • Standard OPD traps models in an illusion of certainty across model scales, domains, and distributional shifts.
  • CaOPD decouples capability distillation from confidence supervision through a simple target replacement mechanism.
  • CaOPD collapses the Overconfidence Gap while retaining capability and avoiding the capability tax typical of RL-based reward shaping.
  • CaOPD enables calibrated confidence in a single forward pass, supporting deployment of uncertainty-aware agents.

8 Limitations and Future Work

CaOPD’s limitations include dependence on base-model in-context learning, parsable confidence formatting, and additional training rollouts. Future work targets finer-grained step-level calibration, adaptive rollout budgets, preference optimization, and cascade routing.

  • Limitations: CaOPD requires sufficient base-model in-context learning capacity, and capability gains remain bounded by the base model’s reasoning capacity.Meaningful calibration improvements appear at 0.6B parameters, but smaller models retain a capability ceiling.
  • Limitations: Verbalized confidence requires a parsable confidence segment, and occasional test-time formatting failures can occur.The limitation is shared with verbalized uncertainty methods.
  • Limitations: Training confidence targets requires K rollouts per prompt, increasing training compute, although deployment remains a single O(1) forward pass.The authors find K = 8 sufficient in practice.
  • Future Work: Future work proposes extending CaOPD from utterance-level confidence to dense step-level uncertainty for long-horizon reasoning.The goal is finer-grained error detection and earlier termination of flawed reasoning chains.
  • Future Work: Adaptive rollout allocation could reduce training cost, while extending CaOPD to DPO and RLHF remains open.One proposed strategy stops early when initial rollouts have unanimous verifier scores.
  • Future Work: CaOPD-calibrated small models could act as lightweight cascade routers that defer uncertain queries to larger models.The cost-accuracy tradeoff of this architecture still requires empirical validation.
  • Theoretical Boundary: If privileged context contains information about success beyond the prompt, the teacher-conditioned confidence cannot be recovered exactly from deployment-time inputs.The formal result identifies the student’s best observable target as its marginal success rate and leaves strictly positive projection error.
  • Theoretical Boundary: Standard OPD therefore makes the student chase an inaccessible confidence target instead of its deployment-time marginal success rate.The mismatch can force memorization of inaccessible context cues and prevent calibrated predictions.

A.2 Proof of Proposition 2

Proposition 2 formalizes entropy collapse: informative privileged context lowers the teacher’s trajectory entropy, and reverse-KL distillation drives the student toward that unjustified certainty. CaOPD decouples capability imitation from confidence supervision by replacing verbalized confidence with student-grounded empirical success.

  • Proof of Proposition 2: If I(A; Z | X) > 0, privileged conditioning yields lower expected trajectory entropy than conditioning on X alone.The statement gives EX,Z[H(πθ(A | X, Z))] < H(A | X).
  • Proof of Proposition 2: Reverse KL is mode-seeking, so matching the concentrated privileged distribution collapses student probability mass onto narrow high-probability token modes.This artificially sharpens student logits despite the absence of privileged evidence at deployment.
  • Proof of Proposition 3: Success-conditioned supervision trains the student in a declarative, confident style that can produce optimism bias when the student later generates incorrect answers.The paper connects this inherited style to overconfidence gaps, including under distribution shift.
  • CaOPD: CaOPD replaces original confidence tokens with the empirical success rate while preserving the student trajectory for capability distillation.The revised completion uses a student-grounded target rather than the teacher’s privileged certainty.
  • CaOPD: Online rollouts keep the empirical confidence target aligned with the student as its capabilities change, avoiding stale-target instability.Fresh rollouts yield higher success rates as the model improves.
  • CaOPD: Sampling-based behavioral consistency is used as a statistically unbiased confidence target and amortized during training into single-pass inference.CaOPD moves the O(K) test-time search cost into training while retaining an O(1) forward pass at deployment.

B.6 Extending CaOPD Without Objective Verifiers: Teacher-Anchored Self-Consistency

Teacher-Anchored Self-Consistency extends CaOPD to settings without objective verifiers by comparing student rollouts with a privileged reference trajectory. It preserves substantial calibration gains, although open-ended Tool Use introduces spurious disagreements and a modest capability reduction.

  • Teacher-Anchored Self-Consistency: When objective verification is unavailable, TA-SC replaces task rewards with agreement between student rollouts and a privileged teacher-conditioned reference.The reference is generated under privileged conditioning, and semantic equivalence can be judged by an LLM.
  • Teacher-Anchored Self-Consistency: TA-SC prevents consistently wrong student rollouts from defining confidence by anchoring agreement to the privileged reference.Eight identical wrong answers can therefore yield µ̂SC(x) = 0/8 and a confidence target of 0.0.
  • Empirical Validation: TA-SC slightly degrades calibration relative to an objective verifier but still substantially outperforms standard SDFT.On Science Q&A, ECE increases from 0.266 to 0.271 with TA-SC.
  • Empirical Validation: In Tool Use, Accuracy decreases from 70.6% to 68.3% and SPR from 0.555 to 0.547, while ECE falls from 0.321 to 0.240.The passage attributes the capability drop to false-negative equivalence checks in open-ended JSON generations.
  • Why Verbalized Confidence?: Verbalized confidence is targeted because it supports black-box, single-pass deployment, despite severe overconfidence after standard post-training.CaOPD addresses this deployment-friendly but miscalibrated uncertainty paradigm.
  • Internal Logit Entropy: CaOPD’s confidence replacement does not degrade reasoning accuracy because autoregressive prefixes exclude the replacement at reasoning positions.Figure 2 reportedly shows CaOPD’s accuracy trajectory shadowing SDPO throughout training.

C.1 Datasets and Models

The evaluation spans Science Q&A and Tool Use across representative open-weight models and the broader Qwen3 scale range. It measures capability, calibration, confidence discrimination, and continual-learning retention while analyzing rollout size and training dynamics.

  • Datasets: Experiments cover Chemistry Science Q&A and ToolAlpaca Tool Use, testing calibration in knowledge-intensive and long-horizon sequential reasoning.Chemistry uses approximately 75% training, 5% validation, and 20% test splits; ToolAlpaca follows its original train-test split.
  • Models: The primary testbeds are Qwen3-8B and Olmo-3-7B-Instruct, supplemented by Qwen3 models from 0.6B to 32B parameters.The scaling analysis includes 0.6B, 1.7B, 4B, 8B, 14B, and 32B models.
  • Metrics: OCG measures the difference between expected verbalized confidence and empirical accuracy, with positive values indicating overconfidence and near-zero values indicating calibration.The metric is intended to expose the direction of conditional miscalibration that ECE and Brier Score can obscure.
  • Metrics: SPR measures whether confidence is strictly higher for correct than incorrect responses, assigning zero points to tied predictions.This makes uniform maximum-confidence outputs receive SPR = 0 rather than credit for ties.
  • Continual learning: Under continual learning, SDFT can recover Tool Use accuracy while failing to preserve calibration mapping.The reported CT accuracy recovers to 0.647, matching the ID baseline, while SPR collapses.
  • CaOPD evaluation: CaOPD samples student rollouts to estimate empirical confidence, and rollout size K = 8 is identified as a compute-efficient calibration sweet spot without changing task accuracy.Across K ∈ {1, 2, 4, 8, 16, 32}, accuracy remains flat while small K values produce quantized overconfident targets.

D.5 Full API Evaluation Results and The Scaling Law of Miscalibration

The API evaluation finds widespread confidence saturation: many models report near-certain confidence despite substantially lower accuracy. CaOPD provides competitive calibration for a fine-tuned 8B model, though its accuracy comparison with zero-shot APIs is not strictly equivalent.

  • Evaluation: 12 leading proprietary and open-weight LLMs are evaluated on Chemistry Science Q&A to assess the Scaling Law of Miscalibration.The evaluation spans GPT-5.x, Claude 4.x, Gemini 3.1, DeepSeek-V3.1, and Qwen3.5-397B families.
  • Scaling Law of Miscalibration: 56%–72% accuracy coexists with 0.99–1.00 mean confidence in many models, demonstrating severe confidence saturation.The passage identifies particularly strong saturation in Gemini and DeepSeek models.
  • API comparison: Claude-Opus-4.6 achieves Accuracy: 83.8%, Mean Confidence: 0.838, and ECE: 0.027, while standard-tier models do not replicate this calibration.The passage presents this result as evidence that reliable uncertainty signaling may require substantial capability overhead or targeted calibration.
  • CaOPD result: CaOPD-trained Qwen3-8B achieves 80.6% accuracy, with ECE reduced from 0.157 to 0.141 and SPR increased from 0.571 to 0.669 without accuracy degradation.The model is fine-tuned on Chemistry, whereas the API models are evaluated zero-shot, so the accuracy comparison is not strictly apples-to-apples.

E.1 Case Study 1: Restoring Capability and Calibrated Confidence

Tool-use case studies show CaOPD preserving high confidence on a correct action while expressing low confidence on an incorrect action. The examples contrast calibrated uncertainty with standard SDFT’s overconfident failures.

  • Correct tool selection: In the holiday-information task, CaOPD selects getHolidayDetails correctly with confidence 0.90, while standard SDFT selects searchHolidays incorrectly with confidence 0.95.CaOPD follows the direct API path; SDFT hallucinates an unnecessary search step.
  • Correct tool selection: The holiday example shows that CaOPD retains high confidence when its reasoning trajectory and tool selection are correct.The reported confidence is high rather than globally suppressed.
  • Hard negative: In the Bible-verse task, both models hallucinate optional API parameters, but CaOPD reports confidence 0.10 compared with SDFT’s 1.00.Both predictions are incorrect, while CaOPD’s lower score signals uncertainty about the optional parameters.
  • Deployment implication: The case study frames low confidence on incorrect API calls as a signal that can support human fallback or API-schema reevaluation in production.The paper contrasts this with an absolute confidence of 1.00, which would allow a flawed action to execute autonomously.

F Broader Impact: Calibrated Small Models as Cascade Routers

CaOPD-trained small models can express confidence that reflects their limited capability, enabling a proposed cascade-routing pattern. The routing cost-accuracy trade-offs remain empirically unvalidated.

  • Small-model calibration: A CaOPD-trained 0.6B model achieves approximately 58% accuracy with mean confidence roughly 0.3, whereas SDFT reports confidence 1.0 despite failing nearly half the time.The lower CaOPD confidence provides a downstream decision signal that the saturated SDFT confidence does not.
  • Cascade routing: A calibrated small model can answer high-confidence queries cheaply and defer low-confidence queries to a larger, more expensive model.This cascade requires calibrated confidence because otherwise the routing threshold is meaningless.
  • Limitation: Empirical cascade-routing performance, including cost-accuracy trade-offs under different thresholds, is left for future work.The paper identifies cascade routing as a promising application direction rather than a validated deployment result.
Loading 2604.16830v1…