Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

• RLVR rewards correctness without specifying how confident a model should be for each answer, and models trained this way are systematically overconfident.Source: §1 · p. 1
• Verbalized confidence reports add variance, collapse to a few values shared by correct and incorrect answers, and make calibration non-differentiable.The calibration loss must therefore work through a scalar reward rather than a direct gradient.Source: §1 · pp. 1–2, Figure 1 · p. 2
• CREDO reads confidence as the relative probability of reserved tokens and trains it by differentiable regression, while policy gradients continue to optimize correctness.Source: §1 · p. 2
• CREDO weights answer learning by the gap between confidence and outcome, improving accuracy in both mathematical and code reasoning.Confident mistakes and unexpected successes are more informative than rollouts where confidence and outcome agree.Source: §1 · p. 2
• Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, with gains extending to abstention and selective prediction.Source: §1 · p. 2

2.1 SETUP: RLVR AND THE CALIBRATION OBJECTIVE

• RLVR rewards verified correctness, while calibration requires confidence reports to match the model’s actual accuracy; correctness-only rewards do not target that relationship.The calibration condition is that answers reported at confidence v are correct at rate v.Source: §2.1 · p. 3

2.2 THE VERBALIZED CHANNEL

• A confidence channel pairs an emission map with a training path, and the verbalized channel samples its report rather than computing it deterministically.This channel choice links sampling to the scalar-reward route for calibration.Source: §2.2 · p. 3
• Verbalized methods sample a numeral from the rollout prefix and parse it as confidence, so calibration can enter RLVR only through a scalar reward.RLCR and DCPO vary their reward designs, but share this emission and training path.Source: §2.2 · p. 3

2.3 COSTS OF THE VERBALIZED CHANNEL

• Single-digit effective levels in GRPO, RLCR, and DCPO reflect confidence reports clustering so correct and incorrect answers share values; verbalized reports also incur a variance tax or determinize.Under squared loss, the tax equals the report variance; under strictly convex loss, Jensen’s inequality gives a positive excess loss.Source: Figure 2 · p. 4, §2.3 · p. 4
• A deterministic readout avoids sampling variance, and a differentiable report at one position provides an exact pathwise calibration gradient; a verbalized numeral does not.The conditional mean of a verbalized report removes variance but cannot be optimized through the sampled-value reward.Source: §2.3 · p. 4

2.4 DESIGN REQUIREMENTS

• A suitable channel must compute confidence deterministically, avoid vocabulary-grid ties, and let calibration differentiate through a report computed during the rollout forward pass.The paper identifies a reserved-token pair’s relative probability as a simple functional meeting these requirements.Source: §2.4 · p. 5, Figure 3 · p. 5

3.1 ROLLOUT FORMAT AND THE READOUT

• CREDO reads confidence from the relative probability of two reserved tokens, producing a continuous readout from the output distribution during rollout generation.The readout is trained to represent confidence by the calibration loss.Source: §3.1 · p. 5, §3 · p. 5

3.2 THE CALIBRATION LOSS

• The calibration regression uses a within-group hybrid target, whose Bayes-optimal readout shrinks conditional correctness toward prompt-level accuracy.Decreasing γ reduces target variance but increases shrinkage bias relative to conditional correctness; prompt-level mean consistency does not in general imply calibration.Source: §3.2 · pp. 5–6

3.3 POLICY LOSSES AND CREDIT ASSIGNMENT

• Policy-gradient credit is assigned separately to answer and confidence-analysis segments, while the readout value is trained by the exact calibration-loss derivative.Each segment uses its own standardized reward advantage; the policy gradient and calibration loss share parameters but no credit.Source: §3.3 · p. 6

3.4 DISCREPANCY WEIGHTING: THE READOUT AS A TRAINING SIGNAL

• Discrepancy weighting gives answer learning more weight on rollouts whose outcome differs more from the model’s confidence.The discrepancy is centered within each group, and the weights have group mean one before clipping, redistributing learning within the group.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation covers mathematical and code reasoning with accuracy and three calibration measures, comparing Qwen3-8B baselines under a shared training protocol.Mathematics uses DeepScaleR training and seven evaluation sets; code uses DeepCoder and four sets, with ECE, AUROC, and Brier reported.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 ACCURACY AND CALIBRATION IN AND OUT OF DOMAIN

• CREDO leads all four reported metrics in both in-domain domains, and its calibration advantage persists and remains largest after domain shift.Accuracy is comparable across methods out of domain; the RLCR code-ECE match in domain does not survive the shift.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 ABLATION AND ATTRIBUTION

• Replacing verbalized confidence with digit expectations improves baseline ranking metrics, but does not close CREDO’s calibration gap; neither calibration-loss nor reasoning ablations preserve the full method’s performance.Removing calibration training sharply worsens ECE and Brier in both domains and lowers mathematical accuracy; removing confidence reasoning or its policy-gradient credit lowers accuracy in both domains.Source: §4.2 · p. 7, §4.3 · p. 8, Table 2 · p. 8
• Moderate discrepancy weighting beats κ=0 on accuracy, positive tested α values restore calibration, and γ=0.5 gives the best accuracy and Brier in both domains.At γ=1, code ECE is lower; the additive reward shape is best or tied in mathematics and competitive in code.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 DOWNSTREAM BENEFITS OF CALIBRATION

• At evaluated coverage budgets, CREDO answers most accurately and has the lowest aggregate error in both domains, with only the two tightest code budgets statistically tied with DCPO.Verbalized reports yield only a dozen or two operating points, while the readout provides hundreds.Source: §4.4 · p. 8, Table 3 · p. 9

Confidence estimation

• Confidence estimation spans verbal numerical reports, model probabilities or internal representations, sample agreement, and trained dedicated readouts.Chuang et al. (2025) train a confidence-token readout under supervised finetuning for routing and rejection.Source: §5 · p. 9

Calibration in RLVR

• CREDO trains a dedicated readout on-policy inside RLVR against a hybrid calibration target, then uses confidence–outcome discrepancies to rescale answer advantages.Unlike LaSeR, which uses its score as a self-reward signal, CREDO targets calibration.Source: §5 · p. 9
• RLVR calibration methods use response likelihood or token statistics, or score sampled numerical confidence with a reward.RLCR [Damani et al., 2026] folds a proper scoring rule into the reward, while DCPO [Ma et al., 2026] separates credit for answer and confidence tokens.Source: §5 · p. 9

6 CONCLUSION

• CREDO attains the best accuracy and calibration on mathematical and code reasoning, and its confidence supports finer abstention control.The conclusion also identifies open-ended tasks with graded rewards as a natural next step and suggests reconsidering uncertain steps during generation as future work.Source: §6 · p. 9
• CREDO reads confidence deterministically from the output distribution, trains it by differentiable regression, and uses confidence–outcome disagreement to weight answer learning.The conclusion reports that this signal improves accuracy in both domains.Source: §6 · p. 9

A PROOFS

• The proof passage represents a verbalized report as a parsed numeral sampled from the policy given the rollout prefix.The report takes values from a finite set Q within [0, 1].Source: A PROOFS · p. 14

A.1 PROOF OF PROPOSITION 1

• For squared loss, sampling adds a per-prefix tax exactly equal to report variance; for any strictly convex loss, a non-degenerate report has a positive tax.A deterministic report avoids this tax, but the proof also notes that it concentrates each prefix on one value in Q.Source: §A.1 · p. 14

A.2 PROOF OF PROPOSITION 2

• A sampled numeral has no unbiased pathwise gradient, so calibration can use its loss value only through a scalar-reward estimator.The proof states that a differentiable report at one position instead admits an exact chain-rule derivative with zero estimator variance given the rollout.Source: §A.2 · p. 14
• Exact pathwise training requires a differentiable report available inside the rollout-generating forward pass, which a verbalized numeral does not provide.Its conditional mean requires summing over possible numeral strings, whereas a single-position readout uses logits already computed in that pass.Source: §A.2 · pp. 14–15

A.3 PROOF OF PROPOSITION 3

• The Bayes-optimal readout mixes conditional correctness with question-level accuracy, while its average across rollouts at a question equals the policy’s accuracy p(x).The proposition gives the mixing weight as λ = γ + (1−γ)/G.Source: §A.3 · p. 15
• At G=8 and γ=0.5, the conditional target-variance coefficient is 0.344 rather than 1 at γ=1, but this alone establishes neither lower prediction error nor lower gradient variance.The hybrid target reduces target variance conditional on x while introducing a population-level Brier cost relative to µ.Source: §A.3 · p. 15
• For fixed γ<1, increasing group size does not generally eliminate the readout’s bias relative to conditional correctness µ.The limiting bias is (1−γ)(p(x)−µ).Source: §A.3 · p. 15

B.1 MODEL AND PROMPTS

• Experiments use Qwen3-8B with reserved H and L special tokens; the readout prompt requests one of those tokens instead of a decimal confidence.Both prompts ask for step-by-step solutions and an uncertainty analysis; the coding versions place the answer in a Python block.Source: §B.1 · p. 16

B.2 TRAINING

• Table 4 gives the training configuration, listing domain-specific values as mathematics / code.Source: Table 4 · p. 16

B.3 EVALUATION AND METRICS

• Evaluation uses temperature 0.7; mathematics and code training use DeepScaleR and DeepCoder, respectively, with evaluation sets specified in Table 5.The base model’s error bars use three evaluation seeds; trained runs use their training seed.Source: §B.3 · p. 16, Table 5 · p. 17
• Mathematics correctness is decided by symbolic equivalence, while code correctness is decided by executing programs against problem tests.Source: §B.3 · p. 17
• The reported measures include accuracy, ECE, Brier score, AUROC, and selective-prediction metrics AURC, acc@τ, and Thr.Selective prediction orders answers by confidence; tied confidence values limit achievable coverages to tie-group boundaries.Source: §B.3 · p. 17

C.1 ADDITIONAL MODEL SCALES

• CREDO leads on all four metrics at each of three model scales, with a wider calibration advantage at larger scales.Accuracy rises with model size for every method; digit-expectation rows also consistently outrank their verbalized counterparts in AUROC.Source: §C.1 · p. 18, Table 6 · p. 18

C.2 EMISSION AND TRAINING-PATH DECOMPOSITION

• CREDO outperforms every ablation on every metric; replacing its readout with digit-expectation emission causes the largest drop, while reward-only training also hurts.A standalone BCE head gives reasonable calibration but trails CREDO on every metric.Source: §C.2 · p. 18, Table 8 · p. 19

C.3 DISCREPANCY WEIGHTING ON VERBALIZED BASELINES

• Discrepancy weighting on RLCR or DCPO does not reproduce CREDO’s accuracy gains, because verbalized confidence lacks the needed per-rollout variation.Digit-expectation weighting does not close the gap; DCPO accuracy drops in both domains, while mathematics calibration worsens and several code metrics improve.Source: §C.3 · p. 19, Table 9 · p. 19

C.4 SHAPE OF THE ANSWER REWARD

• The additive answer reward is best or tied on every mathematics metric, and neither tested variant dominates it in code.The gated form removes the correctness signal after a format failure; removing the format term has a milder accuracy effect and mixed calibration results.Source: §C.4 · p. 20, Table 10 · p. 20

C.5 CALIBRATION CURVES

• CREDO’s calibration curve spans the confidence range and tracks the diagonal closely, while verbalized baselines show few points and mostly overconfidence.Figure 6 pools three seeds and uses equal-mass bins with Wilson 95% intervals; Table 1’s quantitative ECE uses ten equal-width bins.Source: §C.5 · p. 20, Figure 6 · p. 20

D.1 PER-SET RESULTS

• Tables 11–14 decompose Table 1 by evaluation set, domain, and transfer setting, with entries reported as mean ± std over three seeds.The Average column reproduces Table 1; DeepScaleR and DeepCoder are the in-distribution test splits.Source: §D.1 · p. 20, Table 11 · p. 21, Table 12 · p. 21

D.2 SENSITIVITY TO THE COEFFICIENTS

• Table 15 lists every evaluated setting of the three coefficients, with all settings run on seed 43.Source: §D.2 · p. 22, Table 15 · p. 23