PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-time Calibration Learning for Large Language Model Reasoning

Zizhuo Zhang, Xiong Peng, Jingwei Sun, Rong Yao, Shixiong Kai, Mingxuan Yuan, Bo Han

arXiv:2610.02695v1cs.LGcs.AI

TL;DR

Existing calibration learning relies on ground-truth labels, limiting its use on unlabeled test tasks where calibration may shift across domains. TTCL builds correctness and calibration supervision from multiple model responses and adapts on target data. Experiments report consistent accuracy and calibration improvements, including for pre-calibrated models under domain shift.

  • Problem

    Existing calibration learning uses ground-truth correctness labels, while calibration learned on a source domain may not transfer to a new target task.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    TTCL derives correctness and calibration self-supervision from multiple responses and separately applies the rewards to answer and confidence tokens.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    TTCL consistently improves accuracy and calibration on mathematical reasoning and factual QA, and further adapts pre-calibrated models when source-domain transfer is limited.

    Source: §5.2 · p. 8, §6 · pp. 9–10

  • Takeaways & Limitations

    TTCL provides a label-free surrogate for calibration learning that can adapt confidence estimation directly to target tasks at test time.

    Source: §1 · p. 2

    Using the current rollout distribution for supervision can reinforce answer consistency until rollouts collapse to an incorrect answer with confidence approaching 1.0.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incorporate calibration learning into reinforcement learning (RL), jointly optimizing answer correctness and verbalized confidence using ground-truth correctness supervision. However, their reliance on labeled data limits their applicability in practical test-time settings, where ground-truth labels are unavailable and calibration may need to adapt to newly encountered target tasks. To address this challenge, we propose Test-Time Calibration Learning (TTCL), a label-free framework that jointly adapts reasoning accuracy and verbalized confidence directly on unlabeled target-task data. Specifically, TTCL derives self-supervision signals for both correctness and calibration from multiple model-generated responses, enabling calibration learning at test time without ground-truth labels. Theoretical analysis further establishes TTCL as a bounded surrogate for the ideal calibration objective. Extensive experiments on mathematical reasoning and factual question answering demonstrate that TTCL consistently improves both accuracy and calibration across diverse models and tasks. On base models, TTCL achieves an average relative accuracy improvement of +40.13% and an ECE reduction of +70.80% across eight benchmarks. Moreover, TTCL can further improve both accuracy and calibration for already calibrated models under domain shift, particularly when source-domain calibration transfers poorly to target tasks. In the math-to-factQA setting, TTCL achieves an average relative accuracy gain of +20.35% and reduces ECE by +53.83%. The source code is released at https://github.com/tmlr-group/TTCL.

1 INTRODUCTION

Existing calibration learning uses labeled correctness supervision and may transfer poorly across domains, motivating adaptation on unlabeled target-task data. TTCL derives separate self-supervision for correctness and calibration from multiple responses, and experiments report consistent improvements in reasoning and calibration.

1 INTRODUCTION

  • Calibration learning combines correctness and confidence rewards, but existing methods rely on labeled data and may transfer poorly from source to target tasks.The introduction highlights RLCR's limited calibration on factual NQ after calibration on math DAPO-Math-14K, despite transfer to AMC.Source: §1 · p. 2
  • TTCL derives correctness and calibration self-supervision from multiple responses, applying the rewards separately to answer and confidence tokens.A target cache maintains lagged supervision targets to reduce the risk of reinforcing instantaneous predictions.Source: §1 · p. 2
  • TTCL is presented as a bounded surrogate for ideal calibration learning, with experiments reporting consistent reasoning and calibration improvements across models and benchmarks.The stated contributions span conceptual, technical, theoretical, and experimental advances.Source: §1 · p. 2

2 PRELIMINARY

The preliminary section defines calibration learning and ECE, then frames test-time adaptation as a response to ground-truth label requirements. It also illustrates that source-domain calibration may transfer unevenly across tasks.

2 PRELIMINARY

  • A confidence estimate is calibrated when predictions at that confidence are correct at the corresponding rate; calibration learning commonly optimizes confidence against observed correctness.The Brier score is given as a proper scoring rule that encourages predicted confidence to match the probability of correctness.Source: §2 · p. 2
  • Existing RL calibration rewards combine binary correctness and Brier-score calibration terms, but both require the ground-truth answer.The paper therefore focuses on adapting confidence estimation using unlabeled test data.Source: §2 · p. 3
  • ECE measures discrepancy between predicted confidence and empirical accuracy, with lower ECE indicating better calibration.Figure 2 shows RLCR calibration transferring well from DAPO-Math-14K to AMC but remaining limited on NQ.Source: §2 · p. 3, Figure 2 · p. 3

3 TTCL: TEST-TIME CALIBRATION LEARNING

TTCL adapts confidence directly to unlabeled target tasks, deriving correctness and calibration targets from response consensus and stabilizing them with a target cache. Its rewards update reasoning and answer tokens separately from confidence tokens.

3.1 WHY TEST-TIME CALIBRATION LEARNING (TTCL)?

  • TTCL addresses costly or unavailable labels and limited cross-task transfer by adapting confidence directly on unlabeled target-task data.RLCR calibration learned on DAPO-Math-14K transfers well to AMC but remains limited on NQ.Source: §3.1 · p. 3
  • ECE falls from 0.730 to 0.290 when TTCL adapts RLCR-calibrated Qwen3-8B on unlabeled NQ data.The adaptation requires no additional ground-truth annotations.Source: §3.1 · p. 4

3.2 TTCL SELF-SUPERVISION SIGNALS AND OPTIMIZATION OBJECTIVE

  • TTCL uses the rollout group's most-supported answer as its pseudo correctness target and each generated answer's empirical support as its confidence target.This supervision assumes that answers with stronger consensus are more likely to be correct and should receive higher confidence.Source: §3.2 · p. 4
  • Correctness and calibration advantages update different response components: reasoning and answer tokens versus confidence tokens.This separation is intended to avoid undesirable interference between the two rewards.Source: §3.2 · pp. 4–5

3.3 TARGET CACHE FOR STABLE LEARNING

  • A target cache supplies frozen, temporally lagged answer distributions for supervision and updates them with an EMA for subsequent epochs.The cache is introduced to mitigate feedback that could collapse rollouts to one incorrect answer with confidence approaching 1.0.Source: §3.3 · p. 5

4 THEORETICAL ANALYSIS OF TTCL

TTCL uses answer frequency as a label-free proxy for conditional correctness and optimizes a bounded surrogate of the ideal calibration objective. The bound depends on the estimation error of that frequency-based target.

4 THEORETICAL ANALYSIS OF TTCL

  • Theorem 4.1 bounds the gap between TTCL’s calibration objective and the ideal objective by 2ϵ for any confidence estimator.Here, ϵ is the expected absolute error between the frequency-based confidence target and latent conditional correctness probability.Source: §4 · pp. 5–6
  • The empirical frequency of an answer is an unbiased and consistent estimator of its population probability, which can proxy conditional correctness when the answer distribution reflects model uncertainty.This supports using answer frequency as TTCL’s label-free confidence target.Source: §4 · p. 6
  • Conditional correctness probability is the ideal calibration target: matching confidence to it yields perfect calibration, and minimizing expected Brier score yields that estimator.The theoretical bridge connects answer frequency to conditional correctness and then to calibrated confidence.Source: §4 · p. 6

5 EXPERIMENTS

TTCL is evaluated on mathematical reasoning and factual QA, including base and RLCR-calibrated models. Across the reported experiments, it improves accuracy and calibration, adapts under domain shift, and largely preserves performance on other domains.

5.2 MAIN EXPERIMENTAL RESULTS

  • +41.35% in ACC and +73.12% in ECE on mathematical reasoning, and +31.02% in ACC and +66.37% in ECE on factual QA, are TTCL’s average relative improvements for base models.The prose reports consistent improvements across the two benchmark groups; Tables 1 and 2 present the base-model results.Source: §5.2 · p. 7
  • TTCL improves ACC and ECE on RLCR-calibrated models in both math-to-FactQA and FactQA-to-math settings, where source-domain calibration transfers imperfectly.It also further improves accuracy and calibration in the in-domain math-to-math setting.Source: Table 4 · p. 7, Table 3 · p. 7, §5.2 · p. 8
  • TTCL generally achieves higher ACC and AUROC and lower ECE than existing label-free calibration baselines, while RLCR serves as a label-leakage reference.The compared label-free baselines mainly use post-hoc confidence estimation without model training, so their ACC remains unchanged; TTCL can match or surpass RLCR in calibration in some cases without labels.Source: §5.2 · p. 8, Table 5 · p. 8

5.3 FURTHER ANALYSIS

  • TTCL-trained models maintain performance comparable to their pre-adaptation counterparts on five other-domain benchmarks while improving target-task calibration.The evaluation covers scientific reasoning, instruction following, multi-domain knowledge, and truthfulness.Source: §5.3 · p. 8, Table 6 · p. 8
  • Combining correctness and calibration rewards achieves a better balance: correctness-only optimization preserves accuracy but worsens calibration, while calibration-only optimization lowers ECE but degrades accuracy.Figure 4’s ablation compares TTCL with variants retaining only one of the two rewards.Source: §5.3 · pp. 8–9, Figure 4 · p. 8
  • TTCL achieves lower ECE and higher ACC and AUROC than Pseudo-RLCR in most cases, supporting frequency-based confidence supervision over binary pseudo-correctness supervision.Pseudo-RLCR uses majority-vote pseudo-labels for both correctness and confidence supervision; TTCL uses answer frequency for confidence.Source: §5.3 · p. 9, Table 7 · p. 9
  • During training, TTCL narrows the gap between initially overconfident mean confidence and empirical accuracy while improving accuracy.Figure 5 traces accuracy and mean confidence over training.Source: §5.3 · p. 9, Figure 5 · p. 9
  • After TTCL, accuracy across confidence bins moves closer to the ideal calibration diagonal across math and factual QA, for base and pre-calibrated models.Figure 6 visualizes confidence against empirical accuracy.Source: §5.3 · p. 9, Figure 6 · p. 9

6 CONCLUSION

TTCL adapts confidence to target tasks without labels, using multiple model-generated responses for self-supervision; the paper also presents a bounded-surrogate analysis and reviews its relation to prior work and implementation.

6 CONCLUSION

  • TTCL consistently improves accuracy and calibration on mathematical reasoning and factual QA, and can further adapt pre-calibrated models when source-domain calibration transfers poorly.The conclusion also identifies test-time calibration in unlabeled deployment settings as a direction for further research.Source: §6 · pp. 9–10

A GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

  • The experiments use GRPO, with mean-only advantage normalization because standard-deviation normalization can make continuous Brier-reward updates unstable or extreme.The paper states that its focus is test-time confidence calibration, rather than comparing GRPO variants.Source: A GROUP RELATIVE POLICY OPTIMIZATION… · p. 14

B RELATED WORK

  • TTCL is positioned against label-free RL methods focused on correctness and calibration-learning methods that use labeled supervision and may transfer poorly across domains.The related-work discussion distinguishes post-hoc confidence estimation from training models to express calibrated confidence.Source: B RELATED WORK · pp. 14–15

C PSEUDOCODE OF TTCL

  • The target cache is initialized with the initial policy, held fixed during each epoch, and updated at epoch end using the newly observed answer distribution.Source: C PSEUDOCODE OF TTCL · p. 15

D FURTHER THEORETICAL ANALYSIS OF TTCL

  • The theoretical analysis establishes the frequency-based probability target’s statistical properties, identifies conditional correctness as the ideal calibration target, and proves a bounded-surrogate result for TTCL.Source: D FURTHER THEORETICAL ANALYSIS OF TTCL · p. 15

D.1 PROOF OF PROPOSITION 4.2

This proof establishes empirical answer frequency as an unbiased and consistent estimate of the model’s answer probability, supporting its use as a label-free proxy for correctness.

D.1 PROOF OF PROPOSITION 4.2

  • The empirical frequency of an answer across sampled responses is an unbiased estimator of its population probability and converges to that probability.The proof represents each sampled response’s match to the candidate answer as an indicator with expectation equal to the model’s answer probability.Source: §D.1 · pp. 15–16
  • TTCL initializes its cache from the initial policy’s empirical answer distribution, freezes it during each epoch, and updates it afterward using an exponential moving average.Source: §D.1 · p. 16
  • When the model’s answer distribution reflects its uncertainty about correctness, repeated-response frequency provides a proxy for conditional correctness without ground-truth labels.Source: §D.1 · p. 16

D.2 PROOF OF PROPOSITION 4.3

The supplied material connects conditional correctness to perfect calibration and Brier-score minimization, states TTCL’s surrogate bound, and provides baseline and implementation context.

D.2 PROOF OF PROPOSITION 4.3

  • Conditional correctness is sufficient for perfect calibration and is the confidence target that minimizes expected Brier score.The proof states that confidence equal to conditional correctness satisfies the perfect-calibration condition.Source: §D.2 · p. 17

D.3 PROOF OF THEOREM 4.1

  • For any confidence estimator, the absolute difference between TTCL’s objective and the ideal calibration objective is bounded by 2ε.This establishes TTCL as a bounded surrogate for the ideal objective.Source: §D.3 · p. 18

E.1 DETAILS OF BASELINES

  • The baselines include label-dependent RLCR, confidence self-evaluation, verbalization, repeated confidence elicitation, and base-model re-evaluation.RLCR uses ground-truth answer labels for its correctness and calibration rewards.Source: §E.1 · pp. 18–19

E.2 MORE IMPLEMENTATION DETAILS

  • The implementation uses distinct answer-format prompts for mathematical reasoning and factual QA, and adjusts batch size and epoch count by benchmark size.Smaller benchmarks use smaller batches and more epochs to provide sufficient training steps.Source: §E.2 · p. 20

E.3 MORE BENCHMARK DETAILS

The appendix describes mathematical-reasoning and factual-QA benchmarks, visualizes calibration changes, and details evaluation metrics and sampling choices.

E.3 MORE BENCHMARK DETAILS

  • Evaluation covers mathematics benchmarks MATH500 (a 500-question MATH subset), AMC, and AIME24–AIME26, plus factual-QA SimpleQA-Verified (1,000 questions), ChineseSimpleQA, and 200 randomly sampled NQ test examples.Source: §E.3 · pp. 20–21
  • The appendix also evaluates broader capabilities across scientific reasoning, instruction following, multi-domain knowledge, and truthfulness.The listed benchmarks include ARC, GPQA-Diamond, IFEval, MMLU-Pro, and TruthfulQA.Source: §E.3 · pp. 22–23, Table 14 · p. 25
  • The visualization captions report that TTCL narrows confidence–accuracy gaps on base-model math and factual-QA benchmarks and on calibrated models under domain shift.The out-of-domain caption says source-domain calibration transfers poorly in both Math-to-FactQA and FactQA-to-Math settings.Source: Figure 7 · p. 22, Figure 8 · p. 22, Figure 10 · p. 23

E.4 MORE DETAILS OF METRICS

  • Accuracy measures the proportion of correct answers, while ECE measures the discrepancy between predicted confidence and empirical correctness.The ECE computation uses 10 bins.Source: §E.4 · p. 23
  • AUROC measures how well confidence ranks correct responses above incorrect ones, unlike ECE’s measure of absolute confidence–correctness alignment.Source: §E.4 · p. 24
  • The evaluation samples eight responses per question on smaller benchmarks and one response per question on MATH500, SimpleQA, and ChineseSimpleQA.Source: §E.4 · p. 24

F.1 MORE RESULTS OF CALIBRATION PERFORMANCE

TTCL improves calibration across model and domain settings, while also improving accuracy on base models. Its confidence adjustments track empirical accuracy rather than simply reducing confidence.

F.1 MORE RESULTS OF CALIBRATION PERFORMANCE

  • Base models often show mean confidence considerably above empirical accuracy, with large Brier scores on some benchmarks.Brier score measures squared discrepancy between confidence and correctness; lower scores indicate better calibration.Source: §F.1 · p. 25
  • TTCL consistently lowers Brier scores and narrows the gap between mean confidence and empirical accuracy across base and RLCR-calibrated models.For base models, these changes accompany improved answer accuracy; TTCL also adapts confidence when source-domain calibration transfers poorly.Source: §F.1 · p. 25
  • For RLCR-calibrated Qwen3-4B on MATH500, TTCL raises mean confidence from 0.772 to 0.913, close to 90.4% accuracy.This example illustrates that calibration improvement need not come from indiscriminately suppressing confidence.Source: §F.1 · p. 25

F.2 MORE VISUALIZATIONS OF TTCL CALIBRATION EFFECTS

  • Across base and calibrated models, TTCL improves confidence-accuracy alignment in-domain and adapts estimates when source-domain calibration transfers poorly.The visualizations cover mathematical reasoning, factual QA, and cross-domain transfer.Source: §F.2 · p. 26

F.3 MORE TRAINING DYNAMICS OF TTCL

TTCL training moves confidence toward empirical accuracy while improving task accuracy. Without a target cache, current-rollout supervision can instead reinforce confidently repeated wrong answers.

F.3 MORE TRAINING DYNAMICS OF TTCL

  • Across models and target benchmarks, TTCL moves mean confidence toward empirical accuracy while improving task accuracy during test-time learning.Base-model overconfidence is progressively corrected, while calibrated models further adapt confidence to target-domain performance.Source: §F.3 · pp. 26–27

F.4 REWARD HACKING WITHOUT TARGET CACHE

  • Without a target cache, repeated incorrect rollouts can raise their frequency-based confidence target toward 1.0, producing reward hacking.Fixing targets within each epoch weakens immediate feedback between the current policy and its self-supervision signals.Source: §F.4 · p. 27, Figure 13 · p. 27

F.5 MORE RESULTS OF GENERAL CAPABILITIES

TTCL generally preserves models’ performance on broader capabilities while improving target-task accuracy and calibration. Detailed IFEval results likewise show instruction-following performance remains comparable after adaptation.

F.5 MORE RESULTS OF GENERAL CAPABILITIES

  • TTCL-trained models generally maintain performance comparable to their pre-adaptation counterparts across benchmarks of general capabilities.The reported benchmarks include scientific reasoning, instruction following, multi-domain knowledge, and truthfulness.Source: §F.5 · p. 27

F.6 DETAILED RESULTS OF IFEVAL

  • Across target tasks and model settings, IFEval performance after TTCL remains generally comparable to before training, with slight improvements in several cases.Table 15 reports prompt-level and instruction-level strict and loose accuracy separately.Source: §F.6 · p. 28

F.7 CALIBRATION RESULTS OF EXISTING LABEL-FREE RL METHODS

The evaluated label-free RL methods improve accuracy to some extent but retain poor calibration. TTCL case studies illustrate confidence aligned with correctness, including adaptation when source-domain calibration transfers poorly.

F.7 CALIBRATION RESULTS OF EXISTING LABEL-FREE RL METHODS

  • Evaluated label-free RL methods improve accuracy to some extent, but their calibration remains relatively poor, with generally high ECE and mean confidence above 0.8.Mean confidence is typically above 0.9 for the Qwen models; these methods lack an explicit calibration objective.Source: §F.7 · p. 28

F.8 CASE STUDY

  • TTCL case studies show models assigning high confidence to correct answers and relatively low confidence to incorrect answers.When source-domain calibration transfers poorly, TTCL can adapt confidence to the target distribution and improve calibration.Source: §F.8 · pp. 28–29
Loading 2610.02695v1…