PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Test-Time Scaling via Self-Calibration

Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, Jiaxin Huang

arXiv:2503.00031v1cs.LGcs.AIcs.CL

TL;DR

Fixed-sample Best-of-N and Self-Consistency can waste computation on easy queries and undersample difficult ones, while raw LLM confidence is unreliable. The paper introduces Self-Calibration to distill Self-Consistency-derived confidence into the model for one-pass estimation, then uses it for adaptive scaling. Across three LLM architectures and six datasets, the methods outperform baselines at matched budgets, including a MathQA improvement from 81.0 to 83.6 with 16 responses.

  • Problem

    Fixed sampling ignores query difficulty, while LLM confidence is often overconfident and unreliable for efficient test-time scaling.

  • Method

    Self-Calibration distills Self-Consistency-derived confidence into an LLM using synthetic tuples, enabling one-forward-pass confidence and confidence-based sampling strategies.

  • Results

    Across three LLM architectures and six datasets, confidence-based approaches consistently outperform baselines under the same sampling budget; MathQA accuracy rises from 81.0 to 83.6 with 16 responses.

  • Takeaways & Limitations

    Reliable confidence estimation and dynamic sampling substantially enhance the effectiveness and efficiency of test-time scaling.

  • Takeaways & Limitations

    Raw P(True) confidence often lacks calibration and can be overconfident on incorrect answers, motivating supervised calibration.

Abstract

from arXiv · show

Increasing test-time computation is a straightforward approach to enhancing the quality of responses in Large Language Models (LLMs). While Best-of-N sampling and Self-Consistency with majority voting are simple and effective, they require a fixed number of sampling responses for each query, regardless of its complexity. This could result in wasted computation for simpler questions and insufficient exploration for more challenging ones. In this work, we argue that model confidence of responses can be used for improving the efficiency of test-time scaling. Unfortunately, LLMs are known to be overconfident and provide unreliable confidence estimation. To address this limitation, we introduce Self-Calibration by distilling Self-Consistency-derived confidence into the model itself. This enables reliable confidence estimation at test time with one forward pass. We then design confidence-based efficient test-time scaling methods to handle queries of various difficulty, such as Early-Stopping for Best-of-N and Self-Consistency with calibrated confidence. Experiments on three LLMs across six datasets demonstrate the effectiveness of our approach. Specifically, applying confidence-based Early Stopping to Best-of-N improves MathQA accuracy from 81.0 to 83.6 with a sample budget of 16 responses, indicating the efficacy of confidence-based sampling strategy at inference time.

1 Introduction

Repeated-sampling methods improve LLM response quality but use a fixed number of samples, motivating confidence-based adaptive scaling. Self-Calibration distills Self-Consistency-derived confidence into the model and supports sampling strategies that improve accuracy and reduce computation.

  • Motivation: Fixed-response sampling can waste computation on simple queries and provide insufficient exploration for difficult ones.Best-of-N and Self-Consistency use a fixed sample count regardless of query difficulty.
  • Motivation: Heuristic stopping rules can limit generalizability across tasks and models, motivating task-independent, model-agnostic confidence criteria.Prior strategies may rely on manually designed features, such as stopping after repeated identical responses.
  • Self-Calibration: Self-Calibration distills Self-Consistency-derived confidence into LLMs for accurate one-forward-pass confidence estimation without human-labeled data.The method constructs pseudo training tuples containing queries, answers, and confidence values.
  • Self-Calibration: Confidence-based Early Stopping for Best-of-N and confidence-weighted Self-Consistency dynamically adjust inference-time sampling.Early Stopping selects a response once its calibrated confidence reaches a target threshold.
  • Results: Across three LLM architectures and six datasets, the proposed approaches outperform baselines under the same sampling budget.On MathQA, both methods improve accuracy from 81.0 to 83.6 with an average budget of 16 responses.
  • Results: Confidence-weighted Self-Consistency saves 94.2% of samples while reaching 85.0 accuracy on MathQA.The comparison uses standard Self-Consistency with the trained Llama-3.1-8B-Instruct model.

2 Repeated Sampling

Repeated sampling generates multiple candidate responses and selects an answer using scoring, majority voting, or adaptive agreement. Adaptive Self-Consistency reduces sampling by stopping when answer agreement crosses a threshold, while the Self-Calibration framework adds confidence-based training and selection.

  • Repeated Sampling: Repeated sampling generates multiple responses and uses a verifier or scoring mechanism to select a final answer.The framework is presented as a way to enhance response quality by selecting among generated candidates.
  • Best-of-N: Best-of-N assigns scores to candidate responses and selects the response with the highest score.Scores may come from an additional reward model or a confidence generator.
  • Self-Consistency: Self-Consistency selects the most frequent response among multiple candidates through majority voting.Aggregating diverse outputs provides robustness compared with relying on one highest-scoring response.
  • Adaptive Self-Consistency: Adaptive Self-Consistency adjusts sample counts using cumulative and relative frequencies of generated answers.Sampling continues until the maximum relative frequency exceeds a predefined threshold.
  • Self-Calibration: Self-Calibration trains confidence estimation from synthetic query-answer-confidence tuples and uses calibrated scores for inference-time scaling.The framework samples responses, queries confidence, groups answers, and computes Soft Self-Consistency scores during training.
  • Adaptive Self-Consistency: Adaptive Self-Consistency reduces computational costs by limiting required samples while maintaining high final-answer accuracy.Its stopping rule is based on answer agreement among generated responses.

3 Self-Calibration

The Self-Calibration framework creates confidence-labeled training data without human annotations, then trains LLMs to estimate calibrated confidence in one forward pass. It combines intrinsic confidence, response agreement, and diverse sampling while preserving reasoning ability during training.

  • Self-Calibration synthesizes query–response–confidence tuples without ground-truth answers or human annotations for training.
  • The framework trains the model to output calibrated confidence scores in one forward pass using synthetic training tuples.
  • Soft Self-Consistency combines intrinsic P(True) estimates with agreement among sampled responses to produce more reliable confidence targets.
  • Dynamic Temperature sampling increases temperature when output entropy is low to promote response diversity while preserving quality.
  • Training combines SmoothL1 confidence loss with Chain-of-Thought generation loss, selecting responses above confidence threshold η to protect reasoning quality.

4 Confidence-Guided Test-Time Scaling

Confidence-guided test-time scaling adapts sampling and aggregation using calibrated response confidence. The methods stop sampling once confidence is sufficient or weight responses according to their confidence.

  • Early Stopping for Best-of-N terminates sequential sampling when a response reaches confidence threshold τ, reducing unnecessary computation.
  • Self-Consistency with Confidence weights each sampled response by its confidence instead of treating all responses equally during voting.
  • Adaptive Self-Consistency uses confidence as the weight when calculating relative response frequencies.

5 Experiments

Experiments evaluate Self-Calibration and confidence-based test-time scaling across three language models and in-domain and out-of-domain reasoning datasets. Results show improved calibration, accuracy, and sampling efficiency under matched budgets.

  • Experiment Setup: Experiments use three open-source LLMs and evaluate mathematical, commonsense, numerical, and spatial reasoning across in-domain and out-of-domain datasets.The evaluation includes ARC-Challenge, Object-Counting, MathQA, GSM8K, SVAMP, and ARC_easy.
  • Evaluation Metrics: ECE measures the discrepancy between predicted confidence and actual accuracy, with lower values indicating better calibration.Calibration is evaluated alongside AUC and accuracy on in-domain and out-of-domain datasets.
  • Self-Calibration Results: Self-Calibration reduces GSM8K ECE from 13.70 to 3.79 and improves accuracy from 77.44% to 80.43%.On MathQA, Qwen2.5-7B-Instruct accuracy increases from 49.85% to 64.18%, demonstrating gains on an out-of-domain task.
  • Ablation Study: Removing dynamic temperature or soft self-consistency increases ECE and/or decreases accuracy, while replacing L1-smooth loss with MSE reduces accuracy on both tasks.The ablation evaluates MathQA and Object Counting and indicates that each component contributes to calibration or reasoning performance.

6 Analysis

The analysis compares self-generated confidence with reward models and examines performance across sampling budgets and confidence-query prompts. Self-generated confidence matches reward-model performance while using far fewer additional resources, and alternative prompts remain effective.

  • Confidence versus reward models: Reward models can require normalization, increase inference time, and consume additional GPU memory.These constraints make universal thresholding and large-scale deployment less efficient.
  • Confidence versus reward models: Self-generated confidence achieves similar Best-of-16 accuracy to same-sized reward models across all evaluated datasets.The approach adds approximately 10 tokens instead of requiring an additional reward model.
  • Performance across sample budgets: Confidence-guided methods generally outperform their original counterparts as the sample budget increases.All methods improve accuracy with more responses, but the confidence-guided variants usually perform better.
  • Performance across sample budgets: At small sample budgets, Best-of-N can outperform early stopping because a low threshold may stop before discovering a better response.This highlights a trade-off between early computational savings and sufficient exploration.
  • Prompt robustness: Six alternative confidence-querying prompts produce comparable performance with only minor variations across datasets.The result suggests calibration is not tied to one exact inference prompt.

7 Related Work

Related work covers adaptive test-time computation, model calibration, and LLM-based verification. These lines of work motivate confidence-guided sampling and situate the paper among methods that evaluate or select generated reasoning.

  • Adaptive test-time computation: Prior test-time scaling methods allocate computation through tree search, sparse activation, step-wise pruning, or staged elimination.These approaches improve efficiency by controlling which reasoning paths receive further computation.
  • Model calibration: Calibration research aims to align model confidence with accuracy, using scaling, binning, and verbalized-confidence techniques.The paper addresses the same alignment problem in the setting of LLM inference.
  • LLM verification: LLM verifiers assess generated responses or reasoning paths to improve the selection of accurate outputs.Examples include trained verifiers, LLM judges, next-token verifiers, and iterative reasoning-trace methods.

8 Conclusion

The paper proposes reliable confidence estimation and dynamic test-time scaling without labeled data. Its experiments report better baseline performance under matched sampling budgets, supporting confidence-guided inference as a more efficient scaling strategy.

  • Contribution: Self-Calibration improves LLM confidence estimation in one forward pass without requiring labeled data.It distills Self-Consistency-derived confidence into the model.
  • Contribution: Dynamic sampling strategies use calibrated confidence for Early-Stopping Best-of-N and confidence-weighted Self-Consistency.These methods adjust inference computation according to the model’s estimated confidence.
  • Findings: The proposed approaches consistently outperform baselines under the same sample budget.The reported conclusion links reliable confidence estimation with improved test-time scaling effectiveness and efficiency.

B Full Main Results

The full results examine performance under a small sample budget and confidence-querying prompt variants. Confidence signals are especially useful when sampling is constrained, while higher budgets allow early stopping to exploit promising reasoning paths.

  • Small-budget results: With sample budget 4, an additional confidence signal helps filter noisy or incorrect responses when reasoning-path exploration is limited.The confidence-augmented method selects the most promising candidate under tight sampling constraints.
  • Larger-budget results: As the sample budget grows, Early Stopping can terminate after encountering a correct reasoning path, especially with a high confidence threshold.More candidate solutions increase the chance of finding a correct path before termination.
  • Prompt evaluation: Six alternative confidence-querying prompts are evaluated for inference-time robustness.The prompts include six differently worded correctness questions.
  • Prompt evaluation: The experiments report results for alternative prompts on tuned Llama-3.1-8B-Instruct.The comparison is presented in Table 8.

D Results for Different Sample Budgets

This section compares inference strategies across sample budgets, models, and datasets, including ARC_Challenge, Object Counting, and MathQA. Table 7 reports accuracy comparisons across three language models, while Figures 4–9 provide dataset- and model-specific comparisons.

  • Additional sample budgets: Additional results examine performance under different sample budgets for other datasets and models.
  • Llama-3.1-8B-Instruct: Figures 4–6 compare inference strategies on ARC_Challenge, Object Counting, and MathQA using Self-Calibration trained Llama-3.1-8B-Instruct.
  • Accuracy comparisons: Table 7 compares test-time scaling methods across three language models on Object Counting, MathQA, and ARC_Challenge.The table defines sample budget as the average number of responses sampled per query and reports confidence-augmented improvements over baselines in parentheses.
  • Qwen-2.5-7B-Instruction: Figures 7–9 compare inference strategies on ARC_Challenge, Object Counting, and MathQA using Self-Calibration trained Qwen-2.5-7B-Instruction.

E.1 Training Data Generation

The experiments use specified response-generation, training-objective, optimization, parameter-efficient fine-tuning, and dataset-mixture settings. Training-data selection balances examples across confidence intervals to support calibration robustness.

  • Training data generation: Training-data generation uses N = 32 responses per query and dynamic-temperature settings T0 = 0.8, M = 0.8, γ = 1.0, and τ0 = 0.001.
  • Training objective: The training objective uses η = 0.75 to filter responses for generation-ability training and w = 0.1 to balance the two losses.
  • Training process: Training uses AdamW with learning rate 5 × 10−5, 100,000 training samples, 1,000 evaluation samples, batch size 1, 64 accumulation steps, and one epoch.
  • Parameter-efficient fine-tuning: LoRA uses rank r = 32, scaling factor α = 16, and dropout 0.05, while causal language modeling data comprise 0.7 of the training examples.
  • Dataset mixture: GSM8K and SVAMP each contribute 15% of training and evaluation samples, while seven other datasets each contribute 5%.The 5% datasets are SciQ, CommonsenseQA, Winogrande, OpenBookQA, ReClor, ARC-Easy, and LogiQA.
  • Confidence-balanced selection: Samples are evenly distributed across confidence intervals so each confidence bin contributes the same number of training examples.The stated purpose is to reduce overrepresentation of particular confidence ranges and improve calibration robustness.
  • Response generation: Response generation uses temperature 1.0.
Loading 2503.00031v1…