PaperlayerView exact revision on arXiv

Source-linked AI summary

Believe Your Model: Distribution-Guided Confidence Calibration

Xizhong Yang, Haotian Zhang, Huiming Wang, Mofei Song

arXiv:2603.03872v1cs.LG

TL;DR

Confidence distributions contain information about answer correctness, but prior work has not fully used that structure for test-time answer selection. DistriVoting decomposes confidence scores with GMM and filters candidates, while SelfStepConf adjusts inference using step-level confidence to increase distribution separation. Across 16 models and 5 benchmarks, the methods consistently improve performance over the evaluated alternatives.

  • Problem

    Prior work observes distinct confidence distributions for correct and incorrect trajectories but mainly uses them to assess confidence reliability rather than guide answer selection.

  • Method

    DistriVoting decomposes mixed confidence scores into positive and negative components with GMM, filters candidates, and uses negative-distribution votes to reject false positives; SelfStepConf intervenes with step-level confidence.

  • Results

    Across 16 models and 5 reasoning benchmarks, DistriVoting consistently improves performance, while SelfStepConf further increases separation between positive and negative distributions.

  • Takeaways & Limitations

    The results support using confidence-distribution structure alongside confidence scores to improve test-time voting reliability.

  • Takeaways & Limitations

    The voting-accuracy analysis assumes normal distributions and mutually independent samples, including one correct answer and multiple incorrect answers.

Abstract

from arXiv · show

Large Reasoning Models have demonstrated remarkable performance with the advancement of test-time scaling techniques, which enhances prediction accuracy by generating multiple candidate responses and selecting the most reliable answer. While prior work has analyzed that internal model signals like confidence scores can partly indicate response correctness and exhibit a distributional correlation with accuracy, such distributional information has not been fully utilized to guide answer selection. Motivated by this, we propose DistriVoting, which incorporates distributional priors as another signal alongside confidence during voting. Specifically, our method (1) first decomposes the mixed confidence distribution into positive and negative components using Gaussian Mixture Models, (2) then applies a reject filter based on positive/negative samples from them to mitigate overlap between the two distributions. Besides, to further alleviate the overlap from the perspective of distribution itself, we propose SelfStepConf, which uses step-level confidence to dynamically adjust inference process, increasing the separation between the two distributions to improve the reliability of confidences in voting. Experiments across 16 models and 5 benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches.

1 Introduction

Test-time scaling improves reasoning by generating more candidate answers, but answer selection remains difficult without test-time labels or rewards. DistriVoting addresses this gap by using confidence-distribution structure, with SelfStepConf further separating positive and negative distributions.

  • Test-time scaling generates multiple answers or increases token computation, yet evaluating answer quality during inference remains difficult without labels or rewards.
  • Prior work finds that correct and incorrect trajectories typically follow distinct confidence distributions, but mainly uses this relationship to assess confidence reliability.
  • DistriVoting models confidence as two Gaussian components, selects potential positive answers, filters likely negatives, and rejects false positives using negative-distribution votes.
  • SelfStepConf applies confidence during individual trajectories to provide real-time supervision and further alleviate overlap between positive and negative distributions.
  • Experiments span 16 models and 5 reasoning benchmarks, showing consistent DistriVoting gains and additional improvements from SelfStepConf.

2 Preliminaries

The paper defines trajectory confidence from token probabilities and models sampled confidence values as positive and negative normal distributions. It analyzes how their separation relates to voting accuracy and motivates SelfStepConf as a way to increase that separation.

  • 2.1 Confidence of Trajectory: Trajectory confidence uses token negative log-probabilities to assess the quality of a generated reasoning trajectory during and after inference.
  • 2.1 Confidence of Trajectory: The trajectory-level confidence calculation can focus on a tail subset of generated tokens containing the answer and use top-k token probabilities.
  • 2.2 Confidence Distribution: Repeated sampling forms an original confidence distribution that is modeled as positive and negative normal distributions, with higher and lower means respectively.
  • 2.3 Distributions Distance and Voting Accuracy: Theorem 2.1 states that SelfStepConf amplifies separation between positive and negative prediction distributions, directly improving voting accuracy.
  • 2.3 Distributions Distance and Voting Accuracy: The distribution-distance analysis assumes normal densities with positive variances and considers weighted, mutually independent samples with one correct answer and multiple incorrect answers.
  • 2.3 Distributions Distance and Voting Accuracy: The integral ratio R(µ1, µ2) increases strictly with the mean gap δ = µ1 − µ2 when the means differ.
  • 2.3 Distributions Distance and Voting Accuracy: Experiments in §5.2 validate that SelfStepConf achieves superior voting performance through effective distribution separation.

3 Methodology

The method combines SelfStepConf for dynamic confidence-guided reflection with DistriVoting, which separates trajectory confidence distributions and filters candidates before hierarchical voting.

  • SelfStepConf: SelfStepConf monitors step-wise confidence and triggers self-reflection when confidence declines significantly during reasoning.The method compares step confidence with an adaptive threshold and activates reflection when the relative change falls below a control parameter.
  • SelfStepConf: When confidence drops below the threshold, reflection is injected by swapping probabilities for the highest-probability token and a reflection token.The swapped distribution is sampled at temperature 0 for the specified reflection-token steps, while token confidence remains unchanged.
  • GMM Filter: GMM models unlabeled trajectory confidences with two Gaussian components and maps the higher-mean component to potentially positive trajectories.The resulting positive and negative trajectory sets provide separate confidence sets for subsequent filtering.
  • Reject Filter: The reject filter uses negatively weighted confidence from potentially incorrect trajectories to identify negative answers and remove false positives from the voting pool.When positive and negative answers differ, the candidate pool is further filtered based on the negative answer before final voting.
  • Hierarchical Voting: Hierarchical voting partitions confidence values into sub-intervals, performs weighted majority voting within intervals, and then combines interval answers.This accounts for varying correct-to-incorrect answer ratios across confidence intervals.
  • Overall pipeline: The two filtering processes remove true negative and false positive answers, enhancing confidence reliability in final voting.The main results table evaluates SelfStepConf and DistriVoting across benchmarks under a budget of 128 and 64 repetitions.

4 Experiment

The experiments evaluate DistriVoting and SelfStepConf across models, benchmarks, clustering methods, and sampling budgets. Results consistently favor adaptive distribution-based filtering and enhanced confidence separation.

  • Main comparisons: 76.64% and 75.79% improve over 74.75% and 75.22% with WSC, while 76.95% and 75.99% improve over 76.32% and 75.79% with DIS.These comparisons evaluate adaptive GMM filtering against fixed Top50 filtering across two models.
  • Main comparisons: DistriVoting consistently outperforms WSC across all three models under both Top50 and GMM Filter settings.The comparison tests distribution-aware voting against naive weighted voting.
  • SelfStepConf: SSC provides substantial and consistent gains across all models for both WSC and DistriVoting approaches.The GMM and GMM* comparison assesses whether SSC improves confidence distribution discrimination.
  • Clustering ablation: GMM achieves 1.78× the efficiency of K-Means and 5.49× that of MeanShift while outperforming both in trajectory correctness prediction accuracy.The resulting confidence quality and voting performance also favor GMM for bimodal normal distributions.
  • Budget ablation: SSC consistently outperforms BasicInference across budgets 8, 16, 32, 64, 128, and 256, while adaptive GMM filtering surpasses fixed Top50 filtering.DistriVoting has significant advantages over conventional methods when Budget ≥ 16 and comparable performance at smaller budgets.

5 Analysis

The analyses examine how filtering, confidence separation, adaptive thresholds, answer distributions, and inference-time reflection affect voting. Together, they link improved trajectory selection and confidence separation to stronger voting behavior, while showing that SSC improves sampling efficiency without expanding reasoning limits.

  • Filtering effects: Acc and WAcc increase from all samples to GMM-filtered candidates and then to Reject-Filtered final samples.The two stages respectively improve the correct-sample ratio in the voting trajectory pool.
  • SSC distribution separation: SSC’s confidence distribution shows less overlap than BasicInference, indicating greater separation between positive and negative distributions.The paper also gives a theoretical link between increased µ_pos−µ_neg separation and higher voting accuracy.
  • SSC distribution separation: SSC increases the confidence-density difference from 0.165 to 0.251 in the positive interval at high confidence ranks.The higher density indicates more reliable voting information after distribution separation.
  • Adaptive threshold selection: Optimal fixed top-thresholds vary across benchmarks at 6%, 9%, 36%, 42%, and 96%, motivating adaptive selection by DistriVoting.DistriVoting selects trajectories at both benchmark and individual-question levels.
  • Gaussian interpretation: The top 8 frequent answer distributions approximately follow Gaussian patterns but overlap significantly, with the top-1 and top-2 means differing by only 0.022.Clustering confidence by correctness avoids mapping confidence through answer identities and reduces information loss.
  • Inference behavior: SSC achieves higher pass@1 than BasicInference at K=1, but their pass@K performance converges as K increases.The reported pattern characterizes SSC as improving sampling efficiency rather than expanding fundamental reasoning limits.
  • Inference behavior: After reflection triggers at steps 21 and 55, SSC maintained high confidence and produced a correct answer, whereas BasicInference’s confidence declined and its answer was incorrect.On this trajectory, SSC used 86 steps and 7,750 tokens versus 91 steps and 9,472 tokens for BasicInference.

6 Conclusion

The paper concludes that distribution-guided voting addresses confidently wrong predictions by combining confidence-distribution filtering with dynamically improved inference confidence.

  • DistriVoting: DistriVoting uses GMM Filter and Reject Filter to remove likely true-negative and false-positive samples before voting.HierVoting is used to compensate for performance deficiencies when filter quality is low.
  • SelfStepConf: SelfStepConf dynamically adjusts inference to increase the distance between positive and negative distributions and improve confidence reliability.The conclusion presents this distributional separation as complementary to DistriVoting’s filtering process.

Impact Statement

The impact statement describes a confidence-based test-time scaling method that uses model-internal information to enhance voting accuracy. It situates the work within test-time scaling and intrinsic-information research.

  • Stated impact: The method is described as enhancing voting accuracy using solely model-internal information.The statement identifies advancing machine learning as the method’s goal.
  • Research context: The work is situated alongside test-time scaling research that increases inference depth or improves information use across multiple sampling results.This context describes current approaches to improving final-answer accuracy.
  • Research context: Prior studies suggest that intrinsic information generated during reasoning can reflect response quality and support applications in training and test-time scaling.The statement presents this as related background rather than a result established by the present paper.

B.1 Proof of Theorem 2.1

The proof shows that increasing the separation δ between two normal confidence distributions increases a right-tail ratio and a lower bound on voting accuracy. Consequently, voting accuracy is strictly increasing in δ under the stated assumptions.

  • δ = µ1 − µ2 measures the nonnegative difference between the two distribution means, assuming µ1 > µ2.
  • The proof establishes positivity of the derivative using the standard normal PDF, positive variances, and non-trivial positive weights.
  • The ratio of right-tail integrals, split at the midpoint of the means, is strictly increasing as δ increases.
  • The probability that the correct weighted sum exceeds the incorrect weighted sum provides a lower bound for voting accuracy.
  • Since the lower bound increases strictly with δ, the voting accuracy Pvote(δ) also increases strictly with δ.

C.2 Implementation Details

The implementation combines step-level confidence monitoring, reflection-triggered inference adjustment, GMM-based filtering, and hierarchical weighted voting. It uses specified segmentation, model, prompt, and parameter settings to produce final answers.

  • Experimental setup: The evaluation uses Qwen3 models from 0.6B to 32B, additional DeepSeek, Qwen2.5-Math, and Llama models, across five mathematical reasoning benchmarks.
  • SelfStepConf: SelfStepConf monitors token and step confidence during generation, triggering reflection when the confidence-based condition is met.The process updates an exponential moving threshold and injects reflection tokens when the indicator IR equals 1.
  • DistriVoting: DistriVoting fits a GMM to partition trajectories into positive and negative groups before applying a reject filter and refitting the positive set.The final answer is selected through hierarchical weighted voting over the filtered trajectories.
  • Voting: Hierarchical voting divides the confidence range into NC intervals, aggregates answers within intervals, and applies weighted majority voting.
  • Voting: Weighted majority voting scores each unique answer using trajectory weights and returns the highest-scoring answer.
  • Parameters: The main parameter settings include k = 20 for token confidence and NC = 10 for base voting.The implementation details also identify α and δ as SelfStepConf parameters controlling threshold updates and reflection triggers.

E.2 Analysis of α

The sensitivity analyses find robust performance across α and δ settings, while explaining how these parameters control threshold smoothing and reflection strictness. They also show that GMM filtering reduces the value of further confidence stratification.

  • α sensitivity: α controls how strongly the previous confidence threshold influences its exponential moving update.Larger α produces a smoother threshold curve, whereas smaller α makes the threshold follow the current step confidence more closely.
  • α sensitivity: Across α from 0.1 to 0.9, all metrics except time vary minimally, motivating the choice α = 0.8.
  • α sensitivity: As α increases, the confidence threshold varies less across steps, matching α’s role in the threshold update.
  • δ sensitivity: δ controls how much step confidence must drop relative to the threshold before SelfStepConf triggers reflection.Larger δ makes the reflection check stricter, while smaller δ tolerates larger confidence drops.
  • δ sensitivity: δ is insensitive across the reported metrics except time, and δ ≤ 0.7 produces zero reflections, so the implementation selects δ = 0.8.
  • NC sensitivity: GMM-filtered voting methods remain similar across NC values, unlike DIS-Top50 at NC = 1, because filtering leaves trajectories with relatively uniform quality.

F.2 Step Split Ablation Study

The ablation compares fixed-window, sentence-level, entropy-level, and paragraph-level step partitioning for SelfStepConf. Paragraph-level splitting is identified as the most suitable option for reasoning tasks, while efficiency varies across splitting methods.

  • Step partitioning: SelfStepConf uses paragraph boundaries marked by “\n\n” to preserve logical integrity during dynamic test-time step adjustment.
  • Step partitioning: High-entropy tokens are treated as potential fork points, but a minimum step length of 200 prevents excessively short steps.The method uses an entropy threshold of 0.672 for confidence checks.
  • Ablation results: Compared with fixed windows, sentence-level splitting, and entropy-level splitting, paragraph-level partitioning is reported as more suitable for SelfStepConf.The comparison considers Avg Pass@1, steps, tokens, confidence, reflections, and time.

G.1 Main Experiments and Ablation Results on More Models

Across additional Qwen3 models and reasoning modes, DistriVoting and SelfStepConf retain their reported advantages, while ablations identify GMM filtering as central and confidence separation as a consistent pattern. Supplementary analyses also show that threshold optima vary by model and benchmark, whereas answer-category distributions approximately follow Gaussian patterns.

  • Main results: DistriVoting consistently outperforms WSCVoting across Qwen3 models, filter strategies, thinking modes, and non-thinking modes.The reported main results use a budget of 128 trajectories and 64 repeats.
  • Ablation results: SelfStepConf consistently improves voting performance over basic inference across the tested ablation settings.The ablation attributes this pattern to SSC inference rather than to a single voting configuration.
  • Ablation results: The GMM Filter is the crucial DistriVoting component, while the Reject Filter provides benefit only after the confidence distribution is effectively split.The ablation compares GMM filtering with a naive Top50 filter and no filtering.
  • Confidence analysis: Across additional models, SelfStepConf produces higher confidence quality than BasicInference under the reported AUROC and WAcc evaluations.The supplementary analysis uses Qwen3-8B, Qwen3-14B, Qwen3-14B-NonThinking, and Qwen3-32B.
  • Threshold sensitivity: The applicable fixed top-threshold varies across models on the same benchmark and across benchmarks for the same model.This variation supports using top50 as a fixed-threshold filter despite the lack of one universal optimum.
  • GMM clustering: Each frequently selected answer’s distribution approximately follows a Gaussian pattern, consistent with the paper’s clustering analysis.The visualization examines the top 30 most frequent answers using DeepSeek-R1-8B on HMMT2025.
Loading 2603.03872v1…