PaperlayerView exact revision on arXiv

Source-linked AI summary

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

Feiyang Li, Shengjing Liu, Qi Zhan, Sijie Cheng, Weiqing Wang, Hongwen Chen, Yuxuan Yang, Wen Wang, Yile Wang, Hui Huang

arXiv:2609.38070v1cs.AI

TL;DR

Probability-based confidence methods can be overconfident, and it is unclear whether selected-token probabilities themselves explain calibration gains. DTC counts strong next-token disagreement along a reasoning trajectory; across mathematical benchmarks, it improves calibration over probability-based, verbalized-confidence, and UQAC baselines.

  • Problem

    Trajectory-level probability aggregates can be overconfident, and UQAC’s product leaves unclear whether calibration gains come from selected-token probabilities or their count.

    Source: §1 · p. 2

  • Method

    DTC counts tokens whose next-token distributions have JSD above a threshold, then maps the count alone or with a full-sequence probability score to path-level confidence.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Across mathematical benchmarks, DTC improves calibration over sequence likelihood, verbalized confidence, and UQAC; DTClin and DTCprod achieve average ECE values of 13.0 and 13.2.

    Source: §5.1 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    Divergent-token count serves as a calibration signal in white-box and black-box settings, and applying it to verbalized scores reduces overconfidence.

    Source: §7 · p. 10

    Black-box calibration assumes the relationship between divergent-token count and path reliability remains reasonably stable across model pairs, using same-family auxiliary models.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertainty of their answers. Current methods for estimating the confidence of large language models are generally based on probabilities of selected key tokens, but the underlying mechanism remains unclear. Our pilot study finds that replacing selected token probabilities with coarse substitutes can also improve calibration, motivating us to further explore effective signals of model confidence. We introduce Divergent Token Confidence (DTC), a framework that estimates confidence by counting tokens at which two models strongly disagree during decoding. DTC identifies these divergent tokens using the Jensen-Shannon divergence between next-token distributions evaluated along the same reasoning trajectory. We find that their count is almost negatively associated with answer accuracy, thereby serving as a simple yet effective signal for uncertainty quantification. DTC supports both white-box and black-box evaluation using auxiliary models, without explicit training and affecting the generation process. Experiments across multiple model families and six mathematical benchmarks demonstrate improved calibration over probability-based and verbalized baselines. Under white-box evaluation, the count-only estimator achieves an average expected calibration error of 13.0%, compared with 32.7%-42.4% for standard full-sequence confidence methods. In black-box settings, it also improves calibration over the original verbalized scores. For example, mean expected calibration error falls from 32.1%-40.2% to 13.7%-16.3% on DeepSeek-V3.2. These findings provide new insights for improving reasoning uncertainty quantification in large language models. The code is released at https://github.com/szu-tera/DTC.git.

1 INTRODUCTION

The introduction questions whether selected-token probabilities drive calibration gains, then motivates divergent-token count as an alternative confidence signal. It presents DTC as a method for estimating reasoning-path confidence in white-box and black-box settings.

1 INTRODUCTION

  • Reasoning confidence can help determine whether to rely on model outputs, and confidence mismatch may amplify unreliability in safety-critical scenarios.Prior work also reports that confidence can be leveraged to improve a model’s reasoning performance (Fu et al., 2026).Source: §1 · pp. 1–2
  • Existing trajectory-level probability aggregates can be overconfident, while UQAC’s product combines selected-token probabilities with selected-token count.This leaves unclear whether calibration gains come from the probabilities or the count.Source: §1 · p. 2
  • Replacing selected probabilities with trajectory-mean probabilities or a shared constant generally improves calibration, suggesting those specific probabilities may not be necessary.The shared-constant replacement makes the score depend only on the number of selected tokens.Source: §1 · p. 2
  • Tokens whose next-token distributions exceed a JSD threshold are counted as divergent, and their count is negatively associated with path accuracy.Figure 1 depicts the selection process and decreasing path accuracy as divergent-token counts rise.Source: §1 · p. 2, Figure 1 · p. 2
  • DTC improves calibration over full-sequence likelihood-based, verbalized-confidence, and UQAC baselines across model families and mathematical benchmarks.It also reduces overconfidence when applied to verbalized scores on existing trajectories.Source: §1 · p. 2

2 RELATED WORK

Prior work estimates confidence using token probabilities or verbalized scores, and another line uses signals from multiple models or model perturbations. This paper focuses on divergent-token count as a bounded score interpretable as confidence and usable for calibration.

Probability-based confidence and verbalized confidence

  • Probability-based methods aggregate token signals, while relevance-aware approaches weight them by token, sentence, or contextual relevance; CoT-UQ and UQAC use reasoning-token relevance.Black-box verbalized methods elicit confidence after a response or alongside reasoning, and coupled elicitation prompts may alter CoT and lower accuracy or yield overconfidence in mathematical reasoning (Wang et al., 2025).Source: §2 · p. 3

Cross-model signals for confidence estimation

  • Cross-model approaches use consensus, consistency, base-model signals, or token entropy, whereas this work targets a bounded confidence score that supports calibration.Prior token-entropy methods aggregate signals from the generating model, perturbed models, or additional models along a CoT trajectory.Source: §2 · p. 3

3 PRELIMINARIES AND PILOT STUDY

This section defines reasoning uncertainty as confidence in the reliability of a realized reasoning path, reviews token-probability confidence scores, and probes why selected-token methods calibrate. The pilot motivates counting cross-model divergent tokens, whose count tracks answer accuracy.

3.1 REASONING UNCERTAINTY AND CALIBRATION

  • Reasoning uncertainty concerns final-answer reliability conditioned on the query and realized reasoning path, operationalized as calibrated confidence in answer correctness.Prediction uncertainty instead concerns answers before conditioning on a realized path.Source: §3.1 · p. 3

3.2 SEQUENCE CONFIDENCE FROM TOKEN PROBABILITIES

  • Full-sequence scores aggregate token probabilities, while UQAC selects answer-related CoT tokens and combines their probabilities; this leaves probability and selected-token count entangled.The section uses ECE to quantify miscalibration, with lower ECE indicating better calibration.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 DO SELECTED-TOKEN PROBABILITIES EXPLAIN UQAC’S CALIBRATION GAINS?

  • Replacing selected-token probabilities generally improves calibration over UQAC, and a shared constant makes the score depend only on selected-token count.The count-as-exponent variant has the lowest ECE across model sizes, while the global-constant variant generally remains better calibrated than UQAC.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4 MOTIVATION: COUNTING DIVERGENT TOKENS

  • Divergent tokens are positions where two models’ next-token distributions disagree beyond a JSD threshold; the count measures strong disagreement along a shared trajectory.The paper uses JSD for symmetry and boundedness and sets the threshold separately for each model family on a small validation set.Source: §3.4 · p. 5
  • For black-box targets, the dual-auxiliary approach compares two external models from the same family and assumes the count–reliability relationship remains reasonably stable across model pairs.Source: §3.4 · p. 5
  • Accuracy decreases nearly monotonically as divergent-token count increases in both single- and dual-auxiliary settings.This supports the count as a signal of reasoning-path reliability.Source: §3.4 · p. 5

4 DTC: DIVERGENT TOKEN CONFIDENCE

DTC converts divergent-token counts into path-level confidence through a count-only estimator or a product estimator that also uses trajectory-mean probability. The count mapping depends only on the number of divergent tokens, while the product estimator distinguishes trajectories sharing a count.

4 DTC: DIVERGENT TOKEN CONFIDENCE

  • DTClin maps divergent-token count directly to confidence, while DTCprod combines the count with trajectory-mean probability.DTCprod uses generator probabilities in white-box settings and the larger auxiliary model on the frozen trajectory in black-box settings.Source: §4 · p. 5
  • DTClin confidence decreases linearly with count and reaches a floor; main experiments use a = 0.95, b = 0.05, and n = 10.Once divergent tokens are selected, this mapping depends only on their count.Source: §4 · p. 5
  • DTCprod uses trajectory-mean probability to give finer-grained confidence to trajectories sharing the same divergent-token count.The method also allows the count to be combined with other overconfident scores.Source: §4 · pp. 5–6

5 EXPERIMENTS

In white-box evaluation, divergent-token estimators improve calibration over probability-based baselines; in black-box evaluation, they lower ECE on default and verbalized-prompt trajectories.

5.1 WHITE-BOX SETTING

  • White-box experiments cover four mathematical benchmarks and eight models from the Qwen2.5, Qwen3, and Gemma3 families.Baselines include standard full-sequence confidence, its refinements, verbalized estimation, and a supervised process-reward reference; thresholds are set using each family’s smallest model and 100 randomly drawn MATH training problems.Source: §5.1 · pp. 6–7

5.2 BLACK-BOX SETTING

  • On DeepSeek-V3.2, DTClin’s mean ECE is 13.7–16.3, compared with 32.1–40.2 for original verbalized scores.Across generators on Default CoT, DTClin achieves mean ECE values of 11.5–12.7; it also lowers ECE for every model–dataset pair on verbalized trajectories.Source: Table 2 · p. 8, §5.2 · p. 9

5.1 WHITE-BOX SETTING

  • Among the refinements, UQACmean has the lowest average ECE (23.6) and highest AUROC (66.7), but its ranking trails standard methods.BaseCal provides limited calibration gains, while UQAC variants reduce ECE more substantially.Source: §5.1 · pp. 7–8
  • 13.0 (DTClin) and 13.2 (DTCprod) average ECEs accompany AUROCs of 75.3 and 80.8, versus standard methods’ 32.7–42.4 ECE range.DTClin uses the divergent-token count alone; DTCprod also uses path-mean probability and attains higher average ranking at similar ECE.Source: §5.1 · pp. 7–8

5.2 BLACK-BOX SETTING

  • Black-box tests use four contest benchmarks and three generators, with Qwen2.5 auxiliary models scoring trajectories without access to generator logits.Verbalized baselines alter the CoT trajectory, so DTC is also applied to those prompted trajectories for comparison.Source: §5.2 · pp. 8–9

6 ANALYSES AND DISCUSSIONS

Adding DTC to verbalized confidence lowers ECE across reported comparisons, with little or sometimes improved AUROC. Figure 5 compares DTClin’s threshold sensitivity across model sizes and benchmarks.

6 ANALYSES AND DISCUSSIONS

  • Figure 5 compares DTClin’s sensitivity to θ across 7B, 14B, and 32B generators on AMC23 and AIME24, marking θ=0.50 as the white-box operating point.Solid lines represent AMC23, dashed lines AIME24, and the auxiliary model is fixed at 1.5B.Source: Figure 5 · p. 9
  • Table 3 scopes a comparison of verbalized scoring against scoring with DTC for DeepSeek-V3.2.Across the cited tables, DTC lowers ECE; AUROC changes little and sometimes improves.Source: Table 3 · p. 9, §6 · p. 9
  • DTC lowers ECE in every reported comparison, often by more than 20 percentage points; on DeepSeek-V3.2, ECE falls by 12.3–35.9 points and AUROC changes by −0.4 to +2.0 points.The authors describe combining verbalized confidence with DTC as a better ECE–AUROC tradeoff.Source: §6 · pp. 9–10, Table 3 · p. 9

7 CONCLUSION

The conclusion presents divergent-token counts as a signal of reasoning-path reliability and confidence. It reports improved calibration from both DTC estimators across mathematical benchmarks, including when generator logits are unavailable.

7 CONCLUSION

  • DTC counts tokens where two models strongly disagree about the next token, using the count to estimate reasoning-path confidence.DTClin uses the count directly; DTCprod combines it with a full-sequence probability score, with larger counts lowering the score.Source: §7 · p. 10
  • On mathematical benchmarks, both DTC estimators improve calibration over sequence likelihood, verbalized confidence, and UQAC with or without generator logits.Source: §7 · p. 10
  • Applying the count to a verbalized score also reduces overconfidence.Source: §7 · p. 10

AI USE STATEMENT

The authors disclose uses of generative AI in implementation, writing, and reference work, while distinguishing those uses from synthetic-data generation and mathematical claims. The appendix describes the scope of its experimental implementation details.

AI USE STATEMENT

  • Generative AI assisted with evaluation and plotting code, translation, manuscript and figure drafting and editing, reference formatting, and readability improvements.The authors state that they reviewed AI-assisted text, citations, and code and checked reported numbers against their evaluation pipeline.Source: AI USE STATEMENT · p. 10

A EXPERIMENTAL IMPLEMENTATION DETAILS

A.1 DATASETS AND MODELS

The appendix details mathematical benchmarks and model pairings for white-box and black-box evaluation, followed by sampling, answer evaluation, and baseline procedures.

A.1 DATASETS AND MODELS

  • White-box experiments use MATH-500, AMC23, AIME24, and AIME25, while black-box experiments use AIME24, AIME25, HMMT25, and HMMT26.White-box generators are paired with smaller same-family auxiliaries; the three black-box generators share an external Qwen2.5 auxiliary pair.Source: §A.1 · p. 15, Table 4 · p. 15

A.2 SAMPLING AND PROMPTING

  • Generation uses a zero-shot step-by-step prompt, and black-box confidence protocols add instructions before scoring the resulting trajectories without regenerating them.Sampling settings and trajectory counts vary by generator and benchmark.Source: §A.2 · pp. 15–16

A.3 ANSWER EXTRACTION AND EVALUATION

  • Confidence is evaluated on the same generated trajectory using teacher forcing, while accuracy includes truncated responses and unextractable answers are excluded from UQ evaluation.Correct and incorrect predictions are subsampled to balance classes for ECE and AUROC, but not for accuracy.Source: §A.3 · p. 16

A.4.1 WHITE-BOX BASELINES

  • White-box baselines include length-normalized sequence likelihood, mean token probability, normalized predictive entropy, and a paired-base-model probability score.Source: §A.4.1 · p. 16

A.4.2 BLACK-BOX BASELINES

  • Black-box verbalized baselines elicit a confidence number, probabilities for two candidate answers, or a distribution over candidate answers.For open-ended questions, the distribution method includes “None of the above”; the reported confidence is the selected answer’s probability.Source: Figure 6 · p. 17, Figure 7 · p. 17, §A.4.2 · p. 18

B COMPLETE RESULTS

The appendix supplies additional white-box and black-box results, verbalized-confidence comparisons, and black-box sensitivity analyses. The supplied passages describe table scope and selected sensitivity findings but do not provide result-table values.

B COMPLETE RESULTS

  • The appendix tables add omitted results; accuracy uses every sampled response, while ECE and AUROC use class-balanced sets.An arrow indicates rescoring the same trajectories, and a dash marks a missing or inapplicable metric.Source: B COMPLETE RESULTS · p. 18

B.1 FULL WHITE-BOX RESULTS

  • The additional white-box results report AUROC for the same single-auxiliary setup as the main-text ECE results.PRM is a supervised reference and is excluded from the unsupervised best- and second-best comparison.Source: §B.1 · p. 18, Table 5 · p. 19

B.2 FULL BLACK-BOX RESULTS

  • The Qwen3-4B and Qwen3-30B-A3B black-box tables report accuracy, ECE, and AUROC across four contest benchmarks, with averages.DTC and PRM rescore frozen Default CoT trajectories; within each confidence protocol, DTC rows use the same trajectories as that protocol.Source: §B.2 · p. 18, Table 6 · p. 20, Table 7 · p. 20

B.3 VERBALIZED CONFIDENCE WITH DTC

C.1 BLACK-BOX AUXILIARY SIZE

  • The lowest ECE on AIME25 is similar across auxiliary-model pairs in the dual-auxiliary DTClin comparison.The search varies θ in steps of 0.05; Figure 9 reports the θ with lowest ECE and its AUROC.Source: §C.1 · p. 21, Figure 9 · p. 21

C.2 BLACK-BOX THRESHOLD CURVES

  • At the black-box operating point θ=0.70, ECE remains low for DTClin on DeepSeek-V3.2 trajectories.Figure 10 varies θ with the 1.5B auxiliary fixed and shows AIME25 and HMMT25 curves.Source: §C.2 · p. 21, Figure 10 · p. 21

C.3 COUNT AND RATIO

Across five appendix checks, divergent-token count remains associated with lower accuracy after length normalization, while DTC’s divergence, threshold, saturation-length, and offset settings are examined for sensitivity. The reported comparisons support JSD, n = 10, and k = 4 under the tested settings.

C.3 COUNT AND RATIO

  • Accuracy falls as divergent-token count rises, and length-normalized ratio also declines, so path length alone does not explain the association.On MATH-500, Qwen2.5-7B accuracy drops from about 0.97–0.99 at count zero to 0.24–0.29 at counts of 11 or more; count and ratio correlations are r = −0.985 and r = −0.937.Source: §C.3 · p. 21, Figure 11 · p. 22

C.4 CHOICE OF DIVERGENCE MEASURE

  • JSD is steadier near the main-experiment threshold than KL, whose required threshold is larger and less stable across models.ECE is U-shaped against θ for all three divergence scores, while AUROC rises where ECE falls.Source: §C.4 · p. 22, Figure 12 · p. 23, Figure 13 · p. 23

C.5 THRESHOLD AND SATURATION SENSITIVITY

  • Although the ECE-minimizing threshold shifts by family, main-text thresholds remain low-ECE, and the comparison across saturation lengths supports n = 10.The white-box comparison uses the smallest Qwen2.5, Qwen3, and Gemma3 generators, averaged over MATH-500, AMC23, AIME24, and AIME25.Source: §C.5 · p. 22, Figure 14 · p. 24

C.6 BLACK-BOX SATURATION LENGTH

  • Black-box DTClin comparisons across three generators and two Qwen2.5 auxiliaries also support saturation length n = 10.The curves are averaged over AIME24, AIME25, and HMMT25.Source: §C.6 · p. 22, Figure 15 · p. 24

C.7 OFFSET k FOR DTCprod

  • DTCprod’s ECE minimum shifts with offset k and model family, but k = 4 still gives low ECE at the main-text thresholds.The white-box comparison covers the smallest generator in each of the Qwen2.5, Qwen3, and Gemma3 families.Source: §C.7 · p. 22, Figure 16 · p. 25
Loading 2609.38070v1…