PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Think with Confidence

Yichao Fu, Xuewei Wang, Yuandong Tian, Jiawei Zhao

arXiv:2508.15260v1cs.LG

TL;DR

Test-time parallel thinking improves LLM reasoning but can incur high computational overhead and diminishing returns. DeepConf filters low-confidence reasoning traces using local confidence signals, improving accuracy and efficiency; online mode reduces token generation by up to 84.7% versus standard parallel thinking.

  • Problem

    Parallel thinking can incur substantial computational overhead and diminishing returns because majority voting treats reasoning traces equally despite quality variation.

  • Method

    DeepConf uses local confidence measurements to filter low-confidence reasoning traces during or after generation in online and offline modes.

  • Results

    Across models and challenging datasets, DeepConf improves reasoning performance and computational efficiency; online mode reduces token generation by up to 84.7% versus standard parallel thinking.

  • Takeaways & Limitations

    DeepConf supports test-time compression as a practical and scalable approach for more efficient LLM reasoning.

  • Takeaways & Limitations

    Global confidence aggregation can obscure intermediate reasoning failures and requires complete traces, preventing early termination of low-quality generations.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown great potential in reasoning tasks through test-time scaling methods like self-consistency with majority voting. However, this approach often leads to diminishing returns in accuracy and high computational overhead. To address these challenges, we introduce Deep Think with Confidence (DeepConf), a simple yet powerful method that enhances both reasoning efficiency and performance at test time. DeepConf leverages model-internal confidence signals to dynamically filter out low-quality reasoning traces during or after generation. It requires no additional model training or hyperparameter tuning and can be seamlessly integrated into existing serving frameworks. We evaluate DeepConf across a variety of reasoning tasks and the latest open-source models, including Qwen 3 and GPT-OSS series. Notably, on challenging benchmarks such as AIME 2025, DeepConf@512 achieves up to 99.9% accuracy and reduces generated tokens by up to 84.7% compared to full parallel thinking.

1 INTRODUCTION

DeepConf addresses the diminishing returns and inefficiency of majority-vote parallel thinking by filtering low-confidence reasoning traces using local confidence signals during or after generation. Across multiple benchmarks and models, it improves or maintains accuracy while substantially reducing token generation, including 99.9% AIME 2025 accuracy and up to 84.7% fewer tokens.

  • Motivation: Majority voting improves reasoning accuracy but incurs substantial computational overhead, and its performance can saturate or degrade as more traces are added.Treating all traces equally allows low-quality reasoning paths to dominate voting.
  • Limitations: Global confidence measures can hide local reasoning failures and require complete traces before filtering, preventing early stopping of low-quality trajectories.Local confidence fluctuations can provide useful signals for estimating trace quality at intermediate reasoning steps.
  • Method: DeepConf combines parallel thinking with local confidence-aware filtering to discard low-confidence traces during or after generation, reducing unnecessary tokens while maintaining or improving accuracy.It supports both offline and online modes without requiring complete trace generation in the online setting.
  • Evaluation: DeepConf is evaluated over AIME 2024/2025, HMMT 2025, BRUMO25, and GPQA-Diamond using DeepSeek-8B, Qwen3-8B/32B, and GPT-OSS-20B/120B across 64 repetitions per setting.The evaluation compares DeepConf with standard majority voting on reasoning performance and generated-token usage.
  • Results: 99.9% accuracy on AIME 2025 with GPT-OSS-120B makes DeepConf@512 outperform cons@512 at 97.0% and pass@1 at 91.8%, while online control cuts tokens by up to 84.7%.The offline result uses no tools, and Fig. 1 highlights these key results.

2 CONFIDENCE AS AN INDICATOR OF REASONING QUALITY

Model-internal token-distribution metrics provide supervision-free signals of reasoning-trace quality, with average trace confidence distinguishing correct from incorrect paths. However, global trace aggregation can hide intermediate failures and requires complete generation before filtering.

  • Confidence as an Indicator of Reasoning Quality: Model-internal token-distribution metrics distinguish high-quality reasoning trajectories from erroneous ones without external supervision, as shown by Kang et al. (2025).
  • Token Entropy: Low token entropy reflects a peaked, high-certainty distribution, whereas high entropy reflects uncertainty in the next-token prediction.
  • Token Confidence: Token confidence is defined from the average log-probability of the top-k tokens, with higher values indicating greater certainty and lower values indicating uncertainty.Here, k denotes the number of top tokens considered.
  • Average Trace Confidence: Higher average trace confidence indicates a greater likelihood that a reasoning path is correct, effectively separating correct from incorrect traces.This trace-level measure, also termed self-certainty, aggregates token-level confidence across generated tokens and is demonstrated in Figure 2 using HMMT25 data.
  • Limitations: Average trace confidence can mask intermediate reasoning failures and cannot terminate low-quality generations early because it requires complete traces.A few high-confidence tokens may obscure numerous low-confidence segments, reducing computational efficiency.

3 DEEP THINK WITH CONFIDENCE

Deep Think with Confidence uses fine-grained confidence signals to estimate reasoning-trace quality and improve performance or efficiency in offline and online thinking. It introduces localized confidence metrics, confidence-aware aggregation and filtering, and adaptive online termination and sampling.

  • Confidence Measurements: Group confidence averages token confidence over overlapping spans, while bottom 10% confidence aggregates the lowest-confidence groups to capture localized reasoning failures.The method uses sliding windows, such as 1024 or 2048 previous tokens, and empirically finds that the bottom 10% captures the most problematic segments.
  • Confidence Measurements: Bottom 10% and tail confidence better separate incorrect from correct traces than mean confidence, making them more effective for trace-quality estimation.Figure 2 compares these confidence measurements and reports the stronger separation; the bottom 10% captures problematic reasoning segments across models and datasets.
  • Offline Thinking: Offline thinking weights majority votes by trace confidence and can filter to the top η = 10% or η = 90% of traces.Top 10% emphasizes the highest-confidence paths, whereas top 90% preserves more diversity and reduces bias by including broader reasoning alternatives.
  • Online Thinking: Online thinking uses lowest group confidence to terminate unpromising traces dynamically, conserving computation when confidence falls below a critical threshold.DeepConf-low and DeepConf-high combine offline warmup with adaptive sampling; their thresholds retain the top η = 10% or η = 90% highest-confidence warmup traces.
  • Online Thinking: Adaptive sampling continues generation until consensus reaches a preset threshold or the fixed trace budget B is exhausted, adjusting trace counts to problem difficulty.Consensus is measured by the majority vote weight relative to total vote weight.

4 EXPERIMENTS

Across five reasoning benchmarks and five open-source LLMs, DeepConf improves accuracy over majority voting while substantially reducing generated tokens through confidence-based filtering and adaptive early stopping. The experiments evaluate offline confidence weighting and online DeepConf variants under a consistent trace-sampling framework.

  • Experimental Setup: The evaluation covers five open-source models from three families and five benchmarks spanning mathematical competitions and graduate-level STEM reasoning.The models are DeepSeek-8B, Qwen3-8B, Qwen3-32B, GPT-OSS-20B, and GPT-OSS-120B; the benchmarks are AIME24, AIME25, BRUMO25, HMMT25, and GPQA.
  • Experimental Setup: Experiments compare Pass@1, unweighted Cons@K, confidence-weighted Measure@K, and filtered Measure+top-η%@K using 4,096-trace pools and 64 independent runs.Offline and online evaluations resample working sets from the common pool, enabling consistent comparisons between voting and adaptive generation.
  • Offline Evaluation: Confidence-aware filtering generally outperforms majority voting, with η=10% producing gains such as 82.3%→87.4% for DeepSeek-8B on AIME25 and 85.3%→90.8% for Qwen3-32B on AIME24.GPT-OSS-120B reaches 99.9% on AIME25, while local and global confidence measures both identify useful high-confidence traces.
  • Offline Evaluation: Lowest Group Confidence filtering with η=10% consistently improves DeepSeek-8B offline accuracy over the best majority-voting result across AIME24, AIME25, BRUMO25, and HMMT25.The method uses the least-confident 2,048-token group in each trace before confidence-weighted voting.
  • Online Evaluation: DeepConf-low reduces generated tokens by 43–79% at K=512 while matching or improving majority-vote accuracy in most settings, although occasional accuracy drops occur.DeepConf-high provides a more conservative 18–59% token reduction; DeepConf-low improves DeepSeek-8B on AIME24 by 5.8% but reduces Qwen3-32B BRUMO25 accuracy by 0.9%.
  • Online Evaluation: DeepConf achieves mean token savings of 62.88% for DeepConf-low and 47.67% for DeepConf-high while maintaining equivalent accuracy to majority voting on DeepSeek-8B.The online results retain the offline pattern that η=10% filtering usually provides the largest accuracy gains, with occasional dataset-specific drops.

5 FUTURE WORK

Future work will extend DeepConf to reinforcement learning, where confidence-based early stopping may guide policy exploration and improve training sample efficiency. It will also address overconfidence in incorrect reasoning paths through stronger confidence calibration and uncertainty quantification.

  • Future directions: DeepConf could be extended to reinforcement learning using confidence-based early stopping to guide policy exploration and improve sample efficiency during training.This direction targets training-time policy exploration rather than only test-time reasoning.
  • Future directions: A key limitation is that models can exhibit high confidence on incorrect reasoning paths.Future work should directly address this observed failure mode.
  • Future directions: More robust confidence calibration techniques and uncertainty quantification are additional priorities for future work.

6 CONCLUSION … B ABLATION STUDY

DeepConf improves reasoning accuracy and computational efficiency across models and challenging datasets, including scales from 8B to 120B parameters. It builds on test-time scaling and confidence estimation by using confidence-aware reasoning to improve accuracy–compute trade-offs beyond conventional aggregation.

  • 6 CONCLUSION: DeepConf significantly improves reasoning performance and computational efficiency in ensemble voting across state-of-the-art models and challenging datasets.The method achieves substantial accuracy improvements while saving tokens, with benefits consistent across models from 8B to 120B parameters.
  • A.1 TEST TIME SCALING: Test-time scaling improves reasoning by allocating more inference computation through longer single trajectories or parallel sampling and aggregation.Representative depth-scaled models include o1, DeepSeek R1, Kimi K1.5, Qwen3, and Grok-4.
  • A.2 EFFICIENT REASONING: Efficient reasoning methods seek better accuracy–compute trade-offs through adaptive sampling and richer aggregation.ESC, RASC, Adaptive-Consistency, Dynamic Voting, and Dynasor reduce sample counts while preserving accuracy.
  • A.3 CONFIDENCE ESTIMATION: Confidence estimation quantifies output reliability using token-level entropy, uncertainty scores, self-certainty, or learned confidence tokens.These approaches provide complementary signals for evaluating model outputs.
  • A.3 CONFIDENCE ESTIMATION: Global confidence can rank or select completed reasoning candidates and outperform majority voting while generating fewer tokens.Global confidence is computed at the sequence level and applied post hoc after candidate completion.
  • A.3 CONFIDENCE ESTIMATION: DeepConf differs from post-hoc global-confidence methods by integrating a lightweight local confidence signal into test-time reasoning.This distinction enables confidence to assess individual traces as part of the reasoning-and-aggregation process.

B.1 ABLATION ON CONSENSUS THRESHOLDS · B.2 ABLATION ON WARMUP SAMPLING SIZE.

The ablations identify τ=0.95 as the best consensus threshold for preserving accuracy while reducing tokens, while larger warmup samples generally stabilize thresholds and improve online accuracy relative to offline results. Warmup effects remain model- and dataset-dependent.

  • B.1 ABLATION ON CONSENSUS THRESHOLDS: The online method checks modal agreement before each new trace and stops sampling when agreement exceeds τ.The baseline uses budget-only stopping without confidence-based early stopping.
  • B.1 ABLATION ON CONSENSUS THRESHOLDS: τ=1.0 weakens token savings without reducing accuracy, making it more conservative than τ=0.95.The ablation evaluates consensus thresholds at voting budgets B∈{32,512}.
  • B.1 ABLATION ON CONSENSUS THRESHOLDS: τ=0.95 preserves accuracy while saving 15.4%/52.8% tokens at B=32/512 for DeepConf-low and 22.0%/54.7% for DeepConf-high.The comparison uses Qwen3-32B on AIME24 against a budget-only baseline after Ninit=16 warmup traces.
  • B.2 ABLATION ON WARMUP SAMPLING SIZE: Warmup sizes Ninit∈{8,16,32} are compared at B=512 under budget-only online DeepConf with DeepConf-low’s top η=10% confidence traces.The comparison is reported across models and datasets.
  • B.2 ABLATION ON WARMUP SAMPLING SIZE: Increasing Ninit stabilizes the empirical confidence distribution used to set threshold s and generally brings online accuracy closer to the offline baseline.This is reflected by smaller |∆Acc| values in practice.
  • B.2 ABLATION ON WARMUP SAMPLING SIZE: The relationship between warmup size, threshold, and accuracy depends on the model and dataset.Therefore, larger Ninit does not imply a uniformly better threshold-accuracy outcome.

B.3 ABLATION ON FILTERING PERCENT. · B.4 ABLATION ON CONFIDENCE METRICS

The ablations show that more aggressive filtering often improves accuracy, although the best retention threshold depends on the dataset and model. Confidence metrics also matter: head-based signals are weak or harmful, while tail- and mean-based signals frequently improve results, including 89.6% on DeepSeek-8B AIME25.

  • B.3 ABLATION ON FILTERING PERCENT.: The filtering ablation sweeps retention from the top 90% to the top 10% using Lowest Group Confidence with group size 2,048 and evaluates voting sizes B ∈{1, . . . , 512}.Best accuracy is reported for each model–dataset pair in the offline setting.
  • B.3 ABLATION ON FILTERING PERCENT.: Table 4 examines online warm-up size Ninit at fixed B = 512 for DEEPCONF-Low, comparing online accuracy and token usage with the offline baseline.The bolded warm-up size is the one whose online accuracy is closest to the offline baseline.
  • B.3 ABLATION ON FILTERING PERCENT.: More aggressive filtering generally improves accuracy across DeepSeek-8B, Qwen3-8B, and Qwen3-32B, although the optimal retention percentage varies by dataset.Top 10% is frequently best, while top 25% or top 50% can be optimal for some model–dataset combinations.
  • B.4 ABLATION ON CONFIDENCE METRICS: At voting size B = 512, the confidence-metric ablation compares majority voting, mean confidence, and head-, tail-, and mean-based filtering or weighting across model–dataset pairs.The aggregation rules differ in per-trace confidence computation and whether filtering occurs before voting; Mean@10% selects the top 10% before confidence-weighted voting, while Tail(2k)@90% retains the top 90% by tail confidence.
  • B.4 ABLATION ON CONFIDENCE METRICS: Head-based confidence typically matches majority voting and can reduce accuracy, whereas tail- and mean-based signals frequently yield gains.Early tokens often reflect setup, paraphrase, and exploratory planning rather than discriminative information for final correctness.
  • B.4 ABLATION ON CONFIDENCE METRICS: 89.6% accuracy is reached by a tail variant on DeepSeek-8B AIME25, demonstrating a strong result with an 8B model.The supplied passage also identifies GPT-OSS-120B AIME25 as a notable tail-variant case, but does not provide its value.

C SCALING BEHAVIOR FOR OFFLINE DEEPCONF

Offline DeepConf combines confidence filtering with confidence-weighted majority voting across 20 model–dataset settings and voting sizes up to 512. Top 90% is more stable, while Top 10% achieves higher average gains but can regress when confidence is incorrectly concentrated.

  • Scaling behavior: The offline evaluation spans 5 model configurations, 4 datasets, and voting sizes B ∈{1, . . . , 512}, using Lowest Group Confidence with two retention settings.The datasets are AIME24, AIME25, BRUMO25, and HMMT25; GPQA-Diamond results are additionally reported in Appendix E.
  • Scaling behavior: Top 90% matches or slightly outperforms unweighted majority voting consistently, with low variance from −0.21% to +0.73% and average improvement of +0.17%.This makes Top 90% the safer choice when stability is paramount.
  • Scaling behavior: Top 10% improves performance in 12/20 settings by +0.26% to +9.38%, but drops in eight settings by −4.69% to −0.31%.The regressions occur when confidence concentrates on an incorrect answer, producing “confidently wrong” traces.
  • Scaling behavior: +11.62% average improvement over single-sample voting is achieved by Top 10%, compared with +10.57% for Top 90%, across diverse models and mathematical reasoning benchmarks.Top 10% gains range from +5.26% to +20.94%, while Top 90% gains range from +5.83% to +16.88%.

D SCALING BEHAVIOR FOR ONLINE DEEPCONF

Online DeepConf improves the token–accuracy trade-off across models and datasets, with DeepConf-low delivering the largest savings and DeepConf-high providing a more conservative alternative. Adaptive early termination outperforms budget-only DeepConf at matched voting ensemble sizes.

  • Online setting: Online evaluation warms up 16 traces to calibrate τ, stops traces when current group confidence falls below τ, and aggregates completed traces with confidence-weighted majority voting.DeepConf-low uses the 90th-percentile threshold and DeepConf-high uses the 10th-percentile threshold.
  • At matched budget: 43–84% fewer tokens: DeepConf-low usually matches or improves majority-voting accuracy at voting size 512, while DeepConf-high saves roughly 16–59%.Examples include DeepSeek-8B/AIME24 with −77.9% tokens and +5.8 pp, and Qwen3-32B/AIME24 with −66.8% tokens and +4.7 pp.
  • At Comparable Accuracy: 19–96% token reduction: adaptive DeepConf-low typically maintains matched majority-voting accuracy, while DeepConf-high achieves 13–84% savings with near-equivalent performance.Some model–dataset cases between B ∈[32, 512] do not achieve matched accuracy with reduced token consumption.
  • At Comparable Accuracy: Adaptive DeepConf consistently dominates budget-only DeepConf on the token–accuracy Pareto frontier at identical voting ensemble sizes.For DeepSeek-8B/AIME24 at 512 votes, Low uses 0.782 × 108 versus 1.512 × 108 tokens.

E GPQA-DIAMOND RESULTS

On GPQA-Diamond, DeepConf offline methods match or exceed majority voting at peak accuracy, while online adaptive policies reduce token usage more than fixed policies at the same voting budget. DeepConf-high generally preserves majority-voting accuracy, whereas DeepConf-low seeks greater savings with possible underperformance on DeepSeek-8B.

  • Offline results: DeepConf matches or exceeds majority voting in peak accuracy on GPQA-Diamond across DeepSeek-8B, Qwen3-8B, and Qwen3-32B.These offline results use Lowest Group Confidence and are shown in Fig. 10.
  • Offline results: Keeping Top-10% outperforms majority voting on Qwen3-8B and Qwen3-32B, while Keeping Top-90% roughly matches it on DeepSeek-8B.On DeepSeek-8B, Keeping Top-10% performs slightly lower than the baseline.
  • Online results: Adaptive policies consistently reduce token usage more than the fixed method at the same voting budget in online GPQA-Diamond evaluation.The online scaling results are shown in Fig. 11.
  • Online results: DeepConf-high generally maintains majority-voting accuracy, whereas DeepConf-low pursues larger computational savings but may underperform on DeepSeek-8B.These online findings align with the offline results.

F EXPERIMENTAL SETTINGS … G.6 ADDITIONAL NOTES

The paper specifies reproducible decoding and DeepConf configurations, then describes a minimal vLLM integration that enables confidence-based early stopping through the compatible API. The implementation maintains windowed confidence in logprob processing, checks it during decoding, and records triggered stops for traceability.

  • F EXPERIMENTAL SETTINGS: Experiments fix model-specific temperature, top-p, top-k, and maximum generation length while using each model’s native tokenizer and recorded decoding settings.For Qwen3 and GPT-OSS, prompts request step-by-step reasoning with boxed answers; GPT-OSS also uses its official system prompt and high reasoning effort, while DeepSeek-8B uses its official system prompt.
  • F EXPERIMENTAL SETTINGS: Generation terminates at an end-of-sequence token or maximum length, with final answers extracted from \boxed{...}.This defines the post-processing and stopping conditions used across the experiments.
  • F EXPERIMENTAL SETTINGS: DeepConf uses two online configurations—low (η=10%) and high (η=90%)—with Ninit, η, τ, and voting budget B specified in Table 12.Ninit is the offline-warmup trace count, η retains top traces for Ttop, τ is the consensus threshold, and B is the maximum trace budget.
  • G.1 ENVIRONMENT AND COMMIT: The implementation uses vLLM commit 31f09c615f4f067dba765ce5fe7d00d880212a6d with Python 3.12.0 and CUDA 12.8.DEEPCONF is implemented with minimal changes to the serving framework.
  • G.2 WHAT CHANGED (HIGH LEVEL): Only two vLLM locations change: LogprobsProcessor maintains sliding-window confidence and exposes check_conf_stop(), while output_processor.py performs one early-stop check before RequestOutput construction.The edits add confidence state, initialization, updating during sampling, and invocation in the decode loop.
  • G.3 HOW TO ENABLE (OPENAI-COMPATIBLE API): The feature is enabled per request through extra_body["vllm_xargs"], forwarded to SamplingParams.extra_args, and requires logprobs=True with top_logprobs>=2.window_size controls the confidence window, threshold sets the cutoff, and top_k=0 optionally disables top-k truncation.
  • EXACT EDITS (COPY-PASTE GUIDANCE): The copy-paste guidance adds imports, dataclass fields, request initialization, a stop-check helper, confidence updates, and a decode-loop invocation without changing subsequent RequestOutput construction.The recommended patch is organized across logprobs.py and output_processor.py, with the latter retaining its downstream logic.
  • G.6 ADDITIONAL NOTES: Confidence is the moving average of negative mean candidate logprobs over a fixed window; when triggered, decoding stops with FinishReason.STOP and a traceable <gconf<THR>> annotation.The stop check is invoked in the output processor’s decode loop after confidence updates.
Loading 2508.15260v1…