Source-linked AI summary
Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
Wei-Lin Chen, Liqian Peng, Tian Tan, Chao Zhao, Blake JianHang Chen, Ziqian Lin, Alec Go, Yu Meng
TL;DR
Longer Chain-of-Thought traces are unreliable indicators of reasoning quality because they may reflect overthinking rather than effective computation. The paper measures effort through depth-wise prediction revision, evaluates deep-thinking ratio across reasoning benchmarks and models, and uses it to select promising samples more efficiently. Deep-thinking ratio correlates positively with accuracy and supports Think@n, which matches or exceeds standard self-consistency while reducing inference cost.
Problem
Token counts do not consistently indicate reasoning quality because longer reasoning can reflect overthinking and performance degradation.
Method
The paper defines deep-thinking ratio by identifying tokens whose predictive distributions undergo sustained revision in deeper layers before converging.
Results
Across four reasoning benchmarks and multiple model families, deep-thinking ratio has the strongest and most stable relationship with task performance, with average correlation r=0.687.
Takeaways & Limitations
Think@n uses short-prefix deep-thinking ratios to select and aggregate samples, matching or surpassing standard self-consistency at approximately half the inference cost.
Takeaways & Limitations
Overly restrictive settling thresholds and depth fractions can make positive correlation trends slightly more unstable by filtering informative high-computation tokens.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.
1. Introduction
The paper argues that token count is an unreliable proxy for reasoning quality because longer reasoning can reflect overthinking, and proposes measuring internal depth-wise prediction revision instead. Across reasoning benchmarks and models, deep-thinking ratio supports accuracy assessment and enables more efficient test-time scaling.
- Motivation: Longer reasoning traces do not consistently improve accuracy and can reflect overthinking that degrades performance.Prior evidence includes inverted-U relationships and inverse scaling between Chain-of-Thought length and performance.
- Core idea: Deep-thinking ratio measures inference-time effort through tokens whose predictions undergo sustained revision in deeper layers before converging.This contrasts with surface-level measures such as output length.
- Evidence: Across four mathematical and scientific benchmarks and GPT-OSS, DeepSeek-R1, and Qwen3 models, deep-thinking tokens strongly correlate with task accuracy.The benchmarks are AIME 2024, AIME 2025, HMMT 2025, and GPQA.
- Evidence: Deep-thinking ratio substantially outperforms length-based and confidence-based baselines as an indicator of reasoning success.The paper compares DTR with both types of commonly used proxies.
- Application: Think@n prioritizes samples with high deep-thinking ratios and can match or exceed standard self-consistency while reducing inference costs through early rejection.Unpromising generations are rejected using short prefixes.
2. Measuring Deep-Thinking Ratio
The method identifies tokens that require deeper-layer prediction revision before stabilizing, then aggregates these token-level decisions into a sequence-level deep-thinking ratio. It uses JSD to compare intermediate and final-layer distributions and classifies tokens by their settling depth.
- Preliminaries: Intermediate hidden states are projected into vocabulary space with the language-modeling head to obtain predictive distributions at each layer.The final-layer hidden state is projected using the unembedding matrix, and the same approach is applied to intermediate states.
- Token-level measurement: At each generation step, Jensen–Shannon divergence compares an intermediate-layer distribution with the final-layer distribution.JSD is used because it is symmetric and bounded.
- Token-level measurement: Later convergence toward zero indicates prolonged distributional revision, whereas early convergence indicates fewer subsequent updates.The trajectory of divergence across layers operationalizes the distinction between more and less internal thinking.
- Token classification: A token’s settling depth is the first layer where its divergence falls below a fixed threshold, and the deep-thinking regime is defined by a depth fraction.The method classifies a token as deep-thinking when its settling depth lies in that regime.
- Sequence-level measure: Deep-thinking ratio is the proportion of sequence tokens that settle in the deep-thinking regime.A higher DTR means a larger fraction of tokens undergo extended computation before stabilization, without implying that early-settling tokens are suboptimal.
3. Deep-Thinking Ratio Reflects Task Accuracy More Reliably
Across eight model variants and four reasoning benchmarks, DTR provides a more stable and accurate-associated measure of inference-time effort than token-count and confidence-based proxies. Its relationship with accuracy remains positive across parameter settings, though overly restrictive criteria can reduce stability.
- Evaluation setup: DTR is evaluated across eight reasoning-model variants and four benchmarks using Pearson correlations between quantile-binned effort scores and accuracy.The evaluation compares DTR with length-based and confidence-based measures under a shared ranking-and-binning protocol.
- Length-based measures: Token count has mean correlation r = −0.59 with accuracy, indicating that longer generations are associated with lower performance in these evaluations.Reverse token count has the same magnitude with the opposite sign, but serves only as a post hoc statistical adjustment.
- Confidence-based measures: Confidence measures achieve mean correlations r = 0.219 ∼ 0.605, but their associations vary from strong positive to weak or negative across models and benchmarks.The evaluated confidence measures include log probability, negative perplexity, negative entropy, and self-certainty.
- DTR results: DTR reaches the highest average correlation, r = 0.687, outperforming reverse token count and Self-Certainty across the evaluated settings.DTR remains positive in 30 of 32 model–benchmark settings, with only two negative values.
- Hyper-parameter effects: Higher settling thresholds or lower depth fractions reduce DTR, while depth-fraction changes preserve positive slopes and settling threshold has the larger effect on correlation.The tested configuration (g, ρ) = (0.5, 0.85) balances stability and correlation; overly restrictive settings can make trends less stable.
4. Deep-Thinking Tokens Enable Efficient Test-Time Scaling
Repeated sampling enables test-time scaling, but selecting samples by deep-thinking ratio can preserve accuracy while reducing inference cost through early stopping. Think@n achieves the strongest accuracy-cost trade-off across the evaluated methods.
- Method: Think@n uses DTR to preferentially select and aggregate higher-quality samples from repeated generations.The protocol compares Think@n with self-consistency, mean, length-based, and confidence-based aggregation methods.
- Evaluation: Early stopping is explicitly included when measuring the inference cost of Short@n, Self-Certainty@n, and Think@n.Cost is reported as the average number of generated tokens per problem.
- Prefix ablation: 50-token prefixes achieve higher Think@n accuracy than longer prefixes and match full-sequence performance while reducing inference cost.The result is reported in the AIME 2025 prefix-length ablation and motivates fixing l_prefix = 50 in the main experiments.
- Results: Think@n matches or exceeds Cons@n accuracy while requiring approximately half the inference cost.The averaged results identify Think@n as having the best overall Pareto-optimal accuracy-cost trade-off.
- Results: Length-based selection performs poorly: Short@n modestly reduces cost but underperforms Cons@n, while Long@n performs worse without saving cost.These results support DTR as a more informative selection signal than raw sequence length.
5. Related Work
Related work challenges the assumption that longer Chain-of-Thought always improves reasoning and investigates how internal layer representations can guide prediction and analysis. This paper builds on both lines by using internal depth-wise token prediction changes rather than surface-level length alone.
- Relationship between CoT Length and Performance: CoT length can follow an inverted-U relationship with accuracy, with performance deteriorating beyond an optimal length.The related work attributes deterioration to factors such as error accumulation.
- Relationship between CoT Length and Performance: Prior methods exploit conciseness through shortest-chain voting, first-completed-trace selection, or rejection of longer responses during training.The paper differs by proposing a mechanistic internal signal instead of relying solely on brevity heuristics.
- Leveraging Internal Information in LLMs: Intermediate language-model representations can encode meaningful information before the final layer and can be projected into vocabulary space for token prediction.Prior analyses describe autoregressive transformers as iteratively refining coarse next-token guesses across layers.
- Leveraging Internal Information in LLMs: Layer-specific transformations and comparisons between higher and lower layers have been used to improve interpretability and amplify factual signals.These approaches connect internal depth structure with more interpretable predictions and factuality.
- Leveraging Internal Information in LLMs: Interpretability studies suggest early layers favor generic guesses, whereas later layers perform contextual or fine-grained refinement.This depth-wise view motivates examining how token predictions evolve before final convergence.
6. Conclusion
The paper introduces deep-thinking ratio as an internal measure of inference-time reasoning effort and uses it in Think@n for efficient sample selection. Its results support shifting attention from reasoning length to how deeply token predictions are revised.
- Conclusion: DTR measures inference-time reasoning effort by tracking whether token predictions stabilize or undergo sustained revision across model layers.The measure is presented as a more reliable signal than surface-level token length or confidence.
- Conclusion: Think@n uses DTR for early selection and aggregation, achieving comparable or better performance than standard self-consistency while substantially reducing inference cost.The strategy prioritizes internally deeper-thinking samples rather than simply longer reasoning traces.
- Conclusion: The findings suggest that effective reasoning may be characterized more by internal computation than by the number of generated tokens.The conclusion identifies deeper, more computationally intensive reasoning as a direction for future work.
A. Comparison of Different Distance Metrics for DTR
The method compares JSD, KLD, and cosine-based measures for identifying deep-thinking tokens, with JSD producing the strongest and most consistent accuracy correlations on AIME 25 and HMMT 25.
- A. Comparison of Different Distance Metrics for DTR: JSD, KLD, and cosine similarity provide alternative distance metrics for computing deep-thinking ratio (DTR).KLD replaces JSD for distributional divergence, while cosine similarity compares intermediate- and final-layer hidden states directly.
- A. Comparison of Different Distance Metrics for DTR: JSD-based DTR achieves the strongest positive accuracy correlations on AIME 25 (r=0.869) and HMMT 25 (r=0.895).These results support using JSD in the paper’s DTR definition.
- A. Comparison of Different Distance Metrics for DTR: Cosine-based DTR shows weaker, unstable correlations on AIME 25 (r=0.633) and HMMT 25 (r=0.172).The HMMT 25 association is especially weak compared with the JSD result.
- A. Comparison of Different Distance Metrics for DTR: KLD can become artificially small because high-entropy early predictions assign probability mass to tokens later driven near zero.This asymmetric, numerically sensitive behavior may explain KLD’s inconsistent correlations.
B. DTR Under Different GPT-OSS Reasoning Levels
Across GPT-OSS-120B reasoning configurations, lower reasoning levels produce higher DTR, whereas higher levels achieve better accuracy with smaller DTR. Because DTR averages over sequence tokens, longer reasoning chains can reduce its value and limit cross-mode comparability.
- B. DTR Under Different GPT-OSS Reasoning Levels: Lower GPT-OSS-120B reasoning levels exhibit higher DTR, while higher levels yield smaller DTR and better task accuracy.This pattern appears on both AIME 25 and GPQA-D despite identical model weights and differing system prompts.
- B. DTR Under Different GPT-OSS Reasoning Levels: Higher reasoning levels may redistribute computation from depth to sequence length, using longer reasoning chains with less deep revision per token.The passage presents this as a potential explanation for the observed DTR pattern.
- B. DTR Under Different GPT-OSS Reasoning Levels: Longer sequences reduce DTR because the total generated-token denominator increases, limiting direct comparability across models or model modes.DTR is defined as the proportion of deep-thinking tokens averaged over all generated tokens.
C. Additional Analysis of Think@𝒏
Think@n benefits from larger candidate pools and can increasingly outperform self-consistency as the sample count grows. Its performance peaks when retaining the top 50% of samples, while retaining too few or too many degrades accuracy.
- C. Additional Analysis of Think@𝒏: Think@n improves monotonically as the number of sampled responses increases from n∈{16, 32, 48}.Its advantage over Cons@n becomes more pronounced at larger sample counts.
- C. Additional Analysis of Think@𝒏: Larger candidate pools enable Think@n to preferentially select better samples and achieve stronger gains over Cons@n.More samples make the correct answer cluster larger and more likely to appear.
- C. Additional Analysis of Think@𝒏: Performance peaks at η=50% when Think@n retains the top-scoring samples for voting.The evaluated retention levels are η∈{25%, 50%, 75%}.
- C. Additional Analysis of Think@𝒏: Retaining too few or too many samples reduces accuracy by weakening voting robustness or admitting lower-quality samples.The results support η=50% as a stable operating point.
D. Prompts
The paper provides the inference prompts used for its mathematical and scientific reasoning evaluations, including separate prompts for math tasks and GPQA.
- D. Prompts: Table 4 contains the inference prompt used for AIME 2024, AIME 2025, and HMMT 2025.These are the paper’s listed math-task prompts.
- D. Prompts: Table 5 contains the inference prompt used for GPQA.The GPQA prompt is provided separately from the math-task prompts.
E. Qualitative Examples
The qualitative example contrasts an incorrect and a correct OSS-120B-medium solution to an AIME 2025 problem. The incorrect solution is much longer and has lower DTR, while the correct solution is shorter and has higher DTR.
- The example presents an AIME 2025 question with its ground-truth answer and two OSS-120B-medium outputs.
- 27,724 tokens and DTR 13.9 characterize the incorrect output, compared with 3,725 tokens and DTR 19.0 for the correct output.
- The incorrect reasoning explores coordinate geometry, considers alternative points, and leaves the rectangle orientation unresolved.