PaperlayerView exact revision on arXiv

Source-linked AI summary

Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking

Chia-Hsuan Lee, Sihui Dai, Mingyang Zhou, Isha Slavin, Hsuan Su, Shi-Xiong Zhang, Sambit Sahu, William Campbell

arXiv:2607.00482v2cs.CL

TL;DR

Overthinking can make reasoning traces longer without improving answers, while identifying helpful versus harmful reflection is costly. DASH uses intermediate answer checkpoints for segment-level credit, achieving the best accuracy on challenging math benchmarks at 59.45%.

  • Problem

    Reasoning models overthink through behaviors such as hedging, re-verifying, and switching approaches, while identifying helpful self-reflection requires costly step-wise labels.

  • Method

    DASH compares intermediate answer checkpoints with ground truth and assigns segment-level advantages to reasoning that moves toward or away from correctness.

  • Results

    59.45% average accuracy: DASH outperforms GRPO at 56.95%, Dr.GRPO at 58.13%, and the base model at 55.65% on challenging math evaluation.

  • Takeaways & Limitations

    Efficient reasoning training can target when reflection stops helping rather than universally shortening responses, using answer checkpoints without process labels or external judges.

  • Takeaways & Limitations

    DASH requires extractable intermediate answers, limiting applicability to domains with verifiable checkpoints, and its evaluation covers only mathematical reasoning.

Abstract

from arXiv · show

Reasoning language models frequently overthink: generating extended chains of behaviors such as hedging, approach abandonment, and self contradiction that consume tokens without improving answers. We show that these behaviors are not merely a consequence of length; even when controlling for response length, incorrect traces exhibit higher rates of unproductive self-reflection than correct ones. Addressing this requires identifying where self-reflection helps vs hurts, but obtaining these step-level annotations is costly. We observe that intermediate answer commitments within reasoning traces can provide a cheap proxy: by comparing each final answer candidate in the trace to the ground truth, we can determine whether subsequent reflection is productive without any additional supervision. Building on this insight, we propose DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness. On competition-level math benchmarks, DASH achieves the highest accuracy where overthinking is prevalent (Average Accuracy: 59.45% vs. 58.1% Dr.GRPO vs. 56.95% GRPO) while reducing overthinking behaviors and achieving more productive self-correction than baselines.

1 Introduction

Reasoning models can overthink through unproductive behaviors that sometimes worsen answers, while identifying helpful self-reflection step by step is costly. DASH uses intermediate answer checkpoints to assign segment-level credit and improves accuracy and productive self-correction.

  • Longer reasoning can hurt: hedging, re-verifying, and switching approaches may lead models to incorrect final answers.These behaviors motivate retaining productive self-reflection while suppressing unproductive self-reflection.
  • Step-wise labels for helpful self-reflection require process reward models, LLM-as-a-judge, or manual annotation, motivating a cheaper alternative.The paper identifies step-level supervision cost as a significant challenge.
  • Intermediate answer commitments enable supervision-free assessment by comparing each checkpoint with ground truth to determine whether later reflection improved or degraded the answer.Models may explicitly state or box an intermediate result before continuing to reason.
  • DASH divides traces into segments between consecutive answer checkpoints and assigns advantages according to whether each segment moves toward or away from correctness.This converts traces drifting from correct to incorrect intermediate answers into informative training examples rather than wasted negatives.
  • 0.92 vs. 0.47/trace: DASH’s correct traces show twice as many contradiction-then-resolution patterns as GRPO’s, alongside fewer blind approach abandonments.These behavioral metrics support more productive self-correction rather than superficial accuracy gains.
  • 59.45% average accuracy: DASH outperforms GRPO at 56.95%, Dr.GRPO at 58.13%, and the base model at 55.65% where drift is severe.The comparison spans four competition-level math benchmarks and the challenging suite with severe drift prevalence.

2 Analyzing Overthinking in Reasoning Traces

The section argues that response length alone cannot characterize overthinking, motivating linguistic signals that distinguish unproductive reasoning behaviors. Analysis shows that incorrect traces contain more unproductive self-reflection than correct traces even at matched lengths, motivating selective suppression of harmful reflection.

  • Limitations of Length-Based Analysis: Length alone misses the specific overthinking behaviors exhibited by reasoning models.Prior efficiency work primarily treats longer traces as less efficient and applies length penalties or early stopping.
  • Linguistic Signals: Four regex- or n-gram-based signals measure repetition, hedging, abandonment, and contradiction without learned components.These signals target circular reasoning, uncertainty markers, strategy switches, and self-contradiction markers.
  • Linguistic Signals: 4.1–4.3× more common: abandonment is the strongest individual failure predictor across all tested models.The signal counts explicit strategy switches such as declaring an approach wrong and trying another way.
  • Empirical Analysis: Accuracy decreases monotonically with response length, indicating that additional computation in longer traces is largely unproductive.Figure 2 analyzes Nemotron-4B traces on AIME while grouping responses by length.
  • Empirical Analysis: Even within the same length bucket, incorrect traces show higher rates of numerical recomputation and self-contradiction than correct traces.The gap is largest for numerical recomputation, whose density is consistently elevated in incorrect traces across all buckets; self-contradiction shows a more moderate trend.
  • Motivation: These findings motivate discouraging self-reflection that leads to incorrect answers while preserving reflection that aids error correction.The proposed direction aims to improve reasoning efficiency and accuracy without simply truncating traces.

3 Drift-Aware Advantage Shaping

DASH identifies answer drift by tracking intermediate answer commitments and assigns segment-level credit according to whether reflection moves toward or away from correctness. It preserves useful self-correction while penalizing harmful continuation, redundant confirmation, and post-correction drift.

  • Drift Detection: Answer drift occurs when a trace reaches the ground-truth answer at an intermediate commitment but ends with an incorrect final answer.Intermediate commitments are extracted by matching boxed answers, explicit answer statements, and natural-language answer commitments.
  • Segment-Level Credit: DASH divides rollouts into neutral, positive, and negative segments based on whether each checkpointed answer is correct or incorrect.Positive segments receive credit for reaching correct answers, while negative segments represent shifts to incorrect candidates.
  • Overthinking Control: The method reduces overthinking by geometrically discounting repeated correct confirmations and escalating penalties for continuing through negative segments after a correct answer.The first correct arrival receives full credit; redundant confirmations receive diminishing credit, while negative-segment penalties are capped to prevent gradient explosion.
  • Conditional Treatment: Pre-answer reasoning in drift traces receives a weak positive signal rather than being penalized, because the failure occurred after producing a correct answer.Correct traces receive the standard advantage, drift traces receive a weak positive signal, and purely incorrect traces receive no gradient on the pre-answer segment.
  • Reward Shaping: Drift traces receive partial-credit rewards that rank them above purely incorrect traces but below correct traces before advantage computation.The shaped reward decreases as the post-drift portion of the trace grows.

4 Experimental Results

DASH achieves the strongest performance on most challenging math benchmarks, avoids GRPO’s AIME25 regression, and improves self-correction. Its gains persist across reasoning models from different families, indicating that segment-level credit assignment generalizes beyond one backbone.

  • Experimental setup: Evaluation covers OlympiadBench, MinervaMath, AIME 2024, and AIME 2025, using avg@32 for AIME benchmarks and pass@1 for the other two.The experiments primarily use Llama-3.1-Nemotron-Nano-4B-v1.1 and additionally train on Phi-4-reasoning-plus and OLMO-3-think.
  • Main benchmark results: DASH achieves the highest accuracy on 3 of 4 challenging math benchmarks, including OlympiadBench, MinervaMath, and AIME24.The result is especially notable on complex benchmarks where overthinking is severe.
  • Main benchmark results: +4.7 AIME25 accuracy over base lets DASH avoid GRPO’s −0.7 regression, which can penalize valid reasoning prefixes after drift.GRPO improves over base by +6.4 on OlympiadBench and +1.3 on AIME24 but degrades on AIME25.
  • Self-correction: DASH often achieves stronger self-correction than GRPO and Dr.GRPO, while the Brevity Bonus sacrifices AIME accuracy because of limited self-correction.Self-correction is measured as recovery from a wrong intermediate answer to a correct final outcome.
  • Generalization across model families: +0.5, +1.5, and +4.1 average-accuracy gains over vanilla GRPO occur on Nemotron-4B, OLMO-3-think-SFT-7B, and Phi-4-reasoning-plus-14B, respectively.The models differ in pretraining data, scale, and tuning recipe.

5 Analysis

DASH’s components are complementary, with reward shaping and the length penalty driving much of its improvement. Compared with DR-GRPO, it reduces unproductive overthinking while using contradiction as productive self-monitoring and error correction.

  • Component ablations: The full DASH method reaches the best average of 57.5, while every component ablation performs worse, indicating complementary components.Setting drift partial-credit δ to zero reduces the average to 54.6 (−2.9), the largest decline.
  • Component ablations: Reward shaping and the length penalty drive the bulk of DASH’s gains, primarily improving reasoning efficiency and stability rather than raw accuracy.
  • Overthinking signal profile: 59.4% vs. 58.1% DR-GRPO avg@32: DASH achieves the highest accuracy with lower abandonment, hedging density, and length-outlier rate at nearly identical response lengths.DASH records abandonment of 1.07 vs. 1.80 DR-GRPO (−41%), hedging density of 0.53 vs. 0.61 (−14%), and length-outlier rate of 0.004 vs. 0.004 (−17%), with response lengths differing by +0.03%.
  • Contradiction as self-monitoring: Contradiction rises to 0.96 vs. 0.85 DR-GRPO (+13%), reflecting increased self-monitoring that usually co-occurs with accuracy gains and successful error resolution.On Minerva Math, DASH instead has fewer contradictions and higher accuracy; DR-GRPO more often responds to confusion with hedging and abandonment.
  • Answer drift case studies: In two AIME 2025 cases, the base model found correct intermediate solutions but drifted into alternatives, exhausted its token limit, and produced no final answer.The drift-hybrid model stabilized after a single validation in both cases.

6 Related Work

Prior work documents excessive reasoning in LLMs, including over-allocation of computation and diminishing benefits from continued reasoning. This work instead trains models to recognize when to stop within a reasoning trace, motivated by self-doubt after correct answers.

  • Chen et al. (2024) named overthinking in o1-like models and showed that models over-allocate compute to simple problems.
  • Su et al. (2025) found a U-shaped relationship between reasoning length and accuracy, while Wei et al. (2025) formalized that continued reasoning almost never improves answers after peak correctness probability.
  • Rather than relying on continued reasoning to self-correct, this work trains models to recognize when to stop within a trace, tracing the mechanism to self-doubt after correct answers.

7 Conclusion

DASH reduces overthinking by assigning segment-level credit from answer checkpoints, rewarding reasoning that reaches correctness and penalizing reflection that drifts away. It targets abandonment, contradiction, repetition, and excessive length while making remaining contradictions more productive.

  • DASH uses answer checkpoints as free supervision to reward segments leading to correct answers and penalize segments leading away from them.A trace that reaches a correct answer and later diverges reveals both a useful reasoning prefix and a harmful reflective suffix.
  • Overthinking is not merely response length: comparable-length incorrect traces contain more abandonment and unresolved contradiction.
  • On competition math benchmarks, DASH achieves the best AIME25 accuracy while reducing repetition, abandonment, and length outliers.
  • Efficient reasoning training should teach when reflection stops being useful rather than making models universally shorter.Checkpoint-based segment credit provides this approach without process labels or external judges.

Limitations

DASH’s evaluation and applicability are constrained by model scale, the need for extractable intermediate answers, and a mathematics-focused evaluation. The method also entails accuracy trade-offs on some easy benchmarks and inherits the base model’s safety limitations.

  • Most experiments use a 4B model, so training dynamics may differ at larger scales despite apparently scale-invariant drift patterns.The limitation concerns extrapolating results to larger models.
  • DASH requires extractable intermediate answers for drift detection, limiting applicability to domains with verifiable checkpoints.Mathematics and code with test cases are identified as applicable domains, while open-ended reasoning would need alternative drift indicators.
  • −1.7pp vs. base is the Nemotron accuracy decrease on MATH-500, creating a trade-off against gains on hard problems.The passage characterizes this decrease on an easy benchmark as a real trade-off for practitioners.
  • The evaluation is limited to mathematical reasoning, leaving generalization to other reasoning domains unestablished.
  • DASH inherits the base model’s safety properties and failure modes without introducing additional safety mitigations.

Ethics Considerations · A Signal Implementation Details · B Signal Motivation and Prior Work

The paper introduces reproducible overthinking signals grounded in model traces and verified answers, motivates them through prior work, and reports sharply divergent reasoning lengths and accuracies on hard problems. It uses public datasets and open-weight models without human subjects, personal data, human annotation, or external learned judges.

  • Ethics Considerations: The study uses publicly available mathematical datasets and open-weight models, deriving all training signals from model-generated traces and verifiable groundtruth answers.No human subjects or personal data were involved, and no human annotation or external learned judges were used.
  • A Signal Implementation Details: Repetition is scored as maximum 5-gram Jaccard similarity across nonadjacent 200-token windows with 50-token stride, flagged above 0.4.Prior work describes low-temperature repetition as circular loops driven by self-reinforcing attention mechanisms.
  • A Signal Implementation Details: Recomputation flags unique numeric values repeated at least 3 times within 10-token computation contexts.Prior work links numerical loops to reasoning impasses and reports 20.3% token reduction when confirmatory self-verification is suppressed.
  • A Signal Implementation Details: Length outliers use within-prompt-group z-scores, flagging z > 2.0 and scaling scores as min((z−2)/2, 1).This adaptively captures pathological long responses without penalizing legitimately long solutions.
  • A Signal Implementation Details: Hedging uses keyword density, while abandonment counts approach-reversal phrases per trace.Explicit reconsideration precedes approach abandonment in over 67% of negative flips, abandonment increases with token count, and “alternatively” is 4.1–4.3× more common in incorrect Nemotron traces.
  • A Signal Implementation Details: Contradiction is counted through regex matches for inconsistency phrases, reflecting contradictions found in 17.7% of all ChatGPT sentences.Extended reasoning chains increase opportunities for self-contradiction and degenerate outputs.
  • B Signal Motivation and Prior Work: 88% accuracy occurs below 2K tokens, whereas 0.6% accuracy occurs in the 10K–25K token dead zone on hard Nemotron-4B problems.The bimodal length pattern motivates within-group normalization that avoids penalizing legitimately long solutions.

C Full Overthinking Signals On Nemotron-4B

On AIME 2024, density-normalized comparisons show that incorrect traces exhibit more recomputation than correct traces within fixed length buckets, while raw counts amplify this difference. Contradiction counts can nevertheless be productive, with DASH achieving higher accuracy on nearly all benchmarks despite elevated contradictions.

  • Length-controlled signals: Density-normalized panels distinguish behavioral rates from trace length by reporting signal counts per 100 words within length quintiles.Gaps between correct and incorrect traces therefore reflect intrinsic rate differences rather than only longer responses.
  • Numerical recomputation: 14.6 versus 4.9 recomputation events per 100 words in the shortest bucket and 9.9 versus 5.2 in the next show incorrect traces repeatedly re-derive quantities.The gap persists in every length quintile except the longest, indicating repeated recomputation is associated with failure itself.
  • Raw-count accumulation: 444 versus 131 recomputation events in the shortest bucket and 752 versus 394 at mid-length show that incorrect traces accumulate more repeated derivations in absolute terms.Higher recomputation rates combine with greater trace length, multiplying the total effect.
  • Contradiction signal: DASH achieves higher accuracy on all but AIME 2025 despite elevated contradiction counts on three of four benchmarks, suggesting many contradictions are productive.On Minerva Math, DASH combines lower s4 with higher accuracy, consistent with avoiding unproductive contradiction loops.

D Contradiction Signal: Discrimination Analysis … J Qualitative comparison of reasoning traces

DASH distinguishes productive self-correction from unproductive contradiction and spiraling, assigning credit within reasoning traces rather than controlling length alone. Across benchmarks and qualitative examples, it reduces drift while preserving targeted correction, with training and related-work analyses positioning it as a segment-level, drift-aware approach.

  • D Contradiction Signal: Discrimination Analysis: DASH’s higher contradiction signal is predominantly productive: it detects, diagnoses, and resolves errors, unlike baseline traces that abandon errors and drift.The raw s4 contradiction count conflates these phenomena.
  • D Contradiction Signal: Discrimination Analysis: 0.96 vs. 0.85 average contradiction counts: DASH increases contradictions 13% relative to DR-GRPO while improving accuracy across four benchmarks.On AIME 2024 and AIME 2025, contradictions rise 24% and 33%, respectively.
  • D Contradiction Signal: Discrimination Analysis: On Minerva Math, DASH reduces contradictions 13% while improving accuracy 4.1pp, indicating that simpler problems can avoid contradiction loops altogether.On AIME 2024, increased error detection accompanies a 3.1pp accuracy gain.
  • F Training Configuration; G.2 Additional Length Control Methods; I.1 Length Control in RL for Reasoning: DASH uses segment-level credit assignment with decoupled normalization, drift-shaped advantages, and penalties for repeated confirmation rather than token count alone.Its training configuration includes grpo_drift_hybrid, α+ = 1.0, α− = 1.0, length penalty α = 3.0, and enabled segment length penalty.
  • D.1 Qualitative Example: Self-Correction vs. Spiraling; J Qualitative comparison of reasoning traces: On AIME25 Problem 20, GRPO repeats blind substitutions until truncation, whereas DASH resolves the tangency misunderstanding at 59% of the trace and reaches the correct computation.The qualitative comparison reports 8 GRPO parameterizations versus 4 DASH contradictions, with DASH’s trace 43% shorter.
  • E Training Dynamics: DASH maintains the lowest KL divergence, while GRPO + Brevity Bonus suffers entropy collapse to 0.21, indicating targeted length reduction rather than aggressive policy deviation.Standard GRPO shows gradually increasing entropy and KL.
  • G.1 Adaptive Reasoning Depth; G.2 Additional Length Control Methods; G.3 Token-Level Credit Assignment, Continued; I.2 Token- and Segment-Level Credit Assignment: Unlike response-level adaptive-depth methods, DASH shapes credit within one trace and uses intermediate answers to detect drift without additional models, sampling, or learned components.Related methods include Thinkless, TEMPO, and SPO, while VAPO reshapes value estimation rather than the policy gradient.
  • G.4 Inference-Time Early Exit, Continued; H Artifact Licenses; I.1 Length Control in RL for Reasoning; I.3 Inference-Time Early Exit; J Qualitative comparison of reasoning traces: DASH complements inference-time early exit and differs from token-count controls by targeting answer drift; the study uses listed scientific artifacts under non-commercial research licenses.Table 4 shows DASH committing after brief verification and avoiding self-verification loops.
Loading 2607.00482v2…