PaperlayerView exact revision on arXiv

Source-linked AI summary

Stop When Reasoning Converges: Semantic-Preserving Early Exit for Reasoning Models

Dehai Min, Giovanni Vaccarino, Huiyi Chen, Yongliang Wu, Gal Yona, Lu Cheng

arXiv:2605.17672v1cs.CL

TL;DR

Long reasoning models often continue generating after reaching a solution, while answer-level early-exit signals may not capture reasoning convergence or preserve coherent reasoning. PUMA combines semantic-redundancy detection with answer verification to remove redundant continuation, achieving 26.2% average token reduction across five models and five benchmarks while preserving accuracy and retained CoT quality.

  • Problem

    Existing early-exit methods mainly use answer-level signals, leaving a need for efficient stopping that also preserves answer accuracy and semantically complete reasoning chains.

  • Method

    PUMA detects semantic redundancy within reasoning trajectories and uses answer-level verification to identify reliable early exits during decoding.

  • Results

    Across five LRMs and five challenging reasoning benchmarks, PUMA achieves 26.2% average token reduction while preserving accuracy and retained CoT quality.

  • Takeaways & Limitations

    Reasoning-level semantic redundancy is a robust, transferable, and learnable signal for efficient reasoning across text, code, and zero-shot vision-language tasks.

  • Takeaways & Limitations

    PUMA may be less effective for very short, poorly structured, or difficult-to-segment outputs, and broader domains may require additional calibration.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) achieve strong performance by generating long chains of thought (CoT), but often overthink, continuing to reason after a solution has already stabilized and thereby wasting tokens and increasing latency. Existing inference-time early-exit methods rely primarily on answer-level signals, such as confidence or trial-answer consistency, to decide when to stop. However, these signals mainly reflect answer readiness rather than reasoning convergence: they may trigger before the model has finished exploring or self-correcting, causing premature exits that can degrade final-answer accuracy and leave the retained reasoning chain semantically incomplete. We identify reasoning-level semantic redundancy as a complementary signal for semantic-preserving early exit: when successive steps no longer add novel progress and instead revisit established conclusions, the reasoning trajectory has likely converged. Building on this insight, we propose PUMA, a plug-and-play framework that combines a lightweight Redundancy Detector with answer-level verification. The detector flags semantically redundant candidate exits, while verification confirms whether stopping is safe, allowing PUMA to remove redundant continuation while preserving both answer accuracy and a coherent reasoning prefix. Across five LRMs and five challenging reasoning benchmarks, PUMA achieves 26.2% average token reduction while preserving accuracy and retained CoT quality. Additional experiments on code generation, zero-shot vision-language reasoning, and learned stopping-policy internalization further demonstrate that reasoning-level redundancy is a robust, transferable, and learnable signal for efficient reasoning. Our code is available at \url{https://github.com/giovanni-vaccarino/PUMA}.

1 Introduction

PUMA addresses inefficient overlong reasoning by detecting semantic redundancy within reasoning trajectories and verifying candidate exits. It reduces unnecessary tokens while preserving final-answer accuracy and the semantic coherence of retained chains of thought.

  • Motivation: Long CoT improves LRM performance and supports interpretability, but unnecessary continuation creates inefficiency and makes naive truncation inadequate.Effective early exit must preserve both final-answer accuracy and a coherent, semantically complete retained reasoning chain.
  • Motivation: Inference-time early exit is deployment-friendly, whereas training-based methods require retraining and prompt-based compression can suppress necessary intermediate reasoning.These tradeoffs motivate a lightweight method that operates during inference without per-model retraining.
  • Key insight: Within-trajectory semantic similarity provides a convergence signal when recent reasoning steps stop adding new logical or semantic content.This transfers semantic entropy’s focus on meaning-level similarity from diversity across outputs to progress within a reasoning trajectory.
  • Method: PUMA combines a lightweight Redundancy Detector, which flags semantically redundant candidate exits, with answer-level verification of trial-answer stability and confidence.The detector determines where to consider stopping, while verification determines whether the candidate exit is reliable; a Loop Breaker handles persistent late-stage redundancy.
  • Results: 26.2% average token reduction is achieved across five LRMs and five challenging reasoning benchmarks while preserving final-answer accuracy and retained CoT quality.The token savings also translate into practical wall-clock speedups, indicating that efficiency gains do not come at the expense of reasoning-chain quality.

2 Related Work

Related work frames efficient reasoning around LRM overthinking and inference-time early exit. Existing early-exit methods primarily use answer-level or token-level signals, which may detect readiness without measuring semantic progress.

  • Overthinking and efficient reasoning in LRMs: Overthinking describes extended CoT reasoning that generates more tokens than necessary despite improving performance.Later tokens may enter a converged phase and become largely redundant.
  • Overthinking and efficient reasoning in LRMs: Existing efficient-reasoning methods intervene at different stages, including training-based approaches that reshape reasoning through length-penalized reinforcement learning.
  • Inference-time early exit: Inference-time early-exit methods use answer-level signals such as trial-answer confidence or agreement across consecutive probes.These signals estimate answer readiness but do not determine whether the reasoning trajectory is still making semantic progress.
  • Inference-time early exit: Token-level early-exit signals track decoding artifacts, including the rank of the </think> token, exit-associated neurons, and reflection-triggered features.

3 Methodology

PUMA performs online, step-level early exit by detecting semantic redundancy as reasoning converges, then verifying answer readiness before stopping. Its primary verified-exit path is supplemented by a late-stage Loop Breaker for trajectories that remain redundant without satisfying verification conditions.

  • Overview: PUMA models a reasoning chain as segmented steps followed by a final answer and operates online over this step sequence.Steps are segmented at natural paragraph or step boundaries.
  • Answer Verification: After a redundancy flag, Answer Verification probes a trial answer from the current prefix and evaluates confidence and consistency across a verification window before stopping.The checks require a confident, consistent candidate answer without a material confidence decline; otherwise generation resumes.
  • Overview: PUMA is plug-and-play during decoding, requiring no model-weight or base-prompt modifications, and combines Verified Early Exit with a Loop Breaker fallback.Verified Early Exit is the primary stopping mechanism.
  • Redundancy Detector: The lightweight Redundancy Detector uses a LoRA-fine-tuned Qwen3-Embedding-0.6B model with an InfoNCE objective to distinguish novel progress from restatement, re-derivation, or looping.It compares the current step with previous reasoning steps; the default lookback is k=1.
  • Loop Breaker: The Loop Breaker activates after a minimum step threshold when m consecutive redundant steps occur, using the highest-confidence observed trial answer and a weak minimum-confidence gate.It addresses trajectories that produce redundancy without satisfying all Verified Early Exit conditions.

4 Experimental Setup

The experimental setup evaluates PUMA across diverse reasoning benchmarks and language models, using deployment-compatible baselines and joint accuracy-efficiency metrics. Experiments run on GH200 hardware with standardized inference settings and repeated random seeds.

  • Benchmarks and models: PUMA is evaluated on five challenging benchmarks spanning competition mathematics, olympiad-level STEM, and graduate-level scientific reasoning.The benchmarks are MATH-500, AIME24/25, OlympiadBench, and GPQA-Diamond.
  • Benchmarks and models: The main experiments cover five LRMs from diverse families and scales, including 7B–32B dense and mixture-of-experts models.The listed models are DeepSeek-R1-Distill-Qwen at 7B/14B/32B, Llama-3.1-Nemotron-Nano-8B, and Qwen3-30B-A3B-Thinking.
  • Baselines: Comparisons emphasize deployment-compatible efficiency methods that neither modify model weights nor require training model-specific hidden-state probes, with Full-CoT as the reference.The baselines include prompt-based methods such as No-Think, Concise CoT, Chain of Draft, and Plan-and-Budget.
  • Metrics: The main metrics are accuracy (Acc), average generated tokens (Tok), and token reduction TR = (1 −Tokmethod/TokFull-CoT) × 100, reported jointly to assess efficiency without sacrificing answer quality.Token-reduction statistics are weighted by benchmark size, while accuracy follows benchmark-level reporting conventions.
  • Implementation Details: Experiments use vLLM on a single node with 4×NVIDIA GH200 Grace Hopper superchips, recommended reasoning settings, and results averaged over three random seeds.The seeds are 0, 42, and 123; the Redundancy Detector is served on the same node.

5 Experimental Results

PUMA achieves substantial token savings while preserving accuracy and retained reasoning quality. It also converts these savings into practical speedups and transfers beyond text-only reasoning to code and vision-language tasks.

  • Accuracy–Efficiency Tradeoff: 26.2% average token reduction is achieved across five LRMs and five benchmarks while preserving accuracy.PUMA removes redundant continuation without sacrificing final-answer quality, with a slight average accuracy gain over Full CoT.
  • Accuracy–Efficiency Tradeoff: PUMA is more accuracy-preserving than prompt-based compression and offers a more reliable accuracy–efficiency tradeoff than answer-level early-exit baselines.On Qwen3-30B-A3B-Thinking, CCoT, CoD, and Plan-and-Budget reach 54.3%, 45.3%, and 59.6% accuracy, respectively, versus 81.7% for Full CoT.
  • Retained Reasoning Quality: PUMA achieves the highest average retained-chain quality, ranking first in coherence, conciseness, and justification while remaining close to Full CoT in completeness.The evaluation uses an LLM-as-Judge protocol covering completeness, coherence, conciseness, and justification.
  • Wall-Clock Efficiency: 1.40× and 1.28× average speedups are achieved on DS-7B and DS-14B, respectively, while the Redundancy Detector adds only 0.4–1.1% overhead.PUMA turns token savings into practical speedups despite the overhead associated with trial-answer probing.
  • Transfer Beyond Text-Only Reasoning: 18–19% token reductions on LiveCodeBench come with at most 1.5 points of pass@1 change, while PUMA is also evaluated on MathVista and MathVision.These experiments test transfer to code generation and vision-language reasoning beyond text-only mathematical and scientific QA.

6 Analysis and Discussion

PUMA’s ablations show that its redundancy-triggered verification components jointly balance accuracy, token efficiency, and probe efficiency. Further analyses reveal model-specific mixtures of exit modes, while stopping supervision can be internalized through fine-tuning without deploying PUMA modules.

  • Component ablation: 60.2% accuracy and 35.6% token reduction mark PUMA’s best accuracy–efficiency balance in the DS-7B ablation.The full configuration combines candidate triggering and verification components.
  • Component ablation: 13.0 percentage points of token reduction are lost without the Loop Breaker, confirming its role as an efficiency fallback for sustained redundancy.Token reduction falls from 35.6% to 22.6%.
  • Component ablation: 6.5 points of accuracy are lost without the Confidence Gate, while removing Answer Consistency lowers accuracy by 2.9 points despite the former variant’s higher token reduction.The ablation identifies Answer Verification as important for preserving accuracy.
  • Exit behavior: Verified exits, Loop Breaker exits, and full reasoning occur in model-specific proportions, reflecting differing redundancy patterns across three representative LRMs.Figure 4 also analyzes correctness transitions for verified and Loop Breaker exits relative to Full CoT.
  • Stopping-signal internalization: PUMA-selected exit positions supervise SFT, DPO, and GRPO fine-tuning on 12K math problems, with all trained variants evaluated without PUMA modules at deployment.The internalization experiment uses DS-R1-Distill-Qwen-7B and pure vLLM inference.

7 Conclusion … A.2 Failure Rates of Answer-Level Stopping Signals

PUMA uses reasoning-level semantic redundancy with answer-level verification to stop after reasoning converges, reducing tokens while preserving accuracy and retained-chain quality. The paper also finds substantial post-answer overthinking and evaluates failure modes of confidence- and consistency-based stopping signals.

  • 7 Conclusion: PUMA reduces tokens by 26.2% on average across five LRMs and five challenging reasoning benchmarks while preserving accuracy and improving retained reasoning-chain quality.The framework also delivers practical wall-clock speedups.
  • 7 Conclusion: PUMA may be less effective for very short, poorly structured, or difficult-to-segment outputs because it relies on step-level reasoning traces.Its Redundancy Detector transfers beyond text-only question answering but is primarily trained on text reasoning traces, so broader domains may require additional calibration.
  • 7 Conclusion: Internalization experiments use one base model and math-focused training data, leaving larger-scale and cross-domain learned stopping policies for future work.
  • A.1 Overthinking Prevalence: 41–52% of reasoning tokens are generated after models have already committed to their final answers across five representative LRMs and five benchmarks.The analysis defines the golden step as the earliest step whose intermediate trial answer matches the final answer, with later tokens classified as post-answer redundancy.
  • A.1 Overthinking Prevalence: Even Qwen3-30B-A3B-Thinking spends over half its tokens on post-answer re-verification, rephrasing, and re-derivation.
  • A.2 Failure Rates of Answer-Level Stopping Signals: The study retroactively evaluates confidence and consecutive-trial-answer consistency as answer-level stopping signals on full CoT traces.Confidence uses the geometric mean of token-level log-probabilities, while consistency checks trial answers across consecutive steps.
  • A.2 Failure Rates of Answer-Level Stopping Signals: λ = 0.95 confidence and k=3 consecutive identical trial answers define candidate exits across five models and five benchmarks.Failure rate is the fraction of first-triggered exits whose trial answer is incorrect, averaged over all 25 model–benchmark combinations.

A.3 Counterfactual Analysis: Do Signal Misfires Prevent Self-Correction? … B.6 Trial Answer Induction

The analyses show that answer-level stopping can interrupt recoverable reasoning, while PUMA uses lightweight step-level redundancy detection and verification to identify safer convergence points. Its implementation combines deterministic segmentation, contrastive detector training, conservative local comparisons, and task-specific trial-answer induction.

  • A.3 Counterfactual Analysis: Do Signal Misfires Prevent Self-Correction? / A.4 Threshold Sensitivity of Answer-Level Stopping Signals: 42.4% of confidence-based failures and 54.8% of consistency-based failures were premature exits that interrupted trajectories which would have self-corrected.Premature exits are incorrect triggered answers followed by correct uninterrupted Full-CoT answers; non-recoverable failures remain incorrect without early stopping.
  • A.4 Threshold Sensitivity of Answer-Level Stopping Signals: Threshold sweeps preserve a failure-rate/token-reduction tradeoff, with safer operation points generally reducing the usefulness of standalone answer-level early exit.Confidence thresholds tested λ ∈{0.93, 0.94, 0.95, 0.96, 0.97}, while consistency windows tested k ∈{1, 2, 3, 4, 5, 6, 7, 8}.
  • B Implementation Details / B.1 Reasoning Step Segmentation: PUMA segments reasoning into coherent steps by splitting blank-line-delimited paragraphs, assigning coarse semantic roles, and merging compatible short paragraphs within [Lmin, Lmax] = [200, 1000] characters.The string-based procedure adds negligible overhead and supplies stable units for detector training and online early-exit decisions.
  • B.2 Redundancy Detector Training: The detector is trained contrastively from GPT-5-mini seed annotations and synthesized redundant counterparts, producing 701,641 InfoNCE training rows.The seed stage contains 40,844 annotated examples, while labels distinguish novel steps (y = 0) from redundant steps (y = 1).
  • B.2 Redundancy Detector Training: The trained detector achieves 91.26% pairwise ranking accuracy and, at τsim = 0.35, reaches 91.54% absolute classification accuracy with a 93.58% true-negative rate.Its conservative threshold is selected on held-out calibration data to avoid over-triggering on genuinely novel reasoning.
  • B.3 Choice of Redundancy Signal: Embedding-based redundancy detection preserves and slightly improves accuracy over Full CoT by +1.3 on average while achieving 27.9% token reduction, outperforming NLI-style alternatives on the accuracy–efficiency trade-off.The embedding detector averages about 21 ms per question versus ∼93 ms for both NLI variants, while unfine-tuned embeddings incur a substantial accuracy cost.
  • B.4 Redundancy Detector Lookback Window: Using k = 1, which compares each step with its immediate predecessor, gives the best accuracy-preserving behavior; expanding to k = all changes average ∆Acc from +1.3 to −2.4.Larger lookback windows increase token reduction but can falsely trigger on similarity to earlier steps that no longer reflects useful local progress.
  • B.5 A Semantic-Entropy Perspective on Reasoning Convergence / B.6 Trial Answer Induction: Low local Reasoning Semantic Entropy indicates recent steps have collapsed into redundant semantic patterns, motivating PUMA’s detector as a lightweight proxy for convergence rather than explicitly clustering steps online.At candidate exits, PUMA induces task-specific trial answers: math and multiple choice remain in thinking mode, code closes </think>, and probes are capped at ∼30 or ∼50 tokens.

B.7 Extended Hyperparameter Analysis … C.3 Full Main Results Across Five Models

PUMA remains robust across broad stopping-hyperparameter settings and achieves substantial reasoning-step compression across five models, while full results and benchmark extensions document its evaluation scope and limitations of prompt-only compression.

  • B.7 Extended Hyperparameter Analysis: PUMA is robust across nine alternative hyperparameter configurations, with only two producing negative ∆Acc values, both within 1pp of Full CoT.The negative values are −0.7 at τsim=0.40 and −0.1 at λ=0.99.
  • B.7 Extended Hyperparameter Analysis: The redundancy threshold τsim is most safety-critical, while λ is stable from 0.95 to 0.98 and λ=0.99 sharply reduces token savings.Lowering τsim to 0.30 changes ∆Acc from +2.2 to 0.0; raising it to 0.40 nearly halves token savings, while λ=0.99 produces −6.3pp TR.
  • B.7 Extended Hyperparameter Analysis: The default L = 2 provides the best balance, whereas L = 3 is overly conservative and reduces token reduction without improving accuracy.An additional verified candidate often does not appear before generation continues when L = 3.
  • C Experimental Details and Full Results: PUMA’s main stopping hyperparameters are shared across models, while the Loop Breaker threshold m is tuned per model using held-out AMC23 validation performance.Different models use m=1, m=3, or m=4, and the Loop Breaker is not activated for Qwen3-30B-A3B-Thinking.
  • C.1 Benchmark and Dataset Statistics: The main evaluation spans five challenging reasoning benchmarks, with additional generalization experiments covering code generation and vision-language reasoning.The benchmark domains include competition mathematics, olympiad-level STEM, and graduate-level science; OlympiadBench uses 675 text-only English open-ended math problems.
  • C.2 Existing Assets and Licenses: All main assets are publicly available and cited with their original references, and the work does not redistribute third-party model weights or benchmark datasets.The assets support research evaluation and method development.
  • C.3 Full Main Results Across Five Models: PUMA saves 17.6–28.5% of reasoning steps across models, with deeper compression on longer chains and more conservative savings on shorter chains.DS-7B achieves 28.5% savings and Qwen3-30B-T achieves 27.9%; Table 14 reports all five models’ per-benchmark token counts, accuracy, and token reduction.
  • C.3 Full Main Results Across Five Models: Prompt-only compression is limited because asking a reasoning model to skip thinking is not a reliable substitute for online monitoring and stopping.This limitation motivates process-level early-exit intervention rather than relying solely on prompts.

C.4 Full Component Ablation · C.5 Detailed Latency Analysis · D Budget Tuning Does Not Rescue Prompt-Based Baselines

The ablations show that PUMA’s redundancy filtering and answer-verification components are complementary, while latency results demonstrate that token reduction does not always yield wall-clock speedup. Budget sweeps likewise show that prompt-based baselines do not close their accuracy gap through relaxed word limits.

  • C.4 Full Component Ablation: PUMA’s full component ablation evaluates how redundancy detection, answer verification, and probing overhead affect accuracy and token reduction.The extended analysis covers DS-7B and Qwen3-30B-A3B-Thinking, including AV-only and w/o AC and CG variants.
  • C.4 Full Component Ablation: Removing the RD gate invokes Answer Verification at every eligible reasoning step, increasing token reduction but lowering accuracy and raising probe overhead.Probe overhead rises 3.3× on DS-7B and 4.3× on Qwen3-30B-T.
  • C.4 Full Component Ablation: Disabling either Answer Consistency or the Confidence Gate degrades accuracy, while removing both causes the largest accuracy collapse despite high token reduction.These gates prevent premature exits, complementing the RD gate and Answer Verification.
  • C.4 Full Component Ablation: The ablation identifies complementary roles: the RD gate controls verification timing, Answer Verification filters unreliable candidates, and the Loop Breaker contributes to exit control.The supplied passage states these components work together to balance accuracy and efficiency.
  • C.5 Detailed Latency Analysis: 1.15–1.40× speedup: PUMA consistently accelerates wall-clock inference across settings, whereas DEER is slower than Full CoT at 0.52–0.71×.Dynasor ranges from 1.11× to 0.17× depending on model size.
  • C.5 Detailed Latency Analysis: Positive token reduction does not guarantee end-to-end speedup when verification overhead outweighs the time saved by shorter reasoning traces.This explains why token-reducing methods can still run slower than Full CoT.
  • D Budget Tuning Does Not Rescue Prompt-Based Baselines: Budget tuning does not close PUMA’s accuracy gap with prompt-based baselines CCoT and CoD on DS-7B LiveCodeBench.The sweep uses prompted word budgets, but actual token reduction reflects generated outputs because LRMs may not follow nominal budgets strictly.

E Reasoning Quality Evaluation Details

Reasoning-chain quality is evaluated beyond final-answer accuracy using an anonymized LLM-as-Judge protocol. The evaluation scores retained chains with a four-dimensional rubric and emphasizes relative comparisons under a shared judge and rubric.

  • Evaluation goal: The evaluation measures whether retained reasoning chains remain readable and sufficient, rather than only whether final-answer accuracy is preserved.This directly targets the quality of the retained reasoning prefix after early exit.
  • Judge protocol: GPT-5.4-thinking judges each retained chain from the original question, providing a rationale and scores without gold answers, method names, or correctness labels.Three model–benchmark combinations are evaluated: two GPQA-Diamond settings with 198 questions each and Qwen3-30B-A3B-Thinking on MATH-500 with 500 questions.
  • Rubric: Chains receive scores from 10–100 in increments of 10 for completeness, coherence, conciseness, and justification.The dimensions assess sufficient derivation, logical connectedness, avoidance of repetition or loops, and why the final answer follows.
  • Bias control: Evaluator bias is mitigated by anonymized judging under one rubric, while the judge is excluded from detector training and stopping-hyperparameter selection.Detector supervision uses reasoning-step-pair annotations, whereas the judge evaluates complete retained chains with different inputs and prompts.

F Internalizing PUMA into Model Weights: Implementation Details

PUMA internalization fine-tunes DS-R1-Distill-Qwen-7B with LoRA and removes Redundancy Detector and answer verification at inference. Training uses Lightman-split MATH data to construct supervised, preference, and reinforcement-learning variants with PUMA- or fixed-position exit signals.

  • Implementation: All internalization variants fine-tune DS-R1-Distill-Qwen-7B with LoRA and use pure vLLM inference without RD or AV at test time.LoRA uses rank 64, alpha 128, and all-linear targeting.
  • Training data: The training set contains 12K mathematics-only problems from the Lightman split of MATH-benchmark, with no overlap with the MATH-500 test split.MATH-500 therefore measures in-distribution behavior, while AIME24 and GPQA-D test zero-shot generalization.
  • Per-variant data construction: PUMA-SFT retains verified correct early exits under t∗/|R| < 0.6, yielding ∼6.5K examples, while PUMA-DPO forms ∼5.8K chosen–rejected pairs from correct truncated and Full CoT chains.FixedExit-SFT uses the earliest correct fixed-interval position with K=3, and PUMA-DPO chooses PUMA-truncated chains over Full CoT chains when both are correct.
  • Per-variant data construction: PUMA-RL and FixedExit-RL train on ∼15K pre-expanded samples each, using reasoning prefixes truncated at PUMA-flagged or fixed-interval positions before answer generation.Standard-SFT and Standard-GRPO omit exit-position signals.
Loading 2605.17672v1…