PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Clearly: Improving Reasoning via Redundant Token Pruning

Daewon Choi, Jimin Lee, Jihoon Tack, Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati

arXiv:2507.08806v1cs.AIcs.CLcs.LG

TL;DR

Reasoning traces can contain redundant detours, and the paper asks whether removing them can improve performance. It estimates token importance through self-summarization and prunes low-contributing reasoning chunks; the authors report improved accuracy and memory efficiency across reasoning tasks.

  • Problem

    Reasoning chains can contain repetitive, verbose, or speculative detours, and low-attention chunks may be abandoned while leaving redundant traces.

    Source: §1 · p. 1

  • Method

    The method prompts the model to summarize its ongoing reasoning using an end-of-thinking token, then prunes low-contributing tokens with a chunk-aware eviction strategy.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Accuracy improves across reasoning-heavy tasks; on AMC2023, accuracy rises from 75.0% to 82.5%, while KV cache memory usage falls by 10.3% on DeepSeek-R1-Distill-Qwen-7B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Targeted KV cache pruning can serve as compression while also improving reasoning performance across a range of reasoning tasks.

    Source: §6 · p. 8

    The method relies on visible reasoning traces, and its summarization-based token-importance estimation may not generalize directly to models that do not expose their reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent large language models have shown promising capabilities in long-form reasoning, following structured chains of thought before arriving at a final answer. However, we observe that these reasoning paths tend to include substantial redundancy; analyzing attention patterns reveals that attention scores are widely scattered, particularly incorrect answers exhibit greater attention sparsity. In this paper, we demonstrate that deliberately removing this redundancy in the reasoning process significantly improves performance through clear thinking, i.e., removing distraction. Specifically, we systematically identify reasoning redundancy by measuring token-level attention scores to a special end-of-thinking token, which is appended to an explicit instruction inserted to conclude each intermediate reasoning step. Furthermore, we propose structure-aware pruning that prioritizes removing tokens in low-contributing reasoning chunks over individual tokens. After evicting redundant tokens, we remove the injected end-of-thinking instruction, then resume the reasoning generation. We demonstrate that our method significantly improves overall accuracy across reasoning-intensive benchmarks without any training involved. In particular, our method shows strong performance on challenging mathematical competition benchmarks such as AIME and AMC, where reasoning redundancy is more prevalent.

1 Introduction

Structured reasoning chains let language models break complex problems into smaller steps, and verbalized thoughts can be monitored or intervened in. The paper asks whether removing redundant reasoning tokens can improve performance and proposes test-time pruning to do so.

1 Introduction

  • Reasoning chains often contain repetitive, verbose, or speculative detours, with sparse attention patterns especially evident in incorrect answers.The authors describe these patterns as potentially distracting the model from relevant context.Source: §1 · p. 1
  • The authors ask whether removing redundant tokens on the fly can improve reasoning performance.They motivate the question with low-attention reasoning chunks that appear to be abandoned, leaving redundant traces.Source: §1 · p. 1
  • The proposed test-time method measures token contributions to a summarization-inducing end-of-thinking token and prioritizes pruning low-contributing reasoning chunks.After pruning, the injected end-of-thinking instruction is removed and generation resumes.Source: §1 · p. 2
  • Accuracy improves from 75.0% to 82.5% on AMC2023, while KV cache memory usage falls by 10.3% on DeepSeek-R1-Distill-Qwen-7B.The authors report significant gains on challenging mathematics competition benchmarks, including AIME and AMC.Source: §1 · p. 2

2 Related Work

Related work covers structured long-form reasoning and methods for reducing the memory cost of growing KV caches. The paper distinguishes its approach by targeting redundant reasoning tokens to improve performance, as well as compression.

2 Related Work

  • The section situates the work within reviews of long chain-of-thought reasoning and token-pruning or compression frameworks.The paper describes its approach as test-time pruning of redundant tokens so models can focus on relevant reasoning steps.Source: §2 · p. 2
  • Prior reasoning work uses multi-step intermediate traces to break complex problems into manageable steps, with reported benefits including accuracy, robustness, self-correction, and verification.The section also notes that reasoning may be exposed to users or kept latent, depending on the model.Source: §2 · p. 2
  • Existing KV-cache approaches use token-retention rules, accumulated attention scores, per-step low-attention eviction, or layer-wise compression.The section frames KV-cache growth with sequence length as an inference memory bottleneck.Source: §2 · pp. 2–3
  • Unlike methods primarily aimed at inference efficiency, this paper targets redundant reasoning tokens to improve performance over the full KV cache.The authors also report competitive performance under memory constraints relative to recent token-eviction and compression baselines.Source: §2 · p. 3

3 Not All Tokens Matter for Reasoning

The section examines whether all generated reasoning tokens are needed to reach a correct answer. It reports that attention patterns differ between failed and successful reasoning and motivates using end-of-thinking attention and reasoning chunks to identify redundancy.

3 Not All Tokens Matter for Reasoning

  • The analysis considers multi-step reasoning traces followed by an answer, marked with <think> and </think>.It focuses on whether all previously generated tokens are needed to reach a correct answer.Source: §3 · p. 3
  • Failed reasoning examples contain redundant text and sparse attention, while successful examples show a clearer trajectory and more global attention structure.The comparison suggests attention scores may help identify redundant reasoning steps.Source: §3 · p. 3
  • Reasoning tokens that contribute to the final answer tend to receive high attention from the end-of-thinking token, including chunks that initiate reasoning or summarize key conclusions.The section analyzes this token as a way to quantify token importance.Source: §3 · p. 3
  • Redundant tokens often occur in contiguous chunks, and the end-of-thinking token tends to attend to informative segments while ignoring irrelevant parts.This motivates a pruning strategy that uses the end-of-thinking token together with reasoning-chunk structure.Source: §3 · pp. 3–4

4 Improving Reasoning with Redundant Token Pruning

The section presents a KV-cache eviction policy that uses self-summarization to score reasoning-token importance, then allocates eviction across reasoning steps to favor redundant ones.

4 Improving Reasoning with Redundant Token Pruning

  • The proposed KV-cache eviction policy combines self-summarization-based token scoring with step-aware eviction.It is designed to remove redundant reasoning tokens while improving reasoning models’ effectiveness and efficiency.Source: §4 · p. 4

4.1 Identifying redundant tokens via self-summarization

  • A short prompt shifts the model from reasoning to summarization, and attention to its concluding </think> token scores each token’s contribution to the summary.The token scores are computed separately for each layer and attention head.Source: §4.1 · pp. 4–5

4.2 Step-aware eviction with hierarchical budget allocation.

  • The reasoning trace is divided into semantically coherent steps, whose importance is aggregated across their tokens and attention heads.Each step’s score is the mean importance of its tokens across heads.Source: §4.2 · p. 5
  • The eviction budget is allocated in order of step importance, prioritizing less important steps before removing the lowest-scoring tokens within them.The policy keeps total evictions within the budget and avoids premature removal from important steps.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate reasoning accuracy and KV-cache efficiency, including mathematical and non-mathematical tasks. The reported results show improved accuracy, reduced memory use, and contributions from the method’s structured pruning components.

5.1 Experimental setup

  • Evaluation spans mathematical benchmarks and GPQA, using distilled Qwen and Llama reasoning models, FullKV and KV-compression baselines, and accuracy and cache-use measures.The protocol scores answer accuracy by exact match and measures KV-cache use during generation.Source: §5 · p. 5, §5.1 · pp. 5–6

5.2 Redundant reasoning token pruning improves the performance

  • 57.9% to 63.4% average accuracy under Qwen2.5-7B versus FullKV.The method significantly and consistently outperforms FullKV in accuracy across the reported mathematical benchmarks.Source: Table 1 · p. 6, §5.2 · p. 7

5.3 Redundant token pruning efficiently and effectively reduces KV cache budget

  • At 50% memory usage, the method maintains over 94% of full-KV accuracy.The method achieves superior compression without compromising accuracy, while other methods lose accuracy under aggressive pruning.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 Ablation and analysis

  • Ablations show that removing self-summarization yields inaccurate importance scores, while removing step-aware eviction over-prunes important chunks; the full method performs best.The component analysis reports accuracy on AIME2024 and AMC2023 using Qwen2.5-7B distilled from DeepSeek-R1.Source: Table 3 · p. 7, §5.4 · p. 8
  • Random and H2O pruning yield marginal gains, whereas the method significantly outperforms them by aligning eviction with reasoning-trace structure.The comparison evaluates deliberate pruning against random and structure-agnostic strategies on mathematical reasoning benchmarks.Source: §5.4 · p. 8, Table 4 · p. 8
  • On GPQA Diamond, the method improves performance over FullKV by suppressing iterative intermediate justifications that hurt answer quality.GPQA Diamond is a non-mathematical science reasoning benchmark.Source: §5.4 · p. 8, Table 5 · p. 8

6 Conclusion

The conclusion presents test-time pruning of redundant reasoning tokens as a way to enhance LLM reasoning. It combines forced intermediate summarization with step-aware eviction and requires no retraining or model modification.

6 Conclusion

  • The method estimates token contribution through forced intermediate summarization, then allocates eviction budgets across reasoning steps by aggregated importance.This hierarchical strategy preferentially prunes tokens from redundant reasoning steps; the plug-and-play algorithm requires no retraining or model modification.Source: §6 · p. 8

Limitations

The method’s scope is limited by its reliance on visible reasoning traces, test-time-only pruning, and evaluation focused mainly on factual or symbolic reasoning.

Limitations

  • The method relies on visible thoughts or intermediate supervision, and its importance estimation may not generalize directly to models that do not expose internal reasoning.The authors identify end-to-end models without exposed reasoning as outside the method’s direct applicability.Source: Limitations · p. 9
  • Pruning is applied only at test time, limiting adaptiveness because the method does not train token importance jointly with model weights.The authors suggest joint learning through reinforcement learning or differentiable attention masking as possible future directions.Source: Limitations · p. 9
  • Evaluation focuses mainly on factual or symbolic reasoning, leaving open-domain question answering, commonsense inference, and multimodal reasoning as open challenges.The authors note that these settings may require different redundancy detection or finer-grained reasoning segmentation.Source: Limitations · p. 9

A Experimental Details

Experiments use three distilled reasoning models from the DeepSeek-R1-Distill family.

A Experimental Details

  • The experiments use Qwen2.5-1.5B, Qwen2.5-7B, and Llama3.1-8B from the DeepSeek-R1-Distill family.All checkpoints were downloaded from Huggingface.Source: §A.1 · p. 11

A.2 Implementation

The implementation segments reasoning traces using phrase markers, then applies periodic token eviction or a fixed KV-cache budget during decoding.

A.2 Implementation

  • Reasoning traces are segmented into intermediate steps at phrases from a predefined set that signal reflection, correction, sequencing, or exploring alternatives.Examples include “Wait,” “Alternatively,” “First,” and “Therefore.”Source: §A.2 · p. 11
  • The effectiveness setting removes k tokens from the KV cache across all heads and layers at each generation interval p.The interval is 200 for Qwen2.5-1.5B and 7B, and 100 for LLaMA3.1-8B.Source: §A.2 · p. 11
  • The efficiency setting caps decoding KV-cache length using 25% or 50% of the Full KV baseline’s average length.A recent-token window occupies half the allocated cache; the full problem prompt remains cached and is excluded from the fixed budget.Source: §A.2 · p. 11
  • Table 6 compares Ours with FullKV on six mathematical benchmarks using Qwen2.5-1.5B, reporting accuracy and average KV-cache length.Average accuracy and KV cache are reported in the final column.Source: Table 6 · p. 12

B.1 Additional results on a small reasoning model

The method improves reasoning accuracy on a small reasoning model across multiple mathematical benchmarks, including AMC2023.

B.1 Additional results on a small reasoning model

  • The method significantly improves overall accuracy across multiple reasoning-intensive mathematical benchmarks on the Qwen2.5-1.5B reasoning model.The authors also report effectiveness on AMC2023, a challenging benchmark.Source: §B.1 · p. 12

Related papers

Loading 2507.08806v1…