PaperlayerView exact revision on arXiv

Source-linked AI summary

TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling

Weizhe Lin, Xing Li, Zhiyuan Yang, Xiaojin Fu, Hui-Ling Zhen, Yaoyuan Wang, Xianzhi Yu, Wulong Liu, Xiaosong Li, Mingxuan Yuan

arXiv:2505.17155v2cs.LGcs.AIcs.CL

TL;DR

Large Reasoning Models can incur substantial decoding overhead from extended reasoning, including overthinking and unresolved underthinking. TrimR uses a lightweight verifier to compress reasoning without fine-tuning, and evaluations report runtime reductions of up to 70% with accuracy largely unaffected.

  • Problem

    Extended Chain-of-Thought reasoning incurs substantial decoding overhead, while LRMs can generate redundant overthinking or lengthy, unresolved underthinking.

    Source: §1 · p. 2, §3.3 · p. 5

  • Method

    TrimR uses a lightweight verifier to detect redundant intermediate thoughts and prompt early termination, without fine-tuning the LRM or verifier.

    Source: §1 · p. 2

  • Results

    Runtime reductions reach 70% across evaluated models and benchmarks, with accuracy largely unaffected and drops under 1.7%.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    TrimR combines dynamic reasoning compression with an asynchronous system intended for high-throughput industrial deployment.

    Source: §5 · p. 10

    The underthinking thresholds Rthres and Nthres must be calibrated to allow adequate exploration of alternative reasoning paths before termination.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Large Reasoning Models (LRMs) demonstrate exceptional capability in tackling complex mathematical, logical, and coding tasks by leveraging extended Chain-of-Thought (CoT) reasoning. Test-time scaling methods, such as prolonging CoT with explicit token-level exploration, can push LRMs' accuracy boundaries, but they incur significant decoding overhead. A key inefficiency source is LRMs often generate redundant thinking CoTs, which demonstrate clear structured overthinking and underthinking patterns. Inspired by human cognitive reasoning processes and numerical optimization theories, we propose TrimR, a verifier-based, training-free, efficient framework for dynamic CoT compression to trim reasoning and enhance test-time scaling, explicitly tailored for production-level deployment. Our method employs a lightweight, pretrained, instruction-tuned verifier to detect and truncate redundant intermediate thoughts of LRMs without any LRM or verifier fine-tuning. We present both the core algorithm and asynchronous online system engineered for high-throughput industrial applications. Empirical evaluations on Ascend NPUs and vLLM show that our framework delivers substantial gains in inference efficiency under large-batch workloads. In particular, on the four MATH500, AIME24, AIME25, and GPQA benchmarks, the reasoning runtime of Pangu Pro MoE, Pangu-R-38B, QwQ-32B, and DeepSeek-R1-Distill-Qwen-32B is improved by up to 70% with negligible impact on accuracy.

1 Introduction

Extended Chain-of-Thought reasoning gives Large Reasoning Models strong capabilities but adds decoding cost, and their reasoning can include redundant overthinking or unresolved underthinking. TrimR proposes verifier-based, training-free compression to reduce this overhead while maintaining reasoning performance.

1 Introduction

  • Extended LRM reasoning incurs substantial decoding overhead and inference cost, hindering deployment in resource-constrained settings.Source: §1 · p. 2
  • Overthinking repeats verification after correct steps, while underthinking oscillates among incomplete chains and produces lengthy but inaccurate responses.Source: §1 · p. 2
  • TrimR uses a lightweight verifier to detect overthinking and underthinking in intermediate thoughts, then prompts early termination when redundancy is detected.The approach is training-free and uses answer existence and equivalence checks rather than complex process or outcome reward models.Source: §1 · p. 2
  • The paper models reasoning as optimization in language space and seeks a stopping rule that minimizes inference cost without degrading reasoning performance.Source: §1 · pp. 2–3
  • The contributions include a training-free compression method and asynchronous deployment system, evaluated across dense and MoE models with reduced reasoning cost and token use.Source: §1 · p. 3

2 Related Works

Prior approaches shorten reasoning through model training, self-evaluation, or auxiliary model collaboration, each with deployment or compute trade-offs. TrimR instead uses a lightweight verifier without fine-tuning or added self-evaluation steps.

2 Related Works

  • Training-based compression uses length-reward reinforcement learning, supervised fine-tuning, or latent-space methods, but demands substantial compute and can overfit or degrade general capabilities.Source: §2 · p. 3
  • Self-evaluation methods ask LRMs to estimate confidence or restarting benefit, but added inference steps may increase latency and complicate deployment.Source: §2 · p. 3
  • Collaborative frameworks use auxiliary evaluators or critique models, while some small-model approaches require extra training, state switching, interruption, or fine-tuning.Source: §2 · p. 3
  • TrimR uses a lightweight verifier to detect and truncate redundant reasoning without fine-tuning or adding self-evaluation steps, supporting integration with existing inference frameworks.Source: §2 · p. 3

3 Method

TrimR compresses reasoning during online inference using a small verifier to detect redundant thoughts, underthinking, and repetition. Its method combines thought extraction and verification with termination prompts and an asynchronous deployment system.

3 Method

  • TrimR dynamically prunes redundant Chain-of-Thought generation during online inference using smaller verifiers without fine-tuning either the verifier or LRM.Source: §3 · p. 3

3.1 Thought Detection and Answer Extraction

  • Thoughts are segmented at reflection tokens, and the last several sentences of each segment are extracted as intermediate thought answers for checking.Extremely short thoughts are skipped; asynchronous token detection allows generation to continue while checks run.Source: §3.1 · p. 4

3.2 Overthinking Compression

  • Overthinking compression first checks whether a thought contains a solution, then whether consecutive solution-bearing thought answers are equivalent.The method stops after M consecutive agreements, using compact binary verifier tasks to reduce overhead relative to full-sequence reward models.Source: §3.2 · pp. 4–5

3.3 Underthinking Compression

  • Underthinking is flagged when, after Rthres% of the token budget and Nthres rounds, the model lacks three consecutive answer-bearing chains agreeing on one solution.A guidance prompt then asks it to stop further reasoning and summarize existing insights; the thresholds must leave adequate opportunity to explore alternatives.Source: §3.3 · pp. 5–6

3.6 System Design

  • The system runs an inference engine alongside T3CS, which asynchronously verifies reasoning chains and returns flags for the sampling controller to adjust generation.The deployment description uses vLLM on Ascend NPUs and describes parallel decoding and state history to avoid redundant detokenization and API calls.Source: Figure 3 · p. 6, §3.6 · pp. 6–7

3.4 Repetition Truncation

  • A rolling-hash detector incrementally identifies repeating token-ID subsequences for real-time repetition checks.Source: §3.4 · p. 6

3.5 Guidance Prompts for LRMs Thinking Compression

  • The gentle prompt steers overthinking runs toward a final answer, while the forceful prompt targets underthinking and repetition before the think-end token.Repetition during summary generation is stopped by emitting an end-of-sequence token.Source: §3.5 · p. 6

4 Experiments

TrimR was evaluated on four reasoning benchmarks across multiple models, with substantial runtime reductions and little change in accuracy. Analysis examines compression components, threshold choices, token distributions, and verifier performance.

4.1 Experiment Setup

  • Experiments use four benchmarks, concurrent vLLM requests, and 8 Ascend 910B-64GB NPUs, with outputs capped at 30K tokens.The setup evaluates Pangu Pro MoE, QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and Pangu-R-38B.Source: §4.1 · p. 7

4.2 Main Results

  • Runtime falls by up to 70% across three models and four benchmarks, while accuracy drops by less than 1.7%.Table 1 covers QwQ-32B, Pangu-R-38B, and DeepSeek-R1-Distill-Qwen-32B; R1Q-32B runtime reductions range from 53% to 70%.Source: §4.2 · p. 7, Table 1 · p. 8
  • Pangu Pro MoE runtime falls by more than 25% on the mathematical datasets, except the eightfold-replicated AIME24 dataset.For GPQA Diamond, overthinking and underthinking were triggered only eight times; the paper attributes this to different multiple-choice reasoning patterns.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Analysis

  • Combining overthinking and underthinking compression reduces TPR and token use across models, with accuracy largely maintained or improved.On QwQ-32B/MATH500, the combined method cuts TPR by 15.8% and tokens by 14.3% while maintaining 96.8% accuracy; on R1Q-32B/MATH500, accuracy rises from 90.4% to 92.4%.Source: §4.3 · p. 9
  • On QwQ-32B/AIME24, increasing Rthres from 30% to 40% raises accuracy from 73.3% to 76.6%, and the default thresholds are Rthres = 50% and Nthres = 20.Accuracy degrades when Rthres is below 40%; accuracy plateaus beyond 50% while token count continues to grow.Source: §4.3 · pp. 9–10, Figure 4 · p. 9
  • TrimR shifts token counts toward shorter outputs, reducing the 20–32K-token share from roughly 6% to under 2%.The verifier comparison selects Pangu-7B: it has 87.87% annotation accuracy versus 86.70% for Qwen2.5-7B-Instruct, with MATH500 accuracy of 95.6% versus 95.0%.Source: §4.3 · p. 10

5 Conclusion

TrimR uses a lightweight pretrained verifier to truncate unnecessary intermediate reasoning, improving inference efficiency while maintaining accuracy in the reported benchmarks and large-batch deployments.

5 Conclusion

  • TrimR dynamically compresses reasoning in large reasoning models using a training-free, verifier-based framework to eliminate redundant thinking.The verifier truncates unnecessary intermediate steps.Source: §5 · p. 10
  • TrimR, a training-free verifier-based framework, reduces inference runtime by up to 70% on MATH500, AIME24/25, and GPQA across diverse dense and MoE models, particularly in large-batch industrial settings, without compromising accuracy.Source: §5 · p. 10
  • T3CS provides TrimR’s online deployment system integrated with Ascend NPUs and vLLM for high-throughput deployments.Source: §5 · p. 10

A Algorithm of Underthinking Compression

Underthinking compression relies on overthinking detection and stops further reasoning when a sequence cannot converge within the specified budgets.

A Algorithm of Underthinking Compression

B More Analysis of Thinking Length and Accuracy

The analysis links long, inaccurate R1Q-32B responses with underthinking and describes how decoding length relates to throughput and latency. The appendices specify reflection markers and verifier prompt criteria.

B More Analysis of Thinking Length and Accuracy

  • R1Q-32B accuracy falls from above 80%∼90% to 0% as decoding length increases, while its number of thoughts correlates linearly with generated tokens.The model is described as producing long, often wrong responses; repetition can lead to non-stopping 32K-token outputs.Source: B More Analysis of Thinking Length and… · p. 13, Figure 6 · p. 13

C Decoding throughput/latency over decoding length

  • Latency typically increases linearly with decoding length, and reducing unnecessary tokens can yield super-linear runtime reductions; Figure 7 presents throughput reduction as length increases.The passage identifies compatibility with inference infrastructures such as vLLM and SGLang as a deployment requirement.Source: C Decoding throughput/latency over… · p. 13, Figure 7 · p. 14

D Full List of Reflection Tokens

  • Reflection tokens are newline-prefixed markers—including “Wait,” “But,” and “Alternatively”—used to separate thoughts; plain “but” and “But” are excluded.The newline separator is described as a strong structural separator for DeepSeek R1 and QwQ-32B.Source: D Full List of Reflection Tokens · p. 14

E Detailed Prompts

  • The verifier prompts test whether reasoning content reaches a solution and whether two answers are logically or mathematically equivalent.The prompts request a Yes/No response without explanation.Source: E Detailed Prompts · pp. 14–15

F Overthinking/Underthinking examples

The examples contrast repeated verification after an answer is reached with reasoning that cycles among approaches and produces unsupported answers.

F Overthinking/Underthinking examples

G Detailed Experiment Setup

The evaluation spans mathematical, Olympiad, and expert-written science benchmarks, using concurrent batch decoding on Ascend hardware and a separate best-of-N selection setup.

G Detailed Experiment Setup

  • MATH500 covers 500 math problems, AIME24 and AIME25 provide Olympiad-style problems, and GPQA Diamond contains 198 expert-written science questions.MATH500 spans arithmetic, algebra, geometry, and calculus; GPQA Diamond covers biology, physics, and chemistry.Source: G Detailed Experiment Setup · p. 16
  • Experiments used eight Ascend 910B-64GB NPUs, 192-core Kunpeng-920 CPUs, 1.5TB memory, and up to 128 concurrent decoding batches.All dataset requests were submitted to vLLM concurrently; solutions were scored with the Qwen2.5 math evaluation tool and formatting was postprocessed.Source: G Detailed Experiment Setup · p. 16
  • For best-of-N experiments, Pangu-ORM selected the best solution, with an LRM-to-ORM ratio of 1:1, although production may use a higher ratio.Pangu-ORM is identified as a close-source Outcome Reward Model.Source: G Detailed Experiment Setup · p. 16

H Effectiveness of Repetition Truncation

Repetition truncation reduced inference costs in five GPQA Diamond runs, while the accompanying analysis links truncation to late-stage repetition and prompt guidance.

H Effectiveness of Repetition Truncation

  • 22.0% lower total runtime, 50.10% lower TPR, 20.7% lower token consumption, and a 1% accuracy gain accompanied repetition truncation.Table 6 covers five GPQA Diamond runs.Source: H Effectiveness of Repetition Truncation · pp. 16–17, Table 6 · p. 17
  • Table 6 reports the effects of repetition truncation across five GPQA Diamond runs.Source: Table 6 · p. 17
  • Most early terminations occurred in later reasoning stages, where the model became stuck in attention patterns and failed to generate diverse content.The guidance prompt mitigated infinite repetitive outputs by steering generation toward contextually relevant content grounded in prior analysis.Source: H Effectiveness of Repetition Truncation · p. 17

I Extended Analysis on Distribution

On QwQ-32B and MATH500, TrimR shifts token and reasoning-round distributions toward shorter outputs, with rare underthinking cases remaining comparatively longer.

I Extended Analysis on Distribution

  • After compression, over 80% of samples used fewer than 6,000 tokens, while over 85% completed within 30 reasoning rounds.The token result is a roughly 30% increase; the ≤20-round regime increased by nearly 20 percentage points.Source: Figure 8 · p. 17, I Extended Analysis on Distribution · p. 17
  • Overthinking samples shifted into lower token and round ranges, while underthinking samples—under 5% of cases—remained in mid-range lengths.The token tail beyond 15,000 was nearly eliminated; underthinking cases were truncated to 50–120 rounds.Source: I Extended Analysis on Distribution · p. 17
  • The authors report that TrimR preserves valid reasoning, curtails redundant overthinking, and minimally affects cases needing additional confirmation.They describe efficiency gains in both token and reasoning-round dimensions.Source: I Extended Analysis on Distribution · p. 18

J One Verifier Can Serve Multiple LRMs

A verifier can serve multiple LRM instances: measured capacity supports an ideal 14:1 request ratio, while production uses 8:1 for safety margins.

J One Verifier Can Serve Multiple LRMs

  • Each LRM averaged 9 requests per second, while the verifier processed about 128 requests per second, yielding an ideal 14:1 LRM-to-verifier ratio.Production uses a conservative 8:1 deployment ratio to maintain operational safety margins.Source: J One Verifier Can Serve Multiple LRMs · p. 18
  • After accounting for verifier infrastructure costs, the calculation reports 99.1% preserved performance gains per computational instance.The stated configuration uses eight Ascend 910-64GB units per LRM instance and one per verifier instance.Source: J One Verifier Can Serve Multiple LRMs · p. 18
  • TrimR reduces user waiting times, whereas scaling the cluster cannot further optimize token utilization or user-perceived TPR.Source: J One Verifier Can Serve Multiple LRMs · p. 18
Loading 2505.17155v2…