PaperlayerView exact revision on arXiv

Source-linked AI summary

SEAL: Steerable Reasoning Calibration of Large Language Models for Free

Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, Zhangyang Wang

arXiv:2504.07986v3cs.CLcs.AI

TL;DR

Extended chain-of-thought reasoning can incur redundant computation and harm performance through unnecessary reflection and transitions. SEAL calibrates these reasoning paths by extracting a steering vector offline and intervening in latent space during decoding, improving accuracy while reducing token use across tasks and models.

  • Problem

    Extended reasoning traces create inference overhead, and redundant reflection, transition, and verification steps can degrade answer quality.

  • Method

    SEAL is a training-free two-stage method that extracts a reasoning steering vector offline and adjusts hidden states during decoding.

  • Results

    Across diverse models and benchmarks, SEAL improves accuracy by up to 11% while reducing reasoning tokens by 11.8% to 50.4%.

  • Takeaways & Limitations

    Latent-space calibration provides a lightweight, interpretable way to promote execution and reduce less productive reasoning without fine-tuning or architectural changes.

Abstract

from arXiv · show

Large Language Models (LLMs), such as OpenAI's o1-series have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism. However, recent studies reveal substantial redundancy in the CoT reasoning traces, which not only increases inference latency but also negatively impacts model performance by diverting attention to unnecessary reasoning paths. To address this issue, we investigate the internal reasoning structures of LLMs and categorize them into three primary thought types: execution, reflection, and transition thoughts. Moreover, our analysis reveals that excessive reflection and transition thoughts are strongly correlated with failure cases and these thought categories exhibit clear separation in the latent space. Based on these, we introduce SEAL (Steerable reasoning calibration), a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains. SEAL consists of an offline stage for extracting the reasoning steering vector in the latent space, followed by an on-the-fly calibration of the reasoning trace through representation intervention using the steering vector. Notably, the steering vector exhibits strong transferability across various tasks. Extensive experiments across multiple models (DeepSeek-R1-Distill and QwQ-32B-Preview) and benchmarks (Math500, GSM8K, LiveCodeBench) validate the effectiveness of SEAL, up to a 11% improvement in accuracy while reducing reasoning tokens by 11.8% to 50.4%. Our code is publicly available at https://github.com/VITA-Group/SEAL.

1 Introduction

Extended chain-of-thought helps LLMs handle complex reasoning, but redundant reasoning increases inference overhead and can hurt performance. SEAL identifies problematic reasoning patterns and calibrates them without training, improving accuracy while reducing token use.

  • Extended chain-of-thought reasoning enables LLMs to tackle complex tasks such as mathematics, planning, and code debugging.
  • Redundant thought sequences increase memory-bound inference costs and can degrade performance through verification loops or unnecessary reasoning detours.
  • SEAL categorizes reasoning into execution, reflection, and transition thoughts, whose excess is associated with failed answers and whose representations separate in deep latent space.
  • SEAL extracts a reasoning steering vector offline from about one hundred training samples and adjusts hidden states during inference for training-free calibration.
  • SEAL improves accuracy by up to 11% while saving 11.8% to 50.4% of reasoning tokens across multiple models and benchmarks.

2 Recognizing Reasoning Patterns in LLMs

The paper decomposes responses into consecutive thought blocks and identifies execution, reflection, and transition patterns. Incorrect samples contain more thoughts, largely because reflection and transition steps trigger additional execution and reasoning detours.

  • Responses are segmented into interconnected thoughts using double line breaks, represented as O = (T1, T2, ..., TN).
  • Figure 2 compares thought counts and numerical response lengths between correctly and incorrectly answered Math-500 samples.
  • Incorrect samples contain more thoughts, driven largely by increased reflection and transition thoughts followed by several execution steps.
  • The analysis motivates controlling reflection and transition thoughts to improve reasoning efficiency and effectiveness.

3 Different Reasoning Patterns are Distinguishable in the Latent Space

The study addresses variation in the tokens used for similar thought types by analyzing their layer-wise representations. It uses thought-boundary hidden states and t-SNE to examine latent-space structure.

  • Tokens vary substantially across reflection thoughts, making token-level analysis difficult even within one category.
  • Hidden representations at thought boundaries are collected from transformer layers and projected with t-SNE to analyze conceptual behavior.

4 Steerable Reasoning Calibration

SEAL uses a two-stage, training-free framework: it extracts a reasoning steering vector offline and applies latent-space intervention during decoding. The intervention targets redundant reflection and transition thoughts while preserving execution.

  • 4 Steerable Reasoning Calibration: SEAL combines offline steering-vector extraction with on-the-fly latent-space intervention during decoding.
  • 4.1 Extraction of Reasoning Steering Vector: The extraction process generates and segments reasoning samples, then classifies thoughts as execution, reflection, or transition.
  • 4.1 Extraction of Reasoning Steering Vector: The steering vector is computed from category-specific thought representations to suppress unnecessary reflection and transition while preserving execution.
  • 4.2 Decoding with Latent Space Intervention: During decoding, SEAL updates hidden states as eH = H + α · S, with α controlling intervention strength.
  • 4.2 Decoding with Latent Space Intervention: Latent-space intervention adds negligible computational cost compared with the original forward pass, while offline extraction adds no decoding latency.
  • 4.2 Decoding with Latent Space Intervention: The extracted validation-set steering generalizes across tasks, including transfer from mathematical reasoning to code generation.

5 Experiments

Experiments show that SEAL improves accuracy and reduces reasoning cost across models and tasks, including when its steering vector transfers beyond Math500. Ablations indicate that targeting reflection and transition thoughts, especially in mid-to-late layers with balanced strength, is most effective.

  • Main Results: SEAL improves accuracy while reducing response length across DeepSeek-R1-Distill and QwQ-32B-Preview experiments.The evaluation covers Math500 and generalization across GSM8K and LiveCodeBench.
  • Main Results: 14.1% accuracy improvement accompanies a 28.8% token reduction on hard Math-500 problems.The authors attribute this to avoiding excessive rechecking and unnecessary transition thoughts.
  • Generalization: 0.3% to 10.0% accuracy gains and up to 50.4% lower token usage persist when transferring the Math-derived steering vector across datasets.The transfer includes GSM8K within the same domain and LiveCodeBench in a different domain.
  • Comparison with Logits Penalty: Latent-space steering reduces reworded reflection and transition steps, whereas logits penalties increase those steps despite lowering representative token logits.SEAL suppresses the broader concepts rather than only selected words such as “wait” or “alternatively.”
  • Efficiency: 32.9% to 37.9% average response-time reductions are achieved in end-to-end Math500 evaluation, with 83.65% to 86.61% reductions for the most improved sample.The added steering computation is described as negligible, with a slight throughput improvement of approximately 2 tokens per second.
  • Steering Type Ablation: Weakening execution thoughts degrades performance, while weakening reflection and transition thoughts individually helps and weakening both yields the largest improvement.The resulting formulation is S = HE −HRT.
  • Layer Ablation: Mid-to-late layers provide the best steering performance and generalize strongly across models and tasks.The authors relate this to later layers carrying more conceptual rather than token-level representations.
  • Strength Ablation: A steering coefficient of α = 1.0 performs consistently strongly, while minimizing reflection and transition thoughts does not always improve results.The findings support retaining a balanced number of these reasoning steps.

6 Conclusion

The paper identifies execution, reflection, and transition thoughts as separable reasoning components and introduces SEAL to steer their balance during decoding. SEAL is training-free, reduces less productive reasoning while improving efficiency and accuracy without fine-tuning or architectural changes.

  • 6 Conclusion: SEAL calibrates reasoning by steering latent representations to promote execution and reduce excessive reflection and transition thoughts.It uses a precomputed steering vector and on-the-fly hidden-state intervention during decoding.
  • 6 Conclusion: Execution, reflection, and transition thoughts are highly distinguishable in the latent space.This separability supports more precise control over reasoning trajectories.
  • 6 Conclusion: Excessive reflection and transition thoughts are associated with inefficiencies and errors in extended chain-of-thought reasoning.Related work characterizes overthinking as excessive computation and underthinking as switching without sufficiently exploring promising paths.
  • 6 Conclusion: SEAL improves reasoning efficiency and accuracy without requiring model fine-tuning or architectural modifications.Experiments across various LLMs and benchmarks support its robustness and generality.

B.2 Ablation on different criteria

The ablation study evaluates prefix-based and phrase-based criteria for extracting reflection and transition thoughts. Either subset can still produce competitive steering results, suggesting that rigidly designing all patterns is unnecessary.

  • B.2 Ablation on different criteria: Using only the prefix rule or only the phrase rule still achieves competitive results in steering.The ablation compares each rule subset for extracting reflection and transition thoughts.
  • B.2 Ablation on different criteria: Capturing key concept-level information about reflection and transition thoughts is sufficient without overly rigid pattern designs.This conclusion follows from the comparable performance of rule subsets.

B.3 LLM-Based Labeling and Comparison with Heuristic Method

The paper compares GPT-4o-based thought labeling with heuristic keyword rules for classifying execution, reflection, and transition thoughts. The keyword-based method performs slightly better for SEAL, while LLM-based labeling remains easier to adapt when heuristics are difficult to define or scale.

  • B.3 LLM-Based Labeling and Comparison with Heuristic Method: Keyword-based labeling achieves slightly better end-task performance in SEAL than GPT-4o-based labeling.GPT-4o was prompted with the full reasoning context and current step to assign one of three thought types.
  • B.3 LLM-Based Labeling and Comparison with Heuristic Method: LLM-based labeling remains appealing for adapting thought categories to new domains, languages, or expanded taxonomies.It may be especially useful where heuristic rules are difficult to define or scale.

C SEAL Performance on In-Distribution Steering Vector

The paper tests steering vectors extracted from in-distribution samples on GSM8K and LiveCodeBench. These vectors remain effective but underperform vectors derived from MATH500, supporting the use of a well-curated, diverse dataset for generalizable steering.

  • C SEAL Performance on In-Distribution Steering Vector: GSM8K and LiveCodeBench use in-distribution samples to construct steering vectors for their evaluations.The procedure selects 1000 GSM8K training samples and 480 LiveCodeBench difference-set samples.
  • C SEAL Performance on In-Distribution Steering Vector: Steering vectors derived from in-distribution data are effective but perform worse than those derived from MATH500.The comparison is reported for in-distribution steering experiments.
  • C SEAL Performance on In-Distribution Steering Vector: The authors attribute the performance discrepancy to the relative homogeneity of GSM8K and LiveCodeBench compared with MATH500’s broader difficulty levels and problem types.They argue that broader coverage makes it easier to obtain a generalized steering vector.
  • C SEAL Performance on In-Distribution Steering Vector: The results support that extracting a new steering vector for every task is unnecessary when using a well-curated, high-quality dataset.The authors present this as evidence for a generalizable steering vector.

D Generalization to Open-Ended Planning Tasks

SEAL generalizes beyond mathematics and programming to NaturalPlan, an open-ended natural-language calendar-planning benchmark. Reusing a steering vector extracted from MATH, it improves accuracy and reduces response length relative to baseline.

  • D Generalization to Open-Ended Planning Tasks: SEAL was evaluated on NaturalPlan, an open-ended benchmark for natural-language calendar planning.The experiment tests generalizability beyond structured mathematics and programming domains.
  • D Generalization to Open-Ended Planning Tasks: Reusing the reasoning steering vector extracted from MATH, SEAL consistently improved accuracy and reduced response length versus baseline.The results indicate that reflection and transition overuse also appears in open-ended planning contexts.
  • D Generalization to Open-Ended Planning Tasks: Table 9 reports NaturalPlan calendar-planning results using Deepseek-R1-Distill-7B.

E Results with Sampling-Based Decoding

Under sampling-based decoding, SEAL improves performance while reducing token usage, and its gains exceed those from simply truncating reasoning. The results attribute the benefit to suppressing non-productive reflection and transition thoughts rather than shortening outputs alone.

  • E Results with Sampling-Based Decoding: SEAL achieves improved performance and reduced token usage under sampling-based decoding.Sampling uses temperature t = 0.6 and top p = 0.95, with results averaged over three runs.
  • E Results with Sampling-Based Decoding: SEAL significantly reduces sequence length for incorrectly answered samples by disrupting unnecessary reflection and transition thoughts.It also slightly reduces redundant tokens for correctly answered samples.
  • E Results with Sampling-Based Decoding: A fixed think budget truncation baseline limits reasoning to 3500 tokens, approximately matching SEAL’s average output length.The model inserts an </think> token after the preset budget and then summarizes the answer.
  • E Results with Sampling-Based Decoding: The truncation baseline underperforms SEAL despite producing similarly shorter outputs, showing that SEAL’s advantage is not sequence shortening alone.SEAL instead suppresses specific non-productive reasoning types and produces more accurate reasoning trajectories.
  • E Results with Sampling-Based Decoding: Repeated verification and thought switching can loop a model away from a correct answer, even after it has reached that answer multiple times.Figure 9 illustrates this failure pattern for DeepSeek-R1-Distill-Qwen-7B on a MATH-500 example.
Loading 2504.07986v3…