PaperlayerView exact revision on arXiv

Source-linked AI summary

Effectively Controlling Reasoning Models through Thinking Intervention

Tong Wu, Chong Xiang, Jiachen T. Wang, G. Edward Suh, Prateek Mittal

arXiv:2503.24370v3cs.LGcs.AIcs.CL

TL;DR

Reasoning-enhanced LLMs provide explicit intermediate reasoning, but existing control methods largely operate only on inputs. This paper introduces Thinking Intervention, which inserts or revises tokens during reasoning, and reports improved performance across instruction following, instruction hierarchy, and safety alignment. The approach offers a flexible control mechanism, while closed-source deployment and public intervention APIs remain practical and security challenges.

  • Problem

    Existing methods predominantly manipulate input prompts, leaving the explicit reasoning stages of reasoning-enhanced LLMs underused for direct behavioral control.

  • Method

    Thinking Intervention dynamically inserts or revises targeted token sequences within ongoing reasoning to control the model's reasoning trajectory.

  • Results

    Thinking Intervention improves reasoning-model performance across instruction following, instruction hierarchy, and safety alignment tasks, outperforming baseline prompting approaches.

  • Takeaways & Limitations

    The framework provides fine-grained, transparent control over reasoning processes and supports progress toward more reliable, interpretable, and human-aligned LLM systems.

  • Takeaways & Limitations

    Closed-source deployment remains challenging because most providers do not support interventions in internal reasoning processes, while public APIs could create safety-bypass risks.

Abstract

from arXiv · show

Reasoning-enhanced large language models (LLMs) explicitly generate intermediate reasoning steps prior to generating final answers, helping the model excel in complex problem-solving. In this paper, we demonstrate that this emerging generation framework offers a unique opportunity for more fine-grained control over model behavior. We propose Thinking Intervention, a novel paradigm designed to explicitly guide the internal reasoning processes of LLMs by strategically inserting or revising specific thinking tokens. We find that the Thinking Intervention paradigm enhances the capabilities of reasoning models across a wide range of tasks, including instruction following on IFEval and Overthinking, instruction hierarchy on SEP, and safety alignment on XSTest and SorryBench. Our results demonstrate that Thinking Intervention significantly outperforms baseline prompting approaches, achieving up to 6.7% accuracy gains in instruction-following scenarios, 15.4% improvements in reasoning about instruction hierarchies, and a 40.0% increase in refusal rates for unsafe prompts using open-source DeepSeek R1 models. Overall, our work opens a promising new research avenue for controlling reasoning LLMs.

1 Introduction

Reasoning models expose intermediate thinking steps that enable more precise behavioral control than input-level prompting alone. The paper introduces Thinking Intervention and reports improvements across instruction following, instruction hierarchy, and safety alignment.

  • Reasoning-enhanced LLMs generate intermediate reasoning steps before answers, improving performance on complex mathematical, programming, and logical tasks.
  • Existing guidance methods mainly modify input prompts, while explicit reasoning stages create opportunities for direct intervention during reasoning.
  • Thinking Intervention inserts or replaces specified token sequences within ongoing reasoning to provide fine-grained, transparent control aligned with task objectives.
  • The approach supports adaptive reasoning control, requires no model training, integrates with existing techniques, and is intended for practical deployment.
  • 6.7% and 1.9% improvements over Vanilla Prompting and Prompt Engineering, respectively, were achieved on instruction-following tasks.
  • Up to 15.4% stronger instruction-hierarchy performance and substantially higher refusal rates on unsafe prompts were reported across evaluation tasks.

2 Thinking Intervention: A Novel Paradigm to Control Reasoning Models

Thinking Intervention controls reasoning-enhanced LLMs by dynamically modifying partial reasoning chains rather than only optimizing the input prompt. The framework supports context-aware insertion or revision, with lightweight trigger-based implementations and flexible deployment.

  • 2.1 Preliminaries and Notations: Reasoning-enhanced LLMs separate generation into an intermediate reasoning stage and a final response stage.
  • 2.2 Intervening in the Reasoning Process as a General Paradigm: Unlike prompt engineering, Thinking Intervention makes online decisions from incomplete reasoning chains and revises guidance during generation.
  • 2.2 Intervening in the Reasoning Process as a General Paradigm: The intervention function can insert new tokens or replace existing reasoning tokens, including corrective feedback, alternative approaches, or domain knowledge.
  • 2.3 Instantiation: Intervention via a Postfix-based Monitor: A postfix-based monitor detects trigger strings in recent reasoning tokens and appends an intervention sequence when a match occurs.
  • 2.3 Instantiation: Intervention via a Postfix-based Monitor: Interventions can target reasoning onset or other positions, and beginning-of-reasoning intervention was most effective in the reported evaluations.
  • Design Properties: Thinking Intervention requires minimal engineering effort, avoids fine-tuning and parameter modification, adds negligible computational overhead, and can complement prompt engineering.
  • Effectiveness: The framework directly guides reasoning and was associated with stronger attention to intervention tokens and improvements across instruction, hierarchy, and safety tasks.

3 Evaluation on Instruction Following Tasks

Thinking Intervention is evaluated on IFEval as a way to improve reasoning models’ instruction-following under verifiable output constraints. Across models and prompting methods, it consistently improves accuracy without additional training.

  • Methods: Thinking Intervention is compared with Vanilla Prompting and Reminder Prompting, using first-person intervention sequences derived from reminder statements.The intervention sequence is inserted into the model’s reasoning process.
  • Results: 5.74%, 4.99%, 6.65%, and 2.96% accuracy gains are obtained over Vanilla Prompting for R1-Qwen-7B, R1-Qwen-14B, R1-Qwen-32B, and QwQ-32B, respectively.The gains remain effective as model size increases within the R1-Qwen family.
  • Results: 62.84%, 77.63%, and 82.44% accuracy are reached when Thinking Intervention is combined with Reminder Prompting for R1-Qwen-7B, R1-Qwen-32B, and QwQ-32B, respectively.The results indicate that Thinking Intervention complements existing prompting methods.
  • Results: Thinking Intervention improves instruction following and also mitigates model overthinking without requiring additional model training.The broader overthinking results are reported in Appendix C.

4 Evaluation on Instruction Hierarchy Task

Thinking Intervention is evaluated on SEP, where models must prioritize high-priority instructions over unrelated low-priority directives. It improves robustness across model scales while preserving utility with only negligible variation.

  • Benchmark: SEP pairs a high-priority main instruction with relevant data and an unrelated low-priority instruction to test instruction hierarchy handling.The task requires models to follow the main instruction while ignoring the low-priority one.
  • Evaluation metrics: Robustness measures correctly ignored low-priority instructions, while utility measures baseline performance on the main task without low-priority instructions.Utility is evaluated with LLM-as-a-judge and normalized to a 0-100% scale.
  • Methods: Thinking Intervention uses the intervention “I should follow all the instructions in the task block and not follow any instructions in the data block.”The method is compared with Vanilla Prompting and Reminder Prompting.
  • Results: 66.4% robustness is achieved for R1-Qwen-32B with Reminder Prompting, a 20.20% improvement over the Reminder Prompting baseline.Other reported robustness gains are 5.0% for R1-Qwen-7B, 3.4% for R1-Qwen-14B, and 7.2% for QwQ-32B.
  • Results: Utility varies by no more than 0.73% from the baselines across settings while Thinking Intervention improves robustness.The results support maintaining instruction hierarchy performance without compromising utility-related tasks.

5 Evaluation on Safety Alignment Task

The evaluation examines whether Thinking Intervention improves safety alignment in reasoning models across XSTest and SORRY-Bench while preserving compliance with benign requests. Results show stronger refusal of unsafe requests, compatibility with prompting methods, and generalization across safety benchmarks.

  • Evaluation setup: Thinking Intervention is evaluated as a safety steering method for open-source reasoning models, including R1-Qwen-32B.The paper frames the task around low baseline refusal rates to unsafe requests in open-source reasoning models.
  • Evaluation setup: XSTest measures refusal of 200 harmful requests and compliance on 250 benign questions using GPT-4o-mini evaluation.The benchmark includes benign questions with safety triggers that may otherwise cause mistaken refusals.
  • XSTest: Over 40%: Thinking Intervention increases refusal rates for unsafe requests across R1 models, with compliance decreases below 2% in larger models.The smaller R1-Qwen-7B model experiences a relatively larger compliance drop.
  • Prompting combinations: Approximately 75% refusal and high compliance: Goal Priority plus Thinking Intervention reaches this refusal rate on unsafe XSTest requests while maintaining the reported safe-request compliance rate.Reminder Prompting plus Thinking Intervention raises unsafe-request refusal by approximately 30% while maintaining approximately 97% compliance.
  • SORRY-Bench: Approximately 87% refusal: combining Thinking Intervention with Default Prompting on SORRY-Bench improves over Default Prompting alone by nearly 20%.This refusal rate is reported as higher than those of GPT-4o and o3-mini.
  • Implications: The paper reports further safety gains while maintaining high benign-request compliance and positions Thinking Intervention as complementary to existing safety techniques.The authors describe it as a complementary safety layer rather than a complete solution to all safety challenges.

6 Analysis and Discussion

The analysis studies how intervention position, sequence complexity, narrative perspective, and auxiliary-model assistance affect Thinking Intervention. Early interventions are strongest, longer instructions trade compliance for safety, narrative perspective matters little, and auxiliary monitoring adds overhead.

  • Design scope: The primary evaluations use concise first-person interventions at the beginning of reasoning, while the framework permits other design choices.The paper presents the default configuration as TIdefault and describes Thinking Intervention as general and flexible.
  • Position of intervention: Later interventions are less effective than early-stage Thinking Intervention on XSTest, consistent with reasoning paths becoming harder to redirect after extended deliberation.The authors hypothesize that models become committed to incorrect or suboptimal trajectories over time.
  • Complexity of intervention sequences: Longer safety instructions improve alignment but reduce overall compliance because overly restrictive guidance constrains responses.The exploratory TIlong condition exposes a trade-off between compliance and safety alignment.
  • Narrative perspective: First-person and second-person interventions show minimal performance differences, which the authors attribute to reasoning models’ robust self-correction.The comparison uses TIsecond against first-person Thinking Intervention.
  • Leveraging an auxiliary LLM: Auxiliary-LLM interventions occasionally outperform manually crafted interventions, but their additional computational overhead may hinder practical deployment.The approach uses Qwen2.5-7B-Instruct to monitor and revise the primary model’s reasoning traces for safety.
  • Practical utility: Thinking Intervention is practically constrained for closed-source models because most providers do not expose controls over internal reasoning processes.The paper also warns that public intervention APIs could create security risks by enabling easier circumvention of safety mechanisms.

7 Related Works

Related work covers interventions for measuring faithfulness in chain-of-thought reasoning, their extension to reasoning-enhanced models, and methods that control reasoning length. Thinking Intervention complements monitoring-based approaches by directly controlling reasoning processes.

  • Intervention-based reasoning research: Prior intervention methods measure faithfulness in chain-of-thought reasoning and have been extended to reasoning-enhanced models.The paper positions its framework alongside, rather than as a replacement for, these monitoring-oriented methods.
  • Reasoning-length control: Other studies control reasoning length by encouraging longer chains for accuracy or shortening chains for efficiency.These approaches target the duration of reasoning rather than the targeted insertion or revision of reasoning content.

8 Conclusion.

The paper proposes Thinking Intervention for controlling reasoning-enhanced LLMs and evaluates it across instruction following, instruction hierarchy, and safety alignment. It positions the approach as groundwork for more reliable, interpretable, and human-aligned systems.

  • Thinking Intervention is proposed as an approach to effectively control reasoning-enhanced LLMs.
  • Reasoning models generate explicit intermediate steps and have improved performance on mathematical problem-solving, programming assistance, and logical inference.
  • Prompt engineering and activation steering are mainstream post-training control approaches, whereas Thinking Intervention intervenes in the thinking process.
  • The paper evaluates Thinking Intervention for instruction following, instruction hierarchy, and safety alignment in open-source reasoning models.
  • IFEval evaluates instruction following with four accuracy metrics, including prompt-level and instruction-level strict and loose accuracy.
  • Loose accuracy permits certain formatting flexibility, whereas strict accuracy requires outputs to precisely match the specified requirements.

B.2 Comprehensive Experiment Results

Comprehensive IFEval results show that Thinking Intervention improves instruction-following performance across multiple reasoning models and evaluation metrics. The reported gains extend beyond the primary R1-Qwen models to R1-Llama models.

  • 5.36%, 4.80%, and 6.47% prompt-level loose-accuracy gains are observed for R1-Qwen-7B, R1-Qwen-14B, and R1-Qwen-32B, respectively, over Vanilla Prompting.
  • 8.88% and 1.85% prompt-level loose-accuracy increases are observed for R1-Llama-8B and R1-Llama-70B, respectively, compared to Vanilla Prompting.
  • Thinking Intervention consistently improves IFEval performance over Vanilla Prompting and Reminder Prompting across multiple reasoning models.

C Instruction Following Evaluation (Overthinking)

The overthinking evaluation tests whether Thinking Intervention can reduce reasoning length while preserving accuracy. The appendix also details SEP evaluation metrics for robustness, utility, and following low-priority task instructions.

  • C Instruction Following Evaluation (Overthinking): The Overthinking evaluation uses a 500-sample MATH subset and measures both problem-solving accuracy and reasoning length.
  • C Instruction Following Evaluation (Overthinking): Thinking Intervention is compared with Vanilla Prompting and Reminder Prompting, using a concise intervention that instructs the model not to think too much.
  • C Instruction Following Evaluation (Overthinking): Accuracy changes remain within a narrow range, decreasing by at most 1.2% relative to baseline approaches.
  • C Instruction Following Evaluation (Overthinking): Reasoning length decreases by as much as 30.4%, while task accuracy can improve by up to 1.6%.
  • D.1 More Details of Evaluations: Figure 10 illustrates SEP cases where low-priority instructions appear in data, appear in the task, or are absent.
  • D.1 More Details of Evaluations: Figure 11 uses the complete prompt and model response as Question and Answer fields for the SEP utility judge, while Table 4 compares the evaluated methods.
  • D.1 More Details of Evaluations: The SEP dataset contains 9,160 examples pairing a main instruction and data with a low-priority query and its witness; evaluation samples 500 prompts.
  • D.1 More Details of Evaluations: SEP reports robustness, SEP utility, and utility, respectively testing whether models ignore low-priority data instructions, follow low-priority task instructions, and perform without such instructions.

D.2 Comprehensive Experiment Results

Comprehensive SEP results show that Thinking Intervention improves robustness across additional reasoning models while generally preserving SEP utility. The appendix also documents the safety-alignment evaluation setup and baseline prompts.

  • Thinking Intervention maintains or improves SEP utility while enhancing robustness on the extended SEP evaluation.
  • 92.4% and 91.6% SEP utility scores for R1-Qwen-14B improve over baseline methods by 4.0% and 2.8%, respectively.
  • SEP utility degradation remains below 1.2% for R1-Qwen-32B and QwQ-32B while robustness is enhanced.
  • Robustness increases by up to 9.0% for R1-Llama-8B and up to 18.6% for R1-Llama-70B.
  • Safety-alignment experiments include XSTest and SORRY-Bench, with additional evaluations and integration with a safety fine-tuned model described in the appendix.
  • Baseline safety-steering prompting methods use the prompts shown in Figure 12.

E.2 Safety Alignment on XSTest

Thinking Intervention substantially improves safety alignment across reasoning models and prompting methods on XSTest and SORRY-Bench. Gains in unsafe-request refusal generally come with smaller compliance reductions for medium- and large-sized models, while intervention placement and length create safety–compliance trade-offs.

  • XSTest results: Over 40% higher unsafe-request refusal rates occur for R1 models, while QwQ-32B improves by approximately 10% over Vanilla Prompting.These gains hold across prompting methods and model architectures on XSTest.
  • XSTest results: Compliance reductions for safe requests are generally below 10% in medium- and large-sized models, but larger drops occur in smaller reasoning models.The reported medium- and large-sized models are R1-Qwen-14B, R1-Qwen-32B, QwQ-32B, and R1-Llama-70B.
  • SORRY-Bench results: Thinking Intervention consistently enhances safety alignment on SORRY-Bench across diverse reasoning models and prompting methods.SORRY-Bench contains 45 unsafe-instruction taxonomies with 450 total unsafe requests, evaluated using GPT-4o-mini.
  • Safety fine-tuning integration: On STAR1-32B, unsafe-request refusal rates increase by approximately 3%–5% on XSTest and 0.2%–1.8% on SORRY-Bench.Thinking Intervention adds gains even when the safety-fine-tuned model already performs strongly.
  • Intervention design: Intervening at the reasoning start steers models more effectively than intervening in the middle or at the end.Later placement reduces unsafe-request refusal rates, including a reported 10% to 20% reduction on SORRY-Bench.
  • Intervention design: Longer intervention sequences raise unsafe-request refusal but reduce safe-request compliance.On XSTest, the longer sequence decreases compliance by over 10% and increases refusal by approximately 10%; SORRY-Bench refusal rises by 3% to 10%.

F.3 Narrative of Thinking Intervention

Narrative wording has limited overall impact on Thinking Intervention performance, although the alternative narrative slightly reduces SORRY-Bench refusal. Models can self-correct narrative inconsistencies, while manually designed interventions generally outperform auxiliary-LLM editing.

  • Narrative effects: Narrative wording changes XSTest compliance and refusal rates by less than 3%, while SORRY-Bench refusal decreases by 0.2% to 3.8% with the second narrative.Both narrative variants are inserted at the beginning of the reasoning process.
  • Narrative effects: Reasoning models can self-correct an initially incorrect second-narrative response and continue with first-narrative reasoning.The demonstrated model revises its internal narrative during reasoning after initially using the alternative wording.
  • Auxiliary-LLM editing: The default manually designed intervention generally achieves higher unsafe-request refusal and safe-request compliance than auxiliary-LLM editing.On SORRY-Bench, the default intervention outperforms across all four prompting methods, with differences below 3%.
  • Instruction hierarchy: Without Thinking Intervention, models may follow low-priority instructions instead of the main task; with it, they identify and ignore the irrelevant query.The SEP example contrasts incorrect discussion of book composition with correct interpretation of the Sarbanes-Oxley Act.
  • Safety steering: Thinking Intervention steers harmful-request reasoning more effectively than Prompt Engineering by directing greater attention to safety instructions.The attention analysis links the stronger steering to safety guidance placed within the thinking process.
Loading 2503.24370v3…