PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Chain-of-Thought Steering for Efficient and Controllable LLM Reasoning

Yu Xia, Zhouhang Xie, Xin Xu, Byungkyu Kang, Prarit Lamba, Xiang Gao, Julian McAuley

arXiv:2606.03965v1cs.CLcs.AI

TL;DR

Extended chain-of-thought can improve accuracy but is costly and difficult to control because existing efficiency methods largely constrain length rather than reasoning behavior. ACTS uses a budget-aware controller to steer a frozen reasoner’s strategy step by step, and experiments show full-thinking-level performance with substantial token savings and controllable accuracy-efficiency trade-offs.

  • Problem

    Extended chain-of-thought improves final-answer accuracy but can use tokens inefficiently, while existing efficiency methods leave the model’s reasoning strategy largely implicit.

  • Method

    ACTS formulates reasoning steering as an MDP where a controller observes reasoning history and remaining budget, then emits a strategy and steering phrase for a frozen reasoner.

  • Results

    ACTS matches or outperforms full-thinking baselines with substantial token savings and enables controllable accuracy-efficiency trade-offs across reasoners and task domains.

  • Takeaways & Limitations

    Strategy-level steering provides a complementary control surface for efficient reasoning beyond global length control.

  • Takeaways & Limitations

    Experiments cover three reasoners up to 8B parameters, leaving validation on substantially larger or frontier proprietary reasoners for future work.

Abstract

from arXiv · show

Large language models improve final-answer accuracy through extended chain-of-thought reasoning, but often spend tokens inefficiently and offer little inference-time control. Existing efficient reasoning methods control thinking length by shortening, early-stopping, or compressing traces, leaving how the model thinks implicit. In this paper, we propose Agentic Chain-of-Thought Steering (ACTS), which formulates reasoning steering as a Markov decision process where a controller agent adaptively steers a frozen reasoner during inference. At each step, the controller observes the reasoning trace and remaining thinking budget, then issues a steering action consisting of a reasoning strategy and a steering phrase that initiates the next reasoner step. This enables budget-aware strategy control for efficient reasoning while preserving the reasoner's generation continuity. We initialize the controller agent from our constructed synthetic steering trajectories with multi-budget augmentation, and further optimize it via reinforcement learning with budget-conditioned reward shaping. Experiments across multiple benchmarks show that ACTS matches full-thinking performance with substantial token savings, and enables controllable accuracy-efficiency trade-offs across different reasoners and tasks. The code is available at https://github.com/Andree-9/ACTS.

1 Introduction

ACTS steers a frozen reasoner’s stepwise strategy under a thinking-token budget, addressing the implicit reasoning behavior left uncontrolled by length-focused methods. It is initialized from synthetic multi-budget trajectories and optimized with budget-conditioned reinforcement learning, achieving full-thinking-level accuracy with substantial token savings.

  • Motivation: Length-focused efficient reasoning methods shorten, compress, early-stop, or budget-force traces but leave the underlying reasoning strategy implicit.The paper identifies planning, execution, reflection, and conclusion as functional transitions that motivate behavior-level control.
  • ACTS framework: ACTS formulates budgeted reasoning as an MDP in which a controller steers a frozen reasoner step by step.The controller observes steering history, the current reasoning step, and remaining budget.
  • ACTS framework: Each ACTS action combines a high-level reasoning strategy with a short natural-language steering phrase that initiates the next reasoner step.This provides in-flight, reasoner-agnostic strategy control while preserving continuity of the reasoner’s thinking process.
  • Training: The controller is trained through synthetic steering trajectories with multi-budget augmentation, supervised initialization, and budget-conditioned reinforcement learning.The reward shaping penalizes overusing the budget for correct answers and premature termination for incorrect answers.
  • Results: ACTS matches full-thinking baselines with substantial token savings across multiple benchmarks and supports controllable accuracy-efficiency trade-offs across reasoners and tasks.The paper presents this as the principal empirical outcome of the framework.

2 Related Work

Prior efficient-reasoning methods primarily control how long models think, while other work directly steers reasoning behavior through prompts, activations, or controller models. ACTS is positioned as complementary because it explicitly steers strategy at each reasoning step while allowing budget compliance to arise from learned completion.

  • Length control: Prompt-level, tuning-based, and reinforcement-learning methods reduce reasoning cost mainly by controlling chain-of-thought length.These approaches include brevity instructions, supervised trace compression, and length-aware rewards.
  • ACTS positioning: ACTS complements length-control methods by steering explicit reasoning strategies and phrases at each step, with budget compliance arising from learned completion rather than enforced length.Its control surface targets how the model thinks while retaining budget-aware reasoning.
  • Behavior steering: Other approaches intervene in reasoning behavior through strategic prompts or hidden-representation activation steering.The cited behaviors include reflection and backtracking.
  • Controller-based methods: Explicit controller and meta-cognitive frameworks select reasoning structures or adapt strategies during inference.Analyses of recurring functional structure in long traces motivate these control approaches.

3 Agentic Chain-of-Thought Steering

ACTS steers a frozen reasoner through budget-aware strategy and phrase selection, using an MDP controller trained from synthetic trajectories and reinforcement learning. Its design covers controller initialization, reward shaping, and asynchronous deployment.

  • 3.1 Reasoning Steering as an MDP: ACTS formulates budgeted reasoning steering as an MDP in which a controller guides a frozen reasoner step by step.The controller observes steering history, reasoning steps, and remaining budget while the reasoner generates continuations.
  • 3.1 Reasoning Steering as an MDP: Each controller action combines a predefined reasoning strategy with a natural-language steering phrase that primes the reasoner’s next step.The strategy set includes UNDERSTAND, PLAN, EXECUTE, EXPLORE, CHECK, SUMMARIZE, and CONCLUDE.
  • 3.1 Reasoning Steering as an MDP: The reasoner updates the remaining-budget fraction after each generated step, and episodes terminate on CONCLUDE, an end-of-thinking token, or a maximum step count.The trajectory ends with answer generation conditioned on the completed thinking trace.
  • 3.2 Controller Behavior Initialization: The controller is initialized from synthetic trajectories created by segmenting expert traces, annotating strategies, extracting opening phrases, and applying multi-budget augmentation.The corpus uses OpenR1-Math traces produced by DeepSeek-R1 and removes consecutive repeated strategies or phrases.
  • 3.2 Controller Behavior Initialization: The synthetic corpus exposes temporal strategy structure: UNDERSTAND and PLAN open traces, EXECUTE spans the middle, CHECK rises later, and SUMMARIZE and CONCLUDE dominate near exhaustion.Behavior initialization transfers this distribution into the controller as a prior over when strategies should fire under a given budget.
  • 3.3 Online Reinforcement Learning: Budget-conditioned reward shaping penalizes both correct answers produced after overspending and incorrect answers produced while leaving budget underused.A 10% grace margin around zero remaining budget prevents minor boundary overshoots from triggering penalties.
  • 3.4 Asynchronous Inference: An asynchronous two-server controller-reasoner pipeline addresses deployment overhead from alternating controller and reasoner calls.The passage identifies naive serialization as a source of increased per-sample latency and reduced rollout throughput.

4 Experimental Setup

The evaluation spans three reasoners and five benchmarks, comparing ACTS with efficient-reasoning, early-exiting, and budget-aware baselines using accuracy, token count, and savings. The controller is instantiated and trained with specified supervised-finetuning and reinforcement-learning configurations.

  • Evaluation protocol: ACTS is evaluated across five benchmarks and three reasoners using accuracy, total token count, and token savings over Vanilla.The benchmarks include MATH-500, AIME24, AMC, OlympiadBench, and a science QA benchmark.
  • Baselines: The baselines cover direct full thinking, no thinking, compressed reasoning, dynamic early exiting, and auxiliary budget guidance.These methods are represented by Vanilla, NoThink, CoD, DEER, and BudgetGuidance.
  • Controller training: The controller is initialized through supervised fine-tuning on synthetic steering trajectories and refined with GRPO reinforcement learning.Training uses multi-budget steering data and budget-conditioned controller optimization.
  • Inference setup: Evaluation uses fixed decoding configurations for the controller and reasoner, with ACTS served through separate asynchronous SGLang servers.The controller and reasoner communicate over asynchronous HTTP during inference.

5 Results

Across reasoners and benchmarks, ACTS preserves or improves accuracy while substantially reducing token use and supports smooth budget-controlled accuracy-efficiency trade-offs. Its savings arise from rescue and shortening behaviors, while asynchronous serving keeps throughput close to Vanilla.

  • 5.1 ACTS matches full thinking performance with substantial token savings: ACTS and ACTSπsft consistently reach accuracy comparable to or above Vanilla while substantially reducing average token usage.They improve over BudgetGuidance and DEER on the majority of reported cells.
  • 5.2 ACTS enables controllable accuracy vs. efficiency trade-offs: ACTS budget sweeps produce smooth, largely monotone accuracy-versus-token curves across three reasoners and five benchmarks.The curves are compared from tight budgets to near-Vanilla scale.
  • 5.2 ACTS enables controllable accuracy vs. efficiency trade-offs: With DeepSeek-R1-Distill-1.5B, ACTS surpasses Vanilla on every harder benchmark at substantially lower token cost.A mild accuracy reversal appears at very large budgets on AIME and GPQA for this reasoner.
  • 5.3 Our trained controller agent generalizes to different reasoners and tasks: ACTS transfers from synthetic DeepSeek-R1 math trajectories to Qwen3-8B and science QA, improving GPQA Diamond accuracy over Vanilla by up to +11.9pp.The paper attributes this transfer to steering that counteracts long, confused traces from unguided reasoners.
  • 5.4 Agentic reasoning steering offers interpretable token savings: On DeepSeek-R1-7B, Rescue provides 28% of savings, whereas Shorten provides 42% on Qwen3-8B; Regress contributes less than 5% for both.Rescue converts Vanilla errors into correct answers, while Shorten removes redundant post-solution verification and strategy oscillations.
  • 5.5 Async controller-reasoner inference incurs negligible latency overhead: ACTS matches Vanilla throughput within 1% on Qwen3-8B and within 11% on DeepSeek-R1-Distill-7B, while DEER reaches about half of Vanilla’s throughput.The asynchronous controller-reasoner pipeline uses a 4+4 GPU split for ACTS under the shared eight-GPU comparison.

6 Conclusion

ACTS treats reasoning steering as budgeted strategy control by a controller guiding a frozen reasoner step by step. The framework matches full-thinking performance with token savings, offers controllable trade-offs, and generalizes across reasoners and domains.

  • Conclusion: ACTS formulates reasoning steering as a controller-reasoner Markov decision process under a thinking-token budget.At each step, the controller emits a reasoning strategy and steering phrase for the next reasoner step.
  • Conclusion: The controller is trained from synthetic multi-budget steering trajectories and then optimized with budget-conditioned reinforcement-learning rewards.The reward shaping asymmetrically penalizes overthinking and premature termination.
  • Conclusion: ACTS matches or outperforms full-thinking baselines with substantial token savings and enables controllable accuracy-efficiency trade-offs.The framework also generalizes to different reasoners and task domains.

Limitations

The study’s evaluation is limited to three reasoners of up to 8B parameters from two model families, and ACTS assumes an externally supplied inference-time thinking budget.

  • Evaluation covers three reasoners up to 8B parameters from two model families, leaving larger 70B-scale and proprietary reasoners unvalidated.
  • ACTS assumes that users or service providers supply the thinking-token budget at inference time.The paper leaves autonomous difficulty-aware budget selection for future work.

A Strategies and Steering Phrases

The controller uses seven high-level reasoning strategies, each paired with a short steering phrase that opens the next reasoner step while preserving native generation continuity.

  • The controller’s strategy set contains seven functional reasoning strategies paired with representative steering phrases.The listed phrases are illustrative rather than exhaustive, and the controller can emit free-form phrases conditioned on the running trace.
  • A steering phrase supplies the opening tokens of the next reasoner step and linguistically enacts the selected strategy.The phrase is designed to preserve continuity with the reasoner’s native generation style.

B Steering Trajectory Construction

ACTS initializes its controller from synthetic, annotated reasoning trajectories, augments them across budgets, filters repetitive behavior, and applies the same controller format during inference.

  • The corpus samples 2,000 OpenR1-Math traces uniformly across 512-token bands from 512 to 8192 tokens and segments them at paragraph boundaries ending in sentence-final punctuation.
  • Each reasoning step is annotated with one of seven strategies and an extracted opening steering phrase using Qwen3-32B.The annotator jointly classifies the step and extracts its opening phrase as structured controller action data.
  • Annotated traces are assembled into multi-turn controller dialogues, with assistant-token-only loss during supervised fine-tuning.
  • Trajectory filters cap consecutive identical strategies at five and discard repeated-phrase or excessive alternating-strategy cycles.
  • At inference, controller messages include the question initially and then the latest reasoner step with remaining-budget information.The reasoner retains its native chat template and receives the cumulative trace followed by the controller’s steering phrase.

D Case Studies

The case studies illustrate ACTS steering through explicit strategy labels and starter phrases, contrasting its rescue of an initially wrong path with its shortening of redundant correct reasoning.

  • Case studies: The case studies compare Vanilla with ACTS paired with DeepSeek-R1-Distill-Qwen-7B on two MATH-500 examples.The examples illustrate the Rescue and Shorten mechanisms.
  • Steering annotations: The annotator classifies each reasoning step by opening intent, using strategies such as understanding, planning, and execution.
  • Steering annotations: The assembled steering trajectory displays annotated strategies and extracted steering phrases alongside the original reasoning trace.
  • Controller interaction: The controller receives the question and full budget initially, then the latest reasoner step and remaining budget on subsequent turns.
  • Reasoner interaction: The reasoner preserves its native chat template while ACTS appends the cumulative trace and controller-generated steering phrase inside the thinking block.
  • Rescue: In Rescue, Vanilla commits to an incorrect candidate after miscounting its digit sum, whereas ACTS reaches the correct candidate through structured stepping.
  • Shorten: In Shorten, Vanilla redundantly re-derives 4343, while ACTS performs the division once, verifies once, and concludes.
Loading 2606.03965v1…