PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Agentic Reasoning Through Self-Regulated Simulative Planning

Mingkai Deng, Jinyu Hou, Lara Sá Neves, Varad Pimpalkhute, Taylor W. Killian, Zhengzhong Liu, Eric P. Xing

arXiv:2605.22138v1cs.AIcs.CLcs.LGcs.RO

TL;DR

Reactive agents often lack explicit control over planning, causing longer reasoning without reliable accuracy gains. This paper introduces SR2AM, which combines future-state simulation, learned planning regulation, and reactive execution in an LLM-based agent. Across four interactive reasoning categories, the resulting models are competitive with much larger systems while using fewer reasoning tokens.

  • Problem

    Current reactive approaches leave planning presence, structure, and horizon uncontrolled, so longer reasoning does not necessarily yield better answers or an explicit planning process that can be independently regulated.

  • Method

    SR2AM decomposes agent reasoning into reactive execution, simulative planning through an LLM world model, and a learned configurator that decides when and how deeply to plan.

  • Results

    Across math, science, tabular analysis, and web information seeking, SR2AM models achieve Pass@1 competitive with much larger systems, while v1.0-30B uses 25.8–95.3% fewer reasoning tokens than comparable agentic LLMs.

  • Takeaways & Limitations

    Learned self-regulation produces longer-horizon plans rather than substantially more frequent planning and may extend beyond planning to broader agent self-organization.

  • Takeaways & Limitations

    The evaluation is limited to language-based interactive reasoning, and an LLM used as a language-space world model may have limited predictive power in physical and social worlds.

Abstract

from arXiv · show

How should an agent decide when and how to plan? A dominant approach builds agents as reactive policies with adaptive computation (e.g., chain-of-thought), trained end-to-end expecting planning to emerge implicitly. Without control over the presence, structure, or horizon of planning, these systems dramatically increase reasoning length, yielding inefficient token use without reliable accuracy gains. We argue efficient agentic reasoning benefits from decomposing decision-making into three systems: simulative reasoning (System II) grounding deliberation in future-state prediction via a world model; self-regulation (System III) deciding when and how deeply to plan via a learned configurator; and reactive execution (System I) handling fine-grained action. Simulative reasoning provides unified planning across diverse tasks without per-domain engineering, while self-regulation ensures the planner is invoked only when needed. To test this, we develop SR$^2$AM (Self-Regulated Simulative Reasoning Agentic LLM), realizing both as distinct stages within an LLM's chain-of-thought, with the LLM as world model. We explore two instantiations: recording decisions from a prompted multi-module system (v0.1) and reconstructing structured plans from traces of pretrained reasoning LLMs (v1.0), trained via supervised then reinforcement learning (RL). Across math, science, tabular analysis, and web information seeking, v0.1-8B and v1.0-30B achieve Pass@1 competitive with 120-355B and 685B-1T parameter systems respectively, while v1.0-30B uses 25.8-95.3% fewer reasoning tokens than comparable agentic LLMs. RL increases average planning horizon by 22.8% while planning frequency grows only 2.0%, showing it learns to plan further ahead rather than more often. More broadly, learned self-regulation instantiates a principle we expect to extend beyond planning to how agents govern their own learning and adaptation.

1 Introduction

The paper argues that reactive reasoning without explicit control over planning can waste tokens without reliably improving answers. SR2AM separates reactive execution, simulative planning, and learned self-regulation, achieving competitive accuracy with substantially lower reasoning cost.

  • Uncontrolled chain-of-thought leaves planning presence, structure, and horizon unspecified, increasing token use without reliably improving answers.
  • The proposed decomposition assigns direct action to System I, future-state simulation to System II, and planning control to System III.The configurator determines when and how deeply to plan.
  • SR2AM implements the configurator and simulative planner as distinct stages in an LLM’s chain of thought, with the LLM serving as the world model.The stages operate alongside free-form reasoning and acting.
  • SR2AM-v0.1-8B and SR2AM-v1.0-30B achieve Pass@1 competitive with systems ranging from 120–355B and 685B–1T parameters, respectively.
  • 25.8–95.3% fewer reasoning tokens are used by SR2AM-v1.0-30B than by competitive agentic LLMs of similar scale.
  • RL increases average planning horizon by 22.8% while planning frequency grows only 2.0 percentage points.The reported pattern indicates further-ahead planning rather than more frequent planning.

2 Formalizing Self-Regulated Simulative Reasoning

The formalization models agent behavior as belief-state prediction and action selection through simulative planning, reactive execution, and a configurator that regulates planning. It contrasts this decomposition with unregulated or partially regulated deliberation.

  • 2.1 Agent-Environment Model and Simulative Reasoning: At each step, an agent acts from a world state or inferred belief state while the environment transitions according to the selected action.Partial observability requires inferring belief states from observations.
  • 2.1 Agent-Environment Model and Simulative Reasoning: A world model predicts next belief states for proposed actions, enabling simulated action sequences to approximate optimal behavior without direct access to environment dynamics.
  • 2.1 Agent-Environment Model and Simulative Reasoning: Simulative reasoning proposes candidate actions, predicts their consequences, and selects the sequence maximizing expected long-term progress.The paper presents it as a general-purpose planning mechanism grounded in next-state prediction.
  • 2.1 Agent-Environment Model and Simulative Reasoning: Exact optimization is intractable in practice, so an approximate simulative planner outputs a plan containing the current belief, selected actions, and predicted future states.
  • 2.1 Agent-Environment Model and Simulative Reasoning: The actor uses the plan for fine-grained reasoning and direct action, while predicted states support progress assessment and detection of violated expectations.Reactive execution also enables fast responses when deliberation is unnecessary.
  • 2.2 From Unregulated Deliberation to Self-Regulation: Unregulated reactive policies generate latent deliberation before acting, without explicit simulative plans or regulation of when planning occurs.
  • 2.2 From Unregulated Deliberation to Self-Regulation: Longer reasoning can increase dramatically without improving task success, whereas planning at every step can be prohibitively costly when replanning is unnecessary.
  • 2.2 From Unregulated Deliberation to Self-Regulation: The configurator regulates whether to make a new plan, continue an existing plan, or skip planning, and the formulation can support iterative refinement.Prior effort-adaptive, mode-routing, and workflow-distillation approaches implement only subsets of the full decomposition.

3 Instantiating Self-Regulated Simulative Reasoning

SR2AM instantiates the three-system decomposition for interactive reasoning by using an LLM as a language-space world model and training structured regulation and planning with supervised learning followed by RL. Two data-construction approaches target feasibility and scalability.

  • SR2AM targets mathematical, scientific, tabular, and web information-seeking tasks, where iterative tool use helps smaller LLMs address otherwise larger-model tasks.
  • The LLM itself serves as the world model in language space, while the configurator and planner are integrated with reactive reasoning and acting.
  • Training first uses supervised finetuning on self-regulated simulative reasoning data, followed by RL for task success.
  • v0.1 records decisions from a prompted multi-module system, whereas v1.0 reconstructs configurator decisions and plans from pretrained reasoning traces.The latter approach is described as more scalable and better at preserving free-form reasoning.
  • During inference, the agent receives observations, forms a belief state, and selects tool calls or a final text response.The tool set includes web search, webpage visiting, and a stateless Python sandbox.
  • Supervised data encode configurator decisions and simulative plans alongside free-form reasoning to train SR2AM-v0.1 and SR2AM-v1.0.
  • RL coordinates configurator decisions, planner outputs, and actions across trajectories until a final answer or the maximum step limit.
  • The reward combines answer correctness, trajectory structure compliance, and final-answer extractability, optimized with an adapted GRPO procedure.

4 Experiments

Across 11 benchmarks, SR2AM achieves competitive accuracy with substantially lower reasoning-token use, while ablations and RL analyses support the value of structured, selective, horizon-controlled planning.

  • Experimental Setup: Evaluation covers 11 benchmarks spanning math, science, tabular analysis, and web information seeking, using overall Pass@1 and reasoning tokens per trajectory.Overall Pass@K is the unweighted average across datasets; reasoning tokens exclude environment observations and tool outputs.
  • Main Results: SR2AM-v0.1-8B achieves overall Pass@1 57.0, competitive with larger 30–32B agentic and 120–355B tool-using systems.The model outperforms other systems at the same parameter scale.
  • Main Results: SR2AM-v1.0-30B reaches overall Pass@1 71.3, competitive with DeepSeek-V3.2 at 685B and Kimi-K2.5 at 1.0T parameters.It exceeds GPT-5.4-xhigh text-only reasoning performance of 68.4 and approaches its tool-harness result of 78.3.
  • Main Results: 25.8–95.3% fewer reasoning tokens accompany better or competitive accuracy for SR2AM-v1.0-30B versus stronger 30–32B agentic LLMs.Against MiroThinker-v1.5-30B, it uses 5,518 versus 11,295 reasoning tokens, a 51.2% reduction, at competitive Pass@1.
  • RL Analysis: RL shifts planning toward longer horizons while planning frequency remains stable, increasing average horizon across all task categories.The largest horizon gain is 32.7% in science and the smallest is 20.9% in web; v0.1 RL ends with 34.1% fewer tokens and Pass@1 of 56.2 versus 47.6 for unregulated deliberation.

5 Related Work

Related work largely treats reasoning as uniformly extended or regulates only its amount, whereas the paper positions simulative reasoning as structured future-state prediction.

  • Related Work: Frontier reasoning models apply extended chain-of-thought uniformly, without mechanisms to modulate reasoning horizon or structure.External regularization includes length penalties, supervised compression, and user-specified effort controls.
  • Related Work: Effort-adaptive approaches regulate reasoning amount but operate along a single axis without constructing simulative plans.The paper distinguishes this line from its proposed decomposition of reactive execution, simulative reasoning, and self-regulation.

6 Conclusion

The paper concludes that efficient agentic reasoning benefits from three complementary systems: reactive execution, simulative planning, and learned self-regulation. Across four task categories, this decomposition achieves competitive accuracy at lower reasoning cost, while RL produces longer-horizon rather than more frequent planning.

  • The three-system decomposition combines reactive execution, simulative planning, and learned self-regulation as complementary components.The configurator governs planning, while simulative reasoning predicts future states and reactive execution handles direct action.
  • Across four interactive reasoning task categories, the decomposition yields competitive accuracy at substantially lower reasoning cost.
  • RL with self-regulated simulative reasoning increases planning horizon rather than merely increasing planning frequency.
  • A single simulative planning framework achieves strong performance across diverse tasks without per-domain procedures.

7 Limitations and Future Work

The evaluation is limited to language-based interactive reasoning and uses LLMs as simplified world models. Future work includes richer embodied and multi-agent settings, isolated component diagnostics, improved data and context management, and broader forms of self-regulation.

  • Scope boundaries: The study evaluates self-regulated simulative reasoning only in language-based interactive reasoning, leaving embodied and multi-agent settings for future work.Those settings introduce richer state representations and coordination constraints.
  • World-model assumptions: Using LLMs as language-space world models may limit predictive power in wider physical and social environments.Multimodal world models predicting next states from perceptual inputs are proposed as a future test.
  • Evaluation gaps: The evaluation measures holistic task accuracy and reasoning efficiency rather than configurator and world-model accuracy in isolation.Isolated comparisons with oracle decisions or observed outcomes could diagnose where the decomposition needs improvement.
  • Engineering and future directions: Engineering limitations include retaining full interaction history, relying on publicly available data, and using standard tool interfaces.The paper suggests context management, targeted synthetic data, richer interfaces, and extensions beyond inference-time planning.

C Supervised Data Collection Details

The supervised-data pipeline collects trajectories with bounded actions and correctness filtering, then reconstructs structured plan horizons for scalable training. Web tasks receive additional configurator reasoning, while plan annotations make horizon control explicit.

  • v0.1 collection: For collection using o4-mini, trajectories allow up to T_max = 30 actions and are retained only when they call more than 3 reasoning modules or actions combined.Trajectories are retried up to 3 times and filtered for answer correctness.
  • v0.1 collection: Web-task configurator reasoning is enriched with GPT-4.1 before action selection, which improves performance in practice.Qwen3-Coder-480B-A35B-Instruct is also explored for the same pipeline.
  • v1.0 plan reconstruction: The v1.0 structured plan horizon T′ − t is explicitly controlled by adjusting plan annotations across task categories.DeepSeek-V3.2 generates and jointly annotates trajectories with T_max = 100 steps, retrying up to 5 times until correctness.

D RL Objective and Training Details

The RL pipeline defines trajectory rewards around correctness and structural compliance, then updates the policy with an adapted GRPO objective. Training uses on-policy stabilization, trajectory filtering, and difficulty-based data selection across multiple reasoning datasets.

  • Reward design: The reward r(τ, g) combines three binary signals in a piecewise function prioritizing answer correctness while rewarding structural compliance.This preserves a gradient signal even for unsuccessful trajectories.
  • RL objective: The adapted GRPO procedure samples groups of trajectories, normalizes token advantages at the group level, and optimizes likelihood ratios with clipped bounds.The objective uses asymmetric clipping bounds ϵ_low and ϵ_high.
  • Training stabilization: All model updates remain on-policy, and truncated trajectories are filtered for models of 30B parameters and above.The filtering reduces noisy negative signals that can cause format collapse.
  • Data construction: Difficulty-based filtering produces a final RL mix of 151,218 examples after downsampling web data to 33.3% of its filtered size.The filtering is intended to avoid uniform rewards and uninformative gradients.
  • Data construction: The v1.0 RL dataset contains 20,701 examples assembled from filtered non-web data and balanced v0.1 and v1.0 web subsets.

E.3 Training Hyperparameters

SR2AM uses separate v0.1 and v1.0 training configurations, each combining supervised fine-tuning with reinforcement learning. Training continued to show reward and evaluation gains at its endpoint, suggesting possible undertraining.

  • SR2AM-v0.1: SR2AM-v0.1 uses Qwen3-8B, four supervised-fine-tuning epochs, and reinforcement learning with 8,192-token responses and 40 maximum action steps.Its SFT configuration uses Axolotl, while RL uses Slime.
  • Training procedure: Both instantiations are trained with supervised learning followed by reinforcement learning.The supplied training descriptions specify separate SFT and RL stages for v0.1 and v1.0.
  • SR2AM-v1.0: SR2AM-v1.0 uses Qwen3-30B-A3B-Thinking-2507, four supervised-fine-tuning epochs, and reinforcement learning with 16,384-token responses.Its SFT configuration uses ms-swift, with a 131,072-token context; RL uses the same GRPO configuration as v0.1 except for longer response and context limits.
  • Training status: Reward and evaluation were still improving at the end of training for both instantiations, indicating that the models may be undertrained.The paper identifies scaling training data and compute with more efficient asynchronous training as a future direction.

G.1 Evaluation Protocol

Evaluation uses consistent inference settings across the benchmarks, with duplicated small test sets for stability and overall Pass@K averaged uniformly across datasets. Reasoning efficiency is measured by generated reasoning tokens per trajectory, excluding observations and tool outputs.

  • Inference settings: Evaluation imposes 10-minute per-turn, 5-minute per-tool, and 60-minute overall-response timeouts.Tool-harness models also use a 16,384-token maximum completion per turn.
  • Inference settings: All benchmarks use consistent inference settings to support stability and reproducibility.The protocol also specifies benchmark-specific scoring functions and evaluation settings in the appendix.
  • Stability: Small-sample benchmarks are duplicated 32× for AIME-24 and AIME-25, and 4× for GPQA-Diamond, GAIA-103, and XBench-DeepSearch.Reported metrics are averaged across duplicated runs; HLE uses a 500-question subset.
  • Metrics: Reasoning efficiency is reported as average reasoning tokens per trajectory, excluding environment observations and tool outputs.Included tokens cover generated reasoning, configurator decisions, and plans.

H Evaluation Function Sources

The evaluation-function table documents the sources used to score the paper’s benchmarks. These sources include rule-based graders and adapted official or benchmark-specific evaluation functions.

  • Evaluation-function sources: The listed sources include PRIME-RL/PRIME-derived rule-based grading, General-Reasoner, GAIA, OpenAI simple-evals, HLE, and XBench evaluation functions.Several functions are adapted from benchmark or project implementations, including official evaluation functions where noted.
  • Evaluation-function sources: Table 2 lists the sources of evaluation functions used across all benchmarks.The caption identifies the table as a source map for benchmark evaluation functions.

I Quantitative Evaluation Result by Benchmarks

Table 3 reports per-benchmark results for reference systems and agentic reasoning paradigms on the final test set. It groups systems by paradigm and reports average reasoning tokens per problem alongside benchmark outcomes.

  • Table organization: Table 3 presents per-benchmark results across reference systems and agentic reasoning paradigms on the final test set.Systems are grouped by paradigm and ordered by parameter size within each group.
  • Table organization: Within each paradigm, systems are ordered by parameter size, with ties broken by overall performance in ascending order.GPT-5.4 is listed last where applicable.
  • Metrics: The “# Reas. Tokens” column reports the average number of reasoning tokens per problem.This column provides the table’s reasoning-efficiency measure alongside per-benchmark results.

J Comparison of Self-Regulated Simulative Reasoning Across Models

Across three examples, self-regulated planning improves state tracking and fallback behavior, but excessive verification can make anticipatory planning inefficient. Structured plans distinguish reactive tallies from plans that verify sources or adapt after blocked searches.

  • Example 1: Web Research (GAIA): SR2AM-v1.0-30B verifies the crocodile tally against an authoritative source, unlike reactive plans that accept 6 Nile plus 1 Morelet’s as 7.The example contrasts a single-step “provide final answer” plan with a plan that explicitly checks for other species and cross-validates the result.
  • Example 2: Mathematical Reasoning (AIME 2024): The trained model finds the correct rectangle-solving path in 11 messages by planning a fallback to computational enumeration when search fails.Its plan tracks the current state, recognizes that vertex-only counting is insufficient, and pivots after blocked or flawed sources.
  • Example 2: Mathematical Reasoning (AIME 2024): Unregulated CoT makes a conceptual error by counting only vertex-inscribed rectangles, while explicit current-state tracking flags the assumption for re-examination.The unregulated system reaches 15, whereas the structured plan distinguishes dodecagon vertices from diagonal-intersection points.
  • Example 3: Over-Verification (HLE): SR2AM’s anticipatory verification becomes counterproductive when the correct answer already appears in the first search results, causing 60+ unnecessary messages and 0/3 accuracy.The example attributes this failure to an insufficiently refined stopping criterion for skipping further planning.
  • Plan Reconstruction: Plans are structured sequences of predicted state-action transitions that can be omitted when the agent continues an existing plan.The reconstruction prompt forbids inserting specific future observations, tool outputs, calculations, or final answers into the inferred plan.
Loading 2605.22138v1…