PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent models: Internalizing Chain-of-Action Generation into Reasoning models

Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Xinyan Wen, Jitao Sang

arXiv:2503.06580v1cs.AI

TL;DR

Prompt-based agentic workflows leave reasoning models with limited autonomy in managing tool interactions. AutoCoA trains reasoning models to internalize when and how to act through SFT and RL, and the reported results show stronger goal-oriented task completion than traditional agentic workflows. The paper presents this as an initial exploration, with further training approaches still to be explored.

  • Problem

    Traditional agentic workflows use external prompts to manage tool interactions, while the paper investigates internalizing action generation in reasoning models.

    Source: §1 · pp. 1–2

  • Method

    AutoCoA uses SFT and RL to train reasoning models to interleave CoT and CoA, including action-trigger training and an internal world model for simulated observations.

    Source: §3.2.1 · p. 7, §5 · p. 13

  • Results

    AutoCoA-trained agent models outperform traditional agentic workflows in goal-oriented decision-making and show improved task completion in the reported open-domain QA study.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    The report presents end-to-end CoA learning as a preliminary approach for improving reasoning models’ ability to decide when and how to act toward a goal.

    Source: §1 · p. 2, §5 · p. 13

    This is an initial exploration of CoA learning, with pure RL, improved action-inclusive sampling, and loss design for interleaved CoT and CoA left for future exploration.

    Source: §5 · p. 13

Abstract

from arXiv · show

Traditional agentic workflows rely on external prompts to manage interactions with tools and the environment, which limits the autonomy of reasoning models. We position \emph{Large Agent Models (LAMs)} that internalize the generation of \emph{Chain-of-Action (CoA)}, enabling the model to autonomously decide when and how to use external tools. Our proposed AutoCoA framework combines supervised fine-tuning (SFT) and reinforcement learning (RL), allowing the model to seamlessly switch between reasoning and action while efficiently managing environment interactions. Main components include step-level action triggering, trajectory-level CoA optimization, and an internal world model to reduce real-environment interaction costs. Evaluations on open-domain QA tasks demonstrate that AutoCoA-trained agent models significantly outperform ReAct-based workflows in task completion, especially in tasks that require long-term reasoning and multi-step actions. Code and dataset are available at https://github.com/ADaM-BJTU/AutoCoA

1 INTRODUCTION

The introduction distinguishes agent models from prompted workflows by their internalized, context-aware generation of actions alongside reasoning. It presents AutoCoA as an initial approach for improving task completion through multi-turn tool use, while noting that broader validation remains future work.

1 INTRODUCTION

  • The paper distinguishes the emerging Agent stage from Chatbot and Reasoner stages by its combination of reasoning and autonomous tool use in multi-round workflows.The introduction describes Chatbot and Reasoner as earlier stages and identifies Operator and Deep Research as examples of the Agent stage.Source: §1 · p. 1
  • The paper characterizes scripted agentic workflows as passive, while describing Deep Research as deciding from model behavior when and how to act.It further distinguishes end-to-end fine-tuning for task completion from workflow switching based only on preceding context.Source: §1 · p. 1
  • The introduction proposes Large Agent Models (LAMs) for models using longer thought-action loops to complete more complex knowledge-search tasks.The proposed name follows the account of loops enabling completion of more complex knowledge-search tasks.Source: §1 · p. 2
  • The introduction contrasts prompted CoT elicitation with learning-based approaches that internalize reasoning-step logic as model behavior.It names CoT prompting and learning-based methods including SFT, SFT followed by RL, and direct RL.Source: §1 · p. 2
  • Agent models learn coupled CoT and CoA patterns, initiating tool use when needed and making context-aware execution decisions.Prompting-based workflows simulate thought-action interplay, whereas agent models internalize it as behavior.Source: §1 · p. 2
  • AutoCoA’s initial open-domain QA results show improved task completion compared with agentic workflows.The report focuses on small-scale reasoning models and uses search action as its testbed; larger models, more tools, and broader open-ended tasks remain future work.Source: §1 · p. 2

2 AGENT MODEL

The agent model is framed as a reasoning model trained end-to-end to interleave reasoning and tool actions, using environmental observations to guide subsequent steps. Its development must address both when to act without losing reasoning ability and the cost and instability of real-environment interaction.

2.1 DEFINITION AND FORMALIZATION

  • The interaction paradigm changes from human-model exchanges for Chatbot and Reasoner to a human-model-environment structure for Agent.Agent actions obtain environmental feedback through tools, with multiple rounds of thought, action, and observation before the final response.Source: §2.1 · p. 3
  • An Agent Model is a reasoning model enhanced through end-to-end task-oriented tool training to interleave reasoning, tool actions, and environment observations until task completion.The passage formalizes inference as a POMDP whose state includes the initial environment state, task context, and generated sequence so far.Source: §2.1 · p. 3
  • Unlike workflows managed by external prompts or scripts, the agent model internalizes tool-use logic for autonomous, context-aware invocation.Its inference loop can generate reasoning, invoke a tool, or return an answer, adapting to environmental feedback until completion or an iteration/context limit.Source: §2.1 · p. 4

2.2 CHALLENGES

  • Challenge 1 is teaching the model when to trigger actions while preserving its CoT-generation ability during action-focused fine-tuning.The model must determine when internal knowledge is insufficient and when external information is needed.Source: §2.2 · p. 5
  • Challenge 2 is managing costly, inefficient, and potentially risky environment interactions in a dynamic setting that can destabilize online RL.Search-call time and resources, along with changing network conditions, complicate training and timely adaptation.Source: §2.2 · p. 5
  • The paper uses open-domain QA with web search as its testbed for exploring solutions to agent-model development challenges.The stated setup is similar to Deep Research, with the tool restricted to web search.Source: §2.2 · p. 5

3.1 OVERVIEW

AutoCoA combines SFT and RL to internalize CoA generation in reasoning models through progressive training stages. The overview addresses action timing, trajectory-level action learning, and reduced reliance on real-environment interactions through an internal world model.

3.1 OVERVIEW

  • AutoCoA combines SFT and RL to internalize CoA generation capability in reasoning models.The framework progresses from step-level action triggering to trajectory-level CoA optimization, then from simulated exploration to real-environment interaction.Source: §3.1 · pp. 5–6
  • Pure CoT data is mixed into SFT and RL to preserve reasoning ability and support adaptive action use across tasks of different difficulty.At the beginning of SFT, contrastive examples pair successful tool use with failure without action to train when to trigger actions.Source: §3.1 · p. 5
  • An internal world model is trained to simulate tool calls and generate observations, enabling extensive trajectory rollouts before limited real-environment interaction.The policy first explores potential trajectories using its own generated observations, then makes a limited number of real tool calls for adaptation.Source: §3.1 · p. 5
  • The real-environment substage is intended to help the model adapt to real-world conditions and support robust deployment.It follows simulated exploration in the staged framework.Source: §3.1 · p. 6
  • The overview distinguishes step-level action triggering from trajectory-level CoA optimization and simulated from real-environment interaction.These stages progressively inject capabilities into the model.Source: §3.1 · p. 6

3.2 SUPERVISED FINE-TUNING

The SFT pipeline synthesizes reasoning, tool-use, and contrastive data, then trains models to decide when to act and how to interleave reasoning with actions. Later stages train on complete trajectories, first excluding and then including environment feedback in the loss.

3.2.1 DATA SYNTHESIS

  • The synthesis pipeline builds pure CoT data and tool-interleaved CoA data, with contrastive examples that vary whether an action is inserted or removed.The base is 20,000 HotpotQA training samples; 10,000 are used for CoT data, and another 10,000 for CoA data.Source: §3.2.1 · pp. 6–7
  • For SFT Stages 2 and 3, the training mix combines 5,000 CoA samples with 1,000 CoT samples to balance tool use with tool-free reasoning.Stage 1 uses 1,500 CoT+A contrastive data points.Source: §3.2.1 · p. 7

3.2.2 SFT-STAGE1: COT+A

  • Stage 1 uses shared-prefix contrastive pairs to teach when to act, boosting chosen sequences over rejected ones while retaining correct reasoning-only sequences among the chosen.An auxiliary SFT loss is added because chosen and rejected sequence probabilities can both fall during training.Source: §3.2.2 · pp. 7–8

3.2.3 SFT-STAGE2: COT + COA (W/ OBSERVATION MASK)

  • Stage 2 trains on complete CoT–CoA trajectories while masking environment-feedback tokens from the loss, preserving reasoning as the model learns action steps.The stage teaches action type and parameter generation using the non-feedback tokens.Source: §3.2.3 · p. 8

3.2.4 SFT-STAGE3: COT + COA (W/O OBSERVATION MASK)

  • Stage 3 includes environment-feedback tokens in trajectory training, jointly teaching reasoning, action outputs, and feedback prediction as an implicit world model.This provides an initialization for the subsequent reinforcement-learning stage.Source: §3.2.4 · p. 8

3.3 REINFORCEMENT LEARNING

The RL stage refines interleaved reasoning and action with GRPO, using exact-match and formatting rewards. It moves from low-cost simulated tool interactions to real-environment feedback.

3.3 REINFORCEMENT LEARNING

  • Building on SFT, RL aims to internalize CoA generation and strengthen multi-turn tool use by refining how the model interleaves reasoning and action.The framework uses GRPO for this refinement.Source: §3.3 · p. 8

3.3.1 REWARD FUNCTION

  • Rewards require an exact ground-truth match, while a format penalty discourages outputs that omit required structure such as the ⟨think⟩ tag.The reward criteria are exact matching and adherence to the required format.Source: §3.3.1 · p. 8

3.3.2 POLICY OPTIMIZATION USING GRPO

  • GRPO refines CoT–CoA sequences by sampling action groups and optimizing the policy according to actions’ relative performance.Its advantage function favors actions that outperform others in the same context.Source: §3.3.2 · p. 9

3.3.3 RL-STAGE1: COT + COA (SIMULATED ENVIRONMENT); 3.3.4 RL-STAGE2: COT + COA (REAL ENVIRONMENT)

  • The RL pipeline first explores tool strategies in a simulated environment using the SFT-trained internal world model, then adapts them through real tool interactions and feedback.The real-environment stage targets dynamic and unpredictable scenarios.Source: §3.3.3 · p. 9, §3.3.4 · p. 9

4 EXPERIMENTS

The experiments evaluate AutoCoA on open-domain QA benchmarks against model-only and ReAct baselines, then compare training variants and long-horizon task execution.

4.1 SETTINGS

  • The evaluation covers two single-hop and three multi-hop QA datasets, using exact-match and LLM-judged semantic accuracy.The LLM metric uses Qwen2.5-14B-instruct to judge semantic matches.Source: §4.1 · pp. 9–10

4.2 COMPARED METHODS

  • Baselines include reasoning models without retrieval and base or reasoning models using prompted ReAct workflows.The ReAct tool calls are explicitly prompted rather than internalized.Source: §4.2 · p. 10

Agent model

  • SFT progresses from step-level action triggering to trajectory-level patterns and environment-response prediction; RL settings include simulated-only and mixed simulated/real training.The final SFT stage is described as learning an implicit world model.Source: Agent model · p. 11

4.3 RESULTS

  • Separating when-to-act from how-to-act training improves CoA pattern learning, while direct RL alone yields marginal gains constrained by initial-policy capacity.The reported comparison attributes better thought-action transitions to separating the SFT stages.Source: §4.3 · p. 11
  • After five-sixths simulated training, one-sixth real-environment interaction improves adaptation; simulated training reduces interaction costs but sacrifices some performance.The paper identifies improving world-model fidelity as future work.Source: §4.3 · p. 11
  • All AutoCoA variants substantially outperform the initial policy model with ReAct after CoA learning.Table 1 compares performance across different QA methods.Source: §4.3 · p. 11, Table 1 · p. 12
  • Results for the large R1 model reflect memorized knowledge rather than action capability, and its accuracy is very low on new questions.The passage attributes this caveat to model size and outdated QA questions.Source: §4.3 · p. 12

4.3.1 IMPROVING LONG-HORIZON TASK EXECUTION

  • Without CoA learning, success is concentrated on tasks needing one or two actions; trained agents support longer rounds and maintain relatively high accuracy at five actions.The comparison examines correctly completed tasks and success rates across action counts.Source: §4.3.1 · p. 13

5 CONCLUSION AND FUTURE WORK

AutoCoA is presented as an end-to-end fine-tuning approach for multi-turn tool use, alongside a roadmap from short-term applications to a long-term agent OS. The report describes this exploration as preliminary and identifies further methods and application scopes to pursue.

5 CONCLUSION AND FUTURE WORK

  • End-to-end fine-tuning gives AutoCoA models a stronger ability to decide when and how to act, outperforming traditional workflows in goal-oriented decision-making.The report characterizes its CoA-learning method as an initial exploration, with further approaches still to explore.Source: §5 · p. 13

5.1 AGENT ROADMAP

  • The roadmap frames Operator and Deep Research as execution and reasoning capabilities, with a three-phase path for agent development.The report associates them with Computer Use and Knowledge Research, respectively.Source: §5.1 · p. 14

5.1.1 SHORT-TERM: APPLICATIONS

  • Short-term applications include Agentic RPA for adaptable enterprise workflows and personal assistants that use behavioral logs and personal knowledge bases.The assistant concept includes proactive behavior and multimodal context awareness as future extensions.Source: §5.1.1 · p. 14
  • Domain analysts may need proprietary data and domain knowledge for specialized fields, while coding agents use programming to automate tasks such as API integrations.The report identifies restricted open-domain access and shallow expert analysis as current Deep Research limitations.Source: §5.1.1 · p. 14

5.1.2 MID-TERM: EXPANDING CAPABILITIES

  • Mid-term action executors are envisioned to expand from software APIs to physical-world input and control through interfaces such as IoT devices.These interfaces could support sensing in homes and factories as well as remote operation.Source: §5.1.2 · p. 15
  • Mid-term insight seekers would integrate tools for information acquisition and processing, with future inputs potentially including experimental findings and multimodal feedback.The longer-term vision has agents exploring data, experimental materials, and simulations to generate insights.Source: §5.1.2 · p. 15

5.1.3 LONG-TERM: AGENT OS

  • The long-term agent OS analogy maps agent models to CPUs, tools to programs, and short- and long-term memory to RAM and hard drives.The report says this requires unified representations for multimodal I/O, tools, and memory, plus an agent-oriented instruction set.Source: §5.1.3 · pp. 15–16
  • Agent protocols are proposed for lower-level interaction with tools, vector databases, and other agents; mature ecosystems may also coordinate agents on complex projects.The report describes sharing intermediate results, delegating subtasks, and self-organization as features of future multi-agent collaboration.Source: §5.1.3 · p. 16

A APPENDIX

The appendix passage identifies a table comparing performance across temporal knowledge domains but provides no results.

A APPENDIX

  • Table 2 is titled as a performance comparison across temporal knowledge domains.The supplied passage gives the table title only, not its values or comparisons.Source: Table 2 · p. 18

A.1 DATASETS

The experiments examine how memorized knowledge relates to multi-hop reasoning accuracy using separate evaluations of unseen temporal knowledge and familiar popular knowledge.

A.1 DATASETS

  • The study evaluates how model knowledge memorization impacts multi-hop reasoning accuracy.The evaluation is divided into unknown-knowledge and known-knowledge settings.Source: §A.1 · p. 18
  • The unknown-knowledge test contains 192 three-hop questions, built from 576 sub-questions using 97 recent CCTV News sports reports not exposed during training.The reports were published between January 16 and March 7, 2025.Source: §A.1 · p. 18
  • The known-knowledge evaluation uses a stratified sample of 192 three-hop MINTQA-POP questions covering high-frequency domains familiar from wiki datasets.The selected questions comprise 576 sub-questions.Source: §A.1 · p. 18

A.2 EXPERIMENTAL RESULTS

Results differ sharply between familiar popular knowledge and unseen temporal questions: R1 performs strongly on the former, while both models score much lower on the latter.

A.2 EXPERIMENTAL RESULTS

  • 61.5% (EM) and 64.6% (LLM) for R1 on MINTQA-POP exceed the distilled 7B model’s 5.2% (EM) and 7.3% (LLM).The passage characterizes R1 as relying on memorized patterns and excelling on familiar knowledge domains.Source: §A.2 · p. 18
  • 7.8% (EM) and 9.8% (LLM) for R1 on unseen temporal data accompany a collapse for both models, including 2.6% and 5.2% (LLM) for R1-Distill-Qwen-7B.The passage attributes the sharp decline to knowledge-cutoff dependence and says external tool use is needed to augment internal knowledge.Source: §A.2 · p. 18
Loading 2503.06580v1…