PaperlayerView exact revision on arXiv

Source-linked AI summary

In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

Zhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, Pan Lu

arXiv:2510.05592v2cs.AIcs.CLcs.LGcs.MA

TL;DR

Tool-augmented LLMs struggle with monolithic full-context policies, while most agentic systems remain static or offline-trained. AgentFlow trains a planner inside a live multi-turn system using Flow-GRPO, and its 7B model outperforms specialized baselines across four domains, including larger GPT-4o.

  • Problem

    Existing tool-integrated methods use monolithic full-context policies, while agentic systems are often training-free or decoupled from live multi-turn dynamics.

  • Method

    AgentFlow coordinates planner, executor, verifier, and generator modules through evolving memory, while Flow-GRPO optimizes the planner on-policy with shared trajectory-level rewards.

  • Results

    14.9% search, 14.0% agentic, 14.5% mathematical, and 4.1% scientific average accuracy gains were achieved against top-performing specialized systems.

  • Takeaways & Limitations

    In-the-flow optimization improves planning and tool-calling reliability, with benefits that scale with model size and reasoning turns.

Abstract

from arXiv · show

Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios. Agentic systems offer a promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training decoupled from the live dynamics of multi-turn interaction. We introduce AgentFlow, a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environments, we propose Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates. It broadcasts a single, verifiable trajectory-level outcome to every turn to align local planner decisions with global success and stabilizes learning with group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability, and positive scaling with model size and reasoning turns.

1 INTRODUCTION

Existing tool-augmented LLMs use monolithic policies, while agentic systems decompose work but are usually not trainable in the live interaction loop. AgentFlow addresses this gap with in-the-flow planner optimization and reports broad benchmark gains.

  • Monolithic tool-integrated methods interleave reasoning and tool calls under full context, creating scaling challenges for multi-turn reasoning.
  • Agentic systems decompose tasks across specialized modules and shared communication, but are commonly training-free or orchestrated by handcrafted logic.
  • AgentFlow coordinates planner, executor, verifier, and generator modules through evolving memory while optimizing the planner on-policy inside the multi-turn loop.
  • Flow-GRPO assigns one verifiable final-outcome reward across turns and uses group-normalized advantages to stabilize long-horizon optimization.
  • 14.9% search, 14.0% agentic, 14.5% mathematical, and 4.1% scientific average accuracy gains were achieved against top-performing specialized systems.
  • AgentFlow’s 7B backbone outperformed specialized baselines and larger proprietary models, with analyses showing improved planning, tool-calling reliability, and positive scaling.

2 PRELIMINARY

Outcome-based reinforcement learning trains LLMs toward verifiable rewards, while tool-integrated and agentic paradigms extend reasoning through external tools or specialized modules. Their central distinction is monolithic interleaving versus decomposed collaboration.

  • Outcome-based RL fine-tunes LLM policies to maximize verifiable rewards while remaining close to a reference policy.
  • Group Relative Policy Optimization samples response groups, normalizes reward-based advantages, and applies a clipped objective.
  • Tool-integrated reasoning interleaves model reasoning and tool invocations within a full-context trajectory containing states, actions, and tool observations.
  • Agentic systems assign subproblems to specialized modules with dedicated tools and capabilities within collaborative workflows.
  • These agentic systems are typically training-free, leaving pretrained modules frozen and coordinating them through handcrafted logic or prompting heuristics.

3 IN-THE-FLOW AGENTIC SYSTEM OPTIMIZATION

AgentFlow formulates tool-integrated reasoning as observable multi-turn transitions among planning, execution, verification, and memory updates. Flow-GRPO trains the planner on live trajectories using shared final-outcome rewards to address sparse-reward credit assignment.

  • 3 IN-THE-FLOW AGENTIC SYSTEM OPTIMIZATION: AgentFlow bridges trainable monolithic models and static agentic systems by optimizing a planner inside a multi-turn agentic loop.
  • 3.1 AGENTFLOW: AN IN-THE-FLOW AGENTIC SYSTEM: Four specialized modules—Action Planner, Tool Executor, Execution Verifier, and Solution Generator—coordinate through evolving memory and a toolset.
  • 3.1 AGENTFLOW: AN IN-THE-FLOW AGENTIC SYSTEM: At each turn, the planner uses the query, toolset, and memory to formulate a sub-goal, select a tool, and produce an action.
  • 3.1 AGENTFLOW: AN IN-THE-FLOW AGENTIC SYSTEM: The executor returns an observation, the verifier produces a binary signal, and deterministic memory updates record structured process information until termination or the turn limit.
  • 3.1 AGENTFLOW: AN IN-THE-FLOW AGENTIC SYSTEM: The explicit memory records the reasoning process deterministically, supporting transparency and controllability of multi-turn decisions.
  • 3.2 IN-THE-FLOW REINFORCEMENT LEARNING OPTIMIZATION: Offline training decouples planner optimization from executor, verifier, and generator dynamics, causing distribution shift and limited guidance about important intermediate decisions.
  • 3.2 IN-THE-FLOW REINFORCEMENT LEARNING OPTIMIZATION: Flow-GRPO rolls out the full live system and updates the planner on the states, actions, and tool events induced by the current policy.
  • 3.2 IN-THE-FLOW REINFORCEMENT LEARNING OPTIMIZATION: Every action receives the same verifiable final-outcome reward, propagating trajectory-level success through the reasoning chain.

4 EXPERIMENTS

AGENTFLOW is evaluated across diverse reasoning tasks, training strategies, scaling settings, and qualitative behaviors. Results show broad benchmark gains, benefits from in-the-flow planner optimization, and improved adaptation as model capacity and turn budgets increase.

  • Experimental setup: AGENTFLOW evaluates knowledge-intensive search, agentic reasoning, mathematical reasoning, and scientific reasoning across ten benchmarks.The setup uses Qwen2.5-7B-Instruct modules and five tools, with only the Action Planner trained.
  • Main results: 14.9% in search, 14.0% in agentic reasoning, 14.5% in mathematics, and 4.1% in science are AGENTFLOW’s gains over top specialized tool-integrated systems.The 7B-backbone system also surpasses GPT-4o across all evaluated domains.
  • Training strategies on the planner: 19.0% average accuracy is lost with offline SFT, whereas Flow-GRPO produces a 17.2% average gain over the frozen planner baseline.The comparison keeps executor, verifier, and generator modules fixed while changing planner training.
  • Scaling trends in AGENTFLOW: Flow-GRPO consistently improves performance for both 3B and 7B backbones.Only the planner is fine-tuned, while the other modules and tools remain fixed across backbone-scale comparisons.
  • Scaling trends in AGENTFLOW: Increasing Tmax from 3 to 10 consistently improves outcomes across tasks while increasing the average number of turns consumed.Larger budgets support deeper information retrieval on knowledge-intensive benchmarks.
  • Training dynamics and qualitative analysis: Flow-GRPO-trained planners show adaptive efficiency, stronger self-correction, and spontaneous discovery of new tool-based solution pathways.Training rewards rise while response length eventually shortens and stabilizes, indicating more concise planner behavior.

5 RELATED WORK

Related work contrasts monolithic tool-integrated reasoning with agentic systems that decompose tasks across specialized modules. Existing agentic approaches are often training-free or statically orchestrated, motivating trainable adaptation.

  • Tool-integrated reasoning: Tool-integrated reasoning models interleave reasoning and tool calls under full context, extending outcome-based reinforcement learning to external tools.Early systems focused on single-tool settings, while later systems support multiple tools through prompt-encoded metadata.
  • Agentic systems with tool use: Agentic systems decompose tasks across specialized modules that coordinate through shared memory and communication.Many existing systems rely on handcrafted logic and prompting without training, limiting adaptation of collaborative strategies from experience.

6 CONCLUSION

AGENTFLOW combines in-the-flow planner optimization with Flow-GRPO to address long-horizon, sparse-reward tool use. Experiments report strong cross-domain performance and improved planning, tool reliability, and scaling trends.

  • Conclusion: AGENTFLOW coordinates four specialized modules through evolving memory while optimizing its planner inside the multi-turn loop.The modules are planner, executor, verifier, and generator.
  • Conclusion: Flow-GRPO converts multi-turn reinforcement learning into tractable single-turn updates by broadcasting one verifiable trajectory outcome to every turn.Group-normalized advantages stabilize credit assignment under long-horizon sparse rewards.
  • Conclusion: AGENTFLOW surpasses specialized baselines and larger proprietary models while showing improved planning, tool-calling reliability, and positive scaling with model size and turn budgets.These conclusions are supported by comprehensive experiments and further analyses.

A TRAINING ALGORITHM OF AGENTFLOW

The training algorithm rolls out AGENTFLOW turn by turn, updates shared memory after execution and verification, generates a final solution, and assigns its outcome reward to every turn before updating the planner.

  • Training algorithm: AGENTFLOW initializes shared memory with each query and iteratively samples planner actions, executes them, verifies results, and updates memory until termination.The loop uses the planner policy, tool executor, executive verifier, generator, toolset, and shared evolving memory.
  • Training algorithm: The solution generator produces the final output from the terminal query-memory state after the multi-turn interaction.The algorithm samples o from the generator conditioned on q and M_T.
  • Reward assignment: The algorithm assigns the same verifiable final outcome reward to every action in the trajectory.This broadcasts the trajectory-level reward across all turns for planner optimization.
  • Policy optimization: Flow-GRPO updates the Action Planner with a clipped objective using group-normalized advantages.The full multi-turn objective is assembled from single-turn optimization terms.
  • Notation: The planner policy generates token-sequence actions from states defined by the query, toolset, and evolving memory.The trajectory records state-action pairs over the interaction horizon.

B.2 EQUIVALENCE PROOF FOR OPTIMIZATION OBJECTIVES

The proof shows that Flow-GRPO’s global multi-turn objective is equivalent to expected local token-level optimization under the on-policy state distribution. It also establishes a surrogate-based monotonic improvement guarantee under KL control.

  • Objective equivalence: The proof reduces group-averaged trajectory expectations to single-trajectory expectations through linearity of expectation and i.i.d. sampling.This provides the intermediate step for expressing the objective as a local per-state expectation.
  • Objective equivalence: The trajectory expectation can be rewritten using the on-policy state-visitation distribution, with each state weighted by 1/T.The trajectory advantage remains associated with the trajectory from which each state was sampled.
  • Objective equivalence: Flow-GRPO’s global multi-turn objective is mathematically equivalent to maximizing the expected token-level local objective at each time step.The equivalence holds under standard sampling assumptions with i.i.d. trajectories and fixed finite turn horizon.
  • Convergence analysis: Flow-GRPO’s surrogate objective lower-bounds policy improvement after subtracting a KL-divergence penalty scaled by horizon- and reward-dependent constant C.Optimizing this lower bound is used to guarantee performance improvement between successive policies.
  • Convergence analysis: The analysis concludes that Flow-GRPO guarantees monotonic policy improvement and converges reliably to a locally optimal policy.This conclusion follows from the valid surrogate objective and its improvement guarantee.

C.4 EVALUATION DATASETS

The evaluation spans search-intensive, agentic, mathematical, and scientific reasoning datasets. These benchmarks test multi-step inference, tool use, domain knowledge, and general-agent capabilities.

  • Search-intensive benchmarks: Search-intensive evaluation includes Bamboogle, 2Wiki, HotpotQA, Musique, GPQA, and MedQA.These datasets cover compositional multi-step reasoning, Wikipedia-based question answering, sequential inference, graduate-level science, and medical knowledge.
  • Evaluation scope: The benchmark suite therefore combines sequential inference, web and tool interaction, medical and scientific knowledge, and mathematical problem solving.Together, the datasets cover multiple reasoning domains rather than a single specialized task.
  • Agentic benchmarks: GAIA evaluates general AI systems and agents through sequential reasoning, web navigation, and comprehensive tool utilization.The evaluation uses the text-exclusive portion of the dataset.
  • Mathematical benchmarks: Mathematical evaluation includes GameOf24, AIME24, and AMC23, covering arithmetic construction and advanced competition-style problems.GameOf24 requires forming 24 from four numbers, while AIME24 and AMC23 assess broader mathematical reasoning.

D.1 MAIN RESULT ANALYSIS

AGENTFLOW consistently outperforms baseline systems across search-intensive, agentic, mathematical, and scientific reasoning tasks. Its 7B backbone also exceeds larger monolithic and specialized models, while in-the-flow optimization improves planning and tool use.

  • Overall performance: AGENTFLOW consistently outperforms all baseline models across search-intensive, agentic, mathematical, and scientific reasoning domains.The main results are reported in Tables 1 and 2.
  • Cross-model comparisons: 8.2% average improvement over GPT-4o on search-intensive tasks, with AGENTFLOW reaching 57.3% versus GPT-4o’s 49.1%.The comparison uses the 7B-backbone AGENTFLOW system against the approximately 200B-parameter GPT-4o.
  • Cross-model comparisons: 15.8% average improvement over GPT-4o on agentic tasks, with AGENTFLOW reaching 33.1% versus 17.3%.This result is presented as evidence that the agentic design handles complex multi-turn tasks better than the larger monolithic model.
  • Cross-model comparisons: 16.4% improvement over GPT-4o on mathematical reasoning, with AGENTFLOW reaching 51.5% versus 35.1%.AGENTFLOW also reaches 63.5% on scientific reasoning versus 51.0% for Llama-3.3-70B, a 12.5% improvement.
  • Versatile reasoning: AGENTFLOW reaches 57.3% on search, 33.1% on agentic, 51.5% on mathematical, and 63.5% on scientific reasoning.Against AutoGen with the same backbone, AGENTFLOW improves by 14.9% on search and 19.9% on math.
  • Optimization analysis: Flow-GRPO’s in-the-flow optimization is reported as crucial, while offline SFT causes a 19.0% average accuracy drop versus the frozen baseline.The trained planner improves planning, tool-calling reliability, and solution-path discovery.

D.2 IN-DEPTH ANALYSIS OF OPTIMIZED PLANNING

The analysis examines how Flow-GRPO changes tool selection and how its benefits scale with stronger tools and larger backbones. It also details the structured prompts and iterative workflow used by AGENTFLOW’s modules.

  • Tool scaling: 1.0% on GAIA, 6.0% on AMC23, and 13.0% on HotpotQA are the gains from replacing Qwen2.5-7B tools with GPT-4o-powered tools.The comparison uses GPT-4o-powered Base Generator and Python Coder tools against Qwen2.5-7B-Instruct-powered tools.
  • Optimized tool usage: 6.1% performance improvement accompanies Flow-GRPO’s increased Web Search use on Musique.The planner learns to search more deeply in URLs returned by other search tools.
  • Backbone scaling: Flow-GRPO consistently improves performance for both Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct backbones across the evaluated tasks.Only the planner is fine-tuned, while all four modules and associated tools use the selected backbone.
  • Planning workflow: The Action Planner selects one tool and formulates its justification, context, and sub-goal from the query, memory, and tool metadata.The planner must end its response with Context, Sub-Goal, and Tool Name sections.
  • Execution workflow: The Tool Executor converts the planner’s selected action into Python code and requires at least one assigned tool.execute() call.It uses the selected tool’s metadata, sub-goal, and relevant context to construct the command.
  • Verification workflow: The Execution Verifier checks whether memory is complete, accurate, consistent, and sufficiently supported before deciding STOP or CONTINUE.If information is missing, it identifies useful tools and concludes CONTINUE; otherwise it concludes STOP.

F.1 EXAMPLE 1: EFFICIENT SEARCH FOR SIMPLE TASKS

The example contrasts Flow-GRPO’s efficient search-based solution with an untuned baseline that repeatedly misuses the Python Coder and produces an incorrect answer. The tuned system finds and verifies a valid expression in one action.

  • Success case: Flow-GRPO solves the [1, 1, 1, 13] GameOf24 instance in one Google Search action.The search returns (13 - 1) * (1 + 1) = 24, using all four numbers.
  • Success case: PASS, STOP verifies the Flow-GRPO solution before the generator returns ((13 −1) × (1 + 1)) = 24.The verifier accepts the retrieved expression as sufficient to answer the query.
  • Failure case: The untuned baseline repeats Python Coder attempts after failing to find a matching tool.Its attempted commands include an incorrect arithmetic expression and a command that returns an indentation error.
  • Failure case: The failure-case generator incorrectly concludes that (1 + 1 + 1) × 13 = 24.That expression evaluates to 39, so the generated answer does not solve the stated GameOf24 query.

F.2 EXAMPLE 2: SPONTANEOUS BRUTE-FORCE

This example contrasts Flow-GRPO-tuned and untuned AGENTFLOW on arithmetic search, showing how the tuned planner recovers from failed attempts and reaches a verified solution through targeted tool use.

  • Success case: Flow-GRPO-tuned AGENTFLOW ultimately verifies (1 + 1) * 9 + 6 = 24 using Google Search after several unsuccessful attempts.The execution verifier passes and stops, and the solution generator reports the same final expression.
  • Success case: The tuned planner progresses from initial attempts and a tool error to systematic search that finds valid expressions equaling 24.The systematic-search step reports multiple valid expressions, including (1+1)*9+6.
  • Failure case: The untuned agent enters repeated Python-tool failures and produces an incorrect result instead of reliably solving the arithmetic task.Its failure case includes an erroneous 108 result and repeated variable errors in another task example.
  • Comparison: The comparison attributes the tuned agent’s advantage to targeted queries, strategic tool selection, and recovery from failed searches, rather than merely accessing correct information.The tuned agent refines inputs and switches tools, while the untuned agent remains trapped in inefficient or repetitive paths.
  • Related examples: In related examples, Flow-GRPO supports targeted retrieval of answers about Handel’s Messiah through external search and tool switching.The cited steps identify Part I and the chorus “For unto us a child is born.”

F.6 EXAMPLE 6: RAPID AND CORRECT PHYSICS CALCULATION

This example contrasts tuned and untuned agents on a relativistic-travel question. Flow-GRPO supports correct problem formulation and efficient tool use, whereas the untuned process misinterprets the setup and performs redundant calculations.

  • Success case: Flow-GRPO-enhanced AGENTFLOW identifies relativistic time dilation as the core issue and reaches the correct answer of 81 years efficiently.The solution generator reports a physics-based calculation that accounts for distance and time dilation.
  • Failure case: The failed calculation applies the time-dilation setup to the astronaut’s 22-year lifespan rather than the travel problem’s stated distance and mission conditions.The failure trace explicitly defines t as approximately 22 years and uses that value in the calculation.
  • Success case: The successful example uses sequential tool reasoning, but its documented calculation also notes an inverse-relationship error in the proper-time formula.The success trace reports travel time, a Lorentz factor, and perceived time before flagging the formula application as incorrect.
  • Failure case: The untuned agent incorrectly treats the astronaut’s age as travel time and outputs 84615390.151914 years after redundant calculations.The execution verifier marks the process as a conceptual error and inefficient multi-step process.
  • Broader comparison: Across the broader comparison, tuned multi-tool reasoning builds on prior findings, while untuned behavior can remain inefficient or fail to integrate retrieved information coherently.A separate comparison describes systematic research and verification for identifying Gülçiçek Hatun as Olivera Despina’s mother-in-law.
Loading 2510.05592v2…