PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowReasoner: Reinforcing Query-Level Meta-Agents

Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, Tianyu Pang

arXiv:2504.15257v1cs.AI

TL;DR

Task-level multi-agent systems do not automatically adapt to individual queries, while query-specific search sets are unavailable. FlowReasoner uses reasoning, external execution feedback, and reinforcement learning to generate a system for each query. It consistently outperforms competing approaches across three code-generation benchmarks, including o1-mini.

  • Problem

    Task-level systems are fixed for all queries in a task, limiting automatic adaptation to individual queries.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    FlowReasoner distills DeepSeek R1-based reasoning and enhances query-level system generation with reinforcement learning from external execution feedback and a reward for performance, complexity, and efficiency.

    Source: §1 · p. 2

  • Results

    Across three benchmark datasets, FLOWREASONER-14B outperforms all competing approaches, improving 5 percentage points over MaAS and 10% over o1-mini.

    Source: §5.1 · pp. 7–8

  • Takeaways & Limitations

    The approach generates a personalized multi-agent system for each query rather than relying on a fixed task-level workflow.

    Source: §1 · p. 2

Abstract

from arXiv · show

This paper proposes a query-level meta-agent named FlowReasoner to automate the design of query-level multi-agent systems, i.e., one system per user query. Our core idea is to incentivize a reasoning-based meta-agent via external execution feedback. Concretely, by distilling DeepSeek R1, we first endow the basic reasoning ability regarding the generation of multi-agent systems to FlowReasoner. Then, we further enhance it via reinforcement learning (RL) with external execution feedback. A multi-purpose reward is designed to guide the RL training from aspects of performance, complexity, and efficiency. In this manner, FlowReasoner is enabled to generate a personalized multi-agent system for each user query via deliberative reasoning. Experiments on both engineering and competition code benchmarks demonstrate the superiority of FlowReasoner. Remarkably, it surpasses o1-mini by 10.52% accuracy across three benchmarks. The code is available at https://github.com/sail-sg/FlowReasoner.

1 Introduction

Most multi-agent systems are manually designed, and existing automation still leaves workflow design dependent on human effort or complex search. FlowReasoner addresses the limited adaptability of task-level systems by reasoning from execution feedback to generate a query-specific system.

1 Introduction

  • Most LLM-based multi-agent systems are manually designed, increasing human-resource costs and limiting scalability.Early automation optimizes prompts or hyperparameters, but retains workflows that must be designed for each new scenario.Source: §1 · p. 1
  • Task-level meta-agents produce one system for a task, leaving them unable to adapt automatically to individual queries.The paper’s target is a tailored system for each query rather than a one-size-fits-all system.Source: §1 · p. 1
  • Query-specific search sets are unavailable, so FlowReasoner uses external execution feedback to guide a reasoning-based meta-agent instead of search algorithms.The feedback is used to polish the generated system.Source: §1 · p. 2
  • FlowReasoner is initialized with DeepSeek R1-generated SFT data, then enhanced with RL and a reward for performance, complexity, and efficiency.At inference, it uses deliberative reasoning to generate a novel system for each user query.Source: §1 · p. 2

2 Related Work

Prior work automates parts or all of multi-agent system design, while this paper distinguishes its query-level reasoning approach from task-level workflow optimization. It also situates FlowReasoner among work on LLM reasoning and efficiency.

LLM-Based Multi-Agent Systems

  • Prompt and hyperparameter optimization can automate agent design, but the workflow remains fixed and requires human design for new scenarios.Graph-based approaches automate workflow structure, while code-based approaches represent systems for optimization.Source: §2 · p. 3
  • ADAS [18], AFLOW [58], and MaAS [57] optimize code-represented workflows through meta-agent search, MCTS, and architecture search, respectively.ScoreFlow [49] trains a workflow generator using direct preference optimization.Source: §2 · p. 3
  • Earlier workflow-automation methods generate one system per task, which cannot automatically customize workflows for individual queries.This motivates a query-level meta-agent that generates a system for each query.Source: §2 · p. 3

Reasoning in LLMs

  • FlowReasoner develops an o1-like reasoning model as a query-level meta-agent, avoiding complex search algorithms and carefully designed search sets.The related work describes step-by-step prompting, self-correction, self-critique, debate, and plan-and-solve as reasoning approaches [25] [21] [29] [48].Source: §2 · p. 3

3 Problem Definition

The paper formalizes a multi-agent system as agents and their collaboration workflow, then contrasts manually designed task-level systems with search-based automation. Both task-level approaches produce systems for a task rather than adapting to each query.

3 Problem Definition

  • A user task is a distribution of queries, and a multi-agent system comprises agents plus their collaboration workflow.The paper denotes a user query by q and a task by t = P(q).Source: §3 · p. 3
  • Manual systems use human-designed agents and workflows fixed across a task, incurring human costs and failing to allocate inference resources dynamically across its queries.The paper represents this arrangement as Stask = H(t).Source: §3 · p. 3
  • Search-based systems generate candidate designs and search for an optimal task-level system, but remain one-size-fits-all and depend on time-consuming search sets unavailable for one query.Figure 2 contrasts manual, search-based, and reasoning-based automatic multi-agent systems.Source: §3 · pp. 3–4, Figure 2 · p. 4

4 Meta-Agent FLOWREASONER

FLOWREASONER generates a query-specific multi-agent system by reasoning from external execution feedback, with training that progresses from reasoning-data distillation and SFT warmup to reinforcement learning. It then iteratively refines a system for each query using execution feedback.

4 Meta-Agent FLOWREASONER

  • FLOWREASONER trains a query-level meta-agent to propose a multi-agent system for each user query, then evaluates that system’s result against ground truth.The training pipeline has three stages: reasoning-data distillation, reasoning SFT warmup, and reinforcement of reasoning from external execution feedback.Source: §4 · p. 4, Figure 3 · p. 5

4.1 Learn to Reason

  • R1-671B generates multi-round reasoning and systems with execution feedback, which are combined into training samples for SFT warmup of DeepSeek-R1-Distill-Qwen-7B.The warmup trains the model to produce both a reasoning process and a final multi-agent system from an instruction and query.Source: §4.1 · p. 5

4.2 Reinforce Reasoning from External Execution Feedback

  • GRPO further trains reasoning from external execution feedback, using normalized per-round rewards and subsequent rewards to calculate token-level advantages.Candidate scores incorporate solution pass rate, workflow complexity, and workflow diversity.Source: §4.2 · pp. 5–6

4.3 Generate Multi-Agent Systems with FLOWREASONER

  • For each query, FLOWREASONER represents system nodes and edges in code, applies predefined and custom operators, and iteratively refines the workflow using execution feedback.The pass rate of the proposed solution is the key performance indicator used as feedback.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate FLOWREASONER on code-generation benchmarks against individual models, manually designed workflows, and automated workflow methods. Results compare its performance with baselines and examine model size, training stages, worker choices, generalization, and generated workflows.

5 Experiments

  • Experiments focus on BigCodeBench, HumanEval, and MBPP, comparing individual models, manual workflows, and automated workflow methods.The method uses o1-mini as its worker, runs 10 workflow iterations, and evaluates code accuracy with pass@1.Source: §5 · pp. 6–7

5.1 Experiment Results

  • FLOWREASONER-14B improves overall accuracy by 5 percentage points over MaAS and exceeds o1-mini by 10%.The comparison covers three code benchmarks.Source: Table 1 · p. 7, §5.1 · pp. 7–8
  • Across all three benchmarks, FLOWREASONER-14B outperforms the 7B variant, while SFT+RL improves on SFT alone at each model size.Table 2 compares accuracy across model sizes and training stages.Source: Table 2 · p. 7, §5.1 · p. 8
  • Generated workflows are more complex for the engineering-oriented BigCodeBench task and more concise for the simpler, algorithm-focused HumanEval task.Figure 5 shows examples from both benchmarks.Source: Figure 5 · p. 8, §5.1 · pp. 8–9
  • FLOWREASONER maintains performance when its trained meta-agent is paired with alternative workers, suggesting transferability across execution agents.The tested alternatives include Qwen2.5-Coder, Claude, and GPT-4o-mini.Source: Table 3 · p. 9, §5.1 · p. 9
  • On BigCodeBench, open-source meta-agents paired with o1-mini and no initial workflow often produce error-prone workflows; o1-mini performs best among tested workers with Claude 3.5 as meta-agent.The authors attribute API-based models’ stronger performance likely to superior instruction-following.Source: §5.1 · p. 9

6 Conclusion

The conclusion presents FLOWREASONER as a query-level meta-agent that generates personalized workflows and reports stronger results than manual and automated alternatives across code-generation benchmarks.

6 Conclusion

  • FLOWREASONER generates a personalized multi-agent system for each user query rather than a fixed system for a task category.It uses reasoning-based optimization with external execution feedback and reinforcement learning guided by rewards for performance, complexity, and efficiency.Source: §6 · p. 9
  • FLOWREASONER-14B outperforms manual workflows and existing automated methods across multiple code benchmarks, improving o1-mini’s performance by 10.52% overall on three benchmarks.The conclusion also reports generalization when paired with different worker models.Source: §6 · p. 9
  • The authors conclude that query-specific workflow generation can reduce human resource costs and enhance scalability compared with fixed workflows for whole task categories.They also report improved adaptability and efficiency in multi-agent systems.Source: §6 · p. 9

A More Implementation Details

The implementation details specify workflow-feedback evaluation, the available operators, and training settings for supervised fine-tuning and reinforcement learning.

A More Implementation Details

  • Each workflow is evaluated three times to provide external execution feedback, using six base operators drawn from Aflow.The operators are Code Generator, Format Generator, Ensemble Operator, Review Operator, Revise Operator, and Code Test Operator.Source: A More Implementation Details · p. 14
  • Supervised fine-tuning uses approximately 1,400 R1-generated items, with batch size 1 per device, gradient accumulation over 2 steps, learning rate 1e-5, and 3 epochs.Training uses LLaMAFactory.Source: A More Implementation Details · p. 14
  • The reinforcement-learning settings are scaling factor k=1.1, threshold T=3, rollout number m=5, and a maximum of 5 episodes.Source: A More Implementation Details · p. 14

B More Cases of Workflow

The section introduces additional generated cases, including both successful and failed results.

B More Cases of Workflow

B.1 Successful Cases

Figures 6, 8, and 7 present successful FlowReasoner cases for BigCodeBench, HumanEval, and MBPP, respectively.

B.1 Successful Cases

  • Figures 6, 8, and 7 show successful FlowReasoner cases for BigCodeBench, HumanEval, and MBPP, respectively.Source: §B.1 · p. 14

B.2 Failure Cases

Figures 10, 11, and 9 show failed FlowReasoner workflows for BigCodeBench, HumanEval, and MBPP, respectively.

B.2 Failure Cases

Loading 2504.15257v1…