Source-linked AI summary
Multi-Agent Collaboration via Evolving Orchestration
Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, Xuantang Xiong, Lei Han, Zhiyuan Liu, Maosong Sun
TL;DR
Complex problem solving exposes scalability and efficiency limits in monolithic LLMs and coordination limits in static multi-agent organizations. This paper introduces a centralized puppeteer that dynamically sequences agents and adapts its policy through reinforcement learning, achieving better solutions with lower computational cost while producing compact cyclic reasoning structures.
Problem
Monolithic LLMs and static multi-agent topologies have limited scalability and efficiency for complex tasks as task complexity and agent numbers increase.
Method
A centralized puppeteer dynamically selects and sequences agents from evolving task states, using reinforcement learning to adapt orchestration and prune inefficient trajectories.
Results
Experiments across closed- and open-domain scenarios show superior solution quality with reduced computational cost, alongside compact cyclic reasoning structures.
Takeaways & Limitations
Dynamic, learnable orchestration supports flexible multi-agent collaboration while favoring more compact reasoning structures.
Takeaways & Limitations
The method uses coarse-grained rewards based on final outputs and token usage, assumes fixed agents and tools, and can exhibit mis-coordination or deceptive agreement.
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving. While recent research explores multi-agent collaboration among LLMs, most approaches rely on static organizational structures that struggle to adapt as task complexity and agent numbers grow, resulting in coordination overhead and inefficiencies. To this end, we propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states. This orchestrator is trained via reinforcement learning to adaptively sequence and prioritize agents, enabling flexible and evolvable collective reasoning. Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs. Analyses further reveal that the key improvements consistently stem from the emergence of more compact, cyclic reasoning structures under the orchestrator's evolution. Our code is available at https://github.com/OpenBMB/ChatDev/tree/puppeteer.
1 Introduction
Static multi-agent structures can become inefficient and difficult to scale as tasks and agent populations grow. The proposed puppeteer paradigm dynamically orchestrates agents and evolves their coordination toward effective, compact reasoning.
- Monolithic LLMs face increasing limitations on complex, multifaceted problems requiring tool use and collaboration.
- Static agent topologies lack flexibility and scalability, increasing coordination overhead and inefficiency as tasks and agent numbers grow.
- The centralized puppeteer dynamically selects and sequences agents according to evolving task states, prioritizing effective agents and suppressing less efficient ones.
- Reinforcement learning continuously updates the orchestrator policy, emphasizing strong agent trajectories and pruning less effective ones.
- Across closed- and open-domain scenarios, the approach produces more effective solutions with less computational overhead, associated with compact cyclic reasoning structures.
2 Method
The method models collaboration as sequential, centralized agent selection over a directed reasoning graph. Reinforcement learning optimizes this orchestration using rewards that jointly represent solution quality and computational cost.
- A centralized policy dynamically organizes diverse LLM agents and continually optimizes their collaboration through reinforcement learning.
- Each agent is represented as a foundation model, reasoning pattern, and available external tools, covering intrinsic and tool-augmented behaviors.
- The MAS is modeled as a directed graph whose nodes are agents and whose edges convey dependencies or intermediate information flow.
- Dynamic Orchestration: The orchestrator selects one agent per step from the current global state and task specification, then repeats selection after each state update.
- Dynamic Orchestration: The process stops under a designated criterion, after which an aggregation function combines agent outputs into the overall solution.
- Adaptive Evolution: REINFORCE updates the orchestration policy from complete reasoning trajectories, suppressing unhelpful or costly agents and favoring compact collaboration structures.
- Adaptive Evolution: The reward jointly accounts for solution quality and computational efficiency by penalizing excessive FLOPs or token-level expenditure.
3 Experiments
Experiments evaluate Puppeteer across closed- and open-domain tasks, model subspaces, and agent configurations. The evolved orchestrator improves performance while its learned structures become more compact, cyclic, and efficient.
- Datasets and Metrics: The evaluation spans closed- and open-domain reasoning datasets, including GSM-Hard, MMLU-Pro, and ALFWorld.Closed-domain benchmarks emphasize objective reasoning and accuracy, while ALFWorld connects language reasoning with embodied interaction.
- Baselines and Configurations: Puppeteer evaluates both homogeneous Mono agents and heterogeneous agents in separate Mimas and Titan model subspaces.Mimas contains smaller models, whereas Titan contains larger models; the Mono setting uses one model for all agents.
- Performance: Puppeteer achieves superior average performance during the evolved phase across evaluated tasks, regardless of domain type or model-space size.Puppeteer-Mono also performs robustly across large- and small-scale models.
- Performance: Puppeteer-Mono consistently outperforms competing workflows and multi-agent baselines using the same base model across nearly all evaluated tasks.The reported comparison attributes this to centralized orchestration, optimized reasoning strategies, tool use, and dynamically constructed topologies.
- Performance: 0.6893 to 0.7731: Puppeteer’s average Titan-subspace performance improves after continued optimization.A similar improvement trend is reported in the Mimas subspace.
- Efficiency and Organizational Evolution: Token consumption decreases across almost all settings, while Titan uses fewer active agents and evolved topologies show greater compaction and cyclicality.In Mimas, agent counts remain relatively stable, with token reductions attributed primarily to other orchestration changes; evolved examples contain fewer paths and more cycles.
4 Related Work
LLM-based agents have expanded multi-agent systems’ capabilities, but early fixed structures created rigid coordination and limited flexibility. More adaptive approaches therefore seek dynamic agent selection and optimizable collaboration graphs.
- Autonomous LLM-based agents increasingly support planning, memory, tool usage, complex tasks, dynamic environments, collaboration, and decision-making.
- Early multi-agent systems used fixed handcrafted structures that produced rigid coordination, limited workflow flexibility, and suboptimal agent composition.
- Later methods introduced adaptive orchestration through dynamic agent selection and optimizable graphs for prompt refinement and improved cooperation.
5 Conclusion
The paper presents a centralized, learnable puppeteer that dynamically activates agents in a directed graph-of-thoughts and adapts its policy through reinforcement learning. Experiments report better solution quality at lower computational cost, associated with compact cyclic reasoning structures.
- The framework centrally and dynamically activates LLM agents within a directed graph-of-thoughts, replacing static or manually designed topologies.
- Reinforcement learning adapts the orchestration policy to improve collaboration efficiency and reduce redundant coordination.
- Experiments across diverse tasks achieved superior solution quality while reducing computational cost.
- Analyses link the performance improvements to compact, cyclic reasoning structures promoted by the orchestrator.
- The authors position the framework as a step toward dynamic and scalable multi-agent coordination.
A.1 Token Cost Analysis
The appendix extends cost analysis across mono-agent and multi-agent settings in the Titan and Mimas subspaces. In Titan, lower token consumption is mainly associated with shorter reasoning paths, while Mimas shows less consistent reductions.
- The supplementary analysis covers mono-agent and multi-agent cost results across Titan and Mimas, adding three configurations beyond the main paper’s multi-Titan results.
- In Titan, token consumption decreases across most tasks, primarily because the optimized systems use shorter reasoning paths.
- GSM-hard maintains a relatively stable number of agents, suggesting its cost reduction may instead reflect fewer-token prompts or more concise prompting.
- In Mimas, token consumption decreases for some tasks but lacks a consistent downward trend for others.
- Because mono-agent Mimas uses a shared base model, identifying substantially more token-efficient agents is more difficult.
A.2 Performance–Cost Trade-off Improvement
The performance–cost analysis tracks a score combining task performance and token cost throughout reinforcement-learning optimization. Across most tasks, the score rises as solutions improve and token use becomes more efficient, although limited samples and fixed settings constrain some tasks.
- The cost score is defined as task performance divided by token cost, jointly measuring solution quality and reasoning efficiency.
- The evaluation separates optimization into initial and evolved phases for convenience, not because model behavior changes intrinsically.
- Performance and cost generally improve online with minor stochastic fluctuations under mono-agent and multi-agent settings.
- Across most tasks, performance/cost scores trend upward as optimization discovers more suitable agent organizations and reasoning pathways.
- Some trajectories initially decline before recovering, indicating that early exploration can test weaker configurations before improved structures emerge.
- A small subset of tasks lacks a significant upward trend, possibly because 200 optimization samples or current sequence-length settings are insufficient.
- Further gains may require larger optimization budgets or hyperparameters adapted to task difficulty and model capability.
A.3 Emergent Patterns in MAS Behavior
The puppeteer guides diverse multi-agent trajectories, producing both validated and novel organizational patterns. The framework also extends to embodied tasks, where it supports step-by-step interaction in a simulated environment.
- Emergent MAS trajectories: Puppeteer-guided trajectories exhibit validated structures alongside novel organizational behaviors emerging from agent interactions.These visualizations are presented as evidence of behavioral diversity and coordination capabilities.
- Emergent MAS trajectories: The policy dynamically organizes agents into distinct formations in response to high-level guidance.The resulting patterns are described as demonstrating generalization beyond behaviors explicitly encoded during training.
- Embodied extension: Embodied tasks add real-time environmental interaction and sequential action challenges beyond the paper’s main reasoning, coding, and writing experiments.These tasks require planning, reasoning, and decision-making under dynamic feedback.
- Embodied extension: ALFWorld serves as the embodied testbed, using agent chains capped at 50 steps and one exploration trajectory because of environment constraints.The benchmark bridges textual reasoning with embodied interaction through a simulated environment and natural-language interface.
- Embodied extension: A THOR case study shows successful step-by-step execution of an embodied task, supporting the framework’s applicability to interactive environments.The example task is finding a remote control under a floor lamp.
B Agent Implementation Details
Agents are implemented through role-conditioned and action prompts that distinguish tool use from internal reasoning patterns. Tool prompts produce structured invocation parameters, while reasoning prompts activate planning, critique, reflection, and related cognitive processes.
- Agent action categories: The system categorizes agent actions into Tool-use Agent and Reasoning Agent types.Tool-use agents interact with external environments, whereas reasoning agents implement internal cognitive processes.
- Agent prompts: Role prompts specify each agent’s identity, expertise, and responsibilities, while action prompts guide tool use or reasoning patterns.Each structured prompt pairs a defined action with a language-model invocation template.
- Tool-use prompts: Tool-augmented action prompts generate structured parameters for external tool invocation rather than executing tools directly.Examples include producing search queries, URLs, or other interface parameters.
- Reasoning prompts: Reasoning-pattern prompts activate planning, reasoning, critique, reflection, and sub-question generation without relying on external tools.They support complex problem-solving and decision-making through generative reasoning.
- Training resources: Table 3 reports average computational resources for orchestrator training, but direct cost comparisons with inference-time-search baselines are inappropriate.Online training interleaves parameter updates with multi-agent inference, making isolated GPU-hour measurement difficult.
D Limitations
The framework’s optimization relies on coarse final-output and token-usage rewards, and its current design assumes fixed agents and tools. The authors also identify occasional coordination failures and call for finer supervision and stronger interaction mechanisms.
- Optimization feedback: The Puppeteer currently optimizes with coarse-grained rewards based only on final outputs and token usage.The authors identify missing intermediate feedback as a limitation.
- Optimization feedback: Fine-grained supervision, such as step-level correctness, is proposed to improve optimization efficiency.This is presented as a future refinement rather than a current capability.
- Adaptability: The framework assumes a fixed set of agents and tools, limiting adaptation to task variations during inference.The authors propose dynamic agent or tool modification to improve flexibility and robustness.
- Interaction reliability: Occasional mis-coordination or deceptive agreement among agents motivates more robust interaction protocols and incentive designs.Future work includes reward shaping and adaptive mechanisms for more context-aware orchestration.