PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Self-Orchestration in Decentralized Agent Swarms

Guowei Xu, Zhenting Qi, Amani Kiruga, Himabindu Lakkaraju, Yilun Du

arXiv:2610.22322v1cs.MAcs.AIcs.CL

TL;DR

Existing multi-agent systems often keep orchestration fixed, despite the coupling between participation and agents’ learning. MPT jointly trains action and bidding policies so decentralized swarms can adapt who participates and how collective outcomes provide credit. It outperforms monolithic post-training across three benchmarks, and trained swarms also adapt to related tasks and benefit from larger inference-time populations.

  • Problem

    Many multi-agent systems use fixed orchestration, although participation determines which agents gain experience and learning can change who should act.

    Source: §1 · p. 2

  • Method

    MPT uses bids to determine participation and connects collective outcomes to local credit assignment while jointly training action and bidding policies.

    Source: §1 · p. 2

  • Results

    Across embodied crafting, robot navigation, and deep research, MPT gains over monolithic post-training; on Iron Pickaxe, mean episode reward improves 63.4% relative to Graph-of-Agents.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Trained swarms adapt faster to related objectives and benefit from larger inference-time populations while better tolerating agent unavailability.

    Source: §1 · p. 2, §5 · p. 9

    MPT relies on externally evaluated task rewards, and its effectiveness depends on their quality; subjective outcomes and settings with difficult-to-obtain reliable feedback were not evaluated.

    Source: D POTENTIAL LIMITATIONS · p. 37

Abstract

from arXiv · show

The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. Code and trained models are available at https://github.com/Embodied-Minds-Lab/MPT.

1 INTRODUCTION

The introduction frames decentralized swarms’ fixed orchestration as a limitation because participation and learning are coupled. It presents MPT as a way to learn action and self-orchestration together, with gains across benchmarks and benefits beyond the original tasks.

1 INTRODUCTION

  • Existing multi-agent systems often use predefined interaction structures, leaving orchestration fixed even when agents are trainable.Participation shapes which agents receive experience, while learning can change who should participate.Source: §1 · p. 2
  • MPT jointly learns action policies and self-orchestration in decentralized swarms, using bids to adapt participation as agents improve.Collective outcomes provide local training credit without a central coordinating agent.Source: §1 · p. 2
  • On Iron Pickaxe, MPT achieves a 63.4% relative improvement in mean episode reward over Graph-of-Agents, the strongest alternative.The coordination strategies use the same post-trained action policies.Source: §1 · p. 2
  • Across embodied crafting, robot navigation, and deep research, MPT gains over monolithic post-training, while participation evolves during learning.Both bidding and action benefit from learning, and learned bids provide more effective action-training signals.Source: §1 · p. 2
  • Trained swarms adapt faster to new crafting objectives, while larger inference-time populations improve performance and better tolerate agent unavailability.The paper presents these findings as evidence of reuse, scalability, and robustness.Source: §1 · p. 2

2 METHOD

MPT jointly learns specialized action policies and self-orchestration in decentralized swarms. Agents bid to determine who acts, and bids also connect participation to credit assignment.

2.1 SELF-ORCHESTRATION THROUGH BIDDING

  • At each turn, agents independently bid their willingness to participate; the highest bidder acts, and that bid sets the share of accumulated value passed backward.This allocates turns using local assessments without a central dispatcher.Source: §2.1 · p. 3

2.2 LEARNING TO ACT

  • After an episode, rewards propagate backward through acting agents: each retains a fraction of accumulated value as local credit and passes the remainder to the preceding participant.The selected bid determines the fraction retained; downstream outcomes can therefore provide learning signals for earlier actions.Source: §2.2 · p. 3, Figure 3 · p. 4
  • Each agent updates its parameters with REINFORCE using only action tokens from turns it took and a per-agent mean-credit baseline.The baseline is the agent’s mean training credit over its assigned turns in the batch.Source: §2.2 · p. 4

2.3 LEARNING TO SELF-ORCHESTRATE

  • For contested turns, the method compares the collected trajectory with an alternative rollout where the runner-up acts, then trains bidding preferences toward the higher-return branch.Turns with equal branch returns are discarded; if resampling fails to make the selected bid lower than the runner-up’s, that turn is also excluded from bidding training.Source: §2.3 · p. 4
  • Action and bidding learning alternate in four action-training steps followed by one bidding-training step, optimizing the same parameters within each agent so capabilities and participation can co-adapt.Preference learning increases the relative likelihood of the bid associated with the better outcome.Source: §2.3 · pp. 4–5

3 EXPERIMENTS

Experiments compare MPT with monolithic reinforcement learning across embodied crafting, robot navigation, and deep research, then examine learned participation, coordination, and adaptability. Results cover matched-compute performance, component ablations, transfer to new objectives, inference-time scaling, and dropout robustness.

3.1 SETUP

  • The setup uses three role-specialized agents per task, independent agent parameters, and SFT initialization, comparing MPT with monolithic REINFORCE under matched SFT compute.The agent roles and task environments differ across embodied crafting, robot navigation, and deep research.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • ≈41 vs 20 mean episode reward in crafting, ≈0.9 vs 0.3 in navigation, and ≈27% vs 6% mean accuracy in research favor MPT at matched training FLOPs.Held-out evaluation also reports MPT outperforming monolithic RL on all three tasks.Source: Figure 5 · p. 5, §3.2 · p. 6, Table 1 · p. 6

3.3.1 SELF-ORCHESTRATION EVOLVES DURING TRAINING

  • At decision step 120, Logger participation falls from approximately 80% before RL to 45% after RL, while Miner rises from approximately 10% to 40%.Smith participation rises from approximately 10% to 15%; the Logger remains more prevalent early in episodes.Source: §3.3.1 · p. 6, Figure 6 · p. 6

3.3.3 BOTH BIDDING AND ACTION BENEFIT FROM LEARNING

  • 41.5 mean reward with both components post-trained exceeds 21.5 with trained bidding only and 27.2 with trained actions only; the initial configuration scores 15.4.The fully post-trained configuration achieves 2.69 times the initial configuration’s reward.Source: §3.3.3 · p. 7, Table 2 · p. 7

3.3.2 COMPARISON BETWEEN SELF-ORCHESTRATION AND OTHER COORDINATION STRATEGIES

  • 41.5 vs 25.4 mean episode reward gives MPT a 63.4% relative improvement over Graph-of-Agents, the strongest alternative, using the same post-trained action policies.The comparison evaluates coordination strategies on Iron Pickaxe over 32 seeds.Source: §3.3.2 · pp. 6–7, Table 2 · p. 7

3.3.4 ISOLATING THE CONTRIBUTIONS OF LEARNED SELF-ORCHESTRATION

  • At approximately 5×10^18 training FLOPs, MPT reaches about 42 mean reward, versus 15 for round-robin and 31 for random credit.The ablations remove bidding or replace bids used for credit assignment with random values.Source: §3.3.4 · p. 7, Figure 7 · p. 7

3.4 REUSABILITY, SCALABILITY, AND ROBUSTNESS

  • At approximately 10^18 adaptation FLOPs, the society scores about 35 on Torch and 38 on Stone Bricks, versus 17 and 19 for monolithic RL.Both methods continue training from their respective Iron Pickaxe checkpoints.Source: §3.4 · pp. 7–8, Figure 8 · p. 8
  • Increasing the population from 3 to 12 raises pass@3 on crafting milestones, while at 90% dropout the 12-agent swarm scores about 33 reward versus 19 for the 3-agent swarm.The larger population also improves performance across all five evaluated subtasks and better preserves capability under severe dropout.Source: §3.4 · p. 8, Figure 9 · p. 8, Figure 10 · p. 8

4 RELATED WORK

Related work spans LLM multi-agent systems, multi-agent reinforcement learning, and economic coordination. MPT is distinguished by learning action and participation policies together through weight updates and learned bids.

LLM-based multi-agent systems

  • LLM multi-agent systems use role specialization, communication, memory, workflows, or adaptive routing; MPT instead improves capabilities and self-orchestration through weight updates.Examples include Generative Agents (Park et al., 2023), CAMEL (Li et al., 2023), GPTSwarm, AgentNet (Yang et al., 2025), and Graph-of-Agents (Yun et al., 2026).Source: §4 · p. 9

Multi-agent reinforcement learning

  • Prior multi-agent reinforcement learning addresses decentralized policies, value factorization, credit assignment, communication, or LLM-agent training, but does not connect next-agent selection with downstream reward distribution.MPT couples turn allocation and credit assignment through learned bids.Source: §4 · p. 9

Economics in agents

  • Economic coordination work ranges from market allocation and credit assignment to LLM-agent auctions, while recent LLM economic systems primarily evolve prompts with frozen weights.MPT uses feedback to train both action and bidding policies.Source: §4 · p. 9

5 CONCLUSION

MPT jointly learns agents’ action policies and self-orchestration, using learned bids to connect participation with local credit assignment. Its training alternates action updates with bidding updates based on counterfactual comparisons.

5 CONCLUSION

  • MPT jointly learns individual action policies and self-orchestration in decentralized swarms, using bids to link participation decisions with local credit assignment.This lets agents improve how they act and when they participate without a central coordinator.Source: §5 · p. 9
  • Trained swarms adapt faster to related objectives, benefit from larger inference-time populations, and better tolerate agent unavailability as population size grows.Source: §5 · p. 9

A PSEUDO CODE

  • The pseudocode alternates action and bidding updates, using credit assignment for action training and counterfactual comparisons for bidding training.Action updates use collected trajectories; bidding updates compare the original branch with an alternative rollout.Source: A PSEUDO CODE · p. 16

B.1.1 TASK AND SIMULATOR

This section describes the Minecraft iron-pickaxe task and simulator setup, including staged crafting and mining, role-specialized agents that bid for turns, capped rewards and episode limits, and cold-start SFT using demonstrations from actual interactions.

B.1.1 TASK AND SIMULATOR

  • Agents pursue an iron pickaxe from an empty inventory in randomly seeded Minecraft worlds, following a tech tree from gathering logs through crafting and smelting; runs use 1,614 seeds.The simulator is Minecraft through MineStudio.Source: §B.1.1 · pp. 16–17

B.1.2 REWARD

  • The environment pays capped, staged rewards for 12 tech-tree events, with a maximum episode reward of 140.Event rewards range from 1 for mining a log to 30 for crafting the iron pickaxe.Source: §B.1.2 · p. 17

B.1.3 EPISODE, OBSERVATION AND DECISION

  • Each 120-decision episode gives the policy a 128 × 128 RGB screenshot and structured text state, then executes its natural-language instruction for 40 game frames using frozen STEVE-1.The text includes the task and crafting rules, step, events, world status, recent history, inventory, and previous instruction outcome.Source: §B.1.3 · p. 17

B.1.4 AGENT DESIGN

  • Three separately fine-tuned Qwen3-VL-4B-Instruct agents—LOGGER, MINER, and SMITH—compete in a sealed-bid auction, with role-specific verbs constraining their actions.Bids use five revenue-share levels from 0 to 0.8; the highest bid wins, and each role can issue only instructions beginning with its permitted verbs.Source: §B.1.4 · pp. 17–20
  • The Minecraft task is to obtain an iron pickaxe through staged gathering and crafting, with mining drops dependent on having the appropriate pickaxe equipped.The supplied rules specify that iron ore requires a stone-or-better pickaxe to yield ore, which is then smelted into ingots for the final recipe.Source: §B.1.4 · p. 22
  • The Smith desk wins an auction to issue one instruction per turn, while its parser restricts commands to craft, equip, or smelt and rejects actions with missing ingredients.A low-level STEVE-1 controller executes instructions for about two seconds; crafting verbs are handled exactly by the server.Source: §B.1.4 · pp. 22–23

B.1.5 SFT COLD START

  • Cold-start SFT uses 4,791 Claude Sonnet 5 demonstrations from Minecraft interactions—1,677 LOGGER, 2,380 MINER, and 734 SMITH—on matched compute for monolithic and role-specific agents.All model parameters are fine-tuned in bf16 with AdamW and a cosine learning-rate schedule; each model is trained on a single B200 GPU.Source: §B.1.5 · p. 23

B.1.6 POST-TRAINING

Embodied crafting post-training uses Minecraft worlds generated from 1,614 random seeds, with environments run in parallel and reinforcement learning performed on B200 GPUs.

B.1.6 POST-TRAINING

  • The post-training dataset contains 1,614 random seeds, each generating a different Minecraft world.Source: §B.1.6 · p. 23
  • The team runs 32 Minecraft worlds in parallel on four RTX PRO 6000 GPUs, alongside the STEVE-1 environments.Source: §B.1.6 · p. 23
  • Reinforcement learning is performed on four B200 GPUs.Source: §B.1.6 · p. 23

B.2.1 TASK AND SIMULATOR

The navigation experiments use VLN-CE in Habitat, with specialist agents sharing control through bids and each agent restricted to its assigned action type. The setup includes a waypoint reward, an SFT cold start, and a separate post-training dataset.

B.2.1 TASK AND SIMULATOR

  • VLN-CE evaluates whether an agent follows an R2R route instruction through a Matterport3D house and stops within 3 m of the goal.The evaluation uses Habitat, with 0.25 m forward steps and 15° turns as atomic simulator actions.Source: §B.2.1 · p. 23

B.2.2 REWARD

  • The waypoint reward combines geodesic-distance progress with a success bonus, using λd = 1.0 and λs = 2.5.Success means issuing stop within 3 m of the goal.Source: §B.2.2 · pp. 23–24

B.2.3 EPISODE, OBSERVATION AND DECISION

  • Each episode has at most 64 decisions, and policies receive an instruction plus an eight-frame window of 512 × 512 egocentric RGB images.The model outputs reasoning followed by a turn, forward move, or stop action.Source: §B.2.3 · p. 24

B.2.4 AGENT DESIGN

  • TURNER, MOVER, and STOPPER are separate NaVILA-8B fine-tunes that share observations but own turning, forward movement, and stopping, respectively.A sealed-bid auction selects one agent to act at each step; each specialist bids only when its action can advance the task.Source: §B.2.4 · pp. 24–25

B.2.5 SFT COLD START

  • The SFT cold start uses successful oracle trajectories with 1,931 decisions, annotated by Claude Sonnet 5, and fine-tunes each society agent on its own role’s data.The monolithic agent is fine-tuned on the combined role data.Source: §B.2.5 · p. 26

B.2.6 POST-TRAINING

  • Navigation post-training uses 10,744 R2R-CE training episodes that differ from the SFT episodes.Habitat runs on two RTX PRO 6000 GPUs, while reinforcement learning runs on four B200 GPUs.Source: §B.2.6 · p. 26

B.3.1 TASK AND ENVIRONMENT

The deep-research setup uses a searchable corpus and three browser tools, with episodes ending in a correctness-graded answer. Three specialized agents share the conversation, and the models receive a supervised fine-tuning cold start before evaluation.

B.3.1 TASK AND ENVIRONMENT

  • OpenResearcher deep research asks agents to browse a local web corpus with three tools and return a short answer.Questions come from OpenResearcher-Dataset and the corpus is OpenResearcher-Corpus, served by BM25 search.Source: §B.3.1 · p. 26

B.3.2 REWARD

  • The episode receives a single terminal reward when the LLM judge marks the final answer correct.The reward is binary and tied to the answering turn.Source: §B.3.2 · p. 27

B.3.3 EPISODE, OBSERVATION AND DECISION

  • Episodes last at most 64 decisions, and each observation contains the conversation history, question, prompt, and tool responses.The prompt budget is 16,384 tokens.Source: §B.3.3 · p. 27

B.3.4 AGENT DESIGN

  • SEARCHER, OPENER, and FINDER are separate fine-tuned agents, each restricted to its corresponding browser tool; any may submit the final answer.Agents bid for each turn, with the highest bid winning; the bidding prompt offers five levels from 0 to 0.8.Source: §B.3.4 · pp. 27–29

B.3.5 SFT COLD START

  • A supervised fine-tuning cold start uses browsing trajectories generated by gpt-oss with the same tools and corpus.The 568 assistant turns are divided among SEARCHER (222), OPENER (273), and FINDER (73); each society agent is fine-tuned on the action turns it owns.Source: §B.3.5 · pp. 30–31

B.3.6 POST-TRAINING

Deep-research post-training uses questions distinct from the supervised fine-tuning set and runs reinforcement learning on four H200 GPUs.

B.3.6 POST-TRAINING

  • The post-training dataset contains 5,000 OpenResearcher questions that differ from the SFT questions.Search services run on CPU nodes.Source: §B.3.6 · p. 31
  • Reinforcement learning is performed on four H200 GPUs.Detailed settings are listed in Table 5.Source: §B.3.6 · p. 31, Table 5 · p. 31
  • Table 5 presents hyperparameters for the Deep Research experiment.The passage identifies the table’s scope but does not provide its hyperparameter values.Source: Table 5 · p. 31

C.1 EMBODIED CRAFTING

Episode walkthroughs show how trained agents divide work in embodied crafting and robot navigation, with bids selecting which agent acts at each illustrated decision.

C.1 EMBODIED CRAFTING

  • In the crafting episode, LOGGER gathers logs, SMITH crafts tools and components, and MINER mines materials before SMITH crafts the iron pickaxe.Figure 11 plots cumulative reward and colors each rewarded decision by the agent that won the auction.Source: §C.1 · p. 31, Figure 11 · p. 31
  • The illustrated crafting decisions show SMITH winning bids to craft planks, a crafting table, and a wooden pickaxe, while MINER wins to dig for stone.The figure panels also show LOGGER winning an early bid to chop a tree and SMITH later smelting iron and crafting an iron pickaxe.Source: Figure 13 · p. 32, Figure 14 · p. 32, Figure 12 · p. 32

C.2 ROBOT NAVIGATION

  • At decision 2, MOVER wins and moves forward 75 cm as reward rises by 0.75 and distance falls from 6.9 m to 6.1 m.At decision 17, STOPPER wins with a stop action; the figure reports +2.50 reward, 2.0 m to 2.0 m, and success.Source: Figure 20 · p. 34, Figure 19 · p. 34, Figure 24 · p. 35

C.3 DEEP RESEARCH

A deep-research episode illustrates how the three-agent society searches, opens, and checks sources to answer a question about a film’s filming location. The society returns the reference answer, Libya, after six decisions.

C.3 DEEP RESEARCH

  • The society answers the question about where “Legend of the Lost” was filmed correctly after six decisions, returning Libya.The judge reports an exact match with the reference answer.Source: §C.3 · pp. 36–37
  • SEARCHER queries, OPENER opens the Wikipedia result, and FINDER searches the page before two confirmation lookups and the final answer.The opened page says location shooting took place near Tripoli, Libya.Source: §C.3 · pp. 36–37
  • Table 6 lists every decision in the episode and gives bids in SEARCHER/OPENER/FINDER order.The accompanying example panels show the three bids, the winner’s response, and its tool output.Source: Table 6 · p. 36, §C.3 · p. 36

D POTENTIAL LIMITATIONS

MPT’s evaluation is bounded by its dependence on externally evaluated rewards, same-state counterfactual rollouts, and computationally constrained training scale.

D POTENTIAL LIMITATIONS

  • MPT relies on externally evaluated task rewards to train both action and bidding policies, and subjective or unreliable-feedback settings remain unevaluated.Source: D POTENTIAL LIMITATIONS · p. 37
  • Bidding updates require counterfactual rollouts from the same environment and agent states, leaving extension to irreversible real-world interactions an open problem.Source: D POTENTIAL LIMITATIONS · p. 37
  • Computational constraints limited study scale; training swarms with multiple models larger than 10B parameters each or substantially larger populations was not explored.Source: D POTENTIAL LIMITATIONS · p. 37

Related papers

Loading 2610.22322v1…