PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces

Chenchen Zhang

arXiv:2605.02801v1cs.CL

TL;DR

RL for LLM-based multi-agent systems lacks a unified account of how orchestration decisions and credit assignment should be optimized at realistic scale. This paper frames the problem with replayable orchestration traces and taxonomies of rewards, credit units, and sub-decisions, finding sparse message-level credit, no explicit RL method for learned stopping, and a gap between academic evaluations and publicly reported deployment envelopes.

  • Problem

    Open research lacks joint trace-level benchmarks and explicit RL treatments of multi-agent credit assignment and stopping at industrial-scale orchestration lengths.

  • Method

    The paper models multi-agent interaction as graph-structured orchestration traces and organizes the literature across reward families, credit-bearing units, and five orchestration sub-decisions.

  • Results

    The review finds especially sparse explicit counterfactual message-level credit, no explicit RL training method for learned stopping, and a scale gap between open academic evaluations and publicly reported deployment envelopes.

  • Takeaways & Limitations

    Orchestration traces and the accompanying taxonomies provide a supported starting point for comparing RL methods and identifying infrastructure needs for industrial-scale multi-agent evaluation.

  • Takeaways & Limitations

    The 84-entry corpus is curated rather than exhaustive and lacks database exports, deduplication logs, and a multi-annotator screening protocol.

Abstract

from arXiv · show

As large language model (LLM) agents evolve from isolated tool users into coordinated teams, reinforcement learning (RL) must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped. This paper studies RL for LLM-based multi-agent systems through orchestration traces: temporal interaction graphs whose events include sub-agent spawning, delegation, communication, tool use, return, aggregation, and stopping decisions. Using this lens, we identify three technical axes. First, reward design spans eight families, including orchestration rewards for parallelism speedup, split correctness, and aggregation quality. Second, reward and credit signals attach to eight credit- or signal-bearing units from token to team; explicit counterfactual message-level credit remains especially sparse in our curated pool. Third, orchestration learning decomposes into five sub-decisions: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop. In our curated pool as of May 4, 2026, we found no explicit RL training method for the stopping decision. We connect academic methods to public industrial evidence from Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code. The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces. We release the artifact at https://github.com/xxzcc/awesome-llm-mas-rl, including an 84-entry tagged paper pool, a 32-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces.

1 Introduction

The paper frames reinforcement learning for LLM-based multi-agent systems around orchestration traces: event graphs capturing how teams spawn, delegate, communicate, use tools, aggregate, and stop. It develops taxonomies for rewards, credit, and orchestration learning, while connecting academic methods with public industrial evidence and a curated artifact.

  • Unifying thesis: The paper analyzes LLM-MAS RL through orchestration traces—temporal event graphs—rather than only per-agent trajectories.The trace abstraction covers spawning, delegation, communication, tool use, return, aggregation, and stopping decisions.
  • Scope and artifact: The curated pool contains 84 retained entries: 51 focal LLM-MAS entries and 33 classical-MARL, safety, single-agent-RL, benchmark, and evaluation references.The pool is deliberately curated rather than exhaustive, and supports taxonomies rather than field-wide prevalence estimates.
  • Taxonomy formalism: Its taxonomy covers reward design, eight credit- or signal-bearing units, and five orchestration sub-decisions.The orchestration decisions are when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop.
  • Industrial–academic bridge: The paper bridges open academic methods with Kimi PARL, OpenAI Codex, and Anthropic Claude Code, while treating industrial scale figures as reported deployment envelopes.Kimi K2.5 reported up to 100 sub-agents and 1,500 coordinated steps/tool calls; K2.6 reported 300 sub-agents and 4,000 coordinated steps.
  • Scope and artifact: The artifact includes entry cards and identifies fifteen open problems spanning algorithmic, reward, systems, safety, and evaluation axes.Appendix A provides uniform one-card summaries for thirteen core methods, frameworks, and industrial anchors.

2 Background: From MARL to LLM-MARL

Classical MARL provides the Dec-POMDP, CTDE, and credit-assignment foundations, while single-agent LLM RL provides PPO, GRPO, verifiable rewards, and agentic rollouts. LLM-MAS differs through natural-language actions, variable observations, dynamic agent counts, graph-structured orchestration, and new credit-bearing units.

  • Classical MARL: A Markov game extends an MDP to n agents with individual actions, observations, policies, joint-action transitions, and cooperative, competitive, or mixed-motive rewards.Partial observations yield a decentralized partially-observable MDP (Dec-POMDP).
  • Classical MARL: Classical MARL is organized around centralized training with decentralized execution and value decomposition versus counterfactual baselines.VDN, QMIX, MADDPG, and MAPPO exemplify CTDE, while COMA uses a counterfactual baseline.
  • Single-agent LLM RL: Single-agent LLM RL progresses from preference-based RLHF and RLAIF to verifiable RLVR, reasoning RL with GRPO, and agentic RL for multi-turn tool use.Reward signals become denser and credit assignment moves from trajectories toward tokens, steps, turns, and tool calls.
  • LLM-MAS differences: LLM-MAS uses natural-language messages, tool invocations, and sub-agent spawning as actions, while observations are long, partially summarized, variable, and graph-structured.These properties make classical MARL machinery difficult to apply directly.
  • LLM-MAS differences: LLM-MAS makes the number of agents dynamic and learnable, and introduces credit- and signal-bearing units that classical MARL and single-agent LLM RL do not natively handle.The survey therefore focuses on orchestration traces and multi-agent credit assignment.

3 A Working Abstraction for the Orchestration Trace

This section introduces orchestration traces as temporal event graphs for analyzing LLM multi-agent RL, extending dynamic-agent formalisms with policy-determined events. It frames value conditioning, credit diffusion, and counterfactual identification as central technical consequences of this abstraction.

  • Trace representation: Orchestration traces are rooted, edge-labelled temporal graphs whose policy-determined events include decisions, spawns, messages, tools, returns, and aggregations.Edges encode temporal or causal dependencies, while vertices carry agent, role, and structured-content labels.
  • Scope: The working abstraction is a taxonomy formalism, not a new MARL solution concept, and leaves off-policy evaluation and variable-shape value-function forms for future theory.It does not establish equivalence to existing dynamic-agent MARL formalisms.
  • Relation to existing formalisms: The abstraction extends fixed-agent Dec-POMDP and Markov-game settings with dynamic populations, delegation, communication, tool use, and aggregation events.Dynamic-Dec-POMDP notation permits time-indexed agent sets and spawn/despawn actions under an orchestrator’s privileged control.
  • Value conditioning: Value functions are naturally conditioned on growing or shrinking event graphs rather than fixed-shape joint states, motivating graph- or trace-conditioned critics.A spawn event can make G≤t+1 contain more vertices than G≤t; this is a trace-conditioned value identity, not a convergence claim.
  • Credit and counterfactuals: Uniform allocation of a shared terminal reward across n trace units makes per-unit signals increasingly difficult to distinguish from baseline noise as trace length grows.The exact scaling depends on the noise model, while same-prefix spawn credit remains non-identifiable from realized on-policy traces without branch coverage or structural assumptions.

4 System Forms: How LLM Agent Teams Are Organized

Agent-team topology determines which rewards and credit assignments are definable, while public industrial systems reveal deployment envelopes and harness constraints that exceed most open academic evaluation regimes. The section distinguishes documented orchestration mechanisms from deployment-shape evidence and emphasizes that this scale gap concerns rollout and harness conditions, not independently verified industrial training traces.

  • 4.1 A typology of agent-team topologies: Six recurring topologies are not mutually exclusive, and production systems typically combine two or three; topology constrains definable rewards and credit levels.Centralized orchestrators favor orchestrator-level credit, whereas debate topologies make message-level credit more natural.
  • 4.2 Public industrial evidence and selection rule: Industrial anchors are selected for disclosed orchestration mechanisms, stable interfaces constraining RL design, or long-running parallel workflows beyond open academic benchmarks.Product documentation supports deployment shape, scale, interfaces, and workflows, but does not by itself make an algorithm reproducible.
  • 4.2.1 Kimi Agent Swarm (K2.5 / K2.6): Kimi K2.5 reports a learned orchestrator using PARL, scaling to up to 100 sub-agents and 1,500 coordinated steps / tool calls; K2.6 reports 300 sub-agents and 4,000 coordinated steps.Kimi’s reward combines rperf + λ1rparallel + λ2rfinish, while Critical-Steps provides orchestrator-level credit against pseudo-parallelism.
  • 4.2.1 Kimi Agent Swarm (K2.5 / K2.6): Kimi publicly supports claims about learned orchestration, reward shaping, and orchestrator-level signals, but not reproduction of the full training recipe or a uniform industrial scale-gap claim.The scale-gap argument is anchored by Kimi; Codex and Claude Code primarily evidence harness and workflow shape.
  • 4.2.2 OpenAI Codex (app + harness): Codex illustrates that the deployed unit is a fixed harness—model ⊕tools ⊕prompts ⊕execution logic—whose boundary constrains training observations, actions, and inference.Agent Lightning’s execution/training decoupling is presented as an academic articulation of this constraint.
  • 4.2.3 Anthropic Claude Code (sub-agents + agent teams): Claude Code makes sub-agent spawning, delegation, and aggregation first-class RL objects, while sub-agents increase difficulty of understanding and steering workflows during execution.This creates a credit-assignment-shaped question about where humans can intervene in the trace.
  • 4.3 What systems reveal that papers do not: Academic methods mostly evaluate 10–100-step traces, whereas Kimi reports 1,500–4,000-step traces; no open academic method in the curated pool reports training at those lengths.Credit diffusion becomes qualitatively worse at the longer trace lengths.
  • 4.3 What systems reveal that papers do not: Across the three industrial systems, stable harness boundaries expose constraints largely absent from academic methods, making the gap primarily one of rollout shape and harness conditions.The section frames this as a difference between public deployment envelopes and open academic evaluation regimes, not primarily an algorithmic gap.

5 Systems Engineering: Rollout Cost and Harness Boundary

This section identifies three engineering constraints that determine whether LLM-MAS RL methods can reach industrial scale: rollout cost, the training-frozen harness boundary, and credit fragility on long traces. These constraints expose gaps between academic training regimes and deployment conditions.

  • 5.1 Rollout Cost: Industrial multi-agent rollouts are substantially more expensive than single-agent rollouts because costs accumulate across sub-agent tokens, tool calls, trace events, and orchestration overhead.A single-agent rollout is typically 10^2–10^3 tokens with one or two tool calls; multi-agent cost grows with team activity and trace length.
  • 5.1 Rollout Cost: A Kimi K2.6 operating point with K=300 and roughly 4,000 coordinated steps / tool calls can yield a rollout one to several orders larger than a short single-agent reasoning rollout.The estimate is a schematic proxy whose ratios depend on token lengths, tool latencies, and harness overhead.
  • 5.1 Rollout Cost: Methods requiring large orchestration traces are impractical at industrial rollout cost without interventions such as pipeline parallelism, execution–training decoupling, trajectory compression, or parallel self-search.MarsRL, Agent Lightning, Context-Folding, WideSeek-R1, and MARTI-MARS2 address idle time, asynchronous collection, aggregation length, or rollout-graph width.
  • 5.2 Harness Boundary: The harness, not model parameters alone, is the production deployment unit because it fixes the prompt transformation, tool and spawning interface, input distribution, and output grammar.Fine-tuning through a different harness produces a different policy in the relevant operational sense.
  • 5.3 Credit Fragility: Under uniform shared terminal credit, per-decision signal-to-noise becomes fragile as trace length increases, while academic entries mostly train at T ≲10^2 and Kimi reports T ∼10^3–10^4.Dr. MAS documents instability of naïve GRPO at multi-agent scale, and the section argues benchmarking should use deployment-relevant T.

6 Reward Design for LLM-based MAS

Reward design in LLM-based multi-agent reinforcement learning spans eight composable families, progressing from shared outcomes and per-agent signals to process and system-level orchestration incentives. Orchestration rewards are newly central, but their weighting and scheduling remain largely hand-tuned, with Kimi PARL publicly illustrating transient auxiliary shaping that is annealed toward zero.

  • Reward-family taxonomy: Eight reward families span shared outcomes, per-agent decomposition, process signals, orchestration incentives, and hybrid compositions.The families are not mutually exclusive and are commonly combined through R8 weighting.
  • Reward-family taxonomy: R1–R3 emit terminal rewards, whereas R4–R6 densify feedback throughout traces while introducing attack surfaces such as PRM gaming and judge collusion.The terminal-to-process transition offers stronger gradients but creates additional opportunities for reward hacking.
  • Orchestration rewards: R7 is newly central because it rewards system-level spawn, delegation, parallelism, and aggregation properties without a close single-agent analogue.Its focus is orchestration over multiple agent instances rather than only individual action quality.
  • Orchestration rewards: Kimi PARL combines rperf, rparallel, and rfinish, with auxiliary orchestration-reward hyperparameters annealed to zero so the final policy optimizes the primary task objective.rparallel rewards genuine speedup over a serial baseline, while rfinish discourages pseudo-parallelism by rewarding spawned agents reaching termination.
  • Open problems: Every retained R8 weighting is hand-tuned, with no identified mechanism that automatically balances reward terms using gradient magnitudes or component variance.The paper identifies adaptive MAS-specific composition weighting as a tractable follow-up target.

7 Credit Assignment in LLM-based MAS

LLM-MAS credit assignment extends temporal and agent-level credit with a structural dimension spanning roles, messages, tools, and orchestrator decisions. The surveyed literature organizes signals across eight units, but most methods remain single-level, leaving explicit counterfactual message and orchestrator credit comparatively sparse.

  • Credit-bearing hierarchy: Eight credit-bearing units range from team outcome to token, adding orchestrator and role levels alongside agent, turn, message, and tool signals.The lower four units decompose an agent’s trajectory, while orchestrator credit changes the future agent set itself.
  • Credit-bearing hierarchy: Most RL-oriented entries introduce one novel credit level and inherit standard machinery elsewhere, rather than composing explicit mechanisms across the hierarchy.Examples span agent, role, orchestrator, turn, and message interventions, while published pool entries do not compose explicit mechanisms across all levels.
  • Signal trade-offs: Signal density trades off against outcome attribution: token signals are densest but least team-grounded, whereas team reward is cleanest but emits one value per trace.Role and turn levels occupy an intermediate position; orchestrator and message levels have few decisions per trace but can target structural decisions.
  • Limitations: Counterfactual methods incur substantial computational and structural costs because they estimate returns for alternative traces that were not produced.C3 samples substitute messages, with cost at least linear in message count times sample count and potentially quadratic when substitutions are contextual; SHARP uses Monte Carlo Shapley sampling.
  • Limitations: Stacking credit mechanisms can cause credit collisions when the same pivotal message is rewarded through both message-level and orchestrator-level mechanisms.The survey presents composition as algebraically possible but warns that overlapping gradients may overemphasize some messages.
  • Research coverage: Explicit counterfactual message credit remains especially sparse, while agent- and role-level signals are denser in the surveyed literature.The survey’s takeaway identifies explicit counterfactual message credit and explicit orchestrator credit as sparse columns in the taxonomy.

8 Learning Orchestration: Trajectory →Orchestration Trace

The paper defines LLM-MAS optimization over orchestration traces: temporal interaction graphs that capture branching, delegation, communication, tool use, aggregation, and stopping. It organizes learning around five orchestration sub-decisions, while identifying explicit stopping-policy training and counterfactual spawn evaluation as open problems.

  • Trace formalization: An orchestration trace G = (V, E, ℓ) is a temporal interaction graph whose events, dependencies, and labels represent coordinated multi-agent execution.Unlike a linearly ordered single-agent trajectory, traces support parallel branching, aggregation joins, and orchestrator-to-sub-agent delegation.
  • Orchestration decisions: Five sub-decisions define learnable orchestration: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop.The taxonomy treats the joint orchestrator, sub-agents, and aggregation policy as optimizing a composite reward over G.
  • Spawning: Spawn learning lacks explicit counterfactual estimators because unrealized no-op branches are unobserved, leaving off-policy evaluation as a central gap.Kimi PARL uses parallelism and finish reward shaping, while AgentSpawn and HALO rely on complexity estimation or search heuristics.
  • Stopping: No retained method explicitly trains stopping; existing systems terminate through external completion signals or fixed step-count caps.The proposed objective is expected marginal gain from another orchestration step minus its cost, implemented through a cost-adjusted return.
  • Training and engineering: Training regimes trade cost against expressivity: freezing sub-agents is cheap, joint training is unstable without normalization, and role-specific critics or replay are engineering-heavy.The section also reports serial-collapse and one-dominant-agent-collapse failure modes, while academic dispatchers remain below dynamically growing industrial swarms.

9 Benchmarks and Evaluation

Current LLM-MAS benchmarks mostly measure task success rather than coordination, making compute-driven gains difficult to distinguish from coordination improvements. The section recommends MAS-native, trace-level evaluation spanning four dimensions, stratified by trace length and topology, tested adversarially, and released openly for reproducibility.

  • Benchmark landscape: Most LLM-MAS benchmarks report task success only, while a MAS-native benchmark must report at least three of four dimensions: accuracy, parallelism efficiency, collaboration quality, and protocol overhead.No open benchmark in the pool covers all four dimensions jointly; Kimi Swarm Bench is closed and unauditable, while MultiAgentBench covers only a subset at small T.
  • Benchmark landscape: E1-only evaluation confounds compute with coordination because spawning more sub-agents in parallel can improve task success without demonstrating better credit assignment.Without parallelism-efficiency metrics, these mechanisms are indistinguishable on task success alone.
  • Benchmark design recommendations: Trace-length stratification should report performance across T ∈{10^1, 10^2, 10^3} bands, exposing whether methods degrade as credit diffuses over longer orchestrations.The recommendation distinguishes methods that excel at T = 50 but degrade at T = 500 from methods that scale gracefully.
  • Benchmark design recommendations: Benchmarks should instrument identical tasks across centralized, debate, swarm, and hierarchical topologies, so orchestration effects can be separated from credit-assignment effects.The goal is to expose how much reported gain comes from a method’s preferred topology rather than to identify one universally best topology.
  • Benchmark design recommendations: Each task should include controlled indirect prompt injection, inter-agent message pollution, and shared-memory poisoning, making robustness margins first-class evaluation metrics.These perturbations correspond to attack vectors AV2, AV3, and AV4.
  • Benchmark design recommendations: Open instances, public leaderboards, per-band scores, and replayable orchestration traces would improve cross-method comparability without requiring rollout reruns.Typed event-graph traces allow later work to recompute reward, credit, parallelism, and safety metrics over the same rollout.

10 Safety and Adversarial Robustness in LLM-MAS

LLM-MAS safety remains underdeveloped because multi-agent systems add propagation-specific attack surfaces beyond single-agent safety. Defenses are mostly ad hoc, while adversarial attacker–defender RL and MAS-specific benchmarking are only beginning to emerge.

  • Threat model: Attacks arise from external user input, untrusted tool output, and adversarial agents embedded within the team.Classical jailbreaks and indirect prompt injections coexist with the MAS-specific possibility of a malicious or compromised sub-agent influencing natural-language information flow.
  • Attack surfaces: Published attacks primarily target message and tool levels, whereas orchestrator and role attacks are barely studied and sandbox/system attacks are not publicly documented.Table 15 organizes attack vectors by the same credit- or signal-bearing levels used for RL credit assignment, linking security coverage to structural credit levels.
  • Defenses: Inference-time defenses are mostly ad hoc, including sanitization, allowlists, sandboxing, verification, heterogeneous verifiers, human checkpoints, and scoped credentials.MAGIC is the clearest retained training-time example, using co-evolving attacker and defender agents to manufacture adversarial shielded-RL scenarios.
  • Evaluation: The benchmark landscape is sparse: TAMAS is the most MAS-specific benchmark, while AgentDojo, InjecAgent, WASP, and two ACL 2025 studies cover adjacent or emerging multi-agent settings.None jointly reports safety with collaboration quality (E3) and parallelism efficiency (E2), preventing comprehensive reward–credit–safety comparisons.
  • Threat model: LLM-MAS adds three structural threats beyond single-agent safety: inter-agent message pollution, shared-memory contagion, and orchestrator misdirection.Every spawned sub-agent inherits tool access, messages can carry injections, shared-memory writes can be poisoned, and inter-agent flows grow super-linearly with trace size.

11 Open Problems

The survey identifies fifteen open problems across algorithmic, reward, systems, safety, and evaluation axes. It highlights credit diffusion, unrealized-branch counterfactuals, and MAS-native benchmarks as most load-bearing, while train–inference mismatch, variable-shape replay, and mid-trace steerability are especially deployment-relevant and under-published.

  • Reward problems: P6–P8 show that tool-spam, verifier–policy collusion, and process–outcome reward imbalance lack general reward-design principles.Existing mitigations condition tool rewards on downstream outcomes, use heterogeneous verifier families, or assign role-specific PRMs, but each remains partial or brittle.
  • Systems problems: P9–P11 identify rollout cost, variable-shape replay, and straggler bias as systems bottlenecks without complete solutions.Multi-agent rollouts are 10–100× more expensive than single-agent rollouts, while graph-native replay and asynchronous-rollout bias correction remain absent from the retained pool.
  • Safety problems: P12–P14 show that inter-agent prompt injection, mid-trace intervention, and sandbox isolation become harder under propagation and dynamic spawning.The survey reports attacks and benchmarks for injection and steerability, but no retained paper formalizes steerability as an RL objective, and dynamic-spawn systems do not publicly explain per-agent isolation.
  • Evaluation problems: P15 identifies the lack of an open MAS-native benchmark jointly covering E1–E4 at ≳10^3-step traces as the most tractable infrastructure gap.Without such a benchmark, credit-assignment methods cannot be fairly compared and scaling claims cannot be cross-validated at industrial trace lengths.
  • Takeaway: The survey ranks P1, P4, and P15 as most load-bearing, while P5, P10, and P13 are most deployment-relevant and under-published.The takeaway connects progress on the three load-bearing problems to progress on many others.

12 Limitations

The survey is a curated taxonomy and position paper rather than an exhaustive systematic review. Its limitations concern corpus coverage, single-author tagging, industrial-source interpretability, formal claims, and the rapidly changing literature cutoff.

  • Corpus coverage: The 84-entry corpus is curated for relevance, not a complete bibliography, and its auditable screening log does not provide PRISMA-level reproducibility.The retained pool covers reward design, credit assignment, orchestration learning, systems constraints, benchmarks, and safety in LLM-MAS RL.
  • Annotation: Taxonomy tags were assigned by a single author without blinded multi-annotator review or inter-annotator agreement, so borderline labels are judgements.Borderline cases include surveys, industrial systems with undisclosed training details, and self-evolution frameworks without conventional RL objectives.
  • Industrial evidence: Industrial sources describe deployment shape, scale, harness boundaries, and workflow pressure, but only Kimi K2.5 explicitly discloses RL training of the orchestrator.The survey does not infer undisclosed multi-agent RL objectives from product behavior.
  • Formal claims: The dynamic-Dec-POMDP and orchestration-trace definitions organize vocabulary, while credit-diffusion and non-identifiability claims are conceptual rather than theorems.A full theory would require assumptions about noise, baselines, graph dynamics, off-policy branch coverage, and value-function approximation over variable-shape traces.
  • Moving literature: The literature cutoff is May 4, 2026, so new reports, submissions, benchmarks, and safety evaluations may change coverage and sparsity conclusions.The message-credit, orchestrator-credit, adaptive-deliberation, and trace-level safety cells are especially subject to change.

13 Reproducibility and Artifact Statement

The paper releases an auditable artifact containing corpus metadata, reproducible statistics, a machine-readable orchestration-trace schema, and structural validation. These materials make taxonomy support, sparsity counts, and schema compliance mechanically inspectable, but do not make the review exhaustive or replace multi-annotator review.

  • Artifact components: 84 retained entries and 32 screened-but-excluded decisions are documented in CSV files with controlled metadata and public identifiers or URLs.The retained-entry CSV contains 18 controlled fields.
  • Artifact components: Repository scripts regenerate retained and excluded counts, controlled-field histograms, and cross-tabs reported in Tables 1, 2, and 22.The statistics are scripted for reproducibility from the artifact repository.
  • Artifact components: A machine-readable JSON Schema and minimal valid example define typed orchestration traces, while a dependency-free checker validates fields, event types, edge references, duplicate identifiers, and non-negative costs.The checker operates on a trace JSON file.
  • Scope and inspectability: The artifact makes taxonomy support, sparsity counts, and benchmark-log schema compliance mechanically inspectable, but it is neither an exhaustive literature review nor a substitute for multi-annotator systematic review.Readers can check which entries support a taxonomy cell and regenerate the reported counts.

14 Conclusion

The survey argues that reinforcement learning for LLM-based multi-agent systems should be analyzed through orchestration traces, supported by taxonomies of rewards, credit-bearing units, and orchestration sub-decisions. It highlights an asymmetric industrial–academic evidence gap and proposes unified credit assignment and coordination-focused benchmarks as next steps.

  • Core thesis: Orchestration traces provide the survey’s central abstraction for taxonomy and auditability, replacing an exclusive focus on per-agent trajectories.The abstraction uses an event graph drawn from a dynamic-Dec-POMDP extension and motivates observations about credit diffusion and orchestration non-identifiability.
  • Taxonomies: Eight reward families and eight credit- or signal-bearing units operationalize the survey’s framework for analyzing multi-agent reinforcement learning.Orchestration reward R7 most directly targets spawn, delegation, and aggregation decisions, while the hierarchy runs from team/orchestrator through role, agent, turn, message, tool, and token.
  • Industrial–academic bridge: Academic evaluations at T ≲102 cannot be assumed to transfer to the Kimi-reported deployment envelope at T ∼103–104.The survey attributes this scale gap to rollout-cost scaling, a training-frozen harness interface, and per-decision signal decay; open literature typically uses fixed or moderate-size teams rather than hundreds of sub-agents.
  • Future directions: The survey prioritizes unified credit assignment over orchestration graphs and MAS-native benchmarks that measure coordination rather than success alone.It identifies explicit counterfactual message credit and explicit orchestrator credit as tractable sparse targets, while (E1) alone weakly distinguishes compute gains from coordination gains.
  • Closing perspective: The orchestration trace, credit- and signal-bearing-unit hierarchy, and five-way orchestration sub-decision remain abstractions that can still be selected before industrial conventions calcify.The survey presents these abstractions as a productive starting point for the field.

A Entry Cards: Core RL Methods and Anchors for LLM-MAS … A. Focal LLM-MAS training, benchmark, and adjacent framework entries (40 entries)

The appendix organizes core LLM-MAS RL methods and industrial anchors by credit-bearing unit, spanning orchestrator, role, agent, turn, message, and framework levels. The paper pool table then situates these methods within an 84-entry taxonomy of rewards, credit granularity, orchestration, and scenario tags.

  • A Entry Cards: Core RL Methods and Anchors for LLM-MAS: Thirteen method and industrial-anchor cards uniformly report claims, reward shapes, credit mechanisms, orchestration forms, empirical results, and limitations.The cards are ordered from team/orchestrator to message/tool, mirroring the credit hierarchy.
  • A.1 Orchestrator-level credit: Puppeteer learns a central orchestrator that selects the next sub-agent, using a team outcome and centralized critic while freezing sub-agent policies.Kimi PARL reports orchestration at up to 100 sub-agents and 1,500 coordinated steps, while K2.6 extends the public envelope to 300 sub-agents and 4,000 steps without an independent RL-training claim.
  • A.2 Role-level credit: +14.14% over a single-agent baseline is reported by MALT, which trains generator, verifier, and refiner roles with role-specific process rewards and PRMs.M-GRPO separately estimates planner and sub-agent advantages, while MATPO shares LLM weights across planner and worker roles with role- and tool-conditioned rewards.
  • A.3 Agent-level credit: Agent-level methods range from shared team-reward updates in MAGRPO and MAPoRL to per-agent normalization in Dr. MAS and Shapley marginal credit in SHARP.These methods respectively formalize cooperative MARL, train collaboration behavior after co-training, address cross-agent variance, and combine global, marginal, and tool-process rewards.
  • A.4 Turn-level credit: Turn-level approaches assign stage- or branch-specific advantages, with MarsRL using concurrent pipeline stages and Context-Folding rewarding fold/unfold actions against shared and branch baselines.Context-Folding uses a branch reward approximately equal to the main reward ±0.2 scope adjustment per folded sub-trajectory.
  • A.5 Message-level credit: C3 is the only retained pool entry with counterfactual message-level credit, estimating each utterance’s trace-return change under message replacement or removal.Pivotal messages receive proportionally larger credit under its contextual counterfactual intervention.
  • A.6 Framework-level (cross-cutting): Agent Lightning provides a harness-based framework that decouples execution from training and supports application-defined per-agent and per-tool-call credit primitives.Its framework-level reward dispatch is hybrid, but the specific decomposition depends on the application.
  • B Paper Pool Summary Table: The 84-entry paper pool is organized into evidence buckets, with machine-readable taxonomy fields for RL relevance, reward family, finest credit level, orchestration, scenario, and core status.The focal entries include cooperative MARL, orchestration, role-specific, process, debate, verification, and hybrid methods across centralized, hierarchical, and other settings.

B. Related surveys used for gap analysis (5 entries) … C.8 Artifact update protocol

The paper situates its taxonomy against related surveys, industrial systems, classical and single-agent RL foundations, benchmarks, safety work, and critic/tool-use evaluations. Its artifact provides an auditable, quasi-systematic corpus protocol, machine-readable trace schema, scripted statistics, and controlled update procedure.

  • C. Industrial systems (cases): The industrial cases provide deployment-scale evidence from Agent Swarm, OpenAI Codex, Claude Code, and 16 parallel Claudes building a C compiler.These cases motivate the orchestration framing but are not algorithmic contributions in the paper pool.
  • B–H Related evidence: The evidence pool spans five related surveys, six industrial cases, ten classical MARL works, eight benchmarks, five single-agent foundations, six safety studies, and four critic/tool-use evaluations.The 84 entries are organized into eight manuscript evidence buckets, including 40 focal entries and the listed supporting categories.
  • C Artifact, Search Protocol, and Trace Schema: The protocol is quasi-systematic rather than PRISMA-level reproducible, because raw database exports, deduplication logs, and multi-annotator agreement are unavailable.The search cutoff is May 4, 2026, covering arXiv, ACL Anthology, OpenReview, Semantic Scholar links, official project pages, technical reports, and product documentation.
  • C.2 Tag schema: Each retained record uses 18 controlled fields, while eight entries are marked verified=partial and Kimi K2.6 is verified=yes but rl=partial.Partial entries support adjacent taxonomy cells rather than serving as sole support for central claims; Kimi K2.6 supports deployment-envelope claims.
  • C.5 Claim-to-artifact ledger: The artifact ledger bounds claims to retained-pool fields, scripted counts, or explicitly bounded industrial evidence rather than estimating field-wide prevalence.It separately distinguishes credit granularity from explicit counterfactual credit and documents the sparsest credit cells.
  • C.6 Scripted meta-analysis: The repository statistics script regenerates retained/excluded counts, controlled-field histograms, and reward–credit, orchestration–credit, and category–verification cross-tabs.The headline counts and cross-tabs are generated from the CSV artifact rather than entered manually.
  • C.7 Machine-readable trace schema: The minimal JSON trace schema requires trace id, task id, events, edges, rewards, and costs, with typed orchestration, communication, tool, return, aggregation, human, and safety events.Edges encode temporal, causal, spawn, message, tool-dependency, return, aggregation, and safety-flow relations; content references permit redaction while preserving replayable topology and metadata.
  • C.8 Artifact update protocol: Future updates append candidates to the exclusion log, retain them only when they change a controlled evidence cell, and rerun summary statistics when promoted tags alter counts.The protocol also updates Figure 3, Table 1, Table 2, and the appendix summary table when required.
Loading 2605.02801v1…