Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces

Thesis

• Orchestration traces reframe LLM-MAS RL as temporal interaction graphs governing spawning, delegation, communication, tool use, aggregation, and stopping.The framework complements per-agent trajectories and organizes reward design, credit assignment, and orchestration-process learning.Source: Thesis · p. 5

1.1 Why now: recent developments

• Public industrial evidence reports Kimi Agent Swarm scaling to 100 sub-agents and 1,500 coordinated steps, then 300 sub-agents and 4,000 steps.These are deployment envelopes rather than independently reproduced training traces; Codex and Claude Code provide deployment-shape evidence without disclosing whether coordination is an RL target.Source: §1.1 · p. 5
• The paper targets a gap between surveys of multi-agent architectures, single-agent agentic RL, and LLM agents by triangulating multi-agent RL, post-training, and LLM-agent collaboration.The recent literature includes team RFT, hierarchical GRPO, message-level counterfactual credit, Shapley-based credit, communication learning, and related collaboration methods.Source: §1.1 · p. 6

1.2 Scope and positioning

• The scope is post-training: given an architecture, the paper asks which rewards drive teams, how credit is assigned, and how orchestration itself is learned.It is a taxonomy rather than an exhaustive systems catalogue, benchmark leaderboard, or new benchmark release.Source: §1.2 · pp. 6–7

1.3 Corpus construction and evidence levels

• The curated corpus contains 84 retained entries: 51 focal LLM-MAS entries and 33 supporting references, with 116 audited records including the exclusion log.The pool is constructed through survey seeding, database and project-page searches, citation expansion, and explicit inclusion rules.Source: §1.2 · p. 7, §1.3 · p. 7

1.4 Contributions

• The paper contributes three taxonomies covering eight reward families, eight credit-or-signal units, and five orchestration sub-decisions, plus a dynamic-Dec-POMDP organizing abstraction.It also releases tagged corpus data, scripted statistics, and a machine-readable orchestration-trace schema with validation artifacts.Source: §1.4 · pp. 8–9

1.5 Roadmap

• The roadmap connects industrial and academic system forms to engineering constraints, reward design, credit assignment, orchestration learning, benchmark gaps, and open problems.Sections 6–8 develop the thesis pillars, while later sections address evaluation failures, fifteen open problems, and the concluding synthesis.Source: §1.5 · p. 10

2.1 Classical MARL in one page

• Classical MARL models agents in Markov games or Dec-POMDPs and organizes learning around CTDE and value or counterfactual credit decomposition.CTDE uses a centralized critic during training but decentralized observations at deployment; VDN, QMIX, MADDPG, MAPPO, COMA, Shapley-value credit, and difference rewards instantiate these choices.Source: §2.1 · pp. 10–11

2.2 Single-agent LLM RL in one page

• Single-agent LLM RL progresses from preference-based RLHF and RLAIF to verifiable, reasoning, and agentic RL, with PPO and GRPO as dominant policy-optimization choices.Reward signals become denser and credit moves from trajectories to tokens, steps, turns, actions, and tool calls; GRPO removes the value network through within-group advantage normalization, which requires adjustment for multi-agent settings [15].Source: §2.2 · p. 11

2.3 Why LLM-MAS is not classical MARL

• LLM-MAS differs from classical MARL through language actions, long summarized observations, dynamic agent counts, free-form communication, asynchronous episodes, role heterogeneity, and new credit-bearing units.These units include messages, tool calls, roles, and orchestrator decisions; orchestration traces are therefore graph-structured, and classical machinery does not handle them natively.Source: §2.3 · pp. 11–12

3 A Working Abstraction for the Orchestration Trace

• Orchestration traces are policy-generated temporal interaction graphs whose event vertices include decisions, spawns, messages, tools, returns, and aggregations.The abstraction supplies vocabulary for the survey rather than a fully axiomatized MARL framework, and leaves off-policy evaluation and variable-shape value-function forms open.Source: §3 · p. 12

3.1 Relation to existing formalisms

• Unlike Dec-POMDPs, hierarchical RL, dynamic-population systems, or graph-conditioned MARL alone, the abstraction tracks credit- and signal-bearing events whose graph structure is produced by orchestration.Table 4 positions the framework as an auditable event vocabulary, not a new solution concept.Source: §3.1 · pp. 12–13, Table 4 · p. 13

3.2 Orchestration trace as a Dec-POMDP extension

• The dynamic-Dec-POMDP extension gives the orchestrator spawn and despawn actions, lets sub-agents act under role- and history-dependent spaces, and represents rollouts as labelled causal event graphs.The graph contains orchestrator decisions, spawns, messages, tool calls, returns, and aggregations; classical fixed-agent trajectories are a special case.Source: §3.2 · pp. 13–14

3.3 Value function under variable-shape traces

• With active-agent count and joint-action shape varying, value functions are conditioned on the trace prefix and must account for transitions that grow or shrink the event graph.The graph-conditioned identity motivates trace- or graph-conditioned critics, while sufficient features and amortization across variable shapes remain open.Source: §3.3 · p. 14

3.4 Two organizing observations

• Uniform shared-reward credit becomes less distinguishable for individual units on longer traces, while same-prefix spawn effects are not identifiable from realized on-policy traces without branch coverage or structural assumptions.These are qualitative organizing observations rather than theorem statements; the latter motivates off-policy counterfactual value estimation.Source: §3.4 · pp. 14–15

3.5 The reward–credit dual

• Dense rewards reduce credit diffusion by targeting lower-level orchestration events, whereas sparse outcome rewards shift the burden to credit decomposition, making reward density and credit machinery dual design choices.The takeaway is that the framework organizes later taxonomy sections while remaining an abstraction rather than a new MARL theory.Source: §3.5 · pp. 15–16

4 System Forms: How LLM Agent Teams Are Organized

• System form is foundational because it determines what later RL methods can reward and where credit can flow, while deployment pressures constrain practical training.Relevant pressures include rollout cost, asynchrony, and harness design.Source: §4 · p. 16

4.1 A typology of agent-team topologies

• Six recurring, non-exclusive topologies organize the paper pool, and each constrains which reward families and credit-bearing units are definable.Production systems typically combine two or three patterns; centralized orchestrators favor orchestrator-level credit, while debate favors message-level credit.Source: §4.1 · p. 16, Table 5 · p. 16, Figure 5 · p. 17

4.2 Public industrial evidence and selection rule

• The industrial evidence selection conservatively retains Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code as distinct anchors rather than a census of commercial products.Product documentation supports deployment shape, scale, interfaces, and workflows, but does not by itself establish a reproducible algorithmic method.Source: §4.2 · pp. 17–18, Table 6 · p. 18

4.2.1 Kimi Agent Swarm (K2.5 / K2.6)

• Kimi’s public reports describe learned orchestration with PARL [28], scaling from 100 sub-agents and 1,500 coordinated steps to 300 sub-agents and 4,000 steps.Its reward combines performance, parallelism, and finishing terms, while Critical-Steps supplies orchestrator-level credit; the full training recipe remains unreproducible.Source: §4.2.1 · p. 18

4.2.2 OpenAI Codex (app + harness)

• Codex makes the harness—not the model alone—the deployment unit, and its long-running parallel workflows make rollout cost and scheduling central RL constraints.Its RL recipe is undisclosed; Agent Lightning [40] articulates the fixed-harness training constraint, while MarsRL [36] targets pipeline-parallel scheduling.Source: §4.2.2 · p. 19

4.2.3 Anthropic Claude Code (sub-agents + agent teams)

• Claude Code exposes sub-agents as first-class spawn, delegation, and aggregation objects, while its sixteen-agent compiler case study demonstrates large parallel workflows.Its steerability concern frames mid-execution human intervention and downstream consequences as a credit-assignment-shaped open problem.Source: §4.2.3 · pp. 19–20

4.3 What systems reveal that papers do not

• The systems reveal three publication gaps: rollout-cost realism, explicit harness boundaries, and steerability as an RL target.Academic methods mostly use 10–100-step traces, no pooled method reports Kimi-length training, and no pooled paper formulates steerability as an RL objective.Source: §4.3 · pp. 20–21
• The resulting scale gap concerns rollout and harness regimes rather than independent verification of industrial training traces: topology defines learnable rewards and credit, while Kimi most clearly exceeds open academic envelopes.Codex and Claude Code support harness and workflow comparisons, not claims of comparable disclosed RL trace scale.Source: §4.2.1 · p. 18, §4.3 · p. 21, Table 8 · p. 21

5.1 Rollout cost dominates wall-clock training time

• Industrial multi-agent rollouts can be orders of magnitude larger than single-agent rollouts, while GRPO-style training with G=8 adds an 8× collection multiplier.The cost depends on sub-agent token use, tool calls, orchestration events, and orchestrator inference; pipeline parallelism, execution–training decoupling, and context folding mitigate it.Source: §5.1 · pp. 22–23, Figure 7 · p. 23

5.2 The harness boundary as a training-frozen interface

• The deployment harness fixes the policy’s observation interface and action grammar, so training through a different harness changes the policy operationally; only Agent Lightning [40] states this contract explicitly.The harness includes the prompt template, tool registry, and execution runtime, while only model parameters receive RL gradients.Source: §5.2 · p. 23, Figure 8 · p. 24

5.3 Trace length and credit fragility

• Uniform terminal credit becomes increasingly fragile as trace length grows, destabilizing naïve GRPO at multi-agent scale; role-, message-, and orchestrator-level credit reduce the effective decision horizon [15] [44] [7] [10].Academic entries mostly train at T ≲ 10^2, whereas Kimi’s reported deployment envelope reaches T ∼ 10^3–10^4, making deployment-scale benchmarking essential.Source: §5.3 · p. 24, Figure 9 · p. 25

5 Systems Engineering: Rollout Cost and Harness Boundary

• Three constraints govern industrial-scale LLM-MAS RL: rollout cost makes large-G algorithms infeasible, most academic methods ignore the harness boundary, and uniform-credit signal quality declines with trace length.These constraints correspond to specific open problems and are especially consequential for long-horizon deployments reported by Kimi.Source: §5.3 · p. 24

6.1 Eight families of rewards

• The reward design space comprises eight families organized by signal, granularity, source, hacking risk, and representative methods, progressing from terminal outcomes to process, system, and hybrid rewards.R1–R3 provide terminal signals, R4–R6 densify process feedback, R7 introduces orchestration incentives, and R8 combines families; larger practical methods commonly use R8 compositions.Source: §6.1 · p. 25, Table 10 · p. 26

6.2 The Kimi PARL reward decomposition (worked example)

• Kimi PARL [28] combines task performance with parallelism and completion rewards, then anneals both auxiliary orchestration weights to zero so they scaffold exploration without defining convergence.The public description reports rperf, rparallel, and rfinish; the exact optimizer, data, ablations, and schedules remain undisclosed, limiting independent reproducibility.Source: §6.2 · p. 27, Figure 11 · p. 28

6.3 Reward-hacking failure modes specific to MAS

• MAS reward hacking includes pseudo-parallelism, free-riding, communication padding, tool-spam, and verifier collusion, with mitigations ranging from finish checks and marginal credit to outcome-conditioned tool rewards.Kimi PARL uses rfinish and Critical-Steps against padded traces; SHARP [31], Dr. MAS [15], MATPO [43], Agent Lightning [40], Debate-as-Reward [51], and CriticLean [48] address related failure modes.Source: §6.3 · pp. 27–28

6.4 Discussion: the unanswered weighting question

• Interaction-derived rewards increasingly use discussion dynamics, successful traces, majority support, or meta-level feedback to manufacture training signals without forming a new primitive family.The survey also hypothesizes, but does not establish, that richer shaping may invite more gaming and produce smaller gains than sparse shaping; direct comparison is unavailable.Source: §6.4 · pp. 28–29
• R8 weights remain hand-tuned, with no retained method automatically balancing components or providing a principled schedule for transient R7 incentives.The survey identifies measurable gradient or variance-based balancing and convergence-indicator schedules as open targets, while Kimi’s annealing description remains unspecified.Source: §6.4 · p. 29

7.1 The credit- and signal-bearing unit hierarchy

• LLM-MAS organizes credit across eight units from team outcome to token, adding structurally distinctive orchestrator, role, and message levels to temporal and agent-level credit.The orchestrator level can change the future agent set, while messages and tools mediate inter-agent information flow; Figure 12 identifies orchestrator and message levels as especially sparse.Source: §7.1 · p. 30, Figure 12 · p. 30

7.2 A two-dimensional taxonomy of entries

• The taxonomy shows most methods target one novel level, while orchestrator and message credit remain sparse; C3 [7] is the only retained method estimating counterfactual message-level credit.Puppeteer, Kimi PARL, and WideSeek-R1 explicitly signal orchestrator decisions through different mechanisms, whereas only two entries are tagged at message level.Source: Table 11 · p. 31, §7.2 · p. 32

7.3 Why naive single-agent credit assignment fails

• Naive team reward plus per-agent GAE fails under reward diffusion, asymmetric contribution, and spawning counterfactuals that distinguish sub-agent value from the value of spawning itself.Public Kimi anchors reach 1,500 coordinated steps/tool calls [28] and 4,000 coordinated steps, while Dr. MAS [15] reports instability not cured by hyperparameter tuning alone.Source: §7.3 · p. 32

7.4 Open algorithmic questions

• Open questions include composing credit mechanisms, balancing dense process signals against sparse outcomes, extending Shapley credit to dynamic agent sets, and crediting the decision not to spawn.The surveyed pool contains no entry addressing unrealized no-spawn branches as of May 4, 2026.Source: §7.4 · p. 33

7.5 Discussion: composing the sparse and dense levels

• Credit density trades off against outcome attribution: token signals are dense but locally detached, team rewards are clean but sparse, and orchestrator or message signals are sparse but consequential.DEPART and MARSHAL [75] mix sparse task rewards with denser role or turn signals, but exact full-hierarchy partitioning remains absent.Source: §7.5 · p. 34
• Counterfactual credit is costly and compositional stacking risks double-counting, yet no surveyed method provides an exact reward partition across credit channels.C3 [7] substitutes messages, SHARP [31] uses Monte Carlo Shapley sampling, and neither learned counterfactual estimation nor importance sampling has a published instance in the pool.Source: §7.5 · p. 34

8.1 The orchestration trace as a first-class object

• An orchestration trace G = (V, E, ℓ) is a temporal interaction graph whose events, causal dependencies, and labels represent orchestration decisions, agent actions, tools, messages, returns, and aggregation.A single-agent trajectory is a linearly ordered special case, whereas multi-agent traces support branching, joining, and delegation; optimization uses composite reward R and joint policy πθ.Source: §8.1 · pp. 34–35

8.2 Methods by orchestration sub-decision

• The five learnable decisions are when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop, but no retained method explicitly trains the stopping decision.The taxonomy treats each decision as a policy target with its own training signal and open problems; Figure 15 identifies coverage for O1–O4 but not O5.Source: §8.2 · p. 35, Figure 15 · p. 36, §8.5 · p. 39

8.2.1 O1: When to spawn

• Spawn learning remains dependent on shaping or search proxies: Kimi PARL [28] rewards parallelism and finish behavior, while AgentSpawn [9] and HALO [20] use complexity estimation or MCTS.None uses an explicit counterfactual estimator for the unrealized no-op branch, leaving principled off-policy evaluation of spawn alternatives open.Source: §8.2.1 · p. 36

8.2.2 O2: Whom to delegate to

• Delegation methods include Puppeteer’s [10] centralized critic, ParaManager’s [76] unified agent/tool action space, and WideSeek-R1’s [68] learned width scaling, but dynamic dispatch to hundreds of agents remains unaddressed.The target signal is a per-delegation return differential identifying which agent would perform best in context.Source: §8.2.2 · p. 36

8.2.3 O3: How to communicate

• Communication learning spans resolution-based rewards [51], contextual message interventions [7], QMIX-style topology optimization [23], and LatentMAS’s [84] continuous channel, while explicit message-level credit remains the sparsest signal.These point solutions lack a unified information-theoretic account of which information orchestrators should exchange.Source: §8.2.3 · p. 37

8.2.4 O4: How to aggregate

• Aggregation is modeled as a policy output in M-GRPO [19] and Context-Folding [55], but both rely on lossy, uncalibrated LLM summarization rather than explicit uncertainty modeling.Training can use team reward or proxies such as whether the aggregate contains the fact needed for the next decision; Context-Folding uses rbranch ≈rmain ± 0.2 scope adjustment.Source: §8.2.4 · p. 37

8.2.5 O5: When to stop

• Explicit stopping-policy RL is absent: surveyed systems stop through external completion signals or fixed step caps, despite the natural objective of trading marginal accuracy gain against orchestration cost.Adding a stop action trained against cost-adjusted return is identified as a clear open direction.Source: §8.2.5 · p. 37

8.3 Orchestrator training regimes

• Training regimes trade cost against expressivity: frozen-sub-agent orchestration is cheapest, joint training updates all policies but needs normalization, and decoupled central-critic training is most expressive and engineering-heavy.Regime A cannot improve sub-agent skill; regime C requires separate replay buffers per role.Source: §8.3 · p. 38, Figure 16 · p. 38

9 Benchmarks and Evaluation

• Most benchmarks report task success alone, so they cannot distinguish genuine coordination from gains caused by additional compute.A benchmark is MAS-native only when it reports at least three of four evaluation dimensions; most benchmarks in the pool report E1 only.Source: §9 · p. 39, §9.1 · p. 40

9.1 Four dimensions for auditing multi-agent evaluation

• MAS-native auditing requires task accuracy, parallelism efficiency, collaboration quality, and protocol overhead.These dimensions include speedup and utilization, message redundancy and consensus, token costs, error amplification, and prompt-injection flow; LatentMAS [84] shows collaboration can occur without natural-language messaging.Source: §9.1 · p. 40

9.2 Benchmark landscape by domain

• No open benchmark covers all four dimensions jointly; Kimi Swarm Bench [28] is closed, TAMAS [25] focuses on safety, and MultiAgentBench [83] covers only subsets.Credit-assignment methods also use different benchmarks, preventing direct comparison of their mechanisms.Source: Table 12 · p. 40, §9.2 · pp. 40–41

9.3 The benchmark gap

• E1-only evaluation rewards compute rather than coordination, underrepresents safety, and omits the 4,000-step traces reported for Kimi K2.6.These omissions hide credit-diffusion failures and deployed-system risks such as prompt injection, shared-memory poisoning, and tool-parameter escalation.Source: §9.3 · p. 41

9.4 What a good MAS-native benchmark would look like

• A good MAS-native benchmark should report E1–E4, stratify trace lengths, vary centralized, debate, swarm, and hierarchical topologies, and include controlled adversarial conditions.These design choices expose compute-versus-coordination confounds, scaling behavior, topology effects, and robustness margins.Source: §9.4 · p. 41
• Open instances, public leaderboards, per-band scores, and replayable orchestration traces are needed for reproducibility and fair cross-method comparison.The proposed typed event-graph schema enables later recomputation of reward, credit, parallelism, and safety metrics from the same rollout; no open benchmark jointly reaches E1–E4 at the Kimi-reported long-trace envelope.Source: §9.4 · pp. 41–43

10 Safety and Adversarial Robustness in LLM-MAS

• LLM-MAS adds multi-agent-specific threats through inter-agent message pollution, shared-memory contagion, and orchestrator misdirection beyond inherited single-agent safety risks [17] [12].Attack surfaces grow with team size, natural-language outputs can propagate compromised instructions, and runtime spawning complicates advance isolation audits.Source: §10.1 · p. 43, §10 · p. 43

10.1 Threat model for LLM-MAS

• Threats enter through external user input, untrusted tool outputs, or compromised agents and messages, while classical MARL safety methods do not address natural-language information flow [17] [12] [14].Adversarial agents can be compromised at spawn time or poisoned during execution, with contagion propagating through shared memory [41] [25].Source: §10.1 · pp. 43–44

10.2 Attack taxonomy

• Table 15 maps attacks to credit- and signal-bearing levels: message and tool attacks dominate published work, whereas orchestrator, role, sandbox, and system attacks remain barely studied or undocumented.The taxonomy treats credit-assignment levels as corresponding attack pivots, linking security analysis to the orchestration hierarchy.Source: Table 15 · p. 44, §10.2 · p. 44

10.3 Defense landscape and benchmarks

• Inference-time defenses are mostly ad hoc, while MAGIC [65] uses co-evolving attacker and defender agents for adversarial safety training; no comparable trace-level constrained-optimization formulation was found.Benchmarks remain sparse: TAMAS [25] is the most MAS-specific, alongside adjacent injection benchmarks and two ACL 2025 studies, none jointly evaluating safety, collaboration quality, and parallelism efficiency.Source: §10.3 · pp. 44–45

10.4 The under-addressed problem: steerability

• Steerability—human inspection and intervention during a partially completed orchestration trace—has not been formalized as an RL objective, despite its alignment with hierarchical credit assignment.Effective intervention requires identifying responsible earlier decisions and downstream decisions invalidated by edits; C3’s [7] message-level counterfactual machinery could potentially support this attribution.Source: §10.4 · p. 45

11.1 Algorithmic

• Algorithmic problems span fragile credit over 10^3–10^4-step traces, shared-reward free-riding, diversity collapse, unrealized-branch counterfactuals, and generalization from k to k′ agents.Dr. MAS [15], LangMARL [71], CoLLM-MAAC [38], MARSHAL [75], SHARP [31], C3 [7], and MARTI-MARS2 [61] address partial symptoms, but major general principles remain missing.Source: §11.1 · p. 46

11.2 Reward

• Reward problems include tool-spam and fabricated success, verifier–policy collusion, and dense process rewards overwhelming sparse team outcomes.Agent Lightning [40] and MATPO [43] condition tool rewards on downstream outcomes; heterogeneous verifier families and role-specific PRMs in MALT [44] are partial mitigations.Source: §11.2 · pp. 46–47

11.3 Systems and engineering

• Systems gaps are dominated by multi-agent rollout costs of 10–100× single-agent training, variable-shape traces, and straggler-induced asynchronous bias.MarsRL [36] and Agent Lightning [40] provide partial cost solutions, but graph-native replay, matching advantage normalization, and straggler-robust bias correction remain missing.Source: §11.3 · p. 47

11.4 Safety

• Safety problems concern prompt-injection propagation, absent mid-trace steerability objectives, and unverified sandbox isolation for dynamically spawned sub-agents.TAMAS [25], AgentDojo [12], WASP [14], and MAGIC [65] establish benchmarks or countermeasures, while documented topology attacks and memory poisoning [41] expose unresolved risks.Source: §11.4 · p. 47

11.5 Evaluation

• P15 identifies the key evaluation gap: no open benchmark jointly covers E1–E4 at ≳10^3-step traces, and no retained method learns stopping as an RL policy.Progress on P1, P4, and P15 is described as load-bearing because it would unlock progress elsewhere; P5, P10, and P13 are especially deployment-relevant and under-published.Source: §11.5 · pp. 47–48

• The 84-entry corpus is curated for relevance rather than exhaustive, and although record-level decisions are auditable, the review is not PRISMA-reproducible.The manuscript provides neither database-result exports and deduplication logs nor a multi-annotator screening protocol.Source: §12 · p. 48
• Taxonomy tags were assigned by a single author without blinded annotation or inter-annotator agreement, so borderline entries represent taxonomy judgements rather than objective labels.Examples include RL-relevant surveys, industrial systems with undisclosed training details, and self-evolution frameworks lacking conventional RL objectives.Source: §12 · p. 48
• Industrial sources are evidence about deployment shape, scale, harness boundaries, and workflow pressure, not reproducible algorithms or inferred multi-agent RL objectives.Kimi K2.5 is the only pooled industrial source explicitly disclosing RL training of the orchestrator.Source: §12 · p. 48
• The dynamic-Dec-POMDP and orchestration-trace definitions organize vocabulary, while credit-diffusion and non-identifiability claims remain conceptual rather than theorem-backed.Figure 9 is schematic, and a full theory would require assumptions about noise, baselines, graph dynamics, off-policy branch coverage, and variable-shape value approximation.Source: §12 · p. 48
• The literature cutoff is May 4, 2026, so subsequent industrial reports, submissions, benchmarks, and safety evaluations may change coverage and sparsity conclusions.The message-credit, orchestrator-credit, adaptive-deliberation, and trace-level safety cells are especially time-sensitive.Source: §12 · p. 48

• The artifact records 84 retained entries and 32 screened-but-excluded decisions with controlled metadata, public identifiers or documentation handles, and URLs.Source: §13 · p. 48
• Repository scripts regenerate retained/excluded counts, controlled-field histograms, and cross-tabs used in Tables 1, 2, and 22.Source: §13 · p. 49
• A machine-readable JSON Schema, minimal valid example, and dependency-free checker support typed orchestration traces and validate fields, events, references, identifiers, and cost values.The checker is a lightweight sanity check rather than a complete JSON Schema implementation.Source: §13 · p. 49
• The artifact makes taxonomy support, sparsity counts, and minimal benchmark-trace conformity mechanically inspectable, but it is neither exhaustive nor a substitute for multi-annotator systematic review.Source: §13 · p. 49

• Orchestration traces organize the field beyond per-agent trajectories, while credit diffusion and spawn-decision non-identifiability remain motivating observations rather than formal theorems.The trace abstraction supports taxonomy and auditability, but tight rates and full proofs are deferred.Source: §14 · p. 49
• The taxonomies identify eight reward families, eight credit-bearing units, and five orchestration sub-decisions, with counterfactual message-level credit sparse and explicit RL training for stopping absent.Orchestration reward R7 most directly targets spawning, delegation, and aggregation, with useful weight varying over training.Source: §14 · p. 49
• The industrial–academic bridge is asymmetric: Kimi Agent Swarm publicly trains an orchestrator, whereas OpenAI Codex and Anthropic Claude Code document deployment shapes without establishing RL training of orchestration.Publicly documented deployment features include parallel workflows, harness boundaries, and dynamic spawning.Source: §14 · p. 49
• Academic methods evaluated at T ≲102 cannot be assumed to transfer to Kimi’s reported T ∼103–104 deployment envelope, because rollout cost, frozen harnesses, and signal decay constrain scaling.The gap is primarily infrastructural, involving variable-shape replay (P10), rollout cost (P9), and MAS-native benchmarks at Kimi-like scale (P15).Source: §14 · p. 49
• The survey prioritizes unified orchestration-graph credit assignment and MAS-native benchmarks measuring coordination at Kimi-reported trace lengths rather than success alone.Table 11’s sparse counterfactual message- and orchestrator-credit cells are tractable targets, while (E1) alone weakly distinguishes coordination from compute.Source: §14 · pp. 49–50
• Safe, steerable long-horizon orchestration requires addressing human intervention, inter-agent prompt injection, and sandbox isolation under dynamic spawning, as emphasized by MAGIC [65], Claw Groups, and Anthropic’s trustworthy-agents framework.These deployment-shaped problems have only begun to receive attention through training-time adversarial games and related frameworks.Source: §14 · p. 50

A Entry Cards: Core RL Methods and Anchors for LLM-MAS

• Thirteen core methods and anchors are presented in uniform cards ordered from team/orchestrator to message/tool, mirroring the credit hierarchy.Each card reports a claim, reward shape, credit mechanism, orchestration form, headline result, and limitation.Source: A Entry Cards: Core RL Methods and… · p. 58

A.1 Orchestrator-level credit

• Puppeteer [10] learns a central delegation orchestrator with a critic over team outcomes, while Kimi PARL [28] trains dynamic parallel spawning at reported scale.Puppeteer freezes sub-agents, whereas Kimi PARL combines performance, parallelism, and finish rewards; K2.6 is deployment-envelope evidence only.Source: §A.1 · p. 58

A.2 Role-level credit

• +18.38% over a single-agent baseline is reported for MATPO [43], alongside +14.14% for MALT [44] and AIME 86.5 →93.3 for M-GRPO [19].These methods use role-specific or hierarchical advantages, with limitations including role-balance requirements, hand-engineered rubrics, and fixed hierarchy assumptions.Source: §A.2 · p. 59

A.3 Agent-level credit

• Agent-level methods broadcast or decompose team rewards: MAPoRL [47] establishes post-co-training collaboration, Dr. MAS [15] stabilizes GRPO, and SHARP [31] uses Shapley credit.SHARP’s sampling cost grows combinatorially, while MAGRPO [37] formalizes cooperative MARL but leaves free-riding unaddressed.Source: §A.3 · pp. 59–60

A.4 Turn-level credit

• MarsRL [36] uses pipeline-parallel rollouts with stage-specific advantages, while Context-Folding [55] assigns branch-aware advantages to fold/unfold actions for long-horizon training.MarsRL requires factorable reasoning stages, and Context-Folding remains tied to an agent-specific harness.Source: §A.4 · pp. 60–61

A.5 Message-level credit

• C3 [7] is the pool’s only retained method with explicit counterfactual message-level credit, estimating each utterance’s intervention-based trace return.Its computational cost grows with trace length and message count.Source: §A.5 · p. 61

A.6 Framework-level (cross-cutting)

• Agent Lightning [40] decouples execution from training across agent harnesses and supplies agent- and tool-level credit primitives, but remains a framework rather than an algorithm.It aligns training with industrial harness boundaries such as Codex and Claude Code.Source: §A.6 · p. 61

B Paper Pool Summary Table; A. Focal LLM-MAS training, benchmark, and adjacent framework entries (40 entries)

• Table 16 organizes all 84 pool entries by evidence bucket, while the focal listing records reward, credit, orchestration, scenario, and core-status tags across methods and adjacent frameworks.The CSV is the machine-readable source of truth, with its category histogram reported separately in Table 1.Source: B Paper Pool Summary Table · pp. 61–62, A. Focal LLM-MAS training, benchmark,… · p. 62

C. Industrial systems (cases) (6 entries); D. Classical MARL (conceptual toolkit) (10 entries); E. Benchmarks cited in Sec. 9 (8 entries); G. Safety / Adversarial Robustness (cited in Sec. 10) (6 entries)

• Industrial cases provide deployment-scale evidence, while classical MARL and benchmark resources supply conceptual and evaluation context rather than direct evidence of the paper’s central training claims.The pool distinguishes industrial cases from algorithmic contributions and marks supporting classical-MARL, benchmark, safety, and foundation entries accordingly.Source: C. Industrial systems (cases) (6… · p. 64, D. Classical MARL (conceptual toolkit)… · p. 64, E. Benchmarks cited in Sec. 9 (8… · p. 64

B. Related surveys used for gap analysis (5 entries); C. Industrial systems (cases) (6 entries); D. Classical MARL (conceptual toolkit) (10 entries); E. Benchmarks cited in Sec. 9 (8 entries); F. Single-agent RL & LLM-RL foundations (5 entries); G. Safety / Adversarial Robustness (cited in Sec. 10) (6 entries); H. Critic / tool-use evaluation cited in Sec. 6–9 (4 entries)

• The 84-entry pool is organized into eight manuscript evidence buckets, with core, case, and supp labels indicating relevance to the survey’s argument rather than RL-method status.Source: Table 16 · p. 65

C Artifact, Search Protocol, and Trace Schema; C.1 Search strings and screening counts; C.3 Borderline decisions

• The protocol is quasi-systematic rather than PRISMA-level reproducible, exposing search families, screened records, exclusions, tags, and borderline decisions without raw database exports, deduplication logs, or multi-annotator agreement.The search cutoff is May 4, 2026, and sources include arXiv, ACL Anthology, OpenReview, Semantic Scholar links, official project pages, technical reports, and product documentation.Source: C Artifact, Search Protocol, and Trace… · p. 66, §C.1 · p. 66, Table 17 · p. 67

C.2 Tag schema

• The retained-entry CSV uses 18 fields, while eight entries are verified only partially and Kimi K2.6 is verified for deployment evidence but only partially classified as RL.Partial evidence is used to populate adjacent taxonomy cells, not as sole support for central claims; Kimi K2.6 supports deployment-envelope claims rather than independent RL-training claims.Source: §C.2 · p. 66

C.6 Scripted meta-analysis

• The repository statistics script regenerates retained and excluded counts, controlled-field histograms, and reward–credit, orchestration–credit, and category–verification cross-tabs from the CSV.These scripts support the claim that message-level and orchestrator-level tags, especially explicit credit mechanisms, are sparse.Source: §C.6 · p. 67, §C.5 · p. 67

C.7 Machine-readable trace schema

• The minimal JSON trace schema requires identifiers, events, edges, rewards, and costs, covering orchestration, spawning, messaging, tools, returns, aggregation, human intervention, and safety events.Content references allow replayable topology, reward channels, costs, and safety flow while redacting private prompts, credentials, and tool outputs; a dependency-free validator checks example traces.Source: §C.7 · pp. 67–68

C.8 Artifact update protocol

• Future updates must log candidates, retain them only when they change controlled evidence cells, and rerun the statistics outputs when promoted entries alter tag counts.This procedure keeps curation auditable and supports consistent revision of borderline inclusion and exclusion decisions.Source: §C.8 · p. 68, Table 19 · p. 69, Table 22 · p. 71

C.4 Evidence matrix; C.5 Claim-to-artifact ledger

• The evidence matrix separates algorithmic, training, deployment, and scale support so product documentation is not treated as reproducible algorithmic evidence.The claim-to-artifact ledger similarly bounds each central survey claim to retained fields, scripted counts, or explicitly bounded industrial evidence rather than field-wide prevalence.Source: §C.5 · p. 67, Figure 18 · p. 69, Table 20 · p. 70