PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Yifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao, Kun Wan, Qingyun Wu, Huazheng Wang

arXiv:2605.24202v2cs.AIcs.LG

TL;DR

The paper asks when reinforcement learning improves multi-agent LLM workflows and what training dynamics govern success or failure. Across workflows, tasks, and model scales, it finds that multi-agent RL usually improves over base models, while policy sharing creates workflow- and task-dependent tradeoffs rather than uniform stability.

  • Problem

    Prior work lacks a systematic account of when multi-agent RL improves workflows and why training succeeds or fails.

  • Method

    The study evaluates shared- and isolated-policy RL across three workflows, two task domains, and three model scales in a controlled experimental grid.

  • Results

    Multi-agent RL improves over base models in most evaluated cells, while Isolated-Policy reaches higher peaks but suffers more late-training degradation and Shared-Policy redistributes failure through dominant-role capture.

  • Takeaways & Limitations

    Policy sharing is an auditable workflow-level design choice requiring task- and topology-specific selection and monitoring of per-role drift, not aggregate accuracy alone.

  • Takeaways & Limitations

    Each workflow–task–scale–policy cell uses a single training seed, so cell-level effect sizes should be interpreted cautiously.

Abstract

from arXiv · show

Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while Shared-Policy training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under Isolated-Policy, parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under Shared-Policy, asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs.

1 Introduction

The paper systematically studies when multi-agent RL improves LLM workflows across workflow topologies, tasks, model scales, and policy-sharing strategies. It finds that RL usually improves over base models, while Isolated-Policy and Shared-Policy trade off peak performance against late-training stability rather than yielding uniformly reliable training.

  • Empirical Findings: Multi-agent RL usually improves over base models, but the effect depends jointly on workflow, task, and model scale rather than policy sharing alone.The study was motivated by the lack of a systematic account of when multi-agent RL succeeds or fails.
  • Study Design: The study spans Eval-Opt, Voting, and Orch-Workers workflows; math and code tasks; 0.6B, 1.7B, and 4B models; and shared or isolated policies.Each setting is compared with both a base-model control and a matched-scale, matched-task single-agent RL control.
  • Empirical Findings: Isolated-Policy more often reaches higher peak accuracy but is more prone to late-training degradation, creating a ceiling-versus-floor tradeoff.The paper characterizes this as a policy-sharing strategy tradeoff rather than a uniformly stable training choice.
  • Empirical Findings: Shared-Policy is conservative on the upside yet remains subject to late-training drift, so policy sharing does not eliminate failure.The two strategies route training pressure differently across roles and workflow structures.

2 Related Work

Prior work extends reinforcement learning to multi-agent LLM workflows through workflow-specific objectives, adaptive deliberation policies, and parameter-efficient fine-tuning. Related studies also examine optimization instability, diversity collapse, and role drift in multi-agent LLM reinforcement learning.

  • Multi-agent RL training of LLM workflows: Recent methods adapt GRPO-style objectives to multi-agent workflow structures, including agent- and turn-wise grouping, tree-structured sampling, and Dec-POMDP formulations.The cited approaches include AT-GRPO and MAGRPO.
  • Multi-agent RL training of LLM workflows: Other work learns adaptive meta-policies for deliberation, develops conceptual frameworks for multi-agent reinforcement fine-tuning, and analyzes gradient-norm imbalance across agents.Examples include rank-based RL for deliberation, LoRA-adapter frameworks, and agent-wise normalization to address global advantage-normalization imbalance.
  • Diversity collapse and role drift in LLM RL: Related research addresses diversity collapse and role drift through alternative divergence choices for verifiable-reward RL and lightweight protocol-level monitoring in multi-agent collaboration.These directions target sample-diversity preservation and detection or repair of role drift.

3 Experimental Setup

The experiments form a controlled task × scale × policy-routing × workflow grid using Math and Code tasks with Qwen3 models at three scales. They compare Eval-Opt, Voting, and Orch-Workers under Shared-Policy and Isolated-Policy routing.

  • Experimental Grid: Experiments vary task, model scale, policy-routing strategy, and workflow in a controlled grid spanning mathematical reasoning and code generation.The workflow dimension covers three topologies with distinct role structures and communication patterns.
  • Tasks and Models: Math uses DAPO-Math-17K for multi-step reasoning, while Code uses DeepCoder for code-oriented problem solving across Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B.The same Qwen3 family keeps architecture fixed while varying scale.
  • Workflows: Eval-Opt separates generation and evaluation through a generator and evaluator, testing role specialization versus dominance in optimization dynamics.The evaluator judges answers, provides verdicts and critiques, and may induce revision.
  • Workflows: Voting combines three generators with an aggregator, while Orch-Workers hierarchically connects an orchestrator, three workers, and a synthesizer.Both workflows include same-role multiplicity, with Orch-Workers placing repeated workers in a hierarchical pipeline.
  • Policy Routing: Shared-Policy updates one policy from all workflow roles, whereas Isolated-Policy assigns each distinct role its own role-specific policy adapter.The comparison tests whether a single policy can support multiple functional roles without interference.

4 Results

Multi-agent RL usually improves validation accuracy over base models, but its gains and failure modes depend jointly on workflow, task, scale, and policy-sharing route. Isolated-Policy reaches higher peaks with larger instability, while Shared-Policy generally has lower amplitude without uniformly stable later-training validation.

  • Overall accuracy gains: Multi-agent RL reaches higher validation accuracy than the corresponding base model in the large majority of workflow, scale, and task cells.On most cells, multi-agent accuracy also matches or exceeds the same-scale, same-task single-agent RL baseline.
  • Policy-sharing tradeoffs: Isolated-Policy more often achieves the higher matched-cell accuracy, but its advantage can shrink or reverse later in training.Across the 1.7B × Math dynamics, IP rises faster and reaches a higher peak across all three workflows, then falls back toward or below SP.
  • Policy-sharing tradeoffs: Isolated-Policy exhibits larger training-side instability amplitude across matched cells, whereas Shared-Policy typically plateaus earlier and lower with smaller amplitude.The compared diagnostics include maximum token-level policy ratio, maximum gradient norm, and entropy-collapse depth.
  • Workflow and task interactions: The magnitude and direction of multi-agent improvement vary jointly with workflow and task, and workflow rankings are not constant across tasks or scales.Eval-Opt, Voting, and Orch-Workers differ at matched scale and task, while Math and Code also behave differently.
  • Failure patterns: Shared-Policy’s smaller training-side amplitude does not guarantee stable later validation: Orch-Workers-SP-4B-Math falls off a late-training cliff, while Voting-SP-4B-Math holds a plateau but shifts toward verbose aggregator justifications.Some Shared-Policy cells also have larger token-level policy-ratio amplitude than their Isolated-Policy counterparts.

5 Role-Level Gradient Dynamics

Role-level gradient pressure explains the distinct failure shapes of isolated- and shared-policy training. Isolated policies amplify coherent same-role updates, while shared policies can be captured by roles contributing disproportionate gradient mass, producing workflow- and task-dependent drift.

  • Mechanistic overview: Isolated-policy and shared-policy training share role-level gradient pressure but produce different manifestation geometries selected by workflow topology and policy routing.The isolated-policy mechanism has one gradient source whose surface depends on workflow; shared-policy training has two sources of gradient asymmetry with task- and workflow-conditional surfaces.
  • Shared inference consequence: Both mechanisms can narrow parallel-slot policies, increasing agreement on the same wrong answer when the narrowed mode is incorrect.This wrong-answer agreement is higher than under the base model on the wrong-answer subpopulation.
  • Isolated-policy dynamics: Parallel same-role agents on shared or related prompts receive coherent, co-varying updates through one isolated policy, amplifying that role’s per-step gradient.This condition applies to Voting IP and Orch-Workers IP, where N > 1 same-role agents receive the same outcome reward.
  • Isolated-policy dynamics: In Voting-IP-1.7B-Math, generator χ2 and training perplexity climb sharply while aggregator metrics remain near their first logged values, followed by descending validation accuracy.The trajectory decomposes by role identity: the generator is the parallel role and continues drifting after validation accuracy peaks.
  • Isolated-policy dynamics: At matched workflow, task, and scale, Orch-Workers-IP-1.7B-Math peaks higher and falls farther than the SP run, whose lower peak remains near terminal performance.The IP curve rises and descends, whereas the SP curve rises and plateaus.
  • Shared-policy dynamics: Shared-policy capture occurs when one role contributes more or more distinctive per-step gradient mass, shifting the shared policy toward that role’s distribution and outputs.The captured role’s evaluation-time outputs become typical of the dominant role.
  • Shared-policy dynamics: Token-distribution asymmetry yields task-specific capture, including code-like evaluator emissions, long-form evaluator re-solving, and generator-style drift in a terse aggregator slot.These surfaces occur in Eval-Opt-SP-0.6B-Code, Eval-Opt-SP-1.7B-Math, and Voting-SP-4B-Math, respectively.
  • Shared-policy dynamics: Per-episode frequency asymmetry in Orch-Workers-SP-4B-Math produces worker-shape capture, global training-side amplitude escalation, and terminal validation-accuracy descent.The workflow has 3 workers, 1 orchestrator, and 1 synthesizer, giving workers more episode slots.

6 Conclusion

Multi-agent RL usually improves LLM workflow performance over base models, but training stability depends jointly on workflow, scale, task, and policy sharing. Isolated-Policy and Shared-Policy route training pressure through distinct role-level mechanisms, making policy sharing an auditable workflow-level design choice rather than a universal safety default.

  • Multi-agent RL usually improves workflow performance over base models, but its training trajectories are unstable across workflow, scale, task, and policy-sharing conditions.
  • Isolated-Policy often reaches higher accuracy peaks but can suffer late terminal degradation cliffs, while Shared-Policy produces a distinct failure surface.
  • Role-level gradient dynamics arise from workflow topology and policy routing: Isolated-Policy amplifies same-role gradients, whereas Shared-Policy enables capture by the dominant role.These mechanisms route training pressure through different channels and generate characteristic failure surfaces.
  • Policy sharing should be selected as an auditable workflow-level design choice using workflow topology, role multiplicity, and task fit, with per-role drift monitored alongside aggregate accuracy.

A.1 Limitations

The study’s claims are limited by its LoRA-only training substrate, outcome-only rewards, and single-seed evaluation. These choices leave adapter-capacity effects and process-reward behavior uncharacterized, while making cell-level effect sizes less certain.

  • LoRA substrate: LoRA-only training leaves the interaction between adapter capacity and the base-model role prior uncharacterized, although the reported cross-role gradient mechanisms are expected under full-parameter training.The experiments use LoRA adapters, reported to match full fine-tuning on RL workloads at these scales.
  • Outcome-reward setting only: Outcome-only rewards score final answers and do not establish how process rewards on intermediate role outputs would alter the reported patterns.Process rewards may suppress, amplify, or recompose those patterns by changing each role’s reward geometry.
  • Single seed per cell: Each workflow × task × scale × policy cell uses one fixed seed, so cross-cell consistency does not replace repeated seeds and cell-level effect sizes require caution.The consistency of Isolated-Policy-versus-Shared-Policy cliff signatures across matched cells is treated as a substitute for repeated seeds, not an equivalent.

A.2 Single-Agent RL Baseline Details

The SA-RL baseline matches multi-agent runs in model, task, hyperparameters, and training-step budget while using a single-role workflow and adapter. This isolates multi-agent training effects and enables a generator-only transfer diagnostic within multi-agent workflows.

  • Matched SA-RL baseline: SA-RL matches multi-agent runs on base model, task, hyperparameters, and total training-step budget at each scale.Its rollout uses one generator role and trains a single LoRA adapter with a prompt template matching the multi-agent generator prompt.
  • Matched SA-RL baseline: The baseline differs only in workflow topology and policy routing, enabling residual accuracy columns to isolate multi-agent training’s contribution.SA-RL uses a single-role workflow and single adapter, unlike Isolated-Policy or Shared-Policy.
  • Generator-only transfer: At 1.7B, the SA-RL adapter is additionally applied only to the generator role while supervisor roles remain base-model roles.Table 3 reports validation accuracy across Eval-Opt, Voting, and Orch-Workers on dapo_math and Code, with the SA-RL column reproducing the matched Table 1 value.

A.3 Trajectory-level signatures for the gradient_amplification cells in §5.1

This subsection summarizes trajectory-level signatures for two Isolated-Policy cells, sampled at three checkpoints under dapo_math. It distinguishes generator- and worker-role parallelism and notes the scale mismatch relevant to Orch-Workers validation.

  • Role placement: In Voting-IP-1.7B-Math, same-role parallelism falls on the generator role, whereas in Orch-Workers-IP-1.7B-Math it falls on the worker role.The Orch-Workers cell uses the same columns as Table 6.
  • Scale matching: Panel C2 and Table 6 use different scales, so the scale-matched Orch-Workers Math Isolated-Policy versus Shared-Policy validation-accuracy contrast is reported separately in Fig. 4.Panel C2 uses 1.7B, while Table 6 uses 4B.

A.4 Trajectory-level signatures for the sp_role_capture cells in §5.2

The subsection identifies trajectory-level signatures of role capture in Shared-Policy cells. Gradient asymmetry manifests as evaluator, aggregator, or worker role capture depending on workflow topology and episode-slot allocation.

  • Experimental setup: Three trajectory checkpoints per cell, each covering 100 problems, are used to characterize Shared-Policy signatures under dapo_math or deepcoder_primeintellect.The appendix defines hedging-phrase rates from a fixed phrase list and reports role-specific trajectory statistics.
  • Eval-Opt: In Eval-Opt Shared-Policy cells, evaluator-dominated gradient mass captures the shared slot through a Python solution block on Code or a long re-solve derivation on Math.The captured slot is pulled toward the evaluator’s idiom.
  • Voting: In Voting Shared-Policy, cross-role anchoring keeps token-level shared-policy metrics near initial values while the aggregator trajectory accumulates a captured mode and begins producing the work.The aggregator’s scoring role converges toward the same captured-mode end state as long-form re-solving in Eval-Opt Math.
  • Orch-Workers: In Orch-Workers Shared-Policy, the worker role supplies dominant per-step gradient mass because it occupies three of five episode slots, without per-rollout token-distribution asymmetry.Here, role capture is associated with episode-frequency asymmetry rather than token-distribution asymmetry.

A.5 Hyperparameters … A.8 Compute

The paper combines role-specific LoRA training, workflow- and task-conditional policy-sharing choices, and role-level diagnostics with outcome-based rewards and a shared two-GPU compute stack. Its design implications emphasize that workflow and task fit often matter more for stability than policy sharing alone.

  • A.5 Hyperparameters: Training uses Qwen3 base models with LoRA adapters attached to every linear module, while Isolated-Policy assigns one adapter per role type.The hyperparameter matrix spans Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B cells.
  • A.6.1 Design Implications: Policy-sharing choice is workflow- and task-conditional because Shared-Policy and Isolated-Policy route training pressure through different channels.The appropriate choice depends on workflow, task, and the most fragile role.
  • A.6.1 Design Implications: Isolated-Policy preserves role-distinguishing parameters but can amplify coherent gradients when a repeated same-role agent is also the role most prone to collapse.This makes IP preferable when specialization matters and multiplicity does not coincide with the fragile role.
  • A.6.1 Design Implications: Shared-Policy can suppress same-role amplification but may be captured by a dominant role when roles contribute asymmetrically to per-step gradients.In Voting, capture appears as verbose voter-like output in the aggregator slot while voter-side metrics remain near their first logged values.
  • A.6.2 Monitoring Recommendations: Monitoring should extend beyond aggregate accuracy to per-role metrics, trajectory inspection, and aggregator response shape because aggregate metrics can miss role drift.Recommended signals include per-role perplexity, gradient norm, KL divergence, token concentration, trajectory outputs, and emitted-length distributions.
  • A.7 Reward Functions: Math rewards exact parsed terminal boxed answers, whereas Code rewards the fraction of hidden tests passed by the terminal Python block.Malformed or missing math answers receive a format-error penalty; unparsable, uncompilable, or timed-out code receives 0.
  • A.7 Reward Functions: A single workflow outcome reward is propagated uniformly to every emitted token, while each role update masks tokens emitted by other roles.Shared-Policy aggregates all roles into one adapter gradient; Isolated-Policy restricts each role’s tokens to its own adapter.
  • A.8 Compute: Runs use a frozen base model with LoRA adapters on a single two-GPU node, with memory dominated by adapter and optimizer states plus the vLLM rollout cache.Multiple adapters remain resident and are selected by low-overhead pointer swaps; IP’s additional small-rank adapters add negligible incremental overhead.
Loading 2605.24202v2…