PaperlayerView exact revision on arXiv

Source-linked AI summary

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Zhuoran Li, Yunzhan Li, Xun Wang, Yihan Du, Longbo Huang

arXiv:2610.01882v1cs.LGcs.AI

TL;DR

Online MARL needs policies that capture complex, multimodal coordination without the computational cost of iterative generative sampling. OMAF uses a Transformer-based one-step flow policy with a path-score surrogate and joint optimization, and reports superior performance across MPE and MAMuJoCo tasks.

  • Problem

    Online MARL needs expressive policies for complex, multimodal coordination, while iterative sampling in generative policies is computationally costly.

    Source: §1 · pp. 1–2

  • Method

    OMAF combines a Transformer-based one-step flow policy and approximate path-score surrogate with synchronized joint flow-policy optimization and softmax Q-value estimation.

    Source: §1 · p. 2, §5.2 · p. 8

  • Results

    Across MPE and MAMuJoCo tasks, OMAF achieves superior performance, with up to 3.4× higher returns than baseline methods.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    The results support one-step flow policies as an efficient and expressive approach for online MARL.

    Source: §7 · p. 11

    The convergence guarantee assumes bounded critic and path-score errors, policy-class approximation conditions, smoothness, a local Polyak–Łojasiewicz inequality, and bounded gradient variance.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

1. Introduction

OMAF combines one-step generative policy learning with coordinated optimization to address the tension between expressive policies and efficient online MARL. Across 10 standard tasks, it reports superior performance and training efficiency.

1. Introduction

  • Generative policies model complex, multimodal action distributions, but iterative sampling is costly and becomes especially challenging as multi-agent joint state-action spaces grow.Online MARL also requires sample efficiency because interaction data are limited and coordination is complex.Source: §1 · pp. 1–2
  • OMAF is designed to balance the expressiveness of generative policies with efficient action generation for online multi-agent learning.The introduction frames this balance as an open challenge for online MARL.Source: §1 · p. 2
  • OMAF uses a Transformer-based one-step flow policy and an approximate path score surrogate, with joint softmax Q-value estimation and synchronized flow-policy optimization for stable, sample-efficient coordination.The one-step policy generates continuous actions in a single denoising step, eliminating iterative sampling while retaining expressive coordination.Source: §1 · p. 2
  • Across 10 standard online MARL environments, OMAF achieves the best performance, with up to 3.4× higher returns and 10.5× greater sample efficiency than baseline methods.The environments are drawn from MPE and MAMuJoCo; Figure 1 summarizes performance and training efficiency.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Prior work spans online MARL methods and generative flow policies, while one-step flow-policy optimization for MARL had been considered only in offline learning. This paper addresses the missing online setting.

2. Related Work

  • The related-work section surveys representative online MARL and flow-policy research, with a more comprehensive review deferred to the appendix.The overview separates online multi-agent RL from flow policies in reinforcement learning.Source: §2 · p. 2

2.1. Online Multi-Agent Reinforcement Learning

  • Online MARL work includes CTDE methods such as MADDPG, MAPPO, VDN, QMIX, and QPLEX, alongside generative-policy approaches including HAQO, OMAD, and DDPL.Other work targets factorization, exploration, robustness, scalability, heterogeneous coordination, and decentralized learning.Source: §2.1 · p. 3

2.2. Flow Policies in Reinforcement Learning

  • Flow-policy research covers value estimation, expressive policies, one-step generation, and policy-gradient optimization, but MARL one-step flow policies had been studied only offline.The paper positions its contribution as direct one-step flow optimization in online MARL.Source: §2.2 · p. 3

3. Background

The background formulates cooperative MARL with shared rewards and entropy-regularized objectives, then introduces softmax Q-value estimation and flow matching as relevant tools.

3.1. Multi-Agent Reinforcement Learning and Efficient Value Estimation

  • Cooperative MARL is formulated as a Dec-POMDP with global states, joint actions, shared rewards, and decentralized agent policies; maximum-entropy RL adds an entropy regularizer to encourage exploration.The softmax Q-value provides a smooth alternative to the hard maximum by weighting actions according to exponentiated Q-values, with estimation via importance sampling.Source: §3.1 · pp. 3–4

3.2. Flow Matching

  • Flow matching learns a continuous transport from a tractable source distribution to a desired data distribution by fitting a time-dependent velocity field.After training, sampling follows the associated ODE, which can be discretized with a first-order update.Source: §3.2 · p. 4

4. Theoretical Bridge to OMAF

The theoretical bridge identifies challenges in likelihood-based exploration, decentralized coordination, and one-step reward optimization. It gives population-level guarantees for policy convergence and one-step generation under stated approximation and optimization conditions.

4. Theoretical Bridge to OMAF

  • Online flow policies face difficult likelihood evaluation for entropy regularization, a tension between joint expressiveness and decentralized execution, and a mismatch between one-step transport and reward optimization.The paper identifies reconciling expressive one-step generation with global coordination as an open online MARL challenge.Source: §4 · p. 4

4.1. Theoretical Guarantees

  • The guarantees rely on bounded critic and path-score errors, policy-class approximation conditions, and optimization assumptions including smoothness, a local Polyak–Łojasiewicz inequality, and bounded gradient variance.The paper presents these as population-level guarantees under practical approximations.Source: §4.1 · pp. 4–5
  • The convergence result establishes that flow-policy optimization approaches the soft-optimal decentralized policy up to approximation and optimization errors.The policy is the unique policy-space minimizer when the stated errors vanish.Source: §4.1 · pp. 4–5
  • The one-step result bounds the performance gap using velocity-estimation error, injected velocity noise, and optimal-flow curvature under its stated conditions.Exact one-step recovery follows in the special case where the listed errors are zero.Source: §4.1 · pp. 5–6

5. The Online MARL via one-step Flow model Method

OMAF combines a Transformer-based one-step flow policy with centralized training objectives for efficient, coordinated online MARL. Its optimization couples value guidance, synchronized policy updates, and agent-wise exploration control.

5.1. Decentralized Flow Policy Formulation

  • OMAF factorizes policies for decentralized execution, uses a Transformer velocity network, and generates actions with one Euler step while approximating intractable likelihoods with a path score surrogate.The approximation supports efficient policy optimization while retaining flow-based modeling and one-step action generation.Source: §5.1 · p. 7

5.2. Efficient Online Centralized Training of the Flow Policy

  • Softmax Q-value guidance, synchronized joint policy optimization, and agent-specific temperature tuning combine to support stable, coordinated learning and fine-grained exploration control.The centralized objective accounts for inter-agent behavior, while the adaptive temperatures maintain desired exploration levels.Source: §5.2 · pp. 7–8

5.3. OMAF Algorithm

  • The training algorithm collects trajectories into a replay buffer, updates centralized critics with sampled joint actions, and periodically optimizes all policies and agent-wise temperatures.Initial trajectories use random policies; subsequent collection uses learned flow policies.Source: §5.3 · p. 8

5. The Online MARL via one-step Flow model Method

  • OMAF integrates expressive flow policies and efficient one-step generation with path-score-based maximum-entropy optimization for multi-agent coordination.Its one-step generation replaces the iterative denoising used by diffusion policies.Source: §5 · p. 6, §5.3 · p. 8

6. Experiments

Across MPE and MAMuJoCo experiments, OMAF outperforms representative online MARL, diffusion, and flow-based methods. Ablations examine softmax Q sample count, policy architecture and denoising steps, and temperature tuning.

6.1. Experimental Setup

  • The benchmarks cover Cooperative Navigation in MPE and five MAMuJoCo locomotion environments, compared against actor-critic, diffusion-policy, and flow-based online MARL methods.Methods were trained with three seeds, with mean performance and standard deviation reported over ten evaluation episodes.Source: §6.1 · p. 9

6.2. Experiment Results

  • 10.5× improvement in sample efficiency accompanies OMAF’s superior performance across all 10 MPE and MAMuJoCo tasks against baseline algorithms.Figure 3 compares average episode returns over training steps across three random seeds, with one-standard-deviation shading.Source: Figure 3 · p. 10, §6.2 · p. 10
  • OMAF performs well across task performance, training and sample efficiency, exploration, and scalability, with lower computational requirements than diffusion-based OMAD.Figure 4 presents the comparison across multiple metrics.Source: §6.2 · p. 10, Figure 4 · p. 10

6.3. Ablation Study

  • On Ant 2 × 4, setting softmax Q sample count K to 1 substantially degrades performance, while excessive K slightly reduces training efficiency.An appropriate K stabilizes learning and boosts overall return.Source: §6.3 · p. 11, Figure 5 · p. 11
  • On Ant 2 × 4, the Transformer outperforms GRU and MLP variants, reduced-step OMAD performance collapses, and auto-tuned α yields higher returns than fixed settings.The fixed α settings tested were 0.001, 0.01, 0.025, and 0.1.Source: §6.3 · p. 11

7. Conclusion

The conclusion presents OMAF as a one-step flow-policy approach to online MARL, combining joint optimization and softmax Q-value estimation. The related-work discussion positions it against offline multi-agent flow methods and prior flow optimization focused on single-agent settings.

7. Conclusion

  • OMAF combines expressive flow policies, synchronized joint optimization, and softmax Q-value estimation, and outperforms strong diffusion- and flow-based baselines on MPE and MAMuJoCo using one flow step.The conclusion frames one-step flow policies as a potentially efficient and expressive approach for online MARL.Source: §7 · p. 11

A.1. Online Multi-Agent Reinforcement Learning

  • Prior online MARL methods mostly use unimodal policies, while diffusion-policy approaches incur iterative-denoising overhead; OMAF instead uses one-step flow policies.The related-work passage describes this as a gap in online MARL.Source: §A.1 · p. 19

A.2. Flow Policy in Offline Reinforcement Learning

  • Earlier flow-based MARL methods MAC-Flow and OM2P operate offline and use multi-stage training or distillation; this work addresses online learning.The authors state that, to their knowledge, no prior work directly learns flow policies within online MARL.Source: §A.2 · p. 20

A.3. Flow Policy in Online RL

  • Existing flow-policy optimization methods are predominantly developed for single-agent RL; OMAF extends direct one-step optimization to online MARL under CTDE.The related-work passage identifies one-step flow optimization in online MARL as an unaddressed challenge.Source: §A.3 · p. 20

B. Proofs of Theoretical Results

The proofs connect approximation and optimization errors to policy convergence, then bound the consequences of collapsing a flow trajectory to one step. The flow-policy details provide a tractable transition likelihood for optimization.

B. Proofs of Theoretical Results

  • In the realizable, exact-objective limit, the ideal actor objective uniquely matches the soft-optimal policy within the decentralized factorized class.With practical critic and log-likelihood errors, the objective discrepancy is carried explicitly by ε_Q and ε_ℓ.Source: B. Proofs of Theoretical Results · pp. 20–21
  • Shared-latent coupling and the 1-Lipschitz tanh link endpoint error to policy discrepancy and return loss; if all three error terms vanish, one-step recovery is exact.The proof integrates the conditional discrepancy bound over the observation distribution before transferring it to return loss.Source: B. Proofs of Theoretical Results · p. 22

C. Details of the Flow Policy

  • The flow-policy formulation uses an observation-conditioned Transformer velocity field and a path-wise likelihood surrogate for policy optimization.The formulation derives path-score evaluation through continuous-time change of variables and describes its practical computation.Source: C. Details of the Flow Policy · p. 22

C.1. Derivation of path score surrogate

  • For each agent, actions evolve from Gaussian noise along reverse-time flow dynamics; Euler–Maruyama yields a tractable transition likelihood for evaluation and optimization.The reverse-time process injects stochasticity, and its one-step transition has a closed-form likelihood.Source: §C.1 · pp. 22–23

C.2. Detailed derivation of the path score surrogate

The paper uses a path-score surrogate to make flow-policy likelihoods practical for optimization, with a discrepancy bound conditioned on latent-path entropy. It details a Transformer flow policy and contrasts its one-step, CTDE design with Gaussian, diffusion, and single-agent flow approaches.

C.2. Detailed derivation of the path score surrogate

  • When conditional latent-path entropy is small, the path score approximates the exact action log-likelihood in expectation, with entropy-regularized objective discrepancy bounded by α_i ε_i.The path-score surrogate is presented as enabling efficient value estimation and policy optimization compared with diffusion-based policies.Source: §C.2 · pp. 24–25

C.3. Design of the Flow Policy

  • The observation-conditioned Transformer predicts flow velocity and noise, while its sampled path score accumulates base, perturbation, and tanh-correction terms as a tractable likelihood surrogate.With fixed noise schedule and discretization, the approximation's remaining error comes from numerical stabilization and is negligible when κ is not too small.Source: §C.3 · pp. 26–27

C.4. Relationship to Baseline Algorithms

  • Unlike Gaussian policies, OMAF transforms Gaussian noise through a learned velocity field; unlike iterative diffusion, it generates actions in one step and jointly optimizes decentralized policies under CTDE.The paper contrasts OMAF's multi-agent design with prior flow-based reinforcement-learning methods focused primarily on single-agent decisions.Source: §C.4 · p. 29

D.1. Experimental Environments

The experiments use Cooperative Navigation and multi-agent MuJoCo locomotion tasks to evaluate coordination in continuous control. The appendix also reports shared and task-specific hyperparameters and actor–critic implementation details.

D.1. Experimental Environments

  • Cooperative Navigation tests joint landmark coverage and collision avoidance, while MAMuJoCo tasks distribute robot actuators across agents controlling a shared physical system.The benchmark includes Ant, HalfCheetah, CoupledHalfCheetah, Walker2d, and Swimmer locomotion tasks.Source: §D.1 · pp. 30–31

D.2. Hyperparameters

  • The actor uses a one-step flow policy with an observation-conditioned Transformer, while a centralized twin-critic MLP estimates Q-values from joint states and actions.The target policy uses a hard update (ρ = 0), and the critic uses a discount factor γ = 0.99.Source: §D.2 · p. 32
  • Table 1 summarizes hyperparameters shared across tasks; Table 2 reports task-specific optimization settings, and experiments use three independent random seeds unless noted otherwise.Task-specific settings include policy and critic learning rates, gradient clipping thresholds, and the softmax Q-value temperature β.Source: Table 1 · p. 31, §D.2 · p. 32, Table 2 · p. 32

D.3.1 Additional Ablation Studies

The ablations examine one-step generation, Transformer depth, and softmax temperature; additional analyses assess exploration and performance as agent populations grow.

D.3.1 Additional Ablation Studies

  • On Ant 2×4, the one-step variant achieves the best performance among tested denoising configurations, with more than 6,500 episodic returns and one function evaluation per action.Increasing denoising steps does not consistently improve performance; the authors suggest longer transformation sequences may make Q-guided updates less direct.Source: Figure 7 · p. 33, §D.3.1 · p. 33
  • Transformer depth 3 converges faster and achieves superior final performance, while intermediate settings show variance consistent with potential optimization instability.The authors characterize depth selection as a balance between expressiveness and optimization stability.Source: Figure 7 · p. 33, §D.3.1 · p. 33
  • On HalfCheetah 2×3, moderate softmax temperatures provide a better balance between smoothing Q-values and preserving their relative differences for Q-guided optimization.The tested β values range from 0.001 to 3.Source: Figure 7 · p. 33, §D.3.1 · p. 34

D.3.2 Exploration Details

  • At 250K steps on Ant 2×4, OMAF visits 1,162 state-space bins, the most among compared methods, and explores regions unique relative to diffusion-based OMAD.The analysis discretizes state dimensions 23 and 13 into 1,750 bins; the passage attributes broader exploration to the expressive entropy-regularized flow policy.Source: Figure 8 · p. 34, §D.3.2 · p. 34

D.3.3 Scalability

  • OMAF outperforms considered baselines on Cooperative Navigation with 10 and 20 agents, generating joint actions in a single flow step.The scalability evaluation uses Cooperative Navigation to vary agent count while preserving task structure, since adding agents in physics-based environments changes system dynamics.Source: §D.3.3 · pp. 34–35, Table 3 · p. 35

D.4.1 Training and Evaluation Protocol

The evaluation uses controlled seeds and repeated policy rollouts, then compares learning under a limited interaction budget and over longer training horizons.

D.4.1 Training and Evaluation Protocol

  • Algorithms use three fixed random seeds, and evaluation averages episodic returns over 10 episodes before reporting the mean and standard deviation across seeds.The seeds are 0, 42, and 3407.Source: §D.4.1 · p. 35
  • Within a 2M-step budget, OMAF achieves strong performance in substantially fewer training steps, indicating stronger early-stage interaction efficiency.The protocol compares all baselines within the same 2M training-step horizon.Source: §D.4.1 · p. 35
  • OMAF reaches converged performance earlier, while several baselines continue improving with additional environment interactions.The passage notes that baselines require more interaction steps to realize their full performance.Source: §D.4.1 · p. 35

D.4.2 Discussion about the Performance Metric in the Radar Plot

The radar plot aggregates normalized benchmark performance, exploration, scalability, compute, training-time, and sample-efficiency measures. The discussion also states the continuous-action scope and presents qualitative coordination visualizations.

D.4.2 Discussion about the Performance Metric in the Radar Plot

  • OMAF’s Performance score is 99.54, versus 81.60 for OMAD, averaging within-environment normalized peak returns across ten benchmarks.The score is calculated by normalizing mean peak returns within each environment and averaging across the ten environments.Source: §D.4.2 · p. 36
  • OMAF requires 1427.891 s/100K steps, versus 2981.520 for MAFlowRL, and has the highest Training Time score among evaluated generative-policy methods.The score is mapped so that higher values indicate lower computational cost.Source: §D.4.2 · p. 36
  • OMAF’s average sample-efficiency ratio is 9.51, versus 5.61 for OMAD, with higher scores indicating fewer interactions to reach target returns.The ratio averages across ten environments; thresholds are reached based on the mean learning curve across three seeds.Source: §D.4.2 · p. 36
  • OMAF visits 1,162 bins and receives an Exploration score of 100.00, the highest coverage among six algorithms during Ant 2×4’s first 250K steps.The state-occupancy analysis uses 1,750 bins; the visualization score is normalized using the six algorithms’ minimum and maximum occupied-bin counts.Source: §D.4.2 · pp. 36–37
  • OMAF’s Scalability score is 100.00, ahead of HATD3 at 62.80, based on normalized mean returns at 10 and 20 agents.Scores are normalized independently at each agent scale and then averaged.Source: §D.4.2 · p. 37
  • In the reported training setup, OMAF achieves strong performance using only 12 GPUs, compared with 680 GPUs for diffusion-based OMAD, indicating substantially lower GPU consumption.GPU Hours measures computational cost for 100K training steps on Ant 2×4, using measured training time and aggregate FP32 peak GPU throughput; evaluation time is excluded.Source: §D.4.2 · p. 37

D.4.3 Discussion about the Discrete Action Space

  • The evaluation covers continuous-control benchmarks, not discrete-action settings such as SMAC; discrete flow or categorical-action extensions are left for future work.The paper says its proposed flow policy is designed to model continuous action distributions.Source: §D.4.3 · p. 38

D.5. Visualization

  • Across four MAMuJoCo tasks, OMAF visualizations show stable coordinated motion, with reported returns of 8320.8, 5971.7, 7495.1, and 138.9, respectively.The trajectories cover CoupledHalfCheetah, Walker2d 2×3, Ant 2×4, and Swimmer 2×1.Source: §D.5 · p. 38, Figure 9 · p. 39, Figure 10 · p. 40
Loading 2610.01882v1…