Source-linked AI summary
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
Filippos Christianos, Lukas Schäfer, Stefano V. Albrecht
TL;DR
Sparse-reward MARL makes exploration difficult because agents must learn environment dynamics and joint actions while learning concurrently. The paper proposes SEAC, which shares agents’ experience within an actor-critic framework. Across sparse-reward tasks, SEAC learns faster and reaches higher returns, with harder tasks sometimes solved only by SEAC; its scope remains bounded by stated task assumptions and evaluation settings.
Problem
MARL exploration is difficult because joint action spaces grow exponentially, concurrent learning causes non-stationarity, and sparse rewards make most transitions uninformative.
Method
SEAC updates each agent’s actor and critic using its own experience together with weighted, importance-corrected experience from other agents.
Results
Across ten sparse-reward tasks in four environments, SEAC outperformed independent learning, shared-policy training, and state-of-the-art MARL algorithms in sample efficiency and final returns.
Takeaways & Limitations
SEAC can make the difference between learning to solve harder sparse-reward tasks and not learning at all, while adding less than 3% running time versus independent learning.
Takeaways & Limitations
Future work must relax assumptions on the tasks SEAC can address and evaluate experience sharing across additional MARL algorithms and environments.
Abstract
from arXiv · showhide
Exploration in multi-agent reinforcement learning is a challenging problem, especially in environments with sparse rewards. We propose a general method for efficient exploration by sharing experience amongst agents. Our proposed algorithm, called Shared Experience Actor-Critic (SEAC), applies experience sharing in an actor-critic framework. We evaluate SEAC in a collection of sparse-reward multi-agent environments and find that it consistently outperforms two baselines and two state-of-the-art algorithms by learning in fewer steps and converging to higher returns. In some harder environments, experience sharing makes the difference between learning to solve the task and not learning at all.
1 Introduction
MARL exploration is difficult because agents jointly explore a growing action space while learning dynamics non-stationarily, especially under sparse rewards. SEAC addresses this by sharing experience in actor-critic learning and reports faster learning, higher returns, and low added runtime.
- Motivation: MARL exploration combines environment-dynamics exploration with a joint action space that typically grows exponentially with the number of agents.Concurrent learning also creates non-stationarity, while sparse rewards make most transitions uninformative.
- Motivation: Experience sharing lets agents learn from successful trajectories generated by one another, supporting more even learning progress.In the motivating game, both agents can learn approaches to the goal from different directions after a successful episode.
- Approach: Shared Experience Actor-Critic (SEAC) combines an agent’s gradients with weighted gradients computed from other agents’ experiences.The method is presented as a novel actor-critic MARL algorithm operating similarly to independent learning.
- Results: 70% fewer required training steps and higher final returns were reported for SEAC across four sparse-reward multi-agent environments against several baselines and state-of-the-art algorithms.Comparisons included independent learning, shared-policy training, MADDPG, QMIX, and ROMA.
- Results: Less than 3% additional running time was observed across all environments compared with independent learning.
2 Related Work
Related work improves multi-agent learning through centralized training, teaching, demonstrations, distributed computation, and population diversity. SEAC instead shares trajectories among concurrently learning agents to speed and synchronize MARL training.
- Centralised Training with Decentralised Execution: CTDE methods access all agents’ data during training but learn decentralized policies, while MADDPG and related methods reinforce an agent’s own tried actions.SEAC differs by using shared experience to reinforce agents’ learning.
- Agents Teaching Agents: Teaching approaches exchange expertise or instructions between agents, but most prior work focuses on single-agent reinforcement learning.Multi-agent teaching can share knowledge following interactions between agents.
- Learning from Demonstrations: Learning from demonstrations uses trajectories from other agents or humans to derive or refine policies, or to shape rewards.SEAC instead uses trajectories generated by concurrently learning agents in a multi-agent system.
- Distributed Reinforcement Learning: Distributed reinforcement learning shares computation and data across parallel actors to improve trajectory collection and optimization efficiency.Examples include A3C, IMPALA, and SEED RL.
- Population-play: Population-based training improves exploration and coordination by training diverse agent populations that may share policies while evolving together.
3 Technical Preliminaries
The paper models MARL with Markov games in which agents receive local observations, choose actions, and optimize discounted returns. Policy-gradient and actor-critic methods provide the learning foundation, with shared observation and action spaces assumed here.
- Markov Games: A Markov game specifies agents, states, local observation spaces, action spaces, joint observations and actions, transitions, and individual reward functions.
- Markov Games: Each agent observes locally, while the transition function maps a state and joint action to a distribution over successor states.
- Markov Games: The objective is to find policies whose agents’ discounted returns are maximized with respect to the other policies.
- Assumptions: The formulation assumes identical observation and action spaces across agents but does not require identical reward functions.
- Policy Gradient and Actor-Critic: Policy-gradient methods directly optimize a parameterized policy, while actor-critic methods estimate returns to reduce gradient-estimation variance.The experiments use a modified actor-critic algorithm.
4 Shared Experience Actor-Critic
SEAC extends on-policy actor-critic learning by treating other agents’ trajectories as off-policy data and combining them with each agent’s own experience. The method supports multi-agent generalization, uses a tunable sharing weight, and is also discussed as applicable beyond on-policy actor-critic.
- Assumptions: SEAC assumes local policy gradients from one agent’s experience provide useful learning directions for the other agents, even when reward functions need not be identical.
- Experience Sharing: Each agent supplements its own on-policy trajectory with other agents’ off-policy trajectories during every training step.The shared data are generated by agents executing different policies than the one being optimized.
- Off-Policy Correction: Importance sampling corrects the policy-gradient objective when trajectories come from a behavioral policy different from the optimized policy.
- Losses: The actor and critic losses combine local and other-agent experience, with λ weighting the other agents’ experience.SEAC was largely insensitive to λ and used λ = 1 in the experiments.
- Algorithm: The framework provides pseudocode for gradient updates to each agent’s actor and critic parameters, and its two-agent derivation generalizes to more agents.
- Generality: Experience sharing is also presented as applicable to off-policy methods such as DQN, although deep off-policy methods are described as less stable than on-policy actor-critic.
5 Experiments
SEAC is evaluated across four sparse-reward multi-agent environments against independent, shared-policy, and state-of-the-art baselines. It generally learns faster and achieves higher returns, with experience sharing especially valuable in harder tasks.
- Evaluation setup: SEAC is evaluated in four sparse-reward environments against IAC, SNAC, MADDPG, QMIX, and ROMA.The environments are Predator Prey, SMAC, LBF, and RWARE.
- Results: In sparse Predator Prey, only SEAC learns successfully and does so consistently across seeds.IAC and SNAC are unable to learn to catch the prey.
- Results: In sparse SMAC, SEAC outperforms both actor-critic baselines, although none of the methods learns to win battles reliably.Even QMIX does not successfully solve this sparsely rewarded task.
- Results: On harder LBF tasks, SEAC converges to higher returns than IAC and SNAC, while IAC shows no learning on the largest grid.The easiest LBF variant shows no significant difference, and SEAC tends to converge in fewer timesteps than IAC.
- Results: On the hardest RWARE task, SEAC’s final mean returns are approximately 70% higher than IAC and 160% higher than SNAC.SEAC also converges in fewer steps than IAC.
- Comparisons: Across selected tasks, MADDPG, QMIX, and ROMA show little learning, while shared experience reduces off-policy Q-learning variance but has less impact than in SEAC.MADDPG matches SEAC on sparse PP, and QMIX is comparable on cooperative LBF.
- Efficiency: SEAC increases running time by less than 3% compared with IAC across all environments.This measures computational time rather than environment-sample efficiency.
- Analysis: In RWARE, SEAC’s learning curve begins rising at roughly 1/N of IAC’s timesteps, where N is the number of agents.The paper attributes higher final returns to similar learning rates alongside flexibility for agents to develop different policies.
6 Conclusion
The paper concludes that SEAC improves multi-agent learning by sharing experience while preserving policy differences that support coordination and exploration. It also notes that the method is computationally simple, while identifying broader applicability as future work.
- SEAC outperformed independent learning, shared policy training, and state-of-the-art MARL algorithms across ten sparse-reward tasks in four environments.
- SEAC combines local gradients and concurrently learned similar policies without restricting agents to identical policies.This design supports coordination and exploration while allowing agents to develop distinct behaviours.
- Experience sharing requires barely any additional computational power, extra parameter tuning, or additional networks.
- Future work should relax the assumptions defining tasks to which SEAC applies and evaluate experience sharing across additional MARL algorithms and environments.
Broader Impact
MARL may have applications in autonomous systems and infrastructure, but the paper emphasizes that real-world deployment is currently constrained by unresolved safety, governance, and robustness issues. Strong benchmark performance alone does not establish that models are safe or broadly usable.
- Potential MARL applications include autonomous vehicles, robotic warehouses, internet of things, and smart grids.
- Real-world MARL application is currently not viable because explainability, failure robustness, legal, and ethical problems remain open.These issues are outside the scope of the work.
- Improved MARL can create undue trust in reinforcement-learning models, since good task performance does not establish safety or broad usability.
Funding Disclosure
The paper describes sparse-reward cooperative and mixed cooperative-competitive environments used to evaluate multi-agent learning. Tasks vary in warehouse layout, agent and shelf counts, grid configurations, and cooperation requirements.
- Multi-Robot Warehouse: The multi-robot warehouse simulates robots delivering requested shelves to goals and returning them to empty locations.The environment uses a four-action space and 3 × 3 local observations containing nearby agents and shelves.
- Multi-Robot Warehouse: Warehouse agents receive reward 1 for each requested shelf delivered to a goal, creating a very sparse reward signal.The task also requires finding an empty shelf location afterward.
- Multi-Robot Warehouse: Warehouse difficulty varies with size, agent count N, and requested shelves R; default R = N, while easy and hard settings use R = 2N and R = N/2.Smaller R values make random discovery of the correct shelf increasingly improbable, especially on larger grids.
- Level-Based Foraging: Level-based foraging is a mixed cooperative-competitive grid game in which agents collect food by coordinating when required.Agents and food have levels, and collection succeeds when participating agents’ combined levels meet the food level.
- Level-Based Foraging: The fully cooperative foraging variant requires all agents to load simultaneously because food levels equal the sum of all agents’ levels.The reward is shared among the agents.
B Additional Experimental Details
The experiments implement IAC, SEAC, and SNAC in an A2C-style setup and compare their training behavior, sensitivity, and process time. SEAC is largely insensitive to λ across a broad range, while very low values reduce performance toward IAC.
- Implementation: IAC, SEAC, and SNAC closely follow A2C using n-step returns and parallel sampled environments.The experiments use four parallel processes, five n-steps, gradient clipping at 0.5, and λ = 1.0 in the listed hyperparameters.
- Implementation: Hyperparameters for MADDPG, QMIX, and ROMA were optimized by grid search across learning rate, exploration rate, and batch size in all environments.
- Process Time: Process time was measured for 100,000 environment iterations using Python 3.7 and PyTorch 1.4 on a 6th Gen Intel i7 @ 4.6 Ghz.
- λ Sensitivity: SEAC shows similar training performance across a wide range of λ values, indicating low sensitivity to this hyperparameter.Much lower values closer to 0 decrease performance and eventually converge to IAC at λ = 0.
- Evaluation: Evaluation returns were computed from the best saved model per seed over 100 episodes, with algorithm-specific evaluation policies.
C SEAC Loss Derivation
SEAC derives shared-experience actor and value losses under reward-independence and symmetry assumptions, correcting for differing action distributions with importance sampling. The derivation combines on-policy terms with off-policy updates from another agent’s experience.
- SEAC defines each agent’s policy and value function separately, with policies parameterized by φ1 and φ2 and value functions by θ1 and θ2.
- Off-policy loss: Importance sampling corrects for different action distributions when one agent’s experience is used to train another agent’s policy.
- Assumptions: The reward-independence assumption treats other agents as part of the environment and makes perceived rewards depend only on an agent’s own action.
- Assumptions: The symmetry assumption introduces a state-mapping function f that swaps the agents, supporting corresponding reward and transition relations.
- Loss derivation: The derivation trains an agent’s policy and value function with another agent’s experience, while retaining on-policy terms identical to A2C.
- Off-policy comparison: SEQL samples replay-buffer experiences across agents and applies the same DQN loss to each agent, using equal sample counts from each buffer in the experiments.
D.2 Results
Shared experience improves off-policy Q-learning unevenly across the evaluated RWARE and LBF tasks. Its effects differ from those observed with actor-critic methods, particularly in variance and the magnitude of performance gains.
- On LBF, SEQL’s average returns increased significantly through experience sharing and at its best evaluation exceeded SEAC’s average returns.
- On RWARE, experience sharing considerably reduced variance without significantly changing average returns.
- Off-policy SEQL and IQL showed significantly larger variance than on-policy SEAC, IAC, and SNAC.