Source-linked AI summary
Multi-Agent Generative Adversarial Imitation Learning
Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon
TL;DR
Reward design is difficult for complex tasks, and existing imitation-learning approaches face non-stationarity and multiple Nash equilibria in multi-agent settings. The paper proposes a general Markov-game framework that combines multi-agent inverse reinforcement learning with actor-critic optimization. Experiments show imitation of complex cooperative and adversarial behaviors in high-dimensional environments.
Problem
Existing imitation-learning approaches are not generally suitable for multi-agent settings because environments are non-stationary to individual agents and can have multiple Nash equilibria.
Method
The paper introduces a Markov-game framework that learns parametrized policies for interacting experts using multi-agent inverse reinforcement learning and a practical actor-critic algorithm.
Results
The method imitates complex behaviors in high-dimensional environments involving both cooperative and adversarial interactions.
Takeaways & Limitations
The framework bridges multi-agent reinforcement learning with implicit generative models for general multi-agent imitation learning.
Takeaways & Limitations
The practical approach matches the joint occupancy measure because expert policies are unavailable during training, so it cannot obtain the expert-conditioned occupancy measure.
Abstract
from arXiv · showhide
Imitation learning algorithms can be used to learn a policy from expert demonstrations without access to a reward signal. However, most existing approaches are not applicable in multi-agent settings due to the existence of multiple (Nash) equilibria and non-stationary environments. We propose a new framework for multi-agent imitation learning for general Markov games, where we build upon a generalized notion of inverse reinforcement learning. We further introduce a practical multi-agent actor-critic algorithm with good empirical performance. Our method can be used to imitate complex behaviors in high-dimensional environments with multiple cooperative or competing agents.
1. Introduction
Reward design is difficult for complex tasks, and the challenge intensifies in multi-agent environments with potentially conflicting objectives. Imitation learning addresses reward specification through expert demonstrations, but standard approaches face non-stationarity and multiple equilibria in multi-agent settings; the proposed framework extends imitation learning to this setting.
- Motivation: Reward functions are difficult to design for complex, underspecified tasks and can produce undesirable behaviors when important aspects are omitted.The problem is amplified when multiple agents may have different or conflicting reward functions.
- Motivation: Imitation learning learns desirable behavior from expert demonstrations instead of requiring a manually specified reward signal.
- Challenge: Standard imitation and inverse-reinforcement-learning approaches are difficult to apply generally because multi-agent environments are non-stationary and may contain multiple equilibrium solutions.An agent’s optimal policy can depend on the policies of the other agents, creating multiple mutually optimal responses.
- Approach: The proposed framework learns parametrized policies for multiple experts interacting in the same Markov-game environment by integrating multi-agent reinforcement learning with multi-agent inverse reinforcement learning.It extends GAIL in the single-agent case and uses agent-specific discriminators to distinguish learned behavior from each corresponding expert.
2. Preliminaries
The preliminaries define Markov games as multi-agent environments with joint actions, transitions, individual rewards, and policies. They then introduce reinforcement learning, Nash equilibria, inverse reinforcement learning, and occupancy-measure matching as foundations for the paper’s multi-agent framework.
- 2.1 Markov games: A Markov game specifies states, each agent’s action set, joint-action-dependent transitions, individual reward functions, and stationary Markov policies.The joint policy concatenates the agents’ individual policies, while each agent maximizes its own discounted expected return.
- 2.1 Markov games: A trajectory expectation is generated from an initial-state distribution, policy-sampled actions, and transition-process-sampled successor states.The notation distinguishes joint actions from individual actions and can also describe next-step action sampling without state averaging.
- 2.2 Reinforcement learning and Nash equilibrium: Reinforcement learning maximizes expected return from a reward signal, while causal-entropy regularization selects a unique highest-entropy policy among otherwise optimal policies.The regularizer resolves ambiguity because the entropy function is concave and the set of optimal policies is convex.
- 2.2 Reinforcement learning and Nash equilibrium: In a Markov game, a Nash equilibrium is a policy set from which no agent can improve its reward through unilateral deviation.Because each agent’s optimal policy depends on the others’ policies, equilibrium computation can be formulated with constraints involving value and Q-functions.
- 2.3 Inverse reinforcement learning: Inverse reinforcement learning recovers a reward function that rationalizes expert demonstrations, but the objective is ill-defined without restrictions because many reward functions can explain observed behavior.A convex reward regularizer restricts the reward class, for example to rewards linear in predetermined features.
- 2.4 Imitation by matching occupancy measures: Imitation learning can be viewed as matching policy occupancy measures, the state-action distributions generated while navigating the environment.GAIL instantiates this idea with a discriminator distinguishing expert trajectories from policy-generated trajectories and a generator trained to fool it.
3. Generalizing IRL to Markov games
The framework extends inverse reinforcement learning to Markov games by defining multi-agent equilibria and a reward-recovery objective that compares expert and learner policies. Its formulation handles equilibrium constraints through a Lagrangian margin and generalizes single-agent IRL under stated assumptions.
- Multi-agent imitation learning is difficult because each agent has its own reward and optimality is defined through Nash equilibrium.
- MARL(r) represents stationary Markov policies forming a Nash equilibrium under reward r with maximum discounted causal entropy among equilibria.
- The multi-agent inverse operator selects rewards that create a margin between expert and alternative policy sets despite Nash-equilibrium constraints.
- The constrained equilibrium problem is reformulated through a Lagrangian, enabling constraints to be incorporated into an objective that compares expected rewards.
- A trajectory-based dual formulation sums over agents and trajectories, with multipliers corresponding to trajectory-generation probabilities.
- For a specific multiplier choice, the formulation recovers a difference between summed expected rewards of two policies, yielding an IRL-style performance gap.
- The framework strictly generalizes single-agent IRL when N = 1 and β = 1, while multi-agent analysis requires a unique MARL solution under the stated regularization assumptions.
- Because an expert policy is unavailable during training, the practical method matches occupancy measures between expert and learner joint policies instead.
4. Practical multi-agent imitation learning
The section develops practical MAGAIL variants for centralized, decentralized, and zero-sum reward structures, plus MACK for variance-reduced multi-agent policy optimization.
- Multi-agent generative adversarial imitation learning: MAGAIL uses discriminators as implicit reward functions, training policies to distinguish expert demonstrations from generated behaviors.
- Multi-agent generative adversarial imitation learning: Three reward priors support centralized cooperation, decentralized agents without reward-correlation assumptions, and two-agent zero-sum interactions.
- Multi-agent generative adversarial imitation learning: In zero-sum MAGAIL, expert and policy trajectories are paired for discrimination rather than generated through policy interaction with the environment.
- Multi-agent generative adversarial imitation learning: In decentralized MAGAIL, each agent has an observation-based discriminator, while the agents’ discriminators interact indirectly through the environment.
- Multi-agent actor-critic with Kronecker factors: MACK applies centralized training with decentralized execution, using all agents’ observations and actions in advantage estimation to reduce variance.
- Multi-agent actor-critic with Kronecker factors: MACK does not require knowledge of other agents’ policies or differentiable estimators for discrete actions.
5. Experiments
Experiments evaluate MAGAIL variants in cooperative and competitive particle environments and in cooperative control, including adaptation to changed dynamics and imperfect demonstrations.
- Experiments: MAGAIL variants, GAIL, and behavior cloning are evaluated in particle and cooperative control environments using results averaged over 5 random seeds.
- Cooperative tasks: MAGAIL consistently outperforms behavior cloning across cooperative settings as the number of expert demonstrations increases.
- Cooperative tasks: With 200 demonstrations in Cooperative Communication, centralized MAGAIL reaches expert-level performance while behavior cloning remains far below it with 400 trajectories.
- Competitive tasks: In competitive tasks, decentralized and zero-sum MAGAIL often outperform centralized MAGAIL and behavior cloning, highlighting the importance of selecting a suitable reward prior.
- Cooperative control: With 10 imperfect demonstrations, centralized MAGAIL reduces failure from behavior cloning’s 39.8% to 26.2% and obtains reward 26.57 versus 1.26.
6. Related work and discussion
The discussion positions MAGAIL against single-agent imitation learning and prior specialized multi-agent methods, presenting it as a general Markov-game IRL framework.
- Related work: Behavior cloning uses supervised learning, whereas IRL recovers an unknown reward and then learns a policy through reinforcement learning.
- Related work: Existing multi-agent imitation work commonly assumes specific reward structures, including fully cooperative or particular non-cooperative settings.
- Discussion: The paper proposes a general multi-agent IRL framework bridging multi-agent reinforcement learning and implicit generative models.
- Notation: The notation uses hats to abbreviate value, action-value, and trajectory quantities whose dependence on policy and reward is implicit.
A.1 Proof to Lemma 8
The proof establishes that the Bellman-based constraints characterize Nash equilibrium by connecting constraint violations to unilateral profitable deviations.
- Proof to Lemma 8: For any policy, the Bellman residual is zero when the candidate value function is that policy’s value function.
- Proof to Lemma 8: A Nash equilibrium satisfies the inequalities requiring each agent’s value to dominate every unilateral action value.
- Proof to Lemma 8: If an inequality fails, the corresponding agent can change its actions at the violating state and obtain strictly higher expected return while other policies remain fixed.
- Proof to Lemma 8: When all constraints hold, convex combinations over actions recover the Bellman constraints, and Lemma 8 implies that the policy is a Nash equilibrium.
- Proof to Lemma 8: The argument applies the same unilateral-deviation reasoning to every agent, state, and action.
A.4 Proof to Theorem 6
The proof decomposes the multi-agent inverse reinforcement learning objective across agents, then applies the single-agent GAIL result to recover the same solution.
- The MAIRL objective separates across agents because each reward and policy is independent.
- Each single-agent MARL objective has a unique expert-policy solution under the paper’s assumption.
- Applying Ho and Ermon’s single-agent proposition to every agent yields the same solution as composing MARL with MAIRL.
A.5 Proof to Proposition 7
The proof identifies the adversarial objective with Jensen-Shannon divergence and presents MAGAIL as an iterative multi-agent policy, discriminator, and value-estimation procedure.
- The adversarial objective equals the Jensen-Shannon divergence between policy and expert occupancy measures up to shift and scaling.
- The proof concludes that the two compared quantities are equal after taking the minimum of the adversarial objective.
- MAGAIL initializes policy, discriminator, and value estimators together with expert trajectories and a Markov game.
- Each iteration samples policy trajectories and expert state-action pairs, then updates value estimates, discriminators, and policies.
- The experiments use two-layer multilayer perceptrons, with batch size 1000 for the particle environment and 2048 for cooperative control.
C.2 Detailed Results
The detailed experiments use particle and multi-agent control environments, evaluate cooperative and competitive tasks, and report episode-level variability alongside sample-complexity results.
- Experiments use the particle environment and multi-agent control environment, with means and standard deviations computed over 100 episodes.
- Competitive-task policies are trained from 100 expert trajectories, while cooperative-task policies use varying numbers of demonstrations.
- Environment randomization produces seemingly high standard deviation across episodes because landmark locations and other episode conditions differ substantially.
- Figure 3 normalizes expert performance to one and behavior-cloning performance to zero when showing cooperative-task sample complexity.
C.3 Video Demonstrations
The video demonstrations show expert, proposed-framework, and behavior-cloning trajectories across multi-agent control and particle environments, including a hard-environment failure mode.
- Particle-environment videos compare behavior-cloning agents with agents trained by the proposed framework in navigation and predator-prey interactions.
- The demonstrations include expert trajectories in an easy environment and centralized GAIL and behavior-cloning trajectories in a hard environment.
- In the hard environment, agents mostly fail when the plank falls or bounces off because lower plank weight reduces friction and increases acceleration.