PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Rui Zhao, Jinming Song, Yufeng Yuan, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, Yang Wei

arXiv:2112.11701v3cs.AI

TL;DR

Agents trained through self-play may struggle to coordinate with unencountered human partners, a challenge for human-AI collaboration without human training data. MEP trains a diverse population and a prioritized best-response agent; in Overcooked evaluations, it outperformed comparison methods on average across five layouts with real human partners.

  • Problem

    Agents must coordinate with humans they have not encountered during training, while avoiding the onerous and expensive step of using human data.

    Source: §1 · p. 1

  • Method

    MEP trains a population for individual and pairwise diversity, then trains a common agent by pairing it with population members through prioritized sampling.

    Source: §1 · p. 2

  • Results

    Across five Overcooked layouts with real human partners, MEP outperforms SP, PBT, FCP, and TrajeDi on average.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The paper concludes that entropy maximization may be a promising avenue for achieving diversity and robustness in reinforcement learning.

    Source: §6 · p. 9

    The Population Diversity objective has quadratic runtime in population size, and its unbounded KL divergence may lead to convergence issues.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using any human data. Although such agents can be obtained through self-play training, they can suffer significantly from distributional shift when paired with unencountered partners, such as humans. To mitigate this distributional shift, we propose Maximum Entropy Population-based training (MEP). In MEP, agents in the population are trained with our derived Population Entropy bonus to promote both pairwise diversity between agents and individual diversity of agents themselves, and a common best agent is trained by paring with agents in this diversified population via prioritized sampling. The prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game environment, with partners being human proxy models and real humans. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

1 Introduction

The paper addresses zero-shot human-AI coordination, where agents must collaborate with partners they have not encountered during training. It proposes MEP, which trains a diverse agent population and a common agent paired with population members, and evaluates the approach with human proxies and real humans.

1 Introduction

  • Zero-shot human-AI coordination asks whether agents can collaborate with unencountered humans without requiring human data during training.The paper identifies potential applications including cooperative games, communicative agents, self-driving vehicles, and assistant robots.Source: §1 · p. 1
  • Self-play agents may assume partners follow their own policies, creating distributional shift when paired with humans.In Overcooked, the cited example has an agent waiting for soup from one pot while its human partner fills other pots.Source: §1 · p. 1
  • MEP encourages individual policy diversity and pairwise differences using a safe, computationally efficient Population Entropy surrogate for Population Diversity.The paper describes Population Entropy as a lower bound on the Population Diversity objective with linear runtime complexity.Source: §1 · p. 2
  • MEP trains a common best agent by pairing it with agents from the diversified population using prioritization based on collaboration difficulty.The paper evaluates MEP and other methods in Overcooked with human proxy models and real humans.Source: §1 · p. 2

2 Preliminaries

The preliminaries define the two-player decision process and the human and AI roles used in the paper, then introduce Overcooked and maximum-entropy reinforcement learning. The accompanying population figure describes training agents with both task and population-entropy rewards.

2 Preliminaries

  • A two-player Markov Decision Process specifies states, each agent’s actions, transition dynamics, a discount factor, and a reward function.From one agent’s perspective, the other agent can be treated as part of the environment, yielding a partially observable MDP.Source: §2 · p. 2
  • In the paper’s human-AI setting, the human and AI occupy the two player roles, and the population trains the AI to cooperate with different partners.The human policy is represented as π(H), its model as ˆπ(H), and the AI policy as π(A).Source: §2 · p. 2
  • Overcooked is the coordination testbed, where two players cook soups and must collaborate to achieve a high score.Source: §2 · p. 2
  • Maximum-entropy RL adds expected policy entropy to the reward objective; the population figure applies task and population-entropy rewards to each agent.The preliminaries state that maximum-entropy policies favor exploration and can capture multiple modes of near-optimal behavior.Source: §2 · p. 2, Figure 1 · p. 3

3 Method

MEP trains a diverse population using Population Entropy, then trains a robust agent by prioritizing partners from that population. The method targets both varied partner behaviors and stronger coordination with difficult partners.

3 Method

  • MEP first trains a diverse population, then trains a robust AI agent using prioritized sampling from that population.Source: §3 · p. 3

3.1 Population Diversity

  • Population Diversity combines average policy entropy, encouraging individual exploration, with average pairwise KL divergence, encouraging distinct policies.Source: §3.1 · p. 3
  • The Population Diversity objective has O(n^2) evaluation cost and unbounded KL divergence, which may cause convergence issues when optimized as a reward.Source: §3.1 · p. 3

3.2 Population Entropy

  • Population Entropy is the entropy of the population’s mean policy, a lower bound on Population Diversity with linear runtime and boundedness for categorical distributions.Source: §3.2 · pp. 3–4

3.3 Training a Maximum Entropy Population

  • Population agents are trained with a mean-policy entropy bonus added to task reward, using a copied version of each agent as its task partner.Source: §3.3 · p. 4

3.4 Training a Robust Agent via Prioritized Sampling

  • Because uniform sampling may favor easy partners, MEP prioritizes harder partners to approximate max-min robustness, using rank-based sampling with β controlling prioritization strength.Source: §3.4 · pp. 4–5
  • In the matrix game, MEP converges faster than TrajeDi in both self-play and cross-play return.Source: Figure 3 · p. 5
  • If the training population contains an agent ε-close to a human policy, prioritized sampling yields a human-AI coordination performance lower bound.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate MEP in a collaborative matrix game and Overcooked, using a human proxy and real-human partners. MEP performs strongly against both, while population-entropy reward is associated with greater population entropy and more varied behaviors.

Experimental setup

  • The evaluation includes a matrix game and Overcooked, comparing MEP with SP, PBT, TrajeDi, FCP, and MPD using a human proxy and real players.Overcooked coordination is evaluated across five layouts; the real-human study uses Amazon Mechanical Turk.Source: §4 · pp. 5–6

Toy environment

  • In the collaborative matrix game, MEP converges faster than TrajeDi in both self-play return and cross-play return.The game gives each player an independent choice, with reward determined by the intersection of their choices.Source: §4 · p. 6

Population entropy

  • Population entropy was generally greater for α > 0 than for α = 0 and increased as α grew, supporting the PE reward's intended effect.Here α is the weight of the PE reward.Source: §4 · p. 6, Table 1 · p. 6

Population behavior

  • Agents trained with population-entropy reward displayed less synchronized, less predictable, and more stochastic behaviors than agents trained without it.The paper illustrates this contrast in the supplementary video.Source: §4 · pp. 6–7

Method comparison

  • With the human proxy, MEP outperforms SP, PBT, TrajeDi, FCP, and MPD in all five Overcooked layouts.Performance is measured by average episode reward over 400 timesteps, with standard error across five random seeds.Source: Figure 4 · p. 7, §4 · p. 7

Ablation

  • Both population-entropy reward and prioritized sampling are necessary components for achieving the best human-proxy performance.Figure 4b ablates MEP by removing the PE reward or prioritized sampling.Source: Figure 4 · p. 7, §4 · p. 7

Real-human evaluation

  • Across five layouts, MEP outperforms SP, PBT, FCP, and TrajeDi with real humans, performing on par with Human-Human coordination.The evaluation used Amazon Mechanical Turk and the same procedure as Carroll et al. [6].Source: Figure 5 · p. 8, §4 · p. 8

Qualitative human-AI behavior

  • In real-human play, MEP agents showed flexible task behaviors across layouts, while SP and PBT agents tended to repeat narrower routines.Examples include delivering soup when a partner is busy and adapting soup delivery direction to the human's position.Source: §4 · p. 8

5 Related Work

Related approaches address collaborative-agent robustness through partner data, human models, or policy diversity. MEP instead trains in a zero-shot setting with a diverse population of partner policies and learning-progress-based sampling.

Collaborative-agent training

  • Earlier collaboration methods use partner behavior to select an equilibrium or incorporate a human model, but collecting substantial human data is expensive [27] [52] [6] [23].This paper considers zero-shot training without behavioral data from the human partner [19].Source: §5 · pp. 8–9

MEP in relation to prior work

  • MEP trains a population to maximize a centralized population-entropy objective, then samples partners dynamically by learning progress to target robustness across policies.The paper relates prioritized sampling to PFSP, which was designed for zero-sum games, and derives a connection to cooperative performance lower bounds.Source: §5 · p. 9

Domain randomization

Trajectory diversity

  • Unlike TrajeDi, which optimizes trajectory-level Jensen-Shannon divergence with importance sampling, MEP augments action-level rewards with population entropy.The paper reports superior empirical performance for MEP and notes that TrajeDi's evaluated gradient variance could be unbounded.Source: §5 · p. 9

Fictitious co-play

  • FCP [46] uses checkpoint and seed diversity but, in these experiments, requires a larger population; MEP uses half its population and performs better.The authors report MEP as more effective at obtaining a diverse population for equal computational resources.Source: §5 · p. 9

Other diversity methods

  • DvD [36] and DIPG [31] target single-agent settings, while MEP is designed for multiagent cooperation; the authors describe it as complementary to prior approaches.PSRO-based methods [4] [28] address diverse behavior in games with non-transitive dynamics.Source: §5 · p. 9

6 Conclusion

The conclusion presents MEP as a method for robust human-AI coordination, combining population entropy with learning-progress-based prioritized sampling. It reports greater flexibility and robustness to varied human strategies and identifies combining MEP with other MARL algorithms as future work.

Conclusion

  • MEP uses a population-entropy reward to encourage policy diversity and learning-progress-based prioritized sampling to enhance generalization to unencountered policies.Source: §6 · p. 9
  • MEP-trained agents demonstrate greater flexibility and robustness to various human strategies.Source: §6 · p. 9
  • The paper presents entropy maximization as a promising avenue for diversity and robustness in reinforcement learning, and proposes combining MEP with other MARL algorithms as future work.Source: §6 · p. 9

A Population Entropy Lower Bound

This appendix sets up a lower-bound theorem connecting population diversity and population entropy, and sketches its proof using KL divergence, entropy, and Jensen’s inequality.

Theorem setup

Proof

B Prioritized Sampling and Performance Lower Bound

The AI policy is trained by pairing with population members, with higher sampling priority for harder-to-collaborate agents. The paper states that prioritized sampling improves a lower bound on cooperation across the population, unlike uniform sampling, which gives no worst-case guarantee.

B Prioritized Sampling and Performance Lower Bound

C Relation to Human-AI Coordination Performance

The paper relates population cooperation guarantees to human-AI coordination when the population contains a policy sufficiently similar to the human policy. Its ϵ-close result bounds the collaboration return for similar policies.

C Relation to Human-AI Coordination Performance

  • The stated comparison of collaboration returns is (1 −ϵ)T J(π(2), π(A)) < J(π(1), π(A)) < (1 + ϵ)T J(π(2), π(A)) (35).J(π(i), π(A)) denotes the expected sum reward achieved by π(i) and π(A) collaborating, and T is the total steps in the trajectory.Source: C Relation to Human-AI Coordination… · pp. 15–16
  • If the human policy is ϵ-close to a population policy, the AI policy’s cooperation performance with the human is bounded by the result for that similar policy.The paper defines ϵ-close across states and relates it to expected sum rewards.Source: C Relation to Human-AI Coordination… · pp. 15–16
  • Prioritized sampling optimizes a lower bound on AI-population returns and, if the population is diverse and representative enough to cover human behavior, also on human-AI returns.Source: C Relation to Human-AI Coordination… · p. 16

D Experiment Details

The experiments report average episode reward and standard deviation across five random seeds. The listed MEP settings include the population configuration, entropy weight, and prioritized-sampling parameter.

D Experiment Details

  • Experiments ran each method with 5 random seeds and report average episode reward and standard deviation.Source: D Experiment Details · p. 16
  • The listed training settings include learning rate 8e-4, reward-shaping horizon 5e6, 1.1e7 environment steps per agent, and PPO iterations of 40000 timesteps.Each PPO iteration’s timestep count refers to environment timesteps for each agent-pairing training.Source: D Experiment Details · p. 16
  • MEP used a population of 5, formed from each agent’s beginner, middle, and best checkpoints; its population-entropy weight α was generally 0.01 and β for prioritized sampling was 3.For Forced Coordination, α was 0.04.Source: D Experiment Details · p. 17

E Experimental Results

The supplied experimental results report that MEP converges faster than TrajeDi across the tested hyperparameters. The accompanying figures and tables describe additional method comparisons and parameter settings, without supplying their numerical outcomes here.

E Experimental Results

  • MEP converges faster than TrajeDi under all parameters in the reported hyperparameter search.Source: Figure 6 · p. 17
  • Figure 7 compares average rewards over 400-timestep trajectories with HP roxy, including MEP, SP, and PBT, ablations, and MPD with or without importance-weight clipping.Results include standard errors over five seeds; hatched bars indicate switched starting positions.Source: Figure 7 · p. 18
  • The appendix figures vary TrajeDi’s discount factor and FCP’s population size, while Tables 2–3 cover reward and population entropy across five layouts and α values.The table captions list α values from 0.000 to 0.050.Source: Figure 8 · p. 19, Figure 9 · p. 19, Table 2 · p. 20

Related papers

Loading 2112.11701v3…