PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Utility of Learning about Humans for Human-AI Coordination

Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, Anca Dragan

arXiv:1910.05789v2cs.LGcs.AIcs.HCstat.ML

TL;DR

The paper examines why agents trained to coordinate with AI partners may fail with humans, and evaluates human-aware alternatives in a coordination-challenging Overcooked environment. Agents trained without human data perform well with themselves but much worse with human partners, while incorporating human models improves collaboration. The authors note that the strength and interpretation of this improvement remain bounded by human adaptation, model fidelity, and alternative robustness explanations.

  • Problem

    Agents trained through self-play or population-based training can coordinate with AI partners but may not learn to understand or anticipate human behavior in collaborative tasks.

  • Method

    The paper compares self-play, population-based training, planning, imitation, and reinforcement-learning agents trained or evaluated with behavior-cloned human models in Overcooked.

  • Results

    Agents trained without human data perform very well with themselves but drastically worse with a proxy human, while agents trained with a human model perform significantly better; the same trend appears with real humans.

  • Takeaways & Limitations

    Accounting for human behavior during training is preferable to treating humans as optimal or similar agents, and planning or reinforcement learning with human models outperforms vanilla imitation.

  • Takeaways & Limitations

    The results may partly reflect robustness to wider state distributions rather than human-model fidelity, and real humans can adapt to agent policies in ways the proxy human cannot.

Abstract

from arXiv · show

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be understood by humans. To demonstrate this, we introduce a simple environment that requires challenging coordination, based on the popular game Overcooked, and learn a simple model that mimics human play. We evaluate the performance of agents trained via self-play and population-based training. These agents perform very well when paired with themselves, but when paired with our human model, they are significantly worse than agents designed to play with the human model. An experiment with a planning algorithm yields the same conclusion, though only when the human-aware planner is given the exact human model that it is playing with. A user study with real humans shows this pattern as well, though less strongly. Qualitatively, we find that the gains come from having the agent adapt to the human's gameplay. Given this result, we suggest several approaches for designing agents that learn about humans in order to better coordinate with them. Code is available at https://github.com/HumanCompatibleAI/overcooked_ai.

1 Introduction

Self-play-like training can produce agents that coordinate well with AI partners yet fail with humans because collaborative mistakes and opaque conventions require anticipating human behavior. The paper tests this hypothesis in Overcooked using human models and finds substantial benefits from incorporating human behavior into training.

  • Motivation: Self-play and population-based training have achieved strong performance against human experts in competitive games, despite never training with humans.The paper attributes this partly to distributional shift being less damaging in competitive settings.
  • Motivation: In collaborative games, an unanticipated human mistake can reduce both players’ reward rather than benefit the AI.In the paper’s common-payoff example, the AI expects an optimal human response but receives the worst outcome when the human chooses incorrectly.
  • Motivation: Agents co-trained with AI partners may also learn opaque coordination protocols that humans do not understand, as illustrated by Hanabi hints.Such protocols can fail when paired with a human who does not know how to play their part.
  • Approach: The paper introduces a coordination-challenging Overcooked environment and compares self-play-style agents with agents trained using a behavior-cloned human model.The comparison includes self-play, population-based training, coupled planning, reinforcement learning with a human model, and planning with a human model.
  • Results: Agents trained without human data perform very well with themselves but drastically worse with a proxy human, whereas human-model-aware agents perform significantly better.The same broad pattern appears with real humans, though the human model predicts them less accurately.
  • Implications: The results support accounting for real human behavior during training and favor planning or reinforcement learning with human models over vanilla imitation or self-play alone.The authors recommend improving human models or incorporating them into training populations.

2 Related Work

The related work spans human-robot collaboration, multiagent reinforcement learning, observational self-play, and imitation learning. The paper distinguishes its focus on whether modeling humans at all improves human-AI collaboration.

  • Human-robot interaction: Human-robot interaction commonly plans and learns with non-optimal human behavior models rather than assuming humans are optimal.It also studies robot behaviors that support collaboration.
  • Human-robot interaction: The paper claims to be the first to analyze the optimal-human assumption in deep reinforcement learning and test solutions such as population-based training.This positions the work at the intersection of human-robot interaction and deep RL.
  • Multiagent reinforcement learning: Most multiagent reinforcement learning research focuses on AI-AI interaction rather than human-AI settings.Prior work includes collaborative behaviors and communicative policies in cooperative games such as Hanabi.
  • Related approaches: Observational Self-Play combines imitation learning and multiagent reinforcement learning while assuming test-time agents have settled into an equilibrium to replicate.This differs from the paper’s approach, which allows the human partner’s behavior to remain non-equilibrium.
  • Imitation learning: The paper uses behavior cloning to learn human models from demonstrations, while its main focus is designing agents that collaborate with those models.Behavior cloning itself learns an imitator’s policy from demonstrator data.

3 Preliminaries

The preliminaries define the multiagent decision-process setting and introduce behavior cloning and population-based training as core tools. These methods respectively model demonstrations and evolve a population of reinforcement-learning agents.

  • Multi-agent MDP: A multiagent Markov decision process consists of states, agents with action sets, transitions, and a shared real-valued reward function.The transition function maps the current state and all agents’ actions to a distribution over next states.
  • Experiment layouts: The experiment layouts vary coordination demands, including collisions, asymmetric strengths, forced joint strategies, and non-obvious onion-passing routes.These layouts operationalize both low-level motion coordination and high-level strategy selection.
  • Behavior cloning: Behavior cloning learns a policy by mapping observations to actions from expert demonstrations using supervised learning.Here, the discrete action space makes training a classification problem with cross-entropy loss.
  • Population Based Training: Population-Based Training periodically adapts hyperparameters and selects models within an evolving population of agents.Poor performers are replaced by copies of strong performers with mutated hyperparameters.

4 Environment and Agents

The paper builds a simplified Overcooked environment, learns human behavior models from human-human trajectories, and evaluates self-play, population, planning, imitation, and human-aware agents. Human-aware reinforcement learning and planning are designed to exploit an explicit partner model.

  • Environment: The Overcooked environment requires players to navigate, manipulate ingredients and dishes, and coordinate effectively while preparing and serving food.Its layouts are designed to make multi-step strategy and motion coordination difficult.
  • Environment: Players place three onions in a pot, wait 20 timesteps, transfer the soup to a dish, and serve it for a shared reward of 20.The action space contains up, down, left, right, noop, and interact.
  • Human models: Human models are trained with behavior cloning from approximately 16 human-human trajectories per layout, totaling 18k environment timesteps.Separate models are used as the accessible human model BC and the held-out human proxy HP roxy.
  • Baseline agents: The evaluated self-play agents are PPO self-play and PBT, while coupled planning computes and repeatedly replans joint actions.The planner uses hierarchical action abstractions and near-optimal search under simplifying assumptions.
  • Human-aware agents: PPOBC trains a policy in an environment where the learned human model’s action is part of the dynamics, after self-play initialization and annealing.A separate planning agent assumes access to the other player’s policy and acts near-optimally under that model.

5 Experiments in Simulation

Agents trained with themselves coordinate well with themselves but perform substantially worse with human partners, while human-aware training improves coordination. The same pattern appears in planning and real-human evaluations, although planning requires an accurate human model and gains with real humans are weaker.

  • Experimental setup: Agents are evaluated against the proxy human using cumulative reward over 400 timesteps, with agent type varied across self-play, population-based training, planning, human-model training, and imitation.DRL results average 100 rollouts with standard errors across five seeds.
  • Reinforcement learning: Self-play and population-based agents perform much better with themselves than with the proxy human, while PPOBC outperforms agents trained without human data.PPOBC approaches the gold-standard performance obtained when training directly against the proxy human model.
  • Real-human evaluation: With real humans, PPOBC outperforms self-play methods in three layouts and is roughly on par with the best self-play method in the other two.The real-human effect is weaker than in simulation, but PPOBC remains overall preferable.
  • Planning: Planning shows the same coordination pattern: coupled planning performs well with itself but drops sharply with the proxy human, whereas access to the correct human model improves performance.Using an inaccurate behavior-cloned model can cause planning agents to get stuck in loops.
  • Model quality: The benefit of human-aware training depends on model quality: behavior cloning is relatively effective for the proxy human but can remain inaccurate in some layouts.In Forced Coordination and Counter Circuit, an incorrect human model limits PPOBC performance.

6 User Study

In a real-human user study, PPOBC generally outperformed self-play and population-based training, supporting the benefit of training with a human model, although effects were weaker than in simulation and varied by layout.

  • PPOBC outperformed self-play methods in three layouts and was roughly on par with the best self-play method in the other two.The effect was weaker than in simulation, but PPOBC was overall preferable.
  • F(2, 224) = 6.49, p < .01, with PPOBC significantly outperforming SP (p = .01) and PBT (p < .01).The ANOVA used agent type as a factor and layout and player index as covariates.
  • PPOBC in some cases significantly outperformed human-human performance, suggesting it would also outperform imitation learning in those cases.The suggestion follows from the typical inability of imitation learning to exceed its demonstrator’s performance.
  • Layout differences were attributed speculatively to variation in behavior-cloning model quality and deep-reinforcement-learning training quality.PPOBC performed best where both components were good, while poor human modeling or training limited performance elsewhere.
  • Humans may have weakened the observed simulation gap by adapting to agent policies, unlike the simple HP roxy.This primarily benefited self-play methods with opaque coordination policies and helped PPOBC less because humans had less need to adapt to it.
  • SP and PBT were poor predictive models of humans, whereas BC and HP roxy were good or decent predictive models.Figure 7 reports cross-entropy loss averaged over human-AI data, with standard error across five seeds.

7 Qualitative Findings

Qualitative observations link the performance gap to adaptivity: self-play agents specialize in rigid coordination conventions, while PPOBC adjusts more readily to human behavior.

  • Similar qualitative behaviors appeared in simulation and with real users, although the specific SP and PBT failures differed.
  • Adaptivity to the human: SP agents specialized in one pot and waited unproductively when humans used the other, whereas PPOBC used both pots contextually.The example comes from Asymmetric Advantages and illustrates adaptation to human choices.
  • Leader/follower behavior: SP and PBT expected only clockwise or counterclockwise routes, causing repeated collisions, while PPOBC could act as either leader or follower.PPOBC insisted on a shorter path in one situation but adapted to the human’s route in another.
  • Adaptive humans: Humans learned to anticipate SP and PBT’s fixed delivery protocol, but PPOBC could not exploit this adaptation because its behavior-cloned model was not adaptive.In Cramped Room, humans avoided blocking the agents’ preferred right-down-interact route.

8 Discussion

The paper argues that coordination with humans requires modeling real human behavior rather than assuming partners are optimal or similar. It also identifies limitations and several directions for improving human models, training coverage, test-time adaptation, and learning-human assumptions.

  • Discussion: Agents explicitly trained with a human model achieved significantly better coordination with learned and real humans than agents trained only to coordinate with AI partners.The human-aware agents were also more adaptive and could take both leader and follower roles.
  • Limitations and future work: Separate agents were trained for each layout because the study used a simple human model to isolate the value of human data.The authors present this restricted scope as motivation for exploring more sophisticated human models and algorithms.
  • Future work: Better human models could reduce distributional-shift problems through theory of mind, shared planning, or periodic human data aggregation.The proposed alternatives aim to improve how human models generalize beyond their imitation-learning demonstrations.
  • Future work: Population-based training could incorporate multiple imperfect human models whose aggregate behavior covers a wider range of possible human behavior.This would bias the training population toward human-compatible coordination without requiring every model to be accurate.
  • Future work: Agents could adapt online by selecting among human models with model-based control or by using meta-learning to rapidly adjust to new partners.The paper specifically suggests model predictive control and MAML as possible mechanisms.
  • Future work: The stationary human-policy assumption preserves the Markov formulation but omits human learning and adaptation during gameplay.Recurrent architectures or explicit models of how humans learn are suggested as remedies.

D Population Based Training

Population-based training uses a small evolving population of PPO agents trained across pairings. The section describes its mutation procedure, training setup, and self-pairing reward curves.

  • Population setup: PBT trains a population of 3 PPO agents, with each agent parameterized by the same neural-network structure used elsewhere.The agents are trained through repeated pairings within the population.
  • Population setup: Each PBT iteration trains all possible pairings, then replaces the worst agent with a mutated hyperparameter configuration.Mutations can affect λ, clipping, learning rate, gradient steps, entropy coefficient, and value-function coefficient.
  • Training curves: Figure 10 tracks PPOHP average episode rewards on each layout while pairing the agent with itself or HP roxy according to current self-play annealing.The curves span 400 horizon timesteps.
  • Evaluation: PBT reports mean rewards and standard errors over 5 seeds while linearly annealing reward shaping during training.Training curves and hyperparameters are reported in Figure 11 and Table 4.

E Near-optimal joint planner

The near-optimal joint planner searches for coordinated actions at two levels while using precomputed motion plans to reduce search cost. Its approximations trade generality and exactness for computational feasibility.

  • Planner design: Precomputed optimal joint motion plans provide quickly queried costs for every possible starting and desired goal location.This preprocessing supports the planner’s low-level movement search.
  • Planner design: A high-level A* search selects actions such as getting an onion or serving a dish, mapping each action to joint motion plans.The planner searches over coordinated task-level actions rather than only primitive movements.
  • Approximations: The planner considers player positions but not orientations, reducing the number of motion plans by a factor of 16 at the cost of occasional wasted timesteps.Additional checks reduce this approximation’s impact but are not general.
  • Approximations: Counters are excluded from high-level action selection because incorporating them would greatly increase A* runtime.The authors report that counters matter little in the two evaluated layouts and are minimally used in human gameplay.
  • Approximations: The planner uses a 3-dish delivery look-ahead, although increasing it to 4 dishes is not expected to change rewards substantially on the evaluated layouts.This expectation is based on rollout analysis.
  • Fixed-partner planning: The fixed-partner planner performs two-layered A* search, querying the partner model for one low-level action and searching high-level actions above it.The partner model is made deterministic by selecting its argmax-probability action.

G Planning experiments

Planning experiments were computationally restricted and used deterministic actions, which introduced sources of suboptimality. Planning with the exact human model nevertheless outperformed planning with an inaccurate proxy.

  • Experimental limitations: Planning evaluations used a 100-timestep horizon multiplied by 4, creating a possible source of suboptimality.The shorter horizon was required because of computational complexity.
  • Results: PBC+HP roxy performed much worse than planning with the actual HP roxy test model.The comparison is shown in Figure 5.
  • Results: Deterministic action selection often trapped planning agents in loops lasting the rest of the trajectory, producing little or no reward.Determinism was used to reduce planning complexity.

H Human-AI experiments

The human-AI experiments compare agent types on reward and human-action prediction, including analyses with non-cooperative trajectories retained and numerical safeguards for predictive losses.

  • Figure 12 reports human-AI performance without removing the observed non-cooperative trajectories, producing larger standard errors.Reported standard errors are across human participants for each agent type.
  • PPOBC significantly outperformed PBT on reward when non-cooperative human-AI trajectories were retained.The difference was statistically significant after ANOVA and Tukey HSD correction.
  • Agent type significantly affected reward across the analyzed human-AI data, with F(2, 250) = 4.10 and p < .01.Layout and player index were included as covariates.
  • The PPOBC-versus-SP reward comparison fell just short of statistical significance, with p = .06.
  • Figure 13 reports model accuracy for predicting human behavior in human-AI trajectories, using the same setup as Figure 7 but replacing cross-entropy losses with accuracies.
  • To avoid numerical overflow in Figure 7, correct-action probabilities for all models were lower-bounded at ϵ = 1 × 10−3.This empirically affected PBT and PPOSP most because they were most often confidently incorrect when predicting human actions.
Loading 1910.05789v2…