PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Cooperate with Humans using Generative Agents

Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, Natasha Jaques

arXiv:2411.13934v1cs.LGcs.AIcs.MA

TL;DR

Coordinating with unseen humans is challenging because their strategies are diverse, while simulated partners may not cover human behavior and human interaction data is limited. GAMMA learns a generative model of partner strategies and samples partners to train Cooperators; evaluations with real humans report improved performance over prior methods across data sources.

  • Problem

    AI agents can struggle to coordinate with unseen humans because human behavior is diverse and differs from strategies seen during training.

    Source: §1 · p. 1

  • Method

    GAMMA learns a generative model of partner strategies from human or simulated data and samples novel partners to train a robust Cooperator.

    Source: §1 · p. 2

  • Results

    GAMMA improves real-human performance over multiple lines of prior work, whether its generative model is trained on simulated populations or human data.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors’ latent-space comparison indicates that simulated populations may not cover the range of human players.

    Source: §7 · p. 10

    With current techniques, adding more agents would exponentially increase dataset size, motivating better sampling techniques.

    Source: §7 · p. 11

Abstract

from arXiv · show

Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

1 Introduction

Cooperation with unseen humans is difficult because human strategies vary, while human-only data is scarce and synthetic training may produce conventions that do not match people. GAMMA learns a generative model of partner strategies and reports improved performance with real humans across training data sources.

1 Introduction

  • Human-only training data can be scarce, while self-play may produce non-human conventions that leave agents poorly adapted to novel human strategies.Self-play agents may persist with conventions formed during training when a human uses a different strategy.Source: §1 · pp. 1–2
  • Discrete simulated-agent populations can be computationally difficult to scale while covering the many strategies found in more complex tasks.The paper reports that human data performs better in its comparison, especially in more complex environments.Source: §1 · p. 2
  • Sampling a novel partner strategy at each training episode lets one adaptive Cooperator train across a range of partners.Human-Adaptive sampling targets training partners toward real human coordinators using a small amount of human-provided data.Source: §1 · p. 2
  • GAMMA trains a generative model on human or simulated partner data, then samples novel partners to train a robust Cooperator.The model’s latent representation can encode a partner’s style or skill, and its decoder generates partner actions.Source: §1 · p. 2
  • GAMMA’s latent space can interpolate simulated agents, capture diversity in human behavior, and be sampled to target a population such as human coordinators.The figure contrasts generative-model coverage with the narrower human-proxy coverage on human data.Source: Figure 1 · p. 3
  • GAMMA significantly improves real-human performance over multiple lines of prior work, whether trained from simulated or human data.The evaluation directly compares approaches trained on simulated populations and human data.Source: §1 · pp. 2–3

2 Related Work

Prior work trains Cooperators against simulated populations or human cooperation data, and separately studies latent partner representations. GAMMA uses a generative model to flexibly sample novel partners and incorporate human data.

Training against simulated populations

  • Population-generation methods diversify simulated partners, but GAMMA instead models populations generatively to sample new partners and can incorporate human data.The experiments compare GAMMA with FCP, CoMeDi, and MEP, including when using the same underlying simulated population.Source: §2 · p. 3

Training with human data

  • Human cooperation data is limited in scale and heterogeneous, and GAMMA combines simulated partner data with limited human data to target human coordination.The related-work discussion describes human-data challenges including arbitrary behavior, preferences, and differences in action speed.Source: §2 · pp. 3–4

Inferring the partner type

  • Earlier partner-modeling approaches infer partner type or latent representations, whereas GAMMA uses its generative model to simulate novel partners during training for robust zero-shot coordination.Some approaches require a predefined set of partner policies, while other work infers representations from interaction data.Source: §2 · p. 4

3 Problem Formulation

The paper frames cooperation as a two-player Markov game with a shared reward, but notes that the objective for human-AI cooperation is unclear in that formulation. It models partner variation with a latent variable and defines the Cooperator’s objective over that latent population.

3 Problem Formulation

  • In the two-player Markov game, agents choose actions from policies, receive shared reward, and transition between states, but the human-AI learning objective is unclear.The formulation defines the fully cooperative joint-policy objective, while distinguishing the unresolved human-AI objective.Source: §3 · p. 4
  • A latent variable z represents partner attributes such as style, skill level, or reaction speed, with the human population modeled as a distribution over z.Source: §3 · p. 4
  • Conditioning on z yields an MDP for the Cooperator, whose objective is to maximize expected value across the latent partner distribution.Source: §3 · p. 4

4 GAMMA: Generative Agent Modeling for Multi-agent Adaptation

GAMMA uses generative models to diversify partner strategies, training a Cooperator to coordinate with novel partners and targeting adaptation toward human behavior when human data is available.

4.1 Learning Generative Models of Partner Behavior

  • A VAE models coordination trajectories from human play or paired simulated agents, encoding partner style and generating behaviors beyond the training data.Its decoder predicts the next action from the latent style and the agent’s own past experience; the authors report greater strategy-space coverage than simulated behavior alone.Source: §4.1 · p. 5

4.2 Training a Cooperator with Generative Coordination Models

  • GAMMA samples a latent partner strategy for each episode and uses the generated partner as part of the environment while reinforcement learning trains the Cooperator.The Cooperator is not trained by imitation learning; it learns task-specific coordination behavior.Source: §4.2 · pp. 5–6

4.3 Targeted GAMMA using Human-Adaptive Sampling and Fine-tuning

  • For human-targeted training, encoded human trajectories define a human-centered latent distribution for partner sampling, and the generative model can also be fine-tuned on human data.The authors describe this as focusing less on irrelevant synthetic partners; fine-tuning updates the encoder and decoder.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate GAMMA in Overcooked, asking whether generative partner models improve training with simulated populations and small human datasets. In proxy-model evaluation, GAMMA improves normalized reward across layouts, with the largest gaps on the most strategy-rich layouts.

Experimental setting

  • Overcooked tests two-player coordination in cooking tasks; the added Multi-strategy Counter layout introduces onion-versus-tomato soup choices to make coordination more challenging.The five prior layouts include Counter Circuit, where players can pass items across the counter or move around the ring.Source: §5 · p. 6, Figure 3 · p. 7

Research questions

  • The experiments ask whether generative training improves on simulated partners, leverages small human datasets, and beats competitive baselines using simulated and human data.These are the stated H1, H2, and H3 research questions.Source: §5 · pp. 6–7

Learning from a Simulated Agent Population

  • For simulated-data experiments, the study uses FCP, MEP, and CoMeDi populations to construct coordination data for training generative models.The populations represent diverse simulated partners through varied skill levels, population entropy, or cross-play-based diversity.Source: §5 · p. 7

Learning from Human Data

  • For human-data experiments, the study assumes 20–50 trajectories and compares PPO-BC-GAMMA with PPO-BC, plus human-fine-tuned simulated-data models using human-adaptive sampling.The described fine-tuning variants include decoder-only and full fine-tuning.Source: §5 · p. 7

Human evaluation

  • Real-human evaluation uses 80 online participants on Counter Circuit and Multi-strategy Counter, while held-out human behavior-cloning agents provide a proxy evaluation for eligible methods.The study rates agents over randomized rounds; methods trained directly against the proxy are excluded from that automatic evaluation because they can exploit its metric.Source: §5 · p. 7

Learning from a Simulated Agent Population

  • GAMMA helps the Cooperator converge to higher normalized reward across environments, with consistent improvement across layouts and the largest gaps on Counter Circuit and Multi-Strategy Counter.The figure reports rewards normalized by the highest reward on each layout; the two largest gaps occur on layouts with the most valid cooperation strategies.Source: Figure 4 · p. 8

6 Results

GAMMA improves cooperation with real humans across simulated- and human-data training, with GAMMA-HA achieving the strongest reported performance. Human ratings and qualitative feedback also favor GAMMA methods, while results depend on the quality and diversity of partner data.

6.1 Evaluation in Simulation

  • Against a behavior-cloning human proxy, GAMMA improved performance over baselines across layouts, with larger gains on more complex layouts.The authors attribute this to more efficiently using simulated agents by providing a landscape of partners for Cooperator training.Source: §6.1 · p. 8

6.2.1 H1: Training with Simulated Data.

  • 88 versus 64: On Multi-Strategy Counter, GAMMA trained using MEP’s high-quality population outperformed MEP.The paper reports this as a 38% improvement.Source: §6.2.1 · pp. 8–9

6.2.2 H2: Training with Human Data.

  • Human-data results varied: PPO+BC+GAMMA improved performance only half the time, while Human Adaptive GAMMA outperformed baselines on both layouts.On the more complex layout, PPO-BC was the strongest baseline.Source: §6.2.2 · p. 9

6.3 H3: Can we obtain better performance than competitive baselines?

  • 60% and 43%: GAMMA-HA surpassed the most competitive baselines in Counter Circuit and Multi-Strategy Counter, respectively.The reported improvements are for real-human evaluations.Source: §6.3 · p. 9, Figure 5 · p. 9

6.3.1 Subjective Human Ratings

  • Human ratings generally favored GAMMA methods over their baselines, and GAMMA-HA received a higher proportion of positive ratings on both layouts.CoMeDi was an exception: it received high ratings on Counter Circuit despite poor performance on Multi-Strategy Counter.Source: §6.3.1 · p. 9, Figure 6 · p. 10

6.3.2 Qualitative Findings

  • Participants described FCP + GAMMA adapting to their actions, including adopting a strategy of placing onions on the table to save time.The authors connect this behavior to training over a more diverse range of partners.Source: §6.3.2 · p. 10
  • Participants described GAMMA agents as logical and consistent; the authors hypothesize baseline unpredictability may reflect human behavior outside its training distribution.The authors connect this observation to the importance of covering more of the human data distribution.Source: §6.3.2 · p. 10

7 Conclusion

The paper presents GAMMA, which uses generative models to produce training partners, and reports stronger cooperation performance and human ratings than baselines. Its scope and benefits depend on available data, and scaling beyond two players would increase dataset demands.

7 Conclusion

  • GAMMA uses generative models to produce training partner agents, and the paper reports better quantitative cooperation performance and subjective human ratings than baselines.The conclusion also notes that simulated populations may not sufficiently cover the range of human players.Source: §7 · p. 10
  • The approach depends on reasonably diverse cooperation data; low-quality simulated population data can prevent significant benefits.The paper also notes that its human studies used Prolific participants, who may not represent broader populations, and that its human dataset is limited.Source: §7 · p. 10
  • The study focuses on two-player cooperation; with current techniques, more agents would exponentially increase dataset size, requiring better sampling.Source: §7 · pp. 10–11
  • The authors identify limited human data as a constraint on training the generative model from scratch and propose conditioning the Cooperator policy on the partner-policy embedding as future work.Source: §7 · p. 11
  • The work frames human-cooperative AI as a potential step toward assistive robots that could support older or disabled people and reduce domestic-labor burdens.Source: §7 · p. 11

A Reproducibility

  • The demo website provides code and additional results, while implementation details and experiment hyperparameters are provided to support reproducibility.Source: A Reproducibility · p. 14

B Implementation details

The appendix specifies how partner populations, trajectory datasets, policies, and generative models were assembled and trained, and describes participant screening for the human evaluation.

B Implementation details

  • Simulated datasets used 100K joint trajectories, while Overcooked Cooperators were trained with PPO and reward shaping for the first 100M steps.For FCP, trajectories were generated by evenly sampling policy pairs; the data were split 70/30 for training and validation.Source: B Implementation details · p. 14

C Human Dataset

  • The human dataset included 16, 17, 16, 12, and 15 trajectories across five original layouts, plus 38 trajectories for Multi-strategy Counter.The original-layout trajectories were about 1,200 steps long; Multi-strategy Counter trajectories were about 400 steps, closer to training episode length.Source: C Human Dataset · p. 14

D Hyperparameters

  • Cooperator policies used a fixed GRU-then-CNN architecture, while generative models used encoder and decoder heads for posterior and action reconstruction predictions.Tables 1 and 2 are identified as listing policy-model and VAE-model hyperparameters, respectively.Source: D Hyperparameters · p. 14, Table 1 · p. 15, Table 2 · p. 15

E Computational Resources

  • The main experiments used AMD EPYC processors and NVIDIA A40/L40 GPUs, taking about one day per Cooperator and about 3,600 GPU hours overall.Preliminary experiments searched for hyperparameters and training frameworks.Source: E Computational Resources · p. 14

F Human Evaluation

  • The 54% female and 46% male participant sample skewed young, and participants who failed survey-response attention checks were excluded.39% were ages 18–26, 44% ages 27–37, 11% ages 38–48, and 6% ages 49 or above.Source: F Human Evaluation · p. 15

F.1 Humans are adapting during evaluation

Participants changed their gameplay and improved over repeated trials, while the study also reports Holm-Bonferroni-corrected significance testing for human-AI team scores.

F.1 Humans are adapting during evaluation

  • Human performance improved with trial count, and participants changed or adapted their gameplay during the evaluation.The evaluation allowed users to change strategy at any point; 17.9% self-identified as novice video-game players and 49.5% as novice Overcooked players.Source: §F.1 · p. 15, Figure 8 · p. 16
  • Figure 7 asked participants whether they had experience playing Overcooked, while Figure 8 tracks human performance across trials.The Figure 8 caption identifies performance improvement with trial count as evidence of learning and adaptation.Source: Figure 7 · p. 16, Figure 8 · p. 16

G.1 Human-AI team scores

  • Human-AI team-score p-values were tested with Holm-Bonferroni correction, with results reported in Table 4.Source: §G.1 · p. 16

G.2 Qualitative analysis

User reports and ratings favor GAMMA-augmented agents on adaptation and human-like behavior, while qualitative feedback contrasts coordinated, purposeful behavior with baseline frustration and inefficiency.

G.2 Qualitative analysis

  • Users rated FCP + GAMMA and GAMMA-HA-DFT as more adaptive and human-like than their respective baselines.Figure 9 reports adaptation ratings; Figure 10 reports human-likeness ratings, with the latter comparison also including CoMeDi + GAMMA.Source: §G.2 · p. 16, Figure 9 · p. 18, Figure 10 · p. 18
  • Table 3 reports significant improvements for GAMMA methods, and Table 4 says statistical significance was achieved for a majority of results.Source: Table 3 · p. 17, Table 4 · p. 17
  • Participants described GAMMA agents as coordinated, efficient, and responsive, including moving aside to avoid blocking teammates.Reports included agents completing tasks through item passing, anticipating the teammate’s next step, and helping rather than getting in the way.Source: §G.2 · pp. 16–17
  • Baseline feedback highlighted frustration, inefficiency, and weak cooperation, including random actions and agents getting in teammates’ way.Participants also described baseline behavior as lacking coordination or intelligence and failing to notice items supplied by the human teammate.Source: §G.2 · pp. 17–18

H Additional simulated results

Additional experiments compare simulated- and human-data training, including performance with held-out proxy and self-play partners and human-adaptive sampling.

H Additional simulated results

  • FCP + GAMMA and GAMMA-HA received lower frustration ratings than their respective baselines.Source: Figure 11 · p. 18
  • Across six layouts, GAMMA matched or outperformed comparison methods trained from FCP, CoMeDi, or MEP simulated data against a held-out human proxy.Figure 12 provides the learning curves; its error bars show standard errors of the mean.Source: H Additional simulated results · p. 18, Figure 12 · p. 19
  • With a held-out human proxy, human-adaptive sampling learned faster on Counter Circuit but did not attain better final performance, while PPO-BC was trained to exploit that proxy.Figure 13a presents the proxy evaluation.Source: H Additional simulated results · p. 18, Figure 13 · p. 19
  • GAMMA-HA showed faster learning in both layouts and better adaptation to held-out self-play agents than PPO-BC.The passage notes that PPO-BC’s proxy exploitation hurts adaptation to more diverse real human players.Source: H Additional simulated results · pp. 18–19

I.1 Full fine-tuning can suffer from insufficient human data

Larger regularization mitigates the insufficient-human-data problem for full fine-tuning. Human-data fine-tuning results depend on the amount of data and training approach, while z-conditioning is unstable in preliminary tests.

I.1 Full fine-tuning can suffer from insufficient human data

  • Decoder-only fine-tuning performs consistently strongly on both layouts, while full fine-tuning is strongest on the first but weaker on the second.The second layout has 11 human coordination trajectories versus 37 for the first, and the authors attribute the difficulty to insufficient data and overfitting risk.Source: §I.1 · p. 19

I.2 Large regularization mitigates the problem of insufficient human data

  • A larger KL-divergence penalty improves the new full-model fine-tuning and brings its performance close to the original best decoder-only fine-tuning.The authors report this mitigates the insufficient-data problem.Source: Figure 14 · p. 20, §I.2 · p. 20

J Can we condition the Cooperator policy on z?

  • The z-conditioned Cooperator learns faster but is unstable: performance falls with longer training, and the authors leave it out of the main experiment.In the Multi-strategy Counter, reward declines after its peak as the policy overfits the encoder.Source: J Can we condition the Cooperator… · p. 20, Figure 15 · p. 21
Loading 2411.13934v1…