Source-linked AI summary
Collaborating with Humans without Human Data
DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, Richard Everett
TL;DR
The paper studies how to train agents that collaborate with novel humans without collecting human training data. It introduces Fictitious Co-Play, which trains a best response to diverse self-play partners and their checkpoints, and finds higher scores with novel agents and humans plus stronger human preference than baselines. FCP therefore offers an effective, simple alternative to human-data-dependent training, although its partner-population construction remains costly and potentially difficult to scale.
Problem
Model-free reinforcement learning methods without human data often fail to generalize from training partners to novel human collaborators, while human-model approaches require costly human-data collection.
Method
Fictitious Co-Play trains an agent as a best response to independently trained self-play agents and their checkpoints, creating diversity in partner conventions and skill levels.
Results
FCP agents score significantly higher than SP, PP, and BCP with novel agent and human partners, and humans strongly prefer partnering with FCP agents.
Takeaways & Limitations
Effective human collaboration can be achieved without human training data by exposing agents to a diverse population of self-play partners and checkpoints.
Takeaways & Limitations
FCP requires manually training and selecting diverse partners, which is time consuming and may introduce researcher bias; more complex games may require unrealistically large populations.
Abstract
from arXiv · showhide
Collaborating with humans requires rapidly adapting to their individual strengths, weaknesses, and preferences. Unfortunately, most standard multi-agent reinforcement learning techniques, such as self-play (SP) or population play (PP), produce agents that overfit to their training partners and do not generalize well to humans. Alternatively, researchers can collect human data, train a human model using behavioral cloning, and then use that model to train "human-aware" agents ("behavioral cloning play", or BCP). While such an approach can improve the generalization of agents to new human co-players, it involves the onerous and expensive step of collecting large amounts of human data first. Here, we study the problem of how to train agents that collaborate well with human partners without using human data. We argue that the crux of the problem is to produce a diverse set of training partners. Drawing inspiration from successful multi-agent approaches in competitive domains, we find that a surprisingly simple approach is highly effective. We train our agent partner as the best response to a population of self-play agents and their past checkpoints taken throughout training, a method we call Fictitious Co-Play (FCP). Our experiments focus on a two-player collaborative cooking simulator that has recently been proposed as a challenge problem for coordination with humans. We find that FCP agents score significantly higher than SP, PP, and BCP when paired with novel agent and human partners. Furthermore, humans also report a strong subjective preference to partnering with FCP agents over all baselines.
1 Introduction
The paper asks whether model-free reinforcement learning without human data can produce agents that coordinate effectively with novel human partners. It introduces Fictitious Co-Play (FCP), evaluates training methods in zero-shot coordination, and reports stronger agent and human-partner outcomes for FCP.
- Zero-shot coordination with novel partners remains a longstanding AI challenge, especially when agents must generalize to human collaborators.
- Self-play and population play generally fail to produce robust collaborators for novel humans in pure common-payoff settings.Population play can reduce strategic diversity by encouraging agents to play similarly.
- The study evaluates training methods in zero-shot coordination with agents and then conducts a human-agent study measuring participant preferences.
- FCP trains an agent to best respond to independently trained self-play agents and their historical checkpoints, creating diversity in conventions and skill levels.Different random seeds support symmetry variation, while checkpoints represent different skill levels.
- FCP outperforms SP, PP, and BCP in generalization to novel agents and humans, while humans significantly prefer partnering with FCP agents.
2 Methods
The methods compare partner-training strategies in Overcooked, a two-player common-payoff cooking environment requiring movement and strategic coordination. FCP builds diverse partners through independent self-play and checkpoints, then trains an adaptive best response against them.
- Fictitious Co-Play (FCP): FCP first trains independently initialized self-play partners and saves checkpoints to represent different symmetry-breaking conventions and skill levels.Using multiple checkpoints adds skill diversity without extra training cost.
- Fictitious Co-Play (FCP): FCP then trains an agent as a best response to the frozen partner pool, requiring adaptation to partners rather than adaptation by partners.This supports a general policy across strategies and skill levels.
- Training comparisons: The baselines are self-play, population-play with randomly paired co-trained agents, and behavioral cloning play using a human behavior model.
- Environment: Overcooked is a two-player common-payoff gridworld in which chefs coordinate sequential actions to cook and deliver tomato soup.Players share rewards for successful deliveries and must coordinate both movement and strategy.
- Evaluation: The evaluation uses five kitchen layouts emphasizing different coordination strategies and reports average deliveries across layouts.Results use five random seeds and held-out partner evaluations over ten episodes per agent-partner pair.
- Implementation: FCP and population-play use populations of N = 32 agents, while FCP uses three checkpoints per partner at initialization, mid-training, and convergence.
3 Related work
Related work spans ad-hoc team play, diversity in multi-agent reinforcement learning, and human-agent interaction. This paper extends these lines by evaluating diverse-partner training with humans in a common-payoff cooking environment and comparing subjective preferences.
- Ad-hoc team play: Ad-hoc team play, also called zero-shot coordination, studies collaboration with novel partners using planning, online learning, diversity, and related approaches.
- Diversity in multi-agent reinforcement learning: Prior multi-agent reinforcement learning studies associate behaviorally diverse training populations with stronger generalization to held-out populations than self-play.
- Human evaluation: Evaluation has often remained in agent-agent settings, with relatively few human evaluations in non-competitive domains.
- Human evaluation: The paper applies diverse-partner training to human-agent collaboration and argues that this approach supports effective collaboration.
- Human-agent interaction: Relative to BCP, FCP removes human-data collection for agent training and adds subjective human preference comparisons alongside objective performance metrics.
4 Zero-shot coordination with agents
FCP is evaluated against held-out agent populations as a proxy for generalization to novel partners. It significantly outperforms all baselines across diverse, low-skill, and human-proxy partners, while past checkpoints are critical to its performance.
- The evaluation reports average deliveries across five kitchen layouts, using ten episodes per agent-partner pair and five random seeds.Episodes have length T = 540, and results are aggregated across layouts.
- FCP significantly outperforms all baselines across three held-out agent populations.The evaluation includes a human-behavioral-cloning proxy, diverse self-play agents, and randomly initialized low-skill agents.
- FCP performs especially better than the baselines when paired with randomly initialized, suboptimal partners.The result suggests greater robustness to partners that behave poorly, though the passage frames implications for humans as a suggestion.
- Removing past checkpoints significantly reduces performance, establishing checkpoint diversity as a key component of FCP training.The ablation compares full FCP with FCP−T, which excludes past checkpoints.
- Adding architectural variation does not improve full FCP, but it improves performance when past checkpoints are unavailable.The relevant comparisons are FCP ≈ FCP+A and FCP−T,+A > FCP−T.
5 Zero-shot coordination with humans
An online within-participant study evaluated FCP and baseline agents with human partners using randomized episodes and kitchen layouts. FCP-human teams achieved the highest scores across maps, participants preferred FCP partners, and behavioral analyses linked this performance to movement and pot-use coordination.
- Human-agent study design: The study recruited 114 Prolific participants who played 20 randomized episodes with agent partners and kitchen layouts, reporting partner preferences after every two episodes.Episodes lasted T = 300 steps, or one minute, and each participant played with the full cohort of agents.
- Finding 1: FCP coordinates best with humans, achieving the highest score across maps: FCP-human teams achieved the highest average scores across maps and significantly outperformed all other agent-human teams, with every p < 0.001.FCP performed as well as or better than the other teams on each individual map.
- Finding 2: Participants prefer FCP over all baselines: Participants significantly preferred FCP partners over every other agent, including BCP, with every p < 0.05.Participants also preferred BCP over PP despite no significant difference in completed deliveries between human-BCP and human-PP teams.
- Human-agent study design: Evaluation metrics aggregated human-agent play across kitchen layouts, with 95% confidence intervals calculated over episodes.Individual-layout results were reported in Appendix D.3.
- Finding 1: FCP exhibits the best movement coordination with humans: FCP moved most often on all layouts except Forced, suggesting better movement-strategy coordination with human partners.Humans rarely moved, whereas this behavior appeared in the training distributions for BCP and FCP.
- Finding 2: FCP’s preferences over cooking pots aligns best with that of humans: FCP’s pot-use preference was closest to humans, with a difference of 0.11 for FCP versus 0.14 for humans.Other agents strongly preferred a single pot, unlike typical human behavior of using both pots.
6 Discussion
FCP addresses zero-shot human collaboration by creating diverse training partners, but its effectiveness is bounded by manual partner selection, fixed rewards, limited task scope, and alignment concerns.
- FCP agents scored significantly higher than all baselines with novel agent and human partners, while humans preferred FCP partners.
- Limitations and future work: Manual training and selection of diverse partners is time consuming and may introduce researcher bias.
- Limitations and future work: FCP used N = 32 partners successfully here, but more complex games may require unrealistically large populations.
- Limitations and future work: The method requires a known, fixed reward function and is designed for adaptive collaboration on a single task rather than general preference inference.
- FCP uses diverse partners to represent different symmetry-breaking conventions and skill levels.
- Societal impact: FCP provides no guarantees that policies align with partners’ preferences, intentions, or welfare, creating potential societal risks.
- Societal impact: Human preference data could be used to fine-tune and filter agents before deployment to encourage better alignment with human values.
- Environment: The environment rewards soup delivery, with shared +20 rewards for delivered soups and +1 rewards for depositing tomatoes.
B.4.1 Architecture and hyperparameters
The experiments use separate layout-conditioned networks and handcrafted state features, while the human-model baseline is trained from a limited set of human trajectories.
- Architecture: The agent network contains five layout-specific three-layer MLPs with 256 hidden units per layer.
- Hyperparameters: The behavioral-cloning network is trained across all layouts using a layout feature, batch size 256, and Adam learning rate 0.0003.
- Architecture: The BC agent uses handcrafted features including positions, orientations, held items, relative player position, pot state, and local occupancy.
- Human-model baseline: Human modeling uses 5 human-human trajectories per layout across 5 layouts, totaling 60,000 environment steps, split between Hpartner and Hproxy.
C.1 Evaluation details
Evaluation averages results across random seeds and held-out partner populations, while population-size experiments show that FCP performance plateaus near 32 training partners.
- Seeds and populations: Results for SP, BCP, and FCP use 5 random seeds, while PP uses one 32-agent population and evaluates its first 5 agents.
- Held-out populations: The diverse SP evaluation population varies random seed, architecture, and training time across 60 agents.
- Evaluation protocol: Each evaluated agent-target pair plays 10 games of length T = 540 for every layout.
- Agent-agent evaluation: FCP outperforms all baselines on every map with every held-out partner population.
- Population size: Table 2 reports mean deliveries with standard deviation over 5 random seeds as population size varies.
- Population size: Performance consistently increases with population size and plateaus around N = 32 training partners.
D.1 Experimental design
The human-agent study uses randomized repeated interactions across layouts and partners, finding higher FCP team deliveries and stronger FCP preferences than the baselines and ablation.
- Participants: The study recruited N = 114 U.S. Prolific participants meeting minimum approval-rate and prior-study criteria.
- Procedure: Participants completed tutorials, practice, agent-partner episodes on multiple layouts, and a debrief questionnaire.
- Randomization: Each participant played 20 episodes, with four episodes on each of five layouts, using randomized layout and partner sequences.
- Task performance: FCP teams completed significantly more deliveries than BCP, PP, SP, and the FCP−T ablation.
- Partner preference: Participants significantly preferred FCP over SP, PP, and BCP.
- Partner preference: Participants significantly favored BCP over PP.
D.3 Additional quantitative results
FCP performs comparably or better than all baselines on every evaluated map in human-agent collaborative evaluation. It is also the most consistently preferred agent across maps, although preferences vary on some specific maps.
- Human-agent collaborative evaluation: FCP performs comparably or better than all baselines on every map in terms of deliveries.The figure reports objective and subjective metrics with 95% confidence intervals in panels (a) and (b).
- Human-agent collaborative evaluation: FCP is the most consistently preferred agent across maps, though participants occasionally preferred other agents on asymmetric and forced maps.
D.4 Additional qualitative results
Participants’ feedback emphasizes that effective collaboration depends on partners adapting, coordinating roles, and pursuing compatible goals. Responses also describe differences in partner responsiveness, speed, and ability to move around the human player.
- Participant feedback: Participants valued partners that adapted to their behavior, coordinated roles, and worked toward the same goal.Feedback described independent or chaotic behavior as difficult to accommodate, while role specialization could make teamwork more efficient.
- Participant feedback: Participants noticed differences in partners’ movement behavior, including whether they moved around the human or became stuck until the human moved.
- Participant feedback: Participants preferred partners willing to accept handed-over tomatoes or plates, especially when the human maneuvered more slowly.
- Participant feedback: Participants reported that partner responsiveness varied, with some strategies becoming clear quickly and others requiring time to discover.
E Related work
The authors distinguish their study from related work through differences in population size, observations, learning algorithm, environment implementation, and behavioral-cloning data. These design differences may contribute to differences in reported results, while the BC comparison was constrained by limited data collection.
- Related-work comparison: The authors summarize these experimental-design differences as factors that may contribute to differences between their results and related work.
- Experimental design differences: The study used population-play with N = 32 instead of population-based training with N = 3; PP performed significantly better than SP in these experiments.The authors suggest that the smaller PBT population may have caused diversity collapse, potentially explaining performance differences.
- Experimental design differences: Agents used egocentric observations and VMPO rather than a top-down whole-environment representation and PPO.The fixed observation size enabled one agent to play across all layouts, unlike related work that trained one agent per layout.
- Implementation differences: The authors implemented the environment in DMLab2D and built their own human-agent interaction pipeline, noting that the recreation was likely imperfect.
- Behavioral-cloning comparison: BC agents used 12,000 rather than 18,000 environment steps per layout because of data collection limitations.The authors also used a larger, deeper architecture and slightly different observable features, but observed no noticeable performance difference.