Source-linked AI summary
Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration
Mesut Yang, Micah Carroll, Anca Dragan
TL;DR
Human models can require substantial real-human data, while random initialization offers little structure for learning behavior efficiently. This paper initializes human models from near-optimal behavior, finding improved data efficiency and generalization, as well as better collaboration performance under tested conditions. These results are based on simulated proxy humans, and require further verification with real humans.
Problem
Human models require real behavioral data, but limited data makes it difficult to capture behavioral nuances while retaining expressive models.
Source: §1 · p. 1Method
Results
OBP-based human models increase data efficiency and generalize across environments, and OBP-trained collaborative agents outperform the other tested conditions.
Source: §1 · p. 2Takeaways & Limitations
An optimal behavior prior can make multi-environment human modeling possible and improve downstream collaboration in the tested setting.
Source: §7 · p. 9The experiments use simulated proxy humans, so further work is needed to verify whether the results hold with real humans.
Source: §7 · p. 9
Abstract
from arXiv · showhide
AI agents designed to collaborate with people benefit from models that enable them to anticipate human behavior. However, realistic models tend to require vast amounts of human data, which is often hard to collect. A good prior or initialization could make for more data-efficient training, but what makes for a good prior on human behavior? Our work leverages a very simple assumption: people generally act closer to optimal than to random chance. We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient and able to generalize to new environments. Our intuition is that such a prior enables the training to focus one's precious real-world data on capturing the subtle nuances of human suboptimality, instead of on the basics of how to do the task in the first place. We also show that using these improved human models often leads to better human-AI collaboration performance compared to using models based on real human data alone.
1 Introduction
Human models need real data to capture the nuances of human behavior, but limited data makes learning them efficiently difficult. The paper proposes initializing those models from near-optimal behavior and reports gains in data efficiency, generalization, and downstream collaboration.
1 Introduction
- Real-human-data-only models can underperform humans and fail to generalize to unseen or uncommon situations, creating risks when best-response RL exploits their biases.Such RL may explore states unlike those induced by human-human collaboration.Source: §1 · p. 1
- The paper proposes an optimal behavior prior because people are expected to act closer to optimal than to random chance.The prior structures high-capacity models without reducing their ability to improve as data and coverage increase.Source: §1 · pp. 1–2
- Initializing human models from near-optimal behavior lets human data focus on subtle suboptimal behavior rather than learning basic task behavior.The approach uses optimal-agent parameters as a starting point, with self-play offered as a way to obtain near-optimal policies that can be easier to access than more human data.Source: §1 · p. 2
- Optimal Behavior Prior (OBP) models enable better collaboration than other tested conditions, while traditional behavior cloning models perform worse than self-play agents with simulated humans.The introduction also reports that OBP-based models are more realistic and competent, and generalize across environments without specialized hard-coded state-featurizations.Source: §1 · p. 2
- OBP substantially increases human-modeling data efficiency, suggesting applicability to settings with limited human data.The figure describes reusing a self-play-trained optimal-behavior parameterization as the human model initialization.Source: §1 · p. 2, Figure 1 · p. 2
2 Related work
The paper situates its approach among human modeling, human-AI collaboration, and generalization methods. It learns human behavior from data while using domain randomization to target new environments.
Human modeling
- Behavior cloning learns observation-to-action mappings from expert demonstrations, while search-based modeling can improve performance and use cloning regularization to retain human-like behavior.The authors view OBP as potentially complementary to search-based regularization and potentially competitive with search-based models in isolation.Source: §2 · p. 3
Human-AI collaboration
- Human-AI collaboration methods include handcrafted biases, equilibrium-based approaches, and learned human models used to train best responses; the latter avoids assuming test-time human behavior in advance.The paper's approach is closest to learning a human model and then training a best response to it.Source: §2 · p. 3
Generalization to new environments
- Domain randomization trains agents across a distribution of environments, and the paper uses it to support generalization to new test environments.Source: §2 · p. 3
3 Preliminaries
The preliminaries formalize multi-agent decision processes and explain how a fixed partner policy can be incorporated into an agent's environment. They connect learning a collaborative policy to computing a best response.
Multi-agent MDPs
- An n-player MDP is defined by states, each agent's actions, an initial-state distribution, transition dynamics, discount factor, reward, and horizon.Source: §3 · p. 3
Embedding agents in multi-agent MDPs
- When all policies except one are fixed, the other agents' choices can be incorporated into the transition dynamics of a single-agent MDP.Source: §3 · p. 3
Human-AI collaboration through deep RL
- With a human policy known, an agent can be trained against it; when it is unknown, human data can train a model that is embedded for deep-RL policy training.A sufficiently accurate human model is expected to yield an agent able to collaborate with humans at test time.Source: §3 · pp. 3–4
Best Response (BR)
- A best response maximizes expected collaboration reward against a fixed partner policy, and deep RL serves as an approximate best-response method in complex environments.Source: §3 · p. 4
4 Optimal Behavior Prior (OBP) for efficient human modelling
The paper uses an optimal behavior prior to initialize human models, replacing a random-behavior prior with near-optimal behavior. It obtains that initialization through self-play across environments and also considers freezing learned representations.
4 Optimal Behavior Prior (OBP) for efficient human modelling
- OBP replaces the random-initialization prior with a prior that humans try to act better than random, potentially reducing the data and optimization needed to reach a given human-model quality.In practice, the paper uses weights from a trained near-optimal agent to initialize the human model.Source: §4 · p. 4
- The method trains a near-optimal agent by self-play across the environment distribution and reuses its parameterization to warm-start human-model training.The setup covers human-human gameplay data for environments drawn from the distribution; a deterministic single-environment distribution recovers the traditional single-environment case.Source: §4 · p. 4
- OBP with freezing tests whether optimal-agent representations suffice by freezing the first k layers and fine-tuning the remaining layers.This imposes the assumption that the unfrozen parameters retain enough expressivity to represent human behavior.Source: §4 · p. 4
- The approach assumes access to environments of interest and human-human data from a limited number of diverse environments.It also assumes human behavior is closer to optimality than chance, and that optimal-behavior training and the human model can use the same model form.Source: §4 · pp. 4–5
- The trained general human model can be used to train collaborative AI agents across the environment distribution through domain randomization.Source: §4 · p. 5
5 Experimental setup
The experiments test whether an optimal-behavior prior improves human-model generalization to unseen environments and the task reward of human-aware agents collaborating with real people.
5 Experimental setup
- Two hypotheses ask whether OBP improves human-model generalization to unseen environments and whether human-aware agents trained with OBP-based models earn higher task reward with real humans there.Source: §5 · p. 5
5.1 Multi-environment human models
- Human-model comparisons include behavior cloning from human data, multi-environment self-play, OBP-initialized cloning, and OBP cloning with all but the last layer frozen.Source: §5.1 · p. 5
5.2 Best Response (BR) agents
- Collaborative agents comprise the self-play baseline and approximate deep-RL best responses trained to complement each human-data-based model across environment distribution E.Source: §5.2 · p. 6
5.3 Simulated user study
- The simulated study uses Overcooked kitchens, a proxy human, and held-out layouts spanning difficulty to evaluate models and agents while isolating the prior from real-user adaptation.Source: §5.3 · p. 6, Figure 3 · p. 7
- Simulated-human data comes from 40 layouts, is split equally into training and test sets, and trains full-state models that output probabilities over six actions.Behavior cloning early-stops when test cross-entropy begins to increase; collaborative best responses use PPO, with rollouts sampled across environments for multi-layout agents.Source: §5.3 · p. 7
6 Experimental results
In Overcooked-AI, OBP improves human models’ reward-based realism while validation loss stays similar; OBP-based best responses usually yield the highest collaboration rewards with proxy humans.
6.1 Human modeling results
- SPE is superhuman rather than human-like, while BCE without OBP earns zero reward on every evaluation layout.SPE never waits, unlike real humans and the ground-truth proxy, which wait around 55% of timesteps.Source: §6.1 · p. 7, Figure 4 · p. 8
- OBP improves reward realism but not validation loss: BC variants have almost identical cross-entropy, although OBP models complete tasks.The authors report that the validation-loss results are inconclusive overall.Source: §6.1 · pp. 7–8
- Freezing OBP representations somewhat raises rewards, especially on harder layouts, and slightly lowers validation loss across layouts.The authors attribute the modest data-efficiency gain to additional regularization and shared optimal-agent and human representations.Source: §6.1 · p. 8
6.2 Collaborative AI results
- The simulated user study mostly supports H2, suggesting the collaboration results may also hold with real humans.The paper notes that real-human studies are still needed to verify this.Source: §6.2 · p. 8
- OBP-based best responses achieve the highest human-collaboration rewards on almost all layouts, sometimes by slim margins.The figure compares test collaboration rewards across multi-layout and single-layout conditions.Source: §6.2 · p. 8, Figure 5 · p. 9
- Multi-layout best responses to OBP models can match single-layout counterparts and sometimes do significantly better on easier layouts.The authors suggest robustness and improved representations may help multi-layout training reach better deep-RL optima.Source: §6.2 · p. 9
7 Discussion
The discussion summarizes how an optimal behavior prior supports multi-environment human modeling and downstream collaboration, while noting evaluation and validation limits.
Metrics for evaluating human models
- Reward similarity may be a better human-model metric than prediction loss, but memorizing training demonstrations can produce the same reward as those demonstrations.The authors caution that reward matching is not a bulletproof measure of model quality.Source: §7 · p. 9
Limitations and future work
- The results use simulated proxy humans, so further work is needed to verify whether they hold with real humans; the authors also identify alternative priors as future work.They mention max-entropy agents and speculate about combining behavior cloning with RL to warm-start OBP training.Source: §7 · p. 9
Summary
- OBP makes high-quality multi-environment human modeling possible where straightforward modeling struggles, and the approach performs better on collaborative reward than modeling humans as optimal.The authors summarize this as leveraging the expectation that humans behave better than random.Source: §7 · p. 9
A Additional information about multi-Layout Overcooked-AI
The appendix describes the multi-layout environment, shared agent architecture, and simulated humans used in the experiments.
A Additional information about multi-Layout Overcooked-AI
- The multi-layout environment samples roughly 10^13 7 × 5 layouts with about 75% empty space, and roughly 40% of counters contain an interactive object.Empty squares are cook-occupiable; counters can hold dispensers, a pot, or a serving station.Source: A Additional information about… · p. 14, Figure 6 · p. 14
B.1 Human models and best response agents
- Human models and best-response agents share a convolutional and fully connected network that maps a 26 × 7 × 5 state encoding to probabilities over six legal actions.The authors omit recurrence over state history to reduce training computation.Source: §B.1 · p. 14
B.2 Simulated humans
- Simulated humans greedily select a high-level task, re-plan an action sequence, and execute its first action, with waiting and two levels of Boltzmann irrationality as tunable parameters.The two noise parameters affect low-level action selection and high-level goal selection.Source: §B.2 · p. 14
- The study tests waiting-matching H0, action-matching H1, and reward-matching H2; preliminary experiments found the approach performed similarly against baselines across all three.The main text uses H0, which fits waiting probability while setting both noise parameters to zero.Source: §B.2 · p. 14
C.1 Simulated humans
The appendix reports how simulated-human variants were fitted and compares their prediction loss and reward matching.
C.1 Simulated humans
- The authors fit simulated-human parameters from human data using cross-entropy methods on data collected in prior work [21].The fitted parameters are reported in Appendix Table 1.Source: §C.1 · p. 15
- Preliminary experiments found the method performed similarly relative to baselines with all three simulated-human variants, and the main text uses H0.The variants differ in which aspect of human data their parameters are fit to.Source: §C.1 · p. 15
- H0 achieves the lowest cross-entropy loss with decent reward matching, while H2 has the most human-like reward but much higher cross-entropy loss.H1 has similar cross-entropy loss to H0 but lower reward.Source: Table 1 · p. 15
C.2 Human models and best response agents
The appendix reports self-play and best-response training conditions, the simulated-data protocol, and behavior-cloning training conditions.
C.2 Human models and best response agents
- Multi-layout self-play trains for 3,000 PPO epochs, while single-layout self-play uses 500 epochs; both use batches of 200,000.Multi-layout batches contain 500 episodes on 250 sampled layouts.Source: §C.2 · p. 15
- The entropy schedule makes the biggest difference between multi-layout and single-layout self-play conditions, while SPE matches SP1 reward on all evaluation layouts.Other PPO hyperparameters are kept the same.Source: Table 2 · p. 15
- The simulated study uses 20 pairs of participants, each collaborating on seven layouts: five test layouts and two sampled from the environment distribution.Each pair supplies an episode on each layout.Source: §C.2 · p. 15
- The sampled-layout data aggregates 40 layouts, with the probability of resampling the same layout described as negligible.The resulting dataset is the multi-layout human dataset introduced in the main text.Source: §C.2 · p. 15
- Multi-layout behavior cloning uses 32,000 human-human timesteps, while single-layout counterparts use 16,000; OBP allows lower training loss.The appendix notes that single-layout variants may achieve lower loss through memorization with an over-parameterized network.Source: §C.2 · p. 15, Table 3 · p. 16
- Best-response rollouts use a fixed behavior-cloning partner on 500 freshly sampled layouts per iteration for 500 iterations, and all best-response agents receive OBP and equal training access.Each rollout episode has length 400.Source: §C.2 · p. 16, Table 4 · p. 16
D Additional analysis for human modeling conditions
The analysis separates wait from non-wait actions to investigate why OBP methods matched non-OBP methods on validation loss. OBP models achieve more human-like loss on wait actions, while BCE overpredicts waiting.
D Additional analysis for human modeling conditions
- Roughly 40%-65% of actions in the human data used for simulated-human parameter fitting were waits, likely because the rapid game clock made constructive actions difficult at every timestep.Source: D Additional analysis for human… · p. 16
- OBP methods performed as well as non-OBP methods on validation loss, prompting a breakdown of training datapoints by whether the ground-truth action was wait.Source: D Additional analysis for human… · p. 16
- OBP and OBP+f achieve more human-like loss on wait actions, while OBP helps prevent the wait-probability overprediction seen with BCE.BCE assigns high probability to wait actions; the authors suggest that, without a prior, it favors this easiest route to lowering loss.Source: D Additional analysis for human… · pp. 16–17, Figure 7 · p. 17