PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Tool Use From Multi-Agent Autocurricula

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch

arXiv:1909.07528v2cs.LGcs.AIcs.MAstat.ML

TL;DR

Complex physical tasks are costly to supervise, and undirected exploration scales poorly as environments become more complex. The paper studies whether competitive hide-and-seek with standard reinforcement learning can induce an autocurriculum, then evaluates the resulting capabilities through transfer and targeted tests. It finds up to six strategy adaptations involving human-relevant tool use and proposes transfer-based evaluation, while noting that the environment’s strategy space is bounded and learned representations can be difficult to fine-tune.

  • Problem

    Direct supervision is costly for complex physical tasks, while qualitative evaluation becomes intractable as multi-agent autocurricula grow more open-ended.

  • Method

    The paper trains competing teams in a physics-based hide-and-seek environment using only visibility-based rewards and standard reinforcement learning, then evaluates capabilities through transfer and targeted intelligence tests.

  • Results

    Multi-agent competition induces as many as six distinct strategy adaptations, including collaborative tool use, shelter construction, and ramp-based counter-strategies.

  • Takeaways & Limitations

    Multi-agent autocurricula can produce physically grounded, human-relevant behavior and provide a framework for evaluating targeted capabilities in open-ended environments.

  • Takeaways & Limitations

    The hide-and-seek environment has a bounded strategy space and learned skill representations can be entangled and difficult to fine-tune.

Abstract

from arXiv · show

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

1 INTRODUCTION

Directly supervising complex physical tasks is costly and bounds learned skills by task descriptions, while undirected exploration scales poorly with environment complexity. This work uses competitive hide-and-seek to induce emergent, human-relevant tool use and evaluates capabilities through transfer.

  • Direct reward specification and demonstrations for complex physical tasks are time-consuming and costly.
  • Single-agent reinforcement learning skills are bounded by task descriptions, leaving little room to improve after task mastery.
  • Intrinsic-motivation exploration methods scale poorly with increasing environment complexity, motivating competition-based autocurricula.
  • Hide-and-seek competition induces collaborative tool use, shelter construction, ramp-based counter-strategies, and as many as six strategy adaptations.
  • The paper proposes transfer and fine-tuning as quantitative evaluations and reports targeted tests comparing hide-and-seek agents with intrinsic motivation and scratch baselines.
  • The main contributions include evidence for emergent autocurricula, human-relevant tool use, targeted intelligence tests, and open-sourced environments and code.

2 RELATED WORK

Related work spans self-play, open-ended evolution, multi-agent reinforcement learning, intrinsic motivation, and tool-use learning. The paper builds on these traditions while targeting physically grounded emergent behavior.

  • Self-play research includes genetic algorithms, coevolved simulated creatures, open-ended evolution, and deep reinforcement learning in competitive games.
  • Multi-agent reinforcement learning has produced emergent behaviors including ball passing, interception, and communication.
  • Intrinsic-motivation methods reward exploration through state visitation counts, density estimation, or transition-prediction errors.
  • Tool-use learning is difficult when rewards are unaligned, with prior work studying robotic tools and construction tasks.

3 HIDE AND SEEK

Agents compete in a physics-based, two-team hide-and-seek environment where hiders avoid visibility and seekers maintain it. Objects, walls, rooms, and a preparation phase create opportunities for environmental interaction.

  • Hiders avoid line of sight while seekers seek to maintain vision of hiders.
  • Agents can grab and lock scattered objects, while randomly generated rooms and walls require navigation.
  • During the preparation phase, seekers are immobilized so hiders can flee or modify the environment.
  • The only supervision is the hide-and-seek objective: team rewards are based on whether all hiders are hidden or any hider is seen.
  • Episodes last 240 timesteps, with the first 40% used as a zero-reward preparation phase and a -10 penalty for leaving the 18-meter square.

4 POLICY OPTIMIZATION

The agents use self-play and shared policy parameters in a distributed reinforcement-learning setup. Entity-centric attention enables processing variable numbers of agents and objects while environment statistics track emergent phases.

  • Self-play provides a natural curriculum by matching agents against opponents of an appropriate level.
  • Policies use separate actor and critic networks optimized with PPO and GAE in a large-scale distributed reinforcement-learning framework.
  • Sharing policy parameters preserves all six emergence stages, while separate parameters achieve them with reduced sample efficiency.
  • Entity embeddings with shared weights are processed by masked residual self-attention and pooled into a fixed-size representation that supports variable entity counts.
  • Training progress is tracked through object and ramp movement and locking statistics across game and preparation phases.

5 AUTO-CURRICULA AND EMERGENT BEHAVIOR

Multi-agent competition produces up to six successive strategies, including tool use, coordination, and defensive adaptations. Progression depends on scale and environment randomization, while behavioral statistics track corresponding changes in object interaction.

  • Emergent strategy: Up to six distinct strategies emerge, each creating new pressure for agents to progress to the next stage.These strategies arise without direct incentives to interact with objects or explore.
  • Tool use: After about 25 million episodes, hiders construct shelters by moving boxes together or against walls and locking them in place.After another 75 million episodes, seekers use ramps to jump over obstacles and enter the shelters.
  • Strategic adaptation: After 380 million total episodes, seekers learn to surf a box from a ramp to reach shelters, prompting hiders to lock all boxes before construction.This produced two additional qualitatively new strategies beyond ramp defense.
  • Coordination: Agents must coordinate within teams, with hiders often dividing labor by separately bringing boxes to the construction area.Team-based rewards require collaboration for success.
  • Caveat: New skills do not necessarily increase team reward; small hider teams may lack enough time to lock all boxes during surf defense.This limits reward-based interpretation of skill discovery.
  • Scale and robustness: Larger batch sizes accelerate training while only marginally affecting sample efficiency down to 32,000, whereas 16,000 and 8,000 never converged.The default batch size of 64,000 reached ramp defense after 132.3 million episodes over 34 hours.

6 EVALUATION

The evaluation compares hide-and-seek pretraining with intrinsic-motivation and scratch baselines using behavioral statistics, transfer tasks, and targeted intelligence tests. Results are mixed across tasks, while the authors identify entangled skill representations as a limitation of fine-tuning.

  • Comparison to intrinsic motivation: Behavioral statistics compare net box movement and maximum agent movement across count-based state representations and RND over three seeds.The count-based variants use progressively richer state representations, including box location alone, box location with rotation and velocity, and the full observation space.
  • Comparison to intrinsic motivation: Count-based exploration produces the largest agent and box movement with only 2-D box locations, while higher-dimensional representations substantially reduce movement.RND performs slightly better than count-based exploration in the full-state setting.
  • Transfer and fine-tuning: The evaluation uses transfer to five domain-specific intelligence tests, comparing hide-and-seek pretraining with count-based intrinsic motivation and scratch training.The tests reuse the hide-and-seek action space, observation space, and object types.
  • Transfer and fine-tuning: Across three of five tasks, hide-and-seek-pretrained agents learn faster and achieve higher final reward than both baselines.Figure 6 reports mean normalized performance and 90% confidence intervals across three seeds, except Blueprint Construction, which uses six seeds.
  • Transfer and fine-tuning: Hide-and-seek pretraining performs slightly better than both baselines on Lock and Return, Sequential Lock, and Construction from Blueprint, but not uniformly across the suite.It performs slightly worse than count-based training on Object Counting and matches scratch training in final Shelter Construction reward while learning slightly slower.
  • Transfer and fine-tuning: The authors attribute mixed transfer results to skill representations that are entangled and difficult to fine-tune.They distinguish reuse of learned feature representations from the more difficult reuse of learned skills.

7 DISCUSSION AND FUTURE WORK

Multi-agent self-play with simple rules induced up to six rounds of strategy and counter-strategy, including physically grounded, human-relevant behavior. The authors frame these results as a proof of concept while identifying substantial sample-complexity and environment-design limitations.

  • Up to six distinct strategy and counter-strategy rounds emerged from simple game rules, competition, and standard reinforcement learning at scale.
  • Transfer and targeted intelligence tests were proposed to evaluate learning progress in open-ended environments.
  • The results are a proof of concept for physically grounded and human-relevant behavior, but the environment’s strategy space is inherently bounded.
  • Progress through the six stages requires an enormous amount of experience, likely because the reward functions are not directly aligned with the resulting behavior.
  • Agents exploit small environment-design and physics inaccuracies, including box surfing, shielding while escaping, and simulation artifacts.

A.2 DEPENDENCE OF SKILL EMERGENCE ON RANDOMNESS IN THE TRAINING DISTRIBUTION OF ENVIRONMENTS

Environmental randomization supports richer skill emergence in the hide-and-seek autocurriculum. Reducing randomization produces fewer, sometimes less sophisticated strategies, while a simpler quadrant environment reproduces the main progression.

  • Reducing environmental randomization yields fewer emergent skill stages and sometimes less sophisticated strategies, such as using boxes as movable shields.
  • Table A.1 reports the number of emergence stages across environmental-randomization levels with batch size held constant.
  • Trajectory traces illustrate chronological snapshots of strategies including running and chasing, fort building, ramp use, and ramp defense.
  • The quadrant environment reproduces progression from running and chasing to fort building, ramp use, and ramp defense.
  • In the quadrant environment, hiders block doors with boxes while seekers drag ramps to exterior walls and use them to enter the shelter.

A.4 FURTHER ABLATIONS

Ablations show that access to the full unobscured state in the value function is critical for progressing through the autocurriculum at the tested scale. Pooling and model-size comparisons quantify progress toward ramp defense.

  • Figure A.3 compares episodes and wall-clock time required to reach stage 4 across batch and model sizes, including value-function and pooling architectures.
  • With the same compute budget, masked value functions never progressed past stage 3, whereas omniscient value functions enabled progression toward stage 4.

A.5 EVALUATING AGENTS AT DIFFERENT PHASES OF EMERGENCE

Fine-tuning policies from different emergence phases reveals that some capabilities improve progressively, others are phase-specific, and manipulation performance is largely unrelated to phase. Food-reward variants produce related skill progressions and increasingly goal-directed construction behaviors under specific environment conditions.

  • Phase-wise transfer: Policies from later phases improve navigation and memory capabilities, with Lock and Return performance increasing monotonically across emergence phases.The phase 6 policy performs 20% better than the phase 1 policy on Lock and Return.
  • Phase-wise transfer: Object Counting is strongest in phase 1 but declines in later phases, making the capability transient rather than progressively accumulated.Phase 1 substantially outperforms all baselines and other phases on this task.
  • Phase-wise transfer: Manipulation-task performance is relatively uncorrelated with emergence phase, and the phase 1 policy performs comparably to policies from later phases.Phase 1 precedes the emergence of tool use, yet transfers similarly on manipulation tasks.
  • Food-reward variants: The fixed-food variant produces four progression levels: running and chasing, fort building, ramp use, and ramp defense.Food consumption first decreases as seekers improve, then rises when hiders build forts and again when they defend against ramps.
  • Food-reward variants: Dynamic-food protection requires forts covering all possible food locations, emerging after around 4.5 × 10^10 samples and sooner when the spawning region is smaller.With a region one-sixth of the game area, the behavior emerges after 1.5×10^10 samples; with a one-quarter region, hiders ignore food and build only a small fort.
  • Food-reward variants: In the food protection game, hiders discover cooperative seeker-trapping: they push seekers into a corner and use elongated boxes to build a fort that prevents escape.The strategy emerges after around 7×10^10 samples and includes two hiders pushing while the third follows the preparation behavior.
  • Food-reward variants: Removing the boundary penalty makes corner-blocking behavior emerge twice faster, at 3.5 × 10^10 samples, while other penalty settings yield similar fort-building behavior.With no penalty, one hider remains inside the fort with the seekers; applying the penalty throughout keeps hiders in the center while grabbing objects.
  • Generalization: Trained policies zero-shot generalize to environments with slightly more hiders, ramps, or boxes, although increasing hider count lowers hider reward.The hider reward serves as a measure of generalization because the modifications make safe-shelter construction more difficult.

B.7 POLICY ARCHITECTURE DETAILS

The policy representation embeds objects relative to the agent and processes their interactions with residual self-attention. Evaluation tasks align their observations with the hide-and-seek policy by supplying zeroed placeholder observations.

  • Object representation: Each object is concatenated with the agent’s self representation and embedded using dense layers whose parameters are shared across objects of the same type.For example, all boxes use shared embedding weights.
  • Attention processing: The embedded entities are processed by a residual self-attention block without position embeddings.The architecture is described as similar to the self-attention mechanism of Vaswani et al. (2017).
  • Evaluation alignment: Single-agent evaluation tasks receive fake hider and preparation-phase observations set to zero to align their inputs with the hide-and-seek environment.This observation alignment preserves the expected input structure while removing those multi-agent signals.

C.1 COGNITION AND MEMORY TASK

The paper defines transfer tasks spanning navigation, sequential locking, construction, and shelter-building, with randomized environments, shaped rewards, and task-specific horizons. These tasks evaluate agents’ cognition, memory, object manipulation, and coordinated construction abilities.

  • Cognition and memory tasks: The transfer suite includes lock-and-return and sequential-lock tasks requiring navigation, box locking, ordered actions, and ramp use.Lock-and-return uses six connected rooms and one box; sequential locking uses four boxes placed in an unobserved order across three rooms.
  • Construction tasks: Construction from Blueprint requires agents to place boxes at randomly sampled construction sites in an empty room.Agents observe each site’s position and four corners, and the policy embeds these observations before self-attention.
  • Construction tasks: Blueprint construction rewards proximity between construction-site corners and box corners using a smooth minimum over distances.The episode ends when every site and site corner is within dmin of a corresponding box feature; the number of sites k is sampled from 1 to 4 and n = 8 boxes are used.
  • Construction tasks: Shelter construction asks agents to build a shelter around a randomly placed cylinder using three elongated boxes and five square boxes.The cylinder location and diameter are randomized, and the cylinder must remain sufficiently far from room edges for completion to be physically possible.

D INTRINSIC MOTIVATION METHODS

The paper compares hide-and-seek policies with intrinsic-motivation baselines including count-based exploration and RND, using different state representations and shared architectural choices. These policies are evaluated through fine-tuning on transfer-task intelligence tests.

  • Experimental controls: Intrinsic-motivation agents are trained without ramps except for the Sequential Lock task, which is the only transfer task containing ramps.The hide-and-seek policy architecture and optimization hyperparameters are inherited for the compared policies.
  • Count-based exploration: The count-based baseline discretizes continuous state values into 30 bins and randomly projects them into 16-dimensional discrete embeddings for hashing.Entity embeddings are max-pooled by object type to produce an entity-invariant object representation.
  • Count-based exploration: The decentralized count-based method shares a random projection across rollout workers while maintaining separate state counts for each worker.Its intrinsic reward is computed from visitation counts within each rollout worker.
  • Random Network Distillation: RND uses a fixed target network and a predictor network whose prediction error supplies the intrinsic reward.Both networks omit the LSTM and output 64-dimensional vectors; the squared prediction difference is scaled by 1.0.
  • Intrinsic motivation methods: The intrinsic-motivation comparison includes RND and three count-based exploration variants with different state representations.The study evaluates pretrained policies on intelligence-test suites through fine-tuning.
Loading 1909.07528v2…