Source-linked AI summary
Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel
TL;DR
The paper asks how adaptive systems can generate novel challenges when complex innovations are difficult to discover and research environments are laborious to create. It develops autocurricula as sequences of challenges arising from interacting adaptive units, and argues that these dynamics can support innovation while sometimes cycling without accumulating it.
Problem
The paper addresses how qualitatively novel, complex behavior can arise when spontaneous discovery is unlikely, especially when multiple agents must jointly explore solutions.
Source: Innovations Arise on All Levels of… · p. 4Method
The paper frames interacting adaptive units’ changing strategies and implementation policies as sources of naturally generated sequences of learning challenges called autocurricula.
Source: The Problem Problem · p. 2, Exploration by Exploitation · p. 4, Exogenous Challenges Generate… · p. 5Results
The paper argues that interactions among adaptive units across biological levels generate challenge sequences that perturb adaptive landscapes and can support innovation through exploration by exploitation.
Source: Accumulating Autocurricula and Human… · p. 9Takeaways & Limitations
Autocurricula offer a lens for examining how social interaction generates challenges through competition and cooperation and under what conditions innovation may accumulate.
Source: The Problem Problem · p. 2, Accumulating Autocurricula and Human… · p. 9Autocurricula may cycle through repeated challenges without accumulating innovations, and forgetting past policies can make self-play double back onto itself.
Source: Exogenous Challenges Generate… · p. 6
Abstract
from arXiv · showhide
Evolution has produced a multi-scale mosaic of interacting adaptive units. Innovations arise when perturbations push parts of the system away from stable equilibria into new regimes where previously well-adapted solutions no longer work. Here we explore the hypothesis that multi-agent systems sometimes display intrinsic dynamics arising from competition and cooperation that provide a naturally emergent curriculum, which we term an autocurriculum. The solution of one social task often begets new social tasks, continually generating novel challenges, and thereby promoting innovation. Under certain conditions these challenges may become increasingly complex over time, demanding that agents accumulate ever more innovations.
The Problem Problem
The section frames the “problem problem” as the labor-intensive creation of ever more environments for intelligence research. It proposes that social interaction can generate challenges and learning tasks endogenously, offering a multi-agent perspective on that bottleneck.
The Problem Problem
- In the Go illustration, adding an opponent turns a simple territory task into an emergently complex game that can occupy millions of minds.With only black stones, placing them along the grid’s edge solves the task; white stones introduce barriers.Source: The Problem Problem · p. 1
- Agents in social games must learn both which strategy to choose and how to implement it, while others’ strategic choices change the challenges they face.The paper terms the resulting sequence of challenges an autocurriculum.Source: The Problem Problem · pp. 1–2
- The “problem problem” is the bottleneck of generating enough diverse, complex environments for intelligence research, a laborious process that cannot scale indefinitely.The paper compares the work to creating videogames.Source: The Problem Problem · pp. 1–2
- Social interaction can continually reshape other agents’ adaptive environments, suggesting a way life generates new challenges without relying only on researcher-built environments.The paper points to species innovations and human cultural rules as examples of changes that demand adaptation.Source: The Problem Problem · p. 2
Innovations Arise on All Levels of Life’s Hierarchies
The section treats adaptive units as nested across biological and social scales, with innovations arising at any level. It highlights the difficulty of generating complex novel behavior and uses “adaptive unit” to connect evolutionary and learning processes.
Innovations Arise on All Levels of Life’s Hierarchies
- Adaptive units are nested across levels, from interacting subunits to collectives such as human communities, which the authors treat as intelligent units in their own right.The examples include cells composed of organelles and communities composed of individuals.Source: Innovations Arise on All Levels of… · p. 3
- Innovations expand an adaptive unit’s behavioral repertoire with robust, repeatable problem-solving abilities and can occur at any hierarchical level.Examples range from eukaryotic cells and multicellular organisms to agriculture, industry, and institutions.Source: Innovations Arise on All Levels of… · p. 3
- “Adaptive unit” unifies systems adapting through genetic evolution, reinforcement learning, or cultural evolution, despite their different timescales.Interactions between fast and slow processes can also matter, as in gene-culture co-evolution.Source: Innovations Arise on All Levels of… · p. 4
- Generating useful complex behavior spontaneously is extraordinarily unlikely, and becomes exponentially harder when multiple agents must jointly explore solutions.The section poses this as a challenge for explaining innovations such as walking on the moon, creating the internet, and curing smallpox.Source: Innovations Arise on All Levels of… · p. 4
Exploration by Exploitation
The section presents challenges as changes in adaptive landscapes that can prompt learning and exploration. It proposes that social interaction can generate a sequence of such challenges naturally, with competition and cooperation producing contrasting, complementary dynamics.
Exploration by Exploitation
- A challenge is a change in environment dynamics that alters a policy’s value relative to other policies, positively or negatively.Examples include prey countermeasures weakening predation strategies and an uninhabited island opening new foraging opportunities.Source: Exploration by Exploitation · p. 4
- Challenges can prompt exploration as adaptive units follow shifts in their experienced policy values, without deliberately departing from exploitation.The paper calls this “exploration by exploitation,” contrasting it with a deliberate exploration–exploitation tradeoff.Source: Exploration by Exploitation · p. 4
- Autocurricula are sequences of challenges generated naturally by social interaction, rather than by environmental engineering.The challenges are endogenous or exogenous to the adaptive unit under consideration.Source: Exploration by Exploitation · p. 4
- Competition and cooperation yield contrasting but complementary dynamics underlying the ways social interaction can generate autocurricula.Source: Exploration by Exploitation · pp. 4–5
Exogenous Challenges Generate Autocurricula
In exogenous autocurricula, one agent’s strategic change alters another’s learning landscape, prompting responses that can create further challenges. These dynamics can generate innovation in open strategy spaces, but may instead cycle or regress when strategies are forgotten.
Exogenous Challenges Generate Autocurricula
- A player’s strategic shift changes an opponent’s best response, and implementing each new strategy creates further challenges in a potentially escalating arms race.Players must learn both which strategy to choose and how to implement it through sequences of elementary actions.Source: Exogenous Challenges Generate… · p. 5
- Empirical game-theoretic methods infer strategic properties from outcomes, and Policy Space Response Oracles incrementally add best responses to a normal-form game table.These methods have been used to analyze strategic intransitivities and learned policies in social dilemmas.Source: Exogenous Challenges Generate… · p. 5
- Intransitive dynamics such as rock-paper-scissors can oscillate, repeating underlying challenges without producing new innovations.Implementations may differ across repetitions even as the same challenges recur.Source: Exogenous Challenges Generate… · pp. 5–6
- Forgetting past policies can make newer generations vulnerable to older ones, causing an autocurriculum to revisit past challenges.The paper notes that self-play algorithms often train against diverse sets of older policies as well as the strongest current policy.Source: Exogenous Challenges Generate… · p. 6
- In open games such as Go and Chess, self-play can generate novel innovations, while games with few strategies are expected to approach Nash equilibrium.AlphaGo and AlphaZero use Monte Carlo tree search to improve policies and learn value estimators from generated games.Source: Exogenous Challenges Generate… · p. 6
- Malthusian reinforcement learning extends self-play to variable player counts, where successful subpopulations grow and intensify competition for limited resources.This demographic expansion perturbs established solutions and can help escape poor local optima.Source: Exogenous Challenges Generate… · pp. 6–7
- Social competition has also been proposed as a driver of intelligence evolution, while collective-action challenges may have motivated further cognitive innovations.The cited accounts include competition to manipulate social partners and the challenges of organizing collective action.Source: Exogenous Challenges Generate… · p. 7
Endogenous Challenges Generate Autocurricula
Autocurricula can arise within adaptive units when internal social dilemmas prompt institutions that change incentives. Resolving one dilemma may create successor dilemmas, producing challenges that grow in scope and complexity, though proposed ways to evade this pattern depend on assumptions that may not hold in costly settings.
Endogenous Challenges Generate Autocurricula
- Autocurricula can operate across levels of organization: collectives face external competition and internal conflicts among their adaptive subunits.Examples include colonies competing with peer colonies and cooperation breaking down through renegade reproduction or selfish cellular behavior.Source: Endogenous Challenges Generate… · pp. 7, 9
- Common-pool resource dilemmas cannot generally be solved by unilateral restraint; resolution requires changing the behavior of a critical fraction of participants.Each person’s appropriation reduces what remains for others, while an individual’s restraint alone has too little effect to escape resource degradation.Source: Endogenous Challenges Generate… · p. 7
- Institutions can change participants’ incentives, for example by sanctioning over-exploitation so it is no longer the dominant strategy.The paper notes hints of emergent institutions in multi-agent reinforcement learning, including agents that sanction over-exploiters.Source: Endogenous Challenges Generate… · pp. 7–8
- Proposed ways to evade higher-order dilemmas rely on assumptions that may fail when monitoring or coordination is costly.Asymmetries between altruistic cooperation and punishment may apply when monitoring costs are small, but not in many real-world common-pool settings; reputation mechanisms can also create new attack vectors.Source: Endogenous Challenges Generate… · p. 8
- Resolving a social dilemma can create a second-order free-rider problem, and higher-order institutions may then produce successor dilemmas of increasing scope and complexity.The paper describes this as a social “no-free-lunch” property: resolving one dilemma can cause another to arise.Source: Endogenous Challenges Generate… · pp. 8–9
Accumulating Autocurricula and Human Uniqueness
The authors propose that autocurricula can drive innovation, but may cycle rather than accumulate; they argue humans’ unusually long cultural memory helps avoid such cycles and promote cumulative innovation.
Human uniqueness
- The authors argue that humans’ exceptionally long cultural memory can help avoid intransitive cycles, promoting cumulative accumulation of innovations.They note that autocurricula may otherwise cycle, repeatedly learning and unlearning information without innovations building on one another.Source: Accumulating Autocurricula and Human… · p. 9
Open questions
- The paper leaves open whether autocurricula generate sufficiently diverse challenges to resolve the problem problem.It also poses questions about community-level strategy and implementation, social dilemmas, and interactions across biological levels.Source: Accumulating Autocurricula and Human… · p. 9
Cumulative culture loop
- In the cumulative culture loop, population growth increases competition for resources, while survival-promoting innovations spread and can further increase population size.The authors describe high-fidelity cultural transmission and larger populations’ protection against cultural loss as supporting cumulative cultural evolution.Source: Accumulating Autocurricula and Human… · p. 10, Figure 2 · p. 10
Self-domestication loop
- In the self-domestication loop, coordinated punishment may reduce aggression, increasing tolerance and creating opportunities for better communication and language.Improved communication can in turn support coordinated punishment and more accurate reputation information, potentially allowing institutions to accumulate.Source: Accumulating Autocurricula and Human… · pp. 10–11, Figure 2 · p. 10
Research implications
- The authors conclude that intelligence research focused only on individual cognitive abilities is likely incomplete and should attend to multi-agent dynamics in evolution and learning.They identify multi-agent interactions as key drivers of sustained innovation and perhaps increases in intelligence during human evolution.Source: Accumulating Autocurricula and Human… · p. 11