PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent cooperation through in-context co-player inference

Marissa A. Weis, Maciej Wołczyk, Rajai Nasser, Rif A. Saurous, Blaise Agüera y Arcas, João Sacramento, Alexander Meulemans

arXiv:2602.16301v1cs.AI

TL;DR

Decentralized MARL often struggles to select cooperative equilibria while co-players continually adapt. This paper trains sequence-model agents against diverse co-players, finding that in-context best responses create extortion-based mutual shaping that produces cooperation without hardcoded opponent-learning assumptions or explicit timescale separation.

  • Problem

    Decentralized MARL must learn cooperation despite equilibrium-selection difficulties and non-stationarity from simultaneously adapting co-players.

  • Method

    The paper trains sequence-model agents with Predictive Policy Improvement against diverse co-players, using predictive sequence modeling to learn in-context inference and adaptation.

  • Results

    Diverse co-player training induces in-context best-response strategies whose extortability and mutual shaping drive agents toward cooperative behavior.

  • Takeaways & Limitations

    Standard decentralized reinforcement learning with sequence models and co-player diversity provides a path toward cooperative multi-agent systems without explicit meta-gradients or rigid timescale separation.

  • Takeaways & Limitations

    The Iterated Prisoner’s Dilemma uses a fixed horizon of T=100 steps as a tractable approximation to infinite-horizon behavior.

Abstract

from arXiv · show

Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that account for and shape the learning dynamics of their co-players. However, existing approaches typically rely on hardcoded, often inconsistent, assumptions about co-player learning rules or enforce a strict separation between "naive learners" updating on fast timescales and "meta-learners" observing these updates. Here, we demonstrate that the in-context learning capabilities of sequence models allow for co-player learning awareness without requiring hardcoded assumptions or explicit timescale separation. We show that training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies, effectively functioning as learning algorithms on the fast intra-episode timescale. We find that the cooperative mechanism identified in prior work-where vulnerability to extortion drives mutual shaping-emerges naturally in this setting: in-context adaptation renders agents vulnerable to extortion, and the resulting mutual pressure to shape the opponent's in-context learning dynamics resolves into the learning of cooperative behavior. Our results suggest that standard decentralized reinforcement learning on sequence models combined with co-player diversity provides a scalable path to learning cooperative behaviors.

1. Introduction

The paper addresses why decentralized learning agents often fail to cooperate and proposes that diverse co-player training can induce cooperation through in-context learning and mutual extortion.

  • Self-interested agents must cooperate in mixed-motive interactions, but decentralized MARL faces equilibrium selection and non-stationarity because co-players learn simultaneously.
  • Existing co-player learning-aware methods model opponent updates or separate fast naive learners from slower meta-learners, often relying on explicit learning assumptions.
  • The paper hypothesizes that diverse co-player training naturally yields in-context best-response policies that functionally replace naive-learner parameter updates.
  • Extortion pressures pull agents away from pure defection, while mutual extortion gradients drive them toward cooperation.
  • The proposed decentralized sequence-model setup targets robust Iterated Prisoner’s Dilemma cooperation without explicit meta-versus-inner trajectories or assumptions about opponent learning rules.
  • The method combines self-supervised predictive sequence modeling with reinforcement learning and includes a theoretical characterization relating its training equilibrium to Nash and subjective embedded equilibria.

2. Problem setup and methods

The paper formulates cooperation in partially observable multi-agent games and trains sequence-model agents with Predictive Policy Improvement against mixed populations of learning and tabular co-players.

  • The interaction is formalized as a partially observable stochastic game in which each agent receives local observations and acts from interaction history.
  • The Iterated Prisoner’s Dilemma provides the cooperation test: mutual defection is the single-shot Nash equilibrium, while mutual cooperation yields higher returns.
  • The experiments approximate the infinite horizon with a fixed horizon of T=100 steps.
  • Mixed-pool training combines sequence-model Learning Agents with static Tabular Agents whose 5-dimensional parameters specify cooperation probabilities across initial and previous-action states.
  • Each learning agent plays 50% of episodes against another learning agent and 50% against a uniformly sampled tabular agent, without receiving opponent identifiers.
  • Predictive Policy Improvement uses a sequence model to predict joint actions, observations, and rewards as both a world model and policy prior.
  • PPI iteratively gathers trajectories with an improved policy and retrains the sequence model on the new data, while pretraining begins on interactions between randomly sampled tabular agents.

3. Results

Mixed-pool training produces cooperation by inducing in-context best responses, making agents vulnerable to extortion, and allowing mutual shaping to move policies toward cooperation.

  • Mechanism overview: Diverse co-player training is intended to produce both opponent-policy inference and within-episode best-response adaptation.
  • Mixed training: PPI and A2C agents trained in the mixed pool converge to cooperation in the Iterated Prisoner’s Dilemma.
  • Mixed training: Providing explicit opponent identification or removing mixed-pool training causes agents to collapse to mutual defection.
  • Step 1: In-context best response: PPI agents trained against tabular opponents rapidly adapt to the specific opponent’s best response within an episode.
  • Step 2: Extortion: A new PPI agent trained against a frozen in-context learner learns extortion by exploiting that learner’s tendency to adapt.
  • Step 3: Mutual extortion: When two extortion policies interact, mutual shaping makes their in-context behavior more cooperative, and subsequent parameter updates reinforce that direction.
  • Step 4: Synthesis: Mixed-pool training combines adaptation to tabular opponents with vulnerability to mutual extortion, ultimately driving learning agents toward cooperation.

4. Conclusion

The paper demonstrates that diverse co-player training can induce in-context best-response strategies and cooperative behavior without explicit co-player learning-awareness machinery or rigid timescale separation.

  • Diverse co-player training induces in-context best-response strategies that can support cooperation without explicit meta-gradients or rigid timescale separation.The approach uses sequence-model agents trained on interaction histories from mixed co-player populations.
  • The interaction is modeled as a partially observable stochastic game in which agents act from local observations and histories.The formalization includes states, joint actions, transitions, rewards, observations, discounting, and a finite horizon.
  • In Iterated Prisoner’s Dilemma, agents choose cooperation or defection over 100 rounds while sequence-model agents use full interaction history.Tabular agents use only the latest observation, whereas PPI and A2C agents condition on the full history.
  • PPI combines a learned sequence model with planning-based policy improvement, using predicted actions, observations, and rewards as a world model and policy prior.The method iteratively collects trajectories, adds them to the training dataset, and updates the sequence model.
  • The experiments include mixed-population training, opponent-identity conditioning, removal of tabular opponents, and sequence-model pretraining on 200 000 random-agent trajectories.The opponent-identity ablation prepends a conditioning vector, while the no-tabular-opponents ablation changes the training population and PPI pretraining distribution.
  • A2C uses a GRU-based sequence model with a value head and updates its policy using bootstrapped temporal-difference advantages.Training follows an A2C protocol with value estimation, generalized advantage estimation, advantage normalization, and reward scaling.

B.1. In-episode trajectories for mixed pool training

Early mixed-pool training shows within-episode opponent inference and an initial movement toward cooperation against other learning agents.

  • Figure 3’s early-training trajectories show PPI and A2C inferring opponents in context and beginning to move toward cooperation against learning agents.The trajectories correspond to PPI phase 8 and A2C training iteration 70k.

B.2. Additional results on A2C

Additional A2C experiments reproduce the best-response, exploitation, and cooperation dynamics observed for PPI, while also revealing instability across random seeds.

  • A2C learns best responses to varied tabular agents and obtains approximately 1.25 reward against the Fixed-ICL baseline, versus approximately 0.9 for PPI.The comparison is reported for the newly trained A2C agent playing against the Fixed-ICL baseline.
  • A2C extortion agents initially move toward cooperation, but training instability can return them to defection depending on the random seed.The exploitation dynamics are described as irregular, suggesting a complex adversarial strategy against the A2C Fixed-ICL policy.

A. PPI - Episode early in training (Phase=8) B. A2C - Episode early in training (Iteration=70k)

Early mixed-training trajectories show extortion preceding cooperation, while opponent identification can produce high defection against non-tit-for-tat-like tabular policies.

  • A. PPI - Episode early in training (Phase=8) B. A2C - Episode early in training (Iteration=70k): Both A2C and PPI initially attempt to extort their counterparts, followed by increased cooperation within the episode.Error bars represent standard deviation across 10 random seeds.
  • A. PPI - Episode early in training (Phase=8) B. A2C - Episode early in training (Iteration=70k): Identifying an opponent as a non-tit-for-tat-like tabular policy leads to a high defection ratio.
  • A. PPI - Episode early in training (Phase=8) B. A2C - Episode early in training (Iteration=70k): The A2C results separate best-response emergence, exploitation of in-context learners, and later cooperation that can collapse to mutual defection.The A2C experiments use five random initializations, and the collapse depends on the random seed.

C. Derivation of Predictive Policy Improvement (PPI)

PPI replaces separate value functions and self-models with a single self-supervised sequence model that serves as a behavioral prior over interaction histories. Its surrogate KL-regularized objective is optimized through coordinate ascent and first-order policy improvement, with Q-values estimated by sequence-model rollouts.

  • PPI replaces MPO’s separate value function and self-model with one self-supervised sequence model that predicts actions, observations, and rewards.
  • The surrogate objective J combines expected return with a KL penalty between the policy π and sequence-model prior p_ϕ, forming a lower bound on V(π).The bound is tight when π = p_ϕ.
  • The model p_ϕ(a | x_≤t) acts as a behavioral prior or self-model conditioned on the full interaction history.
  • PPI alternates coordinate ascent over the non-parametric policy π and parametric model ϕ, using a first-order approximation around the current prior policy.The approximation supports a tractable policy-improvement step while preserving the derivative at the prior.
  • Unlike standard MPO, PPI estimates history-conditioned Q-values through Monte Carlo rollouts inside the sequence model, which predicts environment dynamics and co-player responses.

D. Theoretical Analysis of the Equilibrium Behavior of PPI Agents

PPI creates a performative loop in which predictive models induce policies, policies generate data, and the resulting data updates the models. The analysis defines predictive equilibria and proves existence of local and mixed forms under stated assumptions, while global pure equilibria are not guaranteed.

  • D.1. Formal Setup: PPI forms a performative loop: models induce policies, policies induce interaction-history data, and data updates the models.
  • D.2. Predictive Equilibria: Global pure-strategy equilibrium is not guaranteed because deep neural networks produce non-convex objectives and potentially discontinuous argmin sets through mode hopping.
  • D.2. Predictive Equilibria: The paper introduces Local Predictive Equilibrium and Mixed Predictive Equilibrium as relaxed solution concepts for gradient-based and randomized strategies.
  • D.1. Formal Setup: A predictive equilibrium is a stable configuration where each model optimally predicts the data generated by the policies derived from the joint models.
  • D.2.1. Local Predictive Equilibrium: Local Predictive Equilibrium corresponds to first-order stationarity under a fixed data-generating process and matches the convergence criterion of projected gradient descent.
  • D.2.1. Local Predictive Equilibrium: Under compact convex parameter spaces and continuous gradient mappings, at least one Local Predictive Equilibrium exists.

D.2.2. Mixed Predictive Equilibrium

The section establishes existence of mixed predictive equilibria by constructing a continuous improvement map over mixed strategies and applying Schauder’s fixed-point theorem. Under functional convexity of representable models, this existence result yields a pure global predictive equilibrium.

  • Definition and existence: A mixed predictive equilibrium is a tuple of parameter distributions over which no agent can achieve a lower local loss through predictive-model improvement.The proof represents each agent’s strategy as a probability distribution over its compact parameter space.
  • Definition and existence: The construction mixes each current strategy with an improvement measure weighted by strictly positive advantages.The resulting transition map remains within the mixed-strategy space.
  • Existence proof: Compactness, continuity, and bounded advantages make the joint transition map continuous on a compact convex strategy space, so Schauder’s theorem provides a fixed point.The fixed point is then shown to be a Mixed Predictive Equilibrium by contradiction: any strictly better distribution would generate positive improvement mass.
  • Pure equilibrium corollary: Under convexity of the space of representable predictive models, a Global Predictive Equilibrium in pure strategies always exists.The argument converts mixed predictive models into equivalent pure parameters, without requiring convexity in the parameters themselves.
  • Pure equilibrium corollary: Functional convexity is an idealization for finite-capacity networks, although increasing model capacity can make the pure-equilibrium result an increasingly accurate approximation.The scope boundary concerns convexity in functional space rather than parameter space.

D.3. Relationship to Nash Equilibria and Subjective Embedded Equilibria

The section relates predictive equilibria to standard game-theoretic concepts by contrasting fixed-environment Nash reasoning with agents that optimize using internal predictive world models. When those models are perfect on the induced data distribution, a predictive equilibrium corresponds to a subjective embedded equilibrium.

  • Relationship to Nash equilibria: Nash equilibrium assumes agents optimize against a fixed environment whose co-player policies are independent of the focal agent’s current action selection.Predictive agents instead optimize relative to internal models of future trajectory distributions.
  • Subjective embedded equilibria: A Subjective Embedded Equilibrium requires subjective optimality and on-path consistency, while imposing no accuracy constraints on off-path counterfactuals.Each policy must be a strict best response to its internal world model, which must match true dynamics on histories generated by the joint policy.
  • Perfect predictive equilibrium: A Perfect Predictive Equilibrium implies a Subjective Embedded Equilibrium under the stated policy-improvement operator.The theorem establishes the correspondence when predictive world models are perfect.
  • Perfect predictive equilibrium: A Perfect Predictive Equilibrium is a configuration in which agents perfectly model the induced data distribution.This condition supplies the link between predictive equilibrium and subjective embedded equilibrium.
  • Proof of correspondence: At equilibrium, zero KL divergence means the sequence model matches the true data distribution on the equilibrium path, while off-path conditionals remain unconstrained.A valid off-path completion can then make deviations no better than the on-path policy, satisfying subjective optimality.
Loading 2602.16301v1…