Source-linked AI summary
Modeling Others using Oneself in Multi-Agent Reinforcement Learning
Roberta Raileanu, Emily Denton, Arthur Szlam, Rob Fergus
TL;DR
Multi-agent reinforcement learning requires agents to infer others’ hidden goals from behavior in challenging, non-stationary settings. The paper introduces Self Other-Modeling, which uses an agent’s own policy to model others online; across cooperative and competitive tasks, explicit opponent modeling improves performance, reaching approximately 75–80% wins in Recipe versus 15–20% for several baselines.
Problem
Agents must infer other players’ hidden goals from behavior because rewards depend on both agents’ hidden states, but this is difficult in diverse, competing, and non-stationary multi-agent environments.
Method
Self Other-Modeling uses an agent’s own policy to model another player’s behavior, infer the player’s hidden goal, and update that estimate online for action selection.
Results
Explicitly modeling the other player improves performance in cooperative and competitive settings; in Recipe, SOM wins approximately 75–80% against NOM, IPP, and SPP, versus approximately 15–20% for those baselines.
Takeaways & Limitations
SOM agents can estimate hidden goals and converge to better policies, while the approach remains simple, flexible, and compatible with different reinforcement learning algorithms and network architectures.
Takeaways & Limitations
SOM requires longer training than other baselines because it back-propagates through the network at each step, and the work plans to address settings where other players differ from the self.
Abstract
from arXiv · showhide
We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility. The reward function depends on the hidden state (or goal) of both agents, so the agents must infer the other players' hidden goals from their observed behavior in order to solve the tasks. We propose a new approach for learning in these domains: Self Other-Modeling (SOM), in which an agent uses its own policy to predict the other agent's actions and update its belief of their hidden state in an online manner. We evaluate this approach on three different tasks and show that the agents are able to learn better policies using their estimate of the other players' hidden states, in both cooperative and adversarial settings.
1. Introduction
Multi-agent systems require agents to reason about others’ intentions and goals from behavior, but non-stationarity makes this difficult. The paper introduces explicit online goal estimation and reports better performance than treating other agents as part of the environment.
- Reasoning about other agents’ intentions matters because multi-agent systems can contain diverse and competing goals.
- The problem is challenging because multi-agent domains are inherently non-stationary.
- SOM estimates other agents’ unknown goals from behavior and uses those estimates to choose actions.
- In the proposed tasks, explicitly modeling the other player performs better than treating that player as part of the environment.
- The paper frames the setting as a two-player stochastic game with full environmental visibility, hidden goals, and no communication channel.
- Because each reward depends on both agents’ goals, each agent’s optimal policy must account for both goals.
2. Approach
SOM uses an agent’s own policy to model another player, inferring the player’s hidden goal online from observed actions and conditioning action selection on that estimate. The method combines separate self and other roles with differentiable goal optimization during episodes.
- SOM’s network takes the agent’s own goal, an estimate of the other player’s goal, and its observation state to output an action distribution and value estimate.
- Each agent uses two networks: fself computes its own actions and values, while fother infers the other agent’s goal from observed actions.
- The parameters of fother equal those of fself, allowing the agent to model the other player using its own policy.
- At each game step, the agent optimizes the estimated other goal using the previously observed action as supervision, then uses the estimate to choose its action.
- The inference optimizer’s step count is a game-dependent hyperparameter, and recurrent inference reuses the saved pre-inference state across steps.
- The estimated other goal is updated multiple times per step, while self-policy parameters are updated at episode end using A3C.
- Discrete goals use one-hot embeddings and a Gumbel-Softmax sample to obtain gradients through the estimated other goal.
- After optimization, the policy receives the one-hot vector corresponding to the argmax of the updated other-goal estimate.
3. Related Work
Prior work includes domain-specific opponent models, communication-based multi-agent learning, inverse reinforcement learning, and general opponent-modeling frameworks. SOM differs by inferring goals online with its own policy rather than relying on offline reward estimation, explicit strategy inference, or implicit action-prediction representations.
- Earlier opponent-modeling approaches often use domain-specific probabilistic priors or strategy parameterizations.
- Unlike communication-focused multi-agent reinforcement learning, this setting prohibits communication and requires reasoning from observed behavior.
- Inverse reinforcement learning infers an agent’s reward function from behavior assumed nearly optimal, whereas SOM directly infers goals online from observed actions.
- Some related work models how other agents learn or maintains cooperation strategies, while SOM does not explicitly shape the other agents’ learning.
- He et al.’s framework jointly learns policies and opponent behavior models, whereas SOM explicitly estimates goals rather than opponent strategies.
4. Experiments
Across cooperative, adversarial, symmetric, and asymmetric tasks, SOM estimates other agents’ goals and uses those estimates to choose actions. SOM generally outperforms baselines, while inference quality and timing constrain gains, especially in the Door game.
- Evaluation setup: SOM is evaluated on cooperative Coin, adversarial Recipe, and asymmetric cooperative Door games, comparing against three baselines and a true-goal upper bound.All tasks use the Mazebase gridworld environment.
- Baselines: SOM explicitly infers the other agent’s goal, whereas NOM has no other-agent model and IPP/SPP predict actions without explicitly estimating goals.SPP uses a separate opponent network, while SOM supplies an inferred goal to its policy network.
- Coin Game: SOM clearly outperforms all baselines on Coin reward and collects significantly more Other than Neither coins, indicating useful early goal inference.The comparison includes the empirical upper bound provided by TOG, which receives the true other-agent goal.
- Recipe Game: In Recipe, SOM wins approximately 75–80% against NOM, IPP, and SPP, while those baselines win approximately 15–20%; TOG remains substantially stronger.Same-type agent matches win approximately 40–50% of games.
- Door Game: In Door, SOM outperforms the three baselines but by a smaller margin than in earlier tasks because agents must learn both roles before using their policy to infer goals.The result supports SOM’s ability to handle asymmetric cooperative play and infer the other player’s role.
5. Discussion
SOM combines hidden-state inference with explicit modeling of other agents, improving policies and rewards across cooperative and competitive tasks. The approach is flexible but requires longer training and remains to be tested in broader settings.
- Discussion: SOM agents estimate other players’ hidden goals and use those estimates to choose actions, yielding better policies and higher rewards in cooperative and competitive settings.The method outperformed treating the other player as merely part of the environment in the proposed tasks.
- Discussion: SOM requires longer training than other baselines because the network is back-propagated through at every step.Despite this cost, the authors state that online operation is essential for adapting to other agents’ behavior.
- Discussion: SOM is simple and flexible: it requires no extra modeling parameters, works with any reinforcement learning algorithm, and integrates with different policy architectures.The authors also describe generalization to multiple players, environments, and tasks.
- Future work: Future evaluations should address more than two players, mixed strategies, diverse agent types, and deviations from the assumption that others resemble the self.The planned agent types may differ in action spaces, reward functions, roles, or strategies.
- Applications: Potential applications include human-robot interaction, teacher-student interaction, value alignment, and model-based multi-agent reinforcement learning.For model-based reinforcement learning, forward-model accuracy depends strongly on predicting other agents’ behavior.