Machine Theory of Mind

Neil C. Rabinowitz; Frank Perbet; H. Francis Song; Chiyuan Zhang; S. M. Ali Eslami; Matthew Botvinick

Source-linked AI summary

Machine Theory of Mind

Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami, Matthew Botvinick

arXiv:1802.07740v2Feb 21, 2018cs.AI

TL;DR

The paper asks how machines can autonomously model other agents from limited behavioral data, addressing concerns about understanding artificial systems. It uses meta-learning to build a ToMnet and finds that it models varied agents, infers behavioral abstractions, and recognizes false beliefs in POMDP agents.

Problem
Machines are often difficult to understand, motivating autonomous methods for modeling other agents from limited behavioral observations.
Method
The ToMnet uses meta-learning to learn general and agent-specific models from observations of stochastic, algorithmic, and learned agents without assuming optimal policies.
Results
Across experiments, the ToMnet models varied agent populations, discovers behavioral abstractions, and implicitly learns that deep reinforcement learning agents can hold false beliefs.
Takeaways & Limitations
Such models may support multi-agent decision-making, value alignment, flexible cooperation, machine-human interaction, and interpretable AI.
Takeaways & Limitations
Explicit belief inference depends on supervised access to others’ latent beliefs and does not scale to predicting beliefs about every aspect of the world.

Abstract

from arXiv · show

Theory of mind (ToM; Premack & Woodruff, 1978) broadly refers to humans' ability to represent the mental states of others, including their desires, beliefs, and intentions. We propose to train a machine to build such models too. We design a Theory of Mind neural network -- a ToMnet -- which uses meta-learning to build models of the agents it encounters, from observations of their behaviour alone. Through this process, it acquires a strong prior model for agents' behaviour, as well as the ability to bootstrap to richer predictions about agents' characteristics and mental states using only a small number of behavioural observations. We apply the ToMnet to agents behaving in simple gridworld environments, showing that it learns to model random, algorithmic, and deep reinforcement learning agents from varied populations, and that it passes classic ToM tasks such as the "Sally-Anne" test (Wimmer & Perner, 1983; Baron-Cohen et al., 1985) of recognising that others can hold false beliefs about the world. We argue that this system -- which autonomously learns how to model other agents in its world -- is an important step forward for developing multi-agent AI systems, for building intermediating technology for machine-human interaction, and for advancing the progress on interpretable AI.

1. Introduction

The paper frames Machine Theory of Mind as autonomous learning of useful models of other agents from limited behavioral data. Its ToMnet uses meta-learning to form general and agent-specific models, and experiments show capabilities ranging from goal inference to recognizing false beliefs.

Motivation: Human social reasoning represents others’ desires, beliefs, and intentions rather than reconstructing their detailed physical mechanisms.These high-level models are evaluated by how well they support prediction and planning.
Approach: The paper seeks to learn models of other agents autonomously from limited behavioral data, rather than hand-coding generative or rational-planning models.This distinguishes the approach from inverse reinforcement learning, Bayesian inference, Bayesian Theory of Mind, and game theory.
Approach: Meta-learning trains an observer to predict a novel agent’s future behavior from behavioral traces, while learning a prior over commonalities among agents.At test time, observations support rapid adaptation to the new agent.
Approach: The ToMnet separates a general theory of mind in its learned weights from an agent-specific embedding formed from observations at test time.These components correspond to a prior and posterior over agent behavior.
Results: Experiments show ToMnet models random, algorithmic, and deep reinforcement learning agents, including inference of goals, cost-reward tradeoffs, and population-level abstractions.For random agents, it approximates Bayes-optimal hierarchical inference; for algorithmic agents, it performs few-shot inverse reinforcement learning.
Results: When trained on deep reinforcement learning agents in POMDPs, ToMnet learns that agents can hold false beliefs and can explicitly predict their belief states.It also infers what agents can see and therefore tend to believe from behavior alone.

2. Model

The paper formulates modeling unfamiliar agents as an online meta-learning task and implements it with a three-module ToMnet. The observer uses behavioral trajectories to infer agent-specific representations and predict future behavior across varied agent species and gridworld environments.

Task formulation: Agents may differ in observations, rewards, discount factors, and policies, which can be stochastic, algorithmic, or learned and need not be optimal.Agents may also be stateful, although their hidden states do not carry over between episodes.
Task formulation: The observer predicts a novel agent’s future behavior from full or partial trajectories collected across past episodes, optionally using a partial current-episode trajectory.The number of past episodes may vary, including zero, and predictions may target actions or latent states such as beliefs.
ToMnet architecture: The ToMnet has character, mental-state, and prediction modules that respectively encode past behavior, infer current mental state, and predict subsequent behavior.The character net processes past episodes into echar, while the mental state net processes the current trajectory into emental.
ToMnet architecture: The prediction net combines character and mental-state embeddings with a current state to estimate next-step actions and other behavioral quantities.Its outputs include action probabilities, object-consumption probabilities, and successor representations; the network is trained end-to-end with shared and target-specific heads.
Agents and environments: Experiments use parametrically generated gridworld POMDPs with up/down/left/right/stay actions, deterministic dynamics, and consumable objects.The tested agent species include random policies, value-iteration planners, and partially observable agents trained with feed-forward networks or LSTMs.

3. Experiments

Across increasingly rich agent populations, the ToMnet learned priors and agent-specific models from limited behavioral observations, predicting actions, goals, trajectories, and belief-dependent behavior. Its embeddings captured interpretable characteristics such as action statistics, species, preferences, and visibility, while its learned models supported false-belief predictions.

3.1. Random agents: The ToMnet approximated Bayes-optimal online inference for random agents and adapted its predictions to the species distribution encountered during training.It learned priors for policy sparsity and hierarchical inference over mixtures of stochastic and deterministic species.
3.1. Random agents: Its character embeddings organized random agents by empirical action counts, reflecting sufficient statistics for their predicted policies.The embedding solution resembled the Dirichlet posterior structure of the Bayes-optimal estimate.
3.2. Inferring goal-directed behaviour: The ToMnet predicted goal-directed agents’ next actions, consumed objects, and successor representations from full or partial behavioral trajectories.Increasing the number of past observations improved predictions, while zero-shot predictions retained a species-level prior for reasonable behavior.
3.2. Inferring goal-directed behaviour: A single trajectory could reveal stable goal preferences: distant-object pursuit led to confident predictions of the same object, whereas closest-object pursuit produced more cautious predictions.These predictions generalized to new episodes and starting positions.
3.3. Learning to model deep RL agents: For deep RL agents, the ToMnet inferred species and individual characteristics, modeling local observability, persistent state, preferred objects, and belief-dependent corrective behavior.It learned these behavioral statistics even without swap events during training and distinguished visibility factors in variational embeddings.

4. Discussion

The ToMnet learns general and agent-specific models from behavioural observations, but the experiments remain simple and leave substantial scaling and deployment challenges.

Discussion: The ToMnet learns a general model over its training distribution and constructs agent-specific models online from new behavioural observations.It flexibly models different agent species while making few assumptions about their generative decision processes.
Discussion: The experiments were simple demonstrations of the core ideas, and scaling the ToMnet to richer domains remains future work.The authors identify richer predictions, inductive biases, and use of agents’ own experience as future needs.
Discussion: The work was conducted entirely in controlled gridworlds rather than complex 3D visual environments or other rich-state POMDPs.The authors chose gridworlds for the control they provide and propose extension to richer environments.
Discussion: The study did not test situations where the observer itself has limited observability, an important challenge in real-world social interaction.Learning to infer what another agent knows when the observer lacks that information is identified as a future challenge.
Discussion: Future characterisations could include animacy, prosociality, adversarialness, reactivity, and planning ability, potentially enabling machine anthropology.The authors also identify placing the ToMnet inside artificial agents for multi-agent tasks as an important direction.

A. Model description: architectures

The architecture appendix describes implementation details and notes that performance was not substantially optimized through systematic architecture or hyperparameter search.

A. Model description: architectures: The appendix provides the precise architecture details used in the main-text experiments.It serves as a technical specification for the reported models.
A. Model description: architectures: The authors selected sensible-looking architectures and hyperparameters without systematic or substantial optimization.They anticipate that better decisions could improve performance, but leave this outside the study’s scope.

A.1. Common elements.

Across ToMnet variants, trajectories are converted into spatialized state-action representations, encoded into embeddings, and used for prediction with fixed training settings.

A.1. Common elements.: The common preprocessing begins with trajectories defined as sequences of observed state/action pairs.Both the character and mental-state networks consume these trajectories.
A.1. Common elements.: Observed states have shape (11 × 11 × K), while actions are length-5 vectors that are tiled spatially and concatenated with states.The resulting tensor has shape (11 × 11 × (K + 5)).
A.1. Common elements.: All ToMnets use Adam with learning rate 10^-4 and batch size 16, trained for 40k minibatches on random agents and 2M otherwise.These settings define the common training schedule across the reported experiments.
A.1. Common elements.: Each agent receives a variable number of past one-step episodes, and the character embedding is zero when no past episodes are sampled.The number of past episodes is sampled from Npast ∼U{0, 10}.
A.1. Common elements.: The character net encodes spatialized state-action pairs with a convolutional network and convolutional LSTM before producing a 2D agent embedding.Average pooling and a fully connected layer produce echar,i.
A.1. Common elements.: The prediction net combines the agent embedding with a query state and uses convolutional processing to output next-step action probabilities.The final layer produces logits in R5 followed by a softmax.

A.3.1. EXPERIMENT 1: SINGLE PAST MDP

In the single-past-MDP experiment, the ToMnet forms an agent embedding from one complete trajectory and uses it with a new query state to predict actions, consumption, and successor representations.

A.3.1. EXPERIMENT 1: SINGLE PAST MDP: A single past episode supplies one complete trajectory for the character embedding, while the query state is the initial state of a new MDP.No mental-state embedding is required in this experiment.
A.3.1. EXPERIMENT 1: SINGLE PAST MDP: The character net processes the past trajectory with a 5-layer ResNet and LSTM to produce a 2D- or 8D agent embedding.The reported results were not substantially different between the two embedding sizes.
A.3.1. EXPERIMENT 1: SINGLE PAST MDP: The prediction network combines the agent embedding with the query state and branches into heads for action, consumption, and successor-representation predictions.A shared torso supports the three prediction tasks.
A.3.1. EXPERIMENT 1: SINGLE PAST MDP: The action head outputs a softmax policy over five actions.It maps the shared torso representation to 5-dimensional logits before normalization.
A.3.1. EXPERIMENT 1: SINGLE PAST MDP: The consumption head outputs four sigmoid probabilities for whether each object will be consumed by episode end.Each output is a Bernoulli probability.
A.3.1. EXPERIMENT 1: SINGLE PAST MDP: The successor-representation head predicts normalized representations independently for discount factors γ = 0.5, 0.9, 0.99.A softmax is applied independently over the three channels.

A.3.2. EXPERIMENT 2: MANY PAST MDPS, ONLY A SINGLE SNAPSHOT EACH

Experiment 2 forms character embeddings from single state/action snapshots sampled across many past episodes, then uses the prediction net as in Experiment 1.

Data: Character embeddings use one state/action snapshot from each of Npast ∼ U{0, 10} past episodes.The query state is handled as in Experiment 1.
Character net: The character net spatialises, concatenates, and ResNet-processes each snapshot before mapping it to a 2-dimensional contribution.Because each trajectory contributes only one state/action pair, this component has no LSTM.
Character net: Contributions from the Npast past trajectories are summed to form the character embedding echar,i.
Prediction net: The prediction net is unchanged from Experiment 1.

A.3.3. EXPERIMENT 3: GREEDY AGENTS

Experiment 3 evaluates ToMnet on greedy agents using full past trajectories, recent behavior, and explicit predictions of actions, consumption, successor representations, and beliefs. Training averages these losses equally, with an optional variational bottleneck.

Data: Character embeddings summarize four full past-trajectory POMDPs while the observer sees the entire gridworld despite the agents’ partial observability.The current episode is split at a random time into a recent trajectory and query state.
Character net: The character net maps each past trajectory to an 8-dimensional contribution and sums these contributions into echar,i.
Mental net: Recent state/action pairs pass through a 5-layer ResNet and convolutional LSTM to produce the mental-state embedding emental,i ∈ R11×11×32.An empty recent trajectory corresponds to the query state being the initial state.
Prediction net: The prediction torso combines echar,i, emental,i, and the query state, while the consumption head outputs five dimensions for gridworlds containing a subgoal object.
Loss function: Training averages agent losses and gives equal weight to action, consumption, successor-representation, and belief prediction components.These losses are computed from predicted actions, object consumption, discounted successor representations, and 122-dimensional object-belief distributions.
Loss function: When DVIB is used, training adds β-weighted KL terms between posterior embeddings and their priors.
Agent setting: Greedy algorithmic agents have sparse preferences over terminal objects and use value iteration with a discount factor of 1.Their movement and wall penalties are specified separately from the randomly drawn object preferences.

D. Deep RL agent training and architecture

Deep reinforcement-learning agents are trained across varied observation architectures, fields of view, and random seeds, with gridworld observations and optional object-location beliefs.

Training: Deep RL agents use the UNREAL architecture, train for over 100M episode steps, and apply entropy regularisation to encourage exploration.Training uses 16 CPU workers, Adam at 10^-5, and whole-episode BPTT over 50 steps.
Population: The study trains 660 agents across 33 seeds, five fields of view, two architectures, and two network depths.Top-performing agents are selected and partitioned into training and test sets, with reward permutations expanding each condition.
Observations: At each time step, agents receive nine 11×11 feature planes encoding map entities, the agent, and currently unobservable locations.
Beliefs: An auxiliary belief head predicts each object’s location using five 122-dimensional discrete distributions.The distributions cover the 11×11 maze and an additional absent-object state.

E. Additional results

Additional materials report train/test agent organization, losses for alternative embedding configurations, and a Natural Sally-Anne result that does not require swap events during training.

Agent splits: The additional-results material distinguishes train and test agents in its tabular organization.
Loss comparisons: The supplementary table compares losses for models with neither, one, or both character and mental nets, including shuffled-embedding tests.
Natural Sally-Anne: The Natural Sally-Anne performance remains independent of observing swap events during training.This holds when evaluation uses POMDPs with swap probability p = 0 rather than p = 0.1.

F.1. Hypersensitivity of 3 × 3 agents to swap events with swap distance 1

Agents with 3 × 3 fields of view were considerably more sensitive to adjacent swap events than agents with 9 × 9 fields of view, while 5 × 5 and 7 × 7 agents showed intermediate sensitivity. Swap effects were measured using Jensen-Shannon Divergence across policies, successors, and beliefs.

3 × 3 agents were considerably more sensitive to adjacent swap events than 9 × 9 agents.
5 × 5 and 7 × 7 agents showed intermediate sensitivity to adjacent swap events.
Larger-field-of-view agents were often sluggish to respond behaviorally, adjusting their policies more strongly on subsequent steps.A 5 × 5 agent shifted only a small amount of probability toward an immediately visible preferred object before strongly adjusting on the following step.
Jensen-Shannon Divergence measured swap-event effects on agents’ and ToMnet’s predicted policies, successors, and beliefs.The authors chose it for its symmetry and stability, while KL-divergence typically produced more variance.