MA-UITD: Offline Multi-Agent Reward Inference for Characterizing Continuous Social Behavior

1 INTRODUCTION

• Explicit rewards specify which outcomes an individual values and how competing outcomes are traded off, offering an interpretable account beyond action prediction.Inferred rewards, values, and prediction errors can also serve as trial-by-trial variables for analyses of neural recordings.Source: §1 · pp. 1–2
• Social choices reflect both personal preferences and expectations about others, so similar actions can arise from different incentives.Single-agent IRL treats individuals as isolated decision-makers, complicating separation of intrinsic preferences from socially contingent choices.Source: §1 · p. 2
• Fixed behavioral datasets often lack reward labels, known dynamics, or opportunities for new interaction, while observed states and actions may be continuous.Existing methods can require repeated policy updates and environment interaction, or discrete actions and additional rollouts.Source: §1 · p. 2
• MA-UITD uses belief-conditioned successor features to represent unilateral long-term values while marginalizing over other agents’ anticipated behavior.Its offline learning combines inverse temporal-difference consistency with noise-contrastive behavioral learning, without known dynamics or an inner policy-optimization loop.Source: §1 · pp. 2–3
• Across cooperative, competitive, and general-sum simulations, MA-UITD outperforms discrete-action baselines and captures task-relevant reward structure in continuous-action settings.On social-interaction data, it reveals distinct spatial and interaction-dependent incentives underlying leader–follower dynamics.Source: §1 · p. 3

Multi-agent inverse reward learning

• MA-AIRL uses adversarial reward estimation with policy optimization, MAMQL assumes discrete actions and uses additional policy rollouts, and other methods impose further modeling constraints.Latent quantal level-k reasoning requires recursive value iteration; GS-CIOC presumes known dynamics and local linear-quadratic approximations.Source: §2 · p. 3

Successor-feature-based inverse reward learning

• Inverse temporal-difference learning estimates shared cumulants, demonstrator-specific successor features, and preference vectors from fixed demonstrations, but lacks an explicit strategic Markov-game formulation.Its formal setup uses different reward tasks within a common controlled Markov process with discrete actions.Source: §2 · p. 3

Offline reward inference for animal behavior

• Offline reward-inference methods for animal behavior can avoid known dynamics and additional interaction, but prior formulations are single-agent and use discrete actions.Joint experimental trajectories can be available even when constructing a faithful interactive simulator is difficult.Source: §2 · p. 3
• MA-UITD learns belief-conditioned successor features from fixed joint transitions and uses noise-contrastive estimation to avoid the continuous-action normalizer.It recovers explicit agent-specific reward profiles without adversarial optimization, additional environment interaction, or a forward policy-optimization loop.Source: §2 · p. 3

3 PROBLEM FORMULATION

• The setup is an n-agent discounted Markov game with joint state and actions, transition dynamics, and no known reward functions.Each agent’s action space may be discrete or a bounded continuous subset of R^di.Source: §3 · p. 3
• Learning uses only offline joint expert trajectories without observed generating rewards or additional environment interaction.The formulation assumes joint actions lead to state transitions under the game’s transition dynamics.Source: §3 · p. 4
• The method assumes agent-specific arrival-state rewards, assigning value to the resulting state rather than directly to the executed action.Because the state is joint, these rewards can still encode individual preferences and interactions; action-contingent rewards can be represented equivalently when choices produce distinguishable outcome states.Source: §3 · p. 4
• The target is a representative agent-specific reward profile, assessed by whether it ranks demonstrated actions above counterfactuals under each agent’s predictive belief.Rewards in general-sum Markov games are not uniquely identifiable from demonstrations without additional assumptions.Source: §3 · p. 4

4 MULTI-AGENT UNILATERAL INVERSE TEMPORAL DIFFERENCE LEARNING (MA-UITD)

• MA-UITD combines shared reward features, agent-specific preferences, and belief-conditioned successor features to infer explicit agent-specific rewards from fixed joint demonstrations.Each agent’s BCSFs represent the long-term consequences of its own action while averaging over its prediction of other agents’ behavior.Source: §4 · p. 4

4.1 BELIEF-CONDITIONED SUCCESSOR FEATURES (BCSF)

• Because an agent chooses without observing others’ current actions, its unilateral value averages over a belief about their actions; equilibrium and level-k reasoning restrict that belief differently.The reward preferences remain encoded by agent-specific preference vectors, separate from strategic assumptions in predictive beliefs.Source: §4 · p. 4, §4.1 · p. 5

4.2 OFFLINE REWARD INFERENCE WITH MA-UITD

• The offline procedure calibrates predictive beliefs to demonstrated conditional behavior and fits the representation from observed joint transitions without a transition model or policy-optimization loop.Observed transitions provide Monte Carlo samples for the behavioral and temporal-difference objectives.Source: §4 · p. 4, §4.2 · p. 5

4.2.1 CONTRASTIVE BEHAVIORAL LEARNING

• Noise-contrastive estimation fits unnormalized BCSF scores by comparing demonstrated actions with sampled alternatives, avoiding the intractable continuous-action normalizer.The temperature controls assumed behavioral stochasticity, and the alternatives are unilateral while other agents’ behavior remains marginalized.Source: §4.2.1 · p. 6

4.2.2 UNILATERAL INVERSE TEMPORAL-DIFFERENCE LEARNING

• The unilateral inverse temporal-difference objective enforces Bellman consistency on BCSFs using demonstrated transitions, which implicitly average over other agents’ actions and environment transitions.The sample-based objective requires neither the transition kernel nor an explicit opponent model.Source: §4.2.2 · p. 6

4.2.3 OPTIMIZATION AND REWARD RECOVERY

• Training alternates behavioral and temporal-difference updates, then recovers each agent’s reward as the shared feature map weighted by that agent’s preference vector.The shared representation applies across cooperative, competitive, and general-sum games.Source: §4.2.3 · p. 6

5.1 EXPERIMENTAL SETUP

• The simulated evaluation spans cooperative Simple Spread, competitive Simple Push, and general-sum Gems, with discrete actions or bounded two-dimensional displacements.The games use an 80-step horizon; Gems includes private and jointly collected rewards.Source: §5.1 · pp. 6–7, §5 · p. 6

5.2 POLICY AND REWARD RECOVERY

• MA-UITD achieves higher recovered-policy returns than both baselines in Simple Spread and for both roles in Simple Push, and most closely approaches expert performance in Gems.In Gems, MA-UITD and MAMQL recover both cooperative and competitive behaviors, while MA-AIRL recovers only competitive behavior.Source: §5.2 · p. 7, Table 1 · p. 7
• MA-UITD outperforms baselines on both Pearson and Spearman reward-recovery measures across all three simulated environments.Correlations compare predicted and ground-truth immediate rewards on held-out transitions.Source: Figure 1 · p. 7, §5.2 · p. 7
• In Simple Spread, MA-UITD recovers the task’s coverage structure and nearly identical spatial reward landscapes for both agents.When one agent is near a landmark, the other landmark becomes relatively more rewarding for the other agent, encouraging landmark coverage rather than converging on one target.Source: §5.2 · pp. 7–8, Figure 2 · p. 8
• In continuous-action tasks, recovered policies match expert returns in Gems and approach them in Simple Spread, while Simple Push performance remains role dependent.Across held-out trajectories, reward agreement is positive in all three tasks; Simple Spread has the strongest linear agreement (agent-mean PCC 0.828±0.033).Source: §5.2 · p. 8, Table 2 · p. 8, Table 3 · p. 8
• On mouse social-foraging trajectories, leaders’ learned rewards depend primarily on their own location, whereas followers’ rewards emphasize proximity to the partner.Across five pairs, followers valued proximal partner locations more highly than leaders (two-sided paired t-test at d = 0.07, P = 0.026; n = 5 pairs).Source: §5.2 · p. 9, Figure 3 · p. 9

6 DISCUSSION

• MA-UITD uses belief-conditioned successor features to recover explicit agent-specific rewards from fixed demonstrations in continuous-state settings with discrete or bounded continuous actions.Reward inference is conditioned on predictive beliefs estimated from observed behavior.Source: §6 · p. 9
• Rewards outside the support of observed states are weakly evidenced and depend largely on regularization and the function approximator’s inductive biases.The paper advises cautious interpretation of these estimates until additional data support them.Source: §6 · p. 10

AI USE STATEMENT

• The authors report using generative AI for language polishing, literature retrieval and discovery, and coding, but not for ideation, methodology, or experimental design.They state that they reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A ALGORITHMS

• Algorithm 1 takes reward-free joint demonstrations and training settings as input, then returns shared reward-feature parameters and agent-specific parameters.Its loop samples unilateral alternatives and updates agent parameters using the inverse temporal-difference loss.Source: A ALGORITHMS · p. 12

B IMPLEMENTATION DETAILS

• For Simple Spread, reward features use agent–landmark and inter-agent distances, while successor-feature networks also receive positions, velocities, and each agent’s own action; Simple Push and Gems use task-specific variants.In Gems, a shared MLP([32, 32]) torso encodes agent positions; algorithms receive the same task-specific state and action information.Source: B IMPLEMENTATION DETAILS · pp. 12–13

B.1 EXPERT DEMONSTRATIONS.

• Experts were trained with multi-agent ACKTR for discrete Simple Spread and Simple Push, and with MASAC for continuous versions and both Gems versions; datasets span 500–4,000 episodes.Gems demonstrations mix fully cooperative and fully competitive behavior equally.Source: §B.1 · p. 13

B.2 MA-UITD ON DISCRETE ACTION DATASETS

• For discrete-action datasets, MA-UITD replaces the NCE loss with maximum-likelihood estimation.Source: §B.2 · p. 13

B.3 NOISE DISTRIBUTION

• Negative actions are drawn from the expert’s empirical marginal action distribution by uniformly sampling a training transition and using its action at the current state.Source: §B.3 · p. 13

B.4 CONTINUOUS POLICY ROLLOUT

• Continuous-policy rollouts approximate each Boltzmann policy with 256 low-discrepancy candidate actions, sample the executed action categorically, and fix candidate-generation and sampling seeds across model seeds.Source: §B.4 · p. 13

B.5 HYPERPARAMETERS

• The authors conducted a comprehensive hyperparameter search and list MA-UITD settings separately for discrete-action, continuous-action, and real-animal-trajectory datasets.Source: §B.5 · p. 14, Table 4 · p. 14, Table 5 · p. 14

• The workstation had a 12-core AMD Ryzen 9 9900X CPU, 64 GB memory, and one NVIDIA GeForce RTX 5080 GPU with 16 GB; each run used one GPU.Experiments used Python 3.10.20, PyTorch 2.11.0, and CUDA 12.8.Source: §B.6 · p. 15

• Table 7 reports held-out immediate-reward recovery using PCC and SCC, with each cell’s upper and lower values corresponding to those metrics.Entries are mean ± sample standard deviation across three training seeds on the fixed 500-episode test split.Source: Table 7 · p. 15, §C.1 · p. 15

C.2 POLICY IMITATION FOR THE Gems

• For Gems policy imitation, Table 8 reports the percentage of gems collected, averaged across three seeds with 500 stochastic-policy episodes per seed.Source: Table 8 · p. 15, §C.2 · p. 15

C.3.1 RECOVERED REWARD IN DISCRETE-ACTION ENVIRONMENTS

• Figure 4 compares true and recovered rewards against landmark and arrival-state distances for both agents in discrete Simple Spread.Curves show raw MA-UITD, MA-AIRL, and MAMQL rewards, averaged over 4,836 held-out transitions from 500 episodes.Source: Figure 4 · p. 16
• Figure 5 plots true and recovered adversary and good-agent rewards against landmark distances in discrete Simple Push.Curves show raw MA-UITD, MA-AIRL, and MAMQL rewards, averaged over 12,500 held-out transitions from 500 episodes.Source: Figure 5 · p. 17
• Figure 6 compares ground-truth and recovered red- and blue-agent rewards against gem-arrival distances and shows spatial reward maps with gem locations and collection radii.The maps use independent symmetric scales centered at zero; blue denotes negative rewards and red positive rewards.Source: Figure 6 · p. 18

C.3.2 RECOVERED REWARD IN CONTINUOUS-ACTION ENVIRONMENTS

• Figure 7 compares true and raw MA-UITD rewards across continuous-action Simple Push, Gems, and Simple Spread, including a spatial map for Simple Spread.Curves are binned means from 500 held-out episodes per task; spatial maps use independent raw color scales.Source: Figure 7 · p. 19