PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Modelling under Partial Observability for Deep Reinforcement Learning

Georgios Papoudakis, Filippos Christianos, Stefano V. Albrecht

arXiv:2006.09447v4cs.LGcs.MAstat.ML

TL;DR

Agent modelling commonly assumes access to modelled agents’ local trajectories during execution, which may be restrictive when that information is unavailable. LIAM learns representations from the controlled agent’s local information and uses them to condition its policy. Across evaluated multi-agent environments, it generally improves returns over methods without the representations and sometimes approaches an ideal trajectory-access baseline.

  • Problem

    Existing agent-modelling methods commonly assume execution-time access to modelled agents’ local observations and past actions, which may be too restrictive when that information is unavailable.

    Source: §1 · p. 1

  • Method

    LIAM trains an encoder-decoder to reconstruct modelled-agent observations and actions from the controlled agent’s local information, then uses the encoder’s representation to condition its policy.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    LIAM generally achieves higher average returns than the same RL algorithm without learned representations, and in some cases approaches an ideal baseline with modelled-agent trajectory access.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support effective agent modelling using only the controlled agent’s local information during execution across the evaluated multi-agent settings.

    Source: §1 · p. 2

    LIAM requires modelled-agent actions to affect the controlled agent’s observations; otherwise, it learns an average representation over possible trajectories.

    Source: §5 · p. 10

Abstract

from arXiv · show

Modelling the behaviours of other agents is essential for understanding how agents interact and making effective decisions. Existing methods for agent modelling commonly assume knowledge of the local observations and chosen actions of the modelled agents during execution. To eliminate this assumption, we extract representations from the local information of the controlled agent using encoder-decoder architectures. Using the observations and actions of the modelled agents during training, our models learn to extract representations about the modelled agents conditioned only on the local observations of the controlled agent. The representations are used to augment the controlled agent's decision policy which is trained via deep reinforcement learning; thus, during execution, the policy does not require access to other agents' information. We provide a comprehensive evaluation and ablations studies in cooperative, competitive and mixed multi-agent environments, showing that our method achieves higher returns than baseline methods which do not use the learned representations.

1 Introduction

LIAM addresses the restriction that agent modelling often depends on other agents’ local trajectories at execution. It learns representations from the controlled agent’s local information and improves returns over methods without those representations.

1 Introduction

  • Existing agent-modelling methods commonly assume access during execution to the modelled agents’ local observations and past actions, an assumption that may be too restrictive.Limited views, unreliable communication, or unavailable knowledge of other agents’ perception systems can constrain that access.Source: §1 · p. 1
  • The paper asks whether effective agent modelling can rely only on the controlled agent’s locally available information during execution.The setting assumes a fixed set of possible policies for the non-learning agents.Source: §1 · p. 1
  • LIAM learns a compact representation of modelled agents by reconstructing their observations and actions from the controlled agent’s local information.The decoder uses modelled-agent observations and actions as training targets; only the encoder is retained for execution.Source: §1 · p. 2
  • LIAM generally raises average returns over the same RL algorithm without its learned representations, sometimes approaching an ideal baseline with access to the modelled agent’s trajectory.It is evaluated in double speaker-listener, level-based foraging, and modified predator-prey environments.Source: §1 · p. 2

2 Related Work

Prior neural agent-modelling approaches generally assume access to modelled-agent trajectories at execution or evaluate fully observable tasks. LIAM instead learns about trajectory relationships from the controlled agent’s local information.

Learning Agent Models

  • Earlier neural agent-modelling methods generally require modelled-agent trajectories at execution or fully observable evaluation environments.The cited approaches include action reconstruction [He et al., 2016], intention inference [Raileanu et al., 2018], encoder-decoder modelling [Grover et al., 2018], TomNet [Rabinowitz et al., 2018], relational forward models [Tacchetti et al., 2019], and VAE-based modelling [Zintgraf et al., 2021].Source: §2 · p. 2
  • LIAM differs from Poker-domain counter-strategy selection, which adapts over multiple hands, by adapting within a single episode.The Poker approaches select a strategy against an opponent type through a bandit problem [Johanson et al., 2008, Bard et al., 2013].Source: §2 · p. 2

Representation Learning in Reinforcement Learning

Multi-agent Reinforcement Learning

3 Approach

The approach models interactions with fixed-policy agents as a partially observable game. LIAM learns a modelled-agent representation from the controlled agent’s local trajectory and uses it alongside local observations to condition the decision policy.

3.1 Problem Formulation

  • The controlled agent optimizes average return against a set of fixed modelled-agent policies, initially treated as equally probable and held fixed during each episode.The controlled agent has no access to the identity of the policy used by the modelled agent during training or execution.Source: §3.1 · p. 3

3.2 Local Information Agent Modelling

  • LIAM’s recurrent encoder maps the controlled agent’s local trajectory to a latent representation, while a decoder learns to reconstruct the modelled agent’s observations and actions.Training uses information from both agents, but only the controlled-agent encoder is needed at execution; Figure 1 distinguishes the training and execution components.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 Reinforcement Learning Training

  • The RL policy is conditioned on the controlled agent’s local observation and the LIAM representation, which are combined in an augmented observation space.The paper uses A2C in its experiments and does not back-propagate actor-critic gradients into the encoder.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate LIAM in cooperative, mixed, and competitive environments against agent-modelling and task-adaptation baselines. Returns, representation analyses, ablations, and an episode walkthrough examine its performance and how its embeddings work.

4.1 Multi-Agent Environments

  • LIAM is evaluated in double speaker-listener, level-based foraging, and predator-prey, with ten fixed policies created for each environment.The environments cover cooperative, mixed, and competitive settings, respectively.Source: §4.1 · p. 5

4.2 Baselines

  • The five baselines include full-information and no-agent-model bounds, task adaptation, policy-identity classification, and contrastive representation learning.FIAM uses the modelled agent’s trajectory during execution, whereas NAM and VariBad do not use that trajectory during training; CBAM uses policy identities, and CARL uses contrastive learning.Source: §4.2 · p. 6

4.3 Evaluation of Returns

  • Across all three evaluation environments, LIAM’s returns are closer to FIAM’s upper baseline than NAM’s lower baseline.Figure 3 reports episodic evaluation returns with 95% confidence intervals; sampling from the policy distribution yielded significantly higher returns than greedy action selection.Source: Figure 3 · p. 7, §4.3 · p. 7

4.4 Model Evaluation

  • LIAM’s embeddings cluster interactions with the same fixed policy, and action and hidden-colour reconstruction accuracy exceeds 90% after 10 time steps.Action prediction is lower in level-based foraging and predator-prey than in double speaker-listener, where the controlled agent does not always observe the modelled agent.Source: §4.4 · pp. 7–8, Figure 4 · p. 8

4.5 Ablation Study

  • In double speaker-listener, removing either reconstruction target or either encoder input lowers returns, while LIAM-Local has the lowest returns among the ablations.The authors attribute LIAM-Local’s result to not learning the relationship between the local and modelled-agent trajectories; they suggest restrictive KL regularisation may explain LIAM-VAE’s worse performance.Source: Figure 5 · p. 8, §4.5 · pp. 8–9

4.6 Understanding LIAM

  • In the episode walkthrough, the controlled agent revises its colour belief after observing how the modelled agent responds to different messages, then moves toward the correct landmark.At time step 7 it believes its colour is blue; after more than 10 interactions, it recognizes that belief was wrong.Source: Figure 6 · p. 9, §4.6 · p. 9

5 Conclusion

LIAM learns models of other agents’ trajectories from the controlled agent’s trajectories and conditions the resulting model on the controlled agent’s local observations. It improves episodic returns over methods without agent modelling, with A2C+LIAM sometimes nearly matching FIAM.

5 Conclusion

  • LIAM models other agents’ trajectories using the controlled agent’s trajectories and is agnostic to cooperative, competitive, and mixed interactions.It can model an arbitrary number of agents.Source: §5 · p. 9
  • A2C combined with LIAM significantly improves episodic returns over baselines that do not use agent modelling, sometimes nearly matching FIAM, which has modelled-agent trajectories at execution.LIAM is trained alongside the decision policy and conditions the agent model on the controlled agent’s local observations.Source: §5 · pp. 9–10
  • LIAM requires modelled-agent actions to affect the controlled agent’s observations; otherwise, it learns an average representation over possible trajectories.The study focuses on low-dimensional vector observations; extending to pixel observations may require a different representation-extraction architecture, such as CARL.Source: §5 · p. 10
  • Future work includes extending local-information agent modelling to concurrently learning agents and investigating safety against agents that deceive or exploit the model.Source: §5 · p. 10

A Pseudocode of LIAM

LIAM’s pseudocode resets its encoder state, samples fixed policies, and runs parallel environments. It batches the gathered sequences to update the agent-modelling and A2C objectives.

A Pseudocode of LIAM

  • It resets the encoder LSTM’s hidden state, samples E fixed policies, creates E parallel environments, and initializes the modelled-agent action vectors to zero.Source: A Pseudocode of LIAM · p. 14
  • Across H time steps, the pseudocode loops over each environment and performs actions to obtain observations.Source: A Pseudocode of LIAM · p. 14
  • The algorithm gathers sequences from parallel environments into one batch, then takes gradient steps to minimize both the LED and A2C losses.Source: A Pseudocode of LIAM · p. 14

B Fixed Policies

The fixed-policy sets combine hand-designed and learned behaviors across three multi-agent environments. Their construction varies with each environment’s communication, cooperation, or predator-prey setup.

Double Speaker-Listener

  • In double speaker-listener, five pairs of navigation agents learn with MADDPG, while manually selected fixed policies communicate the same colors using different five-dimensional one-hot messages.Each policy has communication-message and navigation-action sub-policies.Source: B Fixed Policies · p. 14

Level-Based Foraging

  • Level-based foraging uses four heuristic policies and three each trained with IA2C and MADDPG; dropout during exploration and evaluation makes MADDPG policies stochastic.The authors observe that these stochastic policies perform significantly better.Source: B Fixed Policies · p. 14

Predator-Prey

  • Predator-prey uses ten fixed policies, each combining three predator sub-policies randomly drawn from four heuristics and six pretrained policies.The pretrained policies come from two sets of three agents trained with MADDPG and IA2C, respectively.Source: B Fixed Policies · p. 14

C.1 Contrastive Agent Representation Learning (CARL)

CARL learns representations from paired trajectories using contrastive learning. It matches the controlled and modelled agents’ trajectories within an episode and contrasts them with trajectories from other episodes.

C.1 Contrastive Agent Representation Learning (CARL)

  • CARL extracts modelled-agent representations from the controlled agent’s local observation and previous action without reconstruction.Training uses trajectories of all other agents, but execution uses only the controlled agent’s local trajectory.Source: §C.1 · p. 15
  • CARL defines positive pairs from the controlled and modelled agents’ trajectories in the same episode, and negative pairs from the controlled trajectory and modelled trajectories in other episodes.Source: §C.1 · p. 15
  • Separate recurrent encoders process the controlled agent’s and modelled agent’s trajectories to generate time-step representations.Source: §C.1 · p. 15
  • The controlled-agent representation feeds the A2C actor and critic; InfoNCE training attracts positive pairs and repels negative pairs.The loss uses cosine similarity and a softmax temperature.Source: §C.1 · p. 15

C.2 LIAM-VAE

LIAM-VAE prioritizes informative latent representations over reconstruction quality, using an encoder to summarize the controlled agent’s history and a decoder to reconstruct the modelled agent’s trajectory. Its prior can use the encoder’s recurrent state or the previous posterior; both choices yielded similar episodic returns.

C.2 LIAM-VAE

  • LIAM-VAE deliberately accepts worse reconstruction to make latent variables more informative about the modelled agent.The stated goal is to preserve information needed for agent modelling, rather than optimize reconstruction alone.Source: §C.2 · p. 15
  • The encoder uses the controlled agent’s sequential observations and actions to produce the mean and logarithmic variance of a Gaussian distribution.The decoder receives the latent variable and reconstructs the modelled agent’s trajectory at each time step.Source: §C.2 · p. 16
  • The prior either uses the encoder’s recurrent hidden state to output a Gaussian distribution or takes the previous time step’s posterior; both choices led to similar episodic returns.The reported returns are shown for the latter prior choice in Figure 5.Source: §C.2 · p. 16
  • The actor and critic receive the mean of the Gaussian posterior when training LIAM-VAE.Source: §C.2 · p. 16

D Implementation Details

The implementation uses recurrent encoder-decoder networks alongside A2C, with hyperparameters selected in double speaker-listener and generally held fixed across environments. In that environment, LIAM’s achieved returns stayed close to the best configuration across most tested learning-rate combinations.

D Implementation Details

  • The encoder is an LSTM with a linear ReLU layer, and feed-forward networks use two hidden layers of 128 nodes.Training uses Adam, with learning rates 3 × 10^-4 for A2C and 7 × 10^-4 for the encoder-decoder loss.Source: D Implementation Details · p. 16
  • Hyperparameters were optimized in double speaker-listener and kept constant in the other environments, except for level-based foraging’s entropy regularization.The entropy regularization was adjusted there because all algorithms showed significant performance gains.Source: D Implementation Details · p. 16
  • The experiments used 10 parallel environments and trained for 40 million time steps in double speaker-listener and 10 million in the other environments.Source: D Implementation Details · p. 16

E Different Learning Rates

  • LIAM’s achieved returns were close to the best configuration for most evaluated combinations of RL and encoder-decoder learning rates.Figure 7 presents achieved returns in the double speaker-listener environment as a heat-map over the learning-rate combinations.Source: E Different Learning Rates · p. 16, Figure 7 · p. 17

F Scalability in the Number of Fixed Policies

Against 100 fixed-policy combinations, LIAM’s returns diverged more from CBAM’s than in the earlier double speaker-listener evaluation. The authors relate LIAM’s advantage to embeddings that encode the modelled agent’s trajectory, communication message, and observation rather than explicitly clustering policy identities.

F Scalability in the Number of Fixed Policies

  • The 100 fixed-policy combinations are formed by taking the Cartesian product of the two double speaker-listener sub-policies.Source: F Scalability in the Number of Fixed… · p. 17
  • With 100 fixed policies, the return difference between LIAM and CBAM increased significantly, with LIAM achieving higher returns.Figure 8 plots episodic evaluation returns and 95% confidence intervals for both methods during training against the 100 policies.Source: Figure 8 · p. 17, F Scalability in the Number of Fixed… · p. 17
  • LIAM’s embeddings are not explicitly clustered by fixed-policy identity; the authors say they encode the modelled agent’s trajectory, communication message, and observation.They argue that this information lets RL learn a policy with higher returns than CBAM in this evaluation.Source: F Scalability in the Number of Fixed… · p. 17
Loading 2006.09447v4…