Source-linked AI summary
Learning Latent Representations to Influence Multi-Agent Interaction
Annie Xie, Dylan P. Losey, Ryan Tolsma, Chelsea Finn, Dorsa Sadigh
TL;DR
Non-stationary agents adapt their policies in response to the ego agent, making repeated interaction difficult to anticipate and coordinate. The paper learns high-level latent strategies and their dynamics from local experience, then trains a policy to influence future strategies. Across simulated domains and robot air hockey, the approach outperforms alternatives and actively influences the other agent.
Problem
Non-stationary, separately controlled, partially observable agents change behavior in response to the ego agent, creating a need to anticipate and model those changes for co-adaptation.
Method
The framework learns latent strategies and their dynamics from local interaction histories, conditions the ego policy on predicted strategies, and optimizes long-term rewards to influence them.
Results
The approach outperforms alternatives across four simulated environments and a real robot air-hockey experiment, while learning to influence the other agent.
Takeaways & Limitations
Representing non-stationary agents with latent strategies enables the ego agent to anticipate behavioral changes and actively guide interaction toward co-adaptation.
Takeaways & Limitations
The learned policy was too brittle for human opponents because edge-based puck blocking caused more failures when humans shot imperfectly.
Abstract
from arXiv · showhide
Seamlessly interacting with humans or robots is hard because these agents are non-stationary. They update their policy in response to the ego agent's behavior, and the ego agent must anticipate these changes to co-adapt. Inspired by humans, we recognize that robots do not need to explicitly model every low-level action another agent will make; instead, we can capture the latent strategy of other agents through high-level representations. We propose a reinforcement learning-based framework for learning latent representations of an agent's policy, where the ego agent identifies the relationship between its behavior and the other agent's future strategy. The ego agent then leverages these latent dynamics to influence the other agent, purposely guiding them towards policies suitable for co-adaptation. Across several simulated domains and a real-world air hockey game, our approach outperforms the alternatives and learns to influence the other agent.
1 Introduction
The paper addresses non-stationary multi-agent interaction by representing other agents’ policies as changing latent strategies and learning how the ego agent’s behavior influences them. It proposes latent-strategy learning and influence, evaluated in simulation and robot air hockey.
- Non-stationary agents continually change their behavior in response to the ego agent, making interaction a changing learning environment.
- The framework targets repeated interactions with separately controlled, partially observable agents without requiring explicit low-level policy modeling.It uses high-level policy representations called latent strategies, which can change over time.
- The ego agent learns latent strategies and models how its behavior changes another agent’s future strategy.
- The ego agent can purposely influence other agents toward strategies better suited for co-adaptation.
- The approach is compared across four simulated environments and a real experiment with two 7-DoF robot arms playing air hockey.It outperforms alternatives and learns to influence the other agent.
2 Related Work
Related work handles non-stationary interaction through opponent modeling, centralized or communicative multi-agent learning, influence objectives, and partial-observability formulations. The paper instead emphasizes latent representations for decentralized interaction with separately controlled agents.
- Prior opponent-modeling methods represent intentions, roles, learning dynamics, or handcrafted and learned behavioral features.Explicitly modeling recursive opponent intentions or policies can become computationally intractable.
- Other multi-agent approaches use centralized training or learned communication, whereas this approach requires neither.This supports operation with humans or decentralized agents.
- Prior influence methods modify learning objectives or study influence within a single interaction, while this approach produces influence without explicit reward encouragement.
- The setting can be viewed as a POMDP whose hidden state is the other agent’s strategy, with structure modeled as a sequence of hidden-parameter MDPs.The strategy is assumed constant within an episode and changing between episodes.
- Unlike hand-designed air-hockey systems that switch strategies based on opponents, the paper learns both opponent strategies and responses.
3 Repeated Interactions with Non-Stationary Agents
The paper models repeated interaction as a sequence of hidden-parameter MDPs in which the other agent’s latent strategy affects transitions and rewards and changes in response to prior trajectories. This formalization supports predicting and influencing future strategies.
- The problem considers dyadic repeated interactions between an ego agent and an opponent or partner, with extension to N other agents.
- The ego agent observes interaction histories while the other agent continually updates its policy, motivating a high-level intention representation for coordination.
- Each interaction is a hidden-parameter MDP where latent strategy z_i affects the ego agent’s transition function, reward function, and trajectory.The trajectory contains states, actions, and rewards over a fixed horizon H.
- The other agent’s next strategy follows Markovian latent dynamics conditioned on its current strategy and the ego agent’s previous trajectory.The formal relation is z_i+1 ∼ f(· | z_i, τ_i).
- Across interactions, the ego agent maximizes cumulative discounted reward while its actions affect the other agent’s future policy.
- The proposed overview combines prediction of latent strategy from previous experience with a strategy-conditioned policy and joint representation learning and model-free reinforcement learning.
4 Learning and Influencing Latent Intent (LILI)
LILI jointly learns to infer another agent’s latent strategy from local trajectories, condition the ego policy on that prediction, and influence future strategies through long-term reward optimization. Its encoder-decoder representation learning is trained alongside SAC-based reinforcement learning.
- LILI predicts the other agent’s next latent strategy from the previous interaction and uses a policy conditioned on that prediction.
- The encoder approximates latent dynamics from local experience, while the decoder reconstructs the next interaction’s transitions and rewards without strategy labels.
- 4.3 Influencing by Optimizing for Long-Term Rewards: The ego policy is trained to maximize rewards across interactions, causing it to generate trajectories that lead to exploitable future latent strategies.
- 4.3 Influencing by Optimizing for Long-Term Rewards: In the motivating example, the ego agent establishes trust to guide the other agent toward a strategy that enables safer speed regulation.
- The encoder and decoder are fully connected networks, while the SAC actor and critic are conditioned on state and latent embedding.
- Training alternates interaction collection with critic, actor, encoder, and decoder updates using replayed interaction pairs.
5 Experiments
The experiments evaluate latent-strategy learning and influence across four simulated environments and real-world air hockey, comparing LILI with learning-based baselines and oracle information. LILI generally improves returns, nearly matches oracle performance, and can shape opponents’ future strategies when doing so is beneficial.
- Experimental setup: The study compares SAC, SLAC, an oracle, LILI (no influence), and full LILI across four simulations and a real robot air-hockey experiment.The environments include Point Mass, Lunar Lander, two driving settings, and air hockey with two 7-DoF robot arms.
- Simulation results: Across all simulated domains, LILI achieves higher returns than existing reinforcement-learning methods and nearly matches oracle performance.The comparison covers the simulated results summarized in Figures 4 and 5.
- Simulation results: In Point Mass, LILI learns to intentionally over- or under-shoot the hidden target, trapping it near the start location instead of greedily reaching it each interaction.The ego agent uses whether it ends inside or outside the circle to influence the target’s clockwise or counterclockwise movement.
- Simulation results: In the remaining simulations, LILI and LILI (no influence) perform identically because each available latent strategy is equally beneficial to the ego agent.This equality means there is no performance advantage from influencing the other agent in those settings.
- Air hockey results: With a robot opponent, LILI blocks 91% of shots in the final 100 interactions, compared with 44% for SAC and 18% when moving randomly.LILI is the only evaluated approach reported to co-adapt with all of the opponent’s changing intentions.
- Air hockey results: In air hockey, LILI influences the striker to fire left 41% of the time, exploiting the ego agent’s bonus for blocking on the left.Because the striker changes firing locations by rule, LILI manipulates it into alternating between left and middle strategies.
6 Discussion
The framework enabled robots to anticipate and influence another agent’s latent strategy for more seamless co-adaptation, but the learned policy was too brittle for human interaction. The study was also constrained by limited pandemic-era access to robots and participants.
- The approach represented non-stationary agents with latent strategies and enabled robots to anticipate and influence strategy changes for co-adaptation.
- LILI worked well against another robot but was too brittle to interact alongside humans.The policy learned to block pucks with paddle edges, causing more failures against imperfect human shots.
- COVID-19 restrictions prevented the planned full-scale user study with 10+ human participants, leaving only a proof-of-concept pilot.The pilot found that LILI was not sufficiently robust for an immediate user study.
A Implementation Details
The implementation uses multilayer perceptrons for representation learning and reinforcement learning, with an eight-dimensional latent space and joint encoder training objectives.
- The encoder and decoder are two-layer, 128-unit MLPs, and the latent space has size 8.
- The policy and critic networks are MLPs with two fully connected layers of size 256.
- Encoder weights are updated using gradients from both the representation objective Jrep and the soft actor-critic critic loss JQ.
B Experimental Details: Simulated Domains
The simulated evaluation spans target-reaching, Lunar Lander, and driving settings with interaction-dependent or time-dependent latent dynamics, including oracle and collision-based task variants.
- In the target-reaching environment, the unknown target moves 0.2 radians clockwise or counterclockwise based on the ego agent’s position after 50 timesteps.
- In Lunar Lander, the unknown launchpad alternates between z = [0.4, 0.2] and z = [−0.4, 0.2] at every interaction.The latent dynamics are independent of the ego agent’s interactions and evolve only with time.
- The Lunar Lander reward includes penalties for velocity, angle, engine use, and lateral control, with state variables defined for position, velocity, and angle.
- The Lunar Lander oracle receives the launchpad position z as an additional observation.
- In the simpler driving domain, the other agent switches into the lane selected by the ego agent in the previous interaction, making same-lane choices collision-prone.
- CARLA adds aggressive alternating lane switches, steering-only observations, and early termination for collisions or lane-boundary violations.
C Experimental Details: Robotic Air Hockey
The robotic air-hockey task uses two Panda arms, with one blocking and one striking; the striker switches among left, middle, and right aiming modes based on prior interaction outcomes.
- Two Franka Emika Panda arms play air hockey, with the ego agent blocking and the opponent striking.
- The opponent aims left, middle, or right and selects its next mode based on the ego agent’s final position in the previous interaction.
- The blocker controls lateral displacement, incurs cost for distance from the puck, and receives +1 for a successful block and 0 otherwise.
D Experimental Results: Analysis of Simulation Results
In the Lunar Lander experiments, LILI without influence converged as successfully as the Oracle, while simpler baselines achieved roughly 50% landing rates.
- LILI (no influence) and the Oracle had near-identical convergence rates and successfully learned to land on the launchpad.Both agents also modeled the opponent’s alternating coordinates.
- SAC, SLAC, and LILAC achieved landing rates near 50% by randomizing directions or consistently choosing the same path.The other agent’s simple dynamics made these baseline behaviors similarly effective.
- LILAC did not outperform SAC or SLAC despite the environment matching its assumptions.The passage attributes this possibility to optimization challenges from regularizing the encoder to a learned prior.
D.2 Driving (2D)
In simulated driving, LILI converged to an optimal solution with near-Oracle sample efficiency, while more complex CARLA interactions exposed distinct baseline failures and learning challenges.
- Driving (2D): LILI converged toward an optimal solution with near-equivalent sample efficiency to the Oracle in simulated driving.SAC and LILAC avoided rear-end collisions but randomized passing lanes, approaching the 50% upper bound.
- Driving (2D): SLAC persistently failed because it consistently attempted to pass from the same lane.
- Driving (2D): Table 2 varies Gaussian noise in the other agent’s step size across σ = 0.0v, 0.2v, 0.4v, 0.6v, 0.8v, and 1.0v.
- Driving (CARLA): CARLA required agents to avoid lane-boundary violations, prevent rear-end collisions, and predict opponent lane switches.Early interaction terminations and increased environmental complexity created several stages of learning difficulty.
- Driving (CARLA): LILI eventually learned to pass successfully on every interaction, whereas SLAC and LILAC converged toward repeated local minima and collided with the opponent.SAC approached the no-information upper bound by mixing its passing strategy.
E Experimental Results: More Complex Strategies
LILI remains effective when other-agent strategies include noisy actions or longer interaction histories, although performance declines as noise and history length increase.
- Agent Strategies with Noise: The noise experiment evaluated Gaussian perturbations to the other agent’s step size at σ = 0.2v, 0.4v, 0.6v, 0.8v, and 1.0v.
- Agent Strategies with History Dependence: For history-dependent strategies, the other agent used the majority of the previous N interactions to choose its movement direction.The tested history lengths were N = 3, 5, 7, 9, and 11, and LILI encoded the last N trajectories.
- Agent Strategies with History Dependence: The authors expect short histories to be more typical in human interaction because of bounded rationality.The experiments therefore emphasize strategies that can be inferred from a short interaction history.
- Agent Strategies with History Dependence: LILI’s final performance degraded as N increased, remained better than SAC for smaller N, and matched SAC at N = 11.The passage attributes the degradation to increasingly difficult latent-strategy modeling.