Source-linked AI summary
Imitating Latent Policies from Observation
Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, Charles L. Isbell
TL;DR
The paper addresses imitation from state observations without expert actions and with few environment interactions. ILPO learns latent policies and forward predictions from observed transitions, then aligns latent actions with real actions through limited interaction. It recovers expert behavior across classic-control and vision-based tasks and outperforms BCO, while depending on discrete actions and deterministic transitions and performing worse with deterministic demonstrations.
Problem
Imitation learning often requires expert actions and extensive environment interaction, whereas ILPO targets policy learning from expert observations with only a few interactions.
Method
ILPO learns a latent policy and forward dynamics model from observed transitions, then learns a state-dependent mapping from latent actions to true actions.
Results
ILPO recovered expert behavior across four classic-control and vision-based domains, performed better than BCO, and matched expert performance in several tasks after few interactions.
Takeaways & Limitations
Learning from observations followed by a small number of action-alignment interactions can support imitation when expert observations are available but unguided exploration is risky.
Takeaways & Limitations
The formulation assumes discrete actions and deterministic state transitions, and performance decreases with deterministic demonstrations.
Abstract
from arXiv · showhide
In this paper, we describe a novel approach to imitation learning that infers latent policies directly from state observations. We introduce a method that characterizes the causal effects of latent actions on observations while simultaneously predicting their likelihood. We then outline an action alignment procedure that leverages a small amount of environment interactions to determine a mapping between the latent and real-world actions. We show that this corrected labeling can be used for imitating the observed behavior, even though no expert actions are given. We evaluate our approach within classic control environments and a platform game and demonstrate that it performs better than standard approaches. Code for this work is available at https://github.com/ashedwards/ILPO.
1. Introduction
The paper targets imitation from state observations without expert actions, reducing costly or risky environment interaction. It learns latent causes of observed transitions, then aligns them with real actions using limited interaction.
- Imitation learning typically requires expert observations and actions plus extensive environment interaction, unlike human learning from observed state trajectories.
- The proposed setting removes expert action access and limits environment interaction because unguided exploration can be risky and costly.
- ILPO first learns an offline latent policy from observed transitions, then uses limited environment steps to map latent actions to true controls.
- A forward dynamics model predicts next states and latent-action priors from state-transition pairs, while greedy selection chooses the most probable next-state explanation.
- Across cartpole, acrobot, mountain car, and CoinRun, ILPO matched expert performance after few interactions and outperformed Behavioral Cloning from Observation.
2. Related work
Related approaches learn from actions, state transitions, rewards, or observation-derived representations, often requiring substantial environment interaction. ILPO instead uses latent multi-modal future predictions with priors to derive policies from observations.
- Traditional imitation learning assumes expert actions are known, requiring data recorded specifically for imitation and limiting action-free learning.
- Behavioral cloning uses supervised learning, while more sophisticated methods model state transitions but may require full dynamics information or numerous environment interactions.
- GAIL and SAIL match demonstrated and agent state-action visitation frequencies using adversarial training or temporal-difference learning.
- Observation-only alternatives may require many environment samples, initial environment learning, or substantial interaction to learn inverse or forward dynamics.
- ILPO predicts a latent multi-modal distribution over future states and directly learns priors over modes, which are crucial for deriving policies.
3. Approach
ILPO learns a latent policy from expert state observations, then uses limited environment interaction to align latent transitions with true actions. The resulting policy can imitate expert behavior without expert action labels.
- 3.3.1. Latent Forward Dynamics: ILPO assumes discrete latent actions explain observed state transitions and uses expert state pairs to learn a multimodal forward dynamics model.The model predicts state differences for latent actions, allowing distinct transition types to be represented rather than collapsing them into an average prediction.
- 3.3.1. Latent Forward Dynamics: The latent dynamics model penalizes only the prediction closest to the observed next state, encouraging separate generators to learn distinct transition clusters.Penalizing every prediction would instead encourage the generators to predict the expected next state.
- 3.3.2. Latent Policy Learning: ILPO concurrently learns a latent policy that assigns probabilities to latent transitions and selects the most likely transition for a given state.Its policy loss uses the expected generated next state while holding individual predictions fixed, encouraging likely transitions to explain the observations.
- 3.4. Step 2: Action Remapping: The action-remapping stage uses limited environment interaction to learn a state-dependent mapping from latent actions to true actions.The mapping is state-dependent because the same latent transition type can correspond to different real actions in different parts of the state space.
- 3.4. Step 2: Action Remapping: At execution, ILPO chooses the most likely latent cause under its learned policy and then the most likely real action under the remapping network.This two-stage policy imitates expert behavior without observing expert actions.
4. Experiments and results
ILPO was evaluated across classic control tasks and the visual CoinRun environment using state observations, limited environment interaction, and comparisons with expert, BC, and BCO policies. It matched expert-level performance in cartpole and acrobot, outperformed BCO across reported settings, but remained below the expert in mountain car and CoinRun.
- Classic control results: ILPO matched expert and behavioral-cloning performance in cartpole and acrobot within 100 environment steps, while outperforming BCO.In cartpole, ILPO achieved this in fewer than 100 steps; acrobot showed the same pattern within 100 steps.
- Classic control results: Neither ILPO nor BCO matched the expert in mountain car, but ILPO outperformed BCO.
- Latent-action analysis: Choosing |Z| = |A| was a good initial guess, while |Z| = 1 performed poorly because all actions collapsed to one latent and state predictions could not be disentangled.The agent could also learn with other latent-action counts.
- Limitation: ILPO requires stochastic demonstrations, and its performance decreased when demonstrations were deterministic.
- CoinRun results: In CoinRun, ILPO underperformed the expert but significantly outperformed BCO on both easy and hard tasks, with occasional expert-level or zero-shot performance from favorable initialization.Results averaged across all seeds included runs with poor initialization; high-dimensional visual inputs required more alignment steps.
5. Discussion and conclusion
ILPO learns to imitate expert behavior from state observations with very few environment interactions, and the authors demonstrate this across four domains. The approach may support observation-based learning across agents with differing embodiments and action spaces.
- Results: ILPO recovered expert behavior in four classic-control and vision-based domains using only expert state observations and very few environment interactions.The authors report fewer required interactions than BCO.
- Limitations and future work: Future work should address ILPO’s assumptions that actions are discrete and state transitions are deterministic.The authors also propose improving action remapping through stronger local consistency across states.
- Extensions: The method may enable learning from agents with different embodiments whose actions are unknown or lack a known correspondence.The authors also identify transfer across environments as a possible extension.
- Extensions: ILPO could provide pre-training for imitation-by-observation methods that later use more sophisticated imitation learning procedures.The authors describe this as complementary to related approaches that use behavioral cloning for pre-training.
A. Hyperparameters
All reported experiments used the Adam optimizer.
- Optimization: The Adam optimizer was used to train ILPO, behavioral cloning, and BCO experiments.The passage states this as the shared optimizer across experiments.
A.1.1. ILPO
The ILPO implementation uses separate latent-policy and action-remapping networks, with state embeddings, latent-action conditioning, and task-specific architectures for vector and image observations.
- Latent policy: The latent-policy network uses Ep with FC128 → lrelu → FC256 and trains for 1000 epochs with batch size 32 and learning rate .0002.The latent action is one-hot encoded and combined with Ep before predicting state dimensions.
- Action remapping: The action-remapping network uses Ea with FC64 → lrelu → FC32 → FC|A| and trains for 1000 steps with batch size 32 and learning rate .002.Its policy conditions on both the current state embedding and latent action.
- Training and comparison: ILPO uses ϵ = .2 for random training actions and ϵ = 0 during evaluation.The implementation also describes state encoding and policy architectures for behavioral cloning and BCO.
- Image-based ILPO: For image observations, ILPO uses convolutional embeddings and deconvolutional generators, training for 10 epochs with batch size 100 and learning rate .0002.The image latent-policy architecture uses Conv30 through Conv120 layers and corresponding deconvolutional layers.
A.2.2. BCO
The BCO implementation uses a state-policy network without latent-action inputs and trains its inverse-dynamics and policy components through repeated environment experience.
- Architecture: BCO overfit when using the same state-encoding architecture as Ep, motivating a smaller convolutional embedding.The revised embedding uses Conv15 followed by Conv30.
- Training: BCO collects 200 experience steps per iteration and trains its inverse-dynamics and policy networks for 500 steps each over 1000 iterations.The reported learning rate is .0001, and training is described as very slow for image-based models.
- Exploration: BCO and ILPO use ϵ = .2 for random training actions and ϵ = .1 during evaluation.This setting is reported for both methods in the image-based comparison.
B. Deterministic Demonstrations
With deterministic demonstrations, ILPO still receives high reward on both classic-control tasks, although stochastic demonstrations are necessary for optimal performance.
- Stochastic demonstrations are necessary for ILPO to achieve optimal performance.Deterministic demonstrations nevertheless produce high reward on both tasks.
- Figure 7 reports classic-control results averaged over 50 ILPO trials, with policy evaluations averaged over 10 runs.