Source-linked AI summary
Generative Adversarial Imitation from Observation
Faraz Torabi, Garrett Warnell, Peter Stone
TL;DR
Imitation from observation seeks to learn tasks from state-only demonstrations, addressing the absence of demonstrator actions in conventional imitation learning. The paper introduces a state-transition-based framework and GAIfO, which performs comparably to action-informed imitation learning and favorably against existing IfO methods in the reported experiments.
Problem
Imitation from observation addresses learning from state-only demonstrations when demonstrator actions are unavailable, a limitation of most existing imitation-learning approaches.
Method
The paper formulates IfO using costs over state transitions and introduces the model-free generative adversarial imitation from observation algorithm GAIfO.
Results
GAIfO performs comparably to conventional imitation-learning methods with action access and favorably against recently developed IfO methods across low-dimensional and raw-visual settings.
Takeaways & Limitations
GAIfO finds imitation policies without action information while performing close to policies learned by techniques that use that information.
Takeaways & Limitations
In visual settings, performance below the expert may reflect a fundamental limitation of learning from visual data, namely partial state observability.
Abstract
from arXiv · showhide
Imitation from observation (IfO) is the problem of learning directly from state-only demonstrations without having access to the demonstrator's actions. The lack of action information both distinguishes IfO from most of the literature in imitation learning, and also sets it apart as a method that may enable agents to learn from a large set of previously inapplicable resources such as internet videos. In this paper, we propose both a general framework for IfO approaches and also a new IfO approach based on generative adversarial networks called generative adversarial imitation from observation (GAIfO). We conduct experiments in two different settings: (1) when demonstrations consist of low-dimensional, manually-defined state features, and (2) when demonstrations consist of high-dimensional, raw visual data. We demonstrate that our approach performs comparably to classical imitation learning approaches (which have access to the demonstrator's actions) and significantly outperforms existing imitation from observation methods in high-dimensional simulation environments.
1. Introduction
Imitation learning addresses reward-design difficulty and slow individual experience by learning from expert demonstrations, but conventional approaches usually require demonstrator actions. This paper develops a state-transition-based framework and GAIfO to learn control directly from state-only demonstrations.
- Reinforcement learning can improve behavior from reward-based experience, but designing feedback for complex tasks is difficult and learning solely from experience can be slow.
- Imitation learning avoids explicit reward-function design by communicating task information through expert demonstrations and is typically faster than reinforcement learning.
- Most imitation-learning research assumes access to both demonstrator states and actions, limiting use of state-only resources such as online videos.
- Existing imitation-from-observation work emphasizes visual perception while often solving control with reinforcement learning over a predefined reward function.
- The paper proposes a state-transition-cost framework and GAIfO, evaluating them with manually defined features and raw visual observations.
- The experiments compare GAIfO favorably with recent imitation-from-observation methods and comparably with conventional imitation learning that uses demonstrator actions.
2. Related Work
Imitation learning includes behavioral cloning and inverse reinforcement learning, but most existing methods require expert actions. Imitation from observation addresses state-only demonstrations through inverse-dynamics, behavioral-cloning, and representation-learning approaches, each with distinct limitations.
- Imitation learning broadly divides into behavioral cloning, which maps states directly to actions, and inverse reinforcement learning, which learns a demonstrator’s cost function.
- GAIL uses generative adversarial networks to bring the imitator’s and demonstrator’s state-action distributions closer together.
- Most imitation learning methods require expert actions, motivating imitation from observation when demonstrations provide only state trajectories.
- Inverse-dynamics approaches infer demonstrator actions from state-only demonstrations, either reproducing one demonstration exactly or using inferred actions for behavioral cloning and generalization.
- Representation-learning approaches use surrogate rewards based on encoded state proximity, while differing in invariance to embodiments, viewpoints, and contexts.
3. Preliminaries
The preliminaries formulate imitation learning in Markov decision processes and describe inverse reinforcement learning as cost-function learning followed by reinforcement learning. GAIL replaces this two-stage procedure with direct policy learning through adversarial discrimination of state-action pairs.
- In the MDP formulation, states and actions are connected by a policy, environment transition dynamics, and a state-action cost function.
- Inverse reinforcement learning learns a cost function that favors expert trajectories over those generated by other policies, then optimizes a policy using that cost.
- GAIL directly learns the optimal policy by training a classifier to distinguish demonstrator and imitator state-action pairs and bringing their distributions closer.
4. A General Framework for Imitation from Observation
Imitation from observation changes the objective from matching expert actions to matching their effects on the environment. This framework therefore represents costs over state transitions, learns a transition-based cost, and uses reinforcement learning to obtain the imitation policy.
- Imitation from observation seeks actions that have the same effects on the environment as the demonstrator’s actions, rather than identical actions.
- The framework defines the cost over state transitions, c : S × S →R, instead of over state-action pairs.
- Inverse reinforcement learning from observation produces a transition-based cost ˜c from expert state-only trajectories.
- The theoretical analysis omits the entropy term, while the paper leaves detailed analysis of its effect to future work.
- An RL algorithm then optimizes ˜c to produce the imitation policy ˜π.
5. Generative Adversarial Imitation from Observation
GAIfO instantiates imitation from observation by matching the expert’s state-transition occupancy measure with a generative-adversarial objective. Its discriminator distinguishes expert from imitator transitions while the policy learns to make generated transitions indistinguishable.
- GAIfO is a specific algorithm derived from the paper’s general imitation-from-observation framework.
- The state-transition occupancy measure represents the distribution of transitions encountered under a policy.
- The framework and algorithm target matching transition occupancy measures, although different policies can induce the same measure.The paper considers this acceptable because IfO primarily concerns the policy’s effect on the environment.
- The practical algorithm alternates discriminator updates with policy updates, using TRPO for the policy optimization.
- The algorithm compares expert and imitator state-transition distributions rather than state-action distributions.The substantive difference from the related adversarial derivation is replacing (s, a) with (s, s′).
- The discriminator classifies state transitions, while the policy generates transitions intended to fool it into treating them as expert data.The discriminator outputs values near zero for expert data and near one for imitator data.
6. Practical Implementation
The practical GAIfO implementation uses neural networks to classify and generate state transitions, then alternates their updates. This formulation is directly applicable to low-dimensional, fully observable, strictly Markovian states but requires additional treatment for visual observations.
- The discriminator is a multilayer perceptron mapping a state transition to a value between 0 and 1.The policy is a multilayer perceptron mapping a state to an action.
- GAIfO collects imitator transitions by executing its policy, then feeds those transitions to the discriminator during iterative training.The figure contrasts expert demonstration transitions with imitator-generated transitions.
- The low-dimensional implementation assumes fully observable states and a strictly Markovian system.Its policy maps a single state to an action and its reward operates on a single state transition.
- Those assumptions are not necessarily valid for visual state representations.The paper therefore treats the described implementation as effective only for low-dimensional state representations.
7. Experimental Setup and Implementation Details
The experiments use PyBullet-based OpenAI Gym domains with TRPO-trained expert demonstrations and compare GAIfO against three imitation baselines. Results are averaged over ten independent trials.
- The evaluation uses domains from OpenAI Gym based on the PyBullet simulator.
- Expert agents are trained with trust region policy optimization, and demonstrations are recorded from the resulting policies.
- Reported figure results are averages over ten independent trials.
- The comparison includes Behavioral Cloning from Observation, which infers missing actions through an inverse dynamics model before behavioral cloning.
- Time Contrastive Networks learn time-indexed state embeddings and define rewards from embedded-state distances.
- GAIL is included as a baseline with access to demonstrator actions, unlike the other listed methods.
8. Results and Discussion
Across low-dimensional and visual experiments, GAIfO generally outperformed existing imitation-from-observation methods and matched action-informed baselines in several domains. In visual experiments, GAIfO performed substantially better than BCO and TCN, while visual partial observability limited performance relative to the expert.
- Low-dimensional State Representations: GAIfO was comparable to GAIL in low-dimensional experiments despite lacking explicit action information.
- Low-dimensional State Representations: GAIfO outperformed other imitation-from-observation algorithms by a large margin in most low-dimensional experiments.In InvertedDoublePendulum, GAIfO achieved performance similar to the expert.
- Low-dimensional State Representations: GAIfO and GAIL performed as well as the expert in InvertedPendulumSwingup, whereas TCN and BCO performed poorly.The authors associate TCN's weakness with task encoding and BCO's with compounding error from consecutive action requirements.
- Visual State Representations: GAIfO outperformed BCO and TCN by a large margin in visual experiments.The comparison used visual-state TRPO as a reward-informed reference, while performance was scaled so random and expert policies scored 0.0 and 1.0.
- Visual State Representations: In visual experiments, GAIfO reached the performance of a TRPO-trained agent using visual states but did not reach the expert's performance.The authors suggest this gap may reflect partial state observability in visual data.
- Visual State Representations: BCO and TCN performed poorly in the visual domains, potentially because of visual dynamics-model complexity, compounding error, or unsynchronized demonstrations.These explanations are presented as hypotheses rather than established causes.
9. Conclusion and Future Work
The paper presents GAIfO as an imitation-from-observation approach that learns better policies without action information and approaches the performance of action-informed techniques. Its analysis leaves policy entropy terms for future investigation.
- GAIfO finds better imitation policies without action information and performs very close to techniques that access demonstrator actions.
- GAIfO removes restrictive assumptions required by some other imitation-from-observation techniques, including time-synchronized demonstrations.
- The analysis does not include policy entropy terms in either the IRLfO step or the reinforcement-learning step.The paper identifies incorporating entropy and investigating its effects as future work.