Source-linked AI summary
Imitation Learning via Off-Policy Distribution Matching
Ilya Kostrikov, Ofir Nachum, Jonathan Tompson
TL;DR
Imitation learning seeks to learn behavior from expert demonstrations without explicit rewards, but traditional distribution matching relies on on-policy data or changes the target objective. ValueDICE reformulates divergence minimization off-policy, learns the imitation policy directly without separate RL or explicit rewards, and achieves strong performance and sample efficiency across benchmark settings.
Problem
Imitation learning avoids designing explicit rewards, but traditional distribution-ratio estimation requires on-policy samples, while replay-buffer alternatives alter the target distribution-matching objective.
Method
ValueDICE transforms KL-divergence minimization into an off-policy objective over a value function and directly computes policy gradients, eliminating separate RL optimization and explicit rewards.
Results
ValueDICE shows strong performance and faster convergence across MuJoCo tasks, including extremely low-data settings, and outperforms state-of-the-art baselines on standard benchmarks.
Takeaways & Limitations
The method provides principled off-policy divergence minimization while simplifying distribution-matching imitation learning to optimization over the policy and value function.
Takeaways & Limitations
Prior off-policy methods such as DAC may achieve good empirical results but provide no guarantee that the learned policy recovers the desired expert distribution after changing the objective.
Abstract
from arXiv · showhide
When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data-inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.
1 INTRODUCTION
Imitation learning learns behavior from expert demonstrations without explicit rewards, while distribution-matching methods compare expert and behavior state-action distributions. ValueDICE reformulates this objective off-policy and learns the policy directly, avoiding explicit rewards and separate RL optimization.
- Motivation: Imitation learning learns behavior policies from expert demonstrations without requiring explicit knowledge of rewards.Expert demonstrations are logged experience from a near-optimal policy interacting with the environment.
- Distribution Matching: Distribution matching frames imitation as minimizing divergence between the expert target distribution and the behavior policy’s induced state-action distribution.The demonstrations are treated as finite samples from the target distribution.
- Problem: Existing distribution-ratio estimation typically requires behavior-policy samples, forcing new environment interactions at every policy update.This creates a data-efficiency problem when environment interactions are expensive or limited.
- Problem: Off-policy alternatives reuse logged data but alter the original divergence objective, so the learned policy is not guaranteed to recover the desired target distribution.These methods measure divergence against the replay-buffer distribution instead of necessarily the behavior-policy distribution.
- ValueDICE: ValueDICE derives a completely off-policy divergence-minimization objective and learns the imitation policy directly without explicit rewards or separate RL optimization.The method computes policy gradients directly from the derived objective and reports state-of-the-art performance on imitation-learning benchmarks.
2 BACKGROUND
The background contrasts behavioral cloning with distribution matching for imitation learning. Behavioral cloning avoids environment interactions but suffers from distributional drift, whereas distribution matching targets state-action distributions and traditionally relies on on-policy data and alternating optimization.
- MDP Setting: An MDP comprises state and action spaces, an initial-state distribution, transition dynamics, rewards, and a discount factor.A behavior policy interacts with the environment to generate state, action, reward, and next-state experience.
- Imitation Learning: Imitation learning observes expert transition demonstrations and seeks a behavior policy that recovers the unknown expert policy without observing environment rewards.Additional environment interaction may be available, but rewards are not observed in those interactions.
- Behavioral Cloning: Behavioral cloning fits a mapping from state observations to expert actions, commonly by minimizing the negative log-likelihood of demonstrated actions.The mapping may be learned using regression or density estimation.
- Behavioral Cloning: Behavioral cloning avoids additional policy-environment interactions but cannot teach recovery after the policy reaches states absent from the demonstrations.This limitation is described as distributional drift.
- Distribution Matching: Distribution matching targets the expert and behavior state-action distributions, which correspond one-to-one with policies under certain conditions.The approach is described as robust to distributional shift.
- Distribution Matching: KL-based distribution matching converts imitation into return maximization using log distribution ratios as rewards, estimated with a GAN-like discriminator objective.Prior methods alternate discriminator updates with policy updates and require on-policy samples from the behavior distribution.
- Limitations: Prior distribution-matching methods are limited by on-policy data requirements and a separate RL optimization between distribution-ratio estimation steps.Replay-buffer substitutions can lose guarantees of matching the expert distribution, especially when the buffer differs substantially from the current behavior distribution.
3 OFF-POLICY FORMULATION OF THE KL-DIVERGENCE
The paper reformulates KL divergence minimization so distribution ratios can be estimated entirely off-policy while still targeting the divergence between the policy and expert distributions. The resulting objective uses expert and initial-state expectations, avoiding replay-buffer substitution and enabling direct policy optimization.
- KL reformulation: The Donsker–Varadhan representation expresses the negative KL divergence as an optimization over a function x whose optimum is the log state-action distribution ratio.The optimal x equals the log ratio up to an additive constant, which is ignored in the infinite-horizon setting.
- KL reformulation: A change of variables x = ν − B_πν transforms the objective into one involving a value-like function ν and Bellman operator B_π.The transformation is chosen to exploit linearity in the second expectation.
- Off-policy objective: The transformed objective replaces the policy-distribution expectation with an initial-state expectation after the second term telescopes.The resulting expression contains a log expectation over expert state-action samples and an expectation over initial states and policy actions.
- Off-policy objective: The objective requires no on-policy samples and estimates the proper divergence between d_π and d_exp rather than between a replay buffer and expert distributions.This avoids the replay-buffer mismatch that can produce poor behavior when d_RB is far from d_π.
4 VALUEDICE: IMITATION LEARNING WITH IMPLICIT REWARDS
ValueDICE jointly learns the policy and a value-like function from a max-min objective, eliminating the separate RL optimization and explicit rewards used in traditional distribution matching. Its policy gradients implement the corresponding RL update through the learned ν function.
- Direct policy learning: The change of variables is valid under a bounded log distribution-ratio assumption, with x restricted to a bounded function family.Specifically, log d_π(s,a)/d_exp(s,a) must lie in a bounded subset K.
- Direct policy learning: ValueDICE formulates policy learning as a max-min optimization over policy π and function ν using the off-policy JDICE objective.The inner ν optimization supports direct computation of policy gradients.
- Implicit rewards: At optimality, ν* acts like a Q-value for rewards given by the log ratio between policy and expert state-action distributions.Policy gradients minimize ν*, thereby maximizing future discounted expert-to-policy log ratios.
- Implicit rewards: ValueDICE performs the RL optimization through policy gradients of the same objective used for distribution matching.The value function ν is learned directly rather than as a critic in a separate actor-critic algorithm.
- Implicit rewards: ValueDICE removes explicit rewards and the need for a separate RL algorithm, leaving an implementation in terms of π and ν.For continuous control, re-parameterization can compute gradients with respect to Gaussian policy mean and variance; practice alternates ν and π updates.
5 SOME PRACTICAL CONSIDERATIONS
The practical implementation uses finite expert and initial-state samples, accepts biased mini-batch estimates for two nonlinear expectations, and can regularize training with replay-buffer data without changing the expert optimum. Virtual trajectories increase state coverage while preserving objective optimality.
- Finite-sample optimization: The empirical objective contains expectations over expert data, initial states, and environment transitions, requiring finite-sample approximations.The expert and transition terms involve logarithms outside expected exponentials, whereas the initial-state term is linear.
- Finite-sample optimization: The experiments use biased mini-batch estimates for expert and transition expectations, while standard mini-batch gradients remain unbiased for the initial-state term.These choices were sufficient for good benchmark performance in the considered domains.
- Replay-buffer regularization: Replay-buffer regularization mixes expert and replay distributions as d_mix = (1 − α)d_exp + αd_RB, adding ν-values on non-expert samples.This can provide learning signals when the policy deviates from expert trajectories.
- Replay-buffer regularization: For any α < 1, the global optimum remains π = π_exp; in practice, the method uses α = 0.1.The mixed objective therefore adds replay-buffer regularization without altering the expert-matching optimum.
- Initial-state sampling: Treating every state in a trajectory as an initial state creates virtual trajectories and expands the initial-state distribution to include all trajectory states.In Markovian environments, this modification does not affect objective optimality with respect to π.
6 RELATED WORK
Prior adversarial imitation methods largely estimate distribution ratios from on-policy samples, while earlier off-policy variants replace the target divergence with one involving replay data. ValueDICE instead learns the policy by directly minimizing an off-policy KL formulation.
- Prior AIL methods: Many AIL extensions retain GAIL’s distribution-ratio estimation form and therefore require on-policy samples.This includes several applications of the adversarial imitation framework.
- Prior off-policy methods: Earlier off-policy approaches train discriminators on replay-buffer samples, changing the estimated divergence from expert-versus-policy to expert-versus-replay.This alteration distinguishes those methods from the paper’s formulation.
- ValueDICE distinctions: ValueDICE uses DualDICE-inspired techniques to learn a policy directly from an off-policy KL objective rather than only estimating data distributions for a fixed policy.It uses the Donsker–Varadhan KL form and trains policy and value with the same objective, unlike REPS’s bi-level optimization.
7 EXPERIMENTS
ValueDICE is evaluated on a sparse synthetic Ring MDP and MuJoCo benchmarks, where it reduces divergence and performs strongly with both very limited and larger demonstration sets.
- Ring MDP: ValueDICE training decreases the true KL divergence D_KL(dπ||dexp) for a stochastic expert.The synthetic MDP permits direct measurement of this divergence during training.
- MuJoCo benchmarks: With one expert trajectory, ValueDICE performs similarly to or better than DAC on all tasks except Walker2d, where it converges to a slightly worse policy.Behavioral cloning usually cannot recover the expert policy in this low-data regime.
- MuJoCo benchmarks: With larger demonstration sets, ValueDICE maintains strong performance and converges faster on all tasks, while behavioral cloning can recover expert performance.Figure 3 reports results given 10 expert trajectories.
- MuJoCo benchmarks: GAIL is too sample-inefficient to make progress in the evaluated low- and high-data scenarios.The comparisons use standard MuJoCo environments and report returns across multiple seeds.
8 CONCLUSION
The paper concludes that ValueDICE performs robust divergence minimization off-policy while directly learning a Q-function without explicitly learning or defining rewards.
- 8 CONCLUSION: ValueDICE outperforms the state-of-the-art on standard MuJoCo imitation-learning tasks.The paper also reports increased performance over baselines in both low- and high-data regimes.
- 8 CONCLUSION: ValueDICE performs divergence minimization in a principled off-policy manner and directly learns a Q-function within the distribution-ratio objective.This avoids a separate explicit reward-learning or reward-definition step.
- 8 CONCLUSION: The paper demonstrates ValueDICE on a synthetic tabular MDP and standard MuJoCo continuous-control benchmarks.The evaluations cover both low- and high-data regimes.
A IMPLEMENTATION DETAILS
The implementation uses common neural architectures, specified optimization rates, regularization, gradient penalties, and four updates per environment step.
- A IMPLEMENTATION DETAILS: All algorithms use MLP networks with two hidden layers and 256 hidden units.The architecture specification applies across the evaluated algorithms.
- A IMPLEMENTATION DETAILS: Discriminators, critics, and ν use Adam with learning rate 10^-3, while actors use learning rate 10^-5.These are the stated optimizer settings for the respective network groups.
- A IMPLEMENTATION DETAILS: The discriminator and ν networks use gradient penalties, and the actor uses orthogonal regularization with coefficient 10^-4.The gradient penalties follow Gulrajani et al. (2017).
- A IMPLEMENTATION DETAILS: The algorithms perform four updates per environment step.This is the stated update-to-interaction ratio.
B ALGORITHMS
The algorithms are implemented from DualDICE pseudocode using expert and policy replay buffers, sampled transitions, and alternating objective updates.
- B ALGORITHMS: The algorithm pseudocode is based on DualDICE and takes an expert replay buffer as input.The procedure initializes a separate replay buffer for policy-generated experience.
- B ALGORITHMS: Policy interactions generate transitions that are added to the replay buffer.The sampled transition is represented as (s, a, s′).
- B ALGORITHMS: The procedure samples starting states and next actions from the policy for batch transitions.The pseudocode includes policy actions at next states and starting episode states.
- B ALGORITHMS: The algorithm computes separate losses on expert data and replay-buffer data.The displayed pseudocode includes expert-data and replay-buffer loss computations involving νψ.
- B ALGORITHMS: The parameters ψ and θ are updated using gradients of the difference between logarithmic and linear objective terms.The pseudocode specifies updates for both ψ and θ after loss computation.
C ADDITIONAL EXPERIMENTS
In the offline regime, ValueDICE outperforms behavioral cloning using only one trajectory, even without replay regularization.
- ValueDICE outperforms behavioral cloning in the offline regime without sampling additional environment transitions.
- Behavioral cloning uses the same regularization as ValueDICE’s actor training in this comparison.
- With 1 trajectory, ValueDICE still outperforms behavioral cloning even without replay regularization.