PaperlayerView exact revision on arXiv

Source-linked AI summary

Imitation Learning via Off-Policy Distribution Matching

Ilya Kostrikov, Ofir Nachum, Jonathan Tompson

arXiv:1912.05032v1cs.LGstat.ML

TL;DR

Imitation learning seeks to learn behavior from expert demonstrations without explicit rewards, but traditional distribution matching relies on on-policy data or changes the target objective. ValueDICE reformulates divergence minimization off-policy, learns the imitation policy directly without separate RL or explicit rewards, and achieves strong performance and sample efficiency across benchmark settings.

  • Problem

    Imitation learning avoids designing explicit rewards, but traditional distribution-ratio estimation requires on-policy samples, while replay-buffer alternatives alter the target distribution-matching objective.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Method

    ValueDICE transforms KL-divergence minimization into an off-policy objective over a value function and directly computes policy gradients, eliminating separate RL optimization and explicit rewards.

    Source: §1 · p. 2, §3 · p. 4, §4 · p. 5

  • Results

    ValueDICE shows strong performance and faster convergence across MuJoCo tasks, including extremely low-data settings, and outperforms state-of-the-art baselines on standard benchmarks.

    Source: §7.2 · p. 9, §8 · p. 9

  • Takeaways & Limitations

    The method provides principled off-policy divergence minimization while simplifying distribution-matching imitation learning to optimization over the policy and value function.

    Source: §3 · p. 4, §4 · p. 5

    Prior off-policy methods such as DAC may achieve good empirical results but provide no guarantee that the learned policy recovers the desired expert distribution after changing the objective.

    Source: §1 · p. 2, §2.2 · p. 3

Abstract

from arXiv · show

When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data-inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

1 INTRODUCTION

Imitation learning learns behavior from expert demonstrations without explicit rewards, while distribution-matching methods compare expert and behavior state-action distributions. ValueDICE reformulates this objective off-policy and learns the policy directly, avoiding explicit rewards and separate RL optimization.

Motivation

  • Imitation learning learns behavior policies from expert demonstrations without requiring explicit knowledge of rewards.Expert demonstrations are logged experience from a near-optimal policy interacting with the environment.Source: §1 · p. 1

Distribution Matching

  • Distribution matching frames imitation as minimizing divergence between the expert target distribution and the behavior policy’s induced state-action distribution.The demonstrations are treated as finite samples from the target distribution.Source: §1 · p. 1

Problem

  • Existing distribution-ratio estimation typically requires behavior-policy samples, forcing new environment interactions at every policy update.This creates a data-efficiency problem when environment interactions are expensive or limited.Source: §1 · p. 1
  • Off-policy alternatives reuse logged data but alter the original divergence objective, so the learned policy is not guaranteed to recover the desired target distribution.These methods measure divergence against the replay-buffer distribution instead of necessarily the behavior-policy distribution.Source: §1 · p. 2

ValueDICE

  • ValueDICE derives a completely off-policy divergence-minimization objective and learns the imitation policy directly without explicit rewards or separate RL optimization.The method computes policy gradients directly from the derived objective and reports state-of-the-art performance on imitation-learning benchmarks.Source: §1 · p. 2

2 BACKGROUND

The background contrasts behavioral cloning with distribution matching for imitation learning. Behavioral cloning avoids environment interactions but suffers from distributional drift, whereas distribution matching targets state-action distributions and traditionally relies on on-policy data and alternating optimization.

MDP Setting

  • An MDP comprises state and action spaces, an initial-state distribution, transition dynamics, rewards, and a discount factor.A behavior policy interacts with the environment to generate state, action, reward, and next-state experience.Source: §2 · p. 2

Imitation Learning

  • Imitation learning observes expert transition demonstrations and seeks a behavior policy that recovers the unknown expert policy without observing environment rewards.Additional environment interaction may be available, but rewards are not observed in those interactions.Source: §2 · p. 2

Behavioral Cloning

  • Behavioral cloning fits a mapping from state observations to expert actions, commonly by minimizing the negative log-likelihood of demonstrated actions.The mapping may be learned using regression or density estimation.Source: §2.1 · p. 2
  • Behavioral cloning avoids additional policy-environment interactions but cannot teach recovery after the policy reaches states absent from the demonstrations.This limitation is described as distributional drift.Source: §2.1 · p. 2

Distribution Matching

  • Distribution matching targets the expert and behavior state-action distributions, which correspond one-to-one with policies under certain conditions.The approach is described as robust to distributional shift.Source: §2.2 · p. 3
  • KL-based distribution matching converts imitation into return maximization using log distribution ratios as rewards, estimated with a GAN-like discriminator objective.Prior methods alternate discriminator updates with policy updates and require on-policy samples from the behavior distribution.Source: §2.2 · p. 3

Limitations

  • Prior distribution-matching methods are limited by on-policy data requirements and a separate RL optimization between distribution-ratio estimation steps.Replay-buffer substitutions can lose guarantees of matching the expert distribution, especially when the buffer differs substantially from the current behavior distribution.Source: §2.2 · p. 3

3 OFF-POLICY FORMULATION OF THE KL-DIVERGENCE

The paper reformulates KL divergence minimization so distribution ratios can be estimated entirely off-policy while still targeting the divergence between the policy and expert distributions. The resulting objective uses expert and initial-state expectations, avoiding replay-buffer substitution and enabling direct policy optimization.

KL reformulation

  • The Donsker–Varadhan representation expresses the negative KL divergence as an optimization over a function x whose optimum is the log state-action distribution ratio.The optimal x equals the log ratio up to an additive constant, which is ignored in the infinite-horizon setting.Source: §3 · p. 4
  • A change of variables x = ν − B_πν transforms the objective into one involving a value-like function ν and Bellman operator B_π.The transformation is chosen to exploit linearity in the second expectation.Source: §3 · p. 4

Off-policy objective

  • The transformed objective replaces the policy-distribution expectation with an initial-state expectation after the second term telescopes.The resulting expression contains a log expectation over expert state-action samples and an expectation over initial states and policy actions.Source: §3 · p. 4
  • The objective requires no on-policy samples and estimates the proper divergence between d_π and d_exp rather than between a replay buffer and expert distributions.This avoids the replay-buffer mismatch that can produce poor behavior when d_RB is far from d_π.Source: §3 · p. 4

4 VALUEDICE: IMITATION LEARNING WITH IMPLICIT REWARDS

ValueDICE jointly learns the policy and a value-like function from a max-min objective, eliminating the separate RL optimization and explicit rewards used in traditional distribution matching. Its policy gradients implement the corresponding RL update through the learned ν function.

Direct policy learning

  • The change of variables is valid under a bounded log distribution-ratio assumption, with x restricted to a bounded function family.Specifically, log d_π(s,a)/d_exp(s,a) must lie in a bounded subset K.Source: §4 · p. 4
  • ValueDICE formulates policy learning as a max-min optimization over policy π and function ν using the off-policy JDICE objective.The inner ν optimization supports direct computation of policy gradients.Source: §4 · p. 5

Implicit rewards

  • At optimality, ν* acts like a Q-value for rewards given by the log ratio between policy and expert state-action distributions.Policy gradients minimize ν*, thereby maximizing future discounted expert-to-policy log ratios.Source: §4 · p. 5
  • ValueDICE performs the RL optimization through policy gradients of the same objective used for distribution matching.The value function ν is learned directly rather than as a critic in a separate actor-critic algorithm.Source: §4 · p. 5
  • ValueDICE removes explicit rewards and the need for a separate RL algorithm, leaving an implementation in terms of π and ν.For continuous control, re-parameterization can compute gradients with respect to Gaussian policy mean and variance; practice alternates ν and π updates.Source: §4 · p. 5

5 SOME PRACTICAL CONSIDERATIONS

The practical implementation uses finite expert and initial-state samples, accepts biased mini-batch estimates for two nonlinear expectations, and can regularize training with replay-buffer data without changing the expert optimum. Virtual trajectories increase state coverage while preserving objective optimality.

Finite-sample optimization

  • The empirical objective contains expectations over expert data, initial states, and environment transitions, requiring finite-sample approximations.The expert and transition terms involve logarithms outside expected exponentials, whereas the initial-state term is linear.Source: §5 · p. 5, §5.1 · pp. 5–6
  • The experiments use biased mini-batch estimates for expert and transition expectations, while standard mini-batch gradients remain unbiased for the initial-state term.These choices were sufficient for good benchmark performance in the considered domains.Source: §5.1 · p. 6

Replay-buffer regularization

  • Replay-buffer regularization mixes expert and replay distributions as d_mix = (1 − α)d_exp + αd_RB, adding ν-values on non-expert samples.This can provide learning signals when the policy deviates from expert trajectories.Source: §5.2 · p. 6
  • For any α < 1, the global optimum remains π = π_exp; in practice, the method uses α = 0.1.The mixed objective therefore adds replay-buffer regularization without altering the expert-matching optimum.Source: §5.2 · p. 6

Initial-state sampling

  • Treating every state in a trajectory as an initial state creates virtual trajectories and expands the initial-state distribution to include all trajectory states.In Markovian environments, this modification does not affect objective optimality with respect to π.Source: §5.3 · p. 7

6 RELATED WORK

Prior adversarial imitation methods largely estimate distribution ratios from on-policy samples, while earlier off-policy variants replace the target divergence with one involving replay data. ValueDICE instead learns the policy by directly minimizing an off-policy KL formulation.

Prior AIL methods

  • Many AIL extensions retain GAIL’s distribution-ratio estimation form and therefore require on-policy samples.This includes several applications of the adversarial imitation framework.Source: §6 · p. 7

Prior off-policy methods

  • Earlier off-policy approaches train discriminators on replay-buffer samples, changing the estimated divergence from expert-versus-policy to expert-versus-replay.This alteration distinguishes those methods from the paper’s formulation.Source: §6 · p. 7

ValueDICE distinctions

  • ValueDICE uses DualDICE-inspired techniques to learn a policy directly from an off-policy KL objective rather than only estimating data distributions for a fixed policy.It uses the Donsker–Varadhan KL form and trains policy and value with the same objective, unlike REPS’s bi-level optimization.Source: §6 · p. 7

7 EXPERIMENTS

ValueDICE is evaluated on a sparse synthetic Ring MDP and MuJoCo benchmarks, where it reduces divergence and performs strongly with both very limited and larger demonstration sets.

Ring MDP

  • ValueDICE training decreases the true KL divergence D_KL(dπ||dexp) for a stochastic expert.The synthetic MDP permits direct measurement of this divergence during training.Source: §7.1 · p. 8, Figure 1 · p. 8

MuJoCo benchmarks

  • With one expert trajectory, ValueDICE performs similarly to or better than DAC on all tasks except Walker2d, where it converges to a slightly worse policy.Behavioral cloning usually cannot recover the expert policy in this low-data regime.Source: Figure 2 · p. 8, §7.2 · p. 9
  • With larger demonstration sets, ValueDICE maintains strong performance and converges faster on all tasks, while behavioral cloning can recover expert performance.Figure 3 reports results given 10 expert trajectories.Source: Figure 3 · p. 8, §7.2 · p. 9
  • GAIL is too sample-inefficient to make progress in the evaluated low- and high-data scenarios.The comparisons use standard MuJoCo environments and report returns across multiple seeds.Source: §7.2 · pp. 8–9

8 CONCLUSION

The paper concludes that ValueDICE performs robust divergence minimization off-policy while directly learning a Q-function without explicitly learning or defining rewards.

8 CONCLUSION

  • ValueDICE outperforms the state-of-the-art on standard MuJoCo imitation-learning tasks.The paper also reports increased performance over baselines in both low- and high-data regimes.Source: §8 · p. 9
  • ValueDICE performs divergence minimization in a principled off-policy manner and directly learns a Q-function within the distribution-ratio objective.This avoids a separate explicit reward-learning or reward-definition step.Source: §8 · p. 9
  • The paper demonstrates ValueDICE on a synthetic tabular MDP and standard MuJoCo continuous-control benchmarks.The evaluations cover both low- and high-data regimes.Source: §8 · p. 9

A IMPLEMENTATION DETAILS

The implementation uses common neural architectures, specified optimization rates, regularization, gradient penalties, and four updates per environment step.

A IMPLEMENTATION DETAILS

B ALGORITHMS

The algorithms are implemented from DualDICE pseudocode using expert and policy replay buffers, sampled transitions, and alternating objective updates.

B ALGORITHMS

  • The algorithm pseudocode is based on DualDICE and takes an expert replay buffer as input.The procedure initializes a separate replay buffer for policy-generated experience.Source: B ALGORITHMS · p. 11
  • Policy interactions generate transitions that are added to the replay buffer.The sampled transition is represented as (s, a, s′).Source: B ALGORITHMS · p. 11
  • The procedure samples starting states and next actions from the policy for batch transitions.The pseudocode includes policy actions at next states and starting episode states.Source: B ALGORITHMS · p. 11
  • The algorithm computes separate losses on expert data and replay-buffer data.The displayed pseudocode includes expert-data and replay-buffer loss computations involving νψ.Source: B ALGORITHMS · p. 11
  • The parameters ψ and θ are updated using gradients of the difference between logarithmic and linear objective terms.The pseudocode specifies updates for both ψ and θ after loss computation.Source: B ALGORITHMS · p. 11

C ADDITIONAL EXPERIMENTS

In the offline regime, ValueDICE outperforms behavioral cloning using only one trajectory, even without replay regularization.

  • ValueDICE outperforms behavioral cloning in the offline regime without sampling additional environment transitions.Source: C ADDITIONAL EXPERIMENTS · p. 11
  • Behavioral cloning uses the same regularization as ValueDICE’s actor training in this comparison.Source: C ADDITIONAL EXPERIMENTS · p. 11
  • With 1 trajectory, ValueDICE still outperforms behavioral cloning even without replay regularization.Source: Figure 4 · p. 11

Related papers

Loading 1912.05032v1…