Imitation Learning as $f$-Divergence Minimization

1 Introduction

• The paper argues that learning a single mode can suffice when demonstrations contain diverse near-optimal choices, interface variability, or differences in expert skill and preferences.Examples include navigating left or right around obstacles [1] and variability in kinesthetic robot-arm demonstrations [2].Source: §1 · p. 1
• Behavior cloning can interpolate between left- and right-steering modes and drive into an obstacle; GAIL [5] shows a similar trend, while InfoGAIL [4] attempts costly recovery of all modes.The obstacle example is illustrated in Fig. 1.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Behavior cloning minimizes KL, GAIL minimizes JS, and DAgger minimizes TV; the paper attributes mode coverage and interpolation to KL and JS, versus mode-seeking subset collapse to reverse KL.The framework examines f-divergences and uses estimators from [6] because they cannot be minimized exactly.Source: §1 · p. 2
• The paper introduces a framework for minimizing f-divergences between learner and trajectory distributions, with algorithms that recover existing methods and efficiently minimize reverse KL.It argues for reverse KL on multi-modal inputs and reports that it collapses to a demonstrator mode in bandit and RL environments, unlike KL and JS interpolation.Source: §1 · p. 2

2 Related Work

• The central question is which objective captures similarity to expert demonstrations, independently of whether learning is model-based or model-free and uses policies or trajectories.Source: §2 · p. 3
• Behavior cloning treats IL as supervised action matching, but small errors can cause distribution mismatch; DAgger [17] can alleviate this, though expert labeling may be impractical [20].The section also notes counterexamples in which the DAgger objective produces undesirable behavior [22].Source: §2 · p. 3
• Reward recovery and Q-value recovery frame IL as making the expert appear optimal [23–25]; relaxing this to value or feature-expectation matching leads to maximum-entropy formulations [28].The authors note that their divergence estimators also match moments.Source: §2 · p. 3
• The paper frames imitation learning as f-divergence minimization, encompassing relative-entropy and symmetric-cross-entropy methods, a connection also independently made in.The section focuses this perspective on multi-modal expert demonstrations.Source: §2 · p. 3
• Adversarial min-max formulations connect IRL to GANs [30] [31]; GAIL [5] and SAM [33] recover policies, while AIRL [34] and EAIRL [35] recover rewards.Source: §2 · p. 3
• For multi-modal demonstrations, prior work clusters data [45], uses InfoGAN latent clusters extended to IL [4] [47], or learns sparse policies with MCTE [48] and sparse mixture-density networks [49].The section also notes that divergence choice has been studied for policy improvement in reinforcement learning [50].Source: §2 · p. 3

3 Problem Formulation

• An f-divergence uses a convex, lower-semicontinuous function f, with different choices recovering KL, Jensen-Shannon, and Total Variation divergences.The distributions must satisfy absolute continuity as specified in the definition.Source: §3 · p. 4
• The framework formalizes imitation as minimizing an f-divergence between expert and learner trajectory distributions.The target is to match the expert and learner trajectory distributions, ρπ*(τ) and ρπ(τ).Source: §3 · p. 4
• Different f-divergence choices lead to different learned policies.Source: §3 · p. 4
• Accurate trajectory-divergence estimation requires many expert samples because trajectory-space size grows exponentially with time.Source: §3 · p. 5
• Theorem 1 establishes that, for any two policies, divergence between their average state-action distributions lower-bounds divergence between their trajectory distributions.The section proposes minimizing the average state-action divergence instead of estimating trajectory divergence directly.Source: §3 · p. 5

4 Framework for Divergence Minimization

• The framework estimates state-action divergence from demonstrations, which also lower-bounds trajectory divergence, then minimizes that estimate.The expert policy is unknown, so the divergence cannot be computed exactly.Source: §4 · p. 5

4.1 Variational approximation of divergence

• Using the variational form of f, VIM estimates divergence from expert and learner samples with a discriminator-based lower bound.The bound reflects both Jensen’s inequality and restriction to the estimator class, turning divergence estimation into discriminative classification.Source: §4.1 · pp. 5–6
• The estimator class should approximate the subdifferential of f; neural discriminators can meet range constraints through a suitable activation function [6].The discriminator is parameterized as an unconstrained function followed by an activation mapping into the domain of the convex conjugate.Source: §4.1 · p. 6
• VIM jointly finds a learner and discriminator at a saddle point using expert demonstrations and learner rollouts, updating the estimator by supervised learning and the policy by gradients.Plugging in different f-divergences yields different algorithms.Source: §4.1 · p. 6

• RKL–VIM is labeled as minimizing reverse KL divergence, with constant factors removed.Source: §1 · p. 6
• Table 1 lists f-divergences, their conjugates, optimal estimators, and activation functions.Source: Table 1 · p. 7

4.2 Recovering existing imitation learning algorithms

• Behavior Cloning minimizes trajectory-distribution KL, equivalent to cross-entropy loss for multi-class classification.Source: §4.2 · p. 7
• JS–VIM is exactly the GAIL optimization with Jensen-Shannon divergence, but without the entropic regularizer [5].Source: §4.2 · p. 7
• DAgger [17] corresponds to Total Variation distance; its iterative supervised learning with an interactive expert addresses non-i.i.d. learning, while its counter-examples [22] can be explained as a divergence artifact.Source: §4.2 · p. 7

4.3 Alternate techniques for Reverse KL minimization via interactive learning

• RKL–VIM has a double lower-bound approximation, may require a complex state-action density-ratio estimator, and can converge slowly.Source: §4.3 · pp. 7–8
• With access to an interactive expert, Reverse KL can be minimized directly over action distributions on states visited by the learner, motivating two interactive I-projection techniques.Source: §4.3 · p. 8
• Variational action-divergence minimization uses fresh expert and learner data each iteration and is easier to approximate than RKL–VIM.Source: §4.3 · p. 8
• Density-ratio minimization uses an off-the-shelf estimator and dataset aggregation for non-i.i.d. learning, needs no policy gradients, and gives a global performance guarantee with an expressive policy class.Source: §4.3 · p. 8

5 Multi-modal Trajectory Demonstrations

• A restricted policy class may be unable to achieve zero divergence for the two demonstrated modes, so the objective should encourage safe failure.The example uses a policy class with fixed-variance Gaussian policies.Source: §5 · p. 8
• KL favors mode-covering and can produce interpolating behavior that crashes into the tree, while RKL favors one demonstrated turn.KL incurs an infinite penalty for failing to support demonstration actions but no explicit penalty for outlier samples; RKL instead heavily penalizes samples improbable under the demonstrator.Source: §5 · p. 9, Figure 2 · p. 9
• RKL mode-collapses to always turning left or right, depending on initialization and mode mixture, yielding a predictable and safe demonstrated trajectory.RKL has no explicit penalty for failing to cover the full expert distribution, but can impose an arbitrarily large penalty for generating improbable samples.Source: §5 · p. 9, Figure 2 · p. 9
• Jensen-Shannon divergence can lead to either mode-covering or mode-collapse depending on the MDP, demonstrations, and optimization initialization.Because JS is symmetric and expectations are taken over both distributions, unsupported states and actions leave the divergence finite; safe mode-collapse is possible but not guaranteed.Source: §5 · p. 9

6.1 Low dimensional tasks

• With enumerated policies, RKL selects a single mode (A or Right), whereas KL and JS select interpolating policies (M or Up).The comparison evaluates each policy’s exactly computed stationary distribution and f-divergence.Source: §6.1 · p. 10, Table 2 · p. 11
• As bandit control noise ϵ0 changes, RKL strongly prefers mode collapse, KL prefers mode covering, and JS lies between them.RKL’s mode-collapse preference has an exception at high control noise.Source: §6.1 · p. 10, Figure 4 · p. 11
• For preferred policies, sample-based KL and JS estimates are often much lower than their true divergences, whereas RKL underestimates less.The comparison examines policies in the 1 percentile of estimations.Source: §6.1 · p. 10, Figure 5 · p. 12
• With continuously parameterized policies optimized by policy gradients, RKL–VIM collapses to a single mode, while KL–VIM and JS–VIM do not.The experiments compare locally optimal policies in the bandit and gridworld settings.Source: §6.1 · pp. 10–11, Table 2 · p. 11

6.2 High dimensional continuous control task

• On Ant, Hopper, and HalfCheetah, RKL–VIM converges to higher reward than JS–VIM, while their performance is similar on Humanoid, Reacher, and Walker2d.The experts were single-modal; mixing α = 0.2 of the true reward into the generator loss helped avoid saddle points and produced reliable, albeit different, convergence.Source: §6.2 · pp. 11–12, Figure 6 · p. 12

7 Discussion

• The f-divergence framework generalizes behavior cloning (KL), GAIL (JS), and DAgger (TV).It provides a common framework for estimating and minimizing these divergences.Source: §7 · pp. 12–13
• For multi-modal demonstrations, reverse KL safely and efficiently collapses to a subset of modes, whereas KL and JS often produce unsafe behavior.The framework presents behavior cloning as KL, GAIL as JS, and DAgger as total variation.Source: §7 · pp. 12–13
• The framework minimizes an approximate divergence estimate, notably a lower bound that is tight when φ has enough capacity; only KL can actually be measured.The passage refers to lower bound (5).Source: §7 · p. 13
• For reverse KL, the unbounded derivative may require exponentially many samples, while no finite-sample upper bound is informative without assumptions on expert or learner distributions.Without distributional assumptions, finite samples cannot accurately estimate support; the only upper bound described is infinity.Source: §7 · p. 13
• Adding Gaussian noise to learner and expert distributions makes them absolutely continuous, upper-bounds divergence magnitude, and may speed convergence by smoothing the policy's cost [54].The discussion emphasizes that the goal is minimizing divergence, not measuring it.Source: §7 · p. 13
• Side information, such as obstacle constraints inferred from data, can supply an auxiliary penalty instead of estimating the cost of learner states the expert avoids.The proposed loss balances maximizing expert-action likelihood with penalizing learner visits to states the expert does not visit.Source: §7 · p. 13
• Future work could unify the framework with MaxEnt moment matching, which matches basis-function moments and adjusts expert and learner state costs symmetrically.By contrast, RKL-VIM exponentially lowers expert-state costs and linearly raises learner-state costs.Source: §7 · p. 13
• Integral probability metrics are another direction: they are measurable by definition, include MMD, total variation, and Earth-mover distance, and preliminary estimator results seem promising [55].They take the form of a supremum over function families φ of the difference in expected values under p and q.Source: §7 · p. 13

A Lower Bounding f-Divergence of Trajectory Distribution with State Action Distribution

• Lemma 2’s information-loss inequality assumes one variable contains all information about the other and derives the bound by factoring joint distributions and applying the generalized log-sum inequality.The proof invokes the condition equating the conditional distributions and then applies the log-sum result.Source: A Lower Bounding f-Divergence of… · pp. 18–19
• Trajectory-distribution f-divergence lower-bounds state-action-distribution divergence because discarding temporal information loses information, and it also lower-bounds state-distribution divergence.The section characterizes this as information loss when temporal detail is discarded.Source: A Lower Bounding f-Divergence of… · p. 19
• The gap between trajectory and state-action divergences is the conditional f-divergence of trajectories containing a given state-action pair, scaled by that pair’s probability.The gap reflects treating those trajectories as separate events rather than combining them as one state-action event.Source: A Lower Bounding f-Divergence of… · p. 20

B Relating f-Divergence of Trajectory Distribution with Expected Action Distribution

• For any policy and feature function, expected feature counts along induced trajectories equal expected feature counts on the induced state-action distribution.This equality follows by marginalizing future states and actions and using the average state-distribution definition.Source: B Relating f-Divergence of Trajectory… · p. 21
• Tallying states visited along trajectories recovers the policy’s average state visitation frequency.This follows as a useful equality derived from the expected-feature-count lemma.Source: B Relating f-Divergence of Trajectory… · p. 21
• The expected-feature-count equality does not hold for f-divergences in general; the upper-bound analysis applies to divergences satisfying the triangle inequality, including total variation and squared Hellinger distance.The paper characterizes these examples as distances.Source: B Relating f-Divergence of Trajectory… · p. 22
• For f-divergences satisfying the triangle inequality, trajectory divergence is upper-bounded by expected action-distribution divergence on states induced by the learner policy.This bound requires expert labels at learner-visited states, so the expert must be queryable from arbitrary states.Source: B Relating f-Divergence of Trajectory… · p. 22

C Existing algorithms as different f-divergence minimization

• Behavior cloning instantiates KL as expert-distribution cross-entropy or M-projection without environment interaction, whereas GAIL [5] instantiates JS and reportedly outperforms it on simulated environments using costlier minimax optimization.Behavior cloning's M-projection can contribute to covariate shift: supervised errors compound over time, and the learner may assign probability to actions the expert never takes.Source: C Existing algorithms as different… · p. 26
• DAgger [17] corresponds to total variation, which upper-bounds trajectory-distribution distance by expected action-distribution distance, yet can under-penalize learners leaving the expert's track even when they imitate perfectly in bad states.The paper contrasts this with reverse KL, which would penalize the off-track behavior more strongly.Source: C Existing algorithms as different… · pp. 26–27, Figure 7 · p. 28

D Reverse KL Divergence via Interactive Variational Imitation

• RKL-iVIM exploits equality to minimize action-distribution divergence on learner-visited states, querying the expert and collecting fresh expert and learner batches each iteration.The expert must be available during training, which may be inconvenient.Source: D Reverse KL Divergence via… · p. 28

E Density Ratio Minimization for Reverse KL via No Regret Online Learning

• Algorithm 3 estimates π_n(a|s)/π*(a|s) from learner-visited states, aggregates the resulting state-cost pairs, and updates the policy by cost-sensitive classification.It uses a density-ratio estimator and a cost-sensitive classifier as inputs.Source: E Density Ratio Minimization for… · p. 30
• As N grows, Theorem 6 bounds inverse KL by (1 + 1/c)γ + ε_class; richer policy classes can reduce ε_class, and improved density-ratio estimation can improve Algorithm 3.The bound assumes the expert assigns at least probability c to every action at every state.Source: E Density Ratio Minimization for… · pp. 31–32

E.1 Proof of Theorem 6

• Theorem 6's proof views Algorithm 3 as follow-the-leader with no-regret losses; its density-ratio error depends on estimator capacity and samples, requires min π*(a|s) ≥ c for KL, and retains linear dependence on T.The passage notes that this linear dependence is not improvable and also appears in DAgger.Source: §E.1 · pp. 32–33

E.2 An Example of Density Ratio Estimation and Its Finite Sample Analysis

• The section analyzes Nguyen et al.’s density-ratio estimator and supplies finite-sample analysis missing from.The estimation problem is to recover r(z)=p(z)/q(z) from two distributions over a finite set.Source: §E.2 · p. 33
• The analysis assumes p(z),q(z)≥c and that the true ratio belongs to finite G, so the ratio is defined and bounded in [c,1/c].The finite-class assumption simplifies the analysis; the text notes that continuous classes can instead use complexity terms such as Rademacher complexity.Source: §E.2 · pp. 33–34
• Uniform convergence over G follows by applying Hoeffding’s inequality to each fixed function and taking a union bound over G.The resulting bound depends logarithmically on |G|.Source: §E.2 · pp. 34–35
• Under assumptions E1 and E2, with probability at least 1 − δ, DRE returns a ratio estimator whose expected squared error under q is bounded by 2C_N.Source: §E.2 · pp. 34–35

F Divide-by-zero Issues with KL and JS

• If p(x)>0 while q(x)=0, sampling only from q misses the location and can fail to capture an infinite divergence, especially for KL and JS.The problem occurs even when p(x) and q(x) can be evaluated exactly.Source: F Divide-by-zero Issues with KL and JS · p. 36

G Experimental Details

• The experimental details set bandit and grid-world noise levels and parameterize their discrete-action policies using cosine-based vectors and softmax.Unless otherwise specified, bandit control noise is ϵ0=0.28; grid-world control and transition noise are ϵ1=0.14 and ϵ2=0.15. Both policy constructions set A=2.5; the bandit policy uses scalar θ, while the grid-world policy uses a matrix θ with one value per state.Source: G Experimental Details · p. 37