Source-linked AI summary
A Divergence Minimization Perspective on Imitation Learning Methods
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, Shixiang Gu
TL;DR
The paper asks why adversarial inverse reinforcement learning outperforms behavioural cloning when demonstrations are scarce. It develops a unified divergence-minimization view through f-MAX and evaluates the resulting hypotheses. The main conclusion is that explicit expert state-marginal matching accounts for the performance advantage, while f-MAX has a forward-KL limitation.
Problem
The paper investigates why adversarial IRL methods outperform behavioural cloning in low-data settings despite both recovering the expert policy at optimality.
Method
The paper introduces f-MAX, an f-divergence generalization of AIRL, to unify imitation-learning methods and evaluate their differences in continuous-control benchmarks.
Results
Explicitly matching the expert’s marginal state distribution, in addition to conditional action distributions, accounts for the major performance gain of IRL methods over behavioural cloning.
Takeaways & Limitations
State-marginal matching can train diverse behaviours from hand-specified state distributions without reward functions or expert demonstrations.
Takeaways & Limitations
f-MAX cannot be used for forward KL divergence and requires a separate method for that case.
Abstract
from arXiv · showhide
In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, due to multiple factors of variation, directly comparing these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective on IL algorithms based on divergence minimization. We present $f$-MAX, an $f$-divergence generalization of AIRL [Fu et al., 2018], a state-of-the-art IRL method. $f$-MAX enables us to relate prior IRL methods such as GAIL [Ho & Ermon, 2016] and AIRL [Fu et al., 2018], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches, and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings conclusively identify that IRL's state-marginal matching objective contributes most to its superior performance. Lastly, we apply our new understanding of IL methods to the problem of state-marginal matching, where we demonstrate that in simulated arm pushing environments we can teach agents a diverse range of behaviours using simply hand-specified state distributions and no reward functions or expert demonstrations. For datasets and reproducing results please refer to https://github.com/KamyarGh/rl_swiss/blob/master/reproducing/fmax_paper.md .
1 Introduction
Imitation learning addresses difficult or underspecified reward-design problems through expert demonstrations. The paper explains why adversarial IRL methods outperform behavioural cloning in low-data settings and develops a divergence-based framework to analyze this difference.
- Imitation learning uses expert demonstrations when reward functions are difficult to optimize or cannot capture desired outcomes.
- Behavioural cloning regresses expert actions, whereas inverse reinforcement learning infers the expert’s reward before training a policy.
- Adversarial IRL methods outperform behavioural cloning especially when only a very limited number of expert trajectories are available.
- The paper introduces f-MAX, an f-divergence generalization of AIRL, to unify imitation-learning methods and explain their algorithmic properties.
- The paper applies this perspective to state-marginal matching and trains diverse behaviours using hand-specified state distributions without rewards or demonstrations.
2 Related Work
Related work connects imitation learning with distribution matching and probabilistic views of reinforcement learning. It also highlights behavioural cloning’s covariate-shift problem and the strong continuous-control performance of IRL methods.
- Control-as-probabilistic-inference work interprets entropy-regularized optimal control as approximate inference or reverse-KL minimization between trajectory distributions.
- Behavioural cloning is simple and widely used, but suffers covariate shift between training and testing; DAgger and DART require interactive expert access.
- IRL methods outperform behavioural cloning on challenging continuous-control problems, with earlier approaches matching policy and expert feature expectations or moments.
3 Background
The background formulates imitation learning through maximum-entropy IRL and adversarial distribution matching. GAIL uses a discriminator to minimize Jensen–Shannon divergence, while related adversarial methods directly recover expert policies.
- An MDP is specified by state and action spaces, dynamics, rewards, an initial-state distribution, and a discount factor; policy marginals describe state-action and state frequencies.
- Maximum-entropy IRL methods can directly recover the policy produced by the full reward-learning process instead of separately recovering reward and policy.
- GAIL: GAIL trains a discriminator on expert and policy state-action pairs while updating the policy with reinforcement learning and causal-entropy regularization.
- GAIL: GAIL’s alternating discriminator-policy procedure minimizes the Jensen–Shannon divergence between expert and policy state-action marginals.
- AIRL: AIRL extends adversarial IRL by recovering the maximum-entropy reward and training the corresponding maximum-entropy policy.
- Adversarial maximum-entropy IRL methods achieve strong performance with very limited demonstrations in standard MuJoCo benchmarks, a failure scenario for standard behavioural cloning.
4 f-MAX: f-Divergence Max-Ent IRL
f-MAX generalizes AIRL by expressing policy learning as minimization of a chosen f-divergence between expert and policy state-action distributions. Its specializations connect AIRL to reverse KL and GAIL to the broader Max-Ent IRL framework.
- f-MAX: f-MAX uses the f-GAN formulation to minimize an arbitrary f-divergence between expert and policy state-action distributions.
- f-MAX: The iterative procedure optimizes a discriminator-like function and policy so that, at convergence, the policy state-action distribution matches the expert’s.
- AIRL: Choosing f(u) := −log u makes f-MAX equivalent to AIRL and interprets AIRL as minimizing reverse KL divergence.
- GAIL: GAIL is the Jensen–Shannon-divergence special case within the class of methods that directly retrieve expert policies.
- Cost-regularized Max-Ent IRL: f-MAX is a subset of Ho and Ermon’s cost-regularized maximum-entropy IRL framework, and AIRL inherits this relation.
5 Understanding the Relation Among Imitation Learning Methods
Writing IL objectives as divergence minimization reveals that BC matches expert actions conditionally, while AIRL and GAIL also match state marginals and use more mode-seeking divergences.
- BC minimizes the expert-conditional forward KL, while AIRL minimizes reverse KL and GAIL minimizes Jensen-Shannon divergence over state-action marginals.The common formulation expresses these methods as statistical-divergence objectives, with GAIL additionally regularized by causal entropy.
- AIRL and GAIL explicitly encourage matching expert state distributions, unlike standard BC, which matches only the conditional action distribution.This distinction motivates state-marginal matching as a possible explanation for IRL's advantage.
- The first hypothesis is that rewards often depend more on states than actions, making explicit state-marginal matching an important learning criterion.
- The second hypothesis is that mode-seeking divergences are more beneficial than forward-KL mode covering when learning high-quality trajectories from limited demonstrations.
- The experiments compare a forward-KL state-action method with Behavioural Cloning and AIRL across varying amounts of expert demonstrations.
6 FAIRL: An Alternative Method for Forward KL
The authors introduce a separate forward-KL method, FAIRL, because f-MAX cannot handle forward KL; modifying AIRL's reward yields an objective proportional to negative forward KL.
- f-MAX is general across many f-divergences but cannot be used for the forward-KL special case.The paper identifies this problem in Appendix E and introduces a separate method for minimizing that divergence.
- FAIRL uses the AIRL discriminator with a modified reward defined from the discriminator logit and its exponential.The reward is specified as h(s, a) := log D(s, a) − log(1 − D(s, a)) and r(s, a) := exp(h(s, a)) · (−h(s, a)).
- The expected FAIRL return is proportional to −KL(ρexp(s, a)||ρπ(s, a)), converting AIRL into its forward-KL counterpart.
- AIRL, GAIL, and FAIRL correspond respectively to reverse KL, Jensen-Shannon, and forward KL divergences.
7 Intuition About Different Divergence Rewards
The discriminator logit equals the expert-to-policy state-action density ratio, providing a common axis for comparing the reward structures induced by AIRL, GAIL, and FAIRL.
- For an optimal discriminator, the logit equals the log density ratio between expert and policy state-action marginals.Figure 1 plots each method's reward as a function of this optimal-discriminator logit.
- AIRL encourages more probability where the expert has more mass than the policy and less where the expert has less mass.
- GAIL discourages the policy from placing more probability mass in regions than the expert does.
- FAIRL encourages slightly undercovered regions, tolerates substantial overcoverage, and severely penalizes regions where expert mass exceeds policy mass.The resulting behavior is interpreted as covering the expert distribution from low-probability regions inward toward its modes.
8 Experiments
Across continuous-control benchmarks, AIRL and FAIRL outperform BC, especially with few demonstrations, supporting state-marginal matching as the main source of IRL’s advantage. Applying f-MAX directly to state marginals also produces diverse behaviors in simulated environments, though state marginals alone do not uniquely identify a policy.
- 8.1 Evaluating Hypotheses: The results attribute IRL’s performance gain primarily to matching expert state marginals in addition to conditional action distributions.FAIRL’s advantage over BC indicates that the gain is not explained mainly by the direction of KL divergence.
- 8.1 Evaluating Hypotheses: AIRL and FAIRL significantly outperform BC across benchmark settings, including the low-data regime.Table 2 evaluates 4, 16, and 32 demonstration trajectories across HalfCheetah, Ant, Walker, and Hopper.
- 8.1 Evaluating Hypotheses: With 32 demonstrations, F/AIRL achieves results similar to DAgger trained on the same expert data.The comparison is reported across the benchmark evaluation described in Table 2.
- 8.2 f-MAX for State Marginal Matching: State-marginal matching applies f-MAX without actions by minimizing D_f(ρ_target(s)||ρ_π(s)) through an iterative optimization procedure.The target can be represented by samples and need not be a realizable expert state-marginal distribution.
- 8.2 f-MAX for State Marginal Matching: State-marginal matching is not itself an imitation-learning algorithm because distinct policies can share identical state marginals.Thus, matching state distributions does not guarantee recovery of the expert policy.
- 8.2 f-MAX for State Marginal Matching: The approach trains diverse behaviors, including multi-modal point-mass trajectories, Fetch exploration policies, Pusher pushing, and sinusoidal drawing.The Pusher results are described as intriguing but remain below what can be achieved with RL and the task’s provided reward.
9 Conclusion
The paper unifies imitation-learning methods through divergence minimization by introducing f-MAX, a generalization of AIRL, and evaluates why IRL outperforms BC. It concludes that additional state-marginal matching is the main factor, while extending f-MAX to learning from target state distributions without expert demonstrations.
- 9 Conclusion: f-MAX generalizes AIRL using f-divergences and provides a unified divergence-minimization view of imitation-learning methods.The framework relates prior IRL methods and supports analysis of their algorithmic properties.
- 9 Conclusion: Experiments in high-dimensional continuous-control benchmarks identify additional state-marginal matching as the main contributor to IRL’s gain over BC.The conclusion reports this result as the paper’s central empirical finding.
- 9 Conclusion: FAIRL modifies AIRL to optimize the forward KL divergence, helping separate divergence direction from the effect of state-marginal matching.The modification addresses a degeneracy of f-MAX in a special case.
- 9 Conclusion: Applying f-MAX to state-marginal matching suggests replacing expert demonstrations with hand-specified target state distributions.The target distributions can guide policies without reward functions or expert demonstrations.
- 9 Conclusion: The divergence formulation relies on technical conditions when converting between infinite-series integrals and sums, including bounded ranges for h and trajectory densities.The stated boundedness condition is one case in which the required interchange holds.
C Corollary: A Simple Derivation and Intuition for AIRL
The derivation identifies AIRL as a reverse-KL, cost-regularized Max-Ent IRL method within the f-MAX framework. Under an optimal discriminator, its policy objective corresponds to minimizing KL(ρπ(s, a)||ρexp(s, a)).
- Choosing f(u) := −log u makes f-MAX minimize the reverse KL divergence KL(ρπ(s, a)||ρexp(s, a)).
- AIRL’s optimal-discriminator policy objective matches the derived f-MAX objective up to a constant.
- f-MAX and AIRL fall under the cost-regularized Max-Ent IRL framework, modulo the causal entropy term.
- The same derivation suggests the same training procedure for f-MAX and AIRL.
- A forward-KL specialization can provide no policy-training signal under its optimal objective.
F Derivation for FAIRL
This section presents the derivation of FAIRL under its discriminator formulation and optimality assumption.
- The FAIRL derivation recalls the relevant definitions before assuming an optimal discriminator.
G Experimental Details for Hypotheses Evaluation
The experiments evaluate imitation methods across continuous-control environments, demonstration budgets, and training configurations. They also test state-marginal matching with hand-specified target distributions, while documenting model choices and practical limitations.
- Experimental setup: Experts use Soft-Actor-Critic policies with two-layer 256-dimensional MLPs and Tanh-Normal action distributions.
- Experimental setup: Demonstration sets contain 4, 16, or 32 trajectories, with trajectories subsampled by a factor of 20.
- Experimental setup: Each method is trained with 3 random seeds, selected by best validation loss, and evaluated on 50 test episodes.
- Model configurations: AIRL and FAIRL use SAC student policies and discriminators with 256-dimensional hidden layers, gradient penalties, observation normalization, and clipped logits.
- Model configurations: BC uses the expert architecture and is fitted by maximum likelihood with observations normalized using demonstration statistics.
- Environments: The evaluation includes Point-Mass, Pusher, and Fetch environments with task-specific horizons and control settings.
- State-marginal matching: State-marginal-matching experiments use parametric curves, sinusoidal traces, and a five-dimensional arm-tip/object target distribution.
- Hyperparameter tuning: AIRL is less sensitive than FAIRL to the two tuned hyperparameters and performs well across a wider range in Ant.