Source-linked AI summary
Imitation Learning with Sinkhorn Distances
Georgios Papagiannis, Yunpeng Li
TL;DR
Imitation learning depends on comparing expert and learner occupancy measures, while specifying rewards can be difficult and divergence methods can be challenging to evaluate in high dimensions. SIL minimizes their Sinkhorn distance with an adversarially learned feature-space cost. Across MuJoCo experiments, it is competitive with baselines and outperforms them in some settings on reward and Sinkhorn-distance metrics.
Problem
Comparing expert and learner occupancy measures is central to imitation learning, while practical reward design is non-trivial and divergence methods can be difficult to evaluate in high dimensions.
Source: §1 · p. 1, §2.2 · p. 4Method
SIL formulates imitation learning as Sinkhorn-distance minimization between occupancy measures, using an adversarially trained critic and feature-space cost to derive a reward proxy.
Source: §1 · p. 2, §3 · pp. 5–6Results
SIL is competitive with baselines across five MuJoCo environments and outperforms them in several settings on both reward and Sinkhorn-distance metrics.
Source: §1 · p. 3, §5 · p. 13Takeaways & Limitations
The adversarially trained critic provides a discriminative signal to the learner policy to facilitate imitation of expert behavior.
Source: §5 · p. 13SIL currently supports only on-policy RL because Sinkhorn-distance computation requires complete trajectories, and it uses more environment interactions than off-policy adversarial IL methods.
Source: §5 · p. 13
Abstract
from arXiv · showhide
Imitation learning algorithms have been interpreted as variants of divergence minimization problems. The ability to compare occupancy measures between experts and learners is crucial in their effectiveness in learning from demonstrations. In this paper, we present tractable solutions by formulating imitation learning as minimization of the Sinkhorn distance between occupancy measures. The formulation combines the valuable properties of optimal transport metrics in comparing non-overlapping distributions with a cosine distance cost defined in an adversarially learned feature space. This leads to a highly discriminative critic network and optimal transport plan that subsequently guide imitation learning. We evaluate the proposed approach using both the reward metric and the Sinkhorn distance metric on a number of MuJoCo experiments. For the implementation and reproducing results please refer to the following repository https://github.com/gpapagiannis/sinkhorn-imitation.
1 Introduction
Imitation learning uses expert demonstrations to address the difficulty of specifying rewards, but existing approaches face trade-offs in sample efficiency, expert queries, training stability, and tractability. SIL formulates imitation as Sinkhorn-distance minimization between occupancy measures and reports competitive or better baseline performance across several MuJoCo settings.
1 Introduction
- Practical reward design is often non-trivial, motivating imitation learning from expert demonstrations as an alternative for learning behavior.The introduction notes applications including autonomous driving and surgical task automation.Source: §1 · p. 1
- Behavioral cloning can suffer from low sample efficiency and poor generalization, while DAgger and DART alleviate these issues but require repeated expert action queries.IRL instead recovers a reward function, but its ill-posedness requires additional constraints and alternating reward-policy optimization increases computational cost.Source: §1 · pp. 1–2
- Adversarial imitation avoids explicit reward inference, but f-divergence and GAN-based approaches can have training instability and mode-covering behavior.Wasserstein-based imitation is robust to disjoint measures, although its dual solution is intractable and PWIL's greedy coupling has difficult-to-quantify approximation error.Source: §1 · p. 2
- SIL minimizes the Sinkhorn distance between expert and learner occupancy measures, using trainable optimal-transport ground metrics to derive a reward proxy.The paper presents this as a tractable OT-based imitation-learning pipeline.Source: §1 · p. 2
- On MuJoCo experiments, SIL achieves comparable results with state-of-the-art methods and outperforms baselines in several settings on both reward and Sinkhorn-distance metrics.The reported comparisons use both the commonly used reward metric and the Sinkhorn distance.Source: §1 · p. 3
2 Background
The background frames imitation learning as matching expert and learner behavior distributions, then introduces optimal transport as a metric-based way to compare distributions. Its Sinkhorn regularization makes the transport computation tractable for the state-action distributions used in adversarial imitation learning.
2.1 Imitation Learning
- An MDP is defined by states, actions, transition probabilities, rewards, and a discount factor; policies induce trajectories and state-action occupancy measures.The occupancy measure is the distribution of state-action pairs generated through interaction with the environment.Source: §2.1 · p. 3
- MaxEnt IRL can be expressed as occupancy-measure matching, and selecting a reward regularizer yields GAIL's Jensen-Shannon divergence objective.More generally, f-divergence IL methods optimize a learner policy using a reward proxy derived from the selected divergence.Source: §2.1 · pp. 3–4
2.2 Optimal Transport
- Optimal transport compares probability distributions through couplings based on the metric of the underlying space, making it effective for degenerate distributions.The transport plan is the joint distribution minimizing total transport cost.Source: §2.2 · p. 4
- Sinkhorn regularization makes the otherwise generally intractable Wasserstein computation efficient by restricting couplings according to an entropy bound.In adversarial imitation learning, the compared distributions are the learner's and expert's state-action distributions.Source: §2.2 · p. 4
3 SIL: Sinkhorn Imitation Learning
SIL trains a learner to match expert state-action occupancy measures by minimizing their Sinkhorn distance, using an adversarially learned feature-space cost to provide a discriminative reward proxy. It is also connected to entropy-regularized MaxEnt IRL through a reward regularizer.
3 SIL: Sinkhorn Imitation Learning
- SIL matches learner and expert occupancy measures by minimizing their Sinkhorn distance and using the resulting reward proxy to train the learner policy.The policy objective with this proxy corresponds to minimizing the Sinkhorn distance between learner and expert trajectories.Source: §3 · pp. 4–5
- The transport-plan reward depends on all state-action pairs in both trajectories, so the same learner pair can receive different rewards across trajectories and complicate optimization.The reward is available only after a complete learner trajectory has been generated.Source: §3 · p. 5
- An adversarially trained critic increases transport costs for non-expert state-action pairs, providing the learner with a discriminative signal.Its cost uses cosine distance in an adversarially learned feature space.Source: §3 · p. 5
- SIL randomly pairs learner and expert trajectories, then alternates critic maximization of their Sinkhorn distance with policy minimization using the learned reward proxy.Random pairing is chosen to keep computational complexity constant as the number of demonstrations grows; complete trajectories make SIL on-policy, and the experiments use TRPO.Source: §3 · p. 6
3.1 Connection to regularized MaxEnt IRL.
- The paper interprets SIL as entropy-regularized MaxEnt IRL: its reward regularizer yields the algorithm, with the causal-entropy term treated as regularization.The procedure updates a critic to maximize the occupancy-measure Sinkhorn distance, then finds a policy to minimize it.Source: §3.1 · p. 7
4 Experiments
Across five MuJoCo tasks, SIL is comparable to other adversarial methods on most settings, with its relative performance varying by environment and metric. Training is generally stable, while replacing the learned critic with a fixed cosine cost stabilizes training but performs poorly.
4 Experiments
- SIL was benchmarked against BC, GAIL, and AIRL across Hopper-v2, Walker2d-v2, Ant-v2, HalfCheetah-v2, and Humanoid-v2, using 2–32 demonstration trajectories (8–32 for Humanoid-v2).Expert and learner policies across imitation-learning algorithms shared 2-layer, 128-unit ReLU MLPs; TRPO was used for adversarial IL and expert performance. The adversarial critic used a 2-layer, 128-unit ReLU MLP.Source: §4 · p. 8, §4.1 · pp. 8–9
4.2 Results
- SIL, AIRL, and GAIL are comparable on most settings; SIL leads on HalfCheetah-v2 and Ant-v2, while AIRL leads on Hopper-v2 and Walker2d-v2 by Sinkhorn distance.GAIL is relatively weak with only two expert trajectories, and behavioral cloning is generally noncompetitive, especially with few demonstrations.Source: §4.2 · p. 9, Table 2 · p. 10
- On Humanoid-v2, SIL outperforms GAIL and AIRL with 8 or 16 trajectories, but GAIL leads with 32; behavioral cloning’s 8-trajectory Sinkhorn advantage comes with low reward.The authors describe SIL’s 8- and 16-trajectory results as more sample-efficient in demonstrations and environment interactions.Source: §4.2 · pp. 10–11
- Reward rankings do not always match Sinkhorn rankings: AIRL earns less reward than SIL on Ant-v2, but less than SIL and GAIL on Walker2d-v2.The paper argues that Sinkhorn distance can be a more reliable evaluation metric for the goal of matching state-action distributions.Source: §4.2 · p. 11, Table 3 · p. 11
- SIL training is stable on Hopper-v2, Ant-v2, and HalfCheetah-v2; despite instability on Walker2d-v2 with two trajectories and Humanoid-v2 with 32, it still learns imitation.Figure 1 tracks the Sinkhorn distance during SIL training.Source: Figure 1 · p. 10, §4.2 · pp. 11–12
- Replacing the adversarially learned critic with a fixed cosine transport cost makes training more stable but fails to achieve good Sinkhorn-distance or reward performance.The authors suggest the critic objective supplies strong signals for matching the expert’s state-action distribution.Source: §4.2 · pp. 12–13, Figure 2 · p. 12, Table 4 · p. 12
5 Conclusion
SIL matches expert and learner state-action distributions by minimizing their Sinkhorn distance, using an adversarially trained critic to guide imitation. Across five MuJoCo environments, it shows competitive performance against baselines, while its current formulation has limitations in data collection and design scope.
5 Conclusion
- Across five MuJoCo environments, SIL exhibits competitive performance compared to baselines while matching expert state-action distributions with Sinkhorn distance.Source: §5 · p. 13
- SIL uses an adversarially trained critic to map state-action observations into a learned feature space and provide a discriminative signal for imitation.Source: §5 · p. 13
- Because computing Sinkhorn distance requires complete trajectories, the current formulation supports only on-policy RL and uses more environment interactions than off-policy adversarial IL methods.Source: §5 · p. 13
- Experiments use a fixed critic architecture, leaving its effect unclear; the framework also omits trajectory temporal dependence from its transport coupling and reward proxy.Source: §5 · p. 13