Source-linked AI summary
Imitation Learning from Imperfect Demonstration
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, Masashi Sugiyama
TL;DR
Imitation learning often relies on optimal demonstrations, which are costly and frequently imperfect. This paper introduces 2IWIL and IC-GAIL, which use confidence scores with unlabeled demonstrations through different reweighting strategies. The authors report theoretical and empirical improvements, while identifying task-specific limitations and future application boundaries.
Problem
Optimal demonstrations are costly and real-world demonstrations are often mixtures of optimal and non-optimal behavior, with only some demonstrations carrying confidence labels.
Method
The paper proposes 2IWIL and IC-GAIL, confidence-based imitation-learning methods that reweight confidence-labeled and unlabeled demonstrations.
Results
The methods outperform baselines by a large margin, show robustness to noise, and exhibit a positive correlation between unlabeled data and performance.
Takeaways & Limitations
Confidence scores assigned to only part of imperfect demonstrations can support learning an optimal policy while utilizing additional unlabeled data.
Takeaways & Limitations
Performance improvements are less significant in Walker2d-v2, where the authors conjecture that confidence information is insufficient, and HalfCheetah-v2 is affected by discriminator local maxima.
Abstract
from arXiv · showhide
Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrations. More specifically, we propose two confidence-based IL methods, namely two-step importance weighting IL (2IWIL) and generative adversarial IL with imperfect demonstration and confidence (IC-GAIL). We show that confidence scores given only to a small portion of sub-optimal demonstrations significantly improve the performance of IL both theoretically and empirically.
1 Introduction
Imitation learning avoids reward design but typically assumes optimal demonstrations, an assumption that often fails because real-world demonstrations mix optimal and non-optimal behavior. The paper introduces confidence-based reweighting methods that use partially labeled imperfect demonstrations.
- Optimal demonstrations are difficult to obtain because human attention and physical conditions limit consistently optimal behavior.Real-world examples such as basketball videos may contain turnovers and other mistakes.
- Confidence scores indicate the probability that a demonstration trajectory is optimal and can be cheaper to collect than optimal demonstrations.Scoring requires knowledge of optimal behavior, whereas performing optimally also requires strict physical conditions.
- The proposed setting assumes only some demonstrations have confidence labels, while the remainder are unlabeled demonstrations.This partially labeled setting is intended to reduce the additional cost of learning an optimal policy.
- The paper studies imperfect demonstrations as mixtures of optimal and non-optimal demonstrations, including data from crowdsourcing and different video sources.Some demonstrations are generated by different policies.
- The paper proposes 2IWIL and IC-GAIL, two reweighting-based methods that use confidence and unlabeled data from different perspectives.2IWIL predicts confidence for unlabeled data, whereas IC-GAIL reweights toward the non-optimal distribution and matches optimal occupancy measures.
- IC-GAIL converges more slowly than 2IWIL but achieves better performance, establishing a trade-off between convergence speed and performance.The paper reports that both methods are theoretically and practically sound.
2 Related work
Prior work learns from non-optimal demonstrations using reward, feature, or trajectory information, while this paper uses confidence scores in a partially labeled setting. Related semi-supervised methods estimate confidence from labeled or soft-labeled data but have differing scalability and theoretical properties.
- DM-IRL assumes a linear reward over state features and uses human feedback estimating accumulated reward.Accumulated reward can be harder to provide than confidence because multiple reward functions may share an optimal policy.
- SSIRL uses optimal and sub-optimal trajectories and transductive SVM to recognize optimal trajectories among sub-optimal ones.The paper differs by receiving confidence scores instead of optimal demonstrations.
- 2IWIL trains a probabilistic classifier from confidence-labeled and unlabeled data by optimizing an empirical risk minimization objective.The objective supports different loss functions, models, and optimizers.
- Related methods estimate unlabeled confidence using Gaussian mixture models, principal component analysis, or kernel density estimation.Some approaches use hard-labeled instances to estimate probabilistic labels or confidence bounds.
- A fuzzy-input k-nearest-neighbors method uses soft labels but is difficult to scale to high-dimensional tasks and lacks theoretical guarantees.Another scheme trains a classifier only from positive confidence-equipped data.
3 Background
The background defines reinforcement learning and occupancy measures, then frames imitation learning as recovering an expert policy without its reward function. GAIL solves this through adversarial occupancy-measure matching between agent and optimal demonstrations.
- Reinforcement Learning: An MDP consists of state and action spaces, transition dynamics, rewards, and a discount factor.The discount factor satisfies γ ∈(0, 1).
- Reinforcement Learning: Reinforcement learning maximizes expected discounted performance over trajectories generated by the initial-state distribution, transitions, and policy.The objective is optimized with respect to the policy.
- Occupancy Measures: The occupancy measure characterizes the distribution of state-action pairs generated by a policy and has a one-to-one correspondence with that policy.It is an unnormalized density, while the normalized occupancy measure can be interpreted as a probability density.
- Occupancy Measures: The normalized occupancy measure represents the probability density of state-action pairs experienced by an agent under a policy.
- Imitation Learning: Imitation learning uses expert trajectories and an MDP without a reward function to optimize an agent policy that recovers the expert policy.
- Generative Adversarial Imitation Learning: GAIL performs adversarial occupancy-measure matching by estimating Jensen-Shannon divergence through binary classification.The discriminator distinguishes agent and optimal state-action pairs, and its loss supplies a reward signal for policy updates.
4 Imitation learning with confidence and unlabeled data
The paper uses scarce confidence labels plus unlabeled imperfect demonstrations to learn optimal policies through two confidence-based imitation-learning approaches. 2IWIL predicts unlabeled confidence scores before importance-weighted GAIL, while IC-GAIL performs end-to-end occupancy-measure matching; the analysis establishes consistency but identifies variance and convergence trade-offs.
- 2IWIL: 2IWIL trains a probabilistic classifier on confidence and unlabeled data, predicts unlabeled confidence scores, then performs GAIL with a reweighted distribution.The classifier should use a strictly proper composite loss to estimate confidence reliably; hinge loss is unsuitable for this purpose.
- Problem setting: Confidence data and unlabeled demonstrations are jointly used to estimate confidence scores and learn the optimal policy.The proposed setting assumes only some demonstration state-action pairs receive oracle confidence labels.
- 2IWIL theory: The minimum-variance classifier estimator uses a covariance-dependent β, while practice fixes β = nu/(nc+nu) under a small-covariance assumption.When β ≠ 0, unlabeled data can reduce empirical variance; the optimal coefficient is clipped to [0,1].
- Limitations and trade-offs: The empirical confidence-risk estimate can become negative and cause overfitting, while IC-GAIL’s agent reward becomes nearly constant when α is small.The former is mitigated with a technique from Kiryo et al.; the latter scales the agent term by α(1−α), slowing learning relative to 2IWIL.
- 2IWIL theory: The semi-conf classification risk can be estimated from confidence and unlabeled data, and its minimizer is asymptotically equivalent to the standard classification-risk minimizer.This provides a consistent estimator of p(y = +1|x), supported by an estimation-error bound involving Rademacher complexity.
- IC-GAIL: IC-GAIL avoids the two-step error accumulation of 2IWIL by directly matching the optimal occupancy measure in an end-to-end procedure.Its theoretical objective is minimized if and only if the learned occupancy measure equals the optimal one almost everywhere, and discriminator estimation is consistent under stated assumptions.
5 Experiments
Experiments evaluate whether 2IWIL and IC-GAIL learn near-optimal policies with limited or noisy confidence and varying unlabeled data. Both methods outperform baselines, remain robust to noisy confidence, and improve as unlabeled data increases.
- Experimental setup: The experiments test near-optimal policy learning with limited confidence, robustness to inaccurate confidence, and the effect of additional unlabeled demonstrations.Experiments use Mujoco tasks, five random seeds, and normalized accumulative reward where 1.0 denotes the optimal policy and 0.0 a uniform random policy.
- Performance comparison: 2IWIL and IC-GAIL outperform the baselines by a large margin under imperfect demonstrations.The experiments use an optimal demonstration distribution with an approximately 33% class prior.
- Performance comparison: 2IWIL converges faster than IC-GAIL, while IC-GAIL still converges to near-optimal policies in four tasks despite omitting the class prior α.The slower IC-GAIL convergence is attributed to scaling the agent term by τ = 0.7.
- Performance comparison: Confidence reweighting improves average return over GAIL (C), but confidence alone is insufficient to derive optimal policies from few confidence instances.GAIL (Reweight) surpasses GAIL (C) in all tasks, while the proposed methods additionally exploit the demonstration mixture.
- Robustness to noisy confidence: Both methods are quite robust to Gaussian noise added to confidence scores in the Ant-v2 experiment.The noise is sampled as ϵ ∼ N(0, σ^2), modeling potentially inaccurate human confidence labels.
- Influence of unlabeled data: Performance for both methods grows as the number of unlabeled demonstrations increases, supporting their usefulness when confidence data is scarce.Figure 3 varies the proportion of unlabeled demonstrations used as input.
6 Conclusion
The paper concludes that 2IWIL and IC-GAIL use confidence and unlabeled data to learn from imperfect demonstrations. Experiments report large-margin baseline improvements, noise robustness, and positive association between unlabeled data and performance, while future work targets broader applications.
- Contributions: 2IWIL and IC-GAIL allow imitation-learning agents to use both confidence information and unlabeled data.2IWIL estimates unlabeled confidence and uses importance sampling, whereas IC-GAIL matches a mixture occupancy measure to the demonstrations’ occupancy measure.
- Empirical findings: Experiments show large-margin baseline improvements, robustness to noise, and a positive correlation between unlabeled data and performance.The approaches are also described as general and extensible to other IL and IRL methods.
- Future work: Future work may extend the confidence-based approach to discrete sequence generation using property indicators such as solubility.The paper gives soluble-chemical generation as an example application.
A.2 Proof of Proposition 4.2
The proposition identifies a minimum-variance estimator within the stated family by selecting the combination coefficient β appropriately.
- Variance minimization: For fixed g, the estimator bR_SC,ℓ(g) has minimum variance when β is set to the expression involving n_u, n_c, covariance, and loss variance.The result applies among estimators of the form in Eq. (7) with β ∈ [0, 1].
- Notation: The clipping operation is defined as clip[l,u](v) = min{max{v, l}, u}.This definition specifies how values are constrained to the interval [l, u].
A.3 Proof of Theorem 4.3
The theorem’s proof bounds the estimation error by decomposing it into several terms and controlling their deviations with concentration inequalities.
- Assumptions: The theorem assumes a hypothesis class G, a ρ_ℓ-Lipschitz loss, and a uniform bound C_ℓ on the loss magnitude.These assumptions constrain the loss used in the estimation-error analysis.
- Proof strategy: The proof analyzes the deviation between empirical and expected risks by bounding four terms independently.The first term is controlled using the bounded-difference property of the confidence-weighted loss.
- Final guarantee: The original estimation error is bounded with probability at least 1 − δ over repeated sampling of training data.The proof combines the independently controlled terms to obtain the final guarantee.
B.1 Proof of Theorem 4.4
The proof shows that the discriminator objective is maximized by a density-ratio form and reduced to Jensen–Shannon divergence between mixture distributions. Minimizing this objective therefore recovers the optimal policy distribution almost everywhere.
- The maximized objective is C(πθ) = −log 4 + 2JSD(p∥p′).The Jensen–Shannon divergence is nonnegative and minimized only when p′ = p.
- For a fixed policy, the discriminator maximizes V(πθ, Dw) at Dw(x) = p′(x)/(p′(x)+p(x)).Here p′ = αpθ + (1 − α)pnon.
- C(πθ) is minimized if and only if pθ = popt almost everywhere.
- The theorem rewrites V(πθ, Dw) as an objective estimable from the mixture distribution and confidence-weighted terms.The transformed objective is eV(πθ, Dw) = Ex∼p[log(1−Dw(x))] + αEx∼pθ[log Dw(x)] + Ex,r∼q[(1−r)log Dw(x)].
B.3 Proof of Theorem 4.6
The proof establishes that empirical discriminator training consistently approximates the original objective under repeated sampling. It combines uniform deviation bounds for the objective’s component terms to obtain the final high-probability estimation guarantee.
- The discriminator hypothesis space is analyzed under boundedness and Lipschitz assumptions on log Dw and log(1−Dw).These assumptions support uniform concentration over discriminator parameters.
- The proof decomposes the population–empirical objective difference into three terms and bounds them independently using McDiarmid’s inequality.The terms correspond to unlabeled, agent, and confidence-weighted samples.
- Two-sided uniform deviation bounds are obtained with probability at least 1−δ/3 after applying the one-sided bound twice.
- The component bounds are combined to bound the original estimation error with probability at least 1−δ.
C Implementation and Experimental Details
The experiments use common neural-network settings and normalized returns, while examining risk-estimator stability, confidence noise, and varying unlabeled-data proportions. Ant-v2 GAIL-based training can degrade rapidly early in training.
- Implementation: All tasks use the same neural-network architecture and hyper-parameters, with two hidden layers of size 100 and Tanh activations.
- Implementation: Normalized average return is 1.0 for the optimal policy and 0.0 for the uniformly random policy.
- Risk estimator: The empirical risk may become negative and cause overfitting, so a max operator is added because the expected risk is nonnegative.
- Experimental figures: The figures examine learning curves across baselines, Gaussian confidence-noise levels, and proportions of unlabeled demonstrations.The noise-figure legend reports Gaussian standard deviations, while the unlabeled-data legend reports the proportion used as demonstrations.
- Experimental caveat: GAIL-based approaches in Ant-v2 can experience rapid performance degradation during early training.