Source-linked AI summary
Causal Confusion in Imitation Learning
Pim de Haan, Dinesh Jayaraman, Sergey Levine
TL;DR
The paper studies how distributional shift causes behavioral cloning to mistake nuisance correlates for action causes, producing causal misidentification and sometimes worse performance with more information. It learns policies from causal graphs and uses targeted environment or expert-query interventions to identify the correct model, finding the phenomenon across benchmark and realistic settings while improving over comparison methods. Its solution has so far been validated in simpler synthetic settings intended to mimic realistic scenarios.
Problem
Distributional shift can cause behavioral cloning to misidentify nuisance correlates as the true causes of expert actions, leading cloned policies to fail after deployment.
Method
The paper maps causal graphs to policies and uses targeted interventions through environment execution or expert queries to identify the correct causal model.
Results
Causal misidentification commonly occurs in natural imitation settings, and access to more information can lead to inferior performance across benchmark tasks and realistic driving settings.
Takeaways & Limitations
Targeted interventions can resolve causal misidentification and address distributional shift using environmental rewards or substantially fewer expert queries than DAgger.
Takeaways & Limitations
The solution has been validated only in somewhat simpler synthetic settings intended to mimic realistic scenarios, leaving extension to realistic settings for future work.
Abstract
from arXiv · showhide
Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out that ignoring causality is particularly damaging because of the distributional shift in imitation learning. In particular, it leads to a counter-intuitive "causal misidentification" phenomenon: access to more information can yield worse performance. We investigate how this problem arises, and propose a solution to combat it through targeted interventions---either environment interaction or expert queries---to determine the correct causal model. We show that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and validate our solution against DAgger and other baselines and ablations.
1 Introduction
Imitation learning faces distributional shift, which can make behavioral cloning identify correlates rather than true causes and perform worse with more information. The paper investigates this causal misidentification and proposes targeted interventions to learn the correct causal model.
- Distributional shift separates expert-generated training states from states encountered under the learned policy, so matching expert actions may not match the task objective.
- Causal misidentification occurs when a cloned policy mistakes nuisance correlates for the true causes of expert actions.Sequential actions can create new nuisance correlates because current actions affect future observations.
- In driving, the model seeing the dashboard wrongly used the brake indicator, whereas masking the dashboard led the model to attend to the pedestrian and drive well.Both policies achieved low training loss, but only the masked-input policy performed well on the road.
- Access to more information can yield worse generalization, and causal misidentification occurs commonly in natural imitation settings, especially with history information.
- The proposed solution learns a mapping from causal graphs to policies and uses targeted expert queries or selected environment executions to identify the correct policy.
2 Related Work
The paper relates imitation-learning distributional shift to causal misidentification and positions targeted interventions as a causal solution. It contrasts this approach with DAgger and dropout-based methods that address related feedback or history-correlation problems.
- Imitation learning inherits a distributional-shift problem because cloned policies encounter unfamiliar states during autonomous execution.
- The paper addresses distributional shift through targeted state interventions using either environmental rewards without extra expert involvement or additional expert queries.
- In expert-query mode, the approach resolves causal misidentification with orders of magnitude fewer queries than DAgger.
- Unlike dropout, which directly applies a mixture policy, the proposed method searches among graph-parameterized policies for the true causal graph and performs significantly better in experiments.
- The paper proposes an interventional causal-inference approach for imitation learning, motivated by the general impossibility of observational causal inference.
3 The Phenomenon of Causal Misidentification
Causal misidentification occurs when behavioral cloning relies on nuisance correlates rather than true causes, causing distributional shift to degrade deployed performance. The paper formalizes this problem causally and demonstrates it across benchmark control tasks and realistic driving settings.
- Behavioral cloning: Behavioral cloning learns a mapping from observed states X_t to expert actions A_t using demonstration state-action pairs.At deployment, the learned policy observes X_t and executes its predicted action.
- Causal structure: Some state factors cause expert actions, while nuisance variables do not but may correlate with actions through past states and actions.The confounder Z_t = [X_t−1, A_t−1] influences state variables, allowing nuisance variables to correlate with expert actions.
- Causal structure: Causal misidentification is the failure of a cloned policy caused by selecting nuisance correlates instead of the true causes of expert actions.Such a policy can achieve low training error and still fail after deployment changes the distributions of Z_t and X_t.
- Causal solution: Modeling the interventional query p(A_t|do(X_t)) is sufficient for robustness to distributional shifts, and under mild assumptions requires learning the correct causal graph.Interventions can reveal that setting the brake light does not affect expert braking behavior.
- Empirical evidence: In benchmark tasks, adding previous-action information creates CONFOUNDED observations, while ORIGINAL observations replace that variable with random noise.The study covers MountainCar, Hopper, Pong, Enduro, and UpNDown using identical policy architectures and demonstrations.
- Empirical evidence: Across benchmark tasks, ORIGINAL approaches expert rewards as dataset size increases, whereas CONFOUNDED needs many more demonstrations or fails to reach equivalent performance.The performance gap is not explained by held-out demonstration losses; in Pong, CONFOUNDED has lower validation loss but lower control rewards.
- Empirical evidence: History-based imitation in realistic driving can achieve better held-out demonstration performance yet substantially worse deployed driving performance, consistent with causal misidentification.History introduces information about nuisance factors such as previous actions, and similar symptoms were observed across multiple driving settings.
4 Resolving Causal Misidentification
The paper resolves causal misidentification by learning policies conditioned on candidate causal graphs, then using targeted expert-query or policy-execution interventions to identify the correct graph.
- Targeted intervention: The proposed pipeline jointly learns policies for candidate causal graphs and searches the hypothesis set using targeted interventions.The interventions can use either expert queries or environmental returns.
- Graph-parameterized policy: The graph-parameterized policy πG(X) = fφ([X ⊙ G, G]) masks observations according to graph G while conditioning the policy on G.Each binary graph variable indicates whether a state variable is treated as a cause of the action.
- Expert-query intervention: Expert-query intervention selects states where candidate policies disagree, then uses expert labels to update the graph distribution.This approach targets informative states and is designed to require fewer expert queries than DAgger.
- Policy-execution intervention: Policy-execution intervention evaluates candidate policies through episode returns, using environmental rewards to infer which causal graphs are more likely.This mode is intended for settings where an expert cannot be queried during training.
- Graph search: To scale beyond individually evaluating exponentially many hypotheses, the method infers a factorized graph distribution and returns its mode.The distribution uses a linear energy model and independent Bernoulli factors over graph variables.
- Atari application: On Atari, UNIF-INTERVENTION improves over CONFOUNDED W/ VAE in all three environments and matches ORIGINAL W/ VAE on Pong and UpNDown.DROPOUT improves only UpNDown, while UNIF-INTERVENTION is the intervention method used for these experiments.
5 Experiments
Experiments across control and Atari tasks test whether targeted interventions recover causal models and close the performance gap caused by confounding, while also examining disentanglement and intervention modes.
- Experimental design: CONFOUNDED performed significantly worse than ORIGINAL across all five evaluated tasks, motivating tests of intervention speed and causal-graph recovery.The experiments cover MountainCar, Hopper, and three Atari games.
- Policy-execution interventions: After tens of policy-execution episodes, UNIF-INTERVENTION and DISC-INTERVENTION approach ORIGINAL performance on MountainCar and Hopper.GAIL reaches comparable Hopper performance only after about 1.5k episodes.
- Expert-query interventions: Expert-query interventions improve over CONFOUNDED with few queries, outperform DROPOUT and DAGGER, but GAIL with 1.5k episodes performs better on Hopper.DAGGER requires hundreds of queries on MountainCar and tens of thousands on Hopper for similar rewards.
- Ablations: DISC-INTERVENTION helps only during early MountainCar interventions and not on Hopper, making targeted intervention the main source of its performance.The exact method used to learn the policy mixture has relatively little effect on the final outcome in these settings.
- Causal-graph recovery: The learned causal graph identifies true causes and excludes nuisance correlates in Pong, whereas a random graph labels important state aspects as nuisance variables.The comparison replaces inferred nuisance dimensions or random dimensions with prior samples before regenerating frames.
- Representation ablation: Entangled MountainCar representations produce significantly lower results than disentangled representations after 30 policy-execution episodes or 20 expert queries.This supports the method’s assumption that state representations are disentangled.
6 Conclusions
The paper identifies causal misidentification as a fundamental imitation-learning problem and proposes a causally motivated resolution, while leaving realistic-scenario validation for future work.
- The paper identifies causal misidentification as a naturally occurring and fundamental problem in imitation learning.
- The proposed resolution is causally motivated, but validation so far uses simpler synthetic settings intended to mimic natural imitation-learning scenarios.
- Extending the solution to realistic scenarios and broader feedback settings remains future work.
B Necessity of Correct Causal Model
Matching interventional queries under faithfulness identifies the expert’s causal graph, showing why the correct causal model is necessary for causal imitation.
- Faithfulness means that all conditional independence relationships in the distribution are represented in the causal graph.
- The proof compares variables independent of the action in a graph with the corresponding set in its mutilated graph.This index-set construction connects interventional behavior to graph structure.
- Under faithfulness, a learner matching the expert’s interventional queries must recover the expert’s causal graph.The proposition concludes G* = Ĝ from equality of the action-independent variable sets.
C Passive Causal Discovery, Faithfulness and Determinism
Passive causal discovery can fail in imitation learning because deterministic state transitions violate faithfulness, making causes appear conditionally independent of actions.
- In CONFOUNDED MountainCar, all state variables correlate with expert actions but become mostly independent after conditioning on the confounder.This leads passive analysis to imply that none of the state variables are causes.
- Deterministic transitions are common in realistic imitation learning and make passive causal discovery inapplicable.The paper therefore requires active interventions to determine the causal model.
- The relevant causal-discovery test examines whether state variables provide information about actions once the confounder is known.
D Variational Causal Discovery
The paper uses variational Bayesian causal discovery to infer graph distributions from demonstrations and trains graph-conditioned policy mixtures while accounting for sparsity and graph diversity.
- The latent-variable architecture samples G from qψ(G|u), uses fφ for graph-specific policies, and trains bη to retain dependence between G and U.
- The variational approach infers a distribution over causal graphs and parameters whose modes fit the demonstration data.Each possible cause corresponds to a binary graph variable, yielding 2^n possible graphs.
- Entropy regularization promotes graph diversity, while mutual information encourages correlation between the latent variable and sampled graphs.A variational lower bound makes the intractable entropy objective tractable.
- The graph prior prefers fewer causes for the action, imposing sparsity during causal discovery.
- Because G is discrete, the method uses the Gumbel Softmax trick for gradients while training the policy network with standard backpropagation.
- A graph-conditioned mixture of predictors jointly trains policies that attend to different sparse subsets of the inputs.The learned qψ(G) represents hypotheses over graphs, while each graph defines an imitation policy.
- The method can initialize the intervention prior with the discovered graph distribution rather than a uniform distribution.
E Additional Results: Diagnosing Causal Misidentification
Across several environments, behavioral cloning can achieve near-zero demonstration losses while the confounded policy earns significantly lower deployed reward, confirming causal misidentification.
- Near-zero training and validation losses can coexist with significantly lower reward for the confounded policy when deployed.The discrepancy appears across several environments and confirms causal misidentification.
F DAgger with many more interventions
DAgger requires substantially more intervention samples to achieve rewards comparable to a non-DAgger imitator trained on the original state.
- Hundreds of samples are required before DAgger reaches rewards comparable to a non-DAgger imitator trained on the original state.
G GAIL Training Curves
GAIL training shows no significant difference between original and confounded states, although training requires many environment interactions.
- GAIL's average training curves on original and confounded states do not differ significantly.
- The curves report average training performance with error bars representing two standard errors of the mean.
- The posterior over causal graphs can be evaluated using likelihoods and a prior, but many graphs make exact inference computationally impractical.
- A structured policy family can improve sample efficiency at the cost of no longer exactly inferring the policy.
- The variational approach introduces a temperature parameter τ.
- The assumed variational family consists of independent distributions.
- The objective can be optimized as an entropy-regularized MDP using a linear Q function learned by regression on off-policy graph-reward pairs.