PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Influence Detection for Improving Efficiency in Reinforcement Learning

Maximilian Seitzer, Bernhard Schölkopf, Georg Martius

arXiv:2106.03443v2cs.LG

TL;DR

RL training can require countless hours of interaction, and control over other entities varies by situation. This paper measures situation-dependent causal influence and integrates it into exploration and off-policy learning, reporting improved sample efficiency in robotic manipulation.

  • Problem

    Training RL agents for challenging tasks can require countless hours of interaction, while an agent’s causal influence over other entities varies with the situation.

    Source: §1 · p. 1

  • Method

    The paper derives a situation-dependent causal influence measure and integrates it into exploration bonuses, action exploration, and experience-replay prioritization.

    Source: §1 · pp. 1–2, §6.1 · p. 7

  • Results

    The influence-based applications show strong sample-efficiency improvements in robotic manipulation; on FetchPickAndPlace, the agent reaches 60% success rate four-times faster than baseline (DDPG+HER) without a bonus.

    Source: §6.1 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The same influence measure can support exploration and off-policy learning, and the paper reports improved data efficiency across these applications.

    Source: §7 · p. 9

    The approach assumes full state observability and a factorization into causal variables, and accurate influence estimates rely on a correct model.

    Source: §7 · p. 9

Abstract

from arXiv · show

Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of \emph{situation-dependent causal influence} based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

1 Introduction

RL training for manipulation can require countless hours of interaction, making sample efficiency a key concern. The paper uses situation-dependent causal influence to guide exploration and learning, reporting improved data efficiency in robotic manipulation.

1 Introduction

  • Sample efficiency is a key concern because training reinforcement learning agents for challenging tasks can require countless hours of interaction.The introduction cites simulation-to-real transfer and heavy physical parallelization [2] as approaches that have achieved impressive results.Source: §1 · p. 1
  • The paper formalizes situation-dependent control, derives a causal influence measure, and presents a practical method to compute it.The motivating example is a robotic arm that can move an object only when contact is made.Source: §1 · p. 1
  • Learning should focus on states where the robot controls the object because those states are initially rare, involve hard-to-model contacts, and enable further manipulation goals.The introduction argues that causal influence should also guide exploration so the agent gathers data containing these relevant states.Source: §1 · p. 1
  • An influence bonus can help an agent discover useful behavior even without task-specific rewards, or improve learning efficiency when combined with task rewards, especially in sparse-reward settings.The paper proposes using the measure as intrinsic motivation or as an exploration bonus.Source: §1 · p. 1
  • The proposed influence measure supports both exploration bonuses and off-policy prioritization, with empirical evaluations reporting improved data efficiency in robotic manipulation environments.The approach can also guide exploration toward actions with higher predicted causal impact as an alternative to ϵ-greedy exploration.Source: §1 · pp. 1–2

2 Related Work

The paper situates its approach among work on sparse causal structure, causal influence measures, and exploration. It emphasizes situation-specific influence and its use in both exploration and learning.

2 Related Work

  • The paper argues that localized, sparse influence can improve RL algorithms, building on assumptions that entities act mostly independently and that influence is spatially localized and sparse in time.It connects these assumptions to independent causal mechanisms and the sparse mechanism shift hypothesis [4] [5].Source: §2 · p. 2
  • Causal influence detection concerns whether changing one causal variable affects another, while quantifying an established causal impact has also been studied through measures such as transfer entropy and information flow [14] [18].The paper notes that causal discovery is unidentifiable in general, though assumptions can permit discovery [12].Source: §2 · p. 2
  • Unlike Pitis et al.’s influence-based counterfactual data augmentation [24], this work also uses influence to guide exploration and reports better action-influence detection than their heuristic.The paper also contrasts its situation-specific approach with mutual-information exploration that does not distinguish the agent’s individual situations [37].Source: §2 · p. 2

3 Background

The background models environments as factored state spaces of mostly independent entities and uses causal graphs to represent transitions. It then motivates a but-for account of causation while noting that this account identifies agent-level influence rather than action-specific causes.

3 Background

  • The environment is modeled as an MDP whose state space factors into entity-specific components, reflecting the assumption that real-world entities mostly behave independently.The global transition graph allows possible effects between components, while a particular situation may have fewer active influences.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Causal Graphical Models

  • A causal graphical model represents one-step transitions with a directed graph and conditional distributions that factorize the joint distribution.Interventions are modeled by replacing a variable’s conditional distribution, so a causal graphical model represents a set of distributions rather than a single one.Source: §3.1 · p. 3
  • Although the global graph is fully connected across time steps, the paper focuses on the local causal model for a particular state, where the agent’s action may affect only some entities.A local model is induced by conditioning on an observed state and removing edges until the conditional distribution is causally minimal.Source: §3.1 · p. 3

3.2 The Cause of an Effect

  • The paper adopts a but-for test: an action is a cause of an outcome when that outcome would differ under another value of the action.The text contrasts this algorithmic test with actual-causation accounts that require a normative outcome or observation of the world without the agent’s intervention.Source: §3.2 · pp. 3–4
  • The but-for test can treat every action as a cause when alternative actions would produce different outcomes, so the method identifies whether the agent influences other entities, not which individual actions are causes.Source: §3.2 · p. 4

4 Causal Influence Detection

The section characterizes control as state-dependent causal influence of actions, measures it with conditional mutual information, and estimates that measure from learned transition distributions.

4 Causal Influence Detection

  • Control of S′_j in state s corresponds to an action-to-next-state causal edge, characterized by dependence between S′_j and A conditional on S=s.Dependence under one policy generalizes across full-support policies; independence under one full-support policy implies no control under any intervention.Source: §4 · p. 4

4.1 Measuring Causal Action Influence

  • Conditional mutual information I(S′_j; A | S=s) is the causal action influence measure, thresholded to classify control.The measure uses a uniform action policy to support policy-independent control detection.Source: §4.1 · p. 4
  • CAI accounts for system uncertainty that actions cannot reduce, and is a lower bound on one-step empowerment that is easier to compute.Unlike empowerment, CAI is entity-specific rather than treating the state space as monolithic.Source: §4.1 · pp. 4–5

4.2 Learning to Detect Control

  • The estimator learns transition distributions, samples K actions to approximate the transition marginal and expectation, and uses a Gaussian assumption to simplify the KL calculation.The finite-mixture estimator is a lower bound converging to true mutual information as K increases, assuming the true conditional density.Source: §4.2 · p. 5
  • Training and sampling policies need overlapping action support; using different policies can bias the model, although training on all data performed better in practice.The authors note that collecting only under the random sampling policy may yield uninteresting data in most environments.Source: §4.2 · p. 5
  • Detection evaluation reports ROC curves for 1DSLIDE and FETCHPICKANDPLACE and average precision under added state noise for FETCHPICKANDPLACE.Table 1 reports AUC, average precision, and best achievable F-score.Source: Figure 2 · p. 6, Table 1 · p. 6

5 Empirical Evaluation of Causal Influence Detection

CAI reliably detects causal influence in both tested environments, while the other baselines do not; its performance declines gracefully as observation noise increases. The evaluation uses simulated ground-truth causal graphs for 1DSLIDE and a movement-range heuristic for FetchPickAndPlace.

5 Empirical Evaluation of Causal Influence Detection

  • The tests cover 1DSLIDE and FetchPickAndPlace, with added Gaussian observation noise in the latter; ground-truth influence labels come from simulation for 1DSLIDE and a movement-range heuristic for FetchPickAndPlace.The target variables are the object coordinates.Source: §5 · p. 6
  • CAI reliably detects causal influence, while none of the compared baselines does; its performance drops gracefully as observation noise increases.The comparisons use five random seeds.Source: §5 · p. 6

6 Improving Efficiency in Reinforcement Learning

The section integrates causal action influence (CAI) into reward-based and action-based exploration and off-policy replay prioritization. These approaches improve sample efficiency in robotic manipulation tasks, with CAI-P matching or exceeding an energy-based baseline and improving over PER.

6.1 Intrinsic Motivation to Seek Influence

  • Using only rCAI as reward, the agent learned to grasp, lift, and hold the object after 2,000 episodes.The reward was measured on object coordinates, without a task-specific reward.Source: §6.1 · p. 7, Figure 3 · p. 7
  • At 60% success, the CAI reward bonus reached that success rate four times faster than DDPG+HER without a bonus.The bonus scale must be selected appropriately; if too high, it dominates the task reward.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 Actively Exploring Actions with Causal Influence

  • Combining the CAI reward bonus with active exploration yielded the largest sample efficiency among the compared exploration strategies.The displayed CAI setting uses 100% active exploration and λbonus = 0.2.Source: Figure 6 · p. 8, §6.2 · p. 8
  • Selecting exploratory actions by predicted causal impact improved sample efficiency roughly twofold over random exploration, and any tested amount of active selection helped.The experiments used a total exploratory-action fraction of 30%.Source: §6.2 · pp. 7–8, Figure 5 · p. 8

6.3 Causal Influence-based Experience Replay

  • CAI-P samples episodes more often when their total influence ranks higher, then samples states uniformly within each selected episode.Episode-level prioritization preserves sampling of non-influence states so TD updates can propagate return information to them.Source: §6.3 · pp. 8–9
  • CAI-P was on par with or better than EBP and improved learning speed over PER by 1.5–2.5× at 60% success.The comparison includes EBP with privileged information and PER.Source: Figure 7 · p. 9, §6.3 · p. 9

7 Discussion

The paper presents situation-dependent causal influence as a tool for improving reinforcement learning efficiency. It reports gains from using the measure in exploration and replay prioritization, while identifying assumptions and model-estimation challenges that limit the approach.

7 Discussion

  • CAI supports exploration bonuses, active action exploration, and replay prioritization, with each application yielding strong sample-efficiency improvements.The paper also identifies credit assignment as a possible future use.Source: §7 · p. 9
  • The approach assumes full observability and an available factorization into causal variables, and accurate estimation depends on a correct model.The authors note that deep networks can struggle to capture action–entity causal relationships.Source: §7 · p. 9
  • The authors identify influence detection between entities as future work for modeling multi-step influences of the agent on the environment.They connect this direction to modeling indirect interventions.Source: §7 · p. 9

A.1 Proof of Propositions

The appendix gives proofs and approximation details for the causal influence measure, then describes the manipulation environments and procedures used to evaluate it.

A.1 Proof of Propositions

  • In the local causal graph, conditional dependence between an action and an entity’s next state corresponds to a direct action-to-state edge.The proof relies on the Markov property, causal minimality, and the assumption of no instantaneous effects.Source: §A.1 · p. 15

A.3 Proof that CAI is a Pointwise Version of Janzing et al.’s Causal Strength

  • CAI is a pointwise version of causal strength for policies whose action distribution is independent of the state.The correspondence uses action, next-state entity, and current state as the respective variables.Source: §A.3 · p. 16

A.4 Approximating the KL Divergence Between a Gaussian and a Mixture of Gaussians

  • The KL approximation averages product and variational approximations, which provide lower and upper bounds, respectively.The resulting approximation is thresholded at zero in practice because it can otherwise be negative.Source: §A.4 · p. 17
  • The KL approximation assumes independent dimensions by parameterizing covariance as a diagonal matrix.Source: §A.4 · p. 17

B.1 1D-Slide

  • In 1D-Slide, influence labels can be derived by testing maximum acceleration in both directions and checking for contact and a changed object state.The agent and object move along a line, and the agent cannot pass the environment’s center.Source: §B.1 · p. 18, Figure S1

B.2 FetchPickAndPlace

  • For FetchPickAndPlace, the heuristic labels control when the object lies within an estimated end-effector movement ellipsoid or contact occurs next step.The ellipsoid is built from a lookup table of end positions for sampled starting locations and maximal actions.Source: §B.2 · p. 18

B.3 FetchRotTable

  • FetchRotTable adds periodic table rotation, creating another source of object movement that CAI must distinguish from the agent’s actions.The state includes table-angle and rotation information; 90% of goals are in the air and 10% are on the table.Source: §B.3 · pp. 18–19

C Additional Results for Influence Evaluation

The additional evaluation examines detection under a harder random-policy dataset and qualitatively compares CAI scores with heuristic influence labels across manipulation scenarios.

C Additional Results for Influence Evaluation

  • On the random-policy dataset, only 3.3% of transitions are positive, detection performance drops but remains good, and performance is relatively robust to the number of sampled actions K.The dataset excludes easier cases where the agent moves the object in the air; the authors caution that more sampled actions may matter in edge cases.Source: C Additional Results for Influence… · pp. 19–20
  • In the brief-touch trajectory, CAI spikes near contact, but misses heuristic-labeled influence while the agent hovers near the object.The authors note that the heuristic ground-truth labels can also mispredict.Source: Figure S4 · p. 20, C Additional Results for Influence… · p. 20

D Additional Results for Reinforcement Learning

The supplementary RL results examine CAI’s score patterns, combinations of its proposed improvements, and variants of exploration baselines.

D Additional Results for Reinforcement Learning

  • After the agent learns to lift the object, CAI scores peak, while binary ground-truth labels assign high scores more uniformly.The plotted CAI score distribution approximates the ground-truth label distribution.Source: Figure S5 · p. 21, D Additional Results for Reinforcement… · p. 21
  • Among the tested VIME variants, using agent and object positions performs best; among ensemble-disagreement variants, using object position performs best.These variants are selected for comparison in the main paper.Source: D Additional Results for Reinforcement… · p. 21
  • Proportional episode-score prioritization learns more slowly than ranked prioritization and does not reach 100% success, while within-episode state prioritization and prioritized HER goals yield no improvement.Importance sampling was not needed for the tested tasks, though the authors note it could be required elsewhere.Source: D Additional Results for Reinforcement… · p. 21
  • Flat-threshold and reward-multiplicative exploration bonuses perform worse than additive bonuses, and ranked or proportional action sampling performs worse than choosing the maximum-CAI action.Annealing the bonus to zero also did not improve results.Source: D Additional Results for Reinforcement… · p. 21
  • Combining two or all three CAI improvements yields a 4–10× increase in sample efficiency over using no CAI, with all three performing best.Prioritization provides the largest benefit on its own, and exploration bonus and active action selection add complementary improvements.Source: D Additional Results for Reinforcement… · p. 21, Figure S6 · p. 22

E Settings of Influence Detection Evaluation

The influence-detection evaluation uses held-out datasets from two robotic tasks, with an additional FETCHPICKANDPLACE test set that has much less object interaction. It compares CAI with entropy and Transformer attention scores and specifies the data and training procedures used to obtain them.

Datasets

  • 1DSLIDE and FETCHPICKANDPLACE use separate training, validation, and test datasets collected from DDPG+HER runs, with the 1DSLIDE test set also including random-policy episodes.FETCHPICKANDPLACE training and validation use 5,000 episodes each; its test set has 7,500 episodes and 45.3% positive samples.Source: E Settings of Influence Detection… · p. 22
  • A second FETCHPICKANDPLACE test set contains 5,000 random-policy episodes and only 3.3% positive samples, reflecting considerably less object interaction.Source: E Settings of Influence Detection… · p. 23

Methods

  • CAI scores use 64 sampled actions, while entropy uses the same trained model and Gaussian conditional-entropy estimation; Transformer attention scores link action-input and object-output positions.The Transformer predicts object position, and its total attention matrix combines the attention matrices across layers.Source: E Settings of Influence Detection… · p. 23

Training

  • Models predict next-state differences and use Adam, with early stopping based on validation MSE; FETCHPICKANDPLACE targets are rescaled by 50.The CAI model uses separate mean and variance output layers; variance is constrained positive, clipped at 200, and spectral normalization reduces training instability.Source: E Settings of Influence Detection… · p. 23

F Settings of Reinforcement Learning Experiments

The RL experiments use goal-conditioned, sparse-reward tasks and shared base settings across algorithms. CAI scores guide replay prioritization or exploration bonuses, alongside specified baseline implementations and training schedules.

Goal-conditioned setting

  • Episodes succeed when the goal is reached on the final step, with sparse rewards of 0 at the goal and −1 otherwise; success rates average 100 evaluations every 200 episodes.On FETCH tasks, goals specify object coordinates, which are added to policy and value-function inputs.Source: F Settings of Reinforcement Learning… · pp. 23–24

Shared RL settings

  • On FETCHPICKANDPLACE, DDPG+HER settings from Ren et al. [68] reached 90% success 4× faster than original settings from Plappert et al. [55].The experiments use these settings across DDPG+HER, CAI, VIME, EBP, and PER, and seed replay memory with 200 random-policy episodes.Source: F Settings of Reinforcement Learning… · p. 24

CAI implementation

  • CAI scores are stored with collected episodes and recomputed after model training; fully recomputing every 1,000 epochs instead of every round causes only a small performance loss.The model trains every 100 episodes, and the training-batch schedule is varied over training.Source: F Settings of Reinforcement Learning… · p. 24

Exploration bonus

Baselines

  • Ensemble disagreement has two variants: one computes bonuses from full-state predictions during collection, while the other uses object position and periodically recomputes bonuses.VIME, PER, and EBP use adapted or independently implemented baseline implementations.Source: F Settings of Reinforcement Learning… · pp. 24–25

Related papers

Loading 2106.03443v2…