Source-linked AI summary
State-Only Imitation Learning for Dexterous Manipulation
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, Jitendra Malik
TL;DR
Dexterous manipulation is sample-intensive, and action-labeled demonstrations are difficult to obtain for settings such as learning from videos. SOIL jointly trains a policy and inverse dynamics model to infer actions for state-only demonstrations; it performs on par with state-action approaches and outperforms RL without demonstrations.
Problem
Dexterous manipulation challenges sample-efficient RL, while action-labeled demonstrations are difficult to collect and cannot be readily obtained from internet videos.
Source: §I · p. 1Method
SOIL jointly trains an inverse dynamics model to predict actions between demonstration states and a policy using RL and those predicted-action demonstrations.
Source: §I · pp. 1–2Results
SOIL performs on par with state-action approaches and considerably outperforms RL without demonstrations, while leveraging demonstrations with different dynamics, morphologies, and objects.
Source: §I · p. 2, §VI · p. 6Takeaways & Limitations
The results suggest state-action demonstrations may not be necessary for imitation learning and that SOIL can use demonstrations with mismatched dynamics, morphologies, and objects.
Source: §I · p. 2
Abstract
from arXiv · showhide
Modern model-free reinforcement learning methods have recently demonstrated impressive results on a number of problems. However, complex domains like dexterous manipulation remain a challenge due to the high sample complexity. To address this, current approaches employ expert demonstrations in the form of state-action pairs, which are difficult to obtain for real-world settings such as learning from videos. In this paper, we move toward a more realistic setting and explore state-only imitation learning. To tackle this setting, we train an inverse dynamics model and use it to predict actions for state-only demonstrations. The inverse dynamics model and the policy are trained jointly. Our method performs on par with state-action approaches and considerably outperforms RL alone. By not relying on expert actions, we are able to learn from demonstrations with different dynamics, morphologies, and objects. Videos available at https://people.eecs.berkeley.edu/~ilija/soil .
I. INTRODUCTION
SOIL addresses the difficulty of collecting state-action demonstrations by learning from states alone. It jointly trains a policy and inverse dynamics model, performing on par with state-action approaches and considerably outperforming RL without demonstrations.
I. INTRODUCTION
- Dexterous manipulation is difficult for RL because high-dimensional action spaces challenge exploration and optimization, while effective reward functions are hard to specify.Imitation learning is presented as a promising avenue for sample-efficient learning and could use internet-scale videos of people.Source: §I · p. 1
- State-action demonstrations can be time-consuming to collect, limited to a subset of tasks, and unsuitable for using internet videos that lack action labels.Requiring demonstrator actions also constrains policy flexibility and can hinder learning from demonstrations collected in different settings.Source: §I · p. 1
- SOIL performs on par with state-action approaches and considerably outperforms RL without demonstrations across four simulated dexterous manipulation tasks.It also leverages demonstrations with different dynamics, morphologies, and objects, while using demonstrator actions degrades performance.Source: §I · p. 2
- SOIL jointly trains a policy with RL and demonstrations using predicted actions, while training the inverse model through self-supervision.The policy can rely on demonstrations early in training and later surpass them.Source: §I · pp. 1–2
II. RELATED WORK
Related work spans reinforcement learning and several forms of imitation and dynamics learning. SOIL differs by combining a task-reward RL objective with inverse-model-predicted actions from state-only demonstrations.
Dexterous manipulation
- RL policies for dexterous manipulation have shown promise in simulation,, but large-scale training is required and generalization to new environments is difficult.Optimization and planning approaches,, are described as often insufficient for complex tasks.Source: §II · p. 2
Imitation learning with BC
- Behavior cloning requires expert actions and caps policy performance at the expert level, while SOIL uses an RL objective and is not so bounded.The related inverse-model method also trains an inverse model with the policy, but its behavior-cloning policy remains limited by expert performance.Source: §II · p. 2
RL with demonstrations
- Prior RL-with-demonstrations methods use off-policy replay or on-policy auxiliary terms, but the discussed approaches require state-action demonstrations and cannot directly use SOIL’s state-only demonstrations.Off-policy methods can be more sample efficient but are described as less stable and scaling worse in high-dimensional spaces,.Source: §II · p. 2
Inverse RL
- Inverse-reinforcement-learning density models can be inefficient for high-dimensional dexterous tasks, and adversarial objectives can collapse into suboptimal modes and destabilize learning.The cited example estimates whether trajectories come from demonstrations or the policy and uses that estimate as reward.Source: §II · p. 2
Learning dynamics models
- Prior inverse-model approaches typically pretrain the model to complete expert demonstrations, whereas SOIL jointly trains it with an RL policy for high-dimensional continuous manipulation.The cited similar approach focuses on games with discrete actions.Source: §II · p. 2
Following demonstrations
- Policies trained to follow demonstrations can reproduce a particular trajectory, whereas SOIL uses a goal-conditioned policy that optimizes task reward.The cited work demonstrates human-motion imitation from videos.Source: §II · p. 2
III. PRELIMINARIES
The preliminaries introduce policy-gradient learning and DAPG, which adds an imitation objective to the policy-gradient update. Figure 2 depicts SOIL’s joint policy and inverse-model training.
Reinforcement learning
- Policy-gradient methods directly maximize expected discounted reward through gradient ascent, and the experiments use natural policy gradient.The vanilla policy-gradient update uses the advantage function Aπ(s, a).Source: §III · p. 2
Imitation learning
- DAPG augments the vanilla policy gradient with an auxiliary imitation term over state-action demonstrations, weighted by scalar hyperparameters λ0 and λ1.The method is attributed to.Source: §III · p. 2
- SOIL trains a policy and inverse model jointly: policy-generated trajectories supervise the inverse model, which predicts actions for state-only demonstrations.Figure 2 labels these as self-supervised inverse-model training (a) and demonstration action prediction (b).Source: Figure 2 · p. 3
IV. STATE-ONLY IMITATION LEARNING
The section frames imitation from states alone as a practical setting because state estimates can be easier to obtain than demonstrator actions, including from videos. SOIL addresses the missing actions by iteratively training an inverse dynamics model and policy together.
A. Problem Setup
- The state-only setting uses demonstrations without actions; state estimates from human videos may be available when human action estimates are not.The setup assumes direct access to hand joint angles, applied joint forces, joint speeds, and object locations, keeping state estimation outside the investigation.Source: A. Problem Setup · p. 3
- Standard behavior-cloning and DAPG objectives require ground-truth actions, so the framework instead predicts actions between consecutive demonstration states.The inverse model takes consecutive states as input and estimates the intervening action.Source: A. Problem Setup · p. 3
- SOIL iteratively trains the policy and inverse model together: policy-generated trajectories provide inverse-model training data, and predicted demonstration actions feed the policy update.This addresses the coupled challenge that random exploration is impractical in the large action space, while policy learning needs accurate inverse-model predictions.Source: A. Problem Setup · p. 3
B. Inverse Dynamics Model
The inverse dynamics model predicts actions between consecutive states and is trained from the agent’s own interactions. It is described as adaptable across settings and agent-specific.
B. Inverse Dynamics Model
- The inverse dynamics model takes consecutive states as input and predicts the action between them.It is trained self-supervised using an L2 loss on state-action triplets collected from policy trajectories.Source: B. Inverse Dynamics Model · p. 3
- Training the inverse model is described as adaptable to demonstrations from different settings and agent-specific, potentially supporting generalization to new settings.The stated example is that the agent’s hand works the same way even when the task is new.Source: B. Inverse Dynamics Model · pp. 3–4
- Across four dexterous manipulation tasks, SOIL performs comparably to DAPG and considerably better than pure RL.The largest gap over pure RL is on object relocation, which the passage identifies as the most challenging task.Source: Figure 3 · p. 4
- In selected episodes, SOIL is comparable to DAPG and more realistic than pure RL on object relocation, with similar trends reported for three other tasks.Those tasks are in-hand pen manipulation, door opening, and tool use.Source: Figure 4 · p. 4
C. Joint Training Procedure
SOIL jointly trains an inverse dynamics model and a policy, using predicted demonstration actions in an auxiliary imitation term. That term is annealed so training can shift from demonstrations toward policy gradients.
C. Joint Training Procedure
- Predicted actions between consecutive demonstration states form state-action pairs that contribute an auxiliary imitation term to the policy gradient.The method adjusts the DAPG objective to use actions estimated by the inverse model.Source: C. Joint Training Procedure · p. 4
- The policy and inverse model are jointly optimized, while the imitation term is annealed toward zero during training.The described schedule uses λ1 < 1 and increases k with training iterations.Source: C. Joint Training Procedure · p. 4
- The procedure uses demonstrations for early exploration, then transitions toward policy gradients that may allow the policy to surpass the demonstrations.The two objectives are optimized jointly during training.Source: C. Joint Training Procedure · p. 4
V. EXPERIMENTS
The experiments evaluate SOIL in simulation against state-action and state-only methods, with ablations and a study of demonstration mismatch. The setup uses a dexterous-hand simulator and four manipulation tasks.
V. EXPERIMENTS
- The evaluation includes controlled comparisons with state-action and state-only methods, ablations, and robustness to demonstration mismatch.Source: §V · p. 4
A. Experimental Setup
- Experiments use MuJoCo with the five-finger, 24-degree-of-freedom ADROIT hand across four tasks, using 25 demonstrations per task and NPG for all methods.State-only methods ignore the actions in the demonstrations.Source: A. Experimental Setup · p. 4
V. EXPERIMENTS
- SOIL outperforms Chamfer-distance state matching and density estimation on both object relocation and in-hand pen manipulation.The passage identifies object relocation as the more challenging of the two tasks.Source: Figure 5 · p. 5
- On object relocation, performance depends on imitation-term scale, and annealing that term to zero is beneficial.The reported comparison says both too-small and too-large weights lead to suboptimal performance.Source: Figure 6 · p. 5
B. State-Action Comparisons
SOIL reaches DAPG’s performance while considerably outperforming RL without demonstrations. Its policies resemble DAPG qualitatively, and ablations connect performance to inverse-model quality and training choices.
Pure RL comparisons
- Across four tasks, SOIL outperforms pure RL, with the largest gap on object relocation and surprisingly strong pure-RL performance on hammer.The authors treat pure RL as a lower bound for SOIL.Source: B. State-Action Comparisons · p. 5
DAPG comparisons
- Across four tasks, SOIL comes close to DAPG, suggesting access to ground-truth actions may not be critical for well-performing imitation.The authors treat DAPG as an upper bound because SOIL’s inverse model is unlikely to predict actions better than ground truth.Source: B. State-Action Comparisons · p. 5
Qualitative examples
- SOIL’s selected episodes look comparable to DAPG and more realistic than pure RL, whose policy completes the task without realistic behavior.DAPG’s behavior is described as much more realistic than pure RL’s.Source: B. State-Action Comparisons · p. 5
Inverse dynamics model
- On hammer, lower inverse-model error correlates with higher policy return.The figure compares inverse-model quality on the left with policy performance on the right.Source: Figure 7 · p. 5
Model-based RL enhancements
- On relocation, replay-buffer aggregation reduces inverse-model training variance, while an ensemble shows no performance improvement.The method adopts the replay buffer and omits the ensemble.Source: Figure 8 · p. 5
Summary
- SOIL reaches state-action DAPG’s performance while considerably outperforming RL without demonstrations.The results suggest expert actions may not be necessary for imitation learning.Source: B. State-Action Comparisons · p. 5
C. State-Only Comparisons
SOIL outperforms state-only matching and density-estimation baselines, especially on challenging object relocation. It also outperforms state-action DAPG when demonstrations differ in dynamics, morphology, or objects.
Density estimation
- The density-estimation baseline distinguishes states from the current policy from states in demonstrations, offering a data-driven alternative to distance-based matching.The authors note that distance matching depends on domain knowledge and careful distance-function tuning.Source: C. State-Only Comparisons · p. 5
Comparisons
- On object relocation, SOIL considerably outperforms state-matching and density-estimation baselines; on easier pen-in-hand manipulation, both baselines work reasonably well but trail SOIL.The state-matching baseline uses Chamfer distance, selected from tested distance-function variants as the best-performing approach.Source: C. State-Only Comparisons · p. 5
Demonstration mismatch
- For object relocation with mismatched dynamics, morphology, or objects, SOIL outperforms state-action DAPG.The authors suggest state-only imitation may be preferable when demonstrator and learner settings differ.Source: Figure 9 · p. 6
D. Ablation Studies
The ablations show that imitation-term scale and scheduling matter, policy performance tracks inverse-model quality, and replay aggregation reduces variance without a clear ensemble benefit.
Imitation term weights
- Too small or too large an imitation-term weight leads to suboptimal performance, while annealing the term to zero considerably improves results.The authors report demonstrations help early exploration but begin to hurt as the policy improves and exceeds them.Source: D. Ablation Studies · p. 6
Inverse dynamics model
- A low-error inverse dynamics model corresponds to high policy return, while a high-error model corresponds to low return.The authors conclude inverse-model quality plays an important role in SOIL’s effectiveness.Source: D. Ablation Studies · p. 6
Model-based RL enhancements
- Replay-buffer training reduces inverse-model variance, but an ensemble of inverse dynamics models provides no clear benefit.SOIL uses the replay buffer and does not use an ensemble.Source: D. Ablation Studies · p. 6
E. Robustness to Demonstration Mismatch
SOIL is reported to match state-action imitation while outperforming RL without demonstrations, and to use demonstrations across changes in dynamics, morphology, and objects. In robustness tests, it outperforms DAPG across those three kinds of mismatch.
E. Robustness to Demonstration Mismatch
- The robustness experiments test object relocation with original demonstrations across different dynamics, morphologies, and objects, where original actions may not help and may hurt.The stated motivation is to test whether state-only imitation is more robust than state-action approaches in these settings.Source: E. Robustness to Demonstration Mismatch · p. 6
- With hand mass increased by 16× and 32×, SOIL outperforms action-based DAPG under different dynamics.The heavier hand requires a different action distribution, including more force and accounting for greater momentum.Source: E. Robustness to Demonstration Mismatch · p. 6
- With two to three fingers removed, SOIL performs better than DAPG in both tested hand configurations.The reduced-hand settings require a different strategy from the full hand.Source: E. Robustness to Demonstration Mismatch · p. 6
- For two object variants, including relocating a banana from ball demonstrations, SOIL outperforms DAPG, with larger gains in the harder setting.The passage also reports a clear gap in the relatively easier cube case.Source: E. Robustness to Demonstration Mismatch · p. 6
VI. CONCLUSION
- SOIL matches state-action approaches, considerably outperforms RL without demonstrations, and leverages demonstrations across different dynamics, morphologies, and objects.The authors present these outcomes as a step toward learning from internet videos of humans.Source: §VI · p. 6