Source-linked AI summary
Deep Dynamics Models for Learning Dexterous Manipulation
Anusha Nagabandi, Kurt Konoglie, Sergey Levine, Vikash Kumar
TL;DR
Dexterous manipulation demands precise coordination of contacts, but existing learning methods can be data-hungry or insufficiently flexible for complex tasks. PDDM combines uncertainty-aware deep dynamics models with improved online model-predictive control. It consistently outperforms prior model-based and model-free methods, including challenging real-world manipulation learned from limited data.
Problem
Dexterous manipulation requires balancing complex contacts, while existing learning approaches can require large datasets or fail to scale to fine-motor tasks.
Method
PDDM combines uncertainty-aware deep dynamics models with gradient-free trajectory optimization and online model-predictive control.
Results
PDDM consistently outperforms prior model-based and model-free reinforcement-learning methods in learning speed and final performance across dexterous manipulation tasks.
Takeaways & Limitations
PDDM learns flexible dexterous skills, including free-floating-object manipulation and user-specified handwriting, using substantially less training data than model-free reinforcement learning.
Takeaways & Limitations
Random-shooting planning scales poorly with planning-horizon and action-space dimensionality and often fails to produce meaningful behavior.
Abstract
from arXiv · showhide
Dexterous multi-fingered hands can provide robots with the ability to flexibly perform a wide range of manipulation skills. However, many of the more complex behaviors are also notoriously difficult to control: Performing in-hand object manipulation, executing finger gaits to move objects, and exhibiting precise fine motor skills such as writing, all require finely balancing contact forces, breaking and reestablishing contacts repeatedly, and maintaining control of unactuated objects. Learning-based techniques provide the appealing possibility of acquiring these skills directly from data, but current learning approaches either require large amounts of data and produce task-specific policies, or they have not yet been shown to scale up to more complex and realistic tasks requiring fine motor skills. In this work, we demonstrate that our method of online planning with deep dynamics models (PDDM) addresses both of these limitations; we show that improvements in learned dynamics models, together with improvements in online model-predictive control, can indeed enable efficient and effective learning of flexible contact-rich dexterous manipulation skills -- and that too, on a 24-DoF anthropomorphic hand in the real world, using just 4 hours of purely real-world data to learn to simultaneously coordinate multiple free-floating objects. Videos can be found at https://sites.google.com/view/pddm/
1 Introduction
Dexterous manipulation requires coordinating many joints and complex contact forces, while repeated contact changes make accurate analytic models difficult. PDDM combines learned dynamics and online planning to address these challenges across simulated and real-world tasks.
- Multi-fingered hands provide redundant degrees of freedom needed to control object movement through forces and motions from multiple directions.
- Repeatedly establishing and breaking contacts makes dexterous manipulation difficult for analytic approaches that require accurate system-physics models.
- PDDM combines uncertainty-aware neural-network models with gradient-free trajectory optimization for deep model-based reinforcement learning.
- The experiments scale from a 9-DoF three-fingered hand rotating a valve to a 24-DoF anthropomorphic hand performing handwriting and free-floating-object manipulation.
- The task suite spans valve rotation, in-hand reorientation, handwriting, and manipulation of Baoding balls in simulation and the real world.
2 Related Work
Prior dexterous-manipulation methods use analytic planning, demonstrations, or learned models, but data-intensive learning remains difficult to apply in the real world. PDDM targets sample-efficient and flexible learning without expert demonstrations.
- Earlier approaches planned finger gaits or in-hand rotation using object geometry, offline simulation, reduced pose bases, or mechanics-based motion planning.
- Deep reinforcement-learning methods can learn manipulation policies but require large amounts of system interaction, limiting real-world applicability.
- PDDM is designed to improve sample efficiency without expert demonstrations while retaining data-driven learning for dexterous skills.
- Model-based reinforcement learning combines efficient learning with flexible task execution through learned high-capacity or probabilistic dynamics models.
3 Deep Models for Dexterous Manipulation
PDDM learns deep dynamics models and uses them for short-horizon, closed-loop online planning. Its ensemble uncertainty, filtered action sampling, and reward-weighted refinement are designed for high-dimensional, precise manipulation.
- PDDM extends deep model-based reinforcement learning to dexterous manipulation by combining model-predictive control with ensemble-based uncertainty estimation.
- The learned transition model approximates unknown dynamics from observed data, then supports online planning rather than directly learning a policy.
- Deep neural networks provide sufficient capacity for complex dexterous interactions, while separate random initializations and data batches form an ensemble.
- At each time step, MPC optimizes a short-horizon action sequence using model-predicted outcomes and executes only the first action before replanning.
- Random Shooting: Random shooting scales poorly with planning horizon and action-space dimension because random sequences often fail to produce meaningful behavior.
- Filtering and Reward-Weighted Refinement: CEM refines sampling distributions using top-scoring action sequences, but PDDM instead uses covariance-aware, softer reward-weighted updates for coordination and precision.
- Filtering and Reward-Weighted Refinement: Filtering couples action samples across time steps to produce smoother sequences and reduce the effective search dimensionality.
4 Experimental Evaluation
The evaluation tests PDDM across increasingly challenging dexterous manipulation tasks, design choices, comparisons with prior methods, and real-world Baoding-ball control. PDDM performs strongly across simulated tasks and learns real-world two-ball rotations from entirely real-world experience.
- 4.1 Task Suite: The evaluation spans valve turning, in-hand cube reorientation, handwriting, and two-object Baoding-ball manipulation, targeting coordination, contact dynamics, precision, and failure-sensitive control.The tasks progress from a simpler valve task to free-floating-object manipulation, precise pencil control, and interacting objects.
- 4.2 Ablations and Analysis of Design Decisions: Large model architectures are crucial, while ensembles reduce early overfitting and overconfident harmful behavior during learning.The experiments report similar trends across tasks and find little difference between randomly resetting and warmstarting model weights.
- 4.2 Ablations and Analysis of Design Decisions: Medium γ values provide the best balance, whereas short horizons encourage greedy planning and long horizons suffer from compounding prediction errors.The PDDM controller with action smoothing and soft updates greatly outperforms the other planning algorithms evaluated.
- 4.3 Comparisons: PDDM consistently outperforms prior model-based and model-free methods in learning speed and final performance, including tasks that prior methods cannot solve.On valve turning, most methods eventually succeed, but policy-gradient methods require prohibitively large amounts of data.
- 4.3 Comparisons: With 8 goal angles, PDDM succeeds at in-hand reorientation while model-free approaches become stuck in local optima and fail to achieve previously attainable goals.The additional goals do not make the task harder for PDDM because it learns a model of interactions rather than specific target angles.
- 4.4 Learning Real-World Manipulation of Baoding Balls: After under 2 hours of real-world training, PDDM reaches about 100% success on 90° Baoding-ball rotations and about 54% on 180° rotations using the 24-DoF Shadow Hand.The experiment uses entirely real-world experience without simulation or prior system knowledge, with camera tracking and automated resets.
5 Discussion
PDDM learns challenging dexterous manipulation skills with multi-fingered hands, requiring less data and supporting flexible task reuse. It succeeds in simulation and on a real-world 24-DoF anthropomorphic hand.
- PDDM learns non-prehensile skills including free-floating object control, finger gaits, and writing user-specified strokes.
- PDDM requires substantially less training data than model-free reinforcement learning and produces flexibly reusable models.
- PDDM substantially outperforms state-of-the-art model-free methods on tasks demanding flexibility, including writing user-specified characters.
- 4 hours of entirely real-world interactions sufficed for successful in-hand manipulation on a 24-DoF anthropomorphic hand.
A Method Overview
PDDM uses online planning with deep dynamics models in an iterative loop: a controller selects actions using model predictions, then the model is retrained on collected data.
- PDDM iteratively alternates model-predictive action selection with training dynamics models on newly collected transitions.The overview includes an ensemble of models, rollouts, action selection, transition collection, and repeated model training.
- The controller uses predictions from the trained predictive dynamics model for action selection.
- An ensemble of randomly initialized models is trained from an initially empty dataset.
- During each rollout, PDDM resets the environment, selects actions with PDDMMPC, executes them, and records state transitions.
B Experiment Details
The experiments use a shared neural-network dynamics-model architecture trained with supervised regression and Adam, alongside task-specific reward and hyperparameter details.
- The dynamics model uses two fully connected hidden layers of 500 units with ReLU nonlinearities and a final fully connected output layer.
- All tasks train this architecture with mean squared error supervised learning and Adam at learning rate 0.001.
- Action spaces are normalized to [−1, 1] across tasks and environments.
- Reward functions and other relevant task details are listed in Table 2.
C Tracker Details
The real-world tracker estimates the Baoding balls’ 3D positions from stereo RGB images without explicit depth, then transforms them into ShadowHand coordinates.
- A camera tracker produces low-latency, robust, accurate 3D position estimates for the Baoding balls.
- A dilated CNN receives a 280x180 RGB stereo pair and outputs 2D center locations and depth for each sphere.
- Pinhole camera equations convert image locations and depth into 3D camera-frame points, followed by calibration into ShadowHand coordinates.
- Training occurs in simulation with real-world fine-tuning, using roughly 100 hand-labeled images to generate over 10,000 training images.