Source-linked AI summary
One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors
Justin Fu, Sergey Levine, Pieter Abbeel
TL;DR
The paper addresses the need for large amounts of experience in robotic reinforcement learning when accurate dynamics are difficult to estimate. It combines prior dynamics knowledge with online local adaptation and MPC, enabling a range of complex manipulation tasks on a single attempt. The method requires a full, Markovian system state for iterative-LQR-based MPC.
Problem
Robotic reinforcement learning often requires substantial experience, while model-based methods depend on dynamics models accurate enough for effective policy learning.
Method
The method uses a prior dynamics model from previous tasks, progressively refits a local linear model online, and plans actions with model predictive control.
Results
The approach achieves a range of challenging manipulation tasks on the first attempt, with a short-temporal-context neural-network prior performing best among the evaluated prior models.
Takeaways & Limitations
Online adaptation lets coarse or inaccurate prior models support one-shot learning across manipulation tasks with nonlinear and varying dynamics.
Takeaways & Limitations
The method requires a full, Markovian state of the system to perform MPC using iterative LQR.
Abstract
from arXiv · showhide
One of the key challenges in applying reinforcement learning to complex robotic control tasks is the need to gather large amounts of experience in order to find an effective policy for the task at hand. Model-based reinforcement learning can achieve good sample efficiency, but requires the ability to learn a model of the dynamics that is good enough to learn an effective policy. In this work, we develop a model-based reinforcement learning algorithm that combines prior knowledge from previous tasks with online adaptation of the dynamics model. These two ingredients enable highly sample-efficient learning even in regimes where estimating the true dynamics is very difficult, since the online model adaptation allows the method to locally compensate for unmodeled variation in the dynamics. We encode the prior experience into a neural network dynamics model, adapt it online by progressively refitting a local linear model of the dynamics, and use model predictive control to plan under these dynamics. Our experimental results show that this approach can be used to solve a variety of complex robotic manipulation tasks in just a single attempt, using prior data from other manipulation behaviors.
I. INTRODUCTION
The paper targets one-shot robotic skill learning by combining prior dynamics knowledge with online adaptation. A coarse global model supplies a prior, while local refitting and predictive control compensate for task-specific dynamics.
- Model-based RL is sample-efficient but typically needs enough experience to learn an accurate dynamics model.
- The method adapts a coarse dynamics model online so it only needs to represent local dynamics accurately.The local model can be linear, while global-model accuracy affects proficiency and adaptation requirements.
- Prior data from different manipulation tasks can initialize learning because online adaptation compensates for task-specific mismatch.
- The approach combines prior dynamics models with MPC based on DDP to choose actions for low-level torque-controlled manipulation.
III. BACKGROUND
The background formulates control as minimizing finite-horizon cost under estimated dynamics. Iterative LQR repeatedly linearizes those dynamics and updates the policy in an MPC loop.
- The control objective is to choose actions that minimize cumulative cost over a finite horizon under dynamics xt+1 = f(xt, ut).
- When true dynamics are unknown, the method estimates ˆf from prior interaction data and optimizes actions under that model.
- Iterative LQR linearizes dynamics around a nominal trajectory, forms a quadratic cost approximation, and repeatedly improves the action sequence.
- The resulting local control law is applied through forward rollouts and repeated backward dynamic-programming passes.
- For MPC, iterative LQR recomputes the policy in real time from the current state, making it suitable for short-horizon control.
- Iterative LQR requires estimates of the linearized dynamics, which are difficult when object dynamics are unknown in robotic manipulation.
IV. MODEL-BASED REINFORCEMENT LEARNING WITH ONLINE DYNAMICS ADAPTATION
The proposed algorithm repeatedly fits a local linear dynamics model using recent observations and a prior trained from previous robot experience. The adapted model then supports MPC action selection.
- The method repeatedly updates a time-varying linear dynamics model from recent states, actions, and a learned dynamics prior.
- Standard linear regression estimates the local dynamics parameters from observed state-action-next-state samples before prior information is incorporated.
- Linear dynamics fitting is reformulated as Gaussian modeling so prior information can be incorporated through a normal inverse-Wishart prior.
- Conditioning the fitted joint Gaussian on the current state and action produces linear-Gaussian dynamics for the next state.
- Prior parameters can be generated from global dynamics models including Gaussian mixture models and neural networks.
B. Online Estimation of Locally Linear Dynamics
The method updates local dynamics estimates online from recent state transitions, combining empirical statistics with prior information and adapting how quickly old data is forgotten. It then uses prediction accuracy to adjust the balance between prior and empirical estimates.
- Online estimates update the empirical mean and covariance from recently observed state transitions during execution.The algorithm initializes these statistics from prior-training data and updates them at each time step.
- β controls how quickly past experiences are forgotten, which is useful when nonlinear dynamics change across local regions.Larger forgetting rates emphasize recent experience, while difficult situations require incorporating more data.
- At each step, the posterior dynamics estimate combines the online empirical statistics with prior parameters before computing control actions.Algorithm 1 evaluates the prior, updates β and N, computes the posterior, and then runs LQR and samples an action.
- The method compares prior-based and empirical state predictions, then adapts β and N according to their relative prediction errors.N controls the relative strength of the prior and is inversely proportional to 1 −β.
C. Algorithm Summary
At every time step, the algorithm updates its empirical and prior-based dynamics estimates, combines them into a posterior model, and uses that model for control. It also adds targeted exploration noise when uncertainty or contact dynamics make repeated optimal actions insufficient.
- The algorithm updates empirical statistics from the latest transition and combines them with a prior using prediction accuracy.The resulting posterior mean and covariance provide the current dynamics estimate.
- The estimated dynamics support a local second-order control procedure for selecting actions during execution.
- For difficult contact situations such as a jammed peg insertion, the method adds Gaussian exploration noise to probe the dynamics.The noise covariance is chosen proportional to Q^-1 from the LQR Q-function.
V. NEURAL NETWORK DYNAMICS PRIORS
The paper considers several dynamics priors and compares them experimentally, including a neural-network-based prior described in detail elsewhere in the method.
- The section frames neural-network priors as one of several prior choices evaluated experimentally.
A. Gaussian and Gaussian Mixture Priors
The paper constructs dynamics priors from Gaussian or Gaussian-mixture models and also converts neural-network predictions into local probabilistic priors. These choices trade representational simplicity against the ability to capture nonlinear or complex dynamics.
- Gaussian and Gaussian Mixture Priors: A Gaussian prior fits a distribution to state-action-next-state vectors and uses its mean and covariance as prior parameters.In practice, setting n0 and m to 1 can outperform setting them to the number of prior data points.
- Gaussian and Gaussian Mixture Priors: A Gaussian mixture prior models nonlinear dynamics through locally distinct mixture components associated with hidden contact profiles.The latest transition is assigned to a mixture component whose mean and covariance define the local prior.
- Gaussian and Gaussian Mixture Priors: Gaussian and mixture priors have limited capacity for globally complex dynamics, motivating neural networks as expressive dynamics representations.The Gaussian is globally linear, while the mixture captures only a small number of locally linear modes.
- Neural Network Priors: A neural-network dynamics prediction is linearized at the current state and action to construct a local prior mean and covariance.The prior state-action covariance is set to αI, while conditional covariance is estimated from prediction and target-state variability.
C. Neural Network Architectures
The dynamics prior uses neural networks to predict accelerations and next states, with temporal context improving contact-dynamics modeling. The prior is trained on interaction data from other tasks and combined with online adaptation to generalize across manipulation behaviors.
- Network architectures: The dynamics networks use two hidden ReLU layers and predict accelerations that are integrated to obtain the next state.The layers contain 60 and 40 hidden units, respectively.
- Network architectures: Adding the previous state and action provides temporal context that improves modeling of contact dynamics.The temporal context helps distinguish contact by comparing prior joint torques with the acceleration actually experienced.
- Prior training: The neural-network prior can be trained on interaction data from different tasks rather than the task being learned.The resulting prior is used to support one-shot learning on new manipulation tasks without task-specific prior data.
- Prior training: Training data combines trials from physical and simulated manipulation tasks with random free-space arm motion.The dataset includes workbench, gears, airplane, car, ring, peg-insertion, and stacking tasks.
- Prior training: Although the prior requires considerable data, combining it with online adaptation allows effective generalization to other tasks.The authors characterize the resulting prior as broadly useful across manipulation behaviors.
VI. EXPERIMENTAL EVALUATION
The method was evaluated on physical and simulated PR2 manipulation tasks, with simulation enabling larger comparisons and physical experiments testing a real system. The evaluation covered insertion, stacking, assembly, and object-placement behaviors under a hold-one-out prior-training scheme.
- Evaluation setup: The evaluation used a physical PR2 mobile manipulator and a simulated PR2 arm in MuJoCo.Simulation supported large-scale method comparisons, while physical tasks demonstrated operation on a real system.
- Physical tasks: Physical experiments covered nail insertion, ring placement, gear assembly, toy-car and airplane assembly, and block stacking.The nail task also included a high-friction variant intended to increase difficulty.
- Simulated tasks: Simulated experiments included peg insertion with different peg shapes and stacking tasks.The simulated task set included cylindrical and square-shaped stacking or insertion variants.
- Control setup: The controller used a 32-dimensional state, operated at 20 Hz, and planned with a 15-timestep MPC horizon.The horizon corresponds to 0.75 seconds.
- Evaluation protocol: The neural-network prior for each tested task was trained on data from all other tasks, implementing hold-one-out cross-validation.No data from the task being tested was included in its prior.
B. Comparisons
The evaluation compares adaptive dynamics models and neural-network priors against non-adaptive and Gaussian-process baselines across simulated and physical manipulation tasks. Results show that online adaptation improves performance, while temporal context and adaptation support first-try success on diverse tasks and robustness testing examines observation errors.
- Online adaptation outperformed non-adaptive variants, demonstrating its importance for models trained on other tasks.
- The neural network prior achieved the best overall performance, particularly when using preceding state and action as context.
- The adaptive method succeeded on a wide range of challenging manipulation tasks on the first try without task-specific data for training the dynamics prior.
- The evaluation covered simulated and physical manipulation tasks, including peg insertion, nail insertion, ring placement, stacking, and assembly.
- Robustness experiments introduced fixed target-position errors and found adaptation particularly helpful on harder tasks.
D. Qualitative Results and Conclusions
The method combines a coarse prior dynamics model with online adaptation and MPC to solve challenging manipulation tasks on the first attempt. Its robustness comes from refining prior knowledge through probing, while the paper identifies task selection and state observability as important boundaries.
- Results: The method achieves a range of challenging manipulation tasks on the first attempt, including under inaccurate prior models and unexpected environmental variation.The authors attribute this robustness to combining prior experience with online model adaptation.
- Results: A neural network prior with short temporal context achieves the best results, while every adaptive variant outperforms its corresponding prior model alone.The experiments compare multiple prior models and their adaptive counterparts.
- Qualitative behavior: The approach spends up to 20 seconds exploring unfamiliar dynamical modes after unexpected contact while progressively improving its model.This probing behavior is presented as central to the approach’s robustness and effectiveness.
- Method: The algorithm combines a coarse prior dynamics model with online adaptation and MPC to perform complex manipulation in highly nonlinear systems.The prior need only provide an initial guess; online adaptation refines a local linear approximation while the controller replans.
- Limitation: Aggregating all other tasks to train the dynamics model may cause negative transfer, motivating clustering or grouping of related prior tasks.The evaluation uses all other tasks, but the authors propose restricting training data to related tasks.
- Limitation: The method requires a full Markovian state for iterative-LQR MPC, which can limit application to robotic manipulation from raw sensory input.Learning a latent state representation is suggested as one possible direction for addressing this boundary.