Source-linked AI summary
Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, Sergey Levine
TL;DR
Real-world robotic reinforcement learning must combine data efficiency with robustness to complex, unknown dynamics. The paper integrates LQR-based model updates with PI2 residual corrections for TVLG policies, extends the approach through GPS to neural networks, and reports strong performance on simulated and real tasks.
Problem
Real-world robotic RL must learn efficiently while handling complex, unknown dynamics that can be difficult to model.
Method
PILQR combines LQR-FLM model-based updates with PI2 model-free residual updates for time-varying linear-Gaussian policies.
Results
The method combines model-based learning efficiency with model-free success on discontinuous dynamics and achieves substantially better sample efficiency than direct model-free deep RL when combined with GPS.
Takeaways & Limitations
PILQR can train arbitrary parameterized policies, including deep neural networks, through GPS, while real-robot tasks were learned with less than an hour of experience and no demonstrations.
Takeaways & Limitations
Trajectory-centric methods require consistent environment resets, and the practical update includes a heuristic modification that no longer aligns with Theorem 1.
Abstract
from arXiv · showhide
Reinforcement learning (RL) algorithms for real-world robotic applications need a data-efficient learning process and the ability to handle complex, unknown dynamical systems. These requirements are handled well by model-based and model-free RL approaches, respectively. In this work, we aim to combine the advantages of these two types of methods in a principled manner. By focusing on time-varying linear-Gaussian policies, we enable a model-based algorithm based on the linear quadratic regulator (LQR) that can be integrated into the model-free framework of path integral policy improvement (PI2). We can further combine our method with guided policy search (GPS) to train arbitrary parameterized policies such as deep neural networks. Our simulation and real-world experiments demonstrate that this method can solve challenging manipulation tasks with comparable or better performance than model-free methods while maintaining the sample efficiency of model-based methods. A video presenting our results is available at https://sites.google.com/site/icml17pilqr
1. Introduction
The paper targets reinforcement learning for robotics that must be both data-efficient and robust to complex, unknown dynamics. It combines model-based and model-free updates for time-varying linear-Gaussian policies and extends the approach through guided policy search.
- Real-world robotic RL must minimize costly trials while handling complex dynamics that are difficult to model accurately.
- Model-based methods are sample-efficient but biased by inaccurate dynamics models, whereas model-free methods handle arbitrary dynamics with lower sample efficiency.
- PILQR combines iterative linear-Gaussian model fitting with PI2 updates to optimize time-varying linear-Gaussian policies.
- Guided policy search enables the method to train general-purpose parameterized policies, including deep neural networks.
- The method learns complex real-robot hockey and power-plug tasks from scratch with less than an hour of experience and no demonstrations.
2. Related Work
Prior trajectory-centric methods trade representational power, sample efficiency, and model assumptions in different ways. This paper combines trajectory-centric policies with guided policy search to extend their use to complex neural-network policies.
- Trajectory-centric representations such as splines, dynamic movement primitives, and TVLG controllers support episodic motions and efficient optimization in robotics.
- Deep neural networks process complex sensory inputs and represent strategies that can succeed from varied initial conditions, but trajectory-centric representations are more limited.
- Guided policy search can combine trajectory-centric reinforcement learning with training complex neural-network policies.
- Model-based trajectory-centric methods are fast and sample-efficient but assume properties such as locally linearizable dynamics or continuous costs.
- Earlier hybrid approaches include using LQR for PI2 initialization or models to generate synthetic samples, whereas this method directly combines model-based and model-free updates.
3. Preliminaries
The preliminaries define policy search over trajectory costs and introduce TVLG controllers, LQR-FLM, and PI2 as the compatible model-based and model-free components. Their shared KL-constrained structure supports combining the updates.
- Policy search optimizes policy parameters for a distribution over actions conditioned on system states, evaluated through expected trajectory cost.
- A TVLG controller uses Gaussian actions with mean K_t x_t + k_t and time-varying covariance Σ_t.
- LQR-FLM fits TVLG dynamics from samples, approximates costs quadratically, and uses KL constraints to keep updated policies near regions where the model is reliable.
- Separate KL constraints are enforced for each linear-Gaussian policy rather than one constraint on the induced trajectory distribution.
- LQR-FLM is efficient but depends on accurate dynamics models and can fail on challenging tasks, while PI2 can achieve better final performance at greater sample complexity.
- PI2 reweights sampled controls toward lower-cost trajectories, with temperature parameters controlling the KL step size.
4. Integrating Model-Based Updates into PI2
The hybrid update decomposes PI2 into a model-based update on approximated costs followed by a model-free correction using residual costs. LQR-FLM supplies the efficient first update, while PI2 addresses modeling errors.
- PI2 separates each policy update into an approximated-cost update and a residual-cost update.
- The first update uses model-based optimization of quadratic cost-to-go under fitted linear-Gaussian dynamics, while the second uses PI2 on arbitrary residual costs.
- LQR-FLM improves efficiency but remains susceptible to inaccurate fitted dynamics, especially in discontinuous systems.
- The algorithm evaluates approximated cost-to-go by simulating the previous TVLG controller under fitted dynamics and its time-varying quadratic cost.
- Stored noise realizations are transformed to obtain unbiased control samples from the updated LQR-FLM policy before residual-cost reweighting.
- In practice, the method modifies the constraint expectation to use the updated distribution; the authors describe this heuristic as working better despite no longer matching Theorem 1.
5. Training Parametric Policies with GPS
PILQR is integrated with MDGPS to train parametric global policies through optimized TVLG local policies. The procedure alternates model fitting and policy updates before using local-policy controls as supervised training data.
- 5. Training Parametric Policies with GPS: MDGPS uses PILQR to optimize simple TVLG local policies for different task initial conditions before training a global parametric policy.The global policy is trained from optimized local-policy controls rather than directly optimized with reinforcement learning.
- 5. Training Parametric Policies with GPS: Optimized controls from local policies form supervised training data for the global policy, allowing it to generalize across multiple local policies.The global policy is denoted πθ and is trained in a supervised manner.
- 5. Training Parametric Policies with GPS: Each PILQR iteration generates trajectories, fits TVLG dynamics, estimates cost residuals, and adjusts the LQR-FLM KL step.The step size is adjusted using residual and sampled costs-to-go before computing the LQR-FLM update.
- 5. Training Parametric Policies with GPS: The algorithm combines an LQR-FLM update with a PI2 update to compute the new time-varying linear-Gaussian policy.The PI2 update uses cost residuals and LQR-FLM actions.
- 5. Training Parametric Policies with GPS: In MDGPS, each local policy is constrained against the old global policy rather than the old local TVLG policy.This constraint change enables PILQR to operate within the MDGPS framework.
6. Experimental Evaluation
Experiments compare PILQR and its MDGPS extension with model-based, model-free, and deep RL baselines across simulated manipulation tasks and real-robot skills. PILQR achieves challenging manipulation behaviors while retaining strong sample efficiency, including successful door opening, hockey, and plug insertion.
- Simulation Experiments: PILQR solves all four gripper-pusher conditions with 400 total episodes per condition, including diverse successful block-pushing behaviors.LQR-FLM and PI2 perform significantly worse on the two more difficult conditions.
- Simulation Experiments: MDGPS with LQR-FLM and PILQR performs competitively with TRPO and DDPG on the reacher task, while MDGPS with PI2 makes little progress.TRPO and DDPG require orders of magnitude more samples according to the log-scaled sample axis.
- Simulation Experiments: MDGPS with PILQR learns a successful door-opening neural network policy using 20 times fewer samples than TRPO; the other methods fail despite extensive tuning.The door task involves grasping a handle and pulling the door through complex contact dynamics.
- Real Robot Experiments: On the real robot, PILQR learns a robust hockey policy that consistently hits the puck into the goal, whereas PI2 and LQR-FLM fail to do so reliably.The method shifts toward PI2 updates as the TVLG dynamics approximation becomes inaccurate.
- Real Robot Experiments: The MDGPS-PILQR neural network hits hockey goals in 90% of rollouts across three goal positions, including one unseen during training.Two goal positions were used for training and one was held out for evaluation.
- Real Robot Experiments: PILQR plugs the power plug on every rollout at convergence, compared with 60% success for LQR-FLM and no successful socket reach for PI2.The task requires fine manipulation because of the plug’s small size.
7. Discussion and Future Work
The paper combines model-based and model-free reinforcement learning updates in PILQR, then extends the approach through GPS to train parameterized policies. Experiments show strong performance on challenging tasks, including real-robot power-plug insertion, while retaining efficiency and broader robustness.
- Discussion and Future Work: PILQR combines model-based LQR-FLM updates with model-free PI2 updates to handle model error while retaining efficient trajectory optimization.The PI2 update operates on residuals between sampled costs and costs estimated by local linear models.
- Discussion and Future Work: On the real-robot power-plug task, PILQR consistently inserted the plug fully by the final iteration, unlike PI2 and LQR-FLM.PI2 failed to reach the socket, while LQR-FLM succeeded only 60% of the time at convergence.
- Discussion and Future Work: PILQR can be integrated with GPS to train arbitrary parameterized policies, including deep neural networks.The trajectory-centric method is used to optimize local policies before training a global policy.
- Discussion and Future Work: The method combines model-based efficiency with model-free success on tasks involving discontinuous dynamics and costs, while achieving substantially better sample efficiency than model-free deep RL when paired with GPS.The authors also note that trajectory-centric methods require consistent environment resets and currently assume continuous action spaces.
8. Appendix
The appendix details the LQR-FLM and PI2 updates, experimental setup, task conditions, and additional simulation results. These details show how the hybrid method is optimized, evaluated across conditions, and compared with its component algorithms.
- 8.1. Derivation of LQR-FLM: LQR-FLM computes actions that minimize a second-order Q-function approximation, yielding linear and maximum-entropy TVLG policies.The policy has Gaussian form p(u_t|x_t) = N(K_tx_t + k_t, Q_uu,t^-1).
- 8.1. Derivation of LQR-FLM: LQR-FLM uses per-time-step KL-divergence constraints to stabilize updates between the old and new policies.The KL constraint limits policy changes while fitting local dynamics and optimizing the approximate cost.
- 8.2. PI2 update through constrained optimization: PI2 updates policy probabilities from sampled trajectory cost-to-go values, assigning greater probability to lower-cost paths before normalization.The update is derived from a sample-based Lagrangian and aggregates paths beginning from the same state-action pair.
- 8.3.1. SIMULATION EXPERIMENTS: The experiments train TVLG policies separately for task conditions, then evaluate neural-network policies across randomized initial conditions.Reacher uses 16 training conditions and 300 randomized test episodes; door opening uses four training corners and 100 randomized test episodes.
- 8.4. Additional Simulation Results: PILQR outperforms other baselines on most additional gripper-pusher conditions, while LQR-FLM matches it on simpler cases.PI2 makes no progress with the same samples but becomes comparable to LQR-FLM with 10 times more samples.
- 8.4. Additional Simulation Results: PI2 improves substantially on reacher with 10 times more samples but still trails the other methods, and it remains unsuccessful on door opening.The authors note that PI2 performance may continue increasing with still more samples.