Source-linked AI summary
Continuous Deep Q-Learning with Model-based Acceleration
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, Sergey Levine
TL;DR
Deep model-free reinforcement learning can require too many samples for continuous-control problems on physical systems. The paper introduces NAF for continuous Q-learning and model-based imagination rollouts, finding faster learning with iteratively refitted local linear models where applicable, while noting scope limitations for complex initial-state distributions and inaccurate models.
Problem
High sample complexity, especially with high-dimensional function approximators, limits model-free deep reinforcement learning on physical systems.
Method
The paper combines NAF, a continuous Q-learning representation, with learned-model imagination rollouts to accelerate model-free reinforcement learning.
Results
The method tends to learn faster and acquire more accurate policies than recently proposed deep actor-critic algorithms, while local linear imagination rollouts substantially improve sample complexity.
Takeaways & Limitations
NAF provides a simpler continuous-control Q-learning approach, and learned local linear models can accelerate it without sacrificing the stated benefits of model-free reinforcement learning on applicable domains.
Takeaways & Limitations
Effective time-varying linear-model fitting requires relatively small initial-state distributions, while imagination rollouts degrade dramatically with imperfect learned models.
Abstract
from arXiv · showhide
Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of model-free algorithms, particularly when using high-dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalized adantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially effective for this, and demonstrate substantially faster learning on domains where such models are applicable.
1. Introduction
The paper addresses high sample complexity in deep model-free reinforcement learning for continuous control, especially with high-dimensional function approximators. It proposes continuous Q-learning and learned-model acceleration as complementary efficiency techniques.
- Motivation: High sample complexity limits model-free reinforcement learning on physical systems despite reduced manual engineering.The concern is especially pronounced for robots and autonomous vehicles, where experience must be collected from real systems.
- Contributions: The paper derives a continuous Q-learning variant and combines it with learned models to accelerate model-free learning.The combination aims to preserve the benefits of model-free reinforcement learning.
- Contributions: NAF enables Q-learning with experience replay for continuous tasks without a separate actor or policy function.Its simpler optimization objective and value-function parameterization improve sample efficiency with large neural networks.
- Model-based acceleration: Imagination rollouts under learned models can accelerate Q-learning, but their effectiveness depends strongly on model accuracy.Perfectly matching models are highly effective, whereas imperfect models cause dramatic degradation.
- Model-based acceleration: Iteratively fitted local linear models combined with local on-policy imagination rollouts produce a large improvement in sample complexity on applicable tasks.The method is evaluated on simulated robotic tasks against prior methods.
2. Related Work
Related work contrasts model-based methods' efficiency with model-free methods' generality and slower learning. Prior hybrid approaches include synthetic experience from learned models, but poor model approximations can undermine their efficiency.
- Deep reinforcement learning: Deep reinforcement learning has been applied to Atari games and diverse simulated and real-world robotic control tasks.These methods aim to automate representation design in reinforcement learning.
- Model-based and model-free learning: Model-based reinforcement learning methods have generally been more efficient, whereas model-free algorithms are more generally applicable but substantially slower.The paper situates its approach within efforts to combine these complementary properties.
- Model limitations: Expressive learned models require substantially more data, and Dyna-Q-like algorithms are vulnerable to poor model approximations.This constrains the efficiency of model-based augmentation when dynamics models are inaccurate.
3. Background
The background defines reinforcement learning as optimizing expected returns under unknown or learned dynamics. It reviews model-free value-based methods, the difficulty of continuous-action Q-learning, actor-critic alternatives, and model-based planning or synthetic experience.
- Reinforcement learning setup: Reinforcement learning optimizes expected cumulative rewards by choosing actions from states under environment dynamics and a reward function.The setup includes an initial-state distribution, transition dynamics, policy, finite horizon, and discount factor.
- Model-Free Reinforcement Learning: When dynamics are unknown, policy gradients and value-function or Q-function learning with function approximation are common model-free approaches.Off-policy value-based methods can in principle improve data efficiency, but continuous tasks often require two function approximators.
- Model-Free Reinforcement Learning: Q-learning minimizes Bellman error while learning a greedy deterministic policy from an action-value function and an exploration policy.The target is fixed during each objective evaluation.
- Continuous actions: Continuous-action Q-learning is difficult because each update requires maximizing a complex nonlinear function over actions.Actor-critic methods address this by learning a separate parameterized actor alongside the critic.
- Value and advantage functions: The value and advantage functions characterize expected returns and action-specific value relative to the policy's value function.These definitions provide the concepts used to describe the paper's Q-function representation.
- Model-Based Reinforcement Learning: Model-based reinforcement learning uses known or learned dynamics for planning methods such as iLQG and synthetic-experience methods such as Dyna-Q.Learned time-varying linear models can support iLQG, while model-generated rollouts supplement real experience.
4. Continuous Q-Learning with Normalized Advantage Functions
NAF makes continuous-action Q-learning tractable by parameterizing the Q-function so its maximizing action is obtained analytically. The implementation uses a value term, a quadratic advantage term, and deep Q-learning machinery.
- NAF representation: NAF is a continuous-action Q-learning method designed for deep neural networks.It represents the Q-function so the maximizing action can be computed analytically during each update.
- NAF representation: The neural network separately outputs a state-value term V(x) and an advantage term A(x,u) parameterized quadratically in nonlinear state features.This structured representation enables analytic maximization while retaining nonlinear value-function approximation.
- NAF representation: The positive-definite matrix P(x|θP) is factorized as L(x|θP)L(x|θP)^T, with a lower-triangular neural-network output and exponentiated diagonal terms.This parameterization supports the quadratic action dependence required by the method.
- Optimization: Because the Q-function is quadratic in the action, its maximizing action is given directly by µ(x|θµ).NAF uses this representation with target networks and replay buffers in a deep Q-learning algorithm analogous to prior work.
- Novelty and scope: The paper presents NAF as simpler than DDPG and as the first combination of this representation with deep neural networks for challenging continuous-control tasks.The authors note that alternative, nonquadratic advantage parameterizations remain possible.
5. Accelerating Learning with Imagination Rollouts
The paper accelerates continuous Q-learning by adding synthetic on-policy experience from learned dynamics models, while preserving model-free fine-tuning. Iteratively refitted time-varying linear models and optional iLQG exploration support this approach, with imagination rollouts switched off as the Q-function improves.
- Exploration: Off-policy iLQG trajectories were evaluated as replay-buffer additions, but narrow off-policy exploration often did not significantly improve Q-learning.The paper links this outcome to Q-learning’s need for noisy on-policy actions.
- Imagination Rollouts: Synthetic on-policy trajectories, called imagination rollouts, augment replay-buffer experience for Q-learning under learned models.The approach is described as a variant of Dyna-Q adapted to deep reinforcement learning.
- Safety and Scope: The method extends model-based imagination rollouts to continuous-control deep reinforcement learning and can use mostly off-policy iLQG rollouts in some safety-critical scenarios.This can reduce reliance on poorly trained policies taking potentially dangerous real-world actions.
- Algorithm: The algorithm can mix neural-network policy actions and optional iLQG actions, then store real and model-generated transitions in separate buffers for Q-network updates.Fictional transitions are generated by simulating multiple steps from sampled real transitions under the fitted model.
- Learned Dynamics Models: Iteratively refitted time-varying linear dynamics models produced better imagination-rollout results than nonlinear neural-network dynamics models in the authors’ experiments.The models are refitted from collected data and introduce assumptions beyond standard model-free reinforcement learning.
- Fine-tuning: Imagination rollouts are intended mainly for early learning and are switched off after a specified number of iterations so model-free experience can fine-tune the policy.The paper notes that on-policy behavior tends to outperform model-based controllers as the Q-function becomes more accurate.
6. Experiments
Experiments compare NAF with DDPG across simulated robotic tasks and test model-based acceleration through off-policy experience and imagination rollouts. NAF performs especially well on precision-sensitive manipulation, while short imagination rollouts with fitted time-varying linear models substantially improve data efficiency within a narrower state-distribution setting.
- Experimental setup: Experiments evaluate NAF and DDPG on simulated robotic control benchmarks, using matched neural-network architectures and replay-based Q-learning updates.The evaluation includes manipulation and locomotion domains, with hyperparameter searches over normalization, learning rate, and exploration noise.
- NAF versus DDPG: NAF learns smoother, more precise reacher behavior than DDPG, stabilizing at the target instead of continuously fluctuating around it.The qualitative difference is also reported for peg insertion and mobile-base gripper tasks.
- NAF versus DDPG: NAF and DDPG perform relatively similarly on locomotion, with NAF faster on swimmer6 and ant but DDPG faster on cheetah and better on walker2d.The paper links NAF’s weaker cases to mode-seeking behavior from its quadratic advantage function.
- NAF versus DDPG: NAF outperforms DDPG on the majority of tasks, particularly precision-demanding manipulation domains that suffer less from the lack of multimodal Q-functions.The authors describe this pattern as promising for efficient learning in real-world robotic tasks.
- Model-based acceleration: Mixing good off-policy iLQG experience does not significantly improve data efficiency, whereas imagination rollouts improve data efficiency or final performance.The comparison uses ground-truth dynamics to isolate the effect of model-based experience.
- Model-based acceleration: Time-varying linear models outperform fitted neural-network dynamics for imagination rollouts on the evaluated tasks, while learned neural-network models negate the rollout benefits.The paper attributes this trade-off to the difficulty of learning expressive dynamics models from few samples.
- Model-based acceleration: Imagination rollouts of length 5 to 10 yield significant improvement over fully model-free NAF and provide 2-5 factors of improvement in data efficiency.On gripper, rollouts are switched off after 130 episodes (20,000 steps), after which performance quickly improves again and Q-learning avoids observed divergence.
- Model-based acceleration: Time-varying linear models with imagination rollouts improve sample efficiency but require relatively small initial state distributions, limiting generality.The authors suggest clustering trajectories and fitting multiple models for more complex initial distributions.
7. Discussion
The paper targets sample inefficiency in model-free deep reinforcement learning for continuous control. It introduces NAF for continuous Q-learning and combines model-free learning with learned-model imagination rollouts, finding faster learning when suitable local dynamics models are available.
- High sample complexity, especially with high-dimensional function approximators, limits model-free reinforcement learning’s applicability to physical systems.
- NAF applies standard Q-learning to high-dimensional continuous domains, simplifying actor-critic methods while retaining nonlinear value approximation and adaptive exploration.
- NAF tends to learn faster and acquire more accurate policies than recently proposed deep actor-critic algorithms, while off-policy model-based exploration only rarely improves sample efficiency.
- Locally modeled imagination rollouts can accelerate model-free continuous Q-learning, whereas learning primarily from off-policy model-based exploration is minimally effective on the evaluated tasks.
8. Appendix
The appendix describes locally linear model-based methods, adaptive exploration, and comparative results for NAF and DDPG across continuous-control domains. It reports where NAF excels, where DDPG performs better, and how imagination rollouts and MPC experience affect learning.
- Model-based control: iLQG constructs locally optimal linear feedback controllers using local linearizations of dynamics and quadratic expansions of rewards.With learned dynamics, trajectories fit time-varying linear models, which iLQG uses to obtain updated controllers under a trust-region constraint.
- Exploration: NAF-based exploration uses the quadratic advantage matrix as Gaussian precision, making actions more deterministic in steep directions and more random in flat directions.Independent Gaussian exploration can be poorly scaled across action dimensions and may miss correlations required by tasks such as swimming.
- Exploration: Using learned precision for exploration produced a better policy on canada2d but no significant difference on the other domains.Figure 3 reports that NAF-P slightly outperforms the best DDPG result, with the precision term introduced only after step 50,000.
- NAF versus DDPG: NAF generally outperforms DDPG, especially on precision tasks, but DDPG converges faster on cheetah and can learn more natural cheetah movements.The appendix also reports that NAF significantly outperforms DDPG on moving gripper, whereas DDPG converges faster than NAF on cheetah.
- Model-based acceleration: Imagination rollouts provide the most consistent acceleration across reacher, cheetah, and canada2d, whereas MPC experience helps mainly on hard manipulation tasks.The reported rollout horizons are l = 10 for reacher and canada2d and l = 5 for cheetah; the dynamics model therefore needs accuracy mainly around data trajectories.
- Experimental domains: Table 3 lists the experimental domains and notes that all domains except ant are two-dimensional.The listed domain descriptions include a fixed-target ant task and a planar bipedal walker constrained not to fall or pitch excessively.