Source-linked AI summary
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto
TL;DR
Visual continuous control from high-dimensional images remains difficult for sample-efficient model-free reinforcement learning. DrQ-v2 addresses this gap with a data-augmented actor-critic algorithm and achieves strong task performance, including humanoid locomotion, with lower computational demands.
Problem
Sample-efficient continuous control from high-dimensional images remains a longstanding reinforcement-learning challenge, with model-free methods limited on difficult locomotion tasks and costly to train.
Method
DrQ-v2 builds on DrQ by combining data augmentation with a model-free actor-critic algorithm and several algorithmic and implementation improvements.
Results
DrQ-v2 outperforms prior model-free methods in sample efficiency across DMC difficulty levels and solves humanoid locomotion directly from pixels.
Takeaways & Limitations
DrQ-v2 provides a conceptually simple, computationally efficient baseline for visual continuous-control research.
Takeaways & Limitations
Dreamer-v2 was evaluated on only 12 of the 24 considered tasks because of its compute requirements.
Abstract
from arXiv · showhide
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2's implementation to provide RL practitioners with a strong and computationally efficient baseline.
1 Introduction
DrQ-v2 addresses persistent limitations in pixel-based continuous control by combining data augmentation with a simpler, more computationally efficient model-free algorithm. It improves performance on difficult tasks while reducing training requirements and clarifying the effects of design choices.
- Visual continuous control remains challenging because high-dimensional images make sample-efficient learning difficult.
- Current model-free methods struggle with quadruped and humanoid locomotion, costly distributed training, and unclear design-choice effects.
- DrQ-v2 combines model-free actor-critic learning with data augmentation and solves complex humanoid tasks directly from pixels.
- DrQ-v2 matches DreamerV2 in sample efficiency and performance while training 4× faster in wall-clock time.
- The implementation trains around 3.5 times faster than the previous implementation, increasing environment throughput from 28 to 96 FPS.
2 Background
The paper frames visual control as reinforcement learning from image observations, represented as an infinite-horizon MDP. Its background connects this formulation to DDPG and data augmentation for learning from pixels.
- Image-based control is formulated as an infinite-horizon MDP with stacked image observations, actions, transitions, rewards, discounting, and initial-state distribution.
- DDPG jointly learns a Q-function and deterministic policy for continuous control using replay-buffer transitions and Bellman-based critic learning.
- DDPG can incorporate n-step returns, which enable faster reward propagation during temporal-difference estimation.
- DrQ extends model-free actor-critic learning with a convolutional encoder and random-shift data augmentation for visual reinforcement learning.
3 DrQ-v2: Improved Data-Augmented Reinforcement Learning
DrQ-v2 combines random-shift augmentation, convolutional image encoding, DDPG with n-step returns, scheduled exploration, and implementation changes for efficient pixel-based control.
- Image Augmentation and Encoder: DrQ-v2 applies random shifts to image observations before encoding them into low-dimensional latent vectors with a convolutional encoder.The encoder computes h = fξ(aug(x)).
- Image Augmentation: Bilinear interpolation is added after random shifts, replacing each pixel with the average of four neighboring pixel values.
- Actor-Critic Algorithm: The algorithm uses DDPG with n-step returns and two Q-functions to estimate critic losses from replay-buffer mini-batches.
- Actor-Critic Algorithm: Critic training applies augmented current and future observations, target actions with clipped exploration noise, and slowly updated target Q-networks.
- Actor-Critic Algorithm: The actor is trained with deterministic policy gradients while actor gradients are not used to update the encoder.
- Scheduled Exploration Noise: Exploration noise decays linearly so training begins with more stochastic exploration and later emphasizes mastering promising behaviors.The schedule uses initial and final standard deviations over a decay horizon.
- Key Hyper-Parameter Changes: DrQ-v2 uses a 10 times larger replay buffer and a mini-batch size of 256 among its key hyper-parameter changes.
- Implementation Efficiency: Custom image augmentation and replay-buffer implementations address CPU–GPU transfer and memory bottlenecks, increasing storage capacity and data-transfer speed.
4 Experiments
DrQ-v2 is evaluated on visual continuous-control tasks across difficulty levels, comparing sample efficiency, wall-clock efficiency, and model-free and model-based baselines. The experiments also include ablations of design choices that guided the final algorithm.
- Experimental setup: The evaluation covers DMC tasks learned from pixels, with comparisons to model-free and model-based methods using sample efficiency and wall-clock time.The study includes hard, medium, and easy task subsets, alongside an ablation study.
- Comparison to model-free methods: DrQ-v2 outperforms prior model-free methods in sample efficiency across easy, medium, and hard benchmarks, with larger advantages on harder exploration tasks.It also solves DMC humanoid locomotion tasks directly from pixels, described as the first successful model-free demonstration by the authors.
- Compute efficiency: 96 FPS enables DrQ-v2 to solve easy, medium, and hard tasks within 2.9, 8.6, and 86 hours respectively.The throughput is reported on a single NVIDIA V100 GPU and is compared against prior model-free methods.
- Comparison to model-based methods: DrQ-v2 can rival Dreamer-v2 in sample efficiency in many cases, although Dreamer-v2 performs better on several tasks.The authors specifically cite Acrobot Swingup and Finger Turn Hard as examples and leave the discrepancy for future work.
- Ablation study: Ablations examine influential algorithmic choices, including switching from SAC to DDPG and increasing replay-buffer size.A 1M replay buffer considerably improves performance on Reacher Hard, while the ablations cover three tasks of varying difficulty.
5 Related Work
Visual RL methods improve image-based control through learned representations and augmented data, while humanoid control remains especially difficult. The paper situates DrQ-v2 among these approaches and uses ablation to identify its design improvements.
- Auto-encoders, variational inference, contrastive learning, self-prediction, and augmented data have advanced visual reinforcement learning.
- Dreamer-v2 performs more computations than DrQ-v2, giving DrQ-v2 an advantage in wall-clock training time.
- Augmented-data methods such as DrQ and RAD helped bridge the gap between state-based and image-based reinforcement learning.
- Humanoid control is difficult because of its large state and action spaces, with prior model-free solutions using proprioceptive observations.
- The DrQ-v2 ablation study incrementally evaluates four key improvements over original DrQ, including changes to the base algorithm and multi-step returns.
6 Conclusion
The paper concludes that DrQ-v2 is a simple model-free actor-critic algorithm for image-based continuous control that masters difficult DMC tasks directly from pixels. It also provides a publicly available implementation intended to support further visual RL research.
- DrQ-v2 is a conceptually simple model-free actor-critic algorithm for image-based continuous control.
- DrQ-v2 masters DMC tasks directly from pixels, including humanoid locomotion tasks previously unsolved by model-free approaches.
- The authors provide an efficient, publicly available PyTorch implementation of DrQ-v2.
A Benchmarks
The benchmark appendix organizes 24 continuous control tasks from the DeepMind Control Suite into easy, medium, and hard categories. Table 1 provides detailed descriptions of these tasks.
- 24 continuous control tasks from the DeepMind Control Suite are classified into easy, medium, and hard benchmarks.
- The benchmark classification provides a summary for each task.
- Table 1 gives a detailed description of the tasks in the easy, medium, and hard benchmarks.
B Hyper-parameters
The hyper-parameter appendix specifies a default experimental configuration while documenting task-specific deviations. These deviations include settings for batch size, returns, replay-buffer size, learning rate, and feature dimension.
- Task-specific deviations: Walker Stand, Walk, and Run use mini-batch size 512 and a 1-step return.
- Task-specific deviations: Quadruper Run uses a replay buffer size of 10^5.
- Task-specific deviations: Humanoid Stand and Walk use learning rate 8 × 10^-5 and increased feature dimension to 100.
- Table 2 presents the default set of hyper-parameters used in the experiments.