PaperlayerView exact revision on arXiv

Source-linked AI summary

The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels

Austin Stone, Oscar Ramirez, Kurt Konolige, Rico Jonschkowski

arXiv:2101.02722v1cs.ROcs.AIcs.CVcs.LG

TL;DR

DM Control lacks visual distractions, leaving pixel-based RL benchmarks without the perceptual variation common in real environments. The paper extends it with scalable camera, color, and background distractions and evaluates current RL methods. Performance declines as distractions intensify, and combinations are more damaging than their individual effects.

  • Problem

    DM Control’s constant camera, lighting, and background do not test whether agents can filter task-irrelevant visual variation.

    Source: §I · p. 1

  • Method

    The paper extends DM Control with scalable camera-pose, color, and background distractions and evaluates SAC and QT-Opt variants with image-cropping methods.

    Source: §I · p. 1, §III · p. 2, §IV · p. 3

  • Results

    Current methods lose performance as distraction intensity increases, and combined distractions degrade performance more than individual distractions, especially in the dynamic setting.

    Source: A. Robustness to Distractions During… · p. 4, §V · p. 4, C. A New Benchmark for Control from… · p. 6

  • Takeaways & Limitations

    The benchmark’s results show that methods performing similarly on DM Control can differ under distractions, making simpler-environment results an imperfect guide to more realistic settings.

    Source: §VI · p. 6

Abstract

from arXiv · show

Robots have to face challenging perceptual settings, including changes in viewpoint, lighting, and background. Current simulated reinforcement learning (RL) benchmarks such as DM Control provide visual input without such complexity, which limits the transfer of well-performing methods to the real world. In this paper, we extend DM Control with three kinds of visual distractions (variations in background, color, and camera pose) to produce a new challenging benchmark for vision-based control, and we analyze state of the art RL algorithms in these settings. Our experiments show that current RL methods for vision-based control perform poorly under distractions, and that their performance decreases with increasing distraction complexity, showing that new methods are needed to cope with the visual complexities of the real world. We also find that combinations of multiple distraction types are more difficult than a mere combination of their individual effects.

I. INTRODUCTION

DM Control supports advances in pixel-based control but lacks the visual distractions that make real-world perception challenging. The Distracting Control Suite adds scalable visual variations while preserving task physics and evaluates how current pixel-based RL methods handle them.

I. INTRODUCTION

  • DM Control ties observation changes to task-relevant state and therefore cannot measure agents’ ability to filter irrelevant visual variations.Its constant camera, lighting, and static background contrast with visual conditions encountered in real robot learning.Source: §I · p. 1
  • The suite adds random color, video-background, and camera-pose distractions that can vary statically or smoothly between frames.Each distraction’s difficulty can be scaled, and the types can be combined.Source: §I · p. 1
  • The suite modifies scene properties at render time while keeping the tasks’ underlying physics and control properties exactly the same as DM Control.This preserves comparability with work on the original benchmark.Source: §I · p. 1
  • The authors analyze state-of-the-art pixel-based RL methods across inference-time distractions, individual distraction types, and combinations of all three.Their reported observations include relative robustness to color, difficulty with combined distractions, and method rankings that differ from DM Control.Source: §I · p. 1

II. RELATED WORK

Prior pixel-based RL work improved performance on DM Control, while domain randomization and visual-distraction studies addressed variation in different ways. The paper presents a benchmark combining distraction types to support comparisons across approaches.

II. RELATED WORK

  • Image-cropping methods such as CURL, DrQ, and RAD improved pixel-based DM Control results to levels comparable with state-based control.RL has also been applied to real-robot training, while domain randomization is an alternative simulation-based approach intended to enable transfer.Source: §II · p. 2
  • Domain randomization varies simulation conditions as a training strategy, whereas this paper treats distractions as part of the problem agents must handle during evaluation.Agents cannot control the distractions or arbitrarily sample more of them.Source: §II · p. 2
  • The suite combines camera, color, and background distractions and studies state-of-the-art methods to unify comparisons across approaches using different distraction sets.The authors frame the benchmark and its defined settings as support for future research.Source: §II · p. 2

III. THE DISTRACTING CONTROL SUITE

The Distracting Control Suite extends DM Control with scalable camera, color, and background variation, available in static and dynamic settings. Camera-pose difficulty controls the viewing range and, in dynamic settings, the scale of camera motion.

III. THE DISTRACTING CONTROL SUITE

  • The suite extends DM Control with random camera-pose, object-color, and background changes, scaled by a difficulty magnitude from 0 to 1.Distractions can change between episodes only (static) or during episodes (dynamic).Source: §III · p. 2
  • Difficulty scales camera-pose range and velocity, color-change magnitude and speed, and the number or blending weight of background videos.The applicable background parameter depends on the experiment.Source: §III · p. 2

A. Camera Pose

  • Camera pose is parameterized by spherical position angles and radius, plus a roll angle; pitch and yaw are not randomly varied.The camera is oriented toward the agent’s current or starting position depending on whether the task uses a tracking or fixed camera.Source: A. Camera Pose · pp. 2–3
  • Dynamic camera motion adds a random-walk velocity to the sampled starting pose, with spatial and roll velocity limits scaled to camera difficulty.The walk is clipped to maximum velocity and camera-pose parameters.Source: A. Camera Pose · p. 3

B. Object Colors

Object-color distraction changes all simulated bodies’ colors within a difficulty-controlled per-channel bound. Colors remain fixed within static episodes or change randomly within that bound in dynamic episodes.

B. Object Colors

  • The color difficulty scalar βrgb, ranging from 0 to 1, sets the maximum allowed change per color channel for every simulated body.Source: B. Object Colors · p. 3
  • At each episode’s start, each color channel is sampled uniformly within βrgb of its original DM Control value.Source: B. Object Colors · p. 3
  • In static episodes colors stay constant; in dynamic episodes they change randomly and are clipped to remain within βrgb of the original color.Source: B. Object Colors · p. 3

C. Background

Background distractors use video scenes projected onto the skybox, with settings for scene count, episode timing, and visibility.

C. Background

  • Backgrounds come from DAVIS 2017 videos projected onto the scene skybox, with the number of videos varied from 0 to 60.A video and frame are sampled at each episode start.Source: C. Background · p. 3
  • The floor is made transparent so backgrounds remain visible, except in walking tasks where the floor is small and task relevant.Ground-plane opacity is 1.0 for cheetah and walker, 0 for reacher, and 0.3 for other tasks.Source: C. Background · p. 3
  • Dynamic backgrounds play forward or backward and reverse direction at video endpoints, producing smooth motion without cuts.In the static setting, the sampled frame remains constant throughout the episode.Source: C. Background · p. 3

IV. METHODS FOR RL FROM PIXELS

The experiments compare SAC and QT-Opt with image-cropping variants, including a DrQ-style modification to QT-Opt’s Bellman-error minimization.

IV. METHODS FOR RL FROM PIXELS

  • SAC and QT-Opt are compared without cropping, with one RAD crop, or with two-crop averaging as in DrQ.For consistency, QT-Opt without cropping is called QT-Opt, while the version with cropping is called QT-Opt+RAD.Source: §IV · p. 3
  • QT-Opt+DrQ modifies Bellman-error minimization by averaging target values over two random image crops and averaging Q estimates in the loss.The target uses cross-entropy divergence and is defined from reward plus discounted next-state value.Source: §IV · p. 3
  • Methods use published hyperparameters without special tuning for the Distracting Control Suite.SAC+DrQ was tuned for pixel-based DM Control, whereas QT-Opt was tuned for state input.Source: §IV · p. 3

V. EXPERIMENTS

The experiments find that distractions impair current pixel-based RL methods, especially in combination, while method rankings differ from standard DM Control.

V. EXPERIMENTS

  • Methods trained without distractions are fairly robust to color and somewhat robust to camera changes, while distraction training adds little robustness except for bounded background gains.For color changes, training with distractions brings only minor improvement over not training with them.Source: §V · pp. 3–4
  • Distraction effects are more than multiplicative, leaving current methods poor even in the easiest settings combining all three types.The three types are color, camera, and background distractions.Source: §V · p. 4
  • QT-Opt with RAD or DrQ performs best on distracting benchmarks, although SAC-based and QT-Opt-based methods perform comparably on standard DM Control.RAD variants generally work equally well or better than DrQ across experiments.Source: §V · p. 4
  • Training with random video backgrounds outperforms training with static backgrounds, but generalization to new backgrounds remains limited and does not improve with more scenes.Source: §V · p. 4
  • All methods use the same DrQ model architecture, with a shared image encoder and three-layer MLPs for the actor and critic.Source: §V · p. 4
  • Results follow 500K environment steps and 100 evaluation episodes; unless noted, means and standard errors use five seeds per task.Table results are bolded for the highest mean or values not statistically different from it at p < 0.1.Source: §V · p. 4

A. Robustness to Distractions During Inference

Testing models trained on standard DM Control shows that robustness depends on distraction type and declines as distraction intensity increases.

A. Robustness to Distractions During Inference

  • The inference test trains methods without distractions, then evaluates each distraction type at varying intensity, with 60 background scenes.Source: A. Robustness to Distractions During… · p. 4
  • Without distractions, one or two cropping augmentations are necessary for high performance, and SAC- and QT-Opt-based methods perform comparably.Source: A. Robustness to Distractions During… · p. 4
  • All methods lose performance as distraction intensity rises, coping best with color, less well with camera changes, and poorly with unseen backgrounds.Top methods lose half their score at camera scale βcam = 0.2, color scale βrgb = 0.6, and background weight βbg < 0.1.Source: A. Robustness to Distractions During… · p. 4

B. Training with Distractions

Training with distractions improves performance for camera and especially background variations, but not color variations. In the dynamic setting, camera and background distractions yield higher scores than in the static setting.

B. Training with Distractions

  • Training with distractions improves camera performance and especially background performance, but not color performance compared with testing models trained without distractions.For background distractions, more training videos reduce performance on those same videos while improving performance on unseen videos before results level off.Source: B. Training with Distractions · p. 4
  • Dynamic camera and background distractions produce higher scores than static distractions when training with distractions.The authors suggest this may reflect exposure to a larger variety of distraction instances during training.Source: B. Training with Distractions · p. 4
  • DrQ-based approaches consistently outperform SAC-based approaches across settings when training with distractions.Source: B. Training with Distractions · p. 4

C. A New Benchmark for Control from Pixels

The combined-distraction benchmark is designed to assess whether pixel-based agents can extract task-relevant information amid visual distractions. Its results show compounding performance degradation, especially in the dynamic medium setting, and a changed ranking of methods.

C. A New Benchmark for Control from Pixels

VI. CONCLUSION

The Distracting Control Suite benchmarks pixel-based control under visual distractions and reveals limits in current methods. QT-Opt-based methods perform better under distractions than SAC-based methods, although the two perform similarly on the original DM Control benchmark.

VI. CONCLUSION

  • Visual distractions challenge current methods, especially in combination; random cropping is essential for good performance, but DrQ does not outperform the simpler RAD approach.Source: §VI · p. 6
  • QT-Opt-based methods perform better than SAC-based methods under distractions despite their similar performance on the original DM Control benchmark.Source: §VI · p. 6
  • The suite is available as a benchmark for pixel-based control with visual distractions.The paper provides a code repository.Source: §VI · p. 6
Loading 2101.02722v1…