Source-linked AI summary
dm_control: Software and Tasks for Continuous Control
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Piotr Trochim, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess
TL;DR
dm_control addresses the need for software and standardized tasks for reinforcement-learning research in articulated-body simulation. It combines MuJoCo bindings, procedural model and task-authoring libraries, and benchmark and example task suites. The package provides reusable infrastructure for testing and comparing physics-based control algorithms, while some interface and task conventions remain bounded by documented exceptions and assumptions.
Problem
Research on reinforcement learning for physics-based control needs reusable software, task-authoring tools, and standardized benchmarks for testing and performance comparison.
Method
dm_control combines MuJoCo wrapper bindings with PyMJCF and Composer libraries, a Control Suite benchmark, and configurable task frameworks for locomotion and manipulation.
Results
dm_control has been used extensively in DeepMind and offers a wide range of pre-designed reinforcement-learning tasks alongside a framework for designing new ones.
Takeaways & Limitations
The Control Suite and associated libraries provide a starting place for testing and comparing reinforcement-learning algorithms for physics-based control.
Takeaways & Limitations
Control Suite conventions include documented exceptions, including LQR-specific action, reward, and termination behavior.
Abstract
from arXiv · showhide
The dm_control software package is a collection of Python libraries and task suites for reinforcement learning agents in an articulated-body simulation. A MuJoCo wrapper provides convenient bindings to functions and data structures. The PyMJCF and Composer libraries enable procedural model manipulation and task authoring. The Control Suite is a fixed set of tasks with standardised structure, intended to serve as performance benchmarks. The Locomotion framework provides high-level abstractions and examples of locomotion tasks. A set of configurable manipulation tasks with a robot arm and snap-together bricks is also included. dm_control is publicly available at https://www.github.com/deepmind/dm_control
1 Introduction
dm_control is a Python-based software package for continuous-control and robotics research, combining MuJoCo bindings, procedural modeling, task authoring, standardized benchmarks, and reusable locomotion and manipulation frameworks.
- Motivation: dm_control addresses physical control, whose states, times, and actions are continuous and whose dynamics follow second-order equations of motion.The package has been used extensively within DeepMind for continuous-control research.
- Software package: dm_control combines a MuJoCo wrapper, PyMJCF, a consistent environment API, and Composer for continuous-control and robotics research.Composer adds model variation and observable modules for task authoring.
- Control Suite: The Control Suite provides standard benchmarks with unified rewards, interpretable learning curves, aggregated suite-wide measures, and uniform, extensible code patterns.It is intended as a standardized playing field for evaluating and comparing continuous-control methods.
- Locomotion: The Locomotion framework supports many task variants through self-contained, reusable components and has supported multiple reinforcement-learning and multi-agent research efforts.Its design was inspired by prior locomotion research and emphasizes composability.
- Manipulation: The manipulation examples use a simulated 6 degree-of-freedom Kinova Jaco-based arm for reaching, placing, stacking, throwing, assembly, and disassembly.Reusable snap-together bricks and reward-function examples support vision, low-level features, or both.
Software Infrastructure
The paper collects dm_control usage examples into a Google Colab notebook for convenient access to the software tutorials.
- Tutorial notebook: Code snippets from Sections 2–5 are collated in a Google Colab tutorial notebook.The notebook is identified as tutorial.ipynb in the dm_control repository.
2 MuJoCo Python interface
The MuJoCo Python interface exposes simulation models, state, rendering, stepping, named indexing, interactive visualization, and low-level bindings through Python.
- Physics engine: MuJoCo is a fast, reduced-coordinate, continuous-time physics engine suited to articulated systems with contacts and low-to-medium degrees of freedom.The interface uses Python bindings to expose MuJoCo structures, enums, and API functions.
- Model loading and rendering: Physics.from_xml_string() loads an MJCF model into a Physics instance, whose render() method returns NumPy pixel arrays.Rendering supports resolution, camera, RGB, depth, segmentation, and selectable Linux backends.
- Simulation data: Physics exposes writable views of MuJoCo’s model and data arrays, requiring slice assignment rather than replacing an entire array.For example, physics.data.qpos[:] succeeds whereas assigning physics.data.qpos fails.
- Stepping and state: MuJoCo advances qpos and qvel through mj_step1() and mj_step2(), separating state-dependent computations from control-dependent computations.Physics.reset_context() ensures state changes are followed by forward computation and up-to-date derived quantities and sensors.
- Named indexing: Physics.named.model and Physics.named.data provide convenient named views for reading and writing model quantities instead of relying on indices.Named indexing supports NumPy-style indexing and convenient access to multi-degree-of-freedom joint slices.
- Debugging and bindings: The viewer provides mouse-based playback and interaction for visually debugging physical models, including agent-discovered physics exploits.The wrapper also exposes MuJoCo functions and enums while automatically converting NumPy arrays to data pointers.
3 The PyMJCF library
PyMJCF provides a Python object model for composing and modifying MJCF scenes procedurally, while binding compiled simulation data back to model elements and supporting debugging.
- Python model manipulation: PyMJCF lets users interact with and modify complex articulated MJCF models programmatically through Python.Its object model is analogous to the JavaScript DOM for HTML.
- Model composition: PyMJCF composes multiple MJCF models while automatically maintaining a consistent, collision-free namespace.The bind() method connects compiled physics data with the PyMJCF object tree.
- Reusable components: The Leg class models an articulated two-joint leg with proportional position actuators, while attach() procedurally connects leg models to a torso.The example uses reusable MJCF elements and Python-defined defaults to assemble the creature.
- Procedural construction: The tutorial constructs reusable creatures with configurable leg counts, attaches them to an arena, renders them, and records locomotion trajectories.Six creatures with 3 to 8 legs are instantiated and driven by sinusoidal open-loop controls.
- Physics binding: physics.bind() provides unified access to associated mjData and mjModel fields, including torso positions and colors used for trajectory plots.This enables simulation outputs and model attributes to be accessed through the corresponding MJCF elements.
- Debugging: PyMJCF debugging can trace XML elements to modifying Python stack frames, with a full-dump mode for errors involving attached-model incompatibilities or broken cross-references.Debug tracking is disabled by default because it is expensive to run.
4 Reinforcement learning interface
dm_control provides a standard reinforcement-learning interface for sequential agent–environment interaction, with structured timesteps, specifications, and configurable rewards. Its environment API also represents termination and discounting for different horizon formulations.
- Reinforcement learning agents interact sequentially with dm_control environments to learn policies that maximise future rewards.
- Each environment step returns a TimeStep containing step_type, reward, discount, and observation fields, with FIRST, MID, and LAST episode markers.
- The Environment class implements the dm_env interface, including reset(), step(), action_spec(), and observation_spec() methods.These methods initialise state, advance simulation, and describe accepted actions and returned observations.
- The discount γ distinguishes terminal formulations: γ = 0 denotes terminal states, while terminal γ = 1 represents truncated infinite-horizon episodes.For γ = 1, a parametric value function may estimate future returns.
- Rewards are generally in [0, 1], with sparse tasks using {0, 1}; tolerance() supports smooth or binary reward terms that preserve this range under averaging and multiplication.Figure 3 describes infinite-support and finite-support tolerance reward functions.
5 The Composer task definition library
Composer structures reinforcement-learning tasks from reusable entities, task logic, and an environment wrapper. It also supports configurable observables, stochastic variation, and ordered lifecycle callbacks for procedural task authoring.
- Composer represents task designs through Entity, Task, and Environment abstractions that combine scene structure, task logic, and agent interaction.Entities form trees, Tasks provide reward and observation logic, and Environments compile models and manage episodes.
- Observables expose simulation-derived quantities and can be configured with enabling, update intervals, buffers, corruptors, aggregators, and delays.These features model sampling rates, sensor noise, temporal aggregation, and latency.
- Composer optimizes observable evaluation by computing only values that can appear in future observations, avoiding discarded intermediate computations.For example, an observable updated every step with buffer_size=1 is evaluated once per control step.
- The variation module adds stochasticity to observables and models, including noise, rotations, distributions, deterministic values, and MJCF- or physics-level variations.MJCFVariator runs before model compilation, whereas PhysicsVariator runs after compilation.
- Composer callbacks execute first at the Task level, then across Entities in depth-first order from the root arena and attachment order.The tutorial also requires overriding _build and _build_observables so the MJCF model exists before observables.
6 The Control Suite
The Control Suite is a stable, standardized benchmark collection for continuous-control agents, with common action, observation, reward, and evaluation conventions. Its development process tests both physical stability and task solvability to reduce exploitable or unintended behavior.
- The Control Suite provides stable, well-tested continuous-control tasks with standardized structures that simplify suite-wide benchmarking and learning-curve interpretation.Unlike more elaborate domains, its tasks are not intended to be modified.
- Except for LQR, actions lie in the unit box a ∈ [−1, 1]^dim(A), while most rewards lie in [0, 1] and some are sparse in {0, 1}.The tolerance() function facilitates the reward structure.
- Default observations are strongly observable: the state can be recovered from one observation, although control-dependent features depend on the previous transition.
- Control Suite tasks are infinite-horizon problems without terminal states or time limits, but agents use discounted returns with γ = e^−h/τ internally.Fixed-length 1000-step episodes provide a practical proxy for evaluating infinite-horizon returns.
- Model and Task verification: Time-step selection trades simulation stability against speed because smaller steps are more stable but require more computation.Learning agents can discover and exploit instabilities caused by discretization.
- Model and Task verification: The authors iterated task designs with multiple learning agents until physics was stable and non-exploitable and each benchmark task was solved by at least one agent.Unsolved tasks were placed in an extra task set.
The suite module
The suite module loads named Control Suite tasks and supports wrappers that modify environment behavior. It includes feature-to-pixel observation conversion and reward-linked visualizations, while documenting known simulation exploits and provenance details.
- suite.load(domain_name, task_name) loads an environment representing a selected task, and suite.BENCHMARKING supports iteration over the benchmark task set.
- Wrappers modify environment behavior, including replacing feature observations with pixels or adding pixel observations alongside existing features.
- Simulation bugs that leak energy can be discovered and exploited by learning agents, a phenomenon identified as Sims’ Law.
- Control Suite models use common colours and textures, which can be modified in proportion to reward as a visual cue.
6.2 Domains and Tasks
The Control Suite organizes physical models into task-specific MDPs spanning classic control, locomotion, manipulation, and procedurally generated systems. Benchmarking tasks are grouped separately from non-benchmarking tasks, with task descriptions specifying bodies, rewards, initialization, and goals.
- Task organization: A domain is a physical model, while a task is an instance with a particular MDP structure and initialization.The swingup and balance cartpole tasks differ in whether the pole starts downwards or upright.
- Classic control: The suite includes classic control domains such as pendulum, acrobot, cart-pole, cart-k-pole, ball in cup, point-mass, reacher, and finger.These tasks cover swing-up, balancing, target reaching, catching, and object rotation.
- Locomotion: Locomotion domains include hopper, fish, cheetah, walker, humanoid, and humanoid_CMU, with rewards targeting posture, velocity, swimming, or running.The humanoid tasks specify desired horizontal speeds of 0, 1, and 10m/s for stand, walk, and run.
- Manipulation: Manipulation domains include a planar manipulator and a procedurally generated stacker, with tasks requiring reaching, insertion, bringing objects, or stacking boxes.The stacker rewards a box at the target while the gripper is away, making stacking necessary.
- Additional domains: The suite also provides procedural swimmers and an analytic LQR domain, while LQR is excluded from benchmarking because its controls and reward are unbounded.The benchmarking results for the BENCHMARKING tasks are documented in the original Control Suite technical report.
6.3 Additional domains
Additional domains extend dm_control with humanoid motion-capture support, quadruped and dog models, and a rodent model designed for comparisons with life-science experiments. These additions broaden the available bodies, sensing settings, and research applications.
- Motion capture: humanoid_CMU supports imitation learning through tools that parse, convert, and play back CMU Motion Capture Database data.The convert() function returns a sequence of configurations for the humanoid_CMU model from an AMC file.
- Quadruped: The quadruped provides flat-ground walk and run tasks plus escape over random terrain and fetch of a moving ball to a target.The escape task uses 20 range-finder sensors, while the fetch task takes place in an enclosed arena.
- Dog: The Pharaoh Dog model includes procedurally created kinematics, skinning weights, collision geometry, muscles, and tendon attachment points.Muscles and tendons are included statically but are not yet part of the dynamical model because detailed anatomical knowledge is required.
- Rodent: The rodent model was built to compare learned behavior with experimental settings in the life sciences and has been studied using visual inputs.Its reference skeleton is not included in the physical simulation.
7 Locomotion tasks
The Locomotion framework composes reusable Walkers, Arenas, and Tasks into navigation, foraging, rough-terrain, and multi-agent soccer environments. It supports procedural arenas, varied bodies, and consistent multi-agent interaction conventions.
- Framework: Locomotion uses Walkers, Arenas, and Tasks as reusable abstractions for specifying complete reinforcement-learning environments.Walkers provide locomotion-specific observation transformations, while Arenas can rescale for different walker sizes.
- Corridor navigation: A corridor example combines a position-controlled CMU humanoid, a wall-obstructed arena, and a task rewarding running at a target velocity.The example uses a target velocity of 3.0, a 0.005 physics timestep, and a 0.03 control timestep.
- Corridor navigation: The CMU humanoid used in the example is the improved V2020 version, with revised height, mass, proportions, actuator gains, and torque limits.These changes target more human-like body properties and position-control behavior.
- Maze navigation and foraging: A procedural maze generator constructs navigation and foraging tasks for bodies including humanoids and rodents.The maze can be scaled to human-sized or rodent-scale settings.
- Multi-Agent soccer: Multi-agent soccer extends the Walker-Arena-Task structure with multiple physically interacting walkers and supports BoxHead and Ant walker types.Synchronous environments exchange sequences of per-agent observation dictionaries and action arrays.
8 Manipulation tasks
The manipulation module supplies a robotic arm, object models, observation variants, reward functions, and configurable brick tasks. Its environments range from reaching and lifting to ordered or unordered brick assembly and reassembly.
- Module and observations: The manipulation module provides a robotic arm, simple objects, and tools for constructing manipulation-task reward functions.Its environments are available in versions with different observation types.
- Module and observations: Feature observations include arm joint positions, velocities, torques, and privileged task-specific information about movable objects.Vision observations instead include a fixed RGB camera view of the workspace.
- Module and observations: All manipulation environments return rewards r(s, a) ∈[0, 1] per timestep and have a 10-second episode limit.Environments can be listed by name or filtered by observation tags such as vision.
- Brick tasks: Some tasks change the physical or informational challenge through movable bases, non-prehensile manipulation, visual goal hints, or order information in feature observations.The large box cannot be grasped by the gripper, while random-order stacking can expose the desired order as indices.
- Brick tasks: The brick tasks cover reaching, lifting, placing, snapping, stacking, and reassembling bricks in fixed or variable orders.The stack_3_bricks task fixes the bottom brick to the floor and requires the top two bricks in a specific order.
9 Conclusion
dm_control provides tools for testing and comparing reinforcement learning algorithms in physics-based control, while supporting new task design and community research.
- dm_control supports reinforcement learning algorithm testing and performance comparison for physics-based control.It also offers pre-designed tasks and a framework for designing new ones.