PaperlayerView exact revision on arXiv

Source-linked AI summary

MuJoCo Playground

Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carmelo Sferrazza, Yuval Tassa, Pieter Abbeel

arXiv:2502.08844v1cs.RO

TL;DR

Robot learning needs fast, realistic, and usable simulation to support repeated reward-design, training, and deployment iterations. MuJoCo Playground addresses this with an open-source MJX-based physics, rendering, and environment stack, demonstrating rapid training and zero-shot transfer across varied robots and inputs. Its main scope boundary is inherited MJX and early-stage vision-training limitations.

  • Problem

    Sim-to-real reinforcement learning requires a realistic, convenient, and fast simulator because reward design and training are iterative and computationally intensive.

  • Method

    MuJoCo Playground integrates GPU-based MJX physics, Madrona batch rendering, and reproducible training environments into an installable open-source pipeline.

  • Results

    100% success rate in 12 real-world trials was achieved for zero-shot pixel-based pick-cube transfer, alongside demonstrations across diverse robotic platforms and policy inputs.

  • Takeaways & Limitations

    The framework makes sim-to-real robot learning accessible across locomotion and manipulation, including state- and pixel-based policies.

  • Takeaways & Limitations

    MuJoCo Playground inherits MJX constraints, including 1–3-minute JIT compilation, contact-scaling issues from static shapes, and early-stage Madrona vision training.

Abstract

from arXiv · show

We introduce MuJoCo Playground, a fully open-source framework for robot learning built with MJX, with the express goal of streamlining simulation, training, and sim-to-real transfer onto robots. With a simple "pip install playground", researchers can train policies in minutes on a single GPU. Playground supports diverse robotic platforms, including quadrupeds, humanoids, dexterous hands, and robotic arms, enabling zero-shot sim-to-real transfer from both state and pixel inputs. This is achieved through an integrated stack comprising a physics engine, batch renderer, and training environments. Along with video results, the entire framework is freely available at playground.mujoco.org

I. INTRODUCTION

MuJoCo Playground targets the iterative, computationally intensive workflow of sim-to-real reinforcement learning with an open-source, GPU-native framework. It combines MJX physics, Madrona rendering, diverse environments, and reproducible training resources to accelerate policy development and deployment.

  • Motivation: Sim-to-real reinforcement learning requires realistic simulation, reward design, policy training, and deployment, with the simulator needing realism, usability, and speed.Reward design and policy training often require repeated iteration because learned behaviors can be irregular or unexpected.
  • Motivation: GPU-based simulation accelerates reinforcement learning by exploiting powerful parallel hardware and keeping agent-environment interaction on device.Locomotion and manipulation tasks that previously required days on multi-host systems can be solved in minutes or hours on a single GPU.
  • Framework: MuJoCo Playground integrates MJX physics, Madrona batch rendering, and training environments into a fully open-source, cross-platform pipeline installable with pip install playground.The complete workflow can run in a single Colab notebook, with most tasks requiring only minutes of training.
  • Framework: The framework simplifies sim-to-real deployment into an interactive process for quickly adjusting parameters and refining robot behavior.The authors report deploying state- and vision-based policies across six robotic platforms in less than eight weeks.
  • Contributions: Its contributions include diverse MJX robotic environments, GPU-rendered end-to-end vision-policy training with zero-shot manipulation transfer, and a complete reproducible training pipeline.Supported platforms include quadrupeds, humanoids, dexterous hands, and robot arms.
  • Environment Suite: Playground includes DM Control Suite, locomotion, and manipulation environment categories, with most DM Control Suite environments reimplemented in MJX as entry-level tasks.The environment suite is designed to familiarize users with MuJoCo Playground.

B. Locomotion

MuJoCo Playground provides GPU-based environments and rendering for locomotion and manipulation, supporting state and pixel observations across simulated and real robotic tasks. Its demonstrations cover multiple robot embodiments and include on-device pixel-based training.

  • Environment Coverage: Locomotion environments cover multiple quadruped and humanoid embodiments, while manipulation environments include prehensile, non-prehensile, dexterous, and vision-based tasks.Examples include Unitree, Berkeley Humanoid, LEAP Hand, Franka, Robotiq, and Aloha platforms.
  • Sim-to-Real Experiments: Sim-to-real experiments span Unitree Go1 locomotion, three humanoid platforms, LEAP Hand manipulation, and Franka arm tasks, including vision-trained pick-cube control.The reported experiments use both proprioceptive state and visual inputs.
  • Batch Rendering: Madrona integrates with MJX to provide GPU-based rendering with lighting, shadows, textures, materials, and customizable visual randomization.The integration connects Madrona rendering to JAX transformations such as jit and vmap.
  • Training: Most state-based environments train in under 10 minutes on a single GPU device.The framework also supports training across different hardware devices and reinforcement-learning libraries.
  • Pixel-Based Training: CartpoleBalance demonstrates pixel-based observations generated on GPU while physics, rendering, and training remain entirely on device.End-to-end pixel training is demonstrated on only one DM Control Suite task, with broader exploration left for future work.

B. Locomotion

The locomotion evaluation tests zero-shot sim-to-real transfer across a quadruped and three humanoid platforms. Policies are trained with domain randomization and then deployed on varied real-world surfaces and terrains.

  • Platforms: Sim-to-real locomotion is evaluated on Unitree Go1, Berkeley Humanoid, Unitree G1, and Booster T1 platforms.The experiments cover both quadruped and humanoid morphologies.
  • Quadruped Locomotion: The Go1 task suite includes joystick locomotion, handstand, footstand, and fall recovery policies.Joystick commands specify forward velocity, lateral velocity, and turning rate, while balance tasks minimize actuator torque.
  • Quadruped Locomotion: Go1 policies transfer robustly to reality, coping with uneven terrain and moderate external perturbations without additional fine-tuning.Training uses domain randomization for sensor noise, dynamics, and task uncertainties, followed by rough-terrain finetuning.
  • Humanoid Locomotion: Flat-ground training lasts under 15 minutes for Berkeley Humanoid and under 30 minutes for Unitree G1 and Booster T1 on two RTX 4090 GPUs.The humanoid experiments follow the quadruped domain-randomization and finetuning strategies.
  • Humanoid Locomotion: Humanoid joystick policies track velocity commands on Berkeley Humanoid surfaces ranging from rigid to soft and slippery terrains.Unitree G1 and Booster T1 achieve stable walking and turning on standard indoor floors with zero-shot policies.

C. Manipulation

MuJoCo Playground demonstrates sim-to-real manipulation across dexterous in-hand reorientation and non-prehensile block manipulation. The framework combines randomized training and hardware-oriented control to support real-world deployment.

  • Manipulation scope: Playground covers dexterous in-hand manipulation and non-prehensile manipulation among its demonstrated sim-to-real tasks.The manipulation results include in-hand cube reorientation and yoga-block reorientation on a Franka Emika Panda.
  • In-Hand Cube Reorientation: The LEAP task repeatedly reorients a 7 cm cube from random initial poses to target orientations in SE(3) without dropping it.The setup uses a low-cost LEAP hand and follows prior in-hand manipulation work.
  • In-Hand Cube Reorientation: Domain randomization, sensor noise, and progressive curriculum training support the LEAP policy’s sim-to-real transfer.Randomized parameters include robot properties, cube mass, and friction; training takes within 30 minutes on two RTX 4090 GPUs.
  • In-Hand Cube Reorientation: The most frequent LEAP failure occurs when the cube wedges between the fingers and palm, causing the policy to stall.Less common failures involve accidental index-thumb interlocking attributed to physical flex in the low-cost hardware.
  • Non-prehensile Manipulation: The Panda task reorients a yoga block from a random pose to a fixed goal using direct 200 Hz torque control.The policy succeeds when the block finishes within 3 cm of the goal position and 10° of the desired orientation.
  • Non-prehensile Manipulation: Stochastic delays and progressive curriculum learning enable robust zero-shot transfer for Panda block reorientation.Training randomizes initial states and hardware latency while progressively increasing displacement and orientation difficulty.

3) Pick-Cube from Pixels:

MuJoCo Playground trains pixel-based manipulation policies and evaluates its GPU-oriented training stack. Pixel observations, randomized visual conditions, and on-device simulation and rendering support real-hardware transfer while revealing data-processing bottlenecks.

  • Pick-Cube from Pixels: The pixel-based Panda policy receives a 64 × 64 RGB image and lifts a randomly located 2 × 2 × 3 cm block 10 cm above the table.The action space is restricted to planar Y-Z Cartesian motion with binary jaw control.
  • Pick-Cube from Pixels: Domain randomization covers lighting, shadows, camera pose, object colors, brightness, and gripping delays of up to 250 ms.A three-dimensional action representation improves training sample efficiency, while larger action spaces remain possible with additional camera observations.
  • Pick-Cube from Pixels: 100% success in 12 real-world trials demonstrates zero-shot transfer of the pixel-based policy to hardware.The policy remained resilient to moderate lighting variation and minor camera shaking.
  • Training Throughput: 2080 s on 1x 4090 versus 670 s on 8x H100 shows shorter LeapCubeReorient training wallclock with larger, faster GPU topologies.Both configurations use the same hyperparameters, including 8192 environments; topology-specific tuning remains future work.
  • Training Throughput: MJX physics paired with Madrona rendering reaches roughly 403,000 Cartpole and 37,000 Franka environment steps per second with pixel observations.Franka physics is over 20x more costly than Cartpole’s, producing lower throughput and weaker sensitivity to image resolution.
  • Training Throughput: Physics, rendering, and inference comprise 9% of Cartpole and 43% of Franka PPO training time, shifting the bottleneck toward CNN updates.Figure 7 measures only physics and rendering, not total training throughput.

2) RL Libraries:

MuJoCo Playground combines open-source GPU physics, rendering, and reinforcement-learning components to support accessible sim-to-real research across robotic platforms and observation types.

  • The framework supports both JAX- and torch-based RL libraries, with PPO implementations achieving successful rewards and gaits within similar wallclock times.
  • Unlike closed-source GPU physics implementations, the framework provides open-source environments and training code that researchers can extend and use across robotics tasks.
  • MuJoCo Playground demonstrates zero-shot sim-to-real transfer across the Leap Hand, Unitree Go1, Berkeley Humanoid, Unitree G1, Booster T1, and Franka arm.
  • MuJoCo Playground uses MJX for GPU-based physics and Madrona for on-device batch rendering, integrating simulation, rendering, and training.
  • The framework enables direct vision-based policy training in simulation with high-throughput rendering, avoiding a distillation step, while vision-based training remains an early-stage capability.

Appendix

The appendix documents environment coverage, training evaluation, locomotion observations and actions, and domain-randomization procedures used to support sim-to-real transfer.

  • MuJoCo Playground re-implements DM Control Suite environments in MJX, with selected XML modifications for solver and contact-parameter performance.
  • PPO and SAC reward curves are evaluated across the DM Control Suite using 5 seeds on a single A100 GPU, with environment-step and wallclock-time views.
  • Training throughput is reported for DM Control Suite, locomotion, and manipulation environments, including 95th percentile confidence intervals for the tabulated results.
  • Locomotion observations include body-frame gravity, base velocities, joint positions and velocities, previous action, and optionally user commands.
  • Humanoid policies additionally receive sine and cosine encodings of each foot’s gait phase to represent phase continuously and smoothly.
  • Joystick tasks use absolute joint positions with a default offset, whereas other tasks use relative joint positions, followed by PD torque control.
  • Training randomizes sensor noise, difficult-to-measure dynamic properties, ground friction, and payload mass to reduce the sim-to-real gap.

B.23. Reward and Termination:

The framework defines locomotion rewards and termination rules, uses asymmetric actor–critic training, and evaluates training across locomotion and manipulation environments and hardware configurations.

  • The total reward is a weighted sum of reward terms and is clipped to remain non-negative.
  • Termination uses inversion detection with a reduced feet-only collision model for joystick policies and a full primitive-based collision model for other tasks.
  • The asymmetric actor–critic gives the actor task observations while providing the critic with uncorrupted signals and additional contact, perturbation, and torque readings.
  • Both policy and value networks use three-layer MLPs with hidden sizes 512, 256, and 128 and Swish activations.
  • Locomotion PPO experiments report reward against environment steps and wallclock time across 5 seeds on one A100 GPU.
  • Different GPU devices and topologies do not materially change Go1JoystickFlatTerrain training time because the environment has limited contacts.
  • Manipulation PPO experiments likewise report reward curves across 5 seeds, with throughput tables and GPU-topology comparisons for LeapCubeReorient.

C.3. RL Training Throughput

The manipulation-training section describes the Leap Hand reorientation setup, its system-identification and randomization procedures, and deployment adjustments for actuator and control limitations.

  • LeapCubeReorient training throughput is tabulated, while reward-versus-wallclock performance is compared across GPU devices and topologies.
  • The Leap Hand task sequentially reorients a randomly initialized cube to sampled target orientations, counting success when orientation falls within 0.4 rad.
  • The policy receives noisy hand and cube-state estimates plus previous commanded joint positions and outputs 16 relative joint positions.
  • Training randomizes friction, masses, offsets, motor properties, PD gains, and sensor noise, using an initial 200 M-step phase followed by 100 M-step fine-tuning.
  • System identification targets the DYNAMIXEL servo actuator to improve the performance and transparency of sim-to-real transfer beyond manual parameter tuning.
  • Because the actuator lacks true current control, training randomizes proportional and derivative gains to mitigate mismatch between ideal and deployed PD behavior.
  • Deployment uses the hand’s open-source software, reduces control frequency from 150 Hz to 20 Hz because of USB-driver jitter, and retains identified torque, stiffness, and damping parameters.

C.44. Vision-based Pose Estimator:

The Franka block-reorientation setup combines randomized simulation, torque-based control, and camera-based pose estimation to support zero-shot transfer to hardware. The policy uses robot and block-state observations with 7D joint-torque actions, while real deployment mirrors the simulated control loop.

  • Simulation Environment: The task reorients a randomly initialized yoga block to a central goal pose using a Franka arm with a fixed closed gripper.The policy pushes, slides, or taps the block within a tabletop workspace.
  • Policy Inputs and Actions: The policy receives noisy block pose, robot joint states, end-effector pose, and target pose, and outputs 7D joint-torque commands with a fixed gripper action.These inputs and actions define the control interface used for the reorientation policy.
  • Training Setup: Training randomizes block pose, robot state, delays, and environment difficulty while using curriculum learning and shaped rewards.The curriculum progressively expands the block’s displacement and orientation range.
  • Hardware Control: Real deployment uses a Franka Panda with direct torque control through FCI at 200 Hz, matching the simulator’s update frequency and modeled communication delays.The interface bypasses low-level PID gain schedules and sends raw joint torques.
  • Vision-based Pose Estimation: Four calibrated RGB cameras track fiducial markers on the block, and a central ROS node averages valid detections into a broadcast 6D pose estimate.Two cameras are placed low and two overhead to improve coverage and reduce occlusions.
  • Results: Policies trained under domain randomization and fast torque-control loops transfer zero-shot to real hardware with minimal environment-alignment engineering.The authors also report workspace and force-limit issues addressed through torque penalties.

C.6. Real-world Franka PickCube from Pixels

The pixel-based PandaPickCubeCartesian task trains a vision policy with randomized rendering conditions and deploys it through a lightweight real-time control stack. The section also benchmarks pixel-training throughput and identifies policy updates, rather than rendering, as the main bottleneck.

  • Task Description: The Franka PickCube task moves and grasps a 2x2x3 cm upright cube, with both object randomization and actions restricted to a fixed Y-Z plane.The restriction supports deployment with a single RGB camera.
  • Training: Training combines sparse lifting and placement rewards with dense progress rewards, randomized lighting, colors, brightness, camera pose, and gripper delays.The policy terminates after task completion to improve sample efficiency.
  • Agent: A lightweight CNN processes one RGB image and outputs a 3-value action consisting of Cartesian motion commands and a gripper command.The input is normalized per sample before CNN inference.
  • Hardware: Deployed policies train within ten minutes on a single RTX 4090 and run inference at 15 Hz on an RTX 3090 paired with a Franka arm and Realsense camera.The deployment stack downsamples camera images to 64x64 pixels and converts predictions into joint commands through inverse kinematics.
  • Pixel Environments: Madrona batch rendering enables GPU-generated pixel observations while physics, rendering, and training remain on-device for the demonstrated pixel environments.The paper provides complete Brax PPO training examples for CartPoleBalance and PandaPickCubeCartesian.
  • Performance Benchmarking: Rendering-speed improvements are the primary driver of measured throughput speed-ups, while pixel-training bottlenecks shift to policy updates.For Cartpole, over 90% of training burden is attributed to network updates.

APPENDIX E REINFORCEMENT LEARNING HYPER-PARAMETERS

The appendix catalogs reinforcement-learning hyperparameters used across MuJoCo Playground environments. It separates default settings from task-, robot-, and vision-specific configurations.

  • Overview: The appendix reports hyperparameters for the RL policies used throughout MuJoCo Playground.The listed algorithms include Brax PPO and Brax SAC configurations.
  • Vision-based Training: Vision-based PandaPickCubeCartesian has a dedicated hyperparameter configuration distinct from the state-based task settings.The appendix separately labels vision-based PPO hyperparameters and the vision-based Panda task.
  • Manipulation: Manipulation-specific tables cover Aloha peg insertion, Panda cabinet opening, Panda cube tasks, Robotiq pushing, and Leap Hand cube rotation or reorientation.These configurations are organized by robot and task rather than presented as one universal setting.
Loading 2502.08844v1…