PaperlayerView exact revision on arXiv

Source-linked AI summary

MuJoCo Playground

Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carmelo Sferrazza, Yuval Tassa, Pieter Abbeel

arXiv:2502.08844v1cs.RO

TL;DR

Sim-to-real reinforcement learning requires repeated reward design and policy training, making rapid iteration important. MuJoCo Playground combines GPU simulation, batch rendering, and robotic environments in an open-source framework. The authors demonstrate deployment from state and pixel inputs across locomotion and manipulation tasks on multiple robotic embodiments.

  • Problem

    Sim-to-real reinforcement learning requires repeated reward design and policy training, making time-to-robot a critical concern.

    Source: §I · p. 2

  • Method

    MuJoCo Playground combines GPU simulation, batch rendering, and robotic environments in an open-source framework for sim-to-real reinforcement learning.

    Source: §I · p. 2

  • Results

    The authors demonstrate sim-to-real deployment from state and pixel inputs across locomotion and manipulation tasks on multiple robotic embodiments.

    Source: §VII · p. 7

  • Takeaways & Limitations

    The framework supports interactive sim-to-real iteration, allowing users to quickly adjust parameters to refine robot behavior.

    Source: §I · p. 2

    Contact computation scales with possible rather than active contacts because JAX requires static shapes at compile time.

    Source: §VI · p. 7

Abstract

from arXiv · show

We introduce MuJoCo Playground, a fully open-source framework for robot learning built with MJX, with the express goal of streamlining simulation, training, and sim-to-real transfer onto robots. With a simple "pip install playground", researchers can train policies in minutes on a single GPU. Playground supports diverse robotic platforms, including quadrupeds, humanoids, dexterous hands, and robotic arms, enabling zero-shot sim-to-real transfer from both state and pixel inputs. This is achieved through an integrated stack comprising a physics engine, batch renderer, and training environments. Along with video results, the entire framework is freely available at playground.mujoco.org

I. INTRODUCTION · II. ENVIRONMENTS · A. DM Control Suite

MuJoCo Playground combines MJX simulation, GPU rendering, and reproducible training for rapid sim-to-real iteration with state- and vision-based policies. Its environments span DM Control, locomotion, and manipulation.

I. INTRODUCTION

  • Realistic, convenient, fast simulation underpins four steps: match reality, encode behavior through rewards, train a policy, and deploy it.The digital twin requires sufficient fidelity, while usability must simplify robot creation, modification, composition, and system identification.Source: §I · p. 1
  • Simulation speed matters because reward design remains iterative: reward-maximizing policies can exhibit unexpected behavior, requiring repeated training and sometimes deployment [7].Even semi-automated reward design [37] requires iteration, making time-to-robot—the delay between requesting behavior and observing the robot’s interpretation—critical.Source: §I · pp. 1–2
  • GPU simulation accelerates RL through greater typical processing power and an entirely on-device interaction loop exploiting parallel throughput [39].On-policy RL [3] [51] particularly benefits from GPU-friendly, wide-batch operations.Source: §I · p. 2
  • Most tasks require only minutes of training in MuJoCo Playground, a fully open-source framework built on GPU-based MJX.Installation uses pip install playground, with cross-platform support and single-GPU training in a Colab notebook. The reproducible pipeline includes notebooks, hyperparameters, and training curves.Source: §I · p. 2
  • Less than eight weeks sufficed to deploy state- and vision-based policies across six robotic platforms.The environment suite demonstrates sim-to-real transfer across quadrupeds, humanoids, dexterous hands, and robot arms.Source: §I · p. 2
  • The Madrona batch GPU renderer enables end-to-end vision-policy training on one GPU, without teacher-student distillation [1], and zero-shot transfer on manipulation tasks.Rendering runs on-device alongside physics and learning.Source: §I · p. 2

II. ENVIRONMENTS

  • Playground groups environments into DM Control, locomotion, and manipulation, with robotic tasks supporting zero-shot sim-to-real transfer across diverse platforms.It directly uses MuJoCo Menagerie robot assets and configurations; supported platforms include quadrupeds, humanoids, dexterous hands, and robot arms.Source: §II · p. 2, §I · p. 2

A. DM Control Suite

  • Within Playground’s MJX environment suite, most DM Control environments from [61] are reimplemented as entry-level tasks for familiarizing users with the framework.These introductory tasks complement the suite’s robotic environments for sim-to-real transfer.Source: A. DM Control Suite · p. 2, §I · p. 2

B. Locomotion … A. DM Control Suite

MuJoCo Playground combines locomotion and manipulation environments with MJX–Madrona batch rendering and demonstrates sim-to-real transfer from state and vision. Most environments train state-based policies in under 10 minutes on one GPU, while pixel-based end-to-end DM Control training is demonstrated only on CartpoleBalance.

B. Locomotion

  • Locomotion environments provide velocity-command tracking for quadrupeds and humanoids, additional Go1 recovery and handstand tasks, and demonstrated hardware transfer.Supported robots include Unitree Go1, Boston Dynamics Spot, Google Barkour [6], Berkeley Humanoid [35], Unitree H1 and G1, Booster T1, and Robotis OP3; commands specify forward/lateral velocities and yaw rate. Hardware demonstrations cover joystick, fall recovery, and handstand policies on Go1, plus joystick locomotion on Berkeley Humanoid, G1, and T1.Source: B. Locomotion · p. 2

C. Manipulation

  • Manipulation environments cover prehensile and non-prehensile tasks, with sim-to-real transfer on Leap Hand and Franka robots, including vision-trained cube picking.Tasks include contact-rich block reorientation with Leap Hand, yoga-block reorientation using high-frequency torque control with a Franka Emika Panda and Robotiq gripper, and Madrona-rendered Franka cube picking. Additional environments include bi-arm peg insertion with Aloha [2].Source: C. Manipulation · p. 2

III. BATCH RENDERING WITH MADRONA

  • MJX integrates with Madrona, a GPU-based entity-component system, to enable vision-based environments using batch rendering.Madrona offers a CUDA software batch ray tracer, used in these experiments, and a Vulkan rasterizer; the ray tracer supports complex lighting, shadows, textures, and geometry materials. Deformable materials, moving lights, and terrain height fields are planned future features.Source: §III · p. 3
  • Low-level JAX primitives connect MJX to Madrona initialization and rendering, supporting jit, vmap, and per-instance visual domain randomization.Cartpole-balance and PandaPickCubeCartesian illustrate vision-based environments and policy training. Randomized geometry size, color, lighting, and camera pose play a crucial role in transferring vision-based policies to hardware.Source: §III · p. 3

IV. RESULTS

  • The results cover reinforcement learning and locomotion/manipulation sim-to-real experiments using proprioceptive state and vision, alongside training discussions across hardware devices and RL libraries.Source: §IV · p. 3

A. DM Control Suite

  • Under 10 minutes on a single GPU is sufficient to train state-based policies for most environments.State-based policies are trained for all available tasks in the MJX port of the DM Control Suite.Source: A. DM Control Suite · p. 3
  • Pixel-based CartpoleBalance keeps physics, rendering, and training entirely on the GPU, but end-to-end vision-based training is demonstrated only for this DM Control task.Other DM Control Suite environments can also be rendered with Madrona; broader training experiments remain future work.Source: A. DM Control Suite · p. 3

B. Locomotion · 1) Quadruped Locomotion: · 2) Humanoid Locomotion:

Locomotion policies transfer from simulation to the Unitree Go1 quadruped and three humanoid platforms. Results demonstrate robust quadruped behaviors without additional fine-tuning and successful humanoid velocity tracking, walking, and turning.

B. Locomotion

  • Sim-to-real locomotion experiments span Unitree Go1, Berkeley Humanoid, Unitree G1, and Booster T1, covering quadruped and humanoid platforms.Section B provides further MDP details, including rewards, observation spaces, and action spaces.Source: B. Locomotion · p. 4

1) Quadruped Locomotion:

  • Quadruped tasks include joystick locomotion [25] [50], torque-minimizing front-leg handstands and hind-leg footstands, and recovery from arbitrary fallen configurations [30].Joystick commands specify root-body forward velocity, lateral velocity, and turning rate. Fall recovery targets a stable home posture.Source: 1) Quadruped Locomotion · p. 4
  • Within 5 minutes on two RTX 4090 GPUs, quadruped policies train on flat ground with restricted commands before rough-terrain fine-tuning with wider command ranges.Domain randomization covers sensor noise, dynamics properties, and task uncertainties.Source: 1) Quadruped Locomotion · p. 4
  • All four Unitree Go1 policies—joystick, handstand, footstand, and fall recovery—transfer robustly without additional fine-tuning, handling uneven terrain and moderate external perturbations.Deployments cover outdoor grass and concrete and indoor surfaces with different friction properties.Source: 1) Quadruped Locomotion · p. 4

2) Humanoid Locomotion:

  • 6 DoF per leg characterize Berkeley Humanoid [35], while Unitree G1 and Booster T1 have 29 and 23 total DoF, respectively.Berkeley Humanoid is low-cost and lightweight; Booster T1 is small-scale. All three platforms are evaluated indoors with slight variations in surface friction and ground compliance.Source: 2) Humanoid Locomotion · p. 4
  • Under 15 minutes of flat-ground training is used for Berkeley Humanoid, versus under 30 minutes for Unitree G1 and Booster T1, on two RTX 4090 GPUs.Humanoids use the quadruped’s joystick task and follow its domain-randomization and fine-tuning strategies.Source: 2) Humanoid Locomotion · p. 4
  • Berkeley Humanoid robustly tracks velocity commands across rigid, soft, and slippery surfaces; Unitree G1 and Booster T1 achieve zero-shot stable walking and turning indoors.Minor tuning for each platform’s dynamics may further enhance performance.Source: 2) Humanoid Locomotion · p. 4

C. Manipulation · 1) In-Hand Cube Reorientation: · 2) Non-Prehensile Block Reorientation:

MuJoCo Playground demonstrates real-world in-hand cube reorientation and zero-shot non-prehensile block reorientation, supported by task-specific randomization and training curricula.

C. Manipulation

  • The manipulation demonstrations span dexterous in-hand manipulation, non-prehensile manipulation, and vision-based grasping, illustrating deployment across diverse real-world manipulation tasks.Source: C. Manipulation · p. 4

1) In-Hand Cube Reorientation:

  • 7 cm cube reorientation on the low-cost LEAP hand requires repeatedly reaching new target orientations from random initial poses without dropping the cube.The task closely follows previous in-hand manipulation works [19].Source: 1) In-Hand Cube Reorientation · p. 4
  • 30 min is the training-time upper bound on two RTX 4090 GPUs for the in-hand policy, using domain randomization, sensor noise, and curriculum finetuning.Randomization covers robot parameters, cube mass, and friction; the progressive curriculum increases pose-estimation noise and action regularization.Source: 1) In-Hand Cube Reorientation · p. 4
  • The LEAP-hand policy shows early signs of robust real-world in-hand cube reorientation.Table I evaluates consecutive successful rotations before failure over 10 trials, reporting the median and mean of the #Rotations metric.Source: 1) In-Hand Cube Reorientation · p. 4, Table I · p. 5
  • Cube wedging between the fingers and palm is the most frequent in-hand failure; hardware flex also occasionally causes index–thumb interlocking.The authors suggest improved camera coverage and more accurate collision geometries could mitigate these failures, leaving these changes for future work.Source: 1) In-Hand Cube Reorientation · pp. 4–5

2) Non-Prehensile Block Reorientation:

  • 200 Hz direct torque control of seven arm joints produces smooth, compliant behavior that transfers effectively to hardware.The policy outputs motor torques rather than joint positions, with the gripper closed.Source: 2) Non-Prehensile Block Reorientation · p. 5
  • The non-prehensile policy reliably reorients a yoga block on hardware without additional fine-tuning and remains resilient to real-world perturbations.A Franka Emika Panda arm with a Robotiq gripper moves the block from random initial poses to a fixed goal. Success requires position error within 3 cm and orientation error within 10°.Source: 2) Non-Prehensile Block Reorientation · p. 5
  • 10 minutes on 16x A100 devices trains the block policy using stochastic delays, randomized initial conditions, and a success-driven curriculum.Training randomizes initial poses, joint positions, and velocities, while action and observation delays model hardware latency. Each success increases the block’s displacement and orientation range to prevent overfitting to easier conditions.Source: 2) Non-Prehensile Block Reorientation · p. 5

3) Pick-Cube from Pixels: · D. Training Throughput · 1) Training Throughput with Batch Rendering:

A pixel-based Franka cube-picking policy achieves zero-shot sim-to-real transfer after ten minutes of simulation training. GPU topology affects training time, while CNN updates become the bottleneck in batch-rendered pixel-policy training.

3) Pick-Cube from Pixels:

  • 64 × 64 RGB observations drive Cartesian commands through closed-form inverse kinematics for a Franka policy restricted to Y-Z movement and binary gripper control.The task is to grasp a randomly positioned 2 × 2 × 3 cm block and lift it 10 cm above the table. The end-effector always points downward.Source: 3) Pick-Cube from Pixels · pp. 5–6
  • Ten minutes on a single RTX 4090 suffices to train the pixel-based cube-picking policy in simulation using visual randomization and stochastic gripping delays.Randomization covers lighting, shadows, camera pose, object colors, brightness, and gripping delays up to 250 ms. Three-dimensional actions improve sample efficiency; full Cartesian or joint-space control also works with additional camera perspectives and training samples.Source: 3) Pick-Cube from Pixels · p. 6
  • A pixel-based policy achieved 100% success in 12 real-world trials, grasping a block and lifting it clear of the table, demonstrating reliable zero-shot transfer to real hardware.The policy was resilient to moderate variations in lighting and minor camera shaking.Source: 3) Pick-Cube from Pixels · p. 6

D. Training Throughput

  • Approximately 2080 s versus 670 s is required for LeapCubeReorient training on 1x RTX 4090 versus 8x H100, respectively, with fixed hyperparameters.All runs use 8192 environments; topology-specific hyperparameter tuning remains future work.Source: Figure 6 · p. 6, D. Training Throughput · p. 6

1) Training Throughput with Batch Rendering:

  • 403,000 and 37,000 steps per second are achieved approximately by Cartpole and Franka, respectively, using MJX physics with Madrona batch rendering.These rates cover physics and rendering, not policy inference or updates. Franka physics costs over 20x more than Cartpole physics, reducing throughput sensitivity to image resolution.Source: Figure 7 · p. 6, 1) Training Throughput with Batch… · p. 6
  • 9% and 43% of total PPO training time covers physics, rendering, and inference for Cartpole and Franka, respectively; CNN updates consume most training time.The bottleneck therefore shifts from collecting data to processing it.Source: 1) Training Throughput with Batch… · pp. 6–7

2) RL Libraries: … VII. CONCLUSION

MuJoCo Playground supports JAX- and PyTorch-based reinforcement learning and sim-to-real locomotion and manipulation from proprioceptive states and pixels. Its open-source simulation and batch-rendering stack enables direct vision-based training, while JAX compilation, contact scaling, and early-stage vision training remain limitations.

2) RL Libraries:

  • Brax and RSL-RL PPO agents achieve successful rewards and gaits within similar wall-clock times, using implementation-specific hyperparameters.All other results in the paper use Brax PPO and SAC implementations.Source: 2) RL Libraries · p. 7

V. RELATED WORK

  • Few robot environments [52] [66] use open-source GPU physics engines, unlike the broad sim-to-real results obtained with IsaacGym [39] and Isaac Lab [41].PhysX GPU is closed-source, preventing researchers from extending it for specific workloads. Genesis supports dynamic constraints and contacts, but its sim-to-real results remain limited to a few locomotion policies.Source: §V · p. 7
  • Zero-shot sim-to-real deployment using MuJoCo is demonstrated on Leap Hand, Unitree Go1, Berkeley Humanoid, Unitree G1, Booster T1, and Franka arm.Environment and training code are provided, similar to [39] [41].Source: §V · p. 7
  • Madrona’s high-throughput batch rendering enables vision-based policies to train directly in simulation without distillation, similar to [41] and [60].Visual sim-to-real gaps are often addressed through domain randomization, but these training methods require exponentially more samples.Source: §V · p. 7

VI. LIMITATIONS

  • 1-3 minutes of JIT compilation can delay Playground tasks; contact computation scales with possible rather than active contacts because JAX requires static shapes.More flexible frameworks such as Warp and Taichi could overcome contact-scaling limitations; this upgrade is under development. Madrona-based vision training remains at an early stage.Source: §VI · p. 7

VII. CONCLUSION

  • MuJoCo Playground combines open-source MuJoCo and Madrona batch rendering, demonstrating sim-to-real locomotion, dexterous manipulation, and non-prehensile manipulation from proprioceptive states and pixels.Policy training is demonstrated across various GPU topologies using JAX- and PyTorch-based reinforcement learning libraries.Source: §VII · p. 7

Appendix … B.22. Domain Randomization:

The appendices document DM Control Suite environments reimplemented in MuJoCo Playground and their PPO [51] and SAC [15] evaluations. Locomotion details cover environments, observations, actions, and training randomization intended to reduce the sim-to-real gap.

A.1. Environments

  • DM Control Suite ([61]) environments are reimplemented in MuJoCo Playground, with selected XML modifications to solver iterations, line search iterations, and contact parameters.Table III identifies the ported environments and modifications made for performance.Source: §A.1 · p. 12, Table III · p. 13

A.2. RL Training Results

A.3. RL Training Throughput

  • 95th percentile confidence intervals accompany Table IV’s brax PPO throughput, measured as environment steps divided by wallclock time.The table covers all DM Control Suite environments on an A100 GPU across 5 seeds, using the RL hyperparameters in Appendix Section E.Source: §A.3 · p. 12, Table IV · p. 16

B.1. Environment

B.21. Observation and Action:

  • Locomotion environments share observations comprising body-frame gravity, base linear and angular velocity, joint positions and velocities, previous action, and optional joystick user commands.Source: §B.21 · p. 17
  • Humanoid locomotion observations include sine and cosine of each foot’s gait phase, providing a smooth encoding for synchronizing actions with the desired gait cycle.Each foot’s phase cycles between −π and π.Source: §B.21 · p. 17
  • Joystick tasks use absolute joint-position actions with a default offset; other tasks use relative joint positions, and a PD controller maps desired positions to torque.Source: §B.21 · p. 17

B.22. Domain Randomization:

  • Training randomization targets the sim-to-real gap through noise on all sensor readings and variation in difficult-to-measure dynamics, ground friction, and payload mass.Randomized dynamic properties include link center-of-mass, reflected inertia, and joint calibration offsets.Source: §B.22 · p. 17

B.23. Reward and Termination: … C.1. Environments

Go1JoystickFlatTerrain training wallclock time differs little across GPU devices and topologies. The sections describe locomotion rewards, asymmetric actor–critic [46] networks, finetuning, evaluation and deployment, and catalogue manipulation environments by robot platform.

B.23. Reward and Termination:

  • Total reward is a weighted sum of reward terms, clipped to remain non-negative.Joystick policies use feet-only collisions and terminate upon inversion. Other tasks use full collision models approximated with geometric primitives.Source: §B.23 · p. 17

B.24. Network Architecture:

  • 512, 256, and 128 are the hidden-layer sizes of both actor and critic MLPs, which use Swish activations.The asymmetric actor–critic [46] gives the critic additional uncorrupted observations and sensor readings, including contact forces, perturbation forces, and joint torques.Source: §B.24 · p. 18

B.25. Finetuning:

  • 200 M flat-terrain timesteps precede 100 M rough-terrain finetuning timesteps for all policies, with Perlin-noise heightfields defining rough terrain.The joystick-specific schedule lists 100 M timesteps with command range {1.5, 0.8, 1.2}, 50 M with {1.5, 0.8, 2π}, then 100 M on rough terrain.Source: §B.25 · p. 18
  • 400 W is the getup training power cutoff; handstand and footstand finetuning progressively reduces the power termination budget from 400 W to 200 W.Getup finetuning adds a joint velocity cost, while handstand and footstand finetuning adds joint acceleration and energy costs.Source: §B.25 · p. 18

B.3. RL Training Results

  • 5 seeds on a single A100 GPU underpin locomotion PPO evaluation using the implementation from, with reward plotted against environment steps and wallclock time.The initial flat region in the wallclock curves measures compilation time for the training and environment code.Source: §B.3 · p. 18, Figure 11 · p. 19, Figure 12 · p. 20

B.4. RL Training Throughput

  • Go1JoystickFlatTerrain training wallclock time shows no material difference across devices and topologies because the environment is simple, with limited foot–floor contacts.Table VII reports training throughput across locomotion environments.Source: §B.4 · p. 18, Figure 13 · p. 21, Table VII · p. 21

B.5. Real-world Setup

  • 50 Hz policy inference uses ONNX Runtime, while a separate model-based estimator runs at the hardware interface’s maximal communication frequency of 500–2000 Hz.Locomotion deployments use C++ and ros2-control with real-time guarantees. Abstract sensor and actuator interfaces also support Gazebo sim-to-sim verification.Source: §B.5 · p. 22

C.1. Environments

  • Table VIII catalogues MuJoCo Playground’s implemented manipulation environments by robot platform.The supplied caption identifies this organizational scope but provides no individual platform or environment entries.Source: Table VIII · p. 23

C.2. RL Training Results … C.43. Real Robot Setup:

Manipulation PPO training is evaluated through reward curves and throughput measurements. LEAP Hand cube reorientation combines randomized simulation, actuator system identification, and hardware-matched deployment.

C.2. RL Training Results

  • 5 seeds on a single A100 GPU are used for manipulation PPO training, with rewards reported against environment steps and wall-clock time.The supplied passages describe evaluation settings but provide no numerical reward comparisons.Source: Figure 14 · p. 23, §C.2 · p. 23, Figure 15 · p. 24

C.3. RL Training Throughput

  • 5 seeds on an A100 underpin manipulation-throughput measurements; LEAP reorientation training is additionally evaluated across GPU devices and topologies.Table IX reports 95th percentile confidence intervals using brax PPO and Section E hyperparameters. Figure 15’s initial flat region measures compilation time for the training and environment code.Source: Table IX · p. 23, §C.3 · p. 23, Figure 15 · p. 24

C.4. Real-world Cube Reorientation with a Leap Hand

  • 0.1 rad in simulation versus 0.4 rad on hardware defines cube-reorientation success, with successive targets sampled at least 90° apart [19] [32].The LEAP Hand implementation includes policy training, a hardware interface, and camera-based object pose estimation. The cube starts randomly above the palm; targets are resampled until it drops or the hand remains stuck for over 30 s.Source: §C.41 · p. 26, §C.4 · p. 26

C.41. Simulation Environment:

  • 200 M training steps precede 100 M fine-tuning steps that introduce 0.1-probability random pose injections and hardware-matched torque limits.Training randomizes friction, cube mass, joint offsets, motor friction, reflected inertia, PD gains, link masses, and sensor noise, adding 2 cm positional and 0.1 rad rotational cube-pose noise. Initial training omits injections and torque limits; subsequent injections mimic pose-estimation failures such as occlusions.Source: §C.41 · p. 26
  • 16 relative joint-position actions accompany an asymmetric actor–critic setup whose actor observes noisy hand positions and velocities, cube-pose estimates, joint-position errors, and previous commands.The critic additionally receives uncorrupted robot pose and velocity, fingertip positions, cube pose and velocity, and perturbation forces.Source: §C.41 · p. 26

C.42. System Identification and Domain Randomization:

  • Actuator system identification addresses reliance on manual tuning in the original LEAP Hand environment, estimating reflected rotor inertia and setting nominal friction.Reflected inertia uses I_a = k_g^2 I_r, with gear ratio k_g = 288.35 and rotor inertia I_r = 1.7 × 10^-8 kg m^2, estimated from disassembly assuming uniform rotor mass distribution. Nominal friction is 10% of maximum torque, with heavy domain randomization covering uncertainty.Source: §C.42 · p. 26
  • DYNAMIXEL actuators lack direct motor-current feedback, so training randomizes k_p and k_d to mitigate deviations from ideal PD control.Training uses PD control, while deployment reconciles different motor units through data-sheet mappings and the assumed torque–current relationship tau = k_t i.Source: §C.42 · p. 26

C.43. Real Robot Setup:

  • 20 Hz replaces 150 Hz in simulation and deployment because of USB-driver jitter, while training torque limits are matched to hardware.Training also uses system-identification-guided stiffness and damping settings.Source: §C.43 · p. 26, §C.41 · p. 26

C.44. Vision-based Pose Estimator: … C.52. Real Robot Setup:

These sections describe vision-based cube-pose estimation from [19] and zero-shot transfer of a block-reorientation policy to a Franka–Robotiq robot. Block training combines randomized simulation and a progressive curriculum, while deployment uses synchronized 200 Hz torque control.

C.44. Vision-based Pose Estimator:

  • 15 Hz cube-pose estimation from [19] uses one camera, despite three being available, to supply the policy with a hand-to-cube transformation.Cube coordinates and detected image keypoints determine the camera-to-world transformation; camera-to-hand calibration then yields the hand-to-cube transformation. Estimation is generally stable despite small jitters or missed detections, and an equivalent SE(3) camera-to-object estimator could substitute.Source: §C.44 · p. 26

C.5. Real-world Non-prehensile Block Reorientation with a Franka-Robotiq Arm

  • A simulation-trained policy transfers zero-shot to a Franka Emika Panda with a Robotiq gripper for reliable, non-prehensile yoga-block manipulation.The task reorients the block without grasping it.Source: §C.5 · p. 27

C.51. Simulation Environment:

  • A closed-gripper policy uses 7D joint torques to push, slide, or tap a randomly initialized yoga block toward a central goal pose.Observations include noisy block-pose estimates, current and recent joint positions and velocities, estimated end-effector pose, and target block pose. Episodes terminate when the block exits the workspace or the end-effector violates safety constraints.Source: §C.51 · p. 27
  • A progressive curriculum expands block displacement and orientation ranges while randomizing initial joint positions, block poses, and action and observation delays.Actions are delayed by 1–3 steps and observations by 6–12 steps to emulate hardware communication and sensing latencies.Source: §C.51 · p. 27
  • Shaped rewards encourage nominal joint configurations, low velocities, end-effector proximity to the block, movement toward the goal, and alignment with the desired angle.These objectives jointly shape robot motion and the block’s target position and orientation.Source: §C.51 · p. 27
  • Ten minutes of training on 16 NVIDIA A100 GPUs produces the policy, with the final checkpoint selected by highest simulated success rate.Episodes contain 3000 steps; action repeat of 4 yields 750 policy decisions per episode. Training heavily randomizes block poses, robot states, and delays.Source: §C.51 · p. 27

C.52. Real Robot Setup:

  • 200 Hz raw joint-torque commands through Franka FCI bypass position and velocity tracking, reducing gain tuning and letting the policy directly control contact forces.The deployed Panda carries a Robotiq 2F-85 gripper and FT-300 force-torque sensor; gravity compensation handles arm weight. Software torque limits, built-in safety stops, and collision thresholds constrain operation, with the policy reported to remain well within these limits.Source: §C.52 · pp. 27–28
  • 200 Hz synchronization between simulation and deployment keeps real-system delays consistent with training and avoids aliasing or missed steps.At each 5 ms tick, a Python node runs the JIT-compiled policy using joint states and estimated block pose received through ROS. It publishes a 7D torque vector to a lightweight C++/ROS node that invokes Franka FCI.Source: §C.52 · p. 28

C.53. Camera-based Block Pose Estimation: … D.3. Bottlenecks in Pixels-based Training

Torque-controlled policies transfer zero-shot to real hardware, and single-camera pixel-based policies support real-world PickCube deployment. Madrona batch rendering provides competitive throughput in rough simulator comparisons, with rendering improvements driving speed-ups and policy updates becoming the training bottleneck.

C.53. Camera-based Block Pose Estimation:

  • Torque-controlled policies trained with domain randomization transfer zero-shot to real hardware, although blocks can leave the workspace and torque penalties mitigate force-limit pauses.Four OpenCV-calibrated RGB cameras—two approximately 40 cm above the table and two overhead around 80 cm—track face-mounted Alvar markers, uniformly averaging valid detections into a 6D pose. ROS fuses timestamped estimates and broadcasts geometry_msgs/PoseStamped at about 30–60 Hz; torque penalties preserve similar behavior with reduced force, and dynamic alignment requires minimal engineering.Source: §C.53 · p. 28

C.6. Real-world Franka PickCube from Pixels

  • Single-camera PickCube deployment restricts object randomization and Cartesian actions to the Y–Z plane, using a lightweight CNN and binary gripper commands.The 2x2x3 cm object starts within a 20 cm Y-axis range; success requires lifting to 17 cm in simulation versus a stable real grasp and at least 10 cm clearance, with white tape providing a progress reference. Actor and critic use two 256-unit dense layers after their CNNs and per-image channel normalization; Cartesian increments use inverse kinematics.Source: §C.6 · p. 30
  • PickCube training combines visual randomization, delayed gripper actions, and reward shaping similar to [45] to support real-world transfer.Randomized lighting, colors, brightness, and camera pose, plus 0–5-step gripper delays of up to 0.25 s, encourage conservative grasping important for transfer. Progress-only dense rewards complement sparse lifting and target rewards; episodes terminate upon completion, and only finger–cube collisions remain enabled to increase throughput.Source: §C.6 · p. 30, Figure 19 · p. 30
  • Within ten minutes on a single RTX 4090, deployed PickCube policies are trained for real-world transfer, with deployment inference running at 15 Hz.Deployment uses a Franka Research arm, Intel D435 Realsense camera, and RTX 3090; a C++/ROS ONNX pipeline processes 64x64 images and uses the training-matched IK solution. Roughly calibrating physical Cartesian increment scale to the training setup substantially improves transfer.Source: §C.6 · p. 30

D.1. RL Training Results

  • 5-seed brax PPO experiments cover CartPoleBalance and PandaPickCubeCartesian pixel environments, reporting reward against environment steps and wall-clock time on a single RTX 4090.CartPoleBalance stacks three grayscale frames into 64x64x3 observations and lowers control frequency with retuned training hyperparameters. Both batch-rendered environments include complete training examples.Source: Figure 21 · p. 32, §D.1 · p. 32, Figure 22 · p. 33

D.2. Performance Benchmarking

  • Madrona MJX batch rendering provides competitive throughput against IsaacLab [41] and Maniskill3 [60] in rough comparisons matching hardware and timestep size.Throughput measures random-action environment transitions using state observations or rendered pixels at increasing resolutions. Inherent simulator differences prevent a fully controlled comparison.Source: §D.2 · p. 32, Figure 23 · p. 34
  • Madrona’s rendering improvements appear to drive measured speed-ups primarily, while faster MJX physics also benefits throughput at lower image resolutions.The time-cost breakdown separates physics from non-overlapping rendering costs and reports inverse frequency, so lower values indicate better performance.Source: §D.2 · p. 32, Figure 24 · p. 35

D.3. Bottlenecks in Pixels-based Training

  • Over 90% of Cartpole’s training burden falls on network updates, while policy updates rather than rendering bottleneck both pixel-based environments.The breakdown isolates physics, rendering, inference, and amortized policy-update costs by differencing successive execution settings. Panda uses 20 Hz control with a 5 ms physics timestep—one decision per 10 simulator sub-steps—but convolutional image processing remains more costly than rendering.Source: Table XI · p. 36, §D.3 · p. 36, Table X · p. 36

APPENDIX E REINFORCEMENT LEARNING HYPER-PARAMETERS … E.3. Manipulation

Appendix E documents reinforcement-learning hyperparameters for MuJoCo Playground environments, covering the DM Control Suite, locomotion, and manipulation. The tables distinguish algorithms, default configurations, and task-specific or vision-based settings.

APPENDIX E REINFORCEMENT LEARNING HYPER-PARAMETERS

  • Appendix E reports the hyperparameters used to train reinforcement-learning policies for all MuJoCo Playground environments.Source: Appendix E · p. 37

E.1. DM Control Suite

E.2. Locomotion

  • Locomotion provides default Brax PPO hyperparameters and task-specific configurations for Go1 behaviors and humanoid control.Dedicated configurations include Go1 terrain control, handstands and footstands, G1Joystick, T1Joystick, and Berkeley Humanoid.Source: Table XV · p. 38, Table XVI · p. 38, Table XVII · p. 38

E.3. Manipulation

Related papers

Loading 2502.08844v1…