PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Rubik's Cube with a Robot Hand

OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, Lei Zhang

arXiv:1910.07113v1cs.LGcs.AIcs.CVcs.ROstat.ML

TL;DR

The paper asks whether simulation-trained models can solve highly complex manipulation on a real robot despite the sim2real gap. It introduces automatic domain randomization and a machine-learning-oriented robot platform, and demonstrates Rubik’s cube solving with a humanoid hand while observing emergent meta-learning and deployment-time adaptation.

  • Problem

    Complex real-world manipulation requires accurate control and state estimation, but collecting sufficient physical training data is difficult and simulation does not exactly match reality.

  • Method

    Automatic domain randomization generates increasingly difficult randomized environments for training control policies and vision state estimators, combined with a custom robot platform.

  • Results

    The combination enables a real humanoid robot hand to solve Rubik’s cubes, while ADR-trained recurrent policies adapt at deployment through updates to their recurrent state.

  • Takeaways & Limitations

    ADR improves sim2real transfer over manual domain randomization for vision and control and is associated with clear signs of emergent meta-learning.

  • Takeaways & Limitations

    The policy appears more likely to drop the cube after prolonged difficult rotations or early in trials, possibly before its recurrent state captures needed information.

Abstract

from arXiv · show

We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: https://openai.com/blog/solving-rubiks-cube/

1 Introduction

Humanoid robots remain difficult to use for complex real-world tasks, while simulation offers scalable training but creates a sim2real transfer problem. The paper addresses this by introducing automatic domain randomization (ADR) and a robot platform designed for machine learning, enabling control and state estimation from simulated data.

  • The system combines recurrent reinforcement learning for control with a separately trained CNN for cube pose and face-angle estimation.The vision estimator and policy are trained from randomized simulations and used together during real-world transfer.
  • Simulation can provide training data at scale, but inaccuracies in the robot and environment create a sim2real transfer problem.Domain randomization has shown that simulation-trained models can transfer to real robots.
  • The paper targets Rubik’s cube manipulation with a Shadow Dexterous Hand using only simulated data for control and state estimation.The task requires greater dexterity, precision, and cube-state accuracy than prior block manipulation.
  • ADR automatically generates a distribution over randomized environments for training reinforcement-learning policies and vision state estimators.The approach replaces substantial manual tuning of domain randomization with an automated process.
  • Control policies trained with ADR show clear signs of emergent test-time learning in their recurrent internal state.The authors interpret training an LSTM over an ADR distribution as implicit meta-learning.

2 Tasks

The paper studies block reorientation and the substantially harder task of solving a Rubik’s cube with a single Shadow Dexterous Hand. Cube solving is framed as a sensing-and-control problem built from sequential flips and face rotations.

  • The study considers block reorientation and Rubik’s cube solving, both performed with a single Shadow Dexterous Hand.The block task extends prior work, while cube solving introduces a more complex manipulation challenge.
  • Rubik’s Cube: A Rubik’s cube has six internal degrees of freedom and 26 connected cubelets, with each face independently rotatable.The cube is solved when all six faces return to a single color.
  • Rubik’s Cube: The cube task uses two sequential subgoals: rotating a face by 90 degrees and flipping another face to the top.Top-face rotations are simpler, so arbitrary face rotations are decomposed into a flip followed by a top-face rotation.
  • Rubik’s Cube: A fair scramble typically applies around 20 moves to a solved cube before testing the robot.The official World Cube Association scrambling method is used.
  • Rubik’s Cube: Existing software computes the solution sequence, leaving the core problem as estimating cube pose and face angles and controlling the hand.The robot must successfully execute each generated subgoal.

3 Physical Setup

The physical platform combines a Shadow Dexterous Hand, motion capture, RGB cameras, and a sensorized Rubik’s cube. Hardware and software modifications improve reliability, grip, calibration, and face-angle tracking for robot experiments.

  • The platform uses a Shadow Dexterous Hand, PhaseSpace tracking for five fingertips, and three RGB cameras for cube pose estimation.The system is housed in a contained, movable cage that simplifies calibration and protects setup reliability.
  • The hand was modified with internally routed sensor cables, more robust components, and larger rubber fingertip surfaces.These changes reduce cable wear, improve component reliability, and increase grip during object interaction.
  • Motor torque limits were manually minimized to avoid physical breakage while preserving reliable commanded positions.Real-time monitoring was also added to identify and resolve physical-setup issues more quickly.
  • The sensorized cube tracks face rotations in real time through custom firmware and a Bluetooth client application.Angle references can be reset through calibration procedures before trials.
  • 12.5 Hz angle updates provide low-latency observations to the robot policy.Across two custom cubes, encoder measurements had a 5.90° absolute mean tracking error and 7.61° standard deviation at reference points.

4 Simulation

The simulation uses MuJoCo models of the hand and Rubik’s cube, with calibrated hand dynamics and contact-based cube physics. The hand model was refined to reproduce coupled-finger behavior, while the cube model emphasizes plausible dynamics for domain randomization.

  • MuJoCo simulates the physical system, while ORRB and Unity3D render synthetic images for vision-estimator training.The setup supports separate physics simulation and rendered-scene generation.
  • The Shadow Dexterous Hand model was improved through dynamics calibration and modeling of physical tendons.These changes aim to better match the physical robot before domain randomization.
  • The original simulation produced visibly different joint positions and incorrect coupled-joint dynamics compared with the physical hand.On the real robot, coupled-joint movement depends on the current position of each joint.
  • A non-actuated spatial tendon and pulleys were added to non-thumb fingers, and parameters were calibrated by minimizing joint-position root mean square error.The authors report that improved coupling and dynamics calibration significantly improve performance.
  • The Rubik’s cube model contains 26 rigid convex cubelets with hinge joints representing face rotation and cubelet orientation.The model visualizes the cubelets, axes, and degrees of freedom in MuJoCo.
  • Cube motion is driven by cubelet shapes, contact forces, and friction rather than custom physics.Edges were beveled because a perfectly cubic model was too unforgiving, and calibration sought a plausible starting point rather than an exact physical match.

5 Automatic Domain Randomization

Automatic domain randomization (ADR) replaces manually fixed randomization ranges with an automatically expanding curriculum over environments. It generates training data and adjusts environment distributions according to model performance for both policies and vision estimators.

  • Motivation: ADR automates the process of expanding randomized simulation environments that previously required manual tuning and repeated robot validation.The method applies to both policy and vision training.
  • Overview: ADR samples environments to generate training data, evaluates model performance, and uses the evaluation to update the environment distribution automatically.The same framework supports policy reinforcement learning and supervised vision training.
  • Curriculum: ADR starts from a single environment and expands the distribution when model performance exceeds configured thresholds.The expansion continues while performance remains acceptable.
  • Benefits: ADR simplifies training through a progressively harder curriculum and removes the need to manually tune increasingly complex randomization ranges.These are identified as ADR’s two key benefits over manual domain randomization.
  • Algorithm: Each environment is parameterized by λ, sampled from Pφ, while ADR dynamically changes φ during training rather than keeping the distribution fixed.The factorized distribution uses lower and upper parameters for each randomized dimension.
  • Algorithm: At each iteration, ADR boundary-samples one environment dimension, records performance, and expands or contracts its boundary when averaged results cross thresholds.Training-data generation samples λ from the current distribution, while centralized storage decouples ADR from model optimization.

6 Policy Training in Simulation

The control policy is trained with reinforcement learning on randomized simulation environments to manipulate blocks and Rubik’s cubes. A recurrent, higher-capacity architecture and long-running distributed training support this approach.

  • Training setup: Control policies are trained with Proximal Policy Optimization and reinforcement learning on large distributions of randomized environments.ADR supplies the randomized training environments.
  • Task formulation: The action space uses 11 bins per actuated joint across 20 joints, with actions represented as relative changes in generalized joint positions.The policy receives rewards for progress and goal achievement and a penalty when the cube or block is dropped.
  • Task formulation: Episodes end after 50 consecutive successes, a dropped object, or timeout, with an 800-timestep limit for the Rubik’s cube task.Block reorientation uses a 400-timestep limit.
  • Architecture: The recurrent policy uses a 2048-unit feed-forward layer followed by a 1024-unit LSTM layer, while the separate value network shares the architecture.The recurrent design provides memory needed for meta-learning.
  • Distributed training: Rubik’s cube training used 64 NVIDIA V100 GPUs and 920 worker machines, continuing for several months while simulation, ADR, hyperparameters, and architecture evolved.The distributed setup uses Rapid for policy training.
  • Policy iteration: Behavioral cloning from teacher policies enabled architecture changes without discarding accumulated training progress, and produced the best ADR policies used in this work.Cloning minimizes differences in teacher and student action distributions and value predictions.

7 State Estimation from Vision

The vision system estimates cube pose and face angles from three camera images using synthetic training data and decomposed angle predictions. Domain randomization is important for transferring these estimators from simulation to reality.

  • State estimation: The policy requires cube position, orientation, and all six face-angle estimates, making state estimation substantially harder than pose prediction alone.Absolute face angles cannot be inferred from a single standard-cube image because of sticker rotational symmetry.
  • State estimation: The system uses either asymmetric center stickers for vision-only estimation or a sensor-equipped Giiker cube to avoid compounding vision angle errors during policy experiments.The modified cube breaks rotational symmetry and permits absolute face-angle estimation from one frame.
  • Limitation: The authors’ long-term goal of solving the task from vision alone with a standard cube remains unresolved, potentially requiring stronger recurrence or end-to-end training.A recurrent vision model was difficult to train to the required performance level.
  • Vision model: Three RGB cameras feed shared-weight ResNet50 backbones whose features are combined to predict cube position, orientation, and face-angle representations.The architecture is illustrated in Figure 14.
  • Vision model: Face-angle prediction is decomposed into an active axis, two active face angles modulo π/2, and the absolute angle of the visible top face.Post-processing tracks all face angles and uses the top-face estimate to correct accumulated rotational offsets.
  • Vision model: 90-bin angle discretization per π radians performs better than direct angle regression.The vision model uses separate losses for pose and decomposed angle predictions.
  • Sim2real transfer: A model trained without rendering domain randomization can achieve low simulation errors but fail dramatically on real-world data.The vision model is trained entirely from synthetic images, so sim2real transfer remains central to the setup.

8 Results

ADR improves simulation-to-simulation and simulation-to-real transfer for control policies and vision models, culminating in Rubik’s cube manipulation on a real Shadow hand. Higher ADR entropy generally corresponds to stronger transfer, while the best policy achieves substantial real-robot cube-solving success.

  • 8.1 Effect of ADR on Transfer: ADR policies transfer increasingly well to manually randomized environments as ADR entropy increases.The sim2sim transfer performance rises with ADR randomization entropy.
  • 8.1.2 Sim2Real: Higher ADR entropy corresponds to increased sim2sim and sim2real transfer, whereas the comparable Manual DR policy performs well in simulation but fails on the robot.ADR automates the tuning process and produces broader training distributions than manual randomization.
  • 8.2 Curriculum: ADR achieves high sim2sim transfer more quickly than fixed-entropy runs, while larger fixed entropy takes longer to train.The authors hypothesize that sufficiently difficult tasks and randomization levels may make training from scratch infeasible.
  • 8.3 Effect of ADR on Vision Model Performance: ADR vision models achieve lower real-image errors than the manual-DR baseline at similar training time, with higher ADR entropy correlating with lower errors.Simulation errors increase as ADR generates harder synthetic tasks.
  • 8.4.1 Quantitative Results: 26.80 successes on average over 10 trials correspond to solving 15-face-rotation cubes 60% of the time and 26-face-rotation cubes 20% of the time.The full threshold is equivalent to solving the Rubik’s cube.
  • 8.4.2 Qualitative Results: The best policy recovers from incorrect face rotations, difficult grasps, and cube slips by adjusting its grasp and retrying, usually succeeding eventually.These behaviors were observed on the real robot during Rubik’s cube solving.

9 Signs of Meta-Learning

The experiments find that ADR-trained recurrent policies adapt internally to changing environment dynamics during deployment. Their hidden states accumulate task-relevant information, supporting emergent meta-learning without explicit supervision for storing physical parameters.

  • Response to Perturbations: After hidden-state resets, completion time spikes because the policy must re-identify the environment, then decreases as memory is rebuilt.The post-reset spike is smaller than at rollout start because the hand has already placed the cube in a beneficial configuration.
  • Response to Perturbations: After environment dynamics are resampled, completion time rises and declines as the policy adjusts, while failures initially become more likely.The policy can act according to outdated dynamics immediately after the change, causing failures before adaptation.
  • Response to Perturbations: Broken-joint perturbations trigger gradual reductions in failure probability, but performance remains below the intact broken-joint baseline and the normal baseline.The authors hypothesize that recurrent-state information or an information advantage may limit recovery after the perturbation.
  • Prediction Accuracy: Prediction accuracy rises from near-random initialization to over 80% for some environment parameters as rollouts progress.The policy is not explicitly trained to store semantically meaningful physical parameters in its recurrent states.
  • Information Gain: For cube size, the inferred parameter distribution converges in under 5.0 seconds with approximately 0.9 bits of information gain, then stabilizes at about 2.0 bits of entropy.Prediction accuracy varies across parameters; cube size exceeds 80%, whereas cube mass is reported at 50–60%.
  • ADR and Internal Representations: Prediction accuracy and hidden-state information are strongly correlated with ADR entropy across block-reorientation policies.Interpretability analysis also identifies recurrent activation patterns corresponding to high-level manipulation skills.

10 Related Work

Related work spans dexterous manipulation, real-robot learning, sim-to-real transfer, adaptive environment distributions, and meta-learning. The paper distinguishes its contribution by combining ADR, a difficult Rubik’s Cube task, and out-of-distribution sim2real evaluation.

  • Dexterous Manipulation: Dexterous manipulation research includes rolling, sliding, finger gaiting, tracking, pushing, re-grasping, pivoting, tilting, tumbling, tapping, and two-palm strategies.Closed-loop approaches can use sensor feedback but still require reasonable robot kinematic and dynamic models.
  • Robot Learning: Deep reinforcement learning has learned manipulation skills either directly on physical robots, through policy search, or with model-free algorithms.Alternative approaches use multiple robots, model-based learning, or human demonstrations to address data and sample-complexity challenges.
  • Simulation and Real-World Evaluation: Many complex dexterous manipulation methods, including prior Rubik’s Cube work, were evaluated only in simulation.Other methods learn directly on robots but often target simpler hands or tasks than the present setting.
  • Sim-to-Real Transfer: Sim-to-real approaches include domain adaptation, progressive networks, inverse-dynamics models, and domain randomization, with several methods requiring real data.Prior work also adapted randomized environment distributions using optimization, trajectories, or discriminators.
  • Meta-Learning: Meta-learning methods such as MAML, SNAIL, and RL2 exploit shared structure across environment distributions to identify and adapt to new cases.The paper’s novelty is the combination of automated curriculum generation, Rubik’s Cube manipulation, and completely out-of-distribution sim2real testing.
  • Task Context: Existing Rubik’s Cube-solving robots are built exclusively for that purpose and, according to the authors, cannot generalize to other manipulation tasks.This distinguishes the paper’s humanoid-hand setting from specialized cube-solving machines.

11 Conclusion

The conclusion introduces ADR as a sim2real-transfer algorithm and reports its combination with a custom robot platform for solving a Rubik’s Cube with a real humanoid hand. It also identifies emergent meta-learning in the recurrent policy state.

  • Contributions: ADR improves vision and control transfer over previously established manual-domain-randomization baselines.The conclusion presents ADR as a central algorithmic contribution for sim2real transfer.
  • Real-Robot Demonstration: ADR and the custom robot platform enable a real Shadow Dexterous Hand to solve a Rubik’s Cube, combining difficult manipulation and state estimation.The conclusion characterizes this as a manipulation problem of unprecedented complexity.
  • Emergent Meta-Learning: ADR-trained policies adapt at deployment time to unseen physical reality through updates to their recurrent state.The authors’ systematic policy analysis finds clear signs of emergent meta-learning.

Appendices

The appendices describe reliability monitoring, data persistence and visualization, and the parameterization of ADR randomizations. These components support consistent robot experiments and formalize how simulator environments are varied.

  • Physical Setup: A monitoring system records, visualizes, alerts on, and persists robot operations to support fully functional and consistent physical experiments.The system also enables real-time detection of robot breakage and debugging investigations.
  • Data Infrastructure: InfluxDB stores native-resolution sensor data together with detailed experimental context for real-time tracking and persistence.
  • Data Infrastructure: Grafana visualizes policy performance, robot sensor data, and robotic-component health using InfluxDB as a data source.
  • ADR Parameterization: ADR randomizations are parameterized through automatically changing distribution boundaries, with simple parameters using lower and upper bounds.More complex randomizations, such as observation noise, use several ADR parameters.
  • ADR Parameterization: Randomized environment parameters use λ_i-style variables and ADR boundary values, while simulator physics parameters are randomized with generic or custom randomizers.Generic randomizers are specified by a noise mode and scaling factor α.

B.1.2 Custom randomizers

The paper defines custom randomizers for physical and visual simulation parameters, then extends ADR with adversarial perturbations to broaden training environments. Random networks provide diverse perturbations, while ADR adjusts their influence automatically.

  • Custom randomizers: Custom randomizers vary cube and robot friction, cube size, joint and tendon limits, action delay, action latency, action noise, gravity, and other simulator parameters.The randomization framework covers both physics and action-level effects, with episode- or step-level noise sampling where specified.
  • ADR and adversarial perturbations: ADR trains on an ever-growing environment distribution and automatically chooses randomization ranges, but cannot randomize effects that are absent from the simulator.This limitation motivates additional adversarial perturbations for unmodeled target-domain effects.
  • ADR and adversarial perturbations: Adversarial perturbations mix robot-policy actions with adversarial actions and apply force or torque perturbations to manipulated bodies.The action mixture uses α to control adversarial influence; force and torque perturbations are applied per body, including each cubelet for the Rubik’s cube.
  • ADR and adversarial perturbations: ADR can automatically adjust perturbation strengths α and β over time and independently sample them across joints or bodies.This creates simulations in which different joints or bodies receive different levels of adversarial influence.
  • ADR and adversarial perturbations: Randomly initialized networks outperform more sophisticated adversarial-training methods for sim2sim transfer, which the authors attribute to greater perturbation diversity.The adversarial network is re-initialized at the beginning of each episode, producing varied perturbations across episodes.
  • Vision randomization: Vision ADR controls both rendered-scene variations and image-level TensorFlow distortions during vision-model training.These two randomization categories affect the appearance of images supplied to the model.

C Implementation Details

The implementation generates task goals and represents Rubik’s cube state through orientation and face-angle information. It also specifies policy, ADR, vision, and post-processing configurations for the two manipulation tasks.

  • Goal generation: Cube and block orientations are sampled with one face pointing upward, using 0.4 radians for orientation alignment and 0.1 radians for cube-face alignment.These tolerances determine when orientations and face angles count as aligned.
  • Goal generation: Rubik’s cube goals alternate between top-face rotations and cube flips after checking orientation and face-angle alignment.When aligned, a 50% branch performs a clockwise or counterclockwise top-face rotation; otherwise, the procedure generates a flip goal.
  • Training configuration: The implementation documents policy inputs, network parameter counts, PPO settings, ADR hyperparameters, and vision-training configurations.Separate tables specify task observations, network size, PPO, ADR, vision architecture, and target error thresholds.
  • Vision training: Vision training uses object-pose augmentation that leaves pose unchanged with 20% probability, rotates it with 40% probability, or jitters position and rotation with 40% probability.The jitter adds Gaussian noise independently to position and rotation.
  • Vision post-processing: Vision post-processing tracks six face angles using the predicted cube orientation, active axis, active face angles, and top-face angle.The procedure updates top and active faces while retaining base and tracked angles for the remaining faces.
  • Vision post-processing: MOVEANGLE maps a base face angle to a target using a modulo π/2 delta, keeping the adjustment within the nearest quarter-turn interval.The algorithm computes δ from the wrapped difference between target b and base a, then returns a + δ.

D Full Results

The full-results section evaluates simulation calibration, physical manipulation, vision prediction, and recurrent policy states. It reports performance tables and analyzes activation patterns associated with manipulation skills.

  • Calibration and physical evaluation: Simulation calibration is evaluated by comparing policies trained with old versus calibrated MuJoCo environments on a simplified face-cube task.The calibration includes parameters related to joint movement, coupling, and dynamics.
  • Calibration and physical evaluation: Physical evaluations report block-reorientation and Rubik’s-cube performance over 10 real-robot trials per policy.The reported measures include success counts, time per success, means with standard errors, medians, and full trial lists.
  • Vision evaluation: Vision models are evaluated on simulated and real data using orientation, position, face-angle, and active-axis prediction errors.Orientation uses quaternion rotational distance, position uses 3D Euclidean distance in millimeters, face angles use degrees, and active axis uses incorrect-label percentage.
  • Meta-learning analysis: Face-rotation meta-learning results are reported as comparable to the cube-flip results in the main body.The appendix includes these results for completeness and describes 10,000 simulated trials with up to 50 face rotations.
  • Meta-learning analysis: A 1024-unit LSTM state is analyzed across 1500 rollout time steps containing successful re-orientations and face rotations.The states are arranged as a 1024-by-1500 matrix and reordered with 1D t-SNE before visualization.
  • Meta-learning analysis: Distinct LSTM activation patterns correspond to successful re-orientations, face rotations, and periods when the policy struggles with face rotations.The analysis reports different neuron-group activity before successful task events and extended activation during difficult rotations.
  • Meta-learning analysis: NMF coloring reveals four distinct neuron-activation groups, each associated with a sequence of complex joint actuations or skill.The yellow group, for example, corresponds to flipping the cube back toward the palm.
Loading 1910.07113v1…