Solving Rubik's Cube with a Robot Hand

Motivation

• Machine learning offers an alternative to hand-programming robots by learning to use sensor information to control complex systems appropriately.The motivation is the broader challenge of building robots as versatile as humans and deploying humanoid systems for complex real-world tasks.Source: §1 · pp. 1, 3

Contribution

• The system solves a Rubik’s Cube with a Shadow Dexterous Hand using only simulated data, addressing unusually demanding manipulation and high-accuracy state-estimation requirements.The approach combines ADR with a robot platform designed to complement the machine-learning system.Source: §1 · p. 3

Method

• ADR automatically generates an ever-growing distribution of randomized environments for training reinforcement-learning policies and vision-based state estimators, improving simulation-to-real transfer.Simulation is attractive because physical data collection is expensive, but inaccuracies in simulation create a sim2real transfer problem.Source: §1 · p. 3

Findings

• Policies trained with ADR show clear signs of emergent test-time learning within their recurrent internal state.The paper attributes this behavior to training a memory-augmented policy over an ever-growing ADR distribution and interprets it as implicit meta-learning.Source: §1 · p. 3

2 Tasks

• The paper uses one Shadow Dexterous Hand for both block reorientation and Rubik’s cube solving, as visualized in Figure 3.Source: §2 · p. 3, Figure 3 · p. 4

2.1 Block Reorientation

• Block reorientation rotates a block to a desired orientation, with success defined as matching the goal within 0.4 radians before a new random goal is generated.Source: §2.1 · p. 3

2.2 Rubik’s Cube

• Rubik’s cube solving is a significantly harder task involving a six-degree-of-freedom cube whose six faces must each return to a single color.Source: §2.2 · p. 3
• The hand performs cube solutions through sequential subgoals that combine flipping a face to the top with a 90-degree clockwise or counter-clockwise top-face rotation.Top-face rotation is simpler than rotating other faces, motivating this combined operation design.Source: §2.2 · p. 4
• The cube is evaluated from fair scrambles produced using the World Cube Association method, typically applying around 20 moves to a solved cube.Source: §2.2 · p. 4
• Because Kociemba supplies the solution sequence, the key challenge is sensing the cube’s pose and six face angles and controlling the hand to achieve each subgoal.Source: §2.2 · p. 4

3.1 Robot Platform

• The robot platform uses a Shadow Dexterous E Series Hand, PhaseSpace fingertip tracking, and three RGB Basler cameras, building on the configuration in.The contained, mobile cage simplifies calibration and reduces disturbances when the hand is removed or replaced.Source: §3.1 · pp. 4–5, §3 · p. 4
• Hand customizations improved reliability and grip, while pulley and tensioner changes extended finger-tendon MTBF by more than threefold and approximately five to ten times, respectively.Cable routing reduced wear, strengthened failure-prone components, and expanded fingertip rubber coverage.Source: §3.1 · p. 5, Figure 4 · p. 5
• Manual motor-torque tuning avoided physical breakage and supported consistent policy performance, complemented by real-time monitoring for faster diagnosis and resolution.Torque limits were minimized while retaining reliable achievement of commanded hand positions.Source: §3.1 · p. 5

3.2 Giiker Cube

• The Giiker cube provides Bluetooth state sensing while vision predictions mature, and its custom internals improve face-angle resolution from 90° to approximately 5°.Experiments used the cube to evaluate control without compounding vision face-angle errors, while still using vision for pose estimation.Source: §3.2 · p. 5, Figure 5 · p. 6

3.2.1 Design

• Six custom encoder boards, linearizing circuits, BLE firmware, and a Node.js client enable 5° face-angle sensing and real-time reconstruction of cube state from periodic readings.The client requests readings and calibration resets, tracking rotations from a solved-cube reference.Source: §3.2.1 · p. 6

3.2.2 Data Accuracy and Timing

• Custom cubes achieved 5.90° absolute mean tracking error and 7.61° standard deviation, while 12.5 Hz angle updates provided low-latency observations for the robot policy.Firmware and software resets dynamically establish zero positions and prevent sensor errors from accumulating across trials; periodic full-turn checks monitor degradation.Source: §3.2.2 · p. 7, §3.2.3 · p. 7

4 Simulation

• The simulation uses MuJoCo for physical dynamics and ORRB, a Unity3D-based remote renderer, to generate synthetic images for vision-based pose-estimator training.Source: §4 · p. 7
• The hand model was refined with dynamics calibration and tendon modeling, while the cube model was made accurate enough to bridge simulation and reality despite unavoidable mismatch.Source: §4 · p. 7

4.1 Hand Dynamics Calibration

• Calibrating coupled-joint dynamics with a non-actuated tendon and pulleys, then minimizing joint-position RMSE, significantly improves performance and defines the simulation used thereafter.The updated model reflects position-dependent coupling observed on the physical robot, including proximal joints bending before distal joints.Source: §4.1 · p. 7, Figure 7 · p. 8

4.2 Rubik’s Cube

• Because training occurs entirely in simulation, the cube model must capture complex contact, friction, locking, snapping, elasticity, and deformation-related behaviors while remaining computationally manageable.MuJoCo supplies stable and fast numerical simulation of body dynamics with soft contacts.Source: §4.2 · pp. 8–9
• The Rubik’s cube simulation models 26 rigid cubelets with 66 degrees of freedom, representing aligned configurations and physically valid intermediate states.Source: §4.2 · p. 9
• The cube model uses beveled meshes, native MuJoCo soft contacts, contact forces, and friction rather than custom physics or spring joints, which were found unnecessary.A rudimentary calibration provides a plausible starting point for domain randomization rather than an exact physical match.Source: §4.2 · p. 9

5.1 ADR Overview

• ADR trains both vision models and control policies by dynamically changing the distribution ranges used to randomize visual appearance, robot dynamics, and other environment properties.Unlike conventional domain randomization, ADR generates training data independently of the model-training algorithm and adapts its ranges throughout training.Source: §5.1 · p. 10, §5 · p. 10
• ADR progressively expands the environment distribution when model performance reaches predefined thresholds, creating a curriculum that eases training and removes manual randomization tuning.The method begins from a calibrated or otherwise concentrated environment and expands randomization only when performance remains acceptable.Source: §5.1 · pp. 10–11

5.2 Algorithm

• ADR tests boundary environments and increases or decreases selected parameter bounds according to averaged performance relative to high and low thresholds.Each environment parameter λ_i is sampled from a distribution P_φ whose parameters φ change dynamically during training.Source: §5.2 · p. 11

5.3 Distributed Implementation

• The distributed implementation decouples ADR from model optimization using centralized Redis storage and parallel workers that evaluate boundaries, generate training data, and update policy or vision models.Policy workers reuse rollout data for training, whereas vision training uses images produced by data producers rather than ADR evaluator data.Source: §5.3 · pp. 11–12

5.4 Randomizations

• The randomization scheme covers simulator and custom robot physics, adversarial unmodeled effects, policy observation noise, and rendered vision conditions such as lighting, cameras, materials, backgrounds, and post-processing.Most scalar parameters use lower and upper ADR boundaries, while coupled effects such as observation noise can require larger boundary-parameter sets.Source: §5.4 · pp. 12–13

6 Policy Training in Simulation

• Policies use PPO and reinforcement learning with ADR to train over a large distribution of randomized environments.This setup is similar to while applying ADR as described in Section 5.Source: §6 · p. 13

6.1 Actions, Rewards, and Goals

• The policy uses 20 joints with 11 discrete action bins each, relative joint-position changes, shaped rewards, success bonuses, drop penalties, and task-specific episode limits.Rewards include distance improvement, +5 for achieving a goal, and −20 for dropping a cube; episodes end after 50 consecutive successes, drops, or timeouts of 400 or 800 timesteps.Source: §6.1 · p. 14

6.2 Policy Architecture

• The recurrent policy retains a ReLU feed-forward layer and LSTM while doubling capacity to 2048 and 1024 units, using a separate value network and 10^-6 L2 regularization.An embed-and-add input design separately embeds observations into 512-dimensional latents, sums them, and applies ReLU, facilitating new observations and shared embeddings.Source: §6.2 · pp. 14–15

6.3 Distributed Training with Rapid

• Distributed training uses Rapid with 32 GPUs and 400 workers for block reorientation, versus 64 GPUs and 920 workers for Rubik’s Cube.Rubik’s Cube training accumulated roughly 13 thousand years of experience while simulation fidelity, ADR, hyperparameters, and architecture were continually improved.Source: §6.3 · p. 15

6.4 Policy Cloning

• Behavioral cloning initializes redesigned policies near teacher performance, preserving weeks or months of training progress while enabling arbitrary architecture changes with the same action space.Student rollouts minimize KL divergence between teacher and student action distributions and L2 loss between value predictions; the best ADR policies used this approach after multiple months of training.Source: §6.4 · p. 16

7 State Estimation from Vision

• The policy requires cube position, orientation, and all six absolute face angles, which cannot be recovered from one standard-cube camera frame because of sticker rotational symmetry.The system therefore uses either asymmetric center stickers for vision-only estimation or a Giiker cube’s sensors for face angles, with the latter used for most experiments.Source: §7 · p. 16
• Fully solving state estimation from vision alone on a standard cube remains an open goal that may require stronger recurrent models or end-to-end joint training with the policy.A recurrent vision model was difficult to train to the necessary performance level within the project’s time constraints.Source: §7 · p. 16, §7.1 · p. 16

7.1 Vision Model

• The three-camera model uses identically parameterized ResNet50 networks with common weights and decomposes face-angle prediction into active-axis, modulo-π/2 active-face, and absolute top-face predictions.Post-processing tracks all face angles, using stateless top-face estimates to correct accumulated multiples of π/2 when a face is placed on top.Source: §7.1 · pp. 16–17
• Without domain randomization, the vision model achieves low simulation errors but fails dramatically on real-world data.Table 2 reports ablations across simulation and physical trials, with real errors computed from PhaseSpace and Giiker ground truth.Source: §7.1 · p. 17, Table 2 · p. 18

7.2 Distributed Training with Rapid

• The model is trained entirely on synthetic images with Rapid’s distributed framework, using ADR-rendered data, focal loss weighting across prediction tasks, and LARS optimization.Rendering workers generate ADR-randomized images, while optimizer nodes perform data-parallel training; focal weighting dynamically balances the position, orientation, and angle losses.Source: §7.2 · pp. 17–19

8.1.1 Sim2Sim

• ADR improves sim2sim transfer as its randomization entropy increases, including near-perfect sim2sim performance after scaling ADR.ADR determines the training distribution, while manually randomized environments serve as a held-out sim2sim test distribution.Source: Figure 15 · p. 19, §8.1.1 · p. 19, §8.1.2 · p. 20

8.1.2 Sim2Real

• ADR policies show improved sim2real transfer as entropy per dimension increases, while manual domain randomization fails on the real robot.The largest ADR policies achieved the best sim2sim and sim2real transfer and significantly beat, increasing mean performance by almost 2× and median performance by more than 3×.Source: Table 3 · p. 20, §8.1.2 · pp. 20–21

8.2 Effect of Curriculum on Policy Training

• ADR’s adaptive curriculum achieves high sim2sim transfer much faster than fixed-randomization training, while larger fixed entropies take longer to learn.The authors hypothesize that sufficiently complex fixed environments may become infeasible because they provide insufficient reinforcement-learning signal.Source: Figure 16 · p. 21, §8.2 · p. 21

8.3 Effect of ADR on Vision Model Performance

• ADR-enhanced vision models reduce real-image prediction errors for block state estimation and Rubik’s cube state estimation compared with manual domain randomization.For Rubik’s cube prediction, higher ADR entropy correlates with lower real-image errors, although simulation errors increase as synthetic tasks become harder.Source: §8.3 · pp. 21–22, Table 4 · p. 22, Table 5 · p. 22

8.4 Solving the Rubik’s Cube

• The Rubik’s cube task combines uncalibrated simulated dynamics, difficult cube-state sensing, and vision or Giiker-based estimation of the cube’s six face angles.ADR automates environment randomization because the cube and its interactions with the hand cannot be accurately captured in simulation.Source: §8.4 · p. 22

8.4.1 Quantitative Results

• 26.80 successes on average enabled the best policy to solve Rubik’s cubes requiring 15 face rotations 60% of the time and 26 face rotations 20% of the time.The policy was continuously trained for multiple months at larger scale; 26 quarter face rotations represent the worst case for only about 3 Rubik’s cube configurations.Source: §8.4.1 · pp. 22–23, Table 6 · p. 23

8.4.2 Qualitative Results

• Under untrained perturbations including altered friction, tied fingers, occlusion, and poking, the policy still performs multiple face rotations and cube flips.These qualitative perturbation experiments were not quantified.Source: Figure 17 · p. 24, §8.4.2 · p. 24
• The best policy often recovers from incorrect rotations, slips, and difficult grasps by rearranging its grasp or restoring the face before continuing the original subgoal.It also aligns faces after flips to avoid interlocking, though it sometimes becomes stuck or drops the cube.Source: §8.4.2 · p. 24

9.1 Definition of Meta-Learning

• Meta-learning is defined as learning about an underlying Markov decision process by updating beliefs about transition dynamics from observed data, without explicit parameter supervision.In this work, the behavior emerges through the policy’s recurrent state and is also termed online system identification.Source: §9.1 · pp. 24–25

9.2 Response to Perturbations

• Policies progressively adapt to dynamics at test time: flip completion converges to approximately 4 seconds, improving roughly 1.6 seconds over the first flip, while perturbations trigger renewed adjustment.Resetting hidden state, resampling dynamics, and breaking a joint produce adaptation patterns; broken-joint perturbations increase failures and never fully match the broken-joint baseline.Source: §9.2 · pp. 25–27, Figure 18 · p. 26

9.3.1 Prediction Accuracy over Time

• Prediction accuracy for environment parameters rises from near-random at rollout start to over 80% for some parameters, showing that LSTM states store useful dynamics information without explicit training for it.Cube-size prediction exceeds 80%, whereas cube-mass prediction is 50−60%, with differences reflecting parameter importance across tasks.Source: §9.3.1 · pp. 27–28

9.3.2 Information Gain over Time

• Cube-size information gain reaches approximately 0.9 bits in below 5.0 seconds, after which predictor entropy converges to 2.0 bits as the policy stores only information needed for action.The convergence timing is consistent with the first flip taking roughly 5−6 seconds and subsequent flips becoming faster.Source: §9.3.2 · p. 28

9.3.3 Prediction Accuracy and ADR Entropy

• Prediction accuracy, and thus information stored in hidden states, is strongly correlated with ADR entropy when predicting cube size at 4.8 seconds of rollout time.The experiment compares four block-reorientation policies trained with increasing ADR entropy.Source: §9.3.3 · p. 28, Table 8 · p. 29

9.3.4 Recurrent state visualization

• Interpretability analysis reveals distinctive recurrent-state activation patterns corresponding to high-level robotic-hand skills during rollouts.The analysis uses neural-network interpretability techniques.Source: §9.3.4 · p. 29

10.1 Dexterous Manipulation

• Prior dexterous manipulation methods use planning or feedback, but planning requires accurate hand and object models while feedback-based methods still require robot dynamics and kinematics models.Planning methods typically execute trajectories open-loop, whereas closed-loop approaches can correct runtime errors but remain difficult for under-actuated, high-degree-of-freedom hands.Source: §10.1 · p. 29

10.2 Dexterous In-Hand Manipulation

• Most prior deep reinforcement-learning approaches for complex in-hand manipulation, tool use, and Rubik’s Cube solving were evaluated only in simulation.Direct robot learning avoids modeling the system but is constrained by limited data, generally restricting policies to simple behaviors.Source: §10.2 · p. 30

10.3 Sim to Real Transfer

• Sim-to-real methods include domain adaptation, dynamics randomization, adversarial training, and adaptive environment distributions, but earlier approaches generally assume real data or target narrower transfers.Prior work also used bi-level optimization, real-world trajectories, discrepancy metrics, discriminators, and adversarial networks to adapt randomized environments or vision distributions.Source: §10.3 · p. 30

10.4 Meta-Learning via Reinforcement Learning

• ADR combined with memory-augmented reinforcement learning, Rubik’s Cube solving, and an out-of-distribution sim-to-real test distinguishes this work from prior meta-learning approaches.The approach is based on RL2 [116], while its novelty lies in combining automated curriculum generation, a challenging task, and sim-to-real evaluation.Source: §10.4 · p. 30

10.5 Robotic Systems Solving Rubik’s Cube

• Existing Rubik’s Cube robots are purpose-built and cannot generalize to other manipulation tasks.This motivates the paper’s use of a humanoid robot hand rather than a system designed exclusively for cube solving.Source: §10.5 · p. 31

• ADR improves sim2real transfer over established manual domain randomization baselines for both vision and control.Source: §11 · p. 31
• Combined with a custom robot platform, ADR enables the Shadow Dexterous Hand to solve a Rubik’s cube as a manipulation task of unprecedented complexity.Source: §11 · p. 31
• ADR-trained policies show emergent meta-learning by adapting during deployment to physical reality unseen during training through recurrent-state updates.Source: §11 · p. 31

A System Monitoring

• The monitoring system recorded, visualized, alerted on, and persisted robot operations, enabling real-time breakage detection, setup regression discovery, and simulation-to-physical policy discrepancy identification.It was designed to ensure experiments used a functional robot and a consistent physical setup.Source: A System Monitoring · p. 38
• InfluxDB tracked and persisted native-resolution sensor data alongside detailed information about each experiment’s context.Source: A System Monitoring · p. 38
• Grafana displayed task policy performance, robot sensor data, and robotic-component health using InfluxDB as its data source.Source: A System Monitoring · p. 38

B Randomizations

• ADR randomizations for policy and vision training are listed in Tables 9 and 11, with their parameterization described in the appendix.Source: B Randomizations · p. 38
• Simple environment randomizations such as gravity use lower and upper ADR boundary parameters φL,i and φH,i, whereas complex randomizations such as observation noise use several parameters.Source: B Randomizations · p. 38
• Default values use subscript (·)0, randomized parameters use ADR boundary values φL and φH, and g(x) is defined as exp(x −1.0).Source: B Randomizations · p. 38

B.1 Simulator Physics Randomizations

• Simulator physics parameters are randomized through either generic or custom randomizers.Source: §B.1 · p. 38

B.1.1 Generic randomizers

• A generic randomizer is specified by a noise mode and scaling factor α, with x denoting the simulator physics parameter being randomized.The available noise modes are listed in Table 10.Source: §B.1.1 · p. 38

B.1.2 Custom randomizers

• Custom randomizers vary cube and robot friction, cube size, and lower and upper joint and tendon limits.Cube friction uses weights w = 1.0 for sliding and w = 2.0 for spinning and rolling, while robot-friction weights are 1.0.Source: §B.1.2 · pp. 38–39

B.2 Custom Physics Randomizations

• Custom physics randomizations model effects omitted by the simulator, including action delay, action latency, action noise, motor backlash [77, C.2], gravity, joint margin, and simulation time step.These models range from simple time delays to motor backlash.Source: §B.2 · pp. 39–40

B.3 Random Network Adversary (RNA)

• RNA addresses unmodeled target-domain effects by perturbing actions through policy mixtures and applying normalized force/torque perturbations to manipulated bodies.ADR automatically adjusts perturbation strengths α and β, which can also be sampled independently per joint or body.Source: §B.3 · pp. 40–41
• Random networks outperform more sophisticated adversarial methods for sim2sim transfer, which the authors attribute to their greater perturbation diversity.The random network is reinitialized each episode and held fixed throughout it.Source: §B.3 · p. 41

B.4 Observation Randomizations

• Observation randomization combines correlated episode-level noise with uncorrelated step-level noise, using multiplicative and additive Gaussian components.The noise scales are controlled by ADR parameters for each observation element.Source: §B.4 · p. 41

B.5 Visual Randomizations

• Vision training randomizes both rendered scene variations from ORRB [16] and TensorFlow image distortions, with ADR controlling their effects on final model inputs.The full set of parameter randomizations is provided in Table 11.Source: §B.5 · p. 41, Table 11 · p. 42

C.1 Goal Generation

• Rubik’s cube goal generation checks orientation and face alignment, then either rotates the top face with 50% probability or flips the cube toward aligned faces with a random upward orientation.Orientation alignment allows any upward-facing face within 0.4 radians, while cube-face alignment requires all six faces within 0.1 radians of a straight-angle configuration.Source: §C.1 · pp. 42–43
• Block reorientation goals sample a random upward orientation, while cube goals use either a top-face rotation or aligned-face configuration depending on the current aligned state.In both tasks, one face is sampled to point directly upward.Source: §C.1 · pp. 42–43

C.2 Neural Network Inputs

• The implementation defines separate network input specifications and parameter counts for the block reorientation and Rubik’s cube tasks.Angles are encoded with sine and cosine, doubling the dimensionality of the underlying angle.Source: §C.2 · p. 43, Table 12 · p. 43, Table 13 · p. 44

C.3 Hyperparameters

• Training configuration is documented through PPO, ADR, vision-training, architecture, and target-error hyperparameter tables.Vision ADR uses 70% and 30% increase and decrease thresholds, respectively.Source: §C.3 · p. 44, Table 14 · p. 44, Table 15 · p. 44
• Vision training augments object poses by leaving them unchanged with 20% probability, rotating them around main axes with 40% probability, or applying Gaussian position-and-rotation jitter with 40% probability.The augmentation choices are applied independently as the specified alternatives for vision training.Source: §C.3 · p. 44

C.4 Vision Post-Processing

• Vision post-processing tracks all six cube-face angles by combining predicted cube orientation, active-axis angles, top-face angles, and base-angle estimates.The procedure identifies the top face and, when aligned, replaces its angle with the predicted top angle; active faces are updated relative to their base angles.Source: §C.4 · p. 46
• The MOVEANGLE routine moves a base face angle to a target using the shortest modulo π/2 delta.It computes δ = mod(b − a + π/4, π/2) − π/4 and returns a + δ.Source: §C.4 · p. 46

D.1 Simulation Calibration

• Simulation calibration is evaluated by comparing policies trained with old versus dynamics- and coupling-calibrated simulations on a simplified two-degree-of-freedom cube task.The calibrated MuJoCo parameters include damping, joint ranges, geometry, tendons, and actuator properties; results are reported in Table 19.Source: §D.1 · p. 46, Table 19 · p. 46

D.2 Physical Trials

• Physical trials report performance for different policies on block reorientation and Rubik’s cube tasks using 10 real-robot trials per policy.Tables 20 and 21 report success statistics, time per success, and full success-count lists for the two tasks.Source: §D.2 · p. 46, Table 20 · p. 47, Table 21 · p. 47

D.3 Vision Model Performance

• The vision model predicts cube orientation, position, top angle, active axis, and active angles, with errors evaluated on both simulated and real data.Orientation uses quaternion rotational distance, position uses 3D Euclidean distance in millimeters, face angles use degrees, and active-axis error is the incorrect-label percentage; tiny active face angles largely explain its high error.Source: §D.3 · p. 47

D.4 Meta-Learning

• Meta-learning results for face rotations are presented in Figure 21 and are reported as comparable to the cube-flip results in the main paper.The figure evaluates average completion time and failure probability over 10,000 simulated trials through 50 rotations under baseline, broken-joint, hidden-state-reset, dynamics-resampling, and joint-breaking conditions.Source: §D.4 · p. 47, Figure 21 · p. 49

E.1 Policy Recurrent States

• The recurrent-state analysis uses a 1,500-step simulated rollout containing successful cube reorientations and face rotations, recording 1,024-unit LSTM hidden and cell states with observations and actions.The states are organized as 1,024-by-1,500 matrices for visualization and analysis.Source: §E.1 · p. 50
• t-SNE-reordered recurrent activations distinguish neuron-group patterns preceding successful block reorientations and face rotations, while prolonged activity marks difficult face rotations.Dashed and solid markers identify successful reorientations and rotations, respectively, in Figure 22(a).Source: §E.1 · p. 50, Figure 22 · p. 51
• NMF coloring reveals four distinct activation groups whose timing aligns with sequences of complex joint actuations, indicating identifiable skills in the cell-state representation.One example is the yellow group, which corresponds to flipping the cube back toward the palm; six NMF factors were used for the visualization.Source: §E.1 · p. 50, Figure 22 · p. 51