Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

• The controller dynamically reorients complex and new objects by any amount.Source: Summary · p. 2
• The system operates in real time.Source: Summary · p. 2
• The controller uses a single depth camera.Source: Summary · p. 2

Motivation

• Real-world deployment also requires arbitrary orientations, generalization to new objects, real-time operation with commodity sensors, and robustness to occlusion and gravity.Downward-facing manipulation is especially difficult because the hand must counteract gravity, while visual pose estimators trained per object limit shape generalization.Source: Introduction · p. 3
• Prior reorientation systems commonly restrict shapes, contact patterns, hand orientation, dynamics, or rotation range, limiting applicability to specific objects, scenarios, or orientations.These constraints arise from the high-dimensional continuous-discrete control problem and include simple convex shapes, fixed finger patterns, no contact changes, upward-facing hands, and quasistatic manipulation.Source: Introduction · pp. 3–4

Method

• Training uses synthetic point clouds followed by rendered-point-cloud finetuning, making the vision-policy pipeline 5× faster while reducing the sim-to-real gap.The faster first stage avoids rendering, addressing training times exceeding 20 days with the available compute resources.Source: Introduction · p. 5
• The controller directly predicts actions from point clouds, avoiding object-specific pose or keypoint definitions and enabling generalization to new shapes.A sparse convolutional neural network processes point clouds for real-time feedback control at 12 Hz.Source: Introduction · p. 5

Results

• The approach provides evidence that sim-to-real transfer is possible for dynamic, contact-rich reorientation of complex objects in less-structured settings.The results show progress toward real-time control while acknowledging substantial room for improving precise reorientation.Source: Introduction · p. 6

Contribution

• The proposed system dynamically reorients new complex-shaped objects through arbitrary rotations using a single commodity depth camera and joint encoders.Experiments include downward-facing-hand manipulation, both with external support and in the air without a supporting surface.Source: Introduction · p. 6

Hardware

• The hardware setup costs less than $5,000 and uses only open-source components, making the platform easier to replicate.This contrasts with prior systems using custom or expensive manipulators and sophisticated sensing equipment.Source: Introduction · p. 6

• The system can be used for reorientation and other dexterous manipulation tasks.Source: Introduction · p. 6
• Its low barrier to entry and evidence of tackling a challenging manipulation task support broader accessibility.Source: Introduction · p. 6
• The platform can democratize research in dexterous manipulation.Source: Introduction · p. 6

Results

• The controller reoriented diverse new objects with complex geometries not used for training, operating in real time at 12 Hz from a single depth camera and finger proprioception.A single controller was trained to reorient 150 objects from arbitrary initial to target configurations, then deployed on real three- and four-fingered manipulators.Source: Results · pp. 6–8

Extrinsic dexterity: object reorientation with a supporting surface

• With table support, three fingers sufficed, and the system was evaluated across diverse object shapes, materials, surfaces, fingertip materials, and finger counts.The supporting-surface experiments represent extrinsic dexterity and include both three- and four-fingered configurations.Source: Reorientation using a three-fingered… · p. 10, Extrinsic dexterity: object… · p. 10

Reorientation using a three-fingered manipulator with rigid and soft fingertips

• 81% of table-supported trials on seven training objects reached 0.4 radians, compared with 45% on five held-out objects; soft fingertips increased held-out success to 55%.These success rates use the prior-work thresholds of 0.4 and 0.8 radians; the 55% result is obtained by switching from rigid to soft fingertips.Source: Reorientation using a three-fingered… · pp. 10, 12

Object reorientation on different supporting materials

• Performance remained similar across five supporting materials—rough cloth, smooth cloth, slippery acrylic, perforated bathtub mat, and uneven door mat—demonstrating controller robustness to diverse materials.Source: Object reorientation on different… · p. 13

Towards object reorientation in air

• Removing the supporting surface caused controllers trained with support to consistently drop the object, producing failures during attempted in-air reorientation.This result motivated the need for a different approach to reorientation without a supporting surface.Source: Towards object reorientation in air · p. 13
• A four-fingered hand substantially outperforms a three-fingered hand for in-air reorientation, although both achieve similar performance with a supporting surface.The proposed reward discourages contact with external support, enabling one controller to operate in air and recover using a supporting surface when available.Source: Towards object reorientation in air · p. 14, Figure 4 · p. 15

SO(3) object reorientation in air

• Median manipulation time is less than 7s across objects and randomly sampled orientation distances in the full SO(3) space, while non-dropped in-air trials retain orientation-error distributions similar to supporting-surface reorientation.Real-world reorientation times are longer than in simulation, likely because contact dynamics differ.Source: SO(3) object reorientation in air · pp. 14, 16
• 56% of trials dropped the out-of-distribution duck-shaped object, compared with 23% for the training-shaped truck, making dropping the most notable source of error.The duck’s simulation dropping rate was around 20%, indicating a sim-to-real gap; when a table was present, the controller could pick up a dropped object and continue reorienting.Source: SO(3) object reorientation in air · p. 16
• The same controller works across in-air, on-table, and dropping-recovery scenarios, with similar on-table error distributions and some generalization to symmetric objects.The controller’s motions vary with target orientation but remain similar across trials sharing the same initial and target orientations, rather than appearing random.Source: SO(3) object reorientation in air · pp. 16–17

Generalization to objects in daily life

• Qualitative tests on household objects show robustness to noisy goal specifications and some ability to generalize to new materials and shapes.Objects were scanned with Scaniverse because CAD models were unavailable, and imperfect scans produced noisy target-orientation clouds.Source: Generalization to objects in daily life · p. 17

Comparison to prior works

• In simulation, the controller achieved 71% success on the duck-shaped object during dynamic reorientation, whereas prior work achieved 60% (3 out of 5 tests) quasi-statically, taking over 70s.That work reoriented only along two axes and within 0.1 radians; a strictly fair comparison was not possible.Source: Comparison to prior works · pp. 17–19
• The contribution is broader-scope reorientation in pragmatic settings, not higher performance on a single object.The authors emphasize expanding generality rather than optimizing one object.Source: Comparison to prior works · p. 19
• The single controller handles multiple objects, shows some generalization to new objects, and dynamically reorients objects in air without specialized perception.These capabilities define the demonstrated expansion in scope.Source: Comparison to prior works · p. 19
• The authors note ample scope for improving performance and state that a fast-learning framework will facilitate future research in enhancing performance and comparing results.Source: Comparison to prior works · pp. 19–20

Discussion

• Real-world reorientation without shape input suggests shape information is less critical than expected, though incorporating shape features may improve precision and generalization.Earlier simulation work raised uncertainty about whether this effect was real or an artifact; real-world evaluation provides greater confidence that reorientation can proceed without shape information (7).Source: Discussion · p. 20
• More fingers can make deep reinforcement learning easier, especially for challenging in-air manipulation, possibly because over-parameterized hardware offers more good solutions.This explanation is presented as a conjecture rather than an established mechanism.Source: Discussion · pp. 20–21
• The system’s technical choices, including point-cloud representations, sparse convolutions, system identification, soft fingertips, and many fingers, may extend to broader vision-based manipulation tasks.The authors connect these choices and the low-cost platform to democratizing dexterous-manipulation research.Source: Discussion · p. 21

Limitations and Possible Extensions

• Higher-frequency feedback and explicit contact, force, or slip sensing may reduce object drops caused by delayed slip detection and incomplete visual contact information.The current system uses occluded vision at 12Hz, so faster processing or more efficient networks may be required for higher-frequency control.Source: Discussion · p. 21
• Inaccurate rotational-distance prediction, particularly below 0.4 radians, limits precision, while larger datasets, RGB sensing, multimodal sensing, and sim-to-real research may improve generalization.These proposals target fine geometric structure, reduced noise, more complete point clouds, and the remaining sim-to-real gap.Source: Discussion · p. 22
• The D’Claw’s finger geometry, actuation, placement, and materials are suboptimal, motivating computational methods for automatically designing task-specific hands.Soft fingertip designs affect longevity, and manually iterating over hardware choices is infeasible.Source: Discussion · p. 22

Materials and Method

• The controller learns to reorient randomly posed objects to user-provided target orientations in SO(3), training across hundreds of simulated objects before zero-shot real-world deployment.The datasets comprise 150 internet-sourced objects and 12 ContactDB objects with non-overlapping shapes.Source: Materials and Method · p. 23

Training the visuomotor policy

• The two-stage Teacher-student2 framework first trains a privileged-state teacher with reinforcement learning, then trains a vision student by supervised imitation and rendered point-cloud fine-tuning.Figure 6 shows the teacher, student, and deployment pipeline; the student directly controls real robots.Source: Training the visuomotor policy · pp. 24–25
• The student policy avoids object-pose estimation because reference frames differ across objects, instead learning end-to-end from raw sensory observations to match teacher actions.Supervised learning is described as more data-efficient than reinforcement learning for raw high-dimensional observations.Source: Training the visuomotor policy · p. 25

Teacher policy: reinforcement learning with privileged information

• A single teacher policy is trained across multiple objects using proprioceptive state, object state, and target orientation, while previous-action input encourages smooth control through 12-Hz joint position controllers.Source: Teacher policy: reinforcement learning… · pp. 25, 27
• The reward combines target-orientation success with dense orientation-distance shaping and penalties for fingertip separation, energy use, and pushing the object away from the hand.The expanded success criterion also penalizes finger and object movements to reduce oscillation around the target.Source: Teacher policy: reinforcement learning… · p. 27
• The teacher reward penalizes object-table contact for airborne reorientation and penultimate-joint use, with the latter producing more natural-looking behavior but not being critical.The teacher is trained with privileged information, and the fingertip-use penalty is described as a shaping term rather than an essential requirement.Source: Teacher policy: reinforcement learning… · p. 28

Student policy - imitation learning from depth observations

• The student policy imitates the privileged teacher with DAGGER using depth-camera and joint-encoder data represented as exteroceptive and proprioceptive point clouds.The goal is represented by the object’s point cloud in its target orientation, and a gated recurrent unit integrates temporal information without direct object-velocity access.Source: Student policy - imitation learning… · pp. 29, 31, 32
• Training with synthetic point clouds reduces student-policy training from 20+ days to less than three days, followed by roughly one day of rendered-point-cloud finetuning that outperforms training from scratch.The two-stage procedure first uses complete, unoccluded synthetic point clouds, then adapts to occluded rendered observations for deployment.Source: Student policy - imitation learning… · p. 31

Reducing the simulation to reality gap

• Sim-to-real transfer combines depth-only sensing and added depth noise with system identification, selective domain randomization, random object disturbances, and soft fingertips.Robot dynamics receive identification and small randomization, while object and environment dynamics receive larger randomization because their material and mass variation is difficult to identify; soft fingertips reduce overshoot.Source: Reducing the simulation to reality gap · p. 33

Identification of robot dynamics

• The manipulator’s URDF is built from CAD to obtain accurate kinematics, while joint damping and stiffness remain dynamics parameters requiring estimation.This establishes the basis for identifying robot dynamics before simulation-based policy training.Source: Identification of robot dynamics · p. 33
• The controller identifies dynamics by searching simulated parameterizations whose motions best match the real robot, using massively parallel Isaac Gym simulations and CMA-ES (48).Because Isaac Gym does not expose its precise dynamics model or parameter gradients, the approach uses black-box optimization; exact matching may remain impossible under approximate simulation and real-world noise.Source: Identification of robot dynamics · pp. 33–34

Real-world deployment

• Real-world deployment combines manipulator joint positions with RealSense depth images into a unified point-cloud input and stops when a learned predictor estimates sufficiently small rotation and action magnitude.The stopping predictor reuses policy-network features to predict |∆θ_t|.Source: Real-world deployment · pp. 34–35

• The supplementary PDF file includes additional materials.Source: List of Supplementary Materials · p. 36
• Other supplementary materials are also provided for the manuscript.Source: List of Supplementary Materials · p. 36

Nomenclature

• In Nomenclature, ¯a denotes a smoothed action command, while A(·) denotes a sequence of action commands.Source: Nomenclature · p. 48
• In Nomenclature, vo is the object’s linear velocity and d is the disturbance force on the object.Source: Nomenclature · p. 48
• In Nomenclature, ¯˙q, ¯a, ¯v, and ¯p are threshold values for joint velocity, action norm, linear velocity norm, and object-hand distance, respectively.Source: Nomenclature · p. 49
• In Nomenclature, lj identifies the jth link on the hand, while M gives the number of links on a hand.Source: Nomenclature · p. 49
• In Nomenclature, pfi denotes the fingertip position of the ith finger and po denotes the object center position.Source: Nomenclature · p. 49
• In Nomenclature, h is the score function for trajectory similarity, and N() denotes a Gaussian distribution.Source: Nomenclature · p. 49

Object datasets

• The 150-object collection spans complex, non-convex shapes, while S contains 12 ContactDB (40) objects without overlap with dataset B, including a five-object OOD subset.Examples include cars, shoes, and animals, selected for asymmetry or reflective symmetry.Source: Object datasets · p. 50

Training setup

• The teacher policy observes joint positions and velocities, fingertip pose and velocities, object pose and velocity, target orientation, and rotation difference, using a three-hidden-layer MLP with 512, 256, and 256 neurons.Source: Training setup · p. 51
• The success criterion combines target-orientation accuracy with small finger and object motions, regularizing policies to slow near completion and mitigating real-world overshooting from sim-to-real gaps.The gaps include control latency, observation noise, and differing dynamics; the expanded criterion serves both as a training reward and an episode-stopping condition.Source: Training setup · pp. 51–52

Real-world setup

• Real-world experiments used three- and four-fingered hands with nine and twelve Dynamixel motors, respectively, observed using one RealSense D415 camera, with motor position control running at 12Hz.Source: Real-world setup · p. 53
• The deployed controller stops successfully using only orientation and fingertip-motion criteria, without directly estimating object motion from point clouds.A predictor reuses policy features to estimate orientation error, while real-world finger motion can be detected by checking ||a_t|| < ā.Source: Real-world setup · p. 54
• Motion capture with six OptiTrack cameras and object markers provides pose-based quantitative error evaluation, but is not required for reorientation.Marker-induced dynamics were sufficiently tolerated, although trials with tracking failures from finger occlusion are discarded.Source: Real-world setup · pp. 54–55

Computing the orientation error for symmetric objects

• For symmetric objects, task completion is ambiguous because multiple orientations can produce the same visual appearance.The approach therefore requires enumerating all equivalent goal orientations, which is challenging to scale broadly.Source: Computing the orientation error for… · p. 55
• On a rectangular cuboid and cube, the orientation error is the minimum distance between the measured pose and all enumerated visually equivalent goal orientations.Their symmetric axes can be readily enumerated in the experiments.Source: Computing the orientation error for… · p. 55

Fingertip Fabrication

• Rigid fingertips are 3D-printed, while soft fingertip skeletons and elastomer molds are fabricated on a Markforge Onyx One and surface-treated for silicone adhesion.Skeletons are sanded with 400-grit sandpaper, corona treated, and primed with Dow DOWSIL P5200 Adhesion Promoter.Source: Fabrication · pp. 55–56
• Soft fingertips combine a rigid 3D-printed inner skeleton with a tacky elastomer outer coating to increase compliance and friction while limiting deformation.The skeleton maintains fingertip shape similarly to a human finger.Source: Fabrication · p. 56

Object Fabrication

• Evaluation objects are 3D-printed from PolyTerra PLA at 0.25 mm layer height, with real household objects added to test transfer to real-world items.The printed objects use different filament colors, and their masses are listed in Table S2.Source: Fabrication · p. 56

Dynamics Identification

• After identifying joint stiffness, damping, and velocity limits from one finger, simulated joints produced responses similar to real joints under the same commands.Each finger joint was identified individually, and its parameter group was transferred to the simulator’s remaining fingers.Source: Dynamics Identification · p. 57
• Real joints responded slightly more slowly than simulated joints because of robot-hand latency, which was not modeled but did not prevent real-world controller operation.Including latency in simulation may further improve real-world performance and is left for future work.Source: Dynamics Identification · p. 57

Robust policy learning

• Policies were trained with Gaussian action and state noise, point-cloud augmentation, and random point dropout to improve robustness to observation and action variation.Point-position noise and point dropout were each applied with probability p = 0.4; dropout removed q ∈[0, 20] percent of points.Source: Robust policy learning · p. 58
• Training randomized robot link mass, joint friction, joint damping, and object mass, friction, and restitution to handle dynamics mismatch.Robot dynamics received small randomization, while object dynamics received large randomization.Source: Robust policy learning · p. 58
• Training also applied random object disturbance forces with probability p = 0.2 at each time step, using random directions sampled in SO(3).The disturbance magnitude was defined relative to object mass through a coefficient-based force specification.Source: Robust policy learning · p. 58

Student policy closely tracks the performance of teacher policy

• Student policies closely tracked the teacher’s performance, although the fully observable low-dimensional teacher achieved the highest success rate at every error threshold.Performance was measured with ECDF curves of orientation error on training and testing object datasets in simulation.Source: Student policy closely tracks the… · p. 58

Symmetric object reorientation

• The controller worked reasonably well on symmetric cuboids and cubes, but produced higher orientation error than on nonsymmetric objects.The authors believe difficulty determining when the goal orientation is reached contributes more to the performance drop than action prediction.Source: Symmetric object reorientation · p. 59

Ablation on the reward terms

• Increasing c1 and c2 improved learning until excessive c1 or c2 = 1.0 caused performance to deteriorate.The reward-term ablation varied c1, c2, and c3, with policies trained on one object without domain randomization.Source: Ablation on the reward terms · p. 60
• 87% in-air success with the contact penalty versus 4.1% without it demonstrates that penalizing table contact benefits in-air reorientation.The ablation trained policies on a single object without domain randomization.Source: Ablation on the reward terms · p. 60

Using a different encoder for goal

• Stacking the goal point cloud with the scene and using one 3D CNN encoder led to faster policy learning than using separate encoders.The comparison is shown between the whole-input encoder and two separate scene and goal point-cloud encoders.Source: Using a different encoder for goal · p. 60

Stage 0: speeding up vision policy training with visual pre-training

• Pre-training vision to predict low-dimensional state information improved learning speed, with scalar rotational-distance prediction converging faster than 6D rotation prediction.The predicted state included object category, rotational distance to the goal, and robot-hand joint positions; scalar-distance prediction outperformed 6D representation for pre-training convergence.Source: Stage 0: speeding up vision policy… · pp. 61–62
• All four visual pre-training schemes produced a substantial and similar speedup in subsequent policy learning.The vision network was pre-trained before fine-tuning the policy with its recurrent component.Source: Stage 0: speeding up vision policy… · p. 62

Ablation on the prediction tasks for vision network pre-training

• Removing rotational-distance prediction makes pre-training easier, with loss decreasing faster, but most severely reduces the benefit of pre-training.The Full model predicts rotational distance, joint positions, and object category using a single embedding architecture.Source: Ablation on the prediction tasks for… · p. 62
• The rotational-distance prediction task is therefore the most useful pre-training task for the vision network.This conclusion follows from the ablation in which that task’s removal most harms the pre-training benefit.Source: Ablation on the prediction tasks for… · p. 62
• Removing joint-position prediction slightly reduces the benefit of pre-training, whereas removing object-category prediction has almost no effect.Each ablation removes one prediction task from the full three-task pre-training setup.Source: Ablation on the prediction tasks for… · p. 62

Analysis for object reorientation in the air in simulation

• The airborne simulation uses realistically rendered point clouds, a 15-second episode horizon, and a stopping rule based on predicted orientation distance and action magnitude.The table initially supports grasping, after which the controller must reorient the object in the air.Source: Analysis for object reorientation in… · p. 63
• 10.3% of 1200 simulated testing episodes involve the object falling immediately after the table is removed, supporting the airborne evaluation as a reasonable real-world approximation.The simulation keeps the table for 24 steps to enable grasping, then removes it and checks immediate falls through step 30.Source: Analysis for object reorientation in… · p. 63
• Most simulated failures occur because objects fall out of the hand, while object #12 has especially many orientation-error failures because its point cloud is nearly symmetric.The controller evaluates Success, Orientation error, Time out, and Object falls across twelve objects, with 100 randomized trials per object.Source: Analysis for object reorientation in… · p. 63
• Median successful reorientation in the full SO(3) space is less than 7 seconds, demonstrating fast and dynamic manipulation.The comparison between simulation and real-world results still reveals gaps in policy performance.Source: Analysis for object reorientation in… · pp. 63–64

Discussion on precise manipulation

• Success falls from 72.3% at 0.4 radians to 25.9% at 0.1 radians across 1,200 simulation tests on 12 objects.Using ground-truth rotational distance to stop the controller raises success to 67.4% at 0.1 radians and 80.9% at 0.4 radians.Source: Discussion on precise manipulation · p. 64, Figure S12 · p. 74
• The success-rate drop is largely caused by failures in predicting rotational distance, which determines when the hand stops.Ground-truth stopping substantially improves both thresholds, indicating that the controller’s manipulation capability is stronger than the nominal 0.1-radian success rate suggests.Source: Discussion on precise manipulation · p. 64
• Only 29.4% of predictions are within 0.1 radians when actual ∆θ ≤0.1 radians, compared with 85.9% predicting below 0.4 radians when actual ∆θ ≤0.4 radians.Figure S14 shows the predictor performs reasonably overall but is insufficiently accurate in the small-distance region relevant to precise manipulation.Source: Discussion on precise manipulation · pp. 64–65, Figure S14 · p. 76
• Distance-prediction errors, observation delays, and command delays make manipulation with ∆θ ≤0.1 radians challenging.The discussion identifies better orientation-distance prediction or a classifier for recognizing when the goal orientation has been reached as research directions.Source: Discussion on precise manipulation · p. 65