PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, Pulkit Agrawal

arXiv:2211.11744v3cs.ROcs.AIcs.CVcs.LGeess.SY

TL;DR

Prior in-hand reorientation systems were limited by object complexity, orientation range, sensing, dynamics, and real-world transfer. This paper develops a simulation-trained controller that uses visual point clouds to reorient novel complex objects with real-world hardware, demonstrating broad rotation capability while acknowledging limits in precise manipulation.

  • Problem

    Prior reorientation systems impose restrictions on object shapes, orientations, sensing, and deployment, while visual policies and sim-to-real transfer remain unresolved.

  • Method

    The paper trains a controller in simulation and uses direct point-cloud action prediction from a commodity depth camera to control real-world reorientation.

  • Results

    The controller reorients new complex objects across the full space of rotations using a single commodity depth camera and joint encoders.

  • Takeaways & Limitations

    The results provide evidence that sim-to-real transfer is possible for dynamic, contact-rich manipulation in less-structured settings.

  • Takeaways & Limitations

    Precise manipulation remains substantially improvable, with success falling from 72.3% to 25.9% at 0.1 radians.

Abstract

from arXiv · show

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments that remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasistatic manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints which make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real-time, with the median reorientation time being close to seven seconds. The controller is trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only uses open-source components that cost less than five thousand dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56 percent of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23 degrees) 75 percent of the time. Videos are available at: https://taochenshh.github.io/projects/visual-dexterity.

Summary

The paper presents a real-time controller that dynamically reorients complex, previously unseen objects by arbitrary rotations using a single depth camera.

  • A real-time controller dynamically reorients complex and new objects by any amount using a single depth camera.

Introduction

In-hand reorientation is a challenging prerequisite for dexterous manipulation, but prior systems impose restrictions on object shapes, orientations, sensing, dynamics, or deployment. This work addresses these constraints with a vision-based controller trained in simulation and transferred to real-world manipulation of complex, novel objects.

  • In-hand object reorientation supports dexterous tasks such as tool use and orienting tools for downstream applications.
  • Downward-facing manipulation is harder because the hand must counteract gravity, and small finger-motion errors can drop the object.
  • The task combines high-dimensional continuous-discrete control with difficult state estimation under finger occlusion and limited availability of plug-and-play touch sensors.
  • Prior systems restrict applicability through simple shapes, fixed finger patterns, upward-facing hands, quasistatic motion, or limited orientations.
  • Prior reinforcement-learning controllers generally required full state information and large simulation data, leaving visual training and sim-to-real transfer unresolved.
  • A two-stage vision-policy pipeline uses synthetic and rendered point clouds, making training 5× faster while reducing the sim-to-real gap.
  • Direct point-cloud action prediction avoids consistently defining pose or keypoints across different objects, supporting generalization to new shapes.
  • The system dynamically reorients new complex objects across the full rotation space using one commodity depth camera and joint encoders.

Results

The controller reorients diverse objects in real time across supported and in-air settings, including held-out shapes and materials. Performance is strongest on training objects and remains effective but less reliable on novel objects, especially during object drops.

  • Training and setup: 150 objects were used to train a single controller from random initial configurations toward target configurations.Training included arbitrary initial transformations within the robot’s finger range.
  • Training and setup: A single depth camera and finger proprioception provide observations, while the target is specified as an object point cloud in SO(3).The controller predicts joint-position changes and does not require motion capture except for evaluation.
  • Table-supported reorientation: 81% of training objects were reoriented within 0.4 radians, compared with 45% for five held-out test objects under the original threshold.Soft fingertips improved held-out success from 45% to 55% without affecting training-object performance.
  • Table-supported reorientation: The controller generalized across supporting materials, with similar performance reported for different materials and some new shapes.Evaluation included tablecloth, uneven mat, acrylic, and perforated bath-mat surfaces, as well as real household objects.
  • In-air reorientation: Only the four-fingered hand achieved in-air reorientation when lifting was incentivized, whereas three and four fingers learned similarly with table support.In-air errors for non-dropped trials were similar to supported reorientation, and the controller also showed recovery after drops.
  • In-air reorientation: The median manipulation time across objects and orientation distances in full SO(3) was less than 7 seconds.Real-world reorientation was slower than simulation, which the authors associate with differing contact dynamics.

Discussion

The discussion presents design choices supporting real-time reorientation of complex and new objects with limited sensing, while identifying precision, dropping, hardware, and sim-to-real gaps as open challenges.

  • Shape information: A single depth camera can support reorientation of diverse and new objects without explicit shape information.The authors caution that shape remains useful, especially for precise manipulation and generalization.
  • Hardware and optimization: More fingers can make deep reinforcement learning easier, especially for challenging in-air manipulation scenarios.The authors conjecture that over-parameterized hardware provides a larger pool of viable reorientation solutions.
  • System design: Two-stage student training, point-cloud representations, sparse convolutions, system identification, and soft fingertips are among the system’s technical choices.The choices are presented as applicable beyond in-hand reorientation to vision-based manipulation of rigid bodies.
  • Accessibility and hardware: The open-source, low-cost platform lowers the barrier to dexterous manipulation research, although its finger geometry and actuation remain suboptimal.The authors suggest automated task-specific hand design as an alternative to manually iterating hardware choices.
  • Limitations and extensions: Higher-frequency feedback may reduce dropping failures, but it requires more efficient networks or greater processing power.At 12Hz, visual feedback may not detect slipping quickly enough during dynamic manipulation.
  • Limitations and extensions: Explicit contact, force, and slip sensing could improve performance beyond the controller’s current reliance on occluded vision.The authors propose augmenting observations with touch sensors.
  • Limitations and extensions: Precision and generalization may improve with larger object datasets, RGB sensing, multimodal sensing, and reduced sim-to-real mismatch.The discussion also identifies inaccurate rotational-distance prediction and hardware design as performance constraints.
  • Conclusion: The system demonstrates real-time reorientation of complex and new objects by arbitrary amounts using a simple, affordable setup.The authors frame this as progress toward broader real-world dexterous manipulation, while noting substantial room to improve precision and drop rate.

Materials and Method

The method trains a single vision-based reorientation controller in simulation and transfers it directly to real-world manipulation. A teacher–student framework addresses learning from high-dimensional sensory observations while supporting diverse object shapes.

  • Problem formulation: The controller reorients objects from random initial poses to user-provided target orientations in SO(3) space.A single policy is trained across hundreds of simulated objects and deployed directly in the real world.
  • Success criteria: The success criterion combines orientation accuracy with penalties for finger and object motion to reduce oscillation near the target.The added motion terms explicitly encourage slowing down at the end of reorientation.
  • Teacher–student training: The teacher policy uses privileged low-dimensional state information, including fingertip velocity, object pose, and object velocity.These quantities are available in simulation but can be difficult to measure in the real world.
  • Real-world sensing: The real-world controller uses joint positions and a depth image from one RealSense camera as observations.The policy is trained to process raw sensory observations rather than relying on a cross-object pose reference frame.
  • Teacher–student training: The student policy learns from raw sensory observations by imitating teacher actions through supervised learning.Supervised learning is used because it is more data efficient than reinforcement learning for learning from high-dimensional observations.
  • Generalization: The policy operates across diverse objects and generalizes to objects with different shapes and physical parameters.The setup uses separate object datasets with no overlapping shapes, supporting evaluation beyond the training shapes.
  • Teacher–student training: The two-stage student framework first uses synthetic complete point clouds, then fine-tunes on rendered point clouds.The framework substantially speeds vision-policy learning and enables deployment on real robots.
  • Problem formulation: The policy is modeled as an MDP using the current observation and previous action, although the setup lacks full state information.Because a single observation does not convey the complete object geometry, the underlying setup is a POMDP.

List of Supplementary Materials

The supplementary materials provide additional documentation and methods for the manuscript.

  • Supplementary materials: The manuscript includes supplementary materials beyond the main text.The supplied passage introduces the supplementary-materials list without enumerating its contents.
  • Supplementary materials: The supplied list does not specify individual supplementary items.Only the existence of supplementary materials is stated in these passages.
  • Supplementary materials: The supplementary PDF file contains additional material for the manuscript.The passage identifies the supplementary PDF as the container for these materials.

Nomenclature

The nomenclature defines symbols for actions, object and finger states, links, motion thresholds, disturbances, and trajectory scoring.

  • Control and disturbance: The notation includes smoothed action commands, action-command sequences, and disturbance force on the object.These symbols describe control inputs and external effects in the reorientation system.
  • Motion and scoring: The nomenclature defines object linear velocity, a Gaussian distribution, and a trajectory-similarity score function.These symbols support motion description, stochastic modeling, and trajectory comparison.
  • Thresholds: Threshold symbols specify limits for joint velocity, action norm, linear velocity, and object distance from the hand.These thresholds are used to characterize motion and separation constraints.
  • Geometry and structure: The notation includes the jth link on the hand.This symbol indexes individual hand links.
  • Geometry and structure: Finger and object geometry are represented using fingertip positions, object center position, and the hand’s number of links.The notation distinguishes per-finger positions from the object center and hand structure.

Experiment details

Experiments use diverse simulated object datasets, two robot hands, point-cloud policies, and real-world hardware to evaluate reorientation and generalization. The study also documents success criteria, compute cost, sensing, and calibration constraints.

  • Object datasets: The Big dataset contains 150 internet-sourced objects spanning complex and non-convex shapes such as cars, shoes, and animals.The Small dataset contains 12 ContactDB objects with no overlapping shapes and supports out-of-distribution evaluation on five objects.
  • Object datasets: The training objects are asymmetric or reflectively symmetric, while symmetric-object reorientation is evaluated empirically.This training choice mitigates multimodality in defining object poses without restricting the claimed evaluation.
  • Policy architectures: The teacher is an MLP with three hidden layers of 512, 256, and 256 neurons, optimized with Adam.The vision policy processes colorless point clouds voxelized at 0.005m resolution.
  • Evaluation criteria: The expanded success criterion requires target orientation, small joint motion, and small object velocity.Finger and object motion criteria act as regularizers that encourage slowing near the end of reorientation.
  • Hardware: The experiments use three-fingered D’Claw and four-fingered hands, with nine and twelve Dynamixel motors respectively.The hands are mounted on an 80/20 aluminum frame.
  • Hardware: The real-world observation combines motor joint positions with a depth image from one RealSense D415 camera.The policy controls the robot using these sensory observations after calibration with dual quaternions.

Fabrication

The system uses 3D-printed rigid and soft fingertip designs, with compliant, tacky silicone increasing friction while an internal skeleton preserves shape. Objects for quantitative evaluation are also 3D-printed, supplemented by real household objects to test transfer.

  • Fingertip fabrication: The fingertip skeletons and elastomer molds are fabricated by 3D printing, with surface preparation used to improve silicone adhesion.Skeletons are sanded, corona treated, and primed before coating.
  • Fingertip design: Soft fingertips combine a rigid inner skeleton with a compliant elastomer coating to increase friction while limiting deformation.The skeleton is intended to maintain fingertip shape similarly to a human finger.
  • Elastomer coating: Ecoflex platinum-catalyzed silicone provides a Shore hardness of 00-10 and a tacky finish after curing.Pigments are mixed into the Ecoflex to produce gray fingertips.
  • Elastomer coating: The uncured elastomer is poured into smoothed molds, degassed, and cured at room temperature for 4 hours.XTC-3D is applied inside the molds to smooth their surface before casting.
  • Object fabrication: Evaluation objects are 3D-printed from PolyTerra PLA at 0.25 mm layer height, while real household objects test transfer beyond printed shapes.Table S2 lists the masses of the printed objects used in real-world experiments.

Overcoming sim-to-real gap

The sim-to-real approach identifies key joint dynamics, trains with randomized dynamics and observation/action noise, and uses step and sinusoidal commands to collect response data. After identification, simulated joints closely match real responses, although real-hand latency remains unmodeled and can cause slower responses.

  • Action commands: Step commands and sinusoidal commands spanning 0.05–1.5 Hz provide the joint-response trajectories used for identification.Command amplitudes are scaled proportionally to each joint’s limit.
  • Dynamics identification: Dynamics identification estimates joint stiffness, damping, and velocity limits from response trajectories collected on one real finger.The identified dynamics group is applied to the remaining fingers in simulation.
  • Sim-to-real response: After dynamics identification, simulated joints produce responses similar to real joints under the same control commands.This similarity is shown across the three joints on a finger.
  • Sim-to-real response: Real joints usually respond slightly more slowly because of robot-hand latency, which is not modeled in simulation.The controller nevertheless works on real robot hands.
  • Remaining gap: Including latency in simulation is identified as a possible future improvement for real-world controller performance.The current controller succeeds on real hands despite omitting this latency model.
  • Robustness randomization: Training adds Gaussian action noise, point-cloud position noise, point dropout, and randomized robot and object dynamics.Object mass, friction, and restitution receive larger randomization than joint dynamics parameters.

Supplementary Discussion

The supplementary discussion examines policy behavior across training and testing objects, symmetric-object ambiguity, reward terms, visual encoders, and pre-training choices. It reports strong teacher–student agreement and fast real-world reorientation, while identifying symmetry and simulation-to-real differences as remaining challenges.

  • Policy evaluation: The teacher policy achieves the highest success rate at every orientation-error threshold, while student policies closely track its performance.Performance is measured with ECDF curves over orientation errors on training and testing object datasets.
  • Symmetric objects: Symmetric objects create multimodal action sequences because multiple poses can visually match the same goal configuration.The authors train on asymmetric or reflective-symmetric objects and test whether the learned controller generalizes to symmetric objects.
  • Symmetric objects: Implicit models could address multimodality, but their computational inefficiency prevents real-time use.The paper’s alternative handles multimodality during training rather than requiring mode distinction at deployment.
  • Symmetric objects: The controller works reasonably well on a rectangular cuboid and a cube, but their orientation errors tend to exceed those of nonsymmetric objects.The authors believe visually similar symmetric poses contribute more to the performance drop than action prediction.
  • Reward ablations: Increasing c1 and c2 initially improves policy learning, but excessive values reduce performance; learning is less sensitive to c3 and c5.Performance begins to deteriorate after c2 = 1.0, while negative c3 and c5 were advantageous.
  • Reward ablations: 87% in-air success with the contact penalty contrasts with 4.1% without it, showing the penalty benefits in-air reorientation.The penalty reduces table dependency by discouraging contact with the table.
  • Visual architecture: Stacking the goal-object point cloud with the scene point cloud enables faster policy learning than using separate 3D CNN encoders.The comparison is between the whole-input encoder and separate scene and goal encoders.
  • Vision pre-training: Pre-training the vision network to predict low-dimensional state information improves policy-learning speed more than most tested representation-learning objectives.The predicted quantities include object category, rotational distance, and robot-hand joint positions.

Ablation on the prediction tasks for vision network pre-training

The ablation shows that rotational-distance prediction is the most valuable vision pre-training task, while joint-position prediction contributes modestly and object-category prediction has little effect. In-air evaluation also reveals that falling is the dominant failure mode, despite successful fast reorientation in the real world.

  • Prediction-task ablation: Removing rotational-distance prediction makes pre-training easier but substantially reduces its downstream benefit, identifying it as the most useful task.Removing joint-position prediction slightly reduces the benefit, whereas removing object-category prediction has almost no effect.
  • In-air evaluation: 10.3% of 1200 simulated testing episodes involve the object falling immediately after table removal.The simulation keeps the table for 24 steps, then removes it to approximate real-world in-air testing.
  • In-air evaluation: The in-air evaluation classifies outcomes as success, orientation error, timeout, or object falls across 100 trials per object.The controller stops when predicted orientation distance and action magnitude fall below the real-world stopping condition.
  • Failure modes: Most failures occur because objects fall out of the hand, while object #12 has especially many large-orientation-error failures because its point cloud is nearly symmetric.The nearly symmetric representation creates a distinct failure pattern for that object.
  • Real-world performance: Simulation and real-world policy performance still differ, although successful real-world reorientation in full SO(3) has a median time below 7 seconds.The result demonstrates fast and dynamic manipulation capability despite the remaining simulation-to-real gap.

Discussion on precise manipulation

The controller achieves moderate-precision reorientation, but performance declines at stricter orientation thresholds because rotational-distance prediction becomes inaccurate near the target. Improving the stopping module could therefore improve precise manipulation.

  • Precise manipulation: 72.3% success at 0.4 radians fell to 25.9% at 0.1 radians in 1200 simulation tests across twelve objects.The authors attribute much of this decline to the module predicting rotational distance for stopping the controller.
  • Precise manipulation: Using ground-truth rotational distance to stop the controller provides a direct test of whether prediction, rather than reorientation itself, limits strict-threshold success.The rotational-distance estimate is used to determine when the hand should stop.
  • Precise manipulation: 67.4% and 80.9% success rates were achieved at 0.1- and 0.4-radian thresholds, respectively.These thresholds reflect the adopted evaluation criteria for reorientation success.
  • Precise manipulation: Only 29.4% of predictions were within 0.1 radians when the actual rotational distance was Δθ ≤0.1 radians, compared with 85.9% when Δθ ≤0.4 radians.The prediction model performs reasonably well overall but is less accurate near the target orientation.
  • Precise manipulation: Inaccurate rotational-distance predictions, together with observation and command delays, make precise manipulation at Δθ ≤0.1 radians challenging.The paper identifies better distance prediction or goal-reached classification as directions for improving orientation accuracy.
Loading 2211.11744v3…