Source-linked AI summary
General In-Hand Object Rotation with Vision and Touch
Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, Jitendra Malik
TL;DR
Generalizable multi-axis in-hand rotation requires handling diverse object geometries and unavailable object properties. RotateIt trains privileged policies in simulation, distills them through a visuotactile transformer, and reports improved rotation performance with visual and tactile sensing, including real-world transfer.
Problem
Generalizable fingertip-based rotation across diverse geometries is challenging because stable force closure must be maintained during continuous multi-axis reorientation.
Method
RotateIt trains an oracle policy with simulated ground-truth object properties, then distills it into a transformer policy using simulated noisy visuotactile and proprioceptive histories.
Results
RotateIt matches oracle performance with combined vision and touch, reduces the out-of-distribution performance drop from 41% with proprioception only to 15%, and rotates multiple real objects by ∼2π radians within 20 seconds.
Takeaways & Limitations
The results support using visual and tactile sensing to improve generalization and enable multi-axis object rotation beyond the training objects.
Takeaways & Limitations
The method assumes objects are not too long and within the robot hand’s mechanical limits, remains frozen after training, and uses only low-dimensional tactile contact locations.
Abstract
from arXiv · showhide
We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.
1 Introduction
RotateIt targets generalizable fingertip-based rotation across multiple axes by combining visual, tactile, and proprioceptive inputs. It trains with privileged object information, then infers that information from multimodal histories for deployment.
- Generalizable fingertip-based rotation remains challenging because diverse geometries require stable force closure during continuous multi-axis reorientation.
- The training pipeline jointly optimizes the privileged encoder and control policy with PPO, then trains the transformer by regression to the privileged representation.
- RotateIt first trains an oracle policy using ground-truth object shapes and physical properties encoded as privileged information.
- A visuotactile transformer infers privileged object representations from historical visual, tactile, and proprioceptive inputs.
- The evaluation covers three principal axes, sensing ablations, latent shape recovery, and real-world rotation of multiple objects.
2 Related Work
Prior work addresses manipulation through classical control, real-world learning, sim-to-real transfer, and visuotactile sensing, but RotateIt combines multimodal learning for general in-hand rotation.
- Classical approaches rely on planning and simplified object models, whereas RotateIt uses multisensory learning without an object model.
- The training-object collection combines EGAD, Google Scanned Objects, YCB, and ContactDB meshes.
- Real-world reinforcement learning faces sample-efficiency and variation limits, motivating simulation-based training without human demonstrations.
- Earlier sim-to-real manipulation studies often focus on one object, while transferring generalizable diverse-object manipulation remains challenging.
- RotateIt combines visuotactile sensing with temporal transformer modeling for general in-hand object rotation, a combination the authors identify as novel.
3 General In-hand Object Rotation with Vision and Touch
The method trains privileged oracle policies and distills them into multimodal policies that infer object properties from realistic sensory streams. Its reward, object encoding, and sim-to-real representations are designed for stable multi-axis rotation.
- Policy Training: Policy training has two simulation stages: privileged oracle training followed by visuotactile-policy training with realistic noisy observations.
- Privileged Information: The oracle encoding combines sampled mesh shape features with physical properties including mass, center of mass, friction, scale, and restitution.
- Observations and Outputs: The oracle policy maps temporal robot proprioception and privileged encoding to 16-dimensional PD-controller targets.
- Reward Function: The reward combines rotation progress with penalties for undesired rotation, pose deviation, linear velocity, work, and torque to promote stable, smooth, energy-efficient behavior.
- Touch: Touch is represented by discretized fingertip contact locations in simulation and tracked deformation keypoints from omnidirectional tactile sensors in the real world.
- Vision: Vision uses object depth, with Segment-Anything extracting the object depth map during real-world deployment to reduce the sim-to-real gap.
- Visuotactile Transformer: A transformer fuses encoded depth, tactile contacts, joint positions, and previous actions over a temporal sequence to predict the extrinsic vector.
- Training: The transformer is optimized by minimizing distances between predicted and ground-truth extrinsic vectors and between predicted and oracle actions.
4 Evaluation Setup
Evaluation uses simulated and real hardware settings to measure rotation performance, including time-to-fall, rotation reward, rotation penalty, and radians rotated. The study emphasizes how visual and tactile sensing affects sensorimotor policy performance.
- Hardware and simulation: Experiments use an AllegroHand with simulated and real-world sensing setups.The hardware includes 16 commanded joints, depth sensing, and omnidirectional fingertip touch sensors.
- Sensor evaluation: Vision and touch improve sensorimotor policy performance over proprioception alone, especially for rotation reward around the x and y axes.Combining vision and touch further improves performance beyond either modality alone.
- Object set: The object dataset combines EGAD, Google Scanned Objects, YCB, and ContactDB, filtered by aspect ratio.Selected objects have width/depth/height aspect ratios below 2.0.
- Metrics: Evaluation measures time-to-fall, rotation reward, rotation penalty, and radians rotated.Time-to-fall is normalized by the 20-second maximum episode length, while radians rotated is used in real-world evaluation.
5 Results and Analysis
Results show that object shape information, vision, and touch improve rotation policies, including under out-of-distribution and real-world conditions. RotateIt also supports multi-axis training, while latent shape recovery is stronger for regular objects than irregular ones.
- Object shape helps policy training: Point-cloud inputs significantly improve performance across all metrics and rotation axes.The comparison evaluates RotateIt against prior work and a quaternion-based variant without point-cloud input.
- Visuotactile sensing: Vision and touch together improve visuotactile policies over proprioception alone and can match oracle-policy performance.The gains are larger for irregular and non-uniform objects.
- Tactile information: Contact locations improve performance, whereas binary contact provides no additional value over proprioceptive history.Discretized contact locations can match the performance of full contact information.
- Out-of-distribution evaluation: 41% performance drop with proprioception only is reduced to 15% with vision and touch for out-of-distribution evaluation.The result identifies visuotactile information as important for OOD generalization.
- Latent representation: Shape understanding is adequate for spherical and cuboidal objects but requires additional sensors for irregular objects such as pears.This supports the larger benefit of vision and touch for complex objects.
- Real-world evaluation: RotateIt successfully rotates multiple real-world object geometries by ∼2π radians within 20 seconds, while Hora cannot finish the task.The evaluation includes objects outside the training set and a policy that remains similar when one touch sensor is assumed unavailable.
- Multi-axis training: The distilled multi-axis policy performs on par with single-axis oracle policies.Training with reinforcement learning alone does not converge in this experiment.
6 Limitations and Future Work
RotateIt demonstrates multi-axis fingertip object rotation across many objects, but its current scope excludes overly long objects and real-world adaptation after training. The authors identify richer tactile processing, visual pre-training, and lifelong learning as future directions.
- RotateIt demonstrates policies that rotate many objects over multiple axes, supporting progress toward general-purpose in-hand manipulation.
- The method assumes objects are not too long and remain within the robot hand’s mechanical limits.
- The deployed policy is frozen after training and cannot use real-world experiences during deployment.
- Future improvements include lifelong learning, richer tactile inputs, and visual pre-training.
A Additional Experiments
Additional experiments show that vision, touch, and transformer processing each improve over the baseline and complement one another. The model remains robust to randomized and slightly out-of-distribution vision noise, but excessive noise makes camera calibration important.
- Vision, touch, and the transformer architecture each significantly improve performance over the baseline and complement one another.
- Training randomization makes the model robust to randomized and slightly out-of-distribution vision noise.
- Excessive vision noise still reduces performance, highlighting the importance of proper camera calibration.
B Implementation Details
The implementation uses large-scale parallel simulation, randomized object and physics conditions, and multimodal network components. Vision-noise evaluation and listed hyperparameters specify the training and deployment setup.
- Training uses 32768 parallel environments distributed across 4 GPUs, with 200 Hz simulation, 20 Hz control, and 20-second episodes.
- Object initialization requires fingertip distance below 10 cm, at least two contacting fingers, and object height above 13.5 cm over the palm center.
- Noisy-vision evaluation uses five settings, while physics parameters are sampled uniformly and network defaults are specified in Tables 6 and 7.
- Training applies random disturbance forces scaled as 2m, where m is the object mass, with decay and resampling over time.
- The reward uses rmax = 0.5, rmin = −0.5, λtorque = −0.1, λlinvel = −0.3, λwork = −2.0, and λrotp = −0.1.
- The visuotactile transformer processes depth, touch, proprioception, and action history, with depth images sized 60 × 60.