Source-linked AI summary
Rotating without Seeing: Towards In-hand Dexterity through Touch
Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, Xiaolong Wang
TL;DR
The paper asks whether a multi-finger robot hand can perform complex in-hand manipulation without vision, using touch as humans do. It introduces Touch Dexterity, which trains a reinforcement-learning rotation policy with dense binary force sensors covering the hand and deploys it directly on a real robot. The system rotates diverse unseen objects using touch-only sensing, with ablations showing that tactile feedback and broad sensor coverage are important.
Problem
Touch-only approaches remain challenged by complex, high-DOF in-hand manipulation, despite tactile information supporting human dexterity without vision.
Method
Touch Dexterity trains a reinforcement-learning policy in simulation using binary touch sensors distributed across the palm, finger links, and fingertips together with the robot’s internal state.
Results
The learned policy is deployed on a real robot to rotate diverse unseen objects using only touch, and experiments report robust performance with sensor ablations.
Takeaways & Limitations
Broad binary tactile coverage can support touch-only in-hand rotation and generalization from simulated training objects to unseen real-world objects.
Abstract
from arXiv · showhide
Tactile information plays a critical role in human dexterity. It reveals useful contact information that may not be inferred directly from vision. In fact, humans can even perform in-hand dexterous manipulation without using vision. Can we enable the same ability for the multi-finger robot hand? In this paper, we present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand (palm, finger links, fingertips). Such a design is low-cost, giving a larger coverage of the object, and minimizing the Sim2Real gap at the same time. We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training. Extensive ablations are performed on how tactile information help in-hand manipulation.Our project is available at https://touchdexterity.github.io.
I. INTRODUCTION
Touch Dexterity addresses the challenge of high-DOF in-hand manipulation without vision by using broad, low-cost binary touch sensing and reinforcement learning. The resulting policy rotates unseen objects on a real robot using touch-only feedback.
- Humans use tactile information to manipulate objects without vision, motivating touch-based dexterity for multi-finger robot hands.
- Existing touch-only methods struggle with complex, high-DOF in-hand manipulation while often emphasizing precise fine-grained contact sensing.
- Touch Dexterity uses 16 low-cost binary force sensors across the fingertips, finger links, and palm, providing broad coverage and minimizing the Sim2Real gap.Each sensor reports touch or no touch; 16 sensors provide up to 2^16 binary states.
- The policy is trained with reinforcement learning on multiple objects in IsaacGym and uses binary touch plus internal robot state for closed-loop control.The learned policy predicts an action at each time step and is deployed directly on the real robot.
- Real-world experiments test 10 diverse objects, including unseen objects, and report robust touch-only rotation with extensive sensor ablations.The rubber duck can be rotated for two cycles without falling despite never appearing during training.
- The work extends prior binary-contact manipulation research from lower-DOF settings toward multi-finger robot-hand dexterity.
III. TACTILE DEXTEROUS MANIPULATION SYSTEM
The system uses binary contact sensing distributed across the Allegro hand to estimate object position and detect critical manipulation contacts. These signals provide touch-based feedback for rotating objects whose motion can otherwise become unstable or cause them to fall.
- The hardware combines a 16-DOF Allegro hand with 16 Force-Sensing Resistors attached across the palm and finger regions.The sensors change resistance when external force is applied.
- Continuous force measurements are binarized into contact signals, simplifying control and reducing the simulation-to-real transfer gap.
- In simulation, each sensor is modeled as a fixed link on a finger or palm link, with simulated net contact force converted to a binary signal.
- The task initializes an object in the palm and requires rotation around a specified axis, while sliding and rolling make explicit feedback necessary.
- The sensors provide object-position information from palm contacts and interaction information indicating whether the active fingertip is contacting the object.
- The control state combines tactile information, joint position, previous target, and task information before producing actions for a PD controller.
IV. LEARNING TOUCH DEXTERITY
The rotation task is formulated as a Markov Decision Process, and the policy uses stacked tactile and proprioceptive history to issue smoothed relative joint commands. Rotation progress is evaluated through the object’s angle around the target axis.
- The in-hand rotation problem is modeled as a Markov Decision Process with state, action, reward, and transition dynamics.
- The policy state includes joint position, binary sensor observations, the previous position target, and the desired rotation axis.Three historical states are stacked with the current state when using an MLP policy.
- Each policy action is a relative 16-dimensional joint command, and the next target is obtained by adding it to the previous target.
- An exponential moving average smooths consecutive actions before the PD controller drives the hand toward the target.The experiments use η = 0.8, and the controller operates at 10Hz in simulation and on the real robot.
- The rotation angle ∆θ is calculated by tracking a vector attached to the object and comparing its projected orientation around the rotation axis.
3) Reward:
The reward combines rotation progress with penalties that promote stable, smooth, low-effort manipulation and discourage falling. The rotation term uses a finite angular difference because simulator angular velocity is noisy.
- Reward design: The reward is a weighted mixture of rotation, velocity, falling, work, torque, and distance components.These terms jointly target rotation quality, stability, smooth finger motion, and object retention.
- Rotation reward: The rotation reward measures the signed angle change of a sampled object-attached vector projected onto the plane normal to the rotation axis.The vector is sampled in the plane, tracked at the next state, and compared around the axis.
- Rotation reward: Simulator angular velocity is avoided because its noise can produce undesirable vibrating motion; finite differences yield more consistent rotation behavior.The authors report this consistency across different runs.
- Stability penalties: Velocity, falling, work, and torque penalties encourage stable rotation, prevent object loss, smooth finger motion, and limit large torques.The work penalty is based on the controller torque and joint velocity.
- Reset strategies: Episodes reset when the object moves too far from the palm center or its major axis deviates too far from the desired rotation axis.These resets reduce unnecessary exploration and undesired rotation directions.
B. Domain Randomization
The system uses domain randomization across object, controller, tactile, observation, and action properties to improve robustness and Sim2Real transfer. Training uses PPO with parallel simulation and a real-time-compatible control schedule.
- Domain randomization: Object position, mass, shape, and friction are randomized so the learned policy encounters diverse object conditions.These variations are intended to help the policy handle different kinds of objects.
- Domain randomization: Controller gains, tactile-sensor dropouts, and contact-sensor delays are randomized to model real-world uncertainty.Activated sensor outputs may be flipped from 1 to 0 with probability p, and delays follow an exponential model.
- Domain randomization: White noise is injected into policy observations and output actions to improve robustness to small perturbations.This is a non-physics randomization strategy.
- Training setup: PPO trains multilayer perceptron policy and value networks, with clipped advantages, a KL threshold, ELU activations, and a learnable state-independent policy standard deviation.The policy outputs a Gaussian action distribution.
- Training setup: Simulation runs 8192 parallel environments at dt = 0.01667s with two substeps, while six-step action execution produces a 10Hz real control frequency.The action is interpreted as the control target.
- Experimental questions: The experiments ask whether tactile information improves training, robustness, generalization, real-world performance, and simulation-to-real alignment.The study first examines z-axis rotation and then demonstrates rotation along the other axes.
A. Experiment Setup
The experiments combine simulated geometric objects with unseen real-world objects and evaluate rotation capability and retention using complementary metrics. Baselines isolate the contributions of tactile sensing, sensitivity, binarization, and closed-loop control.
- Objects: Simulation uses artificial cuboids, cylinders, and balls, while real evaluation includes unseen objects such as a rubber duck and Lego box.The simulated geometries are intended to approximate diverse everyday objects.
- Evaluation metrics: Cumulative Rotation Reward evaluates simulation rotation, Cumulative Rotation Angle evaluates real rotation, and Time-to-Fall measures object retention in both settings.Time-to-Fall records how long the object remains in the palm before falling.
- Baselines: The No-Sensor baseline trains PPO without tactile information and infers interaction only from differences between current and target joint positions.Pressing contact can appear as a discrepancy between these joint quantities.
- Baselines: LS-Sensor raises the activation threshold to θth = 0.2N, reducing sensitivity to slight contacts, while DS-Sensor disables tactile input during evaluation.DS-Sensor tests how much a trained tactile policy uses tactile information.
- Baselines: The real experiments additionally compare an open-loop trajectory policy and CT-Sensor, which uses continuous-valued rather than binarized sensor inputs.These baselines probe task complexity and whether continuous signals create Sim2Real difficulty.
- Comparison scope: The study does not compare against methods requiring very large collections of visual simulation data because those datasets take substantially longer to generate in real-world time.This bounds the comparison scope.
C. Sim: Policy learning with different sensing capabilities.
Simulation experiments show that tactile sensing improves touch-only rotation, especially for multi-object training, and supports robustness and generalization under changed physics and unseen objects.
- In multi-object training, the tactile policy succeeds while baseline policies fail completely, indicating tactile information is essential for touch-only rotation.In single-object training, low-sensitivity tactile sensing can also improve motion security.
- Under unseen friction and mass parameters, the full method shows little performance drop, while No-Sensor and DS-Sensor degrade clearly.LS-Sensor also performs well, suggesting low-sensitivity touch can suffice for single-object robustness.
- Disabling sensor input causes a significant performance drop on both seen and unseen objects, showing tactile information supports generalization.The policies are trained on object set A and evaluated on object set B for unseen-object testing.
E. Real: Dexterity without Vision
On the real robot, the tactile policy rotates seen and unseen objects without vision, while contact patterns align sufficiently between simulation and reality for transfer. Sensor ablations further show that broad fingertip and palm coverage matters.
- The proposed method outperforms all baselines on real multi-object rotation and generalizes from artificial training objects to unseen objects such as apples and tomatoes.No-sensor and open-loop policies rotate at most 180 degrees before becoming stuck or pushing objects off the palm.
- Adaptive tactile feedback lets the policy adjust finger motion near positions where objects may become stuck or fall, unlike sensor-free baselines.Binarized contact signals perform better than continuous signals in generalizability.
- Simulation and real-world contact signals generally align during cuboid rotation, supporting successful Sim2Real transfer.Simulation signals are somewhat denser and richer, but their overall patterns are similar to real observations.
- Removing either fingertip or palm sensors reduces performance to approximately the DS-Sensor level, indicating both sensor groups are essential.
- Touch rollouts can reveal object shape: with touch information, the method reconstructs the ground-truth object after a 20-second rotation, unlike the no-touch condition.The reconstruction study trains on 125 irregular column-shaped objects and collects 55,000 rollouts.
I. Rotation Around Other Axes
The system extends touch-only rotation beyond the z axis to x and y rotations and combines these axis-specific skills into human-directed reorientation primitives, though some objects remain difficult.
- The system rotates most objects around the x and y axes, although particular objects produce lower CRAs.These rotations involve critical contacts with finger-link sides that the current sensor layout does not fully support.
- A denser contact-sensor array over each finger link is proposed as a way to address failures caused by unsupported side contacts.
- Rotation primitives around the x, y, and z axes enable a human operator to reorient objects using high-level commands.The demonstrated shared-control sequence sends keyboard commands for successive axis rotations.
- Touch Dexterity demonstrates touch-only object rotation without vision and reports experiments in both simulation and the real world.
APPENDIX
The appendix documents PPO training, Sim2Real preparation, sensor mapping, and the reward formulation used for Touch Dexterity.
- Training: PPO trains the control policy with specified actor, critic, normalization, clipping, and minibatch settings.The policy uses a three-layer MLP, while the value network uses a four-layer MLP.
- Sim2Real Transfer: Domain randomization is used to improve Sim2Real transfer, with its configuration summarized in Table VI.The supplied appendix identifies the table but does not provide its individual settings.
- Sim2Real Transfer: System identification aligns simulated and real PD-controller responses to impulse and sinusoidal inputs for successful Sim2Real transfer.The paper states that this alignment step is crucial for transfer.
E. More Sensor Response Examples
Additional real-world sensor trajectories collected on different objects exhibit different activation patterns.
- Sensor Response Examples: Real-world sensor activation trajectories differ across the tested objects.These examples extend the reported sensor-response observations beyond the main figure.