PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Ying Yuan, Haichuan Che, Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Kang-Won Lee, Yi Wu, Soo-Chul Lim, Xiaolong Wang

arXiv:2312.01853v3cs.ROcs.CVcs.LG

TL;DR

Robotic manipulation must combine visual and tactile feedback despite their distinct representations and the sim-to-real gaps of both modalities. Robot Synesthesia integrates tactile and camera point clouds in a shared 3D representation, and its policies transfer to real-world tasks including double-ball rotation and generalization to novel objects.

  • Problem

    Robotic manipulation must integrate sparse, low-dimensional tactile input with dense, high-dimensional vision, while transferring learned visuotactile policies across the domain gaps of both modalities.

    Source: §I · pp. 1–2

  • Method

    Robot Synesthesia paints FSR tactile data into the camera point cloud in a unified 3D space and distills a simulator-trained teacher into a PointNet student using visual and tactile inputs.

    Source: §I · p. 2

  • Results

    The policy solves double-ball rotation, generalizes to novel objects in three-axis rotation, and transfers from simulation to real-world tasks without fine-tuning.

    Source: §I · p. 2, D. Real-world Deployment · p. 6, §VI · p. 6

  • Takeaways & Limitations

    The approach supports complex in-hand tasks such as double-ball rotation and generalization to novel objects within the evaluated task scope.

    Source: §I · p. 2, §VI · p. 6

Abstract

from arXiv · show

Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/ .

I. INTRODUCTION

Robotic manipulation must integrate sparse tactile signals with dense visual data while addressing the sim-to-real gaps of both modalities. Robot Synesthesia represents tactile data as a point cloud alongside camera points, and the resulting policy solves double-ball rotation and generalizes to novel objects.

I. INTRODUCTION

  • Robotic integration is difficult because tactile input is sparse and low-dimensional while vision is dense and high-dimensional, requiring models to interpret and combine both.Sim-to-real transfer adds a separate domain gap for each modality, and bridging both at once increases the challenge.Source: §I · pp. 1–2
  • Robot Synesthesia paints FSR tactile data into the camera point cloud in a unified 3D space, preserving spatial relationships among robot links, sensors, and the object.The representation is generated in simulation and reality using robot kinematics, and is intended to reduce compounding vision-and-touch errors during transfer.Source: §I · p. 2
  • The method trains a simulator teacher with oracle state information, distills it into a PointNet student using visual and tactile inputs, and deploys the student in the real world.The tasks include rotating one or two objects along the x, y, and z axes while preventing slips.Source: §I · p. 2
  • The policy solves the challenging double-ball rotation task and generalizes to novel objects in three-axis rotation tasks.Experiments evaluate the policy with eight real-world objects; the analysis also finds tactile points help PointNet locate action-relevant points.Source: §I · p. 2

II. RELATED WORK

Prior dexterous manipulation work spans classical control, reinforcement learning, and multimodal sensing, but existing approaches have limits in object geometry, size, or spatial representation. Robot Synesthesia combines tactile and camera point clouds at the input level.

II. RELATED WORK

  • Earlier dexterous manipulation methods used expert-engineered dynamics models, while later work explored deep model-free reinforcement learning and imitation learning.The related work also describes combining reinforcement learning with demonstrations to improve sample efficiency and produce more natural manipulation behaviors.Source: §II · p. 2
  • Prior in-hand rotation methods used proprioceptive history, binary tactile signals, depth, or vision-and-touch policies, with reported limits including non-convex objects and fingertip-sized objects.The cited optical-tactile approach required continuous object contact with sensors, limiting it to smaller objects rotated on the fingertips.Source: §II · p. 2
  • The approach uses camera point clouds to incorporate 3D information, whereas RGB images do not inherently capture distance information important for manipulation.Depth has also been used in prior work for contact-intensive tasks such as in-hand rotation.Source: §II · p. 2
  • Robot Synesthesia merges tactile and camera point clouds at the input level, treating them as one policy input in a cohesive 3D space.Unlike feature-level fusion with separate modality extractors, this representation combines the modalities before policy processing.Source: §II · p. 3

III. VISUOTACTILE DEXTEROUS MANIPULATION

The system combines camera-derived and robot-augmented point clouds with tactile sensing for in-hand rotation benchmarks. The benchmarks require rotation across axes, object configurations, and object shapes.

A. System Setup

  • The real and simulated setup uses an XArm6, a 16-DOF Allegro Hand with 16 FSR tactile sensors, and a depth camera.FSR measurements are binarized using a predetermined threshold to abridge the sim-to-real gap; the simulator is Isaac Gym.Source: A. System Setup · p. 3
  • Camera point clouds are generated from depth images, while augmented points sampled from the robot mesh encode hand-object spatial relationships.A one-hot vector distinguishes camera-generated points from augmented points, and control runs at 10Hz in both simulation and reality.Source: A. System Setup · p. 3

B. Benchmark Problems

  • The wheel-wrench benchmark requires z-axis rotation without dropping the wrench, while double-ball rotation requires simultaneous manipulation of two balls around each other.The wrench task requires visual identification of the next possible handle; touch tracks rotation, while vision locates both balls because tactile feedback alone cannot distinguish them.Source: B. Benchmark Problems · p. 3
  • Three-axis rotation extends tasks to fixed x- or y-axis rotation and requires handling varied shapes, including objects absent from training.Source: B. Benchmark Problems · p. 3

IV. LEARNING VISUOTACTILE DEXTERITY

The paper formulates in-hand rotation as a Markov decision process and describes its episode handling and state components.

A. Problem Formulation

  • The rotation task is modeled as a Markov decision process with state, action, transition probability, reward, and discount factor, optimizing expected accumulated reward.Source: A. Problem Formulation · p. 3
  • Episodes end at reset conditions or the maximum step count, and training prunes exploration after an object falls off the hand.Source: A. Problem Formulation · p. 3

1) State:

  • The listed state components include Allegro joint positions, binary tactile signals, the rotation axis, and the previous position target.Source: 1) State · p. 3

2) Action:

The policy issues relative hand-control commands that a PD controller applies to the next position target, while the reward combines six weighted components. The student policy uses visual point clouds in a teacher-to-student pipeline, with point clouds selected over RGB because their Sim-to-Real gap is smaller.

2) Action:

  • Relative control commands update the next hand position target through a PD controller, with exponential moving average smoothing set to η = 0.8.The action is 16-dimensional, and the initial smoothed command is zero.Source: 2) Action · p. 3

3) Reward:

  • The reward is a weighted sum of rotation, velocity, distance, torque, work, and control components.The supplied equation names six components but does not specify their weights.Source: 3) Reward · p. 3
  • The teacher policy is trained with reinforcement learning and distilled to a student that receives camera, augmented, and tactile point clouds.The student distinguishes point-cloud sources with one-hot vectors; point clouds were selected because their Sim-to-Real gap is smaller than RGB's.Source: Figure 3 · p. 4, Figure 4 · p. 4

B. Tactile-Visual Synesthesia

Robot Synesthesia represents visual and tactile inputs together in a shared 3D point-cloud space. Active tactile sensors contribute sampled points, giving the policy spatial relationships among observation entities.

B. Tactile-Visual Synesthesia

  • The method projects visual and tactile modalities into one 3D space rather than extracting separate features and merging them later.The representation is described as analogous to simultaneously perceiving touch and visual stimuli.Source: B. Tactile-Visual Synesthesia · p. 4
  • For each tactile sensor with a detected signal, points are sampled from the sensor mesh to form a tactile point cloud.The detected-signal condition is ot,i = 1.Source: B. Tactile-Visual Synesthesia · p. 4
  • Combining the tactile point cloud with the other observations supplies the policy network with spatial relationships among observation entities.Source: B. Tactile-Visual Synesthesia · p. 4

1) Teacher policy training:

A PPO-trained teacher uses low-dimensional state inputs, and its behavior is distilled into a student policy that takes visuotactile observations. The student is pretrained by behavior cloning and fine-tuned with DAgger; artificial objects are used for training and daily objects for real-world testing.

1) Teacher policy training:

  • The teacher policy uses PPO with hand joint positions, binary tactile signals, rotation axis, prior position targets, object motion, and shape features as input.For multi-object generalization tasks, a pretrained PointNet encoder supplies the shape embedding, and the current state is stacked with three historical states.Source: 1) Teacher policy training · p. 4
  • Artificial objects are used for training, while daily objects are used for real-world testing.Source: Figure 5 · p. 5

2) Student policy training:

  • The student policy takes proprioception, binary tactile signals, the rotation axis, prior position targets, and camera, augmented, and tactile point clouds.The point-cloud sources are concatenated with one-hot identifiers, alongside the non-point-cloud inputs.Source: 2) Student policy training · p. 5
  • The student is pretrained by behavior cloning on 5120k teacher transitions and then fine-tuned with DAgger for more robust behavior.PointNet encodes the point clouds, and an MLP processes the resulting latent vector with other inputs.Source: 2) Student policy training · p. 5

V. EXPERIMENTS

The experiments compare Robot Synesthesia with baselines in simulation and on a real robot, asking about modality benefits, teacher-student training, and multimodal processing. The benchmark covers three rotation problems, including real-world testing on daily objects for generalization.

V. EXPERIMENTS

  • The experiments compare the approach with baselines in simulation and reality, examining modality benefits, teacher-student training, and synesthesia-based processing.Source: §V · p. 5

1) Object Dataset:

  • The three benchmark problems use a four-way wheel wrench, two same-sized balls, and simulated objects with common geometries.For real deployment, cubes and other daily objects of distinct sizes and shapes test generalization.Source: 1) Object Dataset · p. 5
  • The wheel wrench is rotated along the z-axis, while the double-ball task uses two same-sized balls.Source: 1) Object Dataset · p. 5

2) Evaluation Metric:

The teacher RL policy outperforms both baselines across the benchmark tasks. Evaluation uses reward in simulation and rotation angle and duration in real-world tests.

2) Evaluation Metric:

  • Cumulative Rotation Reward evaluates simulation performance, while Cumulative Rotation Angle and Time-to-Fall evaluate real-world rotation and episode duration.Time-to-Fall varies when an object falls before the maximum episode length.Source: 2) Evaluation Metric · p. 5
  • Figure 6 shows the teacher policy’s double-ball rotation learning curve averaged over three seeds.The figure caption identifies the task and averaging; it does not provide specific curve values.Source: Figure 6 · p. 5
  • Figure 7 visualizes selected foreground point clouds against observed background point clouds, with red points marking the proposed tactile point cloud.The caption describes the visualization but does not state a comparison outcome.Source: Figure 7 · p. 5

B. Stage I: RL training with different sensing capabilities

  • The proposed approach outperforms PS and Visual RL on all benchmark tasks; Visual RL barely learns high-reward actions within the same training epochs.The learning curves also show higher reward than PS for double-ball and multi-object rotation around the x-axis.Source: B. Stage I: RL training with different… · p. 5

C. Stage II: Imitation learning with different sensing capabilities

Stage II distills teacher policies into visuotactile student policies and compares sensing configurations. The proposed method performs similarly to other visual policies on regular shapes but outperforms baselines on more challenging objects.

C. Stage II: Imitation learning with different sensing capabilities

D. Real-world Deployment

Without real-world fine-tuning, the visuotactile policies show greater benefits in real deployment than for simple geometries in simulation. The analysis links the proposed tactile point clouds to selected tactile and hand-surface points, while the conclusion reports transfer and generalization to novel objects.

D. Real-world Deployment

  • Without fine-tuning, integrating vision and touch has a more significant advantage in real-world deployment than for simple geometries in simulation.The authors describe the proposed tactile point-cloud representation as having a low domain gap and say rudimentary feature concatenation can make modality relationships harder for the policy to comprehend.Source: D. Real-world Deployment · p. 6
  • Visual policies make occasional adjustments to nudge objects toward the palm center, while non-visual policies tend to follow nearly fixed motions.The reported visual-policy adjustments respond to object deviation or becoming stuck.Source: D. Real-world Deployment · p. 6

E. Qualitative Analysis: Visualization of PointNet intermediates

  • PointNet’s max-pooling encoder implicitly selects no more than cout points, with cout denoting its output dimension.The qualitative analysis reports that selected points include tactile points and points from fingertips, finger edges, and the palm.Source: E. Qualitative Analysis: Visualization… · p. 6
  • PointNet selects 42.7% tactile-based points on average; the remaining selected points are mainly finger tips or edges and the palm.The analysis describes this selection as evidence that the encoder extracts meaningful features from the visual-tactile design.Source: E. Qualitative Analysis: Visualization… · p. 6

VI. CONCLUSION

  • The conclusion reports simulator-trained vision-and-touch policies transferring to the real world, solving double-ball rotation, and generalizing to novel objects.The paper identifies goal-conditioned rotation and optical tactile sensors as possible future work.Source: §VI · p. 6
Loading 2312.01853v3…