PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning

Entong Su, Chengzhe Jia, Yuzhe Qin, Wenxuan Zhou, Annabella Macaluso, Binghao Huang, Xiaolong Wang

arXiv:2403.12170v1cs.RO

TL;DR

Tactile-only manipulation must generalize to objects whose geometry and pose are not directly visible, while real-robot RL interaction demands grow with task complexity. This paper trains tactile-input policies in simulation with diverse objects and investigates representations for Sim2Real transfer. Tactile policies achieve strong real-world pivoting results, including approximately 0.80 success for two representations compared with lower angle-estimation baselines.

  • Problem

    Tactile pivoting requires inferring object pose and geometry from contact, while real-robot RL interactions grow with task complexity and diverse-object learning is difficult.

    Source: §I · p. 1

  • Method

    The paper trains tactile-input RL policies in simulation with diverse objects and studies tactile image representations for zero-shot Sim2Real transfer.

    Source: §I · p. 1, §III · p. 2, B. Tactile Image Representation · p. 3

  • Results

    Tactile methods had the highest real-world success and lowest deviation; Tactile-RGB(Aug) and Tactile-Binary(Aug) achieved approximately 0.80 success, versus 0.23 for PCA and 0.60 for ConvNeXt angle estimation.

    Source: A. Necessity of Tactile Sensing · pp. 4–5

  • Takeaways & Limitations

    The trained tactile policy generalized to various unknown objects and previously unknown surfaces in the pivoting task.

    Source: §VI · p. 6

Abstract

from arXiv · show

Using tactile sensors for manipulation remains one of the most challenging problems in robotics. At the heart of these challenges is generalization: How can we train a tactile-based policy that can manipulate unseen and diverse objects? In this paper, we propose to perform Reinforcement Learning with only visual tactile sensing inputs on diverse objects in a physical simulator. By training with diverse objects in simulation, it enables the policy to generalize to unseen objects. However, leveraging simulation introduces the Sim2Real transfer problem. To mitigate this problem, we study different tactile representations and evaluate how each affects real-robot manipulation results after transfer. We conduct our experiments on diverse real-world objects and show significant improvements over baselines for the pivoting task. Our project page is available at https://tactilerl.github.io/.

I. INTRODUCTION

The paper investigates tactile-only object pivoting, where policies must infer object pose and geometry from fingertip contact. It trains with diverse objects in simulation and studies tactile representations to reduce the Sim2Real gap.

I. INTRODUCTION

  • The task is to pivot an object to a target angle using tactile observations, with the policy responding to object pose and geometry.The figure describes simulation-trained policies transferring to the real robot without real-world data.Source: §I · p. 1, Figure 1 · p. 1
  • Explicitly estimating object shape and pose from partial tactile observations can be inaccurate, while real-robot RL interactions grow with task complexity.The text says this interaction burden prevents policies from learning and operating with diverse objects.Source: §I · p. 1
  • The proposed approach trains an RL policy on diverse simulated objects for zero-shot real-robot transfer, then compares original, difference, and binary tactile inputs.The binary representation carries less object information but reduces the Sim2Real gap effectively.Source: §I · pp. 1–2
  • Training on diverse simulated objects yielded better real-robot performance with a more abstract tactile representation than with full tactile information.The study used 22 simulation training objects and 16 unseen real-world test objects.Source: §I · p. 2

II. RELATED WORK

Prior tactile-manipulation work either compresses high-resolution tactile images into estimated states or maps images directly to actions. The paper situates its approach among methods for bridging tactile Sim2Real differences and generalizing across objects.

II. RELATED WORK

  • State-estimation methods use ICP or PCA to reduce tactile images to low-dimensional pose or orientation estimates, while end-to-end RL maps tactile images directly to actions.The passage notes that low-dimensional states do not fully capitalize on detailed contact geometry.Source: §II · p. 2
  • Unlike prior pivoting work applying pose estimation in real-world conditions, this work trains in simulation with diverse objects for broader real-world applicability.Its approach explores tactile-image representations to reduce the domain gap without real-world training or data collection.Source: §II · p. 2
  • Tactile Sim2Real approaches include binarizing signals, high-fidelity FEM-based simulation, image translation, and differentiable tactile simulation.The passage says FEM-based simulators are computationally too costly for RL training, and diverse-object generalization had not been shown for the cited differentiable method.Source: §II · p. 2

III. SIM2REAL TRANSFER OF VISUAL TACTILE READINGS

The proposed system trains tactile-observation RL policies for DIGIT sensors and targets Sim2Real transfer without relying on real-world data.

III. SIM2REAL TRANSFER OF VISUAL TACTILE READINGS

  • The system is designed to transfer RL policies trained with DIGIT tactile observations to the real world.The section introduces simulated visual-tactile data for learning pivoting.Source: §III · p. 2
  • The approach does not depend on real-world data.Source: §III · p. 2
  • To address the tactile domain gap, the paper investigates three distinct representations for encoding tactile data.Source: §III · p. 2

A. Tactile Pattern Rendering in Simulation

The simulation setup renders DIGIT tactile readings for RL training, with two tactile sensors mounted on the gripper’s fingertips.

A. Tactile Pattern Rendering in Simulation

  • The simulator maps physics-engine contact forces to deformation depth and updates tactile images in real time.The mapping also enables object position adjustments based on applied normal forces.Source: A. Tactile Pattern Rendering in… · p. 2
  • RGB tactile images are rendered using Phong’s model, with PyTorch3D replacing the original OpenGL renderer to speed RL training.Direct GPU tensor rendering removes GPU-CPU transfer overhead.Source: A. Tactile Pattern Rendering in… · p. 2
  • Two tactile sensors are mounted on the gripper’s fingertips, and their readings serve as RL policy inputs after processing.Source: Figure 2 · p. 3

B. Tactile Image Representation

The paper addresses tactile sim-to-real and sensor-consistency challenges by abstracting non-essential image details while preserving contact information for decision-making. It compares three tactile representations and applies augmentation to Diff and Binary images.

B. Tactile Image Representation

  • The representations are designed to abstract non-essential tactile-image details while preserving contact information needed for robot decisions.The motivation includes simulation-to-real sensor differences and inconsistent real-world DIGIT behavior from manufacturing and gel variations.Source: B. Tactile Image Representation · p. 3
  • RGB uses the original image, Diff is a grayscale pixel-wise difference from a force-free canonical image, and Binary thresholds Diff into contacted and non-contacted pixels.The Binary threshold is grid-searched per digit to accommodate manufacturing variation and reduce noise, with a trade-off against missing contact information.Source: B. Tactile Image Representation · p. 3
  • Diff and Binary images are randomly scaled between 0 and 1, while right-gripper images are horizontally flipped to align angle information across grippers.These representations and adjustments are used in both simulation and real-world experiments.Source: B. Tactile Image Representation · p. 3

IV. LEARNING TACTILE POLICIES FOR PIVOTING

The pivoting system trains tactile-input RL policies in simulation and transfers them to a real robot. Its setup combines tactile images and proprioception, constrained actions, varied object conditions, PPO training, and a multi-part reward.

IV. LEARNING TACTILE POLICIES FOR PIVOTING

  • The pivoting task requires the robot to rotate an object to a target angle relative to the gripper, relying exclusively on tactile sensing and joint proprioception, without external sensors for pose estimation.Observations include tactile images, robot joint proprioceptive states, and task-related information such as target angle; tactile-image representations have resolution 64 × 64.Source: §IV · p. 3
  • Actions are restricted to end-effector translation in the xz plane and rotation about the y-axis, with gripper width held constant.The restricted action space is intended to simplify the robotic system and speed RL training.Source: §IV · p. 3
  • Domain randomization varies object dimensions and poses, target angle, and support-surface height, using objects from PartNet and Breaking Bad.The listed ranges are 0–20 cm surface height, 13–18 cm object length, 165–195 degrees initial angle, and 90–150 degrees target angle.Source: §IV · p. 3
  • PPO trains policies with shared image encoders whose features are combined with proprioception for both actor and critic networks.Training uses five seeds and stable-baselines3 default hyperparameters.Source: §IV · p. 3
  • The reward combines contact, distance-to-target, angle-to-target, and action-magnitude terms, while the baselines span proprioception-only, oracle and estimated angles, point clouds, DAgger, and tactile inputs.Tactile baselines include RGB, Depth, and Diff, with and without image augmentation; evaluation uses angle deviation and success rate, with success defined as deviation under 15%.Source: §IV · p. 3

V. SIM2REAL TRANSFER OF VISUAL TACTILE READINGS

The paper evaluates zero-shot Sim2Real transfer of tactile policies on a real robot without fine-tuning. The evaluation considers tactile sensing, representation choice, and generalization across multi-category tactile policies.

V. SIM2REAL TRANSFER OF VISUAL TACTILE READINGS

  • Policies are transferred directly to the real robot without fine-tuning.The experiments assess transfer effectiveness in the real-world setting.Source: §V · p. 4
  • The evaluation examines whether tactile sensing is necessary and how different tactile representations perform.The section also investigates generalization of multi-category tactile policies.Source: §V · p. 4
  • The experiments assess generalization of multi-category tactile policies.Source: §V · p. 4

A. Necessity of Tactile Sensing

Real-world evaluation favors tactile-based methods over the compared alternatives, while comparisons with angle estimation, point-cloud, and DAgger policies show transfer challenges for non-tactile approaches.

A. Necessity of Tactile Sensing

  • Without tactile feedback, the baseline had lower success rates and larger angle deviations than the other methods.Its policy also learned slowly and showed higher variance in simulation reward and success rate.Source: A. Necessity of Tactile Sensing · p. 4
  • Real-world success was 0.23 for PCA and 0.60 for ConvNeXt angle estimation, versus approximately 0.80 for Tactile-RGB(Aug) and Tactile-Binary(Aug).Oracle Angle reached a 0.96 success rate in simulation but faced Sim2Real transfer challenges.Source: A. Necessity of Tactile Sensing · p. 4, Table I · p. 5
  • The Point Cloud policy’s success rate fell from 0.62 in simulation to 0.50 in the real world.The authors attribute failures mainly to size and shape differences between real-world and simulated objects.Source: A. Necessity of Tactile Sensing · p. 4
  • The student policy achieved a simulation success rate similar to its teacher, but its real-world performance decreased to 0.50, highlighting tactile sensing’s crucial role in manipulation.The expert policy’s simulation success rate was 0.69.Source: A. Necessity of Tactile Sensing · p. 5
  • Real-world performance was lower than simulation, likely reflecting evaluation differences: 500 simulation episodes versus 30 real-world episodes and less varied real-world scenarios.The passage also suggests tactile feedback helps pivoting decisions by capturing essential angle information.Source: A. Necessity of Tactile Sensing · p. 5
  • Tactile-based methods achieved the highest real-world success rate and lowest angle deviation among the evaluated methods.Source: A. Necessity of Tactile Sensing · p. 5, Table I · p. 5

B. Effect of Tactile Representations

The tactile representations had similar simulation success rates but differed in real-world performance, while image augmentation particularly improved RGB transfer. Augmentation also favored contact patterns over pixel values.

Effect of different Tactile representations

  • RGB, Diff, and Binary achieved similar simulation success rates of 0.65 and 15% angle deviation, but Tactile-RGB underperformed and Tactile-Binary was consistently superior in real-world tests.RGB tactile transfer faced challenges from lighting, color, and pixel variations.Source: B. Effect of Tactile Representations · pp. 5–6

Effect of Image augmentation

  • 0.76 was the real-world success rate for augmented Tactile-RGB, compared with 0.75 in simulation.Image augmentation produced marginal gains for Tactile-Binary and Tactile-Diff across simulation and real environments.Source: B. Effect of Tactile Representations · p. 6
  • Image augmentation can improve Sim2Real transfer by helping policies focus on contact patterns rather than pixel values.The reported performance gains applied to policies using RGB, Binary, and Diff representations.Source: B. Effect of Tactile Representations · p. 6

C. Multi-Category Tactile Policy Generalization

Multi-category tactile policies generalized to diverse objects and previously unseen supporting surfaces in real-world evaluation. Failures were associated with unstable gripping or incomplete or unusual contact.

Effect of Multi-Objects Training

  • 0.80 was the success rate for the multi-category tactile policy, while single-category policies struggled with varied real-world object geometries.Both training datasets contained multiple instances of each object category.Source: C. Multi-Category Tactile Policy… · p. 6

Generalization to unseen supporting surfaces

  • On unseen surfaces, the tactile-based policy achieved a 0.76 success rate with 12% angle deviation despite a performance drop in tactile-based and non-tactile methods.The surfaces varied in friction and stiffness; each policy was evaluated over ten trials.Source: C. Multi-Category Tactile Policy… · p. 6

Failure Cases

  • Failures were associated with unstable gripping that produced incomplete patterns, or incomplete or unusual contact that yielded poor tactile feedback.The paper summarizes the policies as generalizing across multi-categories and performing well on previously unseen supporting surfaces in real evaluation.Source: C. Multi-Category Tactile Policy… · p. 6

VI. CONCLUSION

The study trained a tactile-based reinforcement-learning policy for pivoting and successfully transferred it from simulation to real-world use.

VI. CONCLUSION

  • The pivoting policy transferred from simulation to real-world use and generalized to various unknown objects and previously unknown surfaces.The authors state that they are committed to releasing the simulated environment and training pipeline code.Source: §VI · p. 6
Loading 2403.12170v1…