PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Binghao Huang, Yixuan Wang, Xinyi Yang, Yiyue Luo, Yunzhu Li

arXiv:2410.24091v2cs.ROcs.AIcs.LG

TL;DR

Robotic manipulation requires both vision’s global information and touch’s detailed contact feedback, but combining these modalities remains challenging. 3D-ViTac addresses this with dense flexible tactile sensing and a unified 3D visuo-tactile representation coupled to diffusion-policy imitation learning. The system improves contact-rich manipulation across challenging tasks, particularly when visual occlusion is heavy and during fragile-object or in-hand manipulation.

  • Problem

    Robots must combine global visual information with local tactile contact information, but tactile hardware and multimodal fusion remain challenging.

  • Method

    3D-ViTac uses dense flexible tactile sensors and fuses 3D visual and tactile points in a shared coordinate space for diffusion-policy imitation learning.

  • Results

    Across four challenging real-world tasks, the 3D visuo-tactile representation improves contact-rich manipulation, especially under heavy visual occlusion.

  • Takeaways & Limitations

    Tactile feedback supports detailed contact-state and local-geometry information for fragile-object manipulation and long-horizon in-hand tasks.

Abstract

from arXiv · show

Tactile and visual perception are both crucial for humans to perform fine-grained interactions with their environment. Developing similar multi-modal sensing capabilities for robots can significantly enhance and expand their manipulation skills. This paper introduces \textbf{3D-ViTac}, a multi-modal sensing and learning system designed for dexterous bimanual manipulation. Our system features tactile sensors equipped with dense sensing units, each covering an area of 3$mm^2$. These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships. The multi-modal representation can then be coupled with diffusion policies for imitation learning. Through concrete hardware experiments, we demonstrate that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and executing long-horizon tasks involving in-hand manipulation. Our project page is available at \url{https://binghao-huang.github.io/3D-ViTac/}.

1 Introduction

3D-ViTac addresses the difficulty of combining dense, flexible tactile sensing with global visual perception for contact-rich robotic manipulation. It introduces a unified 3D visuo-tactile system and reports improved performance across challenging real-world tasks, especially under occlusion and during fragile-object or in-hand manipulation.

  • 1 Introduction: Vision and touch complement each other by providing global scene information and detailed contact feedback, improving robustness under occlusion and during in-hand manipulation.
  • 1 Introduction: Existing robotic solutions face hardware limitations from bulky or expensive tactile sensors and information limitations from sparse low-cost sensing.
  • 1 Introduction: 3D-ViTac combines a dense, flexible tactile sensor array with a unified 3D visuo-tactile representation for contact-rich manipulation.The system integrates visual and tactile information for imitation learning while providing continuous contact feedback.
  • 1 Introduction: The proposed sensor array uses 16×16 sensing units per soft finger, totaling 1024 units across the bimanual system, and captures contact presence, normal force, and local patterns.
  • 1 Introduction: Across four challenging real-world tasks, the 3D visuo-tactile representation improves contact-rich manipulation, with tactile sensing especially valuable under heavy visual occlusion.

2 Related Work

The related work frames bimanual manipulation and visuo-tactile sensing as complementary directions, while positioning 3D-ViTac’s compliant, large-area tactile coverage as a response to prior sensing constraints.

  • 2 Related Work: Prior bimanual manipulation methods commonly rely on model-based control and ground-truth environment models that are time-consuming to construct.
  • 2 Related Work: The platform uses four tactile sensor pads, with two pads mounted on each hand’s soft gripper.
  • 2 Related Work: The proposed tactile sensor uses piezoresistive materials to provide larger-scale, compliant coverage on flexible thin fingers.
  • 2 Related Work: Vision offers broad environmental information but often lacks detailed contact information and is vulnerable to visual occlusion, motivating tactile integration.

3 Visuo-Tactile Manipulation System

The system combines thin, flexible resistive tactile pads with soft grippers, custom readout electronics, and synchronized bimanual teleoperation. This design targets broad, compliant contact coverage while supporting multimodal data collection.

  • 3 Visuo-Tactile Manipulation System: The pads are less than 1mm thick and bend with the soft gripper while maintaining effective signal transmission.
  • 3 Visuo-Tactile Manipulation System: Each tactile pad contains a 16 by 16 array of 256 sensing units, with a current spatial resolution of 3mm2 per sensor point.
  • 3 Visuo-Tactile Manipulation System: A piezoresistive layer converts applied pressure into electrical signals, which are read at up to approximately 32.2 FPS; one pad and board cost about $20 excluding the Arduino.
  • 3 Visuo-Tactile Manipulation System: The soft gripper increases contact area, helping stabilize manipulation and consistently reflect object contact patterns and geometry.
  • 3 Visuo-Tactile Manipulation System: Bimanual teleoperation synchronizes tactile, multi-view RGBD, action, and joint-state data at 10 Hz, while real-time tactile visualization helps operators assess grasp contact.

4 Learning Visuo-Tactile Dexterity

The learning system projects visual and tactile observations into a shared 3D coordinate space and conditions a diffusion policy on the resulting representation. This preserves spatial relationships while supporting dense contact-aware action generation.

  • 4 Learning Visuo-Tactile Dexterity: The policy integrates visual and tactile point clouds in one coordinate system, then uses a diffusion policy to generate robot joint-state action sequences.
  • 4 Learning Visuo-Tactile Dexterity: Visual processing merges multi-view depth clouds, crops them to the work region, downsamples to Nvis = 512, and transforms them into the robot base frame.
  • 4 Learning Visuo-Tactile Dexterity: The tactile point cloud places sensing-unit positions and continuous tactile readings in 3D using gripper forward kinematics and robot joint states.
  • 4 Learning Visuo-Tactile Dexterity: The unified representation labels each point as visual or tactile, explicitly encoding their spatial relationships for contact-rich manipulation.
  • 4 Learning Visuo-Tactile Dexterity: The decision module uses PointNet++ as the backbone of a conditional denoising diffusion model that converts Gaussian noise into actions.

5 Experiments

Experiments evaluate 3D-ViTac on four long-horizon manipulation tasks, comparing visual and tactile sensing choices and examining how touch supports force control, occlusion handling, and demonstration quality.

  • Experiment Tasks: Four long-horizon tasks assess fine-grained force control and in-hand object-state information, including fragile-object handling and tool insertion.Egg steaming and fruit preparation require force information, while Hex Key Collection and Sandwich Serving require in-hand state information.
  • Baseline Comparisons: The visuo-tactile policy integrates vision, tactile sensing, and proprioception, with touch providing contact and force feedback across four challenging tasks.Reported failure cases include insufficient force causing eggs to fall, excessive force damaging grapes, and visual-only difficulty adjusting a hex key in hand.
  • Demonstration Data Quality: Users new to the system perform better with both visual and tactile feedback than with vision alone during demonstration tasks.Ten new users were divided between teleoperation with tactile feedback and teleoperation without it.
  • Tactile Information Ablation: Denser and continuous tactile signals consistently outperform the other tested tactile modalities.The ablation varies tactile resolution from 16×16 to 8×8 and 4×4 and includes a binary-signal baseline; continuous signals encode force and local contact patterns.
  • Visual Information Ablation: As visual occlusion increases with fewer cameras, policies using tactile information continue to perform well.Visual point clouds and tactile point clouds are aligned in camera coordinates for the occlusion ablation.

6 Conclusion and Limitation

The paper concludes that dense flexible tactile sensing and unified 3D visuo-tactile representations support challenging contact-rich manipulation. It identifies real-data cost and the lack of tactile simulation as limitations affecting scalability and future generalization work.

  • Conclusion: 3D-ViTac combines a dense flexible tactile array with a unified 3D visuo-tactile representation for contact-rich robotic manipulation.The representation preserves the 3D structure and spatial relationship between vision and touch.
  • Limitations: Collecting multimodal real data is expensive, and the absence of tactile-sensor simulation limits randomization and data scaling.Future work targets physical simulation to improve policy generalizability and robustness.

Supplementary Materials

Supplementary materials describe the tactile sensor construction, physical characterization visuals, sensor ordering, and compact electronics used for signal acquisition.

  • Sensor Construction: The tactile pad uses orthogonal conductive yarns around a piezoresistive Velostat layer in a triple-layer design.Sixteen conductive threads are aligned for each electrode direction and secured with adhesive.
  • Sensor Calibration: Sensor-unit ordering is explicitly defined so tactile positions can be calculated and signals aligned with the physical setup and dataset.The ordering is visualized in Figure 9.
  • Electronics: The tactile reading board measures 45.5 mm × 48.4 mm and uses an Arduino with shift registers and an analog switch to process sensor signals.The compact board is designed for easy installation and scalability.

A.2.1 Physical Characteristics

The tactile sensors show an approximately linear response from 1 N to 9 N before saturation above 9 N, while the full pad maintains consistent readings across regions. Tactile observations also support 6-DoF pose tracking, becoming more accurate with fuller contact and tracking rotation.

  • Individual Sensor Performance: 1 N to 9 N forms an approximately linear tactile-response region, while saturation begins above 9 N.The result comes from testing 10 sensors with forces from 0 to 12 N and applying a logarithmic force axis.
  • Tactile Sensor Pad Consistency: The 16 × 16 pad is evaluated as 8 × 8 blocks across 1 N, 3 N, 5 N, and 11 N loading conditions.Each block sums four sensor readings, yielding 64 measurements per force condition for heatmaps and consistency analysis.
  • Tactile Sensor Pad Consistency: The pad-consistency experiment summarizes regional responses with heatmaps and a box plot of means and standard deviations after outlier removal.The measurements assess whether the sensor remains stable across the full pad under different normal forces.
  • 6-DoF Object Pose Estimation: The pose-tracking method transforms a known object point cloud with a pose hypothesis and weights it by distances between observed and tactile contact points.A standard particle filter uses these observation and weighting functions to determine the object pose.
  • 6-DoF Object Pose Estimation: Pose estimates become more accurate with more complete tactile signals and remain trackable as the object rotates.The experiment estimates object pose without vision, demonstrating potential for in-hand pose estimation.
  • 6-DoF Object Pose Estimation: With tactile input from only one robot side, multiple poses remain plausible and estimates are inaccurate; contact from the other side improves alignment and rotation tracking.The example uses a bimanual setup with four tactile pads and three RealSense cameras, while tactile signals are published at 30 Hz.

B.2.1 Details for the Egg Steaming task

The egg-steaming task tests whether the robot can open an occluding tray, grasp and place a fragile egg safely, and cover the steaming machine without destabilizing the cover. These steps require controlled force and tactile feedback during occluded or narrow-space interactions.

  • Step 1: Open Egg Tray: The robot must open the randomized egg tray sufficiently for its fingers to grasp the occluded egg.Insufficient tray opening causes the subsequent task to fail.
  • Step 2: Grasp Egg: The robot slowly increases force to grasp the heavily occluded egg, while visuo-tactile control retries when stable tactile contact is absent.A vision-only policy may proceed to the next goal despite lacking reliable contact information.
  • Step 3: Place Egg: The robot places the egg in a machine containing two other eggs without knocking them over, using tactile feedback in the narrow space.Tactile information also signals when the egg is no longer contacting the gripper as the hand exits.
  • Step 4: Cover the Steaming Machine: The robot must apply enough force to lift the steaming-machine handle but not so much that the cover flips and falls.The step depends on precise force control for the handle’s unusual shape.

B.2.2 Details for the Fruit Preparation Task

The fruit-preparation task combines bimanual bag opening with grasping and placing grapes under severe visual occlusion. Its central challenge is selecting contact and force strategies that secure fruit without damaging it or losing it in the gripper.

  • Step 2: Open Plastic Bag: The robot must use both hands to open the transparent plastic bag wide enough for the gripper to enter.The transparent bag adds noise to the point cloud, complicating visual perception.
  • Step 3: Grasp the fruit: Grasping fruit inside the transparent bag is the task’s most difficult step because the bag heavily occludes the robot and objects.The visuo-tactile policy can retry until tactile signals stabilize, whereas vision-only grasping is typically attempted once.
  • Step 3: Grasp the fruit: The visuo-tactile policy grasps single or multiple grapes while applying suitable force, whereas vision-only control may break clustered grapes.The tactile-integrated gripper is thin enough to enter gaps between grapes.
  • Step 4: Place grape: The robot must place the grapes on the plate and return to its initial position without allowing excessive grasping force to make them stick to the gripper.Success is evaluated by completing placement and returning to the starting position.

B.2.3 Details for the Hex Key Collection Task

The hex-key task evaluates stable grasping, bimanual in-hand adjustment, and insertion despite variation in the tool’s pose. Slippage or incorrect adjustment makes the downstream insertion difficult or impossible.

  • Step 1: Right Hand Grasp: The initial right-hand grasp must lift the hex key’s tail without significant slippage, because pose variation complicates later steps.The task deliberately exposes only the tail, requiring adjustments before insertion.
  • Step 2: Left Hand Grasp: The left hand grasps the hex-key head to support the subsequent adjustment step.This establishes the bimanual hold used for repositioning.
  • Step 3: In-hand Adjustment: The two hands adjust the hex key until it is perpendicular to the robot’s fingers for easier insertion.Vision-only control usually fails this adjustment, making the following insertion impossible.
  • Step 4: Insertion: Insertion requires compensating for the hex key’s variable in-hand pose even after a successful adjustment.Success means inserting the key into the hole rather than placing it on the table or becoming stuck.

B.2.4 Details for the Sandwich Serving Task

The sandwich-serving task is organized as a four-step bimanual sequence requiring grasping, tilting, retrieval, and placement. Visuo-tactile sensing supports in-hand state tracking when spoon rotation or position changes would otherwise cause failures.

  • Sandwich Serving Task: The robot grasps a serving spoon, tilts a pot without excessive force, retrieves a fried egg, and places it on bread.Success requires minimal spoon slippage, controlled handle grasping, egg retrieval, and successful tilting for placement.
  • Sandwich Serving Task: During egg retrieval, the spoon passively rotates in the hand, but the visuo-tactile policy explicitly tracks its state while the baseline often fails.
  • Sandwich Serving Task: During egg placement, the policy accounts for changes in the spoon’s in-hand position and adjusts the motion to complete the tilt.
  • Additional Tasks: The broader evaluation also includes light-bulb retrieval and peg insertion, each decomposed into grasping followed by a task-specific manipulation.
  • Learning Backbone: PointNet++ provides the learning backbone through hierarchical feature extraction and processing for point-cloud data.
Loading 2410.24091v2…