PaperlayerView exact revision on arXiv

Source-linked AI summary

VisuoTactile 6D Pose Estimation of an In-Hand Object using Vision and Tactile Sensor Data

Snehal s. Dikhale, Karankumar Patel, Daksh Dhingra, Itoshi Naramura, Akinobu Hayashi, Soshi Iba, Nawid Jamali

arXiv:2601.01675v1cs.RO

TL;DR

In-hand 6D pose estimation is difficult because grippers heavily occlude objects and tactile data lacks a standard representation. The paper represents contact surfaces as point clouds and fuses tactile with vision using a dedicated network trained on synthetic visuotactile data. The resulting model outperforms a vision-only baseline and transfers qualitatively from synthetic training to real robots, although orientation degrades under very heavy occlusion.

  • Problem

    Heavy gripper occlusion limits vision-only in-hand 6D pose estimation, while tactile data lacks a standard representation for complementary sensing.

  • Method

    The method represents tactile contact surfaces as object-surface point clouds and fuses them with color-depth vision through separate visual and tactile channels.

  • Results

    The visuotactile model outperforms a same-dataset color-depth baseline and generalizes qualitatively from synthetic training to different real physical-robot setups.

  • Takeaways & Limitations

    Tactile sensing can complement vision for in-hand 6D pose estimation under occlusion, within the demonstrated synthetic-to-real evaluation scope.

  • Takeaways & Limitations

    Orientation inference depends on visual color features, and under very heavy occlusion performance degrades to the baseline.

Abstract

from arXiv · show

Knowledge of the 6D pose of an object can benefit in-hand object manipulation. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot's grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this paper, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot's hand. To address challenges like lack of standard representation for tactile data and sensor fusion, we propose the use of point clouds to represent object surfaces in contact with the tactile sensor and present a network architecture based on pixel-wise dense fusion. We also extend NVIDIA's Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and corresponding tactile point clouds. Results suggest that using tactile data in addition to vision data improves the 6D pose estimate, and our network generalizes successfully from synthetic training to real physical robots.

I. INTRODUCTION

The paper targets in-hand 6D pose estimation under gripper-induced occlusion by combining tactile and vision data. It introduces a tactile point-cloud representation, a fusion network, and synthetic visuotactile data generation.

  • In-hand pose estimation receives limited attention and is challenged by heavy occlusion from robot grippers and environmental objects.
  • Tactile sensors can capture surface geometry occluded from vision, motivating fusion of tactile and vision data for object pose estimation.
  • The network uses pixel-wise dense fusion with separate visual and tactile channels to combine heterogeneous sensor data.
  • The method represents tactile measurements as object-surface point clouds at finger-object contact locations, avoiding dependence on a single tactile sensor format.
  • The extended NDDS generates photo-realistic vision data, corresponding tactile point clouds, and ground-truth poses for 11 YCB objects, with 20000 examples per object.
  • The visuotactile model outperforms a color-depth baseline on the same synthetic dataset and qualitatively generalizes to different real physical-robot setups.

II. BACKGROUND

Prior 6D pose-estimation methods use geometric, probabilistic, or machine-learning approaches, while tactile data remains uncommon. The paper addresses the missing combination of tactile sensing and in-hand pose estimation under heavy self-occlusion.

  • Existing methods include geometry matching, probabilistic estimation, and machine learning, with geometric and probabilistic approaches potentially limited by slow runtime or multiple contacts.
  • Deep learning pose estimators differ in their sensor inputs, including RGB-only methods, RGB-depth methods, and synthetic-data-trained networks.
  • Pixel-wise fusion of RGB and depth data can train on visible object regions, reducing the effects of occlusion.
  • Tactile pose methods estimate local contact shape and rank it against simulated contacts to infer a likely pose distribution.
  • Tactile data has improved occluded-object shape estimation, but prior work did not estimate object pose.
  • The paper focuses on in-hand 6D pose estimation because prevailing RGB or RGB-depth methods do not leverage tactile data despite gripper self-occlusion.

III. NETWORK ARCHITECTURE

The network estimates the camera-frame 6D pose of a known grasped object from color, depth, and tactile contact point clouds. Its visual channel preserves correspondence while fusing color and depth features.

  • The model takes camera color and depth images plus an object-surface tactile point cloud, outputting the object’s 6D pose in camera coordinates.
  • Sensor fusion is divided into visual and tactile channels to maintain correspondence among the heterogeneous inputs.
  • The proposed architecture combines color, depth, and tactile inputs while preserving feature correspondence for pose estimation.
  • The Visual Channel: The visual channel segments the object in the color image before masking depth and converting visible object pixels into camera-coordinate 3D points.
  • The Visual Channel: Color and depth embeddings are processed by separate two-stage CNNs and combined through pixel-wise dense fusion.

2) The Tactile Channel:

The tactile channel combines depth-derived point clouds with tactile contact-surface point clouds, while a global feature supplies cross-modal context. These fused features feed pose prediction and confidence-based selection.

  • The Tactile Channel: The tactile channel combines camera-depth point clouds with object-surface tactile point clouds from all fingers as one 3D point cloud.
  • The Tactile Channel: Representing all finger data jointly makes the network invariant to the robot’s number of fingers and therefore gripper invariant.
  • The Pose Estimator: Color, depth, and tactile embeddings are fused, passed through a CNN, and average-pooled to create a global feature.
  • The Pose Estimator: Figure 3 distinguishes visual-channel, tactile-channel, and global features using blue, green, and yellow, respectively.
  • The Pose Estimator: The pose estimator uses visual-channel, tactile-channel, and global features to predict translation, rotation, and confidence values.
  • The Pose Estimator: The network selects the 6D pose associated with the feature having the maximum confidence level.

C. The Loss Function

The network combines point-wise pose and confidence losses, comparing transformed model points with ground truth while regularizing confidence estimates. The total loss averages these terms across input points.

  • For each input point, the network estimates a translation vector, rotation vector, and confidence level.
  • The point-wise loss compares model points transformed by the estimated pose with the same points transformed by the ground-truth pose.
  • The confidence loss measures the difference between estimated and expected confidence and includes a regularization term wlog(C_i).
  • The total network loss adds point-wise and confidence losses and averages their sum over all points.

A. Tactile Sensor Invariant Pose Estimation

The method represents tactile observations as object-surface point clouds to reduce dependence on tactile sensor type, then combines them with visual data in simulation. Synthetic data are generated from virtual tactile cameras and a main RGB-D camera across randomized grasps.

  • A. Tactile Sensor Invariant Pose Estimation: Tactile observations are represented as object-surface point clouds estimated from sensor data when the gripper contacts the object.Forward kinematics is used to obtain the contact-surface representation.
  • A. Tactile Sensor Invariant Pose Estimation: The simulation extends an Unreal Engine 4 photo-realistic data-generation plugin to produce tactile point clouds alongside color and depth data.
  • A. Tactile Sensor Invariant Pose Estimation: Virtual cameras attached to each gripper phalanx capture contact surface point clouds, while a main camera generates color and depth images.
  • A. Tactile Sensor Invariant Pose Estimation: The dataset contains 11 YCB objects and 20000 samples per object, using 12 virtual tactile cameras and RGB-D images from the main camera.Finger-camera resolution is 32x32, while the main camera captures 640x480 images.
  • A. Tactile Sensor Invariant Pose Estimation: Data collection randomly varies object placement, orientation, gripper preshape, and arm position while recording RGB-D images, tactile point clouds, and object pose.
  • A. Tactile Sensor Invariant Pose Estimation: Domain randomization affects 80% of the dataset, including backgrounds, floor surfaces, spotlight positions, and spotlight colors.The remaining 20% uses no domain randomization.

V. EXPERIMENTS

The network is trained on a synthetic dataset and deployed on two physical robot setups using different grippers and tactile sensors. The experiments measure deployment characteristics and collect ground-truth pose data for evaluation.

  • V. EXPERIMENTS: The dataset is split 4:1 into 16000 training and 4000 test samples for each object, and the network is trained for 1500 epochs.Training uses batch size 16, learning rate 0.0002, w=0.015, and Adam in PyTorch.
  • V. EXPERIMENTS: Setup-I uses a Weiss gripper with two Gelsight sensors on a Sawyer robot and a Kinect2 RGB-D camera.An OptiTrack system records the object’s ground-truth 6D pose.
  • V. EXPERIMENTS: Setup-II uses an Intel RealSense camera and a four-fingered, 16-joint Honda R&D hand equipped with 224 tactile taxels.
  • V. EXPERIMENTS: Network inference takes 10.1 ms, while the complete pipeline including semantic segmentation and ROS overhead takes 109.5 ms.

VI. RESULTS

The experiments test whether combining tactile and vision data improves 6D pose estimation relative to a vision-only baseline. Performance is evaluated using position and angular errors, with angular error computed from estimated and ground-truth quaternions.

  • VI. RESULTS: The proposed method is compared with a vision-only baseline created by removing the tactile channel and retraining on the same color and depth dataset.
  • VI. RESULTS: The vision-only baseline retains only the visual channel, whereas the proposed network uses both tactile and visual inputs.
  • VI. RESULTS: Position error is measured as the Euclidean distance between estimated and ground-truth positions.
  • VI. RESULTS: Angular error is represented by the angle between the estimated and ground-truth unit quaternions.The estimated quaternion is denoted q-hat and the ground-truth quaternion q.

1) Performance on Position and Angular Metrics:

Across object-level and occlusion analyses, the Tactile-Channel network generally reduces position and angular errors relative to the vision-only baseline, including under heavy occlusion. Fewer tactile contacts increase errors, but tactile input remains beneficial over vision alone.

  • Object-level performance: For all objects except cracker box, the network outperforms the baseline in position or angular error with statistical significance.The adjustable wrench position error is 0.36 cm versus 1.14 cm for the baseline; angular improvements are significant for the adjustable wrench and scissors.
  • Occlusion-level performance: Increasing occlusion is evaluated below 80%, between 80% and 85%, and above 85%, with visibility determined from transformed model points and the camera point cloud.Occlusion level is the percentage of model points labelled as not visible using a K-nearest Neighbor threshold.
  • Occlusion-level performance: Under heavy occlusion, position error is 0.4 ± 0.003 cm for the network versus 0.78 ± 0.008 cm for the baseline.The network maintains position performance as occlusion increases, while the baseline shows a significant performance drop.
  • Occlusion-level performance: Under heavy occlusion, angular error is 11.5 ± 0.22 degrees for the network versus 13.8 ± 0.24 degrees for the baseline, a reduction of 2.3 degrees.Angular error still degrades substantially under heavy occlusion, despite outperforming the baseline.
  • Tactile contact points: Reducing tactile contact points from 240 toward 4 increases position and angular error, yet tactile input remains better than the vision-only baseline.The experiment evaluates 4, 12, 36, 60, and 240 tactile contact points.

B. Qualitative Evaluation

Real-robot deployment shows that the synthetic-data-trained visuotactile network is more stable and accurate than the baseline, including as camera occlusion increases. Ablations quantify contributions from the architecture, global feature, and visual data.

  • Qualitative Evaluation: On Robot Setup-I, the network achieves 36.88 ± 1.27 degrees angular error and 2.69 ± 0.13 cm position error versus 64.57 ± 1.40 degrees and 4.76 ± 0.32 cm for the baseline.Both networks receive the same input during real-time deployment and are trained only on the synthetic dataset.
  • Qualitative Evaluation: During deployment, the network produces stable frame-by-frame estimates while the baseline sometimes deviates significantly from the ground-truth pose.The paper attributes the improved estimates especially to tactile information from fingers at the back of the object.
  • Occlusion robustness: With progressively blocked camera views, the network handles higher occlusion levels than the baseline on the real hardware setup.The experiment blocks the main camera’s view with a piece of paper and compares both estimates against ground truth.
  • Ablation study: The ablation study measures gains from the Siamese network, global feature, and visual features across all 11 objects.Reported gains include 0.006 cm and 0.2 degrees from the Siamese network, 0.57 cm and 3.31 degrees from the global feature, and 1.55 cm from visual data.

D. Comparison with State-of-the-Art

The proposed visuotactile approach outperforms vision-only baselines on synthetic data, particularly under heavy occlusion, while transferring qualitatively to real robots. Its performance remains advantageous with sparse tactile input, although very heavy occlusion reduces it to baseline levels.

  • Quantitative comparison: 4.78 degrees: PoseCNN shows an angular error increase compared to the proposed method on the challenging in-hand pose dataset.The analysis also reports significant degradation in position estimation for vision-only RGB under heavy gripper occlusion.
  • Quantitative comparison: The visuotactile network outperforms the vision-only baseline with statistical significance on synthetic data, especially under heavy occlusion.The comparison evaluates the proposed approach against a vision-only baseline trained on the same synthetic data.
  • Tactile-input study: The proposed approach outperforms the baseline even when tactile input is reduced to 4 points.The conclusion reports a study varying tactile inputs from 4 to 240.
  • Real-robot deployment: Qualitative deployments show that a network trained only on synthetic data transfers successfully to real robots with different hardware setups.Figure 10 distinguishes the tactile-channel output from the baseline across deployment setups; Setup-II lacks motion-capture ground truth.
  • Limitations: Under very heavy occlusion, performance degrades to that of the baseline, despite maintaining performance under moderate to heavy occlusion.The reported limitation is tied to orientation inference's dependency on object color features from vision.
Loading 2601.01675v1…