PaperlayerView exact revision on arXiv

Source-linked AI summary

Touch-GS: Visual-Tactile Supervised 3D Gaussian Splatting

Aiden Swann, Matthew Strong, Won Kyung Do, Gadiel Sznaier Camps, Mac Schwager, Monroe Kennedy III

arXiv:2403.09875v3cs.ROcs.CV

TL;DR

Touch-GS addresses the difficulty of directly using raw optical tactile data to supervise 3DGS scenes. It converts touches into an uncertainty-aware GPIS, fuses this with aligned monocular depth, and reports qualitative and quantitative improvements over visual or tactile alternatives in few-view scenes, including reflective and transparent objects.

Abstract

from arXiv · show

In this work, we propose a novel method to supervise 3D Gaussian Splatting (3DGS) scenes using optical tactile sensors. Optical tactile sensors have become widespread in their use in robotics for manipulation and object representation; however, raw optical tactile sensor data is unsuitable to directly supervise a 3DGS scene. Our representation leverages a Gaussian Process Implicit Surface to implicitly represent the object, combining many touches into a unified representation with uncertainty. We merge this model with a monocular depth estimation network, which is aligned in a two stage process, coarsely aligning with a depth camera and then finely adjusting to match our touch data. For every training image, our method produces a corresponding fused depth and uncertainty map. Utilizing this additional information, we propose a new loss function, variance weighted depth supervised loss, for training the 3DGS scene model. We leverage the DenseTact optical tactile sensor and RealSense RGB-D camera to show that combining touch and vision in this manner leads to quantitatively and qualitatively better results than vision or touch alone in a few-view scene syntheses on opaque as well as on reflective and transparent objects. Please see our project page at http://armlabstanford.github.io/touch-gs

I. INTRODUCTION

Touch-GS addresses the limits of visual-only reconstruction by combining monocular depth priors with tactile sensing for few-view 3D scene synthesis. Its GPIS representation preserves continuous surface information and uncertainty from touch data.

  • Robotic interaction requires precise 3D representations, but visual information can be insufficient in cluttered or dark environments where touch is necessary.Source: §I · p. 1
  • Touch-GS targets few-view object-centric reconstruction, including mirrors and transparent objects, where collecting hundreds of training images is impractical.Source: §I · p. 1
  • Prior depth-supervised NeRF methods improve few-view quality but do not use touch data, which can provide more accurate geometric information.Source: §I · p. 1
  • A GPIS represents an object continuously from discrete touches while retaining the Gaussian Process variance associated with the reconstruction.Source: §I · p. 2
  • The method combines visual and tactile data to reconstruct accurate real-world objects from a small number of RGB images with touch input.Source: §I · p. 2

A. Key Contributions

Touch-GS introduces tactile supervision for 3DGS by converting touch measurements into GPIS-based depth information and fusing it with visual inputs. The method is intended to improve few-view synthesis across difficult scenes and can also support other NeRF representations.

A. Key Contributions

  • A GPIS synthesizes tactile measurements into a representation suitable for supervising 3DGS training.Source: A. Key Contributions · p. 2
  • Bayesian fusion of the touch GPIS and monocular depth estimation produces depth and uncertainty images for additional 3DGS supervision.Source: A. Key Contributions · p. 2
  • The method shows qualitative and quantitative improvements in few-view scene synthesis, including scenes with mirrored and transparent objects.Source: A. Key Contributions · p. 2
  • Touch supervision can extend beyond 3DGS to other Neural Radiance Field representations such as Nerfacto and the original NeRF.Source: A. Key Contributions · p. 2

A. Point-based NeRF and 3DGS

  • 3DGS forms images by blending ordered points along rays, and the same training losses can support depth supervision in point-based or volumetric NeRFs.Source: A. Point-based NeRF and 3DGS · p. 2

B. Gaussian Process Implicit Surface

The GPIS models an object as the zero level set of a Gaussian-Process signed distance function. Touch-GS uses its uncertainty and renders the representation into view-dependent depth and uncertainty information for visual-tactile fusion.

III. METHOD

Touch-GS converts noisy, incomplete tactile measurements into a GPIS surface with uncertainty, then uses the resulting depth and variance images to supervise 3DGS training.

III. METHOD

  • The method combines monocular depth estimation, GPIS tactile conditioning and rendering, and visual-tactile fusion before training a 3DGS model.The resulting representation is modular across NeRF, point-based, and volumetric methods.Source: §III · p. 3

A. GPIS Conditioning and Rendering

  • DenseTact measurements are fused into one GPIS surface that fills gaps and represents uncertainty caused by uncovered regions and pose noise.This produces a continuous object estimate from discrete touches.Source: A. GPIS Conditioning and Rendering · p. 3

1) GPIS Conditioning:

  • Touch orientations define surface gradients by adding artificial inside and outside points with negative and positive SDF values around each contact.Contact points receive zero values, while offsets along the estimated normal receive ±δ values.Source: 1) GPIS Conditioning · p. 3
  • The conditioned GPIS uses contact, offset, and interior points with targets {0, −δ, +δ, −ϵ}, while interior slice centroids provide additional negative constraints.The interior points are computed from successive Z-slices.Source: 1) GPIS Conditioning · p. 3
  • The SDF approximation is reported to work well for reasonably shaped objects, although it is not a perfect representation of the true SDF.The covariance function uses a Matérn kernel with ν = 3/2, with distance and kernel parameters optimized during training.Source: 1) GPIS Conditioning · pp. 3–4
  • The optimized SDF rendering process is illustrated with α = .5, so each step halves the distance to the surface.This visual summarizes the variable-step rendering used for the conditioned GPIS.Source: Figure 4 · p. 4

2) GPIS Rendering:

GPIS rendering uses the SDF value to adapt ray-marching steps, accelerating surface intersection while enforcing a minimum step size.

2) GPIS Rendering:

  • Ray-marching advances by Δt determined by the current SDF value until the surface is reached or the step limit is exceeded.The SDF indicates the shortest distance to the surface, allowing empty space to be skipped.Source: 2) GPIS Rendering · p. 4
  • The step rule includes a minimum allowable increment Δtmin and a tunable parameter α controlling the variable step size.This variable-step procedure is shown in Fig. 4.Source: 2) GPIS Rendering · p. 4

3) Depth and Variance Image Generation:

The method renders GPIS depth and uncertainty, then supplements its limited background coverage with aligned monocular depth to form scene-wide supervision.

3) Depth and Variance Image Generation:

  • GPIS rendering computes per-pixel surface depth by ray-surface intersection and evaluates GP uncertainty at the intersection points.Analytic sphere intersection removes rays that cannot hit the coarse 0-level set, and the resulting surfaces are shown in Figs. 5 and 6.Source: 3) Depth and Variance Image Generation · p. 4

B. Monocular Depth Estimation and Alignment

  • GPIS provides high-quality object depth but lacks background coverage, so monocular depth estimation supplements the scene and can help localize the target object.The monocular pathway outputs an estimated absolute-scale depth map.Source: B. Monocular Depth Estimation and… · p. 4
  • Fig. 5 compares input point clouds with uncertainty-colored rendered 0-level sets, showing gap filling and higher uncertainty in interpolated regions.The final column depicts a real-world dataset.Source: Figure 5 · p. 4

1) ZoeDepth Estimation and Alignment:

ZoeDepth outputs are aligned first to sparse scene depth and then to GPIS touch depth, producing complementary depth sources in a shared frame.

1) ZoeDepth Estimation and Alignment:

  • ZoeDepth alone is insufficient for NeRF training, so its output undergoes two alignment stages using real-world sparse depth and touch depth.The approach is compatible with metric or relative monocular depth estimators.Source: 1) ZoeDepth Estimation and Alignment · p. 4
  • Sparse depth alignment jointly estimates a scene scale factor and offset for an aligned ZoeDepth map.The sparse measurements can come from an Intel RealSense or similar depth sensor.Source: 1) ZoeDepth Estimation and Alignment · p. 4
  • Robot kinematics provide camera poses without requiring COLMAP, whose successful reconstruction typically needs at least 30 feature-rich images.COLMAP can provide sparse keypoints, but the method does not depend on that route.Source: 1) ZoeDepth Estimation and Alignment · pp. 4–5
  • GPIS object alignment learns only an offset with scale fixed to 1, shifting object depth while leaving meaningful background depths unchanged.This second stage uses touch depth to refine the object portion of the monocular map.Source: 1) ZoeDepth Estimation and Alignment · p. 5

2) Vision Uncertainty Estimation:

The method estimates visual uncertainty heuristically, fuses monocular and tactile depth pixelwise under Bayesian assumptions, and uses the fused uncertainty during model training.

2) Vision Uncertainty Estimation:

  • Farther monocular depth values receive greater uncertainty, with an added constant increasing tactile priority during fusion.The visual uncertainty is based on ZoeDepth’s reduced accuracy at larger distances.Source: 2) Vision Uncertainty Estimation · p. 5

C. Depth and Touch Fusion

  • The two depth pathways produce depth images in a shared frame with corresponding variances, which are combined into one supervision pair.Training therefore weights monocular and tactile information according to their respective uncertainty.Source: C. Depth and Touch Fusion · p. 5

1) Bayesian Update:

  • Fusion treats dense monocular depth as a normally distributed prior and tactile data as an independent normally distributed measurement.The update is applied pixelwise to monocular-depth and GPIS-rendered outputs, with an example shown in Fig. 6.Source: 1) Bayesian Update · p. 5

D. Model Training

  • The training design prioritizes tighter touch constraints and improved initialization because few-view NeRFs can otherwise become trapped in local minima.These motivations support the proposed uncertainty-weighted loss and modified 3DGS initialization.Source: D. Model Training · p. 5

1) Uncertainty Weighted Depth Supervision:

Touch-GS uses uncertainty-weighted depth supervision and GPIS-based point initialization, evaluated in simulated and real-world settings with few-view inputs.

1) Uncertainty Weighted Depth Supervision:

  • Uncertainty-weighted depth supervision uses variance information to weight depth constraints, combines with color loss, and can decay its depth weight to escape local minima.The loss includes tunable weight w and optional decay λ_i+1 = βλ_i, with β between 0 and 1.Source: 1) Uncertainty Weighted Depth… · p. 5

2) Model Initialization:

  • GPIS depth from each input view is backprojected into the world frame and combined into a point cloud for 3DGS initialization.This avoids relying on COLMAP or potentially noisy depth sensors, especially for reflective scenes.Source: 2) Model Initialization · p. 5

A. Experimental Setup

  • The evaluation uses five equally spaced simulated views and real scenes with 8–151 RGB-D images, 150–500 touches, and 5,000–20,000 points per touch.The real setup uses Kinova Gen 3 RGB-D cameras and a DenseTact 2.0 sensor.Source: A. Experimental Setup · p. 6

B. Simulation Evaluation

Across simulation and real-world tests, touch–vision fusion and GPIS initialization improve reconstruction, including difficult reflective and transparent scenes, while high touch weighting can reduce reflection rendering.

B. Simulation Evaluation

  • 3DGS performs worst in simulation aside from D-MSE, whereas sparse depth improves visual and depth metrics and outperforms sparse depth across all metrics.The results show that depth priors help few-view reconstruction, while touch alignment trades slightly worse depth loss for better object depth loss.Source: B. Simulation Evaluation · p. 6
  • In simulation, GPIS point initialization combined with vision and touch produces a dramatic improvement in visual and geometric quality, while Nerfacto achieves the best result across all metrics.Touch improves object geometry, vision improves background localization, and their combination complements both roles.Source: B. Simulation Evaluation · p. 6

C. Real world Evaluation

  • On the real-world bunny, Touch-GS renders both background and bunny geometry clearly, including a tail observed through touch but not predicted by vision.The comparison uses 8 input views and 40 test views.Source: C. Real world Evaluation · p. 7
  • Across all geometric metrics, Touch-GS significantly outperforms 3DGS and Depth-3DGS on the real-world bunny reconstruction.Depth-3DGS improves overall shape over 3DGS but suffers larger outliers from incorrect bunny localization; Table IV reports Chamfer and Hausdorff distances.Source: C. Real world Evaluation · pp. 7–8, Table IV · p. 7
  • For mirror and prism scenes, Touch-GS renders flat geometry-respecting surfaces despite holes in vision-based reconstructions, but high touch weights hinder reflection and see-through rendering.The authors identify this trade-off as future work while describing the accurate geometries as an initial step for robotic manipulation.Source: Figure 7 · p. 7, C. Real world Evaluation · p. 8

V. CONCLUSION

Touch-GS fuses visual and tactile data to produce 3DGS scenes, improving over standard and depth-supervised baselines. The approach balances coarse RGB information with fine tactile data while pointing toward richer dynamic object representations.

  • Touch-GS produces 3DGS scenes by fusing visual-tactile data and improves qualitatively and quantitatively over standard and depth-supervised 3DGS baselines.Source: §V · p. 8
  • The method addresses the balance between coarse RGB data and fine tactile data for robotic scene representation.Source: §V · p. 8
  • Future work aims to extend the representation to dynamic objects with deformability and frictional properties, enabling a fuller digital twin.Source: §V · p. 8
Loading 2403.09875v3…