Source-linked AI summary
VisTa3D: A Dataset and Benchmark for Thin Object Reconstruction from Vision, Tactile, and 3D Point Clouds
Shania Guo, Yeongsik Seo, Andrew Fu, Mei Hao, Iris Xia, Jiwon Jenny Lee, Xinyi Mary Xie, Hyoungseob Park, Aaron Dollar, Alex Wong
TL;DR
Thin objects remain difficult for 3D reconstruction because they occupy little image and point-cloud space, while existing benchmarks provide limited focused evidence. The paper introduces VisTa3D, a synchronized visual-range-tactile dataset and benchmark with a three-modality baseline, finding that existing methods struggle and tactile fusion improves reconstruction. The dataset contains 387 scenes covering 70 objects across 17 environments.
Problem
Thin objects are underrepresented and difficult to reconstruct because they occupy limited space in RGB images and 3D point clouds.
Method
The paper constructs VisTa3D with synchronized visual, range, tactile, inertial, pose, calibration, and scanned-object ground-truth data, and introduces a three-modality reconstruction baseline.
Results
Existing methods exhibit pronounced errors on thin structures, while tactile integration improves reconstruction with synthetic training and improves further when real data are added.
Takeaways & Limitations
Tactile data provide useful cues for recovering thin structures that are difficult to infer from RGB images and sparse depth maps.
Takeaways & Limitations
The dataset excludes reflective, specular, and semi-transparent objects, remains moderate in scale, and assumes static scenes without moving or deformable objects.
Abstract
from arXiv · showhide
State-of-the-art 3D reconstruction models, whether from visual, range, or both, tend to underperform on thin objects. This is partially due to the small amount of space such objects occupy in RGB images and in 3D point clouds. To test the extent of their errors, we collected the first thin object dataset comprising of synchronized RGB images, depth maps, and tactile response maps, where each frame is associated with inertial measurements, camera pose and calibration, and groundtruth depth and segmentation maps obtained from laser scanning of thin objects. We hypothesize that tactile data can aid in the reconstruction of thin objects as their response maps provide local shape and deformation information. Our dataset, termed VisTa3D, comprises of 387 scenes covering 70 thin objects over 17 environments. We benchmarked current 3D reconstruction models on VisTa3D and found that, indeed, they exhibit low fidelity on thin objects. To test if tactile data can help, we introduce the first visual-range-tactile 3D reconstruction model as a baseline. Code and data: https://huggingface.co/datasets/shaniaguo/VisTa3D.
1 Introduction
Thin objects are difficult to reconstruct because they occupy few pixels and produce sparse range returns, causing occlusion and correspondence loss. VisTa3D benchmarks this challenge with synchronized multimodal data and a tactile-fusion baseline.
- Challenge: Thin objects occupy few RGB pixels and yield sparse range returns, making reconstruction challenging at commercial sensor resolutions.Limited sampling is especially problematic at longer distances.
- Challenge: Existing benchmarks do not focus on thin structures, while current reconstruction models exhibit pronounced errors on them.The evaluation spans monocular depth estimation, multi-view stereo, depth completion, and neural rendering methods.
- Motivation: Tactile sensing provides local surface-shape and deformation measurements that remain effective when visual occlusion or depth-sensor range limits obstruct thin objects.These measurements are presented as complementary to visual and range inputs.
- Dataset: VisTa3D introduces synchronized RGB, depth, tactile, inertial, pose, calibration, ground-truth depth, and segmentation data derived partly from high-fidelity laser scans.The benchmark also includes synthetic data for method pretraining.
- Dataset: 387 scenes cover 70 thin objects across 17 real and synthetic environments.The dataset is organized as a benchmark for thin-object reconstruction.
- Baseline: A baseline integrating vision, range, and tactile modalities improves thin-object reconstruction with synthetic training and improves further when real data are added.The model uses a tactile integration inspired by depth-completion networks.
- Contributions: The benchmark evaluates 11 existing methods under 6 evaluation modes and introduces a vision-tactile-range reconstruction baseline.Its curation pipeline includes high-fidelity laser scanning and postprocessing for associated ground truth.
2 Related Work
Prior work addresses thin structures mainly through 2D segmentation, specialized reconstruction methods, or tactile fusion with other modalities. VisTa3D instead provides a multimodal 3D benchmark and shows that existing methods remain limited on thin structures.
- Reconstruction Methods: Monocular depth estimation methods increasingly use large-scale datasets and foundation models, but the cited work does not establish thin-object performance here.The passage describes general scene reconstruction advances rather than thin-structure evaluation.
- Reconstruction Methods: Multi-view stereo faces correspondence ambiguity on thin objects, and cited recent methods have not been specifically evaluated on thin objects.Thin structures remain difficult because photometric matching can be ambiguous.
- Reconstruction Methods: Depth completion is especially challenging for thin objects because range sensors return few points, motivating propagation and sparse-depth scaffolding methods.The related methods include spatial, non-local, and dynamic propagation approaches.
- Reconstruction Methods: Novel-view synthesis uses volumetric and multiscale representations, but thin structures may disappear because of z-aliasing.Prior work addresses aliasing with multiscale, grid-based, and prefiltering techniques.
- Tactile Sensing and Reconstruction: Tactile reconstruction has been studied alone and alongside vision, depth, or robot kinematics, including implicit signed-distance representations.These studies motivate combining tactile observations with complementary signals.
- Thin-Structure Reconstruction: Thin-structure reconstruction methods address insufficient sampling, view correspondence, and self-occlusion using surface, curve, tubular, structural-prior, and fusion strategies.Approaches include contour connections, robust meshes, silhouette probabilities, and densely sampled light fields.
- Thin Object Datasets: Existing thin-structure datasets predominantly target 2D image segmentation rather than multimodal 3D reconstruction.Examples include large-scale thin-object, powerline, and UAV segmentation datasets.
- VisTa3D: VisTa3D differs by offering visual, range, inertial, and tactile data with poses, calibration, real and synthetic scenes, and scanned-object ground truth.The benchmark evaluates state-of-the-art reconstruction methods and reports continuing limitations on thin structures.
3 Data Collection Methodology
VisTa3D combines synchronized visual, depth, inertial, tactile, and scanned 3D data with a multistage pipeline for producing registered, refined ground truth. The methodology addresses noisy depth, object segmentation, point-cloud alignment, object insertion, and reprojection.
- Sensor Platform and Tactile Collection: The platform captures RGB images, depth maps, and inertial measurements with an Intel RealSense D455, tactile response maps with DIGIT, and high-precision point clouds with a 3DMakerPro Moose scanner.RGB and depth share resolution, and depth maps are registered to the RGB frame.
- Vision, Inertial, Range Data Collection: RGB, depth, and inertial streams are recorded during near-complete 360° trajectories around each object, with inertials synchronized by nearest-neighbor timestamp matching.Scenes use a single trajectory around an object placed on a support surface.
- Pose Estimation: Camera-to-world poses are estimated using ORB-SLAM3 in RGB-D mode and manually inspected, while the recorded inertials are released but not used for pose estimation.Sequences with tracking failures are excluded or reprocessed, and the trajectories support subsequent point-cloud and ground-truth stages.
- Ground-Truth Generation: The refined point cloud replaces noisy object points, is reprojected using camera calibration and inverse poses, and yields ground-truth depth maps and projection masks.A z-buffer preserves nearest object depth inside the mask, while a k-median buffer suppresses floating artifacts outside it.
- Scanning and Point-Cloud Registration: A high-fidelity structured-light scan is registered to the aggregated raw object cloud through segmentation, backprojection, sparse initialization, and iterative refinement.Object masks from RGB images select depth pixels, which are backprojected using camera intrinsics and transformed into world coordinates before registration.
4 Dataset, Benchmark, and a Baseline
VisTa3D combines multimodal thin-object data with a benchmark spanning reconstruction paradigms and a visual-range-tactile baseline. The benchmark evaluates depth estimation and novel view synthesis, with thin-object-focused evaluation exposing persistent difficulties.
- Dataset: VisTa3D provides synchronized RGB, depth, tactile, inertial, camera, calibration, ground-truth depth, and segmentation data, with scanned objects in point-cloud and mesh forms.The dataset supports thin-object reconstruction and related tasks.
- Dataset: 162 synthetic scenes contain 18 thin objects across nine tabletop configurations, with 500 frames per scene and eight manually selected tactile contact locations.Synthetic scenes include RGB, depth, camera, IMU, masks, tactile maps, and meshes.
- Dataset: 225 real-world scenes cover 52 thin objects across eight indoor and outdoor environments with varied backgrounds, lighting, and motion trajectories.The real-world collection includes wires, cables, figurines, plants, and office or household objects.
- Benchmark: The benchmark evaluates MDE, MVS, MDC, and NVS methods, using real data for benchmarking and synthetic data for pretraining.Depth estimation is primary and novel view synthesis is secondary.
- Baseline: Tactile-DC is a visual-range-tactile depth-completion baseline that separately encodes inputs, fuses RGB with sparse depth, and projects tactile features onto the image plane.Projection uses contact locations together with camera intrinsics and poses before decoding a depth map.
5 Experiments
Experiments show that existing reconstruction methods struggle particularly with thin-object regions, while tactile-visual-range fusion substantially improves depth reconstruction. Novel-view synthesis methods can render plausible views, but sparse views, occlusions, and incomplete depth remain challenging.
- Depth Estimation Results: Thin-object-focused evaluation generally lowers A1–A3 scores, showing that thin-object pixels are harder to reconstruct than the full image.Errors are higher near thin structures and object boundaries.
- Depth Estimation Results: OmniDC performs best among existing baselines, yet still shows considerable errors around thin structures and object boundaries.Scaling or linear fitting does not consistently improve it because its main failure mode is thin-structure recovery rather than global scale mismatch.
- Depth Estimation Results: MDE models improve substantially after scale alignment but remain far below MDC, with A1 below 0.10 for UniDepth V1/V2 and below 0.09 for DepthAnything V1/V2.Their remaining errors indicate difficulty recovering thin structures beyond global scale mismatch.
- Depth Estimation Results: MVS methods preserve more global scene structure than MDE but still miss or over-smooth thin details, even when sparse metric depth is available.VGGT produces smoother outputs, while MapAnything shows noisy or incomplete estimates around thin objects and foreground-background transitions.
- Depth Estimation Results: Tactile-DC achieves the best performance, with A1/A2/A3 of 0.844/0.900/0.920 generally and 0.719/0.811/0.852 for object evaluation.It consistently outperforms OmniDC across protocols, especially in thin-object-focused evaluation.
- NVS Results: Nerfacto achieves the best novel-view synthesis performance, while sparse input views and occlusions remain challenging for current methods.Depth supervision does not consistently help when sensor depth is noisy or incomplete around thin structures.
6 Discussion and Limitations
The benchmark excludes some object types because the sensing hardware performs unreliably on reflective, specular, or semi-transparent surfaces. Its moderate scale, static-scene assumption, and lack of task-specific finetuning also limit coverage and evaluation scope.
- Limitations: Reflective, specular, and semi-transparent objects are excluded because the Intel RealSense and 3DMakerPro Moose perform unreliably on them.This constrains the real-world object selection.
- Limitations: The dataset remains moderate in scale and does not cover the full diversity of real-world thin-object materials, thicknesses, and shapes.
- Limitations: The benchmark assumes static 3D scenes and does not consider moving or deformable objects.
- Limitations: MDE and MVS may benefit from task-specific finetuning that the benchmark does not explore.