PaperlayerView exact revision on arXiv

Source-linked AI summary

ShotcreteDepth: A Bi-modal Dataset for Robust Robotic Depth Perception in Shotcrete Construction Environments

Jakub Gregorek, Lars Arnold Dethlefsen, Patrick Schmidt, Mads Essenbæk, Jonas Flink Bentzen, Lazaros Nalpantidis

arXiv:2606.23152v1cs.RO

TL;DR

Reliable depth perception in shotcreting environments is limited by scarce data and challenging conditions. ShotcreteDepth addresses this gap with a bi-modal dataset and lightweight LiDAR annotation tool, showing that tested approaches can perform depth perception under these conditions while trading accuracy against computational efficiency.

  • Problem

    Depth-perception research for niche shotcreting environments lacks data despite challenging construction-site conditions that impair reliable perception.

  • Method

    The paper releases a bi-modal shotcreting dataset for depth-perception development and evaluation alongside a lightweight tool for annotating unreliable LiDAR measurements.

  • Results

    Nine tested approaches across stereo matching, depth completion, and depth estimation each exhibited merits, while larger deep-learning models generally improved accuracy at the cost of runtime.

  • Takeaways & Limitations

    ShotcreteDepth supports evaluation of depth-perception methods in high-turbidity shotcreting environments, including comparisons between efficient and more accurate approaches.

Abstract

from arXiv · show

We introduce ShotcreteDepth, a bi-modal dataset from the construction domain that captures both an active shotcreting process and general construction environments. The dataset comprises stereo RGB imagery and LiDAR point clouds acquired under harsh real-world conditions, including high turbidity and poor illumination. Such conditions adversely affect sensor measurements, leading to incomplete and noisy observations that pose significant challenges for perception systems in autonomous applications. Alongside the dataset, we release a lightweight annotation tool designed for time-efficient labeling of LiDAR point clouds. ShotcreteDepth consists of 11,252 temporally synchronized data samples, of which 220 are annotated for evaluation purposes. The dataset supports research in stereo matching, depth completion, and depth estimation under conditions that closely reflect the operational complexities found in industrial settings. Project repository: https://github.com/dtu-pas/shotcrete-depth

I. INTRODUCTION · II. RELATED WORK · A. Relevant Datasets

ShotcreteDepth addresses the shortage of depth-perception data for harsh shotcreting environments through a bi-modal dataset and a lightweight LiDAR annotation tool. It supports evaluation of stereo matching, depth completion, and depth estimation while complementing existing construction datasets focused on other tasks.

  • I. INTRODUCTION: ShotcreteDepth is a bi-modal dataset capturing a shotcreting environment for developing and evaluating stereo matching, depth completion, and depth estimation methods.
  • I. INTRODUCTION: Shotcreting reinforces unstable underground structures, builds complex geometries, and repairs damaged structures, but its application remains laborious despite extensive mechanization.
  • I. INTRODUCTION: Reliable construction-site perception systems, particularly for niche applications like shotcrete, are hindered by a lack of data.
  • I. INTRODUCTION: A lightweight annotation tool enables users to label unreliable depth measurements in 3D point clouds.
  • I. INTRODUCTION: The paper contributes a shotcreting dataset emphasizing depth perception and evaluates stereo, depth completion, and depth estimation baselines to characterize it.
  • A. Relevant Datasets: Existing construction-related depth datasets address SLAM, segmentation, place recognition, pose estimation, 3D reconstruction, object detection, and depth estimation, while shotcreting datasets target segmentation, structural performance, and deformations.

B. Depth Estimation · C. Depth Completion · D. Stereo Matching

The paper situates depth estimation, depth completion, and stereo matching as complementary depth-perception tasks. It highlights zero-shot monocular estimation, sparse-depth densification guided by images, and stereo correspondence estimation, alongside representative methodological trends.

  • B. Depth Estimation: Depth estimation predicts scene depth from a monocular image, with this work focusing mainly on methods evaluated in zero-shot settings.Ranftl et al. pioneered zero-shot monocular depth estimation.
  • B. Depth Estimation: Many recent depth estimation models use Vision Transformer backbones initialized from DINOv2, alongside pretrained ConvNext and ResNet models.The passage identifies DINOv2-initialized ViT backbones and pretrained ConvNext and ResNet as model foundations.
  • C. Depth Completion: Depth completion densifies sparse depth using guidance from a monocular image.Zero-shot depth completion can exploit pretrained priors from depth estimators or stereo matching architectures.
  • C. Depth Completion: Recent depth completion methods formulate inference as test-time optimization using low-rank adaptation or visual prompt tuning.Ke et al. applies both low-rank adaptation and visual prompt tuning, while prompt tuning and low-rank adaptation are also explored separately by other methods.
  • C. Depth Completion: Diffusion-based depth completion methods iteratively guide the diffusion process with sparse depth.The passage states that faster inference than test-time optimization can be achieved by fine-tuning the diffusion model.
  • D. Stereo Matching: Stereo matching finds correspondences between stereo-camera images to estimate the observed scene’s depth.Semi-Global Matching became an industrial standard because of its favorable accuracy-efficiency trade-off.
  • D. Stereo Matching: Deep learning methods have dominated stereo matching, including cost-volume aggregation and recurrent disparity refinement.Cost-volume aggregation is often memory expensive, whereas other methods perform recurrent disparity refinement.

III. DATASET · A. Overview · B. Environment

ShotcreteDepth is a bi-modal dataset for depth perception in shotcreting and general construction environments under high turbidity and poor illumination. It contains 11,252 synchronized samples, including 220 diverse annotated evaluation samples, collected in confined, artificially lit spaces with rapidly accumulating shotcrete particles.

  • A. Overview: The dataset covers scenes before, during, and after shotcreting, together with general construction environments.
  • A. Overview: It is bi-modal and intended to support development and evaluation of depth perception systems.
  • A. Overview: 11,252 temporally synchronized data samples comprise the dataset.
  • A. Overview: 220 samples are annotated for evaluation, with selection maximizing diversity by skipping similar consecutive frames.
  • A. Overview: The dataset targets depth perception under high turbidity and far-from-optimal illumination.
  • B. Environment: The construction site used artificial lighting exclusively because it was sealed from the outside, while shotcrete particles accumulated rapidly in the air.
  • B. Environment: Construction environments are often confined, as shotcreting is commonly performed in tunnels or mines.

C. Sensors & Modalities · D. Calibration & Synchronization · E. Influence of Turbidity

ShotcreteDepth combines stereo RGB imagery and LiDAR depth using calibrated, time-synchronized sensors. Turbidity particularly degrades LiDAR measurements through missing data, noise, and visible dust clouds.

  • C. Sensors & Modalities: The dataset uses a Roboception rc visard 160c stereo camera with a 4mm lens and a Velodyne PUCK LiDAR.The camera provides 1280 × 960 RGB images and 640×480 disparity and confidence maps; the LiDAR measures depth up to 100m with typical accuracy up to ±3cm.
  • C. Sensors & Modalities: The stereo camera produces disparity and confidence maps using a proprietary Semi-Global Matching implementation running on its Nvidia Tegra K1.The disparity and confidence maps have resolution 640×480.
  • D. Calibration & Synchronization: Intrinsic camera parameters were estimated with Roboception’s built-in calibration tool, while camera–LiDAR extrinsics were obtained using Matlab calibration tools.The extrinsic calibration used Matlab’s Camera LiDAR and Camera Calibration.
  • D. Calibration & Synchronization: LiDAR timestamps were synchronized using GPS-derived PPS signals and NMEA messages, while the camera clock used PTP with the GPS-synchronized acquisition computer.This setup aligns the camera clock with the acquisition computer’s system clock, which was also synchronized with GPS time.
  • E. Influence of Turbidity: Turbidity makes stereo matching more difficult, but LiDAR appears more severely affected and can degrade even when visibility remains reasonable.Shotcrete particles dispersed in the air degrade LiDAR depth-measurement quality.
  • E. Influence of Turbidity: LiDAR degradation during shotcreting has three forms: missing data, noisy measurements, and dust clouds visible in point clouds.Particles may partially or entirely blind the LiDAR, increase point-cloud noise, and remain mostly translucent to the camera and human eyes.

F. Point Cloud Annotation Tool · G. Removing Occlusions · IV. EXPERIMENTS

The paper presents a LiDAR point-cloud annotation tool for labeling shotcrete dust clouds and a sliding-window method for removing occluded points. Experiments use ShotcreteDepth to evaluate stereo, depth-completion, and depth-estimation baselines and characterize the dataset.

  • F. Point Cloud Annotation Tool: The annotation tool enables manual labeling of LiDAR points as “kept by user” or “removed by user”.Programmatic annotations use the corresponding “kept by algorithm” and “removed by algorithm” labels.
  • F. Point Cloud Annotation Tool: Its interface provides 3D LiDAR visualization and point-cloud overlay on the left camera image.The released visualization distinguishes kept points from the dust cloud excluded from evaluation data.
  • F. Point Cloud Annotation Tool: The released tool manually annotates shotcrete dust clouds observed by LiDAR.The interface is designed around assigning labels to individual point-cloud points.
  • G. Removing Occlusions: Occluded LiDAR points are filtered using a sliding window over depth points projected onto the image plane.Points farther than the nearest point by a threshold are removed.
  • G. Removing Occlusions: 10 × 50 pixels and 0.5m are the selected sliding-window size and distance threshold for filtering.The window is enlarged because the dataset has fewer LiDAR scan lines than the referenced method.
  • G. Removing Occlusions: User-marked removals are ignored during filtering to avoid removing valid points near dust clouds.Points removed by the filtering method are marked as “removed by algorithm”.
  • IV. EXPERIMENTS: Experiments test stereo, depth completion, and depth estimation baselines using ShotcreteDepth to elucidate the dataset’s characteristics.The authors used the dataset themselves for these evaluations.

A. Experimental Setup

The evaluation used fixed-resolution inference with annotated and filtered LiDAR as ground truth, while depth completion and depth estimation relied on 500 stereo-derived depth points. Stereo predictions were converted between disparity and depth using the calibrated stereo setup.

  • 640 × 480 pixel inference outputs were upscaled to 1280 × 960 before metric computation against annotated and filtered LiDAR ground truth.
  • 500 uniformly sampled stereo-matching depth points were provided to depth completion models and used to align affine-invariant depth predictions by scale a and shift b.The alignment minimized least square error.
  • Stereo evaluation converted ground-truth depth to disparity, while stereo disparity was converted to depth for sampling sparse depth maps in depth completion.The conversion used the stereo baseline and camera-calibrated focal length.

B. Metrics

The paper evaluates depth estimation, depth completion, and stereo matching with task-specific error metrics, alongside runtime and model size. It also assesses boundary accuracy and completeness because RMSE and MAE may not fully reflect quality with sparse ground truth.

  • Depth estimation: Depth estimation models are evaluated using absolute relative error (REL) and the fraction of pixels with δ < 1.25.N denotes the number of image pixels; d_i and g_i are predicted and ground-truth pixels.
  • Depth completion: Depth completion models are evaluated using root mean squared error (RMSE) and mean absolute error (MAE), reported in meters.The paper notes that these metrics alone may not fully capture quality when ground truth is sparse.
  • Boundary evaluation: Boundary quality is additionally assessed with Pseudo Depth Boundary Error (PDBE) accuracy (Eacc) and completeness (Ecomp).Ground-truth edges for the PDBE metrics are extracted from stereo matching.
  • Stereo matching: Stereo matching methods are evaluated using endpoint error (EPE) in pixels and D1, the percentage of elements exceeding both 3 pixels and 5% of ground-truth disparity.EPE uses the MAE formulation in disparity space.
  • Efficiency: Runtime is reported in seconds for all evaluated methods, measured on an Nvidia GeForce 4090, while model sizes are given in millions of parameters.These measurements provide computational descriptors alongside task-specific accuracy metrics.

C. Models · D. Stereo Matching

The study evaluates selected stereo matching and depth models using specified pretrained checkpoints and camera settings. Neural-network methods are reported as more robust than embedded-hardware-optimized Semi-global Matching under the dataset’s challenging conditions.

  • C. Models: FoundationStereo was evaluated with the ViT-Large checkpoint “23-51-11”.
  • C. Models: StereoAnywhere used a checkpoint pretrained on SceneFlow.
  • C. Models: RAFT-Stereo used the Middlebury checkpoint recommended for in-the-wild images.
  • C. Models: Marigold-SSD used a model trained on density range [0.16%, 5%], while Marigold-DC used the Marigold v1.0 checkpoint.
  • C. Models: The rc_visard camera used default settings except “Quality” was set to “Full” with “Double-Shot” and “Static” enabled.
  • D. Stereo Matching: The selected stereo matching methods were evaluated quantitatively in Tab. II and compared qualitatively in Fig. 5.
  • D. Stereo Matching: Semi-global Matching is optimized for embedded hardware but often produces incomplete depth maps.
  • D. Stereo Matching: Neural-network methods are more computationally demanding but more robust in the dataset’s challenging environmental conditions.

E. Depth Completion · F. Depth Estimation

The study evaluates depth completion from stereo-derived sparse maps and depth estimation under aligned or unaligned metric conditions. It compares multiple completion and estimation methods using quantitative tables and qualitative visualizations.

  • E. Depth Completion: Depth completion uses maps sampled from stereo matching, with Stereo Anywhere selected as the source because it achieves the lowest EPE and D1.Stereo Anywhere also supplies ground-truth edges for PDBE metrics because its output is visually clean.
  • E. Depth Completion: 500 depth points were sampled uniformly for the depth completion evaluation.
  • F. Depth Estimation: Three depth estimation models are evaluated: Depth Anything v3, Marigold-E2E, and MoGe-2.
  • F. Depth Estimation: Sparse maps from depth completion align the shift and scale of affine-invariant estimates from Depth Anything v3 and Marigold-E2E.MoGe-2 predicts metric depth and is evaluated both with and without shift-and-scale adjustment.
  • F. Depth Estimation: Stereo matching comparisons include RAFT-Stereo, FoundationStereo, Stereo Anywhere, and the proprietary SGM implementation on the Roboception rc visard 160 camera.Timing uses an Nvidia GeForce 4090, while camera stereo-matching runtime is represented by the camera FPS.
  • E. Depth Completion: Three depth completion methods are evaluated: Marigold-DC, Marigold-SSD, and VPP4DC.Marigold-DC reports a single-run value and an ensemble-of-10-runs value.
  • E. Depth Completion: Qualitative results compare the depth completion methods visually.
  • F. Depth Estimation: Qualitative depth estimation results are presented for the evaluated methods.

V. DISCUSSION AND CONCLUSION

ShotcreteDepth enables depth-perception evaluation in challenging shotcreting environments through a bi-modal dataset and lightweight LiDAR annotation tool. Experiments show that multiple tested approaches offer task-specific merits, while accuracy and computational efficiency involve a trade-off, and sensor fusion remains a potential direction.

  • Experimental findings: 3 stereo matching, 3 depth completion, and 3 depth estimation approaches each exhibited merits in their corresponding tasks.The experiments demonstrate that depth perception is possible in challenging shotcreting conditions.
  • Experimental findings: Near-real-time operation is feasible with computational efficiency, but this comes at the expense of output accuracy.The passage contrasts computationally efficient operation with the accuracy of large deep learning-based models.
  • Contributions: ShotcreteDepth is a bi-modal dataset for evaluating and developing depth-perception methods in the high-turbidity shotcreting environment.The work used the dataset to test 9 state-of-the-art deep learning methods.
  • Contributions: A lightweight 3D point-cloud annotation tool was developed to remove LiDAR-observed dust clouds from evaluation data.The tool supports annotation under the dataset’s challenging construction conditions.
  • Future direction: Fusing stereo cameras and LiDARs could achieve the best depth perception, but it would require an additional source of depth for evaluation.The sensors are fundamentally distinct and better suited for different scenarios.
Loading 2606.23152v1…