Source-linked AI summary
Masked Depth Modeling for Spatial Perception
Bin Tan, Changjiang Sun, Xiage Qin, Hanat Adai, Zelin Fu, Tianxiang Zhou, Han Zhang, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue
TL;DR
RGB-D cameras can lose dense depth under appearance ambiguities, motivating a method that treats missing measurements as informative masks. LingBot-Depth applies Masked Depth Modeling with RGB context and valid depth, achieving strong depth estimation results and downstream utility across tracking and grasping.
Problem
RGB-D cameras provide pixel-aligned metric depth but suffer corrupted and missing measurements under challenging conditions such as specular reflections and textureless surfaces.
Method
Masked Depth Modeling treats missing depth as natural masks and learns joint RGB-depth representations to reconstruct dense geometry, using scalable real and synthetic training data.
Results
LingBot-Depth achieves strong performance in depth completion and monocular depth estimation, improves a FoundationStereo monocular prior, and supports downstream tracking and grasping.
Takeaways & Limitations
The model provides a spatial perception prior with aligned RGB-depth representations that transfer to video depth estimation, 3D tracking, and open-world grasping.
Abstract
from arXiv · showhide
Spatial visual perception is a fundamental requirement in physical-world applications like autonomous driving and robotic manipulation, driven by the need to interact with 3D environments. Capturing pixel-aligned metric depth using RGB-D cameras would be the most viable way, yet it usually faces obstacles posed by hardware limitations and challenging imaging conditions, especially in the presence of specular or texture-less surfaces. In this work, we argue that the inaccuracies from depth sensors can be viewed as "masked" signals that inherently reflect underlying geometric ambiguities. Building on this motivation, we present LingBot-Depth, a depth completion model which leverages visual context to refine depth maps through masked depth modeling and incorporates an automated data curation pipeline for scalable training. It is encouraging to see that our model outperforms top-tier RGB-D cameras in terms of both depth precision and pixel coverage. Experimental results on a range of downstream tasks further suggest that LingBot-Depth offers an aligned latent representation across RGB and depth modalities. We release the code, checkpoint, and 3M RGB-depth pairs (including 2M real data and 1M simulated data) to the community of spatial perception.
1 Introduction
RGB-D sensors provide real-time, pixel-aligned geometry but produce corrupted and missing depth in visually ambiguous conditions. LingBot-Depth treats these failures as natural masks, reconstructs depth from RGB and valid measurements, and supports downstream spatial tasks.
- Motivation: RGB-D cameras suffer missing and corrupted depth on low-texture surfaces, specular reflections, and complex lighting conditions.These failures violate the requirement for dense, pixel-aligned geometry.
- Approach: MDM uses naturally missing depth measurements as masks rather than discarding sensor failures as noise.The masks reflect geometric and appearance ambiguities and create a harder reconstruction challenge than random dropout.
- Approach: MDM unifies monocular depth estimation and depth completion through masking, producing metric-scale, pixel-aligned, dense depth maps from RGB-D inputs.Masking all depth tokens yields monocular estimation, while masking invalid tokens enables depth completion using valid readings and visual cues.
- Data: The training pipeline combines 1M synthetic samples with 2M real captures containing synchronized RGB images, sensor depth, and stereo pairs.It uses synthetic and real-world data streams to support large-scale MDM training.
- Results: LingBot-Depth achieves competitive depth completion and metric monocular depth estimation while supporting 3D tracking and dexterous grasping.The reported evaluations include stronger monocular priors, zero-shot video depth generalization, and robotic applications.
2 Masked Depth Modeling
Masked Depth Modeling processes RGB and depth as spatially aligned tokens, masks depth according to sensor validity, and reconstructs dense geometry with a convolutional decoder. Its attention visualizations and data pipeline support cross-modal spatial learning at scale.
- Masked Depth Modeling: MDM shifts masked image modeling from RGB appearance reconstruction to depth-map prediction using RGB-D inputs and a ViT-Large encoder.A ConvStack decoder is used for dense geometric reconstruction instead of the shallow Transformer decoder in vanilla MAE.
- Separated Patch Embedding: Separate RGB and depth patch embeddings produce spatially aligned token sequences that integrate appearance context with geometric cues.The model uses patch size 14 and jointly represents near–far relationships, coplanarity, and spatial continuity.
- Positional Embeddings: MDM adds shared 2D spatial and modality-specific positional information to distinguish RGB and depth tokens at corresponding image locations.The positional design encodes both image-plane location and modality identity.
- Joint Embedding: Sensor-induced masking trains a joint embedding from RGB tokens and unmasked depth tokens, enabling depth reasoning under incomplete observations.Missing depth correlates with material, lighting, reflective, and textureless conditions.
- Masking Strategy: Entirely missing depth patches are always masked, partially invalid patches receive a 0.75 masking probability, and valid patches fill any remaining target mask quota.The masking decision is made at patch level, allowing imperfect but informative depth data to support training.
- Decoder: The encoder retains full RGB tokens and unmasked depth tokens, while the decoder discards latent depth tokens and reconstructs depth from contextual representations.The ConvStack uses retained spatially distributed contextual tokens enriched with global context.
- Attention Visualization: Depth-to-RGB attention visualizations show different query patches attending to distinct spatially corresponding RGB regions across an aquarium and an indoor shelf.The visualization supports fine-grained cross-modal geometric–appearance associations.
- Data Curation: The curation pipeline contains 2.1M real-captured samples and 1.0M simulated captures, paired with RGB-D inputs and ground-truth depth maps.The real and simulated streams provide scalable supervision for training.
3 Data Curation Pipelines
The paper curates realistic RGB-D training data through synthetic simulation and scalable real-world capture, preserving sensor imperfections and broad scene diversity. These streams are combined with open-source datasets for large-scale masked depth modeling.
- Motivation: RGB-D data scarcity motivates a curation pipeline that preserves naturally occurring missing-depth patterns from real-world sensing.Existing datasets often avoid challenging conditions or provide near-perfect depth, limiting masked depth modeling.
- Synthetic Data Pipeline: The synthetic pipeline simulates active-camera artifacts by processing rendered stereo pairs with semi-global matching to produce sensor-like depth.Each sample includes aligned RGB, perfect depth, stereo images, ground-truth disparity, and simulated sensor depth.
- Synthetic Data Pipeline: The synthetic corpus renders 10 million samples from 442 indoor scenes, with RGB and perfect depth at 960 × 1280 resolution.Stereo pairs and sensor depth are generated at 720 × 960 before sensor depth is upsampled to the target resolution.
- Real-World Data Pipeline: The real-world capture protocol spans residential, commercial, public, and specialized indoor environments to broaden scene coverage.The synthetic scenes emphasize high-fidelity scene-level layouts rather than isolated-object compositions.
- Real-World Data Pipeline: The real-world pipeline uses a modular 3D-printed rig and unified interfaces to collect RGB-D data from multiple commercial cameras.The system supports active stereo cameras such as Intel RealSense and Orbbec Gemini, plus passive stereo ZED systems.
4 Experiments
Experiments evaluate LingBot-Depth for depth completion, monocular depth estimation, and stereo matching initialization. Across these settings, the model shows strong benchmark performance and improves downstream representations and convergence.
- Experimental Setup: The experiments evaluate depth completion under block-wise masking and sparse SfM inputs, alongside monocular estimation and FoundationStereo initialization.The depth-completion study uses standard benchmarks and two corruption protocols, while monocular evaluation spans diverse indoor, outdoor, synthetic, and real-world datasets.
- Depth Completion: RMSE decreases by over 40% relative to PromptDA on indoor benchmarks under extreme block-wise masking.LingBot-Depth also achieves the lowest errors across all difficulty levels on DIODE-Outdoor.
- Depth Completion: RMSE decreases by 47% indoors and 38% outdoors on ETH-SfM under sparse SfM inputs compared with the best baseline.The method achieves state-of-the-art results on both ETH-SfM splits in the more challenging sparse-input protocol.
- Monocular Depth Estimation: LingBot-Depth initialization produces consistent improvements across all monocular-depth benchmarks compared with DINOv2 initialization.The encoder is used without depth inputs at inference time, indicating that spatial representations are retained for RGB-only depth reasoning.
- FoundationStereo with MDM Pretraining: At epoch 5, FoundationStereo with the proposed encoder reaches HAMMER EPE 0.27 versus 0.46 for the vanilla baseline.At epoch 15, it reports Middlebury EPE 0.75, HAMMER EPE 0.17, and FSD EPE 0.40, achieving the best or comparable results across benchmarks.
- FoundationStereo with MDM Pretraining: The MoGe-based FoundationStereo variant shows early instability, including HAMMER EPE 2.53 and Booster EPE 2.84 at epoch 5.These errors persist through epoch 10, contrasting with the faster convergence of the proposed initialization.
5 Extensions and Applications
The paper extends LingBot-Depth to video completion, camera and object tracking, and dexterous grasping. Refined depth improves temporal consistency, tracking trajectories, and grasp success in challenging transparent and reflective scenes.
- Video Depth Completion: Image-only pretraining still reduces spatial-temporal inconsistency in 30 FPS, 640 × 480 videos captured in challenging scenes.The evaluated scenarios include a glass lobby, rowing machine, gym, and aquarium tunnels.
- Video Depth Completion: Compared with raw Orbbec and ZED depth, LingBot-Depth fills missing regions around transparent and reflective surfaces while preserving fine structures.The model also maintains temporal consistency without explicit temporal modeling or video-specific training.
- 3D Tracking: Refined depth yields smoother and more accurate SpatialTrackerV2 camera trajectories than raw sensor depth in glass-heavy indoor scenes.Raw sensor depth suffers severe drift because of missing depth regions.
- 3D Tracking: SpatialTrackerV2 tracks coherent 3D trajectories for moving objects using the refined depth.Rainbow-colored trails in the visualizations indicate coherent motion patterns across four dynamic scenarios.
- Dexterous Grasping: The transparent storage box has a 50% grasping success rate with refined depth versus N/A with raw depth.The evaluation covers four challenging transparent and reflective objects across 20 trials per object.
- Dexterous Grasping: Refined depth improves grasping success rates across all four evaluated challenging objects compared with raw sensor depth.The grasping pipeline converts completed depth into point clouds and predicts dexterous hand poses with a diffusion policy.
6 Conclusion
LingBot-Depth addresses missing RGB-D measurements by treating them as natural masks and demonstrates utility across depth estimation, video streams, and downstream spatial applications.
- LingBot-Depth treats missing RGB-D measurements as natural masks reflecting appearance ambiguities in depth imaging systems.
- The model is pretrained on 3 million self-curated RGB-D samples and open-source depth datasets using a large Vision Transformer.
- LingBot-Depth performs strongly on depth completion and monocular depth estimation and provides a monocular depth prior for FoundationStereo.
- The model produces high-quality video depth streams with strong spatial and temporal consistency.
- Downstream evaluations validate LingBot-Depth in 3D point tracking and dexterous robotic grasping.