Source-linked AI summary
Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Zhiyuan Zhou, Ruofeng Liu, Taichi Liu, Weijian Zuo, Shanshan Wang, Zhiqing Hong, Desheng Zhang
TL;DR
Depth completion must recover dense metric depth despite sparse sensors and prior methods’ domain, pattern, and alignment limitations. Any2Full adapts a pretrained MDE model through one-stage scale prompting and a Scale-Aware Prompt Encoder. It reports a 32.2% improvement in average AbsREL over OMNI-DC and a 1.4× speedup over PriorDA with the same backbone.
Problem
Depth completion needs dense metric depth from sparse measurements, while existing methods have limited domain and depth-pattern robustness and two-stage MDE integration adds overhead and distortions.
Method
Any2Full reformulates completion as scale-prompting adaptation of pretrained MDE, using a Scale-Aware Prompt Encoder to convert sparse cues into unified prompts for scale-consistent prediction.
Results
32.2% improvement in average AbsREL over OMNI-DC and 1.4× speedup over PriorDA with the same MDE backbone are reported.
Takeaways & Limitations
Any2Full provides a one-stage depth completion framework reported to combine domain-general geometric priors with robustness across diverse depth patterns and efficient inference.
Takeaways & Limitations
One-stage inference obtains final metric depth through closed-form alignment of predicted relative depth, without additional learnable modules.
Abstract
from arXiv · showhide
Accurate, dense depth estimation is crucial for robotic perception, but commodity sensors often yield sparse or incomplete measurements due to hardware limitations. Existing RGBD-fused depth completion methods learn priors jointly conditioned on training RGB distribution and specific depth patterns, limiting domain generalization and robustness to various depth patterns. Recent efforts leverage monocular depth estimation (MDE) models to introduce domain-general geometric priors, but current two-stage integration strategies relying on explicit relative-to-metric alignment incur additional computation and introduce structured distortions. To this end, we present Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates completion as a scale-prompting adaptation of a pretrained MDE model. To address varying depth sparsity levels and irregular spatial distributions, we design a Scale-Aware Prompt Encoder. It distills scale cues from sparse inputs into unified scale prompts, guiding the MDE model toward globally scale-consistent predictions while preserving its geometric priors. Extensive experiments demonstrate that Any2Full achieves superior robustness and efficiency. It outperforms OMNI-DC by 32.2\% in average AbsREL and delivers a 1.4$\times$ speedup over PriorDA with the same MDE backbone, establishing a new paradigm for universal depth completion. Codes and checkpoints are available at https://github.com/zhiyuandaily/Any2Full.
1 Introduction
Any2Full addresses domain and pattern limitations in depth completion by adapting pretrained MDE geometric priors through one-stage scale prompting. Its Scale-Aware Prompt Encoder converts sparse, irregular scale cues into unified prompts, with experiments reporting improved accuracy and efficiency.
- Commodity depth sensors produce sparse or incomplete measurements, motivating recovery of dense metric depth from depth inputs and corresponding RGB images.
- Existing RGBD-fused methods learn priors conditioned on training RGB distributions and specific depth patterns, limiting domain generalization and pattern robustness.
- Two-stage MDE integration adds computational overhead and can introduce structured distortions and depth-pattern-specific biases through explicit relative-to-metric alignment.
- Any2Full reformulates depth completion as a one-stage scale-prompting adaptation of a pretrained MDE model.It distills inter-point scale ratios from sparse inputs to guide scale-consistent predictions while preserving MDE domain generalization.
- The Scale-Aware Prompt Encoder transforms sparse and irregular scale cues into globally consistent, pattern-invariant features under MDE geometric guidance.Its hierarchical processing supports varying sparsity levels and irregular spatial distributions with minimal additional computation.
- 32.2% improvement in average AbsREL over OMNI-DC and 1.4× speedup over PriorDA demonstrate Any2Full’s reported accuracy and efficiency advantages.The speed comparison uses the same MDE backbone.
2 Related Work
Related work spans monocular depth estimation, conventional depth completion, and MDE-integrated completion. Any2Full responds to the spatially inconsistent alignment and pattern-specific limitations of prior feed-forward approaches with one-stage scale prompting.
- Monocular Depth Estimation: MDE infers 3D scene structure from a single RGB image but fundamentally faces scale ambiguity, limiting predictions to relative depth.
- Depth Completion: Conventional depth completion fuses sparse depth and RGB to predict coarse dense depth, then refines it, but struggles under appearance shifts and heterogeneous depth patterns.
- MDE-Integrated Depth Completion: MDE-integrated methods combine pretrained geometric priors with sparse metric cues through test-time adaptation or direct feed-forward pipelines.Test-time adaptation is computationally expensive, while direct feed-forward methods prioritize efficiency but must adapt relative depth to metric prediction.
- MDE-Integrated Depth Completion: PriorDA explicitly aligns relative MDE output with sparse measurements before refinement, but spatially inconsistent mapping can distort geometric priors and introduce structural artifacts.
- Any2Full: Any2Full constructs unified scale prompts for varying depth patterns to integrate MDE geometric priors with sparse depth cues in one stage.
3 Any2Full
Any2Full reformulates depth completion as one-stage scale-prompting adaptation of a pretrained MDE model, using sparse-depth cues to produce scale-consistent metric depth while preserving geometric priors. Its Scale-Aware Prompt Encoder addresses sparsity and irregular patterns through local enrichment, global propagation, and feature-level fusion.
- 3.1 Formulation: Any2Full adapts a pretrained MDE model by encoding normalized sparse depth into scale prompts that guide scale-consistent relative-depth prediction.The relative prediction is converted to dense metric depth through a non-parametric least-squares fit.
- 3.1 Formulation: The framework obtains dense metric depth by applying a closed-form alignment to the predicted relative depth, without intermediate depth prediction or auxiliary refinement networks.This defines its one-stage inference process while adding no learnable alignment module.
- 3.2 Scale Prompting Pipeline: SAPE uses Local Enrichment to couple sparse-depth scale cues with dense MDE geometry, producing local features robust to varying sparsity.Generalized FiLM anchors patch-level depth cues in the MDE latent space through lightweight modulation parameters.
- 3.2 Scale Prompting Pipeline: Global Propagation diffuses fragmented local scale cues across the scene using MDE geometry-guided attention, establishing globally consistent scale-aware features.The resulting features form a unified scale prompt for diverse depth patterns.
- 3.2 Scale Prompting Pipeline: Scale Prompt Fusion injects unified prompts into the MDE decoder through hierarchical FiLM to refine relative-depth features at multiple semantic levels.This feature-level modulation is designed to minimally alter the pretrained MDE representations while injecting scale-related cues.
- 3.3 Training: Training combines random and hole sampling with losses for global scale consistency, local edge preservation, sparse-anchor alignment, and relative structural similarity.The sampling strategies expose the model to varying densities and large contiguous missing regions.
4 Experiments
Experiments evaluate Any2Full across diverse domains, sensor conditions, depth patterns, sparsity levels, backbones, and real-world robotic deployment. Results show robust generalization, pattern invariance, cross-backbone applicability, and strong efficiency.
- Experimental Setup: Any2Full is evaluated on six unseen public datasets spanning indoor and outdoor scenes, visual conditions, and sensor modalities, plus the Logistic-Black robotic warehouse dataset.The public benchmarks include NYU-Depth V2, iBims-1, KITTI DC, DIODE, ETH3D, and VOID; Logistic-Black contains ToF depth missing regions caused by black packages.
- Domain and Pattern Generalization: Any2Full achieves the lowest average rank of 2.3 across all evaluated benchmark scenarios and remains among the top performers on every dataset.The comparison uses AbsREL and RMSE across diverse domains and depth patterns.
- Model Efficiency: Any2Full achieves a 1.4× speedup over PriorDA with the same backbone while reducing AbsREL from 0.038 to 0.007 when built on DA-L.The DA-S variant runs in 0.09 s, about 7× faster than PriorDA and 1000× faster than TestPromptDC.
- Real-World Deployment: Any2Full improves grasping success on challenging black packages from 28% to 91.6% without damage in a robotic warehouse cell.The system handles thousands of packages daily in the reported deployment.
- Cross-Backbone Generalization: SAPE improves MoGe-2 across all evaluated datasets and depth patterns without architectural modification, showing that scale prompting is not specific to Depth Anything.The cross-backbone evaluation uses MoGe-2-Base and Depth Anything backbones.
- Robustness Analysis: Any2Full remains stable across varying sparsity and depth-range conditions, including sparse VIO points, LiDAR scans, and unseen range-truncated inputs.It extrapolates globally consistent scales to unmeasured regions, whereas competing MDE-based feed-forward methods degrade under narrow valid-depth ranges.
- Ablations and Analysis: Ablations show that scale-prompt fusion enables metric prediction across domains, while Local Enrichment and Global Propagation further improve accuracy across patterns and sparsity levels.The study progressively removes both modules to assess their contributions.
5 Conclusion
Any2Full is a one-stage, domain-general, pattern-agnostic depth completion model that adapts monocular depth estimation through scale prompting.
- Any2Full reformulates depth completion as scale-prompting adaptation of a pretrained monocular depth estimation backbone.
- The model combines domain-general geometric priors with flexible adaptation to diverse sparse depth patterns through a scale-aware prompt encoder.
- Any2Full reports superior generalization, speed, and accuracy, with practical effectiveness demonstrated in real-world deployment.
Overview
The supplementary material adds experimental analyses, deployment evidence, implementation details, and discussion of Any2Full.
- The supplement includes additional experiments and analyses in Section A.
- It covers industrial deployment in robotic warehouse grasping in Section B.
- Implementation details of Any2Full are provided in Section C, with additional discussion in Section D.
A.1 Scale Consistency Analysis
The scale-consistency analysis aligns relative depth globally, measures regional scale variation, and shows that scale prompting produces nearly uniform scene-wide scales.
- The analysis aligns predicted relative depth in the disparity domain to ground-truth metric depth using global least-squares scale and shift fitting.
- Aligned metric depth is recovered by reciprocal conversion, after which the scene is partitioned into non-overlapping regions for local scale estimation.
- Regional scale factors are computed as the median ratio of ground-truth to aligned predicted depth within each region.
- The variance of regional scale factors measures inconsistency, with uniform factors near 1 indicating that global transformation suffices.
- Any2Full maintains a nearly uniform scale distribution across scenes, enabling metric recovery using only global alignment parameters.
A.2 Two-Stage Limitation Analysis
The analysis shows that two-stage depth completion pipelines can propagate noise and structural distortions from coarse depth into final predictions, whereas Any2Full yields cleaner reconstructions.
- Two-stage methods often introduce noise and structural distortions when predicting coarse dense depth before refinement.
- On NYU-Depth V2, traditional RGBD-fused and MDE-integrated two-stage pipelines produce blurred details, disrupted structures, and visible artifacts.
- Any2Full produces fewer outliers and cleaner 3D point-cloud reconstructions than TestPromptDC.
- The comparisons include highly irregular sparse inputs from ETH3D-SfM and VOID-150.
Additional Qualitative Results
Additional qualitative evaluations compare Any2Full with PriorDA across domains, depth patterns, and reconstruction settings. The results emphasize cleaner geometric reconstructions and pattern-agnostic behavior.
- Qualitative comparisons use PriorDA as the primary baseline because it robustly preserves fine geometric structures from MDE priors.
- Any2Full produces fewer outliers and cleaner 3D point-cloud reconstructions than TestPromptDC.
- Table 7 compares MDE-guided propagation with standard cross-attention across different depth patterns.
- Any2Full consistently achieves lower AbsREL than the cross-attention alternative across all reported settings.
A.4 Ablation Studies
Ablation studies identify global propagation and the selected loss terms as important components of Any2Full. Attention visualizations explain how propagation extends scale cues across nearby and distant regions.
- Module Ablation: The global propagation module significantly improves performance, especially under range and sparse depth patterns.
- Module Ablation: Global propagation diffuses scale cues across spatially irregular and sparse inputs through complementary nearby-to-distant attention patterns.
- Module Ablation: MDE-guided propagation uses geometric structures rather than sparse depth patterns, preserving robustness across depth sampling distributions.
- Module Ablation: Replacing MDE-guided propagation with standard cross-attention causes noticeable performance drops.
- Loss Ablation: Lgm and Lanchor consistently improve performance, while Lr-ssim may slightly constrain depth expansion in the Range pattern.
B Real-World Industrial Deployment
Any2Full is evaluated in a robotic warehouse setting targeting black packages, where unreliable active depth sensing can cause grasp failures and deformation. The deployment combines sparse-depth processing, scale-consistent supervision, and real-time robotic operation.
- Motivation: Black packages are difficult for active depth sensors because high-absorption surfaces yield unreliable depth measurements.
- Deployment Setup: Any2Full is deployed on an RTX 4080 Super at 10 Hz, satisfying the robotic cell’s real-time constraints.
- Grasping Evaluation: The evaluation compares raw depth with Any2Full-enhanced depth on Large, Medium, and Small black packages using 12 paired trials per size.
- Grasping Evaluation: Raw depth sensing achieves a 39.0% grasp success rate, with 33.3% of successful grasps deforming packages.
- Long-Term Deployment Stability: During a continuous one-week warehouse trial, the system maintained a 98.86% grasp success rate on standard packages.
- Sparse Depth Processing: Sparse metric depth is converted to disparity and normalized using valid-pixel statistics before forming the model input.
- Training Losses: The model jointly supervises global alignment, structural accuracy, sparse anchoring, and relative structure similarity in relative-depth space.
C.4 Evaluation Metrics
Evaluation uses AbsREL and RMSE over valid pixels in metric-depth space, alongside qualitative tests spanning multiple datasets and depth patterns. The supplied discussion also frames Any2Full as complementary to newer metric-depth MDE models.
- Evaluation Metrics: AbsREL measures the average relative absolute difference between predicted and ground-truth metric depth.
- Evaluation Metrics: RMSE reports the root mean squared depth error in meters.
- Evaluation Metrics: Both metrics are computed over all valid pixels.
- Discussion: Depth Anything v3 extends MDE to arbitrary visual inputs, but monocular depth estimation alone still struggles to recover reliable absolute scale without metric measurements.
- Discussion: The authors identify integration with Depth Anything v3 as future work that may improve monocular depth completion and extend arbitrary-view applications.
- Qualitative Evaluation: Qualitative results cover DIODE, KITTI DC, Logistic-Black, and IBims-1 under Hole, Range, Sparse-Random, Sparse-LiDAR, and Mixed patterns.