Source-linked AI summary
FounRef: Robust, Structure-Preserving, and Fast Metric Refinement of Frozen Monocular Foundation Priors with Sparse Anchors
Dan Halperin, Mirko Mählisch
TL;DR
Dense metric depth from a single image is difficult because monocular priors can have imperfect metric calibration, while accuracy, coherent geometry, and runtime are distinct concerns. FounRef uses a frozen monocular prior, sparse metric anchors, anchor validation, and structure-preserving refinement. Across evaluated datasets, it achieves the lowest normal dispersion while remaining metrically competitive and considerably faster than learned alternatives.
Problem
Method
FounRef calibrates a frozen monocular prior with sparse metric anchors, validates anchors against a locally refined reference surface, and propagates corrections along prior-defined surfaces.
Source: §1 · p. 2, §3 · pp. 3–4Results
Takeaways & Limitations
FounRef’s modular design can benefit from stronger future priors and metric sensors without retraining.
Source: §6 · p. 8FounRef remains limited by prior quality and cannot reliably recover geometry the prior omits or interprets incorrectly.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Dense metric depth from cameras is essential to real-world 3D applications, yet achieving accuracy, faithful surface geometry, and fast inference simultaneously remains challenging. Monocular foundation models provide rich, transferable geometric priors but lack reliable metric scale, while depth-completion networks recover metric depth at the cost of geometric fidelity, cross-domain robustness, or speed. We present FounRef, a training-free method that aligns a frozen monocular foundation prior with sparse metric anchors to produce dense metric depth. FounRef is modular by design: its depth prior, anchor source, and refinement solver can each be replaced independently. We instantiate FounRef with MoGe-2 and LiDAR anchors. FounRef validates each anchor against the prior's dense depth prediction, rejecting inconsistencies caused by cross-sensor misalignment that geometry-only filters cannot detect. It then applies global and local metric corrections through a structure-preserving solver, retaining the prior's fine-grained geometry. FounRef requires no task-specific training and operates out of the box across unfamiliar cameras and scenes. On out-of-domain data, it delivers up to 24% lower depth error, 92% lower surface-normal noise, and almost 15x faster inference than DMD3C, a state-of-the-art depth-completion network. By decoupling metric alignment from geometry prediction, FounRef provides an accurate, geometrically faithful, and efficient approach to dense metric depth that can directly benefit from future advances in foundation models and metric sensors.
1 Introduction
Single-image depth provides relative scene structure but cannot reliably establish metric scale across scenes and camera settings, despite the importance of dense metric depth for real-world applications. FounRef grounds a frozen monocular prior with sparse metric anchors while retaining its geometry.
1 Introduction
- Single images do not provide enough geometric information to determine metric scale reliably across arbitrary scenes and camera settings.Dense metric depth is nevertheless essential for applications including autonomous systems, mapping, and 3D dataset construction.Source: §1 · p. 1
- On KITTI, a single configuration change moves the method to 0.663 m at 11.2° without affecting runtime.Figure 1 plots metric error against surface dispersion for 500 frames with identical anchors; marker area represents runtime.Source: Figure 1 · p. 1
- Monocular foundation models provide detailed, transferable geometry, but their metric calibration remains imperfect, especially under domain shift and at long ranges.Practical inputs may consist of RGB, camera intrinsics, and sparse metric anchors such as projected LiDAR.Source: §1 · p. 1
- Depth-completion networks can achieve high accuracy, but their performance depends on training data and sparse-depth patterns, and lower metric error does not guarantee coherent surfaces.Adapting to a new sensor rig or distribution may require retraining or fine-tuning.Source: §1 · pp. 1–2
- FounRef globally calibrates a frozen foundation prior, validates anchors against a locally refined reference, and propagates retained corrections along surfaces while preserving depth discontinuities.It refines rather than replaces the prior, so output remains limited by the prior where anchors are sparse.Source: §1 · p. 2
2 Related Work
Related work spans datasets that test transfer under different sensor and target conditions, monocular priors with detailed geometry but imperfect metric calibration, and learned or optimization-based refinement. FounRef combines a frozen prior with per-image sparse-anchor adaptation and anchor validation.
Depth datasets and evaluation
- KITTI and NYU Depth V2 provide refined dense targets, while nuScenes, Waymo, and GOOSE retain sparse, imperfect projected measurements.Together, these datasets test transfer across sensor configurations, scene types, and target quality.Source: §2 · p. 2
Monocular depth foundation models
- Monocular foundation models provide detailed dense geometry, but remain metrically biased, particularly under domain shift and at long range.FounRef uses their geometry as a frozen prior grounded by direct metric LiDAR anchors.Source: §2 · p. 2
Learned depth completion
- Learned completion methods combine RGB with sparse depth through approaches including non-local affinities, transformer context, and foundation-model distillation.Other related methods include PromptDA, Prior Depth Anything, and the propagation-based BP-Net baseline.Source: §2 · p. 2
- Scene-dependent learned priors require offline training and depend on training distribution, sensor configuration, and input sparsity, whereas FounRef adapts a frozen prior per image without training.OMNI-DC is described as targeting zero-shot transfer across datasets and sparse-depth patterns.Source: §2 · p. 2
Optimization-based depth refinement
- Explicit refinement methods propagate sparse constraints using regularized correction fields, including TGV2, screened Poisson, and an edge-aware bilateral-grid solver.The bilateral solver supports efficient propagation within similar regions while limiting propagation across boundaries.Source: §2 · p. 2
Anchor outlier handling
- FounRef validates anchors against a locally refined dense reference surface, complementing epipolar filtering by rejecting broader cross-modal and temporal inconsistencies.Projected LiDAR anchors can be corrupted by occlusion, calibration error, moving objects, temporal registration, and projection artifacts.Source: §2 · p. 2
3 Method
FounRef refines a frozen monocular depth prior using sparse metric anchors: global calibration establishes metric scale, anchor validation removes inconsistencies, and a structure-preserving local solver propagates corrections. Its correction field adjusts the prior’s surfaces rather than redrawing their structure.
3 Method
- FounRef takes a frozen prior depth map and projected metric anchors as input and produces a dense metric depth map.Prior-invalid pixels are excluded from fitting and assigned the maximum depth cap in the final output.Source: §3 · p. 2
- Global calibration fits a straight line in log-depth space to prior predictions and measured metric anchors.The same sparse-anchor calibration brings different priors to a shared metric scale while preserving their edges and relative layout.Source: §3 · p. 2, Figure 2 · p. 3
- Calibration uses a Huber-loss fit solved by iteratively reweighted least squares, with a per-distance correction interpolated from anchor-error averages across depth bins.The implementation uses K = 24 bins; with too few anchors it uses the fitted line alone, and with none it keeps the prior.Source: §3 · p. 3
- Anchor validation compares each anchor with a locally refined reference surface and retains it only when log-depth disagreement is within a dataset-specific threshold.The log-space test measures relative disagreement; cleaner KITTI data permits a more relaxed threshold than noisier nuScenes and GOOSE data.Source: §3 · p. 3
- Validation aims to reject only a small fraction of unreliable anchors while preserving a dense metric point cloud, especially under temporal accumulation.As a training-free method, FounRef cannot learn to suppress anchor errors from data, making explicit validation necessary.Source: §3 · p. 3
- The local solver adds a smooth correction field in log depth, adjusting the prior’s surfaces rather than redrawing their structure.It solves for a single additive correction field.Source: §3 · p. 3
- The bilateral smoothness prior couples corrections for nearby pixels at similar prior depth, propagating them along predicted surfaces while limiting spread across depth discontinuities.A nearby pole can be corrected without pulling the background toward it.Source: §3 · p. 4
- The solver runs at half resolution and upsamples only the correction, preserving full-resolution prior detail while avoiding averaging returns from different surfaces.A pure shift field is used, and image intensity is excluded because it can imprint texture into recovered geometry.Source: §3 · p. 4
4 Experiments
The experiments compare frozen depth priors and refinement solvers, test anchor filtering, and evaluate metric accuracy alongside surface structure. They also examine the effects of range limits and solver bandwidth.
4 Experiments
- Input anchors are capped at 50 m because monocular priors become less reliable farther away, and far-range error is reported separately.The selected range retains challenging medium distances without letting unreliable far-range estimates dominate comparisons.Source: §4 · p. 4
- Dense-depth metrics are used on KITTI and NYU, while held-out raw metric measurements are the primary evaluation protocol on nuScenes, Waymo, and GOOSE.The held-out 20% of measurements is excluded from refinement; accumulated evaluation targets are filtered using an independent Metric3D-V2 prior.Source: §4 · p. 4
- For each dataset, evaluation uses 500 validation images; accumulated metric targets use measurements from up to approximately ten neighboring frames.Accumulation improves coverage but can introduce motion, ego-motion, occlusion, and sensor-noise artifacts, so the points are filtered before evaluation.Source: §4 · p. 4
- Frozen priors are compared with identical KITTI images, anchors, calibration, and refinement, across dense accuracy, long-range error, surface structure, and runtime.The structure measures distinguish high-frequency noise on flat regions from detail on textured regions; both are needed to assess clean, sharp predictions.Source: §4 · p. 4, Table 3 · p. 6
- FounRef rejects returns through windshields, mirrors, and moving-vehicle silhouettes when they disagree with its refined surface, beyond RePLAy’s projective-occlusion filtering.Filtering is assessed with single-sweep and temporally accumulated anchors.Source: Figure 3 · p. 5, §4 · p. 5
- On a sparse-anchor nuScenes crop, pixel-wise Poisson and TGV produce anchor-centred bumps, whereas FounRef shares corrections on a coarse bilateral grid.The solver comparison holds the prior, calibration, anchors, and weights fixed, isolating the regularizer.Source: Figure 4 · p. 5, §4 · p. 5
- Reducing the bilateral grid’s spatial bandwidth localizes corrections, improving RMSE but increasing normal dispersion.Figure 5 presents four operating points on nuScenes from 0–50 m.Source: Figure 5 · p. 5
5 Discussion and Limitations
FounRef’s cross-domain results balance metric accuracy, surface coherence, and runtime, while its filtering, solver, and prior choices expose distinct trade-offs. Its refinements remain bounded by the geometry represented in the frozen prior.
5 Discussion and Limitations
- MoGe-2 offers the strongest overall combination of metric compatibility, coherent geometry, retained detail, and runtime; Metric3D-V2 has slightly lower refined error but less fine detail.Depth Pro and Depth Anything introduce more high-frequency surface noise.Source: §5 · pp. 6–7
- FounRef removes only a small fraction of anchors, and the rejected measurements are substantially less reliable than those retained.Its surface-consistency test can reject cross-modal and temporal disagreements, including moving-object silhouettes, calibration errors, and occlusion artifacts.Source: §5 · p. 7
- The bilateral solver propagates corrections along prior-defined surfaces and limits them across depth discontinuities, trading a modest metric penalty for cleaner geometry and lower runtime.Source: §5 · p. 7
- DMD3C’s strong KITTI performance does not transfer consistently to nuScenes, Waymo, GOOSE, and NYU Depth V2, and no learned baseline remains uniformly accurate across all five datasets.Other zero-shot and foundation-guided methods improve transfer at the cost of runtime, surface quality, or consistent metric accuracy.Source: §5 · p. 7
- FounRef achieves the lowest normal dispersion across evaluated datasets while remaining metrically competitive and considerably faster than learned alternatives.The more aggressive σs=4 configuration further improves metric accuracy without increasing runtime, but moves toward the accuracy-structure trade-off observed in the solver study.Source: §5 · p. 8
- FounRef cannot reliably recover geometry omitted or misinterpreted by its prior; on GOOSE, preserving visible vegetation or fences can increase point-wise error when LiDAR returns through them.The authors characterize structure preservation as an inductive bias, not an unconditional advantage.Source: §5 · p. 8
- FounRef lies on an empirical Pareto frontier across metric accuracy, surface coherence, cross-domain robustness, and runtime; more aggressive local fitting reduces error while exposing anchor patterns and degrading surface quality.The authors identify a zero-shot refinement model that moves this frontier without sacrificing structural generalization or solver efficiency as a next step.Source: §5 · p. 8
6 Conclusion
FounRef converts a frozen monocular prior and sparse metric anchors into dense metric depth through global calibration, anchor validation, and bilateral refinement. Across five datasets it remains metrically competitive while producing more coherent surfaces at substantially lower runtime.
6 Conclusion
- FounRef is a training-free method that converts a frozen monocular prior and sparse metric anchors into dense metric depth.Source: §6 · p. 8
- Global calibration, surface-based anchor validation, and bilateral refinement provide metric grounding while preserving the prior’s geometry.Source: §6 · p. 8
- Across five datasets, FounRef remains metrically competitive with learned methods while producing more coherent surfaces at substantially lower runtime.Source: §6 · p. 8
A Appendix
The appendix tests FounRef’s stages and robustness to changing or corrupted anchors, then examines solver settings and compares methods through depth, normal, and 3D views. Across these tests, the prior supplies scene geometry while anchors calibrate metric depth, with a measurable accuracy–structure trade-off at finer solver settings.
Evaluation scope
- Appendix experiments use the default configuration and separate frame subsets and screening references, so their absolute values are not directly comparable with the main tables.Comparisons within each appendix table remain internally consistent.Source: A Appendix · p. 9
Pipeline breakdown
- D1 removes half to two thirds of calibrated-prior error, and D2 improves on D1 by 19% on KITTI and 14% on Waymo; GOOSE is 2.7% worse.D1 and D2 use the same solver but different anchors; off-road returns through vegetation explain the GOOSE exception.Source: A Appendix · pp. 9–10, Table 5 · p. 10
Anchor density
- From full density to 1% anchor density, FounRef’s error grows 1.4–2.1× on every dataset, the most uniform response in Table 6.Every method receives the same thinned cloud per frame; FounRef’s prior supplies geometry while anchors fix metric scale.Source: A Appendix · p. 10, Table 6 · p. 11
Anchor perturbation
- A 5% multiplicative anchor-depth error moves each dataset by about 3% or less, whereas a 5-pixel projection shift raises KITTI error by 95%.The evaluation target is the unperturbed held-out cloud, so these changes measure prediction damage; the results motivate accurate sensor synchronization.Source: A Appendix · p. 11, Table 9 · p. 12
Solver sensitivity
- At σs = 4, RMSE improves 21–33% on four outdoor datasets and 6% on NYU versus σs = 16, but corrections concentrate into damaging anchor-local blobs.Normal dispersion changes by only a few percent from σs = 16 to 3, then worsens at σs = 2; NYU is omitted from Figure 9 because its two settings differ by under a centimetre.Source: A Appendix · p. 11, Table 7 · p. 12, Figure 9 · p. 13
- The median normal-dispersion metric misses sparse anchor blobs, although a high quantile of the same local angle would expose such damage.The metric is the median angle between each surface normal and its 15 × 15 local mean over valid pixels.Source: A Appendix · p. 11
Solver components
- Table 8 attributes solve-time changes to conjugate-gradient iteration count and solve resolution; the reported times are 27 and 42 ms for iteration count and 40 ms for resolution.Each component is toggled alone from the default configuration, with RMSE, normal dispersion, and mean solve time compared.Source: A Appendix · pp. 11–12, Table 8 · p. 12
Qualitative comparison
- The same-frame depth maps are hard to distinguish, but re-projected point clouds expose discontinuous baseline surfaces and FounRef’s continuous surfaces.On Waymo, baselines produce stray sheets behind roads; on NYU, all but DMD3C retain the room despite OMNI-DC and Prior-DA being metrically ahead of FounRef.Source: A Appendix · p. 12, Figure 10 · p. 14, Figure 12 · p. 16