PaperlayerView exact revision on arXiv

Source-linked AI summary

M2Depth: Unifying Monocular Depth Foundation Priors with Multi-View Stereo

Byeonggwon Lee, Sanggi Lee, Siwoo Lee, Khang Truong Giang, Soohwan Song

arXiv:2608.20788v1cs.CV

TL;DR

MVS generalizes poorly in unseen, occluded, and limited-overlap regions, while existing DFM-MVS hybrids rely on static one-way fusion. M2Depth tightly couples a DFM with cascade MVS through prior-guided cost refinement and bidirectional mutual refinement. It consistently surpasses state-of-the-art approaches and generalizes strongly to sparse-view settings with an accuracy-efficiency trade-off.

  • Problem

    MVS often struggles with unseen scenes and difficult regions, while existing DFM-MVS methods use static one-way fusion that limits reciprocal correction.

  • Method

    M2Depth combines prior-guided cost-volume refinement with bidirectional mutual refinement between monocular depth and cascade MVS.

  • Results

    The method consistently surpasses state-of-the-art approaches in 3D reconstruction and depth estimation across DTU, TNT, and RobustMVD, and generalizes robustly to sparse-view settings.

  • Takeaways & Limitations

    M2Depth combines monocular structural completeness with MVS metric reliability for generalized 3D reconstruction without sacrificing geometric consistency.

Abstract

from arXiv · show

Deep learning-based Multi-View Stereo (MVS) has advanced significantly but often generalizes poorly to unseen scenes, particularly in occluded areas or regions with limited view overlap. To mitigate this, recent approaches integrate Depth Foundation Models (DFMs) into MVS pipelines to provide monocular depth priors. However, existing methods typically rely on a static, one-way fusion scheme, which fails to fully exploit the complementary strengths of both modalities. We propose a novel framework that overcomes this limitation by tightly coupling a DFM with a cascade MVS pipeline through a bidirectional mutual refinement strategy. Our method leverages MVS depth to resolve the scale ambiguity in monocular predictions, while the monocular depth, in turn, enhances the structural completeness and fine-grained detail of the MVS estimate. Furthermore, we introduce a prior-guided cost volume refinement mechanism that effectively integrates multi-view and monocular information via attention-based fusion and discretized depth bins, thereby promoting local geometric consistency. Extensive experiments demonstrate that our method outperforms state-of-the-art MVS approaches on standard benchmarks, producing more complete and generalizable depth maps with sharp boundaries. Furthermore, although not explicitly designed for sparse-view settings, our framework generalizes remarkably well, competing favorably with even dedicated sparse-view methods while maintaining a superior accuracy-efficiency trade-off.

I. INTRODUCTION

Deep MVS methods improve reconstruction but remain vulnerable to unseen scenes, occlusions, limited overlap, and ambiguous geometry. M2Depth addresses these limits by coupling monocular foundation priors with cascade MVS through mutual refinement and prior-guided cost-volume fusion.

  • Motivation: Deep MVS approaches often generalize poorly to unseen structures and lose accuracy under occlusions or limited view overlap.Ambiguous scene geometry makes these regions especially challenging.
  • Motivation: Depth Foundation Models provide robust monocular predictions in unseen domains but lack metric scale and explicit multi-view consistency.These limitations restrict their direct use for accurate 3D reconstruction.
  • Research gap: Existing hybrid methods use the complementary modality as a static, one-way prior, allowing inaccurate-prior errors to propagate without correction.MVS-centric pipelines remain incomplete in difficult regions, while DFM-centric pipelines remain vulnerable to scale misalignment.
  • Proposed framework: M2Depth tightly couples a DFM with cascade MVS through bidirectional co-refinement, using MVS depth for monocular scale alignment and monocular depth for MVS structure.Both representations are progressively improved throughout the cascade.
  • Proposed framework: Prior-guided cost-volume refinement fuses monocular and multi-view information with bin masks to improve spatial consistency.The framework targets clean, complete depth maps and accurate 3D reconstruction in challenging cases.
  • Results: The method achieves high accuracy and strong generalization across multiple benchmarks, including depth estimation, 3D reconstruction, and sparse-view settings.It is reported to outperform existing MVS methods and dedicated sparse-view methods.

B. Depth Foundation Model

DFM integration has improved stereo and MVS reconstruction, but static priors and specialized sparse-view pipelines leave scale, completeness, and generalization challenges. M2Depth instead uses explicit cascade MVS with prior-guided cost construction and iterative mutual refinement.

  • Background: Depth Foundation Models offer strong zero-shot generalization, while DFM-based stereo methods exploit their monocular depth priors for matching.Their application to MVS remains comparatively limited.
  • Related approaches: MonoMVSNet injects a static monocular prior into MVS, whereas MVSAnywhere uses multi-view information as a prior within a DFM-centric pipeline.These represent two primary directions for combining DFMs and MVS.
  • Proposed approach: M2Depth uses bidirectional co-refinement so MVS and monocular estimates iteratively correct one another, addressing scale ambiguity and MVS incompleteness.This avoids the error propagation and geometric misalignment associated with unidirectional fusion.
  • Sparse-view reconstruction: Unlike specialized sparse-view methods, the framework remains rooted in explicit multi-view geometry and avoids expensive implicit surface optimization.Prior-guided cost construction and mutual depth refinement adapt cascade MVS to sparse-view conditions.
  • MVS pipeline: The pipeline uses a four-stage coarse-to-fine MVS process that progressively refines depth from 1/8 to full resolution.At each stage, cost volumes become probability volumes from which MVS depth is estimated.
  • Prior-guided refinement: Prior-guided refinement combines multi-view and monocular cost volumes with uniformly spaced depth-bin masks as inputs to Cost-Crafter.The monocular prior supplies structural and fine-grained shape cues for depth refinement.

A. Prior Depth-Guided Cost Volume Refinement

Cost-Crafter refines the initial multi-view cost volume by combining stereo evidence with monocular depth structure. Depth-bin tokenization and bidirectional pixel-bin attention provide efficient, depth-aware global reasoning before producing a refined probability volume.

  • Cost-Crafter fuses multi-view and monocular cost volumes with structural bin masks to regularize ambiguous matching regions.The refinement is applied at the first cascade stage, where reliable global geometry establishes a strong coarse representation.
  • The module constructs multi-view costs, monocular costs from soft one-hot depth encoding, and uniformly spaced depth bins as inputs.The multi-view volume aggregates stereo matching costs, while the monocular volume is derived from the prior depth map.
  • Depth-bin tokenization compresses flattened pixel-depth features into M representative tokens, enabling efficient global context modeling while preserving depth-aware structure.Pixels sharing similar prior depth levels are grouped through a replicated bin mask and masked average pooling.
  • Learnable scalar positional biases inject ordinal relationships between depth layers into cross-attention without explicitly forming an additional bias matrix.The query and key projections receive bias terms, while the value projection receives a zero scalar so the bias affects attention weights rather than propagated semantic features.
  • Bidirectional pixel-bin interaction first aggregates global depth cues into bin tokens and then broadcasts refined responses back to pixels.This design improves structural coherence while preserving local detail, including in textureless, occluded, or weakly matched regions.
  • The attention-enhanced volume is reshaped and processed with a 1 × 1 × 1 convolution, 3D LayerNorm, and depth-wise softmax to produce a refined probability volume.The resulting refinement combines monocular priors, bin-level reasoning, and ordinal depth-aware attention.

B. Mutual Depth Refinement

Mutual Depth Refinement iteratively couples monocular and MVS predictions through symmetric branches, using reliable multi-view geometry to align monocular scale and monocular structure to improve MVS completeness. Confidence-aware signals and adaptive anchoring make the process robust to uncertain, sparse-view regions.

  • MVS-Guided Alignment: MDR progressively corrects spatially varying monocular scale errors rather than applying a single global scale-shift correction.MVS-GA uses multi-view geometric cues for local scale alignment, while Mono-GR refines MVS depth with scale-aligned monocular structure.
  • Mutual Depth Refinement: MDR uses symmetric MVS-GA and Mono-GR branches to predict residual updates for the opposite depth modality.Reliable MVS geometry anchors monocular depth, while monocular structure restores regions where photometric matching is weak.
  • Confidence-Guided Refinement: The probability-volume curvature measures peak sharpness, with high curvature indicating clear matching and flat or multimodal responses indicating unreliable regions.This confidence proxy targets uncertainty in textureless or occluded areas.
  • Confidence-Guided Refinement: The refinement uses confidence-weighted cross-modal discrepancies so monocular cues dominate mainly where MVS matching is unreliable.High-confidence MVS pixels act as geometric anchors that discourage large MVS updates, whereas low-confidence regions allow stronger monocular completion.
  • MVS-Specific Design: MVS-specific refinement signals include a geometry feature volume, curvature-based confidence, confidence-gated discrepancy encoding, and persistent EMA confidence updates.These components address multi-view coupling, cross-view uncertainty aggregation, and coarse-to-fine depth evolution.
  • Cascade Propagation: After refinement, scale-corrected monocular depth and structurally enhanced MVS depth are forwarded to subsequent cascade stages as updated priors and depth hypotheses.This iterative bidirectional process maintains local scale alignment and geometry completion across stages.

C. Loss Function

The cascade is trained with complementary losses that supervise depth distributions, intermediate mutual refinements, and local depth ordering across stages.

  • Loss Components: Cross-entropy loss measures depth accuracy at each cascade stage by comparing predicted and ground-truth depth distributions.
  • Loss Components: A decayed ℓ1 mutual refinement loss supervises every intermediate MVS depth map, emphasizing later refinements with r = 0.9.
  • Loss Components: An order-preserving loss penalizes violations in relative depth ordering among sampled valid-region pixel pairs.The loss is zero for correct ordering and increases linearly when predicted ordering disagrees with ground truth.
  • Loss Aggregation: The final training objective aggregates losses across all cascade stages, with λmut = λord = 0.02 balancing the auxiliary terms against cross-entropy.

IV. EXPERIMENT

The experiments evaluate reconstruction quality, generalization, sparse-view robustness, and component contributions across standard and challenging multi-view settings.

  • Evaluation Scope: The method is evaluated on DTU and Tanks and Temples for 3D reconstruction, and on RobustMVD for cross-dataset depth-estimation generalization.
  • Evaluation Scope: Sparse-view robustness is examined on DTU using only three unfavorable input views.
  • Qualitative Results: Qualitative DTU results report highly complete 3D reconstructions and clean depth maps on challenging Scans 48 and 77.
  • Quantitative Evaluation: DTU quantitative results use point-cloud distance metrics for accuracy, completeness, and overall error, together with runtime and GPU memory usage.

A. Implementation Details

Implementation uses a four-stage coarse-to-fine cascade with stage-specific depth hypotheses, feature dimensions, refinement iterations, and standard DTU training protocols.

  • Model Configuration: Cost-Crafter uses M=10 depth bin masks, while mutual refinement iterations are K = [12, 8, 5, 3] across the four stages.
  • Training Protocol: Training and evaluation use the DTU dataset with standard data splits and view-selection protocols.
  • Hardware and Runtime: Testing runs on an Intel i9-13900KS CPU with an NVIDIA GeForce RTX 3090 Ti GPU, using per-sample depth inference runtimes unless otherwise noted.Table IV runtime additionally includes multi-view fusion and benchmark-specific post-processing.

B. Benchmark Performance

Across DTU, TNT, and RobustMVD, the method achieves stronger reconstruction and depth-estimation performance than conventional MVS and DFM-based baselines. It also preserves quality under sparse-view conditions, with added runtime but comparable memory usage.

  • DTU: The method obtains the best DTU accuracy, completeness, and overall errors among conventional MVS and DFM-based approaches.The completeness gain indicates recovery of missing or ambiguous structures and denser reconstruction.
  • DTU: It consistently outperforms MonoMVSNet across DTU’s accuracy, completeness, and overall metrics.The monocular prior guides cost-volume construction and mutual depth refinement rather than serving only as an auxiliary cue.
  • Efficiency: Under sparse-view evaluation, the method maintains robust reconstruction quality with marginal additional cost and comparable memory usage.The added runtime is described as justified by generalization gains over MonoMVSNet.
  • TNT: The method achieves the highest mean F-scores on both TNT intermediate and advanced test sets.
  • RobustMVD: It achieves the best results on DTU and TNT, while performing slightly worse than MVSA on KITTI.

C. Generalization to Sparse-View Reconstruction

The method generalizes strongly to sparse-view reconstruction despite not being specifically designed or trained for that setting. Under limited-overlap protocols, it outperforms both general MVS and dedicated sparse-view methods while retaining an efficient pipeline.

  • Evaluation setting: The sparse-view evaluation uses three unfavorable DTU views with limited overlap as a protocol-level comparison rather than matched training settings.
  • Quantitative comparison: The method achieves the lowest average Chamfer Distance under the three-view SparseRecon protocol.It outperforms both general MVS methods and sparse-view-targeted methods including UFORecon and SparseRecon.
  • Qualitative results: Qualitatively, the method recovers more complete and cleaner geometry than competing methods in challenging sparse-view regions.
  • Two-view reconstruction: In two-view reconstruction, the method produces complete, aligned, scale-corrected surfaces where MonoMVSNet has outliers and MVSAnywhere has scale ambiguity.
  • Efficiency: Its margin over SparseRecon is modest, but it offers a superior accuracy-efficiency trade-off with minimal computational overhead.Dedicated baselines rely on intensive volume-rendering-based surface optimization, increasing runtime and memory costs.
  • Two-view reconstruction: With minimal multi-view cues, the method reconstructs regions without view overlap while preserving detailed, geometrically consistent surfaces.

D. Ablation Study

Ablations show that Cost-Crafter and mutual depth refinement provide complementary benefits: the former strengthens geometric initialization, while the latter improves smoothness and completeness. Their combination, together with auxiliary losses, yields the strongest balanced reconstruction quality.

  • Component integration: Progressively adding Cost-Crafter, MDR, and auxiliary losses consistently improves reconstruction quality over the baseline.
  • Loss ablation: Mutual refinement and order-preserving losses achieve optimal accuracy and completeness with only minor computational overhead.
  • Component roles: Cost-Crafter supplies accurate coarse geometry, whereas MDR produces smoother and denser depth but depends on reliable initial hypotheses.Cost-Crafter alone remains noisy and incomplete in textureless regions, while MDR alone struggles to correct early geometric errors.
  • Overall design: The full framework combines structural priors, bidirectional refinement, and cascade MVS to mitigate scale ambiguity and MVS incompleteness.
  • Efficiency: The DFM encoder uses a DINOv2 ViT-Small backbone with approximately four times fewer parameters than MVSFormer++’s ViT-Base.This modification is intended to improve computational efficiency.
  • Feature design: Frustoconical positional encoding injects camera-frustum-based 3D spatial information before cost-volume construction.

B. Monocular Volume Construction

The monocular volume construction normalizes and softly encodes monocular depth against discrete inverse-depth hypotheses before fusing it with multi-view costs. This design supports geometric initialization, progressive scale correction, and robustness to flawed monocular priors.

  • Monocular volume: The monocular cost volume is derived from a single-image depth prior and supplies structural guidance where multi-view correspondence is ambiguous.
  • Depth normalization: MAD removes statistical outliers before normalization and rescaling, producing more accurate and consistent monocular scale alignment.
  • Depth encoding: Gaussian-weighted soft encoding maps continuous monocular depth onto uniformly sampled inverse-depth hypotheses while modeling local scale uncertainty.
  • Cost fusion: The resulting monocular volume is fused with the multi-view volume through bidirectional cross-attention in Cost-Crafter.
  • Module complementarity: Cost-Crafter provides accurate geometric initialization, while MDR contributes structural smoothness and completeness; together they produce the most balanced results.
  • Progressive refinement: MDR progressively reduces monocular error across cascade stages and aligns the prior with the true metric scale.
  • Robustness: Multiview consistency can override structurally erroneous DFM predictions, recovering correct geometry instead of blindly propagating prior errors.
  • View robustness: Reducing source views from five to three increases the method’s overall and > 2 mm depth errors by 3.6% and 11.3%, respectively.The corresponding baseline increases are 4.1% and 19.4%, indicating stronger sparse-view robustness.

2) Qualitative Results on DTU:

On DTU and broader cross-dataset qualitative evaluations, the method combines monocular structural completeness with MVS geometric accuracy, correcting scale while preserving details and sharp boundaries.

  • DFMrefined corrects the monocular depth scale ambiguity while preserving structural details.
  • Ours fills MVSFormer++ holes and suppresses MonoMVSNet noise while producing fine details and sharp boundaries comparable to the ground truth.
  • Using only two input images, Ours is contrasted with noisy, incomplete MonoMVSNet reconstructions and scale-misaligned MVSAnywhere geometries.
  • On Tanks and Temples, scale alignment enables depth maps that are structurally complete, geometrically consistent, and sharp-boundaried in challenging scenes.
  • On KITTI driving scenes, the method reconstructs dynamic vehicle shapes and preserves smooth road gradients while generating sharp, scale-consistent depth maps.
Loading 2608.20788v1…