PaperlayerView exact revision on arXiv

Source-linked AI summary

M2Depth: Unifying Monocular Depth Foundation Priors with Multi-View Stereo

Byeonggwon Lee, Sanggi Lee, Siwoo Lee, Khang Truong Giang, Soohwan Song

arXiv:2608.20788v1cs.CV

TL;DR

MVS generalizes poorly in unseen, occluded, and limited-overlap regions, while existing DFM-MVS hybrids rely on static one-way fusion. M2Depth tightly couples a DFM with cascade MVS through prior-guided cost refinement and bidirectional mutual refinement. It consistently surpasses state-of-the-art approaches and generalizes strongly to sparse-view settings with an accuracy-efficiency trade-off.

  • Problem

    MVS often struggles with unseen scenes and difficult regions, while existing DFM-MVS methods use static one-way fusion that limits reciprocal correction.

    Source: §I · p. 1

  • Method

    M2Depth combines prior-guided cost-volume refinement with bidirectional mutual refinement between monocular depth and cascade MVS.

    Source: §I · p. 2, A. Prior Depth-Guided Cost Volume… · p. 4, B. Mutual Depth Refinement · p. 6

  • Results

    The method consistently surpasses state-of-the-art approaches in 3D reconstruction and depth estimation across DTU, TNT, and RobustMVD, and generalizes robustly to sparse-view settings.

    Source: §V · pp. 9–10

  • Takeaways & Limitations

    M2Depth combines monocular structural completeness with MVS metric reliability for generalized 3D reconstruction without sacrificing geometric consistency.

    Source: §V · p. 10

Abstract

from arXiv · show

Deep learning-based Multi-View Stereo (MVS) has advanced significantly but often generalizes poorly to unseen scenes, particularly in occluded areas or regions with limited view overlap. To mitigate this, recent approaches integrate Depth Foundation Models (DFMs) into MVS pipelines to provide monocular depth priors. However, existing methods typically rely on a static, one-way fusion scheme, which fails to fully exploit the complementary strengths of both modalities. We propose a novel framework that overcomes this limitation by tightly coupling a DFM with a cascade MVS pipeline through a bidirectional mutual refinement strategy. Our method leverages MVS depth to resolve the scale ambiguity in monocular predictions, while the monocular depth, in turn, enhances the structural completeness and fine-grained detail of the MVS estimate. Furthermore, we introduce a prior-guided cost volume refinement mechanism that effectively integrates multi-view and monocular information via attention-based fusion and discretized depth bins, thereby promoting local geometric consistency. Extensive experiments demonstrate that our method outperforms state-of-the-art MVS approaches on standard benchmarks, producing more complete and generalizable depth maps with sharp boundaries. Furthermore, although not explicitly designed for sparse-view settings, our framework generalizes remarkably well, competing favorably with even dedicated sparse-view methods while maintaining a superior accuracy-efficiency trade-off.

I. INTRODUCTION

Deep MVS methods improve reconstruction but remain vulnerable to unseen scenes, occlusions, limited overlap, and ambiguous geometry. M2Depth addresses these limits by coupling monocular foundation priors with cascade MVS through mutual refinement and prior-guided cost-volume fusion.

Motivation

  • Deep MVS approaches often generalize poorly to unseen structures and lose accuracy under occlusions or limited view overlap.Ambiguous scene geometry makes these regions especially challenging.Source: §I · p. 1
  • Depth Foundation Models provide robust monocular predictions in unseen domains but lack metric scale and explicit multi-view consistency.These limitations restrict their direct use for accurate 3D reconstruction.Source: §I · p. 1

Research gap

  • Existing hybrid methods use the complementary modality as a static, one-way prior, allowing inaccurate-prior errors to propagate without correction.MVS-centric pipelines remain incomplete in difficult regions, while DFM-centric pipelines remain vulnerable to scale misalignment.Source: §I · pp. 1–2

Proposed framework

  • M2Depth tightly couples a DFM with cascade MVS through bidirectional co-refinement, using MVS depth for monocular scale alignment and monocular depth for MVS structure.Both representations are progressively improved throughout the cascade.Source: §I · p. 2
  • Prior-guided cost-volume refinement fuses monocular and multi-view information with bin masks to improve spatial consistency.The framework targets clean, complete depth maps and accurate 3D reconstruction in challenging cases.Source: §I · p. 2

Results

  • The method achieves high accuracy and strong generalization across multiple benchmarks, including depth estimation, 3D reconstruction, and sparse-view settings.It is reported to outperform existing MVS methods and dedicated sparse-view methods.Source: §I · p. 2

B. Depth Foundation Model

DFM integration has improved stereo and MVS reconstruction, but static priors and specialized sparse-view pipelines leave scale, completeness, and generalization challenges. M2Depth instead uses explicit cascade MVS with prior-guided cost construction and iterative mutual refinement.

Background

  • Depth Foundation Models offer strong zero-shot generalization, while DFM-based stereo methods exploit their monocular depth priors for matching.Their application to MVS remains comparatively limited.Source: B. Depth Foundation Model · p. 2

Related approaches

  • MonoMVSNet injects a static monocular prior into MVS, whereas MVSAnywhere uses multi-view information as a prior within a DFM-centric pipeline.These represent two primary directions for combining DFMs and MVS.Source: B. Depth Foundation Model · p. 2

Proposed approach

  • M2Depth uses bidirectional co-refinement so MVS and monocular estimates iteratively correct one another, addressing scale ambiguity and MVS incompleteness.This avoids the error propagation and geometric misalignment associated with unidirectional fusion.Source: B. Depth Foundation Model · p. 2

Sparse-view reconstruction

  • Unlike specialized sparse-view methods, the framework remains rooted in explicit multi-view geometry and avoids expensive implicit surface optimization.Prior-guided cost construction and mutual depth refinement adapt cascade MVS to sparse-view conditions.Source: C. Sparse-View Reconstruction · p. 3

MVS pipeline

  • The pipeline uses a four-stage coarse-to-fine MVS process that progressively refines depth from 1/8 to full resolution.At each stage, cost volumes become probability volumes from which MVS depth is estimated.Source: §III · p. 3

Prior-guided refinement

  • Prior-guided refinement combines multi-view and monocular cost volumes with uniformly spaced depth-bin masks as inputs to Cost-Crafter.The monocular prior supplies structural and fine-grained shape cues for depth refinement.Source: §III · p. 3

A. Prior Depth-Guided Cost Volume Refinement

Cost-Crafter refines the initial multi-view cost volume by combining stereo evidence with monocular depth structure. Depth-bin tokenization and bidirectional pixel-bin attention provide efficient, depth-aware global reasoning before producing a refined probability volume.

  • Cost-Crafter fuses multi-view and monocular cost volumes with structural bin masks to regularize ambiguous matching regions.The refinement is applied at the first cascade stage, where reliable global geometry establishes a strong coarse representation.Source: A. Prior Depth-Guided Cost Volume… · pp. 3–4
  • The module constructs multi-view costs, monocular costs from soft one-hot depth encoding, and uniformly spaced depth bins as inputs.The multi-view volume aggregates stereo matching costs, while the monocular volume is derived from the prior depth map.Source: A. Prior Depth-Guided Cost Volume… · p. 4
  • Depth-bin tokenization compresses flattened pixel-depth features into M representative tokens, enabling efficient global context modeling while preserving depth-aware structure.Pixels sharing similar prior depth levels are grouped through a replicated bin mask and masked average pooling.Source: A. Prior Depth-Guided Cost Volume… · p. 4
  • Learnable scalar positional biases inject ordinal relationships between depth layers into cross-attention without explicitly forming an additional bias matrix.The query and key projections receive bias terms, while the value projection receives a zero scalar so the bias affects attention weights rather than propagated semantic features.Source: A. Prior Depth-Guided Cost Volume… · p. 4
  • Bidirectional pixel-bin interaction first aggregates global depth cues into bin tokens and then broadcasts refined responses back to pixels.This design improves structural coherence while preserving local detail, including in textureless, occluded, or weakly matched regions.Source: A. Prior Depth-Guided Cost Volume… · pp. 4–5
  • The attention-enhanced volume is reshaped and processed with a 1 × 1 × 1 convolution, 3D LayerNorm, and depth-wise softmax to produce a refined probability volume.The resulting refinement combines monocular priors, bin-level reasoning, and ordinal depth-aware attention.Source: A. Prior Depth-Guided Cost Volume… · p. 5

B. Mutual Depth Refinement

Mutual Depth Refinement iteratively couples monocular and MVS predictions through symmetric branches, using reliable multi-view geometry to align monocular scale and monocular structure to improve MVS completeness. Confidence-aware signals and adaptive anchoring make the process robust to uncertain, sparse-view regions.

MVS-Guided Alignment

  • MDR progressively corrects spatially varying monocular scale errors rather than applying a single global scale-shift correction.MVS-GA uses multi-view geometric cues for local scale alignment, while Mono-GR refines MVS depth with scale-aligned monocular structure.Source: B. Mutual Depth Refinement · pp. 5–6

Mutual Depth Refinement

  • MDR uses symmetric MVS-GA and Mono-GR branches to predict residual updates for the opposite depth modality.Reliable MVS geometry anchors monocular depth, while monocular structure restores regions where photometric matching is weak.Source: B. Mutual Depth Refinement · p. 5

Confidence-Guided Refinement

  • The probability-volume curvature measures peak sharpness, with high curvature indicating clear matching and flat or multimodal responses indicating unreliable regions.This confidence proxy targets uncertainty in textureless or occluded areas.Source: B. Mutual Depth Refinement · p. 5
  • The refinement uses confidence-weighted cross-modal discrepancies so monocular cues dominate mainly where MVS matching is unreliable.High-confidence MVS pixels act as geometric anchors that discourage large MVS updates, whereas low-confidence regions allow stronger monocular completion.Source: B. Mutual Depth Refinement · p. 6

MVS-Specific Design

  • MVS-specific refinement signals include a geometry feature volume, curvature-based confidence, confidence-gated discrepancy encoding, and persistent EMA confidence updates.These components address multi-view coupling, cross-view uncertainty aggregation, and coarse-to-fine depth evolution.Source: B. Mutual Depth Refinement · pp. 5–6

Cascade Propagation

  • After refinement, scale-corrected monocular depth and structurally enhanced MVS depth are forwarded to subsequent cascade stages as updated priors and depth hypotheses.This iterative bidirectional process maintains local scale alignment and geometry completion across stages.Source: B. Mutual Depth Refinement · p. 6

C. Loss Function

The cascade is trained with complementary losses that supervise depth distributions, intermediate mutual refinements, and local depth ordering across stages.

Loss Components

  • Cross-entropy loss measures depth accuracy at each cascade stage by comparing predicted and ground-truth depth distributions.Source: C. Loss Function · p. 6
  • A decayed ℓ1 mutual refinement loss supervises every intermediate MVS depth map, emphasizing later refinements with r = 0.9.Source: C. Loss Function · p. 6
  • An order-preserving loss penalizes violations in relative depth ordering among sampled valid-region pixel pairs.The loss is zero for correct ordering and increases linearly when predicted ordering disagrees with ground truth.Source: C. Loss Function · p. 6

Loss Aggregation

  • The final training objective aggregates losses across all cascade stages, with λmut = λord = 0.02 balancing the auxiliary terms against cross-entropy.Source: C. Loss Function · p. 6

IV. EXPERIMENT

The experiments evaluate reconstruction quality, generalization, sparse-view robustness, and component contributions across standard and challenging multi-view settings.

Evaluation Scope

  • The method is evaluated on DTU and Tanks and Temples for 3D reconstruction, and on RobustMVD for cross-dataset depth-estimation generalization.Source: §IV · p. 6
  • Sparse-view robustness is examined on DTU using only three unfavorable input views.Source: §IV · pp. 6–7

Qualitative Results

  • Qualitative DTU results report highly complete 3D reconstructions and clean depth maps on challenging Scans 48 and 77.Source: Figure 5 · p. 7

Quantitative Evaluation

  • DTU quantitative results use point-cloud distance metrics for accuracy, completeness, and overall error, together with runtime and GPU memory usage.Source: Table I · p. 7

A. Implementation Details

Implementation uses a four-stage coarse-to-fine cascade with stage-specific depth hypotheses, feature dimensions, refinement iterations, and standard DTU training protocols.

Model Configuration

Training Protocol

Hardware and Runtime

  • Testing runs on an Intel i9-13900KS CPU with an NVIDIA GeForce RTX 3090 Ti GPU, using per-sample depth inference runtimes unless otherwise noted.Table IV runtime additionally includes multi-view fusion and benchmark-specific post-processing.Source: A. Implementation Details · p. 7

B. Benchmark Performance

Across DTU, TNT, and RobustMVD, the method achieves stronger reconstruction and depth-estimation performance than conventional MVS and DFM-based baselines. It also preserves quality under sparse-view conditions, with added runtime but comparable memory usage.

DTU

  • The method obtains the best DTU accuracy, completeness, and overall errors among conventional MVS and DFM-based approaches.The completeness gain indicates recovery of missing or ambiguous structures and denser reconstruction.Source: B. Benchmark Performance · p. 7
  • It consistently outperforms MonoMVSNet across DTU’s accuracy, completeness, and overall metrics.The monocular prior guides cost-volume construction and mutual depth refinement rather than serving only as an auxiliary cue.Source: B. Benchmark Performance · p. 7

Efficiency

  • Under sparse-view evaluation, the method maintains robust reconstruction quality with marginal additional cost and comparable memory usage.The added runtime is described as justified by generalization gains over MonoMVSNet.Source: B. Benchmark Performance · p. 7, Table IV · p. 8

TNT

RobustMVD

C. Generalization to Sparse-View Reconstruction

The method generalizes strongly to sparse-view reconstruction despite not being specifically designed or trained for that setting. Under limited-overlap protocols, it outperforms both general MVS and dedicated sparse-view methods while retaining an efficient pipeline.

Evaluation setting

Quantitative comparison

  • The method achieves the lowest average Chamfer Distance under the three-view SparseRecon protocol.It outperforms both general MVS methods and sparse-view-targeted methods including UFORecon and SparseRecon.Source: C. Generalization to Sparse-View… · p. 8

Qualitative results

  • Qualitatively, the method recovers more complete and cleaner geometry than competing methods in challenging sparse-view regions.Source: Figure 6 · p. 9

Two-view reconstruction

  • In two-view reconstruction, the method produces complete, aligned, scale-corrected surfaces where MonoMVSNet has outliers and MVSAnywhere has scale ambiguity.Source: Figure 7 · p. 9

Efficiency

  • Its margin over SparseRecon is modest, but it offers a superior accuracy-efficiency trade-off with minimal computational overhead.Dedicated baselines rely on intensive volume-rendering-based surface optimization, increasing runtime and memory costs.Source: C. Generalization to Sparse-View… · p. 9

Two-view reconstruction

D. Ablation Study

Ablations show that Cost-Crafter and mutual depth refinement provide complementary benefits: the former strengthens geometric initialization, while the latter improves smoothness and completeness. Their combination, together with auxiliary losses, yields the strongest balanced reconstruction quality.

Component integration

  • Progressively adding Cost-Crafter, MDR, and auxiliary losses consistently improves reconstruction quality over the baseline.Source: D. Ablation Study · p. 9

Loss ablation

Component roles

  • Cost-Crafter supplies accurate coarse geometry, whereas MDR produces smoother and denser depth but depends on reliable initial hypotheses.Cost-Crafter alone remains noisy and incomplete in textureless regions, while MDR alone struggles to correct early geometric errors.Source: D. Ablation Study · p. 9

Overall design

  • The full framework combines structural priors, bidirectional refinement, and cascade MVS to mitigate scale ambiguity and MVS incompleteness.Source: §V · pp. 9–10

Efficiency

  • The DFM encoder uses a DINOv2 ViT-Small backbone with approximately four times fewer parameters than MVSFormer++’s ViT-Base.This modification is intended to improve computational efficiency.Source: A. Feature Extraction and Fusion · p. 11

Feature design

B. Monocular Volume Construction

The monocular volume construction normalizes and softly encodes monocular depth against discrete inverse-depth hypotheses before fusing it with multi-view costs. This design supports geometric initialization, progressive scale correction, and robustness to flawed monocular priors.

Monocular volume

Depth normalization

  • MAD removes statistical outliers before normalization and rescaling, producing more accurate and consistent monocular scale alignment.Source: Figure 8 · p. 11

Depth encoding

Cost fusion

Module complementarity

Progressive refinement

Robustness

View robustness

  • Reducing source views from five to three increases the method’s overall and > 2 mm depth errors by 3.6% and 11.3%, respectively.The corresponding baseline increases are 4.1% and 19.4%, indicating stronger sparse-view robustness.Source: A. Additional Ablation Study · p. 12, Table VI · p. 13

2) Qualitative Results on DTU:

On DTU and broader cross-dataset qualitative evaluations, the method combines monocular structural completeness with MVS geometric accuracy, correcting scale while preserving details and sharp boundaries.

Related papers

Loading 2608.20788v1…