PaperlayerView exact revision on arXiv

Source-linked AI summary

MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details

Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, Jiaolong Yang

arXiv:2507.02546v1cs.CV

TL;DR

Monocular geometry estimation still lacks a method that simultaneously provides accurate relative geometry, metric scale, and fine-grained detail. MoGe-2 extends MoGe with decoupled scale prediction and refined real-data supervision, achieving superior performance across these capabilities while retaining overall accuracy. Its main limitation is difficulty with extremely fine structures, severe foreground-background scale differences, and out-of-distribution metric-scale ambiguity.

  • Problem

    Existing monocular geometry methods do not simultaneously address geometry accuracy, metric prediction, and fine-grained detail, despite their importance for broad applications.

  • Method

    MoGe-2 decouples affine-invariant point-map recovery from global scale prediction and refines real training data with synthetic-label filtering and completion.

  • Results

    MoGe-2 achieves superior accurate geometry, precise metric scale, and visual sharpness, advancing monocular geometry estimation for real-world applications.

  • Takeaways & Limitations

    The model combines relative geometry accuracy, metric reconstruction, and fine-grained detail recovery in a single open-domain monocular geometry system.

  • Takeaways & Limitations

    The method struggles with extremely fine structures, straight structures under large foreground-background scale differences, and out-of-distribution metric-scale ambiguity.

Abstract

from arXiv · show

We propose MoGe-2, an advanced open-domain geometry estimation model that recovers a metric scale 3D point map of a scene from a single image. Our method builds upon the recent monocular geometry estimation approach, MoGe, which predicts affine-invariant point maps with unknown scales. We explore effective strategies to extend MoGe for metric geometry prediction without compromising the relative geometry accuracy provided by the affine-invariant point representation. Additionally, we discover that noise and errors in real data diminish fine-grained detail in the predicted geometry. We address this by developing a unified data refinement approach that filters and completes real data from different sources using sharp synthetic labels, significantly enhancing the granularity of the reconstructed geometry while maintaining the overall accuracy. We train our model on a large corpus of mixed datasets and conducted comprehensive evaluations, demonstrating its superior performance in achieving accurate relative geometry, precise metric scale, and fine-grained detail recovery -- capabilities that no previous methods have simultaneously achieved.

1 Introduction

MoGe-2 extends affine-invariant monocular geometry estimation with metric scale prediction and sharper detail recovery. It combines a decoupled metric representation with refined real-data training to target accuracy, metric precision, and granularity together.

  • Monocular geometry estimation predicts 3D scene geometry from one image and can additionally recover camera intrinsics for broader 3D applications.
  • Existing MGE methods fall short of simultaneously delivering accurate geometry, metric scale, and fine-grained detail, although all three matter for real-world applications.
  • MoGe-2 extends MoGe’s affine-invariant point-map approach with metric geometry prediction and improved granularity for intricate details.
  • A pragmatic refinement pipeline filters mismatched real-data depths and completes them with edge-preserving inpainting to support sharp detail learning while retaining real-data generality.
  • The model is trained on extensive synthetic and real datasets and evaluated across datasets and metrics for relative geometry, metric scale, and fine-grained detail recovery.
  • The metric framework decouples affine-invariant point-map prediction from global scale estimation, with empirical evidence supporting this design.
  • MoGe-2 achieves state-of-the-art geometry accuracy and sharpness, surpassing prior methods in global and local geometry accuracy.

2 Related Works

Related work spans monocular metric depth estimation and monocular geometry estimation. Prior methods address metric prediction or reconstruct 3D point maps using varied representations and data sources.

  • Early monocular metric-depth methods targeted specific domains such as indoor environments or street views using limited RGBD-camera or LiDAR data.
  • Recent metric-depth methods use diverse datasets for open-domain prediction, including approaches that address ambiguity across data sources with canonical camera transformations.
  • Monocular geometry estimation predicts a scene’s 3D point map from a single image, often by combining depth estimation with camera-parameter recovery.
  • MoGe-2 also supports metric-depth estimation through the z-channel of its predicted point map and reports stronger open-domain metric-depth performance than existing approaches.

3 Methodology

MoGe-2 extends affine-invariant monocular geometry estimation with decoupled metric-scale prediction and refined real-data supervision. The method combines global scale recovery with filtering and completion strategies to preserve relative accuracy while recovering sharper geometric detail.

  • 3.1 Preliminaries: MoGe: MoGe estimates affine-invariant 3D point maps with unknown global scale and shift, using robust alignment and multi-scale local supervision.The approach also recovers camera focal and shift through optimization after predicting the point map.
  • 3. Methodology: MoGe-2 targets the remaining limitations of MoGe: metric scale prediction and fine-grained detail reconstruction.These capabilities are motivated by applications requiring accurate global geometry, metric scale, and sharp local features.
  • 3.2 Metric Scale Geometry Estimation: The method explores shift-invariant point maps and decoupled scale prediction as two alternatives for extending MoGe to metric geometry estimation.Shift-invariant prediction absorbs metric scale into the point map, whereas decoupled prediction maintains an affine-invariant geometry branch and adds separate scale supervision.
  • 3.2 Metric Scale Geometry Estimation: A naive convolutional scale head worsens metric scale predictions, so the model instead uses global CLS-token information for scale estimation.The convolutional head shares the convolution neck with point-map prediction and does not improve relative geometry.
  • 3.2 Metric Scale Geometry Estimation: The final design predicts metric scale from a DINOv2 CLS-token-conditioned MLP while retaining the affine-invariant point-map branch.Global encoder features support scale prediction, while decoupling avoids unstable scale learning and interference with relative geometry.
  • 3.3 Real Data Refinement for Detail Recovery: Real-data refinement filters mismatched or incomplete geometry using synthetic predictions, then completes filtered regions while preserving robust depth and image-aligned details.The process addresses LiDAR synchronization mismatches and SfM omissions of reflective surfaces, thin structures, and sharp boundaries.

4 Experiments

MoGe-2 is evaluated across diverse datasets and metrics for relative geometry, metric geometry, depth, and boundary sharpness. Ablations show that decoupled scale prediction and refined real data preserve accuracy while improving detail.

  • Experimental setup: The model is trained on 24 datasets and evaluated on 10 benchmarks spanning indoor, street, object-scan, and synthetic-animation domains.The training mix includes synthetic, LiDAR-scanned, and SfM-reconstructed datasets.
  • Relative geometry and depth: MoGe-2 outperforms all existing baselines across every relative-geometry metric while remaining comparable to MoGe.This indicates that metric-scale estimation does not compromise relative geometry accuracy.
  • Metric geometry and depth: MoGe-2 largely surpasses existing methods across every metric for metric point maps and metric depth.Evaluation also uses ground-truth camera intrinsics for methods that accept them, reducing the influence of intrinsic-estimation errors.
  • Boundary sharpness: MoGe-2 achieves boundary sharpness comparable to Depth Pro while significantly outperforming it in relative and metric-scale geometry accuracy.Sharpness is measured with boundary F1 on two synthetic and two real-world datasets.
  • Qualitative comparison: Visual comparisons show that MoGe-2 produces sharp metric-scale geometry, whereas MoGe and UniDepth V2 miss significant details.Depth Pro shows reduced geometric accuracy in the open-domain crocodile example.
  • Metric scale ablation: Decoupling metric scale from relative geometry improves overall performance, with an MLP scale head outperforming a convolutional head in metric geometry.The results indicate that global information is important for predicting metric scale.
  • Real data refinement: Refined real-world datasets retain nearly the original geometric accuracy while maintaining reasonable sharpness, unlike exclusively synthetic or raw-real training.Synthetic-only training gives the highest sharpness but significantly lower geometric accuracy; raw-real training improves accuracy but reduces sharpness.

5 Conclusion

MoGe-2 extends monocular geometry estimation with metric scale and fine-grained detail recovery. Its main limitation is difficulty with extremely fine structures, large foreground-background scale differences, and out-of-distribution metric-scale ambiguity.

  • Conclusion: MoGe-2 combines decoupled relative-geometry and global-scale prediction with a data-refinement pipeline for sharper real-data geometry.The method extends MoGe while retaining affine-invariant representations and improving geometric granularity.
  • Conclusion: The model achieves accurate geometry, precise metric scale, and visual sharpness for open-domain monocular geometry estimation.The paper presents these capabilities as advancing applicability to real-world applications.
  • Limitations: The method struggles with extremely fine structures, straight alignment under major foreground-background scale differences, and metric-scale ambiguity in out-of-distribution scenarios.Future work targets improved architectures and additional real-world priors.

A.1 Network architectures

The architecture uses a DINOv2 encoder, a shared multi-scale feature pyramid, and independently decoded prediction heads. UV positional encoding preserves rectangular aspect-ratio information, while scalar outputs use a CLS-token-conditioned MLP head.

  • DINOv2 Image Encoder: DINOv2 supports variable input resolutions through interpolatable positional embeddings and a user-specified image-token count.The image is resized to a patch-aligned resolution determined by the target token count.
  • Convolutional neck and heads: UV positional encoding maps the rectangular image domain into a unit circle while preserving raw aspect-ratio information.The encoding is injected at each neck scale level.
  • Convolutional neck and heads: A shared intermediate feature pyramid feeds heads that independently decode task-specific output maps.This combines multi-scale feature sharing with head-specific prediction decoding.

A.2 Training Data

Training uses publicly available datasets with MoGe-based sampling weights, while evaluation aligns predictions according to the representation and task. Metric depth is evaluated without alignment, whereas metric point maps use optimal translation alignment.

  • Training data: The training datasets are publicly available and use sampling weights following the MoGe protocol.The complete dataset list is provided in Table A.1.
  • Training-data comparison: Table A.2 contextualizes labeled training-frame counts and pretrained backbones across representative monocular geometry methods.The paper cautions that performance does not necessarily correlate with training-data quantity.
  • Evaluation protocol: Relative-geometry predictions are aligned in scale, and in shift when applicable, before error measurement.The protocol covers scale-invariant and affine-invariant point, depth, and disparity representations.
  • Evaluation protocol: Affine-invariant disparity uses least-squares alignment in disparity space, with disparity defined as d_i = 1/z_i.Aligned disparities are truncated before inversion to prevent excessively small or negative values.
  • Evaluation protocol: Metric depth is evaluated without alignment or clamping unless a model’s inference pipeline specifies post-processing.Metric point maps instead use optimal translation alignment with the ground truth.

B.1 Test-time Resolution Scaling

MoGe-2 is evaluated across test-time resolutions to assess geometry accuracy, boundary sharpness, and latency. Its broad-resolution training and lightweight design support robust performance and a favorable latency–performance balance.

  • Resolution Scaling: Training across 1200–3600 image tokens enables evaluation over a broad range of native input resolutions.These token counts correspond approximately to native input resolutions from 4842 to 11882.
  • Geometry Accuracy: Models trained across diverse resolutions and aspect ratios better maintain geometry accuracy under moderate resolution shifts.The comparison contrasts MoGe and UniDepth V2 with fixed-resolution models that degrade when evaluated away from their training setting.
  • Boundary Sharpness: Higher input resolutions and more image tokens generally produce sharper boundaries in dense predictions.Boundary sharpness is evaluated across test-time resolutions for several DINOv2-based methods, while Depth Pro uses a fixed high resolution.
  • Latency Trade-off: Inference latency scales roughly linearly with the number of image tokens.Despite a shared ViT backbone among compared methods, runtime varies with decoder complexity and architectural choices.
  • Latency Trade-off: MoGe-2 demonstrates the most favorable balance between latency and performance across different resolutions.The comparison averages percentile ranks across evaluated datasets and metric and relative geometry accuracy, with latency measured on an NVIDIA A100 using FP16.

B.2 Runtime Analysis

The runtime analysis compares representative test-time configurations using single-frame inference latency and peak GPU memory on an NVIDIA A100.

  • Runtime Metrics: Runtime evaluation measures single-frame inference latency and peak GPU memory usage on an NVIDIA A100 GPU.These metrics compare computational efficiency and resource requirements across different architectures.

B.3 More Visual Results

Additional qualitative comparisons on open-domain images are provided to show visual results beyond the main paper.

  • Qualitative Results: Figures B.4 and B.5 provide more visual results for qualitative comparison.The figures contain additional open-domain image results across two parts.

B.4 Complete Evaluation on Individual Datasets

The appendix supplements averaged evaluations with dataset-level results and additional qualitative comparisons. It also includes per-dataset ablation results.

  • Individual-Dataset Results: Tables B.4 and B.5 report evaluation and ablation results for individual datasets or sets.The paper notes that the main text lists average performance across multiple datasets for qualitative comparison and ablation study.
  • Visual Comparisons: The qualitative comparison includes Depth Pro, UniDepth V2, MoGe, the proposed method, and the RGB image.These labels identify the methods and input image shown in the visual comparison.
  • Visual Comparisons: Figures B.4 and B.5 provide additional visual comparisons on open-domain images.The visual results are split across two figures and are intended to be viewed zoomed in.
Loading 2507.02546v1…