PaperlayerView exact revision on arXiv

Source-linked AI summary

MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details

Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, Jiaolong Yang

arXiv:2507.02546v1cs.CV

TL;DR

Monocular geometry estimation still lacks methods that jointly provide accurate geometry, metric scale, and fine-grained detail. MoGe-2 decouples relative geometry from global scale and refines real-world training data with synthetic labels; it reports strong geometry accuracy, metric-scale prediction, and visual sharpness.

  • Problem

    Existing monocular geometry methods do not simultaneously address accurate geometry, metric scale, and fine-grained detail, all important for real-world applications.

    Source: §1 · p. 1

  • Method

    MoGe-2 separates affine-invariant relative geometry from global scale prediction and refines real-world data with synthetic labels to improve detail while maintaining accuracy.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    MoGe-2 achieves state-of-the-art geometry accuracy and sharpness, significantly surpassing prior methods in global and local geometry accuracy, while providing precise metric scale.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    MoGe-2 advances the applicability of monocular geometry estimation in real-world applications.

    Source: §5 · p. 9

    MoGe-2 struggles with extremely fine structures and aligned straight structures under large foreground-background scale differences, while real-world metric-scale ambiguity can cause out-of-distribution deviations.

    Source: §5 · p. 9

Abstract

from arXiv · show

We propose MoGe-2, an advanced open-domain geometry estimation model that recovers a metric scale 3D point map of a scene from a single image. Our method builds upon the recent monocular geometry estimation approach, MoGe, which predicts affine-invariant point maps with unknown scales. We explore effective strategies to extend MoGe for metric geometry prediction without compromising the relative geometry accuracy provided by the affine-invariant point representation. Additionally, we discover that noise and errors in real data diminish fine-grained detail in the predicted geometry. We address this by developing a unified data refinement approach that filters and completes real data from different sources using sharp synthetic labels, significantly enhancing the granularity of the reconstructed geometry while maintaining the overall accuracy. We train our model on a large corpus of mixed datasets and conducted comprehensive evaluations, demonstrating its superior performance in achieving accurate relative geometry, precise metric scale, and fine-grained detail recovery -- capabilities that no previous methods have simultaneously achieved.

1 Introduction

MoGe-2 extends MoGe [61] toward monocular geometry estimation that combines accurate relative geometry, metric scale, and sharp fine-grained detail. It decouples an affine-invariant point map from global scale and refines real-data labels to improve detail without sacrificing geometry accuracy.

1 Introduction

  • Monocular geometry estimation can lift image pixels into 3D by predicting camera intrinsics, and its desired capabilities include accurate geometry, metric prediction, and fine-grained detail.Metric scale matters for applications such as SLAM, autonomous driving, and embodied AI; sharp features also support image editing and generation.Source: §1 · p. 1
  • A real-data refinement pipeline filters mismatched or false depth values, especially near object boundaries, then uses edge-preserving inpainting to fill missing regions.It enables finer details while maintaining accuracy comparable to training on full, unprocessed real data.Source: §1 · p. 2
  • MoGe-2 achieves superior relative geometry accuracy, metric scale precision, and fine-grained detail recovery, capabilities not simultaneously achieved by previous approaches.The paper reports state-of-the-art results in geometry accuracy and sharpness, including global and local geometry accuracy.Source: §1 · p. 2, Figure 1 · p. 2
  • The proposed metric framework retains an affine-invariant point map and predicts global scale separately, better preserving relative geometry than the shift-invariant alternative.Both strategies mitigate focal-distance ambiguity; the decoupled approach likely benefits from its well-normalized point-map space.Source: §1 · p. 2

2 Related Works

The section positions MoGe-2 against prior work on open-domain metric depth, monocular geometry, fine-detail depth prediction, and RGB-depth data refinement. It identifies metric scale, geometric accuracy, fine detail, and diverse data artifacts as challenges addressed by the method.

Monocular metric depth estimation

  • Open-domain metric-depth methods include Metric3D [73] [23], ZoeDepth [5], and UniDepth [44] [45]; MoGe-2 reports outperforming existing approaches in open-domain metric-depth prediction.Earlier methods [13] [70] [4] [20] focused on specific domains. Metric3D uses canonical camera transformation, ZoeDepth adds domain-specific metric heads to a relative-depth framework, and UniDepth learns from metric and non-metric data.Source: §2 · pp. 2–3

Monocular geometry estimation

  • Prior monocular geometry methods either separate depth and camera recovery [71] [72] [44] [45] or predict point maps end-to-end; MoGe [61] lacks metric scale and finer details.DUSt3R [62] predicts point maps end-to-end for stereo images, while MoGe [61] predicts affine-invariant point maps for monocular input.Source: §2 · p. 3

Depth prediction with fine-grained details

  • Fine-detail depth methods use patch fusion [40] [35], diffusion models [27] [16] [18], or synthetic-data training [67] [7], but face stitching artifacts, domain gaps, or reduced geometric accuracy.MoGe-2 combines synthetic and real data with a real-data refinement strategy to pursue both fine-detail recovery and precise geometry.Source: §2 · p. 3

RGB-depth data misalignment artifacts

  • LiDAR and SfM depth data can contain misalignment, ghost surfaces, and incomplete reconstruction; prior fixes are artifact-specific or computationally expensive, unlike MoGe-2’s unified refinement approach.Prior approaches use stereo cues [56] or epipolar geometry [83] for LiDAR issues and neural rendering [37] to regenerate SfM depth maps.Source: §2 · p. 3

3 Methodology

MoGe-2 extends affine-invariant MoGe to predict metric-scale 3D geometry while refining real training data to recover fine detail.

3.1 Preliminaries: MoGe

  • MoGe predicts affine-invariant point maps with unknown global scale and shift, using ROE alignment and multi-scale spherical-region supervision to improve local geometry.The point map supports relative geometry, but MoGe lacks metric scale and fine-grained detail.Source: §3.1 · pp. 3–4

3.2 Metric Scale Geometry Estimation

  • MoGe-2 retains the affine-invariant geometry branch and predicts scale separately; a CLS-token MLP improves metric geometry over a convolution head while maintaining relative geometry.The CLS token supplies global information for scale prediction; the convolution head worsens metric-scale prediction and does not improve relative geometry.Source: §3.2 · pp. 4–5

3.3 Real Data Refinement for Detail Recovery

  • The refinement pipeline uses synthetic predictions to locally filter mismatched real depths, then completes filtered regions with predicted detail while retaining real depth as a boundary condition.Local alignment avoids bias from inaccurate absolute depth predictions; logarithmic-space Poisson completion aligns reconstructed gradients with predicted depth while preserving robust absolute depth.Source: §3.3 · p. 6

4 Experiments

Across diverse benchmarks, MoGe-2 achieves accurate metric-scale geometry while retaining strong relative geometry and sharp details. Ablations support decoupling scale prediction from relative geometry and refining real-world training data.

4 Experiments

  • The model is trained on 24 datasets: 16 synthetic, three LiDAR-scanned, and five SfM-reconstructed datasets.Dataset weights, loss functions, cropping, and augmentation follow MoGe [61].Source: §4 · p. 7

4.1 Quantitative Evaluation

  • MoGe-2 outperforms all baselines across relative-geometry metrics and achieves results comparable to MoGe, without compromising relative accuracy for metric-scale estimation.The evaluation covers scale-invariant, affine-invariant, and local point maps, plus scale-invariant, affine-invariant, and affine-invariant disparity depth measures.Source: §4.1 · p. 8
  • MoGe-2 largely surpasses existing methods across every metric for metric point and depth accuracy on seven datasets.Evaluation also uses ground-truth camera intrinsics for methods that accept them, reducing the influence of estimated-intrinsics errors.Source: §4.1 · p. 8
  • MoGe-2 achieves boundary sharpness comparable to Depth Pro while significantly outperforming it in relative- and metric-scale geometry accuracy.Boundary F1 is measured on Spring, Sintel, iBims-1, and HAMMER.Source: §4.1 · p. 8

4.2 Qualitative Evaluation

  • Visual comparisons show MoGe-2 produces metric-scale point maps with sharp details, while MoGe and UniDepth V2 miss significant details and Depth Pro loses accuracy on an open-domain crocodile image.Key metric-scale measurements are annotated on ground truth and predictions for comparison.Source: §4.2 · p. 8

4.3 Ablation Study

  • Ablations show that decoupling metric scale from relative geometry improves overall performance, and an MLP scale head outperforms a convolutional head, especially for metric geometry.The results support using global information to predict scale and better separate scale prediction from relative geometry.Source: §4.3 · p. 9, Table 4 · p. 9
  • Refined real-world training data retains nearly the original data’s geometric accuracy with reasonable sharpness, unlike synthetic-only training’s high sharpness but reduced accuracy.Raw real-world data improves geometric accuracy but reduces sharpness; local-error filtering uses predictions from a model trained on synthetic data to remove mismatched real data, and Figure 7 provides a visual comparison.Source: §4.3 · p. 9, Table 4 · p. 9, Figure 7 · p. 9

5 Conclusion · A Implementation Details

MoGe-2 combines relative geometry recovery, global scale prediction, and refined real-data supervision to estimate metric-scale geometry with fine detail. It still struggles with extremely fine structures, foreground-background scale differences, and out-of-distribution metric scale.

5 Conclusion

  • MoGe-2 extends monocular geometry estimation to open-domain images with metric scale and fine-grained detail recovery.Source: §5 · p. 9
  • Decoupling relative geometry recovery from global scale prediction preserves affine-invariant representations while enabling accurate metric reconstruction.Source: §5 · p. 9
  • A data refinement pipeline enhances real data with synthetic labels, improving geometric granularity without compromising accuracy.Source: §5 · p. 9
  • The authors report that MoGe-2 achieves accurate geometry, precise metric scale, and visual sharpness for real-world monocular geometry applications.Source: §5 · p. 9
  • MoGe-2 struggles to capture extremely fine structures, including thin lines and hair.Source: §5 · p. 9
  • Under significant foreground-background scale differences, the method can fail to maintain straight and aligned structures.Source: §5 · p. 9
  • Ambiguity in real-world metric scale can cause deviations in out-of-distribution scenarios.Source: §5 · p. 9
  • Future work will address these challenges through improved network architectures and additional real-world priors.Source: §5 · p. 9

A.1 Network architectures

The architecture combines a variable-resolution DINOv2 encoder with a shared convolutional decoder and task-specific heads. UV positional encoding preserves aspect ratio, while a CLS-conditioned MLP predicts positive scale values.

DINOv2 Image Encoder

  • Four DINOv2 layers (6th, 12th, 18th, and final) are projected, reshaped, and summed to form decoder input at a resolution selected for the target image-token count.The image is resized to 14h × 14w to meet DINOv2’s input requirement, and interpolatable positional embeddings support variable resolutions.Source: §A.1 · p. 15

Convolutional Neck and Heads

  • A shared convolutional neck and task-specific heads use progressive residual blocks and transpose convolutions to upsample features, then bilinear interpolation matches the raw image size.Normalization layers are removed from ResBlocks to reduce inference latency without affecting performance or training stability.Source: §A.1 · p. 15, Figure A.1 · p. 15

Positional Encoding and Scalar Head

  • UV positional encodings preserve image aspect ratio in the shared feature pyramid, while a two-layer CLS-conditioned MLP predicts a positive scale factor via exponential mapping.Each task-specific head independently decodes its output map; the MLP hidden size equals the input feature dimension.Source: §A.1 · p. 15

A.2 Training Data … B Additional Experiments and Results

The appendix identifies MoGe-2’s training-data sources and contextualizes training-set scale, then specifies alignment protocols for relative geometry and evaluation rules for metric depth and point maps.

A.2 Training Data

  • Training datasets are publicly available for academic use, and their sampling weights follow the MoGe [61] protocol.Table A.1 lists the datasets used to train the model.Source: §A.2 · p. 15, Table A.1 · p. 16
  • Because monocular geometry methods lack a shared training set, Table A.2 contextualizes their labeled-frame counts and pretrained backbones; performance need not correlate with training-data volume.The frame-count summary is described as rough.Source: §A.2 · pp. 15–16, Table A.2 · p. 16

A.3 Evaluation Protocol

  • For relative geometry, predictions and ground truth are aligned per image in scale, and in shift when applicable, before errors are measured.The protocol follows MoGe [61] and specifies scale alignment for scale-invariant point maps and depth maps, and scale-and-shift alignment for affine-invariant point maps and depth maps.Source: §A.3 · pp. 16–17
  • Affine-invariant disparity uses least-squares alignment in disparity space, with aligned disparities truncated at 1/zmax before inversion to depth.The truncation prevents excessively small or negative aligned disparities.Source: §A.3 · p. 17

Metric Geometry

  • Metric depth is evaluated without alignment or range clamping unless post-processing is hard-coded in a method’s inference pipeline.Source: Metric Geometry · p. 17
  • Metric point maps are aligned to ground truth using the optimal translation.Source: Metric Geometry · p. 17

B.1 Test-time Resolution Scaling

Training across a broad range of token counts supports robust test-time resolution scaling, with strong geometry accuracy, sharp boundaries, and a favorable latency–performance balance.

B.1 Test-time Resolution Scaling

  • MoGe-2 is trained across 1,200–3,600 image tokens, enabling flexible use across a broad range of input resolutions.For ViT models, token count determines native input resolution through fixed-size image patches.Source: §B.1 · p. 17
  • MoGe-2 delivers the sharpest boundary predictions at each tested resolution and matches Depth Pro’s detail with significantly fewer tokens.Depth Pro operates at a fixed high resolution because of its specialized multi-scale, patch-based architecture.Source: §B.1 · p. 17, Figure B.3 · p. 18
  • Inference latency scales roughly linearly with token count, while MoGe-2’s lightweight design supports fast inference across resolutions.Compared methods share a ViT backbone, but decoder complexity and architectural choices can change runtime.Source: §B.1 · p. 17, Figure B.3 · p. 18
  • Across test-time resolutions, MoGe-2 maintains top geometry accuracy, including beyond its training range, and demonstrates the most favorable latency–performance balance.Its accuracy holds when scaled up for detail or down for faster inference.Source: §B.1 · pp. 17–18, Figure B.3 · p. 18

B.2 Runtime Analysis

The runtime analysis compares methods under representative test-time configurations using single-frame inference latency and peak GPU memory on an NVIDIA A100.

B.2 Runtime Analysis

  • Runtime was measured for each method under its representative test-time configuration on a single NVIDIA A100 GPU.Table B.3 reports the runtime statistics.Source: §B.2 · p. 18, Table B.3 · p. 18
  • The comparison uses single-frame inference latency and peak GPU memory to assess computational efficiency and resource requirements across architectures.Source: §B.2 · p. 18

B.3 More Visual Results

This subsection provides additional visual results for qualitative comparison in Figures B.4 and B.5.

B.3 More Visual Results

  • Additional qualitative comparison results are shown in Figures B.4 and B.5.Source: §B.3 · p. 18

B.4 Complete Evaluation on Individual Datasets

This section supplements average-performance reporting with evaluation results for individual datasets, including baseline comparisons and ablation results.

B.4 Complete Evaluation on Individual Datasets

  • Figures B.4 and B.5 show additional visual results on open-domain images.Both captions recommend zooming in for the best view.Source: Figure B.4 · p. 19, Figure B.5 · p. 20
  • Tables B.4 and B.5 list evaluation results for individual datasets, covering baselines and the ablation study, respectively.The paper notes that its main text listed only average performance across multiple datasets for qualitative comparison and ablation study.Source: §B.4 · p. 19, Table B.4 · p. 21, Table B.5 · p. 22

Related papers

Loading 2507.02546v1…