PaperlayerView exact revision on arXiv

Source-linked AI summary

MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision

Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, Jiaolong Yang

arXiv:2410.19115v3cs.CV

TL;DR

Monocular open-domain 3D geometry estimation is ill-posed, and depth-based pipelines require camera information that is difficult to infer. MoGe directly predicts affine-invariant point maps with robust global and multi-scale local supervision, achieving strong generalization and substantial gains across unseen point-map, depth, and camera-FOV benchmarks.

  • Problem

    Recovering general-scene 3D geometry from a single open-domain image is highly ill-posed, while depth-based reconstruction requires difficult camera-intrinsic estimation.

  • Method

    MoGe directly predicts affine-invariant point maps and trains them with ROE global alignment, multi-scale local geometry loss, and large-scale mixed data.

  • Results

    MoGe ranks top across evaluated tasks and metrics, reducing error by over 35% versus the previous best point-cloud MGE method on eight unseen datasets.

  • Takeaways & Limitations

    The model provides a versatile geometry estimator for monocular point-map, depth, and camera-FOV tasks across open-domain images.

  • Takeaways & Limitations

    Evaluation assumes the camera principal point is at the image center and pixels are square, simplifying translation to a Z-axis shift.

Abstract

from arXiv · show

We present MoGe, a powerful model for recovering 3D geometry from monocular open-domain images. Given a single image, our model directly predicts a 3D point map of the captured scene with an affine-invariant representation, which is agnostic to true global scale and shift. This new representation precludes ambiguous supervision in training and facilitate effective geometry learning. Furthermore, we propose a set of novel global and local geometry supervisions that empower the model to learn high-quality geometry. These include a robust, optimal, and efficient point cloud alignment solver for accurate global shape learning, and a multi-scale local geometry loss promoting precise local geometry supervision. We train our model on a large, mixed dataset and demonstrate its strong generalizability and high accuracy. In our comprehensive evaluation on diverse unseen datasets, our model significantly outperforms state-of-the-art methods across all tasks, including monocular estimation of 3D point map, depth map, and camera field of view. Code and models can be found on our project page.

1. Introduction

MoGe directly estimates affine-invariant 3D point maps from single open-domain images, using tailored global and local supervision to address monocular geometry ambiguity. It achieves strong generalization and ranks highly across point-map, depth, and camera-FOV tasks.

  • Single-image 3D geometry estimation remains challenging because recovering general-scene geometry from one image is highly ill-posed.
  • MoGe directly predicts affine-invariant point maps, allowing depth and camera focal length or FOV to be derived when needed.The representation permits unknown global scale and 3D shift, eliminating focal-distance ambiguity that harms training.
  • ROE global alignment resolves scale and shift for affine-invariant point-map supervision, while multi-scale local loss penalizes discrepancies under independent local affine alignments.The proposed supervisions target outliers and coarse global alignment, as well as ambiguous relative distances between objects.
  • MoGe is trained on large-scale data sourced from various existing datasets for open-domain monocular geometry estimation.
  • Over 35% error reduction versus the previous best point-cloud MGE method, 20%∼30% for MDE, and over 20% for camera FOV are reported on eight unseen datasets.The method ranks top across all evaluated tasks and metrics.
  • The paper presents affine-invariant point maps, global and local supervisions, and substantial improvements over existing methods across MGE, MDE, and camera-FOV estimation.The authors also identify applications including 3D-aware image editing, depth-to-image synthesis, novel-view synthesis, and 3D scene understanding.

2. Related Work

Prior work estimates monocular geometry through depth, point maps, camera parameters, or large-scale training, but these approaches retain camera-intrinsic or focal-distance challenges. MoGe addresses these issues with direct affine-invariant point-map estimation for monocular input.

  • Metric monocular depth methods often depend on RGBD, LiDAR, or calibrated stereo data, restricting applicability to particular domains.
  • Relative depth methods improve data breadth through affine-invariant prediction, but recovering 3D shape from depth still requires known camera intrinsics.
  • Monocular point-map methods predict free 3D points per pixel, using camera-parameter estimation, depth pipelines, or direct point-map prediction.
  • DUSt3R can adapt to monocular scenarios by duplicating the input image, but its scale-invariant point maps can suffer from focal-distance ambiguity.
  • Learning-based camera-intrinsics estimation for in-the-wild images remains challenging, while MoGe derives camera parameters from point maps and reports state-of-the-art accuracy and generalizability.
  • Large-scale datasets and advanced network backbones have become an emerging strategy for improving generalization in monocular geometry and depth estimation.

3. Approach

MoGe directly predicts affine-invariant 3D point maps from single images, then derives depth and camera parameters while handling undefined-geometry regions. Its training combines robust global alignment with multi-scale local and surface-aware supervision.

  • 3.1. Affine-invariant point map: MoGe predicts a 3D point map directly from a single image, with image-aligned X and Y coordinates.The model uses a ViT encoder and convolutional decoder.
  • 3.1. Affine-invariant point map: Affine-invariant representation treats predictions as equivalent under global scale and 3D translation, resolving focal-distance ambiguity in supervision.Unlike scale-only alignment, translation alignment makes supervision consistent when focal length is ambiguous.
  • 3.1. Affine-invariant point map: Depth, camera shift, and focal length can be recovered from the predicted points by minimizing projection error over masked valid regions.The iterative recovery typically converges within 10 iterations and takes approximately 3ms.
  • 3.2. Training Objectives: ROE alignment robustly and efficiently estimates scale and translation for global point-map supervision instead of relying on rough alignment approximations.Truncated absolute residuals improve robustness to outliers, while a parallelized search avoids the high complexity of generic optimization methods.
  • 3.2. Training Objectives: Training uses around 9 million frames from 21 datasets, with tailored loss combinations reflecting differences in label quality.Invalid geometry is masked, and real outdoor sky masks are obtained with SegFormer.

4. Experiments

MoGe is evaluated zero-shot across diverse datasets and tasks, where it achieves strong point-map, depth, and camera-FOV accuracy. Ablations further show that its affine-invariant representation, alignment strategy, local loss, and infinity-region mask are important for performance.

  • Point map estimation: MoGe achieves the lowest average Relp and highest δp across eight diverse point-map datasets for both point-map representations.The datasets span indoor, street-view, object-scan, and synthetic-movie domains.
  • Point map estimation: Region-wise Relp falls from 7.97 to 5.50, an approximately 30% improvement over previous methods.Local point-map accuracy is measured within segmented object regions.
  • Depth map estimation: MoGe exhibits the lowest average Reld across scale-invariant depth, affine-invariant depth, and affine-invariant disparity representations.The evaluation includes conversions between representations for comprehensive comparison.
  • Camera FOV estimation: A mean FOV error of 2.91° and median error of 2.21° on three benchmarks outperform both camera-calibration and point-map estimation methods.The benchmarks are NYUv2, ETH3D, and iBims-1.
  • Ablation study: Affine-invariant point maps outperform three alternative representations across tasks by resolving focal-distance ambiguity that hampers their training.The alternatives include scale-invariant depth with ray maps, affine-invariant depth with shift and ray maps, and scale-invariant camera-space point maps.
  • Ablation study: The proposed alignment method outperforms two alternatives across all tasks, while removing residual truncation degrades performance.The alignment strategy is therefore a critical component of the ablated system.
  • Ablation study: Removing the multi-scale local loss reduces local-geometry accuracy, while removing valid-region masks produces erroneous sky-region points and harms foreground accuracy when large infinity labels are used.The local loss supplements global supervision where distant objects have ambiguous relative positions.

5. Conclusion

The paper concludes that accurate open-domain monocular geometry estimation depends on effective supervision and affine-invariant point-map prediction. With large-scale mixed-data training, MoGe generalizes across tasks and benchmarks and is positioned as a foundation for monocular-geometry applications.

  • Conclusion: MoGe combines affine-invariant point maps with robust global alignment and multi-scale local geometry supervision for open-domain monocular geometry estimation.The representation addresses ambiguous supervision, while the alignment solver supports global shape learning.
  • Conclusion: The trained model demonstrates strong generalizability and significantly outperforms previous methods across multiple geometry tasks and benchmarks.The conclusion covers monocular geometry estimation broadly rather than a single evaluation setting.
  • Conclusion: The authors propose MoGe as a robust foundational model for applications including 3D-aware image editing and depth-to-image synthesis.These applications are presented as potential uses of the model.

A. Algorithm Details

The ROE alignment procedure solves scale and shift alignment for point maps through efficient subproblems, with optional residual truncation for robustness. Its search-based solvers reduce alignment computation while supporting 1D- and 3D-shift cases.

  • Scale subproblem: The untruncated scale subproblem finds candidate extrema by sorting ratios and evaluating derivatives and objective values.Sorting and prefix summations enable O(N log N) processing, followed by O(Nne) evaluation over extrema.
  • Robust truncation: The truncated objective clips each residual at τ to suppress outliers, although this makes the optimization non-convex.The minimum remains searchable at candidate points, allowing the solver to retain an efficient structured procedure.
  • 1D-shift alignment: The 1D-shift solver enumerates candidate correspondences and solves each resulting subproblem, achieving O(N^2 log N) total complexity.The method selects the lowest objective over possible indices after reducing the problem using the optimality condition.
  • 3D-shift alignment: For 3D shift, a shared corresponding index assumption avoids the prohibitive O(N^4 log N) brute-force search and yields an approximately optimal O(N^2 log N) method.The assumption is motivated by locality of surface points within a 3D sphere.

B.1. Training Data

Training uses a mixed collection of datasets with retrieval-based weighting relative to OpenImagesV7. Evaluation data are separately tailored through dataset-specific cropping, masking, and artifact removal for reliable comparison.

  • Training datasets: Dataset weights are balanced by estimating each dataset’s nearest-neighbor retrieval probability from OpenImagesV7 using DINOv2 features.The probability is computed for a randomly selected OpenImagesV7 image finding its nearest neighbor in each training dataset.
  • Training datasets: Training datasets are listed with their tailored loss combinations, while invalid frames may be dropped from the original data.The reported frame counts can therefore differ slightly from the source datasets.
  • Evaluation preprocessing: NYUv2 evaluation removes boundary regions, excludes depths beyond 5 meters, and masks reflective areas because of unreliable Kinect V1 measurements.The official test split contains 654 samples.
  • Evaluation preprocessing: Figure 8 illustrates removal of mirror and boundary artifacts in NYUv2 and DIODE and exclusion of sky regions in Sintel.These operations exemplify the dataset-specific preprocessing used before evaluation.

B.3. Evaluation Protocol

Evaluation aligns predictions with ground truth per image before computing errors, using scale or affine transformations according to the representation. Disparity uses least-squares affine alignment followed by truncation before depth inversion.

  • Alignment protocol: Predictions and ground truth are aligned per image in scale and, when applicable, shift before error measurement.The protocol applies this alignment consistently to models and baselines.
  • Notation: The evaluation notation distinguishes predicted and ground-truth points, depths, valid-pixel masks, and alignment parameters a and b.The parameters a and b are reserved for evaluation alignment to avoid confusion with training-objective symbols.
  • Alignment protocol: For point maps and depth maps, the protocol selects scale-only or affine scale-and-shift alignment according to the representation.The listed cases include scale-invariant and affine-invariant point and depth maps.
  • Disparity evaluation: Affine-invariant disparity is aligned by least squares using predicted disparity and ground-truth disparity defined as d_i=1/z_i.The aligned disparity is truncated by 1/z_max before inversion to prevent excessively small or negative values.

C. More Results

Additional experiments report detailed depth results and qualitative comparisons across zero-shot datasets and in-the-wild images. Comparisons include MoGe and several point-map and depth-estimation baselines.

  • Quantitative results: Table 6 provides detailed depth-estimation results, including affine-invariant depth and disparity evaluations for metric and scale-invariant methods.This broadens comparison across depth representations under a common evaluation setting.
  • Qualitative results: Qualitative comparisons evaluate MoGe against LeReS, UniDepth, DUSt3R, Metric3D V2, and Depth Anything V2 on zero-shot datasets and in-the-wild images.Depth-only baselines are visualized using MoGe’s estimated focal lengths when 3D point-cloud results are required.
  • Qualitative results: Supplementary videos show extensive uncurated comparisons using the first 100 DIV2K images.The comparisons are presented as additional qualitative evidence beyond the main figures.
  • Qualitative results: Figures 12 and 13 present additional reconstruction results for open-domain images.These figures extend the visual demonstration to more open-domain examples.

D. Limitations and Future Work

MoGe remains limited in recovering thin structures, and its single-image design does not inherently maintain temporal consistency for monocular video reconstruction.

  • Limitations: Thin structures remain difficult to capture because of limited network capacity and noisy real-world training data.The paper identifies this as a significant challenge for the model.
  • Future Work: Monocular video reconstruction remains challenging because a single-image model cannot inherently maintain temporal coherence.The paper points to global optimization techniques as a non-trivial requirement for addressing temporal consistency.
  • Limitations: The model can flatten intricate structures, as shown by its failure to recover a thin fence.Figure 9 gives a concrete failure case in which the fence geometry is flattened.
Loading 2410.19115v3…