MoGe-3: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinement

1 Introduction

• Existing models can produce plausible global layouts yet distort fine geometry, especially thin or elongated structures, in ways that viewpoint changes reveal but 2D depth views may conceal.Examples include twisted fences and smeared poles.Source: §1 · p. 2
• The paper attributes these distortions to 2D decoding: image-plane proximity can mix features from geometrically distant surfaces across depth discontinuities, over-smoothing local structure.This mismatch is especially consequential when image proximity diverges from true 3D relationships.Source: §1 · p. 2
• The method retains a strong 2D foundation model for appearance but models geometry with 3D spatial-neighborhood interactions, decoupling image-adjacent pixels across depth discontinuities and associating same-surface pixels via 3D proximity.MoGe-3 uses SSR to iteratively refine the base geometry estimator’s initial point map in a self-guided manner, using sparse 3D convolutions after voxelizing the point map in log-depth coordinate space.Source: §1 · p. 2
• MoGe-3 achieves state-of-the-art performance on standard zero-shot benchmarks, improving local and global geometric accuracy and reducing distortions in thin and elongated structures.Its qualitative results include more accurate fine-scale geometry and cleaner, more consistent structures.Source: §1 · pp. 2–3

2 Related Works

• Monocular geometry methods progressed from relative-depth approaches to metric and generalizable estimation, with the MoGe family reformulating the task as affine-invariant point-map regression.The field’s approaches include MiDaS [18], DPT [30], ZoeDepth, Metric3D / Metric3D-v2 [57], Depth Anything / V2 [54], and UniDepth.Source: §2 · p. 3
• To improve reconstruction fidelity, prior methods use higher-resolution inputs, generative priors, pixel-space diffusion, or neural implicit fields with 2D UV queries.These approaches include Depth Pro [4], Marigold [16], GeoWizard [9], Pixel-Perfect Depth [51], and InfiniDepth [58].Source: §2 · p. 3
• Although prior methods improve reconstruction quality, they still decode 3D geometry in a 2D parameterization, so image-plane proximity can misrepresent 3D spatial relationships and limit geometric accuracy.The authors describe their approach as the first, to their knowledge, to lift monocular geometry modeling into 3D space and aggregate features via sparse 3D convolutions, mitigating erroneous feature mixing across depth discontinuities.Source: §2 · p. 3

3.1 Base Model

• The base model, initialized from MoGe-2 [45], predicts a dense point map and auxiliary outputs, while its DINOv2 encoder [26] supplies features reused during refinement.The point map is supervised up to an unknown affine transform along the optical axis, and predicted metric scale converts it to a metric point map.Source: §3 · p. 3, §3.1 · p. 3

3.2 SSR: Self-Guided Sparse 3D Refiner

• SSR factorizes each point so image-axis coordinates stay fixed while the refiner predicts residuals for log-depth, iteratively improving the input point map.Each cycle voxelizes the current geometry, runs a sparse 3D U-Net, and applies predicted residuals to update the geometry.Source: §3.2 · pp. 3–4

3.2.1 Self-Guided Voxelization

• Self-guided voxelization maps each pixel to a voxel using image coordinates and quantized log-depth, forming a scale-invariant shell with exactly HW occupied voxels.The shell’s 3D adjacency separates pixels across depth discontinuities, reducing feature mixing between distinct surfaces.Source: §3.2.1 · p. 4

3.2.2 Sparse 3D U-Net

• The sparse 3D U-Net encodes voxels with multi-scale sparse convolutions, fuses base-model 2D features at its bottleneck, and predicts log-depth residuals.Dynamic sparsity lets depth-separated structures remain distinct at coarser levels, while shared image-location features can be duplicated across voxels at different depths.Source: §3.2.2 · p. 5

3.2.3 Iterative update

• At each iteration, predicted log-depth residuals update the factorized representation, whose next voxel shell guides the subsequent refinement cycle.The loop progressively refines spatial locality around surface boundaries.Source: §3.2.3 · p. 5

3.3 Losses and Training

• Training uses a composite geometric objective at each SSR iteration and supervises auxiliary base-model outputs, including mask, normals, and global scale.A two-stage schedule first trains residual corrections with detached base features, then jointly fine-tunes the pipeline; the refiner receives synthetic-sample gradients while the base model uses synthetic and real-world data.Source: §3.3 · p. 5

4.1 Implementation Details

• The default model uses a DINOv2 ViT-L backbone with three inference refinement iterations; a ViT-G variant is also trained.Training uses AdamW and a staged schedule, with full training details in Appendix A.Source: §4.1 · p. 6

4.2 Evaluation Setting

• Evaluation spans nine synthetic and real, indoor and outdoor benchmarks, using global, fine-detail, and depth-boundary metrics.Metric-depth evaluation omits PPD and InfiniDepth, whose point maps use MoGe-2-predicted intrinsics.Source: §4.2 · p. 6

4.3 Quantitative Evaluation

• MoGe-3 achieves the best performance across all three global settings and substantially outperforms every baseline on strict local threshold accuracy δ0.01.Global metrics are averaged over nine datasets; local metrics are averaged over Spring and Synth4K.Source: §4.3 · pp. 6–7
• Boundary F1 ranks third overall, while ViT-G improves over ViT-L across most metric categories and three-step ViT-L inference takes 121 ms per frame.Boundary sharpness is a 2D pixel-level measure that does not fully capture 3D geometric quality.Source: §4.3 · p. 7, Table 2 · p. 7

4.5 Ablation studies

• D=200–300 improves local metrics while preserving global accuracy; RGB-input refinement improves local accuracy but remains behind DINO-feature injection.The full model adopts D=200 as a trade-off among geometric precision, occupancy density, and efficiency.Source: §4.5 · pp. 7, 9

4.4 Qualitative Evaluation

• Qualitative comparisons show that MoGe-3 produces higher-fidelity 3D geometry with reduced distortions, especially in complex regions and thin structures.The figure compares point maps and disparity maps on unseen test images.Source: Figure 3 · p. 8

4.5 Ablation studies

• A parameter-matched 2D U-Net yields only marginal global changes and fails to improve local geometry compared with sparse 3D refinement.The ablation variants use a frozen MoGe-2 base and five refinement iterations.Source: Table 3 · p. 9, §4.5 · pp. 9–10
• As inference iterations increase, local metrics improve toward saturation around K=5 while global metrics remain nearly unchanged.Boundary F1 gains arrive mostly in the first iteration, and performance remains stable or improves through K=7 despite training at K=3; Figure 4 visualizes progressively sharper thin structures.Source: §4.5 · pp. 9–10, Figure 4 · p. 9

5 Conclusion

• MoGe-3 lifts geometry refinement into sparse 3D scene manifolds, improving local geometric fidelity while preserving globally plausible geometry.The paper describes this as its proposed new paradigm for fine-detail monocular estimation.Source: §5 · p. 10
• Regression-based refinement struggles with pixel ambiguity along object boundaries, preventing perfectly sharp depth edges and introducing fly-points.The paper suggests generative formulations such as [51] as a possible way to address this limitation.Source: §5 · p. 10

A.1 Detailed Loss Design

• The global affine-invariant loss solves scale and optical-axis shift on a low-resolution prediction, then applies a depth-relative ℓ1 residual.The closed-form alignment solution is described as a weighted median.Source: §A.1 · p. 15
• The radial-partition local loss targets local geometric distortions that global alignment might overlook.It adapts the MoGe local loss with stochastic multi-scale partitioning and scale-shared local alignment.Source: §A.1 · p. 15
• At each evaluation, partitions use either ground-truth or globally aligned predicted geometry, group pixels into 3D regions, and supervise scales α ∈ {4, 16, 64}.Random rotation and scale perturb partition boundaries; the source selection encourages partitions to respect both ground-truth geometry and current foreground–background separation, reducing over-smoothing across local depth discontinuities.Source: §A.1 · p. 15
• Local scale-shift alignments share the global scale because their scale factors typically deviate minimally from it.The fixed scale simplifies the objective, and the resulting approach has linear computational time without compromising the effectiveness of local geometric supervision.Source: §A.1 · p. 15
• The edge-angle loss is defined over pixels whose two d-neighbors are both valid.Source: §A.1 · p. 15
• The appendix also specifies auxiliary-head losses and expands the main-text losses with solver formulations, weight clipping, and implementation hyperparameters.Source: §A.1 · pp. 15–16

A.2 Detailed Training Setup

• Training uses loss weights (λg, λℓ, λe, λm, λn, λs) = (1.0, 1.0, 1.0, 0.1, 0.1, 0.1), AdamW, staged learning rates, and batch size 96 on 16 A100 GPUs.The backbone is frozen for the first 1,000 steps, then warmed up until step 2,000; refiner gradients are detached for the first 5,000 steps.Source: §A.2 · p. 16

A.3 Training Data

• Training combines 17 synthetic datasets with pixel-perfect geometry and eight real or sensor-reconstructed datasets supplying real-world appearance and sensor statistics.The listed datasets include Hypersim and Objaverse among the synthetic sources, and ARKitScenes and ScanNet++ among the real or sensor-reconstructed sources.Source: §A.3 · p. 16

B.1 Local Fine-Grained Mask Construction

• The coarse mask unions multiscale Laplacian residual detections with top-hat detections of thin near structures and bottom-hat detections of thin far gaps.Each detector uses a threshold of three robust standard deviations; the Laplacian residual is computed at scales 8, 16, and 32, and the morphological structuring elements have sizes 3, 5, 9, and 17.Source: §B.1 · p. 16
• SAM2 segments are retained using support-density, overlap-size, and region-size criteria, and the final mask unions selected segments within valid depth.The criteria are ρ_i ≥ 0.3, |S_i ∩ S| ≥ 5, and |S_i| ≤ 0.05HW.Source: §B.1 · p. 16

B.2 Evaluation Protocol

• Local evaluation fits one global scale, then a translation for each eligible SAM2 segment while keeping that scale fixed.For point maps, each segment's shift is a 3D translation; the protocol avoids scale-collapse degeneracy from per-segment affine alignment and field-of-view sensitivity from scale-only alignment.Source: §B.2 · p. 17
• Locally aligned predictions are scored with AbsRel and δ0.01, averaged across segments.Source: §B.2 · p. 17

• Per-dataset global, local, and boundary results are reported in Tables C.1, C.2, and C.3, respectively.Source: C Additional Quantitative Results · p. 17
• Runtime statistics in Table C.4 characterize practical computational cost across methods.Source: C Additional Quantitative Results · p. 17
• Runtime measurements average 200 sampled single-frame inputs on one NVIDIA A100 GPU at batch size 1.Pixel-Perfect Depth and InfiniDepth timings cover only relative-depth prediction, while other models' timings cover the complete point-map and depth-map forward pass.Source: C Additional Quantitative Results · p. 17

D Additional Qualitative Results

• Additional qualitative results highlight fine-grained structures, depth discontinuities, and geometric details across diverse scenes.Figures D.1 and D.2 are described as showing preserved fine structures, sharp boundaries, high-fidelity reconstruction, and robustness in challenging regions.Source: D Additional Qualitative Results · p. 17, Figure D.1 · p. 20, Figure D.2 · p. 21

E Examples of Local Fine-Grained Mask

• The additional table captions cover global metrics, local metrics, boundary F1 scores, and single-frame inference runtimes.The boundary F1 caption specifies radius 1 and datasets iBims-1, HAMMER, Sintel, and Spring; the runtime caption specifies a single NVIDIA A100 GPU.Source: Table C.1 · p. 18, Table C.2 · p. 19, Table C.3 · p. 19
• Figure E.3 shows input images beside red overlays of local fine-grained masks that select thin structures and high-frequency regions for evaluation.Source: E Examples of Local Fine-Grained Mask · p. 17, Figure E.3 · p. 22