Source-linked AI summary
FoundationGeo: Learning Spatial Pixel-Wise Fields for Monocular Metric Geometry
Muxin Liu, Xiaoyang Lyu, Tianhe Ren, Peng Dai, Xiaoshan Wu, Zhiyue Zhang, Jiaqi Zhang, Jiehong Lin, Shaoshuai Shi, Xiaojuan Qi
TL;DR
Monocular metric geometry remains difficult because relative predictions lack physical scale and metric recovery is sensitive to camera intrinsics. FoundationGeo combines a strong two-stage relative-to-metric model with pixel-wise spatial calibration and targeted focal-length augmentation, achieving the strongest overall zero-shot performance across seven benchmarks while leaving broader camera-model coverage as a limitation.
Problem
Monocular metric recovery is ill-posed, and zero-shot performance is vulnerable to camera-intrinsic distribution mismatch.
Method
FoundationGeo trains a DINOv3-initialized relative geometry model, then uses ray-direction and pixel-wise scale fields plus targeted Blender-rendered focal-length data for metric calibration.
Results
FoundationGeo achieves the strongest overall metric depth performance across seven zero-shot benchmarks, improving averaged AbsRel from 15.7 to 14.8 and δ1 from 76.8 to 80.8.
Takeaways & Limitations
Robust metric transfer depends on both spatial calibration of relative geometry and coverage of camera-intrinsic regimes, especially focal length.
Takeaways & Limitations
The current intervention mainly covers focal length and does not fully span principal point, aspect ratio, sensor size, distortion, or other camera-model factors.
Abstract
from arXiv · showhide
We present FoundationGeo, a two-stage framework that explicitly bridges relative and metric prediction via spatial calibration and principled data design. Stage 1 learns a high-fidelity, affine-invariant geometry model by initializing with DINOv3 and training on a curated 10.2M-sample multi-domain corpus with complementary local-detail supervision, yielding sharp boundaries and strong cross-domain generalization. Stage 2 moves beyond global scaling by introducing lightweight pixel-wise calibration fields for metric estimation: a scale field for spatially varying metric alignment and a ray-direction correction field that mitigates directional bias in point-map geometry, together producing metrically consistent 3D point maps. Beyond model design, we identify camera intrinsic coverage, especially focal length distribution mismatch between training and test data, as a key bottleneck for zero-shot metric generalization: performance drops sharply when test intrinsics fall outside the training distribution. To address this, we synthesize additional training data across diverse focal lengths using a Blender-based data engine, repairing under-covered focal regimes and improving robustness under intrinsic shift. Extensive zero-shot evaluations across seven benchmarks show that FoundationGeo significantly strengthens cross-domain robustness, staying near the top across diverse domains while avoiding the sharp cross-domain performance drops observed in other methods. This consistency translates into the best overall performance, surpassing heavier baselines by over 5.2% on average.
1 Introduction
FoundationGeo addresses the relative-to-metric gap by combining stronger affine-invariant geometry learning with spatial calibration and camera-intrinsic-aware data design. Its seven-benchmark zero-shot results show improved cross-domain robustness and strongest overall metric depth performance.
- Motivation: Relative depth models generalize better than monocular metric models, whose scale recovery remains ill-posed because of perspective, intrinsics, and scene-prior ambiguities.
- Related Work: Relative-to-metric methods leverage strong relative predictors with lightweight calibration, but representative MoGe-2 uses only a single global scale.
- Motivation: Pixel-wise calibration addresses spatially varying scale errors and residual ray-direction bias that a single global scale cannot resolve.Finer local scale alignment monotonically lowers AbsRel toward the per-pixel limit, while directional bias causes 3D inconsistency even after scale alignment.
- Method: The method strengthens its relative backbone with DINOv3 initialization, 10.2M curated multi-domain samples, and complementary local-detail supervision.This design targets fine-detail fidelity and cross-domain generalization before metric calibration.
- Data Design: A Blender-based data engine renders diverse focal lengths because performance drops sharply when test camera intrinsics fall outside the training distribution.The resulting augmentation repairs under-covered focal regimes and improves robustness under intrinsic shift.
- Results: FoundationGeo improves averaged AbsRel from 15.7 to 14.8 and δ1 from 76.8 to 80.8 across seven zero-shot datasets.These correspond to reported improvements of 5.7% and 5.2%, respectively.
2 Related Work
Prior work advances relative depth through expressive architectures and large-scale supervision, while metric methods resolve scale ambiguity using camera modeling or relative-to-metric calibration. FoundationGeo follows the latter direction with a two-stage relative geometry and spatial calibration pipeline.
- Relative depth estimation: Relative depth methods use transformer architectures, million-image supervision, diffusion priors, and geometry-aware constraints to achieve accurate, robust estimation.
- Metric depth estimation: Metric depth methods model camera intrinsics, predict metric heads, or jointly estimate depth and camera parameters to recover absolute geometry.
- Relative-to-metric calibration: FoundationGeo first learns affine-invariant point maps, then applies ray-direction correction and spatial scale fields to produce metric point maps.Metric depth and surface normals are derived from the calibrated point map.
3 FoundationGeo
FoundationGeo combines a high-fidelity affine-invariant relative model with spatial fields and targeted focal-length augmentation to improve metric geometry transfer. Its design addresses both within-image calibration errors and camera-intrinsic distribution mismatch.
- 3 FoundationGeo: FoundationGeo is a two-stage framework that converts affine-invariant point maps into metrically consistent 3D points and diagnoses focal-length mismatch as a residual zero-shot bottleneck.The framework first predicts affine-invariant geometry, then applies a ray-direction correction field and a spatial scale field before addressing camera-model coverage.
- 3.1 Upgrade Relative Base Model: The relative base model uses DINOv3 initialization, multi-scale feature fusion, and complementary global, local, normal, edge, and mask supervision over a curated 10.2M-image corpus.The corpus spans indoor, outdoor, driving, synthetic, and in-the-wild domains, with filtering for data quality and valid depth ranges.
- 3.2 Spatial Fields for Metric Geometry: Spatial fields replace a single global scale with pixel-wise scaling and bounded ray-direction correction, producing direction-corrected relative points before metric calibration.The unified objective keeps relative-geometry supervision active while separately supervising metric regression, scale, ray correction, and correction magnitude.
- 3.3 Focal-Length Coverage Analysis and Synthetic Augmentation: Diverse-Focal fine-tuning consistently outperforms Single-Focal across datasets, while targeted focal coverage reduces sensitivity to camera-intrinsic shifts.The Blender engine adds 23,700 images spanning the under-covered focal regime, complementing spatial calibration fields that address within-image drift and directional bias.
4 Experiments
FoundationGeo is evaluated through zero-shot experiments spanning diverse datasets, metrics, and ablations, with results emphasizing balanced cross-domain performance and the value of two-stage spatial calibration and focal-diverse data.
- 4.1 Main Results: FoundationGeo achieves the best average Rank (2.32) across seven metric-depth benchmarks, with the lowest average AbsRel and highest average δ1.It remains competitive across indoor, driving, and challenging domains, unlike methods that degrade sharply on particular datasets.
- 4.1 Main Results: FoundationGeo-Base achieves the best overall relative-depth performance across eight benchmarks, leading on ETH3D, iBims-1, and Sintel.The reported results are 2.85 AbsRel and 99.3 δ1 on ETH3D, 2.66 and 98.9 on iBims-1, and 12.7 and 83.3 on Sintel.
- 4.1 Main Results: Boundary F1 improves with a larger token budget, and FoundationGeo achieves the best overall sharpness despite Depth Pro operating at 1536×1536 native resolution.The Base Model is already comparable to Depth Pro with a 3.6k-token input, while 6k tokens consistently improves boundary F1.
- 4.1 Main Results: FoundationGeo delivers accurate metric point maps across outdoor and indoor scenes spanning meters to centimeters while preserving fine geometric structure and sharp details.The qualitative comparison attributes this stability to spatial fields and targeted focal-length augmentation under domain and depth-scale shifts.
- 4.2 Ablation Study: Spatial fields outperform image-level global scaling, improving average AbsRel from 20.1 to 19.3 and δ1 from 66.8 to 68.0.The fields provide locally adaptive scale and directional calibration for spatially varying drift and ray-direction inconsistency.
- 4.2 Ablation Study: Field-specific objectives further improve average AbsRel from 19.3 to 18.8 and δ1 from 68.0 to 69.7 by anchoring scale and ray-direction roles.The supervision encourages the fields to function as calibration modules rather than redundant geometry predictors.
- 4.2 Ablation Study: Two-stage training improves full-data results from 15.2/78.2 to 14.8/80.8 in AbsRel/δ1 compared with training the metric model from scratch.Separating affine-invariant geometry learning from metric calibration provides a stronger structural initialization.
- 4.2 Ablation Study: Diverse-focal rendered data improves robustness under camera-intrinsic shifts more consistently than fixed-focal rendering, while ray-direction correction improves angular accuracy and average metric-depth performance.Single-regime rendered data is insufficient for broad generalization, whereas focal diversity better addresses intrinsic mismatch.
5 Conclusion
FoundationGeo bridges affine-invariant relative geometry and monocular metric prediction through two-stage learning, spatial calibration, and targeted camera-intrinsic coverage. Across seven benchmarks, it achieves the best overall metric-depth performance while retaining strong relative-geometry quality.
- Conclusion: FoundationGeo achieves the best overall metric-depth performance across seven zero-shot benchmarks without requiring ground-truth camera intrinsics at inference.Its average performance is strongest across diverse domains and camera models.
- Conclusion: The two-stage framework combines DINOv3-initialized relative geometry with pixel-wise scale and ray-direction calibration fields for metrically consistent 3D geometry.Stage 1 uses a curated 10.2M-image corpus, while Stage 2 corrects spatial scale drift and directional bias.
- Conclusion: Targeted Blender-based augmentation adds 23,700 images across under-covered focal regimes to improve robustness under intrinsic shift.The dataset addresses focal-length distribution mismatch between training and test data.
- Conclusion: The Stage-1 model provides the best overall relative-depth performance and competitive boundary sharpness.The results support the value of stronger affine-invariant representations alongside adaptive metric calibration.
Supplementary Material
The supplementary material documents FoundationGeo’s training objectives and implementation details, with Stage-I combining global alignment, multi-scale local supervision, geometric consistency, and reliability masking. It also explains the supplementary dataset, evaluation protocol, and limitations as part of the broader material.
- Supplementary Material: The supplementary material covers training objectives, ray-direction correction, dataset construction, evaluation details, and limitations for FoundationGeo.It complements the main paper with algorithmic explanations and implementation settings.
- A.1 Stage-I: Relative Geometry Objective: Stage-I trains affine-invariant point maps and reliability masks using global, local, normal, edge, and mask supervision.The objective combines coarse global alignment with multi-scale local patch losses and geometric consistency terms.
- A.1 Stage-I: Relative Geometry Objective: Global alignment fits predicted points to ground-truth camera-space points under scale–shift ambiguity, with inverse-depth weighting balancing supervision across depth ranges.The alignment parameters transform the affine-invariant prediction into ground-truth camera space.
- A.1 Stage-I: Relative Geometry Objective: Local patch supervision uses spherical neighborhoods whose sizes adapt to scene depth and camera intrinsics, supporting coarse-to-fine detail preservation.Different supervision sources use different local scales, including synthetic, SfM, and LiDAR data.
- A.1 Stage-I: Relative Geometry Objective: Normal and edge losses preserve piecewise-smooth geometry, local edge directions, sharp transitions, and high-frequency structure.The edge term penalizes angular discrepancies between predicted and ground-truth horizontal and vertical point-map differences.
- A.1 Stage-I: Relative Geometry Objective: The final Stage-I objective is label-type dependent, combining enabled global, local, normal, edge, and mask terms with source-specific weights and alignment resolutions.Synthetic, SfM, and LiDAR supervision activate different local and auxiliary losses.
A.2 Stage-II Training Details
Stage-II retains Stage-I structural supervision while adding metric regression and separate pixel-wise scale and ray-direction calibration losses. Its ray correction is scale-preserving, bounded, and geometrically interpretable through tangent-plane offsets.
- A.2 Stage-II Training Details: Stage-II combines inherited relative-geometry supervision with metric regression, scale-field supervision, ray-direction supervision, and correction-field regularization.The objective preserves structural learning while explicitly disentangling metric calibration fields.
- A.2 Stage-II Training Details: The additional calibration terms use γs = 0.2, γr = 0.1, and γ∆ = 0.05, while metric loss has unit weight.Normal and mask losses use weights 1.0 and 0.1 when enabled.
- A.2 Stage-II Training Details: Scale-field targets are trained in the log domain with scales clamped to [0.05, 20.0], while ray errors are robustly penalized and clamped to [0.05°, 30°].The scale penalty uses β = log(1.25), and the ray penalty uses β = 3°.
- B Ray-Direction Correction Details: Ray-direction correction decomposes each point into range and unit ray, constructs a stable tangent basis, applies bounded 2D offsets, and restores the original range.The update changes direction without absorbing metric scale.
- B Ray-Direction Correction Details: Dynamic reference-axis selection avoids degenerate cross products, while bounded tanh offsets prevent overly large angular corrections.The tangent-plane representation makes the correction numerically stable and geometrically interpretable.
C FoundationGeo Dataset
The FoundationGeo Dataset is a targeted Blender-rendered supplement designed to improve camera-model coverage, especially in under-represented focal regimes. It provides controlled scenes, trajectories, focal settings, and aligned geometric supervision for metric training.
- C FoundationGeo Dataset: The Blender-based FoundationGeo Dataset contains 23,700 rendered images designed to supplement under-covered focal-length regimes.It includes 22,900 main training images, a 400-image object-centric subset, and 400 images excluded from training.
- C FoundationGeo Dataset: The dataset spans seven scenes across five indoor and two outdoor environments, with aligned RGB, depth, focal metadata, and camera poses.The scenes cover room-scale, building-scale, and open-environment layouts.
- C FoundationGeo Dataset: Images are rendered at 1024 × 768 with varying focal lengths to increase camera diversity and target under-represented intrinsic regimes.This makes the dataset a metric-stage supplement rather than generic synthetic augmentation.
- C FoundationGeo Dataset: Manually designed camera trajectories select valid viewpoints and provide perspective variation, structural depth transitions, and clean geometric supervision.The design avoids degenerate views and invalid scene configurations.
D.1 Evaluation Protocol Details
The evaluation uses a unified protocol across benchmark datasets, with dataset-specific preprocessing and separate relative- and metric-depth coverage.
- D.1 Evaluation Protocol Details: The protocol follows MoGe and covers NYUv2, KITTI, ETH3D, iBims-1, Sintel, DDAD, DIODE, Spring, and HAMMER.
- D.1 Evaluation Protocol Details: The Blender-based dataset contains seven indoor and outdoor scenes with representative RGB images and depth maps spanning varied layouts, viewpoints, and geometric structures.
- D.1 Evaluation Protocol Details: Metric depth is evaluated on seven datasets with reliable metric annotations, while relative depth is reported on eight datasets.Spring is excluded because it is used for training, and Sintel is used only for relative-depth evaluation because it lacks metric scale.
- D.1 Evaluation Protocol Details: Preprocessing includes resolution normalization, dataset-specific cropping or downsampling, and cleaning to reduce sensor noise and ground-truth artifacts.
- D.1 Evaluation Protocol Details: Additional filtering removes unreliable boundaries and masks problematic regions such as distant NYUv2 depths, reflective surfaces, sky, and visible ego-vehicle areas.
D.2 Training Data Filtering
Training-data filtering removes samples with domain mismatch, weak geometric cues, or unreliable annotations while retaining images with clear perspective and depth structure.
- D.2 Training Data Filtering: The 10.2M-sample training set is filtered to remove severe domain mismatch, weak geometric cues, and unreliable annotations.
- D.2 Training Data Filtering: Bird’s-eye-view and remote-sensing-style samples are discarded because their imaging geometry differs substantially from the target distribution.
- D.2 Training Data Filtering: Samples with clear perspective and near–far depth ordering are retained as informative geometric training examples.
- D.2 Training Data Filtering: Mislabeled samples with scene-boundary violations or inconsistent depth annotations are removed from training.
E Limitations and Future Work
The paper identifies boundaries in camera coverage, calibration capacity, and synthetic-data diversity, and proposes broader data and model designs as future directions.
- E Limitations and Future Work: Focal-length coverage does not span other camera-model factors such as principal point, aspect ratio, sensor size, and distortion.The current intervention mainly targets focal-length coverage, leaving broader intrinsic variation for future work.
- E Limitations and Future Work: Lightweight spatial calibration fields have limited correction capacity under severe camera shifts, unusual geometry, or strong appearance degradation.
- E Limitations and Future Work: The synthetic dataset is limited relative to real-world open-domain complexity, using seven scenes with restricted layout, object, material, and motion diversity.
- E Limitations and Future Work: Future work includes richer calibration fields and larger, more automated synthetic-data engines with broader scenes, controllable camera parameters, and scalable trajectories.