PaperlayerView exact revision on arXiv

Source-linked AI summary

RoMa-$Ω$: What Feed-Forward 3D Models Know About Image Matching

David Nordström, Xinyue Zhang, Thibaut Loiseau, Vincent Lepetit, Fredrik Kahl

arXiv:2609.09507v1cs.CV

TL;DR

The paper asks what feed-forward 3D models know about image matching, given uncertainty about their matching capabilities and the dominance of DINO features in current matchers. It evaluates their representations and raw geometry, then replaces RoMa v2’s DINO backbone with VGGT-Ω features, producing strong benchmark performance. The resulting RoMa-Ω improves RoMa v2 by +8.1 mAA on WxBS.

  • Problem

    The paper investigates the unclear image-matching capabilities of feed-forward reconstruction models and whether DINO features are the best matching representation.

  • Method

    The study compares zero-shot patch features, raw 3D point predictions, and a full matcher built on feed-forward representations, then substitutes VGGT-Ω features for RoMa v2’s frozen DINO encoder.

  • Results

    +8.1 mAA on WxBS is achieved by RoMa-Ω over RoMa v2 after replacing its DINOv3 backbone with selected VGGT-Ω features.

  • Takeaways & Limitations

    Feed-forward reconstruction representations can support effective image matching even when raw later-layer feature similarity is degraded.

  • Takeaways & Limitations

    The largest experiments are limited to VGGT-Ω, and raw geometry matching degrades under viewpoint and domain shifts.

Abstract

from arXiv · show

Learned image matching has experienced significant progress in recent years, culminating in robust and accurate matchers such as RoMa, whose robustness is often attributed to its use of frozen DINO features. In a parallel development, feed-forward reconstruction models, such as VGGT, have been trained on ever-growing datasets to accurately regress dense 3D point maps and camera poses. The distinction between matchers and feed-forward reconstruction models has become increasingly blurred with the introduction of matching losses in models such as MASt3R and VGGT-$Ω$. This raises a natural question: what do feed-forward 3D models know about image matching? In this work, we answer this question by analyzing three scenarios: (i) zero-shot matching of patch features, (ii) direct matching of 3D point predictions, and (iii) training a full matcher on top of the learned representations. We find that, despite performing poorly in zero-shot matching, especially in later layers, feed-forward reconstruction models provide strong representations for linear probing and full matching pipelines. We further show that, even without any training, their raw predictions alone enable competitive matching, albeit only under moderate viewpoint changes and modality gaps. Based on these insights, we retrain RoMa v2 by replacing its DINO backbone with VGGT-$Ω$. Our resulting model, \ours, outperforms state-of-the-art matchers on a wide range of benchmarks, e.g. +8.1 mAA compared to RoMa v2 on WxBS.

1 Introduction

The paper asks what feed-forward 3D models know about image matching and tests their features, raw predictions, and integration into a full matcher. Replacing RoMa v2’s DINOv3 backbone with selected VGGT-Ω features yields strong matching performance, including gains on difficult benchmarks.

  • 1 Introduction: Image matching supports downstream 3D-vision tasks including Structure-from-Motion, visual localization, and 3D reconstruction.The introduction frames learned dense matchers as important alternatives to sparse keypoint-based pipelines.
  • 1 Introduction: Feed-forward reconstruction models are studied because their implicit image-matching capabilities remain unclear, while existing matchers typically rely on DINO features.The paper positions this comparison as a way to assess whether reconstruction features are better options for image matching.
  • 1 Introduction: The study evaluates feed-forward models through zero-shot patch matching, direct matching of 3D point predictions, and incorporation into a full matching pipeline.These three focus areas compare learned representations with DINO and assess both training-free and trained matching.
  • 1 Introduction: Feed-forward reconstruction models remain strong with linear probes and in full matching pipelines despite degraded raw feature similarity in later layers.The paper relates this degradation to reconstruction-oriented training and the absence of constraints preserving pairwise patch similarity.
  • 1 Introduction: +8.1 mAA on WxBS and +3.5 mAA on HardMatch result from replacing RoMa v2’s DINOv3 backbone with selected VGGT-Ω features.The change produces RoMa-Ω and improves performance on challenging matching datasets.

2 Related Work

Related work presents image matching as a foundational component of 3D vision and contrasts classical multi-stage pipelines with learned and feed-forward alternatives.

  • 2 Related Work: Image matching supplies geometric constraints for tasks ranging from camera-pose estimation to large-scale reconstruction.This establishes matching as a central interface between visual correspondence and 3D reconstruction.
  • 2 Related Work: Classical image matching combines keypoint detection, local feature description, and correspondence assignment through nearest-neighbor search.Learned methods have replaced individual components with learned detectors, descriptors, joint architectures, and matchers.
  • 2 Related Work: Feed-forward reconstruction models seek to infer scene geometry and camera parameters directly, instead of relying on traditional SfM optimization pipelines.Traditional systems alternate correspondence estimation, geometric verification, triangulation, and bundle adjustment.

3 Feed-forward Reconstruction Models for Matching

Feed-forward reconstruction models encode correspondence information, but their raw features often degrade as matching descriptors in later layers. Linear probes and transformer-based matching decoders can recover this information, with VGGT-Ω providing a stronger frozen backbone than DINOv3 in the evaluated pipeline.

  • Feature Extraction from Reconstruction Models: The analysis evaluates frozen intermediate representations directly as dense descriptors, while reconstruction heads are discarded except when raw geometry predictions are studied.The representations arise from an encoder and transformer with alternating frame-wise and global attention layers.
  • Qualitative Feature Analysis: Later reconstruction layers retain correspondence information but produce diffuse, less matching-friendly feature correlations compared with DINOv3.The raw feature similarity can retain the correct correspondence as its maximum even when the correlation map is substantially less localized.
  • Lightweight Matching Experiments: The study measures matching through zero-shot nearest-neighbor retrieval, lightweight linear probing, and a trainable decoder with a DPT head.The dense pipeline keeps the backbone frozen and optimizes only the decoder and prediction head.
  • Lightweight Matching Experiments: Linear probing reverses the zero-shot trend: later reconstruction layers improve dramatically and consistently surpass DINOv3 on MegaDepth-1500 PCK@32px.VGGT-Ω achieves the strongest linear-probe performance among the evaluated reconstruction models and is used thereafter.
  • Integration into a Dense Matching Pipeline: VGGT-Ω consistently outperforms DINOv3 across decoder depths when both serve as frozen backbones in a RoMa v2-style dense matching pipeline.The performance gap narrows as decoder capacity increases, suggesting larger decoders can partly compensate for weaker backbone representations.

4 RoMa-Ω

RoMa-Ω retains the RoMa v2 matching pipeline while replacing its frozen DINO backbone with frozen VGGT-Ω representations. It extracts multi-scale VGGT features, refines the deepest levels through cross-view reasoning, and predicts coarse warps before convolutional refinement.

  • Architecture: RoMa-Ω isolates the backbone effect by replacing frozen DINO features with frozen VGGT-Ω features while keeping RoMa v2’s decoder, heads, losses, and training procedure unchanged.This provides a direct comparison between feed-forward reconstruction features and DINO features for dense matching.
  • Feature Extraction: The model extracts patch features from VGGT-Ω layers 5, 12, 18, and 24, concatenating frame-wise and cross-view token streams into multi-scale representations.
  • Cross-view Refinement: A transformer decoder jointly reasons across both images to refine the deepest features, while the remaining feature levels are left unrefined.
  • Matching Head: The matching head projects refined deepest features into a matching space, correlates them, and combines the resulting embedding with levels 18 and 24 to predict coarse warp and confidence.
  • Architecture: RoMa-Ω follows RoMa v2’s coarse-to-fine design: a DPT head predicts a coarse warp, followed by convolutional refiners that increase prediction resolution.
  • Training: The coarse matcher is trained across multiple image resolutions and aspect ratios for 500K steps with batch size 128, while refiners use 640×640 inputs.

5 Experiments

Raw VGGT-Ω predictions provide competitive matching without training but are limited by severe viewpoint and modality shifts. A trained RoMa-Ω pipeline overcomes these limitations and improves performance across challenging matching, localization, and dense-matching benchmarks, at increased computational cost.

  • Matching from Raw Predictions: Without matching-specific training, VGGT-Ω raw predictions are competitive on ScanNet-1500, with depth warping strongest and slightly exceeding RoMa v2.Depth warping essentially matches pose read directly from predicted cameras, while 3D mutual nearest neighbors lag behind.
  • Matching from Raw Predictions: Raw VGGT-Ω predictions reach 84.3 AUC@20° on RUBIK but fall to 29.0 and 20.2 mAA on WxBS and HardMatch under extreme shifts.The paper attributes this boundary to training mostly on temporally close, easy-to-match video frames.
  • Benchmark results: +8.1 mAA on WxBS and +3.5 mAA on HardMatch are achieved by replacing RoMa v2’s DINOv3 backbone with selected VGGT-Ω features.The final model combines RoMa’s robust generalization with RoMa v2’s sub-pixel accuracy.
  • Benchmark results: +2.4 AUC@20° on RUBIK and +2.3 AUC VCRE (<45px) on Map-free are reported over RoMa v2.The Map-free benchmark evaluates metric camera localization from a single reference image without a map.
  • Dense Matching: RoMa-Ω outperforms RoMa v2 on 5/6 dense-matching datasets, with FlyingThings3D as the only exception because it contains dynamic scenes.The dense-matching comparison covers a wide range of datasets.
  • Runtime Comparisons: Replacing the approximately 300M-parameter DINO backbone with an approximately 1B-parameter VGGT-Ω backbone makes RoMa-Ω around 40% slower and substantially increases VRAM usage.The runtime benchmark uses 560 × 560 images, batch size 8, and an A100.

6 Limitations

The study’s broad claims are constrained by its experimental scope and computational cost. Its largest experiments focus on VGGT-Ω, while the inverse question and practical deployment remain limited or outside scope.

  • The largest experiments are limited to VGGT-Ω, despite the investigation targeting feed-forward 3D models more broadly.
  • The relationship between image matchers and feed-forward reconstruction remains an open direction because the inverse question lies outside this paper’s scope.
  • RoMa-Ω has higher accuracy than previous methods but its practical usefulness is limited by computational cost.

7 Conclusion

The paper compares feed-forward models’ representations and geometry with DINOv3 for matching, then uses VGGT-Ω to replace RoMa v2’s backbone. RoMa-Ω achieves competitive raw-geometry matching and outperforms RoMa and RoMa v2 across benchmarks, despite computational expense and shift sensitivity.

  • RoMa-Ω outperforms RoMa and RoMa v2 across a wide range of benchmarks despite being computationally expensive.
  • Raw VGGT-Ω geometry predictions are competitive with state-of-the-art matchers, but performance significantly degrades under viewpoint and domain shifts.
  • Raw feature correlations degrade in later layers for all considered feed-forward reconstruction models, paralleling feature degradation discussed for DINOv3.

A Training Instabilities

Retraining RoMa v2 with a VGGT-Ω backbone initially suffered from training collapse. Gradient- and update-stabilization precautions were introduced to make training stable.

  • RoMa v2 retraining with a VGGT-Ω backbone became unstable after about 150K steps, when gradient norms grew sharply and loss began increasing.
  • Training was stabilized by tracking gradient EMA and rejecting updates more than 3 times larger than that EMA.

B Training Dynamics

A fair retraining comparison shows clear evaluation gains from using the VGGT-Ω backbone instead of RoMa v2’s DINOv3 backbone.

  • Using the VGGT-Ω backbone instead of DINOv3 produced clear evaluation gains during RoMa v2 training.

C Training Data

RoMa-Ω is trained on a diverse 3D-dataset mix comparable in size to RoMa v2 and LoMa, with training dynamics comparing DINOv3 and VGGT-Ω backbones.

  • The dataset mix is slightly larger than RoMa v2's, while the authors report fair comparisons in prior results and ablations.
  • Training dynamics compare RoMa v2 retrained with DINOv3 against RoMa-Ω using the VGGT-Ω backbone.
  • The training data combine diverse 3D datasets at a scale comparable to RoMa v2 and LoMa.
Loading 2609.09507v1…