Source-linked AI summary
Visual Geometry Grounded Deep Structure From Motion
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, David Novotny
TL;DR
SfM reconstructs camera parameters and 3D scene structure from unconstrained 2D images, but traditional systems use incremental, non-differentiable pipelines. VGGSfM makes the pipeline fully differentiable and end-to-end trainable, achieving better or comparable performance while remaining unable to process thousands of images like traditional frameworks.
Problem
Traditional SfM incrementally estimates cameras and 3D structure, while learned components have mainly enhanced isolated steps within a non-differentiable pipeline.
Method
VGGSfM uses pixel-accurate deep tracks, jointly initializes all cameras and points, and fuses reconstruction components into a fully differentiable end-to-end pipeline.
Results
VGGSfM achieves better or comparable performance than prior frameworks and state-of-the-art performance across benchmark datasets.
Takeaways & Limitations
End-to-end training enables learned adaptation between SfM components, making the framework simpler while improving performance across benchmark datasets.
Takeaways & Limitations
VGGSfM cannot yet process thousands of images as established traditional SfM pipelines can.
Abstract
from arXiv · showhide
Structure-from-motion (SfM) is a long-standing problem in the computer vision community, which aims to reconstruct the camera poses and 3D structure of a scene from a set of unconstrained 2D images. Classical frameworks solve this problem in an incremental manner by detecting and matching keypoints, registering images, triangulating 3D points, and conducting bundle adjustment. Recent research efforts have predominantly revolved around harnessing the power of deep learning techniques to enhance specific elements (e.g., keypoint matching), but are still based on the original, non-differentiable pipeline. Instead, we propose a new deep pipeline VGGSfM, where each component is fully differentiable and thus can be trained in an end-to-end manner. To this end, we introduce new mechanisms and simplifications. First, we build on recent advances in deep 2D point tracking to extract reliable pixel-accurate tracks, which eliminates the need for chaining pairwise matches. Furthermore, we recover all cameras simultaneously based on the image and track features instead of gradually registering cameras. Finally, we optimise the cameras and triangulate 3D points via a differentiable bundle adjustment layer. We attain state-of-the-art performance on three popular datasets, CO3D, IMC Phototourism, and ETH3D.
1. Introduction
VGGSfM replaces the traditional incremental SfM pipeline with a fully differentiable, end-to-end framework that jointly estimates cameras and 3D structure. It uses learned point tracking, collective camera and point initialization, and differentiable bundle adjustment to achieve strong benchmark performance.
- VGGSfM is a fully differentiable SfM pipeline trained end-to-end rather than preserving the original non-differentiable framework.
- Reliable pixel-accurate 2D tracks simplify correspondence estimation by eliminating pairwise-match chaining.
- All cameras are estimated jointly from image and track features, avoiding incremental camera registration and its discrete correspondence-chaining step.
- A differentiable bundle adjustment layer jointly refines cameras and triangulated 3D points within the reconstruction function.
- VGGSfM achieves strong performance on CO3D, IMC Phototourism, and ETH3D, with in-the-wild reconstruction used to assess generalization.
2. Related Work
Classical SfM typically builds reconstructions incrementally from pairwise keypoint matches, while deep methods mainly improve individual pipeline components. Recent differentiable approaches regress poses or depths but face limitations in generalizability, scalability, temporal assumptions, or category scope.
- Classical SfM estimates scene structure and camera motion through global or incremental pipelines, with incremental methods widely adopted.
- Traditional incremental SfM detects keypoints, matches overlapping image pairs, verifies geometry, registers images, triangulates points, and applies bundle adjustment iteratively.
- Deep-learning methods mainly enhance keypoint detection, matching, semi-dense correspondence, or trainable RANSAC while retaining the broader SfM framework.
- Existing differentiable SfM methods often regress camera poses and depths but may have limited generalizability or scalability, temporal assumptions, or category-specific designs.
- VGGSfM adapts deep point tracking to unordered frames, processes all frames jointly, and refines tracks coarsely to sub-pixel accuracy.
3. Method
VGGSfM replaces incremental, partly hand-engineered SfM with a fully differentiable pipeline that tracks points directly, estimates cameras and points jointly, and refines them together. Its four stages are tracking, initial camera estimation, triangulation, and bundle adjustment.
- Method overview: VGGSfM decomposes reconstruction into point tracking, initial camera estimation, triangulation, and bundle adjustment.The tracker produces 2D tracks; subsequent modules estimate initial cameras and points, while bundle adjustment jointly refines cameras and 3D structure.
- Tracking: A deep feed-forward tracker directly outputs reliable trajectories across free-form images, avoiding pairwise-match chaining and its potential drift.Unlike video trackers, it does not assume temporal ordering or smoothness and processes all input frames jointly.
- Tracking: The tracker uses CNN descriptors, multi-resolution cost-volume tokens, and a Transformer to predict tracked locations and visibility indicators.Each query descriptor is correlated with feature maps from all input frames before Transformer processing.
- Tracking: Coarse-to-fine tracking first finds approximate correspondences and then crops P × P patches to refine locations to sub-pixel accuracy.The refinement targets the pixel-level precision required for accurate 3D reconstruction.
- Learnable camera & point initialization: VGGSfM estimates all cameras collectively with a Transformer and generates an initial point cloud from the cameras and 2D tracks.The camera predictor uses global image features and track descriptors; the triangulator then predicts initial 3D points, with preliminary DLT triangulation providing a point cloud.
- Learnable camera & point initialization: The method replaces incremental registration with non-incremental joint processing, simplifying differentiation while retaining bundle adjustment to refine cameras and points together.Traditional SfM gradually registers frames and enlarges the point cloud, whereas VGGSfM initializes cameras and points collectively.
4. Experiments
VGGSfM is evaluated on camera pose estimation, 3D triangulation, tracking, and component ablations across CO3D, IMC Phototourism, and ETH3D. The experiments show strong benchmark performance and benefits from end-to-end training, accurate tracking, and the proposed initialization modules.
- Datasets and protocols: VGGSfM is evaluated for camera pose estimation on CO3Dv2 and IMC Phototourism, and for 3D triangulation on ETH3D.The experiments also include ablations and qualitative tracking evaluation.
- Camera pose estimation: +9 accuracy points for each metric: VGGSfM outperforms existing methods on CO3D.On IMC, it outperforms all other methods on AUC@10 and AUC@5, and ranks second on AUC@3.
- 3D triangulation: 33.96% completeness at the 5cm threshold versus 29.54% for the best prior work on ETH3D.VGGSfM achieves better accuracy and completeness than all listed baselines across thresholds.
- Ablation study: 74.0% AUC@30 versus 70.7% without joint training on Co3D, while IMC AUC@10 rises from 68.35% to 73.92%.The comparison supports the importance of end-to-end joint training and component synergy.
- Ablation study: Pairwise-matching tracks contain many holes because pairwise matching cannot guarantee proper point tracking.The experiment compares predicted tracks and pairwise matches when supplied to PixSfM and VGGSfM.
- Ablation study: 62.30% AUC@10 after removing the fine tracker, compared with 73.92% for the complete model on IMC.The ablation evaluates the coarse-to-fine tracking mechanism.
5. Conclusion
VGGSfM is a fully differentiable SfM approach that simplifies traditional pipelines while achieving better benchmark performance, though it cannot yet handle thousands of images.
- VGGSfM is fully differentiable and benefits from learned adaptation between SfM components.
- The framework is simpler than traditional SfM while achieving better performance across benchmark datasets.
- VGGSfM is implemented entirely in Python to facilitate future modification and improvement.
- VGGSfM cannot yet process thousands of images as traditional SfM frameworks can.
A. Implementation Details
The implementation trains tracking, camera initialization, triangulation, and the full reconstruction framework in stages, using learned modules and differentiable geometric processing. The system combines image and track features to estimate cameras and 3D points, with filtering and bundle-adjustment components supporting reconstruction.
- Training: Training proceeds from tracker pretraining on Kubric to separate module training on Co3D or MegaDepth, followed by joint framework training.
- Data augmentation: Images receive color, blur, and grayscale augmentations, are resized to 512 × 512 with zero padding, and tracks invisible in over 50% of frames are excluded.
- Inference time: For 25 frames and 4096 query points on an NVIDIA A100, tracker, camera initializer, and triangulator inference take about 4.3, 0.9, and 0.2 seconds.
- Camera initializer: The camera initializer predicts an NI × 8 matrix containing quaternion, translation, and focal-length parameters from image, track, and preliminary-camera features.
- Camera initializer: The camera initializer uses harmonic camera embeddings, projected image descriptors, and a trunk transformer to iteratively update camera representations.
- Outlier filtering: A batched 8-point algorithm evaluates 20 sets of 50 point pairs in parallel to approximate RANSAC-like noisy-match filtering.
- Triangulator: The triangulator forms a preliminary point cloud with multiview DLT, computes camera-ray distances and nearest ray points, then predicts the refined cloud with a transformer.
B. Discussions and Ablation
VGGSfM combines direct track prediction, learned camera estimation, repeated reconstruction, and bundle adjustment, whose benefits depend strongly on correspondence filtering. Tracking and bundle-adjustment ablations expose both performance gains and sensitivity to noisy inputs.
- Global SfM comparison: VGGSfM directly predicts tracks, uses a learnable network for camera parameters, and applies reconstruction with bundle adjustment at each testing iteration.
- Bundle adjustment: Removing bundle adjustment reduces AUC@10 from 73.92 to 18.34.
- Bundle adjustment: Using bundle adjustment without track filtering reduces AUC@10 nearly to zero and produces incorrect camera optimization.
- Bundle adjustment: Without bundle adjustment, rotation errors for over 70% of image pairs remain within 5 degrees, with RRE@5° > 70%.
- Tracking ablation: The tracking ablation reports a clear performance drop when the PiPs video-tracking method is used in the framework.
- Tracking ablation: The tracking comparison is presented as evidence for the effectiveness of the proposed tracking solution.