PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Global Camera Poses from Noisy View-Graphs for Structure from Motion

Fadi Khatib, Meirav Galun, Ronen Basri

arXiv:2609.09491v1cs.CV

TL;DR

Accurate camera poses are foundational for 3D reconstruction and view synthesis, yet existing SfM approaches face trade-offs in scalability, coverage, runtime, or supervision. VGPA learns unsupervised view-graph pose averaging with a permutation-equivariant GNN, followed by triangulation and robust bundle adjustment. Across challenging benchmarks, it outperforms deep methods, remains competitive with classical pipelines, maintains high coverage, and runs faster while scaling to large image collections.

  • Problem

    Camera pose recovery is essential for 3D reconstruction and view synthesis, while existing approaches have limitations in scalability, coverage, runtime, or supervision.

  • Method

    VGPA uses a permutation-equivariant, edge-conditioned GNN to aggregate noisy pairwise relative poses without ground-truth supervision, then applies triangulation and robust bundle adjustment.

  • Results

    Across challenging benchmarks, VGPA outperforms deep methods, remains competitive with strong classical pipelines, maintains high camera coverage, and is substantially faster than COLMAP and GLOMAP.

  • Takeaways & Limitations

    VGPA provides an efficient global SfM pipeline that preserves accuracy and coverage on large image collections and sparse retrieval-based view graphs.

  • Takeaways & Limitations

    The method assumes a stationary scene and requires essential matrices and point tracks extracted during preprocessing.

Abstract

from arXiv · show

Camera pose estimation is a key step in 3D reconstruction and view-synthesis pipelines. We present a deep, global Structure-from-Motion framework based on learned view-graph aggregation. Our method employs a permutation-equivariant, edge-conditioned graph neural network that takes noisy pairwise relative poses as input and outputs globally consistent camera extrinsics. The network is trained without ground-truth supervision, relying solely on a relative-pose consistency objective. This is followed by 3D point triangulation and robust bundle adjustment. Our approach is efficient, scalable to more than a thousand images, and robust to graph density. We evaluate our method on MegaDepth, 1DSfM, Strecha, and BlendedMVS. These experiments demonstrate that our method achieves superior rotation and translation accuracy compared to deep track-centric methods while registering more images across many scenes, and competitive results compared to state-of-the-art classical pipelines, while being much faster.

1 Introduction

VGPA learns to aggregate noisy pairwise camera poses into globally consistent extrinsics, then refines structure with triangulation and robust bundle adjustment. It scales to large and sparse view graphs while achieving strong accuracy, coverage, and efficiency across challenging SfM benchmarks.

  • Motivation: Accurate camera poses are important because MVS, NeRF, and Gaussian Splatting rely on them during preprocessing.The motivation applies to 3D reconstruction and view-synthesis pipelines.
  • Results: VGPA registers almost all images and scales to collections containing more than 1,000 images.The large-scene reconstructions contrast with existing image-based deep methods.
  • Method: VGPA uses a permutation-equivariant, edge-conditioned GNN to aggregate noisy pairwise relative poses into globally consistent camera extrinsics.The network is trained without ground-truth supervision using a relative-pose consistency objective.
  • Method: The pipeline predicts global poses, triangulates point tracks, applies robust bundle adjustment, and can optionally reintegrate discarded cameras to increase coverage.In uncalibrated settings, bundle adjustment jointly optimizes intrinsics and extrinsics.
  • Results: VGPA preserves strong registration coverage and accuracy on top-30 retrieval graphs despite large reductions in edge density.This demonstrates robustness beyond exhaustive pairwise matching.
  • Results: VGPA achieves lower camera position and orientation errors than deep track-centric methods while registering more images, and remains competitive with strong classical pipelines.The evaluation covers MegaDepth, 1DSfM, Strecha, and BlendedMVS, including challenging cyclic trajectories.

2 Related work

Prior SfM approaches trade accuracy, scalability, flexibility, or supervision requirements in different ways. VGPA instead learns unsupervised pose averaging over view graphs and is reported to improve accuracy and camera coverage over prior deep factorization methods at lower runtime than strong global baselines.

  • Classical methods: Incremental SfM can be highly accurate but is inefficient for large image collections and depends on image-processing order.Projective factorization solves jointly but is restricted in handling uncalibrated settings, missing data, and outliers.
  • Limitations of prior work: A prior method achieved accurate pose recovery but sometimes over-pruned valid inliers, causing registration failures and reduced image coverage.This limitation motivates attention to coverage as well as pose accuracy.
  • Global methods: Global SfM handles all images simultaneously through manifold averaging of pairwise rotations and translations.Existing methods commonly average orientations first, followed by camera locations and scales.
  • Learned methods: Recent learned methods include rotation-only networks, point-track factorization models, and end-to-end systems that may require supervision or process only small image sets.VGGT jointly predicts poses, dense structure, and point tracks but requires substantial supervised training and has input-scale restrictions.
  • VGPA: VGPA introduces unsupervised permutation-equivariant view-graph pose averaging that surpasses prior deep factorization approaches in accuracy and camera coverage.It is designed as a learned alternative to sequential, factorization, and conventional global SfM methods.

3 Method

The method recovers globally consistent camera extrinsics by aggregating noisy pairwise relative poses on a view graph, then uses the predictions to reconstruct 3D structure. It combines edge-conditioned permutation-equivariant message passing, pose regression, and unsupervised relative-pose consistency training.

  • 3.1 Preprocessing: Noisy pairwise essential-matrix measurements provide relative rotations and translation directions for the pose-averaging input.Essential matrices are decomposed after feature matching and RANSAC, with translation recovered only up to scale.
  • 3.2 Network architecture: The pose-averaging network builds a view graph whose image nodes carry edge measurements and initializes node embeddings with a shared learnable token.Edge-conditioned message passing aggregates neighboring node and edge information with degree-normalized mean aggregation.
  • 3.2 Network architecture: Permutation-equivariant message passing produces per-camera embeddings, which a three-layer regression head maps to camera parameters.The predicted quaternion is normalized, and the resulting cameras are parameterized by rotations and translations.
  • 3.3 Output and loss: The model is trained without ground-truth poses by minimizing rotation and translation disagreement between output-camera relative poses and RANSAC estimates.The output cameras are converted back into pairwise relative transformations before evaluating the consistency objective.
  • 3.3 Output and loss: After a single-pass prediction on an unseen scene, the method fine-tunes on that scene using the same unsupervised objective.The recovered camera poses are then used to triangulate the 3D structure from the point tracks.

4 Experiments

VGPA is evaluated across diverse datasets and baselines, including large-scale and cross-dataset settings. It generally improves accuracy over deep methods, remains competitive with classical pipelines, and preserves performance on sparse view graphs.

  • 4.1 Datasets: The evaluation trains on MegaDepth and tests on novel MegaDepth scenes plus cross-dataset settings including 1DSfM, Strecha, and BlendedMVS.MegaDepth includes scenes with more than 1,000 images, while Strecha and BlendedMVS provide ground-truth camera poses.
  • 4.4 Results: VGPA achieves lower rotation and translation errors than RESfM across most MegaDepth and 1DSfM scenes, while registering more images than RESfM.It is competitive with Theia and GLOMAP and often surpasses them on both accuracy metrics, although GLOMAP typically registers more images.
  • 4.4 Results: VGPA's pose accuracy remains higher than that of efficient deep reconstruction methods on 1DSfM, while those methods are computationally efficient.Table 3 reports mean rotation and translation errors for TTT3R, CUT3R, and FAST3R.
  • 4.4 Results: On Strecha and BlendedMVS, VGPA is consistently more accurate than image-based deep baselines and performs on par with classical pipelines.The comparison includes VGGSfM, MASt3R, VGGT, Theia, COLMAP, and GLOMAP.
  • 4.4 Results: Using a MegaLoc top-30 retrieval graph, VGPA often preserves strong registration coverage and accuracy despite substantially lower edge density than exhaustive matching.Table 5 reports registered images, mean rotation error, and mean translation error for both graph-construction settings.
  • 4.4 Results: Runtime comparisons use identical point tracks and include VGPA's post-preprocessing stages, while ablations show benefits from deeper message passing and pretraining.The ablations report that pretraining combined with fine-tuning performs best across rotation, translation, and pose AUC.

5 Conclusion

VGPA is an unsupervised deep pose-averaging network for multiview SfM that combines graph-based pose consistency with triangulation and robust bundle adjustment. It achieves strong accuracy and coverage across challenging benchmarks while remaining fast and scalable.

  • 5 Conclusion: VGPA outperforms deep methods and remains competitive with strong classical pipelines while maintaining high camera coverage across challenging benchmarks.The conclusion cites MegaDepth and 1DSfM among the evaluated benchmarks.
  • 5 Conclusion: Permutation-equivariant message passing enforces consistency of pairwise rotations and translation directions over the view graph.The network is trained without ground-truth supervision.
  • 5 Conclusion: 3D point triangulation and robust bundle adjustment refine poses and recover the 3D structure.
  • 5 Conclusion: VGPA is substantially faster than COLMAP and GLOMAP, modestly faster than Theia, and scales to large image collections.An optional lightweight reintegration step recovers some discarded views with negligible overhead.

A Implementation details

The implementation trains and evaluates VGPA on NVIDIA A100 GPUs using PyTorch and Adam, with validation-based early stopping and complete view graphs for evaluation.

  • A Implementation details: Training and evaluation use NVIDIA A100 GPUs with 80 GB memory, PyTorch, Adam, and gradient normalization.
  • A Implementation details: Each epoch processes all training scenes, while a held-out validation set supports early stopping.
  • A Implementation details: Validation and test evaluations use the complete view graph, and MegaDepth training takes approximately 7 hours on one A100.

B More details on the ablation

The attention-based ablation replaces the base message-passing GNN with a GATv2 network using measurement nodes and alternating camera–measurement updates, while retaining the prediction pipeline.

  • B More details on the ablation: The ablation replaces the original message-passing GNN with a GATv2-based network while keeping the overall prediction pipeline unchanged.
  • B More details on the ablation: Each relative pose is encoded as an edge embedding, and its measurement becomes a node connected to the two endpoint camera nodes in a bipartite graph.
  • B More details on the ablation: Message passing alternates camera-to-measurement and measurement-to-camera attention updates using GATv2 convolutions.
  • B More details on the ablation: Residual connections, layer normalization, dropout, and a feed-forward MLP follow each attention block before camera embeddings enter the shared regression head.

C Constructing point tracks

The point-track construction follows the preprocessing procedure of prior work, while runtime comparisons begin only after point tracks have been built.

  • C Constructing point tracks: Point tracks are constructed following the preprocessing procedure of reference.
  • C Constructing point tracks: Reported runtimes exclude the point-track construction stage for all methods, including VGPA.
  • C Constructing point tracks: MASt3R runtimes also exclude pairwise matching, which is substantially more expensive.

D Additional results

The additional results evaluate pose accuracy and registration completeness using AUC and registration-aware AUC, alongside view reintegration. They also document a small reproducibility caveat for RESfM caused by nondeterministic bundle adjustment.

  • AUC evaluation: Standard AUC reports error-threshold recall over registered cameras for MegaDepth and 1DSfM.The reported tables include registered-camera counts and AUC values across thresholds.
  • Registration-aware evaluation: Registration-aware AUC penalizes unregistered cameras, measuring pose accuracy and registration completeness together.Pairs containing an unregistered image receive 180° error and contribute zero recall.
  • Evaluation caveat: RESfM results may differ slightly from previously published values because final bundle adjustment is not fully deterministic, but the differences are minor and do not affect the conclusions.The comparison uses a common evaluation protocol across methods.
  • View reintegration: View reintegration results compare the method with and without add-back postprocessing using registered-image counts and mean rotation and translation errors.Table 9 covers MegaDepth and reports input size, outlier fraction, registration count, and pose errors.
Loading 2609.09491v1…