PaperlayerView exact revision on arXiv

Source-linked AI summary

Light3R-SfM: Towards Feed-forward Structure-from-Motion

Sven Elflein, Qunjie Zhou, Sérgio Agostinho, Laura Leal-Taixé

arXiv:2501.14914v1cs.CVcs.LG

TL;DR

SfM systems that use optimization-based global alignment can be slow and memory-intensive on moderately sized image collections. Light3R-SfM uses feed-forward latent attention and a sparse scene graph to reconstruct unordered image sets; it reports a >49× speedup over MASt3R-SfM for 200 images while retaining competitive accuracy.

  • Problem

    Optimization-based global alignment supports accurate 3D reconstruction but brings slow runtime and extensive memory use even for moderately sized image collections.

    Source: §1 · p. 1

  • Method

    Light3R-SfM uses latent attention for feed-forward global alignment and a sparse scene graph to scale pairwise 3D predictions to image collections.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    33 seconds for 200 images versus approximately 27 minutes for MASt3R-SfM (>49× speedup), with performance rivaling state-of-the-art optimization-based SfM techniques.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach provides a data-driven alternative to optimization-dominated SfM and opens research opportunities for feed-forward 3D reconstruction.

    Source: §5 · p. 8

    The current model does not scale to collections of tens of thousands of images, and its pose accuracy at tight thresholds trails state-of-the-art optimization-based methods.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present Light3R-SfM, a feed-forward, end-to-end learnable framework for efficient large-scale Structure-from-Motion (SfM) from unconstrained image collections. Unlike existing SfM solutions that rely on costly matching and global optimization to achieve accurate 3D reconstructions, Light3R-SfM addresses this limitation through a novel latent global alignment module. This module replaces traditional global optimization with a learnable attention mechanism, effectively capturing multi-view constraints across images for robust and precise camera pose estimation. Light3R-SfM constructs a sparse scene graph via retrieval-score-guided shortest path tree to dramatically reduce memory usage and computational overhead compared to the naive approach. Extensive experiments demonstrate that Light3R-SfM achieves competitive accuracy while significantly reducing runtime, making it ideal for 3D reconstruction tasks in real-world applications with a runtime constraint. This work pioneers a data-driven, feed-forward SfM approach, paving the way toward scalable, accurate, and efficient 3D reconstruction in the wild.

1. Introduction

Light3R-SfM replaces optimization-based global alignment with feed-forward latent attention to estimate globally aligned camera poses from unordered images. It targets SfM's runtime and memory costs while aiming to retain competitive accuracy.

1. Introduction

  • SfM jointly recovers camera poses and 3D scene structure from unconstrained images, supporting applications including NeRFs [3] [29], 3DGS [20], MVS [31] [49], and visual localization [34].Traditional pipelines use feature detection and matching, triangulation, and bundle adjustment.Source: §1 · p. 1
  • 33 seconds for 200 images versus approximately 27 minutes for MASt3R-SfM (>49× speedup), with comparable accuracy across collection sizes on Tanks&Temples.The figure compares processing speed and accuracy for various SfM methods.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing DUSt3R-based SfM systems use optimization-based global alignment for accuracy, but incur slow runtime and extensive memory use even on moderately sized collections.MASt3R-SfM reduces computation through retrieval, sparse correspondences, and global bundle adjustment, but retains optimization-based alignment.Source: §1 · p. 1
  • Light3R-SfM predicts globally aligned camera poses from unordered images using latent-space attention to share global information before pairwise 3D reconstruction.Its contributions include replacing classical global optimization with a learnable latent alignment module and using intelligent graph construction to reduce redundant processing.Source: §1 · pp. 1–2

2. Related Work

Related work spans classical incremental and global SfM, learned components and optimization-based deep pipelines, and pointmap-based systems that scale pairwise reconstruction through retrieval or memory. Light3R-SfM is presented as a feed-forward alternative for offline reconstruction from unordered image sets.

Classical SfM

  • Incremental SfM grows from a selected two-view initialization, while global SfM [8] [30] aligns all cameras and is usually faster but less accurate and robust.GLOMAP [30] narrows the accuracy gap by combining camera-position and 3D-structure estimation in a global positioning step.Source: §2 · p. 2
  • Unlike hybrid SfM, which accumulates incrementally reconstructed image subsets, this method estimates each subset's camera poses feed-forward without further global bundle adjustment.Hybrid SfM divides images into subsets and accumulates locally aligned subsets.Source: §2 · p. 2

Optimization-based deep SfM

  • Deep SfM methods learn pipeline components or optimize per-scene networks; ACEZero [7] and FlowMap [40] struggle with image pairs having low visual overlap.Examples include learned feature detection and matching, PixSfM [25] refinement and bundle adjustment, and differentiable VGGSfM.Source: §2 · p. 2

Pointmap-based SfM

  • DUSt3R [51] and MASt3R use pairwise pointmap predictions with optimization-based global alignment, while exhaustive pairwise reconstruction limits their application to small image sets.MASt3R-SfM [12] uses image retrieval to build a sparse scene graph and sparse correspondences to improve runtime and optimization efficiency.Source: §2 · p. 2

Feed-forward SfM

  • MonST3R [58] extends MASt3R for dynamic scenes using offline optical flow, while Spann3R [48] uses spatial memory to align pointmaps to the first frame.Light3R-SfM instead processes constraints across an unordered image collection with latent alignment and graph construction.Source: §2 · p. 3

3. Light3R-SfM

Light3R-SfM reconstructs globally aligned scene geometry and camera poses from unordered image collections through latent feature alignment, sparse scene-graph decoding, and optimization-free global accumulation. Its supervision combines pairwise and global pointmap losses.

3.1. Latent Global Alignment

  • Latent global alignment shares information across image tokens through pooled global tokens, reducing attention complexity to O(N^2+N×T) from naive O((N×T)^2).The authors identify reducing the constant factor for practical N ≈ T as key to scaling to larger image collections.Source: §3.1 · p. 3

3.2. Scene Graph Construction

  • The scene graph uses a shortest path tree with N−1 edges; unlike a minimum spanning tree, it minimizes path costs toward each node to favor a flatter tree.The authors note that deep minimum spanning trees can cause drift during pointmap accumulation.Source: §3.2 · pp. 3–4

3.3. Global Optimization-free Reconstruction

  • For each tree edge, the decoder predicts local pointmaps and confidences, which are traversed breadth-first and registered into a global reconstruction using confidence-weighted Procrustes alignment.The paper describes this alignment as simpler than joint optimization and closed-form, with overhead linear in image count.Source: §3.3 · pp. 4–5

3.4. Supervision

  • Global pointmaps and pairwise local pointmaps are jointly supervised, with λ = 0.1 and samples below 100 valid pixels excluded from the loss.The global loss is intended to enforce accurate, consistent global alignment and implicitly supervises pose accuracy.Source: §3.4 · p. 5

4. Experiments

Light3R-SfM is evaluated on scene-level, object-centric, and driving-scene pose estimation, alongside ablations of its components and inference choices. Results compare accuracy and runtime with optimization-based and feed-forward methods across varied image collections.

4.1. Scene-level Multi-view Pose Estimation

  • 145% higher RRA and 84% higher RTA across five view settings, at approximately half Spann3R’s runtime.For unordered inputs, Spann3R’s offline ordering increases runtime by 36× in the 50-view setting and encounters out-of-memory errors on full sequences.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Object-centric Multi-view Pose Estimation

  • On CO3Dv2 10-view pose estimation, Light3R-SfM is the best feed-forward method across metrics, on par with VGGSfM, and improves RTA@15 by 6.1% over MASt3R*.It is less accurate than DUSt3R and MASt3R-SfM, which use optimization-based global alignment; the reported comparison also finds superior feed-forward performance in pairwise pose estimation.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. Evaluation on Driving Scenes

  • On Waymo driving scenes, Light3R-SfM matches MASt3R-SfM’s accuracy at approximately 195× lower runtime and achieves approximately 4× better RTA@5 than Spann3R at over 6× lower runtime.Figure 3 contrasts the MASt3R-SfM reconstruction’s inaccurate 90° turn with Spann3R predictions that degrade after tens of frames.Source: Table 4 · p. 7, §4.3 · p. 7, Figure 3 · p. 7

4.4. Ablation Studies

  • Adding latent alignment raises RRA@5 by 6.95% and decreases ATE by 15.78% over the ablation baseline; adding global supervision then raises RRA@5 and RTA@5 by 3.14% and 9.03%.Switching the full model’s backbone initialization from MASt3R to DUSt3R drops RRA@5 by 6.87% and RTA@5 by 8.1%.Source: §4.4 · pp. 7–8, Table 5 · p. 8
  • Increasing the pointmap-confidence threshold improves pose accuracy while reducing registration rate, allowing a completeness–accuracy trade-off; the learned method also generalizes across view settings and datasets.Runtime analysis evaluates a 1,106-image Courthouse scene on a V100-32GB, with encoder and decoder batches chunked by 32 to trade runtime against GPU memory.Source: Table 6 · p. 8, §4.4 · p. 8, Table 7 · p. 8
  • The encoder-retrieval scene graph performs close to the ground-truth-overlap Oracle, while a minimal spanning tree lowers RRA@5 by 15.26% and RTA@5 by 25.61%.The Oracle uses ground-truth overlapping scores to construct an optimal spanning tree.Source: §4.4 · p. 8, Table 5 · p. 8
  • After global-latent alignment conditioning, the decoder predicts pointmaps for images recorded in opposite directions, suggesting a representation of the entire scene.The figure presents this as an ablation-study visualization.Source: Figure 4 · p. 8

5. Conclusion

Light3R-SfM performs SfM without traditional matching or global optimization, using latent alignment and a sparse scene graph. The model does not scale to all SfM settings, and its pose accuracy at tight thresholds trails optimization-based methods.

5. Conclusion

  • Light3R-SfM replaces traditional matching and global optimization with latent alignment of pairwise predictions and a sparse scene graph.The authors report that this reduces runtime while providing competitive accuracy.Source: §5 · p. 8
  • The model does not scale to all SfM settings, including collections of tens of thousands of images.Source: §5 · p. 9
  • Pose accuracy at tight thresholds trails state-of-the-art optimization-based methods, likely because learned methods process low-resolution images.Source: §5 · p. 9

Supplementary Material for Light3R-SfM

The supplementary document provides implementation details, additional evaluations, further ablations, and qualitative visualizations.

Supplementary Material for Light3R-SfM

A. Implementation Details

The implementation uses a DUSt3R-style encoder and pointmap decoder with a global alignment module, trained on sampled image graphs. At inference, it extracts poses from globally referenced pointmaps and averages symmetric predictions using confidence weights.

A. Implementation Details

  • The model uses DUSt3R’s ViT-L image encoder, two ViT-B-parameterized pointmap heads, and four global-alignment blocks with eight attention heads.The attention feature dimension is 1024.Source: A. Implementation Details · p. 12
  • Training uses graphs of eight images sampled from Waymo Open Dataset, CO3Dv2, MegaDepth, and TartanAir, with 512-pixel resizing and varied crops.The model trains for 100,000 iterations with batch size 8, initialized from MASt3R pretrained weights.Source: A. Implementation Details · p. 12
  • At inference, camera poses are extracted from global-frame pointmaps using robust focal-length estimation followed by RANSAC-PnP on points above a confidence threshold.The default threshold is 3, or the 90%-quantile if all confidences fall below it.Source: A. Implementation Details · p. 12
  • Inference symmetrizes each edge and combines its pointmaps with the reverse-edge predictions using a confidence-weighted average.The reverse-edge pointmaps are transformed into the relevant reference frame before averaging.Source: A. Implementation Details · p. 12

B. Additional Details on Tanks & Temples [21]

On 100-view Tanks&Temples scenes, Light3R-SfM’s pose-error gap to optimization-based methods shrinks at looser thresholds, while Spann3R remains worse. The resulting coarse poses may suffice for pose-refinement applications that benefit from speed-ups of up to 198×.

B. Additional Details on Tanks & Temples [21]

  • Runtime comparisons use open-source implementations and default parameters on a system with 10 CPU cores, 64GB memory, and one NVIDIA V100 GPU.VGGSfM runtime is not reported for most full sequences and some 200-image sequences because of out-of-memory errors after memory-budget adjustments.Source: B. Additional Details on Tanks &… · p. 12
  • The pose-accuracy gap at tight thresholds between feed-forward and optimization-based methods rapidly shrinks for Light3R-SfM at looser thresholds, while Spann3R is consistently worse.Figure 5 plots relative rotation and translation pose-error CDFs on 100-view Tanks&Temples scenes.Source: B. Additional Details on Tanks &… · p. 13, Figure 5 · p. 13
  • Up to 198× speed-ups may benefit applications that refine poses, such as novel-view synthesis via Gaussian splatting, where coarse poses may suffice.The authors suggest that a fast optimization stage on the regressed outputs could improve tight-threshold performance, leaving this for future work.Source: B. Additional Details on Tanks &… · p. 13

C. Evaluation on 3D Reconstruction

On Waymo, Light3R-SfM and its variant without latent global alignment outperform Spann3R on reconstruction errors for most scenes, while latent alignment improves results on long forward-moving trajectories. The method is less reliable on dynamic scenes, where confident moving regions can impair pose estimates and global reconstruction.

C. Evaluation on 3D Reconstruction

  • The Waymo evaluation measures each scene’s predicted point cloud against sparse lidar ground truth using average nearest-neighbor Euclidean distance.Figure 7 presents the cumulative distribution of per-scene reconstruction errors.Source: C. Evaluation on 3D Reconstruction · p. 13
  • Both Light3R-SfM variants produce smaller Chamfer reconstruction errors than Spann3R for most Waymo scenes, and latent alignment performs significantly better on long forward-moving trajectories.The evaluation compares scene point clouds with sparse lidar ground truth using per-scene Chamfer distance.Source: C. Evaluation on 3D Reconstruction · p. 13
  • Unlike MASt3R-SfM, Light3R-SfM can lag on dynamic scenes because confident dynamic regions can produce wrong pairwise poses and degrade global reconstruction.Figure 6 visualizes Light3R-SfM’s global confidence map for an image containing dynamic objects.Source: C. Evaluation on 3D Reconstruction · p. 13, Figure 6 · p. 13

D. Additional Ablation Studies

The additional ablations examine latent alignment depth, global supervision weight, training graph size, and backbone initialization. They identify trade-offs in alignment accuracy, noisy global supervision, memory use, and the value of pretrained geometric backbones.

D. Additional Ablation Studies

  • Four latent alignment layers improve pose accuracy over two, while doubling to eight brings diminishing returns, motivating L = 4 as a memory/runtime trade-off.These ablations use 224×224 images and the 100-view Tanks & Temples subset unless stated otherwise.Source: Table 8 · p. 14, D. Additional Ablation Studies · p. 14
  • Increasing global supervision weight λ from 0.01 to 0.1 improves pose estimation, but λ = 1.0 decreases performance.The authors attribute the decline to noisier global supervision and favor pairwise supervision as the main optimization signal.Source: Table 9 · p. 14, D. Additional Ablation Studies · p. 14
  • MASt3R and DUSt3R initialization produce comparable results, whereas training the pairwise regressor from scratch performs poorly.The authors identify pretrained geometric foundation models as important components of their approach.Source: Table 10 · p. 14, D. Additional Ablation Studies · p. 15
  • Larger training graphs yield small but consistent performance improvements, but N = 8 is used for the high-resolution model because larger graphs exceed GPU memory capacity.The comparison evaluates N ∈ {3, 5, 8, 10} on Tanks & Temples scenes.Source: Table 11 · p. 14, D. Additional Ablation Studies · p. 14

E. Additional Visualizations

The additional visualizations show reconstructions across Tanks & Temples and ETH3D, qualitative comparisons on Waymo, and typical failure cases. They illustrate both the method’s scene detail and vulnerabilities to retrieval and pairwise-estimation errors.

E. Additional Visualizations

  • Waymo comparisons show Spann3R struggling on longer trajectories and MASt3R-SfM missing boundaries and distant background, while Light3R-SfM recovers poses and fine scene details.Figure 10 orders the comparisons MASt3R-SfM, Spann3R, then Light3R-SfM from left to right.Source: E. Additional Visualizations · p. 15, Figure 10 · p. 17
  • Figures 8 and 9 provide qualitative reconstructions of varied Tanks & Temples scenes and challenging ETH3D scenes.The Tanks & Temples examples include indoor, object-centric, and large-scale landmark scenes.Source: E. Additional Visualizations · p. 15, Figure 8 · p. 16, Figure 9 · p. 16
  • Retrieval failures can produce internally aligned but globally inconsistent sub-reconstructions, while small pairwise-estimation errors can misalign the global reconstruction.Figure 11 shows typical Tanks & Temples failure cases.Source: E. Additional Visualizations · p. 15, Figure 11 · p. 18
  • Table 12 reports per-scene reconstruction runtimes for Tanks & Temples in seconds.Source: Table 12 · p. 19
Loading 2501.14914v1…