PaperlayerView exact revision on arXiv

Source-linked AI summary

VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale

Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic, Laura Leal-Taixé, Qunjie Zhou, Aljosa Osep

arXiv:2602.23361v1cs.CV

TL;DR

Offline feed-forward 3D reconstruction is limited by quadratic computation and memory growth from variable-length KV scene representations. VGG-T^3 distills that representation into a fixed-size MLP through test-time training, achieving linear scaling while retaining global aggregation and supporting visual localization. It reconstructs large image collections substantially faster than softmax-attention baselines and outperforms other linear-time methods on point maps and video depth estimation.

  • Problem

    Offline feed-forward reconstruction has computational and memory requirements that grow quadratically with the number of input images because scene geometry uses a variable-length KV representation.

  • Method

    VGG-T^3 maps the KV space into a fixed-size MLP and optimizes that MLP at test time while retaining the pretrained encoder and decoder.

  • Results

    11.6× faster reconstruction of 1k images and up to 33× faster reconstruction of 2k images are reported, while outperforming linear-time methods on point maps and video depth estimation.

  • Takeaways & Limitations

    A fixed-size MLP scene representation provides a linear-time offline reconstruction formulation with global scene aggregation and unified visual localization.

  • Takeaways & Limitations

    The method remains less accurate than softmax attention, especially in wide-baseline settings, motivating work on reconciling MLP expressivity with quadratic attention accuracy.

Abstract

from arXiv · show

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T$^3$ (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a $1k$ image collection in just $54$ seconds, achieving a $11.6\times$ speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

1. Introduction

VGG-T^3 addresses the quadratic computational and memory scaling of offline feed-forward reconstruction by distilling variable-length KV scene representations into a fixed-size MLP through test-time training. This linear formulation supports large image collections, global scene aggregation, and unified reconstruction-localization capabilities.

  • Offline feed-forward methods match classical reconstruction accuracy but scale poorly in computation and memory as input images increase.
  • Global softmax attention queries a variable-length KV scene representation, causing quadratic scaling with the number of input images.Sparse attention and token merging compress representation length but retain quadratic asymptotic scaling.
  • VGG-T^3 maps the KV space into a fixed-size MLP and optimizes it at test time, making depth-map queries linear in collection size.The pretrained encoder and decoder are retained, while the MLP is trained with reconstruction loss in token space.
  • 48.5 seconds processes a 2k image collection, a 33× improvement over VGGT at 27 minutes.Mini-batching enables single-GPU processing with CPU off-loading and distributed inference by sharding image tokens across GPUs.
  • The optimized MLP stores a compressed scene representation that supports feed-forward localization of novel query views using the same model as reconstruction.Mapping optimizes the MLP, while localization queries the frozen MLP.
  • The paper proposes linear scaling, fixed-dimensional implicit state conversion, scalable inference, and joint feed-forward visual localization and mapping.These contributions include processing large image sets and enabling efficient distributed inference.

2. Related Work

Related work spans classical pipelines, feed-forward multi-view fusion, long-sequence and compressed-attention methods, online models, test-time training, and visual localization. VGG-T^3 differs by decoupling complexity from input-image count while retaining global offline reconstruction.

  • Classical structure-from-motion pipelines achieve accurate large-collection reconstructions when scenes provide sufficient visual overlap and connectivity.They combine feature extraction, correspondence search, pose estimation, and joint refinement of poses and 3D structure.
  • Feed-forward models use attention across image features to recover 3D geometry, camera motion, and scenes with uncalibrated cameras or low visual overlap.
  • Multi-view feed-forward methods aggregate features across views to predict poses and scene geometry through tokenizers, aggregators, and decoders.VGGT, Fast3R, and π3 perform global fusion in token space via softmax attention.
  • Offline multi-view models incur O(n^2) complexity, while chunking and sliding-window approaches decouple global scene state and can drift on unordered image sets.Token merging and block-sparse attention reduce constants but remain asymptotically quadratic.
  • VGG-T^3 decouples computational complexity from input-image count, moving reconstruction from quadratic to linear time.
  • Online methods retain past frames or tokens as working memory, whereas some causal VGGT variants still require quadratic scaling and memory-intensive KV caching.
  • Compared with concurrent TTT3R, VGG-T^3 compresses the global offline KV space into a fixed-size MLP and maintains linear complexity while reporting higher accuracy.
  • Linear-attention methods replace softmax kernels with linear feature maps to obtain linear-time, constant-memory recurrences.

3. Feed-Forward 3D Reconstruction at Scale

VGG-T3 replaces VGGT’s quadratic global attention with test-time training that compresses variable-length KV scene memory into a fixed-size MLP. This linearized representation supports scalable inference, sequence-length analysis, and querying reconstructed scenes with new images.

  • Preliminaries: VGGT’s variable-length KV scene representation requires O(n^2) computation and linearly growing memory as the number of input views increases.Global self-attention pools information across all views but creates the central scalability bottleneck.
  • Method: VGG-T3 replaces global softmax attention with test-time training that maps KV information into fixed-size MLP weights.The MLP acts as a compact scene representation while retaining the pretrained encoder and decoder.
  • Method: ShortConv2D mixes local spatial context into the Value space, changing the learned target from K → V to a context-aware K → V′ mapping.Values are reshaped to image grids, convolved, and flattened before optimizing the test-time objective.
  • Sequence-length generalization: 5× reconstruction error increase occurs when test-time training processes N = 1k instead of N = 100 images, motivating sequence-length generalization analysis.The paper hypothesizes that a fixed number of optimization steps is insufficient for compressing substantially larger scenes.
  • Large-scale reconstruction: O(n) operations replace O(n^2) global-attention complexity, enabling minibatch inference on arbitrarily large collections and distributed multi-GPU processing.Minibatch gradients can be computed independently and synchronized across GPUs.
  • Query-able reconstruction & visual localization: Frozen test-time-optimized MLPs can be queried with novel images to output scene geometry and camera pose for visual localization.Query processing uses the MLPs without updating their parameters, turning the model into a single-image Transformer for the new view.

4. Experiments

Experiments show that VGG-T3 preserves strong geometric reconstruction while scaling linearly, enabling substantially faster large-scale inference and effective visual localization.

  • Evaluation setup: VGG-T3 is evaluated on pointmap estimation, video depth, camera pose estimation, and visual localization against offline and online baselines.The evaluation covers standard benchmarks and large-scale reconstruction settings.
  • Standard benchmarks: VGG-T3 outperforms the O(n) baseline TTT3R on nearly all pointmap benchmarks, reducing error by 2–2.5× on DTU, ETH3D, and NRGBD-D.It remains competitive with O(n^2) baselines and surpasses them on DTU.
  • Standard benchmarks: VGG-T3 outperforms TTT3R on two of three video-depth datasets and performs on par with O(n^2) methods on KITTI.The comparison uses Absolute Relative Error and the percentage of predictions with δ < 1.25.
  • Standard benchmarks: VGG-T3 struggles on camera-pose estimation, potentially because the TTT MLP must represent heterogeneous image and camera-token modalities.The paper identifies this as a direction for future research.
  • Large-Scale 3D Reconstruction: 58 seconds: VGG-T3 reconstructs 1k images in this time, compared with over 11 minutes for VGGT, while maintaining a small Chamfer-distance gap that narrows with scale.This corresponds to an 11.6× speedup over VGGT.
  • Large-Scale 3D Reconstruction: VGG-T3 supports distributed inference through data parallelism, requiring cross-GPU communication only during fast-weight MLP updates.This enables additional speedup for large image collections.
  • Visual localization: Unseen query images can be used for feed-forward visual localization against the implicit map produced during test-time optimization.VGG-T3 outperforms TTT3R on both localization benchmarks, especially Wayspots.
  • Ablations: The ablations show that linearizing a softmax-pretrained model is essential, while test-time training substantially outperforms T2R and LoLCats.ShortConv2D further closes the gap toward softmax attention.

5. Conclusion

The paper presents VGG-T3 as a scalable feed-forward reconstruction model that replaces quadratic attention queries with test-time optimization of a fixed-size MLP. It achieves large speedups while retaining strong global reconstruction performance, though accuracy remains below softmax attention in wide-baseline settings.

  • Conclusion: VGG-T3 maps Keys to Values through test-time optimization instead of querying a variable-length KV representation with quadratic softmax attention.The resulting model scales linearly with the number of input views.
  • Conclusion: 11.6×: VGG-T3 accelerates reconstruction of 1k images, while 2k-image reconstruction is up to 33× faster than the cited baseline.The method also outperforms linear-time methods on pointmap and video-depth estimation.
  • Conclusion: VGG-T3 retains global scene aggregation and provides more accurate depth and point maps than other linear-time methods.This combines online-like scalability with offline global reconstruction.
  • Limitations: A gap remains relative to softmax attention, especially for wide-baseline inputs.The paper identifies reconciling fixed MLP expressivity with quadratic-attention accuracy as future work.

A. Implementation Details

Implementation uses co-visibility-based image sampling, adaptive training augmentations, selective parameter optimization, and inference procedures designed for large collections.

  • Training: Training collections are built by greedy co-visibility sampling, starting from a random image and repeatedly selecting images with co-visibility above 0.3.The co-visibility matrix is precomputed through depth consistency checks.
  • Training: Training uses adaptive 2–24-image batches, approximately 48 images per GPU, aspect-ratio sampling from [0.5, 2.0], resizing to a 518-pixel long side, and independent color jitter.These choices follow VGGT while varying collection composition and appearance.
  • Optimization: The model trains with AdamW and optimizes only global-attention projections plus TTT parameters while freezing the remaining VGGT components.The inner test-time objective uses Muon with one optimizer step during training.
  • Architecture: Only value projections from image patch tokens enter ShortConv2D, while camera and register tokens bypass it.This separates spatial mixing from the camera and register token pathways.
  • Inference: Pointmaps are obtained from the global pointmap head rather than camera-depth unprojection because predicted camera poses are imprecise.Benchmarking uses NVIDIA A100-80GB GPUs.
  • Evaluation: Large-scale evaluation uses subsampled mapping images for localization and equally spaced keyframes for efficient pointmap metrics.Keyframes avoid the high cost of ICP alignment on large image sets.

B. VGGT adjustments

The VGGT baseline is modified for large image collections through memory optimization, distributed inference, and entropy-scaled attention. These changes strengthen its performance comparison at long sequence lengths.

  • Memory and distributed inference: Memory optimization enables VGGT to process up to 1k images on a single 80GB GPU.Unused activations are discarded in the alternating attention module.
  • Memory and distributed inference: Context parallel inference allows VGGT to run on 2k images while retaining FlashAttention2 in global attention layers.The implementation uses Ulysses in TransformerEngine.
  • Entropy scaling: Entropy-scaling sharpens the attention matrix for sequence lengths larger than those seen during training.The scaling is unchanged for training-length sequences; VGGT training uses a maximum of 24 images, corresponding to NT = 32,856 tokens.
  • Entropy scaling: Table 8 reports improved VGGT performance on large image collections with entropy-scaling.The adjustment makes VGGT a significantly stronger baseline.

C. Additional Results

Additional experiments examine test-time optimization and ShortConv2D configurations. Two optimizer steps work best across varied collection sizes, while a 3×3 value filter is sufficient for a strong objective.

  • Number of optimizer steps: Two optimizer steps achieve the best performance across a wide range of image collection sizes.Three or four steps are comparable or slightly worse, while one step degrades beyond training-scale collections.
  • Number of optimizer steps: Without TTT optimization, reconstruction error is high because no global information propagates across tokens.A single step is sufficient for image collection sizes seen during training but not for larger collections.
  • ShortConv2D: A 3×3 ShortConv2D filter on values vi is sufficient to obtain a strong self-supervised objective for TTT.Increasing the filter size to 5×5 does not further increase performance.
  • ShortConv2D: Applying ShortConv2D jointly to keys ki and values vi decreases performance.The authors attribute this to preserving dependency between keys and values under identical spatial mixing.

D. Additional Qualitative Results

Qualitative results show that VGG-T3 reconstructs large image collections quickly, supports localization across substantial viewpoint and temporal changes, and can lose quality on complex scenes with larger spatial extent.

  • Qualitative comparison: VGG-T3 produces 3D-consistent reconstructions for 1k-image collections within one minute, while TTT3R degrades significantly.VGGT produces slightly sharper details but takes more than 11 minutes.
  • Visual localization: VGG-T3 successfully localizes a tourist phone image within a seven-year-older KITTI recording despite temporal and street changes.The query captures upper building regions absent from the car-mounted camera view.
  • Scenes with larger spatial extent: On Waymo scenes with larger spatial extent, VGG-T3 can match VGGT but degrades in some complex scene layouts.The authors identify this as a limitation of current linear-time attention mechanisms.
Loading 2602.23361v1…