PaperlayerView exact revision on arXiv

Source-linked AI summary

VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale

Sven Elflein, Ruilong Li, Sérgio Agostinho, Zan Gojcic, Laura Leal-Taixé, Qunjie Zhou, Aljosa Osep

arXiv:2602.23361v1cs.CV

TL;DR

Offline feed-forward 3D reconstruction is limited by quadratic computation and memory growth from variable-length KV scene representations. VGG-T^3 distills that representation into a fixed-size MLP through test-time training, achieving linear scaling while retaining global aggregation and supporting visual localization. It reconstructs large image collections substantially faster than softmax-attention baselines and outperforms other linear-time methods on point maps and video depth estimation.

  • Problem

    Offline feed-forward reconstruction has computational and memory requirements that grow quadratically with the number of input images because scene geometry uses a variable-length KV representation.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Method

    VGG-T^3 maps the KV space into a fixed-size MLP and optimizes that MLP at test time while retaining the pretrained encoder and decoder.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    11.6× faster reconstruction of 1k images and up to 33× faster reconstruction of 2k images are reported, while outperforming linear-time methods on point maps and video depth estimation.

    Source: §5 · p. 8

  • Takeaways & Limitations

    A fixed-size MLP scene representation provides a linear-time offline reconstruction formulation with global scene aggregation and unified visual localization.

    Source: §1 · p. 2, §5 · p. 8

    The method remains less accurate than softmax attention, especially in wide-baseline settings, motivating work on reconciling MLP expressivity with quadratic attention accuracy.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T$^3$ (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a $1k$ image collection in just $54$ seconds, achieving a $11.6\times$ speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.

1. Introduction

VGG-T^3 addresses the quadratic computational and memory scaling of offline feed-forward reconstruction by distilling variable-length KV scene representations into a fixed-size MLP through test-time training. This linear formulation supports large image collections, global scene aggregation, and unified reconstruction-localization capabilities.

  • Offline feed-forward methods match classical reconstruction accuracy but scale poorly in computation and memory as input images increase.Source: §1 · p. 1
  • Global softmax attention queries a variable-length KV scene representation, causing quadratic scaling with the number of input images.Sparse attention and token merging compress representation length but retain quadratic asymptotic scaling.Source: §1 · pp. 1–2
  • VGG-T^3 maps the KV space into a fixed-size MLP and optimizes it at test time, making depth-map queries linear in collection size.The pretrained encoder and decoder are retained, while the MLP is trained with reconstruction loss in token space.Source: §1 · p. 2
  • 48.5 seconds processes a 2k image collection, a 33× improvement over VGGT at 27 minutes.Mini-batching enables single-GPU processing with CPU off-loading and distributed inference by sharding image tokens across GPUs.Source: §1 · p. 2
  • The optimized MLP stores a compressed scene representation that supports feed-forward localization of novel query views using the same model as reconstruction.Mapping optimizes the MLP, while localization queries the frozen MLP.Source: §1 · p. 2
  • The paper proposes linear scaling, fixed-dimensional implicit state conversion, scalable inference, and joint feed-forward visual localization and mapping.These contributions include processing large image sets and enabling efficient distributed inference.Source: §1 · p. 2

2. Related Work

Related work spans classical pipelines, feed-forward multi-view fusion, long-sequence and compressed-attention methods, online models, test-time training, and visual localization. VGG-T^3 differs by decoupling complexity from input-image count while retaining global offline reconstruction.

  • Classical structure-from-motion pipelines achieve accurate large-collection reconstructions when scenes provide sufficient visual overlap and connectivity.They combine feature extraction, correspondence search, pose estimation, and joint refinement of poses and 3D structure.Source: §2 · p. 2
  • Feed-forward models use attention across image features to recover 3D geometry, camera motion, and scenes with uncalibrated cameras or low visual overlap.Source: §2 · p. 2
  • Multi-view feed-forward methods aggregate features across views to predict poses and scene geometry through tokenizers, aggregators, and decoders.VGGT, Fast3R, and π3 perform global fusion in token space via softmax attention.Source: §2 · p. 2
  • Offline multi-view models incur O(n^2) complexity, while chunking and sliding-window approaches decouple global scene state and can drift on unordered image sets.Token merging and block-sparse attention reduce constants but remain asymptotically quadratic.Source: §2 · p. 2
  • VGG-T^3 decouples computational complexity from input-image count, moving reconstruction from quadratic to linear time.Source: §2 · p. 3
  • Online methods retain past frames or tokens as working memory, whereas some causal VGGT variants still require quadratic scaling and memory-intensive KV caching.Source: §2 · p. 3
  • Compared with concurrent TTT3R, VGG-T^3 compresses the global offline KV space into a fixed-size MLP and maintains linear complexity while reporting higher accuracy.Source: §2 · p. 3
  • Linear-attention methods replace softmax kernels with linear feature maps to obtain linear-time, constant-memory recurrences.Source: §2 · p. 3

3. Feed-Forward 3D Reconstruction at Scale

VGG-T3 replaces VGGT’s quadratic global attention with test-time training that compresses variable-length KV scene memory into a fixed-size MLP. This linearized representation supports scalable inference, sequence-length analysis, and querying reconstructed scenes with new images.

Preliminaries

  • VGGT’s variable-length KV scene representation requires O(n^2) computation and linearly growing memory as the number of input views increases.Global self-attention pools information across all views but creates the central scalability bottleneck.Source: §3 · p. 3, §3.2 · p. 4, §3.1 · p. 4

Method

  • VGG-T3 replaces global softmax attention with test-time training that maps KV information into fixed-size MLP weights.The MLP acts as a compact scene representation while retaining the pretrained encoder and decoder.Source: §3.2 · p. 4
  • ShortConv2D mixes local spatial context into the Value space, changing the learned target from K → V to a context-aware K → V′ mapping.Values are reshaped to image grids, convolved, and flattened before optimizing the test-time objective.Source: §3.2 · p. 5

Sequence-length generalization

  • 5× reconstruction error increase occurs when test-time training processes N = 1k instead of N = 100 images, motivating sequence-length generalization analysis.The paper hypothesizes that a fixed number of optimization steps is insufficient for compressing substantially larger scenes.Source: §3.2 · p. 5, Figure 3 · p. 5

Large-scale reconstruction

  • O(n) operations replace O(n^2) global-attention complexity, enabling minibatch inference on arbitrarily large collections and distributed multi-GPU processing.Minibatch gradients can be computed independently and synchronized across GPUs.Source: §3.3 · p. 5

Query-able reconstruction & visual localization

  • Frozen test-time-optimized MLPs can be queried with novel images to output scene geometry and camera pose for visual localization.Query processing uses the MLPs without updating their parameters, turning the model into a single-image Transformer for the new view.Source: §3.3 · p. 6

4. Experiments

Experiments show that VGG-T3 preserves strong geometric reconstruction while scaling linearly, enabling substantially faster large-scale inference and effective visual localization.

Evaluation setup

  • VGG-T3 is evaluated on pointmap estimation, video depth, camera pose estimation, and visual localization against offline and online baselines.The evaluation covers standard benchmarks and large-scale reconstruction settings.Source: §4 · p. 6, §4.1 · p. 6, §4.3 · p. 8

Standard benchmarks

  • VGG-T3 outperforms the O(n) baseline TTT3R on nearly all pointmap benchmarks, reducing error by 2–2.5× on DTU, ETH3D, and NRGBD-D.It remains competitive with O(n^2) baselines and surpasses them on DTU.Source: §4.1 · p. 6
  • VGG-T3 outperforms TTT3R on two of three video-depth datasets and performs on par with O(n^2) methods on KITTI.The comparison uses Absolute Relative Error and the percentage of predictions with δ < 1.25.Source: §4.1 · p. 6
  • VGG-T3 struggles on camera-pose estimation, potentially because the TTT MLP must represent heterogeneous image and camera-token modalities.The paper identifies this as a direction for future research.Source: §4.1 · p. 7

Large-Scale 3D Reconstruction

  • 58 seconds: VGG-T3 reconstructs 1k images in this time, compared with over 11 minutes for VGGT, while maintaining a small Chamfer-distance gap that narrows with scale.This corresponds to an 11.6× speedup over VGGT.Source: §4.2 · p. 7, Figure 4 · p. 8
  • VGG-T3 supports distributed inference through data parallelism, requiring cross-GPU communication only during fast-weight MLP updates.This enables additional speedup for large image collections.Source: §4.2 · p. 7, Table 4 · p. 7

Visual localization

  • Unseen query images can be used for feed-forward visual localization against the implicit map produced during test-time optimization.VGG-T3 outperforms TTT3R on both localization benchmarks, especially Wayspots.Source: §4.3 · p. 8

Ablations

  • The ablations show that linearizing a softmax-pretrained model is essential, while test-time training substantially outperforms T2R and LoLCats.ShortConv2D further closes the gap toward softmax attention.Source: §4.4 · p. 8

5. Conclusion

The paper presents VGG-T3 as a scalable feed-forward reconstruction model that replaces quadratic attention queries with test-time optimization of a fixed-size MLP. It achieves large speedups while retaining strong global reconstruction performance, though accuracy remains below softmax attention in wide-baseline settings.

Conclusion

  • VGG-T3 maps Keys to Values through test-time optimization instead of querying a variable-length KV representation with quadratic softmax attention.The resulting model scales linearly with the number of input views.Source: §5 · p. 8
  • 11.6×: VGG-T3 accelerates reconstruction of 1k images, while 2k-image reconstruction is up to 33× faster than the cited baseline.The method also outperforms linear-time methods on pointmap and video-depth estimation.Source: §5 · p. 8
  • VGG-T3 retains global scene aggregation and provides more accurate depth and point maps than other linear-time methods.This combines online-like scalability with offline global reconstruction.Source: §5 · p. 8

Limitations

  • A gap remains relative to softmax attention, especially for wide-baseline inputs.The paper identifies reconciling fixed MLP expressivity with quadratic-attention accuracy as future work.Source: §5 · p. 8

A. Implementation Details

Implementation uses co-visibility-based image sampling, adaptive training augmentations, selective parameter optimization, and inference procedures designed for large collections.

Training

  • Training collections are built by greedy co-visibility sampling, starting from a random image and repeatedly selecting images with co-visibility above 0.3.The co-visibility matrix is precomputed through depth consistency checks.Source: A. Implementation Details · p. 10
  • Training uses adaptive 2–24-image batches, approximately 48 images per GPU, aspect-ratio sampling from [0.5, 2.0], resizing to a 518-pixel long side, and independent color jitter.These choices follow VGGT while varying collection composition and appearance.Source: A. Implementation Details · p. 10

Optimization

  • The model trains with AdamW and optimizes only global-attention projections plus TTT parameters while freezing the remaining VGGT components.The inner test-time objective uses Muon with one optimizer step during training.Source: A. Implementation Details · p. 10

Architecture

  • Only value projections from image patch tokens enter ShortConv2D, while camera and register tokens bypass it.This separates spatial mixing from the camera and register token pathways.Source: A. Implementation Details · p. 10

Inference

  • Pointmaps are obtained from the global pointmap head rather than camera-depth unprojection because predicted camera poses are imprecise.Benchmarking uses NVIDIA A100-80GB GPUs.Source: A. Implementation Details · pp. 10–11

Evaluation

  • Large-scale evaluation uses subsampled mapping images for localization and equally spaced keyframes for efficient pointmap metrics.Keyframes avoid the high cost of ICP alignment on large image sets.Source: A. Implementation Details · p. 11

B. VGGT adjustments

The VGGT baseline is modified for large image collections through memory optimization, distributed inference, and entropy-scaled attention. These changes strengthen its performance comparison at long sequence lengths.

Memory and distributed inference

  • Memory optimization enables VGGT to process up to 1k images on a single 80GB GPU.Unused activations are discarded in the alternating attention module.Source: B. VGGT adjustments · p. 11
  • Context parallel inference allows VGGT to run on 2k images while retaining FlashAttention2 in global attention layers.The implementation uses Ulysses in TransformerEngine.Source: B. VGGT adjustments · p. 11

Entropy scaling

  • Entropy-scaling sharpens the attention matrix for sequence lengths larger than those seen during training.The scaling is unchanged for training-length sequences; VGGT training uses a maximum of 24 images, corresponding to NT = 32,856 tokens.Source: B. VGGT adjustments · p. 11
  • Table 8 reports improved VGGT performance on large image collections with entropy-scaling.The adjustment makes VGGT a significantly stronger baseline.Source: B. VGGT adjustments · p. 11, Table 8 · p. 11

C. Additional Results

Additional experiments examine test-time optimization and ShortConv2D configurations. Two optimizer steps work best across varied collection sizes, while a 3×3 value filter is sufficient for a strong objective.

Number of optimizer steps

  • Two optimizer steps achieve the best performance across a wide range of image collection sizes.Three or four steps are comparable or slightly worse, while one step degrades beyond training-scale collections.Source: C. Additional Results · p. 11
  • Without TTT optimization, reconstruction error is high because no global information propagates across tokens.A single step is sufficient for image collection sizes seen during training but not for larger collections.Source: C. Additional Results · p. 11

ShortConv2D

  • A 3×3 ShortConv2D filter on values vi is sufficient to obtain a strong self-supervised objective for TTT.Increasing the filter size to 5×5 does not further increase performance.Source: C. Additional Results · pp. 11–12
  • Applying ShortConv2D jointly to keys ki and values vi decreases performance.The authors attribute this to preserving dependency between keys and values under identical spatial mixing.Source: C. Additional Results · p. 12, Table 9 · p. 12

D. Additional Qualitative Results

Qualitative results show that VGG-T3 reconstructs large image collections quickly, supports localization across substantial viewpoint and temporal changes, and can lose quality on complex scenes with larger spatial extent.

Qualitative comparison

  • VGG-T3 produces 3D-consistent reconstructions for 1k-image collections within one minute, while TTT3R degrades significantly.VGGT produces slightly sharper details but takes more than 11 minutes.Source: D. Additional Qualitative Results · p. 12

Visual localization

  • VGG-T3 successfully localizes a tourist phone image within a seven-year-older KITTI recording despite temporal and street changes.The query captures upper building regions absent from the car-mounted camera view.Source: D. Additional Qualitative Results · p. 12

Scenes with larger spatial extent

  • On Waymo scenes with larger spatial extent, VGG-T3 can match VGGT but degrades in some complex scene layouts.The authors identify this as a limitation of current linear-time attention mechanisms.Source: D. Additional Qualitative Results · p. 12

Related papers

Loading 2602.23361v1…