PaperlayerView exact revision on arXiv

Source-linked AI summary

GPA-VGGT: Geometry-Preserving Acceleration of VGGT

Muyeol Sung, Jangwon Kim, Jongwoo Lim

arXiv:2610.37000v2cs.CV

TL;DR

VGGT’s quadratic global attention limits runtime on long sequences, and existing acceleration methods can lose precision under strict pose evaluation. GPA-VGGT uses training-free attention and merging changes to reduce inference time while preserving accuracy close to vanilla VGGT across evaluated benchmarks.

  • Problem

    VGGT’s global attention is quadratic and dominates runtime on long sequences, while existing acceleration methods can degrade camera-pose precision under strict AUC@3 evaluation.

    Source: §1 · p. 1

  • Method

    GPA-VGGT accelerates inference by preserving queries during K/V merging, restoring represented tokens’ attention mass, and retaining compact shared cross-frame context in early blocks.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    At 200 frames, GPA-VGGT runs in approximately 34 s versus 87 s for vanilla VGGT and stays within 1% of vanilla VGGT on evaluated AUC@3 benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across diverse multi-view geometry evaluations, GPA-VGGT reduces inference time while preserving vanilla VGGT accuracy under standard and strict metrics.

    Source: §6 · p. 8

    The reimplemented AVGGT reproduces the reported qualitative RE10K trend, but its absolute AUC values are not comparable because its subset and sampling are not public.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

Feed-forward geometry models such as VGGT recover camera parameters, depth maps, and point maps from one to hundreds of views in a single forward pass, making them powerful backbones for multi-view 3D perception. However, VGGT's global attention is quadratic in the number of frames and dominates end-to-end runtime on long sequences, which has motivated a range of training-free efficiency methods such as token merging. These methods, however, come with a drop in accuracy that makes them hard to use in place of VGGT. We propose GPA-VGGT, a training-free acceleration method that replaces global attention in early blocks with frame attention augmented by a shared cross-frame K/V set. In the remaining blocks, we merge only keys and values while preserving all queries, restore the attention mass of the tokens each merged key represents, and randomly project the matching metric to 128 dimensions to reduce matching cost. Across diverse multi-view geometry tasks, GPA-VGGT provides training-free acceleration at speedups comparable to prior work while preserving the information required for precise geometric estimation, retaining over 99% of VGGT's pose accuracy under AUC@3.

1. Introduction

VGGT’s quadratic global attention limits long-sequence scalability, while existing acceleration methods can lose camera-pose precision under strict AUC@3. GPA-VGGT combines training-free attention and token-merging changes to accelerate inference while retaining accuracy close to vanilla VGGT.

1. Introduction

  • At 200 frames, GPA-VGGT runs in approximately 34 s versus 87 s for vanilla VGGT and remains within 1% of vanilla VGGT on evaluated AUC@3 benchmarks.It matches the runtime of other baselines while showing a particularly large advantage over them on the more challenging setting.Source: Figure 1 · p. 1, §1 · p. 2
  • Global attention scales quadratically with the number of views and dominates runtime on long sequences, limiting VGGT’s scalability.VGGT jointly predicts camera parameters, depth maps, point maps, and tracks from unposed views.Source: §1 · p. 1
  • Existing accelerators can appear accurate at relaxed pose thresholds but show more noticeable degradation at strict AUC@3, which tests fine-grained geometric precision.The introduction identifies this precision as consequential for using an accelerated model in place of VGGT.Source: §1 · p. 2
  • GPA-VGGT preserves every query while merging keys and values, restores represented tokens’ attention mass, and adds a compact shared cross-frame K/V set in early blocks.It also uses a fixed low-dimensional projection of the matching metric to reduce matching cost.Source: §1 · p. 2
  • The paper identifies query merging, attention-mass distortion, and early removal of cross-frame context as sources of accuracy degradation in existing accelerators.Its proposed components address these issues while retaining training-free evaluation across multi-view geometry benchmarks.Source: §1 · p. 2

2. Related Work

Related work describes VGGT’s attention bottleneck and surveys acceleration approaches, including training-free attention and token-merging methods. It also distinguishes methods relying on additional training or reduced precision from the training-free architectural design targeted here.

2.1. VGGT Acceleration

  • VGGT predicts camera parameters, depth maps, point maps, and tracking features from unposed views; its global attention is the primary long-sequence bottleneck because its complexity is quadratic in token count.Its Transformer alternates frame attention and global attention.Source: §2.1 · p. 2
  • FlashVGGT and LiteVGGT use learned descriptors or geometry-aware cached merging, while Co-Me uses a distillation-trained confidence predictor; QuantVGGT and QVGGT use reduced precision.These approaches rely on additional training or reduced precision and are outside the targeted training-free architectural design.Source: §2.1 · p. 2
  • FastVGGT merges tokens, FasterVGGT uses block-sparse attention, AVGGT subsamples patch tokens after early frame-attention conversion, and HTTM performs head-wise temporal merging.These methods provide acceleration, but degradation is more apparent under strict AUC@3.Source: §2.1 · p. 2

2.2. Token Merging

  • ToMe merges tokens by attention-key similarity and adds log group size to preserve represented attention mass, while ToMeSD omits that correction and uses a merge–unmerge scheme.FastVGGT adopts ToMeSD-style choices for a 3D geometry backbone, where they can cause fine-grained information loss.Source: §2.2 · p. 3

3. Analysis

The analysis attributes token-merging accuracy loss to collapsed token-specific outputs and distorted attention mass. It also finds that early frame attention disrupts shared geometric reference, while compact shared cross-frame context can mitigate that drift.

3.1. Token Merging Disrupts Token-Specific Predictions and Attention Mass

  • Query merging assigns a shared output to group members, collapsing within-group output diversity, whereas K/V-only merging closely overlaps with unmerged outputs in the block-12 diagnostic.The Figure 2 top panel compares 83 tokens represented by 12 merge slots.Source: §3.1 · p. 3, Figure 2 · p. 3
  • Without + log s, merged keys receive only one token’s attention mass and bias attention toward the first frame; the correction restores a distribution close to the unmerged model.The mass distortion causes a larger accuracy loss than query merging, and the two losses are approximately additive.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Early Blocks Still Need Cross-Frame Context

  • At block 8, frame attention displaces the output field by 15.5% of its radius, while a shared cross-frame K/V set reduces displacement to 7.8%.The mean cosine similarity of per-frame drift directions is −0.09, indicating no common drift direction.Source: §3.2 · pp. 3–4
  • Frame attention’s view-specific drift disrupts relative geometry, while shared special tokens and pooled patch summaries progressively align drift and preserve a shared geometric reference.The analysis says compact shared context is sufficient; it does not require the full cross-frame key set.Source: §3.2 · p. 4

4. GPA-VGGT

GPA-VGGT reduces global-attention cost while preserving token-specific queries and early cross-frame context. Its design combines size-corrected K/V merging, sensitivity-aware merge ratios, and projected matching.

4.1. Query-Preserving K/V Merging with Token-Size Compensation

  • Preserving every query while merging only keys and values gives each token an individual output, at an attention cost factor of N/Nmerged versus merging queries.Adding log s to each merged key's logit restores the attention mass of its represented tokens; the approximation is exact when keys within each group are identical.Source: §4.1 · p. 4

4.2. Pooled Cross-Frame Broadcast for Early Layers

  • Early blocks retain cross-frame context through a shared K/V set containing all frames’ camera and register tokens plus pooled patch summaries.Content-adaptive summaries group patches by head-specific cosine similarity to stride-8 seeds, average their keys and values, and use a + log s correction.Source: §4.2 · pp. 4–5

4.3. Core-Block Sensitivity-Aware Merging

  • Blocks 13–16 are more sensitive to merging than other merged blocks, so they use a weaker merge ratio of 0.2 versus 0.9 elsewhere in the merging window.The paper attributes this sensitivity to sharp attention on a few keys, especially cross-view correspondences, whose information can be diluted by averaging.Source: §4.3 · p. 5

4.4. Gaussian Random Projection for Efficient Matching

  • Projecting matching features with a shared random Gaussian matrix from 1024 to 128 dimensions reduces matching cost by approximately 7–8×.The projected cosine scores select destination indices; because features are redundant and near-tied matches are common, the resulting groups remain similarity-based.Source: §4.4 · pp. 5–6

4.5. Custom Kernels

  • Two custom Triton kernels implement attention with the + log s bias inside its online-softmax loop and source–destination argmax without storing the score matrix.Both kernels are numerically equivalent to their PyTorch references.Source: §4.5 · p. 6

5. Experiments

The experiments assess strict-threshold pose accuracy, transfer to point-map reconstruction, runtime across sequence lengths, and component contributions. GPA-VGGT preserves accuracy across the evaluated benchmarks and sequence lengths while accelerating inference.

5.2. Camera Pose

  • GPA-VGGT’s AUC@3 gap from vanilla VGGT does not widen as CO3D sequences grow from 64 to 200 frames, while competing methods remain less accurate.Table 2 covers 64, 100, and 200 input frames.Source: Table 2 · p. 6, §5.2 · p. 7

5.1. Experimental Setup

  • The evaluation follows the π3 protocol, testing pose on CO3D and RE10K and point maps on 7-Scenes and NRGBD, with the same VGGT checkpoint and no fine-tuning.Pose is also tested on 64-, 100-, and 200-frame CO3D sequences; point-map tests use sparse and dense samplings.Source: §5.1 · p. 6

5.2. Camera Pose

  • Under strict AUC@3, competing methods show more pronounced gaps, while GPA-VGGT preserves accuracy across both CO3D and RE10K.AVGGT(2) stays closer to vanilla VGGT on RE10K than on CO3D.Source: §5.2 · p. 7
  • In qualitative reconstructions, GPA-VGGT preserves one clear object boundary and remains visually close to vanilla VGGT, unlike the split edges or layered surfaces seen with other methods.The comparison shows point clouds by method, with red boxes marking enlarged regions.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3. Point-Map Reconstruction

  • GPA-VGGT preserves point-map quality across datasets and sampling densities, with near-vanilla normal consistency in every setting.It is closest to vanilla VGGT on dense NRGBD and sparse 7-Scenes; AVGGT(2) performs best on sparse NRGBD but degrades more on dense NRGBD and sparse 7-Scenes.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4. Runtime

  • At 200 frames, GPA-VGGT achieves a 3.24× speedup; AVGGT(4) reaches 4.60× at 500 frames but incurs substantially larger accuracy loss.At 10 frames no method exceeds 1.2×; at 500 frames FasterVGGT runs out of memory.Source: Table 4 · p. 8, §5.4 · p. 8

5.5. Ablation

  • Pooled cross-frame broadcast recovers 2.82 of the 3.06 CO3D AUC@3 points lost by early global-to-frame conversion, at 0.7 additional seconds.Query-preserving K/V merging then adds acceleration without substantially compromising accuracy; random-projection matching and sensitivity-aware merging further reduce overhead with only a small additional accuracy effect.Source: §5.5 · p. 8, Table 5 · p. 8

6. Conclusion

The paper identifies accuracy losses in accelerated VGGT and presents components intended to address them. Across multi-view geometry evaluations, GPA-VGGT reduces inference time while preserving vanilla VGGT accuracy under standard and strict metrics.

6. Conclusion

  • GPA-VGGT reduces inference time while preserving vanilla VGGT accuracy under both standard and strict metrics across diverse multi-view geometry evaluations.The conclusion identifies query merging, distorted attention mass, and removal of early cross-frame context as sources of degradation.Source: §6 · p. 8
  • Query-Preserving K/V Merging with + log s correction, Pooled Cross-Frame Broadcast, and low-dimensional matching address the identified accuracy-loss sources.The sources are query-merging information loss, incorrect attention mass after K/V merging, and removal of early cross-frame context.Source: §6 · p. 8
  • The conclusion’s scope is diverse multi-view geometry evaluations, with the stated accuracy comparison against vanilla VGGT under standard and strict metrics.It does not specify individual datasets or numerical results in this passage.Source: §6 · p. 8

A.1. Attention with a Per-Key Additive Bias

A per-key additive bias causes a standard attention interface to fall back to a slower implementation. A custom Triton kernel applies the correction within the online-softmax loop to retain fused attention.

A.1. Attention with a Per-Key Additive Bias

  • 2,142 ms versus 948 ms: with + log s correction, the standard-interface fallback is slower than unbiased fused attention on a 200-frame sequence.The fallback takes more time than key merging saves in the merged global-attention block.Source: §A.1 · p. 10
  • The Triton kernel adds the key-index-dependent correction to each score tile inside the online-softmax loop, preserving FlashAttention’s streaming structure.It avoids padding and copying K/V tensors by clamping out-of-range key indices and biasing duplicate columns.Source: §A.1 · p. 10
  • 984 ms: the custom kernel is 4% slower than unbiased fused attention and 2.18× faster than the fallback for 200 frames.The bias is pre-scaled on the host to avoid conversion inside the inner loop.Source: §A.1 · p. 10

A.2. Memory-Efficient K/V Matching

The matching kernel computes only each row’s maximum similarity and argmax rather than materializing the full similarity matrix. Its tile-wise computation reduces temporary memory substantially, with a modest runtime increase over the chunked reference.

A.2. Memory-Efficient K/V Matching

  • 0.003 GiB: the Triton matching kernel’s temporary memory, versus 1.20 GiB for a chunked PyTorch reference and 27.25 GiB for the full matrix at 200 frames.The kernel keeps score tiles in registers and writes only the row-wise maximum and argmax outputs.Source: §A.2 · p. 10
  • The matching operation needs only the row-wise maximum similarity and argmax index, not the complete similarity matrix.The fused kernel partitions feature dimensions into smaller tiles and accumulates partial dot products while computing those outputs.Source: §A.2 · p. 10
  • 482 ms versus 330 ms: the similarity kernel runs slower than the chunked reference, trading roughly 150 ms for substantially lower memory use.Materialized similarity tiles incur transient VRAM costs that can limit long sequences.Source: §A.2 · p. 10

B. Peak VRAM Footprint

GPA-VGGT reduces attention computation without increasing peak VRAM, while AVGGT matches vanilla VGGT’s peak footprint. For the shared cross-frame K/V set, per-head clustering is the selected pooling strategy.

B. Peak VRAM Footprint

  • GPA-VGGT stays within 0.02 GiB of vanilla VGGT across sequence lengths; at 500 frames, FastVGGT uses 6.02 GiB more VRAM, and FasterVGGT uses 13.72 GiB more at 400 frames and exceeds device memory at 500.AVGGT matches vanilla VGGT’s peak exactly across sequence lengths.Source: B. Peak VRAM Footprint · p. 10
  • The unchanged feed-forward sublayer sets full-model peak VRAM, while FlashAttention keeps modified attention memory linear and nearly identical to vanilla VGGT.Query merging and token restoration in FastVGGT add full-length copies and unmerging buffers, raising its attention-memory peak above the feed-forward peak.Source: B. Peak VRAM Footprint · p. 10

C. Pooling for the Shared Cross-Frame K/V Set

D. Merge-Operator Ablation

The ablations support preserving queries and correcting merged-key attention mass, selecting core blocks by output change, and using 128-dimensional matching features. GPA-VGGT also compares favorably with AVGGT and transfers to π3 and VGGT-Ω with backbone-specific settings.

D. Merge-Operator Ablation

  • Query preservation plus the + log s correction nearly recovers no-merging accuracy, with larger benefits at higher merge ratios.The correction restores represented tokens’ attention mass and provides the larger accuracy gain; merging queries makes tokens in a group share one output.Source: D. Merge-Operator Ablation · p. 11

E. Selection of the Core Blocks

  • Blocks 13–16 are selected as VGGT’s core set, giving the smallest output change across four datasets, while no three-block set matches it.The selection measures final global-block output deviation without ground-truth annotations; sets are evaluated together because merging can alter later representations and assignments.Source: Figure S1 · p. 11, E. Selection of the Core Blocks · p. 11

F. Matching Feature Dimension

  • Projecting matching features from 1,024 to 128 dimensions reduces each matching call from 469 ms to 64 ms; destination similarity falls only 0.56% versus the full-metric optimum.Although 76.6% of source tokens select the same destination as with the full-dimensional metric, 128 dimensions is the smallest tested projection that maintains accuracy close to it.Source: Table S4 · p. 12, F. Matching Feature Dimension · p. 12

G. Comparison with AVGGT

  • At equal token budget, GPA-VGGT aggregates removed tokens into their closest retained token and restores attention mass, whereas AVGGT uses one unweighted mean K/V pair.The 128-dimensional Gaussian projection reduces matching overhead, enabling fewer retained K/V tokens than AVGGT(2) alongside lower runtime and higher accuracy.Source: G. Comparison with AVGGT · p. 12

H. Transfer to Other Backbones

  • The design transfers to π3 and VGGT-Ω, but both use lower overall merge ratios than VGGT because patch tokens participate in fewer global-attention blocks.For π3, merging is applied in blocks 10–17 with lower ratios in core blocks 13–15; VGGT-Ω excludes register blocks from cross-frame attention and uses lower ratios in raw blocks 12, 13, 15, and 16. The shared cross-frame K/V set mitigates accuracy loss from replacing early global attention with frame attention, and the adopted configurations accelerate all three backbones while retaining most vanilla accuracy.Source: H. Transfer to Other Backbones · p. 12

I. Additional Qualitative Results

Additional qualitative comparisons show examples from 7-Scenes, NRGBD, and RealEstate10K.

I. Additional Qualitative Results

Related papers

Loading 2610.37000v2…