PaperlayerView exact revision on arXiv

Source-linked AI summary

VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction

Junyi Wu, Fanqing Kong, Leyang Chen, Shaoqiu Zhang, Yulun Zhang

arXiv:2610.01013v1cs.CV

TL;DR

Long-input 3D vision inference is costly because global attention scales quadratically, and sparse selection may favor value-redundant regions. VASC combines value-aware block selection with execution-updated cross-layer memory; across two datasets and backbones, it improves pose and reconstruction quality over sparse baselines, with up to 2.29× faster inference than dense VGGT.

  • Problem

    Global cross-view attention grows quadratically with visual-token count, while sparse selection must choose fixed-budget interactions without sacrificing geometry prediction or savings.

    Source: §1 · p. 2

  • Method

    VASC combines pooled query–key relevance with adjacent-value contrast and execution-updated cross-layer memory to select blocks under a fixed budget.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Across 7Scenes and NeuralRGB-D with VGGT and π3, VASC improves pose estimation and reconstruction over sparse baselines, with up to 2.29× faster inference than dense VGGT.

    Source: §6 · p. 9

  • Takeaways & Limitations

    VASC applies training-free sparse attention while preserving the native sparse kernel and geometry heads.

    Source: §1 · p. 2, §4 · p. 5

Abstract

from arXiv · show

Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse attention method combining value-aware block selection and execution-aware cross-layer memory. Our value-aware block selection integrates pooled query--key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and $\pi^3$ demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to $2.29\times$ faster inference than dense VGGT. Code is available at https://github.com/kosakayamahoo-design/VASC.

1 INTRODUCTION

Feed-forward geometry models predict multiple scene properties, but scaling long inputs remains costly because global cross-view attention grows quadratically. VASC addresses sparse-selection quality and execution history with value-aware selection and cross-layer memory.

1 INTRODUCTION

  • Global cross-view attention grows quadratically with visual-token count, making it a scalability obstacle for feed-forward geometry models on long inputs.VGGT jointly predicts cameras, depth, point maps, and tracks, while π3 develops permutation-equivariant geometry prediction.Source: §1 · p. 2
  • Sparse selection must choose which interactions receive a fixed computation budget while protecting geometry prediction without overhead that erases the savings.Pooled query–key relevance alone can miss distinctive value content, and independent per-layer selection does not track whether earlier demand was served.Source: §1 · p. 2
  • VASC combines pooled query–key relevance with adjacent-value contrast and carries unserved demand across layers, updating memory according to actual execution under a fixed budget.The method preserves per-query capacity, the native sparse kernel, special-token handling, and geometry heads.Source: §1 · p. 2
  • On 7Scenes and NeuralRGB-D, experiments with VGGT and π3 show VASC improves all six pose and reconstruction metrics over corresponding sparse baselines at S75, while achieving up to 2.29× speedup over dense VGGT.Source: §1 · p. 2

2 RELATED WORK

Related work addresses efficient geometry inference through streaming state, attention sparsity, token reduction, and other training-free acceleration strategies. VASC focuses on selecting value-sensitive blocks inside pretrained global attention while tracking whether assigned demand was executed.

2 RELATED WORK

  • Streaming geometry systems maintain spatial or recurrent state, whereas VASC targets interaction selection inside pretrained multi-view backbones’ global attention.The cited landscape also includes approaches for streaming inputs and maintaining long-term state.Source: §2 · p. 3
  • Efficient-attention and token-reduction methods use different training, token, or inference interfaces; VASC preserves child-block identity and changes support passed to the existing sparse kernel.The passage groups exact-attention kernels, structured sparsity, learned token pruning or aggregation, token merging, and sparse access.Source: §2 · p. 3
  • Geometry-model accelerators include token merging, block-sparse proposals, attention allocation changes, quantization, and streaming KV eviction; VASC selects value-sensitive blocks and tracks executed demand across depth.Related history-aware methods reuse sparse indices or exploit importance dynamics, while VASC records the gap between assigned demand and actual execution.Source: §2 · p. 3

3 ANALYZING ATTENTION AND VALUE IN VGGT

The analysis finds that attention-only selection can degrade reconstruction and that attention priority need not align with local value variation. This motivates treating query relevance and value content as complementary signals under a fixed execution budget.

3 ANALYZING ATTENTION AND VALUE IN VGGT

  • Retaining only the highest-ranked 25% of patch-key connections can still degrade reconstruction, despite renormalizing retained weights and leaving frame attention, special-token handling, and model parameters unchanged.The comparison starts from pretrained VGGT and contrasts dense attention with attention-only selection in global attention layers.Source: §3 · p. 4, Figure 3 · p. 4
  • Highly attended connections can carry locally similar values, while lower-ranked connections can contain more distinct content, so attention and neighboring-value similarity need not agree.Value variation is visualized using each value vector’s mean cosine similarity to its four spatial neighbors; lower similarity indicates greater directional variation.Source: §3 · p. 4, Figure 3 · p. 4
  • Under a fixed budget, a high attention rank does not establish how much distinct content is preserved, and falling below the threshold does not establish irrelevance to the query.The relevant trade-off is which value content is represented within available capacity, not only how much attention is retained.Source: §3 · p. 4
  • VASC treats relevance and value variation as demand signals, while the fixed budget determines execution; actual execution updates cross-layer memory for the next layer.This distinguishes which interactions deserve computation from which can be executed.Source: §3 · p. 4, Figure 4 · p. 5

4 METHOD

VASC combines query–key relevance with value contrast to allocate sparse attention, then carries demand–execution imbalance across layers to refine selections within a fixed budget.

4 METHOD

  • VASC combines query–key relevance and value contrast into current demand, then carries demand–execution mismatch across layers while preserving the native sparse kernel.The method retains the host's geometry heads and special-token handling.Source: §4 · p. 5

4.1 VALUE-AWARE BLOCK SELECTION

  • Adjacent pooled value blocks share an RMS contrast signal that modulates query-specific relevance, with demand rescaled and clipped to each query’s capacity.This combines content distinctiveness with relevance at the native sparse-block granularity.Source: §4.1 · p. 5

4.2 BUDGETED EXECUTION

  • Memory augments current demand, and each query selects binary child executions within its capacity; camera and register queries remain dense, while special-only key blocks stay always kept.Selected masks pass directly to the native sparse attention kernel.Source: §4.2 · p. 6

4.3 CROSS-LAYER MEMORY

  • Pair-level memory averages demand and service across query blocks and paired children, adds demand, subtracts execution, and clips accumulated deficits at zero.The bounded memory copy informs selection in subsequent layers.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments evaluate VGGT and π3 on 7Scenes, NeuralRGB-D, and ScanNet, comparing VASC with sparse baselines across sparsity levels and input strides. VASC improves benchmark metrics while delivering speedups over dense VGGT; ablations examine its components and scheduler adaptations.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers all 18 7Scenes test sequences and nine NeuralRGB-D sequences at strides 5 and 10, plus 50 ScanNet scenes with 100 images each.Reported metrics include pose errors and reconstruction accuracy, completeness, and normal consistency; comparisons share checkpoints, inputs, evaluators, geometry heads, and sparse kernels within each backbone.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • VASC improves all six metrics at S75, all pose metrics across every tested setting, and all six π3 metrics at both sparsities and strides.Table 1 compares strides 10 and 5 on 7Scenes and NeuralRGB-D.Source: §5.2 · p. 7, Table 1 · p. 7
  • On ScanNet-50 at S75, VASC reduces ATE, translational RPE, and rotational RPE by 13.98%, 14.40%, and 18.10% versus FasterVGGT, while Chamfer distance remains nearly unchanged.Source: §5.2 · p. 8
  • On NeuralRGB-D at S85, VASC reaches 2.22× and 2.29× full-model speedups over dense VGGT on A6000 and RTX 4090, with 1.66–2.62% latency overhead over FasterVGGT.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 ABLATION STUDY

  • At S75 and stride 10, full VASC improves all six dataset-level metrics over FasterVGGT on both backbones and datasets.The component ablation compares value-aware-only, Faster+memory, and full VASC variants.Source: §5.3 · p. 8, Table 3 · p. 8
  • Adding value contrast to Faster+memory reduces Acc by 5.20%, 10.09%, 15.32%, and 49.07% on VGGT and 3.96%, 13.85%, 13.89%, and 19.97% on π3 from S55 through S85.The reported reconstruction gains on NeuralRGB-D grow as the execution budget shrinks.Source: §5.3 · p. 8
  • At S75 on π3/NRGBD, adding memory reduces Acc by 19.79% and Comp by 11.08%, while NC rises from 0.831 to 0.881; component trends vary by backbone and dataset.Across sparsities, full VASC improves π3 Acc and NC on both datasets; VGGT Comp gains over either component occur on 7Scenes at all four sparsities and NRGBD at S55 and S75.Source: §5.3 · pp. 8–9, Figure 6 · p. 9
  • Adding memory to FastVGGT and AVGGT-R improves Acc and Comp on both datasets; for FastVGGT, Comp improves 1.20% while Acc worsens 1.27%.FastVGGT adaptation is evaluated within the 7Scenes collection used for adapter selection.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

VASC combines pooled query–key relevance with adjacent-value contrast and cross-layer memory to allocate a fixed sparse-attention budget. On 7Scenes and NeuralRGB-D, VGGT and π3 improve over sparse baselines, with up to 2.29× faster inference than dense VGGT.

6 CONCLUSION

  • VASC improves pose estimation and reconstruction over sparse baselines on 7Scenes and NeuralRGB-D with VGGT and π3, with up to 2.29× faster inference than dense VGGT.Source: §6 · p. 9
  • VASC combines pooled query–key relevance with adjacent-value contrast and carries unserved demand into subsequent selection through cross-layer memory.Source: §6 · p. 9
  • Both components operate within a fixed budget while preserving the native sparse kernel and geometry heads.Source: §6 · p. 9

A IMPLEMENTATION DETAILS

The implementation defines block pairing and fixed-capacity selection, then uses value contrast and execution-updated memory to guide allocation. The appendix also documents state handling and the associated observation and rendering protocols.

A.1 BLOCK SCORES AND BUDGETED SELECTION

  • At each global layer, pooled query blocks index physical key/value child blocks, whose adjacent native-order children form fixed pairs, with a final unpaired child treated as a singleton.Source: §A.1 · p. 13

A.2 PROPORTIONAL CLIPPING

  • Proportional clipping bounds each score by one and total score mass by capacity, while a separate binary Top-K mask enforces exact execution capacity.Source: §A.2 · p. 13

A.3 VALUE CONTRAST AND PAIRING

  • RMS contrast measures differential content between paired values, complementing common content that can remain informative even when contrast is zero.An odd final child is paired with itself for zero contrast and counted as a singleton in demand and service.Source: §A.3 · p. 13

A.4 MEMORY NORMALIZATION AND INITIALIZATION

  • Memory normalization applies separately to each batch entry and head, and only to the ranking copy; the raw balance continues to the next layer.Source: §A.4 · p. 14

A.5 RESIDUAL ACCOUNTING AND STATE LIFETIME

  • The raw pair-level state accumulates demand deficits across layers, while its clipped copy is used for ranking; state resets per inference or on layout mismatch and is settled at the next layer or after the final layer.The selected patch-child mask supplies execution; camera and register queries retain dense attention, and pose, depth, and point heads remain unchanged.Source: §A.5 · p. 14

B OBSERVATION AND VISUALIZATION PROTOCOLS

B.1 ATTENTION-ONLY RECONSTRUCTION AND LOCAL VALUE ANALYSIS

This block documents the attention-only reconstruction intervention and visualization protocols, then describes the scope and setup of the main and denser-input benchmark comparisons.

B.1 ATTENTION-ONLY RECONSTRUCTION AND LOCAL VALUE ANALYSIS

  • The attention-only reconstruction diagnostic retains the highest-ranked 25% of patch keys per query and head across 24 global layers, with special-token handling unchanged.The compared conditions include dense inference and unmasked inference with matched chunking; the diagnostic evaluates reconstruction, not speed.Source: §B.1 · p. 15
  • The illustrated attention and value maps show one selected layer, head, query, and key view; their selected regions are manually annotated and are qualitative evidence, not a prevalence estimate or isolated causal test.Table 5 reports the actual masks for the regions.Source: §B.1 · p. 15, Table 5 · p. 15

B.2 QUALITATIVE RECONSTRUCTION RENDERING

  • Figure 5 uses stride-10 reconstructions from Kitchen, Green Room, and Heads/seq-01, with predictions rigidly aligned to ground truth by point-to-point ICP.Source: §B.2 · p. 15

C ADDITIONAL BENCHMARK RESULTS

C.1 COMPLETE MAIN COMPARISON

  • Table 6 gives the complete stride-10 VGGT and π3 comparison, with sparse baseline and VASC pairs matched for realized capacity across complete native sequences.Source: Table 6 · p. 16

C.2 DENSER INPUT SAMPLING

  • Table 7 compares stride-5 results across sparsity with VASC and FasterVGGT paired at identical realized block capacity; this setting retains every fifth frame without an input cap.Source: Table 7 · p. 17, §C.2 · p. 17

C.3 SCANNET-50

On ScanNet-50, VASC improves all four reported pose means over FasterVGGT, while mean Chamfer is nearly unchanged. The FastVGGT comparison reports metric-specific trade-offs, and the timing passage limits coverage to complete uncontaminated runs.

C.3 SCANNET-50

  • VASC improves all four pose means over FasterVGGT by 13.98%, 14.40%, 18.10%, and 15.15%; mean Chamfer improves 0.08%, with improvements on 23 of 50 scenes.The reported ScanNet-50 metrics are lower-is-better macro means across 50 scenes.Source: §C.3 · p. 17, Table 8 · p. 17

C.4 RECONSTRUCTION COMPARISON WITH FASTVGGT

  • Against FastVGGT, VASC improves Comp and NC medians on NRGBD, with trade-offs in Acc and in 7Scenes NC at stride 5.Table 9 reports sequence-averaged point-level medians across all 18/9 sequences.Source: §C.4 · p. 18, Table 9 · p. 18

C.5 FORWARD TIMING

  • Forward timing measures the complete model forward, including geometry heads, but reports only complete uncontaminated matched runs; some 7Scenes dense and VASC entries are absent.Loading, pose alignment, point-cloud evaluation, and ICP are outside the timed region; missing entries are not imputed.Source: §C.5 · p. 18, Table 10 · p. 18

D COMPONENT ABLATIONS

The component ablations compare variants across four backbone–dataset settings at stride 10, reporting six pose and reconstruction metrics.

D COMPONENT ABLATIONS

  • The component study varies value-aware scoring and cross-layer memory, with value-aware-only removing memory and Faster+memory setting paired-value contrast to one.Tables 11–14 report all six metrics, averaged over 18 7Scenes or nine NeuralRGB-D sequences.Source: D COMPONENT ABLATIONS · p. 19

D.1 VGGT

  • Table 11 reports VGGT component results on 7Scenes at frame stride 10.Source: Table 11 · p. 19

S Variant ATE ↓ RPE-t ↓ RPE-r ↓ Acc ↓ Comp ↓ NC ↑

E MEMORY ADAPTATION TO OTHER ATTENTION-REDUCTION METHODS

The section adapts cross-layer memory to FastVGGT and AVGGT-R while retaining each host’s attention-reduction mechanism. Results include improvements in reconstruction aggregates, with pose regressions in one 90% retained-service setting.

E MEMORY ADAPTATION TO OTHER ATTENTION-REDUCTION METHODS

E.1 FASTVGGT

  • For FastVGGT, memory modifies source-protection priority using merge residuals as arrivals and independent source retention as service, while preserving destination-protection slots and the host’s matcher and merge settings.The adapter’s parameters were selected on two four-sequence subsets, so results characterize in-collection adaptation rather than held-out generalization.Source: §E.1 · p. 21

E.2 AVGGT-R

  • AVGGT-R is an independent reproduction with memory spanning global layers 9–23; both arms retain the same QK-derived priority and correction settings, without adding a paired-value factor.The 85% service-retention ratio refers to the host’s retained service support, not the main experiments’ S85 patch sparsity.Source: §E.2 · p. 21
  • On AVGGT-R, Acc and Comp improve by 3.11% and 1.72% on 7Scenes and 6.44% and 3.80% on NeuralRGB-D; at 90% retained service, three 7Scenes pose means regress by 0.09–0.43%.At that service level, all eight 7Scenes reconstruction aggregates improve.Source: §E.2 · p. 21
Loading 2610.01013v1…