PaperlayerView exact revision on arXiv

Source-linked AI summary

Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers

Weitian Wang, Shubham Rai, Cecilia De La Parra, Akash Kumar

arXiv:2610.09274v1cs.CVcs.LG

TL;DR

VGGT’s global attention spans more than 20k tokens even for small scenes, and its non-sparse attention makes sparsity-based acceleration methods a poor fit. BC attention clusters query hash codes within hardware-friendly blocks and combines calibrated hyperplanes with threshold-based error compensation. At γ = 3, BC attention achieves 2.10× to 2.63× latency improvement on all 24 global attention layers during inference and 1.77× to 2.35× on the whole backbone, with negligible accuracy loss (1%).

  • Problem

    VGGT’s global attention spans more than 20k tokens even for small scenes, and its non-sparse attention makes sparsity-based acceleration methods a poor fit.

    Source: §I · p. 1

  • Method

    BC attention clusters query hash codes within hardware-friendly blocks and combines calibrated hyperplanes with threshold-based error compensation.

    Source: §I · p. 2

  • Results

    At γ = 3, BC attention achieves 2.10× to 2.63× latency improvement on all 24 global attention layers during inference and 1.77× to 2.35× on the whole backbone, with negligible accuracy loss (1%).

    Source: B. Latency Performance · p. 6, §VI · p. 7

  • Takeaways & Limitations

    The blockwise design reduces clustering computation and costly on- and off-chip data movement, delivering practical GPU latency improvements for long sequences.

    Source: §I · pp. 1–2

    The approach focuses on local token similarities; the authors identify framewise similarities, especially for temporally related input frames, as future work.

    Source: §VI · p. 7

Abstract

from arXiv · show

The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers with extremely long sequences that causes a significant latency bottleneck. In this paper, we propose blockwise clustered attention (BC attention) to accelerate the global attention layers in VGGT. By limiting the clustering within HW-friendly neighborhood blocks, BC attention reduces the computation overhead of query clustering as well as the costly data movement between on- and off-chip memory. This enables BC attention to scale to long sequences and deliver practical latency improvements on GPUs. Moreover, we introduce a hashing hyperplane calibration method and a threshold-based error compensation method to reduce clustering errors efficiently, which is a bottleneck in the current clustered attention mechanism. Overall, our experiments on GPU demonstrate that calibrated BC attention accelerates the global attention layers by 2.10-2.63$\times$ and the whole backbone by 1.77-2.35$\times$ with negligible loss (1%) for large scenes. With a small performance loss (< 5%), calibrated BC attention further achieves a 2.26-2.87$\times$ latency improvement on the global attention layers and a 1.90-2.55$\times$ improvement on the backbone.

I. INTRODUCTION

VGGT directly infers 3D scene attributes, but its long-sequence global attention is a latency bottleneck. This paper proposes hardware-aware blockwise clustered attention, with calibrated hyperplanes and threshold-based error compensation, to accelerate those layers on GPUs.

I. INTRODUCTION

  • VGGT directly infers key 3D scene attributes from multiple views, while its global attention over all views creates a latency bottleneck.The sequences exceed 20k tokens even for small scenes, limiting efficiency on medium and large reconstructions.Source: §I · p. 1, Figure 1 · p. 1
  • VGGT’s non-sparse attention makes sparsity-based acceleration methods ill-suited, while similar queries make clustered attention a promising alternative.Vanilla clustered attention nevertheless incurs computation and memory-access overheads and is incompatible with highly efficient attention kernels.Source: §I · p. 1
  • BC attention uses hardware-aware blockwise clustering and optimized GPU implementation to deliver practical latency improvements on VGGT with negligible performance loss, while reducing computation overhead and costly on- and off-chip data movement.Source: §I · pp. 1–2
  • Hardware-aware blockwise clustering reduces clustering computation and on- and off-chip data movement for long token sequences.The paper also introduces gradient-based hash-hyperplane calibration using differentiable surrogates.Source: §I · p. 2
  • Threshold-based structured error compensation replaces vanilla clustered attention’s unstructured top-k correction and can use high-performance kernels such as FlashAttention.This is presented as a way to compensate clustering error efficiently.Source: §I · p. 2

II. RELATED WORK

Prior attention-acceleration methods use sparsity or similarity, but VGGT’s attention is comparatively non-sparse. Its query similarities concentrate in neighborhoods, motivating block-limited clustering.

II. RELATED WORK

  • Global k-means clustering has O(kn) cost, with k proportional to n in practice, and large temporary values drive frequent on- and off-chip data movement.Vanilla clustered attention also recomputes top-k weights per query, creating unstructured patterns that are incompatible with efficient kernels such as FlashAttention.Source: §II · p. 2

A. Observations and Insights

  • VGGT attention is nearly uniform in early and late layers and remains flatter than Llama 3.1 8B in middle layers, so sparsity-based methods are not ideal.The distribution supports similarity-based clustering because non-dominant tokens make approximation errors less pronounced.Source: A. Observations and Insights · pp. 2–3, Figure 2 · p. 2
  • Query cosine similarities concentrate near the diagonal across the first three frames, indicating that similar tokens mainly occupy neighborhoods.RoPE is applied at every layer and amplifies positional distinctions, decreasing similarity between distant tokens; this motivates clustering within blocks.Source: A. Observations and Insights · p. 3, Figure 3 · p. 3

B. Hardware-aware Blockwise Clustered Attention

BC attention approximates attention by clustering similar queries and computing attention for cluster centroids. Its hardware-aware blockwise design limits clustering to local blocks, while hyperplane calibration and threshold-based outlier recomputation improve approximation quality.

B. Hardware-aware Blockwise Clustered Attention

  • Clustered attention replaces per-query attention with attention computed for query-cluster centroids, reducing complexity from O(L^2E) to O(kLE).Assignments are obtained by k-means in Hamming space on SimHash codes, and each centroid is the mean of its cluster’s queries.Source: B. Hardware-aware Blockwise Clustered… · p. 3
  • Blockwise clustering reduces the clustering cost from O(IL^2/γ) to O(ILB/γ) by clustering query hash codes within blocks of size B.The smaller temporary values fit in on-chip registers and shared memory; each block requires only one read and one write to HBM, and clustering can converge in fewer iterations.Source: B. Hardware-aware Blockwise Clustered… · p. 3
  • BC attention uses calibrated headwise hash hyperplanes and recomputes outlier outputs using a static layerwise threshold to improve approximation quality.The method combines these mechanisms with blockwise clustering in its attention pipeline.Source: B. Hardware-aware Blockwise Clustered… · pp. 3–4

C. Hyperplane Calibration via Differentiable Surrogate

Random SimHash hyperplanes can produce high-variance clustering, so the section calibrates learnable hyperplanes with a differentiable surrogate before returning to efficient BC attention.

C. Hyperplane Calibration via Differentiable Surrogate

  • SimHash projects queries onto hyperplanes to create binary hash codes, whose Hamming distances support efficient similarity-based query clustering.The implementation uses 32 bits so each query’s hash vector fits in one int32 code.Source: C. Hyperplane Calibration via… · p. 4
  • Randomly sampled SimHash hyperplanes yield high-variance results; learning model-dependent hyperplanes on a calibration set targets downstream losses without adding inference overhead.The method uses headwise learnable parameters for each layer, with 32 hyperplanes per head, and replaces the sign function with signmoid.Source: C. Hyperplane Calibration via… · p. 4
  • A differentiable surrogate replaces non-differentiable BC attention during calibration, minimizing the loss between accurate and approximated outputs before inference switches back to BC attention with calibrated hyperplanes.The surrogate mimics centroid computation and continues through the attention output; the experiments use KL divergence and calibration inputs from one test-set scene.Source: C. Hyperplane Calibration via… · p. 4
  • Soft k-means clusters differentiable hash-code vectors using softmax assignment probabilities, then computes centroids as assignment-weighted query averages.The experiments optimize hyperplanes using KL divergence between the surrogate’s approximated output and the real output.Source: C. Hyperplane Calibration via… · p. 4

D. Threshold-based Clustering Error Compensation

Threshold-based compensation addresses clustering errors by selecting outlier queries using layerwise distance thresholds and combining that work with blockwise centroid aggregation.

D. Threshold-based Clustering Error Compensation

  • Vanilla clustered attention recomputes top-k scores with unstructured operations that are incompatible with FlashAttention and add latency for long sequences with many clusters.Hamming-distance errors can make cluster assignments inaccurate, motivating error compensation.Source: D. Threshold-based Clustering Error… · p. 5
  • The method identifies outliers by comparing each query’s Euclidean distance from its cluster centroid with a layerwise threshold, merging compensation with blockwise centroid aggregation.Marked outliers are excluded from centroid computation, and affected centroids are recomputed.Source: D. Threshold-based Clustering Error… · p. 5
  • Blockwise aggregation loads query blocks and assignments from HBM to SRAM, computes errors and centroids together, and returns a mask for selecting outliers in attention computation.The mask allows different heads to have different outlier counts; FlashAttention’s variable-length kernel handles the resulting attention computation.Source: D. Threshold-based Clustering Error… · p. 5
  • The layerwise threshold is the 90th percentile of query errors across heads, compensating around 10% of queries; thresholds vary across layers but remain consistent across inputs.Fig. 5 presents top-10% outlier thresholds for four random scenes across layers at different depths.Source: D. Threshold-based Clustering Error… · p. 5, Figure 5 · p. 5

A. Approximation Evaluation

Calibrated BC attention preserves reconstruction performance across the reported evaluations, while random hyperplanes and ToMeSD fare worse in the stated comparisons.

A. Approximation Evaluation

  • On ETH3D, calibrated BC attention yields 1% overall performance loss at compression factor γ = 3 and <5% at γ = 4; on high-fidelity DTU, loss is <0.05mm.Source: A. Approximation Evaluation · pp. 5–6
  • Random hyperplanes cause >17% loss on ETH3D, and their reconstruction exhibits noticeable noise versus standard attention and calibrated BC attention.The qualitative comparison is shown in Fig. 6.Source: A. Approximation Evaluation · p. 6, Figure 6 · p. 6
  • ToMeSD performs worse at γ = 3 than random BC attention at γ = 4, even with the top 10% outliers recomputed during merging.ToMeSD merges similar tokens within each frame.Source: A. Approximation Evaluation · p. 6
  • On MapAnything, which uses a similar alternating frame/global attention design, BC attention maintains results similar to the standard model.The paper notes that MapAnything lacks RoPE but has token-similarity patterns similar to VGGT.Source: A. Approximation Evaluation · p. 6, Table III · p. 6

B. Latency Performance

The latency evaluation finds substantial acceleration from BC attention, with larger-scene improvements assessed across 75 to 200 input frames.

B. Latency Performance

  • The comparison covers 75 to 200 input frames, a range used for adequate viewpoint coverage and geometric consistency in large-scene reconstruction.The authors report that BC attention's acceleration becomes more evident for larger scenes because of clustering overhead and latency elsewhere in the model.Source: B. Latency Performance · p. 6
  • At γ = 3, BC attention improves latency 2.10× to 2.63× on all 24 global attention layers and 1.77× to 2.35× on the whole backbone; at γ = 4, the gains are 2.26× to 2.87× and 1.90× to 2.55×, respectively.Experiments use an NVIDIA H200 GPU, PyTorch, Bfloat16 precision, and FlashAttention2 for both standard and BC attention.Source: B. Latency Performance · p. 6

A. Error compensation

Reducing the error compensation rate worsens the reported ETH3D accuracy distance.

A. Error compensation

  • Reducing error compensation from 10% to 5% increases ETH3D accuracy distance from 0.891 to 1.320.The authors cite this change as evidence that sufficient error compensation is needed to maintain BC attention's approximation quality.Source: A. Error compensation · pp. 6–7

B. Block Sizes Choice

The section reports similar performance for block sizes 64 and 128, worse performance at 256, and concludes with calibrated BC attention's acceleration and accuracy results.

B. Block Sizes Choice

  • Block sizes 64 and 128 have similar ETH3D reconstruction accuracy, while 256 performs worse with the same number of k-means iterations.The authors speculate that the larger block size needs more iterations to converge.Source: B. Block Sizes Choice · p. 7, Table IV · p. 7

VI. CONCLUSION

  • Calibrated BC attention achieves 2.1–2.6× speedup on global attention layers and 1.8–2.4× backbone acceleration with 1% accuracy loss; future work could exploit framewise similarities.The proposed approach focuses on local token similarities, and the suggested extension is especially relevant when input frames have temporal relations.Source: §VI · p. 7
Loading 2610.09274v1…