Source-linked AI summary
Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers
Shuhong Zheng, Michael Oechsle, Erik Sandström, Marie-Julie Rakotosaona, Federico Tombari, Igor Gilitschenski
TL;DR
Visual geometry transformers are slowed by the quadratic cost of global attention over many frames. GoToHunt restricts each query’s key/value interactions through hierarchical inter-frame and intra-frame selection, using diversity and layer-aware pruning. The method achieves a superior efficiency–reconstruction-quality trade-off, while remaining subject to the base model’s capacity and difficult scene conditions.
Problem
Global attention over N input frames and L per-frame tokens has O(N^2L^2) complexity, creating a central inference bottleneck for visual geometry transformers.
Method
GoToHunt uses a training-free hierarchical token-selection framework that first selects diverse frames and then applies layer-adaptive token pruning within selected frames.
Results
GoToHunt achieves a superior trade-off between inference efficiency and reconstruction quality compared with existing approaches, occasionally outperforming the base models.
Takeaways & Limitations
The analysis provides practical guidance for selecting tokens and suggests future visual geometry transformers could benefit from routing mechanisms and altered early global-attention design.
Takeaways & Limitations
Inter-frame selection may be less reliable in object-centric, symmetric, or ambiguous scenes, and the method remains bounded by the underlying base model’s capacity.
Abstract
from arXiv · showhide
Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, we introduce a two-stage framework. First, an inter-frame selection step operates at the frame level to identify frames that should be preserved. Second, an intra-frame selection step further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy for inter-frame selection, which ensures broad coverage of the scene. For intra-frame selection, we show that layer-aware sparsification is necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it accelerates visual geometry transformers by over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how our token selection strategy can play a crucial role in future applications of visual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.
1 Introduction
Visual geometry transformers face a quadratic global-attention bottleneck as input frames increase. GoToHunt addresses this with training-free hierarchical token selection, combining diverse frame retention with layer-adaptive within-frame pruning to improve efficiency while preserving performance.
- Motivation: O(N^2L^2) global attention is the dominant inference bottleneck as visual geometry transformers process more frames.The complexity is quadratic in the number of input frames N and per-frame tokens L.
- Approach: GoToHunt selects key/value tokens hierarchically through inter-frame selection followed by intra-frame selection.The first stage retains frames, while the second discards redundant tokens within those frames.
- Inter-frame selection: Diversity-based frame selection is favored because it provides broad scene coverage, whereas similarity- or activation-based criteria can degrade performance.
- Intra-frame selection: Layer-adaptive pruning is used because early layers have diluted attention while middle and late layers show spiking attention values.The strategy applies more conservative pruning where highly activated tokens may be important.
- Results: The training-free GoToHunt solution achieves a superior efficiency–performance trade-off compared with existing acceleration methods.The reported experiments describe competitive speed improvement with minimal performance compromise.
2 Related Works
Related work pursues more practical visual geometry transformers through feed-forward reconstruction and efficiency improvements. Existing efficiency methods include training-free token merging and attention-based pruning, alongside compression-based approaches.
- Feed-forward 3D Reconstruction: Feed-forward 3D reconstruction replaces traditional optimization-heavy multi-view pipelines with models that predict scene geometry directly.The passage identifies DUSt3R and follow-up works as pioneers of this paradigm, though the supplied text ends before describing the predictions fully.
- Efficiency Improvement on Visual Geometry Transformers: FastVGGT introduces training-free token merging that preserves reference and salient tokens while merging the remaining tokens.
- Efficiency Improvement on Visual Geometry Transformers: SparseVGGT analyzes global attention behavior and uses attention-calculation and token-pruning mechanisms to accelerate inference.
- Efficiency Improvement on Visual Geometry Transformers: Compression-based approaches reduce inference cost as another efficiency-improvement direction for visual geometry transformers.
3 GoToHunt: Token Selection for Global Attention
GoToHunt reduces global-attention cost by hierarchically selecting key/value tokens: diverse frames are retained first, then tokens are pruned within selected frames. Its layer-aware intra-frame strategy follows attention entropy, using stronger pruning where attention is diluted and preserving tokens where attention spikes.
- 3.1 Preliminaries and Problem Formulation: Visual geometry transformers process frame-wise and cross-view tokens, but global attention over N × L tokens creates an O(N^2L^2) computational bottleneck.N denotes input frames and L denotes tokens per frame; additional special tokens may also be present.
- 3.1 Preliminaries and Problem Formulation: GoToHunt restricts the key/value tokens available to each query and uses hierarchical inter-frame followed by intra-frame selection.Selection occurs before global attention, first choosing frames and then discarding additional tokens within selected frames.
- 3.2 Inter-frame Selection: Hunting for Good Frames: Inter-frame selection evaluates temporal, co-visibility, and attention-based alternatives before selecting frames that maximize view-space coverage under a fixed budget.The diversity objective is formulated as a K-center problem and approximated with greedy farthest point sampling.
- 3.2 Inter-frame Selection: Hunting for Good Frames: The diversity-based inter-frame strategy outperforms intuitive alternatives and produces anchor views that provide broad scene coverage within the limited frame budget.These anchors are also expected to provide consistent reference views across queries for cross-view processing.
- 3.3 Intra-frame Token Selection: Preserving Necessary Tokens: Uniform intra-frame downsampling across all global-attention layers causes measurable performance degradation, even with a downsampling factor of σ = 2.This motivates analyzing layer-specific attention patterns rather than applying one pruning rule everywhere.
- 3.3 Intra-frame Token Selection: Preserving Necessary Tokens: Early layers have diluted, near-uniform attention, whereas middle and later layers have spiking attention, so pruning should be layer-adaptive.The proposed strategy can replace the earliest global layers with local attention, apply selected downsampling in intermediate layers, and align pruning with layer attention characteristics.
4 Experiments
Experiments evaluate GoToHunt across camera pose, point-cloud, and video-depth tasks, showing strong quality and efficiency trade-offs on long sequences. Additional analyses indicate robustness to layer-threshold choices and non-monotonic effects from expanding the inter-frame budget.
- Experimental Setup: GoToHunt is evaluated with VGGT and π3 across camera pose estimation, point-cloud reconstruction, and video-depth estimation.The evaluation uses multiple datasets and metrics spanning these tasks.
- Camera Pose Estimation: Across camera-pose datasets, GoToHunt achieves superior overall performance and sometimes improves upon the base model.This includes long-sequence 7-Scenes and Neural RGB-D evaluations, as well as TUM-Dynamics.
- 3D Point Cloud Reconstruction: GoToHunt achieves superior overall performance for dense point-cloud reconstruction on scenes with up to 500 frames.The evaluation specifically targets large-scale multi-view reconstruction rather than sparse 3–5-view settings.
- Video Depth Estimation: GoToHunt scales reliably beyond 800-frame video-depth sequences, while SparseVGGT encounters CUDA out-of-memory errors and GoToHunt outperforms the base model.It also achieves superior performance on most metrics compared with Speed3R, which requires costly retraining.
- Inference Efficiency: GoToHunt has near-linear inference-time scaling with input images and achieves a better overall efficiency–performance trade-off than existing approaches.Its constant global-attention key/value budget improves long-sequence efficiency, although LiteVGGT remains slightly faster.
- Parameter Analysis: Performance remains stable across a broad range of intra-frame layer thresholds, while increasing the inter-frame budget can help initially but may later degrade performance.The inter-frame trend is non-monotonic, with larger budgets sometimes approaching full-model results.
5 Discussions
The analysis suggests that token selection is not only an acceleration mechanism but also reveals architectural and training opportunities for visual geometry transformers.
- Improved performance after token selection indicates that current visual geometry transformers may not yet use an optimal architecture.The authors connect this observation to future network-design and training-strategy research.
- Inter-frame findings suggest routing-based attention mechanisms, while intra-frame findings suggest skipping early global-attention layers during training.These are presented as potential directions rather than established methods.
6 Conclusions
GoToHunt formulates visual-geometry-transformer acceleration as token selection and uses hierarchical frame- and token-level sparsification. Experiments show a favorable efficiency–reconstruction-quality trade-off, sometimes exceeding the base models.
- GoToHunt formulates efficiency improvement as selecting a limited set of key/value tokens for global attention.The formulation targets visual geometry transformers directly.
- Its hierarchical framework combines inter-frame selection with intra-frame selection.The two stages first select frames and then prune tokens within selected frames.
- Diversity-based strategies suit inter-frame selection, while layer-adaptive pruning is needed for intra-frame selection.The conclusion presents these as the principal strategy findings.
- GoToHunt achieves a superior trade-off between inference efficiency and reconstruction quality compared with existing approaches, occasionally outperforming base models.The method is training-free and intended as a general acceleration algorithm for visual geometry transformers.
Technical Appendices and Supplementary Material
The supplementary material provides implementation details, strategy analyses, and practical guidance for the proposed selection method. It explains the diversity-based frame-selection objective through feature similarity and coverage.
- The appendix documents the diversity-based inter-frame selection algorithm and additional analyses validating the method.It also reports alternative intra-frame strategies that were explored but not retained in the final method.
- Inter-frame selection uses place-recognition features, cosine similarity, and a K-center objective to maximize scene coverage.The objective is approximately solved with greedy farthest point sampling.
B.1 Token-level Diversity-based Selection
Token-level diversity-based selection (TLD) further reduces redundant tokens within selected frames, but its effectiveness varies by task and its per-frame FPS computation adds substantial overhead. Entropy-guided layer partitioning offers a viable alternative, yet its runtime cost likewise prevents adoption in the final model.
- Token-level Diversity-based Selection: TLD over-selects tokens with FPS, scores redundancy using cross-frame cosine similarity, and retains the least redundant candidates within each selected frame.The procedure promotes diversity both within frames and, to some extent, across frames.
- Token-level Diversity-based Selection: TLD consistently outperforms the standard strategy in nearly all configurations for video depth estimation, but is comparable to or worse than standard selection for pose estimation and 3D reconstruction.The comparison covers video depth estimation in Table A, pose estimation in Table B, and point-map estimation in Table C.
- Token-level Diversity-based Selection: Roughly 5 seconds are required for TLD on a scene with 500 frames, so the strategy is excluded from the official solution.Unlike inter-frame diversity sampling, TLD runs FPS within each selected frame before the second selection stage.
- Mean-pooling Approximation: Replacing local attention with mean pooling produces noticeable degradation across many configurations, indicating that the approximation is not robust enough for the method.This variant is evaluated in Table D using K = 25 and σ = 2 on Neural RGB-D camera pose estimation.
- Layer Partitioning with Entropy Thresholds: Entropy thresholds can adaptively assign layer-wise strategies, but computing attention entropy during inference adds approximately 7 seconds for a 500-frame scene.The adaptive scheme achieves competitive performance but is presented as a reference rather than used in the final model.
C Additional Experimental Results for Sensitivity Analysis
The layer thresholds llocal and lsample remain robust across base models, including π3 and VGGT, supporting their reliability under the reported settings.
- Sensitivity Analysis: The layer thresholds llocal and lsample are robust with π3 and remain reliable when verified using VGGT.The VGGT verification further enhances the reported reliability and soundness of the method.
D Detailed Explanation on Evaluation Metrics
The appendix defines evaluation metrics for camera pose estimation, 3D reconstruction, and video depth estimation, clarifying what each metric emphasizes. It also documents datasets, licenses, baselines, and robustness checks used in evaluation.
- Camera Pose Estimation: ATE measures global trajectory error and captures long-term drift, whereas RPE measures local motion consistency over a fixed interval of Δ = 1.RPE therefore reports relative pose error between adjacent frames.
- 3D Reconstruction: 3D reconstruction uses Accuracy, Completeness, and Normal Consistency to assess point-cloud proximity, ground-truth coverage, and surface-normal alignment.The metrics compare predicted and ground-truth geometry from complementary perspectives.
- Video Depth Estimation: Abs Rel emphasizes relative depth accuracy, while Sq Rel is more sensitive to outliers because it penalizes large errors through a squared term.These metrics characterize different aspects of video depth prediction quality.
- Video Depth Estimation: RMSE measures absolute depth discrepancy and penalizes large deviations, whereas Log RMSE emphasizes relative differences across depth ranges.RMSE can be dominated by distant-region errors, while Log RMSE balances near and far regions more effectively.
- Video Depth Estimation: Threshold Accuracy (δ < 1.25) reports the fraction of predictions within a multiplicative error bound of the ground truth.It provides an intuitive reliability measure rather than averaging prediction errors.
G Limitations
The method depends on place-recognition features and the capacity of its base model, limiting reliability in ambiguous scenes and scalability to extremely large environments. It remains complementary to chunk-based or recurrent approaches, while raising potential misuse concerns.
- Method Scope: Place-recognition features may be less reliable in object-centric, symmetric, or ambiguous scenes, which are also difficult for conventional reconstruction pipelines.This dependency constrains inter-frame selection in those settings.
- Method Scope: The approach remains bounded by the base model and does not alter the overall inference paradigm, so kilometer-scale scenes may require chunk-based processing or recurrent inference.The authors describe the token-selection method as orthogonal and potentially integrable with those techniques.
- Societal Impact: Reduced inference cost may facilitate deployment on resource-constrained platforms and edge devices, including applications such as digital cultural heritage preservation.The paper frames these as potential positive societal impacts.
- Societal Impact: The paper identifies potential misuse for unlawful surveillance that could harm individual privacy.It reports no obvious negative societal impact while noting this possible risk.