Source-linked AI summary
Vision Transformers Need More Than Registers
Cheng Shi, Yizhou Yu, Sibei Yang
TL;DR
ViTs exhibit artifacts across supervision paradigms, but their shared mechanism has been insufficiently explained. The paper attributes them to lazy aggregation of background patches and introduces LaSt-ViT, which selectively integrates patch features into the CLS token. LaSt-ViT consistently improves results across 12 benchmarks under label-, text-, and self-supervision.
Problem
Artifacts such as attention deficits, dense-feature misalignment, and high-norm tokens occur across supervision paradigms, while their shared mechanism remains insufficiently understood.
Method
The paper analyzes Patch Score and Point-in-Box, attributes artifacts to background-based lazy aggregation driven by global dependencies and coarse-grained supervision, and proposes LaSt-ViT for selective foreground-oriented aggregation.
Results
LaSt-ViT consistently improves performance across 12 benchmarks spanning object discovery, segmentation, and open-vocabulary detection under multiple supervision paradigms.
Takeaways & Limitations
ViT artifacts can be understood as a shared background-dominant shortcut, and selectively anchoring the CLS token to foreground-relevant features provides a broad remedy.
Abstract
from arXiv · showhide
Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior.
1. Introduction
The paper identifies ViT artifacts as a shared background-dominant shortcut caused by coarse-grained supervision and global dependencies, then proposes LaSt-ViT to selectively strengthen foreground representations. The method consistently improves performance across 12 benchmarks and supervision paradigms.
- Motivation: ViT artifacts appear across label-, text-, and self-supervised settings, including attention deficits, dense-feature misalignment, and high-norm tokens that harm downstream tasks.Existing methods do not comprehensively address these phenomena, motivating a unified investigation.
- Unified analysis: Patch Score measures similarity between patch features and the CLS token, while Point-in-Box measures whether the highest-scoring patch lies inside the annotated foreground region.Across supervision settings, ViTs assign higher Patch Scores to background patches and obtain lower PiB than ConvNets.
- Analysis and hypothesis: The authors hypothesize that coarse-grained semantic supervision and global dependencies drive lazy aggregation, diffusing foreground semantics into background patches as a shortcut.Removing the top 50% highest-scoring patches has negligible ImageNet-accuracy impact, supporting reliance on background evidence.
- Proposed solution: LaSt-ViT estimates token contributions and selectively integrates informative patch features into the CLS token to regulate background influence during pre-training.The approach is frequency-aware and encourages attention toward foreground objects.
- Results: LaSt-ViT eliminates Patch Score artifacts across supervision types and improves emergent semantic segmentation properties and performance across 12 benchmarks.The reported benchmarks include object discovery, semantic and instance segmentation, and open-vocabulary detection.
2. Related Work
Prior work shows that CLIP-type ViTs can support dense predictions but often exhibit poor alignment in dense tasks. Existing remedies modify attention, add alignment training, or edit activations at test time.
- Text-supervised ViTs: CLIP features enabled zero-shot semantic segmentation through pixel–text alignment, extending vision–language pretraining beyond image-level classification.
- Text-supervised ViTs: Despite greater model capacity and classification accuracy than ResNets, ViTs often perform worse on dense alignment tasks.
- Existing remedies: Existing approaches address misalignment by modifying final attention layers, adding alignment training, or shifting high-norm activations into untrained register tokens at test time.
3. Preliminary
A ViT splits an image into non-overlapping patches, embeds them as tokens, and updates them with transformer self-attention. Global semantics are then obtained through either pooled patch tokens or a learned CLS query.
- Patch embedding: ViTs split an image into non-overlapping P × P patches and linearly project them into N patch tokens of dimension D.The number of patches is N = HW/P^2.
- Transformer encoder: A transformer encoder updates the embedded patch tokens through stacked self-attention blocks.
- Global aggregation: Global aggregation uses either global average pooling over patch tokens or a learnable CLS query concatenated before encoding.The resulting CLS token serves as the global representation in the query-based formulation.
4. Analysis and Hypothesis
ViTs develop a persistent background-dominant bias: high-scoring background patches can be removed with little accuracy loss, while foreground alignment remains poor from early training. The evidence supports a lazy aggregation hypothesis driven by coarse-grained supervision and global dependencies, with localization improvements trading off against classification accuracy.
- Metrics: Patch Score measures each patch’s similarity to the global representation, while Point-in-Box measures whether the highest-scoring patch lies inside the foreground box.For ViTs the global representation is the CLS token; higher Patch Scores indicate stronger alignment with image-level semantics.
- Patch-score artifacts: Foreground patches concentrate at lower Patch Scores, whereas background patches dominate the high-score tail.This distribution indicates that high similarity to the CLS token does not reliably identify foreground content.
- Patch-score artifacts: Removing high-score patches preserves or slightly improves accuracy, while removing low-score patches causes a sharp drop.For ViT-B/16, high-score masking can improve accuracy by +1.2%, whereas masking 70% of low-score patches reduces accuracy by up to 60%.
- Training dynamics: ViT’s Point-in-Box score stays nearly flat around 0.42 →0.44 during training and remains below ResNet’s despite improving classification accuracy.The background bias appears from the beginning of training rather than emerging as a late-stage byproduct.
- Coarse-grained semantic supervision: Reducing background tokens by increasing patch size raises Point-in-Box from 0.44 to 0.52 but lowers top-1 accuracy from 62% to 55%.High-score regions shift toward objects, revealing a localization–recognition trade-off from naive patch coarsening.
- Global dependencies: Restricting global attention raises Point-in-Box but reduces accuracy, supporting lazy aggregation as a shortcut enabled by coarse-grained supervision and global dependencies.The proposed interpretation is that foreground semantics diffuse into abundant background tokens, producing high image-level accuracy but weaker patch-level consistency.
5. Method
LaSt-ViT reformulates CLS-token aggregation as a frequency-aware selection process that favors stable patch features, aiming to reduce background-driven shortcuts. It then uses patch stability and vote counts to anchor aggregation toward foreground regions and support downstream applications.
- LaSt-ViT: LaSt-ViT distinguishes foreground from background by selecting patch tokens that remain stable under channel-wise low-pass filtering.Foreground signals are described as more semantically homogeneous, while backgrounds have greater channel-wise diversity.
- Channel-wise Top-K Pooling: For each channel, the method selects the K patches with the highest stability scores and averages their features into the CLS token.The selected index set contains the K most stable patches for that channel.
- Vote Count: Each patch’s vote count records how often it is selected across channels, with larger counts indicating greater patch importance.The vote count sums channel-wise selection indicators.
- Foreground Anchoring: After LaSt-ViT, highly voted patches align with foreground regions, indicating that the CLS token primarily aggregates foreground tokens.Figure 5 visualizes patches exceeding 50%, 30%, or 20% of the image’s maximum vote count.
- Downstream Use: LaSt-ViT supports unsupervised object localization by using patch scores to construct masks across training methods.The supplied method description also connects patch-feature similarity to open-vocabulary applications using arbitrary text features.
6. Experiment
The experiments evaluate LazyStrike across supervision types and dense vision tasks, finding that it removes feature-norm and patch-score artifacts while improving downstream performance. Gains appear in semantic segmentation, open-vocabulary detection and segmentation, object discovery, and emergent segmentation and feature structure.
- Experiment Setup: LazyStrike is evaluated under fully supervised, text-supervised, and self-supervised training across object discovery, segmentation, and open-vocabulary detection tasks.The evaluation covers multiple downstream settings, including zero-shot semantic segmentation and instance segmentation.
- Artifact Elimination: LazyStrike eliminates the high-norm phenomenon and improves Point-in-Box scores, approaching ResNet performance.Figure 6 attributes the reduction to lower maximum feature values under fully supervised training.
- Emergent Properties: LazyStrike improves emergent coarse segmentation and separates salient foreground structure in PCA visualizations.In supervised coarse segmentation, it reaches 41.9% versus 47.7% for DINO.
- Zero-shot Semantic Segmentation: Across six zero-shot semantic segmentation benchmarks, LazyStrike consistently outperforms baseline models under text supervision.For CLIP ViT-B/16, mIoU rises from 11.2% to 15.2% on Pascal, 6.5% to 12.1% on Cityscapes, and 49.0% to 75.0% on VOC.
- Open-vocabulary Detection and Segmentation: For OV-COCO, LaSt-ViT gains 15.8% and 14.4% over ViT-B and ViT-L baselines on novel categories.For OV-LVIS, it improves rare-category performance by 11.3% and 6.6% for ViT-B and ViT-L.
- Unsupervised Object Discovery: LaSt-ViT achieves the highest CorLoc scores across the evaluated object-discovery datasets and reaches 55.9 images per second.The reported CorLoc scores are 64.4% on VOC 2007, 67.6% on VOC 2012, and 51.6% on COCO.
7. Conclusion
The paper attributes ViT artifacts to lazy aggregation of background patches and proposes LaSt-ViT to focus CLS aggregation on stable foreground-relevant features. The method removes artifacts across supervision types and improves results on 12 benchmarks.
- Conclusion: ViTs often rely on numerous background patches to encode global semantics, producing a lazy aggregation behavior.The paper frames this behavior as the source of observed artifacts.
- Conclusion: LaSt-ViT uses frequency-guided selective aggregation to focus the CLS token on stable, foreground-relevant features.The conclusion presents this as the proposed countermeasure to background-dominated shortcuts.
- Conclusion: The method eliminates artifacts across supervision types and improves performance on 12 benchmarks.The reported scope includes label-, text-, and self-supervised settings.