PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers Need Registers

Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski

arXiv:2309.16588v2cs.CV

TL;DR

Vision transformers develop high-norm outlier tokens in low-information regions, apparently repurposing patch tokens for global computation while discarding local information. The paper adds learnable register tokens to provide that computational role explicitly. Across supervised, text-supervised, and unsupervised training methods, registers remove the output outliers and improve dense prediction and object discovery while smoothing feature maps.

  • Problem

    High-norm artifacts in modern vision-transformer features and attention maps can make DINOv2 incompatible with LOST despite its strong dense-prediction features.

  • Method

    The paper appends learnable register tokens after patch embedding and discards them at the transformer output, leaving [CLS] and patch tokens as image representations.

  • Results

    Registers entirely remove output high-norm tokens across supervised, text-supervised, and unsupervised models, improving dense prediction and object discovery.

  • Takeaways & Limitations

    Providing dedicated register tokens fixes an interpretability and performance issue while producing smoother feature and attention maps for downstream visual processing.

  • Takeaways & Limitations

    The paper does not fully determine which training factors produce artifacts; pretraining paradigm, model size, and training length all appear relevant.

Abstract

from arXiv · show

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

1 INTRODUCTION

Modern vision transformers can produce high-norm outlier tokens in low-information image regions, repurposing them for global computation and discarding local information. Adding register tokens removes these outliers, smooths feature and attention maps, and improves downstream dense prediction and object discovery.

  • Generic visual features remain important because task-specific annotated data is costly or requires specialized expertise.Pretraining on data-rich tasks and reusing the resulting feature extractors offers an alternative.
  • DINOv2 provides strong dense-prediction features but performs disappointingly with LOST, unlike DINO’s interpretable attention-based object discovery.This contrast motivates investigating why DINOv2 behaves differently from DINO.
  • Outlier tokens have roughly 10x higher norms, comprise around 2% of the sequence, emerge in middle layers, and occur in patches similar to their neighbors.DINOv2 has 2.37% of tokens with norm larger than 150, while DINO does not show the same feature-norm artifacts.
  • Compared with ordinary tokens, outlier tokens retain less positional and local pixel information but provide stronger image-classification accuracy, indicating recycled global information.The proposed interpretation is that the model discards redundant local information and repurposes those tokens for global computation.
  • Appending additional register tokens causes outlier tokens to disappear, increases dense-prediction performance, and produces significantly smoother feature maps.The smoother maps also enable object discovery methods such as LOST with updated models.

2 PROBLEM FORMULATION

DINOv2 exhibits high-norm token artifacts that emerge in sufficiently large, trained models, concentrate on redundant patches, and retain global while losing local information. The paper proposes adding learnable register tokens so global computations no longer repurpose patch tokens.

  • 2 PROBLEM FORMULATION: Modern vision transformers can produce attention-map artifacts, with DINOv2 combining strong local features and undesirable attention artifacts.
  • 2.1 ARTIFACTS IN THE LOCAL FEATURES OF DINOV2: In a 40-layer ViT-g, outliers emerge around layer 15 and after one third of training, while only the three largest model sizes exhibit them.
  • 2.1 ARTIFACTS IN THE LOCAL FEATURES OF DINOV2: High-norm tokens occur on patches highly similar to their four neighbors, indicating that they contain redundant local information.
  • 2.1 ARTIFACTS IN THE LOCAL FEATURES OF DINOV2: Linear probes show that high-norm tokens encode less patch position and pixel information than normal tokens.
  • 2.1 ARTIFACTS IN THE LOCAL FEATURES OF DINOV2: High-norm tokens achieve much higher image-classification accuracy than regular patch tokens, suggesting they store more global image information.
  • 2.2 HYPOTHESIS AND REMEDIATION: The proposed hypothesis is that large, sufficiently trained models recycle redundant patch tokens to store and process global information, potentially harming dense prediction.
  • 2.2 HYPOTHESIS AND REMEDIATION: Adding learnable register tokens gives global computations dedicated sequence elements; patch and [CLS] tokens remain the output representations, while the training causes outlier tokens to disappear.

3 EXPERIMENTS

The experiments evaluate register tokens across supervised, text-supervised, and self-supervised ViTs, showing artifact removal without performance regression and improved object discovery in key models.

  • Artifact removal: Register-trained models eliminate large-norm output tokens across DINOv2, OpenCLIP, and DeiT-III.This quantitatively confirms the qualitative removal of feature-map artifacts.
  • Downstream evaluation: Using registers does not degrade frozen-feature evaluation and slightly improves performance in some cases.OpenCLIP zero-shot classification remains unchanged; OpenCLIP linear evaluation changes from 59.9 to 60.1.
  • Register ablation: One register removes visible artifacts, while additional registers improve downstream performance on ImageNet, ADE-20k, and NYUd.The ablation trains DINOv2 ViT-L/14 models with 0, 1, 2, 4, 8, or 16 registers.
  • Object discovery: Adding registers significantly improves LOST object discovery for DINOv2 and DeiT-III, but slightly worsens OpenCLIP performance.For DINOv2 on VOC2007, corloc improves by 20.1 points, from 35.3 to 55.4.
  • Register behavior: Register tokens develop differentiated attention patterns, with some attending to distinct image regions rather than behaving identically.This differentiation emerges naturally during training and was not explicitly required.

4 RELATED WORK

Related work covers pretrained visual feature extraction, self-supervised Vision Transformers, additional transformer tokens, and attention-map analysis for visual representations.

  • Feature extraction: Pretrained visual features support downstream tasks, while annotation expertise and collection costs motivate generic representations.Earlier approaches used handcrafted features such as SIFT; later methods enabled end-to-end training.
  • Self-supervised learning: Self-supervised Vision Transformers learn image structure through pretext tasks such as masked pixel reconstruction.MAE is cited as an example that reconstructs hidden image areas before fine-tuning.
  • Additional tokens: Transformer extensions add special tokens to provide information, enable extra computation, or gather information for model outputs.Examples include BERT [SEP] tokens and AdaTape tape tokens.
  • Attention maps: Attention-map research includes visualizing [CLS]-to-patch attention and methods that modify optimization or steer attention toward useful image regions.DINO popularized clean attention-map visualization for local visual structure.

5 CONCLUSION

The conclusion characterizes high-norm artifacts as recycled tokens from low-information regions and proposes additional tokens that remove them while improving dense prediction and object discovery.

  • Conclusion: The study finds high-norm outlier tokens in multiple popular vision models and identifies output norm as a simple detection signal.The broader phenomenon is not limited to DINOv2.
  • Proposed fix: The proposed fix appends additional input tokens that are discarded as outputs, absorbing the computation associated with high-norm outliers.The method removes artifacts in supervised and self-supervised models.
  • Outcomes: Register tokens improve dense prediction and object discovery while removing artifacts from DINOv2, DeiT-III, and OpenCLIP.The conclusion presents this as evidence that the solution generalizes across training paradigms.
  • Artifact location: Outliers tend to occur near feature-map borders, where object-centric images often place less-informative background.The authors connect this spatial pattern to token recycling by the base model.

B COMPLEXITY ANALYSIS

Register tokens add little parameter overhead and modest computational cost, while intermediate LOST computations become visibly cleaner for DeiT-III and DINOv2.

  • Complexity: Four registers increase FLOPs by below 2%, while the parameter increase remains negligible.Sixteen registers can increase FLOPs by up to 6%.
  • Complexity: Sixteen registers produce the largest reported FLOP increase, reaching up to 6%.The common four-register configuration has substantially lower overhead.
  • Intermediate computations: Adding registers drastically improves intermediate LOST visualizations for DeiT-III and DINOv2, with a smaller effect for OpenCLIP.The comparison concerns the appearance of intermediate computations rather than a reported benchmark value.

C ANALYSIS OF LOST PERFORMANCE

Registers improve LOST’s seed expansion for DeiT-III and DINOv2 by removing artifacts, while OpenCLIP’s score changes little despite visibly cleaner maps.

  • For OpenCLIP, registers remove artifacts but have little impact on the LOST score.OpenCLIP remains comparable to DINOv2 without registers and DeiT-III with registers.
  • OpenCLIP seed expansion artifacts appear in keys and queries without registers, whereas values remain artifact-free with or without registers.With registers, the LOST score focuses more on the object and becomes smoother.
  • The analysis examines whether registers replace high-norm tokens and assume their computational role.

D.1 NORMS

Registers absorb the high-norm outlier behavior: patch-token outliers disappear, while the information and probing behavior of ordinary patches remain largely unchanged.

  • With four registers, DINOv2 patch-token norm distributions no longer contain high-norm outliers.The high-norm tokens are entirely contained among the registers.
  • The norms of registers appear quantized relative to the earlier outliers, a phenomenon left for future work.
  • Adding registers does not significantly modify linear-probing scores for [CLS] or patch-token representations.The outlier behavior transfers to the newly added register.
  • For normal patches, local information remains similar in models trained with and without registers.This supports removing outlier behavior without substantially changing information in other patches.

D.3 POSITIONAL FOCUS

Registers exhibit distinct positional attention patterns, generally covering broad image areas like the [CLS] token rather than localized patch regions.

  • Register attention maps differ, with register 3 emphasizing borders and other registers focusing more centrally.Register 2 focuses slightly more on upper image areas, suggesting specialization.
  • Registers have broad-support attention maps similar to [CLS], unlike the more localized maps of typical patch tokens.Because [CLS] carries global information, the authors suggest registers also carry global information.

E MASKED AUTOENCODERS

Masked autoencoder feature maps show no observed artifacts, unlike the attention artifacts found across heads in a DINOv2 model without registers.

  • The MAE output feature map shown in its first three principal components contains no reported artifacts.
  • The authors hypothesize that MAE avoids artifacts because training uses only a local patch-token loss rather than global information aggregation.
  • Artifacts appear across all attention heads in the DINOv2-L model trained without registers, although some heads emphasize them more than others.
  • Random selection of patch tokens introduces variability, so the reported classification scores include standard deviations for this choice.

H QUALITATIVE RESULTS

Register tokens produce cleaner attention and feature maps across DeiT-III, OpenCLIP, and DINOv2, while removing visible norm outliers from patch tokens.

  • Register tokens provide cleaner attention maps with considerably fewer artifacts across DeiT-III, OpenCLIP, and DINOv2.The comparison uses models trained with and without register tokens.
  • Register tokens visibly improve the feature maps produced by the evaluated models.
  • Artifact patches correspond to token-norm outliers in all three evaluated models.
  • Training with registers removes large-norm tokens from the model outputs.
Loading 2309.16588v2…