Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

• Prior work introduced dedicated register tokens to reduce outlier artifacts and improve spatial regularity and dense-prediction performance.Source: §1 · p. 1
• The study addresses open questions about what register tokens encode, how they compare with outlier tokens, and whether their semantic and computational roles differ.Source: §1 · p. 1
• The authors train SAEs on targeted DINOv2 token subsets and interpret features with top-activating examples, VLM descriptions, UMAP, CLIP checks, and latent ablations.Source: §1 · p. 1
• Register features align more with high-level semantics, while outlier features cluster around structural and texture-dominant patterns; register-feature ablations disrupt representation stability more.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The paper frames outlier tokens as complementary to register tokens rather than redundant, associating them with lower-level structural and texture-heavy information.Source: §1 · pp. 1–2

• DINO and DINOv2 learn semantic and spatial structure without explicit supervision, but high-norm artifacts have been observed in Huge and Giant DINOv2 models (Darcet et al., 2023).Source: §2 · p. 2
• High-norm outliers are a small subset of patch tokens, about 2%, often found in background regions; registers add capacity for global computation and reduce the need for patch-token outliers.Register tokens were shown to restore spatial regularity and improve dense downstream performance (Darcet et al. (2023)).Source: §2 · p. 2
• Prior work has not established the semantic content of outliers or their relationship to register-based computation; one account links outliers to a softmax bottleneck (Jiang et al., 2025).Source: §2 · p. 2
• Sparse autoencoders decompose dense activations into sparse latent features, and vision studies have identified concepts from textures to objects; this work applies SAEs to register and outlier tokens.Source: §2 · p. 2
• Figure 2 contrasts attention maps with PCA-based semantic maps after ablating the top five activating register features, showing disrupted object focus and semantic coherence.Source: Figure 2 · p. 3
• Automated visual-feature explanations use VLMs, including a steering-based feature-injection approach (Ferrando et al. (2026)); this paper instead examines register and outlier token roles in DINOv2.Source: §2 · p. 3

3. Methodology

• The study partitions layer-8 activations into register, outlier, and standard patch groups and trains separate SAEs using standard and register-augmented DINOv2 Small checkpoints.Activations come from ImageNet-1k; outliers are the top 2.37% of layer-8 patch tokens by L2 norm.Source: §3 · p. 3
• Activations are scaled by a global subset-specific scalar to unit expected squared norm, and the SAE loss combines reconstruction with an L1 sparsity penalty.Source: §3 · p. 3
• SAEs encode activations into sparse latents and decode them into reconstructions; feature quality is assessed with L0 sparsity and fraction of variance explained.Source: §3 · pp. 3–4

3.1. Automated Interpretability

• For feature interpretation, the authors caption the top nine activating patches per feature, examine UMAP organization, and cross-check semantic alignment in CLIP space.Source: §3.1 · p. 4

3.2. Causal Intervention

• For each image, the intervention zeros the five most active SAE features and decodes the altered latent representation.Source: §3.2 · p. 4
• Intervention effects are measured through PCA comparisons of final-layer patch representations and attention maps averaged across heads to assess focus on the primary semantic object.Source: §3.2 · p. 4

4.1. SAE Training

• Register full achieves the highest explained variance (0.809), Base full the highest cosine similarity, and specialized-token reconstructions are less aligned (≈0.63).Outlier only and Base full use original DINOv2 activations; Register only and Register full use register-augmented DINOv2-small activations.Source: §4.1 · pp. 4–5
• All SAE models retain nearly all features as alive, with no substantial dead-neuron problem.This qualification accompanies the reconstruction comparisons.Source: §4.1 · p. 5
• Across SAE regimes, most latents rarely fire, while Outlier only and Register only show slightly denser activations than full-token models.The authors describe this rarity as intended behavior for sparse, interpretable models.Source: §4.1 · p. 5

4.2. Clustering Activations

• Register features align with higher-level semantic content, while outlier features cluster around structural, background, and texture information, consistent with the texture-sink hypothesis.The CLIP-space cross-check agrees with the original SAE-VLM labels.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3. Causal Hierarchy

• Ablating the top five activating register features reduces representation stability by 48.17%; outlier-derived ablations have a negligible effect, about 150 times smaller.Attention maps also show degraded localization of the primary object after register-feature ablation.Source: §4.3 · p. 5, Table 1 · p. 5
• Ablating the bottom five register features also drops similarity by 47.05%; mean-ablating a second register causes an abrupt drop to ≈0.50 after ≈0.01 for one.With four default registers, the authors suggest roughly a one-register safety margin before texture-sink suppression fails and outlier-like signals reappear.Source: §4.3 · p. 5
• Across diverse images, top-activating latents converge on a fixed five-index Global Register Signature, suggesting a universal stabilization or noise-sequestration role.The authors attribute similar top-five and bottom-five ablation effects to exceeding register sequestration capacity rather than to the semantic identity of removed features.Source: §4.3 · pp. 5–6

5. Conclusion

• Register-derived features are more strongly associated with high-level semantic concepts, whereas outlier-derived features are more often associated with texture-dominant background structure.The authors suggest outliers may reflect a structured representational role tied to nonobject-centered visual information.Source: §5 · p. 6
• Ablating top register-derived features substantially disrupts representation stability and downstream attention maps, while outlier-derived feature ablations have a much smaller effect under the same protocol.The interventions support distinct functional roles, but do not identify the routing circuit or when routing emerges during training.Source: §5 · p. 6, §5.1 · p. 6
• The findings suggest DINOv2 variants separate semantic and structural information across distinct representational mechanisms, and provide a foundation for studying interpretable token-level computation.The authors frame this as a foundation for analyzing, regularizing, and designing future ViT architectures.Source: §5 · p. 6

5.1. Limitations

• Semantic and causal analyses are limited to layer 8 of DINOv2 Small; checkpoint availability across other scales and layers does not establish generalization.The authors also note that finite SAE compute budgets and qualitative interpretability tools limit the strength of mechanistic conclusions.Source: §5.1 · p. 6
• The interventions do not identify the circuit routing information into register or high-norm patch tokens, or when that routing emerges during training.Thus, the reported evidence distinguishes token roles without resolving their circuit-level origins.Source: §5.1 · p. 6

5.2. Future Work

• Future work will repeat semantic labeling, texture-versus-object scoring, and matched causal ablations across DINOv2 Small, Base, and Large at layers 4, 8, and 11.The planned comparisons include bootstrap confidence intervals and tests of whether scale or depth changes the register–outlier gap.Source: §5.2 · pp. 6–7
• A proposed circuit study will use texture perturbations, matched controls, and activation patching to test candidate paths from patches through attention heads or MLPs to registers or outliers.The plan includes testing the necessity and sufficiency of the highest-scoring subgraph by ablating and restoring its edges.Source: §5.2 · p. 7
• The authors also plan to track token norms, SAE features, semantic scores, and causal mediation across chronological teacher checkpoints to study when the behavior emerges during pretraining.They propose holding images, token-selection percentiles, and evaluation code fixed, and using a change-point model within a single training trajectory.Source: §5.2 · p. 7