"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers

1 Introduction

• The paper addresses the limited understanding of which concepts ViTs encode and how those concepts interact across layers, despite their broad generalization.This gap matters because internal structure is presented as useful for auditing, shortcut diagnosis, and targeted interventions.Source: §1 · pp. 1–2
• Existing saliency and feature-importance methods emphasize relevant input regions or features, while SAE-based circuit methods primarily capture input-dependent behavior rather than general behavior.The introduction also notes that dataset-scale attribution for a global view is expensive and may not be fully input-invariant.Source: §1 · p. 2
• CLTs provide global, input-invariant concept interactions from learned weights and instance circuits that reveal concepts and pathways used for a specific prediction.The introduction states that CLT-based circuits are more faithful than SAE-based circuits.Source: §1 · p. 2
• Global concept circuits can identify shortcut dependencies, while instance circuits localize concepts and interactions for interventions that reduce reliance on spurious cues.The authors state that their discovery approach does not require domain knowledge.Source: §1 · p. 2
• 11.0%: the method outperforms existing counterparts on the Waterbird dataset in spurious-correlation removal.The introduction also proposes comparing global circuits across foundation models to examine how supervision paradigms and datasets influence model behavior.Source: §1 · p. 3
• Global circuits can be compared across pretrained models such as CLIP, DINO, and ViT to examine how supervision paradigms and datasets influence model behavior.The authors present this comparison as a principled basis for model optimization and selection.Source: §1 · p. 3

2.1 Sparse Autoencoders (SAEs)

• Reading circuits with SAEs requires training multiple single-layer dictionaries and integrating circuit-discovery methods to quantify feature interactions.This approach mainly captures input-dependent interactions and requires computationally expensive dataset-scale attribution for a global view.Source: §2.1 · pp. 3–4

2.2 Cross-Layer Transcoders (CLTs)

• CLTs encode each layer’s MLP input into sparse hidden activations and reconstruct its output from activations at current and previous layers.Their weights are jointly optimized to minimize reconstruction error across layers with a sparsity penalty, encouraging disentangled, interpretable concepts.Source: §2.2 · p. 4
• Input-dependent sparse codes support fine-grained attribution for instance circuits, while fixed learned weights expose input-independent global concept interactions.The encoder and decoder weights remain unchanged across inputs once the CLT is trained.Source: §2.2 · p. 4

3.1 Adapting CLTs to ViTs

• For ViTs, CLTs include CLS tokens with image tokens, aggregate features across token positions to simplify circuits, and choose sparsity 35 and dictionary size 6144 based on a metric tradeoff.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Recovering “world knowledge” from global concept circuits

• Global circuits derive from CLT encoder and decoder weights: each edge sums linear paths between features, then co-activation reweighting reduces interference from rarely co-active features.Source: §3.2 · p. 6
• Mean-ablating top-10 global-circuit concepts drops CLIP mean accuracy from 0.60 to 0.22; reweighted edges have more impact than raw weights, supporting concept relations beyond statistical artifacts.Source: Table 1 · p. 7, §3.2 · pp. 7–8
• Class subgraphs backtrack from top-activating target-class features through strongly connected upstream features; global circuits also support shortcut detection by comparing edge strength with co-activation without human annotation.Source: §3.2 · pp. 6–7

3.3 Revealing decision making process via instance concept circuits

• Instance circuits retain active, influential concepts for one image; attribution patching, token-position aggregation, and pruning by logit contribution and edge influence yield compact prediction pathways.Source: §3.3 · pp. 8–9
• CLT instance circuits consistently outperform SAE across CLIP, DINO, and supervised ViT; on CLIP they surpass SAE by 24.4% in faithfulness and 22.1% in completeness.Source: Figure 5 · p. 9, §3.3 · pp. 9–10

4.1 Automatic spurious correlation discovery

• Global edge weights measure model reliance, while feature co-activation measures class co-occurrence; low co-activation with high global weights flags spurious concepts.The target is first identified as the concept with top influence on the class logit; co-activation is measured across target-class images from ImageNet, LVIS, and Visual Genome.Source: §4.1 · pp. 10–11
• The method identifies “graffiti” for “Freight Car” and “bird feeder” for “Hummingbird,” alongside causal and non-discriminative concepts.The discovered spurious concepts align with those reported by previous work.Source: §4.1 · p. 11

4.2 Spurious correlation removal

• 23% improvement in worst-group accuracy, versus 5% for SAE; the method also outperforms SpLiCE using automated concept selection.The worst group is waterbirds on land; the comparison is reported in Table 2.Source: Table 2 · p. 11, §4.2 · p. 12

4.3 Model comparison

• CLIP has dense deep-layer interactions, DINO connects shallow and middle layers, and ImageNet-supervised ViT relies more on adjacent-layer interactions.The comparison aggregates global edge weights by layer and examines the “Cat” class subgraphs; the ViT example encodes “cat” by layer 8.Source: §4.3 · pp. 12–13, Figure 7 · p. 13

Interpretable machine learning

• Input-attribution and feature-importance methods highlight evidence but typically offer limited insight into how internal representations compose across layers.The discussion covers saliency, perturbation and occlusion, example-based, local-surrogate, and feature-importance approaches.Source: §5 · pp. 13–14

Concept-based interpretability

• Concept-based methods use human-meaningful concepts, while SAE pipelines decompose activations into model-native features but typically emphasize instance-dependent behavior.Recovering a reliable global picture with SAEs can require expensive dataset-scale attribution and aggregation; this approach instead uses CLTs’ input-invariant weights while supporting instance-level circuit extraction.Source: §5 · p. 14

Mechanistic interpretability

• Mechanistic interpretability reverse-engineers networks through weights, features, and circuits; this work extracts instance and global concept circuits in ViTs using CLTs.The circuits support shortcut auditing and model comparison.Source: §5 · p. 14

6 Conclusion

• The framework provides input-invariant global circuits and prediction-specific instance circuits for reading concept interactions in ViTs.Global circuits summarize parameter-encoded structure; instance circuits reveal active concepts and pathways for a particular prediction.Source: §6 · p. 14
• CLTs primarily capture MLP pathways, attention structure only indirectly; concept interpretation requires human inspection, and global graphs do not fully capture causal relations.Source: §6 · p. 15

• The passage names concept circuits of vision transformers.Source: Supplementary Material for “World… · p. 20

A Details of Cross-layer Transcoders

• JumpReLU retains a layer's pre-activation only when it exceeds the learnable threshold τ; otherwise, it outputs zero.The pre-activation is the input to the activation function.Source: A Details of Cross-layer Transcoders · p. 20
• The CLT objective combines layerwise reconstruction error with sparsity and pre-activation losses.The pre-activation loss is included to prevent dead neurons, and Lspa denotes the sparsity penalty.Source: A Details of Cross-layer Transcoders · p. 20
• The dictionary size is f, and each layer's decoder vector for feature i is formed by concatenating its decoder vectors.The passage also identifies i as the pre-activation of the i-th feature in layer l.Source: A Details of Cross-layer Transcoders · p. 20

B Implementation Details

• CLTs are trained on image and CLS tokens from CLIP-ViT-B/32, CLIP-ViT-B/16, ImageNet-supervised ViT-B/16, and DINO-ViT-B/16.Training uses the ImageNet training set for 250,000,000 tokens with batch size 4096.Source: B Implementation Details · p. 20
• Training uses Adam with learning rate 5×10−5, β1 = 0.9, and β2 = 0.999; the learning rate decays to zero during the final 20% of steps.The sparsity penalty λ is linearly ramped from zero to its final value.Source: B Implementation Details · p. 20
• The implementation sets c = 4, λ2 = 3 × 10−5, and initializes JumpReLU threshold τ to 0.03.λ1 and dictionary size are swept on CLIP-ViT-B/32, and the best hyperparameters are chosen for all models.Source: B Implementation Details · p. 20

C Additional Visualizations

• Layers 0–3 feature visualizations correspond to the progression from color and line concepts.Figures 8 and 9 show layers 0–1 and 2–3, respectively.Source: C Additional Visualizations · p. 21, Figure 8 · p. 22, Figure 9 · p. 23
• Layers 4–7 feature visualizations correspond to shape and part concepts.Figures 10 and 11 show layers 4–5 and 6–7, respectively.Source: C Additional Visualizations · p. 21, Figure 10 · p. 24, Figure 11 · p. 25
• Layers 8–11 feature visualizations correspond to object and abstract concepts.Figures 12 and 13 show layers 8–9 and 10–11, respectively.Source: C Additional Visualizations · p. 21, Figure 12 · p. 26, Figure 13 · p. 27