Scale Alone Does not Improve Mechanistic Interpretability in Vision Models

1 Introduction

• Mechanistic interpretability investigates neural networks’ internal information processing, often at the level of individual units as atomic building blocks.The paper connects unit-level analysis to identifying a unit’s sensitivities and potentially building a fine-grained understanding of network circuits.Source: §1 · p. 2
• The motivating hypothesis is that larger models might dedicate more units to learned features, reducing superposition and making unit representations less ambiguous.The paper also notes the possibility that larger-dataset models make more human-like decisions by relying on non-spurious or human-aligned features.Source: §1 · p. 2
• The study finds no practically relevant interpretability differences among the investigated models; neither scale nor other design choices make individual units more interpretable on their own.The authors argue that interpretability must be explicitly improved through new model design choices or training objectives, and release the IMI dataset as a test bed for automated measures.Source: §1 · p. 2

2 Related Work

• Unit-level investigations have historical links to neuroscience, while feature visualization developed from early work into a widely used method for probing unit activations.Feature visualization was first proposed by Erhan et al., developed by later work, and popularized by Olah et al..Source: §2 · p. 2
• Prior experiments found feature visualizations helpful but no more useful than highly activating natural exemplars, and later work showed visualizations can be unreliable or misleading.These experimental evaluations include Borowski et al. [4] and Zimmermann et al.; Geirhos et al. [15] demonstrated that feature visualizations are not guaranteed to be reliable.Source: §2 · p. 3
• Mechanistic interpretability targets internal information processing, unlike post-hoc explanations of specific decisions for end users, and also includes circuits and training dynamics.The broader scope includes circuit analysis and phase changes during training [34].Source: §2 · p. 3
• The paper adopts falsifiable, human-centric evaluation and extends the psychophysical task of Borowski et al. [4] and Zimmermann et al. to arbitrary models.The related-work discussion notes disagreement about definitions and measurement scales, alongside prior work focused on the utility of post-hoc explanations.Source: §2 · p. 3

3.1 Measuring the Mechanistic Interpretability of Many Models

• The study compares nine ImageNet-compatible models across parameter scale, architecture and computational blocks, training data scale, and adversarial robustness.The models range from GoogLeNet at 6.8 million parameters to ConvNeXt-B at 89 million parameters; training data comparisons include ImageNet samples and 400 million LAION samples.Source: §3.1 · p. 3
• For each model, researchers randomly select 84 units by first sampling a layer uniformly, while excluding early convolution layers and restricting ViT units to position-wise feedforward layers.The selected layers include convolution and normalization layers and outputs of skip connection blocks; GoogLeNet sampling focuses on the last layers of inception blocks.Source: §3.1 · p. 3
• In a 2-AFC task, participants identify the more strongly activating query image, aided by 18 natural or synthetic reference images for the target unit.The task begins with the most and least activating ImageNet samples as queries; sampling less extreme activations increases difficulty and probes broader understanding of unit behavior.Source: §3.1 · p. 4, Figure 2 · p. 4
• At least 63 participants per model complete trials for 84 units, with each unit seen by 30 participants; the study records over 130,000 responses across 767 units and nine models.Participants receive practice and feedback, and quality checks exclude inattentive or unreliable workers; the ratio of correct answers is the interpretability score, with random guessing at 0.5.Source: §3.1 · p. 4

3.2 Scaling Feature Visualization to Many Models

• For feature visualizations, the authors tune optimization steps and regularization strength using relative activation and binary search to make visualizations comparable in unit activation.The regularizer controls visual diversity; the procedure addresses hyperparameters previously optimized primarily for GoogLeNet.Source: §3.2 · p. 4
• Figure 3 compares per-unit scores for natural exemplars and synthetic visualizations across nine models and reports little influence from model size or training schemes.The caption also notes that differences in natural-exemplar scores across models are mostly non-significant; no generally accepted ViT feature-visualization method was available, so the authors refrain from using one proposed procedure.Source: Figure 3 · p. 5, §3.2 · p. 5

4.1 Scaling Models Does not Coincide with Improving Interpretability

• Interpretability varies little across nine models and does not improve with model or dataset scale; ConvNeXt and ViT perform worse than GoogLeNet.The study reports little to no difference overall, despite sampling models with different scales and training paradigms.Source: §4.1 · pp. 5–6

4.2 Feature Visualizations are Less Helpful than Exemplars for all Models

• Feature visualizations do not explain unit activations better than natural exemplars across models, although robust ResNet-50 improves over standard ResNet-50 in the synthetic condition.Natural exemplars remain more helpful, and the relationship between unit interpretability across the two methods is only moderately correlated for a few models, with no general trend.Source: §4.2 · p. 6

4.3 Which Layers are More Interpretable?

• For most models, layer position has little correlation with interpretability; Clip ResNet and Clip ViT are outliers with strong, highly significant correlations.The authors conclude that large-scale pretraining might benefit later layers while sacrificing interpretability in early layers.Source: §4.3 · pp. 6–7, Figure 5 · p. 7

4.4 Do our Findings Depend on the Difficulty of the Task?

• Humans perform best in the easy task and worst in the hard task; synthetic feature-visualization performance drops close to chance when difficulty increases only slightly, to medium.Medium, hard, and very hard queries use the 99th, 95th, and 85th activation percentiles, respectively; reference images always come from the activation-distribution tail.Source: §4.4 · pp. 7–8

4.5 IMI - A Dataset to Learn Automated Interpretability Measures

• The authors present IMI as a starting point for developing automated interpretability evaluations that could support direct optimization of model interpretability.They note that human psychophysical evaluations are costly and infeasible for evaluating every unit across entire models.Source: §4.5 · p. 9
• The IMI dataset contains more than 130,000 human responses and query images and explanations for 767 units across nine models.Of the responses, 76,000 passed quality assertions; the dataset is a collection of labels and metadata without fixed features expected to predict interpretability. The authors report no reliable relationship between interpretability and activation centrality or sparseness.Source: §4.5 · p. 9

• The study tested a selected set of models because human psychophysical evaluations are costly; some models, including the largest vision model presented so far [7], were unavailable.The authors consider a drastic change from testing more models unlikely based on current trends.Source: §5 · p. 9
• The approach assumes features are encoded in individual units rather than population codes, and the authors note that other coding schemes are not excluded.The two investigated interpretability methods are only partially correlated, so other explanation methods might yield different conclusions.Source: §5 · p. 10
• There is no clear gold standard for assessing interpretability, and a different experimental paradigm could alter the results.The authors argue that their paradigm closely mirrors practical mechanistic interpretability and encourage further research on interpretability methods.Source: §5 · p. 10
• None of the investigated axes—model size, dataset size, architecture, or training scheme—positively affected vision-model interpretability.The authors conclude that interpretability does not come for free through increased downstream task performance.Source: §5 · p. 10

• Participants complete 2-Alternative-Forced-Choice trials by identifying the positively activating query image for a network unit.Each trial includes two query images and two sets of nine reference images; one reference set is positively activating and the other negatively activating.Source: §A.1 · p. 15
• Reference images are either natural images or synthetic Feature Visualizations, while participants judge which set better fits the positive query.Participants are not told the true semantics of the task and receive at least five hand-picked practice trials.Source: §A.1 · p. 15
• Figure 8 shows the two query images between two nine-image reference blocks, with feedback framing the correct query green and an incorrect one red.Source: Figure 8 · p. 15

A.2 Sampling Images for the Psychophysical Tasks

• For each unit, the image-sampling procedure creates multiple trials from ranked activations, dividing reference-image candidates into nine groups and sampling one from each group.The authors initially create t = 20 trials and use 10, reserving the rest for a potential later experiment.Source: §A.2 · p. 15

A.3 Amazon Mechanical Turk

• Crowd-worker participation is restricted by location, prior task experience, approval rate, and task-quality checks.Workers must pass demo and catch trials, meet time requirements, and avoid selecting the same query position in more than 90% of trials.Source: §A.3 · p. 16
• The study collected 133′310 trials, of which 76′000 passed all quality checks; rejected responses were retained in the IMI dataset.Each model and condition was sampled until 63 unique participants passed the checks.Source: §A.3 · p. 16

A.4 Scaling Feature Visualization to Many Models

• Natural-image exemplars can confound a unit’s relevant feature with co-occurring visual content, while Feature Visualization generates synthetic images through activation-maximizing gradient ascent.A diversity regularizer is used to reduce mode collapse for units sensitive to multiple concepts.Source: §A.4 · p. 16
• Feature Visualization hyperparameters do not generalize well across models and units, so the authors adapt optimization steps and choose a diversity weight separately for each unit.Optimization stops adaptively based on gradient magnitude; the selected diversity weight is the largest tested value that still yields activations at least as strong as natural images.Source: §A.4 · p. 17
• The authors did not evaluate ViTs in the synthetic condition because the available Feature Visualization approach did not produce coherent images without large-scale noise augmentation.That augmentation effectively performs a random search in image space for activating images.Source: §A.4 · p. 17
• Figure 9 compares three randomly selected units per model using maximally and minimally activating visualizations generated without the diversity regularizer.Source: Figure 9 · p. 18

• 86 units were required by an a-priori power analysis targeting a practically relevant 0.1 difference in correct-trial proportions (Cohen’s d = 0.67).The analysis used a Mann-Whitney-U test with α = 0.01 and β = 0.95.Source: §A.5 · p. 19
• 30 independent trials per unit were needed to accept a standard deviation of 0.1 in estimating each unit’s probability of a correct response.Source: §A.5 · p. 19
• 55 trials per participant—40 real, 10 instruction, and 5 catch trials—were chosen to balance estimation needs against focus and frustration concerns.Source: §A.5 · p. 19
• The final design used 84 units and required 63 participants after rounding the calculated unit count down from 86.The calculation used 30 measurements per unit and 40 real trials per participant.Source: §A.5 · p. 19

B.1 Extended Visualizations of Results in Sec. 4

• Interpretability gaps between easy and harder tasks show that well-interpretable units do not necessarily remain interpretable as task difficulty increases.The figure compares easy–medium and easy–hard performance gaps across relative layer position and easy-condition performance for ResNet-50 and Clip ResNet-50.Source: Figure 10 · p. 20
• Synthetic feature-visualization interpretability differences are mostly non-significant across models in the corrected Conover test.Natural-exemplar significance values are shown separately in Fig. 3.Source: Figure 11 · p. 20
• Natural- and synthetic-explanation interpretability are strongly correlated for some models, but this relationship does not hold for all.The comparison uses Spearman rank correlations between proportions correct; asterisks mark significant correlations.Source: Figure 12 · p. 21
• Higher confidence accompanies greater correctness for almost all models with natural exemplars, but for only two models, if at all, with synthetic visualizations.Participants also give higher confidence ratings to natural examples across all models.Source: Figure 13 · p. 22
• Using the 63 units shared across difficulty levels does not change the qualitative finding that performance rapidly declines as tasks become harder.An oversight meant only 63 of the 80 sampled units were identical across difficulty levels.Source: Figure 14 · p. 23

B.2 Analysis of Quality Checks

• The quality-check figure shows distributions of participants passing or failing each of five checks, alongside distributions of the check values and their allowed thresholds.The black and red lines mark minimally required and maximally allowed values, respectively.Source: Figure 15 · p. 24

B.3 Distribution of Activations

• Activations for 15 randomly chosen GoogLeNet units have unimodal distributions without pronounced tail modes.Activations were normalized by each unit’s largest absolute activation to lie between −1 and 1; the figure marks the 5th, 15th, 85th, and 95th percentiles.Source: Figure 16 · p. 25

B.4 Are Activation Patterns in Feature Maps Predictive of a Unit’s Interpretability?

• Local activation-map contrast has no strong relationship with unit interpretability, with Spearman correlations from −0.24 to 0.14.The analysis tests whether spatially concentrated activation might be easier for people to detect than activation spread across an image.Source: §B.4 · p. 25
• Neither pixel-wise nor channel-wise activation sparseness shows a strong relationship with unit interpretability.Pixel-wise sparseness is the fraction of non-positive feature-map values averaged over ImageNet validation images; channel-wise sparseness is the fraction of images where the entire feature map is non-positive.Source: §B.4 · p. 25

C Broader Impacts

• The authors note that interpretability explanations can create false trust, potentially encouraging deployment of models that produce incorrect or undesired real-world results.They also identify potential over-trust in domains better left to human judgment for ethical reasons, such as policing or the justice system.Source: C Broader Impacts · pp. 25–26

D Computational and Financial Cost

• Feature visualizations for 100 randomly selected units used 25 Nvidia 2080Ti GPUs in parallel for about 12 hours across all models except ConvNeXt (about 24 hours on average), across seven models.The electricity requirement was 600 kWh, translating under the stated price and emissions assumptions to about 300 USD and 256 kg CO2e. Total experiment cost, including MTurk fees, was around 12,000 USD.Source: D Computational and Financial Cost · pp. 26–27

E Further Screenshots of Psychophysics Trials

• The trial screenshots show participant consent materials and examples of natural and synthetic task instructions, including feedback after correct and incorrect responses.The instruction examples are drawn from twelve possible trials.Source: Figure 20 · p. 29, Figure 21 · p. 30, Figure 22 · p. 31