Source-linked AI summary
Why Does Robustness Reduce Superposition?
Adam Elimadi
TL;DR
The paper asks why adversarial training reduces superposition, extending prior work that established the association without explaining its mechanism. Using toy models, interference analysis, feature counts, and synthetic data with known feature identities, it finds that adversarial training drops non-robust features, leaving fewer features to represent and thereby reducing superposition.
Problem
Prior work connected adversarial vulnerability to superposition and found that adversarial training reduces superposition, but did not explain why training changes the number of represented features.
Method
The paper combines interference geometry and represented-feature counts with synthetic data whose robust and non-robust feature identities are assigned in advance.
Results
Adversarially trained models drop more features, and the dropped features correspond exactly to the non-robust feature set.
Takeaways & Limitations
The findings support a causal chain in which abandoning non-robust features leaves fewer features to encode in the same dimensional space, reducing superposition.
Takeaways & Limitations
The findings are constrained to toy models, and validation on real-world models using tools such as sparse autoencoders remains a next step.
Abstract
from arXiv · showhide
The study of adversarial examples and their origins remains an open area of research. Mechanistic interpretability, and superposition in particular, offers new avenues for approaching this problem. Gorton & Lewis (2025) demonstrate that adversarial examples arise from superposition and show empirically that adversarial training reduces superposition, yet provide no mechanistic account of why this occurs. We present an empirical explanation inspired by the feature taxonomy of Ilyas et al. (2019), tracing the following chain of causalities: adversarial training abandons non-robust features, leading to fewer total features to represent, resulting in less superposition.
1 Introduction
The paper addresses the missing mechanism linking adversarial training to reduced superposition. It argues that adversarial training drops non-robust features, leaving fewer features to represent and reducing superposition.
- The paper explains why adversarial training reduces superposition by connecting feature selection to representational capacity.This addresses the mechanism left unexplained by prior empirical work.
- The study combines indirect evidence from interference geometry and represented feature counts with direct evidence from known feature identities.
- Adversarially trained models consistently drop more features than standardly trained models across the sparsity sweep.
- Dropped features carry higher average interference, while retained features cluster near antipodal interference (≈−1), minimizing cross-feature interference.
- On synthetic data with a known robust/non-robust partition, the dropped features correspond exactly to the non-robust feature set.
2 Background
The paper defines superposition through sparse toy-model representations and characterizes adversarial examples as loss-maximizing perturbations within an ℓ2 constraint. It uses interference geometry to describe how feature directions affect reconstruction.
- Superposition occurs when a model encodes more features than there are neurons, with sparsity controlling feature activity in the toy models.
- A feature x_i is active when x_i exceeds the sparsity threshold S; otherwise it is inactive.
- Interference measures non-orthogonality between feature directions in activation space.
- More positive inner products between feature directions increase the likelihood that spurious activations corrupt feature reconstruction.
- An adversarial example is an imperceptible perturbation that maximizes MSE loss within an ℓ2 ball of radius ε.
- The attack adds small noise before applying a one-step ℓ2 gradient attack to avoid gradient masking.
3 Setup
The paper uses a sparse toy autoencoder to study how feature packing and interference vary with sparsity, while defining metrics for superposition, representational power, and feature retention. Models are compared under standard and adversarial training across selected sparsity levels.
- Toy model: The toy model maps m-dimensional inputs through n hidden neurons and reconstructs them with a ReLU decoder.The setup uses W ∈ R^n×m, h = Wx, and x̂ = ReLU(W⊤h + b).
- Sparsity and superposition: Higher sparsity leads the model to represent more features and exhibit more superposition.
- Metrics: Superposition is measured with a normalized Frobenius-norm quantity Φ that increases with superposition and sparsity.The paper adapts n/∥W∥F by using squared Frobenius norm per dimension.
- Metrics: Representational power is the squared Frobenius norm of W, measuring total energy across feature directions.Because Φ = P/n, the two quantities are proportional when n is fixed.
- Feature measurement: Feature columns are classified as represented or dropped by thresholding their ℓ2 norms at τ = 0.01.
- Interference measurement: Interference is derived from the off-diagonal Gram matrix and summarized per feature by row-wise totals, separately for kept and dropped features.
- Training protocol: The study trains standard and adversarial models across a sparsity sweep using mixed clean and adversarial examples.The adversarial objective balances clean and robust accuracy with α = 0.5; models are trained for 150,000 steps at learning rate 10^-3.
4 Representational Power and Feature Dropping
Adversarial training lowers superposition and representational power by dropping more features than standard training across the sparsity range. The dropped features have more costly interference geometry, while retained features tend toward antipodal directions that reduce cross-feature interference.
- 4.1 Replication: Adversarially trained models consistently exhibit lower superposition than standardly trained models, with the gap widening as sparsity increases.
- 4.2 Representational Power and Feature Dropping: Adversarially trained models drop more features than standardly trained models across all evaluated sparsity levels.
- 4.3 Feature Dropping and Interference: Dropped features exhibit higher interference than retained features, or are less close to the antipodal value −1 when both groups have negative interference.
- 4.3 Feature Dropping and Interference: Retained features cluster near interference −1, corresponding to antipodal feature directions that maximally suppress cross-feature activation.
5 Robust and Non-Robust Feature Recovery
The paper constructs a synthetic dataset with known robust and non-robust feature identities and shows that adversarial training selectively drops the non-robust features. It explains this behavior through their higher adversarial loss contributions and costly interference geometry.
- 5.1 Structured Data Construction: The synthetic dataset partitions features into robust and non-robust sets with high- versus low-amplitude signals, retaining the partition as ground truth.The construction uses ar = 6.0 for robust features and anr = 0.2 for non-robust features.
- 5.2 Results: Adversarial training drops exactly |N| = 70 non-robust features at every sparsity level, while standard training drops none at S ∈ {0.88, 0.90}.This occurs even when the model has enough sparsity to represent all features.
- 5.2 Results: The model-dropped indices correspond exactly to the ground-truth non-robust set N at every sparsity level.The comparison directly evaluates trained-model drops against the partition assigned during data generation.
- 5.3 Theoretical Intuition: Non-robust features are costly because adversarial flips add loss penalties and their positive interference causes spurious cross-feature activations.The loss penalty compounds as more non-robust features are flipped, while retained features cluster near interference −1.
- 5.3 Theoretical Intuition: Adversarial training reduces superposition by dropping more geometrically costly non-robust features, leaving fewer features to encode in the same dimensional space.The proposed explanation is conceptual rather than a formal derivation.
6 Discussion
The discussion concludes that adversarial training reduces superposition by removing non-robust features, while identifying open questions about retained-feature geometry and real-world validation. The findings remain constrained by the toy-model setting.
- 6 Discussion: Adversarially trained models reduce superposition by dropping more features, and the dropped features correspond precisely to non-robust features.This forms the paper’s proposed causal chain from feature selection to reduced superposition.
- 6 Discussion: The discussion leaves open why adversarial training preserves remaining-feature geometry and preferentially aligns features in opposite directions.These questions concern interference values and antipodal superposition.
- 6 Discussion: The findings are constrained by toy models, motivating validation on real-world models with mechanistic tools such as sparse autoencoders.The paper specifically identifies moving beyond controlled toy settings as the next step.
- 6 Discussion: The paper asks whether superposition reduction improves interpretability and whether robust/non-robust feature trade-offs can reduce polysemanticity without sacrificing critical representations.These are proposed directions rather than established results.
A Data Generation Code
The appendix provides the full procedure for generating the structured dataset with a ground-truth robust/non-robust partition.
- A Data Generation Code: The appendix gives the full procedure used to construct the structured robust/non-robust dataset.The listing is identified as structured data generation with a ground-truth partition.