Towards Understanding the Robustness of Sparse Autoencoders

Ahson Saiyed; Sabrina Sadiekh; Chirag Agarwal

Source-linked AI summary

Towards Understanding the Robustness of Sparse Autoencoders

Ahson Saiyed, Sabrina Sadiekh, Chirag Agarwal

arXiv:2604.18756v1Apr 20, 2026cs.LGcs.AIcs.CLcs.CR

TL;DR

LLMs remain vulnerable to optimization-based jailbreaks, while the robustness implications of SAE representations are underexplored. The paper inserts pretrained SAEs into residual streams at inference time without changing weights or blocking gradients, and reports lower jailbreak success, reduced transferability, and configuration-dependent robustness.

Problem
The paper asks how SAE-based representations affect LLM robustness to jailbreak attacks that exploit internal gradient structure.
Method
The study inserts pretrained SAEs into transformer residual streams at inference time across four model families and evaluates white-box, black-box, transferability, sparsity, and layer-placement behavior.
Results
SAE routing consistently reduces jailbreak success rates and cross-model adversarial-suffix transferability across the evaluated settings.
Takeaways & Limitations
Sparse projection can function as a lightweight representation-level robustness intervention, consistent with a bottleneck that constrains adversarial optimization geometry.
Takeaways & Limitations
In black-box evaluation, robustness improvements depend on the detector: reductions are observed under HarmBench but are smaller under other detectors.

Abstract

from arXiv · show

Large Language Models (LLMs) remain vulnerable to optimization-based jailbreak attacks that exploit internal gradient structure. While Sparse Autoencoders (SAEs) are widely used for interpretability, their robustness implications remain underexplored. We present a study of integrating pretrained SAEs into transformer residual streams at inference time, without modifying model weights or blocking gradients. Across four model families (Gemma, LLaMA, Mistral, Qwen) and two strong white-box attacks (GCG, BEAST) plus three black-box benchmarks, SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline and reduce cross-model attack transferability. Parametric ablations reveal (i) a monotonic dose-response relationship between L0 sparsity and attack success rate, and (ii) a layer-dependent defense-utility tradeoff, where intermediate layers balance robustness and clean performance. These findings are consistent with a representational bottleneck hypothesis: sparse projection reshapes the optimization geometry exploited by jailbreak attacks.

1 Introduction

The paper investigates whether sparse autoencoders can make LLM representations more robust to optimization-based jailbreaks. It studies inference-time SAE insertion across models and attacks, finding lower jailbreak success and reduced transferability.

Motivation: LLMs remain vulnerable to jailbreaks that exploit gradient-aligned directions in intermediate activations.The paper frames adversarial success as closely coupled to the geometry of internal representations.
Motivation: SAEs impose a structured bottleneck by reparameterizing dense residual-stream activations into sparse, feature-aligned components.The robustness implications of these representations remain unclear, especially given reported sensitivity to perturbations and reconstruction-induced behavior shifts.
Approach: The study inserts pretrained SAEs into transformer layers at inference time without modifying weights or blocking gradients.This preserves end-to-end differentiability and isolates the representational effect of sparse projection on adversarial optimization.
Scope: The evaluation spans Gemma, LLaMA, Mistral, and Qwen models, with additional layer-placement analysis using Qwen2.5.The supplied introduction passage names three model families for the main study and Qwen2.5 for controlled depth analysis.
Findings: SAE-augmented models reduce jailbreak success rates by up to 5× and systematically reduce adversarial-suffix transferability.The study evaluates GCG, BEAST, and black-box jailbreak benchmarks.

2 Related works

The related work connects gradient-based jailbreak attacks, sparse-autoencoder interpretability, and existing LLM defenses. The paper’s gap is an analysis of how sparse residual-stream bottlenecks affect optimization dynamics and transferability.

Adversarial Jailbreak Attacks: Recent jailbreak methods target alignment objectives, with GCG optimizing adversarial suffixes using gradients and BEAST using beam-search heuristics.The related-work discussion places these methods among token-level, paraphrasing, prompt-injection, and indirect-exploit attacks.
Sparse Autoencoders for Interpretability: SAEs decompose dense transformer activations into sparse latent features that support feature discovery, attribution, and interventions.Prior work also reports limitations, including sensitivity of SAE latents to small input perturbations.
LLM Defense: Existing jailbreak defenses include stochastic smoothing, input-level noise, decoding interventions, finetuning, and adversarial training.These approaches involve different robustness–utility tradeoffs and intervention points.
Research Gap: The paper addresses limited analysis of how sparse encode–decode bottlenecks alter adversarial optimization dynamics and cross-model transferability.It contrasts this focus with prior evaluations centered primarily on aggregate ASR and utility trade-offs.

3 Method

The method routes transformer residual activations through pretrained sparse autoencoders during inference while preserving model weights and gradient access. It evaluates this intervention across models, attacks, and black-box settings.

SAE-Augmented Models: The intervention inserts pretrained SAEs into intermediate transformer layers as sparse encode–decode operators during inference.The evaluation covers Gemma-2, LLaMA-3, Mistral, and Qwen2.5 models ranging from 2B to 70B parameters.
SAE-Augmented Models: At layer ℓ, the SAE replaces residual activation h_ℓ with reconstructed activation ĥ_ℓ, which subsequent transformer blocks process.The encoder and decoder are defined as z = E(h) and ĥ = D(z).
SAE-Augmented Models: The latent representation is sparse and typically uses a 16× expansion, with offline training combining reconstruction error minimization and an ℓ1 sparsity penalty.This produces a sparse feature basis that approximately reconstructs the original residual activation.
Attack Interface: Gradients propagate through the differentiable SAE, so GCG optimizes suffixes through the SAE-modified representation space.BEAST instead uses beam search and evaluates candidate suffixes directly on the SAE-augmented model.
Evaluation: The experiments use single intermediate-layer SAEs for Gemma, LLaMA, and Mistral, plus multi-layer Qwen2.5 experiments for insertion-depth analysis.The threat model assumes adversaries append suffixes to prompts to induce harmful outputs; evaluation includes white-box and query-only attacks.

4 Experiments

The experiments evaluate effectiveness, transferability, configuration dependence, and optimization dynamics across multiple models and attack protocols. They combine ASR measurements with sparse-feature and gradient-spectrum analyses.

Research Questions: The study addresses four questions covering attack effectiveness, transferability, SAE sparsity and depth, and adversarial optimization dynamics.These questions organize the experimental evaluation.
Model Suite: The model suite spans six Gemma-2, LLaMA-3, and Mistral models from 2B to 70B parameters, with controlled depth experiments on Gemma-2-9B and Qwen2.5.Single intermediate-layer SAEs are used for the main suite, while multi-layer suites support placement analysis.
Attack Protocol: GCG uses 500 gradient steps with 20-token suffixes, while BEAST uses beam-search parameters k1=k2=15 and search depth L=20.The protocols evaluate PROMPT, BASE, and SAE configurations.
Measurements: The evaluation combines adaptive white-box attacks, three query-only jailbreak datasets, ASR, and mechanistic measures of sparse-feature and gradient behavior.ASR is the fraction of prompts producing harmful outputs, assessed with three independent evaluators.
Results: Under BASE transfer evaluation, median ASR decreases from 55.0% to 19.05% for GCG and from 19.35% to 9.7% for BEAST.Figure 1 reports standard errors over approximately 218 HarmBench prompts per model–condition.

5 Results

SAE insertion reduces jailbreak success and cross-model transfer across gradient-based, non-gradient, and black-box evaluations. Robustness depends on sparsity and insertion depth, while sparse routing changes adversarial optimization geometry.

Median GCG ASR decreased from 55.0% for baseline models to 19.05% for SAE-augmented models across six evaluated models.The reduction was statistically significant (p = 0.0087).
Median BEAST ASR decreased from 19.35% for baseline models to 9.7% for SAE-augmented models.The reduction was statistically significant (p = 0.0411).
SAE-augmented models consistently achieved lower ASR than baseline models across three black-box benchmarks, although improvement varied by evaluator and prompt distribution.
Transferability: Under GCG, median same-configuration transfer fell from 8.75% for BASE →BASE to 2.65% for SAE →SAE.Cross-configuration transfer was asymmetric: BASE →SAE was 7.50%, versus 12.65% for SAE →BASE.
Transferability: Under BEAST, median same-configuration transfer fell from 15.25% for BASE →BASE to 5.35% for SAE →SAE.Cross-configuration transfer remained asymmetric, with BASE →SAE at 8.20% and SAE →BASE at 11.90%.
Parametric dependence: For Gemma-2 9B, BASE →SAE ASR increased from 0.9% at L0 = 11 to 21.2% at L0 = 310 as sparsity decreased.LLaMA-3.1-8B-Instruct showed the same monotonic pattern when active-feature count k increased.
Parametric dependence: Early-to-mid insertion layers provided the strongest transfer suppression, reaching 0.9% ASR at Gemma Layer 5 and 0.0% at LLaMA Layer 7.Intermediate layers offered a better robustness–clean-behavior tradeoff, while late layers weakened the defense.
Mechanism: Adversarial suffixes had Jaccard feature overlap of 0.36 ± 0.16, versus 0.09 ± 0.05 for random baselines, a 4× difference.The results indicate convergence on a shared sparse feature subspace, with higher overlap within than across model families.

6 Discussion

SAE routing improves robustness by disrupting adversarial optimization while preserving gradients, but its benefits depend on sparsity, layer placement, attack direction, and evaluation protocol.

Optimization dynamics: SAE integration increases GCG optimization difficulty by 47–74% across model families.The result covers n = 218 attack runs and uses higher final loss as an indicator of less effective suffix generation.
Robustness and transfer: SAE-augmented models reduce jailbreak success rates and cross-model transfer under strong white-box attacks despite full gradient access.This supports a representation-space change rather than gradient masking as the source of robustness.
Configuration trade-offs: Increasing sparsity through lower L0 produces stronger compression and lower transfer ASR.The ablation reports a clear dose–response relationship between SAE sparsity and robustness.
Configuration trade-offs: Intermediate insertion layers balance robustness and clean performance, whereas early layers are stronger defenses and late layers are largely ineffective.The defense–utility tradeoff is reported consistently across Gemma, LLaMA, and Qwen.
Optimization budget: Additional optimization can improve reverse transfer against SAE models, but the defense against base-optimized suffixes remains stable or strengthens.For example, Base→SAE ASR changes from 10.0%→4.0% for Gemma and 34.0%→34.0% for LLaMA.
Evaluation scope: Black-box robustness is detector-dependent: improvements are observed under HarmBench, while effects under other detectors are smaller.Thus, conclusions depend partly on how harmful behavior is defined and measured.

7 Conclusion

The study finds that inserting pretrained SAEs into residual streams improves jailbreak robustness and reduces transferability without retraining or gradient blocking. The proposed explanation is that sparse projection creates a representational bottleneck that reshapes adversarial optimization.

Conclusion: Pretrained SAE routing consistently reduces jailbreak success rates and cross-model transferability across multiple model families and scales.The intervention operates in transformer residual streams at inference time.
Conclusion: Sparse projection alters adversarial optimization geometry and concentrates gradient signal into fewer directions.The paper presents these mechanistic analyses as support for a representational bottleneck explanation.
Conclusion: SAEs developed for mechanistic interpretability can also function as lightweight robustness interventions without retraining or gradient blocking.This conclusion follows from the reported activation-space intervention and preserved optimization access.

A Experimental Setup

The experimental setup specifies SAE configurations, controlled layer and sparsity ablations, and refusal-detection categories used to evaluate attack success.

Experimental Setup: SAE configurations record layer placement, model dimension, and dictionary width for each base model.These parameters define the configurations used in the experiments.
Experimental Setup: Layer-placement ablations vary insertion depth at fixed sparsity, while sparsity ablations vary L0 or TopK k at a fixed layer.LLaMA-3.1-8B uses matched-training layer and sparsity variants from the andyrdt SAE suite.
Experimental Setup: Refusal-detection pattern categories are documented in Table 10, with complete regex patterns available in supplementary code.These categories support refusal-based attack-success measurement.

B Additional ASR Results

Additional results show detector-dependent black-box effects and provide a complete GCG ASR accounting that distinguishes baseline, SAE, absolute, and relative changes.

Black-box results: Under HarmBench, SAE reduces mean ASR from 0.232 →0.160 and median ASR from 0.205 →0.116.Under WildGuard, reductions are more modest, while refusal-based ASR remains extremely high.
Black-box results: WildGuard reductions are smaller, with mean ASR changing from 0.448 →0.414 and median ASR from 0.440 →0.405.The passage states these changes fall within overlapping confidence intervals.
Black-box results: Refusal-based heuristic ASR remains approximately 96–97% mean and approximately 99% median, with no meaningful difference.This demonstrates that black-box outcomes vary substantially by detector.
GCG ASR results: Table 11 reports GCG ASR for baseline and SAE models alongside absolute percentage-point and relative changes.Negative absolute changes indicate reduced ASR, while positive changes indicate increased ASR.

C Cross-family validation (Qwen2.5-7B)

Evaluation on Qwen2.5-7B-Instruct extends the study beyond the primary model families using a fixed, publicly available multi-layer SAE suite. SAE insertion reduced baseline-to-SAE adversarial transfer, with robustness varying by layer placement.

Cross-family validation (Qwen2.5-7B): The Qwen2.5-7B-Instruct evaluation uses a publicly available multi-layer SAE suite for a fixed checkpoint.This setup enables controlled variation of insertion depth without modifying model weights or attack procedures.
Cross-family validation (Qwen2.5-7B): Qwen2.5-7B-Instruct exhibits asymmetric cross-configuration transfer under SAE insertion.The supplied passage reports this asymmetry as consistent with observations in Gemma and LLaMA.

D Comparison with other protections

The paper contrasts SAE integration with defenses centered on input perturbation, auxiliary models, decoding, or adversarial training. It argues that the sparse bottleneck preserves gradients while changing the representation and optimization geometry used by jailbreak attacks.

Comparison with other protections: GCG and BEAST exploit smooth internal optimization landscapes to construct adversarial suffixes.The motivation is to disrupt attack feasibility rather than only detect anomalous prompts.
Comparison with other protections: Randomized-smoothing defenses can increase computational cost and require securing auxiliary models for paraphrasing or response judging.These methods include stochastic input perturbations and auxiliary LLMs.
Comparison with other protections: Noise-based defenses report ASR–utility trade-offs as noise scale changes, but related protections often leave optimization dynamics and cross-model transferability uncharacterized.The comparison includes noise-based, decoding-time, and adversarial-training approaches.
Comparison with other protections: SAE integration inserts a sparse encode–decode bottleneck into the residual stream while preserving full gradient flow.The intervention reparameterizes the internal representation space rather than blocking gradients or modifying model weights.
Comparison with other protections: Up to 5× reduction under GCG accompanies poorer baseline-to-SAE suffix transfer and measurable changes in optimization dynamics.The reported effects include reduced attack success rates and pronounced transfer asymmetry.
Comparison with other protections: Gradient updates show increased successive-step alignment, reduced effective rank, and larger spectral gaps after SAE integration.These changes indicate concentration into fewer dominant directions and reduced exploratory capacity.
Comparison with other protections: Layer placement strongly affects transfer robustness: Layer 7 reaches 0.0% BASE→SAE ASR, whereas later layers are weaker.This result is reported for Qwen2.5-7B-Instruct with a 131K-width SAE.