PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Redistribution in Sparse Autoencoders

Ziye Mei, Haotian Ma

arXiv:2610.21240v1cs.LGcs.AIcs.CL

TL;DR

Prompt transformations can redistribute SAE activity, yet a feature change on one input may not recur, and recurrence alone does not establish downstream relevance. The paper introduces RSS to select consistent recurring changes and test fixed supports beyond selection. In two frozen safety readouts, small RSS supports retain recurrent-failure ranking utility relative to full erosion.

  • Problem

    Prompt transformations can alter SAE feature responses, but it is unclear which sparse changes recur across related inputs or matter to downstream computations.

    Source: §1 · p. 1

  • Method

    RSS selects large feature responses that recur in a consistent direction, evaluates fixed supports on excluded variants, and decomposes their contributions to a fixed linear readout.

    Source: §1 · pp. 1–2

  • Results

    Median RSS supports of 20 and 19 features yield recurrent failure AUROCs of 0.935 and 0.958, compared with 0.965 and 0.964 using full erosion.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RSS identifies recurring sparse representation changes and quantifies their contribution to a fixed downstream readout without equating representation change with failure.

    Source: §5 · pp. 9–10

    The results concern the tested transformations and frozen linear readouts, with the main functional holdout using new realizations of the same operators.

    Source: §5 · p. 10

Abstract

from arXiv · show

Sparse autoencoders (SAEs) decompose language model activations into sparse features, but prompt transformations designed to preserve the underlying request can redistribute activity across them. A feature change observed on one transformed prompt need not recur, and even a recurring change need not affect a downstream computation. We introduce the redistribution sensitivity score (RSS), which identifies large feature responses that recur in a consistent direction, and evaluate the resulting supports on variants excluded from selection. Across 27 pretrained SAEs spanning eight language models, the tested prompt transformations produce a reproducible redistribution profile on these excluded variants. We then separate recurring representation change from signed readout contribution and decision failure. For a fixed linear readout, an exact decomposition shows how response magnitude, readout coefficients, directional alignment, and the clean margin determine downstream consequences. In two frozen safety readouts, combining RSS support erosion with the full clean margin yields recurrent failure AUROCs of 0.935 and 0.958 using median supports of 20 and 19 features, compared with 0.965 and 0.964 using full erosion. RSS therefore provides a way to identify recurring sparse representation changes and quantify their contribution to a downstream readout without equating representation change with failure. Code is available at https://github.com/OpenSAE/Redistribution-Sensitivity-Score

1 Introduction

Prompt transformations can redistribute SAE activity, but observed feature changes may not recur or matter to downstream computations. The paper introduces RSS to select recurring, directionally consistent changes and evaluates their held-out behavior and readout effects.

1 Introduction

  • RSS addresses the limits of tracking one feature or measuring aggregate activation distance by identifying which sparse-basis changes recur across related inputs.The paper calls changes in activity distribution across the sparse basis representation redistribution.Source: §1 · p. 1
  • Across 27 pretrained SAEs, RSS identifies recurring, directionally consistent feature changes and evaluates fixed supports on variants excluded from selection.The tested transformations produce a reproducible redistribution profile on those excluded variants.Source: §1 · p. 2
  • For a fixed linear readout, an exact decomposition connects recurrent feature changes to readout effects and threshold crossing through response magnitude, readout weight, directional alignment, and clean margin.Recurring representation change is distinct from downstream effect: feature weights may be small, contributions may cancel, and score changes may not cross the decision threshold.Source: §1 · p. 2
  • Median RSS supports of 20 and 19 features yield recurrent failure AUROCs of 0.935 and 0.958, versus 0.965 and 0.964 using full erosion.These results are from two frozen safety readouts.Source: §1 · p. 2

2 Related Work

Prior work examines SAE feature quality and sensitivity to related inputs, as well as prompt variation and behavioral sensitivity. This paper distinguishes its focus on recurring feature redistribution and its consequences for a fixed readout.

2 Related Work

  • Prior SAE studies assess reconstruction, sparsity, downstream use, and changes in feature responses across related inputs.Feature sensitivity has also been measured as activation reliability across semantically related texts.Source: §2 · p. 2
  • Prompt sensitivity studies examine behavior changes from format, ordering, or wording, while this work studies recurring redistribution within the sparse basis under controlled prompt variation.Other cited work studies stable representation directions and SAE responses to optimized perturbations.Source: §2 · pp. 2–3
  • Unlike linear probes or feature interventions, the fixed-readout analysis measures prompt-induced readout changes carried by features selected without readout information.The fitted readout is held fixed.Source: §2 · p. 3

3 Method

The method selects recurring feature changes without readout information, then evaluates their contributions to a fixed linear readout and the margin that remains. It also tests the same support on held-out variants and uses readout counterfactuals to localize changes.

3 Method

  • RSS selects recurrent feature responses without using the readout, then quantifies their contributions to a fixed linear readout and remaining margin.A large change on one variant need not recur, so RSS first forms a magnitude-based candidate pool, applies recurrence and directional-consistency criteria, and retains the largest eligible responses.Source: §3 · p. 3, §3.1 · p. 3

3.1 Redistribution Sensitivity Score

  • For one pretrained SAE, layer, base prompt, and transformation setting, RSS uses average absolute feature changes to form a candidate pool and selects eligible features by response magnitude.Eligibility requires recurrence r_f ≥ ρ and directional consistency c_f ≥ κ; MeanRSS summarizes selected-support response magnitude and is unavailable for an empty support.Source: §3.1 · p. 3

Proposition 1 (Constrained magnitude characterization of RSS)

  • With n = 5 and ρ = κ = 0.8, at least four observations share the dominant direction and at most one opposes it, so changing or removing one observation cannot reverse that direction.The constrained magnitude characterization applies when at least K features are eligible; the selected support and directions are fixed on selection variants and evaluated on disjoint held-out variants.Source: Proposition 1 (Constrained magnitude… · pp. 3–4

3.2 Readout Effects of Redistribution

  • For a fixed linear readout, the method isolates feature-change contributions exactly; equal response magnitudes can have different effects because direction and cancellation differ.The readout uses activations scaled with clean training data, while RSS selection remains based on raw feature changes.Source: §3.2 · p. 4

3.3 Support Fidelity and Residual Margin

  • Residual-margin measures combine average erosion with the clean margin, while Margin Contribution Retention measures the magnitude of selected contributions rather than signed margin fidelity.Signed fidelity also depends on transfer of selected contributions and the net contribution of omitted features.Source: §3.3 · p. 5

3.4 Localization, Readout Counterfactuals, and Selective Review

  • The same support is evaluated on each held-out variant because average contributions do not resolve individual transformed decisions.Restoring selected readout inputs to their clean values yields a counterfactual margin; cases ranked by selection-variant residual margin are evaluated against held-out errors.Source: §3.4 · p. 5

4 Experiments and Results

Across the tested transformations, RSS-selected features preserve redistribution patterns on excluded variants, and their readout effects depend on signed contribution and remaining margin. Compact RSS supports also retain offline audit utility, but do not replace the full clean computation.

4.1 Systematic and Persistent Redistribution Across Pretrained SAEs

  • Across 27 SAEs, normalized context and wrapper MeanRSS exceeded local edits by 0.874 (95% interval [0.867, 0.881]); RSS gave stronger held-out separation than comparison methods in 24–26 SAEs.The study used 1,000 prompts, and the transformation profile also remained stable under disjoint realization and operator holdouts.Source: §4.1 · pp. 5–6, Figure 2 · p. 6, Figure 3 · p. 7

4.2 Feature Persistence and Functional Relevance in Frozen Readouts

  • In Gemma and Llama, RSS supports preserve held-out response directions and substantial readout contribution, although readout-aware selection retains more contribution.Response rates are 0.876 versus 0.816 for Gemma and 0.871 versus 0.794 for Llama. At K = 20, RSS retains 43.0% and 56.2% of absolute held-out contribution, versus over 90% for readout-aware ranking.Source: §4.2 · pp. 6–7, Figure 4 · p. 8

4.3 Frozen Readout Failure Depends on Alignment and Remaining Margin

  • Subtracting RSS support erosion from the full clean margin yields recurrent failure AUROCs of 0.935 for Gemma and 0.958 for Llama, versus 0.965 and 0.964 using full erosion.These are offline rankings; erosion is computed on selection variants and recurrent failure on held-out variants. Failures concentrate at low full residual margin.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4 Compact Supports Preserve Readout Change and Audit Utility

  • At a 20% offline review budget, RSS residual margins capture 89.2% of Gemma errors and 92.8% of Llama errors, including 87.3% and 94.4% of harmful false negatives.The median supports contain 20 and 19 features, versus 415 and 862 nonzero coefficients in the full readouts; the compact object is not a replacement probe.Source: §4.4 · p. 9

5 Discussion and Conclusion

RSS treats recurring redistribution as a distinct property of sparse representations and connects it to downstream readout effects without equating representation change with decision failure. The discussion emphasizes that recurring changes selected without readout information explain held-out readout variation, within the tested setting of frozen linear readouts.

5 Discussion and Conclusion

  • Recurring redistribution is a distinct property of sparse representations, with structured patterns observable beyond the variants used to select supports.The discussion contrasts feature-level recurrence with tracking one feature or measuring aggregate representation distance.Source: §5 · p. 9
  • Readout effects depend on response magnitude, readout weight, directional alignment, and clean margin, so large changes need not cause failure while smaller aligned changes can matter near a decision boundary.Figure 6 evaluates reconstructed held-out logit changes, restored errors, and errors captured at a 20% offline review budget.Source: §5 · pp. 9–10, Figure 6 · p. 10
  • Supports selected without readout information explain substantial held-out readout variation, with downstream evaluation limited to frozen linear readouts and new realizations of the same operators.The paper presents this as a bridge from recurring sparse changes to functional consequences within a fixed computation.Source: §5 · p. 10

Supplementary Material

The supplementary material collects notation, proofs, and extended representation and readout analyses. Its mathematical sections establish selector and readout properties, while population response bounds are explicitly not a guarantee for directions selected from finite data.

Supplementary Material

  • The supplementary material provides proofs, notation, complete protocols, and extended representation and readout results.Table S1 summarizes notation defined for one pretrained SAE, layer, base prompt, and transformation setting unless stated otherwise.Source: Supplementary Material · p. 14, Table S1 · p. 14

S1 Mathematical Properties and Proofs

  • The mathematical section proves selector properties and a readout-contribution factorization, then separates transfer error from omitted contributions and derives sufficient conditions for preserving decisions and average-margin rankings.Protocol parameters are listed in Supplementary Table S2.Source: S1 Mathematical Properties and Proofs · p. 14

Proof of Proposition 1

  • RSS maximizes the sum of h_f among k-feature subsets of E_M by retaining its k members with the largest h_f.When the candidate pool contains at least K eligible features, features outside the pool cannot improve the constrained top-K objective; cutoff ties can yield additional maximizers.Source: Proof of Proposition 1 · p. 14

Proof of Proposition 2

  • For n = 5 and ρ = κ = 0.8, at least four responses share the dominant direction, and changing or deleting one observation cannot reverse it.The proof uses r_f ≥ ρ and c_f ≥ κ > 1/2 to establish Equation 7.Source: Proof of Proposition 2 · p. 15

Population Response Statistics

  • The population bound |µ_f| ≤ E|δ_f| ≤ |µ_f| + √v_f relates mean response, absolute response magnitude, and variation, but gives no population guarantee for a finite-sample-selected direction.These bounds do not enter Proposition 2.Source: Population Response Statistics · p. 15

Proof of Proposition 3

  • For a nonzero contribution, its sign differs from sign(−y w_f s_f) only through the sign of η_U_f; the zero-response case follows from the definitions.The passage states this as the final step in the proof of Proposition 3.Source: Proof of Proposition 3 · p. 15

Held Out Support Fidelity and Preservation

Held-out support fidelity separates transfer of the selected support from signed contributions omitted from it. The resulting conditions address decision-sign preservation and average-margin ranking, not preservation of every individual decision.

Held Out Support Fidelity and Preservation

  • Held-out residual-margin error is separated into selected-support transfer and signed contribution omitted from the support.The analysis makes cancellation among omitted features explicit.Source: Held Out Support Fidelity and… · p. 16
  • MCR controls omitted absolute contribution, while signed fidelity also depends on cancellation; setting the cancellation factor to one gives a worst-case bound without cancellation.Cancellation-factor values near one indicate little cancellation, while values near zero indicate strong cancellation.Source: Held Out Support Fidelity and… · pp. 16–17
  • Under Equation 30, the omitted signed contribution is too small to cross zero, so the selected-support and full margins have the same sign.This is a sufficient condition for preserving an individual held-out decision.Source: Held Out Support Fidelity and… · p. 17
  • Equation 32 preserves ordering between average held-out margins when one case’s lower bound exceeds the other’s upper bound, but does not preserve counts of individual negative-margin variants.The condition compares possible transfer and omission errors for each case.Source: Held Out Support Fidelity and… · p. 17

Dilution Across the Full Dictionary

Averaging across the full dictionary can dilute changes concentrated in a small responsive support. The readout analysis then separates those feature responses from their signed contribution and the remaining decision margin.

Dilution Across the Full Dictionary

  • When k ≪ F features change with average magnitude a, the full-dictionary mean is approximately (k/F)a, while the responsive-support mean preserves scale a.Thus, averaging across the full dictionary reduces the response by a factor of k/F.Source: Dilution Across the Full Dictionary · p. 17

Exact Readout Decomposition

  • Replacing selected transformed activations with their clean values reverses a transformed probe error exactly when the contribution outside the support leaves positive signed margin.The language model itself is unchanged by this readout counterfactual.Source: Exact Readout Decomposition · p. 18

S2 Definitions of Plotted Quantities

  • The plotted quantities separate representation profiles, feature contributions, and readout outcomes, with available-case averages defined within each SAE before aggregation.Across the six settings, each SAE’s normalized MeanRSS values have mean one.Source: S2 Definitions of Plotted Quantities · p. 18

S3 Frozen Protocols and Reproducibility

The frozen protocols fix prompt transformations and target alignment before SAE analysis, and separate clean probe fitting from transformed evaluation. They specify how representation supports and safety-readout outcomes are selected and evaluated.

S3 Frozen Protocols and Reproducibility

Representation Corpus Protocol

  • The separate five-variant analysis evaluates all 90,000 expected cases across four models; 18 cases (0.02%), all under local edits, remain unavailable rather than being assigned zero.This count is distinct from transformed texts in the main 15-variant protocol.Source: Representation Corpus Protocol · p. 19

SAE Specifications

Prompt Provenance and Transformations

  • The 1,000 clean English prompts were generated once with fixed acceptance criteria, and six transformations alter context, presentation, punctuation, or noncritical words while preserving the target phrase exactly once.The local punctuation and lexical edits differ from context and wrapper transformations that alter surrounding material or presentation.Source: Prompt Provenance and Transformations · p. 20

Target Token Alignment

  • Evaluation uses the final non-special token overlapping the preserved target span, and the five-variant analysis reports no mapping failures across 90,000 cases.Wrappers and prefixes may shift token indices while the evaluated surface span remains fixed.Source: Target Token Alignment · p. 20

Handling Repeated Realizations

  • For the 15-variant analysis, exact duplicate texts are collapsed within operators, and the five operators remain equally weighted even when duplicate removal leaves different realization counts.The selected-support table is available for 161,996 of 162,000 cases; unavailable cases remain missing.Source: Handling Repeated Realizations · p. 20

Safety Corpus and Probe Training

  • Safety probes use 900 harmful and 900 benign prompts, standardized clean training activations, and clean-validation selection of regularization and threshold without transformed or test outcomes.The threshold targets at most 5% empirical false-positive rate on clean benign validation logits.Source: Safety Corpus and Probe Training · p. 21

S4 Extended RSS Atlas and Calibration

The supplementary analyses examine how the redistribution profile varies across models, prompts, and feature supports, and test selector and summary robustness. Context and wrapper transformations change active feature sets more strongly than local edits, while alternative selectors can preserve similar profiles despite selecting different features.

Profiles Across Models and Layers

  • Figure S1 resolves the aggregate transformation profile by model and layer to assess whether context-wrapper versus local-edit separation is shared across sampled SAEs.The figure plots normalized MeanRSS profiles across sampled layers of eight models.Source: Profiles Across Models and Layers · p. 21, Figure S1 · p. 21

Support Organization

  • Figure S2 compares selected feature identities across transformation settings using Jaccard overlap, since similar response magnitudes need not involve the same features.It reports pairwise overlap across settings and averages within each model.Source: Support Organization · p. 21, Figure S2 · p. 22

Support Availability and Active Feature Changes

Selector Diagnostics

  • The alternative selector yields mean off-diagonal Jaccard 0.316 versus 0.354 for the main selector, yet normalized MeanRSS correlates at 0.986 and same-setting support overlap is 0.698.The transformation profiles are nearly identical even though exact selected features differ; the main and alternative selectors have mean selected counts of 18.32 and 16.18, respectively.Source: Support Availability and Active… · p. 23, Selector Diagnostics · p. 25, Figure S5 · p. 25

Content Categories and Model Composition

Selector and Summary Robustness

  • Robustness analyses vary selector parameters, magnitude summaries, normalization, and feature scaling; their counts distinguish preserving the full six-setting order from preserving only its top or bottom setting.Table S11 reports profile Spearman correlation and the context-wrapper minus local-edit gap after scaling feature changes by clean-prompt activation scale.Source: Selector and Summary Robustness · p. 25, Table S9 · p. 26, Table S10 · p. 26

Intensity Protocol

These analyses vary transformation intensity and the allocation of variants between selection and evaluation, then assess held-out support behavior and safety-readout outcomes. They distinguish representation response, margin erosion, and recurrent-failure ranking as separate quantities.

Intensity Protocol

  • The four intensity levels increase transformation strength within a setting, and support selection is performed independently at each prompt and level.Rank correlations are computed within each prompt, SAE, and transformation before equal averaging across SAEs.Source: Intensity Protocol · p. 26, Table S12 · p. 26

Selection–Evaluation Split Sensitivity

Held Out Support Validation

S5 Extended Audit Results

  • The mismatched-RSS comparator uses a support selected for another transformation, removes repeated feature indices, and truncates it to the current RSS support size.It is unavailable when too few unique features remain.Source: S5 Extended Audit Results · p. 28

Probe Performance

  • The safety-probe analyses report clean test performance and compare clean versus transformed AUROC while holding fitted coefficients and calibrated thresholds fixed.Table S18 reports AUROC, false-negative rate, and false-positive rate.Source: Probe Performance · p. 28, Table S18 · p. 28

Results by Transformation

  • +0.338 [0.236, 0.443] for benign prompts and −0.374 [−0.498, −0.259] for harmful prompts are the context-wrapper versus local-edit differences in Gemma margin erosion.The opposing class-specific differences show that a shared representation profile does not imply a shared direction of readout effect.Source: Results by Transformation · pp. 28, 30, Figure S7 · p. 29, Table S19 · p. 29

Intensity Analysis on the Safety Corpus

  • The safety intensity analysis compares margin erosion and MeanRSS across four levels, with monotonicity counts restricted to clean-correct prompts eligible for that analysis.For Gemma, the subsets contain 254 prompts for irrelevant context and format wrappers, and 222 for punctuation noise; Llama subsets contain 241 and 212, respectively.Source: Intensity Analysis on the Safety Corpus · p. 30, Figure S8 · p. 30, Table S20 · p. 30

Full Risk Results

  • Risk-signal comparisons distinguish correlation with held-out erosion from AUROC and average precision for recurrent failure, so tracking erosion well need not yield the strongest failure ranking.Table S21 evaluates these signals on a common held-out cohort.Source: Full Risk Results · p. 31, Table S21 · p. 31

Relevance of the Selected Features on New Variants

The section tests whether RSS-selected features remain relevant on held-out variants and whether those features help explain or predict downstream readout behavior. Additional analyses examine other Llama layers, selective review, and MeanRSS-guided probe training.

Relevance of the Selected Features on New Variants

  • Margin Contribution Retention at K = 20 compares held-out absolute contribution retained by RSS, mismatched RSS, and contribution-ranked supports; higher MCR means less omitted absolute contribution.For a fixed case, supports share the same denominator, while signed error also depends on cancellation.Source: Relevance of the Selected Features on… · p. 31, Table S24 · p. 32

Additional Llama Layers

  • The Llama 3.1 8B layer 8 and 16 audits report varying functional metrics, so response persistence and retained audit utility are not interchangeable.The protocol includes direction persistence, contribution retention, residual-margin AUROC, localization, and restoration.Source: Additional Llama Layers · p. 32, Table S28 · p. 33

Localization and Readout Counterfactuals

Selective Review

  • Selective review ranks cases by residual margin and measures residual error among cases left after the highest-risk cases are removed.The normalized pAURC integrates residual error over review budgets from 0 to 40%; positive ΔpAURC indicates lower residual error than random ranking.Source: Selective Review · p. 33, Table S30 · p. 34

Probe Training with MeanRSS Sampling

  • MeanRSS-guided sampling lowered harmful false-negative rates in the reported runs, but the training extension did not establish a consistent AUROC advantage over uniform sampling.Across five runs, Gemma’s average macro AUROC difference was +0.0011 (standard deviation 0.0051); Llama had lower clean, mean, and worst-setting AUROC with MeanRSS sampling.Source: Probe Training with MeanRSS Sampling · pp. 34–35, Table S32 · p. 35

Related papers

This paper cites

Loading 2610.21240v1…