PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability

Zhenting Huang, Bo Jiang, Junnan Liu, Zhixing Tan, Qianren Mao

arXiv:2609.37857v2cs.AI

TL;DR

SAE features are useful for interpretation only if they remain stable when meaning is expressed in different surface forms. This paper studies that reliability in TopK SAEs, diagnoses boundary-related losses, and introduces pairwise rank stabilization, which improves rare-feature sensitivity while preserving reconstruction quality and alive coverage.

  • Problem

    The paper asks whether SAE features remain stable units of analysis across meaning-preserving changes in surface form.

    Source: §1 · p. 1

  • Method

    The paper measures feature sensitivity under paraphrases, diagnoses TopK boundary margins, and uses pairwise rank stabilization to target source–paraphrase ordering failures.

    Source: §1 · p. 2, §3.1 · p. 4, §5.2 · p. 7

  • Results

    +8.83pp rare-feature sensitivity on held-out sources with pairwise rank stabilization, while preserving reconstruction quality and alive coverage.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TopK SAE evaluation should consider feature reliability under semantic variation and selection-boundary geometry alongside reconstruction, sparsity, and feature count.

    Source: §7 · p. 9

    The evidence concerns activation reliability; whether dropped features represent lost human-interpretable concepts remains open.

    Source: §6 · p. 9

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are increasingly scaled to wider dictionaries to recover fine-grained structure from large language model activations. However, a feature is useful for interpretation only if it remains a stable unit of analysis when the same meaning is expressed in different surface forms. We study this reliability question for TopK SAEs via feature sensitivity. Experiments demonstrate that scaling selectively reduces the sensitivity of rare features, while common features remain comparatively stable. A controlled width\(\times k\) factorial experiment identifies the active budget k as the root cause: the degradation arises from the selection boundary rather than dictionary width alone. We attribute this failure to the geometry of TopK selection. The active margin, the distance to the cutoff, predicts feature loss without thresholds. Guided by this margin diagnosis, we introduce pairwise rank stabilization. Our method targets ordering failures at the cutoff and improves rare-feature sensitivity by \(8.83\) percentage points, while keeping reconstruction and alive-feature coverage near the baseline. Overall, our results suggest that wide TopK SAEs should be evaluated not only by reconstruction, sparsity, and feature count, but also by feature reliability under semantic variation and boundary geometry for stable interpretability.

1 INTRODUCTION

The paper asks whether SAE features remain reliable units of analysis across meaning-preserving changes in wording. It finds that rare-feature sensitivity declines with TopK scaling and proposes a boundary-focused repair.

1 INTRODUCTION

  • Feature sensitivity measures whether a feature active on a source remains active under meaning-preserving paraphrases.The paper treats this as a reliability probe for SAE-based interpretation.Source: §1 · p. 2
  • Along the practical scaling path, rare-feature sensitivity falls from 0.650 to 0.585 while common features remain highly stable.The evaluation uses source-disjoint paraphrases and groups features by activation frequency.Source: §1 · p. 2
  • A controlled factorial identifies active budget, rather than dictionary width, as the factor carrying rare-feature sensitivity loss.Width marginal means are nearly flat while reconstruction error improves over the same range.Source: §1 · p. 2
  • TopK boundary geometry explains the selective loss: rare active features lie closer to the cutoff and are easier to dislodge after paraphrase-induced shifts.Pairwise rank stabilization keeps source-active features ranked above hard paraphrase competitors at the boundary.Source: §1 · p. 2
  • +8.83pp rare-feature sensitivity on held-out sources with pairwise rank stabilization, while reconstruction quality and alive coverage are preserved.The method targets source–paraphrase ordering failures near the TopK cutoff.Source: §1 · p. 3

2 RELATED WORK

Related work motivates treating reliability as a property of the interpretive unit, not just of explanations built around it. It also positions TopK selection boundaries and paraphrase-based sensitivity as the paper’s diagnostic focus.

Units of Analysis in Mechanistic Interpretability

  • Mechanistic interpretability relies on units that support explanation, circuit localization, and intervention, but learned sparse bases are not always the only faithful basis for analysis.This motivates treating unit reliability as part of usability.Source: §2 · p. 3

Sparse Autoencoders and Scaling

  • SAE width can expose finer structure, while TopK’s per-example rank cutoff makes active-set changes and margins measurable under paraphrasing.Other selectors resolve competition at the batch level or through learned per-feature thresholds.Source: §2 · p. 3

SAE Evaluation Beyond Reconstruction

  • Evaluation work shows that reconstruction alone cannot establish feature quality, and paraphrases test whether the same latent remains active when meaning is preserved but wording changes.The paper uses feature sensitivity to study scaling, boundary-based prediction, and repair.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define TopK feature activity and the paper’s reliability and frequency diagnostics. They also specify the margin concepts and evaluation practices used to assess boundary stability.

3.1 SETUP

  • A TopK SAE keeps the k highest-scoring latents and applies ReLU; a feature is active only when its post-ReLU value is positive.A feature can leave the active set because competitors outrank it, or be zeroed despite remaining selected.Source: §3.1 · p. 4
  • Feature sensitivity measures whether a feature active on the source remains active on meaning-preserving paraphrases.Dropouts can arise from falling below the kth competitor score or from a non-positive score after selection; the paper reports these sources separately.Source: §3.1 · p. 4
  • Frequency buckets classify features as rare, medium, or common by activation frequency; they are diagnostic groups, not semantic labels.Alive coverage counts features with positive activation at least once, while features below the rare lower bound are excluded from bucketed averages.Source: §3.1 · p. 4
  • The top-k margin measures the score gap at the selection cutoff, and the active margin measures an active feature’s score minus that cutoff.Near-cutoff examples use m < 0.01 as an operational warning signal; hard paraphrase competitors sit just below the cutoff.Source: §3.1 · p. 4
  • Source texts are partitioned disjointly across training, frequency estimation, source selection, and evaluation, with held-out sources providing independent confirmation.Qwen and Gemma checks use the same scale, partitioning, and evaluation setup.Source: §3.1 · pp. 4–5
  • Interventions are reported with sensitivity, near-cutoff share, reconstruction, and alive-feature coverage so reliability gains can be read alongside quality tradeoffs.Table 1 reports rare-feature sensitivity marginal means across a 3 × 3 width-by-k factorial with three seeds per cell at the 10M token budget.Source: §3.1 · p. 5, Table 1 · p. 5

4 WIDE TOPK SAES CAN LOSE FEATURE SENSITIVITY

Along the studied scaling path, rare-feature sensitivity declines while common-feature sensitivity remains comparatively stable. A controlled factorial points to active budget rather than width as the main factor, and active margins identify features prone to paraphrase-induced flips.

4.1 SCALING EXPOSES A SENSITIVITY FAILURE FOR RARE FEATURES

  • At 10M tokens, rare-feature sensitivity drops from 0.650±0.015 at width 3072 to 0.585±0.017 at width 12288, while common-feature sensitivity changes from 0.998 to 0.956.The rare-feature decline also appears at 200k tokens (0.802 to 0.793) and 1M tokens (0.779 to 0.748).Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 WHICH FACTOR DRIVES THE DECLINE: THE ACTIVE BUDGET

  • Rare-feature sensitivity falls by 14.8pp on average from k = 32 to k = 128, while width marginal means span about 1.9pp.The near-cutoff share rises from 10% to 27% to 54% as k grows and stays flat across width; at width 12288, reconstruction MSE improves from 1.35 to 0.89 as k grows.Source: §4.2 · p. 5

4.3 THIN TOP-k MARGINS DIAGNOSE THE FAILURE

  • At width 12288 with k = 128, 54.0% of examples have m < 0.01, versus 10.3% at width 3072, and instances closest to the cutoff flip most often.Figure 2 shows near-cutoff examples increasing along the practical scaling path and active-margin bins predicting source–paraphrase flips.Source: §4.3 · p. 6, Figure 2 · p. 6
  • Active margin predicts paraphrase dropout at AUROC 0.728 on GPT-2 layer 8, with drop rates from 94.1% nearest the cutoff to 0.6% farthest away.Across six additional GPT-2 depths, AUROC ranges from 0.747 to 0.850; the within-model ranking requires no cross-model score scale.Source: §4.3 · p. 6, Table 2 · p. 6
  • Across source–paraphrase pairs, active instances nearer the cutoff flip more often and move toward or below it, linking margin distance to the individual drops behind the aggregate sensitivity gap.The proposed repair target is to keep source-active features ranked above their paraphrase-side competitors at the cutoff.Source: §4.3 · p. 6

5 RANK STABILIZATION FROM THE MARGIN DIAGNOSIS

The section tests whether the margin diagnosis can guide training and finds that pairwise rank stabilization improves rare-feature sensitivity while keeping reconstruction and alive-feature coverage near baseline.

5 RANK STABILIZATION FROM THE MARGIN DIAGNOSIS

  • The training comparison tests whether the margin diagnosis can guide a repair, comparing consistency regularization, boundary pressure, boundary-local controls, and pairwise rank stabilization.Rank stabilization gives the largest rare-feature reliability gain on validation while keeping reconstruction MSE and alive-feature coverage near baseline.Source: §5 · p. 7

5.1 FROM OVERALL AGREEMENT TO BOUNDARY ORDERING

  • Whole-mask agreement is not the target: active sets can overlap heavily yet disagree about the feature a downstream audit follows.A thin gap between the kth and (k + 1)th features can flip regardless of mask overlap.Source: §5.1 · p. 7

5.2 PAIRWISE RANK STABILIZATION TARGETS THE DIAGNOSED FAILURE

  • Pairwise rank stabilization penalizes paraphrase hard negatives just below the TopK boundary when they would outrank source-active features after rephrasing.The band size R defines the narrow competitive band, and the standard reconstruction objective trains alongside the ordering term.Source: §5.2 · p. 7

5.3 MAIN REPAIR RESULT

  • +8.83pp: on the held-out collection, rank stabilization raises rare-feature sensitivity from 0.467 ± 0.007 to 0.555 ± 0.011 while reconstruction MSE and alive coverage remain near baseline.Across five seeds, mean reconstruction MSE moves from 0.880 to 0.888 and mean alive coverage from 96.70% to 96.66%.Source: Table 3 · p. 8, §5.3 · p. 8
  • +12.3pp: on validation, rank stabilization raises rare-feature sensitivity from 0.5846 to 0.7073, with reconstruction MSE and alive coverage close to baseline.Global margin lowers near-cutoff share to 26.8% but raises reconstruction MSE from 0.880 to 3.584 and lowers alive coverage from 96.70% to 73.21%.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 ROBUSTNESS, COST, AND SCOPE

  • Boundary pressure has reconstruction and coverage costs; across models, rank stabilization improves sensitivity for crowded-boundary Qwen but is near neutral for sparser-boundary Gemma.At 200k, 1M, and 10M tokens, the wide baseline remains less sensitive to rare features than the narrower one; Qwen improves from 0.929 to 0.946, while Gemma is −0.8pp.Source: §5.4 · pp. 8–9

6 DISCUSSION AND LIMITATIONS

The discussion limits the claims to feature activation reliability under meaning-preserving rewrites and recommends reporting sensitivity alongside standard SAE metrics.

6 DISCUSSION AND LIMITATIONS

  • The evidence concerns activation reliability, leaving open whether dropped features represent lost human-interpretable concepts or whether the gains improve downstream interpretability.The rare, medium, and common frequency buckets are diagnostic partitions, not a semantic taxonomy.Source: §6 · p. 9
  • Feature sensitivity should be reported alongside reconstruction and sparsity; it costs one paraphrase pass per evaluation set and separates checkpoints with near-identical standard metrics.Global margin pressure serves as a mechanistic reference, moving the boundary at a reconstruction and coverage cost.Source: §6 · p. 9

7 CONCLUSION

The conclusion frames TopK SAE reliability as a feature-level concern under semantic variation and links rare-feature drops to selection-boundary geometry.

7 CONCLUSION

  • In TopK SAEs, paraphrase-related failure concentrates in rare features, follows selection-boundary geometry, and can be predicted by margins near the top-k cutoff.Pairwise rank stabilization targets the source–paraphrase ordering failure behind the drops.Source: §7 · p. 9
  • SAE evaluation should report feature reliability alongside reconstruction and feature count because these measures can move in opposite directions along the same scaling path.Source: §7 · p. 10

Appendix organization.

The appendix documents the experimental protocol, qualitative drop audit, statistical conventions, implementation details, and resources underlying the main claims.

Appendix organization.

  • The main GPT-2 reliability probe uses source-disjoint paraphrase pairs, with model comparison and hyperparameter selection on validation and independent confirmation on held-out data.The collection has 295 retained source–paraphrase pairs, split by original source text into approximately 147 auxiliary-training and 148 held-out evaluation pairs.Source: Table A.1 · p. 14, Appendix organization · p. 14
  • The qualitative audit records selection-boundary events rather than confirmed active-feature losses, because source-side scores can be non-positive despite top-k selection.It contains 49 deduplicated rare drop cases and 20 retained comparisons; the rare drops have median source active margin 0.285 and median paraphrase rank 158.Source: Appendix organization · p. 14, Table A.2 · p. 15
  • Appendix means and tests use independent SAE training seeds as the statistical unit, with SEM error terms and two-sided Welch tests unless otherwise noted.The main rank-stability configuration uses λ = 0.004 and R = 8, selected on validation.Source: Appendix organization · p. 15
  • Reported runs fit on one NVIDIA A100-80GB, and the anonymized release is to document run commands, code hashes, output paths, model identifiers, and software versions.The experiments use publicly released pretrained models and standard open-source libraries without redistributing a new pretrained model or scraped text dataset.Source: Appendix organization · p. 15

A.1 FULL GPT-2 EVIDENCE

The GPT-2 evidence localizes the reliability loss to rare features and a crowded TopK cutoff, while identifying rank stabilization settings that improve rare-feature sensitivity without the costs of global boundary pressure.

A.1 FULL GPT-2 EVIDENCE

  • The validation statistic averages over 918 rare-feature source-side instances; each source text has 131 rare active features on average.Frequency buckets are estimated separately on a 1M-activation sample.Source: §A.1 · p. 16, Table A.3 · p. 16
  • The evidence tables cover token-budget sweeps, cutoff-linked paraphrase flips, global boundary pressure, and the λ × R sweep used to select the main configuration.The reported rank-stabilization setting improves rare-feature sensitivity without the reconstruction and alive-coverage costs of global boundary pressure.Source: §A.1 · p. 16
  • All 16 λ × R cells improve rare-feature sensitivity over the 0.5846 baseline by +4.9 to +12.3 percentage points, placing the selected setting inside a broad basin.The selected configuration is λ = 0.004, R = 8.Source: §A.1 · p. 16
  • Rare-feature sensitivity at width 12288 stays below narrower settings at 200k, 1M, and 10M tokens, while common-feature sensitivity remains near ceiling.The same evidence links the aggregate sensitivity loss to paraphrases displacing active features nearest the source-side cutoff.Source: §A.1 · p. 16

A.2 DEPTH CHECK ACROSS GPT-2 LAYERS

The depth check finds boundary crowding and frequency-dependent sensitivity across the inspected GPT-2 layers, with a layer-10 repair result and a 1M-token scope boundary.

A.2 DEPTH CHECK ACROSS GPT-2 LAYERS

  • At layer 10, pairwise rank stabilization improves rare-feature sensitivity by 5.2 percentage points with negligible reconstruction and alive-feature changes.The depth check uses width 12288, k = 128 SAEs on layers 2, 8, and 10, with N = 3 independently trained baselines per layer.Source: §A.2 · p. 16
  • All inspected layers show substantial near-cutoff shares and frequency-dependent sensitivity, with layer 2 combining the highest near-cutoff share and lower rare-feature sensitivity.This ordering matches the paper's margin account; the magnitude of the pattern varies with depth.Source: §A.2 · p. 16, Table A.7 · p. 17
  • The depth runs use 1M-token caches, so their absolute rare-feature sensitivity is higher than the corresponding main-table row; full 10M-token evidence remains centered on GPT-2 layer 8.The separate width sweep reports a sensitivity drop concentrated in rare features, with common-feature sensitivity stable across token budgets.Source: §A.2 · p. 16, Table A.4 · p. 17
  • Global margin-aware training reduces near-cutoff examples, but reconstruction, alive-feature coverage, and rare-feature sensitivity reveal costs to optimizing global clearance.Table A.5 describes these as comparisons of global boundary pressure at the 10M-token budget and smaller-budget sweep rows.Source: Table A.5 · p. 17
  • Active feature instances nearest the source cutoff have substantially higher paraphrase drop rates, connecting the aggregate sensitivity statistic to active-set flips.The margin bins also report source and paraphrase mean top-k ranks.Source: Table A.6 · p. 17

A.3 TOKEN BUDGET AND MODEL FAMILY SCOPE

The scope checks relate rare-feature sensitivity to boundary crowding across token budgets and model families, formalize margin-based drop predictions, and compare targeted controls and internal usability proxies.

A.3 TOKEN BUDGET AND MODEL FAMILY SCOPE

  • Across 200k, 1M, and 10M token budgets, the wide baseline has lower rare-feature sensitivity than the narrower baseline, and the margin objective is lower still.Within each model family, the share of near-cutoff examples describes boundary crowding, and repair movement is largest where that share is largest.Source: Figure 3 · p. 18, §A.3 · p. 18

A.4 A BOUNDARY VIEW OF FEATURE SENSITIVITY

  • A source-active feature drops when paraphrase-induced margin change crosses its negative source margin; under comparable perturbation laws, smaller margins predict higher drop probability.In a public JumpReLU SAE, drop rate falls from 61.8% in the nearest-threshold bin to 1.5% in the farthest.Source: §A.4 · p. 19

A.5 ADDITIONAL CONTROLS

  • The additional controls distinguish global top-k-gap enlargement, a smooth hard-negative boundary, rare-feature clearance, and rank stability against paraphrase hard competitors.The specificity audit reports low rare-to-common migration and low false positives on unrelated text under boundary manipulation.Source: §A.5 · p. 20

A.6 SAE-INTERNAL USABILITY PROXIES

  • The SAE-internal proxies measure true paraphrase retention, mismatch activation, retention gap, and paraphrase-to-source feature contribution mass rather than downstream task performance.The summaries are for rare features and are reported as mean±SEM over seeds.Source: Table A.9 · p. 20, §A.6 · p. 20

AI USE DISCLOSURE

  • GPT-4o generated the paraphrase pairs used as a reliability probe, and each pair was manually reviewed for meaning preservation.The authors state that no model output served as a label or ground truth.Source: AI USE DISCLOSURE · p. 20
Loading 2609.37857v2…