PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens

Matteo He, William F. Shen, Xinchi Qiu, Nicholas D. Lane

arXiv:2609.01936v1cs.CLcs.AIcs.LG

TL;DR

Lens readings can reflect both hidden states and the readout used to decode them, and fitted lenses may report different tokens for the same state depending on their fitting corpus. The paper introduces Sparse Readout Prism, which decomposes logits and logit differences into sparse readout-feature contributions using only readout weights. SRP reconstructs more tested logit differences than geometric baselines, while dominant features remain stable across corpus-dependent token readings.

  • Problem

    Token rankings cannot cleanly distinguish intermediate computation from readout structure or, for fitted lenses, from the fitting corpus used to transport states.

  • Method

    SRP applies sparse coding to readout weights and decomposes token logits or logit differences into shared feature contributions plus residuals.

  • Results

    8.9 to 17.3 percentage points more of tested logit differences are reconstructed than by the strongest of six row-geometry baselines, while dominant readout features remain stable across fitting conditions.

  • Takeaways & Limitations

    SRP provides a corpus-independent reference for separating changes in a lens’s hidden-state reading from changes introduced by its readout instrument.

  • Takeaways & Limitations

    SRP explains selected readout scores and readout organization, but circuit- or generation-level questions require interventions beyond the readout analyses presented.

Abstract

from arXiv · show

A language model's prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading reflects both the hidden state and the readout (the unembedding matrix) used to decode it. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states. We call this dependence corpus conditionality. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses. Replacing the original readout with SRP's sparse approximation reconstructs 8.9-17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows. Ablating features shifts logit differences in proportion to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses.

1 Introduction

Lens readings combine information from hidden states with the readout used to decode them, making token-based interpretations ambiguous and corpus-dependent. SRP instead analyzes sparse readout features, providing a corpus-independent reference for comparing lens readings.

  • Motivation: Lens methods decode intermediate states through the readout, so token rankings can reflect readout structure rather than intermediate computation alone.The logit lens uses the unchanged unembedding, while fitted lenses first transport states into the LM-head space.
  • Motivation: Token identity is too coarse for polysemous tokens and too fine for distinct tokens that share readout structure.The token bug shares one row across software and insect senses, while different tokens can share dominant directions.
  • Corpus conditionality: Fitted lenses can report different languages for the same fixed hidden state when their fitting corpora differ, a dependence called corpus conditionality.This makes inference from reported tokens to a working language underdetermined.
  • Sparse Readout Prism: SRP changes the reported unit from token identities to sparse readout features while leaving state transport unchanged.Its features expose readout structure and support comparisons across tokens, contexts, layers, and lenses.
  • Contributions: 8.9 to 17.3 percentage points more of tested logit differences are reconstructed than by the strongest of six row-geometry baselines.Feature ablations shift scores in proportion to their SRP contributions.
  • Empirical implication: The dominant readout feature remains stable even when fitted lenses report different surface languages for the same hidden state.This supplies a reference fixed in the weights for separating changes in state from changes in the instrument.

2 Related Work

The paper relates SRP to prior work on lens methods, output geometry, sparse coding, and activation decomposition. Its novelty is applying sparse coding to trained transformer readout rows while enforcing score reconstruction.

  • Lens methods: Existing lens methods report intermediate computations in token identity, whether they use the unchanged unembedding or a learned transport map.SRP changes the analysis unit without changing the lens, enabling comparison across differently fitted lenses.
  • Output geometry: Prior work identifies shared structure in unembedding rows through weight tying, output geometry, and tokenizer segmentation.These approaches motivate studying readout-row organization directly.
  • Sparse row dictionaries: SRP extends sparse coding from static word embeddings to the rows of a trained transformer’s LM head.Unlike neighborhoods, clusterings, and principal directions, SRP is constrained to sum to the selected logit or margin.
  • Activation decompositions: Activation SAEs decompose hidden-state representations, whereas SRP decomposes readout directions using one basis shared across lenses and layers.An activation SAE also depends on the corpus producing its hidden states, unlike SRP’s readout-based basis.

3 Sparse Readout Prism

SRP factorizes the LM head’s readout rows into sparse shared features and residuals, then decomposes selected logits or contrasts into feature contributions. A single readout dictionary makes these decompositions comparable across contexts, layers, and lenses while retaining exact residual accounting and fidelity diagnostics.

  • Setup: A lens transforms a layer state into a decoded state, which the LM head scores by dot products with unembedding rows.SRP factorizes the readout matrix alone; the lens changes only the decoded state presented to it.
  • Factorizing the LM head: SRP fits a sparse autoencoder to LM-head rows, representing each row with a shared offset, sparse feature directions, and a residual.The feature directions describe readout organization, while coefficients specify each row’s feature usage.
  • Selected scores: A selected token score or contrast becomes one signed contribution per readout feature plus a residual in logit units.Token logits use one row, while margins use differences between rows; zero-sum contrasts cancel shared components.
  • Comparability: For a fixed selected direction, decompositions vary across contexts, layers, or lenses only through the decoded state.The coefficient structure is fixed by the selected direction and readout SAE, and dominant features are compared in a shared coordinate system.
  • Fidelity: The explicit residual makes each decomposition exact by construction, while fidelity diagnostics measure how well reconstructed rows represent original scores.Diagnostics include residual-based error, sign agreement, and coverage; replacement fidelity evaluates preserved readout distributions and rankings.

4 The Decomposition Is Faithful, Predicts Intervention Effects, and Survives Retraining

SRP faithfully decomposes readout scores, predicts local intervention effects, and remains useful across dictionary retraining and feature comparisons.

  • 4 The Decomposition Is Faithful, Predicts Intervention Effects, and Survives Retraining: Across eight models, SRP reconstructs readout scores, while replacement preserves held-out argmax agreement at 0.89–0.90 for Qwen3.5 and Ministral and 0.75–0.76 for reasoning-distilled readouts.The two Gemma-4 readouts use a nonlinear softcap that SRP does not factorize.
  • 4.2 SRP Outperforms Every Baseline Built on Row Geometry: SRP reaches coverage 0.72–0.81 across six softcap-free readouts and outperforms all six row-geometry baselines, with non-overlapping intervals.It also has the lowest mean, 95th-percentile, and maximum absolute error among the seven methods.
  • 4.2 SRP Outperforms Every Baseline Built on Row Geometry: At matched SRP width and sparsity, k-means trails SRP by 18 points of coverage, while shuffled codes and random support remain below 0.10.These controls test whether SRP’s performance reflects more than dictionary capacity or arbitrary support.
  • 4.2 SRP Outperforms Every Baseline Built on Row Geometry: The same feature contributions predict measured local margin changes with r2 = 0.83–0.93, versus random-control r2 ≤0.06.Measured changes preserve the predicted sign and rank, although slopes exceed one on some smaller Qwen readouts.
  • 4.4 The Decomposition Survives Retraining: Across retrained dictionaries, same-contrast feature-group lists share mean Jaccard 0.21–0.24, versus 0.014–0.018 for unrelated contrasts.The same-contrast overlap is 13–15× higher, although feature directions themselves have median nearest-dictionary cosine ∼0.33 at 32× width.
  • 4.5 Context Decides Which Features Carry a Token: For ambiguous words, every word beats its majority baseline across one, two, four, or eight selected features, while the full decoded state remains the stronger sense predictor.A nearest-centroid probe on the full state reaches 0.96, 0.97, and 0.92 across the reported settings.

5 A Fitted Lens Reports the Language of Its Fitting Corpus

Fitted lenses can report different languages for the same hidden state because their learned maps depend on fitting corpora, but SRP finds stable dominant readout features across lenses and settings.

  • 5 A Fitted Lens Reports the Language of Its Fitting Corpus: Corpus conditionality means fitted-lens token readings reflect both the hidden state and the fitting corpus, so token rankings cannot distinguish those sources.This dependence applies to fitted lenses; direct unembedding-based latent-language studies do not use a fitting corpus.
  • 5 A Fitted Lens Reports the Language of Its Fitting Corpus: Holding the hidden state fixed while changing only the fitting corpus can produce different decoded states and token readings, whereas SRP’s basis is learned only from model weights.The SRP reference excludes translation pairs, language labels, and fitting corpora.
  • 5 A Fitted Lens Reports the Language of Its Fitting Corpus: English- and Chinese-fitted Jacobian lenses report different scripts on 39 of 80 prompts, including 9 of 10 factual-recall prompts, despite processing identical frozen states.Their outputs converge at the final layer, but intermediate readings can support opposing interpretations of the model’s working language.
  • 5.2 Different Tokens Share a Dominant Feature: The same SRP feature dominates under both lenses on 77 of 80 prompts (96%, 95% CI [0.90, 0.99]), with f23180 contributing 75% and 63% in the Figure 4 example.Null comparisons produce no matches, and agreement also appears in 5/6 digit and 5/6 proper-noun controls.
  • 5.2 Different Tokens Share a Dominant Feature: Across lens constructions, English–Chinese translators retain feature agreement on 67/80 prompts, while English–German lenses retain it on 84/90 despite shared scripts.The matched null comparisons are at most 3/159 and 1/240 for the translator setting, and 1/204 and 0/270 for the shared-script setting.
  • 5.3 Corpus Size and Lens Construction: Tripling fitting corpora leaves feature agreement at 77/80 for English–Chinese and raises it from 84/90 to 85/90 for English–German, while token disagreement changes in opposite directions.The reported pattern does not support a common estimation-noise explanation for both language pairs.

6 Future Work

Future safety audits can use SRP to compare recurring feature contributions across prompts, languages, adversarial inputs, and checkpoints.

  • 6 Future Work: Safety audits can define concrete readout-score contrasts and compare feature contributions across paraphrases, languages, adversarial prompts, and checkpoints.The paper recommends reporting reconstruction error, transfer, distributional cost, and effects on other outputs.

7 Conclusion

Sparse Readout Prism makes readout features an alternative unit for interpreting lens readings, complementing token-based analysis with a reference fixed in model weights. It supports decomposition, attribution, and controlled readout edits across lenses and layers.

  • SRP provides one dictionary per model, allowing every lens and layer to decompose readings into the same feature terms without further fitting.The shared terms also support auditing what the fitting corpus contributed to a reading.
  • Readout features expose structure that token identities can obscure, including multiple senses within one token and shared structure across surface forms and scripts.This makes organization of the model’s output space measurable from weights alone.
  • SRP feature terms extend to layer-by-layer attribution and provide candidate directions for constrained edits on the readout side.These terms compose with direct logit attribution, while edit performance varies across the evaluated Qwen models.

Limitations

The paper’s limitations concern basis dependence, controlled empirical scope, and the boundary between readout explanations and broader model behavior. Feature labels and quantitative conclusions require care because they depend on dictionary choices and readout-specific quantities.

  • Fitting and basis choice: SRP feature directions depend on dictionary size, sparsity k, training recipe, and random seed, despite stability checks across selected operating points.The paper reports fidelity sweeps and stability across seeds and recipes, while the one-time fit takes roughly 6–12 GPU-hours on one NVIDIA A40.
  • Empirical scope: The cross-lens study is limited to Qwen3.5-9B and specified English–Chinese and English–German settings, while readout edits are evaluated on Qwen3.5-2B.Cross-model losses for edits are reported in Appendix N.
  • Interpretation boundary: SRP explains selected readout scores and readout organization, but circuit-level and generation-level questions require interventions beyond the readout-side analyses.Feature labels may also inherit tokenizer and vocabulary geometry.
  • Interpretation boundary: Quantitative conclusions rely on feature ids, signed contributions, and residual errors, so feature labels should not be treated as independent semantic ground truth.The paper specifically notes that labels may inherit tokenizer and vocabulary geometry.

Ethical Considerations

The paper uses public checkpoints and held-out decoded states rather than human-subject data or deployed systems, while acknowledging sensitive-text and dual-use considerations. Its ethical scope remains tied to measured readout behavior rather than claims about tokens, outputs, or system-level safety.

  • The study collects no new human-subject data and evaluates no deployed systems, using public checkpoints and held-out decoded states including C4 continuations.C4 may contain offensive, private, copyrighted, or sensitive text.
  • The authors make no claims about the social validity of individual tokens, labels, or generated outputs and do not redistribute checkpoints or raw C4 text.Released SRP dictionaries carry artifact-specific license and usage notes.
  • SRP’s ability to move specified lexical scores is dual-use, so the paper reports probes for distribution shift and off-target effects while leaving safety filters and deployment controls outside scope.System-level evaluation of moderation and deployment controls is explicitly beyond the work.
  • The evaluation is organized around readout fidelity, model scale, portability, tokenizer effects, row norms, special tokens, and matched controls.Primary claims concern selected readout scores, especially token logits and contrastive logit differences.

D.1 Qwen3.5-9B Fixed-k = 128 Recipe Selection

The Qwen3.5-9B sweep selects a no-PCA TopK recipe at the strict k = 128 budget, then evaluates larger widths and sparsity budgets as separate operating regimes. Increasing capacity improves reconstruction and decision fidelity, while budgets below k = 128 strand substantial dictionary capacity.

  • Recipe selection: The selected Qwen3.5-9B recipe fixes k = 128, uses no PCA, and trains for 20k steps after 5k screening and convergence checks.The recipe uses D = 32768, row-seeded initialization, hybrid_50freq_50uniform sampling, and a delayed prism-loss ramp.
  • Recipe selection: +0.0025 rowEV was gained from 15k to 20k steps, supporting the 20k stopping rule.The convergence ladder also reports stable decision-fidelity and usage metrics at the stopping point.
  • Recipe selection: The selected no-PCA TopK recipe improves over the previous long-trained matryoshka baseline on reconstruction, top-1 agreement, KL, and feature usage.The main drivers identified are row-seeded initialization, delayed prism ramping, plain TopK, and D = 32768 width.
  • Capacity sweep: The D = 131072, k = 256 run achieves the best rowEV, top-1, and KL in the Qwen3.5-9B capacity sweep.Its rare feature ladder remains high at 0.300 → 0.294 → 0.292, reported as a usage statistic.
  • Budget and usage: Lowering the matched-width budget from k = 256 to k = 64 raises the dead-feature rate to about half the dictionary and degrades rowEV from 0.80 to 0.68 and KL from 0.23 to 0.37.At k = 32, about two thirds of the dictionary is stranded, and the same progression appears in Qwen3.5-0.8B.
  • Transfer and cross-family comparison: Across native Qwen settings, additional capacity generally improves both rowEV and top-1 agreement, while Ministral and R1 rows show that row reconstruction and decision fidelity can separate across families.The cross-family comparisons motivate treating rowEV, top-1, KL, and usage as distinct evaluation quantities.

E.4 Readout Row-Norm Tail Analysis

Reasoning-distilled checkpoints have wider and heavier-tailed readout row-norm distributions than base models. This helps explain why uniform row reconstruction can remain strong while decision-level fidelity separates.

  • Row-norm distributions: Distilled checkpoints have heavier upper row-norm tails, with max/median 1.7–2.0 versus 1.5–1.6 for base-model rows.R1-Distill-Llama-8B has the most extreme upper tail.
  • Held-out contrasts: Held-out contrastive reconstruction on R1-distilled checkpoints keeps sign agreement near the five-model baseline, but relative-error coverage is lower under the stricter setting.The comparison uses the same contrast-bank version as the five-model run.
  • Row-norm distributions: The lower tail is compressed in distilled rows, with minimum norms rising from 0.25–0.38 on base models to 0.42–0.57.The coefficient of variation also rises from 0.12 on base models to 0.14–0.16 on distilled models.
  • Interpretation: The wider norm distribution is consistent with preserved row reconstruction but separated decision fidelity, because high-norm rows disproportionately affect top-1 and KL after softmax.The analysis motivates norm-aware SAE recipes as a natural extension.

F.1 Additional Metric Analyses

Additional analyses identify support selection, rather than coefficient fitting or low-rank structure, as the main remaining reconstruction bottleneck. They also show that row-level reconstruction and readout-level fidelity can favor different dictionary choices, with substantial implementation and replication requirements.

  • Support selection: Support selection contributes about 0.026 rowEV of headroom, whereas coefficient refitting contributes only about 0.004 on the Qwen3.5-9B k = 128 recipe.Dense rank-128 reconstruction reaches only 0.101, far below the sparse dictionary.
  • Support selection: +0.038 rowEV comes from support selection in the 32×, k = 256 run, compared with +0.006 from coefficient refitting.The sparse code is about 5.9× better than the same-rank dense reference by unexplained-variance reduction.
  • Support selection: Across four smaller models, changing the support improves reconstruction much more than refitting coefficients on the encoder’s existing support.This identifies support selection as the dominant recipe-side bottleneck.
  • Metric mismatch: The D = 65536, k = 256 dictionary truncated to k = 128 has higher rowEV than the fixed-k = 128 recipe, but top-1 and KL favor the fixed-k = 128 recipe.The comparison reports 0.695 versus 0.621 rowEV, while top-1/KL are 0.721/0.469 versus 0.846/0.296.
  • Alternative recipes and controls: Plain TopK exceeds matched matryoshka variants in rowEV at lower cost, while dense rank-matched references and null controls perform substantially worse.The support-selection comparisons include signed-OMP ceilings and shuffled or random-support controls.
  • Reproducibility: Exact replication depends on matching checkpoint revisions, tokenizers, normalization and softcap paths, score filters, and single-token eligibility rules.The authors report explicit reconstruction error and sign agreement for local score decompositions and release code, checkpoints, banks, and audit protocols.

G.2 Metric Definitions and Fidelity Results

The fidelity analyses evaluate reconstruction at both the selected-score and full-contrast levels, using sign agreement, relative error, coverage, and error distributions. SRP consistently outperforms direct-geometry alternatives, while errors and sign disagreements concentrate near zero exact margins.

  • Selected-score fidelity: Across eight k = 256 readout SAEs, SRP preserves contrast sign for 0.914–0.958 of selected scores, with median ρ0.5 = 0.068–0.255.These results use cluster-bootstrap 95% intervals on selected readout scores.
  • Selected-score fidelity: Qwen3.5-0.8B/2B/9B settings achieve sign agreement 0.940–0.951 and ρ0.5 < 0.5 fractions 0.790–0.818.For confident Qwen margins with |mexact| ≥2, top-1 agreement rises to 0.97–1.00.
  • Baseline comparisons: SRP reaches coverage 0.774 with median ρ0 = 0.081 on Qwen3.5-2B, versus 0.622 and 0.260 for nearest row ridge.PCA-256 falls to 0.272 at matched realized compactness, while the redesigned harness reports 0.780 for SRP.
  • Baseline comparisons: The matched-capacity k-means centroid dictionary still trails SRP’s coverage on every model.The primary comparison uses methods built directly from WU geometry and identical score banks and metrics.
  • Error distributions: SRP has the lowest mean, 95th-percentile, and maximum absolute error among the seven methods.Per-model SRP mean absolute error is 0.43–1.22 logits, compared with 1.16–2.95 for nearest-row ridge.
  • Margin stratification: Relative-error failures concentrate among the 14% of contrasts with |mexact| < 0.5, while the 61% with |mexact| ≥2 achieve sign agreement 0.996 and coverage 0.966.Raising the minimum exact margin improves sign agreement but reduces retained coverage.

H Sense-Labelled Evaluation

The evaluation tests whether sparse readout features align with labelled word senses and whether their explanations reproduce across independently trained dictionaries. Selected feature groups achieve strong held-out sense recovery, while decomposition structure remains more reproducible than individual feature directions.

  • Sense alignment: 0.90, 0.92, and 0.80 balanced accuracy recover held-out senses, versus a permutation null near 0.41.Every word on Qwen3.5-2B, 19 of 20 on Qwen3.5-9B, and 12 of 13 on R1-Llama-8B exceed the null’s 95th percentile.
  • Sparsity ladder: The full decoded state reaches 0.96, 0.97, and 0.92 nearest-centroid accuracy, while the full 256-dimensional account reaches 0.95, 0.97, and 0.89.Compressing to eight features costs 5 to 9 points against the full account and 5 to 12 points against the full state.
  • Alternative framing: Per-word classifiers using unweighted projections outperform signed contributions at 0.87 against 0.84, 0.90 against 0.76, and 0.73 against 0.67.The factorization makes coefficient weighting a constant per-coordinate rescaling for a fixed word, while signed terms identify directions supporting a selected score.
  • Reproducibility: About 90% of used feature groups have above-chance counterparts in independently trained dictionaries, and roughly 40% have near-exact single matches.A median group is 83–90% covered by at most three features, indicating feature splitting rather than wholly unstable grouping.
  • Reproducibility: Held-out SRP recovers 72–75% of the stable grouping core, compared with 43–49% for kNN and 21–25% for clustering.Individual decoder directions do not recur across seeds, but tested contrast explanations reproduce above the null’s 90th percentile at approximately 15× the null level.
  • Ablation validation: Feature ablations produce realized local readout-side score changes that agree with predicted contributions at r2 = 0.83–0.93, versus random-control r2 ≤0.06.The validated claim is local to a realized forward pass; circuit-level necessity, sufficiency, and generation-level effects require separate interventions.

K Cross-Lens Study Protocol, Controls, and Per-Family Results

The cross-lens study compares lenses differing in fitting corpus or construction while decomposing their readings in a shared readout-only basis. Across English–Chinese and English–German settings, token-level readings vary, but dominant SRP features are substantially more stable.

  • Cross-lens results: The same SRP feature dominates under English and Chinese lenses on 77 of 80 cross-lens prompts (96%, CI [0.90, 0.99]).Both unrelated-token and shuffled-pairing null floors are empty, at 0/159 and 0/240 matches.
  • Per-family results: The English and Chinese lenses report different scripts on 9/10 factual-recall prompts, while the dominant feature agrees on 9/10.The worked antonym example shows both surface forms carried by shared feature f112 across comparison layers.
  • Per-family results: The English–German bank reaches dominant-feature agreement on 84/90 prompts (0.93, CI [0.86, 0.97]), against unrelated-token and shuffled-pairing floors of 1/204 and 0/270.This second language pair removes the script confound because both languages use the Latin script.
  • Worked examples: For the multilingual antonym example, groß is carried by f6764, large by f12474, and big and 大 share f112.The contrast comes from LM-head row codes alone in this worked example.
  • Construction and corpus effects: Across lens constructions, the same dominant feature appears on 76/80 prompts when the fitting corpus is fixed, while corpus changes still alter surface readings.Under the ridge family, the corpus-conditionality pattern reproduces and the same SRP feature remains dominant on 67/80 prompts.
  • Caveats and scope: The study covers one model, two language pairs, two fitted-lens constructions, and two fitting scales, with Chinese and German fitting corpora containing incidental English text.The incidental English makes token-level divergence conservative, and the ridge translator is a minimal tuned-lens instantiation.

L.7 What SRP Adds Beyond the Logit Lens

SRP supplements token-level logit-lens readings by resolving local contrasts into signed readout-feature contributions. Case studies, label audits, and direct-attribution checks show that these terms expose semantic structure while preserving additive score accounting.

  • Domain contexts: Shell prompts share a shell-related direction, while Java prompts separate beverage-related directions from programming-framework and language directions.Each label lists the top associated unembedding rows for its feature.
  • Prompt-injection case: The ordinary lens can favor an undesirable action, while SRP separates deletion-related features from summary and communication features.In the prompt-injection example, deletion terms support delete and positive terms support summarize.
  • Feature-label audit: 85 of 99 claim-bearing labels name coherent lexical families, with 66 of those 85 also matching the panel’s named sense and side.The remaining 14 labels are token-form artifacts, and no on-target label opposes the prose’s claimed side.
  • Direct attribution: SRP terms form an additive identity within ordinary direct logit attribution for pairwise and vocabulary-mean contrasts.The decomposition expresses a selected readout score as feature contributions plus residual terms over a realized forward pass.
  • Direct attribution: The verify-minus-assume margin is +4.31 exactly, compared with a +4.36 component-attribution sum and +4.28 sparse-feature sum.Verification and checking features provide positive mass concentrated in later layers against a smaller assumption-family group.

N Constrained Readout-Side Edit Test

The constrained readout-side test edits SAE decoder directions selected from discovery tokens and evaluates their effects on intended and held-out lexical logit differences. On Qwen3.5-2B, SRP outperforms label-free baselines at matched KL, but this advantage is model-specific.

  • Test design: SAE decoder-direction edits move constrained lexical logit differences and are evaluated with held-out terms, scale sweeps, comparison rows, and off-target probes.The test differs from state-side feature ablation by editing the readout directly.
  • Test design: The intervention selects directions from five discovery terms and evaluates them on ten held-out lexical terms using a one-step logit edit.The primary grid uses 16 prompts, 10 intervention scales, and five methods, producing 7200 evaluated rows.
  • Feature specificity: 12 of the top 13 profanity-associated SAE features have specificity 1.0 against 20 control terms, while the remaining feature reaches 0.988.The selected directions are organized into recognizable lexical families, including f-word, sh-word, hell/damn, and insult-related directions.
  • Intervention outcome: At scale 16, the readout SAE direction edit matches the oracle token bias held-out flip rate, while discovery token bias has zero held-out effect.The match occurs at saturation; at scale 4, the oracle flips 20 of 23 held-out candidates versus 11 for the SAE edit.
  • Cross-model outcome: The advantage is model-specific: SRP loses to the mean-row direction on Qwen3.5-0.8B and Qwen3.5-9B, while R1-Distill-Qwen-7B is mixed.The supported claim is therefore scoped to the models and matched-KL comparisons tested.
Loading 2609.01936v1…