Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models

• SAEs learn sparse feature directions that approximately reconstruct model activations, offering a compact vocabulary for describing information in hidden states.The paper argues that these features should move beyond post-hoc inspection into practical model-development workflows.Source: §1 · p. 3
• Qwen-Scope releases 14 SAE groups across 7 Qwen3 and Qwen3.5 variants, covering dense and mixture-of-experts architectures.The open-source suite is intended to support community-driven interpretability research on Qwen models.Source: §1 · p. 3
• Feature-level interventions steer language, concepts, and preferences without modifying model weights.The introduction frames steering as an application of SAE features demonstrated in Qwen-Scope.Source: §1 · p. 3
• Feature coverage serves as a proxy for full-scale benchmarking and supports analysis of redundancy and representational concentration across evaluation sets.Source: §1 · p. 3
• SAE features support multilingual toxicity classification and safety-oriented data synthesis.Source: §1 · p. 3
• In SFT, an auxiliary loss suppresses language-specific activations to reduce code-switching; in RL, repetition features generate rare negative rollouts as training signals against endless repetition.Together, the applications position SAE features as a reusable interface for diagnosing behavior, steering outputs, analyzing evaluation data, constructing data, and improving post-training.Source: §1 · p. 3

2.2 Training in Practice

• The release includes 14 SAE groups across 7 Qwen backbones, covering all layers of dense and MoE models; Qwen3.5-27B uses the instruct variant, while the others use base models.For MoE models, wider SAEs are also released to capture finer-grained features.Source: Table 1 · p. 4, §2.2 · p. 4
• Each SAE reconstructs residual-stream activations for one transformer layer of one backbone, using an overcomplete latent representation and Top-k values of 50 or 100.SAEs are trained on activations sampled from in-house pretraining data.Source: §2.2 · p. 4
• Dense SAE widths scale with model hidden size, while MoE releases include widths up to 64× hidden size.The wider MoE SAEs are intended to capture more fine-grained representation structure.Source: §2.2 · p. 4, Table 1 · p. 4
• An auxiliary loss with weight 1/32 reduces dead features, and almost all released SAEs have a negligible number by the end of training.The training setup also filters extremely large-L2-norm activations, which occur most often in Qwen3-1.7B and Qwen3-8B, especially at the first input token.Source: §2.2 · p. 4

2.1 Why Sparse Auto-Encoders?

• The resulting layer-wise SAE feature dictionaries are reused for steering, evaluation analysis, data-centric workflows, and post-training applications.The toolkit builds on sparse latent representations that activate only for a narrow subset of inputs, supporting model analysis and intervention.Source: §2.2 · p. 4, §2.1 · p. 4

3.1 What is Steering?

• Steering modifies a residual-stream hidden state as h′ ←h + αd, where d is an SAE feature direction and α amplifies it when positive or suppresses it when negative.The model then continues its forward pass using the modified representation.Source: §3.1 · p. 5

3.2 How to Identify Features for Steering

• Contrastive identification compares SAE feature activations across positive and negative examples, while automatic interpretation summarizes strongly activating contexts with a language model to describe features for downstream steering.Figure 2 illustrates the contrastive identification and feature-injection steering pipeline.Source: §3.2 · pp. 5–6, Figure 2 · p. 5

3.3 Case Studies of SAE Steering

• Suppressing a highly activated Chinese-language feature removed unintended Chinese text from an English response, while activating a classical-Chinese feature shifted modern Chinese continuation toward classical style while preserving prompt semantics.The case studies use SAE activations to diagnose the language-mixing failure and feature intervention to correct it or control style.Source: §3.3 · p. 6, Figure 3 · p. 6

4.1 SAE Feature Extraction

• For each sample, the SAE latent representation at the last token position defines its active features, whose benchmark-wide footprint represents what the benchmark probes.The encoder’s Top-k ReLU activation is implicit in the latent representation.Source: §4.1 · p. 7

4.2 Benchmark Redundancy

• Performance-based redundancy measures how well a subset preserves model rankings using expected Kendall’s τ across subset sizes; its area under the curve is higher when fewer samples suffice.Computing this measure requires evaluating models on the full benchmark and many sampled subsets.Source: §4.2 · p. 8
• The evaluation-free proxy instead measures how quickly activated features saturate across random subsets, adjusting coverage-curve area by N/|F(D)| to account for total feature diversity.The correction distinguishes benchmarks with the same normalized coverage curve but different numbers of distinct activated features.Source: §4.2 · pp. 8–9
• Across 17 benchmarks, Spearman rank correlation between performance-based redundancy R(D) and feature redundancy R̂(D) is ρ ≈0.85, supporting the latter as an evaluation-free proxy.High feature redundancy suggests that fewer samples may preserve model rankings, while low redundancy may call for retaining or collecting more data.Source: Figure 5 · p. 9, §4.2 · p. 9

4.3 Inter-Benchmark Similarity Analysis

• Asymmetric overlap shows capability containment: 63% of GSM8K’s features are covered by MATH, while code benchmarks cluster and MMLU-Pro and SuperGPQA cover 0.56–0.68 of TheoremQA.The reverse GSM8K–MATH overlap is 0.10, reflecting that MATH probes a broader feature set.Source: Figure 6 · p. 10, §4.3 · p. 10
• After controlling for general ability with MMLU, partial Pearson correlation is 75.5%, compared with direct overlap–performance correlations of 68.4% (Pearson) and 60.7% (Spearman).The correction addresses the confounding effect of general model ability on performance correlations.Source: §4.3 · pp. 10–11, Table 2 · p. 11
• Evaluation suites can retain low-overlap benchmarks, consolidate high-overlap ones, and consider benchmarks with low overlap against the suite as candidates to fill capability gaps.The authors note that high redundancy can also be desirable for reducing evaluation variance or ensuring broad within-domain coverage.Source: §4.2 · pp. 9–10, §4.3 · p. 11

5 Application: Data Classification

• The pipeline discovers features by comparing their firing rates on toxic versus clean examples in a fixed selection split, then uses them directly to classify held-out data.Source: Figure 7 · p. 12
• A small set of toxicity-biased SAE features yields a strong rule-based classifier without an added supervised head or gradient-based fitting after the SAE is fixed.This demonstrates their use as actionable prediction variables rather than only tools for post-hoc analysis.Source: §5 · p. 12
• Toxicity-related SAE directions include features shared across languages and features that transfer from English to other languages, while layer selection and less discovery data improve efficiency.Source: §5 · p. 12

5.1 SAE-Based Toxicity Classifier

• The method first discovers toxic SAE features on a fixed selection split, then applies them directly as detectors on held-out data, keeping predictions sparse and interpretable.It does not require interpreting features in advance, and each positive prediction can be linked to a small set of features and their layer.Source: §5.1 · p. 12

5.1.1 Toxic Feature Discovery

• Across 13 languages, toxic features are selected per layer by their difference in firing frequency on toxic versus clean examples, using 4,000 discovery and 1,000 evaluation examples per language.The balanced split contains 2,000 toxic and 2,000 clean discovery examples, plus 500 of each class for evaluation; selected features retain interpretable toxic and clean firing frequencies.Source: §5.1.1 · pp. 12–13

5.1.2 Rule-Based Classification with Selected Features

• The classifier predicts toxicity when any selected feature fires at any token position, without training a classifier head or learning feature weights after selection.This OR-rule makes positive predictions traceable to the feature, layer, and token position that triggered them.Source: §5.1 · p. 12, §5.1.2 · p. 13
• English held-out F1 exceeds 0.90 for both Qwen3 models, with strongest performance in middle-to-late layers and limited benefit from increasing K beyond a small value.The results indicate that toxicity is captured by a sparse set of selective latent features.Source: §5.1.2 · p. 13, Figure 8 · p. 13
• Toxic SAE features show structured sharing across languages, and English-discovered features transfer well to many languages, especially in larger models.Figure 9 reports feature overlap and held-out transfer F1 across languages.Source: Figure 9 · p. 14

5.2.1 Shared Toxic Structure Across Languages

• Feature overlap is highest among typologically closer languages, especially European pairs, and strongest in middle layers across both Qwen3-1.7B and Qwen3-8B.More distant language pairs have substantially weaker overlap; Qwen3-8B shows somewhat stronger and more stable overlap overall.Source: §5.2.1 · p. 14

5.2.2 Transfer of English-Discovered Features

• top1-diff is a reliable proxy for selecting evaluation-free layers while retaining nearly all achievable performance.Figure 10 compares held-out F1 over layers with the layer selected by the strongest discovered feature for Qwen3-1.7B and Qwen3-8B across English, Russian, French, and Chinese.Source: Figure 10 · p. 15
• English-discovered features transfer strongly to English and several European languages, including Russian and French, but less well to Arabic, Chinese, and especially Amharic.The test applies English-discovered features directly to held-out data in other languages, without rediscovering features.Source: §5.2.2 · pp. 14–15
• Scaling to Qwen3-8B improves the level and stability of cross-lingual transfer, while shifting optimal layers deeper.The results suggest English-discovered detectors can be effective starting points for multilingual detection, particularly in larger models.Source: §5.2.2 · p. 15

5.3.1 Layer Selection and Multi-Layer Composition

• Across languages and both model sizes, the layer selected by top1-diff is usually the best or very close, avoiding much of the cost of a full held-out layer sweep.top1-diff is the strongest toxic-clean frequency gap in a layer and serves as an evaluation-free proxy for layer quality.Source: §5.3.1 · p. 15

5.3.2 Data Efficiency of Feature Discovery

• Using only 10% of the original toxic-feature data achieves 99% of original classification performance across 13 languages for both Qwen3-1.7B and Qwen3-8B.As the discovery budget grows, overlap with the full-data feature set rises quickly, suggesting the most stable toxic-biased features are found early.Source: Figure 12 · p. 16, §5.3.2 · p. 16

5.3.1 Layer Selection and Multi-Layer Composition

• Multi-layer composition is most useful when single-layer evidence is weak, improving harder cases by combining a small number of top-ranked layers.The classifier keeps only the best feature from each selected layer, providing targeted robustness while remaining sparse and inspectable.Source: §5.3.1 · pp. 15–16, Figure 11 · p. 16

5.3.2 Data Efficiency of Feature Discovery

• Once a good general SAE is available, a small labeled set can identify toxic features and build an effective, interpretable detector.Classification performance remains close to the full-data baseline even with a much smaller feature-discovery set.Source: §5.3.2 · p. 16

6.1 Feature-Driven Safety Data Synthesis

• The approach shifts data construction from prompt categories to representation-level targets: identify missing or weak safety features, generate examples from their descriptions, and retain examples that activate the intended directions.It aims to reinforce concepts the model already represents but has not yet turned into reliable post-training behavior, rather than recreating the full pretraining distribution.Source: §6.1 · p. 17, §6 · p. 17

6.1.1 Target Feature Discovery

• Target selection uses seed-corpus feature coverage to prioritize safety-relevant features, with a judge-model semantic filter defining eligibility and uncovered features prioritized first.Coverage is a coarse feature-space support estimate: it records whether a feature is reached at least once, not its frequency or activation strength; weakly covered features can also be targets.Source: §6.1.1 · pp. 17–18

6.1.2 Data Synthesis from Feature Descriptions

• For each target feature, the pipeline creates vanilla and adversarial prompts, pairs harmful requests with refusals and benign requests with helpful answers, and retains examples only when they activate that feature.Feature descriptions guide generation, while representation-level verification—not text relevance alone—determines retention.Source: §6.1.2 · pp. 18–19

6.2 Toward Controllable Safety Post-Training

• Evaluation tests whether feature-driven synthesis improves target-feature coverage and whether those gains translate into a better safety–utility tradeoff after SFT.The experiments assess coverage efficiency against natural sampling and unconstrained safety-related synthesis, then evaluate downstream SFT behavior.Source: §6.2 · p. 19

6.2.1 Training and Evaluation Setup

• Experiments use Qwen3-8B with a layer-30 residual-stream SAE of approximately 65k latents, WildJailbreak data, and LoRA SFT mixing Alpaca, real safety, and synthetic safety examples.The key downstream comparison fixes the total safety-data budget and replaces random synthetic examples with feature-driven ones.Source: §6.2.1 · pp. 19–20

6.2.2 Coverage Efficiency of Feature-Driven Synthesis

• At the matched budget, feature-driven synthesis reached 99.74% target-feature coverage, nearly saturating the inventory while natural sampling and random synthesis left substantial gaps.Coverage is measured over target features activated by at least one retained synthetic example.Source: §6.2.2 · p. 20, Figure 14 · p. 20

6.2.3 Results with Synthetic Data

• At an 8k safety-data budget, 4k real plus 4k feature-driven synthetic examples achieved 77.75 safety accuracy versus 71.75 with natural sampling and 72.00 with random synthesis.The feature-driven mixture also improved IFEval, TruthfulQA, MMLU, and GSM8K over random synthesis; the reported robustness check using Gemini-3-Flash produced performance very close to the main setup.Source: §6.2.3 · pp. 20–21, Table 3 · p. 21

7.1 Unexpected Code-Switching

• Unexpected code-switching is unintended-language generation, and standard SFT lacks explicit negative supervision against it because it only encourages matching the target response.The section motivates training-based intervention because inference-time steering does not persistently improve model behavior and may affect unrelated tasks.Source: §7 · p. 22

7.2 Feature Analysis

• Chinese-feature pre-activations rose before and peaked at code-switches across five models, while directional ablation reduced switching more at larger coefficients and irrelevant-feature ablation had negligible effect.This suggests elevated target-language feature activation can precede switching, and the ablation effect is language-specific.Source: §7.2 · p. 23

7.3 Method

• SASFT ranks SAE features by language monolinguality, then adds a feature-activation regularizer to cross-entropy, suppressing target-language features on other-language data while excluding target-language examples.The regularizer uses each feature’s pre-estimated average pre-activation as its baseline, which avoids using zero when the average is negative.Source: §7.3 · pp. 23–24

7.4 Main Results

• Over 50% reduction in code-switching occurred in most settings, with complete elimination for Qwen3-1.7B on Korean; SASFT maintained or marginally improved performance on six multilingual benchmarks.The evaluation covered five models across Gemma-2, Llama-3.1, and Qwen3, three target languages, and both 210k and 110k dataset settings.Source: Table 4 · p. 24, §7.4 · p. 24, Table 5 · p. 24

8 Application: Reinforcement Learning

• SAE steering proved unsuitable for generating positive rollouts because it could not ensure precise multi-step reasoning and could degrade fluency, motivating its use to induce negative samples instead.Fluency degradation is less consequential for negative samples because the model learns to avoid rather than imitate them.Source: §8 · p. 25

8.1 Feature Analysis

• Repetition-related SAE features rise and remain elevated during endless loops, causally promote repetition under amplification, and also activate during benign repetition.Because the features are shared with normal repetition, suppressing them during training could impair behaviors such as repeating a user’s question or answer choices.Source: §8.1 · pp. 25–26, Figure 20 · p. 26, Figure 21 · p. 27

8.2 Method

• The method adds the repetition feature direction to the residual stream at each generation step; a larger steering coefficient α produces stronger repetition in the generated output.For each rollout group, it adds one steered output expected to repeat alongside the normally sampled outputs.Source: §8.2 · pp. 26–27

Algorithm: SAE-Guided DAPO with Rare Negative Augmentation

• Algorithm 6 samples G − 1 ordinary outputs and one additional SAE-feature-steered output per question using coefficient α, scores all outputs with R, computes token-level advantages, and updates the policy by maximizing DAPO.Source: Algorithm: SAE-Guided DAPO with Rare… · p. 27

8.3 Experimental Setting

• The evaluation compares SAE-guided augmentation with vanilla DAPO on cold-start SFT models at three Qwen3 scales, tracking repetition on held-out prompts and testing standard benchmarks.The benchmark suite includes MMLU, Flores, HellaSwag, LogiQA, IFEval, and MGSM.Source: §8.3 · pp. 27–28

8.4 Main Results

• Across Qwen3-1.7B, Qwen3-8B, and Qwen3-30B-A3, SAE-guided RL reduces repetition faster and further than vanilla RL, while general-benchmark effects remain mixed and task-dependent.The method provides a stronger reduction in repetition but does not uniformly improve general-purpose capability.Source: Figure 22 · p. 28, §8.4 · p. 28, Table 7 · p. 28

9.1 Summary

• Qwen-Scope provides layer-wise SAE features for multiple Qwen3 and Qwen3.5 backbones, covering dense and mixture-of-experts architectures under a unified training pipeline.The authors present it as useful for practical model-development workflows beyond post-hoc interpretation and invite community exploration.Source: §9.1 · p. 29

9.2 Exploring Directions

• Future work can use Qwen-Scope to study SAE features across reasoning branches, identify causally important steps, and track reasoning trajectories under resampling or intervention (Macar et al., 2026; Bogdan et al., 2025).The passage notes that single-forward-pass analysis may be insufficient for long chain-of-thought, multi-step sampling, or latent and vector-space reasoning.Source: §9.2 · p. 29
• Future work can test whether SAE features, combined with probes and activation-based monitors, reveal risks such as deception, hidden objectives, jailbreak susceptibility, and hallucination early and robustly (Goldowsky-Dill et al., 2025; Parrack et al., 2026; Marks et al., 2025).Source: §9.2 · p. 29
• Qwen-Scope can compare model internals before and after fine-tuning, reinforcement learning, or other interventions by tracking feature changes and shifts in activation directions (Minder et al., 2026).This could reveal whether post-training leaves readable traces in activation space.Source: §9.2 · p. 29
• Future work can study how SAE feature interventions affect generalization, robustness, and safety, and how interpretable directions can enter training pipelines (Casademunt et al., 2025).The report describes features as control knobs that can be amplified or suppressed at inference, used as auxiliary SFT signals, or used to construct rare negative examples for RL.Source: §9.2 · p. 29
• SAE features can connect training data to internal feature coverage to identify under-covered behaviors, prioritize examples, guide synthetic data generation, and attribute undesirable behavior to influential data regions (Coalson et al., 2025; Li et al., 2024).Source: §9.2 · p. 29

9.3 Social Impact

• The authors urge researchers and developers not to use Qwen-Scope or Qwen models in ways that violate human ethical values or generate and disseminate harmful information.The passage prohibits misuse of interpretability tools to interfere with model capabilities or produce harmful content.Source: §9.3 · p. 29