PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models

Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, Xiaodong Deng, An Yang, Fei Huang, Dayiheng Liu, Jingren Zhou

arXiv:2605.11887v1cs.CLcs.LG

TL;DR

Large language models remain difficult to inspect and control, while SAE features have mainly been used for post-hoc analysis. Qwen-Scope releases broad Qwen SAE coverage and demonstrates feature-level interfaces for steering, evaluation, data workflows, and post-training, showing their reuse across model-development tasks.

  • Problem

    LLMs are largely opaque, and SAE research has underexplored how interpretable features can support concrete model-development workflows.

  • Method

    Qwen-Scope provides 14 SAE groups across 7 Qwen variants and applies their features to steering, evaluation analysis, data-centric workflows, and post-training.

  • Results

    The demonstrations show that SAEs can serve as reusable representation-level interfaces for diagnosing, controlling, evaluating, and improving language models.

  • Takeaways & Limitations

    SAE features can connect model internals to practical workflows spanning inference control, benchmark analysis, data construction, and post-training.

  • Takeaways & Limitations

    Feature coverage is only a representation-level proxy and does not by itself guarantee improved downstream behavior.

Abstract

from arXiv · show

Large language models have achieved remarkable capabilities across diverse tasks, yet their internal decision-making processes remain largely opaque, limiting our ability to inspect, control, and systematically improve them. This opacity motivates a growing body of research in mechanistic interpretability, with sparse autoencoders (SAEs) emerging as one of the most promising tools for decomposing model activations into sparse, interpretable feature representations. We introduce Qwen-Scope, an open-source suite of SAEs built on the Qwen model family, comprising 14 groups of SAEs across 7 model variants from the Qwen3 and Qwen3.5 series, covering both dense and mixture-of-expert architectures. Built on top of these SAEs, we show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions: (i) inference-time steering, where SAE feature directions control language, concepts, and preferences without modifying model weights; (ii) evaluation analysis, where activated SAE features provide a representation-level proxy for benchmark redundancy and capability coverage; (iii) data-centric workflows, where SAE features support multilingual toxicity classification and safety-oriented data synthesis; and (iv) post-training optimization, where SAE-derived signals are incorporated into supervised fine-tuning and reinforcement learning objectives to mitigate undesirable behaviors such as code-switching and repetition. Together, these results demonstrate that SAEs can serve not only as post-hoc analysis tools, but also as reusable representation-level interfaces for diagnosing, controlling, evaluating, and improving large language models. By open-sourcing Qwen-Scope, we aim to support mechanistic research and accelerate practical workflows that connect model internals to downstream behavior.

1 Introduction

Qwen-Scope addresses the gap between inspecting SAE features and using them to develop models. It releases broad Qwen SAE coverage and demonstrates applications for steering, evaluation, data workflows, and post-training.

  • Qwen-Scope releases 14 SAE groups across 7 Qwen3 and Qwen3.5 variants, covering dense and mixture-of-experts architectures.
  • Steering: SAEs can influence language, concepts, and preferences through feature-level interventions without modifying model weights.
  • Evaluation: Feature coverage serves as a representation-level proxy for studying benchmark redundancy and representational concentration across evaluation sets.
  • Data-centric workflows: SAE features support multilingual toxicity classification and safety-oriented data synthesis.
  • Post-training: SAE-derived signals guide supervised fine-tuning and reinforcement learning to reduce code-switching and endless repetition.
  • Together, the applications position SAEs as reusable representation-level interfaces for diagnosing, controlling, evaluating, and improving language models.

2 Training in Practice

Qwen-Scope trains layer-specific sparse autoencoders across Qwen dense and MoE backbones. The resulting feature dictionaries are reused throughout the report for intervention and analysis.

  • Qwen-Scope releases 14 SAE groups across 7 Qwen backbones, with SAEs trained for every transformer layer.
  • Each SAE reconstructs residual-stream activations as sparse latent features for a specific model layer, enabling intervention on feature activations rather than raw hidden states.
  • Top-k activation retains the largest 50 or 100 latent activations, while MoE models additionally receive wider SAEs of up to 64× the hidden size.
  • Auxiliary loss and large-norm activation filtering are used to stabilize training and reduce dead features.
  • The training setup yields layer-wise SAE dictionaries reused for steering, evaluation analysis, data-centric workflows, and post-training applications.

3 Application: Steering with SAEs during Inference

SAE steering identifies behavior-related feature directions and injects them into hidden states during inference. Qwen3 case studies show both suppression of unwanted language mixing and controlled stylistic generation.

  • Contrastive steering compares SAE activations on positive and negative examples to identify discriminative feature directions before intervention.
  • What is Steering?: Feature steering modifies a hidden state as h′ ← h + αd, where positive α amplifies and negative α suppresses the selected SAE direction.
  • Case studies: Suppressing a highly activated Chinese-language feature removes unexpected Chinese text from an English response while preserving the intended response.
  • Case studies: Activating a classical-Chinese feature shifts modern Chinese continuation toward a classical literary style while preserving the prompt’s semantic direction.
  • These interventions provide an interpretable mechanism for debugging and controllable generation without updating model weights.

4 Application: Evaluation

Qwen-Scope uses SAE-derived feature representations to analyze benchmark redundancy and capability overlap without relying exclusively on expensive model evaluations. The framework estimates redundancy through feature-coverage saturation and compares benchmarks through asymmetric and symmetric feature overlap.

  • Framework: The framework extracts SAE feature footprints, measures intra-benchmark redundancy through coverage saturation, and analyzes inter-benchmark similarity through feature overlap.A benchmark is considered redundant when many samples activate the same features, causing coverage to saturate early.
  • Benchmark Redundancy: Performance-based redundancy requires evaluating many models on sampled subsets, motivating a model-evaluation-free feature-based proxy.The direct approach scales with M × N forward passes, whereas SAE features provide a compact representation of what each benchmark probes.
  • Benchmark Redundancy: Spearman ρ ≈0.85 across 17 benchmarks suggests feature redundancy is a reasonable evaluation-free proxy for performance-based redundancy.The correlation holds across benchmarks with substantially different sizes, including GSM8K and SuperGPQA.
  • Benchmark Redundancy: High feature redundancy can support cheaper iterative model ranking, while low redundancy may require retaining or collecting more evaluation data.The authors frame this as a trade-off between reduced evaluation cost and a modest amount of reliability, depending on practitioner priorities.
  • Inter-Benchmark Similarity Analysis: Asymmetric overlap captures directional capability containment: GSM8K features are 63% covered by MATH, whereas MATH features are only 10% covered by GSM8K.The overlap matrix also groups code benchmarks together and shows broader knowledge benchmarks subsuming specialized ones.
  • Inter-Benchmark Similarity Analysis: After controlling for general model ability, partial Pearson correlation between feature overlap and performance-based similarity reaches 75.5%.The correction addresses a rising-tide confound in which broadly improving models inflate performance correlations between benchmarks with limited shared capability.
  • Inter-Benchmark Similarity Analysis: Feature overlap can guide evaluation-suite design by retaining low-overlap benchmarks and considering high-overlap benchmarks for consolidation.This use case is intended to identify distinct capability coverage without running model evaluations.

5 Application: Data Classification

This section uses sparse autoencoder features as a direct, interpretable interface for multilingual toxicity classification. The approach discovers toxic-biased features and applies them as a rule-based classifier without an additional classification head.

  • 5 Application: Data Classification: The pipeline discovers toxic SAE features on a selection split and applies them directly as a rule-based classifier on held-out data.Each prediction can be traced to a feature, layer, and token position.
  • 5 Application: Data Classification: A small set of toxicity-biased SAE features yields a strong classifier without additional supervised-head training or post-SAE gradient fitting.The features also support cross-lingual analysis and efficiency improvements.

5.1 SAE-Based Toxicity Classifier

The classifier selects SAE features that fire more often on toxic than clean examples, then predicts toxicity when any selected feature activates. This produces a sparse, interpretable detector with strong English performance and traceable decisions.

  • 5.1 SAE-Based Toxicity Classifier: The experiments evaluate Qwen3-1.7B and Qwen3-8B across 13 languages using their corresponding SAE feature dictionaries.Feature discovery is performed independently at each transformer layer from residual-stream activations.
  • 5.1 SAE-Based Toxicity Classifier: The method ranks features by the difference between toxic and clean firing frequencies and classifies an example when any selected feature fires.No additional classifier head or post-selection feature weighting is learned.
  • 5.1 SAE-Based Toxicity Classifier: Held-out English F1 exceeds 0.90 for both Qwen3-1.7B and Qwen3-8B using only a small set of toxic-biased SAE features.Increasing the feature count beyond a small value provides limited additional benefit.
  • 5.1 SAE-Based Toxicity Classifier: The resulting predictions remain locally interpretable because each positive decision traces to a selected feature, layer, and token position.The classifier is sparse and discrete rather than a dense learned boundary.

5.2 Cross-Lingual Generalization of Toxic Features

Toxicity-related SAE features share some cross-lingual structure, especially in middle layers, but overlap and transfer vary with linguistic distance. English-discovered features transfer well to some languages and less effectively to more distant ones.

  • 5.2 Cross-Lingual Generalization of Toxic Features: The findings support a layered structure in which toxicity features are neither fully language-agnostic nor purely language-specific.Some shared directions are sufficient to support meaningful cross-lingual transfer.
  • 5.2 Cross-Lingual Generalization of Toxic Features: Feature overlap is highest among typologically closer languages and weaker for more distant pairs, indicating that toxicity representations are not fully language-agnostic.The comparison uses overlap among independently discovered top toxic-feature sets.
  • 5.2 Cross-Lingual Generalization of Toxic Features: Shared toxic structure is most pronounced in middle layers across both model sizes, with Qwen3-8B showing somewhat stronger and more stable overlap.These layers provide the clearest substrate for multilingual toxic-feature discovery.
  • 5.2 Cross-Lingual Generalization of Toxic Features: Cross-lingual transfer is useful but uneven: English-discovered features transfer strongly to several European languages while Arabic, Chinese, and Amharic remain harder.Performance declines with linguistic distance rather than transferring uniformly.

5.3 Toward Efficient and Practical Classification

The section develops practical ways to reduce the cost of SAE-based toxicity classification. A feature-gap statistic usually identifies a near-best layer, while selective multi-layer composition and smaller discovery sets preserve useful performance.

  • 5.3.1 Layer Selection and Multi-Layer Composition: The top1-diff layer is usually the best or nearly best evaluation layer across languages and both model sizes.This enables evaluation-free layer selection before a full held-out sweep.
  • 5.3.2 Data Efficiency: Using only 10% of the original toxic-feature discovery data achieves 99% of the original classification performance for both models.The result is reported as macro-average best F1 across 13 languages.
  • 5.3.1 Layer Selection and Multi-Layer Composition: Multi-layer composition is most useful when single-layer evidence is weak, improving harder cases while retaining a sparse classifier.Layers are ranked by top1-diff and only a small number of top-ranked layers are combined.
  • 5.3.2 Data Efficiency: The results motivate direct SAE-feature classification because a small labeled discovery set can identify stable toxic-biased features without training an additional classifier.Feature overlap with the full-data set rises quickly as the discovery budget grows.

6 Application: Data Synthesis

The paper uses SAE features to identify under-covered safety directions, synthesize targeted examples, and select data by representation-level activation. This feature-driven approach improves coverage and downstream safety–utility tradeoffs under fixed budgets.

  • Motivation: The method targets concepts the model already represents but has not converted into reliable safety behavior, rather than recreating the full pretraining distribution.This addresses long-tail safety situations that natural sampling may cover inefficiently or noisily.
  • Data Synthesis from Feature Descriptions: The pipeline identifies missing safety-relevant SAE features, generates examples from their descriptions, and retains examples that activate the intended features.This moves data construction from corpus-level sampling to representation-level targeting and verification.
  • Data Synthesis from Feature Descriptions: Feature coverage is a coarse support estimate: it records whether a feature is reached at least once, not activation frequency or strength.The proxy therefore does not by itself guarantee improved downstream behavior; that relationship requires post-training evaluation.
  • Coverage Efficiency of Feature-Driven Synthesis: 99.74% target-feature coverage under a matched budget, nearly saturating the target set unlike natural or random safety-related synthesis.Feature-driven synthesis explicitly targets missing internal directions and verifies that generated examples activate them.
  • Results with Synthetic Data: 77.75 safety accuracy versus 71.75 for natural sampling at the same 8k budget, while also achieving the strongest IFEval and TruthfulQA scores.The 8k mixture contains 4k real safety examples and 4k feature-driven synthetic examples.
  • Results with Synthetic Data: Replacing 4k random synthetic examples with 4k feature-driven examples raises safety accuracy from 72.00 to 77.75 and improves IFEval, TruthfulQA, MMLU, and GSM8K.The matched comparison attributes the gain to targeted supervision rather than synthetic data alone.

7 Application: Supervised Fine-tuning

The paper analyzes unexpected code-switching through language-specific SAE features and uses those features to guide supervised fine-tuning. SASFT reduces code-switching across models and languages while preserving multilingual capabilities.

  • Motivation: Unexpected code-switching is a low-frequency multilingual failure in which the model generates text in an unintended language.Standard SFT lacks an explicit negative signal against this behavior because it primarily matches the target response.
  • Mechanistic Analysis: Language-feature pre-activations rise before code-switching, and directional ablation reduces the code-switching ratio more as the ablation coefficient increases.Ablating irrelevant language features has negligible effect, supporting the language-specificity of the identified directions.
  • Motivation: Inference-time feature ablation can mitigate code-switching but requires intervention during decoding, motivating parameter-level suppression through SASFT.SASFT internalizes the intervention during training rather than applying it externally at each decoding step.
  • Method: SASFT identifies language-specific SAE features and adds an auxiliary suppression loss alongside standard cross-entropy during training.A monolinguality score ranks candidate features, while the regularizer reduces their activations on training residual streams.
  • Main Results: SASFT reduces code-switching by over 50% in most settings and completely eliminates it in some configurations, while maintaining or improving multilingual benchmark performance.The evaluation covers five models from Gemma-2, Llama-3.1, and Qwen3 across Chinese, Russian, and Korean targets.

8 Application: Reinforcement Learning

Qwen-Scope uses SAE features to diagnose repetition, induce targeted negative rollouts, and improve online RL. SAE-guided rare-negative augmentation reduces repetition more effectively than vanilla RL, while downstream capability effects remain mixed and task-dependent.

  • 8.2 Method: An initial attempt to generate higher-quality positive rollouts failed because steering alone could harm reasoning accuracy and fluency.The method consequently shifted toward generating negative samples, where fluency degradation is less consequential.
  • 8.2 Method: Because standard online RL rarely encounters endless repetition, the method injects SAE-steered repetitive rollouts as explicit negative training signals.The approach uses DAPO without Dynamic Sampling and adds one steered negative sample to each rollout group.
  • 8.1 Feature Analysis: Bidirectional steering establishes that repetition features are causally linked to endless repetition: suppression reduces repetition, while amplification induces it.This relationship is demonstrated across repetitive and normal samples.
  • 8.1 Feature Analysis: The same repetition features also activate during benign repetition, so suppressing them directly could impair normal behaviors such as repeating instructions or answer choices.The features therefore capture a broader notion of repetition rather than only pathological output collapse.
  • 8.4 Main Results: SAE-guided rare-negative augmentation consistently reduces repetition more effectively than vanilla RL across Qwen3-1.7B, Qwen3-8B, and Qwen3-30B-A3.The repeat ratio drops sharply early in training and continues toward a very low level, unlike vanilla RL.
  • 8.4 Main Results: Table 7 shows broadly competitive general capability after SAE-guided RL, but downstream effects are mixed and task-dependent rather than uniformly improved.Some benchmarks show small gains, while others regress relative to vanilla RL or the pre-RL model.

9 Conclusion

The conclusion presents Qwen-Scope as an open-source SAE foundation for practical model development and outlines future directions for connecting interpretability with control, auditing, data workflows, and post-training. It also acknowledges that current interpretability research lacks sufficient safeguards against misuse.

  • 9 Conclusion: Qwen-Scope provides layer-wise SAE features for multiple Qwen3 and Qwen3.5 dense and mixture-of-experts backbones under a unified training pipeline.The released modules support community-driven exploration of Qwen model internals and applications.
  • 9 Conclusion: The report presents SAE features as reusable interfaces for diagnosis, control, evaluation, data construction, and post-training improvement.The conclusion frames these applications as practical uses beyond post-hoc interpretation.
  • 9.1 Future Directions: Future work includes reasoning-model interpretability, internals-based monitoring, model diffing, interpretability-driven training, and data-centric feature coverage.These directions target internal reasoning trajectories, hidden risks, post-training changes, training controls, and under-covered behaviors.
  • 9.3 Social Impact: The authors acknowledge that current interpretability research does not yet provide sufficient safeguards against misuse.The paper states that Qwen-Scope and Qwen models should not be applied to harmful or unethical uses.

Authors

The paper lists core contributors and additional contributors.

  • Authors: Core contributors are Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, and Baosong Yang.
  • Authors: Additional contributors are Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, An Yang, Fei Huang, Dayiheng Liu, and Jingren Zhou.
Loading 2605.11887v1…