PaperlayerView exact revision on arXiv

Source-linked AI summary

SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?

Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu

arXiv:2609.09113v1cs.AIcs.CLcs.LG

TL;DR

Autonomous AI research needs internal monitoring because training pipelines that rely on external behavior can miss what models learn. SAEScientist-Bench evaluates agents that use SAE tools to discover concept-relevant features, finding genuine discovery but a large gap from experts, especially in causal steering.

  • Problem

    RSI research largely automates model training while lacking standardized evidence that agents can monitor and audit internal representations for safe, mechanistic understanding.

  • Method

    SAEScientist-Bench tests 10 frontier agents across 20 tasks, using contrastive probes to search a 131K+ Gemma-2-9B-IT SAE dictionary and comparing submissions with expert references on rank, selectivity, and steering.

  • Results

    Agents show genuine discovery and approach Expert on activation selectivity, but causal steering remains substantially lower: 31.47 versus 57.75.

  • Takeaways & Limitations

    Experimental model understanding, including hypothesis testing and interpretation of feedback, is measurable as a capability for closed-loop autonomous AI research.

  • Takeaways & Limitations

    The benchmark is limited to concept-driven single-feature discovery in Gemma-2-9B-IT across two Gemma Scope residual-stream SAE layers with a fixed probe interface.

Abstract

from arXiv · show

While research on recursive self-improvement (RSI) has predominantly automated model training pipelines, reliable autonomous development demands a missing pillar: post-hoc monitoring and auditing to understand what models learn and ensure safe alignment. Mechanistic interpretability tools are essential to bridge this gap, among which Sparse Autoencoders (SAEs) serve as a cornerstone by isolating interpretable features for model inspection and steering. In this paper, we introduce SAEScientist-Bench to evaluate whether AI agents can act as scientists utilizing SAE tools for autonomous mechanistic discovery. Given a target concept, an agent designs contrastive probes and navigates a Gemma Scope dictionary of 131K+ features in Gemma-2-9B-IT to discover the optimal feature, evaluated against curated expert reference features anchored on Neuronpedia across activation rank, concept selectivity on contrastive texts, and causal steering. Across 10 agent configurations and 20 tasks, frontier agents demonstrate genuine discovery capabilities and lead different evaluation dimensions, but remain well behind the expert baseline, approaching expert levels on separating target concepts from contrastive controls while lagging substantially in causal generation steering. Further analysis reveals that although agents can design contrasts to rule out spurious candidates, they frequently misinterpret experimental measurements. These results establish experimental model understanding as a measurable capability for closed-loop autonomous AI R&D. Our code is available at https://github.com/Trae1ounG/SAEScientist.

1 INTRODUCTION

SAEScientist-Bench addresses the lack of standardized evaluation for AI agents conducting SAE-based mechanistic discovery. It tests whether agents can design contrastive probes, identify useful features, and interpret experimental evidence across diverse tasks.

  • Motivation: Current RSI research largely automates training pipelines while treating models as black boxes evaluated through external task performance.These pipelines emphasize data curation, algorithm design, and post-training.
  • Motivation: Internal opacity makes behavioral evaluation insufficient for diagnosing failures or guaranteeing alignment, motivating continuous post-hoc monitoring of learned representations.External optimization can remain vulnerable to reward hacking, specification gaming, and deceptive alignment.
  • Motivation: SAEs expose interpretable feature directions that support both auditing whether concepts are learned and steering model behavior.They decompose polysemantic activations into isolated features for inspection and intervention.
  • Benchmark: SAEScientist-Bench evaluates AI agents as scientists across 20 concept-discovery tasks using contrastive probes and a unified framework of activation rank, selectivity, and causal steering.Agents navigate pretrained SAE dictionaries in Gemma-2-9B-IT and are compared with Neuronpedia-anchored expert references.
  • Results: Across 10 frontier agents, Kimi K3 leads overall performance, while agents approach Expert on activation selectivity but remain substantially behind on causal steering.Top activation selectivity is 92.91 versus Expert 98.92, while steering is 31.47 versus 57.75.
  • Evaluation: The benchmark combines standardized scoring with behavioral analyses of probes, candidate comparisons, and steering outputs to identify where autonomous discovery succeeds or fails.The analyses examine how agents interpret experimental evidence.

2 PRELIMINARIES: SAE FEATURES AND STEERING

Sparse Autoencoders encode model hidden states into sparse feature activations and reconstruct them with a linear decoder. Their decoder directions can also be added during inference to test causal effects on generation.

  • SAE features: A Sparse Autoencoder maps a token-level hidden state h ∈ R^d into a sparse, nonnegative feature vector z ∈ R^m.The language model supplies hidden states at a chosen layer.
  • SAE features: The SAE reconstructs the hidden state with a linear decoder, balancing reconstruction fidelity against sparsity during training.In Gemma Scope, JumpReLU thresholds restrict activation to a small subset of features per token.
  • SAE features: Each feature coordinate f represents an isolated latent feature, and its scalar activation z_f measures that feature’s expression on the input text.The decoder direction d_f = W_dec[f, :] corresponds to feature f’s representation direction.
  • Feature steering: Activation steering adds a feature’s decoder direction d_f to the model hidden state during inference to alter downstream generation.The intervention strength α controls the magnitude of the modification.
  • Benchmark coverage: Figure 3 summarizes benchmark category shares and task counts by SAE layer, with example concepts and Expert features for each category.All tasks receive equal weight, and the full task set is listed in Appendix A.1.

3 SAESCIENTIST-BENCH

SAESCIENTIST-BENCH evaluates autonomous SAE feature discovery across 20 tasks using held-out evaluation and three complementary scores. It compares submitted features with expert references on activation rank, selectivity, and causal steering.

  • Benchmark design: Each task asks an agent to discover one feature from a 131,072-feature Gemma Scope dictionary for a target concept in Gemma-2-9B-IT.The agent receives task metadata and uses a probe interface to retrieve or measure candidate features before submitting one feature ID.
  • Measurements: The benchmark evaluates submitted features on activation rank, activation selectivity, and causal steering, averaged across tasks.Rank measures prominence, selectivity measures separation from contrastive controls, and steering measures downstream generation change under intervention.
  • Measurements: Activation Rank compares a submitted feature’s average dictionary rank on positive texts with the Neuronpedia expert reference.Text-level activation uses the mean of the three largest non-special-token activations across each positive sequence.
  • Measurements: A rank score of 100.0 indicates parity with the expert baseline, while higher or lower values indicate greater or lower prominence, respectively.The score is scaled to a 100-point expert reference, with values above 100.0 indicating a feature ranking ahead of the expert.
  • Measurements: Activation Selectivity uses AUROC to separate positive texts from pooled hard-negative and neutral controls on a [0, 100] scale.A score of 100.0 denotes complete separation, while 0 indicates chance-level or reversed discrimination.
  • Measurements: Causal Steering measures net target-expression increase beyond the stronger control condition during feature intervention, while instruction preservation and degeneration are evaluated separately.The steering score is scaled to [0, 100], and alternative-feature intervention strength is calibrated on held-out prompts to satisfy a non-degeneration threshold.
  • Aggregate scoring: The default overall score is the unweighted arithmetic mean of the three dimension scores, with the expert baseline scoring 85.56.Reported benchmark scores average this overall measure across all 20 tasks.

4 EXPERIMENTAL SETUP

The experiments evaluate 10 frontier agent configurations on 20 discovery tasks through repeated autonomous investigations. After submission, a fixed pipeline validates activations and causal steering, and results are averaged across three runs per configuration and task.

  • Agents and harnesses: The study evaluates 10 representative frontier agent configurations spanning Kimi, Claude, Grok, Gemini, GLM, and OpenAI model families.Sol and Luna use the Codex harness, while the other listed models are deployed in Cursor.
  • Evaluation procedure: Each agent conducts an autonomous multi-turn investigation through the probe interface before submitting a chosen feature ID.The post-submission pipeline measures activations on a frozen evaluation set and performs causal steering with greedy generation.
  • Repeated evaluation: Every model–harness configuration performs three independent end-to-end investigations per task, with means and sample standard deviations reported across runs.Benchmark-level scores average all 20 tasks equally while analyses examine dictionary navigation, contrastive probing, evidence interpretation, and causal effects.

5 RESULTS

Across 20 tasks, frontier agents demonstrate genuine SAE feature-discovery capabilities but remain substantially below expert performance in causal intervention. Their outcomes depend on search strategy, hypothesis testing, evidence interpretation, and balancing selectivity against behavioral impact.

  • Overall performance: Kimi K3 achieved the highest Overall score at 65.82, while Opus 5 led Activation Rank at 75.35, Kimi K3 led Activation Selectivity at 92.91, and Grok 4.6 led Causal Steering at 31.47.
  • Overall performance: Top agents approached the Expert baseline on Activation Selectivity, reaching 92.91 versus 98.92, but lagged sharply in Causal Steering, reaching only 31.47 versus 57.75.
  • Search strategy: Case studies exposed evidence-interpretation failures and trade-offs between feature purity, coverage, causal impact, and generation stability.Opus 5 selected a feature activating more strongly on an English control than the Portuguese target, while Kimi accepted a leakier real-estate feature that increased steering but raised degeneration from 32.5% to 52.5%.
  • Search strategy: Rigorous direct validation separated agents that tested hypotheses from those relying on initial retrieval rankings, even when they selected the same feature.Opus repeatedly tested the anonymized Expert candidate, while Sonnet often overlooked it without direct validation.
  • Search strategy: Agents used distinct search philosophies: Sol varied probes, Opus screened broad candidate sets, and Kimi pursued targeted hypotheses.Sol averaged 115.1 probe texts and tested 20.2 candidates, whereas Opus averaged 77.6 probe texts and tested 80.2 candidates.
  • Activation evaluation: Held-out evaluation revealed coverage gaps and weak activation strength hidden by scalar selectivity scores, showing why generalization, rank, and selectivity must be assessed together.Sol and Expert both achieved AUROC 1.000, but Sol’s weaker target activations led to Activation Rank 3.97 versus Expert’s 100.00.
  • Causal steering: Strong activation separation did not guarantee causal control: only features discovering the Expert feature 41424 reliably induced fluent Portuguese, while several others produced zero target relevance.

6 RELATED WORK

Related work spans mechanistic interpretability, automated SAE feature interpretation, and autonomous agents for recursive self-improvement. These lines of research motivate evaluating agents that conduct interactive, hypothesis-driven interpretability experiments.

  • Mechanistic interpretability: Early mechanistic interpretability analyzed neurons and localized circuits, while SAE features address polysemanticity and support causal intervention.
  • Mechanistic interpretability: Pretrained SAE libraries such as Gemma Scope make large feature dictionaries accessible for mechanistic analysis and steering.
  • Automated interpretation: Automated methods use language models to explain neurons and SAE features, while interactive agents extend this work to experiments, feature discovery, circuit analysis, and hidden-behavior auditing.
  • Autonomous AI research: RSI research evaluates autonomous systems that iteratively advance capabilities through workflows including machine-learning engineering, paper reproduction, and scientific insight rediscovery.

7 LIMITATIONS

The benchmark’s conclusions are bounded by its concept-driven, single-feature setting, fixed evaluation resources, and automated steering judgments. Broader models, tasks, features, and validation procedures remain necessary for a fuller assessment.

  • The benchmark evaluates concept-driven single-feature discovery in Gemma-2-9B-IT using two Gemma Scope residual-stream SAE layers and a fixed probe interface.
  • Its scope should expand to diverse model families, broader dictionary widths, multi-feature circuit discovery, and open-ended hypothesis generation.
  • Candidate evaluation uses fixed text suites, frozen expert baselines, and automated LLM-as-a-judge steering assessments, motivating human validation, multiple references, and judge ensembles.
  • The benchmark evaluates post-hoc auditing but does not yet integrate discovered features into model editing, unlearning, or continual alignment loops.

8 CONCLUSION

SAESCIENTIST-BENCH evaluates autonomous mechanistic-interpretability research across frontier agents and 20 tasks. Agents approach expert performance in activation-space discrimination but retain a substantial causal-steering gap, making experimental understanding a measurable target for recursive self-improvement.

  • SAESCIENTIST-BENCH provides a standardized benchmark for agents conducting autonomous mechanistic interpretability research.
  • Across ten frontier configurations and 20 tasks, agents approached expert performance in distinguishing concepts in activation space but remained substantially behind in causal steering.
  • Effective discovery depended on rigorous hypothesis testing, informative counterexamples, and accurate interpretation of experimental feedback rather than exploration volume alone.
  • The benchmark establishes an auditable foundation for verifying internal representations in recursive self-improvement loops.

REPRODUCIBILITY STATEMENT

The paper’s appendices organize the benchmark materials, investigations, measurements, generations, task comparisons, and execution records for reproducibility.

  • Appendix A specifies the tasks, agents, and evaluation protocol.
  • Appendix B documents the investigations, while Appendices C and D provide activation measurements and complete generations.
  • Appendix E compares tasks and repeated runs, with underlying records preserving model identifiers and execution traces.

AI USE STATEMENT

Generative AI tools supported manuscript preparation, while the authors conducted the research and analytical work. The section also documents the benchmark’s execution and evaluation procedures.

  • AI USE STATEMENT: Generative AI tools were used only for writing assistance, text polishing, and editorial phrasing.
  • AI USE STATEMENT: The human authors conducted the conceptualization, benchmark design, experiments, data collection, and analytical interpretations.
  • Execution protocol: The benchmark runs three complete evaluations over 20 tasks per configuration, replacing failed attempts with retries and enforcing one scored result per episode.
  • Execution protocol: Agents use identical task specifications and probe interfaces, a writable workspace, disabled network access, and a standard 60-minute timeout per task.
  • Steering evaluation: Steering evaluation calibrates alternative-feature intervention strength on held-out prompts before applying the selected scale to final evaluation instructions.

B.2 EXTENDED AGENT INVESTIGATIONS

Extended investigations show agents using contrastive probes to compare candidate SAE features, test coverage and confounds, and balance selectivity against downstream generation stability.

  • Candidate search: Sol’s Portuguese investigation expands retrieval beyond the top 80 features and compares candidates on new texts when the eventual selection is absent from initial lists.
  • Contrastive investigations: Agents compare candidate features with targeted probes that test language, topic coverage, morphology, formatting, and symptom specificity.
  • Coverage tests: Expanding a Portuguese thank-you phrase into a full sentence raises the selected feature’s activation from 1.01 to 8.48, while a raw word list produces 2.58.
  • Candidate trade-offs: Candidate 84434 has higher AUROC (1.000 vs. 0.917) and Target Effect (0.944 vs. 0.900), but degeneration reaches 52.5% versus 32.5% for candidate 17219.
  • Candidate trade-offs: Across nine independent runs, candidate 17219 is preferred in seven instances, reflecting a trade-off between feature purity and downstream output stability.
  • Evidence interpretation: Kimi rejects multilingual candidates using control activations and selects feature 41424, identical to the Neuronpedia Expert.

D.2 EVALUATION DIAGNOSTICS

Evaluation diagnostics examine alternative-feature steering, output quality, and cross-model relationships. They show that stronger concept induction can conflict with instruction preservation and that benchmark rankings align with general intelligence unevenly across metrics.

  • Alternative selections: Alternative-feature evaluations report Target Effect, instruction preservation, and output degeneration across 13 tasks where agents select non-Expert candidates.
  • Steering diagnostics: Clinical steering induces disease-specific terminology while simultaneously exposing tension between aggressive concept induction and preserving the user’s prompt.
  • Cross-agent outputs: Portuguese self-introduction outputs receive varied target-relevance and preservation ratings, including 4/4 for Claude Opus 4.8 and GPT-5.6 Sol.
  • External comparison: The benchmark Overall score correlates with the AA Intelligence Index at ρ = 0.800, while Activation Selectivity reaches ρ = 1.000.

E.2 PERFORMANCE ACROSS TASKS AND CATEGORIES

Performance varies substantially by task and category: agents lead on different steering settings rather than one system dominating every outcome. Repeated investigations also distinguish consistent feature choices from retrospectively best discoveries.

  • Steering by task: Kimi and Sol obtain positive real-estate Steering, Opus 5 obtains zero there, and Opus 5 alone obtains nonzero Spanish Steering among most agents.Cat yields the same result for every agent.
  • Steering by task: Figure 10 reports each agent’s mean Steering on every task, averaging three investigations and enabling same-target, same-layer comparisons.L9 and L20 identify the SAE layer in the figure.
  • Comparing categories: Kimi’s professional Steering is 45.03 versus Opus 5’s 32.19, while Opus leads Kimi on reports, 27.50 versus 12.81.Category means weight tasks equally within categories, while Overall weights all 20 tasks equally.
  • Repeatability across all tasks: Table 29 separates consistently repeated feature selections from the highest-Steering feature found across three attempts.The comparison covers mean versus best Steering across 20 tasks.
  • Repeatability across all tasks: Steering scores are also reported across independent discoveries of four concepts, exposing variation between repeated investigations.The figure presents scores across independent discoveries rather than a single selection.
Loading 2609.09113v1…