PaperlayerView exact revision on arXiv

Source-linked AI summary

From Input to Output: A Flexible Agent for Dual-End Interpretation of Sparse Autoencoder Features

Dewen Liu, Zixuan Li, Jonathan Pan, Zhao Wu, Zijun Yao, Juanzi Li, Xiaozhi Wang

arXiv:2609.35367v1cs.CL

TL;DR

Existing SAE interpretation methods characterize activating inputs and intervention-induced outputs, but often leave their functional connection implicit, while input evidence collection can require costly corpus scans. DAFI gathers evidence and refines input, output, and functional interpretations through component-specific feedback. It improves both input- and output-side scores over component-matched baselines and provides a framework for examining how the endpoints relate.

  • Problem

    Existing methods characterize SAE features from input activations or output interventions, often leaving their functional connection implicit, while corpus scans can be costly for a few features.

    Source: §1 · p. 2

  • Method

    DAFI evaluates input-side, output-side, and functional interpretations separately, then uses component-specific feedback to gather evidence and refine them.

    Source: §1 · p. 2

  • Results

    DAFI achieves a 13.1-point higher Input score than SAGE (91.4% versus 78.3%) and a higher Output score than Token Change (66.7% versus 27.8%).

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    Shift and Break account for 70.7% of analyzed features with reliable endpoint interpretations, indicating that input and output semantics are often non-equivalent.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Sparse autoencoders (SAEs) are an important tool for mechanistic interpretability, but interpreting their many features remains challenging. Existing methods characterize input-side activation patterns and output-side intervention effects, yet often leave their functional connection implicit, while input-side evidence collection typically relies on costly large-corpus scans. We introduce functional interpretation, which characterizes an SAE feature as a mapping from its activating input semantics to its output effects under intervention, and present Dual-End Agentic Feature Interpretation (DAFI), an agent that actively gathers evidence and refines input-side, output-side, and functional interpretations through component-specific feedback. Its short-context token probing enables on-demand activation evidence collection without a full corpus scan. On GemmaScope, DAFI improves Input score by 13.1 percentage points over SAGE and Output score by 38.9 points over Token Change, while being substantially more token-efficient than a general-purpose coding agent. Skills distilled from successful refinements raise the held-out joint pass rate from 58.0% to 92.0% and improve both interpretation quality and efficiency when transferred to a new model-SAE setting. Across features with reliable endpoint interpretations, 70.7% exhibit non-equivalent input and output semantics. On AxBench, DAFI also improves steering-feature selection over output-score filtering. Code is available at https://github.com/THUAIS-Lab/DAFI.

1 INTRODUCTION

SAE feature explanations can capture only an input or output endpoint, while their functional connection is often implicit and corpus scans are inefficient for small feature sets. The paper proposes three-component interpretation and DAFI, which gathers evidence and selectively refines interpretations.

1 INTRODUCTION

  • Input-side corpus scans amortize across many features but are inefficient for small feature sets.Reliable interpretation also requires heterogeneous evidence and refinement operations that depend on diagnostic feedback.Source: §1 · p. 2
  • DAFI treats interpretation as input-side, output-side, and functional components, with component-specific diagnostics and an overall criterion requiring all three to pass.The functional component concerns whether and how the input and output endpoints connect.Source: §1 · p. 2
  • Features with reliable endpoint interpretations often fall into Shift or Break categories, so their input and output semantics are not equivalent.The taxonomy is used to examine whether and how endpoints connect.Source: §1 · p. 2
  • DAFI actively acquires evidence and selectively refines failed components, producing functional interpretations alongside input- and output-side ones.On matched GemmaScope features, it outperforms component-matched input- and output-side baselines.Source: §1 · p. 2
  • Short-context token probing collects targeted input-side activation evidence on demand without a full corpus scan, while achieving scores comparable to corpus-derived examples.DAFI distills refinements into reusable skills that improve quality and efficiency on unseen features and transfer across LLM–SAE settings.Source: §1 · p. 2

2 RELATED WORK

Prior automated SAE feature interpretation characterizes activating inputs or intervention-induced outputs, while the functional connection between them is often left implicit. DAFI adds that relational target and retains successful refinements as reusable experience.

2 RELATED WORK

  • SAEs represent dense LLM activations with sparse feature coefficients and provide finer-grained units than polysemantic neurons.Interpreting these features is presented as a way to understand model mechanisms.Source: §2 · p. 2
  • Input-side methods interpret activating examples, whereas output-side methods characterize changes in model outputs; prior approaches may focus on one endpoint or combine both into one interpretation.Input-side evidence collection can require a full corpus scan, costly when only a few features are targeted; output-side methods lack fine-grained iterative diagnostics.Source: §2 · p. 2
  • DAFI defines a functional interpretation as a mapping from a feature’s input activation pattern to its causal output effects.It evaluates input, output, and functional interpretations separately and uses component-specific feedback to guide refinement.Source: §2 · p. 2
  • Figure 2 presents DAFI’s three interpretation metrics and its agent loop with reusable skills.The loop and metrics are the paper’s approach to interpretation and refinement.Source: Figure 2 · p. 3
  • DAFI retains successful revisions as case skills and distills recurring patterns into general skills for subsequent features.This follows a broader self-evolving-agent approach that accumulates and reuses experience across tasks.Source: §2 · pp. 2–3

3 METHOD

DAFI treats SAE feature interpretation as a triplet of input, output, and functional hypotheses, evaluates each component separately, and uses diagnostic feedback to guide evidence collection and refinement. Successful revisions become reusable skills for later interpretations.

3.1 PROBLEM FORMULATION: ENDPOINT AND RELATIONAL INTERPRETATIONS

  • DAFI seeks hypotheses for a target feature in a language model and SAE, with the refinement round indexed by t.The triplet consists of input-side, output-side, and functional hypotheses.Source: §3.1 · p. 3
  • The functional hypothesis describes how a feature links the semantics that activate it to the output bias induced by intervening on it.Input hypotheses describe preferential activation; output hypotheses describe intervention-induced semantic bias, and the components are evaluated separately for targeted refinement.Source: §3.1 · p. 3

3.2 METRIC DESIGN

  • The Input score combines activation coverage with boundary rejection, testing whether described triggers activate the feature and nearby out-of-scope prompts remain inactive.Both components must individually reach 0.8 for the stricter input-side pass criterion.Source: §3.2 · p. 4
  • The Output score rewards positive intervention-induced logit-change mass captured by the hypothesis and penalizes describing suppressed tokens as promoted.The Functional score tests whether the hypothesis connects activating input semantics to intervention-induced output semantics, grounded in evidence from both endpoints; a score of at least 4 passes.Source: §3.2 · p. 4

3.3 AGENT DESIGN

  • DAFI’s agent loop evaluates and updates an input, output, and functional hypothesis triplet, using feedback and evidence to choose the next refinement or test.It repeats until all criteria pass or refinement stops, then returns the best-supported complete triplet.Source: §3.3 · p. 4
  • DAFI can collect input evidence through corpus-derived examples or short-context token probing, and can vary intervention strength, token position, and prompt context to inspect output changes.Successful revisions are stored as case skills, while recurring patterns become general skills used across features and transferred across model–SAE settings.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate DAFI’s input-side, output-side, and functional interpretations, testing quality, computational cost, design choices, skill reuse, endpoint relations, and steering-feature selection.

4.2 MAIN RESULTS AND EVALUATION RELIABILITY

  • DAFI achieves a 13.1-point higher Input score than SAGE (91.4% versus 78.3%) and a higher Output score than Token Change (66.7% versus 27.8%), while using fewer tokens than SAGE.The component-matched comparisons use 100 features.Source: Table 1 · p. 6, §4.2 · p. 6
  • GPT-5.6-Sol agrees with DeepSeek-V4-Pro on 87.1% of Output score and 80.4% of Functional score pass decisions; Functional score correlates with two researchers’ ratings (Spearman ρ = 0.809 and 0.777).In the 40-feature human evaluation, binary agreement was 95.0% and 92.5%, respectively.Source: §4.2 · p. 6

4.3 DESIGN ANALYSIS

  • 15.8 versus 9.3 percentage-point Input score gains favor short-context probing over Neuronpedia initialization, although final scores are 88.5% versus 91.4%.For one feature, initialization and follow-up scans process 0.59M input token positions, 87.4% fewer than the 4.72M-position corpus pass.Source: Table 2 · p. 7, §4.3 · p. 7
  • 34.0 percentage points of joint pass-rate improvement accompany fewer refinement rounds as skills accumulate, with most pass-rate gains appearing within the first 20 features.The held-out suite is assessed after every 20 features during sequential interpretation of 120 features.Source: Figure 3 · p. 7, §4.3 · p. 7
  • Transferred skills improve all three interpretation scores on Qwen-Scope features while reducing refinement rounds and token usage; the largest score gain is 7.3 percentage points on Output, and token usage falls by 14.5%.The transfer evaluation uses 100 Qwen-Scope features.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 WHAT DUAL-END INTERPRETATION REVEALS ABOUT SAE FEATURES

  • Shift and Break account for 70.7% of 215 features with passing endpoint interpretations, and Shift alone accounts for 63.7%.Equivalent preserves meaning, Shift links distinct meanings through a supported mapping, and Break lacks a supported connection.Source: §4.4 · p. 8, Figure 4 · p. 8
  • Compared with Neuronpedia explanations, DAFI’s Shift cases characterize activating inputs, promoted outputs, and the functional relation between them more comprehensively and precisely.The interpretations explain feature roles during inference by describing how they shape generation in activating contexts.Source: §4.4 · p. 8, Table 5 · p. 9

4.5 APPLICATION CASE: IDENTIFYING STEERABLE SAE FEATURES

  • DAFI-based feature selection achieves a 30.5% higher area under the score-coverage curve than output-score filtering and a 23.2% improvement over the unfiltered baseline.The comparison uses 500 Concept500 pairs with the same AxBench generations and held-out scores.Source: §4.5 · p. 9

5 CONCLUSION

DAFI connects input-side, output-side, and functional interpretations, coordinating selective refinement and reusable skills to improve interpretation efficiency. The appendices describe its repair pipeline, refinement behavior, explanation prompts, and endpoint-relation labels.

5 CONCLUSION

  • DAFI coordinates input-side, output-side, and functional hypotheses, using diagnostic feedback and reusable skills to refine interpretations.The conclusion also identifies lower-cost interpretation and downstream model editing as future work.Source: §5 · p. 9

A AGENT SYSTEM DETAILS

  • Selective reruns preserve unaffected evidence, while best-round selection retains a complete interpretation triplet and its associated scores.The repair policy changes the smallest component supported by the observed failure and reuses unaffected evidence.Source: A AGENT SYSTEM DETAILS · p. 12, Table 6 · p. 12

A.1 CLAUDE CODE BASELINE INSTRUCTIONS

  • The Claude Code baseline uses a nine-step pipeline with separate input, output, and functional interpretations, quality gates, and an iteration loop of up to 10 rounds.The instructions require diagnosing failed gates, recording each execution in a new round directory, and maintaining an experience log.Source: §A.1 · pp. 12–13

B REFINEMENT DYNAMICS AND STABILITY

  • Across 250 features, refinement averages 2.92 rounds per feature; 145 features use no more than two rounds, while 8 reach the 10-round cap.Most runs finish early, with a long tail of difficult cases.Source: B REFINEMENT DYNAMICS AND STABILITY · p. 14, Figure 6 · p. 14
  • The minimum input component decreases for 0.4% of features, while Output and Functional scores decrease for 5.2% and 6.4%, respectively.Aggregate scores improve across all three interpretation components, but individual diagnostic scores are not guaranteed to improve monotonically.Source: B REFINEMENT DYNAMICS AND STABILITY · p. 14, Table 7 · p. 14

C EXPLANATION AND JUDGE TEMPLATES

  • Component-specific prompts use activation contexts for input hypotheses, intervention deltas for output hypotheses, and both endpoint hypotheses for functional explanations.The functional prompt cannot introduce an unobserved intermediate mechanism; the judge returns a Functional score, rationale, and relation label.Source: C EXPLANATION AND JUDGE TEMPLATES · p. 14

C.1 SEMANTIC RELATION CLASSIFICATION

  • The relation classifier labels endpoint pairs Equivalent when meaning is preserved, Shift when distinct meanings have an identifiable mapping, and Break when evidence supports no coherent connection.Its decision order checks Break first, then Equivalent, and otherwise selects Shift.Source: §C.1 · p. 15

D LAYER-WISE REFINEMENT ANALYSIS

Refinement improves reported metrics across five GemmaScope layers, while layer depth relates differently to input reliability and functional interpretation quality. The appendices report held-out skill accumulation and evidence-volume results, including a reversal at higher throughput.

D LAYER-WISE REFINEMENT ANALYSIS

  • Mean Input score rises from 83.3% to 93.3%, mean Output score from 32.5% to 69.5%, and mean Functional score from 3.0 to 4.0; joint passes increase from 20/250 to 200/250.Improvements occur at all five layers, and final Input score exceeds 90% at every layer.Source: Table 9 · p. 16, D LAYER-WISE REFINEMENT ANALYSIS · p. 16

D.1 LAYER DEPTH SEPARATES INPUT-SIDE RELIABILITY FROM FUNCTIONAL INTERPRETATION QUALITY

  • Layer 0 reaches 0.960 for both refined activation coverage and boundary rejection, while final Functional score is 4.28 at Layer 24 versus 3.94 at Layers 0 and 18.Strong activation-side recoverability does not by itself imply a clearer functional relationship to downstream effects; the passage cautions this is a layer-level tendency, not evidence every deep feature is a better intervention target.Source: §D.1 · p. 16

E SKILL ACCUMULATION DETAILS

  • As skills accumulate, held-out joint pass rate rises from 58.0% to 92.0%, while mean agent-loop rounds per feature fall from 4.00 to 2.42.The endpoint comparison shows improved interpretations with less agent effort despite intermediate fluctuations.Source: E SKILL ACCUMULATION DETAILS · p. 16, Table 10 · p. 17

F.1 INPUT-SCORE COMPONENT BREAKDOWN

  • Table 11 separates Input score into activation coverage and boundary rejection for Neuronpedia and one-template short-context-probing initialization.The supplied table caption identifies the comparison scope as 100 matched GemmaScope features.Source: Table 11 · p. 17, §F.1 · p. 17

F.2 EVIDENCE-COLLECTION VOLUME

  • Under the reported average-case accounting, short-context probing uses 87.4% fewer input token positions for one targeted feature, supporting on-demand use for small feature sets.A shared inference service could record requested scans for all active target features, amortizing adaptive probing across concurrent interpretations; this infrastructure optimization is left to future work.Source: §F.2 · p. 18

F.3 COMPLEMENTARY EVIDENCE FROM SHORT-CONTEXT PROBING

Short-context probing can reveal activation evidence absent from available corpus examples and can adapt its context after an uninformative scan. A separate blinded human study reports agreement on functional-interpretation plausibility and relation labels.

F.3 COMPLEMENTARY EVIDENCE FROM SHORT-CONTEXT PROBING

  • Layer 18 Feature 1830's short-context scan exposed multilingual locative tokens absent from 45 available Neuronpedia examples, and its most specific Italian-article hypothesis achieved 1.0 activation coverage and 1.0 boundary rejection.The available examples were dominated by the English token “the”; the absent tokens may still occur in the underlying corpus.Source: §F.3 · p. 18
  • For Layer 18 Feature 3450, adding the “Despite” context after an uninformative initial scan yielded 39 activating tokens and a final input hypothesis with 1.0 activation coverage and 1.0 boundary rejection.The initial full-vocabulary scan found no activation, while the expanded “Despite” scan found a peak activation of 9.75; the semantic interpretation remained a hypothesis evaluated downstream.Source: §F.3 · pp. 18–19, Table 13 · p. 18
  • For Layer 6 Feature 8800, “endforeach” ranked among the top two tokens in all six templates and was absent from the available Neuronpedia examples.Its activation ranged from 23.375 to 32.500, with a mean of 27.458; absence from those examples does not establish absence from the underlying corpus.Source: §F.3 · p. 19

G HUMAN VALIDATION DETAILS

  • Two blinded researchers independently rated a stratified 40-feature subset on input fit, output fit, functional plausibility, and endpoint relation.A human plausibility pass was defined as a rating of at least 4 on a five-point scale.Source: G HUMAN VALIDATION DETAILS · p. 19
  • Annotators' functional-plausibility ratings correlate at 0.865, with quadratic-weighted κ of 0.841 and 92.5% agreement within one point; relation-label agreement is 70.0% with κ of 0.576.System Functional scores match human binary decisions on more than 92% of features; the authors interpret the study as evidence of agreement with human plausibility judgments.Source: G HUMAN VALIDATION DETAILS · p. 19

H CROSS-MODEL JUDGE CONSISTENCY

A second judge agrees with the original on most thresholded Output and Functional decisions, though Functional score is more judge-sensitive. The supplied transfer table caption describes layer-level results on Qwen-Scope but gives no outcome values.

H CROSS-MODEL JUDGE CONSISTENCY

  • GPT-5.6-Sol agrees with DeepSeek-V4-Pro on 87.1% of Output score and 80.4% of Functional score pass decisions.The lower Functional rank correlation indicates greater sensitivity to judge choice, although thresholded conclusions are described as reasonably stable across the two judges.Source: H CROSS-MODEL JUDGE CONSISTENCY · pp. 19–20, Table 15 · p. 20

I CROSS-SAE LAYER BREAKDOWN

  • Table 16 reports layer-level transfer results on Qwen3-1.7B-Base with Qwen-Scope W32K residual-stream SAEs using Top-k activation with k = 50, with quality and efficiency gains that vary across layers.The table caption states that each layer contains 20 sampled features and that “All” gives the 100-feature aggregate.Source: I CROSS-SAE LAYER BREAKDOWN · p. 20, Table 16 · p. 20

J BUDGET COMPARISON DETAILS

The budget comparison reports Claude Code’s feature passes and token costs under restricted and unrestricted execution. It also distinguishes the paper’s output-side evaluation from an output-side description by requiring evidence to support a mapping from activating context to output pattern.

J BUDGET COMPARISON DETAILS

  • The restricted comparison sets Claude Code’s round-start budget from DAFI’s per-feature token usage and prevents another round after the budget is reached.Claude Code still completes one round even if that round exceeds the budget.Source: J BUDGET COMPARISON DETAILS · pp. 20–21, Table 18 · p. 21
  • Under unrestricted execution, Claude Code passes 19 features using 332.39M tokens total; under restriction, it passes 2 features using 77.06M tokens total.Unrestricted use averages 13.296M tokens per feature, with a median of 8.67M and a maximum of 52.87M; restricted execution uses a round-start budget, not a hard cap.Source: J BUDGET COMPARISON DETAILS · p. 21, Table 18 · p. 21

K OUTPUT SCORE EVOLUTION AND OUTPUT-SIDE VALIDATION

  • Output-side evidence must support a functional interpretation mapping input context X to output pattern Y, with score failures guiding the agent’s repair.This goes beyond describing output effects alone.Source: K OUTPUT SCORE EVOLUTION AND… · p. 21

K.1 OUTPUT SCORE DEFINITION

The Output score evaluates whether an output hypothesis captures intervention-induced effects while penalizing hypotheses that also describe suppressed tokens. Its revisions target over-broad explanations, and layer-wise results and a case study illustrate its use.

K.1 OUTPUT SCORE DEFINITION

  • The Output score rewards positive intervention mass captured by the hypothesis and penalizes covered suppressed-token mass, normalizing both by total positive mass.A low coverage score means the hypothesis misses promoted evidence, while a high penalty means it also covers tokens the intervention suppresses.Source: §K.1 · pp. 21–22
  • The score evolved from top-10 positive-token recall to include top-30 positive and negative evidence, adding pressure against broad hypotheses and calibrating steering per feature.VocabProj is a geometric proxy, while intervention-based Output score ties effects to contexts where the feature already fires.Source: §K.1 · p. 22
  • 0.042 versus 0.123 is the across-layer standard deviation of Output scores for DAFI and Token Change, respectively; DAFI scores higher at every layer.The comparison uses 100 matched features, with 20 from each layer; lower standard deviation indicates greater consistency.Source: §K.1 · p. 23, Table 21 · p. 23

L ADDITIONAL CASE STUDIES

  • After five refinement rounds, feature L18 F6750’s scores were Input score 1.000, Output score 0.934, and Functional score 5 out of 5.The final input hypothesis captured cross-lingual tokens expressing adding or having been added while excluding adjacent concepts such as “extra” and “supplementary.”Source: L ADDITIONAL CASE STUDIES · p. 23

M THRESHOLD ROBUSTNESS

Adaptive refinement raises joint pass rates over initial explanations and remains beneficial across the tested threshold settings. The sensitivity analysis varies one threshold at a time while holding the other two at default values.

M THRESHOLD ROBUSTNESS

  • 200 of 250 adaptive explanations pass jointly, compared with 20 of 250 initial explanations.The two sets use the same scoring protocol and default thresholds.Source: M THRESHOLD ROBUSTNESS · p. 23
  • Adaptive refinement raises the joint pass rate at all 27 tested threshold operating points, by 6.4 to 72.0 percentage points.The grid varies one threshold at a time; default thresholds are 0.8 for both input components, 0.5 for Output, and 4 for Functional.Source: M THRESHOLD ROBUSTNESS · p. 23, Table 22 · p. 23

N OUTPUT SCORE FILTER ANALYSIS

On Gemma-2-2B under the AxBench protocol, the output-score filter does not transfer the reported 9B gains. The paper scopes this comparison to matched generations and notes that cross-study protocol differences prevent isolating a single cause.

N OUTPUT SCORE FILTER ANALYSIS

  • Cross-study absolute scores are confounded by differences in model scale, steering-factor selection, instruction sampling, and judge model.The matched filter comparison controls instruction sampling and judging by using the same seeded sample and judge within the study.Source: N OUTPUT SCORE FILTER ANALYSIS · p. 24
  • On the same generations, the ratio rule selects factors with mean holdout score 0.105, versus 0.140 for the AxBench rule.The paper uses the AxBench rule for its main comparison because it is the benchmark’s official protocol and is applied identically to both filters.Source: N OUTPUT SCORE FILTER ANALYSIS · p. 24
  • On Gemma-2-2B under the AxBench protocol, high Output scores co-occur less reliably with fluent, instruction-following generations, and the filter does not transfer from the reported 9B setting.The authors do not rerun the ratio rule on 9B, so these results do not establish its effectiveness under the original 9B protocol; they describe a null result consistent with, not a disproof of, the positive 9B result.Source: N OUTPUT SCORE FILTER ANALYSIS · p. 24

O REPRODUCIBILITY DETAILS

The reproducibility protocol fixes evaluation thresholds, candidate selection, intervention settings, and feature sampling across the reported studies. The ethics statement describes the human evaluation and cautions that language-model outputs may inherit bias.

O REPRODUCIBILITY DETAILS

  • Input success requires activation coverage and boundary rejection of at least 0.8 each, Output score at least 0.5, and Functional score at least 4.Input score is the harmonic mean of activation coverage and boundary rejection; joint success requires one consistent interpretation triplet to pass all three criteria.Source: O REPRODUCIBILITY DETAILS · p. 24
  • Candidate pool and round budget are fixed before evaluation, and the selected triplet is the complete internally consistent one with the highest eligible round priority.Ties go to the earlier round; intervention uses the top 30 positive and negative token changes, at most five steering prompts, and defaults to the maximally activating token.Source: O REPRODUCIBILITY DETAILS · p. 24
  • GemmaScope evaluation uses 200 skill-accumulation and 50 held-out features, while Qwen-Scope transfer uses 20 features from each of five layers.GemmaScope’s 100-feature comparison subsets come from the 250-feature pool; methods share feature lists and applicable evaluation settings, and token counts include the complete agent context.Source: O REPRODUCIBILITY DETAILS · p. 25

P ETHICS CONSIDERATIONS

  • Two experienced authors rated interpretations independently while blinded to system scores and each other’s labels; language-model outputs may inherit biases and warrant empirical evidence and human review.No external participants were recruited, and no compensation or personal, sensitive, or behavioral data were involved.Source: P ETHICS CONSIDERATIONS · p. 25

Related papers

Loading 2609.35367v1…