PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradients for Interventions and Activations for Detection: Targeted Feature Learning in Language Models

Jonathan Drechsel, Steffen Herbold

arXiv:2609.32355v1cs.CLcs.LG

TL;DR

Many interpretability questions concern concepts specified in advance, while SAE concepts are commonly identified post hoc. The paper compares targeted features across signals and estimators, finding that detection and intervention favor different methods.

  • Problem

    SAEs learn broad feature dictionaries whose relation to specified concepts is usually identified post hoc, while many interpretability questions target a concept in advance.

    Source: §1 · p. 1

  • Method

    The paper compares six targeted methods formed by crossing activation values, activation gradients, and parameter gradients with contrastive means and a learned one-dimensional encoder-decoder.

    Source: §1 · p. 2

  • Results

    Across models and tasks, activation-value methods achieve the strongest detection, whereas gradient-based methods achieve the strongest intervention.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Detection and intervention are complementary measures of feature quality, and targeted feature quality depends jointly on signal and estimator.

    Source: §1 · p. 2, §7 · p. 9

    Evaluation covers three language models and 15 tasks; generalization to other architectures, scales, and feature types remains to be established.

    Source: §6 · p. 9

Abstract

from arXiv · show

Model-internal features can be studied through both their ability to identify a specified concept and their causal effect when manipulated, e.g., through steering or weight editing. A prominent approach to feature learning is Sparse Autoencoders (SAEs), which learn broad feature dictionaries whose relation to particular concepts is typically identified post hoc. However, many interpretability questions are instead hypothesis-driven and concern a concept specified in advance. We study this setting as targeted feature learning, where a single feature is constructed for such a predefined concept. We present a controlled comparison across three model signals (activation values, activation gradients, and parameter gradients) and two estimators (contrastive mean and a learned one-dimensional encoder-decoder), yielding six targeted methods, with CAA and GRADIEND as existing instances and four new methods covering the remaining combinations. We compare these methods against pretrained SAEs across 15 tasks and three language models, evaluating both detection and causal intervention. Across models, the strongest detection performance is achieved by contrastive activation value methods, whereas the strongest intervention performance is achieved by gradient-based methods. Overall, our results show that targeted feature quality depends jointly on the model signal and estimator, with detection and intervention capturing complementary properties.

1 INTRODUCTION

Targeted learning constructs a feature for a concept specified in advance, complementing SAE dictionaries whose concepts are usually identified post hoc. The paper compares targeted methods across signals and estimators, finding a trade-off between detection and intervention.

1 INTRODUCTION

  • Targeted learning constructs a single feature for a predefined concept, avoiding post-hoc selection and potentially addressing SAE feature splitting and absorption.The intended feature should distinguish the concept from relevant alternatives and support interventions that change corresponding model behavior.Source: §1 · p. 2
  • Existing CAA and GRADIEND methods vary in signal, estimator, and intervention space, so comparing them alone cannot isolate which choice explains feature quality.The benchmark also differs from AxBench: it targets the feature-inducing position, avoids LLM-dependent data generation and steering evaluation, and evaluates throughout model depth.Source: §1 · p. 2
  • The controlled comparison crosses activation values, activation gradients, and parameter gradients with contrastive means and IEND estimators.Activation gradients bridge existing approaches by adding gradient information while retaining activation-space intervention.Source: §1 · p. 2, Figure 2 · p. 2
  • Activation-value methods achieve the strongest detection, while gradient-based methods achieve the strongest intervention across models and tasks.The comparison covers six targeted methods, 15 tasks, three language models, and pretrained SAEs as an untargeted reference.Source: §1 · p. 2
  • The study introduces ACTIEND, AGIEND, CAGA, and CGA to complete the 3 × 2 design of targeted methods.The methods are compared against pretrained SAEs across 15 tasks and three language models.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans targeted activation directions, untargeted sparse feature dictionaries, and gradient-based feature construction. This study extends comparisons across both signal space and estimator.

Targeted concept representations and activation steering

  • AxBench evaluates concept detection and steering across representation methods, while this study extends comparisons to activation and parameter gradients and varies the estimator.Prior activation-direction work includes CAA and mean-difference directions; probing can establish decodability without showing a coherent or causally used feature.Source: §2 · p. 3

Sparse feature learning

  • SAEs learn broad activation dictionaries without a prespecified concept, making post-hoc identification and intervention dependent on selecting or combining latents.SCAR and Guided SAEs add concept supervision but retain a larger dictionary rather than constructing only the specified feature.Source: §2 · p. 3

Gradient-based feature construction and intervention

  • GRADIEND learns one targeted feature from factual-alternative parameter gradients, whereas this study learns reusable targeted features from activation gradients as well.Gradient SAEs learn from activations and use output gradients to rank estimated influence, unlike the activation-gradient methods described here.Source: §2 · p. 3

3 TARGETED FEATURE LEARNING

Targeted feature learning constructs a single feature from a concept specified in advance, using contrastive factual and alternative targets. The section defines three model signals, two estimators, and how the resulting features support detection and intervention.

3.1 FROM FEATURE DISCOVERY TO TARGETED FEATURE LEARNING

  • Targeted learning uses the feature specification during construction, unlike untargeted SAE features identified after learning; matched factual and alternative targets provide its supervision.The target examples share context while differing in the feature realization.Source: §3.1 · pp. 3–4

3.4 TWO TARGETED FEATURE ESTIMATORS

  • Crossing the three signals with contrastive means and IEND yields six methods, including CAA and GRADIEND and four newly introduced combinations.Contrastive means estimate a direction from contrastive differences; IEND learns a scalar bottleneck and decoder to reconstruct transitions.Source: §3.1 · p. 4, §3.4 · pp. 5–6

3.2 SPECIFYING FEATURES THROUGH CONTRASTS

  • Pairwise contrasts produce a symmetric axis between two classes, while one-sided contrasts learn target-class presence against non-target alternatives.For binary features, pairwise learning yields one polar feature, whereas one-sided learning yields two independently learned target-presence features.Source: §3.2 · p. 4

3.3 THREE SIGNAL SPACES

  • The three signals are activations, gradients with respect to activations, and gradients with respect to parameters; activation-value methods require the target in the input before extracting the signal.Gradient signals can be extracted from the model state preceding the target, but intervention must still occur before its prediction.Source: §3.3 · p. 5

3.5 FEATURE DETECTION AND FEATURE INTERVENTION

  • Detection uses the feature score, while intervention applies a class-oriented direction in activation or parameter space; SAEs provide a post-hoc-selected activation-feature reference.For targeted methods, intervention strength controls movement toward or away from the selected class.Source: §3.5 · p. 6

4 EXPERIMENTAL SETUP

The experiments evaluate three language models and 15 tasks with shared data splits, measuring detection and intervention. Validation selects method-specific settings, with intervention constrained to limit language-model degradation.

Models and tasks

  • The benchmark covers GPT-2, Pythia-70M, and Llama-3.1-8B across 15 semantic, grammatical, factual, language, and algorithmic tasks, using pretrained SAEs as a baseline.The tasks include pairwise and one-sided feature specifications.Source: §4 · p. 6

Protocol and model representations

  • All methods use identical train-validation-test splits; features are constructed on training data, settings selected on validation data, and final scores evaluated on test data.Validation selects IEND checkpoints, SAE latents, detection thresholds, intervention strengths, and representation scopes.Source: §4 · p. 6

Detection evaluation

  • Detection combines target-versus-neutral and, where applicable, target-versus-rival AUROC and specificity using the minimum applicable criterion.Taking the minimum prevents strong performance on one criterion from compensating for failure on another.Source: §4 · pp. 6–7

Intervention evaluation

  • Intervention averages strengthening and weakening changes in target probability, with magnitudes independently selected under a language-model score constraint of LMS(I) ≥ 0.99 LMS(I0).For pairwise features, strengthening is tested on rival examples and weakening on target examples; one-sided features use target examples for both.Source: §4 · p. 7

Representation selection

  • Validation selects the representation scope for test detection and intervention; IEND uses fixed joint scopes, while other methods also evaluate individual layers and an all-layer scope.ACTIEND and AGIEND use joint residual-stream activations, and GRADIEND uses a joint parameter-gradient scope.Source: §4 · p. 7

5 FEATURE DETECTION AND INTERVENTION PERFORMANCE

Across methods, activation-value features lead detection while gradient-based features lead intervention, showing that the two measures capture complementary properties. Estimator, construction, task, and representation scope further shape the comparison.

Signal and estimator

  • For gradient signals, IEND typically raises intervention while often reducing detection; for activation values, it typically lowers both measures relative to the contrastive mean.Within contrastive means, CGA improves intervention over CAA at a detection cost, while CAGA’s pattern varies across models.Source: §5 · p. 7

Feature construction

  • On matched tasks, one-sided construction generally matches pairwise performance and often yields slightly higher detection, with similar intervention for most methods.The aggregate comparison is not directly comparable because one-sided methods include additional tasks; one-sided construction requires fewer feature models and supports concepts without a natural opposing class.Source: §5 · p. 7

Overall performance

  • Activation-value methods, particularly CAA, achieve the strongest detection, while gradient-based methods, particularly GRADIEND, achieve the strongest intervention overall.SAEs are comparatively modest on both measures and do not match the strongest targeted methods.Source: §5 · p. 7, Figure 3 · p. 8

Bayesian method comparison

  • Bayesian pairwise comparisons place activation-value methods strongest for detection and GRADIEND strongest for pairwise intervention, followed by AGIEND.Under one-sided construction, GRADIEND and AGIEND remain statistically unresolved.Source: Figure 4 · p. 8, §5 · p. 8

Task-level variation

  • Task performance varies substantially: LANGID and EMOTION are hardest on both measures, while intervention varies more strongly across tasks and methods than detection.For EMOTION, SAEs fail to recover a feature latent while targeted methods retain non-trivial detection; some one-sided tasks approach chance-level detection, and RAVEL tasks, GENDER, and REPETITION show some of the largest intervention effects, especially for gradients.Source: §5 · p. 8

Representation scope

  • Single-layer representations outperform all-layer representations for held-out detection in 80.4% of SAE and contrastive-mean comparisons, while intervention differences are smaller and less systematic.Restricting those methods to all-layer representations preserves the main signal-level pattern: activation values lead detection and gradients lead intervention.Source: §5 · p. 8

6 DISCUSSION AND LIMITATIONS

The discussion distinguishes the signals best suited to detection from those best suited to intervention, and considers positional alignment and representation scope as factors in that difference. The evaluation is bounded to three models, 15 tasks, and interventions at specified target tokens.

Detection versus intervention

  • Activation values are strongest for detection, whereas gradients are strongest for intervention; AGIEND shows that the intervention advantage does not require parameter-space editing.CAA is particularly strong for detection, while AGIEND and GRADIEND are particularly strong for intervention.Source: §6 · p. 9

Positional asymmetry

  • Targeted activation contrasts cannot be learned before the target when factual and alternative examples share a prefix, but gradient signals can be derived at positions predicting that target.This positional alignment may contribute to, but does not fully explain, the intervention advantage of gradients.Source: §6 · p. 9

Decoupling detection and intervention

  • Detection benefits more consistently than intervention from localizing a feature to an informative layer, motivating a possible hybrid of localized activation inputs and contrastive gradient targets.Whether such strengths can be combined in one learner or reflect a more fundamental trade-off remains open.Source: §6 · p. 9

Limitations

  • The evaluation covers three language models and 15 tasks; generalization to other architectures, scales, and feature types remains unestablished.Intervention is measured at the specified target token and does not capture effects on longer generations; the target-localized setup also prevents direct matched comparison with broader benchmarks such as AxBench (Wu et al., 2025).Source: §6 · p. 9

7 CONCLUSION

The conclusion presents a controlled comparison of targeted feature-learning choices and finds detection and intervention to be complementary measures of feature quality. It also records the paper’s AI-use and reproducibility practices.

7 CONCLUSION

  • Across three models and 15 tasks, detection and intervention are complementary: contrastive activation values provide particularly strong detection, while gradient-based methods achieve the strongest causal intervention.The findings motivate methods that combine these strengths, including hybrid and layer-selective IEND formulations.Source: §7 · p. 9

AI USE STATEMENT

  • Generative AI assisted with methodology discussions, implementation, literature discovery, drafting, and editing; authors reviewed and revised the work and take responsibility for its final content.The tools were not used to generate synthetic datasets.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The study releases code and redistributable datasets, with scripts to regenerate synthetic data and reconstruct third-party tasks from public sources.Fixed seeds are used for training and evaluation, with validation data for selection and test data reserved for final evaluation.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASET DOCUMENTATION

The appendix documents a 15-task benchmark, its data construction and provenance, and the model and software resources used in the experiments.

A DATASET DOCUMENTATION

  • Task datasets include reused, adapted, and newly generated examples, with generated datasets and generators released alongside the study.The appendix also describes task-specific constructions, including IOI and RAVEL adaptations, LANGID examples, and KEYVALUE retrieval.Source: A DATASET DOCUMENTATION · pp. 15, 17
  • The task inventory records feature classes, construction regimes, data provenance, and effective labeled split sizes across the benchmark.P denotes tasks evaluated in both pairwise and one-sided regimes; O denotes a factual class paired with a constructed counterpart.Source: A DATASET DOCUMENTATION · p. 15, Table 3 · p. 16
  • Table 4 illustrates labeled examples, retaining the masked span and local wording while identifying each factual target and its feature class.For one-sided tasks, the counterfactual class is listed in Table 3 but does not add a second base row.Source: Table 4 · p. 16

B.1 IMPLEMENTATION / B.2 MODELS

  • The study extends the gradiend package and evaluates three pretrained decoder-only models, using pretrained SAE resources as the reference.The model checkpoints and SAE resources are listed in Table 6; the study repository contains task, method, and evaluation code.Source: §B.1 · p. 18, §B.2 · p. 18, Table 6 · p. 18

B.3 COMPUTATIONAL COST AND MEMORY FOOTPRINT

The appendix reports observed compute costs and specifies how signals, representations, interventions, and method comparisons are constructed and evaluated.

B.3 COMPUTATIONAL COST AND MEMORY FOOTPRINT

  • IEND variants are consistently more expensive than corresponding contrastive-mean methods, especially gradient-based AGIEND and GRADIEND; parameter-gradient methods also require substantially more memory.These are rough runtime indicators for evaluated implementations, not intrinsic complexity estimates; SAE pretraining is excluded from the reported cost.Source: §B.3 · pp. 18–19, Figure 5 · p. 19

B.4 SIGNAL EXTRACTION

  • Activation values average residual-stream representations over the filled target span, while gradient signals condition on the first target token through the causal-language-modeling loss.Activation gradients differentiate that loss with respect to the preceding prediction-position residual stream; parameter-gradient methods differentiate it with respect to parameters.Source: §B.4 · pp. 19–20

B.5 TRAINING AND SELECTION

  • Representation selection uses validation detection for SAE and contrastive-mean methods, whereas IEND methods use all-layer scopes rather than a layer sweep.Neutral training examples are used only by IEND methods; validation data also supports threshold and intervention-strength selection.Source: §B.5 · p. 21

B.6 INTERVENTION EVALUATION

  • SAE interventions apply decoder directions for selected latents at the selected layer or simultaneously across corresponding layers.Candidate intervention magnitudes and preservation constraints are selected using validation data.Source: §B.6 · p. 21

B.7 AGGREGATION

  • Task scores average evaluated target classes equally, and higher-level summaries average task scores within models and then model means equally.Pairwise constructions evaluate both endpoints of each unordered class pair; one-sided constructions and SAE yield one score per target class.Source: §B.7 · pp. 21–22

B.8 STATISTICAL ANALYSIS

  • Bayesian method comparisons treat each matched model–task combination as one observation, with practical equivalence defined by a 0.01 ROPE.Individual examples, feature classes, and pairwise contrasts are not treated as independent observations.Source: §B.8 · p. 22

C DETAILED RESULTS

  • The detailed-results section provides additional analyses supporting and extending the main results.Figure 6 presents posterior probabilities that each row method performs better, worse, or practically equivalently to each column method.Source: C DETAILED RESULTS · p. 22, Figure 6 · p. 22

C.1 STATISTICAL ANALYSIS

The section presents fuller pairwise Bayesian comparisons and a representation-scope comparison based on held-out performance.

C.1 STATISTICAL ANALYSIS

  • Figure 6 reports posterior probabilities and decisions for whether each row method performs better, worse, or practically equivalently to each column method.It supplies the full pairwise comparison underlying the summary in Figure 4b.Source: §C.1 · p. 22
  • Table 8 defines scope differences as single-layer minus all-layer held-out performance, with win rates counting positive differences.The table caption specifies that single-layer representations are selected by validation detection.Source: Table 8 · p. 23
  • Figure 7 displays the distribution of single-layer minus all-layer differences for detection and intervention, with positive values favoring the selected single layer.Source: Figure 7 · p. 23

C.2 LAYER LOCALIZATION AND REPRESENTATION SCOPE

The analyses find that detection often favors localized representations, while intervention, SAE choices, task family, feature specification, and optimization each show distinct patterns or constraints.

C.2 LAYER LOCALIZATION AND REPRESENTATION SCOPE

  • Single layers improve held-out detection for targeted contrastive-mean methods, but intervention shows no comparable localization advantage and the main comparison remains qualitatively unchanged with all-layer scopes.For SAE, validation preference for single layers does not yield a consistent held-out detection advantage; exhaustive layerwise analyses are limited to GPT-2 and Pythia-70M for computational tractability.Source: §C.2 · pp. 23–24, Figure 8 · p. 24

C.3 SPARSE AUTOENCODER VARIANTS

  • Increasing selected SAE latents from k = 1 to k⋆ changes performance modestly, while pre-target selection lowers detection across models and has model-dependent intervention effects.Pre-target selection increases intervention on Llama-3.1-8B but provides no advantage on Pythia-70M or GPT-2, so filled-target SAEk=1 remains the main reference.Source: §C.3 · p. 25, Figure 11 · p. 25

C.4 TASK CHARACTERISTICS

  • Detection is generally high on factual, grammatical, and semantic tasks, while language tasks show larger method differences; intervention is largest on factual tasks and small on language tasks.Semantic and algorithmic tasks generally fall between those intervention extremes, while grammatical tasks tend to show somewhat weaker effects.Source: §C.4 · p. 26, Figure 12 · p. 26

C.5 PAIRWISE AND ONE-SIDED FEATURE SPECIFICATIONS

  • One-sided construction often modestly improves detection and usually matches pairwise intervention, while remaining applicable when natural examples are available only for the target class.The direct comparison excludes IOI, REPETITION, FUNCCOMP, KEYVALUE, and INDUCTION because both constructions are not available for those tasks.Source: §C.5 · p. 26, Figure 13 · p. 26

C.6 IEND OPTIMIZATION ACROSS SIGNAL SPACES

  • Optimization choices for IEND are signal-dependent: ACTIEND uses a larger decoder learning rate and 100 steps, while gradient-based variants retain shared rates and 500 steps.The supporting optimization experiments are small, single-model, single-task ablations and are intended to motivate choices rather than establish general laws.Source: §C.6 · p. 27

C.7 COMPLETE TASK-LEVEL RESULTS

This section reports task-level quantities underlying the aggregate results and provides per-model views of averaged results.

C.7 COMPLETE TASK-LEVEL RESULTS

  • For completeness, Tables 9–14 report task-level quantities underlying all aggregate results; cited captions identify detection and intervention raw results for Pythia-70M and GPT-2, plus detection raw results for Llama-3.1-8B.Source: §C.7 · p. 27, Table 9 · p. 29, Table 10 · p. 30
  • Figures 14–16 provide model-based views of the averaged results shown in Figure 4a.Source: §C.7 · p. 27
  • Figures 15 and 16 show GPT-2 and Llama-3.1-8B detection and intervention scores per task, respectively.Source: Figure 15 · p. 28, Figure 16 · p. 28
Loading 2609.32355v1…