PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioral Compositionality of Sparse Autoencoder Representations

Haotian Ma, Ziye Mei, Frederic Sala

arXiv:2610.21261v1cs.LGcs.AIcs.CL

TL;DR

SAE features are used to inspect computation and build joint explanations or interventions, but their downstream effects may not add up. This paper measures behavioral residuals across feature subsets and compares learned SAE coordinates with matched rotations. Learned coordinates have lower first-order residuals, yet pairwise terms improve held-out coalition ordering across the tested settings.

  • Problem

    SAE evaluations characterize reconstruction, sparsity, and interpretability, but do not establish when individually characterized features compose into faithful joint explanations or interventions.

    Source: §1 · p. 1

  • Method

    The paper measures downstream responses to subsets of selected SAE contributions with SAE Additivity Residuals (SAR), comparing learned coordinates with sum-preserving rotations and modeling higher interaction orders.

    Source: §1 · pp. 1–2

  • Results

    Learned SAE coordinates have lower first-order residuals than tested rotations, while pairwise terms improve held-out coalition ordering in all 18 settings and outperform parameter-matched random third-order controls.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Individual feature effects can remain insufficient for joint behavioral explanations, even though learned SAE coordinates support lower-order descriptions than the tested rotations.

    Source: §5 · p. 10

    First-order additivity is guaranteed when the downstream continuation is affine over the tested activation states.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Sparse autoencoders (SAEs) represent model activations with sparse, linearly decoded features, but the downstream effects of those features need not compose additively. We introduce SAE Additivity Residuals (SAR), which measure the behavioral variation left unexplained above a chosen interaction order. Under independent feature subset sampling, population SAR is exactly the normalized energy of interactions above that order and bounds coalition ordering error for the optimal low order approximation. To distinguish compositionality associated with the SAE representation from that induced by downstream computation, we compare learned SAE coordinates with sum preserving random rotations of the same selected subspace and find lower first order residuals for the learned coordinates across all 18 model and behavior settings. The coordinates are nevertheless not fully additive: pairwise terms improve held out coalition ordering in every setting and outperform parameter matched models using randomly selected third order terms. In a combined steering experiment, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and reduces ordering error from 0.1444 to 0.0606. Together, these results show that SAE coordinates support comparatively compositional behavioral descriptions while faithful joint explanations and interventions still require interactions beyond individual feature effects. Code is available at https://github.com/OpenSAE/SAE-Behavioral-Compositionality.

1 Introduction

The paper asks whether the downstream behavioral effects of individually characterized SAE features compose into faithful joint explanations and interventions. It introduces a framework for measuring this compositionality and finds that learned SAE coordinates support lower-order descriptions than tested rotations, while pairwise interactions improve coalition predictions.

1 Introduction

  • Existing SAE evaluations characterize reconstruction, sparsity, and interpretability, but do not establish when independently characterized features can form faithful joint explanations or interventions.The paper motivates this gap because circuits, explanations, and interventions combine multiple components.Source: §1 · p. 1
  • The framework retains or removes selected decoded contributions while holding other activation content fixed, then propagates the patched activation through frozen upper layers to measure downstream responses.The resulting response over feature subsets tests whether individual effects predict joint effects.Source: §1 · p. 1, Figure 1 · p. 2
  • SAE Additivity Residuals (SAR) measure behavioral variation unexplained above a chosen interaction order, and under independent subset sampling this residual equals the normalized energy of higher-order interactions.The residual also controls coalition-ordering error for the optimal low-order approximation.Source: §1 · p. 2
  • The main experiments cover three model and SAE pipelines and six behavioral readouts, including copying, agreement, pronoun preference, CounterFact, IOI, and greater-than comparison.Additional analyses span seven pipelines and multiple representation depths.Source: §1 · p. 2
  • Learned SAE coordinates have lower first-order residuals than tested sum-preserving rotations, while pairwise terms improve held-out coalition ordering in all 18 settings and outperform random third-order controls.The framework compares learned coordinates with rotations of the same selected subspace and evaluates interaction terms against parameter-matched controls.Source: §1 · p. 2
  • Pairwise modeling raises rank fidelity from 0.5874 to 0.7790 in a combined SAE steering experiment.The introduction also reports that SAR and reconstruction quality differ in a matched six-SAE comparison.Source: §1 · p. 2

2 Related Work

Prior SAE evaluations characterize properties such as reconstruction and interpretability, while work on explanations and interventions combines multiple features or components. The paper positions its contribution as measuring whether learned SAE coordinates compose behaviorally downstream and how their interactions are distributed across orders.

2 Related Work

  • Prior SAE work examines reconstruction, sparsity, interpretability, and higher-order decoder terms, but not whether fixed SAE coordinates compose behaviorally after downstream model computation.The cited work includes reconstruction residuals, latent concepts, multidimensional feature descriptions, and polynomial SAEs.Source: §2 · p. 3
  • Multi-feature explanations, circuits, and internal interventions combine components, but characterizing components individually does not establish a faithful low-order joint description.This motivates measuring compositionality directly in downstream responses.Source: §2 · p. 3
  • Interaction-index research studies joint effects, while this paper treats learned sparse SAE features as structured players and examines downstream interactions across orders.Appendix C relates the framework to interaction indices and coalition weighting.Source: §2 · p. 3

3 Behavioral Compositionality of SAE Representations

The section constructs a controlled set-function of downstream behavior over selected SAE-feature subsets, then uses low-order approximation residuals to characterize interactions and coalition-order preservation.

3.1 From SAE features to controlled subset responses

  • The controlled response retains selected decoded features while fixing the remaining activation and downstream model; sum-preserving random rotations provide a matched coordinate baseline.The fixed remainder includes reconstruction residual and unselected content, so the full coalition recovers the original activation.Source: §3.1 · pp. 3–4

3.2 Measuring interaction beyond individual feature effects

  • SAR_r is the fraction of behavioral variance unexplained by the best order-r model; under independent subset sampling it equals interaction energy above r and vanishes when order r suffices.SAR decreases as interaction order increases; SAR_1 captures interactions beyond individual effects, while SAR_2 leaves structure beyond pairwise effects.Source: §3.2 · pp. 4–5

3.3 Connecting low order approximation to coalition ordering

  • Small SAR_r guarantees small behavior-weighted ordering error for the optimal order-r approximation, although the converse need not hold.The ordering metric emphasizes reversals between coalitions with larger behavioral differences and downweights near ties.Source: §3.3 · p. 5

3.4 Estimating compositionality from sampled coalitions

  • Because exhaustive evaluation costs 2^K, low-order models are fit on sampled coalitions and evaluated on unseen subsets; estimated SAR includes finite-sample fitting error.The held-out predictive residual is distinct from population SAR because it also reflects fitting error.Source: §3.4 · p. 6

4 Experiments

Across three main model and SAE pipelines, six behaviors, and additional depth and steering analyses, the experiments test how well feature effects compose and how interaction terms improve coalition predictions. First-order residuals vary by behavior and representation; learned SAE coordinates outperform tested rotations, while pairwise models improve held-out coalition ordering.

4 Experiments

  • The main comparisons cover three model pipelines and six behavioral readouts, with 1,000, 100, 800, and 150 prompts per setting across the fixed-layer, rotation, fidelity, and pairwise analyses.Each prompt uses the top K = 16 active SAE contributions and 384 sampled coalitions, split into 256 for fitting and 128 for evaluation.Source: §4 · p. 6

4.1 Behavioral compositionality varies across behaviors and representations

  • First-order residuals are positive in all 57 model, layer, and behavior combinations, and their magnitude varies across behaviors and representations.Simple copying and subject verb agreement have lower residuals than CounterFact in all three main pipelines.Source: §4.1 · p. 6, Figure 2 · p. 7

4.2 Compositionality depends on coordinate choice under subset interventions

  • Across all 18 comparisons, learned SAE coordinates have lower median d SAR1 than sum-preserving rotations of the same selected subspace.For Mistral subject verb agreement, the medians are 0.0218 for learned coordinates and 0.1196 for rotations.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 Estimated SAR is associated with first order coalition fidelity

  • Across 18 model and behavior combinations, d SAR1 correlates negatively with rank fidelity in 17, and the correlation is at most −0.6 in 14.From the lowest to highest d SAR1 decile, rank fidelity falls and selection regret rises in all three models.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 Feature pairs recover missing coalition structure

  • 0.9782 to 0.9959 average rank fidelity, with selection regret falling from 0.0272 to 0.0051, when pairwise terms replace first-order-only models across all 18 settings.Pairwise models also outperform parameter-matched controls with randomly selected third-order terms in all 18 combinations.Source: §4.4 · p. 8

4.5 Estimated compositionality varies across depth

  • Across nine early-to-late comparisons, d SAR1 decreases in eight and bρ2 increases in all nine; SAE coordinates also have lower d SAR1 than rotations at all 33 matched sites.Random rotations show the first-order decrease in eight of nine comparisons, and Llama profiles are nonmonotonic near the final sampled layer. Across seven pipelines, simple copying has lower latest-layer d SAR1 in six cases and higher bρ2 in all seven.Source: §4.5 · pp. 8–9, Figure 6 · p. 9

4.6 Pairwise modeling better predicts the aggregate steering response

  • 0.5874 to 0.7790 rank fidelity and 0.1444 to 0.0606 behavior weighted ordering error with pairwise modeling on the overall 1,400-question MMLU-Pro steering response.The held-out residual estimate decreases from 0.6424 to 0.3891; domain-level effects are heterogeneous.Source: §4.6 · p. 9, Table 1 · p. 9

4.7 Reconstruction quality and SAR differ in a matched SAE comparison

  • In a matched comparison across six SAEs at one Gemma-3 1B site, SAR relates positively to sign error and selection regret across three behaviors, unlike FVUx.FVUx is negative or near zero for sign error and near zero for selection regret, so reconstruction quality and SAR capture different variation in joint behavioral fidelity in this comparison.Source: §4.7 · p. 9, Table 2 · p. 10

5 Discussion and Conclusion

SAE coordinates are additive for activation reconstruction, but that does not make their downstream behavioral effects additive. The experiments show comparatively lower first-order residuals for learned coordinates than tested rotations, while pairwise terms improve coalition prediction; these conclusions depend on the selected contributions, readout, and intervention distribution.

5 Discussion and Conclusion

  • SAE coordinates can have lower first-order residuals than tested rotations without making individual feature effects independent.Residual interactions can still matter for faithful joint explanations and interventions.Source: §5 · p. 10
  • Behavioral compositionality is distinct from reconstruction quality and individual feature interpretability.SAR measures the interaction order needed for joint behavior directly.Source: §5 · p. 10
  • The conclusions depend on selected contributions, behavioral readout, and intervention distribution, rather than holding as a global property of an SAE dictionary.Source: §5 · p. 10

A Proofs and Mathematical Details

The appendix establishes the interaction and projection identities underlying SAR, then describes the sampled-coalition estimator and its finite-sample fitting analysis.

A.1 Notation and equivalent bases

  • At p = 1/2, the indicator and Walsh bases span the same low-order function space through an invertible change of coordinates.The finite-sample analysis uses the indicator basis, while Walsh coordinates expose orthogonality.Source: §A.1 · p. 13

A.2 Proof of the behavioral interaction decomposition

  • Conditioning on features outside a subset and averaging over its memberships derives the behavioral interaction decomposition and its finite-difference expression.The proof uses the orthonormal basis property to establish the decomposition.Source: §A.2 · pp. 13–14

A.3 Projection properties

  • SAR_r is zero exactly when the response lies in A_r; the projection residual has mean zero and SAR is invariant to affine rescaling of the response.If the continuation is affine on tested states, the response is first-order and SAR is zero.Source: §A.3 · p. 14

A.4 Proof of the coalition ordering theorem

  • The ordering theorem bounds population behavior-weighted ordering error, and the normalized squared residual equals SAR_r when the approximation is the projection.The proof defines the residual as the difference between the response and its approximation.Source: §A.4 · p. 14

B.1 Projection estimator

  • For each prompt, the estimator fits a low-order projection on sampled coalitions and evaluates its residual and held-out variance on a separate split.The reported estimator is formed from these held-out quantities.Source: §B.1 · pp. 14–15

B.2 Walsh coordinates and fitting error

  • At p = 1/2, monomial and Walsh coordinates span the same fit class; Walsh orthogonality clarifies population conditioning and finite-sample fitting error.Full-rank fits give the same predictions in either basis.Source: §B.2 · p. 15

B.3 Finite sample diagnostics

The diagnostics distinguish stable comparisons from less stable residual magnitudes, while the following sections clarify SAR’s interpretation, intervention game, and scope.

B.3 Finite sample diagnostics

  • All sampled designs were full rank; the median second-order Walsh Gram condition number was 41.02 (95th percentile 49.88), and monomial and Walsh predictions agreed to numerical precision.The diagnostics used the 2,700 prompts from the explanation-fidelity experiments.Source: §B.3 · p. 15
  • Across fresh coalition draws, ordering conclusions were more stable than absolute residual magnitudes: rank fidelity varied by 0.37% and 0.16%, versus 13.7% and 18.2% for d SAR1 and d SAR2.The fitting-budget analysis also found pairwise ordering gains in all 18 cells at 224 and 256 fitting coalitions, while d SAR2 changed by about 21%.Source: §B.3 · pp. 15–16

B.4 Prompt aggregation and paired endpoints

  • Prompt summaries use medians for SAR quantities unless otherwise stated, means for primary pairwise fidelity measures, and prompt-bootstrap confidence intervals.Paired early- and late-layer comparisons resample the same prompt IDs.Source: §B.4 · p. 16

C.1 Interaction indices and low order fits

  • Unlike interaction indices that assign values to players or coalitions, SAR aggregates residual energy above a chosen order; under product sampling, pairwise coefficients are scaled context-averaged second differences.Low-order coefficients alone do not determine residual energy above that order, so estimation also evaluates held-out behavior.Source: §C.1 · pp. 16–17

C.2 Coalition weighting and variance shares

  • Shapley-style coalition weighting correlates feature memberships and removes the orthogonal variance-share interpretation of the SAR sequence; product sampling preserves it.The two weightings answer different questions.Source: §C.2 · p. 17

C.3 Alternative intervention games

  • The chosen intervention game keeps the reconstruction residual and unselected content fixed, anchoring the full coalition at the original activation.Reconstructing from retained features alone would discard those components and instead evaluate the SAE reconstruction.Source: §C.3 · p. 17

C.4 Scope of the zero point

  • The compositionality measure concerns downstream computation on tested states in the selected SAE coordinates, a subspace of dimension at most K.A continuation may be nonlinear outside this subspace while inducing an additive response on the tested states.Source: §C.4 · p. 17

D.1 Behavior Specifications

  • The behavior specifications list prompt sources and readouts; greater-than labels vary by tokenizer, while corresponding comparison templates and prompt identities are fixed.Gemma-3 and Llama use A/B labels, whereas Mistral uses True/False.Source: §D.1 · p. 18

D.2 Steering Specifications

The steering analysis fixes a selected feature set and evaluates coalition effects on MMLU-Pro; supplementary specifications describe the broader pipelines and affine calibration checks.

D.2 Steering Specifications

  • The steering experiment uses 16 positively correlated Llama-3.1 8B features with fixed published coefficients, evaluating coalition accuracy over 1,400 MMLU-Pro questions.The questions comprise 100 from each of 14 domains.Source: §D.2 · pp. 18–19, Table 7 · p. 18

D.3 Model and SAE Pipeline Specifications

  • The model and SAE pipeline specifications identify the model families, SAE releases, top-K active-feature selection, and fixed prompt, feature, coalition, and split identities within comparisons.The three main pipelines use unquantized model evaluation.Source: §D.3 · p. 19, Table 8 · p. 19

E Measurement Calibration

F Coordinate Comparisons

The sections compare coordinate and depth analyses and describe the explanation-fidelity evaluations, with tables documenting their full comparisons and endpoints.

F Coordinate Comparisons

  • Eight sum-preserving rotations per prompt test coordinate choice; across nine settings, half-magnitude removals lower d SAR1 and raise first-order rank fidelity in eight.Increasing feature inclusion from p = 0.5 to p = 0.75 preserves pairwise rank improvement in all nine settings.Source: F Coordinate Comparisons · p. 20, Table 10 · p. 20

G.1 Fixed Layer Behavior Comparisons

  • Table 11 reports fixed-layer behavior estimates for each combination using 1,000 prompts, with 95% prompt-bootstrap confidence intervals.Source: Table 11 · p. 21, §G.1 · p. 21

G.2 Paired Early and Late Comparisons

  • Table 12 reports paired early-to-late estimates for three detailed depth profiles, with paired-prompt bootstrap intervals for reductions.Source: Table 12 · p. 21, §G.2 · p. 21

G.3 Simple Copying Across Seven Pipelines

G.4 Greater Than Depth Profiles

H Explanation Fidelity Results

  • Sign error and selection regret supplement rank fidelity and behavior-weighted ordering error as explanation-fidelity measures.Sign error counts opposite signs for the response and residual; selection regret compares the best observed coalition with the approximation’s selected coalition.Source: H Explanation Fidelity Results · p. 23

H.1 Estimated SAR and First Order Fidelity

  • Table 15 reports first-order explanation fidelity for the lowest and highest d SAR1 deciles within each model.Figure 4 provides the corresponding decile comparison.Source: Table 15 · p. 23, §H.1 · p. 23

H.2 Pairwise Approximation

  • Tables 16–17 report first-order and pairwise explanation errors and mean prompt-level rank fidelity for all 18 model–behavior combinations.The supplied table descriptions establish coverage but do not give the comparison values.Source: Table 16 · p. 24, Table 17 · p. 24, §H.2 · p. 24

H.3 Behavior Weighted Ordering Error

The appendix reports held-out behavior-weighted ordering errors, compares pairwise terms with matched higher-order controls, and summarizes steering, SAE-basis, and prompt-level examples.

H.3 Behavior Weighted Ordering Error

  • Table 18 reports held-out behavior-weighted ordering errors for first-order and pairwise approximations across 18 model and behavior combinations.The values use the original fitting and evaluation split.Source: Table 18 · p. 25, §H.3 · p. 25

H.4 Pairwise Terms and Matched Higher Order Capacity

  • The matched controls retain the intercept and 16 first-order terms while replacing 120 pairwise terms with sampled third-order terms.Fifty controls are fitted per prompt using the pairwise model’s fitting and evaluation coalitions.Source: §H.4 · p. 26, Table 19 · p. 26

H.5 Steering Results Across MMLU-Pro Domains

  • Table 20 summarizes held-out residual estimates, rank fidelity, and F(2) for steering combinations across MMLU-Pro domains.The first-order and pairwise models are fitted on one coalition sample and evaluated on a separate sample.Source: Table 20 · p. 27, §H.5 · p. 27

H.6 Reconstruction and Behavioral Compositionality Across Six SAEs

  • Figure 8 compares median prompt-level FVUx and median d SAR1 across six SAE bases for copying, subject-verb agreement, and IOI.The accompanying text identifies this as the comparison across six SAE bases referenced in the main text.Source: Figure 8 · p. 28, §H.6 · p. 28

H.7 Examples

  • Figure 9 and Table 22 present selected prompts, their behavior scores under feature and feature-pair ablations, and residual values.First-order and pairwise predictions are included when available.Source: Figure 9 · p. 28, §H.7 · p. 28, Table 22 · p. 29

I Robustness and Protocol Checks

The robustness checks vary explanation size and sampling protocols, with the reported depth trends persisting under higher coalition inclusion probability and a larger prompt sample.

I.1 Sensitivity to the number of selected features

  • Changing K adds or removes SAE coordinates, so the reported values track compositionality as explanation size changes rather than convergence of a fixed quantity.Table 23 reports prompt-median d SAR1 for active feature sets at Gemma-3 1B layer 17, alongside affine controls and an overall MMLU-Pro steering row.Source: §I.1 · p. 29, Table 23 · p. 29
  • Increasing coalition inclusion probability from 0.5 to 0.65 preserves the decrease in d SAR1 across sampled layers for CounterFact, IOI, and simple copying.Mean and median aggregation yield the same depth directions.Source: §I.1 · p. 29
  • With 1,000 prompts, d SAR1 decreases in eight of nine comparisons and bρ2 increases in all nine.These checks preserve the main summary results.Source: §I.1 · p. 29

Related papers

Loading 2610.21261v1…