PaperlayerView exact revision on arXiv

Source-linked AI summary

When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoders

Xiaozuo Shen, Yifei Cai, Tian Tan, Rui Ning, Chunsheng Xin, Hongyi Wu

arXiv:2609.36294v1cs.LGcs.CL

TL;DR

Existing structured SAEs constrain feature relations to single-parent trees, while post-hoc graphs do not ensure reliable recovery. AG-SAE evaluates complete parent sets and uses the resulting mixed-topology graph to guide SAE training and dictionary refinement. It reports stronger structural reliability on real activations and exact recovery in a controlled toy model.

  • Problem

    Structured SAEs impose single-parent relations, while post-hoc multi-parent graphs can include redundant or spurious associations, leaving reliable mixed-topology recovery unresolved.

    Source: §1 · pp. 1–2

  • Method

    AG-SAE evaluates each feature’s complete parent set against null, subset, and alternative explanations, then uses the induced graph to guide SAE training and dictionary refinement.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    69.56% overall joint validity exceeds the full-space post-hoc baseline by 28.93 percentage points, and AG-SAE leads evaluated methods on real-activation reliability measures.

    Source: §4.3 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The recovered mixed-topology structure serves as an unsupervised training signal for the dictionary and supports feature-level causal interventions beyond reconstruction.

    Source: §1 · pp. 2–3, §5 · p. 9

Abstract

from arXiv · show

Sparse autoencoders (SAEs) expose interpretable features in large language model activations, yet existing structured SAEs impose single-parent trees or forests, while post-hoc graphs permit multiple parents but neither guide feature learning nor ensure reliable relation recovery. We introduce the Adaptive Graph Sparse Autoencoder (AG-SAE), a structure-guided training paradigm that treats each feature's complete parent set as an atomic structural hypothesis and lets evidence select zero, one, or multiple parents. By competing complete parent sets against null, subset, and alternative explanations, AG-SAE identifies jointly necessary multi-parent relations while rejecting redundant or spurious alternatives and verifying that each child contributes beyond its parents. The induced topology over SAE features then defines a differentiable structural loss that guides SAE training, while topology-guided refinement mitigates feature absorption and uses persistent reconstruction gaps exposed by the learned structure to initialize new features. The entire graph is then induced again from the revised dictionary by reassessing every feature's complete parent set, closing the dictionary-graph self-consistency cycle. Experiments demonstrate exact mixed-topology recovery in a controlled toy model, greater relational reliability and semantic validity than structured and post-hoc baselines on real LLM activations, and stronger feature-level causal interventions than conventional SAE features. AG-SAE thereby turns recovered mixed-topology feature structure into an unsupervised training signal that improves the dictionary, enables reliable feature organization beyond the topological limitations of trees, and exhibits stronger causal control beyond reconstruction.

1 INTRODUCTION

SAEs expose interpretable features, but existing approaches either leave feature relations unmodeled or constrain them to single-parent trees. AG-SAE recovers complete parent sets and uses the induced structure to guide dictionary learning and refinement.

1 INTRODUCTION

  • Single-parent trees can omit dependencies needed to explain a child, while reconstruction, sparsity, and feature-quality metrics do not assess relational fidelity.Post-hoc multi-parent edges also risk spurious relations from co-occurrence, similarity, or redundancy.Source: §1 · pp. 1–2
  • AG-SAE uses the induced topology to guide feature-specific encoder and decoder updates, reallocates parent-aligned content, and seeds new features from persistent reconstruction residuals.It re-induces the graph after dictionary updates, closing a dictionary–graph self-consistency cycle.Source: §1 · pp. 2–3
  • AG-SAE reports greater relational and semantic fidelity than structured SAE and post-hoc baselines, plus more effective causal interventions than conventionally trained SAE features.The cited introduction describes these as experimental findings, without giving quantitative values.Source: §1 · pp. 2–3
  • AG-SAE evaluates complete parent sets against null, subset, and alternative explanations, allowing zero, one, or multiple parents while rejecting redundant or spurious relations.It also verifies that each child contributes beyond its parents.Source: §1 · pp. 2–3

2 RELATED WORK AND POSITIONING

Prior approaches either constrain learned feature relations to single-parent trees or recover multi-parent associations post hoc, where spurious relations remain a concern. AG-SAE positions complete-parent-set evaluation as a way to extend SAE training while assessing those relations.

2 RELATED WORK AND POSITIONING

  • Structured SAEs learn feature-level relations but restrict children to one parent; post-hoc analyses reveal multi-parent associations that co-activation and redundancy can make spurious.The passage cites structured SAE work (Muchane et al., 2025; Luo et al., 2026; Cao et al., 2026) and post-hoc work (Grandien et al., 2026).Source: §2 · p. 3
  • The identified gap is extending SAE training beyond single-parent trees while distinguishing jointly necessary parent sets from redundant or spurious associations.The passage says existing approaches do not resolve this gap reliably from LLM activations.Source: §2 · p. 3
  • AG-SAE evaluates complete parent sets against null, subset, and alternative explanations, then uses the induced topology to guide training and refinement before re-inducing the graph.Appendix A is identified as providing the full discussion.Source: §2 · p. 3

3 ADAPTIVE GRAPH SPARSE AUTOENCODER

AG-SAE iteratively learns a feature dictionary and mixed-topology graph, using complete parent-set induction to guide training and dictionary refinement. It re-induces the graph under the updated dictionary and stops when graph and feature contributions stabilize.

3 ADAPTIVE GRAPH SPARSE AUTOENCODER

  • Each refinement cycle induces a graph from the current dictionary, trains the SAE with reconstruction and graph-guided structural losses, refines the dictionary, and re-induces the graph.The structural loss feeds the induced relations back into dictionary optimization.Source: §3 · pp. 3–4

3.1 COMPLETE PARENT-SET INDUCTION I

  • Complete Parent-Set Induction jointly evaluates candidate parent sets, allowing different parent-set sizes to compete and retaining a set only when its score exceeds alternatives by a margin.Candidates must provide parent-set activation coverage and jointly match the child representation; the child must also add reconstruction value beyond its parents.Source: §3.1 · p. 4

3.2 MIXED-TOPOLOGY-GUIDED DICTIONARY REFINEMENT R

  • Absorption Realignment separates parent-aligned from child-specific representation, routing the former toward parent directions while retaining the latter with the child.The allocation adapts during training and is intended to mitigate local feature absorption.Source: §3.2 · p. 5
  • Residual Completion uses persistent residuals associated with active parent–child relations to fill representational gaps, reusing a matching feature or learning a new sparse feature.The new feature captures a persistent residual component exposed by the learned relation.Source: §3.2 · p. 5
  • Full Re-Induction reassesses every feature’s parent set after dictionary refinement, and AG-SAE stops when graph structure and feature contributions stabilize across cycles.Stability is assessed on held-out activations using graph and feature-contribution changes.Source: §3 · p. 4, §3.2 · p. 5

4 EXPERIMENTAL EVALUATION

The evaluation tests mixed-topology recovery on a controlled toy model, relational reliability on real LLM activations, causal interventions, and core-component ablations. Across these settings, AG-SAE recovers reliable relations and concentrates stronger causal control in leading features.

4.1 EXPERIMENTAL SETUP

  • The study compares matched four-scale dictionaries and training budgets, then freezes dictionaries, graphs, and thresholds for disjoint report-only evaluation across toy, real-activation, causal-intervention, and ablation tests.The main setting is Gemma-2-2B layer-13 residual-stream features on MiniPile, with cross-model and cross-domain evaluations also specified.Source: §4 · pp. 5–6, §4.1 · p. 6

4.3 RELIABLE STRUCTURE LEARNING FROM REAL LLM ACTIVATIONS

  • 69.56% Joint validity exceeds full-space Vanilla SAE Post-hoc by 28.93 points; AG-SAE leads all six single- and multi-parent reliability measures.For multi-parent relations, AG-SAE reaches 97.00% PSV, 91.88% NR, and 75.00% SV.Source: Table 1 · p. 6, §4.3 · p. 7
  • 23.55% of parented features are multi-parent, and AG-SAE learns an eight-edge path across nine feature levels despite the four predefined levels in Tree SAE and HSAE.Across refinement cycles, held-out relation gain rises from 4.56% to 9.12%.Source: Table 1 · p. 6, §4.3 · p. 7

4.2 EXACT MIXED-TOPOLOGY RECOVERY IN A CONTROLLED TOY MODEL

  • At matched sparsity, AG-SAE recovers all 24/24 toy-model parent sets and rejects all 32/32 hard negatives, while Tree SAE and HSAE recover none of eight two-parent sets.Vanilla SAE also recovers all features with R2 = 1.00, but has only default no-parent matches.Source: §4.2 · p. 7

4.3 RELIABLE STRUCTURE LEARNING FROM REAL LLM ACTIVATIONS

  • For child 7707, all three parents are active in 91.32% of child-active positions, while any individual parent activates with the child in at most 38.44% of its activations.The learned complete parent set is {5963, 14785, 3242}.Source: §4.3 · pp. 7–8
  • The complete parent set scores 0.5497 on held-out fit, above the best single parent (0.2452), two-parent subset (0.4543), and same-cardinality alternative (0.4627).Removing any parent lowers the score by at least 0.0954; the child also contributes a normalized innovation gain of 0.06177.Source: §4.3 · p. 8

4.4 AG-SAE FEATURES ENABLE STRONGER CAUSAL INTERVENTIONS

  • 21/26 concepts improve in both intervention directions; addition rises from 4.22 to 4.79 logits and ablation from 3.23 to 3.91 logits, with matched counterfactuals tying the advantage to AG-SAE.Causal interventions use 416 held-out prompts across 26 first-letter concepts.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

4.5 CORE COMPONENT ABLATIONS

  • On a fixed dictionary and candidate space, complete parent-set induction raises multi-parent PSV from 34.80% to 66.13%; one realignment update reduces absorption from 42.08% to 28.11%.EV, L0, and dictionary capacity are identical in the induction comparison; realignment changes EV by −0.000040 and mean L0 by +0.0258.Source: §4.5 · p. 9

5 CONCLUSION

AG-SAE learns mixed-topology feature relations by evaluating complete parent sets and feeding the induced graph back into dictionary learning and refinement. The paper reports exact toy recovery, stronger real-activation relation reliability, and stronger causal interventions in leading features.

5 CONCLUSION

  • AG-SAE reinduces complete parent sets after dictionary refinement; it recovers all 24 toy-model parent sets, achieves 69.56% overall joint validity on real activations, and improves both intervention directions for 21 of 26 concepts.The conclusion frames recovered structure as an unsupervised training signal with functional benefits beyond reconstruction quality.Source: §5 · pp. 9–10

AI USE STATEMENT

  • Generative AI tools assisted with literature search and summarization, code editing and debugging, experimental-design feedback, result interpretation, and manuscript polishing.Source: AI USE STATEMENT · p. 10

A RELATED WORK

B ADDITIONAL METHOD DETAILS

Appendix B details how AG-SAE induces complete parent sets, trains against their structural criteria, refines the dictionary using the graph, and re-induces the graph from the revised dictionary.

B.1 COMPLETE PARENT-SET INDUCTION I

  • Complete Parent-Set Induction compares each candidate with subsets and equal-size alternatives, then adds accepted parent sets jointly when they preserve acyclicity.Children left unassigned retain empty parent sets.Source: §B.1 · p. 16

B.2 STRUCTURAL TRAINING LOSS LI

  • The fixed-graph structural loss supplies differentiable surrogates for induction criteria, while Full Re-Induction reassesses the updated dictionary using the original criteria.The surrogates include a smooth activation-coverage gate and discourage the child-specific decoder remainder from vanishing into parent-aligned allocation.Source: §B.2 · pp. 16–18

B.3 MIXED-TOPOLOGY-GUIDED DICTIONARY REFINEMENT R

  • Graph-guided refinement realigns child contributions through parent coordinates and uses conditioned residuals to reuse validated features or initialize new ones for later re-induction.A new feature does not inherit the parent set associated with its residual; Full Re-Induction determines its complete parent set.Source: §B.3 · pp. 18–19

B.4 FULL RE-INDUCTION AND JOINT STABILITY

  • Full Re-Induction reassesses every feature, including new ones, and the returned graph is induced from the returned dictionary after structural and feature-contribution stability tolerances are met.Existing parent sets can change as feature representations update.Source: §B.4 · p. 20

C THEORETICAL ANALYSIS OF GRAPH STABILITY IN FULL RE-INDUCTION

The theoretical analysis establishes that, under nondegeneracy conditions, sufficiently small dictionary changes preserve the graph produced by Full Re-Induction.

C THEORETICAL ANALYSIS OF GRAPH STABILITY IN FULL RE-INDUCTION

  • Under a nondegenerate stabilized state, sufficiently small dictionary changes preserve all complete parent sets, preventing repeated structural changes from blocking joint stopping.The assumptions include strict acceptance margins and strict selection gaps between the selected candidate and feasible alternatives.Source: C THEORETICAL ANALYSIS OF GRAPH… · p. 21, §C.1 · p. 22

C.1 SETUP AND ASSUMPTIONS

  • The analysis fixes a finite empirical induction configuration and requires a valid reference state with strict acceptance and selection margins.The configuration includes retrieval, fitting, comparison, and control-calibration data, batch contexts, randomness, and tie rules.Source: §C.1 · pp. 21–22

C.2 LOCAL STABILITY OF FULL RE-INDUCTION

Local stability follows by preserving the discrete induction decisions and controlling the continuity of refitted scores and thresholds around a nondegenerate reference state.

C.2 LOCAL STABILITY OF FULL RE-INDUCTION

  • Full-column-rank fitting makes NNLS solutions continuous locally, and the resulting innovation and direct-support scores are continuous.The proof also establishes continuity of calibrated thresholds and acceptance margins.Source: §C.2 · pp. 22–23
  • Within a neighborhood of a reference state satisfying Condition C.1, Full Re-Induction returns the same complete parent sets and global DAG.This includes features with zero, one, or multiple parents.Source: §C.2 · p. 22
  • Preserving the discrete execution branch keeps supports and conditional samples fixed; strict score gaps then preserve the acyclic-selection choices.The proof applies this argument over at most M selections.Source: §C.2 · p. 23
  • A stability radius can be bounded by the decision margins relative to their sensitivity to dictionary state, while preserving full rank and positive energies.The stated bound also preserves persistent ties.Source: §C.2 · p. 23

C.3 EVENTUAL GRAPH STABILITY AND JOINT STOPPING

Under the stated conditions, a stabilizing dictionary eventually yields a stable re-induced graph, so both graph and feature-contribution changes meet any fixed positive joint-stopping tolerances.

C.3 EVENTUAL GRAPH STABILITY AND JOINT STOPPING

  • For any fixed positive γG and γF, both joint-stability inequalities hold from some finite cycle onward.The theorem’s proof combines eventual graph equality with vanishing feature-contribution change.Source: §C.3 · p. 24
  • At a stopping cycle, the returned graph is induced from the returned dictionary, although positive tolerances can permit stopping before the graph is exactly constant.The empirical trajectories show dF and dG decreasing after G(2), consistent with the analyzed stabilization regime.Source: §C.3 · p. 24
  • After the dictionary stabilizes near a nondegenerate limit, Full Re-Induction eventually preserves all complete parent sets and the graph stabilizes.The result assumes a fixed finite, nonempty feature-identity set after some finite cycle and the stated continuity conditions.Source: §C.3 · pp. 23–24

D.1 ADDITIONAL ANALYSIS OF THE REFINEMENT PROCESS AND LEARNED MIXED TOPOLOGY

On Gemma-2-2B layer-13 MiniPile activations, refinement consolidates a mixed-topology graph with substantial multi-parent structure, stable relation assignments, and depth beyond the dictionary-bank hierarchy.

D.1 ADDITIONAL ANALYSIS OF THE REFINEMENT PROCESS AND LEARNED MIXED TOPOLOGY

  • After G(2), dF falls to 9.25% and dG to 3.06% by the final transition, while mean held-out relation gain rises from 4.56% at G(0) to 9.12% at G(5).Accepted Residual Completion updates fall from 12,245 in the first transition to 12 in the last, alongside consolidation of the dictionary–graph pair.Source: Figure 5 · p. 25, §D.1 · p. 25
  • Roots become less prevalent and leaves more prevalent as bank width increases, while 394 features exceed the three-edge strict scale-progression bound and graph depth reaches eight.The recovered graph includes within-bank and reverse-scale edges, so its paths are not constrained to a four-level progression.Source: §D.1 · p. 26
  • 1,670 of 7,091 parented features (23.55%) have multiple parents, and the two- and three-parent cohorts have Joint validity rates of 65.50% and 64.75%.These Joint rates differ by 0.75 percentage points; the arity-specific table evaluates PSV, NR, and SV on the same relations.Source: §D.1 · pp. 25–26, Table 2 · p. 26

D.2 ADDITIONAL ANALYSIS OF THE LEARNED MULTI-PARENT RELATION

Complete-parent comparisons support a specific multi-parent relation and a nonredundant child contribution; related experiments report validated mixed topology on PubMed under a corpus change.

D.2 ADDITIONAL ANALYSIS OF THE LEARNED MULTI-PARENT RELATION

  • The complete parent set for feature 7707 scores 0.5497, versus 0.2452 for its best single parent, 0.4543 for its best two-parent subset, and 0.4627 for the strongest evaluated matched alternative.Removing any parent reduces the score by at least 0.0954; replacing feature 5963 with 5295 trails the learned set by 0.0870.Source: §D.2 · p. 27, Table 3 · p. 27
  • On the 442 report positions where the child and all three parents are active, adding feature 7707 yields a normalized innovation gain of 0.06177.The separately fitted parent-only and parent-plus-child reconstructions show that the child retains representational value beyond its parents.Source: §D.2 · pp. 27–28

D.3 RESULTS ACROSS MODELS AND DATASETS

  • On PubMed, multi-parent relations comprise 24.82% of parented features and maximum graph depth reaches 11, reproducing the main mixed-topology pattern.Multi-parent shares decrease from 49.39% to 20.65% as bank width grows, with roots less prevalent and leaves more prevalent in wider banks.Source: §D.3.1 · p. 28, §D.3 · p. 28

D.3.1 VALIDATED MIXED TOPOLOGY PERSISTS UNDER A CORPUS CHANGE

  • Under the PubMed corpus change, multi-parent Joint validity is 60.75%, while single-parent Joint validity is 71.75%.More than three fifths of relations in each cohort simultaneously satisfy predictive, nonredundancy, and semantic criteria; model, layer, scales, and protocols are held fixed.Source: §D.3 · p. 28, §D.3.1 · p. 29

D.3.2 VALIDATED MIXED TOPOLOGY PERSISTS UNDER A NEW MODEL AND DICTIONARY CONFIGURATION

In Qwen3.5-2B-Base with a larger two-scale dictionary, AG-SAE recovers mixed topology and scale-dependent roles, with validated relations and graph depth beyond the scale hierarchy.

D.3.2 VALIDATED MIXED TOPOLOGY PERSISTS UNDER A NEW MODEL AND DICTIONARY CONFIGURATION

  • 3,137 of 9,562 parented features (32.81%) have multiple parents, and roots become less prevalent while leaves become more prevalent in the wider bank.This setting uses 49,152 total features across two scales, compared with 30,720 across four scales in the main setting.Source: §D.3.2 · p. 29
  • The graph reaches depth nine despite a one-edge maximum under strict progression from the narrower to wider bank, with 2,880 features beyond that bound.Those 2,880 features are 30.1% of all parented features.Source: §D.3.2 · p. 29
  • Single-parent and multi-parent Joint rates are 63.13% and 62.25%, respectively, so more than three fifths of each cohort satisfy all three validity criteria.The reported criteria are predictive, nonredundancy, and semantic validity.Source: §D.3.2 · p. 29

D.4 ADDITIONAL CAUSAL INTERVENTION RESULTS

Across first-letter concepts, AG-SAE improves bidirectional causal control over Vanilla SAE, including at Top-1 despite identical activation rates. Matched continuation tests show that this gain is specific to full AG-SAE training among the evaluated procedures.

D.4 ADDITIONAL CAUSAL INTERVENTION RESULTS

  • Under matched initialization, data order, training budget, and feature identities, only full AG-SAE training strengthens addition and ablation jointly, with advantages of 0.794 logits and 1.055 logits, respectively.Additional conventional SAE optimization and allowing decoder scale to vary do not reproduce the gain.Source: §D.4 · p. 30
  • AG-SAE gains 0.674 logits in bidirectional control across 21/26 concepts and 0.681 logits at Top-1 across 20/26, despite identical 85.1% activation rates.The paired gains over Vanilla SAE are reported in Table 6.Source: Table 6 · p. 30, §D.4 · p. 30
  • The gain spans addition and ablation: AG-SAE improves them by 0.573 and 0.685 logits across 24/26 and 21/26 concepts, respectively.Both paired confidence intervals are strictly positive; exact sign-test p-values are 1.05 × 10−5 and 0.0025.Source: §D.4 · p. 30

D.5 ADDITIONAL COMPONENT ABLATIONS

Complete parent-set induction improves structural reliability on a fixed dictionary, and dictionary optimization further improves it. Absorption realignment reduces absorption, while refinement trajectories are consistent with progressive dictionary–graph stabilization.

D.5 ADDITIONAL COMPONENT ABLATIONS

  • Complete Parent-Set Induction raises multi-parent PSV from 34.80% to 66.13% and PSV∩NR from 31.68% to 64.25% on the same dictionary and candidate space.Dictionary Optimization further raises PSV to 73.30% and PSV∩NR to 69.63%, gains of 7.17 and 5.38 percentage points over induction alone.Source: §D.5 · pp. 30–31, Figure 7 · p. 31
  • Absorption Realignment updates 12,607 of 15,306 diagnosed cases and resolves 10,817; Absorption Fraction falls from 42.08% to 28.11%.Full Absorption falls from 38.92% to 27.56%, and target absorption from 98.12% to 22.51%, with EV changing by only −0.000040 and mean L0 by +0.0258.Source: §D.5 · p. 31
  • Residual Completion updates decline from 12,245 to 12, while feature-contribution and graph changes fall to 9.25% and 3.06%, consistent with progressive dictionary–graph stabilization.Mean held-out relation gain rises from 4.56% to 9.12%, and the multi-parent population settles at 1,670 after peaking at 1,708.Source: §D.5 · p. 31

E EXPERIMENTAL SETUP AND HYPERPARAMETERS

  • The experimental setup specifies construction, baselines, evaluation protocols, and core training hyperparameters for the reported results.The supplied setup passage describes these components without reporting their detailed values.Source: E EXPERIMENTAL SETUP AND HYPERPARAMETERS · p. 32

E.1 TOY MODEL CONSTRUCTION AND RECOVERY EVALUATION

  • The toy model has 24 features in a three-level mixed-topology graph, with child activation requiring all immediate parents to be active.Exact parent-set recovery requires recovering the child and all true parents, with the complete predicted immediate parent set equal to ground truth; evaluation also tests rejection of hard negatives.Source: §E.1 · p. 32, Figure 8 · p. 32

E.2 LLM ACTIVATIONS AND BASELINE COMPARISONS

  • LLM comparisons use Gemma-2-2B block 13 activations from MiniPile and reference widths of 2,048, 4,096, 8,192, and 16,384.The setup also describes post-hoc Vanilla SAE, Matryoshka SAE, Tree SAE, and HSAE baselines, plus additional PubMed and Qwen experiments.Source: §E.2 · pp. 32–33

E.3 TRAINING CONFIGURATION

The paper specifies matched training budgets and AG-SAE hyperparameters, then evaluates structural, semantic, and causal outcomes under separate protocols. Semantic judgments use frozen feature descriptions, while causal interventions test addition and ablation on held-out prompts.

E.3 TRAINING CONFIGURATION

  • AG-SAE uses matched update budgets for the main baseline comparison, stopping at joint dictionary–graph stability or the budget ceiling, whichever comes first.The returned graph is freshly induced from the final dictionaries; conventional baselines receive their specified training schedules.Source: §E.3 · p. 33

E.4 AG-SAE HYPERPARAMETERS

  • Gemma and Qwen use parent sets of at most three, 24 retrieved candidates, and an exact-search pool of 12; training is capped at 16 cycles before fresh re-induction.The structural weight targets a 3% gradient contribution with a 2,000-update ramp, and joint stability is assessed over 262,144 validation samples.Source: §E.4 · p. 34

E.5 STRUCTURAL AND SEMANTIC EVALUATION

  • Structural evaluation freezes dictionaries, graphs, and thresholds before reporting; PSV compares complete parent sets with null, subset, and matched alternatives, while NR tests parent-only reconstruction.Semantic evaluation freezes feature descriptions, uses identical prompts and evidence counts across methods, and requires every parent to be relevant for multi-parent coverage.Source: §E.5 · pp. 34–35

E.6 CAUSAL INTERVENTION PROTOCOL

  • Causal evaluation compares AG-SAE and Vanilla SAE on 26 first-letter concepts using 416 held-out prompts and matched random-direction and unrelated-feature controls.Feature identities, thresholds, and addition scales are frozen before evaluation; addition and ablation are measured at the queried token’s layer-13 residual stream.Source: §E.6 · p. 36
Loading 2609.36294v1…