PaperlayerView exact revision on arXiv

Source-linked AI summary

HiPACE: Hierarchical Phase-Boundary Analysis and Controlled Evaluation of Feature Absorption in Sparse Autoencoders

Jinyuan Zhang, Peng He, Yin Yuan, He Hu, ShengShuo Jiao

arXiv:2609.29551v1cs.AI

TL;DR

Prior work documents feature absorption but does not predict when a shared parent direction is cost-optimal for an active semantic family. HiPACE derives a hierarchical phase boundary and tests its structural consequences in synthetic and real SAE dictionaries. The synthetic transition matches the prediction across tested cells, and the real-dictionary and intervention analyses support family-level structure and behavioral relevance.

  • Problem

    Prior work documents feature absorption empirically but does not predict when a shared parent direction is the cost-optimal representation of an active semantic family.

    Source: §1 · p. 1

  • Method

    HiPACE derives a phase boundary for hierarchical coding and evaluates its structural consequence with family-level SAE measurements, frozen unseen-family validation, sibling nulls, and mechanistic follow-up.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The synthetic onset agrees with λc(k⋆, α) within ±15% on all 30/30 cells, and real Pythia-160m dictionaries recover the predicted ordering on discovery and frozen holdout families against randomized sibling groups.

    Source: §1 · p. 2, Phase-Boundary Validation (RQ1) · p. 6

  • Takeaways & Limitations

    The results support a family-level structural signal in real SAE dictionaries and behavioral relevance of the recovered family subspace, within the tested settings.

    Source: §7 · p. 8

    The active-count experiment bridges the theory and empirical settings, but equivalence of their training objectives is not claimed.

    Source: §6 · p. 8

Abstract

from arXiv · show

Sparse autoencoders (SAEs) decompose LLM activations into sparse dictionary atoms, so that each distinct concept gets its own feature. One recurring behavior complicates this premise: feature absorption, in which a parent concept and its children--fruit and {apple, banana, pear}, say--collapse into a shared family direction. Prior work documents absorption empirically; missing is a closed-form prediction of when the shared direction is the cost-optimal representation of an active semantic family. This paper closes that gap. For a hierarchical Bernoulli generator with $k$ active children and residual scale $α$, the $L_0$-penalized reconstruction objective admits a closed-form phase boundary $λ_c(k,α)=α^2 k/(k-1)$: above it, pure parent absorption is strictly cheaper than pure child coding. Building on this boundary, we introduce HiPACE, an evaluation protocol that tests the boundary's structural consequence in real SAE dictionaries--measuring parent--child decoder structure over WordNet families, freezing the discovery-selected statistic before testing on unseen families, and contrasting genuine families against randomized sibling nulls. The boundary proves sharp in its native regime, predicting the synthetic transition within $\pm15%$ on all 30 tested cells. In Pythia-160m SAEs, the parent--child decoder gap recovers the predicted ordering with partial correlations up to $-0.93$ that sustain on the locked holdout and exclude sibling nulls ($p=0.002$). Controlled activation composition connects the theory's active-child count to the recovered family directions, and residual-stream interventions show that signed family directions increase parent-category logits, reversing under sign flip and vanishing under random controls--establishing causal sufficiency at the family-subspace level.

1 Introduction

Feature absorption can allocate a shared family direction to a parent and its children, but prior empirical accounts do not predict when this is cost-optimal. HiPACE derives a threshold and evaluates its structural and behavioral consequences in synthetic and real SAE settings.

1 Introduction

  • SAEs decompose LLM activations into sparse combinations of dictionary atoms, and feature absorption can replace distinct child features with a shared family direction.Prior work documents absorption empirically, but does not predict when that representation is cost-optimal.Source: §1 · p. 1
  • For a hierarchical Bernoulli generator, comparing shared-parent coding with child-specific residual coding yields λc(k, α) = α^2k/(k−1), turning absorption into a testable phase boundary.Here k is the number of active children and α is the child-residual scale.Source: §1 · p. 1
  • HiPACE measures parent–child structure across semantic families, freezes discovery-selected evaluation choices before testing unseen WordNet families, and uses randomized sibling groups to test dependence on genuine hierarchy.The protocol also connects the active-count prediction to real activations and tests recovered family directions with residual-stream interventions.Source: §1 · pp. 1–2
  • The paper contributes a hierarchical phase-boundary theory, including mixed-dictionary analysis and conditional threshold shifts for Matryoshka and Archetypal SAEs.The introduction also situates HiPACE as a family-level evaluation and mechanistic test of the theory's structural consequence.Source: §1 · p. 2
  • HiPACE's synthetic boundary matches all 30 tested cells, while real Pythia-160m dictionaries recover the predicted ordering on discovery and frozen WordNet holdout families against randomized sibling groups.Controlled activation composition and residual-stream interventions further connect active-child count to family directions and show that signed family directions change parent-category logits.Source: §1 · p. 2
  • The introduction positions the paper's contribution and then directs readers to its theory, evaluation, experiments, and discussion of scope.This is the paper's stated organization, not an experimental finding.Source: §1 · p. 2

2 Related Work

Prior work studies SAE design, absorption, sparse-coding thresholds, and dictionary benchmarks, but HiPACE differs by predicting hierarchical absorption costs and evaluating structure at the family level. Its frozen holdout and randomized sibling comparisons make that evaluation confirmatory and hierarchy-specific.

2 Related Work

  • SAE variants reshape sparsity or dictionary structure, and the paper predicts downward threshold shifts for Matryoshka and Archetypal training, which its synthetic sweeps confirm.The cited discussion also describes TopK, JumpReLU, Matryoshka, and Archetypal design differences.Source: §2 · p. 2
  • Prior absorption and hedging work documents latent collapse or encoder–decoder mismatch, but does not predict when collapse is cost-optimal; HiPACE supplies a closed-form threshold.The reported median encoder/decoder cosine of Pythia-160m modal latents is +0.94, which the paper uses to argue hedging does not explain its structural results.Source: §2 · p. 2
  • Classical sparse-coding and toy-superposition work studies threshold behavior in asymptotic or flat-feature settings, whereas HiPACE derives a closed-form critical penalty for hierarchical parent–child features.The paper also tests the structural consequence in learned dictionaries.Source: §2 · p. 2
  • Existing dictionary evaluations often score one prompt or feature at a time, while HiPACE audits parent–child decoder structure across semantic families and freezes its statistic before testing unseen WordNet families.The family-level audit reflects a generator-side prediction that does not depend on a unique recovered dictionary.Source: §2 · p. 2
  • HiPACE differs through a closed-form hierarchical absorption boundary, a family-level structural audit matched to its prediction, and a frozen-holdout protocol with randomized sibling nulls.These are the paper's three stated axes of distinction.Source: §2 · pp. 2–3
  • The paper's evaluation motivation links hierarchical feature competition to a predicted dictionary phase boundary and family-level measurement, frozen validation, and mechanistic testing.The figure caption distinguishes k, the active-child count, from K, the family size.Source: Figure 1 · p. 3

3 Methodology

HiPACE links a hierarchical family model and phase-boundary analysis to family-level SAE measurement, frozen validation, and mechanistic follow-up. In its pure-strategy comparison, the critical sparsity penalty marks when shared-parent coding becomes cheaper than child coding, with mixed dictionaries and architecture-specific assumptions qualifying the prediction.

3 Methodology

  • HiPACE combines a hierarchical family model, a phase-boundary analysis, family-level parent–child measurement, frozen holdout validation, and controlled activation and intervention follow-up.The evaluation translates the theory's family-level prediction into a structural measurement rather than requiring one particular latent to win on each prompt.Source: §3 · p. 3

Hierarchical Phase-Boundary Analysis

  • The model treats a unit-norm parent direction and mutually orthogonal unit-norm child residual directions, with α controlling residual magnitude.Each child is independently active with probability p, and k is the active-set size.Source: Hierarchical Phase-Boundary Analysis · p. 3
  • The L0-penalized reconstruction objective prices sparsity with λ, while TopK SAEs use a hard firing budget as an empirical analogue rather than an equivalent objective.The paper explicitly notes that λ is a marginal sparsity price, not a hard firing budget.Source: Hierarchical Phase-Boundary Analysis · p. 3
  • Theorem 1 says pure parent absorption is preferred to pure child coding iff λ ≥ λc(k, α) = α^2k/(k−1) under the hierarchical Bernoulli generator.The theorem compares the single-atom parent dictionary with the per-child dictionary; k is the number of active children and α is the residual scale.Source: Hierarchical Phase-Boundary Analysis · p. 4
  • The pure-strategy threshold does not cover mixed dictionaries, which can include a parent and residual atoms even when neither pure strategy dominates.The paper aggregates over the binomial active count and uses its modal value k⋆ for the macro 50%-onset prediction.Source: Hierarchical Phase-Boundary Analysis · p. 4
  • The cost equality follows because child coding has zero reconstruction error and sparsity cost λk, whereas parent absorption pays λ once plus residual error α^2k.Equating these costs gives λ(k − 1) = α^2k.Source: Hierarchical Phase-Boundary Analysis · p. 4
  • Under the stated Matryoshka condition that the smallest-level atom is pinned to the parent, the residual-firing threshold is λ = α^2, independent of k.This is a conditional shift hypothesis; the precise parent-alignment condition is deferred to supplementary material.Source: Hierarchical Phase-Boundary Analysis · p. 4
  • The Archetypal shift heuristic assumes contamination of convex-hull-constrained atoms by other residuals, which scales child-coding error by k/K and shifts the crossover.The passages state this heuristic's mechanism but do not provide a complete threshold expression.Source: Hierarchical Phase-Boundary Analysis · p. 5

Three absorption notions.

The paper distinguishes absorption notions and evaluates the structural prediction at the semantic-family level. The evaluation separates statistic development from a frozen unseen-family test and uses randomized siblings as a specificity control.

Three absorption notions.

  • Prior absorption measures target whether a latent fires on arbitrary subsets of concept members or whether it fires on all members, rather than the pure-dictionary notion used here.The paper cautions that reconciling claims across these absorption notions is non-trivial.Source: Three absorption notions · p. 5

Family-Level Structural Evaluation

  • HiPACE measures parent–child decoder structure for WordNet families because the theory predicts allocation across a semantic family, not the identity of one winning latent.Its correlations control for family size, which affects both the threshold proxy and the number of available child directions.Source: Family-Level Structural Evaluation · p. 5
  • The discovery-selected layer-6 family-mean statistic and layer-11 centered first principal component are frozen before testing an unseen WordNet holdout that excludes discovery parents and lemmas.Randomized sibling nulls substitute children from other families while preserving family size where possible.Source: Family-Level Structural Evaluation · p. 5
  • The frozen holdout is the confirmatory test, while discovery develops the statistic and sibling-null and intervention controls stress-test robustness around that core.The controlled design distinguishes theory-fixed synthetic testing, discovery selection, frozen validation, null testing, and exploratory mechanistic follow-up.Source: Family-Level Structural Evaluation · p. 5
  • The preregistered instance-level top-1 consensus endpoint exposed a 1/K family-size baseline, so the family-level statistic was developed on discovery data and its holdout pipeline frozen before scoring.The paper reports the preregistered endpoint's outcome and protocol amendments in the supplement.Source: Family-Level Structural Evaluation · p. 5

4 Experiments

The experiments test the predicted transition in controlled hierarchical representations, then evaluate family structure and behavioral relevance in real SAE dictionaries. The results support the boundary and family-level signal across synthetic validation, a frozen holdout, sibling-group controls, activation composition, and intervention.

Research Questions / Experimental Settings

  • The four research questions cover synthetic boundary validation, real-dictionary recovery and generalization, family specificity, and mechanistic relevance; real-SAE experiments use Pythia-160m and WordNet families.The study uses public TopK SAEs at layers 3, 6, 9, and 11, with 50 accepted discovery families and locked holdout filtering.Source: Research Questions · p. 5, Experimental Settings · p. 5

Phase-Boundary Validation (RQ1)

  • The observed absorption onset agreed with λc(k⋆, α) within ±15% in all 30 tested cells, with median absolute relative error 0.026 and maximum 0.053.Trained TopK SAEs recovered the predicted crossover within one budget in 12 of 16 cells; four K = 16 deviations retained a parent atom alongside child residuals in the mixed regime.Source: Phase-Boundary Validation (RQ1) · p. 6

Hierarchical Structure Recovery (RQ2)

  • Discovery partial correlations reached −0.681 (L6 mean gap) and −0.933 (L11 centered-PC1 gap), with frozen holdout values −0.605 and −0.914, respectively.Both layer-adaptive statistics replicated on unseen families.Source: Hierarchical Structure Recovery (RQ2) · p. 6

Specificity and Robustness (RQ3)

  • Both primary holdout statistics reached the minimum attainable p = 0.002 under 500 randomized sibling draws, while the nonprimary L11 mean statistic fell within the null (p = 0.108).The nulls do not jointly match token frequency, decoder norm, hierarchy depth, and lexical similarity.Source: Specificity and Robustness (RQ3) · p. 6

Mechanistic Relevance (RQ4)

  • As k rose from 1 to 4, parent-direction activation increased: L6 mean ρ = +0.963 and median Δ = +32.20; L9 mean +0.963 and +40.26; L11 PC1 +0.846 and +156.81.The controlled composition probe links active-child count to the trained SAE without assuming natural-language prompts isolate k.Source: Mechanistic Relevance (RQ4) · p. 6
  • Parent-signed L6 injection increased the true-parent logit margin, reversed under sign flip, and exceeded random controls; the child direction produced the same median shift.The result localizes behavioral relevance to the family subspace rather than a unique parent atom.Source: Mechanistic Relevance (RQ4) · p. 6

5 Discussion

The discussion frames the boundary as a cost-based account of when shared parent coding is favored over distributed child residuals. It argues for family-level, holdout-based evaluation while limiting causal attribution to the family subspace.

What the boundary explains

  • The boundary identifies when sparse coding favors a compact shared direction over distributed child residuals; synthetic experiments validate this transition where the assumptions hold by construction.In real SAEs, its measurement connects both to family-level decoder gaps and to active-count response under activation composition.Source: §5 · p. 6

Implications for SAE evaluation

  • SAE absorption evaluations should specify the predicted structural unit, separate metric development from validation, test genuine families against matched or randomized groupings, and use interventions for behavioral relevance.Layer dependence cautions against treating one decoder atom as canonical across the model.Source: §5 · p. 7

Scope of the causal evidence

  • Comparable parent and child intervention effects localize causal evidence to a family-aligned subspace rather than a unique parent atom.A finer decomposition requires orthogonalized parent-versus-child interventions and family-level confidence intervals.Source: §5 · p. 7

6 Limitations

The limitations concern endpoint development, differences between the theoretical and implemented objectives, and the scope of generalization and causal identification. The reported intervention evidence establishes sufficiency under controlled perturbations, not necessity during natural prompting.

Endpoint development and preregistration

  • The preregistered instance-level proxy returned ρ = +0.41 at the primary L6 cell, opposite the predicted ordering, with baseline analysis tracing inflation to family size.The family-level statistic was developed on discovery data and frozen before the independent WordNet holdout was scored.Source: §6 · p. 7

Theory–implementation gap

  • The derivation assumes an explicit L0 penalty and sample-level active count k, whereas the real-model SAEs use TopK gating and structural analysis uses family-level measurements.The active-count experiment bridges the settings empirically; equivalence of the training objectives is not claimed.Source: §6 · pp. 7–8

Generalization and residual confounding

  • The structural pipeline is established on Pythia-160m and one public SAE family, while WordNet relations are only a proxy for internal conceptual hierarchy.A joint matched null for frequency, decoder norm, hierarchy depth, and lexical similarity is the stated extension for remaining confounding.Source: §6 · p. 8

Causal identifiability

  • Interventions establish sufficiency under controlled residual-stream perturbations, but necessity during natural prompting remains for follow-up.Comparable parent and child effects place attribution at the family-subspace level.Source: §6 · p. 8

Reproducibility

  • The submission provides code, configurations, and per-run outputs, with total compute under two GPU-hours on one contemporary GPU.All amendments were dated before the affected measurements.Source: §6 · p. 8

7 Conclusion

HiPACE derives an active-count phase boundary for controlled synthetic transitions and evaluates its structural consequence in real SAE dictionaries. Frozen-holdout, sibling-null, composition, and intervention results support a family-level signal, while matched-null construction and parent-versus-child decomposition remain follow-ups.

Conclusion

  • HiPACE predicts controlled synthetic transitions and finds a genuine family-level signal in real SAEs, with behavioral relevance at the family-subspace level.The matched-null construction and parent-versus-child decomposition are identified as immediate follow-ups.Source: §7 · p. 8

A Proof of Theorem 1

The proof compares pure child coding with parent absorption to derive a cost threshold, then analyzes mixed dictionaries and conditional shifts for SAE variants. Synthetic and dictionary-structure checks test parts of this account.

A Proof of Theorem 1

  • For k ≥ 2, parent absorption costs α^2k + λ versus λk for child coding, so it is preferred iff λ ≥ λc(k, α) = α^2k/(k − 1).The generator uses a parent direction and orthogonal child residuals; the macro 50%-onset substitutes the modal active count k⋆ into this per-k boundary.Source: A Proof of Theorem 1 · p. 9

B Cost comparison for Proposition 2 (mixed strategy)

  • For 1 < m < k, the mixed strategy beats parent absorption when λ < α^2 but beats child coding when λ > α^2{1 + 1/(k − m)}, so it dominates neither across any λ interval.The comparison distinguishes dictionary composition from per-sample firing; a trained dictionary can contain parent and residual atoms even when neither pure strategy dominates.Source: B Cost comparison for Proposition 2… · p. 10

C Conditional argument for Proposition 3 (Matryoshka shift)

  • Conditional on the smallest Matryoshka level being parent-aligned, each residual fires when λ < α^2, shifting the threshold below the vanilla bound by α^2/(k⋆ − 1).The argument requires an additional centering, symmetry, or optimization assumption to make the smallest-level atom parent-aligned.Source: C Conditional argument for Proposition… · p. 10

D Heuristic for Proposition 4 (Archetypal shift)

  • Under the convex-hull contamination assumption, the Archetypal threshold is α^2k(1 − k/K)/(k − 1), with a downward shift that vanishes as K grows.The heuristic assumes reachable child approximations mix other children’s residuals; as k approaches K, the threshold tends to zero.Source: D Heuristic for Proposition 4… · p. 10

E Synthetic verification: trained-SAE plot and variant signatures

G Structural-recovery detail

  • Middle layers recover the family-mean parent direction, while deep layer L11 recovers the PC1 parent direction; both structural claims survive bootstrap intervals and fall outside randomized-sibling correlation nulls.Source: G Structural-recovery detail · p. 11

I C.4: hedging decomposition

The hedging decomposition checks whether feature hedging explains the reported structural signal. The remaining sections report estimator robustness, document the preregistration, and distinguish three notions of absorption.

I C.4: hedging decomposition

  • After partialling out hedge, the Spearman correlation between λc and abs_mag remains +0.43, unchanged from baseline; the same partial after encoder–decoder cosine is also +0.43.Across 50 accepted families, encoder–decoder cosine has mean and median +0.94, and median hedge score is 0.014.Source: I C.4: hedging decomposition · p. 11
  • After partialling out K, the Spearman correlation between λc and abs_mag is +0.14, with K alone explaining approximately two-thirds of the signal.Source: I C.4: hedging decomposition · p. 11

J C.3: ˆv estimator robustness

  • Across three ˆv estimators on the same 50 families, ρ remains positive, with a spread of 0.37.Source: Figure 9 · p. 11

K Pre-registration document (verbatim) and amendments

L Three absorption notions

  • Theorem 1 targets structural absorption, while Chanin et al. [2024]’s metric and Bussmann et al. [2025]’s reported reduced absorption target gerrymandered firing; these measures concern different notions.The paper distinguishes structural dictionary alignment, per-sample firing, and whether a child atom skips subsets of concept members.Source: L Three absorption notions · p. 12

M Pre-Registered Instance-Level Proxy and Negative Result

The pre-registered instance-level consensus proxy failed in the predicted direction on the locked family set, and robustness checks point to family-size chance effects as the dominant confound. The paper characterizes this as metric-mediated non-transfer, while transfer under a K-baseline-corrected metric remains open.

M Pre-Registered Instance-Level Proxy and Negative Result

  • The locked study retained 50 of 80 selected WordNet families after mechanical filters, using a Pythia-160m TopK SAE hooked after layer 6.The verdicts use locked criteria; the external replication was pre-committed to publication regardless of outcome, with no undeclared reruns to seek a passing correlation.Source: M Pre-Registered Instance-Level Proxy… · pp. 12–13
  • The endpoint scores top-1 consensus firing across test-template singletons, while using family size K in place of the theorem’s active-child count is a proxy modeling choice.Absorption magnitude is the fraction of singletons whose top-1 latent matches the family’s modal top-1 latent.Source: M Pre-Registered Instance-Level Proxy… · p. 12
  • Among 50 locked families, the proxy yielded Spearman ρ = +0.41, opposite the pre-registered direction (permutation p = 0.996).Only 0.4% of 5,000 label permutations reached a correlation this negative.Source: M Pre-Registered Instance-Level Proxy… · p. 12
  • No robustness variant reached the ρ ≤−0.6 acceptance threshold: estimators were +0.41, +0.10, +0.46; layers were +0.34, +0.41, +0.42, −0.35; MLP replications were +0.12, +0.49.The locked verdict was checked across three estimators, four residual layers, and the pre-committed Pythia-160m/410m MLP replication.Source: M Pre-Registered Instance-Level Proxy… · p. 12
  • Subtracting the 1/K chance baseline from abs_mag collapses correlations near zero; K dominates and pulls λc into the wrong-sign association, while α2 shows no detectable signal.The proxy substitutes family size K for the theorem’s active-child count.Source: M Pre-Registered Instance-Level Proxy… · p. 13
  • The diagnostic sweep points toward genuine non-transfer because width-invariance is also violated, while depth-dependence and the 410m α2 baseline suggest some theoretical structure survives at larger scale.The authors describe the finding as metric-mediated non-transfer and leave transfer under a K-baseline-corrected metric open.Source: M Pre-Registered Instance-Level Proxy… · p. 13
  • In both matched MLP replication arms, partialling out K changes ρ by at most 0.05 and eliminates the spurious signal, supporting K-baseline contamination across model size and hook.The arms used Pythia-160m layers.6.mlp and Pythia-410m layers.12.mlp.Source: M Pre-Registered Instance-Level Proxy… · p. 13

Related papers

Loading 2609.29551v1…