PaperlayerView exact revision on arXiv

Source-linked AI summary

When Is an SAE Feature Interpretable? A Validation Ladder for EEG Foundation Models

Yucong Cao, Chenqi Li, Tingting Zhu

arXiv:2609.34091v1cs.LG

TL;DR

The paper asks when a perturbation-responsive SAE latent deserves a physiological label, given that discrete latents invite human-readable names without ensuring semantic identifiability. It proposes a validation ladder and tests it across EEG settings; the apparent alpha signature weakens under distortion control and selected latents tend to be negatively associated with clean-EEG alpha. The authors conclude that perturbation sensitivity alone does not establish what a latent represents.

  • Problem

    SAEs make individual units easy to name, but discreteness does not establish that a latent represents the physiological property suggested by its perturbation response.

    Source: §1 · p. 1

  • Method

    The paper proposes a validation ladder testing response, distortion-adjusted response, specificity, independent observation, and—separately—task-readout relevance.

    Source: §1 · p. 2

  • Results

    7.3× raw alpha-over-sham firing change falls to 0.28× after removed-spectral-energy normalization and exceeds one in 0/27 settings; selected latents tend to track lower relative alpha power on clean EEG.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Perturbation sensitivity alone does not justify an SAE feature’s semantic name; the evidence should match the strength of the representational or task-relevance claim.

    Source: §1 · p. 2, §5 · p. 5

    The conclusions concern semantic claims about fitted SAE latents, not clinical validity or neural causality; band-stop filtering cannot isolate a biological oscillator, and the normalizations bound rather than remove that confound.

    Source: §4 · p. 5

Abstract

from arXiv · show

Sparse autoencoders (SAEs) decompose dense model activations into discrete latents, making individual features easy to interpret--and easy to misinterpret. In EEG foundation models, this creates a tempting inference: if removing alpha-band activity strongly changes a latent's activation, one might conclude that the latent represents alpha activity. Across 27 settings spanning three backbones, three EEG datasets, and three network depths, this interpretation initially appears compelling: alpha removal changes latent firing 7.3 times more than an equal-width sham notch (95% CI [6.2, 8.7], bootstrapped over settings). However, the alpha filter also deletes far more signal than the sham. After normalizing by removed spectral energy, the ratio falls to 0.28 (95% CI [0.22, 0.36]) and exceeds one in none of the 27 settings. Latents selected for their response to alpha removal are, on clean EEG, slightly anti-correlated with relative alpha power (mean r = -0.073), giving no support for a simple alpha-detector reading. Motivated by this failure case, we propose a validation ladder for semantic interpretations of SAE latents: it asks in turn whether a latent responds, whether that response survives controlling for how much signal the intervention removes, whether it is specific rather than broadly fragile, and whether the proposed property is visible on unperturbed data--while separately testing the stronger claim that the latent matters to a task classifier. Perturbation sensitivity alone does not establish what an SAE latent represents.

1 Introduction

The paper asks when a perturbation-responsive SAE latent merits a physiological label, and presents a validation ladder to distinguish response from representation and task relevance. Across 27 settings, the raw alpha response weakens after distortion control, while selected latents tend to track lower—not higher—relative alpha power on clean EEG.

1 Introduction

  • 7.3× raw alpha-over-sham firing change falls to 0.28× after normalization by removed spectral energy, exceeding one in 0/27 settings.The settings span three backbones, datasets, and network depths.Source: §1 · p. 1
  • On clean EEG, latents selected because alpha removal moves them tend to fire when relative alpha power is lower, not higher.Selection uses perturbation responses, whereas evaluation uses clean signals.Source: §1 · p. 2
  • The validation ladder tests response sensitivity, distortion-adjusted response, specificity, and independent observation; readout intervention is reserved for claims of task-head relevance.Passing the representational path supports a physiological-property claim, not necessarily functional importance to a task.Source: §1 · p. 2
  • The contribution is an operational hierarchy for excluding alternative explanations before assigning physiological semantics, rather than a new individual diagnostic.The paper distinguishes this question from whether features are sparse, reconstructive, monosemantic, or causally editable, and from which input structures influence predictions.Source: §1 · p. 2
  • The case study combines a forked validation protocol with evidence that specialist-like latents neither track clean-EEG alpha in the intended direction nor are preferentially important to a linear readout.DINOv3 is included to test whether the validation problem also appears with a foundation model pretrained in another modality.Source: §1 · p. 2

2 Experimental design

The experiments compare frozen EEG-pretrained and cross-modal backbones across public EEG tasks and network depths. They train SAEs, perturb clean test trials, control responses for input distortion, and assess selectivity, clean-signal correspondence, and readout relevance.

2 Experimental design

  • The study spans 27 settings formed by three frozen backbones, three public EEG classification sets, and early, middle, or late network depth.The backbones are CBraMod, REVE, and DINOv3; the datasets cover motor imagery, mental arithmetic, and imagined speech.Source: §2 · p. 2
  • SAEs are trained on layer-normalized token activations from early, middle, or late blocks, with k = 32 and reconstruction MSE plus AuxK dead-latent recovery.Dictionary expansion is 16× except for DINOv3, where token support motivates 4× or 8×; the dictionaries reach test FVU [0.059, 0.228] with L0 = 32.Source: §2 · p. 2
  • The validation ladder separates the representational path—stages 1–3 plus independent observation—from stage 4, which tests task-head relevance.The figure states that neither claim is supported in this case study.Source: Figure 1 · p. 3
  • Each clean test trial is paired with spatial or spectral perturbations, including an 8–12 Hz alpha notch and width-matched 30–34 Hz sham, while SAEs remain frozen.Responses are assessed with sparse-code cosine, active-set Jaccard, and absolute firing change; response is normalized by time-domain distortion or removed spectral energy.Source: §2 · p. 3
  • Specialist-like alpha latents are selected by top-quartile alpha selectivity and no-greater-than-median channel-dropout fragility, then compared with matched controls in readout tests.Clean-alpha correspondence uses activity-matched non-specialists; group ablations measure changes in logistic-probe balanced accuracy, with frozen and retrained probe variants.Source: §2 · p. 3
  • Confidence intervals are percentile 95% intervals from 10,000 bootstrap resamples of settings, using 27 settings or 21 for task analyses.The resampling unit is settings, not individual latents.Source: §2 · p. 3

3 Results

Across the three results subsections, the large raw alpha response weakens under distortion control, alpha selectivity does not indicate linear-readout importance, and clean EEG does not support the intended alpha-detector interpretation.

3.1 A large raw alpha response collapses under distortion control

  • 7.3× raw alpha/sham firing response falls to 0.28× after spectral-energy normalization and exceeds one in 0/27 settings.The normalized ratio is not distortion-independent under this design.Source: §3.1 · pp. 3–4, Table 2 · p. 4
  • Channel-dropout robustness decreases with depth across all nine backbone–dataset combinations, most strongly for DINOv3.The depth trend applies to both random and contiguous dropout.Source: §3.1 · p. 4

3.2 Alpha selectivity is not readout importance

  • Mean Pearson r between alpha selectivity and activation-weight importance is 0.107 (CI [0.068, 0.146]) across 21 task-analysis settings.The coupling remains weak in all nine BCI-middle retrainings (|r| ≤0.32).Source: §3.2 · p. 4
  • Zeroing specialist-like groups changes frozen-probe balanced accuracy by +0.0055 on average (CI [−0.007, 0.027]), versus −0.028 for fragile groups and −0.019 for random controls.The ablation ordering is seed-dependent on the BCI middle setting, and fragile latents fire 8.9× more; the authors read the result directionally as no preferential importance to this linear readout.Source: §3.2 · p. 4

3.3 Independent observational validation fails in the intended direction

  • Specialists’ signed correlation with relative alpha power is −0.073 on average and negative in 23/27 settings.Their mean absolute correlation is 0.143, below fragile latents (0.246) and slightly above random controls (0.097).Source: §3.3 · p. 4
  • Activity matching leaves specialists with a small |r| edge (0.143 vs. 0.101) but does not reverse the negative sign or validate a simple alpha detector.The tests do not rule out structured spatial–spectral features.Source: §3.3 · p. 4

4 Discussion and limitations

The validation ladder excludes specific alternative explanations but is not a universal feature finder; the paper also identifies boundaries on its evidence and scope.

4 Discussion and limitations

  • The ladder is an operational protocol for excluding specific alternatives, not a universal feature finder, and the 7.3× alpha signature does not establish an alpha-representing claim.A positive control on another physiological axis is left for future work.Source: §4 · p. 4
  • Figure 2A compares per-setting alpha selectivity with activation-weight importance, while 2B compares clean-activation alpha-power correlations for specialists and random controls.The signed correlation is negative on average, although specialists’ mean absolute correlation is slightly above controls.Source: Figure 2 · p. 5
  • Task-readout products cover 21/27 settings because REVE MA/Speech exports were unavailable.The readout relevance measure is for a logistic probe on mean-pooled codes.Source: §4 · p. 5

5 Conclusion

The conclusion is that a strong alpha-removal response alone does not justify assigning an alpha semantic label to an SAE feature.

5 Conclusion

  • A 7.3× alpha-over-sham SAE response does not survive distortion control, track clean-EEG alpha in the intended direction, or identify units a linear readout depends on.The paper argues that semantic names should be justified by the evidence stage, not perturbation response alone.Source: §5 · p. 5
  • The study reanalyzes public EEG datasets, recruits no participants, and makes no clinical predictions.Consent and ethics procedures are inherited from the original releases.Source: §5 · p. 5

A Additional experimental details

The additional details specify layer grouping, clean-signal validation, and activity-matched controls for comparing specialist latents.

  • Layer groups combine consecutive transformer blocks after per-layer ℓ2 normalization, with backbone-specific block ranges.CBraMod and DINOv3 use blocks 0–3, 4–7, and 8–11; REVE uses 0–6, 7–14, and 15–21.Source: A Additional experimental details · p. 6
  • Clean-signal validation examines top-five test activations for 298 selected latents, while all-trial Pearson correlation remains the primary test.Welch spectra quantify relative target-band power on the top-activation tails; the all-trial test avoids conditioning on activation extremes.Source: A Additional experimental details · p. 6
  • Activity-matched controls are unused non-specialists from the same dictionary, selected by clean firing-rate and mean-activation z-distance.Mean matched and specialist firing rates are both 0.374; mean activations are 0.0060 and 0.0062, respectively.Source: A Additional experimental details · p. 6

B Per-backbone raw alpha/sham contrast

Table 3 reports raw alpha-versus-sham outcomes averaged by backbone across dataset–depth settings.

  • Table 3 reports mean active-set Jaccard and mean |∆f| under alpha and sham interventions, averaged over nine settings per backbone.The raw rows are omitted from the main table because normalized 27-setting ratios carry the inferential claim.Source: Table 3 · p. 7
  • The comparison pairs an 8–12 Hz alpha notch with a 30–34 Hz sham notch.Source: Table 3 · p. 7
  • The table’s raw outcomes are summarized by backbone rather than presented as the main inferential result.Source: Table 3 · p. 7

C Depth-dependent channel-dropout robustness

Channel dropout destabilizes sparse codes with depth across all nine model–dataset combinations, with the largest effect in DINOv3.

  • Late-layer active-set overlap is lower than early-layer overlap for random and contiguous dropout, with mean late-minus-early Jaccard of −0.17 and −0.20, respectively.Source: C Depth-dependent channel-dropout… · p. 7
  • The contiguous-dropout depth effect is strongest for DINOv3 (∆Jaccard ≈−0.35), intermediate for CBraMod (≈−0.19), and mild for REVE (≈−0.07).Source: C Depth-dependent channel-dropout… · p. 7
  • Across all nine model–dataset combinations, deeper SAE representations are less stable to spatial perturbation, including in the cross-modal vision backbone.The passage distinguishes this architectural observation from the paper’s primary validation-ladder claim.Source: C Depth-dependent channel-dropout… · p. 7

D Full 21-setting ablation chart

The full ablation chart presents frozen-probe balanced-accuracy changes across 21 settings; setting-level bootstrap results quantify the alpha/sham and readout comparisons.

D Full 21-setting ablation chart

  • Figure 3 displays frozen-probe balanced-accuracy change after zeroing size-matched latent groups across 21 settings.The main-text Figure 2 replaces it with a two-panel summary intended to be readable at workshop scale.Source: Figure 3 · p. 7, D Full 21-setting ablation chart · p. 7

E Setting-level bootstrap and activity matching

F Licenses, assets, and compute

The paper lists usage terms for its three EEG datasets and three backbones, and reports the compute resources and estimated cost of the experiments.

F Licenses, assets, and compute

  • The three EEG datasets have research-use or competition-use terms, and Mental Arithmetic is CC BY 4.0 [Tangermann et al., 2012, Brunner et al., 2008; Zyma et al., 2019, Goldberger et al., 2000; Jeong et al., 2022, Lee et al., 2020].BCI Competition IV 2a is research use; BCI Competition 2020 Track 3 is competition/research use.Source: F Licenses, assets, and compute · p. 8
  • CBraMod and REVE weights or positions are used as released by their authors, while DINOv3 ViT-B/16 is covered by the DINOv3 License (Meta) [Wang et al., 2025; El Ouahidi et al., 2025; Siméoni et al., 2025].The CBraMod reference implementation is BSD-3-Clause.Source: F Licenses, assets, and compute · p. 8
  • The reported pipeline used NVIDIA A100 GPUs on a shared academic Slurm cluster and is estimated at about 300 GPU-hours, excluding exploratory runs not used in the paper.The anonymized supplementary code archive includes training and analysis scripts; checkpoints and raw EEG are omitted because of size and are available from cited public sources.Source: F Licenses, assets, and compute · p. 8

NeurIPS Paper Checklist

The checklist reports that the paper’s claims match its scope and that key experimental and reproducibility details are provided. It also records the study’s ethics, impact, asset, and human-subject disclosures.

NeurIPS Paper Checklist

  • The checklist says the abstract and introduction state the empirical scope and that Sections 3–4 distinguish observations from physiological or causal claims.The authors answer yes that the claims accurately reflect the paper’s contributions and scope.Source: NeurIPS Paper Checklist · p. 8
  • The paper reports that Section 2 and supplementary code provide the models, data splits, preprocessing, SAE construction, perturbations, metrics, probe protocol, and exact configurations needed for reproducibility.Datasets and backbone implementations are public, while an anonymized archive of training and analysis scripts is supplied; checkpoints and raw EEG are omitted because of size.Source: NeurIPS Paper Checklist · pp. 9–10
  • The checklist says the paper specifies its experimental splits, preprocessing, SAE settings, perturbation bands, metrics, classifier protocol, and supplementary training details.The authors answer yes that training and test details are specified.Source: NeurIPS Paper Checklist · p. 11
  • Main-text means are reported with percentile 95% confidence intervals from 10,000 bootstrap resamples of settings, using 27 settings or 21 for task analyses.The resampling unit is the setting rather than the latent; initialization variability is also quantified for the BCI middle-layer setting under three SAE seeds.Source: NeurIPS Paper Checklist · p. 11
  • The reported experiments used NVIDIA A100 GPUs on a shared academic Slurm cluster, with an estimate of about 300 GPU-hours excluding unused exploratory runs.The checklist answers yes that compute resources are sufficiently disclosed.Source: NeurIPS Paper Checklist · pp. 11–12
  • The checklist reports that the study reanalyzes public datasets, makes no clinical or individual-level claims, discusses societal impacts, and did not recruit participants or collect new human-subject data.It also records that no new generative model, personal data, or high-risk dataset is released, and that existing assets are credited with terms of use listed.Source: NeurIPS Paper Checklist · pp. 12–14

Related papers

Loading 2609.34091v1…