PaperlayerView exact revision on arXiv

Source-linked AI summary

ICA Lens: Interpreting Language Models Without Training Another Dictionary

Sida Liu, Feijiang Han

arXiv:2606.11722v1cs.LGcs.AIcs.CL

TL;DR

SAEs provide powerful interpretability dictionaries but can be costly to train and evaluate, motivating a lighter first lens. ICALens makes ICA practical for LLM activations and finds competitive probing and stronger targeted perturbation performance than public SAEs at small-to-medium budgets.

  • Problem

    SAEs offer fine-grained feature dictionaries but require substantial data, storage, tuning, and compute, motivating lighter first-step interpretability tools.

  • Method

    ICALens applies FastICA to normalized, whitened LLM activations and provides an efficient, auditable workflow for inspecting recovered components.

  • Results

    Across models, ICA is competitive with SAEs in sparse probing and achieves stronger targeted probe perturbation performance under small-to-medium intervention budgets.

  • Takeaways & Limitations

    ICA is a compact, complementary first lens that can help determine where more costly overcomplete dictionary learning is warranted.

  • Takeaways & Limitations

    Standard FastICA is compact and cannot provide the large overcomplete vocabulary that modern SAEs are designed to learn.

Abstract

from arXiv · show

Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior. Sparse autoencoders (SAEs) have become the standard tool for this purpose, but using them as the default first lens often requires training, storing, and evaluating large overcomplete dictionaries. This bottleneck limits rapid exploration and raises a fundamental question: how much interpretable structure is already visible from activation geometry before training another neural dictionary? Our intuition is simple: many interpretable directions are selective on tokens, and these directions should look less Gaussian than random directions. We therefore revisit independent component analysis (ICA), a classical method for finding non-Gaussian directions, as a compact lens for language-model interpretability. We find that ICA has been underestimated for LLM interpretability, because prior uses often relied on off-the-shelf ICA implementations that are brittle on LLM activations and lacked systematic tools for inspecting and evaluating the recovered directions. To bridge these gaps, we introduce ICALens, the first practical workflow for stable, efficient, and auditable ICA analysis of LLM representations. It combines an optimized GPU-parallel FastICA pipeline with LLM-specific stability recipes and better fitting diagnostics, enabling efficient and reliable layer-wise analysis. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICALens efficiently recovers compact, human-interpretable directions without per-layer gradient-based dictionary training. On SAEBench, ICA is competitive with public SAEs in sparse probing and outperforms them in targeted probe perturbation under small-to-medium budgets. These results suggest that ICA should not be viewed as a weak baseline, but as an efficient and complementary first lens for exploring language-model representations.

1. Introduction

ICALens revisits ICA as a lightweight first lens for interpreting LLM representations, motivated by the selective, non-Gaussian structure already visible in activation geometry. It provides a stable, efficient, auditable workflow and shows that ICA recovers interpretable directions competitive with SAEs without per-layer dictionary training.

  • Motivation: SAEs are powerful but early analyses may instead benefit from a compact set of interpretable directions that identifies layers, exposes concepts, and guides lightweight interventions.This motivates a first lens that avoids training another large overcomplete dictionary.
  • Motivation: Selective interpretable directions should differ statistically from random projections, motivating non-Gaussianity as a measurable footprint in activation space.Such directions may activate on limited contexts, prefer one sign, or respond to lexical, syntactic, semantic, or discourse patterns.
  • Motivation: Public SAE decoder directions are substantially more non-Gaussian than random directions, suggesting that directly maximizing non-Gaussianity can recover structure learned through sparse reconstruction.ICA uses this signal after centering and whitening to find a small set of statistically exceptional directions.
  • Contribution: ICALens introduces a practical workflow for stable, efficient, and auditable ICA analysis of LLM representations.The paper addresses brittle off-the-shelf ICA by targeting high-dimensional activations, token outliers, slow components, convergence failures, and limited prior interpretability evaluation.
  • Results: Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICA recovers interpretable directions, remains competitive with SAEs in sparse probing, and outperforms public SAEs in targeted probe perturbation.The introduction also reports that ICA components are more non-Gaussian than random directions and public SAE decoder directions, while recovering related yet non-redundant structure.

2. Related Work

Related work has established sparse autoencoders as the dominant approach for extracting interpretable feature coordinates from language-model activations. ICALens revisits ICA as a lightweight, compact complement that seeks non-Gaussian independent directions and offers a more complete workflow for LLM interpretability.

  • Sparse autoencoders: Sparse dictionary learning has produced more monosemantic, interpretable, and causally useful latents than neurons or classical decompositions.This success made sparse autoencoders the dominant route for converting language-model activations into inspectable feature coordinates.
  • Independent Component Analysis: ICA searches for statistically independent, non-Gaussian directions in a linear representation space as a classical lightweight alternative.Prior work recovered interpretable ICA components in images, fMRI, word embeddings, and multimodal embedding spaces.
  • This work: Our work develops a more stable ICA fitting pipeline, an annotation and analysis platform, and a broader evaluation suite for modern LLM interpretability.The paper positions ICA as a compact complement to SAEs: lower-capacity than an overcomplete dictionary, but cheaper to fit, easier to browse, and competitive with public SAEs while outperforming evaluated lightweight baselines.

3. ICALens: Making ICA Practical for LLM Activations

ICALens makes ICA practical for LLM activations by combining GPU-parallel FastICA with normalization, robust convergence acceptance, and adaptive refitting. These safeguards address brittle convergence while preserving stable components and exposing fitting diagnostics.

  • Core ICA procedure: FastICA centers and whitens layer activations, then searches for a rotation whose coordinates are maximally non-Gaussian.ICALens uses the parallel FastICA variant implemented in PyTorch for efficient GPU fitting.
  • Practical challenges: Naive FastICA is brittle on LLM activations because high dimensionality, large-norm token positions, and slow or oscillating components can dominate convergence.A single unstable tail can cause an entire layer to be rejected even when most directions have stabilized.
  • Stability recipes: The p95-LIM fallback accepts fits when most components stabilize, while adaptive refitting lowers the component count only when necessary.With 1M fitting rows and normalization, the p95 fallback increases accepted layers from 8 to 10 and reduces total iterations from 3,107 to 2,741.
  • Diagnostics: ICALens records component-wise convergence statistics and accepted component counts, making fitting difficulty auditable across layers.Full-dimensional fits accept m = d, whereas fits with m < d still provide valid non-Gaussian directions.

4. What Does ICA Recover? Non-Gaussianity and Context Dependence

ICA directions are substantially more non-Gaussian than random projections and SAE decoder directions, supporting non-Gaussianity as an interpretable statistical signature. ERF further shows that ICA recovers a local-to-contextual spectrum whose composition shifts gradually across layers.

  • 4.1 Non-Gaussianity: ICA directions are substantially more non-Gaussian than random directions and SAE decoder directions under a common row-normalized activation geometry.Excess kurtosis near zero indicates Gaussian projections, while larger values indicate heavier-tailed or otherwise exceptional distributions.
  • 4.1 Non-Gaussianity: Non-Gaussianity is a shared signature of learned feature directions, while ICA makes this bias explicit by directly searching for exceptional projection statistics.SAEs exhibit elevated non-Gaussianity despite not being trained to maximize kurtosis.
  • 4.2 Effective Receptive Field: ERF measures the shortest left-context suffix that recovers a component’s signed response while keeping it among the top 15 absolute-score components.The tested context window has Kmax = 11, with unrecovered examples assigned ERF 11.
  • 4.2 Effective Receptive Field: ICA components span a local-to-contextual spectrum: many are recoverable from the target token or a short suffix, while a substantial fraction require longer context.This shows ICA captures contextual directions in addition to lexical detectors.
  • 4.2 Effective Receptive Field: Layer specialization changes through gradual shifts in component mixtures rather than a sharp handoff: early layers emphasize local components, while later layers contain more medium- and long-context components.Both early and later layers retain components outside their dominant context range.
  • 4.2 Effective Receptive Field: −0.41 to −0.50: across all three models, Spearman correlations between excess kurtosis and ERF are negative, so higher-kurtosis components tend to be more local.Broad-context components generally have lower kurtosis and larger ERFs.

5. Human Inspection of ICA Components

Human inspection shows that ICA components are broadly labelable, auditable, and interpretable across lexical, contextual, and construction-level phenomena. Token-centered analyses further show that components combine dynamically according to context rather than encoding fixed one-to-one concepts.

  • Inspection protocol: Across 150 sampled components, 142 received non-unclear labels and 127 received high-confidence labels spanning lexical, syntactic, semantic, positional, and conditional patterns.The explorer supports passive inspection and targeted prompts to test whether labels remain consistent across contexts.
  • Inspection protocol: Among 127 audited labels, 121 were supported, 6 were partially supported, none were rejected, and 112 scored at least 8.The secondary expert audit functions as a consistency check on the initial annotations.
  • Contextual decomposition: The four occurrences of bank share identical embedding representations but separate into context-dependent mixtures of lexical, phrase-level, semantic, positional, and longer-context components in later layers.Financial and river-related readings can begin from left context alone, while some occurrences retain finance-related components because autoregressive conditioning exposes only the preceding text.
  • Contextual decomposition: C67 responds across financial context near bank, deposit, check, withdraw, and cash, while C273 tracks arrival- or purpose-related constructions across multiple sentences.These traces indicate broader contextual or construction-level directions rather than detectors for single words.
  • Token embeddings: Embedding-layer ICA recovers shared and distinguishing factors in related token pairs, including royalty for king/queen, parenthood and gender for father/mother, and occupation plus gender for actor/actress.The embedding control examines fixed lexical representations before contextual information enters the model.

6. Feature Utility under Sparse Probing and Intervention

ICA provides a compact, competitive feature basis for sparse probing and is particularly advantageous for targeted interventions at small-to-medium budgets. Its advantage diminishes at larger intervention budgets, where overcomplete public SAEs become more competitive.

  • Sparse probing: All methods use identical sparse-probing datasets, feature budgets, and train/test splits, with results averaged across representative layers for each model.Matryoshka SAE comparisons are limited to Gemma 2 2B layer 12 because released checkpoints are available only there.
  • Sparse probing: ICA is competitive with public SAEs for sparse probing and consistently outperforms PCA, ITDA, and compact Matryoshka SAE variants.These results indicate that non-Gaussianity is more useful than variance for sparse concept prediction, and reducing SAE dictionary size alone does not close the gap.
  • Targeted probe perturbation: Targeted Probe Perturbation evaluates whether ablating top-ranked features reduces the target-class probe score more than non-target probe scores.For ICA, signed component scores are handled through the same two-sign wrapper used in sparse probing before inverse reconstruction.
  • Targeted probe perturbation: ICA is strongest relative to public SAEs at small-to-medium intervention budgets, combining rapidly increasing intended effects with comparatively small unintended effects.This makes a compact ICA basis advantageous when inspecting, editing, or intervening on a small set of task-relevant directions.
  • Targeted probe perturbation: At larger intervention budgets, public SAEs become more competitive and can become stronger because their overcomplete dictionaries provide more features for accumulating class-relevant latents.The same number of ablations consumes a larger fraction of ICA’s compact basis, making broad disruption harder to avoid.

7. How ICA Relates to SAEs

ICA and SAEs recover partially overlapping but non-identical directions: many ICA components have SAE neighbors, while others are weakly matched. Their token-level behavior also differs, with SAE features tending to be localized and ICA directions often varying smoothly across related tokens.

  • Methodological relationship: SAEs optimize reconstruction and sparsity over large overcomplete dictionaries, whereas ICA extracts compact non-Gaussian directions after normalization, centering, and whitening.The methods therefore expose activation-space directions under different objectives.
  • Direction overlap: Nearest-neighbor comparisons show partial ICA-SAE alignment: many ICA components have nontrivial SAE neighbors, but many are only weakly matched by any single SAE direction.This pattern appears across all three models and remains stable across depth, rather than reflecting a one-to-one correspondence or a few exceptional layers.
  • Functional agreement: Functional agreement is strongest for mid- and high-overlap pairs, whose nearest SAE features often activate on the same target-token activation, whereas low-overlap neighbors are usually inactive.The comparison uses each ICA component’s top-activation context and measures the nearest SAE feature’s activation and rank there.
  • Interpretability audits: ICA-SAE overlap can cross-check feature interpretations, revealing that some public Neuronpedia labels are incomplete or overly narrow.The paper treats public labels as useful hypotheses rather than fixed ground truth because labels may be updated and may not fully describe feature behavior.
  • Token-level behavior: Token-wise plots show SAE features often peaking at one token or a short span, while ICA directions vary more smoothly across neighboring tokens and can track related financial words.In the GPT-2 Small example, ICA follows the interpretation of bank across deposit, check, withdraw, and cash.

8. Conclusion

ICALens provides a stable, efficient, and auditable workflow for applying ICA to LLM representations, recovering interpretable directions without training another overcomplete dictionary. The results position ICA as a compact first lens that complements rather than replaces SAEs by revealing when activation geometry already contains useful structure.

  • Workflow: ICALens normalizes activations, fits a compact GPU-parallel FastICA basis, and supports inspection, annotation, and evaluation of signed non-Gaussian directions.The workflow is designed for stable, efficient, and auditable analysis of LLM representations.
  • Empirical scope: Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICALens recovers human-interpretable components and useful feature coordinates.It also links non-Gaussianity to context dependence through ERF and supports sparse probing and targeted probe perturbation.
  • Positioning: ICA is a compact first lens for language-model activations, not a replacement for sparse autoencoders.SAEs remain suited to high-resolution feature discovery through large overcomplete dictionaries optimized for sparse reconstruction.
  • Positioning: ICALens helps analysts assess how much interpretable structure is visible from activation geometry before deciding whether heavier dictionary learning is worthwhile.This frames ICA as a complementary tool for prioritizing where SAE training costs may be justified.

9. Limitations and Future Directions · A. Additional FastICA Fitting Diagnostics

The paper identifies compactness and state-focused analysis as key boundaries for ICA, while proposing transformation-level, actionable, and task-specific extensions. Additional diagnostics visualize FastICA convergence across fitting-set sizes and model families using component-wise limits and a 10−4 threshold.

  • 9. Limitations and Future Directions: Standard FastICA returns at most d components in a d-dimensional fitting space, preventing ICA from matching the overcomplete vocabularies targeted by modern SAEs.For difficult layers accepted with fewer than d components, reading remains useful for inspection and probing, while intervention edits use pseudoinverse reconstruction.
  • 9. Limitations and Future Directions: Future work could increase ICA capacity beyond compact bases to explore higher-capacity representations.
  • 9. Limitations and Future Directions: Current experiments analyze residual-stream states, whereas future studies could fit ICA to MLP outputs, attention outputs, residual updates, or shared bases across layers.These transformation-level analyses would provide a lightweight counterpart to model-wide SAE releases and component-level tools for studying model computation.
  • 9. Limitations and Future Directions: Steering and automatic annotation are proposed as promising downstream applications of ICA directions.The paper’s goal is to help analysts quickly find, inspect, and test compact directions in model activations.
  • 9. Limitations and Future Directions: Because ICA is cheap to refit, it may help discover task- or distribution-specific directions across datasets for actionable mechanistic interpretability.
  • 9. Limitations and Future Directions: The authors will maintain an updated website list of follow-up projects and open questions.
  • A. Additional FastICA Fitting Diagnostics: Layer-wise FastICA convergence diagnostics are shown for GPT-2 Small using 1k, 100k, and 1M fitting rows.
  • A. Additional FastICA Fitting Diagnostics: Representative convergence curves across model families plot component-wise FastICA limit values over iterations, with percentile bands, maxima, and a 10−4 convergence threshold.Solid colored lines show medians, shaded regions show the 5th–95th percentile interval, dashed black lines show the maximum, and dotted horizontal lines mark the threshold.

B. Additional Human Interpretation Results

The section presents additional human-interpretation analyses of ICA components, covering contextual decomposition of polysemous words and familiar analogy word sets across several language models.

  • Figure 20 shows contextual decomposition of a polysemous word in Gemma 2 2B.
  • Figure 21 shows contextual decomposition of a polysemous word in Qwen 3.5 2B Base.
  • Figure 22 shows embedding-layer ICA components for familiar analogy word sets in GPT-2 Small.
  • Figure 23 shows embedding-layer ICA components for familiar analogy word sets in Gemma 2 2B.

C. Complete Random Annotation Tables

The random ICA component audit reports manual labels, confidence, effective receptive fields, and excess kurtosis for selected components. Most listed components receive high-confidence token, phrase, form, positional, or contextual labels, while a smaller set remains unclear or low-confidence.

  • Audit scope: Table 7 audits selected random ICA components using manual labels, confidence, ERF, type, and excess kurtosis.The reported sign corresponds to each component’s strongest absolute-score example.
  • High-confidence labels: High-confidence components capture diverse patterns, including Have, code-specific =", two leading whitespaces, Array, and non-breaking space U+00A0.Examples include L0/C9 with Have at excess kurtosis 163 and L2/C341 with Two leading whitespaces at 91.
  • High-confidence labels: Other high-confidence components identify semantic, syntactic, positional, and contextual features such as Google/Big Tech Company, comma plus a/the, post-title article start, and Create/Develop.The table also includes labels such as Where, Sort, Control, Advertise, Only, and Word fragments.
  • Broader patterns: Additional components represent phrases and broader patterns, including Register, End, sensitive identity, legal document format, money/fund, and follow-or-violate-rules.Their listed excess kurtosis values range from 14 for Register to 268 for End.
  • Unclear findings: Low-confidence or unclear components include repeated-entry context, memorized code strings, and several unidentified patterns.Examples include L6/C441’s repeated labeled/enumerated entry after intervening context and L23/C1250’s memorized string in a code snippet.

D. Secondary Contrastive Label Audit

The secondary audit found that many randomly selected ICA components received high-confidence, interpretable labels, often refining initial token or construction descriptions. It also identified weaker cases involving broader discourse patterns and ambiguous functional interpretations.

  • Audit setup: The audit scored high-confidence annotations on a 0–10 scale and suggested revised labels for the audited random ICA components.The rows were drawn from random_components_n50_seed0.
  • High-confidence annotations: Many component labels achieved scores of 9 or 10, including array, data, non-breaking-space, auxiliary, lexical-token, and grammatical-construction features.Examples include L2/C890 Data (10), L5/C404 Non-breaking space (10), L7/C351 Do, Did (10), and L21/C399 Adj. prefer after “be” (7).
  • Contextual and structural features: The audit also supported interpretable discourse and structural labels, including accusatory/blame discourse, immigration-policy news discourse, repeated code/list line endings, and parallel/list element expectations.These examples received scores from 7 to 8.

E. Token-Wise Activation Patterns in Gemma and Qwen · F. Explorer Interface Screenshots

The paper compares token-wise ICA and SAE activation patterns on matched Gemma 2 2B and Qwen 3.5 2B Base sentences, while documenting explorer interfaces for both models. In each token analysis, the visualized directions are selected from the top-2 ICA components and top-2 SAE features across the sentence.

  • E. Token-Wise Activation Patterns in Gemma and Qwen: For Gemma 2 2B, Figure 24 compares token-wise ICA and SAE patterns for the same bank sentence.The visualization plots the union of selected directions across all token positions.
  • E. Token-Wise Activation Patterns in Gemma and Qwen: Gemma’s token analysis selects the top-2 ICA components for each token by absolute score.These selected ICA directions are combined across token positions for the plotted pattern.
  • E. Token-Wise Activation Patterns in Gemma and Qwen: Gemma’s comparison also selects the top-2 SAE features for each token by activation.The ICA and SAE selections are plotted together as a union across the sentence.
  • E. Token-Wise Activation Patterns in Gemma and Qwen: For Qwen 3.5 2B Base, Figure 25 applies the same token-wise ICA-and-SAE visualization to a matching bank sentence.The figure plots the union of selected directions across all token positions.
  • E. Token-Wise Activation Patterns in Gemma and Qwen: The Qwen analysis uses the top-2 ICA components by absolute score and top-2 SAE features by activation for each token.This repeats the selection procedure used in the Gemma comparison.
  • F. Explorer Interface Screenshots: For Gemma 2 2B layer 12, Figure 26 shows separate ICA Explorer and SAE Explorer interface screenshots.The screenshots document interfaces for inspecting the two representation analyses.
  • F. Explorer Interface Screenshots: Figure 27 provides explorer interface screenshots for Qwen 3.5 2B Base layer 12.The figure extends the interface documentation to the Qwen model.
Loading 2606.11722v1…