PaperlayerView exact revision on arXiv

Source-linked AI summary

Parts-of-Speech as Emergent Categories in SAE Latent Space

Alessandro Bondielli, Lucia Passaro, Serena Auriemma, Alessandro Lenci

arXiv:2609.29362v1cs.CL

TL;DR

It remains unclear whether linguistic categories in language-model representations correspond to localized units or distributed dimensions, and PoS offers a controlled testbed for this question. The paper analyzes SAE activations to study PoS recoverability and latent organization, finding compact, category-dependent groups rather than one-to-one latent-category mappings.

  • Problem

    It remains unclear whether linguistic categories are localized in individual SAE dimensions or distributed across representations, and PoS offers a controlled testbed for examining this organization.

    Source: §1 · pp. 1–2

  • Method

    The study uses PoS categories as a controlled testbed to analyze how morpho-syntactic information is organized in SAE latent space.

    Source: §6 · p. 9

  • Results

    PoS distinctions are recoverable from sparse activations and supported by compact, category-dependent latent groups; their union preserves strong multi-class classification and the groups remain stable on held-out data.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    PoS is represented in SAE latent space as localizable but distributed features, and cross-category co-activations partly track lexical, positional, and annotation-driven regularities.

    Source: §6 · p. 9

    The study analyzes one base model, LLaMA-3-8B, and one public SAE, so generality across models, layers, SAE variants, and sparsity regimes remains unassessed.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Sparse AutoEncoders (SAEs) offer a promising way to inspect language model representations, but it is still unclear what kind of linguistic structure their latents expose. We use part-of-speech (PoS) categories as a controlled test case to study whether morpho-syntactic information is encoded by individual latents or by structured groups of features. We find that PoS distinctions are highly recoverable from SAE activations, but do not align with one-to-one latent / category mappings. This recoverability is not reducible to lexical memorisation, and Open and Closed PoS classes differ substantially. Categories are supported by compact groups of sparse latents, with substantial variation across tags. These groups remain stable on held-out data, while also showing overlap between related categories. Our results show that SAEs localise morpho-syntactic information in a distributed and category-dependent form rather than through atomic grammatical features.

1 Introduction

The paper asks whether linguistic categories are localized in individual SAE latents or distributed across groups, using PoS as a controlled testbed. It finds recoverable PoS distinctions organized in compact, category-dependent latent groups rather than one-to-one latent-category mappings.

1 Introduction

  • SAE latents may encode linguistic categories as localized units or distributed dimensions, a distinction that matters for assessing what SAE interpretability reveals.PoS tags provide a controlled testbed because they are discrete, independently annotated, and linguistically interpretable, while differing in frequency, lexical openness, and syntactic function.Source: §1 · pp. 1–2
  • The authors analyze token-level SAE activations with probing, feature-salience and coverage analyses, then validate selected latent groups on held-out and controlled data.They also test whether the groups' union supports multi-class PoS classification.Source: Figure 1 · p. 1, §1 · p. 2
  • The study asks whether PoS categories are encoded in SAE activations, how that encoding is organized, and whether it is stable across categories, datasets, and evaluation settings.Source: §1 · p. 2
  • PoS categories align with structured groups of sparse features, whose size and organization vary across categories and differ between Open- and Closed-class tags.Source: §1 · p. 2
  • The selected groups are compact yet effective: their union preserves strong multi-class classification, remains stable on held-out data, and overlaps across related categories.Source: §1 · p. 2

2 Related Work

Prior work uses probes and SAEs to study linguistic information, but probing accuracy alone does not establish how information is organized or used. This paper addresses the limited evidence on theoretically motivated linguistic categories in SAE latent space.

2 Related Work

3 Method and Materials

The study uses the English GUM treebank and a controlled set of lexical items to examine PoS-related SAE activations. It extracts token representations from LLaMA-3-8B and builds a sparse feature matrix for probing.

3.1 Dataset

  • The naturally occurring dataset is the UD English GUM treebank, with 14,353 sentences, 252,284 tokens, and all 17 Universal PoS tags; training is used for discovery and test data for held-out evaluation.Source: §3.1 · p. 3
  • A controlled dataset of 180 lexical items, focused mainly on nouns and verbs, tests whether associated latents activate in minimal, well-formed sentences.Noun and verb forms are varied across templates and tense, with adjacent modifiers and punctuation variants included.Source: §3.1 · p. 3

3.2 Processing Pipeline

  • The pipeline encodes layer-30 residual-stream activations from LLaMA-3-8B with an SAE, then assigns each UD token the activations of its leftmost overlapping subword.The leftmost subword is chosen as the position where token identity first becomes available to the model.Source: §3.2 · pp. 3–4
  • For probing, token activations are projected into a sparse matrix of 130,246 attested latents, with activation strengths as entries and zero for inactive features.Source: §3.2 · p. 4

4 Experiments

The experiments separate PoS recoverability, localization of category-relevant latents, and validation of those groups. They also include controls for lexical identity and comparisons with alternative feature baselines.

4 Experiments

  • The experimental design separates testing whether PoS distinctions are recoverable from examining latent organization and assessing stability across splits and evaluation settings.Source: §4 · p. 4

4.1 Recoverability of POS Information (RQ1)

  • One-vs-rest classifiers test whether each PoS category is linearly recoverable from sparse SAE activations, using gold UD tags as supervision.The setup uses L1-regularized logistic regression with balanced class weights and five-fold cross-validation on the GUM training split.Source: §4.1 · p. 4

4.2 POS Organization in Latent Space (RQ2)

  • Positive logistic-regression coefficients rank category-supporting latents; coverage estimates how many top-ranked features are needed to reach the target threshold.The analysis compares category-specific compactness, using per-class thresholds to avoid bias from class imbalance and differing coefficient profiles.Source: §4.2 · pp. 4–5
  • The union of minimal-coverage latent sets is tested as the input to a multinomial classifier, asking whether selected features suffice for joint PoS prediction.Source: §4.2 · p. 5

4.3 Validation on Held-Out Data (RQ3)

  • Selected groups are evaluated on held-out GUM data and controlled examples, with a cross-PoS activation matrix assessing whether groups are category-specific or shared.Source: §4.3 · p. 5

4.4 Controls and Baselines

  • Controls test whether random labels assigned by word type can reproduce probe performance and compare SAE probes with raw embeddings, raw layer-30 activations, and random latent subsets.Source: §4.4 · p. 5

5 Results

POS distinctions are recoverable from SAE activations, but the information is organized in category-dependent groups of latents rather than one latent per category. These groups preserve multi-class prediction and remain active on held-out data, while overlapping across categories.

5.1 RQ1: Recoverability of POS Information

  • POS distinctions are consistently recoverable from SAE activations, with closed-class and low-variability labels easier to predict than more heterogeneous or less frequent categories.Nouns and verbs are the best-performing open POS; probing alone does not reveal how the information is organized.Source: §5.1 · pp. 5–6, Figure 2 · p. 6

5.2 RQ2: POS-related Latent Groups

  • Positive probe weights identify multiple contributing latents per POS, with open-class classifiers generally using more non-zero latents than closed-class and other classes.Coverage varies by tag: some categories reach 95% with few latents, while others require broader groups, ruling out a one-latent-per-tag mapping.Source: §5.2 · pp. 6–7, Figure 3 · p. 6
  • A compact set of 498 latents, 12% shared between two or more POS, supports multi-class performance comparable to the full SAE representation.The confusion matrix reports predictions for each gold POS using this compact feature set.Source: §5.2 · p. 6, Figure 5 · p. 7

5.3 RQ3: Validation on held-out data

  • On held-out treebank tokens, most selected latent groups retain recall of at least 0.95, while off-diagonal co-activation varies across POS and shows overlap.The results support stability, but qualify category-specificity: groups remain active for their target categories without being fully exclusive to them.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4 Controls and Baselines

  • Random-label controls score 0.54 Accuracy/0.42 Macro F1 versus 0.88/0.97 for the real probe, so POS recoverability is not simply reducible to lexical memorisation.The SAE probe has comparable performance to raw residual-stream probes using approximately eight times fewer features; the authors do not claim SAE probing is superior.Source: §5.4 · p. 8

5.5 Linguistic analysis of SAE activations

  • Open-class categories show more spurious co-activation, including adjective overlap with NOUN, ADV, and PROPN and adverb overlap with several categories.The reported patterns are associated with shared lemmas, morphological relations, syntactic contexts, and surface-form cues.Source: §5.5 · p. 9
  • Related POS latents can activate before the corresponding word appears, but their active counts grow when a word of that category is added to a controlled template.The authors describe these patterns as largely additive across tokens.Source: Figure 7 · p. 8, §5.5 · p. 9
  • Closed classes tend to have compact, selective representations, while open classes involve broader groups that also capture lexical, morphological, and contextual regularities.Compactness tracks lexical variability: invariant word forms can be covered by form-specific latents, so the ordering is not direct evidence of graded abstraction.Source: §5.5 · pp. 8–9

6 Conclusion

POS distinctions are recoverable from SAE activations through compact but category-dependent groups of sparse features, rather than one-to-one latent mappings. These groups can support classification, while their overlaps also reflect other linguistic regularities.

6 Conclusion

  • Each POS category is supported by a compact group of sparse features, with Open-class groups more diffuse and using more active latents than Closed-class groups.A small union of category-specific latents preserves strong multi-class classification performance and remains stable on held-out treebank data.Source: §6 · p. 9
  • POS is represented as sets of localizable but distributed SAE features, not as atomic grammatical features.The authors recommend evaluating latent-level interpretability against theoretically grounded category inventories rather than top-activating examples alone.Source: §6 · p. 9
  • Cross-category co-activations partly track lexical, positional, and annotation-driven regularities, motivating study of richer linguistic levels and typologically diverse languages.The authors identify finer-grained linguistic phenomena as future directions.Source: §6 · p. 9

Limitations

The study’s conclusions are bounded by its focus on POS, one model and SAE, coefficient-based localization, and a small controlled dataset. These choices leave broader linguistic coverage, causal interpretation, and generality across models untested.

Limitations

  • POS tags cover only one linguistic abstraction level; finer morphology, dependency relations, semantic roles, and discourse phenomena remain outside the study.The authors note that latent organization may differ across these phenomena.Source: Limitations · p. 10
  • Using one base model and one public SAE leaves generality across models, layers, training procedures, and sparsity regimes unassessed.The authors say comparisons across multiple models, layers, and SAE variants would be needed to assess generality.Source: Limitations · p. 10
  • Linear probing coefficients identify POS-predictive latents, but do not establish whether the model uses them for morpho-syntactic processing.The authors state that causal interventions would be needed to establish such use.Source: Limitations · p. 10
  • The small controlled dataset centers mainly on nouns and verbs, so it does not cover English’s full syntactic and lexical variability.Broader controlled data could test effects of lexical ambiguity, word order, morphology, and sentence complexity.Source: Limitations · p. 10

A.1 Layer Selection

Layer checks find broadly stable POS representations despite some depth-wise changes, while the appendix documents the extraction setup, computational cost, and released artifacts.

A.1 Layer Selection

  • Activations come from the residual stream after the last layer’s MLP, close to the model output, and are encoded with the corresponding SAE.The study also replicates selected experiments across layers to check stability.Source: §A.1 · p. 12
  • Across layers, pooled mean active latents rise 3105→3697, while L* changes 530→501→498 and compact-feature Macro F1 rises 0.77→0.79→0.81.The authors interpret the pattern as gradual densification and depth-wise redistribution rather than qualitative reorganization; probing performance is generally stable with minor variations.Source: §A.1 · p. 12, Figure 8 · p. 13

A.2 Experimental setup

  • A single 5-fold cross-validation training/test run on the GUM training set takes roughly four hours using all available CPU cores.Representation extraction and SAE encoding take roughly 0.16 GPU hours per layer on the reported setup.Source: §A.2 · p. 12

A.3 Artifacts and Intended Use

  • The study uses a pretrained EleutherAI SAE checkpoint for interpretability research and the GUM treebank’s text and UD annotations for probing and evaluation.The stated uses align with the artifacts’ intended purposes and licenses.Source: §A.3 · pp. 12–13
  • Released artifacts include token-level SAE activations aligned to UD POS tags, a 180-item controlled dataset, and probing and salience code.Activation files use token indices in the original GUM release rather than redistributing source text, respecting per-source licensing.Source: §A.3 · p. 13

B.1 RQ1: Recoverability of POS information

Top classifier coefficients differ by POS class: several Closed-class categories have a dominant latent, whereas most Open-class categories have flatter distributions. PUNCT is especially well classified, and some latents recur across categories.

B.1 RQ1: Recoverability of POS information

  • Closed-class coefficients peak at DET (75751, ≈5.5), CCONJ (34665, ≈4.3), and PRON (6631, ≈2.7), while Open-class coefficients are flatter.Among Open-class POS, only VERB has a clearly dominant feature; ADJ (0.80), PROPN (0.77), and ADV (0.73) have no single salient latent.Source: §B.1 · p. 13, Figure 9 · p. 13
  • PUNCT is the best-classified category overall, driven by latent 117946 (≈5.5).Source: §B.1 · p. 13
  • Latents recur across Closed-class categories, including 72975 in AUX/CCONJ/PART, 116300 in CCONJ/NUM/PRON, and 86665 in DET/PRON/PUNCT.The authors suggest these recurrences may indicate polyfunctional latents.Source: §B.1 · p. 13

B.2 RQ2: POS-related Latent Groups

Open-class POS generally have more nonzero latents and classifier coefficients than closed- and other-class POS, with INTJ and ADP as exceptions. Compact-feature classification is relatively insensitive to C, while τ trades performance against feature count.

B.2 RQ2: POS-related Latent Groups

  • Open-class POS have more nonzero latents and coefficients overall, while other-class POS have the fewest nonzero coefficients on average.Open-class coefficient counts also show the highest variability.Source: §B.2 · p. 14, Table 4 · p. 14, Table 5 · p. 14
  • INTJ is an exception among open-class POS, with few nonzero latents, while ADP has more latents than typical closed-class POS.The authors suggest ADP's adverbial-clause uses may contribute to its pattern; INTJ's low treebank support may contribute to its limited active features.Source: §B.2 · p. 14
  • Compactness is measured as coverage divided by the number of salient latents per POS, and Table 7 reports multiclass performance using the compact feature set L∗.The supplied passages define the measure and identify the evaluation, but do not provide the table's performance values.Source: §B.2 · p. 14
  • Classifier performance changes by about 5 points as τ varies, even with three times fewer features; changing C yields almost no difference.The reported cross-validation standard deviation for C is near zero.Source: §B.2 · p. 14, Figure 13 · p. 15

B.3 RQ3: Validation on held-out data

Held-out activations show that category-associated latent groups fire across POS tokens, while controlled-template distributions are mostly stable across sentence types. Distinctiveness varies by tag, with INTJ and X least distinctive.

B.3 RQ3: Validation on held-out data

  • In most POS cases, 1–4% of the category-associated latents fire on all tokens of that category in the Treebank test set.The analysis also counts tokens with no activation from the category's selected latent group, but the supplied passage gives no fractions.Source: §B.3 · p. 15, Figure 14 · p. 16
  • INTJ and X have the least distinctive activations, and the results show no clear POS-class trend in distinctiveness.Table 8 reports per-category zero-activation fraction, Mcc, and distinctiveness.Source: §B.3 · p. 15, Table 8 · p. 16
  • Across controlled sentence types, most POS activation distributions largely overlap; NOUN shifts slightly with an adjacent adjective in transitive constructions, while VERB shifts only marginally.AUX has two peaks in the have/had templates, reflecting the present-versus-past forms.Source: §B.3 · pp. 15–16

B.3.1 First-token activations in the controlled dataset

Adding a numbered prefix to the controlled templates shifts the first-token activations: latents active on the initial pronoun instead activate on the initial numeral.

B.3.1 First-token activations in the controlled dataset

  • Prepending “1.” to the templates shifts all latents that fire on the original first-token PRON to the new first-token NUM.The reported example uses “I see/saw a [ADJ] [NOUN]”; the authors say all tested cases showed the same behavior.Source: §B.3.1 · p. 16, Figure 19 · p. 17
Loading 2609.29362v1…