PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models

Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff

arXiv:2610.01821v1cs.LGcs.CL

TL;DR

LLM alignment is difficult to assess across layers, models, training stages, and languages when standard metrics compare whole activation spaces under linear assumptions. This paper adapts nonlinear concept discovery to LLM token activations and uses CBA to compare concept manifolds. Its analyses report depth-structured alignment and model-dependent sharing and change across languages and training stages.

  • Problem

    How LLM representations align across layers, model scales, training stages, and languages is difficult to assess with global metrics that rely on linear assumptions and can miss nonlinear concept structure.

    Source: §1 · pp. 1–2

  • Method

    The paper adapts NLMCD to discover density-supported concept manifolds in token activations and introduces CBA, a fuzzy Rand-style score comparing pairwise membership relations.

    Source: §2 · p. 3

  • Results

    The analyses report depth-structured alignment, later-layer shifts toward mixed syntactic-semantic concepts, and multilingual sharing and cross-model alignment that vary by model family and training stage.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the studied models and stages, concept alignment and multilingual sharing are not uniform but depend on depth, model family, and training stage.

    Source: §1 · p. 2

    Evaluation excludes models above ∼32B parameters and mixture-of-experts architectures, while HDBSCAN’s compute cost limits analysis to O(100k) tokens.

    Source: §5 · p. 9

Abstract

from arXiv · show

Understanding information processing in large language models (LLMs) requires dissecting the geometric organization of their internal token representations. While existing mechanistic interpretability (MI) methods seek to extract concepts, they are constrained by a strong linearity assumption challenged by evidence of non-linear feature manifolds. We move beyond linear concepts by adapting Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from computer vision to token-level LLM activations, modeling concepts as low-dimensional manifolds. To compare concept manifolds across layers and models, we introduce a concept-based alignment (CBA) score, a generalized Rand index that measures geometric proximity without explicit feature matching. Our analysis yields six key findings: (i) a neighboring-layer sanity check shows CBA is more sensitive than PCA- or CKA-based linear baselines; (ii) layer-by-layer alignment matrices reveal two block structures in intermediate and late layers, consistent across models and obscured by linear metrics; (iii) concept composition remains syntax-dominated through most of the network before giving way to increasingly mixed syntactic-semantic concepts in later layers, with increasing output-orientation toward the final layers; (iv) multilingual concept sharing between English and Mandarin is training-dependent rather than universal, strongest in Qwen, weaker in Llama, and absent in GPT-2; (v) inter-model alignment mirrors this structure, with strong correspondence between same-family Qwen models of different scale but weak alignment across model families; and (vi) across Tulu-3 training stages, alignment is highest between adjacent stages, with the largest shift between the base model and SFT, while subsequent preference-alignment stages (DPO, RLVR) leave early layers largely unchanged and RLVR mostly preserves DPO's concepts in late layers.

1 INTRODUCTION

The paper addresses how to compare LLM representations across layers, model scales, training stages, and languages when global alignment metrics rely on linear assumptions. It adapts manifold-based concept discovery and introduces CBA, then applies these tools to alignment questions across models and languages.

1 INTRODUCTION

  • Global metrics such as CKA and CCA compare activation spaces under linear assumptions, despite evidence that relevant LLM structures can be nonlinear and multidimensional.The authors argue that such metrics can miss differences in underlying concept structure.Source: §1 · pp. 1–2
  • NLMCD treats concepts as density-supported manifolds, while CBA compares pairwise relations between independently discovered concept spaces without explicit feature matching.The approach adapts concept discovery to LLM activations and focuses on alignment.Source: §1 · p. 2, Figure 1 · p. 2
  • CBA is more sensitive than PCA- and CKA-based baselines, and layer-by-layer alignment reveals intermediate and late blocks across models.The paper also validates its clustering and characterizes the discovered concepts.Source: §1 · p. 2
  • Concepts shift from predominantly syntactic toward more mixed syntactic-semantic content in later layers, with increasing output-orientation near the final layers.These are the paper’s reported concept-composition and orientation patterns.Source: §1 · p. 2
  • English–Mandarin concept sharing varies by model family, same-family Qwen models align strongly across scales, and alignment across families is weak.Across Tulu-3 stages, alignment is highest between adjacent stages, with the largest shift from base to SFT; RLVR mostly preserves DPO concepts in late layers.Source: §1 · p. 2

2 NON-LINEAR CONCEPT DISCOVERY

The method discovers fuzzy nonlinear concepts from aligned token activations at each layer, then compares concept spaces across layers, models, and training stages. A fixed discovery profile and shared token sampling support these comparisons.

2 NON-LINEAR CONCEPT DISCOVERY

  • The study samples 100,000 shared token positions from 5,000 sentences per model and language, records residual outputs across layers, and L2-normalizes activations.Models range from 768–5,120 hidden dimensions and 12–64 layers.Source: §2 · pp. 2–3
  • Each layer’s activations are embedded to 50 dimensions with UMAP, then clustered by HDBSCAN into density-supported concepts with soft memberships and explicit noise.The same hyperparameter profile is used across layers and checkpoints; membership rows need not sum to one.Source: §2 · p. 3
  • CBA compares two fuzzy concept partitions on aligned observations through their pairwise membership relations, with a score of one for identical relations.For crisp assignments, CBA reduces to the Hüllermeier Rand index.Source: §2 · p. 3
  • The model suite covers GPT-2, Llama-3.1-8B-Instruct, and Qwen3-4B/8B/32B, with the Qwen scale sweep and matched-scale Qwen–Llama comparisons.Qwen3 is used instead of natively multimodal Qwen variants to avoid confounding scale and language effects with training-paradigm differences.Source: §2 · p. 3
  • The Tulu-3-8B base, SFT, DPO, and final checkpoints enable comparison of supervised fine-tuning separately from preference and RL optimization.The suite shares initialization with the Llama-3.1-8B checkpoint.Source: §2 · p. 3
  • For within-model comparisons, CBA, PCA-512, linear CKA, and RBF CKA use the same deterministic sample of 5,000 token rows across layers.Concept discovery uses 5,000 sentences per language; coverage, DBCV, and embedding NRMSE provide diagnostics.Source: §2 · p. 3

3 RELATED WORK

Related work uses global alignment metrics and studies cross-space alignment without explicit feature correspondence, while concept discovery has commonly emphasized linear structures. The paper positions CBA as a concept-level alignment approach and identifies a gap in comprehensive LLM alignment studies.

3 RELATED WORK

  • CKA is widely used for neural-representation alignment, while NLMCD offers finer-grained alignment through a generalized Rand index over fuzzy clusterings.The paper examines this measure in the context of LLMs.Source: §3 · p. 4
  • Relative representations and Gromov-Wasserstein alignment compare spaces without explicit matching, but operate on whole spaces or point clouds rather than individual concepts.CBA instead attributes alignment scores to discovered concept manifolds.Source: §3 · p. 4
  • LLM concepts are often modeled as linear subspaces discovered with sparse autoencoders, whereas this work uses general manifolds instead of combining SAE directions.The passage situates this choice alongside findings on nonlinear substructures.Source: §3 · p. 4
  • Prior LLM alignment studies have used CKA, but the authors identify a lack of comprehensive intra- and inter-model alignment analysis.The cited work also includes cross-architectural alignment and representation-based fingerprints.Source: §3 · p. 4

4 RESULTS

Across models, concept-based alignment reveals layer structure that linear metrics can obscure, while concept composition and cross-model comparisons vary with depth, language training, and fine-tuning stage. Concept discovery also yields substantial coverage, supporting its technical validity for LLM activations.

Result 0: Concept Quality Across Layers

  • Concept discovery has median coverage of 0.4469 and median DBCV of 0.2743, supporting its technical validity despite lower coverage and clustering validity in later layers.Coverage declines from about 0.65 in early layers to about 0.4 later; NRMSE falls from 10–12 to 3–4.Source: §4 · p. 4, Figure 2 · p. 5

Result 1: Sanity check

  • CBA identifies adjacent-layer neighbors across models, while its Neighbor z score is more informative and its plots expose detail that linear CKA misses.The neighboring-layer test averages over nine checkpoints; CBA also represents concepts as substructures for finer-grained comparison.Source: §4 · p. 5, Table 1 · p. 5, Figure 3 · p. 6

Result 2: Layer-wise alignment

  • Layer-alignment matrices show recurring band-diagonal and block structures, including a shared middle block across Qwen models and finer central structure in Qwen3-32B.All matrices have pronounced diagonal structure; the larger models generally show multiple band-diagonal and block regions.Source: §4 · pp. 5–6, Figure 4 · p. 6

Result 3: Descriptive/qualitative insights

  • Syntactic concepts remain prevalent through much of the network before mixed syntactic-semantic concepts rise later, alongside more concepts with both input and output evidence.Aggregated syntactic and semantic shares are 53.2% and 39.8% in Qwen3-8B, versus 45.1% and 45.7% in Llama-3.1-8B-Instruct.Source: §4 · p. 6, Figure 5 · p. 7

Result 4: Multilingual activation

  • English–Mandarin shared-concept excess is near zero in GPT-2 and higher in multilingual Llama and Qwen, making sharing training-dependent rather than universal.Mean excess is 0.0242 for Llama and 0.0487 for Qwen3-4B; it is 0.0430 for Qwen3-8B and 0.0392 for Qwen3-32B.Source: §4 · p. 7, Table 2 · p. 7
  • Translation-shared semantic excess peaks in early layers and declines toward the middle, but much total excess is unlabeled, so semantic dominance across all concepts is not established.Qwen3-8B and Qwen3-32B also show a local syntactic peak in their first four layers.Source: §4 · pp. 7–8, Figure 6 · p. 8

Result 5: Cross-model alignment

  • Qwen model pairs show strong depth-structured alignment, whereas Llama-3.1-8B-Instruct versus Qwen3-8B has substantially lower scores with weaker diagonal and middle-layer structure.The Qwen alignment includes early, middle, and near-output regions; Qwen3-8B versus Qwen3-32B has an extended middle block.Source: §4 · p. 8, Figure 7 · p. 8

Result 6: Finetuning case study

  • Tulu-3 alignment is highest between adjacent training stages, with DPO and RLVR near 0.85–0.86 across most layers and larger changes relative to the base model.Alignment remains above 0.8 across stages; most comparisons drop after layer 14, with a larger drop in comparisons involving the base model.Source: §4 · pp. 8–9, Figure 8 · p. 9

5 LIMITATIONS AND FUTURE WORK

The paper bounds its evaluation by model scale, architecture, compute, and extraction-corpus coverage, and documents AI use, ethics, and reproducibility.

5 LIMITATIONS AND FUTURE WORK

  • The study evaluates neither models above approximately 32B parameters nor mixture-of-experts architectures, and its HDBSCAN-dominated compute limits extraction to O(100k) tokens.The authors also note that concepts depend on the extraction corpus, so some fine-grained semantic concepts may not have been activated.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI assisted implementation and suggested improvements, but not conceptual development, hypotheses, experimental design, result interpretation, or proof writing.The authors state that synthetic-data generation and theoretical-model development were not applicable.Source: AI USE STATEMENT · p. 10
  • Authors report reviewing AI-assisted work, validating code with unit tests and ablation checks, and taking responsibility for the final content.At least two authors inspected and validated LLM-generated or modified code.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The authors state that they comply with the ICLR Code of Ethics and provide the submission's source code in an anonymized repository.The ethics statement says all authors read and agree to comply; the reproducibility statement makes the full source code available.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 MODEL SELECTION

The checkpoint suite is selected to support targeted comparisons while preserving architectural and training comparability within a fixed compute budget.

A.1 MODEL SELECTION

  • Nine checkpoints from four families were chosen for architectural purity, comparability to prior MI work, targeted result coverage, and compute feasibility.The study excludes broad sweeps and models above approximately 30B parameters under its compute budget.Source: §A.1 · p. 16
  • GPT-2 is the mechanistic-interpretability anchor for prior SAE/NMF comparisons and the English-only contrast in multilingual analysis.Its small size and English-only pretraining make it the multilingual contrast case.Source: §A.1 · p. 16
  • Llama-3.1-8B-Instruct provides the approximately 8B cross-architecture reference and base weights for the finetuning-stage case study.It is retained because Llama 4 is a multimodal mixture-of-experts model outside the architectural-purity criterion.Source: §A.1 · p. 16
  • Qwen3-4B, 8B, and 32B provide a within-family scale sweep, while Qwen3-8B pairs with Llama-3.1-8B-Instruct for matched-scale comparisons.The dense Qwen3 line preserves text-only training and multilingual coverage; Tulu-3 checkpoints expose intermediate stages for separating supervised from preference/RL-based drift.Source: §A.1 · p. 16

A.2 EMBEDDING DISTANCE DIAGNOSTIC

The embedding-distance diagnostic compares pairwise distances in normalized residual activations with distances in their saved UMAP embeddings.

A.2 EMBEDDING DISTANCE DIAGNOSTIC

  • 20,000 uniformly sampled pairs of distinct token rows compare Euclidean distances between L2-normalized residual activations and saved UMAP embeddings.Pairs use seed 42, and a token row may occur in multiple pairs.Source: §A.2 · p. 16
  • The raw distance error is normalized by the mean original activation-space distance.The reported score is NRMSEraw.Source: §A.2 · pp. 16–17
  • Figure 2 reports the embedding-distance score as part of concept quality across layers.The figure caption identifies NRMSE as the raw normalized error between activation-space and reduced-embedding pairwise distances.Source: §A.2 · p. 17

A.3 CONCEPT ANNOTATION

Cluster annotations use token and context evidence to produce English concept labels and category classifications; these labels do not enter discovery metrics.

A.3 CONCEPT ANNOTATION

  • Clusters are labeled from frequent decoded tokens, causal sentence contexts, and next-token predictions, with greedy decoding producing 2–6-word English labels.Separate prompts classify English concepts by type and evidence orientation, and classify translation-shared concepts.Source: §A.3 · p. 17

A.3.1 CONCEPT NAMES

Concept labels are generated from token, context, and model-depth evidence, with prompt instructions adjusting how that evidence should be interpreted.

A.3.1 CONCEPT NAMES

  • The naming prompt supplies model, layer, corpus, tokenizer, input-token, predicted-token, and example-sentence context for labeling each cluster.Source: §A.3.1 · p. 17
  • Labels should name a shared property in 2–6 words, favoring specific grammatical functions for morphological fragments over generic labels.Source: §A.3.1 · pp. 17–18
  • Predicted next tokens receive less weight early and more weight late, while boundary-only and begin-of-sequence clusters are interpreted through their contexts and predicted continuations.Source: §A.3.1 · p. 18

A.3.2 ENGLISH CONCEPT TYPE AND EVIDENCE ORIENTATION (R3)

The R3 prompt jointly classifies each English concept cluster by type and evidence orientation using token and context evidence.

A.3.2 ENGLISH CONCEPT TYPE AND EVIDENCE ORIENTATION (R3)

  • The classifier receives up to 20 cluster tokens, ten predicted next tokens, and four contexts truncated to 200 characters, without concept names or model identity.Source: §A.3.2 · p. 18
  • Concept type is semantic, syntactic, mixed, or unclear, based on whether meaning-bearing or grammatical evidence organizes the cluster.Source: §A.3.2 · p. 18
  • Orientation distinguishes input-context evidence from predicted-next-token evidence, treating predictions from the final language-model head as auxiliary rather than a direct intermediate-layer readout.Source: §A.3.2 · p. 18

A.3.3 BILINGUAL SHARED-CONCEPT TYPE (R4)

The supplementary sections describe bilingual shared-concept classification and present supporting analyses of alignment, concept properties, multilingual activation, and finetuning-stage comparisons.

A.3.3 BILINGUAL SHARED-CONCEPT TYPE (R4)

  • Bilingual shared concepts are classified by organizing principle using activation tokens and representative parallel examples, rather than translation status alone.Source: §A.3.3 · p. 19

A.4 SUPPLEMENTARY RESULTS FOR RESULT 1: SANITY CHECK

  • The sanity-check table reports per-model nearest-neighbor alignment and mean best-neighbor z-scores, with nearest-neighbor status indicating whether the top-scoring other layer is adjacent.Source: Table 4 · p. 19, §A.4 · p. 19

A.5 SUPPLEMENTARY RESULTS FOR RESULT 2: LAYER-WISE ALIGNMENT

  • CBA layer-alignment matrices are shown for six models using a shared color scale, with self-alignment diagonals masked.Source: §A.5 · p. 19, Figure 9 · p. 20

A.6 SUPPLEMENTARY RESULTS FOR RESULT 3: DESCRIPTIVE/QUALITATIVE INSIGHTS

A.7 SUPPLEMENTARY RESULTS FOR RESULT 4: MULTILINGUAL ACTIVATION

  • For some cases, English–Mandarin token mass and shuffled-pair mass largely coincide, motivating their difference as a measure of multilingual activation.Source: §A.7 · p. 20, Figure 11 · p. 21

A.8 SUPPLEMENTARY RESULTS FOR RESULT 6: FINETUNING CASE STUDY

Loading 2610.01821v1…