PaperlayerView exact revision on arXiv

Source-linked AI summary

Scale, Drivers, and Limits of Representational Alignment Across Large Language Models

Jialin Wu, Meenakshi Khosla

arXiv:2610.14476v1cs.CLcs.AI

TL;DR

Aggregate similarity can obscure whether language models agree across geometric scales and which linguistic properties support their agreement. The paper introduces NW-RSA and CF-RSA to examine scale and controlled perturbations, finding strong broad geometry but weaker agreement in local neighborhoods and selective sensitivity to linguistic changes.

  • Problem

    A single similarity score can obscure alignment differences between local and broad geometry, while natural-text comparisons may not isolate finer semantic or syntactic distinctions.

    Source: §1 · pp. 1–2

  • Method

    NW-RSA varies proximity-based stimulus-pair weights across geometric scales, while CF-RSA compares intact representations with another model’s representations under controlled perturbations.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Models share broad representational geometry but agree substantially less on local neighborhoods, and alignment also persists within linguistic equivalence classes.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Claims of convergence should specify the geometric scale, stimulus distinctions, and input properties to which alignment is sensitive.

    Source: §6 · p. 9

    The model panel is limited to open-weight models, and the English stimuli cover a limited range of constructions and topics.

    Source: H LIMITATIONS · p. 29

Abstract

from arXiv · show

Comparing large language models can reveal shared representational structure and systematic differences between them. However, aggregate similarity measures can obscure how agreement varies across geometric scales and which linguistic properties support it. We introduce two complementary methods to resolve where models agree and what supports that agreement. Standard representational similarity analysis (RSA) summarizes similarity across all stimulus pairs, potentially obscuring differences in alignment between nearby and distant inputs. Neighborhood-Weighted RSA (NW-RSA) addresses this limitation by weighting stimulus pairs according to their proximity in representational space, placing alignment from local neighborhoods to global geometry on a common scale. Counterfactual RSA (CF-RSA) tests which linguistic properties support alignment by comparing intact representations in one model with representations under controlled linguistic perturbations in another. Across diverse language models, we find strong agreement in coarse representational geometry but substantial divergence in local neighborhoods. Alignment is observed both across broadly sampled natural text and within linguistic equivalence classes that preserve particular aspects of meaning or structure. Counterfactual analyses reveal selective sensitivity: changes to logical scope reduce alignment substantially more than the tested changes to hierarchical syntax, event roles, or discourse structure. Alignment also decreases when sentence structure or lexical meaning is disrupted in word lists and jabberwocky sentences, suggesting that the regularities of natural language support stronger convergence. These findings refine claims of representational universality by showing that strong overall similarity need not imply agreement in how models represent finer linguistic distinctions.

1 INTRODUCTION

The paper asks whether apparent representational convergence depends on the geometric scale and linguistic distinctions being measured. It introduces complementary comparisons across geometric neighborhoods, linguistic equivalence classes, and controlled changes to meaning and form.

1 INTRODUCTION

  • A single similarity score can obscure whether models agree in broad layout but disagree locally, and natural text may not test finer semantic or syntactic distinctions.The paper therefore frames convergence as dependent on both the comparison method and the stimulus set.Source: §1 · pp. 1–2
  • The framework separates geometric locality from linguistic locality and asks how controlled perturbations and off-manifold inputs affect alignment.Its questions cover pairwise geometry versus sentence neighborhoods, equivalence classes, interpretation-changing perturbations, and removal of lexical meaning or structure.Source: §1 · p. 2, Figure 1 · p. 2
  • NW-RSA varies pair weights from nearby to distant stimuli on a common scale, while CF-RSA compares intact representations with controlled perturbations in another model.The paper reports strong global but weaker local alignment, greatest reduction from logical-scope changes among tested counterfactuals, and weaker alignment when lexical meaning or sentence structure is removed.Source: §1 · p. 2
  • The findings qualify broad interpretations of the Platonic representation hypothesis and claims of shared representations.The passage cites Klabunde et al. (2023) and Lee et al. (2025).Source: §1 · p. 2

2 RELATED WORK

Prior work compares representations using global distance patterns, local neighborhoods, or mappings between feature spaces, while linguistic studies test particular properties within individual models. This paper combines geometric-scale comparisons with controlled linguistic inputs and perturbations to examine what supports cross-model agreement.

2 RELATED WORK

  • Standard RSA summarizes agreement across all pairwise dissimilarities, whereas mutual k-NN compares local neighbor sets, so their scores do not directly establish whether alignment is stronger locally or globally.NW-RSA instead varies pair weights within one correlation measure while retaining distance ranks, recovering global RSA when weights are uniform.Source: §2 · pp. 2–3
  • Prior convergence studies include increasing alignment among more capable vision models, while calibrated scaling trends reportedly persist for neighborhood membership but largely disappear for global similarity.Naturalistic corpora also vary topic, lexical content, syntax, and meaning simultaneously.Source: §2 · p. 3
  • Linguistic tests and probing study individual-model distinctions or property accessibility, whereas CF-RSA tests which stimulus properties support agreement between models.It compares one model’s intact geometry with another’s intact–perturbed dissimilarities.Source: §2 · p. 3

3 FRAMEWORK

The framework compares representational alignment across geometric scales, controlled linguistic distinctions, and perturbations. It uses RSA-based diagnostics alongside fitted mappings to separate what models share from which input properties support that agreement.

3.1 REPRESENTATIONAL COMPARISON METHODS

  • The framework uses NW-RSA and CF-RSA for neural-representation comparisons and evaluates Procrustes and linear predictivity as alternative alignment definitions.The shared stimulus set is represented by model-specific RDMs computed from sentence-level hidden states using correlation distance.Source: §3.1 · p. 3

3.1.1 NEIGHBORHOOD-WEIGHTED RSA

  • NW-RSA varies proximity-based pair weights within one correlation measure, enabling comparison from local to global geometric alignment.Ranks are computed once and retained as weights change; at maximal neighborhood size or τ →∞, NW-RSA reduces to global RSA.Source: §3.1.1 · pp. 3–4
  • Binary NW-RSA weights retain pairs that are mutual neighbors in either model, while graded weights emphasize nearby pairs more as τ decreases.Increasing k expands binary neighborhoods; graded weights distribute more evenly as τ grows.Source: §3.1.1 · p. 4

3.1.2 COUNTERFACTUAL COMPARISON

  • CF-RSA compares one model’s intact RDM with another’s intact–perturbed dissimilarities to test which input properties support alignment.The perturbations alter selected aspects of interpretation while preserving much surface lexical content; the decrease from global RSA quantifies sensitivity, and identity perturbation recovers global RSA.Source: §3.1.2 · p. 5
  • For fitted mappings, the paper holds the mapping learned on intact inputs fixed during counterfactual evaluation to test whether the correspondence survives the change.Procrustes compares equal-dimensional representations up to rotation and reflection.Source: §3.1.2 · p. 5, §3.1.1 · p. 5

3.2 CONTROLLED LINGUISTIC STIMULUS SETS

  • Linguistic equivalence classes define neighborhoods by fixed linguistic properties rather than representational distance, covering semantic, syntactic, entity, discourse, and task equivalence.Each class has 150 groups with one anchor and 50 neighbors; RSA among neighbors is compared with RSA over the same class’s anchors.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

The experiments compare a panel of language models across natural, controlled, and perturbed stimulus sets. Alignment is aggregated over model pairs, with each experiment using its own global RSA baseline.

4 EXPERIMENTAL SETUP

  • The model panel contains 16 checkpoints spanning encoder-only, GPT-2, Llama, Qwen, and Mistral models, with primary analyses on penultimate-layer mean-pooled hidden states.Appendix E examines robustness across layer depth.Source: §4 · p. 6
  • Five stimulus families cover diverse natural sentences, linguistic equivalence classes, controlled mismatches, paraphrase and scrambling variants, and a 2 × 2 manipulation of lexical meaning and sentence structure.The mismatches target hierarchical syntax, event roles, logical scope, or discourse structure.Source: §4 · p. 6
  • Alignment is averaged over model pairs and plotted with ±1 SEM bands, while each experiment computes its own global RSA baseline.Absolute RSA values are comparable within, but not necessarily across, experiments.Source: §4 · p. 6

5 RESULTS

Across model pairs, broad geometric alignment exceeds alignment among nearby inputs, while alignment within most linguistic equivalence classes remains comparable to the global baseline. Controlled perturbations show selective sensitivity to logical scope, and disrupting sentence structure or lexical meaning reduces alignment.

5.1 GEOMETRIC ALIGNMENT DEPENDS ON THE SCALE OF COMPARISON

  • NW-RSA rises from near 0.10 at kprop = 0.005 to ≈0.31 at kprop = 0.2, approaching the ≈0.53 global ceiling as neighborhoods widen.Graded weighting shows the same pattern, with NW-RSA approaching global RSA by τ ≈0.1; DCLM and FineWeb align more than both Control sets.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 ALIGNMENT PERSISTS WITHIN LINGUISTIC EQUIVALENCE CLASSES

  • Class-specific RSA is near the anchor baseline of 0.48 for most classes, but lower for syntactic equivalence at 0.33.Across layers, the syntactic class is lowest under all three metrics; class differences are largest in the first layer and narrow mainly in early layers.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

5.3 TESTING WHICH LINGUISTIC PROPERTIES SUPPORT ALIGNMENT WITH COUNTERFACTUAL ANALYSES

  • Mismatching logical scope reduces RSA from 0.43 to 0.18, while mismatches in discourse structure, event roles, and hierarchical syntax leave alignment nearly unchanged.The same ordering holds for Procrustes and linear predictivity; the logical-scope gap is largest at every depth, though smallest in the first layer.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4 TESTING ALIGNMENT UNDER PARAPHRASE AND WORD SCRAMBLING

  • Paraphrasing reduces RSA from 0.47 to 0.44, while scrambling and word lists reduce it to 0.22 and 0.21, respectively.Across layers, scrambled inputs fall further behind intact sentences through the middle of the network, while paraphrases move closer to intact sentences.Source: §5.4 · p. 8

5.5 ALIGNMENT UNDER REDUCED LINGUISTIC REGULARITY

  • Natural sentences align best; word lists and jabberwocky sentences yield RSA of 0.38 and 0.34, and nonword lists yield 0.31.Single noun, verb, or adjective substitutions leave RSA about 0.48; substantial alignment persists even for nonword lists.Source: §5.5 · p. 9, Figure 6 · p. 9

6 DISCUSSION

Language-model alignment depends on geometric scale and input distinctions: broad geometry is shared more strongly than local neighborhoods, while linguistic equivalence classes retain alignment. Perturbations reveal selective sensitivity, and the authors argue that convergence claims should specify the scales and properties tested.

6 DISCUSSION

  • Models share broad representational geometry but agree substantially less on relative dissimilarities within local neighborhoods.Strong overall similarity therefore need not extend to finer representational distinctions.Source: §6 · p. 9
  • Alignment also persists within linguistic equivalence classes, which define locality by linguistic properties rather than representational distance.This does not contradict the local–global NW-RSA gap because linguistic equivalence does not require geometric proximity.Source: §6 · p. 9
  • Logical-scope changes reduce alignment more than tested changes to syntax, event roles, or discourse, while paraphrases preserve it more readily than scrambling.Alignment is highest for natural sentences and decreases when lexical meaning or sentence structure is removed, although substantial agreement remains for nonword lists.Source: §6 · p. 9
  • Scrambling effects develop through middle layers, but logical-scope and regularity effects appear from the first layer and grow later.These sensitivities are therefore not confined to initial lexical representations.Source: §6 · p. 9
  • The authors recommend specifying geometric scale, stimulus distinctions, and input properties when making convergence claims; NW-RSA and Counterfactual RSA provide tools for testing them.The paper frames these tools as moving beyond whether models align to what they share and under which conditions.Source: §6 · pp. 9–10

A MODEL PANEL

  • The model panel spans encoder-only masked language models, earlier decoder-only models, and recent open-weight LLMs.Unless noted otherwise, representations are read from each model’s penultimate layer and mean-pooled over non-special, non-padding tokens.Source: A MODEL PANEL · p. 15

B STIMULUS CONSTRUCTION

The appendix documents stimulus sources and construction pipelines for equivalence classes, paraphrases, and off-manifold conditions. The human-authored stimulus check finds smaller differences among word lists, jabberwocky sentences, and nonword lists than on the authors’ stimuli.

B STIMULUS CONSTRUCTION

  • The stimulus experiments draw on public datasets, generated equivalence-class and paraphrase sets, and rule-derived off-manifold conditions.Table 2 summarizes stimulus source and scale for each experiment.Source: B STIMULUS CONSTRUCTION · p. 15, Table 2 · p. 16
  • Each of five equivalence types has 150 anchors with 50 neighbors apiece, built through generation, code checks, and separate LLM audits.The complete dataset audit checks equivalence violations, unnatural outputs, duplicate wording, repetitive templates, and cross-group distinctness.Source: B STIMULUS CONSTRUCTION · p. 15
  • The syntactic equivalence type required by far the most regeneration, while the discourse type required the least.Invalid generation calls are excluded from the discarded-sentence total, making that total a lower bound.Source: B STIMULUS CONSTRUCTION · p. 16, Table 3 · p. 17
  • Paraphrase stimuli pair 750 anchors with meaning- and truth-condition-preserving paraphrases; rule-derived variants scramble all words or retain shuffled content words and negation.The auditor checks each pair for changes to participants, quantity, negation, modality, time, location, cause, or outcome.Source: B STIMULUS CONSTRUCTION · p. 17
  • Off-manifold stimuli use 750 human-written anchors and rule-derived conditions, including pseudoword substitutions, shuffled word lists, and graded noun, verb, or adjective replacements.A rule-based audit re-derives each condition and checks alignment with its anchor; Table 5 summarizes which conditions retain meaning and structure.Source: B STIMULUS CONSTRUCTION · p. 17, Table 5 · p. 18
  • On human-authored SWJN stimuli, differences among word lists, jabberwocky sentences, and nonword lists are smaller than on the authors’ stimuli under all three metrics.The analyses use RSA, Procrustes, and linear predictivity.Source: B STIMULUS CONSTRUCTION · p. 18

C DATA AND CONTENT SAFETY

The appendix describes public and generated stimulus sources and their safety screening. It also identifies the human-authored off-manifold comparison and its alignment metrics.

C DATA AND CONTENT SAFETY

  • The study combines public datasets, LLM-generated and audited equivalence-class and paraphrase stimuli, and rule-derived off-manifold conditions.The public datasets are used within their licenses.Source: C DATA AND CONTENT SAFETY · p. 18
  • Public-dataset text is screened with Llama-Guard-4-12B and named-entity recognition, while generated equivalence-class and paraphrase prompts forbid specified sensitive content and receive a separate audit.Off-manifold conditions are rule-derived from human-written anchors and add no generated text.Source: C DATA AND CONTENT SAFETY · p. 18
  • Figure 8 reports the human-authored SWJN off-manifold analysis at the penultimate layer using RSA, Procrustes, and linear predictivity.The figure summarizes means over model pairs with ±1 SEM.Source: Figure 8 · p. 19

D ALTERNATIVE ALIGNMENT MEASURES

Alternative alignment measures broadly preserve reported patterns across geometric scale, equivalence classes, and perturbations. A weighting variant also retains the shape of the graded NW-RSA curves.

Alternative alignment measures

  • The appendix supplies Procrustes and linear-predictivity versions of the equivalence-class, perturbation, and off-manifold analyses, read at the penultimate layer.Figure 12 presents the off-manifold mapping-based metrics; the supplied text does not state its outcome.Source: D ALTERNATIVE ALIGNMENT MEASURES · p. 20, Figure 12 · p. 22

Geometric scale

Linguistic equivalence classes

Paraphrase and word scrambling

E ALIGNMENT ACROSS LAYERS

Alignment patterns vary with network depth and with the geometric scale, linguistic class, or perturbation being compared. Logical scope shows the largest counterfactual gap, while sentence regularity and paraphrase status also shape layer-wise alignment.

E ALIGNMENT ACROSS LAYERS

  • The layer-wise analyses repeat equivalence-class, counterfactual, perturbation, and off-manifold comparisons under RSA, Procrustes, and linear predictivity.Relative depth counts each model’s effective functional units; depth 1 is the last such unit and differs by a normalization step from the penultimate hidden state used for the bar comparisons.Source: E ALIGNMENT ACROSS LAYERS · p. 23

Linguistic equivalence classes

  • RSA alignment declines with depth across classes, while Procrustes and linear predictivity show rising syntactic alignment through the middle layers; the syntactic class remains lowest at every depth.Under RSA, the entity class moves from above the global sample to below it after the earliest layers; under Procrustes and linear predictivity, semantic, entity, and task classes start high and decline.Source: E ALIGNMENT ACROSS LAYERS · p. 23, Figure 13 · p. 24

Paraphrase and word scrambling

  • Paraphrases overtake scrambled inputs in early layers, while scrambled inputs align nearly as well as intact sentences at the first layer before declining with depth.Beyond the earliest layers, word lists stay close to scrambled inputs; under RSA, they align far below other conditions in the earliest layers, especially the first.Source: E ALIGNMENT ACROSS LAYERS · p. 23, Figure 15 · p. 25

Counterfactual perturbations

  • The logical-scope mismatch produces the largest counterfactual gap at every depth; it widens through the middle layers, then narrows toward output, while other gaps remain small.The event-role gap is negligible, and the discourse-structure gap is absent at the first layer and grows only slightly.Source: E ALIGNMENT ACROSS LAYERS · p. 23, Figure 14 · p. 24

Reduced linguistic regularity

  • Natural sentences align best at every depth, with larger later-layer gaps to word lists, jabberwocky, and nonword lists; single-word replacements stay close to natural sentences.Across metrics, the gap changes differently with depth; in human-authored stimuli, nonword lists align as well as or better than sentences in earliest layers, but sentences align best beyond them.Source: E ALIGNMENT ACROSS LAYERS · p. 23, Figure 16 · p. 25, Figure 17 · p. 26

F IMPLEMENTATION DETAILS

The implementation extracts and compares pretrained-model representations using a shared preprocessing and distance pipeline, then evaluates local-to-global and counterfactual alignment on specified stimuli. The work uses existing assets under their stated research and licensing terms.

F IMPLEMENTATION DETAILS

  • Sentence representations are means of non-special, non-padding token hidden states, mean-centered before correlation distance is computed.Each model receives one forward pass per stimulus, with hidden states read from the penultimate layer.Source: F IMPLEMENTATION DETAILS · p. 27
  • Equivalence-class RSA averages results across 50 neighbors for each of 150 groups, with a within-class global baseline averaged across five classes.The global baseline uses the 150 anchor sentences, one per group, within each class.Source: F IMPLEMENTATION DETAILS · p. 27
  • NW-RSA varies locality with binary k-NN masks or graded temperature weights, while CF-RSA correlates one model’s intact RDM with another’s within-model cross-condition RDM.Both NW-RSA weighting schemes recover global RSA when locality opens to the full set; each counterfactual phenomenon uses 750 matched item pairs.Source: F IMPLEMENTATION DETAILS · p. 27
  • Alignment is reported as the mean with ±1 SEM over model pairs, and contrasts are read from whether those intervals separate.No training or fine-tuning is performed; the pipeline uses forward passes, hidden-state extraction, and lightweight matrix correlations.Source: F IMPLEMENTATION DETAILS · p. 27

G EXISTING ASSETS AND LICENSES

  • The analysis uses existing assets for non-commercial research, without fine-tuning, modifying weights, or redistributing third-party weights, and follows their usage terms.The assets include pretrained checkpoints and stimulus datasets; exact identifiers and versions are to accompany the code released upon acceptance.Source: G EXISTING ASSETS AND LICENSES · pp. 27–28, Table 6 · p. 28

H LIMITATIONS

The findings are bounded by the model panel, stimulus coverage, and the specific counterfactual perturbations examined.

H LIMITATIONS

  • The model panel includes only open-weight models with feasible hidden-state extraction, leaving generalization to closed and substantially larger models untested.Source: H LIMITATIONS · p. 29
  • Stimuli are English and cover a limited range of constructions and topics, while web-derived and generated datasets retain possible noise or unintended variation.Separate generation and auditing models and manual checks reduce, but do not eliminate, these concerns.Source: H LIMITATIONS · p. 29
  • Counterfactual results concern specific perturbations that may alter multiple input properties, establishing sensitivity to tested changes without uniquely identifying their cause.Source: H LIMITATIONS · p. 29

Related papers

Loading 2610.14476v1…