PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Global and Local Geometry of Language Model Embeddings

Andrew Lee, Melanie Weber, Fernanda Viégas, Martin Wattenberg

arXiv:2503.21073v3cs.CLcs.LG

TL;DR

How similar is the geometry of token embeddings across contemporary language models? This paper compares global orientations and local structure using LLE and intrinsic dimension, finding shared geometry and semantically coherent low-dimension clusters. It also introduces EMB2EMB for linearly transforming steering vectors across models.

  • Problem

    The geometric similarities of token embeddings across contemporary language models are less characterized than alignment across embedding techniques, modalities, and languages.

    Source: §1 · p. 1

  • Method

    The paper compares global token orientations, local LLE weights, and a PCA-based intrinsic-dimension measure, also examining semantic coherence with ConceptNet.

    Source: §1 · p. 2, §5.2 · p. 5, §5.3 · p. 6

  • Results

    Token embeddings often share global orientations and local structures across models, while lower intrinsic dimensions are associated with semantically coherent clusters.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    EMB2EMB linearly transforms steering vectors for reuse across language models, including models with different dimensions.

    Source: §1 · p. 2, §7 · p. 10

    Some tokens with zero cosine similarity may be undertrained tokens present in the tokenizer but absent from the training data.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

Researchers have recently suggested that models share common representations. In our work, we find numerous geometric similarities across the token embeddings of large language models. First, we find ``global'' similarities: token embeddings often share similar relative orientations. Next, we characterize local geometry in two ways: (1) by using Locally Linear Embeddings, and (2) by defining a simple measure for the intrinsic dimension of each embedding. Both characterizations allow us to find local similarities across token embeddings. Additionally, our intrinsic dimension demonstrates that embeddings lie on a lower dimensional manifold, and that tokens with lower intrinsic dimensions often have semantically coherent clusters, while those with higher intrinsic dimensions do not. Based on our findings, we introduce EMB2EMB, a simple application to linearly transform steering vectors from one language model to another, despite the two models having different dimensions.

1 Introduction

This work characterizes geometric properties of token embeddings in contemporary language models and studies similarities across models. It finds shared global and local structure, including low-dimensional, semantically coherent clusters, and introduces a way to transfer steering vectors across models.

Motivation

  • The study addresses similarities in contemporary language-model embedding geometry, complementing prior alignment work across embedding techniques, modalities, and languages.It focuses on geometric properties of token embeddings and similarities across language models.Source: §1 · p. 1

Findings

  • Token embeddings from models in the same family often have similar relative orientations, and this alignment appears to persist in hidden layers.The paper also introduces EMB2EMB, which linearly transforms steering vectors for reuse across models with different dimensions.Source: §1 · p. 2
  • Comparing Locally Linear Embedding weights shows that language models often construct similar local representations.LLE approximately reconstructs a token’s local space as a weighted sum of its nearest neighbors.Source: §1 · p. 2
  • A simple intrinsic-dimension measure indicates that token embeddings may lie on lower-dimensional manifolds, with lower-dimension tokens forming more semantically coherent clusters.The paper also reports that tokens have similar intrinsic dimensions across language models.Source: §1 · p. 2

2 Related Work

Prior work examines representation similarity, embedding geometry, and alignment across techniques, modalities, and languages. This paper focuses on geometric similarities among language-model embeddings and relates them to transferring steering vectors across models.

Shared Representations & Geometry

  • Representation similarity has been studied through layer stitching, while the Platonic Representation Hypothesis proposes convergence across large models and modalities given vast training data.The paper positions its contribution as demonstrating geometric similarities in language-model embeddings.Source: §2 · p. 2

Geometry of Embeddings

  • Embedding geometry research covers embeddings’ role in the residual stream and their encoding of relational and hierarchical information.The cited examples include word2vec relations and hierarchical information in contemporary language-model token embeddings.Source: §2 · p. 2
  • Unlike studies of word-embedding instability, this work reports numerous geometric similarities among contemporary language-model embeddings.Source: §2 · p. 2

Word Embedding Alignment; Steering Vectors

  • Prior alignment research spans embedding techniques, modalities, and cross-lingual settings; related steering-vector work shows such vectors can be linearly transformed across language models.The paper cites linear projections for cross-language word embeddings and Gromov-Wasserstein distance as examples of alignment approaches.Source: §2 · pp. 2–3

3 Preliminary

The preliminary section sets notation for a Transformer’s token embeddings, hidden states, residual updates, and output projection. It distinguishes models that reuse embedding weights for unembedding from those that learn separate weights.

Transformer notation

  • A learned embedding matrix E ∈ IR^V×d assigns each input token an embedding, which forms the initial hidden state h0.Source: §3 · p. 3
  • At each layer, a learned Transformer block updates the hidden state, and its output is added back through a residual connection.Source: §3 · p. 3
  • After the final layer, the hidden state is projected back to the model’s embedding space; tied models reuse E, whereas untied models learn separate embedding and unembedding matrices.Source: §3 · p. 3

4 Shared Global Geometry of Token (Un)Embeddings.

Across three model families with shared tokenizers but differing sizes, the paper compares token-space geometry. Most models show similar relative orientations within families, though untied Llama3 models distinguish embedding from unembedding geometry.

4 Shared Global Geometry of Token (Un)Embeddings.

  • The study compares GPT2, Llama3, and Gemma2 model families, whose differently sized models share a tokenizer within each family.Source: §4 · p. 3

4.1 Model Families Share Similar Token Orientations (Usually)

  • The comparison samples 20,000 shared tokens per model and correlates the entries of pairwise cosine-similarity matrices to assess relative geometry.Source: §4.1 · p. 3
  • Most models show high Pearson correlations between pairwise token-similarity matrices, indicating similar relative orientations within model families despite differing embedding dimensions.Llama3 base and instruction-tuned counterparts have correlations near 1; untied Llama3 models show high correlations for unembeddings but not embeddings.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
  • For untied Llama3 models, high orientation correlations occur in the unembedding space but not the embedding space; the authors suggest final hidden states may converge across models.They note that tied models’ embedding-geometry similarity may be an artifact of shared embedding and unembedding weights.Source: §4.1 · p. 4
  • GPT-NeoX-20B and Olmo-7B, trained on different datasets despite sharing a tokenizer, have a Pearson correlation of 0.32, below scores within the same family.Source: §4.1 · p. 4

5 Local Geometry of Token Embeddings

Both LLE weights and intrinsic-dimension measurements reveal local geometric similarities across language models. The intrinsic-dimension measure also links lower-dimensional token neighborhoods with semantic coherence, while correlations vary across model families.

5.1 Locally Linear Embeddings of Tokens

  • Most token embeddings have high cosine similarity between LLE weights across models, indicating similar local structure, including in Llama3 unembeddings.The comparison uses k = 10 for the Llama3 unembedding result; similar trends are reported for GPT2, Llama3 embeddings, and Gemma2.Source: §5.1 · p. 5, Figure 2 · p. 5

5.2 Measuring Intrinsic Dimension

  • Intrinsic dimension is the number of principal components needed to explain 95% of variance among a token’s k nearest neighbors; lower-ID tokens form more semantically coherent clusters.Absolute ID values vary with the number of neighbors and variance threshold, so the authors focus on relative values; Table 1 also describes higher-ID clusters as showing syntax-level patterns.Source: §5.2 · p. 5, §5.3 · p. 6, Table 1 · p. 6

5.3 Low Intrinsic Dimensions Indicate Semantically Coherent Clusters

  • GPT2 and Llama3 show consistently high negative correlations between semantic coherence scores and intrinsic dimensions, whereas Gemma2 shows no correlation.The authors conjecture that Gemma2’s many special, emoji, and non-English tokens make it less suitable for ConceptNet; its vocabulary has 256k tokens, versus 128k for Llama3 and 50k for GPT2.Source: §5.3 · pp. 6–7

5.4 Similar Intrinsic Dimensions Across Language Models

  • Intrinsic dimensions are often highly correlated across models within GPT2 and Llama3, while Gemma2 shows low local similarity.The study compares intrinsic dimensions for 500 randomly sampled tokens; the authors leave further investigation of Gemma2’s result for future work.Source: §5.4 · p. 7, Figure 3 · p. 7

6 EMB2EMB: Transferring Steering Vectors

EMB2EMB transfers steering vectors between language models by linearly mapping unembedding spaces. Experiments report similar steering effects across models, including models with different embedding dimensions.

6 EMB2EMB: Transferring Steering Vectors

  • EMB2EMB fits a least-squares linear map from 100,000 sampled source unembedding vectors to target vectors, then applies it to a source steering vector.The map connects spaces with different dimensions.Source: §6 · p. 8
  • Tests cover seven Llama3 behaviors with two-choice evaluations and GPT2 toxicity reduction on 1,199 REALTOXICITYPROMPTS prompts.The Llama3 behaviors include corrigibility, hallucination, sycophancy, and refusal; GPT2 toxicity scores are evaluated with Perspective API.Source: §6 · p. 8
  • Transferred vectors produced trends similar to original steering in most Llama3-8B cases and similar results across GPT2 source models in most cases.Figure 4 covers Llama3-8B steering from 1B and 3B; Figure 5 compares GPT2 toxicity steering with same-model and transferred vectors.Source: §6 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 9

Steering Vector NEAREST NEIGHBORS

  • Transferred steering vectors often place tokens related to the target behavior among their nearest neighbors, as do original vectors.The vectors are projected into the unembedding space for nearest-neighbor inspection.Source: Table 3 · p. 10, Steering Vector NEAREST NEIGHBORS · p. 10
  • Residual connections may let an intervention at an earlier layer continue to affect the final hidden state despite nonlinear transformer-block operations.Each block’s output is added to the hidden state at its layer.Source: Steering Vector NEAREST NEIGHBORS · p. 10

7 Discussion

The discussion consolidates evidence that language-model embeddings share global and local geometry, including semantically coherent clusters at lower intrinsic dimensions. It presents EMB2EMB as an application of these findings and outlines a constrained LLE formulation.

7 Discussion

  • Language-model embeddings share similar global and local geometric structures, with lower intrinsic dimensions associated with semantically coherent clusters.The discussion characterizes global similarity through relative orientations and local similarity through LLE weights and intrinsic dimension.Source: §7 · p. 10
  • EMB2EMB linearly transforms steering vectors across language models as an application of the observed geometric similarities.The discussion also identifies transfer learning, distillation, and model efficiency as possible application areas.Source: §7 · p. 10

A Closed Form Solution for LLE

  • LLE weights are constrained to sum to one, while regularization makes the matrix C_i invertible in the closed-form derivation.The derivation rewrites neighbor offsets as z_j = x_j − x_i before formulating the constrained minimization.Source: A Closed Form Solution for LLE · p. 15
  • Across Figures 6–8, the captions report that language models share similar local geometry and identify these as additional results for GPT2, Llama3 (embeddings), and Gemma2, respectively.Source: Figure 6 · p. 16, Figure 7 · p. 16, Figure 8 · p. 16
  • Table 4 reports Spearman correlations between intrinsic dimension and Semantic Coherence Score; negative correlations indicate that lower dimensions accompany more coherent clusters.Its k denotes the neighbors used for intrinsic dimension, not those used for the coherence score.Source: Table 4 · p. 17

B Additional Results for LLE Similarity

Additional LLE similarity results are shown for GPT2, Llama3 embeddings, and Gemma2.

B Additional Results for LLE Similarity

C Undertrained Tokens

The authors investigate whether tokens with zero cross-model LLE cosine similarity are undertrained, comparing their candidates with tokens reported by Land & Bartolo (2024).

C Undertrained Tokens

  • The authors hypothesize that zero LLE cosine similarity marks undertrained tokens, and report that Gemma 2 has no undertrained-token reference set for comparison.They say these tokens may be present in the tokenizer but absent from training data, leaving embeddings randomly initialized and not properly updated.Source: C Undertrained Tokens · p. 17
  • 0.415 recall and 0.066 precision: the Llama3-8B procedure matched 924 of Land & Bartolo’s 2,225 reported undertrained tokens.The procedure yielded 13,967 candidates.Source: C Undertrained Tokens · p. 17
  • 0.11 recall and 0.029 precision: the GPT2-Medium procedure matched 106 of Land & Bartolo’s 967 reported undertrained tokens.The procedure yielded 3,581 candidates.Source: C Undertrained Tokens · p. 17

D Spearman Correlations between Semantic Coherence Scores and Intrinsic Dimensions

This section reports broader-hyperparameter Spearman correlations and scatter plots relating intrinsic dimension to Semantic Coherence Scores across several models.

D Spearman Correlations between Semantic Coherence Scores and Intrinsic Dimensions

  • Table 4 reports Spearman correlations between intrinsic dimension and Semantic Coherence Score across a broader range of hyperparameters.The k for intrinsic dimension is distinct from the k used to calculate SCS.Source: D Spearman Correlations between… · p. 17
  • Figures 9–11 plot intrinsic dimensions against Semantic Coherence Scores for GPT2-Medium, Llama3-3B, and Gemma2-2B.The supplied captions identify the plotted variables and models without stating a result.Source: D Spearman Correlations between… · p. 17, Figure 9 · p. 18, Figure 10 · p. 18
  • Table 5 contains example data from Rimsky et al. (2023) used to evaluate steering of Llama3.The caption does not specify the evaluation outcome.Source: Table 5 · p. 19

E Example of Steering Data

The section describes the steering evaluation setup and gives example prompts. It also points to results for steering Llama-3B with vectors transferred from Llama3-1B and Llama3-8B.

E Example of Steering Data

F Additional Examples of Transferred Steering for Llama3

Loading 2503.21073v3…