PaperlayerView exact revision on arXiv

Source-linked AI summary

IsoScore: Measuring the Uniformity of Embedding Space Utilization

William Rudman, Nate Gillman, Taylor Rayne, Carsten Eickhoff

arXiv:2108.07344v2cs.CLcs.LG

TL;DR

Prior work has suggested that contextualized embeddings do not use vector-space dimensions uniformly, but existing isotropy measures have not been shown to measure that property adequately. The paper introduces IsoScore and reports that it is the only tool available that accurately measures variance uniformity across dimensions, while cautioning against existing measures.

  • Problem

    Studies report nonuniform use of dimensions in contextualized embeddings, while existing methods do not truly measure isotropy, defined by variance uniformity across dimensions.

    Source: §1 · p. 1

  • Method

    The paper introduces IsoScore to measure variance uniformity across dimensions and uses a testing suite to assess methods against essential isotropy conditions.

    Source: §1 · p. 1

  • Results

    IsoScore is the only tool identified as mean agnostic, robust to scalar covariance changes, rotation invariant, linear in dimensions used, and stable with highly isotropic subspaces.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors caution that conclusions from existing methods for measuring isotropy in contextualized embeddings may be misleading or inaccurate.

    Source: §6 · p. 9

    The paper argues that existing methods have fundamental shortcomings and none of the described methods truly measure isotropy.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to measure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appropriate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously designed tests, we demonstrate that IsoScore is the only tool available in the literature that accurately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP literature that have been derived using brittle metrics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as resulting conclusions will be misleading or altogether inaccurate.

1 Introduction & Background

The paper addresses claims that contextualized embeddings underuse vector dimensions alongside the lack of adequate measures of isotropy. It introduces IsoScore and evaluates existing methods against explicit criteria.

1 Introduction & Background

  • Prior studies report that contextualized embeddings do not uniformly use all vector-space dimensions, while isotropy is defined by variance distributed uniformly across dimensions.Source: §1 · p. 1
  • A disk can use both in-plane dimensions uniformly yet fail to use all dimensions when embedded in three-dimensional space.Source: §1 · p. 1, Figure 1 · p. 1
  • IsoScore measures how uniformly a point cloud’s variance is distributed across vector-space dimensions.The paper presents it as the first score to incorporate the mathematical definition of isotropy into its formulation.Source: §1 · p. 1
  • IsoScore is global, robust to changes in mean and scalar covariance, rotation invariant, linear in dimensions used, and not skewed by highly isotropic subspaces.Source: §1 · pp. 1–2
  • The paper outlines essential conditions for measuring isotropy, tests methods against them, and shares an efficient Python implementation of IsoScore.Source: §1 · p. 2
  • The paper argues that none of the existing methods accurately measure isotropy.It identifies fundamental shortcomings in state-of-the-art tools.Source: §1 · p. 2

2 Related Work

Prior studies describe contextualized embeddings as anisotropic, “narrow cones” (Ethayarajh, 2019; Gao et al., 2019). This section reviews common spatial-distribution measures and their shortcomings.

2.1 Word Embeddings

2.2 Existing Methods

  • The Average Random Cosine Similarity Score is defined as 1 minus the average cosine similarity of 100,000 randomly sampled point pairs.Source: §2.2 · p. 2
  • The Partition Score uses a quotient involving a partition function, with values near 0 believed to indicate anisotropy and values near 1 isotropy.Source: §2.2 · p. 3
  • The ID Score normalizes estimated intrinsic dimensionality by the ambient dimension to provide a score of isotropy.Source: §2.2 · p. 3
  • The VarEx Score measures the variance explained by the first k principal components relative to k/n, but requires choosing k in advance.This makes comparisons across vector spaces with different dimensions difficult, particularly when the vector space is large.Source: §2.2 · p. 3
  • Existing tools have fundamental shortcomings, and none of the described methods truly measure isotropy, making claims based on them misleading.Source: §2.2 · p. 3

3 Measuring Embedding Space Utilization

The section defines isotropy as uniform variance across dimensions and formalizes how many dimensions a point cloud uniformly utilizes. It sets criteria an isotropy metric should satisfy, including invariance to mean, scale, and rotation, and sensitivity to dimension use and subpopulations.

3.1 Definition of Isotropy

  • Isotropy means variance is uniform across dimensions (covariance proportional to identity); the paper excludes spaces of dimension one and single-point data.Source: §3.1 · p. 3

3.2 Dimensions utilized

  • Uniformly utilizing k dimensions means k nonzero diagonal variances are equal, with remaining dimensions unused; IsoScore rises linearly with utilized dimensions in R9.Table 2 illustrates the linear relationship.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Essential Properties of Isotropy

  • An isotropy score should ignore changes to the mean and positive scalar rescaling of covariance, returning near-maximal isotropy for covariance λI_n.Source: §3.3 · p. 4
  • A score should decrease as maximum variance dominates and remain unchanged under rotations, which PCA reorientation canonicalizes.Source: §3.3 · p. 4
  • A score should rise linearly with uniformly utilized dimensions and reflect the relative sizes of line and sphere subpopulations.The section calls this global stability and uses the “skewered meatball” test.Source: §3.3 · pp. 4–5

4 IsoScore

IsoScore measures the uniformity of variance across dimensions after PCA reorientation and normalization. Its geometric interpretation is approximately the fraction of ambient dimensions uniformly utilized, especially in high-dimensional spaces.

4 IsoScore

  • IsoScore is introduced as a metric of uniform spatial utilization for point clouds.Source: §4 · p. 5

4.1 Formal Definition of IsoScore

  • PCA reorients the point cloud, IsoScore normalizes its variance vector, measures its distance from the all-ones vector, then maps the resulting dimension-use estimate to [0,1].The normalized variance vector has Euclidean norm √n.Source: §4.1 · pp. 5–6

4.2 Geometric Interpretation for IsoScore

  • For large ambient dimension, IsoScore approximates the fraction of dimensions uniformly utilized; scores of 0 and 1 correspond to one and all dimensions, respectively.The estimated number of utilized dimensions lies in [ι(X)n, ι(X)n + 1] ∩ [1,n].Source: §4.2 · p. 6

5 Experiments

The experiments test isotropy metrics against definition-derived properties and apply IsoScore to contextualized embeddings. IsoScore finds low uniform variance utilization, without the deeper-layer decline or necessary narrow-cone geometry suggested by prior work.

5.1 Testing methods against the essential properties

  • Average random cosine similarity and partition score are not mean-agnostic, while IsoScore is, because it is a function of covariance.For isotropic Gaussians, average random cosine similarity approaches 0 as the mean-to-covariance ratio increases; zero-centering can therefore make it report maximally isotropic scores without changing variance distribution.Source: §5.1 · p. 7
  • IsoScore and intrinsic dimensionality alone remain invariant to scalar covariance changes and score a zero-mean isotropic Gaussian as 1 for every positive scale.IsoScore normalizes the covariance diagonal in Step 4.Source: §5.1 · p. 7, Figure 6 · p. 7
  • The ID Score fails the maximum-variance test: its estimate remains 2.0 across point clouds whose variance becomes less evenly distributed.IsoScore's Steps 4 and 5 increase the isotropy defect as covariance mass becomes less equitable.Source: §5.1 · p. 7, Figure 6 · p. 7
  • Only IsoScore, ID Score, and VarEx Score return constant values under the tested rotations; the approximate partition score depends too strongly on basis.IsoScore is rotation invariant by design.Source: §5.1 · p. 7, Table 3 · p. 7
  • Only IsoScore matches the ideal fraction-of-dimensions pattern; IsoScore, average cosine similarity, and partition score stay near 1 as ambient dimension increases.The ID Score fails to reach 1 when all dimensions are used and decreases as dimensionality increases; VarEx Score is unstable under that increase.Source: §5.1 · pp. 7–8, Figure 7 · p. 8
  • The partition score, IsoScore, and intrinsic dimensionality are the only global estimators in the skewered-meatball test.The test varies the relative number of points sampled from a 3D isotropic Gaussian and a 1D anisotropic line.Source: §5.1 · p. 8, Figure 7 · p. 8

5.2 Isotropy in Contextualized Embeddings

  • Every considered contextualized embedding model scores below 0.18 on IsoScore; scores do not significantly decline in deeper layers, and embeddings need not form narrow cones.The findings also indicate that contextualized models use fewer dimensions than previously thought.Source: §5.2 · p. 9
  • Average random cosine similarity estimates vary with sampling: this study sampled 250,000 pairs, versus as few as 1,000 in prior studies of millions of tokens in 768 dimensions.The authors attribute differences in reported scores as likely due to sampling noise and report that the sample count should be orders of magnitude larger than the dimension.Source: §5.2 · p. 9

6 Conclusion & Future Works

The authors conclude that IsoScore meets the stated properties for measuring uniform spatial utilization, unlike existing methods. They suggest further work on isotropy and model performance, and on applying IsoScore beyond NLP.

6 Conclusion & Future Works

  • Existing methods fail to accurately measure isotropy, while IsoScore is mean-agnostic, robust to scalar covariance changes, rotation invariant, scales with dimensions used, and stable with highly isotropic subspaces.The authors caution that conclusions drawn using existing methods may be misleading or inaccurate.Source: §6 · p. 9
  • The authors propose investigating whether decreasing isotropy could improve language-modeling performance and whether IsoScore could regularize fine-tuned embeddings against isotropy.They note that prior claims linking increased isotropy to better performance used inaccurate methods.Source: §6 · p. 9
  • Because point clouds arise in many deep-learning applications, the authors present IsoScore as a tool for studying and refining models beyond NLP.Source: §6 · p. 9

A Interpretation: IsoScore as a Summary Statistic

IsoScore is interpreted approximately as the fraction of dimensions uniformly utilized by a point cloud.

A Interpretation: IsoScore as a Summary Statistic

  • An IsoScore near α indicates that approximately n · α dimensions of Rn are uniformly utilized by the point cloud.For example, a score near 0.5 indicates that around half the dimensions are utilized; the authors report a linear increase as more dimensions are uniformly used in R9.Source: A Interpretation: IsoScore as a… · p. 10

B Pre-existing metrics, in detail

This appendix describes existing spatial-utilization metrics, including their definitions and interpretive caveats. It also states a bound on IsoScore and identifies the aims of its detailed interpretation.

B Pre-existing metrics, in detail

  • Average Cosine Similarity is defined as one minus the average cosine similarity of randomly sampled distinct point pairs, but the paper cautions that its conventional isotropy interpretation does not hold.The score is kept between 0 and 1 under this convention.Source: B Pre-existing metrics, in detail · p. 11
  • The Partition Score approximates a ratio based on the minimum and maximum partition functions over unit vectors, since computing the full optimization is intractable.The approximation uses eigenvectors of XTX.Source: B Pre-existing metrics, in detail · p. 11
  • The ID Score normalizes an intrinsic-dimensionality estimate by ambient dimension, while the variance-explained ratio requires choosing the number of principal components in advance.The paper uses maximum-likelihood estimation for intrinsic dimensionality and normalizes variance explained by the first k components by k/n.Source: B Pre-existing metrics, in detail · p. 11

C Bounds on IsoScore

D Interpretation of IsoScore, in Detail

  • The detailed interpretation appendix aims to formalize IsoScore as approximately the fraction of uniformly utilized dimensions and to justify it as an indicator of uniform spatial utilization.Source: D Interpretation of IsoScore, in Detail · p. 12

D.1 IsoScore Reflects the Fraction of Dimensions Uniformly Utilized

The appendix interprets IsoScore as an estimate of how many ambient dimensions a point cloud uniformly uses, and, in high dimensions, as the fraction of dimensions used. The interpretation depends on an assumption about the isotropy defect of uniformly used dimensions.

D.1 IsoScore Reflects the Fraction of Dimensions Uniformly Utilized

  • The heuristic’s endpoint cases are one dimension used at IsoScore 0 and all n dimensions used at IsoScore 1.Source: §D.1 · p. 12
  • For X in R3, an IsoScore near 1/2 supports inferring that X uniformly occupies approximately two dimensions.The appendix presents this as reverse reasoning from IsoScore to dimension use.Source: §D.1 · p. 12
  • IsoScore corresponds to uniform use between k and k + 1 dimensions, under the heuristic’s assumption about isotropy defect.The appendix identifies this with the inverse isotropy-defect mapping.Source: §D.1 · pp. 12–13
  • For the 2D Gaussian examples, the estimated dimensions are 1.9996, 1.6105, and 1.0281 when x = 1, 3, and 75, respectively.The latter cases are interpreted as using somewhere between one and two dimensions, and mostly one direction, respectively.Source: §D.1 · p. 13
  • IsoScore near 1/2 indicates approximately half of the ambient dimensions are used; the appendix says this relationship can be made rigorous for n ≥ 2.The cited passage introduces a derivation but does not include its resulting formula.Source: §D.1 · p. 13
  • In large ambient spaces, IsoScore is approximately the fraction of dimensions uniformly utilized.The heuristic is justified for covariance matrices of the form I(⌊αn⌋)_n.Source: §D.1 · p. 13

D.2 The IsoScore for I(k)

For covariance matrices with k nonzero entries and n − k zero entries, the appendix identifies I(k)_n as the matrix with the maximal IsoScore. The argument compares the score of this balanced covariance with other matrices using k dimensions.

D.2 The IsoScore for I(k)

  • For fixed k, the appendix states that the possible IsoScores of covariance matrices using k of n dimensions form a specific interval.The supplied passage introduces the proposition establishing that range but does not give its bounds.Source: §D.2 · p. 14
  • I(k)_n has the maximal IsoScore among covariance matrices with k nonzero entries and n − k zero entries.Source: §D.2 · p. 14
  • The proof reduces the comparison to showing that no J in Diag+(n, k) scores above I(k)_n.Jensen’s inequality supplies the bound (a1 + ··· + ak)^2 ≤ kn used in the proof.Source: §D.2 · p. 14

E Numerical Experiments

The numerical experiments specify tests of invariance, dimension utilization, and global stability through controlled point-cloud sampling. The supplied passages describe test designs and setups, not their outcomes.

E Numerical Experiments

  • The mean- and scalar-invariance tests vary a common mean from 0 to 20 and a covariance scale λ from 1 to 25, respectively.They use 100,000-point Gaussian samples in 10D and 5D.Source: E Numerical Experiments · p. 14
  • The maximum-variance test increases one diagonal covariance entry x from 1 to 75 while the other nine entries remain 1.The sample is a 100,000-point 10D Gaussian with zero mean.Source: E Numerical Experiments · p. 14
  • The rotation-invariance test rotates a 2D Gaussian point cloud by 120° and 240°, then applies PCA reorientation while retaining dimensionality.The baseline covariance has diagonal entries 1 and off-diagonal entries 0.8.Source: E Numerical Experiments · p. 14
  • The fraction-of-dimensions test varies k from 1 to 25 in 25D Gaussian samples whose first k covariance entries are 1 and remaining entries are 0.Each point cloud contains 100,000 samples.Source: E Numerical Experiments · pp. 14–15
  • The high-dimensional test samples 100,000 points from identity-covariance Gaussians as dimension increases from 2 to 100.The global-stability test adds up to 150,000 isotropic 3D Gaussian points to 1,000 points sampled from a line in 3D.Source: E Numerical Experiments · p. 15

F Geometry of Isotropy

The distributions in Figure 10 all have covariance proportional to the identity and receive IsoScore 1, yet their point-cloud geometries differ substantially.

F Geometry of Isotropy

  • Uniform, Poisson, Student-T, and ChiSquare samples all receive IsoScore 1 despite having vastly different geometries.Each distribution has covariance proportional to the identity, so variance is equally distributed in all directions.Source: F Geometry of Isotropy · p. 15, Figure 10 · p. 15
Loading 2108.07344v2…