Source-linked AI summary
A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings
Alexandre Quemy
TL;DR
Nearest-neighbor intrinsic-dimension estimators are biased by a hub of short token-embedding rows near the origin. The paper identifies and removes or normalizes this hub, showing that reported dimension estimates become stable rather than growing with model size.
Problem
Token-embedding ID estimates appear heavy-tailed when measured per token and increase with parameter count when measured over the vocabulary, complicating interpretation of representation geometry.
Method
The paper identifies short-norm rows near the origin as a nearest-neighbor hub and evaluates ID estimates after removing, trimming, adding back, or normalizing these rows.
Results
After hub removal, the Pythia estimate changes from 27–122 across 160M–12B parameters to 10–17, while the per-token heavy tail also disappears across eleven models.
Takeaways & Limitations
The reported inflation is an artifact of Euclidean neighbor-ratio estimation on rows with unequal lengths, and cosine-distance or row normalization removes it.
Takeaways & Limitations
The study leaves the unembedding matrix aside and identifies further work on hub-token behavior and tied versus untied models.
Abstract
from arXiv · showhide
A token-embedding table holds a hub of short rows near its origin, and we show that this cluster biases what nearest-neighbor intrinsic-dimension (ID) estimators report. Because of the concentration of measure, a token is closer to the central cluster than to any other token, so its first two neighbors are both hub rows at nearly the same distance. As a result, the ID estimators such as TwoNN return a dimension far above the real ID. Measured one token at a time, dimension is a heavy-tailed distribution. Measured on the full vocabulary, it grows with the model's parameter count. However, when we remove the hub, the heavy tail disappears and the measured dimension collapses to a narrow range for eleven models, from GPT-2 to models such as K3 and GLM-4.7. The hub acts as a switch: a few hundred rows are enough to fully inflate the estimate. We reproduced an experiment stating that the intrinsic dimension (ID) of Pythia's token-embedding table grows with the parameter count, from $27$ to $122$ between 160M and 12B parameters. We show that this result disappears when the hub is removed: the table then reads $10$ to $17$ at every size. The hub contains a subset of the population that under-trained-token detectors flag, but on Pythia the hub that we detected and removed as a whole was updated during training: what seem to characterize these rows is simply their length, not an absence of updates. Finally, we show that normalizing the rows instead of removing them gives the same lower reading.
1. Introduction
The paper argues that short-norm token rows near the origin form a hub that inflates intrinsic-dimension estimates, producing heavy-tailed local readings and parameter-count growth in global readings. Removing or normalizing away the hub eliminates these patterns.
- Per-token intrinsic dimension is heavy-tailed, with GPT-2’s median near 389.
- Full-vocabulary ID rises from 25 on Pythia-410M to 122 on Pythia-12B.
- Short-norm rows near the origin form a hub whose removal makes the heavy tail disappear and ID flat with parameter count.
- Across eleven models, removing the hub lowers full-vocabulary estimates by up to 90%, while equal-sized control trims raise them.
- On Pythia, trimming changes the reported ID from 27–122 across 160M–12B models to 10–17.
- Normalizing every row to unit length agrees with hub removal on nine tables by replacing Euclidean with cosine distance.
2. Related work
Related work develops nearest-neighbor and local intrinsic-dimension estimators while documenting anisotropy, hubness, and heavy-tailed token-level dimension readings in embedding spaces.
- TwoNN uses second-to-first neighbor-distance ratios, while Levina–Bickel estimates use a full k-neighbor set and later higher-order extensions.
- Nearest-neighbor estimators assume locally uniform sampling from a single manifold and are exposed to hubness near the data centroid.
- GPT-2 token-level dimension estimates have median 389 and interquartile range 2–531, indicating heterogeneous heavy-tailed readings.
- Token embeddings are anisotropic, with a few directions carrying most variance between tokens.
- Figure 1 compares per-cell readings, trimmed median-cell readings, full-vocabulary TwoNN, and random subsets matched to cell size.
3. Method and sources of dimension inflation
The method trims short-norm rows, recomputes nearest-neighbor ID estimates, and explains inflation through near-equal distances from hub rows to points in a high-dimensional cloud.
- All geometric quantities are computed from the input embedding matrix E ∈ R^V×D, excluding the unembedding matrix.
- TwoNN estimates dimension from µ = r2/r1, excluding tied or zero-distance ratios and retaining large ratios that lower the estimate.
- MLE-5 uses k = 5 neighbors and aggregates per-token Levina–Bickel estimates with a harmonic mean.
- Concentration of measure makes hub rows nearly equidistant nearest neighbors for cloud points, pushing r2/r1 toward 1.
- A few hub points can inflate estimates because each becomes the nearest pair for thousands of other points.
- Random-projection cells contain at most 800 rows, providing hundreds of ratios so one token cannot dominate a cell estimate.
- The trim removes the shortest-norm decile, with sweeps at 1%, 5%, and 20%, then reruns estimators without those rows as neighbors.
4. Full-vocabulary and per-region readings
Before trimming, global ID readings can grow with model size and inflation appears throughout tables rather than in one isolated region. After trimming, regional estimates align closely with the trimmed full-table value.
- Table 1 reports pre-trim readings for thirteen models, including larger Pythia tables used to reproduce prior results.
- On ten of eleven tables, full-vocabulary estimates are 10–40 for TwoNN and 13–71 for MLE-5.
- Qwen3.8-27B reaches 294 for TwoNN and 754 for MLE-5 before trimming, then 42 and 74 after trimming.
- For Pythia, MLE-5 grows from 27 to 122 between 160M and 12B parameters, reproducing published values across six models.
- After trimming, per-cell estimates match the trimmed full-table value within about 10% for every model.
5. A hub of short rows accounts for the inflation
The shortest-norm decile forms a hub whose removal sharply reduces intrinsic-dimension estimates, eliminating their parameter-count trend across Pythia. The effect is reversible with a small number of rows, predictable from the norm tail, and largely reproduced by unit-length normalization.
- Eight of eleven tables collapse after removing the shortest-norm decile, reducing MLE-5 by 41–90% and TwoNN by 22–86%.Random and longest-norm controls move estimates upward or inconsistently instead.
- A few hundred short-norm rows are sufficient to inflate estimates, while returning a random quarter of the removed decile restores the untrimmed values.For example, Pythia-160M changes from 10 to 26 and GLM-4.7 from 18 to 78 when rows are returned.
- The norm tail predicts inflation: deeper shortest-row norms relative to the median coincide with higher ID estimates across models.The tail is defined as the 1st-percentile norm divided by the median norm; Gemma-4 lacks the inflation, whereas Qwen3.8-27B has the deepest tail.
6. The per-token tail and the under-trained tokens are the hub
The per-token heavy tail is induced by the short-row hub rather than carried by intrinsically high-dimensional tokens. Under-trained-token detectors overlap partly with this hub, but Pythia’s hub rows were updated during training and are distinguished by their final lengths.
- Per-token tail: The per-token ID tail disappears after removing the shortest-norm decile: the median falls from 52 to 24, with no token above 300.The removed rows themselves have median ID 14, so the hub induces the tail in other tokens.
- Per-token tail: 93.5% of untouched tokens receive lower ID estimates after hub removal, with a Spearman correlation of 0.88 between prior ID and its decline.The longest rows are most inflated: 54% of the longest quarter exceeds 300, versus 4% of the shortest quarter.
- Radius dependence: At small radius, the released test’s median falls from 5071 to 41 and the share above 300 drops from 82% to none after trimming.At large radius, the reading changes only from 46 to 57; formal singularity rejections remain 3 versus 2.
- Under-trained tokens: The detector’s lower-ranked deciles overlap only 6% with the norm-based hub, although 50 of 97 prompted verified tokens fall in that decile.On Pythia-6.9B, all 36 verified tokens fall in the decile, but its detector indicator is the embedding norm itself.
- Under-trained tokens: Pythia’s removed rows moved as far as other rows during training, with cosine distance 0.02 from their initial positions, despite ending with short norms.The evidence is therefore weaker for treating the entire hub as under-trained.
7. Discussion and conclusion
The paper concludes that short-row hubness is an artifact of Euclidean nearest-neighbor ID estimation, not an intrinsic property of token tables. Removing or normalizing the hub yields stable readings, while the causes of norm tails and their relation to training remain unresolved.
- Discussion: Euclidean neighbor-ratio estimators inflate ID because short rows near the origin become nearly equidistant nearest neighbors for many tokens.After hub removal, regional readings lie within about 10% of one another and the full-vocabulary estimate is their harmonic mean.
- Conclusion: After trimming, Pythia’s six-model reading is 9.5 to 17 instead of 27 to 122, and Robinson et al.’s median falls from 507 to 41.Three versus two manifold-hypothesis rejections remain, and the paper does not explain those tokens.
- Practical implication: Reporting estimates before and after removing the shortest decile, or on normalized rows, provides a practical hub diagnostic; norm histograms offer a cheap preliminary check.Disagreement between raw and adjusted readings signals a hub.
- Open questions: The paper does not know why tables grow norm tails and could not prove that the hub is under-trained.Its hypothesis about tied versus untied updates remains unconfirmed because Pythia’s removed rows moved as much as other rows during training.
- Open questions: Future work leaves the unembedding matrix, residual-stream behavior of hub tokens, and tied-versus-untied differences in ID estimation unresolved.These are explicitly identified as future research directions.
Appendix A. Complete trim results
The appendix enumerates trim designs and controls for full-vocabulary TwoNN and MLE-5 readings. It also tests alternative trim fractions, re-insertion, subsampling, padding rows, and normalized embeddings.
- Table 3: Table 3 compares shortest-norm trimming with random, longest-norm, and farthest-from-centroid 10% removals for TwoNN and MLE-5.Random values are medians over three seeds, and removed rows are excluded as neighbors.
- Additional appendix tests: Appendix tables vary the removed fraction from 1% to 20%, reinsert quarters of the removed decile, test random subsamples, and repeat trimming on normalized embeddings.Padding-only tests and padding rows retained during shortest-decile trimming are also included.
- Table 3: The farthest-from-centroid decile is omitted because it reads within one unit of the longest-norm decile on every table.The caption specifies the control-arm interpretation for Table 3.
Appendix B. Case studies: the negative controls and the extreme table
The case studies distinguish genuine hub-driven inflation from controls and examine the most extreme table. Short-row trimming collapses inflated readings, while negative controls change little or only partially.
- Negative controls: 12.6 →12.8 for MLE-5 on Gemma-4 shows that trimming changes essentially nothing in this negative control.Gemma-4 has equal row norms by construction, despite its 262k-token vocabulary.
- Negative controls: 14.7 →12.8 for GPT-2 shows only a mild trimming effect, with shortest rows corresponding to frequent rather than rare tokens.
- Partial collapse: 20 →15 for Qwen2.5-0.5B after trimming, with normalization also giving 15, indicates partial rather than full collapse.The table has no row below half the median norm, consistent with a shallow short-row tail.
- Extreme table: 294 for TwoNN and 754 for MLE-5 make Qwen3.8-27B the most inflated table before trimming.The shortest decile contains 3,924 rows below half the median norm, and removing it reduces the two readings by 86% and 90%.
- Extreme table: 754, 223, 90, and 74 are the Qwen3.8-27B MLE-5 readings after returning 0%, 1%, 5%, and 10% of rows, respectively.Most of the hub lies in the shortest few percent; the decile is the instrument setting rather than the hub’s size.
Appendix C. Reproducibility
The reproducibility protocol audits released input embedding matrices in float32 and uses exact full-vocabulary nearest neighbors with fixed seeds and no training. It defines the norm-tail, correlation, and coverage measurements used in the audit.
- Protocol: All tables are released-checkpoint input embedding matrices analyzed in float32.
- Protocol: Exact nearest neighbors are computed over the full vocabulary, and trimmed rows are removed both as points and as neighbors.
- Protocol: Zero-first-neighbor rows are excluded from both readings, including GLM-4.7’s 128 zero-norm rows and 127 duplicate pairs.
- Audit measures: Tail measures shortest-row norms relative to the median and counts rows below half the median norm.
- Audit measures: ρS is the Spearman correlation between row norm and log WikiText-103 count, while Cov. is the vocabulary fraction seen at least once.
- Checkpoint audit: Table 11 records vocabulary width, embedding width, padded rows, zero and tiny rows, and exact duplicates for each checkpoint.
Appendix D. Supplementary figures
Synthetic calibration plants a known-dimensional bulk with a measured-geometry hub in 768 dimensions, while supplementary sweeps test hub re-insertion on real tables. The calibration separates estimator compression from hub-induced inflation.
- Synthetic calibration: 5/14/28/45 are the no-hub readings at planted dimensions d = 5/15/40/80, showing increasing underestimation above d ≈15.This is the known compression behavior of neighbor-ratio estimators.
- Synthetic calibration: Up to 20% hub rows leave the cell median at the planted value in the synthetic calibration.
- Real-data sweep: Figure 4 presents the re-insertion sweep on Pythia-160M and GLM-4.7, corresponding to Table 5.
- Synthetic calibration: Figure 5 compares planted dimensions with readings after adding a hub clump whose geometry matches real embedding tables.