Source-linked AI summary
Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
Songhao Wu, Zhongxin Chen, Yuxuan Liu, Heng Cui, Cong Li, Rui Yan
TL;DR
LLMs underperform as zero-shot text embedding models partly because their embeddings overexpress frequent, uninformative tokens. The paper identifies an unembedding-matrix subspace behind this behavior and filters it with EmbedFilter, improving performance while reducing effective dimensionality.
Problem
LLMs exhibit a persistent performance gap as zero-shot text embedding models, limiting their use for text embedding tasks.
Method
EmbedFilter is a simple linear transformation that filters the edge spectrum subspace encoded by the LLM unembedding matrix.
Results
14.1% improvement on MTEB is reported without training overhead, with experiments across multiple backbones showing superior zero-shot embedding performance.
Takeaways & Limitations
EmbedFilter enables effective dimensionality reduction while lowering index storage overhead and accelerating retrieval.
Takeaways & Limitations
The method uses an empirical word-frequency distribution sampled from RedPajama because the models’ pretraining datasets are undisclosed.
Abstract
from arXiv · showhide
Large language models exhibit impressive zero-shot capabilities across a wide range of downstream tasks. However, they struggle to function as off-the-shelf embedding models, leading to suboptimal performance on massive text embedding benchmarks. In this paper, we identify a potential cause underlying this deficiency. Our motivation stems from an unexpected observation: text embeddings tend to align with frequent but uninformative tokens when projected onto the vocabulary space. We argue that this excessive expression of high-frequency tokens suppresses the model's ability to capture nuanced semantics. To address this, we introduce EmbedFilter, a simple linear transformation designed to refine text embeddings derived from LLMs directly. Specifically, we uncover that the unembedding matrix within LLMs encodes a latent space that is actively writing these frequent tokens into embedding space. By filtering out this subspace, EmbedFilter suppress the influence of high-frequency tokens, thereby enhancing semantic representations. As a compelling byproduct, this enables an inherent dimensionality reduction, lowering index storage and speedup retrieval while fully preserving the refined embedding quality. Our experiments across multiple LLM backbones demonstrate that LLMs equipped with EmbedFilter achieve superior zero-shot downstream performance even with significantly reduced embedding dimensions. We hope our findings provide deeper insights into the mechanisms of LLM-based representations and inspire more principled designs to improve text embeddings training. Our code is available at https://github.com/CentreChen/EmbFilter.
1 Introduction
LLMs perform well zero-shot yet remain suboptimal as off-the-shelf text embedding models. The paper attributes this gap to frequent-token bias and introduces EmbedFilter to remove its underlying subspace.
- LLMs show a persistent performance gap when used as zero-shot text embedding models, limiting their adoption for embedding tasks.
- Logit Lens analyses find that high-frequency, semantically uninformative tokens dominate text embeddings’ highest decoding probabilities across model families.The pattern suggests a vocabulary-level bias that is not tied to particular input semantics.
- The paper links this bias to anisotropic embeddings pulled toward an average-token region, which can overshadow unique semantic features.
- Logit Spectroscopy identifies an edge spectrum subspace in the unembedding matrix that actively writes frequent tokens into embedding space.Filtering the average token’s projection onto this subspace significantly disrupts frequent-token logits.
- EmbedFilter removes the identified subspace with a simple linear transformation encoded in the unembedding matrix and available without further training.
- 14.1% improvement on MTEB is reported without any training overhead, alongside dimensionality reduction, faster retrieval, and lower storage requirements.
2 Background
The paper frames text embeddings as pooled LLM representations and reviews prompt-based extraction alongside mechanistic tools that project hidden states into vocabulary and spectral spaces.
- Text Embeddings: Text embedding extraction maps an input sentence through an LLM backbone and pooling strategy into a dense vector whose similarity should reflect semantic similarity.
- Text Embeddings: The unembedding matrix normally maps hidden states to vocabulary space for token prediction, but the paper proposes exploiting it to enhance text embeddings.
- Prompt Engineering: Prompt engineering can improve LLM-derived embeddings, but gains may be modest, prompt-sensitive, inconsistent, or computationally expensive.
- Mechanistic Interpretability Tools: Logit Lens projects intermediate representations into vocabulary space to inspect how activations shape token predictions.
- Mechanistic Interpretability Tools: Logit Spectroscopy extends this analysis by projecting representations onto spectral components of the unembedding matrix.
- Mechanistic Interpretability Tools: Filtering a representation’s projection onto selected right singular vectors enables measurement of contributions from different spectral subspaces.
3 Discovery of Edge Spectrum Subspace
The analysis links LLM embedding anisotropy and frequent-token alignment to an edge-spectrum subspace encoded by the unembedding matrix. Filtering this subspace reduces frequent-token influence while preserving greater sensitivity to semantic content.
- LLM-derived embeddings concentrate in a narrow subspace and often align with high-frequency tokens carrying little semantics.
- Logit Spectroscopy identifies an edge-spectrum subspace that drives the emergence of high-frequency tokens in the average-token representation.
- The unembedding matrix and corpus word frequencies are used to reverse-engineer an “average” token representation.
- Δ𝜋 measures cumulative logit shifts for the k most frequent tokens after filtering each singular subspace; larger values indicate stronger influence on those tokens.
- With k=100, Δ𝜋 is larger at the spectral edges, while infrequent and random tokens show significantly lower edge-spectrum sensitivity than frequent tokens.
4 Text embedding with EmbedFilter
EmbedFilter removes the edge-spectrum subspace from raw LLM text embeddings using a linear transformation encoded by the unembedding matrix. Because the retained transformation preserves distances, the method supports lower-dimensional embeddings with reduced storage and faster retrieval while maintaining or improving downstream performance.
- 4 Text embedding with EmbedFilter: EmbedFilter filters the edge-spectrum subspace from raw LLM-derived text embeddings as a simple linear post-processing transformation.
- 4.1 Methodology Formulation of EmbedFilter: The Bulk Spectrum Transformation retains mid-range singular components while excluding vectors associated with the largest and smallest singular values.
- 4.1 Methodology Formulation of EmbedFilter: EmbedFilter refines existing embeddings by mapping them through the retained bulk-spectrum components for downstream tasks.
- 4.1 Methodology Formulation of EmbedFilter: The orthogonality of the singular-vector matrix makes the transformation distance-preserving, so selected components can replace the full transformation without changing similarity theoretically.
- 4.1 Methodology Formulation of EmbedFilter: Dimensionality reduction lowers index-storage overhead and retrieval cost by reducing memory bandwidth demands and distance-computation complexity.
- 4.1 Methodology Formulation of EmbedFilter: Experiments report significant dimensionality reduction while maintaining or exceeding downstream task performance.
5 Experiment
Experiments show that EmbedFilter improves zero-shot text embedding performance across MTEB setups while retaining gains under substantial dimensionality reduction. Ablations and comparisons support its effectiveness, lightweight deployment, and distinction from generic dimensionality reduction or calibration.
- Main Results: Up to a 14% increase in MTEB overall performance is maintained when embeddings are reduced to 1/8 of their original dimension.EmbedFilter consistently outperforms evaluated baselines across setups, while prompt-engineering methods show performance fluctuations.
- Main Results: EmbedFilter adds only a lightweight linear transformation and remains effective within MetaEOL and GenEOL prompting pipelines.These pipelines can require iterative commercial-LLM calls or multiple embeddings per sentence, whereas EmbedFilter is presented as lower-overhead post-processing.
- Dimensionality Reduction: The filtering ratio τ reduces embedding dimensionality to 1/τ of the original size, theoretically scaling similarity computation by τ and index storage by 1/τ.Larger τ values correspond to lower memory use and faster retrieval.
- Dimensionality Reduction: With EmbedFilter, Llama-3.1-8B-Instruct can outperform established pre-LLM embedding baselines while using smaller representation dimensions.The dimensionality-reduction evaluation is reported on MTEB.
- Ablation Studies: Randomly selecting or truncating half the dimensions underperforms vanilla PromptEOL, indicating that EmbedFilter’s gains are not explained by dimensionality reduction alone.The comparison uses Matryoshka-style truncation and random dimension selection.
- Ablation Studies: EmbedFilter outperforms filtering variants targeting dominant, secondary, or bulk singular-value subspaces, while nearly matching a task-calibrated upper-bound configuration without calibration.The inverse operation performs worst, and the authors leave optimal asymmetric edge-spectrum filtering for future work.
- Calibration Baselines: EmbedFilter outperforms whitening on Qwen without supervision from calibration data, whereas the whitening comparison uses NLI supervision.The authors attribute this advantage to statistical features captured by the LLM unembedding matrix during pretraining.
- Interpretation and Limitations: The authors describe EmbedFilter as primarily heuristic and leave deeper investigation of its underlying mechanisms to future work.They also offer a whitening-like interpretation within bulk spectral space.
6 Conclusion
The paper interprets suboptimal zero-shot LLM embeddings through the unembedding matrix and introduces EmbedFilter to remove the responsible edge-spectrum space. Experiments report improved zero-shot performance together with reduced effective dimensionality, storage, and retrieval costs.
- Conclusion: The unembedding matrix analysis identifies an edge spectrum space responsible for encoding high-frequency tokens into the embedding space.EmbedFilter is motivated as a linear transformation that filters out this spectrum space.
- Conclusion: Applying EmbedFilter produces superior zero-shot improvements across multiple LLM backbones and text embedding tasks.The conclusion presents this as the central experimental finding.
- Conclusion: Filtering implicitly reduces effective embedding dimensionality, lowering index storage overhead and accelerating retrieval.The conclusion frames this as a practical consequence of the filtering design.
A Details of the Main Experimental Setup
The main experimental setup evaluates EmbedFilter on MTEB tasks using standard metrics and a limited retrieval subset. The appendix also records the prompts used for the embedding configurations.
- Task Coverage: The evaluation covers MTEB semantic textual similarity, classification, clustering, pair classification, re-ranking, retrieval, and summarization tasks.Retrieval evaluation is limited to eight datasets because of computational-resource constraints.
- Metrics and Protocol: The reported metrics follow MTEB recommendations, with Spearman correlation computed from cosine similarity for the relevant evaluation.The experimental framework follows the official MTEB implementation.
- Prompt Configurations: PromptEOL and ECHO prompts are provided across models, with “text” denoting the sentence to be embedded.The appendix supplies the actual prompt templates used in the experiments.
B Equivalence Transformation Proof
The section defines a projection matrix, invokes the identity property of 𝜏, and concludes the proof of the identity in equation 1.
- The projection matrix is introduced as the section’s main defined object.
- Because 𝜏 is the identity, the stated transformation follows.
- The section concludes by completing the proof of the identity in equation 1.
- Figure 4 shows Δ𝜋 distributions for high-frequency, low-frequency, and randomly sampled tokens on Qwen.