Source-linked AI summary
Whitening Sentence Representations for Better Semantics and Faster Retrieval
Jianlin Su, Jiarun Cao, Weijie Liu, Yangyiwen Ou
TL;DR
BERT sentence embeddings contain semantic information but suffer from anisotropy that limits direct similarity matching. The paper applies whitening to make the representation more isotropic and adds dimensionality reduction; experiments show improved benchmark performance alongside lower storage and faster retrieval.
Problem
BERT sentence embeddings suffer from anisotropy and related frequency and dispersion biases, making their semantic features difficult to exploit directly with simple similarity metrics.
Method
The paper applies whitening to transform sentence vectors to zero mean and identity covariance, and uses dimensionality reduction to reduce their output size.
Results
On 7 semantic textual similarity benchmark datasets, whitening generally improves performance, while dimensionality reduction further boosts performance, reduces storage, and accelerates retrieval.
Takeaways & Limitations
Whitening provides a simple post-processing alternative that can improve sentence similarity performance while making representations smaller and retrieval faster.
Takeaways & Limitations
The approach relies on the heuristic hypothesis that isotropic vectors correspond to a Standard Orthogonal Basis suitable for cosine similarity.
Abstract
from arXiv · showhide
Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. Previous work has shown that the anisotropy problem is an critical bottleneck for BERT-based sentence representation which hinders the model to fully utilize the underlying semantic features. Therefore, some attempts of boosting the isotropy of sentence distribution, such as flow-based model, have been applied to sentence representations and achieved some improvement. In this paper, we find that the whitening operation in traditional machine learning can similarly enhance the isotropy of sentence representations and achieve competitive results. Furthermore, the whitening technique is also capable of reducing the dimensionality of the sentence representation. Our experimental results show that it can not only achieve promising performance but also significantly reduce the storage cost and accelerate the model retrieval speed.
1 Introduction
BERT-based sentence embeddings can suffer from anisotropy, limiting semantic similarity matching. The paper proposes whitening, with dimensionality reduction, as a simple post-processing solution that improves performance while reducing vector size.
- Motivation: BERT sentence embeddings occupy anisotropic spaces, with word-frequency bias and sparse dispersion complicating cosine- or dot-product similarity.Contextual word embeddings can have an average pairwise cosine similarity of 0.99.
- Related approaches: Prior work addressed anisotropy using theoretical analysis, embedding regularization, or normalizing flows that map sentence distributions toward isotropic Gaussians.BERT-flow uses an invertible neural-network-based normalizing flow.
- Method: Whitening transforms sentence vectors to zero mean and identity covariance, providing a simple post-processing approach to anisotropic sentence embeddings.The method also introduces dimensionality reduction to improve the whitening procedure’s effect.
- Results: The method is reported to improve performance and achieve state-of-the-art results on most of 7 standard benchmark datasets.Dimensionality reduction further boosts performance, optimizes memory storage, and accelerates retrieval speed.
- Results: 7 semantic textual similarity tasks demonstrate that the method improves model performance significantly while reducing vector size.The result is reported across seven semantic textual similarity tasks.
2 Related Work
Related work tackles sentence representation through post-processing, supervised or unsupervised training, and architectures designed for efficient semantic matching.
- Post-processing: Earlier post-processing methods removed dominant directions or adjusted dense representations to reduce anisotropy in sentence embeddings.These approaches target common information or the geometry of dense low-dimensional representations.
- Anisotropy: Research found that upper layers of ELMo, BERT, and GPT-2 are more context-specific but also more anisotropic than lower layers.BERT-flow subsequently transformed sentence embeddings toward an isotropic distribution.
- Sentence embedding methods: Other sentence embedding methods train on SNLI, Reddit conversations, or unsupervised objectives, while poly-encoders and siamese networks support semantic comparison.These methods include siamese DAN, siamese transformers, Universal Sentence Encoder, poly-encoders, and siamese/triplet BERT variants.
3 Our Approach
The approach argues that cosine-based comparison assumes a standard orthogonal basis, then uses whitening to enforce isotropy in BERT sentence embeddings. Whitening-k additionally removes low-variation dimensions while preserving an efficient computation based on mean and covariance statistics.
- 3.1 Hypothesis: Cosine similarity’s coordinate formula is valid only under a Standard Orthogonal Basis, motivating transformation of anisotropic sentence embeddings before comparison.The paper connects poor cosine-based similarity performance to sentence vectors belonging to a non-standard coordinate basis.
- 3.2 Whitening Transformation: Whitening transforms sentence vectors so their mean is 0 and covariance matrix is the identity matrix.The transformation applies a linear mapping to existing sentence embeddings and targets an isotropic distribution.
- 3.2 Whitening Transformation: The whitening matrix is obtained from the covariance decomposition, with the orthogonal component preserving distances while converting covariance into a diagonal matrix.The paper uses SVD, Σ = UΛU^T, and derives W from the decomposition.
- 3.3 Dimensionality Reduction: Whitening-k retains the first k columns of W, discarding low-variation dimensions; this is theoretically equivalent to applying Principal Component Analysis.The retained components correspond to the descending order of the diagonal elements of Λ, while k is an empirical hyperparameter.
- 3.4 Complexity Analysis: The required mean and covariance can be computed recursively, using O(1) space and O(N) time for large-scale sentence collections.The algorithm requires only the mean vector and covariance matrix of the sentence vectors, enabling limited-memory processing.
4 Experiment
Experiments evaluate whitening across seven semantic textual similarity datasets under unsupervised and NLI-supervised settings, including dimensionality reduction. Whitening generally improves performance over flow-based baselines while reducing embedding size and enabling faster retrieval.
- Evaluation setup: Seven semantic textual similarity datasets are evaluated using Spearman’s rank correlation between cosine similarities and gold labels.The benchmarks include STS 2012–2016, STS Benchmark, and SICK-Relatedness.
- Experimental settings and baselines: The experiments compare raw and whitened BERT variants with BERT-flow and other sentence-embedding baselines under aligned settings.Both BERTbase and BERTlarge are tested, with first-and-last-layer averaging used as the default configuration.
- Results without supervision of NLI: With BERTlarge, 384 dimensions yield better results, while whitening remains competitive with BERT-flow and improves STS-B, STS-13, and STS-14 by roughly 1 point.The dimensionality and improvement claims apply to the reported unsupervised comparison.
- Results with supervision of NLI: With NLI supervision, SBERT-whitening outperforms the corresponding flow baseline on several tasks for both BERTbase and BERTlarge.SBERTbase-whitening is better on STS-13, STS-14, STS-15, and STS-16; SBERTlarge-whitening is better on STS-B, STS-14, STS-15, and STS-16.
- Effect of Dimensionality k: Dimensionality reduction trades embedding size against task performance, with the optimal reserved dimension varying across tasks.Figure 1 plots reserved dimension against Spearman’s correlation and marks each subfigure’s optimum; one example reaches 66.52 at dimension 109, 1.08 points above BERTbase-flow (NLI).
5 Conclusion
The paper concludes that whitening is a simple, effective alternative for alleviating anisotropy in sentence embeddings. Adding dimensionality reduction further improves performance while reducing memory storage and accelerating retrieval.
- Conclusion: Whitening provides a simple but effective approach for alleviating anisotropy on seven semantic similarity benchmark datasets.The conclusion presents whitening as an alternative to more sophisticated approaches.
- Conclusion: Dimensionality reduction can further boost performance while optimizing memory storage and accelerating retrieval speed.The conclusion links the reduced representation size with storage and retrieval benefits.