Source-linked AI summary
Is Cosine-Similarity of Embeddings Really About Similarity?
Harald Steck, Chaitanya Ekanadham, Nathan Kallus
TL;DR
The paper asks why cosine-similarity can behave inconsistently as a semantic measure for learned embeddings. Using regularized linear matrix-factorization models with analytical solutions, it shows that cosine-similarities can be arbitrary or non-unique and depend on regularization, motivating caution about their use.
Problem
Cosine-similarity is widely used for semantic similarity, yet it can perform worse than unnormalized dot-products and its behavior lacks a clear explanation.
Method
The paper analyzes learned embeddings in linear matrix-factorization models using closed-form solutions and compares regularization schemes affecting their similarities.
Results
Cosine-similarities can be arbitrary, non-unique, or controlled by regularization, even when unnormalized dot-products are well-defined and unique.
Takeaways & Limitations
Cosine-similarity should not be used blindly because learning methods and regularization can make resulting embedding similarities meaningless or opaque.
Abstract
from arXiv · showhide
Cosine-similarity is the cosine of the angle between two vectors, or equivalently the dot product between their normalizations. A popular application is to quantify semantic similarity between high-dimensional objects by applying cosine-similarity to a learned low-dimensional feature embedding. This can work better but sometimes also worse than the unnormalized dot-product between embedded vectors in practice. To gain insight into this empirical observation, we study embeddings derived from regularized linear models, where closed-form solutions facilitate analytical insights. We derive analytically how cosine-similarity can yield arbitrary and therefore meaningless `similarities.' For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. We discuss implications beyond linear models: a combination of different regularizations are employed when learning deep models; these have implicit and unintended effects when taking cosine-similarities of the resulting embeddings, rendering results opaque and possibly arbitrary. Based on these insights, we caution against blindly using cosine-similarity and outline alternatives.
1 Introduction
Learned embeddings can support downstream models and data-driven similarity, but cosine-similarity sometimes performs worse than unnormalized dot-products. The paper investigates these inconsistent observations by analyzing regularized linear models.
- Learned embeddings represent discrete entities as dense vectors across domains such as language modeling and recommendation.
- They can serve as model inputs or provide data-driven semantic similarity between previously discrete entities.
- Cosine-similarity emphasizes directional alignment because embedding-vector norms are treated as less important.
- Despite widespread successful use, cosine-similarity sometimes performs worse than unnormalized dot-products between learned embeddings.
- The paper studies these inconsistent observations by deriving analytical results for learned embeddings in linear models.
2 Matrix Factorization Models
The paper analyzes linear matrix-factorization models to show how regularization and parameter non-uniqueness affect cosine-similarities of learned user and item embeddings.
- Model formulation: Matrix factorization estimates a low-rank reconstruction XAB⊤ ≈ X, with A and B producing user and item embeddings.The model represents user-item scores through the unnormalized dot-product (XAB⊤)u,i = ⟨x⃗u · A,b⃗i⟩.
- Regularization schemes: The first objective regularizes the product AB⊤, whereas the second separately regularizes the factor matrices P = XA and Q = B.The first scheme is associated with denoising or input dropout; the second is analogous to weight decay.
- Solution non-uniqueness: The first objective is invariant to arbitrary diagonal rescaling A → AD and B → BD^-1, creating multiple solutions with identical learned models.Rotation invariance also exists, but cosine similarity is invariant to rotations and not generally to these column rescalings.
- Cosine-similarity dependence: Because diagonal rescaling changes row normalizations, cosine-similarities for item-item, user-user, and user-item pairs depend on the arbitrary matrix D.The normalization matrices generally cannot compensate exactly for different choices of D.
- Consequences for similarities: Different choices of D can make distinct item embeddings have zero cosine-similarity, while user-item rankings remain identical to those from unnormalized dot-products.The zero-similarity case makes each item similar only to itself; for user-item scores, row normalization is an irrelevant constant for ranking items for a given user.
- Second objective: With separate factor regularization, the solution is unique up to rotations, so cosine-similarity of the learned embeddings is unique.This contrasts with the first objective, whose cosine-similarities are arbitrary and not unique despite an invariant learned model.
3 Remedies and Alternatives to Cosine-Similarity
The paper recommends avoiding unexamined post hoc cosine-similarity and instead controlling normalization during learning or applying similarity in the original data space.
- Train models directly with cosine similarity, potentially using layer normalization to facilitate this objective.
- Project learned embeddings back into the original space and apply cosine-similarity to the resulting smoothed user representations.The proposed representation is XÂB⊤, whose rows serve as user embeddings in the original space.
- Post hoc normalization can reduce semantic similarities compared with normalization or popularity-bias reduction performed before or during learning.Suggested approaches include standardizing X, negative sampling, and inverse propensity scaling.
4 Experiments
The experiments use simulated user-item interactions with known cluster structure to test whether learned item embeddings recover ground-truth semantic similarities.
- The experiment evaluates cosine-similarity by measuring recovery of item cluster structure when the ground-truth semantic similarities are known.
- The simulation contains 20,000 users and 1,000 items assigned randomly to five clusters.
- User interactions are sampled from cluster preferences and item popularity, with user interaction counts drawn from a power law.
- The study trains both objectives with λ = 10,000 and λ = 100, respectively, using a low-rank constraint k = 50.
- Figure 1 compares ground-truth cluster similarities with three rescalings under Eq. 1 and the unique cosine similarities from Eq. 2.
Conclusions
The paper concludes that cosine-similarity can be opaque or non-unique when applied after dot-product training, and that this concern may extend to deep models.
- Cosine-similarity is commonly used for semantic similarity between learned user and item embeddings, but regularization can implicitly determine latent-dimension scaling.
- The paper cautions that cosine-similarity effects in deep models may be even more opaque because different layers can receive different combinations of regularization.