Source-linked AI summary
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, Chris Dyer
TL;DR
Word vectors lack a standardized evaluation approach, so word similarity tasks are widely used as a fast proxy based on human judgments. The paper analyzes problems in these tasks and reviews proposed solutions, concluding that they can produce incorrect inferences and that alternative evaluation methods need further research.
Problem
Word vectors lack a consensus evaluation method, while word similarity tasks are widely used as an inexpensive intrinsic proxy based on human similarity judgments.
Method
The paper surveys existing literature to comprehensively analyze problems associated with evaluating word vectors on word similarity datasets and summarize existing solutions.
Results
The study suggests that word similarity tasks can lead to incorrect inferences when evaluating word vectors.
Takeaways & Limitations
The paper calls for further research on evaluation methods and suggests task-specific downstream evaluation until a better intrinsic method is found.
Takeaways & Limitations
Frequently used word vector models often ignore word senses, although multiple-vector approaches have been developed to represent polysemy.
Abstract
from arXiv · showhide
Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.
1 Introduction
Word similarity tasks became a popular, fast intrinsic evaluation for word vectors because they compare vector-based similarity with human judgments, despite the absence of consensus on evaluation methods. The paper comprehensively analyzes problems with these tasks and summarizes existing solutions.
- There is no community consensus on the best way to evaluate word vectors.
- Word similarity evaluation measures how well vector representations match human similarity ratings for word pairs.
- Cosine similarity ranks word pairs by vector-space similarity, while Spearman’s correlation compares those rankings with human judgments.
- Word similarity evaluation is attractive because it is computationally inexpensive and fast, enabling faster prototyping and development.
- The paper surveys existing literature to identify problems in word similarity evaluation and summarize available solutions.
2 Problems
The paper identifies several problems with word-similarity evaluation, including ambiguity in human judgments, dataset overfitting, weak links to downstream performance, significance issues, frequency effects, and polysemy. It reviews proposed remedies but finds that important limitations remain.
- 2.1 Subjectivity of the task: Word similarity is subjective and often conflates similarity with relatedness, producing anomalous human ratings that can penalize appropriate representations.For example, cup and coffee are rated more similar than car and train despite being related rather than similar.
- 2.2 Task-specificity: Task-specific embeddings can be unfairly penalized because they encode similarities useful for their training task rather than general semantic similarity.For POS tagging, cat and man may be treated as similar because both are nouns.
- 2.3 Overfitting: Word-similarity datasets lack train-development-test partitions, so optimizing on them can overfit the test set and researcher-created splits can make studies incomparable.The datasets are also small, averaging approximately 781 word pairs, which makes further partitioning problematic.
- 2.4 Low correlation with extrinsic evaluation: Word-similarity performance has no strong correlation with downstream NLP-task performance, weakening its validity as a proxy for extrinsic evaluation.The paper therefore calls for alternative evaluation approaches.
- 2.5 Absence of statistical significance: Statistical significance is often omitted, so apparent gains between vector models may reflect optimizer instability rather than reliable differences.Rastogi et al. propose a minimum required difference for significance based on correlations among model rankings and human rankings.
- 2.6 Frequency effects in cosine similarity: Cosine similarity is polluted by frequency-based effects, while normalization methods have uncertain applicability as absolute distances for word-similarity evaluation.Words with similar frequency can appear closer in vector space than their meanings warrant.
- 2.7 Inability to account for polysemy: Many evaluation datasets assume one sense per word, which can penalize models that correctly represent a different sense or use multiple vectors for polysemous words.The paper notes that most common models ignore word senses, although multiple-vector approaches exist.
3 Conclusion
The paper concludes that word-similarity evaluation can lead to incorrect inferences about word-vector models and requires better evaluation methods. Until then, it recommends comparing models on downstream tasks, accepting that rankings may differ by task.
- The study concludes that word-similarity evaluation can lead to incorrect inferences and calls for further research on evaluation methods.
- Until better intrinsic evaluation exists, the paper recommends task-specific comparison on downstream NLP tasks.Different tasks may produce different model rankings because vector models capture information with varying usefulness for each task.