Source-linked AI summary
Evaluating Word Embedding Models: Methods and Experimental Results
Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, C. -C. Jay Kuo
TL;DR
The paper addresses how to evaluate and select word embedding models when different evaluators emphasize different qualities. It reviews models and intrinsic and extrinsic evaluation methods, experimentally compares six models, and studies evaluator consistency through correlation analysis. The results show that evaluator behavior varies, some evaluators align more closely with NLP tasks, and SGNS-based models tend to perform better in the reported tasks.
Problem
Selecting a good word embedding model remains difficult because evaluators emphasize different aspects and models do not perform uniformly across evaluations and tasks.
Method
The study compares intrinsic and extrinsic evaluators across six embedding models trained on the same wiki2010 corpus and analyzes consistency using correlations.
Results
Different evaluators focus on different aspects, some correlate more with NLP tasks, and SGNS-based models tend to outperform other models in the reported evaluations.
Takeaways & Limitations
Evaluator choice should depend on the application task, while broader evaluation metrics remain needed to assess overall embedding quality.
Abstract
from arXiv · showhide
Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evalutors.
I. INTRODUCTION
Word embeddings represent semantic and syntactic information for NLP, but determining what makes a good model remains open. The paper reviews embedding models and evaluation methods, emphasizing the need to compare intrinsic and extrinsic evaluators comprehensively.
- Word embeddings are real-valued representations learned from large unlabeled corpora that encode semantic and syntactic meanings.
- Intrinsic evaluators assess representations independently of specific NLP tasks, while extrinsic evaluators measure downstream-task performance using embeddings as input features.
- Existing evaluators emphasize particular aspects, so models can rank differently across intrinsic evaluations and tasks.
- Prior correlation studies were limited, motivating a more comprehensive analysis designed to reduce bias and support task-specific model selection.
- The paper reviews popular embedding models, desired representation properties, intrinsic and extrinsic metrics, experiments, and correlation-based consistency analysis.
- CBOW predicts a center word from surrounding context, whereas skip-gram predicts surrounding words from a center word.
C. Co-occurrence Matrix
Co-occurrence-matrix methods represent words using global or window-based word-context statistics. GloVe combines co-occurrence information with a weighted least-squares objective, while SVD offers a lower-dimensional representation with known practical drawbacks.
- A co-occurrence matrix records how often words appear in documents or within specified windows around center words.
- SVD factorizes the co-occurrence matrix and uses its first k columns to obtain k-dimensional word embeddings targeting semantic information.
- SVD-based embeddings face imbalance in word frequency, sparsity, high dimensionality, and computational complexity.
- GloVe combines SVD-style global statistics with log-linear modeling through a weighted least-squares objective containing word-word co-occurrence counts.
- GloVe processes only nonzero co-occurrence entries and is described as more efficient and more accurate because it incorporates co-occurrence counts.
D. FastText
The section describes methods that enrich word representations with subword, n-gram, and dictionary information. These approaches target rare-word estimation, broader contextual relations, and semantic similarity.
- FastText represents each word as a bag of character n-grams, explicitly using subword information to improve rare-word embeddings.
- ngram2vec incorporates word-word, word-ngram, and ngram-ngram co-occurrences while enlarging the reception window.
- ngram2vec is reported to improve word analogy and word similarity performance and to learn difficult negation pairs or phrases.
- dict2vec learns representations from dictionary entries together with a large unlabeled corpus, bringing semantically related words closer in vector space.
G. Deep Contextualized Model
The paper frames word representations and evaluators around multiple linguistic properties and practical evaluation requirements. These include capturing contextual and morphological distinctions, testing broadly and reliably, correlating with downstream performance, and remaining efficient and statistically discriminative.
- Good representations should distinguish contextual details such as plurality and tense rather than conflating them.
- Models should represent multiple senses and use context to distinguish meanings of polysemous words.
- Representations should capture phonetic, morphological, syntactic, and other facets, including changes caused by tense or prefixes.
- Reliable models should produce consistent performance despite different random initializations.
- Good geometry spreads frequent unrelated words while clustering rarer words around frequent words.
- Evaluators should use varied, reliable data and test many embedding properties comprehensively.
- Intrinsic scores should correlate with downstream NLP performance, while evaluators should remain computationally efficient and statistically discriminative.
IV. INTRINSIC EVALUATORS
Intrinsic evaluators assess word embeddings independently of downstream NLP tasks by testing semantic or syntactic relationships directly. Word similarity and analogy tests probe different properties but each has important interpretive limitations.
- Intrinsic evaluators test representations independently of specific NLP tasks by measuring syntactic or semantic relationships between words.
- Word Similarity: Word similarity correlates distances between word vectors with human-perceived semantic similarity to assess whether representations capture human judgments and distributional semantics.
- Word Similarity: Cosine similarity compares word vectors using normalized vector directions, making the evaluator computationally inexpensive and robust to scaling.
- Word Similarity: Word similarity can conflate distributional similarity with morphology, collocation, or relatedness, and its correlation with downstream-task performance may be low.
- Word Analogy: Word analogy evaluates whether vector relationships preserve selected facets by solving for a target word from an analogy such as write : writing :: read : reading.
- Word Analogy: The 3CosMul method is observed to offer better balance across different aspects than the cosine-normalized 3CosAdd approach.
- Word Analogy: Many models score under 30% on analogy tests, with synonymy and antonymy especially difficult; accuracy also decreases as source vectors move farther from the target.
C. Concept Categorization
Concept categorization groups word vectors into categorical subsets and evaluates the purity of the resulting clusters. Its interpretation is constrained by dataset, clustering, computational, and subjectivity challenges.
- Concept categorization separates a set of words into categorical subsets, such as grouping sandwich, tea, pasta, and water into two categories.
- The evaluator computes each word vector, clusters vectors into n categories, and scores cluster purity based on category consistency.
- Datasets lack standardized splits and prescribed clustering methods, while clustering can be computationally expensive for many words and categories.
- Clustering may be unreliable when word-vector distributions are uneven or clusters are not clearly defined.
- Subjective, overlapping word categories make the uncompromising cluster-purity metric inadequate for evaluating every cluster's quality.
- Concept categorization can reveal frequency effects and hubness by showing whether frequent words cluster together.
D. Outlier Detection
Outlier detection evaluates whether word clusters are semantically coherent by identifying the word least compactly related to the others. It offers a relatively objective geometry-based metric, but remains subject to human interpretation and hub-related weaknesses.
- Outlier detection finds the word that does not belong to a given group and tests the semantic coherence of vector-space clusters.
- Human performance is extremely high on outlier detection compared with word similarity tasks, providing a clearer and less subjective gold standard.
- The evaluator computes each word’s compactness as the average of pairwise semantic similarities within the cluster, selecting the lowest-scoring word as the outlier.
- Frequent-word hubs can reduce performance when rarer words are not clustered around the frequent words they relate to.
- Subjectivity remains because people may interpret relationships among word groups differently, and human judgments rely on perceived word importance.
- QVEC correlates embedding dimensions with manually constructed linguistic vectors, but subjective vectors make its scores less representative of downstream-task performance.
V. EXPERIMENTAL RESULTS OF INTRINSIC EVALUATORS
The intrinsic evaluation experiments compare six word embedding models across word similarity, analogy, concept categorization, outlier detection, and QVEC. Results vary by evaluator and dataset, although SGNS-based models generally perform strongly.
- Evaluators: Five intrinsic metrics are evaluated: word similarity, word analogy, concept categorization, outlier detection, and QVEC.
- Experimental Setup: Six models—SGNS, CBOW, GloVe, FastText, ngram2vec, and Dict2vec—are trained on the same medium-sized wiki2010 corpus for consistent comparison.Official toolkits and default training settings are used where specified.
- Word Similarity: ngram2vec performs best on the 13-dataset word similarity evaluation, while Dict2vec performs best on the Rare-Word dataset.The Rare-Word result is attributed to Dict2vec’s dictionary-based fine-tuning, which treats infrequent words equally with others.
- Concept Categorization: SGNS-based models—SGNS, ngram2vec, and Dict2vec—outperform the other models on all three concept categorization datasets.
4) Outlier Detection:
The outlier-detection evaluation uses two datasets and two metrics, revealing that model rankings can change substantially across datasets.
- Outlier Detection: Outlier detection uses WordSim-500 and 8-8-8, with both Accuracy and Outlier Position Percentage (OPP) calculated.WordSim-500 contains 500 clusters, while 8-8-8 contains eight clusters.
- Results: GloVe performs best on WordSim-500 but has the worst accuracy on the 8-8-8 dataset.The paper relates this discrepancy to differences in the properties of the two datasets.
- Interpretation: The study uses correlation analysis to investigate the inconsistency between outlier-detection results across datasets.
C. Named-entity Recognition
The extrinsic evaluation covers named-entity recognition alongside other downstream tasks using standard datasets and evaluation procedures. The supplied passages specify the NER task and shared sequential-labeling setup but do not report NER-specific results.
- Named-entity Recognition: Named-entity recognition identifies information units such as person, location, and organization names, along with numeric expressions.
- Dataset and Evaluation: The NER experiment uses the CoNLL’03 shared-task dataset with standard data splits and evaluation criteria.PTB and CoNLL’00 are used for the corresponding POS-tagging and chunking experiments.
3) Neural Machine Translation:
The NMT evaluation uses testing perplexity and finds ngram2vec, Dict2vec, and SGNS among the top-performing models. The broader consistency analysis compares intrinsic and extrinsic evaluators across sixteen models using Pearson correlation.
- Neural Machine Translation: Lower perplexity indicates better NMT performance because it corresponds to lower prediction entropy.The evaluation reports testing perplexity on 20,000 held-out sentences.
- Neural Machine Translation: ngram2vec, Dict2vec, and SGNS are the top three word models for NMT, consistent with word similarity results.
- Neural Machine Translation: SGNS-based models tend to outperform other models, although ngram2vec requires more time to process n-gram training data.Pre-trained GloVe and FastText models also do not provide better results in the reported evaluation tasks despite larger training datasets and fine-tuning.
- Consistency Analysis: The consistency study uses Pearson correlation across sixteen word models and reverses the sign of NMT perplexity because lower values are better.The collection combines eight models with eight variance-normalized versions.
- Intrinsic Evaluators: Word analogy and concept categorization are reported as effective intrinsic evaluators, while outlier detection and QVEC have important weaknesses.Outlier detection is affected by small or infrequent-word-heavy datasets, and QVEC depends on low-quality lexicon-based vectors.
- Intrinsic Evaluators: The paper recommends considering word similarity, word analogy, and concept categorization jointly because intrinsic evaluators can differ across downstream tasks.
B. Consistency of Extrinsic Evaluators
The consistency analysis finds that evaluator-task relationships vary by downstream task. It also emphasizes that no embedding model consistently performs well across all tasks, motivating task-specific evaluation and improved metrics.
- Consistency of Extrinsic Evaluators: POS tagging, chunking, and NER show no high correlation with intrinsic evaluators because their performance depends on sequential information extraction.Word meaning plays a subsidiary role in these tasks.
- Consistency of Extrinsic Evaluators: Sentiment analysis has stronger correlation with word analogy because it focuses on combining word meaning through dimensionality reduction.
- Consistency of Extrinsic Evaluators: NMT is characterized as sentence-to-sentence conversion in the consistency analysis.
- Consistency of Extrinsic Evaluators: The paper calls for computationally efficient evaluation metrics with high correlation to extrinsic scores, while noting that linguistic relations in embedding spaces remain difficult to decode.
- Consistency of Extrinsic Evaluators: No word embedding model consistently performs well across all tasks, making a universal model challenging to design.Task-specific data can be used to train models for specific tasks, while generic data may be inefficient or hurt performance.