PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Word Embeddings Evaluation Methods

Amir Bakarov

arXiv:1801.09536v1cs.CL

TL;DR

Word-embedding evaluation lacks consensus because semantics and evaluation targets remain difficult to define, while existing datasets and methods have important limitations. The paper surveys and classifies intrinsic and extrinsic approaches, their datasets, and key challenges. It concludes that extrinsic evaluation is useful for specific downstream tasks but cannot serve as a global quality metric, and that evaluation must also address bias and subjectivity.

  • Problem

    The field lacks consensus on how to evaluate word embeddings, while semantic relationships and evaluation targets remain difficult to define.

  • Method

    The paper systematically surveys and classifies widely used and experimental intrinsic and extrinsic evaluation methods, datasets, and challenges.

  • Results

    Extrinsic evaluation can assess embeddings for a specific downstream task but cannot provide an absolute global quality metric because downstream-task scores do not correlate.

  • Takeaways & Limitations

    A complete evaluation perspective must consider task specificity alongside subjective human assessments and biases in embedding spaces.

  • Takeaways & Limitations

    Intrinsic assessments may be biased by subjective factors, and it remains unclear whether conscious judgments capture natural-language semantic structure.

Abstract

from arXiv · show

Word embeddings are real-valued word representations able to capture lexical semantics and trained on natural language corpora. Models proposing these representations have gained popularity in the recent years, but the issue of the most adequate evaluation method still remains open. This paper presents an extensive overview of the field of word embeddings evaluation, highlighting main problems and proposing a typology of approaches to evaluation, summarizing 16 intrinsic methods and 12 extrinsic methods. I describe both widely-used and experimental methods, systematize information about evaluation datasets and discuss some key challenges.

1 Introduction

Word embeddings are widely used real-valued word representations, yet their nature, limitations, and evaluation remain unsettled. The paper surveys existing evaluation approaches and organizes them into a systematic framework.

  • Word embeddings are popular real-valued representations produced by distributional semantic models, but their nature and limitations remain poorly understood.
  • The field lacks consensus on evaluation, with downstream-task performance and cognitive-science experiments serving different research interests.
  • The paper systematizes and classifies widely used, criticized, and experimental approaches to word-embedding evaluation.
  • Its organization covers recent advances, extrinsic and intrinsic methods, and future challenges in word-embedding evaluation.

2 Brief History

Research on evaluating distributional semantics developed from early task proposals and model comparisons, but persistent methodological problems remain. These include ambiguous semantics, inadequate datasets, weak agreement between evaluation classes, and inconsistent significance testing.

  • Early work addressed distributional-semantics evaluation in 2007, followed by a 2010 survey of tasks and a 2011 comparison of DSM performance.
  • Semantic evaluation is constrained because the target relationships between words, including similarity and relatedness, lack clear definitions.
  • Many evaluation datasets lack separate training and test sets, allowing embedding models to be adjusted directly toward the evaluation data.
  • Intrinsic and extrinsic performance scores do not correlate, leaving the more adequate evaluation class unclear.
  • Evaluation research also faces subjective rating scales, inadequate penalties for overestimating dissimilarity, and missing statistical significance tests.
  • Many evaluation questions remain open despite substantial prior research.

3 Extrinsic evaluation

Extrinsic evaluation measures embeddings through supervised performance on downstream NLP tasks and includes tasks such as chunking, named-entity recognition, sentiment analysis, tagging, classification, and detection tasks. It is useful for task-specific comparisons but cannot provide a global quality score because downstream-task scores do not correlate.

  • Extrinsic evaluation uses embeddings as feature vectors in supervised models, measuring their quality through performance on downstream NLP tasks.
  • The survey selects tasks where embeddings were used rather than attempting to cover all existing NLP tasks.
  • The surveyed task inventory includes noun-phrase chunking, named-entity recognition, sentiment analysis, and shallow syntax parsing.
  • The inventory also includes part-of-speech tagging, text classification, metaphor detection, and paraphrase detection.
  • Extrinsic evaluation can give an adequate score when embeddings target one specific downstream task.
  • Creating gold-standard downstream datasets is complex, and human-assessor subjectivity remains an issue.
  • Because performance scores across downstream tasks do not correlate, extrinsic methods cannot yield an absolute or global embedding-quality metric.

4 Intrinsic evaluation

Intrinsic evaluation compares embeddings with human judgments, direct model comparisons, manually constructed lexical resources, or language-internal data. The paper organizes these approaches into a broader typology while noting uncertainty about whether conscious judgments adequately represent natural-language semantics.

  • Human-judgment methods: Intrinsic evaluation compares embeddings with human judgments on word relations, often using manually created word sets and aggregated measures such as Spearman correlation.Absolute intrinsic evaluation contrasts human assessments with embedding-derived data; comparative intrinsic evaluation instead identifies the best model within a given set.
  • Human-judgment methods: Conscious assessments may be biased by subjective interpretations of word relations, and their ability to capture natural-language semantic structure remains unclear.Assessors have time to reflect on answers, which can introduce variability because meaning lacks a clear definition.
  • Typology: The proposed intrinsic typology distinguishes conscious and subconscious evaluation, drawing on psycholinguistic off-line and on-line data-collection methods.Subconscious approaches seek less reflective responses and increasingly use neuroimaging methods previously used in psycholinguistics.
  • Typology: Thesaurus-based methods compare embeddings with knowledge bases, semantic networks, and thesauri manually constructed by linguists or ontology engineers.These methods form a separate class because they do not rely on in vivo human-assessment experiments.
  • Typology: Language-driven methods compare embeddings with data internal to language, including word forms, speech signals, or corpus co-occurrence frequencies.The paper treats these as another category distinct from both human experiments and manually constructed knowledge bases.
  • Scope of evaluation: The paper defines evaluation broadly as finding embedding correlations with data that hypothetically could carry lexical-semantic information, despite uncertain representativeness.This framing avoids excluding methods solely because the semantic information in their data cannot yet be estimated correctly.

4.1 Conscious intrinsic evaluation

Conscious intrinsic evaluation compares embedding-based word relationships with human judgments or structured linguistic tasks. The surveyed methods include similarity, analogy, thematic fit, coherence, and related categorization tests, while facing subjectivity and metric limitations.

  • Word semantic similarity: Word semantic similarity compares embedding distances with human judgments of semantic distances between word pairs.It is widely used, but the survey notes substantial criticism of its subjective assessments.
  • Word semantic similarity: Subjective ratings may reflect more than semantics, including morphological, graphematic, linguistic, psychological, and social factors.The survey also notes that similarity judgments do not correlate reliably with extrinsic evaluation performance.
  • Word analogy: Word analogy evaluates whether vector arithmetic can recover a word relationship from three given words.Metrics include 3CosAdd, 3CosMul, PairDir, and Analogy Space Evaluation, but the adequacy of relationship-transfer metrics remains contested.
  • Word analogy: Analogy datasets span morphological and semantic relations, including BLESS, Semantic Neighbors, WordRep, BATS, and SemEval-2012.The listed resources vary substantially in size, class structure, and relation types.
  • Thematic fit: Thematic-fit methods test whether embeddings identify nouns that appropriately fill a verb’s thematic role.They require role-annotated corpora, and the best representation for role-specific embeddings remains unclear.
  • Semantic coherence: Coherence evaluation asks human assessors to identify a randomly selected outlier among a target word and its nearest neighbors.This tests whether local neighborhoods form coherent semantic spaces.

4.2 Subconscious intrinsic evaluation

Subconscious intrinsic evaluation uses behavioral and neurophysiological signals rather than explicit semantic judgments. The surveyed signals include reading-time priming, brain activity, and eye movements, but data consistency, semantic specificity, cost, and availability constrain their use.

  • Semantic priming: Semantic priming measures whether a target word is read faster after a semantically related prime than after another word.The method operationalizes lexical relatedness through differences in reading time.
  • Semantic priming: The Semantic Priming Project contains 6 337 word pairs collected from 768 subjects for 1 661 target words.Pairs vary by stimulus onset asynchronies of 70 and 200ms and by priming task.
  • Neural activation patterns: Neural activation patterns are difficult to use consistently because they do not correlate across many subjects with different brain sizes and structures.It is also unclear how much of the signal reflects lexical semantics rather than orthographic or phonological properties.
  • Neuroimaging: fMRI uses blood-flow changes and EEG uses electrical responses such as N400 as neuroimaging-based gold standards for evaluation.Both methods treat measured brain responses as potential information about lexical semantics.
  • Eye movements: Eye-tracking evaluation represents each word with reading features such as fixation duration and regression counts.The Provo Corpus provides 26 features for 1 185 words, while GECO provides 48 features for 987 words.

4.3 Thesaurus evaluation

Thesaurus evaluation compares word embeddings with structured lexical or conceptual representations derived from knowledge resources. The surveyed alternatives include thesaurus vectors, dictionary graphs, attribute databases, semantic networks, and distributional significance tests.

  • Thesaurus vectors: QVEC evaluates embeddings against thesaurus vectors whose dimensions represent knowledge categories such as WordNet super-senses.Each component records the word’s occurrences in documents associated with a category.
  • Thesaurus vectors: Wikipedia can provide a broader document-based resource for thesaurus-vector evaluation through Explicit Semantic Analysis.The survey notes its use in cross-language information retrieval.
  • Dictionary vectors: Dictionary-based evaluation builds word graphs whose edge weights count one word’s occurrences in another word’s definitions.The graph is transformed into dictionary vectors used as a gold standard.
  • Dictionary vectors: Dependency-based dictionary graphs replace definition frequencies with counts of one word serving as a syntactic head of another.This variant is intended to capture relations expressed in phrases such as “a cat is an animal.”
  • Distributional comparison: The cross-match test compares two sets of word vectors trained on the same corpus to estimate their correlation and statistical significance.Its evaluation role is based on determining whether two high-dimensional samples could come from the same distribution.
  • Attribute representations: Attribute-based methods associate paired words with distinctive features and calculate distances from their attribute representations.BLESS contains 200 word pairs, while the Feature Norms Dataset contains 24 963 pairs.
  • Semantic networks: Semantic-network evaluation compares embedding similarity with graph-based similarity derived from manually constructed lexical networks.WordNet contains 155 287 words in 117 659 synsets, while DBpedia contains about 4.22 million words.

4.4 Linguistic-driven methods

Linguistic-driven methods evaluate embeddings against properties of linguistic form and usage. The surveyed approaches use phonosemantic distance or n-gram co-occurrence frequency as gold standards, but task-specific open datasets remain limited.

  • Phonosemantic methods: Phonosemantic evaluation assumes that phoneme or character patterns may correlate with word semantics.Levenshtein distance can quantify phonosemantic differences between word pairs.
  • Phonosemantic methods: The survey identifies no open datasets designed specifically for phonosemantic evaluation, with studies using their own data.This limits direct comparability across such studies.
  • N-gram methods: N-gram-based evaluation treats corpus bigram co-occurrence frequency as a gold standard for distances between phrase-group word vectors.The method expects embedding distances for phrase components to correlate with phrase frequency.
  • N-gram methods: Representative corpora or n-gram-frequency resources such as the Google 1T Frequency Dataset can supply evaluation data.The survey presents both corpora and dictionaries as possible sources of co-occurrence frequencies.

5 Future challenges

The survey identifies 28 evaluation methods while highlighting future challenges from emerging multilingual and multisense embeddings, representation questions, fairness bias, and uneven data availability.

  • 28 evaluation methods overall—16 intrinsic and 12 extrinsic—are systematized alongside existing problems and future challenges.The survey covers mainstream and less well-known approaches.
  • Multilingual and multisense embeddings require different evaluation approaches because their vectors represent translations or multiple senses.Mainstream word-similarity datasets may be even less applicable to these models than to classic monolingual, monosense embeddings.
  • Future work must address distributional representations of phrases and sentences, vector-component interpretability, and links to formal and cognitive semantics.
  • Fairness-related gender, racial, and sexual-orientation biases should be considered in evaluating whether embeddings are good models.The survey links evaluation completeness to solving bias-detection problems.
  • Evaluation quality depends heavily on available data, making extensive assessment possible for English but difficult for low-resource languages such as Urdu.The survey calls for language-independent data to address this imbalance.
Loading 1801.09536v1…