PaperlayerView exact revision on arXiv

Source-linked AI summary

From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

Jose Camacho-Collados, Mohammad Taher Pilehvar

arXiv:1805.04032v3cs.CLcs.AI

TL;DR

Word embeddings provide useful semantic representations but conflate a word’s multiple meanings into one vector, limiting lexical meaning modeling. This survey reviews unsupervised and knowledge-based sense representations, their evaluation and applications, and their properties. It concludes that sense-level representations offer a framework for distinguishing meanings while highlighting limitations involving contextual evaluation, interpretability, and reliance on scarce sense-annotated data.

  • Problem

    Word embeddings conflate multiple meanings of a word into one representation, limiting their ability to capture distinct lexical meanings.

  • Method

    The survey synthesizes unsupervised sense models learned from text and knowledge-based models defined using external sense inventories, alongside their evaluations, applications, and properties.

  • Results

    The survey organizes sense representation techniques into unsupervised and knowledge-based branches and reviews their evaluation procedures, applications, interpretability, granularity, domain adaptability, and compositionality.

  • Takeaways & Limitations

    Moving from words to senses provides a more fine-grained framework for modeling unambiguous lexical meaning.

  • Takeaways & Limitations

    Sense evaluation can depend on context-sensitive modeling and may be affected by artifacts in similarity benchmarks, while unsupervised senses may not align with human distinctions.

Abstract

from arXiv · show

Over the past years, distributed semantic representations have proved to be effective and flexible keepers of prior knowledge to be integrated into downstream applications. This survey focuses on the representation of meaning. We start from the theoretical background behind word vector space models and highlight one of their major limitations: the meaning conflation deficiency, which arises from representing a word with all its possible meanings as a single vector. Then, we explain how this deficiency can be addressed through a transition from the word level to the more fine-grained level of word senses (in its broader acceptation) as a method for modelling unambiguous lexical meaning. We present a comprehensive overview of the wide range of techniques in the two main branches of sense representation, i.e., unsupervised and knowledge-based. Finally, this survey covers the main evaluation procedures and applications for this type of representation, and provides an analysis of four of its important aspects: interpretability, sense granularity, adaptability to different domains and compositionality.

1. Introduction

Word embeddings efficiently capture useful syntactic and semantic information but conflate a word’s different meanings. The survey addresses this limitation by reviewing sense-level representations, their learning sources, evaluation, and applications.

  • Word embeddings are low-dimensional vectors learned from massive text data that store syntactic and semantic information.They have improved applications including machine translation, syntactic parsing, text classification, and question answering.
  • A word embedding conflates all possible meanings of an ambiguous word into one representation, limiting semantic discrimination.The survey illustrates ambiguity through words whose meanings depend on context.
  • Word senses are individual meanings of words, and accurately representing senses is important for language understanding in NLP systems.Sense inventories list possible meanings, while frequent words tend to have more senses.
  • The survey reviews sense-representation approaches that either learn senses directly from text or exploit lexical-resource sense inventories.It also covers evaluation procedures, applications, and analyses of the approaches’ main properties.
  • The survey organizes its coverage around theoretical foundations, unsupervised and knowledge-based techniques, evaluation, downstream applications, and comparative analysis.The comparison addresses the main properties of the reviewed representations.

2. Background

Vector space models represent words as points whose proximity reflects contextual similarity, while neural word embeddings learn low-dimensional vectors from text. Because single vectors conflate multiple meanings, sense representations model individual word senses through unsupervised or knowledge-based approaches.

  • 2.1.1 Vector Space Models: Word-based vector space models represent words as points, with similar contexts implying proximity and similar semantics.Traditional vectors use normalized frequencies of co-occurring words in a corpus.
  • 2.1.1 Vector Space Models: Conventional vector space models can produce hundreds of thousands or millions of dimensions, motivating SVD and neural dimensionality reduction.The vocabulary determines the dimensionality of these representations.
  • 2.1.2 Word Embeddings: Neural word embeddings learn low-dimensional vectors from text and provide generalizable prior knowledge for NLP systems.Their generalization power has supported strong performance when integrated into neural network architectures.
  • 2.1.2 Word Embeddings: Word2vec includes CBOW and Skip-gram architectures: CBOW predicts a target word from surrounding context, whereas Skip-gram predicts context words from the target.Both architectures use input, hidden, and output layers, with vocabulary-sized input and output layers.
  • 2.2 Meaning Conflation Deficiency: Representing every word type with one vector conflates multiple meanings, which can pull semantically unrelated words together and hamper semantic understanding.The survey illustrates this deficiency with the rodent and computer-device senses of mouse.
  • 2.3 Sense Representations: Sense representations address meaning conflation by assigning independent representations to individual word senses, using unsupervised or knowledge-based sense distinctions.Unsupervised methods learn senses from text corpora, while knowledge-based methods use lexical resources.

3. Unsupervised Sense Representations

Unsupervised sense representations learn distinct word meanings from text corpora, using either separate induction and representation stages or joint training. The survey also discusses contextualized embeddings and limitations including fixed sense counts, undisambiguated context words, unclear distinctions, and scalability.

  • Unsupervised approaches: Unsupervised models induce word senses from contextual semantics in text corpora and represent each sense using corpus-derived statistical information.
  • Unsupervised approaches: Clustering-based models first induce senses and then learn their representations, whereas joint-training models perform induction and representation learning together.
  • Clustering-based models: Early context-clustering methods addressed reliance on sense-annotated data and external resources but required independent representations for every context, limiting scalability.
  • Joint models: Joint models couple clustering with embedding learning and can incorporate broader context, including whole-sentence information or globally modeled topics.
  • Limitations: Unsupervised models commonly assume a fixed number of senses despite natural variation in polysemy, and their context words are generally not disambiguated.
  • Limitations: Because induced distinctions are often unclear, unsupervised sense representations can be difficult to map to well-defined concepts and integrate into downstream models.

4. Knowledge-Based Semantic Representations

Knowledge-based semantic representation techniques use external knowledge resources to construct sense and concept representations, complementing approaches that learn only from text corpora.

  • Knowledge-based techniques exploit external knowledge resources to construct representations of senses and concepts.
  • The survey distinguishes knowledge-based methods from unsupervised models that learn word senses directly from text corpora.
  • The section covers knowledge resources, their use for improving word vectors, and representations of senses, concepts, and entities.

4.1 Knowledge Resources

Knowledge resources differ in construction, coverage, and semantic organization. The survey reviews expert-made, collaboratively constructed, automatically constructed, and merged resources used for representation learning.

  • Knowledge resources are broadly categorized as expert-made or collaboratively constructed, with each category offering distinct advantages and limitations.
  • The survey focuses on WordNet, Wikipedia-related resources, and merged resources such as BabelNet and ConceptNet.
  • Expert-made resources: WordNet organizes concepts into synsets composed of one or more synonymous lexicalizations and connects them through lexical and semantic relations.
  • Automatically constructed knowledge resources are not yet used in practice and generally lack recall.
  • Collaboratively-constructed resources: Wikipedia provides continuously growing multilingual encyclopedic coverage, while Freebase supplied structured entities and relations before its information was partly transferred to Wikidata.
  • Merged resources: Merged resources combine complementary information across resource types; BabelNet uses WordNet-style synsets, whereas ConceptNet models words.

4.2 Knowledge-Enhanced Word Representations

Knowledge-enhanced word representations combine corpus information with lexical-resource knowledge to improve semantic coherence, coverage, specialization, and multilingual construction.

  • Lexical resources add information unavailable in text corpora to improve the semantic coherence or coverage of word vectors.
  • Early methods modified neural language-model objectives to integrate lexical relations while learning word embeddings.
  • Retrofitting improves pre-trained word embeddings by incorporating relations from semantic networks.
  • Extensions apply retrofitting to multilingual ConceptNet embeddings, explicitly model pairwise relations, or analyze semantic-network structure with Personalized Page Rank.
  • Specialized embeddings target applications such as similarity, relatedness, and dialogue state tracking using monolingual and cross-lingual linguistic constraints.
  • External resources support cross-lingual vector spaces without requiring a large parallel corpus.

4.3 Knowledge-Based Sense Representations

Knowledge-based sense representations de-conflate words into individual senses defined by external sense inventories. The survey organizes methods by their use of textual definitions, semantic-network structure, and pre-trained word embeddings.

  • Knowledge-based techniques represent individual senses according to an external sense inventory rather than learning senses solely from text.
  • Textual definitions: Textual-definition methods initialize sense embeddings from pre-trained word vectors or convolutional neural networks, then use them for disambiguation and sense learning.
  • Semantic networks: Semantic-network methods exploit graph structure, using concepts as nodes and relations as edges to support disambiguation and representation learning.
  • Pre-trained embeddings: Post-processing approaches transform pre-trained word embeddings into sense embeddings using semantic-network information or modified Word2vec objectives optimized with EM.
  • Pre-trained embeddings: AutoExtend constrains each word vector to equal the sum of its sense vectors and each synset vector to equal the sum of its lexicalizations.

4.4 Concept and Entity Representations

Concept and entity representations use knowledge bases to embed entities and relations, mainly for knowledge base completion. The survey also covers entity-only graph methods and hybrid representations combining knowledge-base and textual cues.

  • Knowledge-base embeddings: Knowledge-base embedding research constructs entity and relation representations from knowledge graphs, particularly for predicting missing relations in incomplete knowledge bases.
  • Knowledge-base embeddings: TransE models relations as translations, aiming for e1 +⃗r ≈⃗e2 for every knowledge-base triple.
  • Knowledge-base embeddings: Subsequent models modify TransE through improved relational mappings, separate entity and relation spaces, dynamic entity-relation mappings, pre-trained initialization, unified objectives, or new regularization.
  • Entity representations: Entity-only methods such as DeepWalk represent graphs as artificial sentences generated by random walks.
  • The review is not comprehensive because the knowledge-base embedding literature contains many papers, with broader surveys covering the area in greater depth.
  • Hybrid representations: Hybrid models combine knowledge-base and text-corpus cues, often using Wikipedia and resources such as WordNet or BabelNet, and support disambiguation applications.

5. Evaluation

Meaning representations are evaluated intrinsically for vector-space quality and extrinsically within downstream tasks. Sense-based intrinsic evaluation adapts word benchmarks through sense-pair similarity and contextualized comparisons, but contextual meaning discrimination remains difficult.

  • Intrinsic benchmarks assess vector-space quality independently of downstream applications, while extrinsic benchmarks assess effectiveness within practical tasks.
  • Intrinsic evaluation: MaxSim evaluates two words by selecting the most similar pair of their senses.
  • Intrinsic evaluation: AvgSim instead averages pairwise similarities across all possible senses of the two words, commonly using cosine similarity between sense vectors.
  • Contextual evaluation: Contextual datasets such as SCWS evaluate word similarity using the contexts in which target words occur, with context-aware measures including MaxSimC and AvgSimC.
  • Contextual evaluation: Sense representations generally outperform word-based models on SCWS, but simple disambiguation strategies may not be optimal.
  • Contextual evaluation: A binary context-based evaluation found that recent sense and contextualized representations distinguish meanings only slightly better than a simple baseline and lag behind human agreement.
  • Extrinsic evaluation: Extrinsic evaluations include text categorization, sentiment analysis, document similarity, word sense induction, and disambiguation.

6. Applications

Sense representations have been applied across information retrieval, machine translation, classification, disambiguation, knowledge-base tasks, and other NLP applications. Their integration remains less immediate than that of word representations because it generally requires disambiguation or induction, whereas contextualized representations adjust dynamically to context.

  • Sense representations remain less mature in downstream applications because their integration generally requires an additional word sense disambiguation or induction step.
  • Applications include information retrieval, machine translation, text classification, sentiment analysis, document similarity, disambiguation, knowledge-base completion, and visual object discovery.
  • Sense representations: A lack of sense-annotated multilingual corpora has constrained some applications, although efforts have semi-automatically disambiguated large parallel corpora.
  • Sense representations: Unsupervised sense embeddings did not significantly influence several NLP tasks in one study, with increased word-embedding dimensionality producing similar results.
  • Sense representations: Knowledge-based sense integration requires explicit or implicit disambiguation, and a graph-based method improved text classification when the input text was sufficiently large.
  • Contextualized representations: Contextualized representations dynamically adjust word representations to context without explicitly replacing words with discrete sense embeddings.
  • Contextualized representations: Their dynamic integration into neural architectures has supported evaluation across many NLP tasks, with improvements reported over conventional static word embeddings.

7. Analysis

The analysis contrasts unsupervised and knowledge-based sense representations across interpretability, domain adaptability, granularity, and compositionality. Their suitability depends on the setting, with each approach retaining distinct advantages and limitations.

  • Interpretability and domain adaptability: Knowledge-based representations are more interpretable because sense-inventory entries provide definitions, examples, images, relations, and translations.They can also inject prior information from lexical resources into downstream models.
  • Interpretability and domain adaptability: Knowledge-based models may fail to represent novel senses absent from their underlying inventories, whereas knowledge-enhanced approaches only partially alleviate this constraint.Updating sense inventories is not generally straightforward, and even enhanced methods require senses to be present in the input semantic network.
  • Interpretability and domain adaptability: Unsupervised senses are learned from corpora and can reflect domain-specific distinctions, but their interpretations may not match human sense distinctions.They may therefore be difficult to distinguish without additional interpretability methods.
  • Sense granularity: WordNet lists 41 senses for run, but many map to only correr or operar in Spanish, illustrating why fine-grained distinctions may not benefit multilingual tasks.Merging distinctions into supersenses has been beneficial in several downstream applications.
  • Sense granularity: Dynamic unsupervised sense learning can provide a more realistic distribution of senses than fixing the same number of senses for every word.This discussion links sense granularity to how representations are learned, not only to how inventories are designed.
  • Compositionality: Compositional models can select the contextually intended sense of an ambiguous word, and sense representations have been introduced through induction, exemplars, and related methods.Evaluation has often used paraphrase detection, while integration into neural compositional models remained unexplored for question answering and information retrieval.

8. Conclusions

The survey reviews semantically grounded meaning representations and organizes sense models into unsupervised and knowledge-based approaches. It concludes that important questions remain about sense definition, multilinguality, common-sense reasoning, and downstream integration.

  • Contributions: The survey shows that word embeddings capture useful semantic properties but conflate different meanings, motivating representations that distinguish individual senses.It categorizes these models as unsupervised when senses are induced from corpora and knowledge-based when they are defined by external inventories.
  • Open challenges: Although some sense models are effective in practice, common-sense reasoning has not been deeply explored and most evaluations remain English-only.Only a few works address other languages or multilinguality.
  • Open challenges: The best strategy for integrating sense models into downstream applications remains unclear, including whether explicit pre-disambiguation is necessary.Contextualized embeddings suggest dynamically learning senses for each context without an explicit pre-disambiguation step.
  • Open challenges: The definition of sense and the appropriate representation paradigm remain open questions involving discreteness, knowledge-resource links, and context-dependent learning.Different approaches may suit different applications or domains, without a clear general conclusion.
Loading 1805.04032v3…