Source-linked AI summary
sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings
Andrew Trask, Phil Michalak, John Liu
TL;DR
Most word embeddings assign one vector to each word even when it has multiple senses, while prior multi-sense methods can be expensive and are not always designed for downstream NLP models. The paper uses supervised labels to produce sense-specific embeddings that a consuming model can select efficiently, and reports improved dependency parsing across six languages. Its approach is positioned against prior methods’ repeated-training costs and WordNet coverage limits.
Problem
Most word embeddings encode multiple meanings in one vector, while existing multi-sense approaches can require expensive clustering and repeated neural-embedding training.
Method
The model uses supervised NLP labels to disambiguate word instances and train embeddings that predict word senses from surrounding senses.
Results
Disambiguating Part-of-Speech with sense2vec reduced dependency-parsing error in all six languages, with an average reduction greater than 8%.
Takeaways & Limitations
Sense-disambiguated embeddings provide a mechanism for downstream NLP models to select appropriate word senses and improve syntactic dependency parsing.
Takeaways & Limitations
Related clustering-based alternatives remain constrained by the vocabulary of the English WordNet resource and its limited cross-Part-of-Speech links.
Abstract
from arXiv · showhide
Neural word representations have proven useful in Natural Language Processing (NLP) tasks due to their ability to efficiently model complex semantic and syntactic word relationships. However, most techniques model only one representation per word, despite the fact that a single word can have multiple meanings or "senses". Some techniques model words by using multiple vectors that are clustered based on context. However, recent neural approaches rarely focus on the application to a consuming NLP algorithm. Furthermore, the training process of recent word-sense models is expensive relative to single-sense embedding processes. This paper presents a novel approach which addresses these concerns by modeling multiple embeddings for each word based on supervised disambiguation, which provides a fast and accurate way for a consuming NLP model to select a sense-disambiguated embedding. We demonstrate that these embeddings can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced senses such as sarcasm. We further evaluate Part-of-Speech disambiguated embeddings on neural dependency parsing, yielding a greater than 8% average error reduction in unlabeled attachment scores across 6 languages.
1 INTRODUCTION
Neural word representations compress language into vectors that capture semantic and syntactic information, but single-vector embeddings mix a word’s multiple senses. This superposition can obscure context-specific meaning and impair NLP classifiers.
- Neural representations: Neural language models embed vocabularies into lower-dimensional spaces that model word-sequence probabilities.These representations form vector-space models encoding semantic and syntactic information.
- Applications: Distributed word representations support tasks including Part-of-Speech tagging, Named Entity Recognition, similarity querying, transliteration, and dependency parsing.
- The sense problem: Single-vector embeddings force each word to encode all potential meanings, producing a mixture of senses in vector space.
- The sense problem: Sense superposition obscures context-specific meaning and can negatively affect NLP classifiers that use the embeddings as input.Separating word senses into distinct embeddings alleviates the resulting confusion for an NLP model.
2 RELATED WORK
Prior work models multiple word senses through context clustering and prototype embeddings, but these approaches incur selection, clustering, repeated-training, or resource-coverage costs. The related work motivates a supervised alternative that can reduce computational overhead and better support downstream NLP use.
- Single-vector embeddings: word2vec learns continuous word embeddings that encode semantic and syntactic meaning but ignore word order.
- Single-vector embeddings: wang2vec incorporates word order for syntactic representations, but its single vector cannot model polysemic words with multiple meanings.The method improves syntactic modeling at the cost of computation speed.
- Context-clustering approaches: Context-clustering approaches create multiple prototype vectors, but selecting senses and clustering contexts add computational overhead.
- Context-clustering approaches: Some clustering-based models require training neural embeddings multiple times, while WordNet-based models are limited by English WordNet vocabulary and sparse cross-Part-of-Speech links.
3 THE SENSE2VEC MODEL
sense2vec uses supervised NLP labels to assign multiple sense embeddings, replacing unsupervised clustering with an efficient mechanism for downstream models to select the appropriate sense. Experiments distinguish Part-of-Speech and sentiment-related senses, including nominal, verbal, and sarcastic uses.
- 3 THE SENSE2VEC MODEL: Supervised NLP labels determine each word instance’s sense, replacing unsupervised clustering and enabling a classifier to select the appropriate embedding.The approach avoids repeated embedding training and a separate clustering step.
- 3 THE SENSE2VEC MODEL: sense2vec counts labeled word uses, initializes sense embeddings, and trains CBOW, Skip-gram, or Structured Skip-gram to predict senses from surrounding senses.
- 3.1 SUBJECTIVE EVALUATION - SUBJECTIVE BASELINE: Single-sense embeddings mix the meanings of polysemous words such as bank, apple, so, bad, and perfect, sometimes omitting an interpretation entirely.The baseline bank vector combines proper-noun, verb, and noun uses.
- 3.2 SUBJECTIVE EVALUATION - PART-OF-SPEECH DISAMBIGUATION: Part-of-Speech labeling separates apple’s fruit and company uses, distinguishes three bank uses, and disambiguates nuanced uses of so.
- 3.3 SUBJECTIVE EVALUATION - SENTIMENT DISAMBIGUATION: Sentiment labeling separates conventional and sarcastic interpretations, with positive bad associated with sarcasm and positive perfect clustering with a sarcastic interpretation.
4 NAMED ENTITY RESOLUTION
sense2vec disambiguates named entities by assigning separate embeddings to entity senses, including both single-word and multi-word sequences.
- sense2vec distinguishes “Washington” as either a PERSON or a GPE.
- The method disambiguates both multi-word sequences and single-word sequences of text.
- Hillary Clinton is associated with titles she held during the dataset’s time span.
5 NEURAL DEPENDENCY PARSING
The paper evaluates sense2vec embeddings for neural dependency parsing against wang2vec embeddings, using POS-specific sense embeddings as parser inputs. POS disambiguation reduced parsing error in all six languages, with an average reduction greater than 8%.
- The evaluation compares sense2vec and wang2vec embeddings on neural syntactic dependency parsing tasks.
- The baseline uses structured skip-gram embeddings without Part-of-Speech disambiguation, while sense2vec disambiguates terms by Part of Speech.
- The parser uses POS-specific sense2vec embeddings as inputs and determines the Part-of-Speech label from gold-standard treebank tags.
- Greater than 8% average error reduction was achieved across all six languages with Part-of-Speech disambiguation.
6 CONCLUSION AND FUTURE WORK
The paper proposes supervised labeling to disambiguate word senses, reducing the computational overhead of word-sense modeling and giving NLP tasks a mechanism to select appropriate embeddings. It also reports improved syntactic dependency parsing across multiple languages and identifies broader supervised-label applications as future work.
- The proposed model uses supervised NLP labels to disambiguate between word senses.
- Supervised context labeling reduces the computational overhead of word-sense modeling.
- Disambiguated embeddings provide a mechanism for consuming NLP tasks to select the appropriate sense embedding.
- The embeddings increase syntactic dependency parsing accuracy across a variety of languages.
- Future work will examine other supervised labels and consuming NLP tasks.