Source-linked AI summary
Word Embeddings: A Survey
Felipe Almeida, Geraldo Xexéo
TL;DR
The paper addresses the need for a comprehensive account of word-embedding strategies and their underlying assumptions. It surveys prediction-based and count-based approaches, finding that embeddings encode syntactic and semantic regularities and are useful across many NLP tasks.
Problem
There was no comprehensive survey covering modern word-embedding developments, despite embeddings’ usefulness in downstream NLP tasks and their encoded semantic information.
Method
The survey describes recent strategies for constructing fixed-length, dense, distributed word vectors from word co-occurrence statistics under the distributional hypothesis.
Results
Word embeddings encode syntactic and semantic regularities and are useful for tasks including chunking, question answering, parsing, and sentiment analysis.
Takeaways & Limitations
Embedding advances have produced increasingly accurate and faster representations available through widely used toolkits such as Word2Vec, gensim, FastText, and GloVe.
Takeaways & Limitations
The distributional approach assumes that words with similar contexts have the same meaning, and n-gram models use limited context windows.
Abstract
from arXiv · showhide
This work lists and describes the main recent strategies for building fixed-length, dense and distributed representations for words, based on the distributional hypothesis. These representations are now commonly called word embeddings and, in addition to encoding surprisingly good syntactic and semantic information, have been proven useful as extra features in many downstream NLP tasks.
1 Introduction
Word embeddings represent words as dense, distributed, fixed-length vectors built from word co-occurrence statistics under the distributional hypothesis. The survey reviews prediction-based and count-based strategies and their relevance to NLP.
- Word and document vectors support operations such as addition, subtraction, distance measurement, and use in machine-learning algorithms.
- Neural language models project raw word vectors onto an embedding layer to support generalization and reduce the effects of dimensionality.
- Embeddings encode syntactic and semantic relationships and can serve as standalone features across many NLP tasks.
- The distributional hypothesis assumes that words appearing in similar contexts have similar meanings.
- Word embeddings are dense, distributed, fixed-length word vectors built from word-context co-occurrence statistics.
- Prediction-based models use language models to predict the next word, whereas count-based models use global word-context co-occurrence counts.
2 Background: The Vector Space Model and Statistical Language Modelling
The vector space model makes text amenable to mathematical operations, while statistical language models estimate word distributions and next-word likelihoods. These foundations motivate modern embeddings but face dimensionality and generalization challenges.
- The Vector Space Model: Vector representations support mature linear-algebra and statistical methods as well as machine-learning algorithms for NLP.
- The Vector Space Model: The vector space model represents each document as a term-based vector, enabling similarity calculations and search-result scoring.
- Statistical Language Modelling: Statistical language models estimate the distribution of words and calculate the likelihood of a next word given its preceding context.
- Statistical Language Modelling: An n-gram language model uses a limited context window; satisfactory results have been observed with contexts as small as three words.
- Statistical Language Modelling: Next-word prediction is performed by maximum likelihood estimation over all words in the vocabulary.
- Statistical Language Modelling: These models face high dimensionality for vocabularies of about 100,000 words and difficulty generalizing to unseen sequences.
- Statistical Language Modelling: Neural networks and log-linear models have produced better language-model results as measured by perplexity.
3 Word Embeddings
Word embeddings are fixed-length vector representations learned through prediction-based or count-based strategies. Prediction-based models evolved from neural language models through efficiency improvements and methods designed specifically to learn embeddings, while later work showed that the vectors encode syntactic and semantic relationships.
- 3 Word Embeddings: Word embeddings are fixed-length vector representations induced either from local context with prediction-based models or global corpus statistics with count-based models.Prediction-based methods resemble neural language models, whereas count-based methods use corpus-wide word counts and frequencies.
- 3 Word Embeddings: Neural language models introduced embedding layers that project raw word vectors before subsequent network layers, helping address dimensionality and improve generalization.The embedding layer was initially part of a broader language-modeling architecture.
- 3 Word Embeddings: Early neural language-model research focused on efficiency because long training times hindered development despite better language modeling than n-gram models.Approaches included reducing normalization costs, hierarchical softmax, and noise-contrastive estimation; importance sampling reported a 19-fold training-time gain.
- 3 Word Embeddings: Collobert and Weston designed a model specifically to learn embeddings, using full contexts and positive-versus-false examples rather than training a language model.The false examples replaced the actual center word with a random vocabulary word, an approach later called negative sampling.
- 3 Word Embeddings: RNN-derived embeddings were found to encode syntactic and semantic regularities, including male-female and singular-plural relationships represented by vector arithmetic.The relationships are illustrated through a two-dimensional projection of high-dimensional embeddings.
- 3 Word Embeddings: FastText-related work replaced word embeddings with compositional n-gram embeddings to exploit morphological information and generalize to unseen words.The authors reported better results than SGNS, particularly for German, French, and Spanish.
4 Conclusion
Word embeddings support many NLP tasks and are built through both prediction-based and count-based approaches. Widely used toolkits have made them increasingly accurate and faster for practical use.
- Word embeddings have been found useful for Chunking, Question Answering, Parsing, and Sentiment Analysis.
- Prediction-based models estimate the probability of the next word from a sequence, whereas count-based models use global word-context co-occurrence statistics.
- Widely used toolkits include Word2Vec, gensim, FastText, and GloVe.
- These toolkit advances have produced ever more accurate and faster word embeddings ready for NLP tasks.
5 Further Work
Further work includes task-specific tuning, connections between prediction-based and count-based models, and composing word vectors into representations of larger linguistic units.
- Task-specific tuning of embeddings has been associated with improved results for NLP tasks.
- SGNS has been suggested to be equivalent to a slightly modified PMI-weighted word-context matrix, potentially linking the two model families.
- Recent work adapts distributional compositionality methods to compose neural word embeddings into sentence and document representations.
A Log-linear Models and Neural Embeddings
Log-linear models represent conditional probabilities for language-modeling relationships between contexts and target words. With neural embeddings, the formulation uses context inputs, target-word labels, and learned parameters, and can be trained as a neural network.
- Log-linear models are probabilistic devices for modeling conditional probabilities between word contexts and target words.
- The log-linear formulation provides a template for each output unit.
- In neural language modeling, y is the target-word label, x is its context, and v is a learned row vector in the shared weight matrix.
- The formulation can be viewed as a neural network with one linear hidden layer connected to a softmax output layer.
- The model can be trained with gradient-based methods and extended with regularization terms.