Source-linked AI summary
Word Emdeddings through Hellinger PCA
Rémi Lebret, Ronan Collobert
TL;DR
Neural language models produce useful embeddings but can be difficult and time-consuming to train. This paper replaces their embedding computation with Hellinger PCA over word co-occurrence statistics, while retaining neural networks for task-specific adaptation. The resulting embeddings achieve similar or better performance on NER and movie review tasks.
Problem
Neural language model embeddings are successful but may be difficult and time-consuming to train, motivating simpler embedding computation.
Method
The paper computes embeddings through Hellinger PCA of a word co-occurrence matrix and uses neural architectures to fine-tune them for specific NLP tasks.
Results
H-PCA embeddings perform similarly to other embeddings on NER and movie review tasks, with tuned H-PCA embeddings obtaining the best NER F1 scores.
Takeaways & Limitations
Hellinger PCA offers a practical alternative to neural language models for generating embeddings, while deep learning provides task-specific fine-tuning.
Abstract
from arXiv · showhide
Word embeddings resulting from neural language models have been shown to be successful for a large variety of NLP tasks. However, such architecture might be difficult to train and time-consuming. Instead, we propose to drastically simplify the word embeddings computation through a Hellinger PCA of the word co-occurence matrix. We compare those new word embeddings with some well-known embeddings on NER and movie review tasks and show that we can reach similar or even better performance. Although deep learning is not really necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to specific tasks.
1 Introduction
The paper proposes obtaining effective word embeddings with mostly linear operations on word co-occurrence statistics, rather than relying solely on difficult, time-consuming neural architectures. It also examines neural adaptation of generic embeddings for task-specific performance.
- The method computes word embeddings from co-occurrence statistics using PCA and compares them with neural and spectral baselines across NLP tasks.Compared embeddings include CW, Turian, HLBL, and LR-MVL.
- Assuming an appropriate metric, PCA can produce embeddings as good as those from deep-learning architectures.
- Generic embeddings tend to support syntactic tasks such as POS tagging, chunking, and NER.
- Neural architectures can adapt existing embeddings for semantic tasks such as sentiment classification, increasing general performance.
2 Related Work
Prior work developed distributed word representations to capture structural relationships and meaning from context, while highlighting the need to adapt generic embeddings for semantic tasks.
- Word order is important for capturing semantic information, motivating distributed representations that encode structural relationships between words.
- Neural language models learn word vectors jointly with statistical language models and inspired later neural embedding architectures.
- Co-occurrence statistics provide a natural basis for embedding words with similar contexts into a common vector space.
- Generic embeddings improve performance on many NLP tasks but may require sentiment-specific information for sentiment classification.
3 Spectral Method for Word Embeddings
The spectral method converts word co-occurrence counts into probability distributions and applies Hellinger-based PCA to obtain lower-dimensional word representations.
- 3.1 Word co-occurrence statistics: A neural language model learns next-word probabilities, whereas corpus counting can directly retrieve those distributions from unlabeled text.
- 3.1 Word co-occurrence statistics: Raw co-occurrence frequencies count how often each context word follows a sequence T.
- 3.1 Word co-occurrence statistics: The resulting probability distributions form an N × |D| co-occurrence matrix over all possible word sequences.
- 3.2 Hellinger distance: Hellinger distance compares the distributions associated with two words and is chosen for simplicity, symmetry, and being a true distance.
- 3.2 Hellinger distance: The Hellinger distance is directly related to the Euclidean norm between the distributions’ square-root vectors.
- 3.3 Hellinger PCA: PCA reduces vocabulary-dependent probability distributions to a lower-dimensional space while minimizing reconstruction error under the Hellinger distance.
4 Architectures for NLP tasks
The paper uses neural architectures for sentence and document NLP tasks, with lookup-table embeddings, contextual windows, structured tag inference, document filters, and end-to-end training or fine-tuning.
- 4 Architectures for NLP tasks: A multilayer neural network supports end-to-end task training while reducing reliance on hand-designed features.
- 4.1 Sentence-level Approach: The sentence-level model feeds word embeddings through classifiers and CRF-type inference to tag each word in a sentence.
- 4.1 Sentence-level Approach: Sliding windows of N = (2n + 1) words produce context-dependent representations for the middle word.
- 4.1 Sentence-level Approach: The sentence model represents tag dependencies with transition scores and infers the best tag path using the Viterbi algorithm.
- 4.2 Document-level Approach: The document-level classifier applies trained filters over sliding windows and sends each filter’s maximum document-wide response to a linear classifier.
- 4.2 Document-level Approach: In movie reviews, filter signs can indicate positive or negative sentiment.
- 4.3 Embedding Fine-Tuning: A lookup-table layer initializes existing embeddings, which are then trained by backpropagation for the target task.
5 Experimental Setup
The experiments compare H-PCA embeddings with established neural and spectral embeddings on NER and movie-review tasks, while examining computational cost, normalization, and fine-tuning. H-PCA performs competitively across both tasks, with fine-tuning improving performance and Hellinger-based embeddings outperforming E-PCA.
- Experimental design: The evaluation compares H-PCA with CW, Turian, HLBL, and LR-MVL embeddings on NER and movie-review tasks.The study also evaluates E-PCA and fine-tuning effects.
- Experimental design: 1,652 million words comprise the English corpus assembled from Wikipedia, Reuters, and WSJ.Text was lowercased, number sequences were replaced with “NUMBER,” and the data were tokenized with the Stanford tokenizer.
- Evaluation tasks: NER uses CoNLL 2003 with word embeddings and a capital-letter feature, while the movie-review experiment uses 50,000 balanced IMDB reviews.The NER benchmark reports F1, and the movie-review benchmark reports classification accuracy.
- Embeddings normalization: Normalization depends on the network architecture: λ = 0.1 works best for the document-level approach, whereas λ = 1 is better for H-PCA in the sentence-level approach.The embeddings are normalized to avoid saturation issues in the network architectures.
- Results: H-PCA matches neural-language-model embeddings on NER and performs as well as the other embeddings on movie reviews.Fine-tuned H-PCA obtains the best NER F1 scores, while fixed CW embeddings slightly outperform fixed H-PCA on NER; the fixed movie-review difference is about 3%.
- Results: Fine-tuning embeddings by backpropagation increases general performance on both NER and movie-review tasks.The increase is generally higher for movie reviews, where fine-tuning is associated with a high semantic component.
6 Conclusion
The paper concludes that Hellinger PCA of a word co-occurrence matrix offers a fast, practical alternative to neural language models, while deep learning remains useful for task-specific adaptation.
- Hellinger PCA produces appealing word embeddings from a word co-occurrence matrix.
- The method achieves similar results on NLP tasks using a N × 10,000 co-occurrence matrix computed with only one word of context.
- PCA of the N × 10,000 matrix is fast and not memory consuming, making the method a practical alternative to neural language models.
- Deep learning can finetune embeddings over specific NLP tasks.