Source-linked AI summary
Enriching Word Vectors with Subword Information
Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov
TL;DR
Word-vector models overlook word-internal structure, making rare and unseen forms difficult to represent, especially in morphologically rich languages. This paper extends skipgram with character n-gram representations and shows that the resulting vectors outperform non-subword and morphology-based baselines across evaluations.
Problem
Word-vector models ignore internal word structure, limiting representations for rare or unseen forms in morphologically rich languages.
Method
The model extends skipgram by representing each word as the sum of vectors for its character n-grams and the word itself.
Results
The proposed subword model outperforms baselines and morphology-based methods across the reported evaluations, with gains especially evident for morphologically rich languages.
Takeaways & Limitations
Character n-gram representations provide useful word vectors for rare and out-of-vocabulary words, including when training data are limited.
Abstract
from arXiv · showhide
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character $n$-grams. A vector representation is associated to each character $n$-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.
1 Introduction
The introduction frames word representations as efficient distributional and neural methods that typically assign one vector per word, overlooking internal morphology. The paper addresses this limitation by representing words through character n-grams and evaluating a subword-aware skipgram model across nine languages.
- Background: Word representations are commonly learned from large unlabeled corpora using co-occurrence statistics or neural prediction models.The introduction traces this tradition from distributional semantics to feedforward and log-bilinear neural models.
- Problem: Most existing techniques assign each vocabulary word a distinct vector without parameter sharing or internal morphological information.This limits their ability to capture relationships among related word forms.
- Problem: Morphologically rich languages such as Turkish and Finnish contain many rare or unseen word forms, making word-level representations especially limiting.French and Spanish verbs often have more than forty inflected forms, while Finnish nouns have fifteen cases.
- Contribution: The paper represents words as sums of character n-gram vectors and extends continuous skipgram to incorporate subword information.The approach is evaluated on nine languages with different morphological properties.
2 Related work
Related work incorporates morphology into word representations and learns language representations directly from characters. Character-level approaches include recurrent, convolutional, and restricted-Boltzmann-machine models applied across multiple NLP tasks.
- Morphological word representations: Morphological word representations use feature-based methods to improve modeling of rare words, including applications to morphologically rich languages such as Turkish.Alexandrescu and Kirchhoff introduced factored neural language models representing words as sets of features, which can include morphological information.
- Character level features for NLP: Character-level models discard word segmentation and learn language representations directly from characters.These models have been applied to language modeling, text normalization, part-of-speech tagging, and parsing.
- Character level features for NLP: Convolutional neural networks trained on characters have supported part-of-speech tagging, sentiment analysis, text classification, and language modeling.These applications are reported by dos Santos and Zadrozny, dos Santos and Gatti, Zhang et al., and Kim et al.
- Character level features for NLP: Sperr et al. introduced a restricted Boltzmann machine language model in which words are encoded as a set of character n-grams.This provides another character-based modeling family alongside recurrent and convolutional neural networks.
3 Model
The model extends skipgram with negative sampling by representing each word as the sum of vectors for its character n-grams, while retaining a separate word representation. Boundary symbols, n-grams of lengths 3–6, and hashing define the subword dictionary and enable parameter sharing across words.
- Model framework: The model replaces distinct word-only scoring with representations that incorporate internal word structure through subword units.It is derived from the skipgram framework with negative sampling, which predicts observed context words against randomly sampled negatives.
- Subword representation: Each word is represented as a bag of character n-grams, including boundary symbols < and > and the word itself.Boundary symbols distinguish prefixes and suffixes from other character sequences, while including the whole word provides an additional word-specific representation.
- Subword representation: The model extracts all character n-grams with lengths from 3 through 6, distinguishing boundary-marked sequences from identical internal sequences.For example, <her> differs from the trigram her because the boundary symbols encode its position within a word.
- Subword representation: A word vector is computed as the sum of the vectors associated with the n-grams appearing in that word.If G_w is the set of n-grams in word w and z_g is the vector for n-gram g, the word representation aggregates the corresponding z_g vectors.
- Parameter sharing: Sharing n-gram representations across words enables the model to learn reliable representations for rare words.The shared subword parameters allow information from related character sequences to contribute across different word representations.
- Subword dictionary: The model hashes character n-grams with FNV-1a into K = 2.106 buckets to bound memory requirements.Each word is ultimately identified by its dictionary index together with the set of hashed n-grams it contains.
4 Experimental setup
Experiments train the proposed model and baselines on multilingual, preprocessed Wikipedia data using shared optimization and evaluation settings. The character n-gram model is slower than the skipgram baseline but remains implemented in publicly available C++.
- Optimization: Optimization uses stochastic gradient descent on the negative log likelihood with linearly decaying step sizes and parallel Hogwild updates.The step size is γ0(1 − t/TP), where T is the training-set size, P is the number of passes, and γ0 is fixed.
- Configuration: Both models and baselines use 300-dimensional vectors, five negative samples, context sizes uniformly sampled from 1 to 5, and a 10^-4 rejection threshold.Negative-sampling probability is proportional to the square root of unigram frequency.
- Implementation: 1.5× slower training: the character n-gram model processes 105k words/second/thread versus 145k for the skipgram baseline.In most experiments, comparisons use the C implementation of skipgram and cbow from the word2vec2 package; the model is implemented in C++ and publicly available.
- Data: Models are trained on Wikipedia dumps in nine languages, except for comparisons to previous work in Sec. 5.3.The languages are Arabic, Czech, German, English, Spanish, French, Italian, Romanian, and Russian.
- Data: Raw Wikipedia data are normalized, shuffled, and processed for five passes over each dataset.Normalization uses Matt Mahoney’s pre-processing perl script.
5 Results
Across similarity, analogy, comparison, data-size, and n-gram experiments, the subword skipgram model generally improves word representations, especially for rare words and morphologically rich languages. Its benefits depend on task, language, and character n-gram choices, with semantic analogy performance showing important limitations.
- Word similarity: Subword representations outperform baselines on all word-similarity datasets except English WS353, while unseen-word vectors are at least as good as null-vector representations.The model computes out-of-vocabulary vectors by summing character n-gram vectors; the null-vector variant is called sisg-, and the unseen-word variant sisg.
- Word similarity: Character n-grams help more in Arabic, German, and Russian than in English, French, and Spanish, partly because they capture declensions and German compounds.The passage gives German “Tischtennis” and “Tennis” as an example of character-level similarity helping represent compounds.
- Word analogies: Morphological information improves syntactic analogies and helps most in morphologically rich languages, but does not improve semantic analogies and degrades German and Italian performance.The effect is tied to character n-gram length; choosing n-gram sizes optimally reduces the semantic-analogy degradation.
- Character n-gram size: Character n-gram lengths of 3–6 provide satisfactory performance across English and German, while longer n-grams are important and n ≥3 improves analogy results over n ≥2.Columns corresponding to n ≤5 and n ≤6 work best, particularly for German compounds; larger n-grams also help semantic analogies.
6 Qualitative analysis
The proposed subword representations produce better nearest neighbors for complex, technical, and infrequent words than skipgram, while important character n-grams often correspond to meaningful morphemes. For out-of-vocabulary words, n-gram similarities can recover meaningful subword correspondences.
- Nearest-neighbor analysis: Nearest neighbors for complex, technical, and infrequent words are better with the proposed approach than with the skipgram baseline.The comparison uses cosine-similarity nearest neighbors for selected words in Table 7.
- Morpheme analysis: The analysis ranks n-grams by the cosine change caused by omitting each n-gram from the summed word representation.For each word, the n-grams whose removal produces the most different representation are shown in Table 6.
- Morpheme analysis: German compound nouns often separate into valid morphemes, as Autofahrer yields Auto and Fahrer among its most important n-grams.English compounds such as lifetime and starfish also show this separation.
- Morpheme analysis: English important n-grams can represent affixes, while French examples capture verb inflections such as ais>, ent>, and ions>.The examples include kindness and unlucky in English.
- OOV analysis: For OOV words, n-gram matches recover meaningful correspondences: micro and circuit match chip, scarce matches rarity, -ness matches -ity, and -adolesc- matches young.These patterns are observed by comparing n-gram cosine similarities in selected English RW word pairs.
7 Conclusion
The paper concludes that incorporating character n-grams into skipgram provides a simple way to learn word representations using subword information. The model trains quickly without preprocessing or supervision and outperforms baselines that ignore subword information.
- 7 Conclusion: Character n-grams are incorporated into the skipgram model to learn word representations from subword information.The approach is related to an idea introduced by Schütze (1993).
- 7 Conclusion: The model trains fast and requires neither preprocessing nor supervision.Its simplicity enables these practical advantages.
- 7 Conclusion: The model outperforms baselines that do not take subword information into account.The conclusion explicitly contrasts the proposed approach with baselines lacking subword information.