Source-linked AI summary
Mimicking Word Embeddings using Subword RNNs
Yuval Pinter, Robert Guthrie, Jacob Eisenstein
TL;DR
Word embeddings fail for OOV words absent from their training data, motivating a method that predicts distributional vectors from spellings. MIMICK learns this mapping at the type level without the original corpus, and improves tagging over word-based baselines across 23 languages while complementing supervised character models in low-resource settings.
Problem
OOV words lack embeddings because unlabeled corpora cannot cover every language term, limiting embedding-based downstream methods.
Method
MIMICK learns a function from word spellings to distributional embeddings using a character-level RNN trained over the existing vocabulary.
Results
MIMICK improves performance over a single UNK-embedding baseline across 23 languages and achieves competitive results against supervised character embeddings, especially in low-resource settings.
Takeaways & Limitations
MIMICK provides a task-agnostic tool for inferring OOV vectors from limited-vocabulary models without accessing the originating corpus.
Takeaways & Limitations
The reported model uses characters as input, while future work may consider morphemes, phonemes, or bitmap representations for ideographic characters.
Abstract
from arXiv · showhide
Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data. However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIMICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings. Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learning is performed at the type level. Intrinsic and extrinsic evaluations demonstrate the power of this simple approach. On 23 languages, MIMICK improves performance over a word-based baseline for tagging part-of-speech and morphosyntactic attributes. It is competitive with (and complementary to) a supervised character-based model in low-resource settings.
1 Introduction
MIMICK addresses OOV embeddings by learning to predict distributional vectors from word spellings with a character-level RNN. Evaluations show improvements over a single-UNK baseline across 23 languages and competitive, complementary performance against supervised character embeddings, especially in low-resource settings.
- OOV words challenge embedding-based methods because unlabeled data cannot cover every language term, especially in low-resource languages.
- MIMICK-RNN learns word embeddings from spellings by training a character-level recurrent neural network with embeddings as targets.The model predicts vectors for OOV words in downstream tasks.
- Nearest-neighbor analysis shows that MIMICK vectors capture both word-shape and lexical features for rare and unseen words.The method produces reasonable neighbors for abbreviations, names, novel compounds, and orthographic errors.
- 23 languages show significant improvement over a single UNK-embedding backoff method for joint part-of-speech and morphosyntactic tagging.
- MIMICK obtains competitive results against supervised character embeddings and is particularly effective and complementary in low-resource settings.The authors position it as a tool for tagging with limited labeled data.
2 Related Work
Prior work composes rare-word embeddings from morphology or character structure, but typically learns the composition model through objectives over large unlabeled corpora. Supervised subword models instead learn task-specific representations from labeled data, while morphosyntactic tagging provides a multilingual evaluation setting.
- Compositional models for embedding rare and unseen words: Morphology-based approaches compose embeddings from morphemes or morphological parses but struggle with names and foreign words containing unseen morphemes.
- Compositional models for embedding rare and unseen words: Character-based approaches avoid unseen-morpheme problems by composing embeddings from character representations or character n-grams.
- Compositional models for embedding rare and unseen words: These compositional models learn how subword units form word embeddings by optimizing objectives over large unlabeled corpora.
- Supervised subword models: Supervised subword models learn character-based embeddings within end-to-end tagging systems, so their coverage depends on labeled data.
- Morphosyntactic attribute tagging: Morphosyntactic tagging evaluates token attributes such as gender, number, case, and tense using multilingual Universal Dependencies annotations.
3 MIMICK Word Embeddings
MIMICK learns to generate embeddings for unseen words from their spellings using a Word Type Character Bi-LSTM, trained against pretrained embeddings. The section evaluates its nearest neighbors and downstream applicability while documenting OOV prevalence and tokenization conditions.
- Model formulation: MIMICK learns a function from word forms to pretrained embeddings, enabling embeddings for unseen words outside the original vocabulary.The model approximates existing word-to-vector assignments and applies the learned function to new words.
- Model formulation: A Word Type Character Bi-LSTM reads a word’s character sequence, and a multilayer perceptron computes the resulting word embedding.The architecture combines forward and backward character-LSTM representations before applying affine transformations and a nonlinear function.
- Training and resources: Training matches predicted embeddings to pretrained vectors by minimizing squared Euclidean distance, with backpropagation updating the character, recurrent, and output parameters.The pretrained embeddings used in the experiments come from Polyglot, whose datasets provide 64-dimensional vectors for the 100,000 most frequent words.
- Evaluation setting: 29.1% or more of word types are OOV in at least half of the 23 evaluated languages.The token-level median OOV rate is 9.2%.
- Evaluation caveat: Tokenization differences between Polyglot and Universal Dependencies may adversely affect OOV counts and resulting model performance.The evaluation retains this real-world mismatch rather than applying subtokenization corrections.
- Intrinsic evaluation: Nearest-neighbor examples show learned word shape, typo robustness, suffix-based part-of-speech patterns, compounding, and some morphosyntactic attributes, but weak semantics.The analysis covers English, Hebrew, Spanish, and Tamil examples; semantic information is limited because training lacks contextual input.
- Intrinsic evaluation: MIMICK recovers about half of the OOV-related performance loss in the Stanford RareWord “All pairs” condition and outperforms VarEmbed.FastText is treated as an upper bound because its vocabulary is 25 times larger and misses only 44 pairs in this dataset.
4 Joint Tagging of Parts-of-Speech and Morphosyntactic Attributes
The tagging model extends a sentence-level bidirectional LSTM with attribute-specific projection layers, using MIMICK vectors to initialize OOV words. Joint training predicts POS tags and morphosyntactic attributes, although POS accuracy declines when the tasks are combined.
- Model: The model duplicates a projection layer for each morphosyntactic attribute, producing attribute-specific probability distributions for every token.Its input is each token’s underlying sentence-level LSTM state.
- Model: Word embeddings are initialized from Polyglot when available and from MIMICK vectors for OOV words.The tagging LSTM therefore receives compositional embeddings only where the pretrained vocabulary lacks a word.
- Training objective: The training objective sums negative log probabilities for POS tags and morphosyntactic attributes across each tagged sequence.A weighted alternative was considered because attribute sparsity differs across languages, but the simpler sum loss was retained.
- Training objective: POS tagging was less accurate when learned jointly with morphosyntactic attributes in all evaluated settings.The authors suggest that attribute loss may act as POS-unrelated noise, but present this as a possible explanation.
5 Experimental Settings
The experiments evaluate MIMICK across 23 sufficiently annotated languages using Universal Dependencies, with POS accuracy and micro F1 for morphosyntactic attributes. Comparisons include UNK, MIMICK, character-augmented, FastText, and other embedding initializations.
- Languages and data: The tagging evaluation uses 23 languages selected from the 44-language intersection between Polyglot embeddings and UD, excluding many under-annotated languages.The selected languages cover varied morphological patterns, while Vietnamese has no morphosyntactic attributes by design.
- Metrics: POS tagging is measured by overall token-level accuracy, while morphosyntactic attribute tagging is evaluated with micro F1.Micro F1 compares each token’s value for each attribute with the gold assignment.
- Compared models: The No-Char baseline uses Polyglot embeddings, lowercased backoff when available, and the Polyglot UNK vector otherwise.MIMICK instead supplies inferred embeddings for OOV words without lowercased backoff.
- Compared models: CHAR→TAG appends a character-level LSTM output to Polyglot-based word embeddings and increases training time threefold.The Both model combines MIMICK initialization with this character-level LSTM.
- Compared models: FastText provides 300-dimensional embeddings for 2.51 million lowercase-only forms and has no UNK vector.Other examined non-Polyglot embedding models performed substantially worse than Polyglot.
- Optimization: All models use two LSTM layers with hidden size 128, 40 training epochs, and dropout of 0.5, with 80 epochs for 5K settings.CHAR→TAG models use 20-dimensional character embeddings and one hidden layer of size 128.
6 Results
Across languages and training regimes, MIMICK improves OOV handling for POS and morphosyntactic tagging, with especially strong gains in several language families and low-resource settings. It also outperforms or complements character-based alternatives in many low-data conditions.
- The experiments report full-resource results and 5,000-token low-resource results, with additional training-set sizes shown in Figure 2.
- MIMICK embeddings significantly improve over Polyglot UNK embeddings for OOV tokens in nearly all settings across both tagging tasks, languages, and corpus sizes.
- Gains are especially apparent for Slavic languages and Chinese, while morphosyntactic tagging also benefits in agglutinative languages such as Tamil and Turkish.The passage attributes these patterns to rich word representations, large character inventories, and stable morpheme representations.
- These improvements are particularly substantial for test-set tokens outside the UD training set, where limited labeled data constrains CHAR→TAG’s learned orthographic mapping.
7 Conclusion
MIMICK infers OOV embeddings from pre-trained, limited-vocabulary models without accessing their originating corpora, and improves word-based sequence tagging across diverse languages. The approach is particularly useful in low-resource settings, while the paper identifies alternative subword units as future work.
- MIMICK infers OOV word-embedding vectors from pre-trained, limited-vocabulary models without needing access to the originating corpus.
- Table 5 reports POS tagging accuracy on the UD 1.4 test set, while Table 6 reports morphosyntactic attribute micro-F1; statistical comparisons use McNemar’s test and a bootstrapped Z-test, respectively.
- Figure 2 compares agglutinative and Slavic languages across training-token counts starting at 500, with error bars showing standard deviations over five subsamples.
- The method improves annotated sequence-tagging performance across languages differing in family, orthography, and morphology.
- The character-input MIMICK model could be extended to morphemes, phonemes, or bitmap representations of ideographic characters.