Source-linked AI summary
Learning to Compute Word Embeddings On the Fly
Dzmitry Bahdanau, Tom Bosc, Stanisław Jastrzębski, Edward Grefenstette, Pascal Vincent, Yoshua Bengio
TL;DR
Rare words are difficult to represent because they occur too infrequently for task-trained embeddings to learn well, while UNK tokens discard their specific meanings. The paper predicts their embeddings from auxiliary data using a network trained end-to-end for the downstream task, and reports improvements across its evaluated tasks.
Problem
Rare words occur too infrequently for standard end-task training to learn informative representations, while OOV words may instead share a single UNK embedding.
Source: §1 · p. 1Method
The method predicts rare-word embeddings from auxiliary semantic data, such as dictionary definitions or spelling, using encoders trained jointly with the downstream task.
Source: §1 · p. 2Results
Auxiliary data improves performance on reading comprehension, textual entailment, and language modeling, with reported gains over spelling-only models on question answering and semantic entailment.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
The approach can use relatively small amounts of auxiliary data to narrow the gap between task-only embeddings and data-rich pretrained embeddings.
Source: §1 · p. 2, §6 · p. 10Dictionary coverage can be missing, definitions may rely on words outside the training vocabulary, and some meanings require nontrivial commonsense inference.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.
1 INTRODUCTION
Rare words are difficult to represent because they occur too infrequently for standard supervised methods to learn informative embeddings, while treating them as OOV discards word-specific meaning. The paper proposes predicting rare-word embeddings from auxiliary semantic data and evaluates this approach on three established tasks.
1 INTRODUCTION
- Standard supervised methods need many occurrences to learn useful rare-word representations, while UNK tokens erase distinctions among words outside a restricted vocabulary.Large vocabularies are parametrically expensive, yet rare words can still be too infrequent to learn well.Source: §1 · p. 1
- The method predicts word representations from auxiliary data describing semantics, with encoders trained jointly for the downstream task.The paper focuses on dictionary data, spelling, or both, and can combine multiple auxiliary sources.Source: §1 · p. 2
- Experiments compare task-trained embeddings with on-the-fly embeddings and fixed pretrained GloVe vectors across reading comprehension, textual entailment, and language modeling.The paper reports improved performance from auxiliary data and qualitative evidence of exploiting synonymy and semantic relatedness.Source: §1 · p. 2
2 RELATED WORK
Prior approaches represent rare words through pretrained text embeddings, spelling, or external knowledge. This paper distinguishes its approach through task-specific end-to-end learning from dictionary definitions and other textual auxiliary data.
2 RELATED WORK
- Pretrained embeddings from large raw-text corpora offer an alternative to task-only training, but require substantial data and may not align with the task.The paper notes that pretrained embeddings may be difficult to obtain in technical domains and are not guaranteed to help the target task.Source: §2 · p. 2
- Character-based OOV methods can help with morphology, but spelling alone may not reveal semantics, especially for named entities.The paper presents spelling as one auxiliary source and generalizes the spelling-based approach of Ling et al. (2015) to other definition types.Source: §2 · pp. 2–3
- Unlike Hill et al. (2016), who train definition embeddings for reverse dictionary and crossword tasks, this paper trains a dictionary reader end-to-end for a specific task.This avoids the potentially suboptimal auxiliary ranking cost and reliance on high-quality pretrained embeddings described for Hill et al. (2016).Source: §2 · p. 3
- The approach uses textual knowledge and task-directed end-to-end learning, rather than restricting auxiliary information to graph-structured knowledge bases.The paper contrasts its approach with knowledge-base embedding work (Faruqui et al., 2015).Source: §2 · p. 3
3 ON THE FLY EMBEDDINGS
The method replaces a single UNK representation for unseen words with embeddings computed from their dictionary definitions. Definition readers can use several encoders, and their outputs can be combined with ordinary word embeddings.
3 ON THE FLY EMBEDDINGS
- For an unseen test-time word, the standard restricted-vocabulary setup uses the shared UNK embedding, losing its specific meaning.Even an in-vocabulary word seen rarely may have an uninformative learned embedding.Source: §3 · p. 3
- A definition reader f maps the token embeddings of a word’s dictionary definition to a definition embedding for that word.The paper considers mean pooling, mean pooling followed by a learned linear transformation, and the final state of an LSTM.Source: §3 · p. 4
- When a word has multiple definitions, their embeddings are mean-pooled; definitions are included for matching headwords or possible lemmas of lower-cased words.The paper simplifies notation by assuming one definition per word thereafter.Source: §3 · p. 4
- Definition embeddings can be used alone for OOV words or combined with ordinary embeddings, and definitions for input words can be processed in parallel.The combination may use a learned transformation or direct summation; when no definition is available, its embedding is a zero vector.Source: §3 · p. 4
- This first version excludes definitions for word combinations and does not recursively use definition embeddings for unknown words inside definitions.Both extensions are identified as future work.Source: §3 · p. 4
4 EXPERIMENTS
The experiments test dictionary definitions and spelling as auxiliary data for question answering, entailment, and language modeling, comparing task-trained models with pretrained GloVe. Auxiliary information improves results over scratch baselines, while GloVe remains ahead in reported comparisons.
4.1 QUESTION ANSWERING
- On SQuAD, spelling helps more than dictionary definitions, but combining them adds 1.1 points over spelling alone and leaves GloVe 1.1 points ahead.The authors suggest spelling's larger gain may reflect the dictionary's lower coverage.Source: §4.1 · p. 5
- On SQuAD test data, the dictionary-and-spelling model scores 1 point above spelling alone; lemmatized, lowercased spelling does not outperform spelling.Only S, SL, and SD were evaluated on the test set because evaluation was performed manually by the SQuAD authors.Source: §4.1 · p. 5
4 EXPERIMENTS
- Auxiliary information improves results over scratch baselines across SQuAD, SNLI/MultiNLI, and OBW; dictionary definitions bridge about 40% of the pretrained-embedding gap on both entailment datasets.On SQuAD, adding external information improves exact match by 3.7–10.5 points; the entailment comparison reports a consistent gain on both datasets.Source: §4.1 · p. 5, Table 1 · p. 6, §4.2 · p. 7
4.1 QUESTION ANSWERING
- Definitions help match rare terms such as “scientist” to named entities, but missing definitions, rare definition words, and difficult inferences constrain their use.For example, “projectile” and “planet” fall outside the training vocabulary, while inferring that a historian is a profession requires common-sense reasoning.Source: §4.1 · p. 6
4.2 ENTAILMENT PREDICTION
- Dictionary-enabled models significantly outperform the baseline on SNLI sentences containing rare words; the figure also shows fruit embeddings separated from tool and vehicle embeddings.Dictionary definitions bridge about 40% of the gap between training from scratch and using pretrained embeddings on both SNLI and MultiNLI.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
4.3 LANGUAGE MODELLING
- On OBW, external information helps in all reported settings, dictionary-plus-spelling is slightly better than spelling alone, and GloVe has the best perplexity.Switching from 1% to the full training set shrinks all gaps; lemma-plus-lowercase performs worse than dictionary-based models.Source: Table 3 · p. 9, §4.3 · p. 9
- For words following OOV words in the restricted 1% OBW setting, adding definitions to spelling bridges half of the 6-point PPL gap between spelling and GloVe.The relative model ranking mostly stays the same on this rare-word slice, while performance differences become larger.Source: §4.3 · p. 9
5 DISCUSSION
Definition-based embeddings improve over spelling-only embeddings on question answering and semantic entailment, while qualitative QA examples support how definitions help. Rare words within auxiliary definitions remain an unresolved challenge.
5 DISCUSSION
- Spelling and dictionary definitions can both produce useful on-the-fly embeddings, but spelling may not reveal meaning directly, as entailment experiments confirm.The approach is presented as a way to learn end-to-end from other auxiliary sources as well.Source: §5 · p. 9
- Dictionary definitions improve over spelling-only embeddings on question answering and semantic entailment, and qualitative QA examples support the proposed explanation for that gain.The discussion also suggests adding more auxiliary data and phrase-level definitions, such as definitions for “Mark Twain.”Source: §5 · p. 10
- The authors suggest that adding more auxiliary data and definitions for phrases could further help the approach.They mention first sentences from Wikipedia and WordNet definitions of phrasal verbs as possible additions.Source: §5 · p. 10
- Rare words inside auxiliary data remain unaddressed; recursively computing their embeddings could be costly and dictionary definitions can contain cycles.The authors identify asynchronous training as a direction for future work.Source: §5 · p. 10
6 CONCLUSION
On-the-fly embeddings from limited auxiliary data are presented as an alternative between task-only learning and pretraining with a large corpus. The method may also let engineers or users adjust systems by editing auxiliary information.
6 CONCLUSION
- On-the-fly embeddings from limited auxiliary data offer an alternative between task-only learning and pretraining when large representative corpora are unavailable.The authors also suggest that users or engineers may adjust system behavior by editing auxiliary information, and that domain adaptation could use domain-specific definitions.Source: §6 · p. 10