Source-linked AI summary
Definition Modeling: Learning to define word embeddings in natural language
Thanapon Noraset, Chen Liang, Larry Birnbaum, Doug Downey
TL;DR
Word similarity and analogy evaluate embedding semantics indirectly, motivating a more direct test through natural-language definitions. The paper models definitions from words and their embeddings, and finds that gated target-word influence improves performance while character-level information adds further gains.
Problem
Similarity and analogy tasks evaluate lexical information in embeddings indirectly, leaving open whether embeddings can generate direct, transparent dictionary definitions.
Source: §1 · p. 1Method
The paper trains recurrent definition models on word-definition pairs, controlling target-word influence and testing complementary character-level features.
Source: §1 · pp. 1–2Results
The gated update improves definition-modeling perplexity by ∼10%, the character-level CNN adds ∼5%, and reverse-dictionary accuracy gains 11.8% absolute over Hill et al. (2016).
Source: §1 · p. 2Takeaways & Limitations
Definition-model errors may provide insight into shortcomings in the information encoded by word embeddings.
Source: §1 · p. 2, §7 · p. 7Whether definition models can improve word embeddings or standard language models remains future work.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Distributed representations of words have been shown to capture lexical semantics, as demonstrated by their effectiveness in word similarity and analogical relation tasks. But, these tasks only evaluate lexical semantics indirectly. In this paper, we study whether it is possible to utilize distributed representations to generate dictionary definitions of words, as a more direct and transparent representation of the embeddings' semantics. We introduce definition modeling, the task of generating a definition for a given word and its embedding. We present several definition model architectures based on recurrent neural networks, and experiment with the models over multiple data sets. Our results show that a model that controls dependencies between the word being defined and the definition words performs significantly better, and that a character-level convolution layer designed to leverage morphology can complement word-level embeddings. Finally, an error analysis suggests that the errors made by a definition model may provide insight into the shortcomings of word embeddings.
1 Introduction
The paper asks whether embeddings can generate natural-language definitions, offering a more direct view of their lexical semantics than similarity and analogy tasks. It introduces definition modeling and recurrent models that control target-word influence and incorporate character-level information.
1 Introduction
- Definition modeling estimates the probability of a textual definition from the word and its embedding, and tests words unseen during training.Dictionary definitions provide a more transparent view of embedding semantics than indirect similarity and analogy evaluations.Source: §1 · p. 1
- A gated update controls how much information about the target word contributes to generating each definition word, while a character-level CNN adds complementary morphological information.The gate reflects the observation that only some definition words depend strongly on the word being defined.Source: §1 · pp. 1–2
- The gated model improves definition-modeling perplexity by ∼10%, the character-level CNN further improves it by ∼5%, and reverse-dictionary accuracy gains 11.8% absolute over Hill et al. (2016).The paper also reports that errors may reveal shortcomings in the information encoded by embeddings.Source: §1 · p. 2
2 Previous Work
Earlier dictionary research extracted semantic information or learned embeddings from definitions, while this work generates definitions from fixed embeddings. The task is also distinct from other embedding-to-text generation because it maps a word-level input to a sequence.
2 Previous Work
- Earlier dictionary studies extracted semantic information or built taxonomies, whereas this work generates definitions from existing, fixed embeddings.Other prior work used definition text to learn embeddings or compositionality and evaluated reverse-dictionary retrieval.Source: §2 · p. 2
- Embedding-to-text approaches also covered phrase reconstruction, document reconstruction, image captions, and spoken dialogue, but definition modeling is a word-to-sequence task.The paper describes its architectures as inspired by sequence-to-sequence models while distinguishing their input-output setup.Source: §2 · p. 2
3 Dictionary Definitions
This section reviews definition structures, describes corpus preparation and analysis, and specifies the definition-modeling and dictionary-ranking tasks.
3.1 Definition Content and Structure
- Definitions often use genus followed by differentiae, but may also reverse that order, give a synonym, or follow miscellaneous structures.The genus is a generalized class and the differentiae distinguish the word from others in that class; patterns and styles can vary by dictionary.Source: §3.1 · p. 2
3.2 Corpus: Preprocessing and Analysis
- The corpus preparation removes non-definitional fields and usage examples and separates entries that may contain multiple definitions.The desired data format has one defined word and one definition per entry.Source: §3.2 · p. 2
- The corpus samples around 20k words from the 50k most frequent Google Web 1T words and has a vocabulary of 29k after cleaning and pruning.Table 2 reports basic corpus statistics, with splits mutually exclusive in the words being defined.Source: §3.2 · p. 3, Table 2 · p. 3
- Manual analysis of 680 definitions from 100 words found genus-plus-differentiae to be dominant in both WordNet and GCIDE, while other structures also occur.Table 3 reports counts of labeled structures in the two dictionaries.Source: §3.2 · p. 3, Table 3 · p. 3
3.3 Dictionary Definition Tasks
- Definition modeling estimates P(D|w*) for an input word using embeddings estimated from a corpus separate from the definition-training corpus.Test-corpus perplexity measures modeling performance, while generation and reverse- and forward-dictionary ranking provide additional evaluations.Source: §3.3 · p. 3
4 Models
The paper models definition generation as predicting a definition from the word being defined and its embedding, using recurrent neural networks. It compares ways to condition the model on that word and explores adding morphological and hypernym features.
4 Models
- Each definition token is predicted from preceding tokens and the word being defined, with an RNN language model summarizing inputs in a hidden state.The target word conditions generation alongside the definition prefix.Source: §4 · p. 3
4.1 Model Architectures
- The Gated model uses update and reset gates to regulate hidden-state updates and how much information from the target word is used.The architecture is motivated by giving the target word more influence on semantically informative portions of definitions than on function-word patterns.Source: §4.1 · p. 4
- Seed supplies the target word only at the first step, Input supplies its embedding at every step, and Hidden uses it to update the recurrent state.These architectures serve as baselines for comparison with Gated.Source: §4.1 · p. 4
4.2 Other Features
- The model concatenates the target-word embedding with additional feature vectors, including character-derived affix features and hypernym embeddings.The hypernym vector is a frequency-weighted sum of the top five hypernyms from WebIsA.Source: §4.2 · p. 4, Hypernym Embeddings · p. 4
Affixes
- A character-level CNN applies varied-length kernels and max pooling to padded one-hot character sequences to detect affixes and represent possible morphological regularities.The approach is motivated by the role of morphemes in word meaning and the difficulty of automatically identifying morphemes and affix semantics.Source: Affixes · p. 4
5 Experiments and Results
Experiments evaluate generalization to unseen words across three tasks, using fixed pretrained embeddings and validation-selected hyperparameters.
5.1 Definition Modeling
- Gated update S+G improves Seed perplexity, unlike S+I and S+H; affixes improve further, while hypernym embeddings’ contribution is unclear.The test-set perplexity is lower for better models.Source: Table 4 · p. 5, §5.1 · p. 5
5.2 Definition Generation
- S+G+CH achieves higher overall BLEU while balancing two dictionary styles; the gate helps most over Seed, and affixes improve further.The baselines have low BLEU when evaluated against the other dictionary; hypernym embeddings have unclear contribution.Source: Table 5 · p. 5, §5.2 · p. 5
- Removing repeated phrases and retaining only the most likely output raises BLEU by 2 for Seed and S+G+CH+HE.The repeated-phrase cleanup addresses patterns found in generated definitions.Source: §5.2 · p. 5
5.3 Reverse and Forward Dictionary
- Our models perform well on reverse and forward dictionary tasks, measured by accuracy at top k and R-Precision, despite optimizing definition likelihood; they have more parameters than prior models.RNN w2v cosine performs better than BOW w2v cosine, a difference the authors suggest may reflect their preprocessing.Source: Table 6 · p. 6, §5.3 · p. 6
6 Discussion
The discussion evaluates generated definitions qualitatively and analyzes how their errors may reflect information captured in word embeddings.
6.1 Qualitative Evaluation and Analysis
- S+G+CH+HE* ranked second after Dictionary on average, significantly above Seed* (p < 0.002) and borderline above NE (p < 0.06).Table 7 reports the percentage of times definitions were ranked in each position and their average rank.Source: Table 7 · p. 6, §6.1 · p. 6
- Gate activations make outputs more dependent on the defined word for content words and less dependent for function words; the hypernym gate stays relatively constant.The authors report that the hypernym embedding does not contribute to performance.Source: §6.1 · p. 6
- Gated models generate better definitions than the compared alternatives, while character information can help select content words such as “mathematics” for “mathematical.”Nearest-embedding definitions can make important errors, such as confusing “feminine” with “masculine.”Source: §6.1 · p. 6
6.2 Error Analysis
- In manual error analysis, 140 of 200 labeled definitions contained some degree of error; fluency problems (types 1–3) likely stem from the definition model rather than shortcomings in embeddings.Source: §6.2 · pp. 6–7
- Other errors may reflect embedding gaps: definitions can omit implicit details, as with a captain’s leadership role, or confuse antonyms.With randomized rather than learned Word2Vec embeddings, test perplexity worsened to 100.43.Source: §6.2 · p. 7
7 Conclusion
The paper introduces definition modeling and evaluates RNN-based models on definition generation and dictionary tasks. It concludes that gated updates and character information improve performance, while some errors may reveal embedding shortcomings.
7 Conclusion
- The gated update, which controls the defined word’s influence at each time step, improves accuracy, and a character-level convolutional layer further improves performance.The conclusion also identifies well-trained word embeddings as crucial to the models.Source: §7 · p. 7
- The paper introduces definition modeling and evaluates RNN language-model architectures on definition generation and reverse and forward dictionary tasks.Source: §7 · p. 7
- Some generated-definition failure modes may provide insight into shortcomings of word embeddings.The authors identify investigating whether definition models can improve embeddings or standard language models as future work.Source: §7 · p. 7