Source-linked AI summary
Learning to Understand Phrases by Embedding the Dictionary
Felix Hill, Kyunghyun Cho, Anna Korhonen, Yoshua Bengio
TL;DR
Learning useful representations for phrases and sentences is difficult because there is no obvious gold standard. This paper trains neural models to map dictionary definitions to word embeddings, then applies them to reverse dictionaries and crossword question answering, where they perform comparably to or better than commercial systems.
Problem
Useful phrase and sentence representations are difficult to learn because they lack an obvious gold standard for training, evaluation, and comparison.
Method
Neural RNN and BOW embedding models map dictionary definitions or sentences to pretrained embeddings of the words they define.
Results
Across reverse-dictionary and crossword-answering tasks, the models perform comparably to or better than commercial systems, including outperforming commercial solvers on longer clues.
Takeaways & Limitations
Dictionary-based training provides a practical bridge between lexical and phrasal semantics for these two language-understanding applications.
Takeaways & Limitations
The crossword evaluation narrows the task because the architecture lacks a knowledge base or web-scale information and does not address fitting answers into a grid.
Abstract
from arXiv · showhide
Distributional models that learn rich semantic word representations are a success story of recent NLP research. However, developing models that learn useful representations of phrases and sentences has proved far harder. We propose using the definitions found in everyday dictionaries as a means of bridging this gap between lexical and phrasal semantics. Neural language embedding models can be effectively trained to map dictionary definitions (phrases) to (lexical) representations of the words defined by those definitions. We present two applications of these architectures: "reverse dictionaries" that return the name of a concept given a definition or description and general-knowledge crossword question answerers. On both tasks, neural language embedding models trained on definitions from a handful of freely-available lexical resources perform as well or better than existing commercial systems that rely on significant task-specific engineering. The results highlight the effectiveness of both neural embedding architectures and definition-based training for developing models that understand phrases and sentences.
1 Introduction
The paper addresses the difficulty of learning and evaluating representations for arbitrary-length phrases by using dictionary definitions as training bridges to word representations. It applies these models to reverse dictionaries and general-knowledge crossword question answering.
- Phrasal and sentence representations lack an obvious gold standard, making both learning and model comparison difficult.
- Dictionary definitions provide phrases whose composed meanings correspond to the meanings of the words they define.This lets word vectors serve as targets for learning coherent phrasal representations.
- The approach experiments with recurrent neural networks and feedforward bag-of-words embedding models, using Word2Vec representations as lexical targets.
- The reverse dictionary returns candidate words from user descriptions and performs comparably to or better than commercial systems using only a handful of dictionaries.
- The crossword question answerer uses dictionary definitions and Wikipedia openings, with both BOW and RNN models outperforming bespoke commercial solvers particularly for longer clues.
2 Neural Language Model Architectures
The paper compares RNN and BOW architectures for mapping variable-length definitions into fixed-length word embeddings. RNNs summarize sequences through recurrent states, while BOW sums projected word embeddings; training uses pretrained lexical representations and distance-based objectives.
- RNN architecture: RNNs encode variable-length definitions, descriptions, or sentences and map them to representations of the words those inputs define.
- RNN architecture: The RNN updates an internal activation with each input word, so its final state acts as a summary of the sentence.
- Long Short Term Memory: LSTMs mitigate vanishing or exploding gradients that can cause RNNs to neglect information near the beginning of longer sentences.
- Long Short Term Memory: LSTM gates control input incorporation, memory retention or forgetting, and the extent to which memory contributes to the output state.
- Long Short Term Memory: The LSTM memory summary is projected into the target embedding space, allowing target and activation dimensions to differ.
- Bag-of-Words NLMs: BOW encodes a definition by summing projected word embeddings and is a special RNN case with identity update and nonlinear functions.
- Embedding initialization: Pretrained input embeddings can add linguistic or conceptual knowledge and expand the effective vocabulary at test time.
- Training objective: Models learn to place each definition near its defining word's pretrained embedding using cosine distance or a rank loss.
3 Reverse Dictionaries
The paper evaluates neural reverse dictionaries against commercial and simple composition baselines, finding strong generalization to unseen descriptions and coherent candidate outputs. It also extends the approach cross-lingually, while noting that prior academic systems cannot be directly compared because their models and evaluations are unavailable.
- 3 Reverse Dictionaries: Reverse dictionaries return candidate words for a user’s phrase, description, or definition, supporting writers seeking a word for an idea.The paper contrasts this neural approach with information-retrieval systems using hand-engineered sentence features and proprietary commercial systems.
- 3.2 Comparisons: Direct comparison with prior academic reverse-dictionary systems is unavailable because their models and evaluations are not publicly released.The evaluation therefore compares against commercial systems, especially OneLook; Dictionary.com returned no candidates for over 96% of input definitions.
- 3.3 Reverse Dictionary Evaluation: Supervised RNN, BOW, and OneLook models clearly outperform unsupervised composition baselines, with elementwise addition outperforming multiplication.Multiplication almost never returns the correct word as the nearest neighbor of the composition.
- 3.4 Results: On unseen concept descriptions, neural models generalize well: they match OneLook approximately in top-ten retrieval, exceed it within the top 100, and show lower ranking variance.OneLook has a lower median rank but behaves more inconsistently across individual queries.
- 3.4 Results: RNNs do not significantly outperform BOW models, whose order-invariant outputs can still recover concepts effectively from permuted descriptions.The result emphasizes the contribution of lexical semantics to interpreting definitions.
- 3.5 Qualitative Analysis: NLM candidates can generalize beyond training data and often form syntactically and semantically coherent sets, whereas OneLook may return broadly related words of varied types.Examples include climate-related candidates absent from the relevant training definitions and functionally appropriate verbs or deverbal nouns.
- 3.6 Cross-Lingual Reverse Dictionaries: The RNN architecture can be modified into a bilingual reverse dictionary, and the embedding-space model avoids online bilingual mappings and some word-sense ambiguity errors.The bilingual model returns candidates in one language from descriptions in another, with applications for translators or transcribers.
- 3.7 Discussion: The evaluation initially tests dictionary embedding only on tasks it was trained to perform, before examining transfer to a novel task.This marks a scope boundary in the reverse-dictionary analyses.
4 General Knowledge (crossword) Question Answering
The paper evaluates dictionary embedding models as general-knowledge crossword question answerers, using clue sets of different lengths and comparisons with unsupervised and commercial systems. Performance is strongest on longer questions, where embedding models outperform the comparison systems, while shorter clues reduce that advantage.
- Task and motivation: General-knowledge crossword answering narrows open-domain question answering by providing answer length and clues that can resemble definitions or category requests.The evaluation focuses on language understanding rather than fitting answers into a crossword grid.
- Evaluation: The evaluation uses 150 longer Eddie James questions and 150 shorter Guardian Quick Crossword questions, excluding multiword answers and applying additional length filters to the short set.The long set excludes clues shorter than four words; the short set excludes clues longer than four words, and 30 short clues are single-word clues.
- Benchmarks and comparisons: The comparison includes RNN and BOW neural language models, elementwise Word2Vec addition, One Across, and Crossword Maestro, with candidates restricted to the clue’s specified answer length.Commercial systems are evaluated through web interfaces, so only accuracy@10 is reliably recorded for them.
- Results: On long questions, the best RNN ranks the correct answer third on average and places it among the top ten over 60% of the time.All dictionary embedding models show a clear advantage over the commercial systems and the unsupervised baseline on this question type.
- Results: The embedding models’ advantage diminishes on shorter questions, where the unsupervised baseline and One Across achieve comparable accuracy.Short clues more often reduce to synonym generation or category associations, which existing systems and Word2Vec representations can handle well.
- Qualitative analysis: Qualitative examples show that embedding models generate semantically consistent candidates and answer some factual questions despite training on definitions and introductory sentences.The BOW model infers connections for a Schoenberg query even though the query and definition share only the word “composer.”
5 Conclusion
Dictionary definitions provide valuable training data for neural language models that represent phrases and sentences. The approach supports competitive reverse dictionaries, crossword question answering, and an objective evaluation task, while leaving clear directions for improvement.
- Dictionary lexical resources can train neural language models to interpret and represent phrase and sentence meaning.
- Two architectures—a recurrent long-short-term memory model and a linear bag-of-words model—explicitly map dictionary phrases toward lexical meaning.
- Reverse-dictionary models perform comparably to leading commercial applications despite using many fewer definitions, without linguistic preprocessing or task-specific engineering.
- Crossword models outperform commercial systems on questions containing more than four words, and the results suggest possible extensions to broader question answering, dialogue, and information retrieval.
- The released code, data, evaluation sets, and tools support future research, including the reverse-dictionary task as an objective test of phrase composition.
- The models are not trained on question-like language, and richer world knowledge may require an external memory module.