Source-linked AI summary
The LAMBADA dataset: Word prediction requiring a broad discourse context
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, Raquel Fernández
TL;DR
Existing language models may exploit local statistical patterns without demonstrating broad text understanding, motivating a benchmark that tests discourse-level word prediction. The paper introduces LAMBADA, a human-normed narrative dataset designed for this purpose, and finds that tested models perform far below humans, making it a challenging benchmark.
Problem
Language-model evaluations must distinguish statistical pattern extraction from genuine understanding of broader discourse context.
Method
LAMBADA uses narrative passages where humans can predict the target word from the whole passage but not the preceding sentence, and evaluates language models through word prediction.
Results
None of the tested models came remotely close to human performance on LAMBADA in preliminary experiments.
Takeaways & Limitations
LAMBADA provides a challenging benchmark for developing models capable of accounting for broad context in natural language.
Takeaways & Limitations
The evaluation is preliminary and may improve with more extensive tuning or more sophisticated mechanisms such as attention.
Abstract
from arXiv · showhide
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.
1 Introduction
LAMBADA is introduced to test whether language models use broad discourse context rather than merely producing locally plausible text. Preliminary experiments found that existing models with longer-term memory remained far from human performance.
- Statistical generalization from large corpora can make neural systems appear to understand language more deeply than they do.
- Locally appropriate responses can become incoherent when a system fails to incorporate the broader discourse context.
- LAMBADA frames broad-context language understanding as a classic word-prediction task.
- Existing language-model architectures, including systems designed to retain longer-term contextual memories, were tested on the dataset.
- None of the tested models came remotely close to human performance in preliminary experiments, establishing LAMBADA as a challenging benchmark.
2 Related datasets
LAMBADA differs from related broader-context benchmarks in both its narrative-completion setup and the inference it requires. Word prediction is presented as a natural, simple, and data-flexible way to probe text understanding.
- CNNDM asks models to recover a removed named entity from a short article summary using the broader article context.
- Unlike CNNDM, LAMBADA requires understanding a plausible continuation of a narrative fragment or dialogue rather than summarizing an article.
- CBT uses book excerpts with a randomly removed word, but its passages were not filtered to ensure that only broader context enabled human guessing.
- In a large proportion of CBT cases, annotators who guessed from broader context could also guess from the last sentence alone, while about one fifth remained unsolved even with broader context.
- Word prediction is attractive because it is natural to norm with non-expert humans, simple to formulate, and trainable from virtually unlimited raw text.
3 The LAMBADA dataset
LAMBADA is a word-prediction dataset built from narrative passages where the target is guessable from broad discourse context but not the preceding sentence alone. Its construction and analysis emphasize broad-context phenomena, including explicit cues, discourse inference, and varied linguistic structure.
- Dataset design: LAMBADA passages contain an average of 4.6 context sentences and require guessing the last word of a target sentence.The context is the minimum number of complete preceding sentences containing at least 50 tokens.
- Data source: The dataset uses Book Corpus novels and separates source novels into disjoint training and development-plus-testing partitions.The corpus contains 5,325 novels and 465 million words after duplicate and offensive-material filtering.
- Data collection: Human filtering retained passages whose targets were guessable from the whole passage but not from the target sentence alone.Collection used sequential whole-passage and sentence-only judgments after automated filtering.
- Dataset design: 10,022 passages form LAMBADA, divided into 4,869 development and 5,153 test passages from disjoint novels.Average passage length is 75.4 tokens for development and 75 tokens for test passages.
- Dataset analysis: More than 80% of LAMBADA passages include the target word in context, compared with less than 15% of the input passages.The target can still require linguistic and general cognitive skills even when explicitly present.
- Dataset analysis: Proper nouns make up 48% of targets, common nouns 37%, and verbs 7.7%, while common nouns also reflect bridging, synonymy, and event-participant inference.About 16% of items lack the target lemma in context; in about one third of examined cases, the missing word was closely related by root, part of speech, or synonymy.
4 Modeling experiments
The experiments evaluate language models and baselines on LAMBADA, where performance is uniformly poor despite reasonable control-set results. The findings indicate that broad-context understanding, rather than shallow passage statistics or exact-word heuristics, is central to the task.
- Computational methods: The study evaluates RNN, LSTM, N-Gram, cache N-Gram, Memory Network, and CBOW-based baselines on LAMBADA.Sup-CBOW uses a bag-of-words passage representation, while the evaluated model set includes architectures intended to handle broader context.
- Results: About 1/5 accuracy on the control task is achieved by the N-Gram models and LSTM, but LAMBADA performance is drastically worse for all models.The contrast shows that the models can perform standard language modeling while struggling on the broad-context benchmark.
- Results: 7% accuracy from the capitalized-word heuristic does not suffice, while the shallow Sup-CBOW baseline performs very poorly.The result argues against solving LAMBADA through the passage’s proper-noun bias or simple bag-of-words exploitation.
- Results: N-Gram models outperform neural-network models comparatively on perplexity and rank, with N-Gram w/cache achieving perplexity 768.Nevertheless, even N-Gram w/cache is effectively unable to guess the correct word.
- Limitations: The evaluation is preliminary and may improve with more extensive tuning or mechanisms such as attention.The authors characterize the experiments as a proof-of-concept study and caution that the tested models are not a definitive evaluation of all possible approaches.
- Limitations: Because LAMBADA filters for passages where simple language models fail, standard language models are expected to fail on it by design.The authors frame this construction as an inherent difficulty that future models should overcome because humans can solve the task.
5 Conclusion
The paper presents LAMBADA as a benchmark for broad discourse understanding in word prediction. Its preliminary results motivate developing models that capture non-local phenomena through longer-term memory and reasoning.
- Conclusion: LAMBADA tests whether language models can use broad discourse context to predict a word that humans find difficult from the last sentence alone.The dataset targets non-local phenomena in narrative passages and contrasts whole-passage access with sentence-only access.
- Conclusion: The authors hope LAMBADA will stimulate novel language models that genuinely capture the reflected non-local phenomena.They plan to support this direction through a public competition based on the dataset.
- Conclusion: Longer-term memory coupled with reasoning about stored information is proposed as a crucial component of successful models.This is presented as the authors’ hunch despite the disappointing performance of the tested vanilla Memory Network.
- Conclusion: Human performance on word prediction is presented as a promising basis for benchmarks targeting computational text understanding.The authors position broad-context influence as one example of this broader benchmarking strategy.