Source-linked AI summary
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Felix Hill, Antoine Bordes, Sumit Chopra, Jason Weston
TL;DR
The paper asks whether language models use wider context to capture meaning, rather than mainly predict frequent function words. It introduces the Children’s Book Test and compares models with different memory representations. Explicit long-term context helps semantic-word prediction, with intermediate-sized window memories and self-supervised retrieval producing the strongest reported results across CBT and CNN QA.
Problem
The paper addresses limited evaluation of how language models use wider context for semantically informative words rather than frequent syntactic function words.
Method
The paper introduces the Children’s Book Test and compares models that encode previously read context using different memory representations, including window-based memories and self-supervised retrieval.
Results
Explicit context representations outperform conventional neural language models on semantic content words, while intermediate window sizes and self-supervised retrieval achieve the strongest named-entity results on CBT and CNN QA.
Takeaways & Limitations
Memory representations encoding sub-sentential windows can retain and retrieve useful semantic information more effectively than single-word or full-sentence representations for these tasks.
Takeaways & Limitations
The conclusion notes that optional Memory Network features are evaluated separately in an appendix, and the human comparison may be affected by annotators not reading the training novels beforehand.
Abstract
from arXiv · showhide
We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.
1 INTRODUCTION
The paper introduces the Children’s Book Test to examine how models use wider context, especially for semantically informative words. Explicit memory representations and appropriately sized context windows improve semantic prediction, while function-word prediction shows a different pattern.
- The Children’s Book Test evaluates whether models exploit local and wider context when predicting missing words in children’s books.It distinguishes prediction of different word types, including named entities, nouns, verbs, and prepositions.
- Humans use wider context for named entities and nouns, but not for higher-frequency verbs or prepositions.
- RNNs with LSTMs predict prepositions and verbs well but lag behind humans on nouns and named entities because their predictions rely mainly on local context.
- Memory Networks can exploit explicit wider-context representations to improve prediction of named entities and nouns over conventional language models.The authors connect this advantage to applications requiring coherent semantic processing or language generation.
- Window-based memory representations have an optimal size between single words and full sentences, and self-supervised memory access improves named-entity prediction.The preferred window size depends on the word class being predicted.
2 THE CHILDREN’S BOOK TEST
The Children’s Book Test constructs multiple-choice prediction questions from 21 consecutive sentences in children’s books. It separates word classes so models can be evaluated on both contextual semantic content and syntactic function words.
- The CBT forms each example from 21 consecutive sentences, using the first 20 as context and the 21st as a query containing a missing word.
- Each question asks a model to select the missing answer from 10 candidate words drawn from the context and query.The answer, query, context, and candidate set are denoted a, q, S, and C.
- The benchmark evaluates Named Entities, Common Nouns, Verbs, and Prepositions using nine incorrect candidates matched to the answer’s word type.
- Unlike average-perplexity evaluations, the CBT supports focused analysis of lower-frequency content words that carry more of the meaning in language.This is intended as a proxy for semantic coherence in applications such as translation, dialogue, and question answering.
- Compared with the MSRCC, the CBT supplies wider context beyond a single sentence, uses 10 rather than 5 candidates, and has 10,000 rather than 1,040 test questions.
- CNN QA instead asks models to identify missing entities in paraphrased summaries of online news articles, with named entities anonymised.
3 STUDYING MEMORY REPRESENTATION WITH MEMORY NETWORKS
The paper compares Memory Networks that encode context as words, windows, or sentences and retrieve supporting memories through attention. It finds that window memories and self-supervised retrieval provide an effective strategy for semantic prediction.
- Memory Networks are used to study how different context encodings affect semantic processing of naturally occurring language.
- 3.1 ENCODING MEMORIES AND QUERIES: Lexical memory stores each word in a separate slot and adds time features to encode word order.
- 3.1 ENCODING MEMORIES AND QUERIES: Window memory stores text windows centered on occurrences of candidate words, with window size tuned on validation data.A separate dictionary for each window position performed best among the tested window encodings.
- 3.1 ENCODING MEMORIES AND QUERIES: Sentential memory stores complete context sentences, producing exactly 20 memories for each CBT question.
- 3.2 END-TO-END MEMORY NETWORKS: The end-to-end model retrieves supporting memories by comparing query and memory embeddings, then returns an answer distribution from the retrieved memories.
- 3.3 SELF-SUPERVISION FOR WINDOW MEMORIES: Self-supervision infers a supporting window from the correct answer and trains the model to score it above memories associated with other candidates.
- 3.3 SELF-SUPERVISION FOR WINDOW MEMORIES: At test time, candidate scores sum the attention scores of all corresponding windows rather than using only the highest-scoring window.This relaxes the max operation so multiple windows can contribute information.
4 BASELINE AND COMPARISON MODELS
The paper compares non-learning baselines, classical and neural language models, contextual models, and human performance on the Children’s Book Test.
- Comparison models: The evaluation includes language-modelling and machine-reading architectures alongside simple frequency and non-learning ranking baselines.The compared systems include n-gram and cache models, supervised embeddings, LSTM RNNs, and contextual recurrent models.
- Non-learning baselines: The sliding-window baseline ranks candidates by scanning overlapping query–candidate windows across the context.Each candidate is associated with ten windows formed by concatenating it with the query and sliding the result across the context.
- Non-learning baselines: The word-distance benchmark aligns the query with candidate occurrences in the context and selects the occurrence with the lowest alignment penalty.The penalty is computed from the nearest positional matches between query words and the aligned context subsequence.
- Neural models: The embedding model scores each candidate from an encoded input passage and learned input and output word-embedding matrices.The paper evaluates variants using the entire context and query, the query alone, or query windows, including position-specific representations.
- Neural models: LSTM language models were trained on 5.5M words with minibatch SGD, using 512-dimensional hidden states and word embeddings in the best model.A context-plus-query variant could read the entire context before answering the query.
- Human comparison: Human results were collected on 10% of the CBT test set, while Table 2 reports the overall CBT test-set comparison.The human evaluation involved 15 native English speakers and 2,000 answers across question types.
5 RESULTS
Performance varies sharply by predicted word type: conventional language models handle syntactic words well, while Memory Networks better capture nouns and named entities using wider context. Window-based memories with self-supervised retrieval perform especially strongly, and related results support sub-sentential context representations.
- Modelling syntactic flow: Conventional language models predict prepositions and verbs well but perform less accurately on named entities and nouns.
- Capturing semantic coherence: Memory Networks predict common nouns and named entities more accurately than conventional language models by exploiting wider context.LSTMs without attention perform similarly on these word classes with or without context, consistent with limited long-range retention.
- Getting memory representations ‘just right’: Window memories centred on candidate words are more useful for predicting named entities and nouns than word-level or sentence-level memories.Lexical memories work better for prepositions and verbs, whereas sentence memories perform poorly in general.
- Self-supervised memory retrieval: Self-supervised window-based Memory Networks outperform all other models at predicting named entities and common nouns.Training uses hard attention selection among window memories, while the question representation remains a simple window-based strategy.
- News article question answering: On CNN QA, the unsupervised window model matches the best prior approach, while adding self-supervision greatly surpasses the state of the art.Removing entities already appearing in the bullet summary improves performance further; the CNN QA dataset contains 93k news articles.
- News article question answering: Best-performing reading models support informative text representations as sub-sentential chunks, with Memory Network gains linked to hard attention and self-supervision.The paper identifies jointly learning information access and use as difficult, while self-supervision makes Memory Network learning more tractable.
6 CONCLUSION
The paper presents the Children’s Book Test as a benchmark for wider-context use and finds that sub-sentential memory windows, trained with self-supervision, support semantic prediction across CBT and CNN QA.
- The Children’s Book Test separates syntactic function-word prediction from semantically informative-word prediction to measure contextual language understanding.
- Memories encoding informative sub-sentential windows are most useful, although the best chunk size depends on whether the task concerns semantic or syntactic words.
- Memory Networks using this principle and simple self-supervision surpassed other methods at named-entity prediction on CBT and CNN QA.
- The appendix contains an ablation study removing optional Memory Network features.
A EXPERIMENTAL DETAILS
The experiments tune model hyperparameters on validation data, including embedding dimensions, learning rates, memory sizes, hops, and window sizes for CBT and CNN QA.
- Hyperparameters were selected by validation-set grid search, covering embedding dimension, learning rate, window size, hops, and maximum memory size.
- The section identifies the optimal CBT and CNN QA hyperparameter settings but does not report their corresponding performance values.
- CBT window-based models use window size b = 5, while the listed CNN QA window-based models also use b = 5.
- The listed CBT configurations include lexical, window, sentential, and self-supervised window Memory Networks with different dimensions, learning rates, memory sizes, and hop counts.
- CNN QA configurations include window-memory models with self-supervision and ensembles of 7 or 11 models.
D EFFECTS OF ANONYMISING ENTITIES IN CBT
Anonymising candidate entities and words has relatively little effect on named-entity prediction but a larger effect on syntactic tasks such as prepositions and verbs.
- The experiment replaces the ten candidates in each question with anonymised placeholders across training, validation, and test sets.
- Anonymisation has relatively low impact on named entities but a larger impact on preposition and verb prediction.
E CANDIDATES AND WINDOW MEMORIES IN CBT
The study compares candidate-based and broader window or target training for Memory Networks, finding similar superiority with self-supervision but speed costs for broader alternatives.
- In the main CBT results, window memory consists of windows over candidates, with training targeting the true answer against dictionary words.
- “All windows” includes every possible window, although extra windows do not affect CBT candidate scores at evaluation time.
- “All targets” trains with the true answer against all dictionary words and does not require known candidates.
- Language-model training uses all targets and all windows because multiple-choice questions do not cover every word.
- All alternatives provide superior named-entity and common-noun results with self-supervision compared with training without self-supervision.
- Using all windows or language-model training instead of candidate windows affects training and testing speed.