Source-linked AI summary
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Urvashi Khandelwal, He He, Peng Qi, Dan Jurafsky
TL;DR
Neural language models’ use of prior context is poorly understood. The paper uses test-time context ablations on LSTMs across PTB and WikiText-2 to examine context length, order, and copying. It finds roughly 200 useful tokens on average, with nearby order-sensitive context differing sharply from distant context and cache support especially important for copying faraway words.
Problem
How neural language models use prior linguistic context, especially beyond a sentence, remains poorly understood.
Method
The authors apply test-time ablations to a pretrained LSTM on PTB and WikiText-2, dropping, shuffling, and replacing prior context words.
Results
About 200 context tokens are useful on average; word order matters mainly within recent context, while the cache helps copy words from long-range context.
Takeaways & Limitations
The findings clarify that nearby and distant context serve different roles and provide evidence for how cache-based models help LSTMs copy distant words.
Takeaways & Limitations
Separating model behavior from language characteristics is difficult because model and data are tightly coupled with confounding factors such as vocabulary and dataset size.
Abstract
from arXiv · showhide
We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the distant history. The model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context (beyond 50 tokens), suggesting the distant past is modeled only as a rough semantic field or topic. We further find that the neural caching model (Grave et al., 2017b) especially helps the LSTM to copy words from within this distant context. Overall, our analysis not only provides a better understanding of how neural LMs use their context, but also sheds light on recent success from cache-based models.
1 Introduction
The paper examines how LSTM language models use prior context beyond the sentence level, addressing limited understanding of their contextual behavior. Ablation studies reveal distinct roles for nearby and distant context and clarify how caching supports copying.
- NLM context use remains largely unexplained despite their success being attributed to long-range dependency modeling.
- The study extends sentence-level LSTM analyses to long-range context and asks how much context is used, how regions differ, and how copy mechanisms help.
- The authors perturb pretrained LSTM contexts by changing length, word order, and target-word presence to measure effects on performance and copying.
- About 200 context tokens are useful on average, with word order relevant only within the 20 most recent tokens and distant context represented roughly semantically.
- The LSTM can regenerate some nearby words but relies heavily on the cache to copy words from long-range context.
2 Language Modeling
This section defines language-model prediction and evaluation through conditional probabilities, negative log likelihood, and perplexity. The experiments compare loss changes under contextual ablations.
- Language models estimate the conditional probability of a target word from its preceding context.
- Training minimizes negative log likelihood over the corpus, while held-out performance is commonly evaluated with perplexity.
- Ablation comparisons focus on differences in dev-set NLL, which are equivalent to relative improvements in perplexity.
3 Approach
The authors evaluate a standard LSTM by perturbing context at test time across PTB and WikiText-2, using dev-set results averaged over multiple random seeds. This design measures the contribution of contextual information without retraining on perturbed inputs.
- Ablation analysis measures performance changes when contextual information is removed during evaluation.
- The study perturbs pretrained contexts by dropping, shuffling or reversing, and replacing tokens, while retaining standard training.
- The infinite-context setting feeds all tokens preceding the target word to the model.
- Experiments use a standard LSTM and add a cache only in Section 6.2 to investigate external copying.
- The datasets are PTB, with 0.9M training tokens and a 10K vocabulary, and Wiki, with 2.1M tokens and a 33K vocabulary.
- Results are reported on dev sets, confirmed against test sets, and averaged across three models trained with different random seeds.
4 How much context is used?
The LSTM uses roughly 200 context tokens on average, but context requirements vary by dataset, hyperparameters, word frequency, and part of speech. Nearby and content-word context contribute more strongly than distant or function-word context.
- An effective context size is the token count achieving loss comparable to infinite context, defined here as a 1–2% perplexity difference.
- About 200 tokens are used on average, determined by progressively varying the number of recent tokens supplied at test time.
- Truncation drops tokens farther from the target, and the resulting loss increase measures the importance of the omitted context.
- A 1% perplexity increase occurs beyond 150 tokens on PTB and 250 tokens on Wiki, indicating dataset-specific effective context sizes.
- Changing dropout, model size, or truncated-backpropagation settings changes perplexity but not the context-usage trend.
- Infrequent words require more than 200 tokens, whereas frequent-word loss is insensitive to missing context beyond the most recent 50 tokens.
- Content words need more context than function words: nouns and verbs use distant context, while determiners rely mainly on the last 10 tokens.
- Effective context size is dynamic because it depends on the target word.
5 Nearby vs. long-range context
The LSTM distinguishes nearby from long-range context: recent word order and content/function-word information matter most near the target, while distant word identity remains useful without ordered structure.
- Word order: Local word order matters strongly within the most recent 20 tokens, approximately the length of an average sentence.Permuting local 20-token spans has much less effect when the span is farther from the target.
- Word order: Global word order matters only within the most recent 50 tokens; beyond that, shuffling or reversing context words does not affect performance.The study uses 50 tokens as the boundary between nearby and long-range context.
- Word identity: Faraway word identity remains relevant even though faraway word order does not, indicating a rough semantic representation of long-range context.Replacing words causes a different effect from permuting them, showing that the model is not simply insensitive to distant changes.
- Word types: Dropping function words near the target increases loss by about 15%, while beyond a sentence only content words have a sizeable influence.Function words remain relevant nearby because they support grammaticality and syntactic structure.
6 To cache or not to cache?
The analysis separates how LSTMs use nearby and distant context for copying, then examines when a neural cache improves or harms this behavior. The cache especially supports copying from long-range history but can mislead the model when the target is absent.
- 6.1 Can LSTMs copy words without caches?: Replacing a nearby-copyable target with another vocabulary token hurts more than dropping it, indicating that the target’s identity—not merely grammatical context—supports copying.The figure caption reports no comparable replacement effect for words that can only be copied from far away.
- 6.1 Can LSTMs copy words without caches?: LSTMs directly copy nearby-context words more often, while using rough semantic information to generate words available only in distant context.For Cfar, removing distant context increased perplexity by 12%, versus 2% when only target occurrences were removed; for Cnear, the corresponding increases were about 3.5% and almost 9%.
- 6.2 How does the cache help?: The neural cache records hidden states and interpolates a history-based word distribution with the model’s vocabulary distribution, upweighting words from the prior context.Its flatness is controlled by θ, and the cache is evaluated with all prior context to match the referenced setup.
- 6.2 How does the cache help?: The cache helps distant-copyable words more than nearby-copyable words: without caching, perplexity rises 28% versus 22% on PTB and 53% versus 32% on Wiki for Cfar versus Cnear.This complementarity reflects the standard LSTM’s weaker ability to regenerate words from long-range context.
- 6.2 How does the cache help?: Caching harms about 36% of PTB words and 20% of Wiki words that cannot be copied from context, because the cache distribution becomes flat when the target is absent.The paper suggests allowing the model to ignore the cache when it is highly uncertain.
7 Discussion
The study interprets LSTM context behavior and discusses how these findings may guide model improvements, while acknowledging that model behavior and language characteristics are difficult to disentangle.
- Implications: Adaptive word dropout could reflect that some word types are more important than others.The authors also suggest allowing the cache distribution to be ignored when it is extremely uncertain.
- Implications: Nearby-versus-long-range differences suggest that explicit memory representations could capture information orthogonal to the LSTM.This implication is proposed for memory models that feed context representations to the LSTM.
- Broader evaluation: The empirical methods are model-agnostic and can compare model classes beyond test-set perplexity.Suggested comparison axes include average context usage and robustness to shuffled contexts.
- Limitations: Separating model behavior from language characteristics is difficult because the model and data are tightly coupled.Vocabulary size and dataset size are among the confounding factors identified by the authors.
- Limitations: PTB and Wiki provide some variation in content, writing style, and structure, but further work across model classes and languages is suggested.Wiki articles are 4-6x longer on average and include titles and paragraph or section markers.
- Comparison: A prior PTB result reported close to an 8% perplexity increase at 13 context tokens, compared with 25% in this setup.The authors attribute the difference to restricted-context training and a different error propagation scheme in the prior model.
8 Conclusion
The conclusion summarizes the empirical findings about LSTM context use, order sensitivity, and cache-assisted copying, while identifying sentence-level interactions as future work.
- A standard LSTM effectively uses about 200 context tokens on two benchmark datasets, regardless of hyperparameter settings such as model size.
- Word order matters more in nearby context than in long-range context.
- The LSTM regenerates some nearby words but relies heavily on caches to copy words from far away.
- Deeper understanding of sentence-level interactions remains future work because the observations are reported at the token level.
A Hyperparameter settings
The appendix specifies the vanilla LSTM configuration and regularization choices used in the study.
- The model is a vanilla LSTM with dropout applied to recurrent connections, embedding weights, and all input and output connections.
- The architecture ties word-embedding and softmax weights.
- Training uses variable-length backpropagation sequences and the averaging SGD optimizer.
B Additional Figures
The additional figures extend the main analyses across datasets and perturbation settings, showing how context order, word type, context length, copying, and cache behavior affect loss.
- Appendix scope: The appendix reports complementary figures for analyses presented in the main text, with conclusions unchanged.Results are averaged across three models trained with different random seeds; error bars show standard deviation or 95% confidence intervals depending on the chart.
- Context order: Shuffling or reversing 300 context tokens does not affect loss beyond 50 tokens in PTB.
- Word types: Dropping both content and function words near the target increases loss, while beyond 20 tokens content words are more relevant on Wiki.
- Context length: Changing model hyperparameters preserves the context-usage trend while changing model performance.
- Copying: Words copyable only from long-range context are more sensitive to dropping distant words than to dropping the target.
- Copying: For words copyable from nearby context, dropping the target has a much larger loss effect than dropping long-range context.
- Neural cache: The neural cache fails when its distribution is flat and succeeds when its distribution is peaky on Wiki.