Source-linked AI summary
Does Episodic Memory Help Close the Lexical Frequency Gap in Sensitivity to Syntactic Contrasts? A Test Using Retrieval-Augmented Language Models
Jing Liu, Najoung Kim
TL;DR
Language models remain sensitive to lexical frequency in syntactic-contrast evaluations, raising whether episodic memory can compensate for weak representations of rare items. The paper tests kNN-LM retrieval across syntactic phenomena, frequency bands, and model scales, finding that retrieval consistently narrows the gap, especially when structural information is available, but does not fully close it.
Problem
The paper addresses the gap between language models’ lexical-frequency sensitivity and the robustness observed in human syntactic-contrast judgments.
Method
The study uses kNN-LMs as a functional episodic-memory proxy and evaluates frequency-stratified syntactic contrasts across three phenomena and two pretraining scales.
Results
Retrieval consistently narrows the performance gap between high- and low-frequency items across phenomena and model scales, with structural information critical to the compensation.
Takeaways & Limitations
Episodic retrieval is supported as a possible mechanism for compensating for weak parametric representations, particularly through structurally informed retrieval.
Takeaways & Limitations
The frequency gap remains open, and the conclusions are limited to controlled English minimal-pair tests, two models, and a functional rather than biological CLS analogy.
Abstract
from arXiv · showhide
Grammatical knowledge and how it is empirically tested are typically considered robust to the frequency of the lexical items in the expressions. However, neural network-based models of grammaticality exhibit high sensitivity to lexical frequency. We draw upon Complementary Learning Systems theory to test the hypothesis that robustness to lexical frequency can arise via a hippocampal episodic memory mechanism, which enables rapid encoding and retrieval of specific experiences and allows learners to leverage them when processing rare patterns. We use retrieval-augmented language models as an instantiation of such an episodic memory mechanism (specifically, $k$-nearest-neighbor language models that augment parametric models with explicit instance storage), and test whether this augmentation helps close the lexical frequency gap that vanilla language models exhibit in syntactic contrast tests. Using syntactic contrasts with frequency-stratified test items, we find that retrieval augmentation narrows the performance gap between high- and low-frequency items, consistent with episodic memory compensating for weak parametric representations. This benefit is consistent across different syntactic phenomena and across models pretrained on child-realistic and large-scale data. Additionally, we show that structural information is critical for effective retrieval, whereas semantic similarity alone provides little benefit. While these are promising proof-of-concept results supporting our hypothesis, the frequency gap is narrowed rather than fully closed. Based on our analyses, we propose preferential reweighting of retrieved instances, better representations and retrieval strategies for structural information, and flexible configurations of storage and retrieval as promising future directions for improving the implementation of episodic memory in language models.
1 Introduction
Language models show persistent lexical-frequency biases, especially for low-frequency items, unlike the robustness recently observed in human syntactic-contrast judgments. The paper tests whether episodic retrieval can compensate for weak representations of rare patterns.
- Language-model performance on linguistic evaluations correlates with the training frequency of test-example lexical items.
- Low-frequency items remain a persistent challenge for neural language models, even when grammatical contrasts are otherwise comparable.
- Complementary Learning Systems theory distinguishes rapid hippocampal encoding and retrieval from gradual neocortical extraction of statistical patterns.
- When parametric knowledge is weak for rare items, episodic memory can compensate by directly retrieving relevant past encounters.
- The study uses kNN language models as an episodic-memory proxy, combining a datastore of explicit training instances with an underlying parametric network.
- Across syntactic phenomena and model scales, retrieval narrows but does not fully close the lexical-frequency gap.
2 Related Work
Prior work establishes frequency-related weakness in parametric language-model representations and motivates compensating for it through inference-time retrieval. CLS theory provides the memory framework, while kNN-LMs provide the paper’s computational implementation.
- Language models perform worse on syntactic minimal pairs containing low-frequency words than on otherwise comparable high-frequency pairs.
- Prior findings link corpus frequency distributions to model behavior and show difficulty acquiring long-tail knowledge with sparse supporting evidence.
- These results suggest that lexical frequency creates unequal strength in parametric representations, with low-frequency items learned less robustly.
- This paper asks whether retrieval can compensate for frequency bias at inference time rather than relying only on improved training.
- CLS theory proposes rapid hippocampal episode encoding alongside slower neocortical learning of statistical regularities.
- kNN-LMs supplement parametric knowledge by interpolating the base model’s next-token distribution with distributions from nearest-neighbor datastore contexts.
3 Method
The study evaluates parametric and retrieval-augmented GPT-2 models on frequency-stratified syntactic minimal pairs. It varies retrieval representations and configurations while defining frequency gaps within each model and corpus.
- Design: The analysis compares vanilla models with kNN-LMs on targeted syntactic contrast datasets divided by content-word frequency.
- Models: The two base models are GPT-2 XL with 1.5B parameters and GPT-2 Small with 124M parameters trained on child-directed speech.
- Retrieval model: kNN-LMs store contextual representation–token pairs in a datastore for direct retrieval during inference.
- Retrieval model: At inference, retrieved neighbors produce a kNN probability that is interpolated with the base language-model distribution.
- Evaluation: The evaluation covers subject-verb agreement, wh-questions, and relative clauses using grammatical–ungrammatical minimal pairs.
- Frequency stratification: Frequency bands use high-frequency f > 10^4 and low-frequency f < 10^3 based on mean content-word frequency.
- Frequency stratification: Frequency gaps and their narrowing are measured within each model and corpus, so absolute thresholds are not directly comparable across pretraining settings.
- Retrieval representations: Retrieval variants separate full contextual embeddings from averaged contextualized and uncontextualized semantic representations.
4 Results
Retrieval augmentation improves syntactic-contrast performance across phenomena and pretraining scales, with especially strong benefits for low-frequency items. The frequency gap narrows, but low-frequency performance remains lower after retrieval.
- Overall retrieval benefits: kNN augmentation improves accuracy across all three syntactic phenomena and both pretraining scales, with relative clauses benefiting most.
- Frequency effects: 0.140 versus −0.013: child-realistic retrieval gains are positive for low-frequency items but near zero for high-frequency items.These are per-frequency-band performance deltas, with 95% confidence intervals [0.115, 0.166] and [−0.030, 0.004], respectively.
- Frequency effects: β = 0.52, p < 0.001: retrieval has a significantly larger effect for low-frequency than high-frequency items.
- Frequency effects: The high-frequency retrieval effect is not significant (β = −0.04, p = 0.62).
- Remaining gap: Low-frequency items still yield lower accuracy than high-frequency items across all tested phenomena and model scales after retrieval augmentation.
5 Analysis
Retrieval benefits depend primarily on structurally matched examples rather than semantic similarity alone, while retrieval configuration effects vary across phenomena, frequency bands, and model scales.
- Importance of Structural vs. Semantic Information: Semantics-only retrieval performs at or below the no-retrieval baseline across most conditions, with limited low-frequency relative-clause exceptions.G-Avg produces substantial low-frequency relative-clause gains but substantially degrades high-frequency performance; FastText and GloVe show smaller large-scale gains.
- Importance of Structural vs. Semantic Information: Structurally matched retrieved examples consistently outperform unmatched sets for low-frequency items across phenomena and model scales, especially for relative clauses.Similarity-based selection adds smaller gains, but without structural matches it does not outperform random selection with structural matches.
- Effects of Retrieval Configuration: Sequence-level retrieval is numerically strongest or tied strongest for low-frequency items across phenomena and model settings, although granularity effects are small.The comparison uses k = 16 and τ = 3.
- Effects of Retrieval Configuration: Context-window effects are non-monotonic: preferred window size depends on phenomenon, frequency band, and model.For example, τ = 10 is strongest for relative clauses, while τ = 3 is strongest for low-frequency subject-verb agreement.
- Effects of Retrieval Configuration: For low-frequency items, k = 16 generally outperforms k = 1 and k = 1024, whereas high-frequency effects are smaller and mixed.The intermediate setting improves aggregation of exemplar evidence, while k = 1024 can introduce noise from less relevant neighbors.
- Effects of Retrieval Configuration: No single retrieval configuration is optimal across all phenomena, models, and frequency strata.Sequence-level retrieval, larger context windows, and k = 16 provide robust overall performance but do not uniformly dominate.
6 Discussion
The discussion interprets kNN-LM retrieval as a functional episodic-memory mechanism that narrows, but does not eliminate, lexical frequency disparities. It identifies reweighting, structural retrieval, and adaptive configurations as routes for further improvement while delimiting the evidence.
- Discussion: kNN-LM retrieval provides substantial compensation for low-frequency items, but the lexical frequency gap remains only partially reduced.The authors frame this as a how-possibly explanation and a functional analogy to Complementary Learning Systems.
- Preferential reweighting: Similarity-only retrieval lacks preferential weighting by rarity or informativeness, which may yield suboptimal examples for difficult phenomena such as relative clauses.The proposed remedy is to prioritize novel or informative instances during retrieval or storage.
- Better representations and retrieval strategies for structural information: Because structural information drives retrieval benefits, stronger structural representations and retrieval strategies could further reduce the frequency gap.The discussion links this direction to the persistent difficulty of retrieving structurally similar examples.
- Flexible configurations of storage and retrieval: The absence of a universal configuration suggests that effective episodic retrieval should adapt granularity, context scope, and neighbor count.Fixed settings are therefore unlikely to be uniformly optimal across syntactic phenomena.
- Limitations: The evidence is limited to controlled English minimal-pair tests, three phenomena, and two language models, leaving generalization to naturalistic language use and human behavior open.The kNN-LM also models only a functional CLS analogy, while semantic and structural comparisons do not cleanly isolate the two information sources.
7 Conclusion
Using kNN-LMs as a functional proxy for episodic memory, the study finds that retrieval consistently narrows the lexical frequency gap across syntactic phenomena and model scales. Structural information is critical, but the gap is not fully closed, motivating adaptive and structurally informed retrieval improvements.
- Conclusion: Retrieval consistently narrows the performance gap between high- and low-frequency items across syntactic phenomena and model/data scales.The study uses kNN-LMs as a functional proxy for hippocampal episodic memory.
- Conclusion: Structural information plays a critical role in the compensatory benefits of retrieval augmentation.The conclusion identifies stronger structural representations and retrieval strategies as a future improvement direction.
- Conclusion: The frequency gap remains incompletely closed, with low-frequency lexical items still producing systematically lower performance after retrieval augmentation.The authors present the findings as proof-of-concept support for the CLS hypothesis rather than a complete account of human robustness.
A.1 Pretrained models and datasets
The study uses two decoder-only Transformer models that differ in parameter scale and pretraining data. The child-realistic condition combines several naturalistic language sources, while GPT-2 XL uses large-scale web text.
- Pretraining data: The child-realistic corpus combines child-directed speech, conversational dialogue, children’s stories, and subtitles.These sources also appear in the 2023 BabyLM corpus release.
- Pretraining data: GPT-2 XL provides the large-scale condition, with 1.5B parameters and approximately 40GB of WebText pretraining data.
- Model architectures and training: The child-realistic model uses GPT-2 Small with 124M parameters, is trained from scratch for 10 epochs, and follows the BabyLM Challenge setup.Training uses the standard autoregressive language-modeling objective with cross-entropy loss.
A.2 Evaluation set construction
Evaluation focuses on syntactic phenomena relevant to the research questions by selecting targeted subsets from three established linguistic benchmarks.
- Evaluation set construction: The evaluation subsets are drawn from BLiMP, Zorro, and BIG-bench.
- Evaluation set construction: These selected subsets were designed to focus evaluation on the syntactic phenomena studied in the paper.
- Evaluation set construction: The benchmark subsets are discussed in the following evaluation sections.
A.2.1 Source benchmarks
The evaluation draws on BLiMP, Zorro, and BIG-bench paradigms covering subject–verb agreement, wh-questions, and relative clauses. Analyses test frequency-stratified retrieval gains, structural matching, and retrieval configurations across child-realistic and large-scale models.
- BLiMP: BLiMP contains 67,000 linguistically targeted minimal pairs, while the study selects agreement paradigms relevant to subject–verb agreement.BLiMP items are automatically generated from linguist-crafted grammar templates and have 96.4% aggregate human agreement.
- Zorro: Zorro provides restricted-vocabulary grammar tests, including selected paradigms for agreement, wh-questions, and relative clauses.Its 23 paradigms contain 2,000 minimal pairs per paradigm and use BabyBERTa vocabulary items.
- BIG-bench: BIG-bench contributes selected subject–verb agreement and relative-clause subtasks to the evaluation.
- Statistical analyses: The analysis estimates per-frequency-band retrieval gains with 95% bootstrap confidence intervals based on 10,000 resamples over test items.The reported accuracy delta is kNN − baseline for k = 16, τ = 3, sequence-level retrieval.
- Statistical analyses: A mixed-effects logistic regression tests Frequency × Model Type while including random intercepts for items and phenomena.High-frequency items and the baseline model are the reference levels.
- Statistical analyses: β = 0.52, p < 0.001 for the Frequency × Model Type interaction, while retrieval has no detectable high-frequency effect (β = −0.04, p = 0.62).The interaction indicates larger retrieval effects for low-frequency items.
- Structural match analysis: Structural matches are identified for subject–verb agreement, wh-questions, and relative clauses using dependency parsing and part-of-speech information.Relative-clause detection additionally uses relativizers, relcl dependencies, and noun-phrase embedding.
- Structural match analysis: Retrieval benefits are greater when neighbors contain structurally matched examples, especially for low-frequency items and structurally complex relative clauses.Table 8 reports the proportion of retrieved neighbors containing the target structure, and Table 9 compares retrieval effects across structural-match conditions.