Source-linked AI summary
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Yasaman Razeghi, Robert L. Logan, Matt Gardner, Sameer Singh
TL;DR
Few-shot numerical reasoning may depend on how frequently instance terms appeared during pretraining, raising questions about robust generalization. The paper measures frequency–accuracy relationships across numerical tasks and GPT models, finding consistently large gaps favoring high-frequency terms. It argues that reasoning evaluations should account for pretraining data while acknowledging that the study does not establish causality and is limited to numerical reasoning.
Problem
The paper asks whether strong few-shot numerical reasoning reflects robust generalization beyond pretraining data, since benchmark interpretation often lacks reference to the model’s training corpus.
Method
The study counts unigram and low-order co-occurrence frequencies for instance terms in the Pile, then relates them to accuracy across addition, multiplication, and unit-conversion tasks.
Results
Across three Pile-pretrained EleutherAI/GPT model sizes and numerical reasoning experiments, performance gaps between highest- and lowest-frequency terms are consistently large, exceeding 70% in some cases.
Takeaways & Limitations
Reasoning evaluations should take pretraining corpora into account, because overlooking their influence can mislead assessments of model reasoning skills.
Takeaways & Limitations
The study makes no causal claim, may contain uneliminated confounders, and is limited to numerical reasoning tasks.
Abstract
from arXiv · showhide
Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper, we investigate how well these models reason with terms that are less frequent in the pretraining data. In particular, we examine the correlations between the model performance on test instances and the frequency of terms from those instances in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models (pretrained on the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and unit conversion). Our results consistently demonstrate that models are more accurate on instances whose terms are more prevalent, in some cases above $70\%$ (absolute) more accurate on the top 10\% frequent terms in comparison to the bottom 10\%. Overall, although LMs exhibit strong performance at few-shot numerical reasoning tasks, our results raise the question of how much models actually generalize beyond pretraining data, and we encourage researchers to take the pretraining data into account when interpreting evaluation results.
1. Introduction
Few-shot numerical performance may reflect pretraining-data frequency rather than robust reasoning. The paper investigates this concern and finds accuracy strongly associated with term frequency.
- Large language models perform reasoning-related tasks in zero- and few-shot settings, including arithmetic.
- Over 20% separates GPT-J-6B’s example multiplication performances for 24 and 23, despite aggregating across multiplicands and seeds.
- A robust arithmetic reasoner should perform similarly on queries whose operands have different pretraining frequencies.
- GPT-J-6B accuracy strongly correlates with first-term frequency when multiplying numbers 0–100 by operands 1–50.
- The study analyzes addition, multiplication, and unit conversion by relating instance-term frequencies in pretraining data to model accuracy.
- More than 70% separates average accuracy for the highest- and lowest-frequency terms in some experiments across three GPT model sizes.
2. Background and Methodology
The paper measures whether numerical reasoning performance varies with pretraining frequency. It defines term-frequency statistics and summarizes their association with accuracy using a top-versus-bottom quantile performance gap.
- Background: In-context learning places training examples in prompts rather than changing the model’s internal parameters.
- Background: Prior arithmetic results may reflect answers encountered during pretraining, motivating analysis of statistical overlap with evaluation terms.
- Methodology: The study measures term frequencies, connects them to reasoning accuracy, and introduces the performance gap as its comparison metric.
- Frequency: Numerical instances contain input terms x_i, consisting of positive integers or time units, and a derived positive-integer output y.
- Frequency: Frequency counts occurrences of subsets of an instance’s terms within a specified pretraining-data window, including selected unigram and co-occurrence statistics.
- Frequency: Frequency is symmetric for term pairs even when operand order remains important for representing the task instance.
- Performance Gap: The performance gap compares average accuracies for instances in the top and bottom frequency quantiles.
- Performance Gap: For multiplication, the gap over ω{x1} compares average accuracy for the 10% most frequent and least frequent first operands.
3. Experiment Setup
The experiments use publicly available EleutherAI GPT models and the released Pile corpus to study numerical reasoning tasks. The setup constructs frequency-aware arithmetic, operation-inference, and time-conversion evaluations with few-shot prompts.
- Setup: The experimental setup measures pretraining-data effects on few-shot numerical reasoning across different language models.
- Models and Corpus: The models are GPT-J-6B, GPT-Neo-1.3B, and GPT-Neo-2.7B, whose architectures and pretraining corpus are publicly available.
- Models and Corpus: The Pile is an 800GB English dataset spanning 22 academic or professional data sources, used to count integers with fewer than seven digits.
- Tasks: The study creates arithmetic, operation-inference, and time-unit-conversion datasets targeting mathematical capabilities.
- Data Construction: The data-construction pipeline uses pretraining term counts to develop reasoning queries and render them into model-input prompt templates.
- Tasks: Time conversion represents values such as 24 hours to minutes using the primary number and time units as additional operands.
- Evaluation: Few-shot prompts use 0, 2, 4, 8, or 16 examples, with performance averaged over five random prompt selections.
4. Results
Across arithmetic, operation inference, and time-unit conversion, model accuracy is generally higher for instances with more frequent pretraining terms. This frequency-related performance gap often increases with more shots and appears across model sizes, though decade-to-year conversion is an exception.
- Arithmetic: Arithmetic accuracy correlates positively with term frequency, with especially large gaps for multiplication and persistent frequency effects for addition.
- Operation Inference: Operation-inference models perform better on frequent instances, including on this lower-accuracy task that requires inferring the operation and computing the answer.
- Time-Unit Conversion: Time-unit conversion generally shows increasing performance gaps with more shots, except decade-to-year conversion, where performance is nearly perfect with 8 shots and gaps remain small.
- Model Size: Smaller models are less accurate overall but remain focused on instances containing more frequent pretraining terms.
- Across 11 datasets, performance gaps are generally high across multiple definitions of instance-term frequency.
- Performance gaps commonly increase as the number of shots grows, indicating that additional prompt examples do not eliminate frequency-related differences.
5. Related Work
Related work examines prompting, memorization, pretraining frequency, dataset artifacts, and documentation of pretraining data. This paper connects those concerns to numerical and temporal reasoning in language models.
- Prompting: Prior prompting research uses in-context examples to probe factual, commonsense, mathematical, and other knowledge acquired during pretraining.
- Impact of Frequency on LM Performance: Studies of pretraining frequency examine how it affects memorization, reasoning with facts, and generalizable syntax rules.
- Memorization: Memorization research defines memorization through accuracy differences when training points are included versus excluded, and applies influence functions to language models.
- Training Artifacts Challenge Evaluation: Work on training artifacts shows that lexical cues, question-passage overlap, and related biases can affect evaluation of language-model generalization.
- Documenting Pretraining Data: Researchers have called for better documentation of datasets, although the size of language-model pretraining corpora makes this challenging.
- Numeracy and Temporal Reasoning in LMs: Related numeracy research includes methods for improving temporal and numerical reasoning and evidence that math solvers rely on simple heuristics.
6. Discussion and Future Work
Few-shot reasoning evaluations can be misleading when they ignore pretraining-data effects, including simple unigram overlaps. The authors recommend incorporating pretraining information into evaluation while noting causal, task-scope, and pattern-complexity limitations.
- Prior evaluation methods focused on overlap between training or pretraining data and test instances, but simple unigram overlaps can also affect model performance.
- Evaluation of reasoning capabilities should take the pretraining corpus into account, and reasoning claims require robustness to pretraining effects.
- Future evaluations could report performance gaps between high-frequency and low-frequency terms, although a zero gap may be necessary but not sufficient for demonstrating reasoning.
- The study does not make a causal claim and may contain uneliminated confounders, motivating causal-inference methods and training interventions.
- The analysis aggregates relatively simple patterns and is limited to numerical reasoning tasks, leaving broader pattern complexity and other reasoning capabilities for future research.
7. Conclusion
In-context language-model performance on numerical reasoning tasks can be significantly affected by low-order pretraining co-occurrence statistics. This raises questions about how much models actually reason when solving these tasks.
- Low-order co-occurrence statistics in pretraining data can significantly impact in-context language-model performance on numerical reasoning tasks.