Source-linked AI summary
When Do You Need Billions of Words of Pretraining Data?
Yian Zhang, Alex Warstadt, Haau-Sing Li, Samuel R. Bowman
TL;DR
The paper asks what knowledge or skills large pretrained language models gain from billions of words that smaller-data models do not. It compares four probing measures across RoBERTa models trained on different data volumes and finds that linguistic representations mature far earlier than downstream NLU performance. The results point to additional skills, including commonsense knowledge, as important at larger scales.
Problem
Prior probing work mostly studies models pretrained on fixed, usually billion-word datasets, leaving unclear what data-rich models know that smaller-data models do not.
Method
The study evaluates MiniBERTas and RoBERTaBASE with classifier probing, minimum description length probing, BLiMP acceptability judgments, and SuperGLUE fine-tuning across pretraining volumes.
Results
Linguistic features are mostly learned by 100M words, whereas SuperGLUE performance improves most rapidly beyond 1B words and likely continues improving at larger scales.
Takeaways & Limitations
Encoding linguistic features is necessary but likely does not explain large-scale NLU gains; other skills, potentially including commonsense knowledge, matter as well.
Takeaways & Limitations
The results do not explain what causes NLU performance to improve with large quantities of pretraining data, and the authors call for causal probing and broader settings.
Abstract
from arXiv · showhide
NLP is currently dominated by general-purpose pretrained language models like RoBERTa, which achieve strong performance on NLU tasks through pretraining on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? We adopt four probing methods---classifier probing, information-theoretic probing, unsupervised relative acceptability judgment, and fine-tuning on NLU tasks---and draw learning curves that track the growth of these different measures of linguistic ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that LMs require only about 10M or 100M words to learn representations that reliably encode most syntactic and semantic features we test. A much larger quantity of data is needed in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models.
1 Introduction
The paper asks what large-scale pretraining adds beyond linguistic features learnable from less data. It probes RoBERTa models across data volumes and finds that linguistic knowledge saturates earlier than downstream NLU performance.
- Pretrained language models learn linguistic features and world knowledge from datasets containing tens or hundreds of billions of words.
- Existing probing studies largely examine models trained on fixed, usually billion-word datasets, leaving data-volume learning requirements unclear.
- The study probes MiniBERTas and RoBERTaBASE across pretraining volumes from 1M to about 30B words using four evaluation methods.
- Three probing methods find RoBERTa pretrained on 100M words has linguistic knowledge close to RoBERTaBASE pretrained on about 30B words.
- RoBERTa requires billions of pretraining words for substantial improvements on downstream NLU tasks.
2 Methods
The study evaluates MiniBERTas and RoBERTaBASE across increasing pretraining volumes with four probing approaches. Scores are normalized by task, plotted against data volume, and summarized with fitted logistic learning curves.
- The evaluation also includes RoBERTaBASE pretrained on about 30B words and three randomly initialized RoBERTaBASE models.
- The four methods are classifier probing, minimum description length probing, unsupervised BLiMP acceptability judgments, and SuperGLUE fine-tuning.
- All 16 models are tested on every task, with most results min-max normalized so 0 and 1 mark the worst and best model scores.
- Learning curves plot normalized scores against pretraining volume and fit a logistic function after log-transforming the x-values.
3 Classifier Probing
Classifier probing measures how well frozen MiniBERTa representations encode linguistic features through downstream classifiers. Most feature learning occurs before 100M words, while Winograd coreference improves mainly at much larger scales.
- Method: Classifier probing freezes MiniBERTa representations and trains MLP classifiers for ten edge-probing tasks, including syntactic and semantic features.For pairwise tasks, attention-pooled span representations are classified against candidate labels; single-span tasks omit the second span.
- Overall results: 90% of attainable overall classifier-probing improvements are estimated to occur with <20M words of pretraining data.The estimate is based on a best-fit logistic learning curve; overall performance averages normalized F1 across ten tasks.
- Overall results: Most feature learning occurs with <100M words, with curves rising sharply below 1M words and nearly saturating by 100M words.This pattern is reported across most individual probing tasks.
- Task groups: Winograd coreference improves significantly only between 1B and 30B words, unlike the faster learning of most syntactic and semantic features.Because Winograd coreference tests commonsense knowledge and reasoning, the authors infer that these features require more data to encode.
- Task groups: 90% of syntactic-learning improvements are reached with about 10M words, while semantic learning continues rising slightly after 100M words.The aggregated curves separate syntactic, semantic, and commonsense tasks; syntactic learning rises slightly earlier than semantic learning.
4 Minimum Description Length Probing
MDL probing measures the bits needed to transmit task labels from model representations, balancing decoder complexity against prediction loss. Across edge probing tasks, most feature-codelength reductions occur before 100M words, though Winograd Coreference is anomalous.
- MDL sums data codelength and model codelength, capturing prediction loss and the cost of transmitting decoder parameters.Simpler decoders can increase data codelength, creating a tradeoff between the two terms.
- The online code partitions training data into 11 portions, trains successive MLP decoders, and sums held-out losses plus an initial uniform-prior cost.The portions range from 0% to 100% of the training data.
- Most feature-codelength reduction occurs with fewer than 100M words, with syntactic-feature MDL decreasing even sooner.The analysis uses one classifier head per candidate label, including tasks with multiple correct labels.
- Winograd Coreference results are idiosyncratic, likely because the probes failed to learn the task, and its learning curve lacked an adequate logistic fit.
- Data codelength decreases monotonically with richer representations, while model-codelength trends are inconsistent across tasks.On many tasks, model codelength begins decreasing at larger pretraining volumes, suggesting more accessible feature information.
5 Unsupervised Grammaticality Judgement
BLiMP evaluates unsupervised grammatical acceptability across 67 minimal-pair tasks spanning 12 linguistic phenomena. Overall gains are concentrated between 1M and 100M words, but data requirements vary substantially by phenomenon.
- BLiMP uses 67 minimal-pair tasks to test morphological, syntactic, and semantic phenomena through unsupervised likelihood comparisons.A model is correct when it assigns higher likelihood to the acceptable sentence.
- The greatest overall BLiMP improvement occurs between 1M and 100M words, reaching within 9 accuracy points of humans at 100M words.Additional data produces only 6 more accuracy points.
- Agreement phenomena improve almost entirely between 1M and 10M words, whereas wh-dependencies show slower learning through 100M words.The contrast is consistent with local, frequent agreement dependencies versus rarer, long-distance wh-dependencies.
- Figure 5 groups BLiMP results by linguistic category and compares them with human agreement and RoBERTaLARGE performance.The benchmark contains 12 categories of phenomena.
- Quantifier phenomena are not effectively learned even by RoBERTaBASE, likely reflecting subtle semantic or pragmatic contrasts.The examples include distinctions such as “Nobody ate more than” versus “at least” two cookies.
6 Finetuning on NLU Tasks
The study fine-tunes MiniBERTas on selected SuperGLUE tasks to assess downstream language understanding. Unlike probing and BLiMP performance, SuperGLUE improvements generally require more than 1B pretraining words and continue through 30B words.
- SuperGLUE comprises eight classification-based language-understanding tasks, of which five are evaluated at these data scales.
- For most SuperGLUE tasks, the fastest improvement occurs beyond 1B words, with no significant saturation by 30B words.The possible exception is CommitmentBank.
- The SuperGLUE curves suggest that some key NLU skills are not learned with fewer than billions of words.The authors therefore expect continued improvement with 10 to 100 times more pretraining data.
7 Discussion
The discussion contrasts rapid acquisition of linguistic representations with the much slower improvement of downstream NLU performance. It considers commonsense knowledge, feature use, and probing-method limitations as possible explanations.
- Classifier probing, MDL probing, and acceptability judgments improve rapidly between 1M and 10M words, with little improvement beyond 100M.
- SuperGLUE performance improves most rapidly above 1B words and likely continues improving at much larger data scales.
- At least some skills used for typical NLU tasks require billions of words, although many edge-probed linguistic features are already encoded by 100M-word models.
- Limited commonsense knowledge below 1B words may help explain the delayed SuperGLUE improvement, but the experiments do not fully assess commonsense acquisition.
- Encoding a linguistic feature does not guarantee that a model can use it in downstream tasks or generalize with it preferentially.
- The similar classifier-probing and MDL curves do not establish that feature information and accessibility improve at the same rate.
8 Related Work
Related work examines probing-method validity, data-volume effects on linguistic knowledge, and downstream performance across architectures, languages, and pretraining scales.
- Warstadt et al.’s experiments differ by measuring preference for linguistic over surface features during fine-tuning, which may not correspond straightforwardly to MDL.
- Classifier probes can learn from training data and may fail to distinguish extractable features from features actually used by models.
- Different probing methods can yield contradictory results, motivating evaluation with multiple methods.
- Earlier acceptability studies generally find rapid linguistic-knowledge gains up to about 10M words, followed by slower improvement.
- Raffel et al. report that roughly 500M words can match T5 performance with 34B words, contrasting with the present RoBERTa findings.
- French-language studies report substantial learning from small-scale pretraining, including similar question-answering performance after 100MB and larger datasets.
9 Conclusion
The conclusion summarizes a data-volume study of linguistic representations and NLU performance, then identifies unresolved causes and broader scope questions.
- The study tracks language models’ acquisition of linguistic-feature representations as pretraining data increases.
- Linguistic features are mostly learnable with 100M words, whereas NLU task performance requires far more data.
- The results do not explain what causes NLU task performance to improve with large quantities of data.
- Future work should examine causal probing, feature-learning interactions, other architectures, pretraining tasks, and more human-like data domains.
A Appendices
The appendices provide comparisons with larger pretrained models and supporting hyperparameter and BLiMP result tables.
- Figure 7 compares absolute edge-probing development-set results with BERTLARGE test-set results from Tenney et al. (2019b).
- Figure 8 compares absolute SuperGLUE results with RoBERTaLARGE results from Liu et al. (2019).
- Table 1 lists hyperparameter search ranges for the SuperGLUE tasks.
- Table 2 reports BLiMP results alongside scores from 5-gram, LSTM, TXL, GPT-2, and BERTBASE systems.