PaperlayerView exact revision on arXiv

Source-linked AI summary

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjape, Adina Williams, Tal Linzen, Ryan Cotterell

arXiv:2504.08165v1cs.CL

TL;DR

Humans learn language from far less data than language models, motivating a test of whether low-resource training can improve data efficiency and cognitive plausibility. The BabyLM Challenge compares diverse systems under fixed data budgets and finds strong performance from particular architectures and training strategies, while identifying unresolved evaluation and scaling questions.

  • Problem

    Humans acquire language from less than 100 million words, while language models typically use 3 or 4 orders of magnitude more data and remain behind humans on some evaluations.

  • Method

    The BabyLM Challenge evaluates 31 submissions trained under progressively looser data restrictions on grammar, downstream performance, generalization, and related tasks.

  • Results

    Winning systems using LTG-BERT outperformed BabyLM baselines and Llama 2, while strong results also came from shorter input sequences and teacher–student training.

  • Takeaways & Limitations

    The most consistent gains came from modified architectures, training objectives, principled preprocessing, and hyperparameter searches; curriculum learning was generally unsuccessful.

  • Takeaways & Limitations

    Results for scaling model size were mixed, and controlled studies across architectures and datasets are needed to determine whether scaling up or down is preferable.

Abstract

from arXiv · show

Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements.

1 Introduction

The BabyLM Challenge targets the gap between human and language-model data efficiency by organizing low-resource training under standardized evaluation. It reports diverse approaches, with LTG-BERT-based winners outperforming BabyLM baselines and Llama 2.

  • Language models require hundreds of times more input than humans while humans remain the state of the art in language learning.
  • The challenge invites NLP, linguistics, and cognitive science researchers to train models in data settings resembling human language learners.Its goals include improving cognitive-model relevance, finding data-efficient algorithms, and democratizing training research.
  • The evaluation pipeline targets zero-shot grammar, finetunability on language-understanding tasks, model inductive bias, and semantic and discourse phenomena.The challenge also contributes novel zero-shot evaluations for semantic and discourse-level behavior.
  • 31 papers explored architectures, objectives, preprocessing, hyperparameters, curriculum learning, and teacher–student training.The strongest overall models used the LTG-BERT architecture and improved over BabyLM baselines and Llama 2.
  • Future challenges should use harder, more varied evaluations, explore underexamined approaches such as multimodality, and incentivize compute efficiency.

2 Motivation

The motivation is to understand how humans learn language from far less data than modern language models and to use data-limited training for cognitive modeling, faster experimentation, and broader research participation.

  • Children receive less than 100 million word tokens by age 13, whereas modern language models typically use 3 or 4 orders of magnitude more data.This discrepancy motivates questions about human language-learning efficiency and its implications for language-model improvement.
  • Data-limited training can support cognitively plausible models, faster iteration on architectures and hyperparameters, and research beyond highly funded industry groups.
  • When trained on developmentally plausible data volumes, language models perform poorly on benchmarks of human-like syntactic and semantic behavior.
  • Closing the data-efficiency gap can help identify which aspects of human learning support efficient acquisition and reduce differences between humans and models.
  • Optimizing language-model pipelines for a fixed data budget remains relatively underexplored.
  • The challenge extends work on small-dataset models and community efforts centered on data-limited training.

3 Guidelines and Timeline

The challenge defined three data-use tracks, allowed creative submissions outside track eligibility, and supported participation through public infrastructure and a staged evaluation timeline.

  • Tracks: Strict, Strict-Small, and Loose tracks allowed 100 million, 10 million, and 100 million English-language training tokens respectively, with Loose permitting additional non-linguistic data.
  • Tracks: Repeating data across epochs did not count as additional text, although the organizers acknowledged that iterative processing differs from human learning.
  • Tracks: Submissions using external linguistic data could be presented and published but could not win a track.
  • Community building: The public forum helped establish the community, clarify rules, debug the evaluation pipeline, and distribute announcements.At paper writing, it had over 250 members.
  • Timeline: The timeline progressed from the December 2022 announcement through dataset release, public evaluation, hidden evaluations, and the June 2023 model-submission deadline.

4 Pretraining Corpus

The BabyLM corpus was designed around children’s input, controlling both scale and domain while providing standardized splits and minimal preprocessing for Strict and Strict-Small training.

  • Size: The Strict corpus contains under 100 million words, while Strict-Small contains under 10 million words, approximating developmentally plausible input volumes.The Strict-Small set corresponds roughly to input during the first two to five years of development.
  • Size: Unlike contemporary models trained on trillions of words, the corpus is intended to resemble the quantity and domain of children’s language experience.
  • Domain: Approximately 56% of the corpus comes from transcribed or scripted speech, reflecting the predominant input available to hearing children.This contrasts with standard corpora consisting mostly of text intended to be read and potentially edited.
  • Domain: About 40% of the corpus comes from child-directed or child-appropriate sources, including children’s books, educational videos, and simplified English.
  • Domain: The corpus includes multiple domains because fewer than 10 million words of transcribed child-directed speech are available and child-directed speech is only part of children’s input.
  • Data preparation: The organizers released train, development, and test splits for ten sources, with the Strict-Small training set randomly sampled from Strict training data.
  • Data preparation: Preprocessing was minimal, preserving original newlines in most texts while applying source-specific extraction and cleaning procedures.

5 Evaluation

The evaluation pipeline measures grammatical ability, downstream NLP performance, generalization, and alignment with human language learning. It combines established benchmarks with hidden and supplementary tasks designed to test broader linguistic knowledge and reduce overfitting.

  • Grammatical ability: BLiMP evaluates whether language models assign higher probability to acceptable than unacceptable sentences in minimal pairs.
  • Downstream tasks: A subsample of (Super)GLUE assesses paraphrase detection, sentiment, inference, question answering, acceptability judgments, and commonsense reasoning.
  • Generalization and human alignment: Hidden MSGS, BLiMP supplement, and AoA tasks broaden evaluation and help prevent submissions from overfitting to BLiMP and (Super)GLUE.MSGS tests whether models generalize according to linguistic or surface features; AoA compares model surprisal-derived predictions with children’s word-acquisition ages.
  • Supplementary linguistic tests: Supplementary test suites probe lexical entailment, subject–auxiliary inversion, dialogue turn-taking, and question–answer congruence.These tasks target linguistic relationships and dependencies beyond the original BLiMP coverage.
  • Generalization and human alignment: The AoA task compares language-model word surprisals with children’s ages of acquisition, using mean absolute deviation from human AoA values.
  • Evaluation caveat: Performance on the AoA-style evaluation should be interpreted primarily as alignment with humans and usefulness as a cognitive model, not as general language-model quality.

6 Submissions Summary

The challenge attracted broad participation and a wide range of training approaches. Its organizers catalogued submissions by method and released leaderboard, predictions, scores, and metadata to support comparison and analysis.

  • Participation: 31 papers and 162 models were submitted, with participants sometimes entering multiple tracks.
  • Approaches: The submissions covered architecture design, hyperparameter tuning, curriculum learning, and teacher–student model training.
  • Approach typology: The organizers assigned each submitted model one or more labels from a nine-approach typology to quantify methodological patterns.
  • Transparency and resources: The official leaderboard and participant-consented models, predictions, task scores, and metadata were released through Dynabench and GitHub.

7 Results & Analysis

Across tracks, the strongest BabyLM results approached Llama 2 and human performance, while architecture and data preprocessing were more consistently productive than curriculum learning or multimodal inputs.

  • Overall Results & Track Winners: Strict models were strongest overall, but only two exceeded the best Strict-Small model on GLUE despite ten times more training data.Loose models generally performed worse in aggregate than Strict-Small models, despite potential access to additional non-linguistic data.
  • Overall Results & Track Winners: Many BabyLM models approached the Llama 2 skyline and human performance; the top BLiMP model was about 3% below the human level.GLUE results involved finetuning, whereas BLiMP provided a grammatical evaluation without that same downstream-task setup.
  • Winning Submissions: ELC-BERT won both the Strict and Strict-Small tracks, with LTG-BERT’s backbone architecture appearing to account for much of the success.LTG-BERT combines additional layer normalization, GEGLU modules, disentangled attention, and scaled weight initialization.
  • Winning Submissions: The Loose-track Contextualizer created a dataset with as many training samples as a 4B-word dataset from only 100M words, outperforming 40 training epochs on the same samples.It generated additional samples by combining chunks of text from different contexts and repeating this process 40 times per chunk.
  • Common Methods: Curriculum learning was attempted by 13 teams (41.9%), but most variants did not produce consistent improvements across BabyLM evaluation tasks.Curricula varied sentence ordering by surprisal, lexical frequency, length, or syntactic complexity, among other schemes.
  • Common Methods: Knowledge distillation, shorter training sequences, and preprocessing changes often improved results, whereas multimodal submissions did not reliably improve overall accuracy.Model-size results were mixed: some teams benefited from scaling up, while others performed well with very small models.

8 Future BabyLM Challenges

Future BabyLM Challenges should refine evaluation, standardize effective training practices, broaden participation, and explicitly address compute efficiency and organizational support.

  • Future challenge design: 31 papers produced lessons spanning preprocessing, hyperparameter tuning, architectures, curriculum learning, and teacher–student training.The authors propose using these findings to improve future challenge iterations.
  • Future challenge design: Strict-Small attracted significantly more submissions than the other tracks, partly because it was least computationally expensive and constrained the search space.The authors suggest more specific and constrained Loose tracks focused on directions such as multimodality or expert annotations.
  • Future challenge design: Future pipelines could standardize effective preprocessing, starting hyperparameters, and strong architectural baselines.One proposed preprocessing change is sorting data according to a curriculum method that improved performance.
  • Compute efficiency: Training for hundreds of epochs is not cognitively plausible and does not make experimentation more accessible on a university budget.The authors therefore suggest rewarding compute efficiency in future iterations.
  • Organizational support: A larger dedicated evaluation-support team could maintain the pipeline and support more submission types, including systems outside HuggingFace.The current pipeline was difficult for a single organizer to maintain and update.

9 Conclusions

The BabyLM Challenge showed that language modeling can be improved under smaller and more cognitively plausible data constraints, with gains concentrated in several training choices.

  • Conclusion: The challenge asked whether language modeling could improve using smaller and more cognitively plausible datasets.This framing connects low-resource modeling with cognitive modeling research.
  • Conclusion: The most consistent gains came from modified architectures, new training objectives, principled preprocessing, and hyperparameter searches.Curriculum learning produced significant improvements in one case.
  • Conclusion: Future work can build on these findings to improve language modeling for low-resource settings and cognitive modeling research.The authors present the findings as a basis for subsequent BabyLM work.

A Data Source Descriptions

The BabyLM pretraining sources combine child-directed interaction, dialogue, children’s literature, encyclopedic text, subtitles, educational material, and broader literary collections.

  • Dialogue and child language: CHILDES provides about 5M words from transcribed adult–child interactions involving American English-speaking children ages 0–6.The processed data remove child utterances and tokenize the interactions.
  • Dialogue and child language: The British National Corpus contributes about 10M words from its dialogue portion.The source is a 100M-word multidomain British English corpus from the second half of the twentieth century.
  • Children’s literature: Children’s literature sources include the Children’s Book Test and a manually selected Children’s Stories Text Corpus.The Children’s Book Test contains over a hundred books, while the stories corpus was compiled for story-generation research.
  • Literary and reference text: Project Gutenberg supplies a curated selection of over 50k public-domain literary books totaling over 3B tokens.Its metadata supports filtering by language and date.
  • Literary and reference text: Other sources include Simple English Wikipedia, English Wikipedia, OpenSubtitles, QED educational subtitles, and Switchboard telephone conversations.Simple English Wikipedia uses shorter sentences and high-frequency vocabulary, while Switchboard contains conversations between pairs of strangers.

B Evaluation Data Details

The evaluation details describe vocabulary-based filtering, task-specific dataset sizes, and results for grammatical and generalization evaluations, while limiting direct comparison with prior full datasets.

  • Evaluation data: Evaluation examples lacking lexical overlap with the Strict-Small pretraining corpus were filtered, partially controlling for mismatch in language style.Other factors, including sentence length and syntactic complexity, were not controlled by this procedure.
  • Evaluation data: The filtered evaluation data support comparisons among models evaluated on the challenge’s task versions, but not direct comparison with prior full-dataset results.The filtering uses subsets of the original training and evaluation examples.
  • Evaluation data: Table 4 reports the numbers of training and test examples for each evaluation task after filtering by pretraining-corpus vocabulary.The table is intended to document the resulting task sizes.
  • Generalization results: MSGS Matthews correlation coefficients were largely negative, although ELC-BERT and Contextualizer showed high positive average scores in some comparisons.The reported scores indicate a tendency toward surface features for many models and stronger linguistic-feature preferences for selected models.
  • Generalization results: Strict-trained models generally had higher MSGS MCCs than Strict-Small models, but the pattern depended on the compared features and architecture.The authors interpret this as evidence that more data generally strengthen syntactic inductive biases, without making the relationship universal.
  • Grammatical evaluation: BLiMP results showed that larger models did not consistently perform better, while Boot-BERT generally performed best among BabyLM submissions.ELC-BERT and Boot-BERT are both based on LTG-BERT, which the authors identify as a promising starting architecture.
  • Grammatical evaluation: The best BabyLM submissions outperformed Llama 2 on ISLAND EFFECTS, whereas Llama 2 was stronger on QUANTIFIERS.Most models achieved consistent but mediocre results on QUANTIFIERS.

D.3 BLiMP Supplement

BLiMP Supplement results reveal sharp differences across linguistic abilities: some constructions are easy, while adversarial cross-sentential dependencies and lexical entailment remain difficult.

  • Lexical entailment: HYPERNYM performance is near chance for all models, including skylines, leaving lexical entailment beyond their demonstrated ability.The authors caution that unnatural logical statements and assumptions about invalid statements may partly explain this result.
  • Question–answer congruence: Only top-performing Strict-track models exceed chance on tricky question–answer congruence examples, while RoBERTa substantially outperforms them.The tricky set exposes susceptibility to locally coherent distractors and insufficient attention to long-distance wh-word dependencies.
  • Syntactic and discourse abilities: Subject–auxiliary inversion is relatively easy, with the best models reaching near-perfect accuracy.This contrasts with the stronger discrimination observed for question–answer congruence and turn-taking.
  • Syntactic and discourse abilities: Turn-taking is highly discriminative: some models perform near chance, whereas the best exceeds 90% accuracy and ELC-BERT outperforms the skylines.The authors suggest transcribed dialogue may contribute because it occupies a relatively large share of BabyLM training data.
  • Downstream confirmation: (Super)GLUE results generally place ELC-BERT first in the Strict and Strict-Small tracks, with Boot-BERT also highly effective in Strict.These downstream results broadly confirm LTG-BERT’s effectiveness for smaller-corpus pretraining and suggest curriculum learning can sometimes help.

E Age of Acquisition Prediction Results

The AoA task evaluates how closely models predict when words are acquired. Most submissions were in Strict-Small, where none surpassed OPT-125M, although several came close.

  • Strict-Small: No Strict-Small model achieves closer AoA predictions than the OPT-125M baseline, though many perform very close to it.The table reports mean average deviation in months, with lower values indicating better predictions.
  • Strict: BabyStories achieves a score very close to the OPT-125M baseline in the Strict track.The comparison uses the best-performing baseline within each track.

F Summary of Each Submission

The submissions explore architectures, preprocessing, curricula, training duration, multimodal inputs, and distillation. Results favor some targeted interventions, but curriculum learning is inconsistent and often underperforms random ordering.

  • Data and initialization: Targeted grammar induction and data augmentation improve some baselines, but grammar-induced embeddings show similar gains to random initialization and augmentation relies on GLUE-derived patterns.These results make the source of the apparent improvements less uniform across approaches.
  • Training setup: Reducing context length from 128 to 32 produces significant and consistent improvements, whereas proposed curricula lack consistent benefits.The comparison covers the BabyLM Strict-Small data and contrasts proposed, reversed, and absent curricula.
  • Architectures: ELC-BERT modifies LTG-BERT by learning weighted combinations of all previous layer outputs; all variants outperform the BabyLM RoBERTa baseline, without one variant clearly winning.Learned weights emphasize the previous few layers and the static embedding layer.
  • Curriculum learning: Curriculum learning based on linguistic complexity finds improvements over random ordering in Strict-Small, but other curricula underperform or fail to exceed random baselines.Evidence ranges from modest gains to no improvement across dependency, frequency, entropy, context-length, surprisal, and complexity-based orderings.
  • Teacher–student training: Baby Llama distills two teacher models into a 58M-parameter student that outperforms the baselines, its teachers, and a same-sized model trained from scratch.The teachers are a 300M-parameter Llama model and a 700M-parameter GPT-2 model trained on the Strict-Small corpus.
  • Architectures: StructFormer’s hierarchical bias improves classic masked-language-model pretraining most on single-sentence or syntactic tasks, less consistently on multi-sentence semantic tasks.The reported gains therefore vary with evaluation scenario.
  • Training duration: Training longer helps several approaches: DistilBERT improves at 60 versus 20 epochs, and ChapGTP continues improving through 200 epochs.Other work similarly reports benefits from many epochs, including small BabyBERTa models.
Loading 2504.08165v1…