PaperlayerView exact revision on arXiv

Source-linked AI summary

Specialization, Diversity, and Scaling in Child-Scale Language Models

Steven Y. Feng, Michael C. Frank

arXiv:2610.36679v1cs.CLcs.AIcs.LG

TL;DR

Children experience far less language than modern language models train on, and evidence about learning from small naturalistic datasets is limited. This paper trains small GPT-2 models on naturalistic and synthetic child-directed data while varying amount, source, and diversity. Models specialize to their training environments, while broader input diversity improves pooled prediction without a similarly clear grammatical-benchmark gain.

  • Problem

    Evidence about learning from small naturalistic datasets is limited, despite the large difference between children’s estimated exposure and language-model training data.

    Source: §1 · p. 1

  • Method

    The study trains GPT-2-style models on BabyView and TinyDialogues data, varying training amount, source, and diversity while keeping the learner mostly fixed.

    Source: §3.2 · p. 3

  • Results

    Models specialize to individual children’s environments, and more children’s input improves pooled held-out prediction without a similarly clear grammatical-benchmark gain.

    Source: §8 · p. 8

  • Takeaways & Limitations

    At small data sizes, which environments supply the words and how well they match evaluated language matter alongside training-data quantity.

    Source: §7 · p. 7

    Conclusions are limited to the tested models, datasets, and evaluations; BabyView samples only part of children’s language and multimodal experience.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child's language environment predict that environment substantially better than other children's environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child's input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.

1 Introduction

Children experience far less language than modern language models train on, but small naturalistic datasets raise questions about more than input quantity. The paper varies data source, amount, and diversity to examine specialization and learning in child-scale models.

Approach

  • The study keeps the learner mostly fixed while varying training data, using small GPT-2 models and a Kneser-Ney 5-gram count-model control.The models are used to study properties of small language datasets, not to reproduce how children learn.Source: §1 · pp. 1–2

Motivation

  • Children’s environments differ in vocabulary, topics, speakers, routines, and linguistic structures, so equal word counts can provide different input distributions.The paper calls whether evaluated language is represented in training “distributional support” and the similarity of training and evaluation distributions “match.”Source: §1 · p. 1

Data

  • BabyView provides naturalistic transcripts from children’s everyday environments, while TinyDialogues supplies synthetic child-directed dialogue at larger scales.Dense recordings from individual children remain smaller than estimates of children’s total exposure.Source: §1 · p. 1

Research questions

  • The study asks how models specialize to individual children, what changes when a fixed budget spans more children, and how natural and synthetic learning curves differ.Source: §1 · p. 2

Main findings

  • Models specialize to their training environments, and a 5-gram reproduces much of the cross-child pattern; mixing children improves pooled prediction more clearly than grammatical performance.Source: §1 · p. 2

2 Related work

Prior work studies language learning with limited data, scaling, and input distributions, but much evidence comes from different settings or much larger datasets. This study builds on developmental-scale work while varying the composition and match of input.

Developmental-scale language modeling

  • Developmental-scale work includes the BabyLM challenge and small models trained on child-directed, CHILDES-derived, or synthetic dialogue data.Source: §2 · p. 2
  • Earlier single-child studies and a BabyView study examined small datasets; this work adds matched budgets, more draws and seeds, cross-child comparisons, persistence, and diversity tests.It mostly holds the model and training pipeline fixed while varying input, following a controlled-rearing approach (Warstadt and Bowman, 2022).Source: §2 · p. 2

Scaling and data efficiency

  • Most scaling-law evidence comes from datasets far larger than child-scale data, where optimization and unseen vocabulary or constructions may matter more.The paper characterizes scaling in the child-scale regime rather than proposing or testing new scaling laws.Source: §2 · pp. 2–3

Input distribution, domain match, and variation across children

  • Prior work links training–evaluation resemblance and training-data mixtures to downstream performance, while developmental studies document variation in children’s input.These findings motivate separating input amount from composition and evaluation match.Source: §2 · p. 3

3 Experimental framework

The framework compares naturalistic and synthetic corpora, small GPT-2-style models, and a count-based baseline across prediction and language benchmarks. It uses matched word budgets and replicated analyses, while treating the evaluations and comparisons as specific to their settings.

3.1 Training corpora

  • BabyView supplies about 4.67M filtered English words; TinyDialogues supplies synthetic age-2 child-directed dialogue at training sizes up to 12.8M words.BabyView scaling uses nested subsets from 25k to 3.2M words plus a separately reported 4.67M-word corpus; supplementary TinyDialogues variants alter age composition or generator source.Source: §3.1 · p. 3
  • Individual-environment and diversity experiments use matched 100k-word corpora, varying child identity or the number of contributing children; a 250k-word diversity replication is secondary.The primary 100k-word budget permits matched single-child comparisons across 16 eligible children.Source: §3.1 · p. 3

3.2 Models and training

  • Scaling uses GPT-2-style micro and small models with corpus-specific tokenizers, while individual-environment, persistence, and diversity analyses use GPT-2 micro with a shared tokenizer.Models train for 20 epochs; fixed-recipe comparisons match sampled word occurrences, not optimization steps or compute.Source: §3.2 · p. 3

3.3 Kneser-Ney 5-gram baseline

  • The Kneser-Ney 5-gram is a local-word-statistics diagnostic, not a perfectly matched architecture ablation; vocabulary-aligned scoring addresses its different handling of unseen words.KN-5 maps unseen words to one unknown symbol, whereas BPE assigns probability to each word’s spelling.Source: §3.3 · p. 4

3.4 Evaluation

  • Evaluation covers held-out prediction, grammatical benchmarks, and lexical semantics, with absolute cross-corpus BPW differences not treated as controlled corpus effects.Zorro results are called grammatical-benchmark performance rather than direct grammatical competence; fixed-budget diversity focuses on prediction and grammaticality.Source: §3.4 · p. 4

3.5 Statistical analysis

  • Replicated runs use bootstrap, child-cluster, permutation, or HC1 procedures according to the analysis; power-law fit checks were specified before examining results.Source: §3.5 · p. 4

4 Individual environments leave strong distributional signatures

Models trained on one child’s environment predict that environment better than other children’s, and this specialization is largely lexical and distributional. It extends beyond the child’s own speech and remains detectable months later, while synthetic distributional differences can also produce it.

4.1 Own-environment prediction advantage

  • 0.77 BPW lower loss on same-environment held-out language than on other children’s environments; all 16 children show the advantage.The 95% interval is [0.62, 0.92].Source: §4.1 · p. 4

4.2 Beyond the child’s own speech

  • 0.775 BPW versus 0.769 BPW same-environment advantage when evaluation uses only other people’s speech versus all speech; removing the child’s speech from training worsens prediction by 0.26 BPW.The removal comparison is across 10 children, while Zorro performance is essentially unchanged.Source: §4.2 · p. 5

4.3 Lexical and distributional differences

  • About 95% of the specialization effect comes from words unseen in training, although they make up only about 6% of evaluation words.Unseen words include names, household-specific terms, rare objects, and transcription artifacts; the study does not fully separate these sources.Source: §4.3 · p. 5

4.4 5-gram reproduces cross-child pattern

  • r = .91 correlation between KN-5 and neural cross-child prediction patterns across 240 comparisons excluding same-child pairs.This shared pattern suggests local distributional statistics explain much of the structure; raw effect sizes are not directly compared because KN-5 uses a closed vocabulary.Source: §4.4 · p. 5

4.5 Specialization persists for months

  • ∼0.46 BPW same-environment advantage remains 6–12 months later, weakening by ∼0.04 BPW/month.Because time and child age increase together, the decline is not interpreted as a causal effect of time itself.Source: §4.5 · p. 6

4.6 Distributional differences & specialization

  • As synthetic source distributions become more different, the same-environment advantage increases; at similar distributional differences, its overall size can resemble BabyView’s.Synthetic specialization is less concentrated in unseen vocabulary, and distributional difference does not fully predict the advantage across manipulations.Source: §4.6 · p. 6

5 Diversity improves pooled prediction more clearly than grammar

At a fixed 100k-word budget, adding children’s environments improves prediction of pooled held-out language, with gains tracking distributional match. The grammatical-benchmark effect is smaller and less consistent.

5 Diversity improves pooled prediction more clearly than grammar

  • Pooled held-out prediction improves substantially as training data include more children; BPW decreases by about 0.124 for each unit increase in ln k.Unigram Jensen-Shannon divergence explains more performance variation than the number of contributing children alone (R2 = .93 versus .67).Source: §5 · p. 6
  • At 100k words, Zorro shows no clear trend with the number of contributing children (p ≈.28), and per-token Zorro and BLiMP are likewise null.Thus, the clear pooled-prediction improvement does not have a similarly clear grammatical-benchmark counterpart at this budget.Source: §5 · pp. 6–7
  • At 250k words, per-token Zorro is positive (p = .044), but primary Zorro is weaker (p = .085) and BLiMP remains null.The authors therefore conclude that grammatical benefits are smaller and less consistent, not that diversity never helps grammatical generalization.Source: §5 · p. 7

6 Learning across data sizes and sources

Naturalistic BabyView and synthetic age-2 TinyDialogues show different benchmark patterns, while grammatical differences vary by phenomenon. Count models remain competitive at low data, and the tested low-data curves do not pass the prespecified single-power-law fit checks.

6.1 Scaling differs by data source

  • BabyView generally scores higher on Zorro, while TinyDialogues often performs better on word-similarity benchmarks; the fixed common-word-pair difference is −0.057.That BabyView-minus-age-2 WordSim difference is for GPT-2 small across 800k–3.2M words; at 25k words, BabyView reaches 59.4% on Zorro versus 48.4% at matched random initialization.Source: §6.1 · p. 7

6.2 Grammar differences by phenomenon

  • Across 800k–3.2M words, NPI licensing accounts for much of BabyView’s Zorro advantage (about 32 percentage points), but TinyDialogues leads on several other categories.The KN-5 comparison reverses the NPI difference in favor of age-2 TinyDialogues; the neural advantage is concentrated in only/even, and the authors report no full mechanistic explanation.Source: §6.2 · p. 7

6.3 Count models stay competitive at low data

  • GPT-2 small overtakes KN-5 between 800k and 1.6M words on BabyView and between 200k and 400k on age-2 TinyDialogues.With validation-tuned training, the BabyView crossover moves to 400k–800k words, while the age-2 range is unchanged.Source: §6.3 · p. 7
  • A single standard power law fails the prespecified fit checks for the tested low-data learning curves, a result limited to these models and fitting procedure.The curves improve overall with data; this is not a rejection of power-law scaling more generally.Source: §6.3 · p. 7

7 Discussion

The discussion argues that environmental match and broader linguistic generalization need not move together: specialization is substantial, yet much is concentrated in a small fraction of evaluated words. The authors also stress that source-specific benchmark patterns do not establish a generally superior training source or explain how children generalize.

7 Discussion

  • Specialization is concentrated in a small fraction of evaluated words and is largely reproduced by a simple count model, while grammatical-benchmark differences among children are comparatively small.This suggests environmental match and broader linguistic generalization need not vary together.Source: §7 · pp. 7–8
  • BabyView-trained models generally perform better on Zorro, while TinyDialogues often performs better on word-similarity benchmarks; neither source is established as generally better.The 5-gram baseline does not reproduce the neural grammatical pattern, so simple local word statistics cannot explain all source differences.Source: §7 · p. 8
  • The experiments do not test the proposed explanation that child generalization may require stronger inductive biases or regularization together with social, sensory, and multimodal information.They instead identify distributional differences a theory of child language learning must address.Source: §7 · p. 8
  • These results do not show that GPT-2 learns as children do; future work could test unseen-child transfer, larger or compute-matched models, and grounded video and audio.The authors also propose separating names and transcription artifacts from other unfamiliar words.Source: §7 · p. 8

8 Conclusion

The conclusion is that small-data language-model performance depends on which environments supply the words, not only on the amount of data. Input diversity and match to the evaluation language also matter.

8 Conclusion

  • Models specialize strongly to individual children’s environments, especially in vocabulary and other distributional features, and this specialization remains detectable months later.Including more children at a fixed training budget improves pooled child-language prediction without a similarly clear grammatical-benchmark gain.Source: §8 · p. 8
  • Naturalistic BabyView and synthetic dialogue support different grammatical and lexical-semantic learning patterns as training data increase.Simple count models help identify which patterns can be explained by local statistics.Source: §8 · p. 8
  • The content and diversity of input, and its match to the language being predicted, matter alongside data quantity.The conclusion frames these findings as evidence that quantity alone is not enough to describe learning with small datasets.Source: §8 · p. 8

Limitations

The findings are limited to the models, datasets, and evaluations studied, and should not be read as modeling children’s full language input or learning. The diversity and scaling comparisons also have specific boundaries in their data and evaluation settings.

Limitations

  • The findings cover only the models, datasets, and evaluations studied, not children’s full language input or human linguistic knowledge.BabyView samples only part of children’s language and multimodal experience; even the pooled 4.67M-word corpus is smaller than estimates of children’s cumulative exposure, and the individual-child experiments use 100k words.Source: Limitations · p. 9
  • GPT-2-style models are general statistical learners, not cognitive models of how children themselves learn language.A comparison with children’s learning might still be informative.Source: Limitations · p. 9
  • Scaling conclusions are specific to the tested recipes, models, and data sizes; the largest synthetic sets lack matched BabyView counterparts.TinyDialogues reflects its generation setup, and the diversity experiment tests pooled held-out language from observed children, not transfer to a new household.Source: Limitations · p. 9

Ethical Considerations

The study uses existing BabyView recordings under approved consent and ethics procedures and collects no new participant data. It avoids reproducing identifying information; TinyDialogues is entirely synthetic.

Ethical Considerations

  • The analysis uses existing BabyView data collected under approved consent and ethics procedures, with no new participant data collected or identifying information reproduced.TinyDialogues is entirely synthetic.Source: Ethical Considerations · p. 9

AI Use Statement

The analyses test how training recipes, vocabulary alignment, corpus differences, and evaluation choices shape small-model results. They report persistent own-environment prediction advantages, while documenting limits in power-law fit adequacy and uncertainty estimates.

AI Use Statement

  • The authors used generative AI for experimental planning, implementation, analysis, literature search, and writing assistance, and reviewed and verified AI-assisted outputs.GPT-2 micro scaling plots use one corpus draw and three seeds for the full-corpus BabyView anchor; larger age-2-only extensions have no naturalistic counterpart.Source: AI Use Statement · p. 9, H Raw Scaling for GPT-2 Micro · p. 15

B Power-Law Fit Diagnostics

  • No core BabyView or age-2 TinyDialogues power-law fit passed all five prespecified criteria, so the standard power law did not adequately summarize these observed curves under this stringent test.For BabyView micro, weighted R2 was 0.965 and AICc was 71.9 for the power law versus 58.5 for the log-linear alternative; this does not rule out power laws under other setups or fitting procedures.Source: B Power-Law Fit Diagnostics · pp. 11–12

C KN-5 Vocabulary Alignment and Grammar Controls

  • Vocabulary alignment scores GPT-2 and KN-5 on identical held-out in-vocabulary units; all four primary-checkpoint crossover ranges were unchanged, while BabyView small’s final-checkpoint aligned crossover was 800k–1.6M words rather than 1.6M–3.2M.KN-5 has a closed vocabulary, whereas GPT-2 models byte-level spellings and may use longer context.Source: C KN-5 Vocabulary Alignment and… · p. 12
  • On Zorro, the pooled BabyView-minus-age-2 TinyDialogues contrast was about +1.69 percentage points for KN-5 (95% interval [+0.34, +2.96]), versus about +4.51 for neural models.For NPI licensing, the comparison reversed: about −13.9 points for KN-5 versus +32.4 for GPT-2; the planned item-level lexical-frequency explanation was not identifiable, and the measurable neural effect was concentrated in the only/even subset.Source: C KN-5 Vocabulary Alignment and… · p. 12, Figure 6 · p. 13

D Cross-Child Specialization Checks

  • 0.769 BPW (95% interval [0.623, 0.921]) was the all-speech own-environment advantage across 16 children, and unseen words accounted for about 94.8% of the non-target-speech advantage.GPT-2 and KN-5 cross-child prediction patterns correlated at r = .908 after excluding same-child comparisons.Source: D Cross-Child Specialization Checks · pp. 12–13, Figure 7 · p. 14

E Statistical Procedures and Analysis-Plan Provenance

  • Scaling uncertainty intervals resampled whole corpus-draw trajectories and then model seeds, but only three corpus draws limited bootstrap precision; multiple comparisons were not corrected.The full-corpus BabyView anchor had one fixed corpus draw and three seeds, so its range reflects seeds only.Source: E Statistical Procedures and… · p. 13

F Persistence: Full Analysis and Replication

  • The own-environment advantage declined by −0.0376 BPW/month (95% interval [−0.0575, −0.0218]); chronological-prefix replication yielded −0.0321 BPW/month (95% interval [−0.0463, −0.0199]).The analyses cannot distinguish elapsed-time effects from changes associated with age, or both.Source: F Persistence: Full Analysis and… · p. 14, Figure 8 · p. 15

G Synthetic Differences and Specialization

  • Synthetic specialization increased monotonically with the primary manipulation, but interlocutor-only variation yielded about 0.976 BPW at JSD 0.0542, roughly 4.1 times the comparable main-series value.BabyView’s 0.769 BPW advantage at mean JSD 0.106 was compatible with the primary synthetic relationship, but was more concentrated in unseen vocabulary; synthetic environments do not capture all properties of real children’s environments.Source: G Synthetic Differences and… · p. 15, Table 3 · p. 15

I.1 WordSim Evaluation Sensitivity

WordSim contrasts between training sources vary with benchmark and evaluation procedure, while supplementary comparisons show that age composition and data diversity relate differently to prediction and grammatical benchmarks.

I.1 WordSim Evaluation Sensitivity

  • Final-layer, mean-subtoken pooling is the fixed WordSim rule; best-layer selection is optimistic and raises an untrained score from −0.007 to 0.028.The archived scores do not isolate last-subtoken from mean-subtoken pooling.Source: §I.1 · pp. 15–16, Table 4 · p. 16
  • −0.017 mean Spearman correlation separated BabyView from age-2 TinyDialogues across 800k–3.2M words; on shared-vocabulary pairs, the difference was −0.057.BabyView was also lower than mixed-age TinyDialogues by −0.046 across all pairs and −0.074 on the shared-vocabulary subset; these benchmark contrasts do not establish that BabyView is worse for learning word meanings.Source: §I.1 · p. 16

I.2 Random initialization and 25k-word Zorro

  • +33.5 percentage points for subjective pronoun case contrasted with −13.7 points for ellipsis in BabyView’s 25k-word gains over initialization.Other selected gains included negative-polarity-item licensing (+29.5) and local attractors (+26.0), while subject–verb agreement (+0.1), determiner–noun agreement (approximately zero), and irregular forms (−0.8) remained near chance; these are descriptive, not evidence of broad grammatical-rule acquisition.Source: §I.2 · p. 17

J Age-Composition Comparisons (Supplementary)

  • +0.061, +0.055, and +0.063 mean Spearman differences favored mixed-age over age-2 TinyDialogues at 3.2M, 6.4M, and 12.8M words.Mixed-age Zorro differences were +3.77 points at 3.2M, +2.74 at 6.4M, and +4.55 at 12.8M; held-out prediction depended on the evaluation distribution. These synthetic-data comparisons do not establish whether mixed-age input is better or worse than naturalistic input.Source: J Age-Composition Comparisons… · pp. 17–18, Figure 11 · p. 18

K Fixed-Budget Diversity: Supplementary Tests

  • −0.1236 BPW per unit ln k was the pooled held-out prediction slope at 100k words; the primary Zorro slope was +0.2003 percentage points per ln k.At 250k words, pooled BPW also improved with additional children (slope −0.1465; p = 0.0001), while primary Zorro evidence was less clear and BLiMP showed no clear effect; these relationships do not establish causation or generalization to new children.Source: K Fixed-Budget Diversity:… · p. 18

L Child-Speech Removal Control

  • Removing a child’s speech worsened pooled held-out prediction by +0.257 BPW versus random removal, but Zorro and BLiMP accuracy were similar.The pooled difference occurred for all 10 children (exact signflip p = 0.002); grammatical-benchmark differences were not significant.Source: L Child-Speech Removal Control · pp. 18–19

M Developmental Outcomes (Supplementary)

  • Adding model word-learning order did not meet the predefined Wordbank improvement criterion, and its CDI advantage disappeared after accounting for frequency.Wordbank gave ∆AIC = +0.66 against an improvement criterion below −2; for CDI, the own-environment advantage was ∆AUC = 0.0229, but frequency alone performed slightly better and adjustment reversed the advantage (∆AUC = −0.0147).Source: M Developmental Outcomes (Supplementary) · p. 19

N Computational Resources and Implementation Details

  • The experiment ledger estimates approximately 429.5 GPU-hours for training and GPU-based evaluation: 411.8 in paper-contributing experiment families and 17.7 in pilots or other investigations, with process-time estimates potentially overcounting overlapping GPU work.Source: N Computational Resources and… · p. 19, Table 6 · p. 19
Loading 2610.36679v1…