PaperlayerView exact revision on arXiv

Source-linked AI summary

Children, but not language models, show accelerating returns in word learning

Michael C. Frank

arXiv:2608.17120v1cs.CL

TL;DR

The key question is whether children’s vocabulary growth accelerates beyond ordinary accumulation and whether language models show the same pattern. The study compares learning returns on a common metric and finds that children’s returns accelerate while language models’ remain constant-proportional.

Abstract

from arXiv · show

Children learn hundreds of words over the first years of their lives, in a process that begins slowly but quickly picks up speed. Prior models describe vocabulary growth as evidence accumulation over time. Here we show that the process is best characterized as accelerating accumulation: children learn more from each additional unit of linguistic experience than they did from the one before. In contrast to children, language models -- even those trained on child-directed speech -- do not accelerate. Instead, they show constant proportional returns on new data, consistent with scaling laws. Children learn using many orders of magnitude less training data than language models; their increasingly efficient use of their learning input is a candidate explanation.

An accelerating accumulator model of vocabulary growth

The section proposes an accelerating accumulator model in which children’s later language input contributes more to vocabulary learning than earlier input. It formalizes learning with child ability and word difficulty, while allowing children to differ in baseline efficiency and acceleration.

An accelerating accumulator model of vocabulary growth

  • The model treats each word as a bucket filled by tokens of input and represents production probability with a Rasch model of child learning ability and word difficulty.It uses ratio-scale measures of tokens heard per hour and total vocabulary types, which can be log-transformed because they have true zeros.Source: An accelerating accumulator model of… · p. 2
  • Earlier accumulator models recover interpretable differences in word “bucket size” and offer a graded account of vocabulary spurts, while supralinear vocabulary growth alone permits multiple mechanisms.Word difficulty can reflect properties beyond frequency, so the current work estimates difficulty freely and focuses on variation between children.Source: An accelerating accumulator model of… · p. 2
  • When κ > 1, later language input contributes more to learning than earlier input; κ = 1 instead gives the pure accumulator’s linear returns on experience.The model describes ability with a baseline-efficiency intercept, an acceleration term over developmental time, and a constant for waking hours.Source: An accelerating accumulator model of… · p. 2
  • The proposed model lets children vary in baseline accumulation efficiency and acceleration, extending longitudinal growth models with variation in both intercept and slope.The formulation is close to earlier models but had not previously been proposed (8, 9, 28).Source: An accelerating accumulator model of… · pp. 2–3

Fitting the accumulator model

The accelerating accumulator best fit children’s vocabulary growth across five longitudinal datasets, with estimates consistently rejecting pure accumulation. Its population acceleration improved prediction for most children when enough prior CDI administrations were available.

Fitting the accumulator model

  • The analysis fit Bayesian models to item-level longitudinal reported-production CDI data from 1,841 monolingual, typically developing children aged 8–36 months across five datasets.The datasets covered American English, Norwegian, and Japanese; only children with at least three longitudinal administrations were included.Source: Fitting the accumulator model · p. 3
  • Acceleration estimates ranged from 10.6 [10.5, 10.8] to 13.3 [13.2, 13.3], robustly rejecting the pure-accumulator value κ=1 in every dataset.Estimates remained essentially unchanged with a larger dataset and in a hierarchical model pooling across datasets.Source: Fitting the accumulator model · p. 3
  • A 2PL IRT model fit better, but κ remained far above 1; CDI item selection had minimal effects on κ, and linear-growth models did not fit better.Word-level age-of-acquisition estimates were very similar under the 2PL IRT model.Source: Fitting the accumulator model · p. 3
  • Between-child acceleration standard deviations ranged from 3.2 to 6.6, and 99.5% of children were estimated above κ=1; acceleration improved prediction for 91.9% with at least five prior CDIs.Typical CDI series were too sparse to reliably predict an individual child’s acceleration beyond the population mean.Source: Fitting the accumulator model · p. 3
  • LOO ELPD > 760 for all five datasets favored the accelerating accumulator over the next-best model, whose fits closely matched non-parametric GAMLSS curves.Figure 1B shows the fitted trajectories across datasets; the model’s fit also closely tracked GAMLSS beta-regression curves used for CDI percentile norms.Source: Fitting the accumulator model · p. 3, Figure 1 · p. 4

Accelerating accumulation implies less exposure is needed for later-acquired words

The accelerating accumulator implies that children need fewer exposures to learn words as they age. The decline is exponential within word classes, unlike the pure accumulator’s much shallower prediction.

Accelerating accumulation implies less exposure is needed for later-acquired words

  • Within each word class, estimated exposures before learning decline exponentially with age, whereas the pure accumulator predicts a far shallower decline and delays many early words until adolescence.Figure 2 plots estimated cumulative exposures against the age when 50% of children produce each word.Source: Figure 2 · p. 5, Accelerating accumulation implies less… · p. 6
  • A toddler may hear “book” thousands of times before producing it, while a preschooler may repeat “ibex” after hearing it a handful of times.These examples illustrate the changing number of exposures associated with word learning over childhood.Source: Accelerating accumulation implies less… · p. 5
  • The estimated efficiency gain may be understated because the analysis assumes every word exposure is equally meaningful, although exposures vary in information.Words may be overheard rather than directed to the child, or lack physical context that supports clear inference.Source: Accelerating accumulation implies less… · p. 6

Language models do not accelerate

Language models showed no accelerating returns in either word-level learning or learner-level returns, unlike children, whose proportional returns increased with additional input.

Language models do not accelerate

  • Language models had constant proportional returns on new data, while individual children’s proportional returns increased.Proportional return measures how much of the remaining possible loss is removed by a given amount of data.Source: Figure 3 · p. 7, Language models do not accelerate · p. 8
  • The comparison linked vocabulary growth to scaling laws, which model language-model loss as entropy plus a power-law term in dataset size.The loss was expressed as L = E + BD^β.Source: Language models do not accelerate · p. 8
  • For each word, the analysis fit a sigmoid to its surprisal trajectory across training; the slope, in κ units, measured how quickly evidence accumulated.This linked the language-model analysis to the logistic curves used in the child Rasch model.Source: Language models do not accelerate · p. 8
  • Per-word κ estimates clustered around 1 in GPT-2 trained on 24M words of child-directed speech, and developmental models trained on 0.5M–24M words likewise showed no acceleration.The developmental models were each trained to convergence on increasingly large data subsamples.Source: Figure 3 · p. 7, Language models do not accelerate · p. 8
  • Children showed far greater κ variation than language models, even when language-model training data were varied.The word-level comparison found high κ estimates in children, in contrast to language models’ lack of acceleration.Source: Figure 3 · p. 7, Language models do not accelerate · p. 8

Discussion

Children’s word learning accelerates over development, whereas language models show constant proportional returns on additional experience. The discussion considers possible developmental and learning-to-learn mechanisms, while emphasizing limits in the evidence and the need to align comparisons with language models.

Discussion

  • Later in development, children need fewer exposures to learn new words, while language models—even child-data-trained models—show constant proportional returns.On a shared metric, children’s returns increase with additional experience; language models’ do not.Source: Discussion · p. 8
  • Changes in learning, memory, and processing speed could contribute to acceleration by helping older children extract more signal from each utterance (24, 37–39).In the model’s terms, older children’s “buckets” require fewer drops to fill; faster processing is linked to growing vocabulary (24, 39).Source: Discussion · pp. 8–9
  • Children accelerate by “learning to learn”: using the language they know to become more effective learners; their increasingly sophisticated inferential abilities let them extract progressively more information from the linguistic signal (40, 41).Source: Discussion · p. 9
  • Changes in input quantity or ordering are unlikely to explain acceleration, but richer input or child-caregiver interaction could contribute and merits study (44–46).Parents do not typically fine-tune language to children’s developmental level, and age- or simplicity-based curricularization has not increased language-model performance.Source: Discussion · p. 9
  • Language models lack children’s developmental maturation; in-context learning typically requires vastly more data, and rapid few-shot word learning may arise through meta-learning rather than pretraining (47–49).Establishing linking hypotheses to compare and align children and language models is a critical goal for future work (50).Source: Discussion · p. 9
  • The accelerating accumulator fits children’s vocabulary growth, but evidence comes from statistical fit rather than causal manipulation, and parent-report CDI measures also reflect verbal production (1, 18, 19).Testing the accumulator as a mechanism and linking it to direct vocabulary measures are important next steps.Source: Discussion · p. 9

Supplemental Information · Materials and Methods

The methods combine longitudinal vocabulary datasets, hierarchical Bayesian accumulator models, and word-level language-model surprisal curves to compare returns on input and acquisition slopes.

Vocabulary data

  • All longitudinal datasets with at least three observations per child were included, and a quality filter excluded 36 children (1.9%).The filter removed administrations with vocabulary decreases above 25% or increases exceeding 40% of total vocabulary per month; nearly all exclusions came from English (Marchman).Source: Vocabulary data · p. 16

Bayesian models

  • Bayesian models were fit separately to each dataset, with individual child variation around population means and convergence requirements of R̂ < 1.01 and bulk ESS above 400.Fits used 4,000 post-warmup draws; priors were weakly informative, with acceleration centered around the pure accumulator model.Source: Bayesian models · p. 16

Learning efficiency

  • Average monthly exposures for each word were estimated as N_j = R · t50(j) · p_j, combining input rate, age of acquisition, and corpus word probability.Age of acquisition used weighted word difficulties from the three English M3 models, and word probabilities came from North American English CHILDES corpora.Source: Learning efficiency · p. 16

Language model training

  • Language models were trained on 24M-word CHILDES, BabyLM, and ClimbMix data, with surprisal measured for 609 CDI words across held-out contexts.Training included children’s own utterances; analyses varied training runs, dataset sizes, and dataset composition, while using a shared BPE tokenizer.Source: Language model training · p. 17

Proportional Return on Input

  • γLM = β ≈ 0.32 at every data budget for language models, whereas children’s proportional return rises toward each child’s κ_i as vocabulary saturates.The language-model return is the share of remaining reducible loss removed per e-fold of data; the child measure is defined from vocabulary production loss.Source: Proportional Return on Input · pp. 17–18

Computing Acceleration in LMs

  • The child word-acquisition logit slope equals κ_i, while the language-model slope is 1/(scale_w ln 10) per natural-log input, enabling comparison on a common scale.Language-model slopes were estimated by fitting four-parameter logistic curves to each word’s surprisal across training checkpoints.Source: Computing Acceleration in LMs · p. 18

Supplementary Text … Datasets

The supplement relates prior accumulator models to the paper’s framework, explains why cross-sectional data cannot identify acceleration, and describes the datasets and quality exclusions used in the analyses.

Comparison with Prior Models

  • The accelerating accumulator decomposes child-specific ξ into input rate r and learning efficiency α, placing closely related prior models within a shared framework.The paper notes that its exact model form has not previously been described in the literature.Source: Comparison with Prior Models · p. 18
  • Pure accumulator (20) corresponds to M0; (8) adds acceleration through exponent κ and maps to M1, while (9) varies α but not κ, mapping to M2.Source: Comparison with Prior Models · p. 18
  • Leverage model (60) changes vocabulary-growth shape and timing but cannot create acceleration absent from word-difficulty distributions; (21) maps to M0 plus the fitted 2-parameter logistic variant.Source: Comparison with Prior Models · pp. 18–19

Non-identifiability in Cross-Sectional Data

  • A single cross-sectional observation cannot identify a child’s acceleration because it may reflect either efficiency ξ or acceleration κ differences.Figure 4 illustrates the same simulated data as either stable between-child intercept differences or varying growth slopes.Source: Non-identifiability in Cross-Sectional… · p. 19, Figure 4 · p. 19

Datasets

  • Table 1 reports dataset characteristics; the main analyses use children with 3+ administrations for more precise κ estimates, with robustness also checked using 2+ administrations.Source: Datasets · p. 19, Table 1 · p. 20
  • Quality checks flagged and excluded some observations, speculated to reflect data-entry or correspondence errors common in archival data; the paper describes their effect as small.Figure 5 marks flagged administrations and children excluded entirely in red.Source: Datasets · p. 20, Figure 5 · p. 20

Full Model Comparison Results · Individual-level Cross-validation · Comparison to a Non-Parametric Quantile Model

M3 is the best model in the full leave-one-out comparison. Prospective prediction results depend on training depth, while M3 tracks non-parametric and empirical quantiles across five datasets.

Full Model Comparison Results

  • Table 2 compares leave-one-out ELPD across models against M3, identified as the best model, for both datasets.The datasets include children with at least three administrations and the larger sample with at least two.Source: Full Model Comparison Results · p. 20, Table 2 · p. 21

Individual-level Cross-validation

  • At k=2, M2 predicts better, but M2 and M3 tie by k=5; with two observations, M3’s per-child slope absorbs measurement error.The prospective test fit each child’s latest k observations and predicted the next administration three months later, using 371 Norwegian children with at least six longitudinal CDIs.Source: Individual-level Cross-validation · p. 21, Table 3 · p. 21
  • Because both M2 and M3 include population acceleration, this cross-validation does not test whether acceleration is present.It instead suggests that estimating each child’s acceleration requires sustained, precise longitudinal measurement.Source: Individual-level Cross-validation · p. 21
  • M2, with a free population acceleration exponent, predicted held-out administrations better than M2-zero at every depth.M2-zero fixes the exponent at κ=1 while retaining each child’s efficiency term, so the comparison tests whether acceleration predicts individual trajectories.Source: Individual-level Cross-validation · p. 22, Table 4 · p. 22

Comparison to a Non-Parametric Quantile Model

  • Across all five datasets, M3 tracks the non-parametric GAMLSS fit and empirical quantiles, providing a low-dimensional compression of the data’s quantile structure.The comparison fit GAMLSS beta-regressions with penalized splines on the mean and scales.Source: Comparison to a Non-Parametric… · p. 22, Figure 6 · p. 22

Robustness Across Model and Data Settings · Sensitivity to Data-Quality Exclusions · Linear vs. Logarithmic Age

Across five datasets, linear-age acceleration models were dispreferred to the log-age model, while estimates of acceleration remained robust across model settings and data-quality exclusions. The pooled model closely reproduced independent-fit parameters, and changing exclusion filters did not substantially change estimates in the two affected datasets.

Robustness Across Model and Data Settings

  • Acceleration estimates were substantially similar for separate fits with 2+ administrations and a hierarchical fit with 3+ administrations, relative to the main 3+ separate-fit setting.Table 5 presents LOO fits and acceleration estimates across these settings.Source: Robustness Across Model and Data… · pp. 22–23, Table 5 · p. 23
  • The pooled hierarchical model reproduced the independent-fit parameters closely while estimating each dataset’s mean efficiency and acceleration.Children and words were nested within dataset, with each dataset assigned its own mean ξ and κ.Source: Robustness Across Model and Data… · p. 23
  • In the pooled model, dataset-level variation in efficiency and acceleration was drawn toward typical values for datasets with few children, while remaining essentially unconstrained for larger datasets.Item difficulty was estimated separately within each dataset because the datasets represent different languages.Source: Robustness Across Model and Data… · p. 23

Sensitivity to Data-Quality Exclusions

  • Changing exclusion filters did not substantially change acceleration estimates in the English (Marchman) and Norwegian datasets.The refits compared no filter, loose, reported, and tight settings. The loose filter used 40% decline and 60 percentage-point monthly-rise cutoffs; the main filter used 25% and 40%, and the tight filter used 15% and 25%.Source: Sensitivity to Data-Quality Exclusions · p. 23, Table 6 · p. 24
  • The exclusion-filter sensitivity analysis covered four settings, but only the English (Marchman) and Norwegian datasets, where the filter removes observations.The settings were no filter, loose, reported, and tight; the table specifies the thresholds for the loose, main, and tight filters.Source: Sensitivity to Data-Quality Exclusions · p. 23, Table 6 · p. 24

Linear vs. Logarithmic Age

  • The linear-age model was dispreferred across all five datasets compared with the log-age model.Table 7 reports the LOO ELPD advantage of log-age M3 over its linear-age counterpart; positive values favor log-age.Source: Linear vs. Logarithmic Age · p. 24, Table 7 · p. 24

2PL Model Comparison

Supplementary analyses test whether the acceleration results depend on item-model specification, item selection, input assumptions, or estimator choice. They support the children–language-model contrast, while the pure accumulator fails to reproduce children’s age-of-acquisition pattern.

2PL Model Comparison

  • The 2PL fit better than the 1PL in every language, but other model results were unchanged and per-word acquisition ages closely aligned.Difficulty–discrimination correlations were small to modest, providing little support for the concern that they inflate 1PL acceleration estimates.Source: 2PL Model Comparison · pp. 24–25, Table 8 · p. 25

Sensitivity to Item Selection

Variation in Acceleration

  • Across datasets, children’s fitted acceleration appeared right-skewed, although the generally normal-looking parameter distributions may partly reflect the model’s random-effects structure.The authors caution against strongly interpreting the apparent distributional shape.Source: Variation in Acceleration · p. 26, Figure 8 · p. 27

Input Estimation

  • Assuming constant input, the calibration estimates 561K tokens per month; measured input-rate slopes over age were shallow and non-significant across four datasets.A child one standard deviation above or below the median input rate hears ×1.71 or ÷1.71 as many tokens by the same age, shifting the curve horizontally on the log-token axis.Source: Input Estimation · p. 28, Figure 9 · p. 29

Age of Acquisition in the Pure Accumulator Model

Matched Estimation of Acceleration

Other Architectures

  • Per-word acceleration distributions in four architectures trained on standard language-model corpora resembled one another, indicating the reported results are not specific to GPT-2 (34).The architectures were BERT, GPT-2, BiLSTM, and LSTM.Source: Other Architectures · p. 32, Figure 12 · p. 32

Related papers

Loading 2608.17120v1…