PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Critical Period Effects in Language Acquisition through Neural Language Models

Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell, Alex Warstadt

arXiv:2407.19325v2cs.CL

TL;DR

The paper asks whether critical-period effects in language acquisition arise from innate maturation or from experience-driven learning. It tests language models by varying L2 exposure timing and adding a plasticity-reducing regularizer; the models lack the effects without that intervention, while both emerge when it is imposed.

  • Problem

    It is unclear whether critical-period effects reflect innate brain maturation or experience-driven stabilization, and language models offer a way to test whether the effects extend beyond humans.

    Source: §1 · p. 1

  • Method

    The authors train language models from scratch on bilingual data with varied L2 exposure timing, then add Elastic Weight Consolidation partway through training to simulate reduced plasticity.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    Language models do not show both tested critical-period effects without an imposed plasticity reduction, but both emerge when it is introduced.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings challenge the claim that critical-period effects necessarily follow from statistical learning and provide only weak evidence for an innate mechanism.

    Source: §1 · p. 2, §8 · p. 13

    The experiments neither strongly support the strong innate claim nor establish EWC as a plausible mechanism for human plasticity loss; other statistical learners could show effects through experience.

    Source: §7.2 · p. 12

Abstract

from arXiv · show

Humans appear to have a critical period (CP) for language acquisition: Second language (L2) acquisition becomes harder after early childhood, and ceasing exposure to a first language (L1) after this period (but not before) typically does not lead to substantial loss of L1 proficiency. It is unknown whether these CP effects result from innately determined brain maturation or as a stabilization of neural connections naturally induced by experience. In this study, we use language models (LMs) to test the extent to which these phenomena are peculiar to humans, or shared by a broader class of language learners. We vary the age of exposure by training LMs on language pairs in various experimental conditions, and find that LMs, which lack any direct analog to innate maturational stages, do not show CP effects when the age of exposure of L2 is delayed. Our results contradict the claim that CP effects are an inevitable result of statistical learning, and they are consistent with an innate mechanism for CP effects. We show that we can reverse-engineer the CP by introducing a regularizer partway through training to simulate a maturational decrease in plasticity. All in all, our results suggest that L1 learning on its own may not be enough to induce a CP, and additional engineering is necessary to make language models more cognitively plausible.

1 Introduction

The paper uses language models to test whether critical-period effects follow from experience alone or require human-specific mechanisms. Models do not show the tested effects without an imposed reduction in plasticity, but both emerge when one is introduced.

1 Introduction

  • The experiments train models from scratch on bilingual data while varying the age of L2 exposure to test L2 acquisition and L1 attrition.The paper asks whether LMs acquire L2 more easily with simultaneous L1 exposure and whether they retain L1 after extensive training on it.Source: §1 · pp. 1–2
  • Without an imposed plasticity reduction, language models differ from humans on both tested critical-period effects, challenging the view that these effects necessarily follow from statistical learning.The findings are consistent with, but provide only weak evidence for, a biologically programmed developmental stage.Source: §1 · p. 2
  • With Elastic Weight Consolidation added partway through training, both critical-period effects emerge together, simulating a loss of neural plasticity.The regularizer is introduced to reverse-engineer the effects in models.Source: §1 · p. 2
  • Transfer training from L1 to L2 yields similar or better performance than joint training on both languages, rather than the human-like disadvantage expected from delayed L2 exposure.The paper presents this as a novel conclusion from its experiments.Source: §1 · p. 2

2 Background: The Critical Period

The critical period includes distinct effects on L2 acquisition and L1 attrition, which this study examines separately. Competing accounts attribute reduced plasticity with age either to innate mechanisms or to experience-driven learning dynamics.

2 Background: The Critical Period

  • The critical period comprises effects on L1 acquisition, L2 acquisition, and L1 attrition; this study focuses on the latter two.The authors note that simulating late L1 exposure would require training on non-linguistic data, so it is outside this work’s scope.Source: §2 · p. 2, §2.1 · p. 2

2.1 Critical Period for L2 Acquisition

  • For L2 acquisition, increasing age of exposure is associated with greater learning difficulty and worse outcomes, while controlled computational tests of this effect had not previously been conducted.Prior work on transfer learning did not manipulate age of exposure while controlling total input and focused mainly on tasks other than language modeling.Source: §2.1 · pp. 2–3

2.2 Critical Period for L1 Attrition

  • L1 attrition is loss of L1 proficiency through lack of exposure, typically constrained to earlier ages; neural networks can also lose original-domain proficiency through catastrophic forgetting.Continual-learning approaches include adapters, regularizers, or further training in the original domain.Source: §2.2 · p. 3

2.3 Theories

  • Innate accounts link age-related plasticity loss to language-specific or neurobiological mechanisms, whereas experiential accounts treat it as a consequence of learning and network convergence.Experiential accounts include the proposal that training makes units less prone to change over time.Source: §2.3 · pp. 3–4

3 The Role of LMs in Studying the CP

Language models let researchers control learning mechanisms and environments in ways unavailable with human subjects. This study uses that leverage to test theories of critical-period effects, while positioning its reverse-engineering effort at the computational level.

3 The Role of LMs in Studying the CP

  • Computational models offer control over learning mechanisms and environments, but their differences from humans limit how directly findings bear on human learning.The authors describe their reverse-engineering effort as computational-level work; the resulting models more closely resemble human learners in the relevant property.Source: §3 · pp. 4–5
  • Current LMs can provide evidence about some theoretical claims, but they differ from humans and cannot by themselves settle every account of language acquisition.The paper frames reverse-engineering human-like properties as a way to make models more relevant to human learning.Source: §3 · pp. 4–5
  • The strong innate claim says innate constraints are necessary for critical-period effects, while the strong experiential claim says those effects necessarily follow from successful statistical learning.The latter can be challenged by a successful connectionist learner that does not show the effects.Source: §3 · p. 4

4 Research Questions and Methodology

The study tests critical-period effects by varying L1 and L2 exposure schedules and evaluating model performance. It also examines whether reducing plasticity induces the effects and whether results depend on language similarity.

4 Research Questions and Methodology

  • The main experiments vary when models acquire L1 and L2, then evaluate both languages without changing model architecture or learning objectives.The research questions also ask whether reduced plasticity induces the effects and whether they depend on language similarity.Source: §4 · p. 5
  • The similarity analysis repeats the experiments across language pairs that differ in similarity, assessing robustness to the language-pair choice.It also examines the relationship between language similarity and critical-period effects.Source: §4 · p. 5

4.1 Training conditions

  • Five training schedules manipulate exposure timing while keeping datasets, per-language data size, and epochs consistent where possible.Total exposure is equal across conditions except for the monolingual and sequential-interleaved conditions.Source: §4.1 · p. 5
  • The schedules include simultaneous interleaved bilingual training, sequential L1-then-L2 training, and continued L1 exposure interleaved with L2.Sequential training abruptly halts L1 exposure; sequential-interleaved training continues using the same L1 dataset.Source: §4.1 · pp. 5–6

4.2 Enforcing Plasticity

  • Elastic Weight Consolidation adds a Bayesian-inspired regularizer partway through training, penalizing deviations from a prior defined by L1 training to simulate reduced plasticity.The regularization strength is controlled by λ; λ = 0 means no programmed decrease in plasticity.Source: §4.2 · p. 6

5 Experimental Setup7

The experiments train language models from scratch on bilingual language pairs, using controlled language choices, training conditions, and English proficiency evaluations. The setup uses multilingual data assembled from three domains, though the available datasets are not fully developmentally plausible.

Languages

  • Most experiments pair English with German or Finnish, chosen as relatively high-resource languages that are respectively related and unrelated to English.An additional experiment broadens the L1 pool across language families and scripts, including English and Java endpoints.Source: §5 · p. 6

Datasets

  • Training data combines OpenSubtitles, Gutenberg, and Wikipedia because no multilingual datasets fully represent the type and volume of language input children encounter.The authors judge these sources to balance diversity, quality, and quantity among limited public options, while acknowledging they are not fully developmentally plausible.Source: §5 · pp. 6–7

Models and training

  • The study trains 125M-parameter RoBERTa and 137M-parameter GPT-2 models from scratch; sequential conditions start L2 training from final L1 checkpoints with a new optimizer.A Bayesian hyperparameter search selects model configurations, and EWC strength is chosen to make L1 and L2 performance roughly equivalent at training's end.Source: §5 · p. 7

Evaluation

  • English proficiency is evaluated with character-normalized perplexity, zero-shot BLiMP grammatical contrasts, and fine-tuned GLUE and superGLUE tasks.Only English is used for evaluation because comparable benchmarks are unavailable across the studied languages; the authors prioritize controlled evaluation over exploratory breadth.Source: §5 · p. 7

6 Experiments

Across the experiments, monolingual training is the strongest baseline, while sequential L2 training can sharply reduce L1 performance. EWC can preserve L1 to some degree, but the results show a trade-off with L2 learning; language-relatedness effects are limited across evaluations.

Experiment 1: Regular training

  • Figure 2 summarizes L2 (English) perplexity and BLiMP during training, plus GLUE performance at the end, aggregated across model configurations and German/Finnish L1s.Training patterns are generally similar across conditions except interleaved training, with differences appearing mostly in final performance.Source: Figure 2 · p. 8, §6 · p. 8
  • Monolingual training performs best, while interleaved training is slightly worse than sequential training; reducing plasticity with EWC preserves L1 at a cost to final L2 performance.The selected EWC strength reduces L2 performance by roughly 1.5 nats when L1 is preserved to monolingual levels.Source: §6 · p. 8

Experiment 2: Reversing the language order

  • Sequential training rapidly erases L1 knowledge after L2 exposure begins, whereas continued L1 exposure in the sequential-interleaved condition mitigates L1 loss.In the monolingual condition, L1 performance continues improving through training; GLUE shows a smaller drop than BLiMP after L1 exposure ceases.Source: §6 · p. 9

EWC trade-off

  • EWC strength trades off L1 retention against L2 learning: strong regularization can maintain L1 near initial levels but almost completely impair L2, while weak regularization affects L2 less but fails to preserve L1.The selected λ values target roughly equivalent L1 and L2 performance rather than either extreme.Source: §6 · pp. 9–10

Experiment 3: Training to convergence

  • With 48 training epochs, sequential training still shows substantial L1 loss, while longer training does not significantly improve final L2 performance under EWC.The authors report that EWC appears to impose an L2 performance lower bound for a given λ rather than merely slowing L2 learning.Source: §6 · p. 10

Experiment 4: Diversifying the language pool

  • L2 perplexity generally worsens as L1 becomes less related to English, but the reported effect is small and BLiMP results do not show the same pattern consistently.The authors report that BLiMP supports a relatedness effect only under EWC, suggesting ordinary models remain plastic enough to learn L2 grammar across language choices.Source: §6 · p. 11

7 Discussion

The discussion finds that typical language models do not show the human pattern of delayed L2 learning or retained L1 under deprivation. It argues that these results challenge strong experiential accounts, while the models and training choices limit what can be inferred about humans.

7.1 Research Questions

  • L2 performance was worse or no different in INTERLEAVED than SEQUENTIAL models, while MONOLINGUAL models outperformed INTERLEAVED models on all three metrics.INTERLEAVED models also showed more gradual and delayed improvement than either comparison condition.Source: §7.1 · p. 11
  • After L1 exposure ceased, SEQUENTIAL models rapidly lost substantial L1 performance; continued L1 exposure prevented this loss, unlike the human pattern described.Self-talk could provide a source of L1 exposure even when external exposure ceases.Source: §7.1 · p. 11, §7.2 · p. 11
  • SEQUENTIAL-EWC preserved L1 substantially better than SEQUENTIAL training but harmed final L2 performance, with L2 curves converging relatively quickly after plasticity reduction.Preserving L1 to monolingual levels harmed L2 performance by roughly 1.5 nats; human-like performance was not quantitatively established.Source: §7.1 · p. 11
  • Language relatedness affected L2 learning as expected only on some evaluations: PPL supported the expected pattern, while BLiMP did so only with EWC.Without EWC, the authors suggest models remain plastic enough to learn L2 grammar regardless of relatedness.Source: §7.1 · p. 11

7.2 Theoretical Implications

  • The results challenge the strong experiential claim that successful learning algorithms necessarily show CP effects, but provide only weak support for an innate maturational account.The authors also allow that other statistical learners may develop CP effects through experience, and that humans could be such learners.Source: §7.2 · pp. 11–12

7.3 Limitations and Future Work

  • The models may not generalize to human learning: Transformers differ from humans, use transcribed speech rather than multimodal input, and do not fully match human BLiMP agreement.The authors report 75 ± 1% BLiMP accuracy for GPT-2 models in SEQUENTIAL and INTERLEAVED conditions, versus 89% human agreement.Source: §7.3 · p. 12
  • Restarting the learning rate at L2 training may artificially increase model plasticity and contribute to the absence of CP effects.The authors identify learning-rate schedules and their interaction with learning and CP effects as topics for future work.Source: §7.3 · pp. 12–13
  • Future work could test other regularizers, language-modular architectures, and multimodal training to address limits of EWC, shared parameters, and text-only input.EWC is described as a computational-level model of an innate biological CP; multimodal training could also enable tests of CP effects for L1 learning.Source: §7.3 · p. 13

8 Conclusion

The conclusion argues that language models remain more plastic later in learning than humans. Comparing their learning trajectories offers evidence against CP effects arising necessarily from experience alone, while leaving innate explanations weakly supported.

8 Conclusion

  • Language models remain more plastic later in learning than human learners, and the results weigh against CP effects being necessarily induced solely by experience.The results are consistent with, but provide only weak evidence for, innate mechanisms being necessary to explain CP phenomena.Source: §8 · p. 13

A Data Preprocessing

The preprocessing pipeline cleans and combines the source data, then prepares language-pair datasets and tokenized input for model training.

Cleaning

  • Cleaning removes extra spaces, non-breaking spaces, and dataset-specific characters; Java code is stripped of docstrings and comments.Examples of removed characters include dialogue lines and music symbols in OpenSubtitles and paragraph delimiters in Gutenberg.Source: A Data Preprocessing · p. 20

Unifying

  • Data sources are downsampled and mixed, generally using a 2:1:1 OpenSubtitles:Gutenberg:Wikipedia ratio, before splitting into 83% train, 8.5% validation, and 8.5% test.Experiment 4 instead samples OpenSubtitles and Wikipedia in equal parts.Source: A Data Preprocessing · p. 20

Interleaving

  • INTERLEAVED datasets alternate ordered L1 and L2 text blocks, presenting the same blocks in the same order as SEQUENTIAL training.The sequence alternates L1 block 1, L2 block 1, L1 block 2, and so on.Source: A Data Preprocessing · p. 20

Size alignment

  • Training dataset sizes are aligned across languages using BPE token counts, making input quantities more comparable despite language-specific information per word.The passage notes that BPE token counts are used because tokens are the LMs' input unit and information per token is more comparable.Source: A Data Preprocessing · p. 20

Tokenization

  • Bilingual byte-level BPE tokenizers use a 32,000-token vocabulary and minimum frequency of 2, then divide tokenized text into 512-token input blocks.MONOLINGUAL training uses a monolingual tokenizer instead.Source: A Data Preprocessing · p. 20

B Data

The appendix identifies tables for collected-data statistics, dataset sizes in the main experiments and Experiment 4, and GPT-2 and RoBERTa configurations.

B Data

C Hyperparameters

D Evaluation Tasks

The evaluation-task section describes examples from BLiMP and subsets of GLUE and SuperGLUE used in the experiments. It outlines tasks spanning grammaticality, inference, paraphrase, sentiment, and comprehension.

GLUE

  • GLUE tasks include acceptability judgments in CoLA, sentiment classification in SST-2, and paraphrase judgments in MRPC and QQP.CoLA uses Matthews Correlation Coefficient; the other listed tasks use Accuracy for SST-2 and F1 for MRPC and QQP.Source: D Evaluation Tasks · p. 21
  • MNLI and MNLI-MM assess textual entailment, with MNLI-MM using out-of-domain dev and test data.Both use Accuracy; the QNLI description begins by identifying it as automatically derived from SQuAD.Source: D Evaluation Tasks · p. 21

BLiMP

Other tasks

  • The task descriptions also include SQuAD v1.1 question–paragraph pairs and RTE judgments about whether one text's meaning can be inferred from another.The SQuAD passage specifies Accuracy; the RTE passage also specifies Accuracy.Source: D Evaluation Tasks · p. 22

SuperGLUE

  • The listed SuperGLUE tasks include BoolQ yes–no reading comprehension, MultiRC multi-sentence comprehension, and WSC ambiguity resolution using world knowledge and reasoning.The passages identify Accuracy for BoolQ and WSC, and F1 for MultiRC.Source: D Evaluation Tasks · p. 22

E Derivation of EWC Regularization for Language Modeling

This section derives a regularizer for retaining L1 knowledge during L2 training by approximating the L1 posterior around its likelihood mode. It then uses a diagonal Fisher approximation to make the regularizer practical.

Posterior formulation

  • The derivation treats L1 and L2 datasets as conditionally independent given model parameters and factors the posterior into an L2 likelihood and the L1 posterior.The model is described as generating sentences from both L1 and L2.Source: E Derivation of EWC Regularization for… · p. 23

Posterior approximation

  • A second-order Taylor approximation around the L1 likelihood mode is used to construct a Gaussian approximation of the L1 posterior.The derivation omits the Taylor expansion's first term because expansion occurs at a local optimum.Source: E Derivation of EWC Regularization for… · p. 23
  • The prior is a zero-centered spherical Gaussian, and Bayes' rule expresses the log posterior as the L1 log likelihood plus the log prior, minus a constant.The constant log p(DL1) does not affect optimization.Source: E Derivation of EWC Regularization for… · p. 23

Approximation

  • The exact Hessian-based calculation is impractical because it requires second-order differentiation and involves vocabulary- and parameter-dimension costs.The passages cite O(|Σ|) computation for the next-symbol distribution and O(d^2) unique matrix entries.Source: E Derivation of EWC Regularization for… · p. 24
  • A Monte Carlo approximation estimates the Fisher-related quantity using sampled next symbols; retaining only its diagonal reduces the number of entries to O(d).When K is much smaller than |Σ|, the sample-based approximation provides a speed-up.Source: E Derivation of EWC Regularization for… · p. 24

A Simple Regularizer

  • The resulting regularizer is intended to help a model trained on L1 data retain that knowledge during L2 training.Its coefficients are tunable; λ is tuned on held-out data, while μ is set to zero in the experiments.Source: E Derivation of EWC Regularization for… · p. 24

Related papers

Loading 2407.19325v2…