PaperlayerView exact revision on arXiv

Source-linked AI summary

What Artificial Neural Networks Can Tell Us About Human Language Acquisition

Alex Warstadt, Samuel R. Bowman

arXiv:2208.07998v2cs.CL

TL;DR

The paper asks how artificial learners can provide relevant evidence about human language acquisition despite major differences between model and human learning scenarios. It develops an ablation-based framework for testing whether hypothesized advantages are necessary and argues that success by relatively unbiased models can provide proofs of concept, while current model–human differences and inductive-bias comparisons remain limiting factors.

  • Problem

    Current artificial learners and humans differ substantially in data exposure, grounding, interaction, and inductive bias, weakening inferences from models to human language acquisition.

  • Method

    The paper analyzes how to generalize model learnability results and advocates ablation experiments using learners without significant advantages over humans.

  • Results

    Artificial learners can provide proofs of concept that specific linguistic behaviors are learnable under impoverished conditions, with randomly initialized neural networks offering relatively weak linguistic inductive bias.

  • Takeaways & Limitations

    Positive results from appropriately constrained model learners can inform causal debates about whether environmental inputs or innate biases are necessary for human language learning.

  • Takeaways & Limitations

    Generalization remains difficult when models and humans differ in relevant advantages, and tightly estimating human inductive bias requires substantial empirical work.

Abstract

from arXiv · show

Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language. However, the learning environments and biases of current artificial learners and humans diverge in ways that weaken the impact of the evidence obtained from learning simulations. For example, today's most effective neural language models are trained on roughly one thousand times the amount of linguistic data available to a typical child. To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans. If an appropriate model successfully acquires some target linguistic knowledge, it can provide a proof of concept that the target is learnable in a hypothesized human learning scenario. Plausible model learners will enable us to carry out experimental manipulations to make causal inferences about variables in the learning environment, and to rigorously test poverty-of-the-stimulus-style claims arguing for innate linguistic knowledge in humans on the basis of speculations about learnability. Comparable experiments will never be possible with human subjects due to practical and ethical considerations, making model learners an indispensable resource. So far, attempts to deprive current models of unfair advantages obtain sub-human results for key grammatical behaviors such as acceptability judgments. But before we can justifiably conclude that language learning requires more prior domain-specific knowledge than current models possess, we must first explore non-linguistic inputs in the form of multimodal stimuli and multi-agent interaction as ways to make our learners more efficient at learning from limited linguistic input.

1 Introduction

Artificial neural networks can inform human language-acquisition debates when their learning scenarios avoid unfair advantages over humans. The paper proposes ablation-based tests of whether hypothesized innate biases or environmental inputs are necessary for target linguistic knowledge.

  • 1 Introduction: Current language-model studies often use unrealistic scenarios, including massive internet-scale corpora, limiting their direct relevance to human language learning.Neural networks have nevertheless made substantial progress on grammatical acceptability and other language behaviors.
  • 1 Introduction: Artificial learners are most informative when they provide proofs of concept that target linguistic knowledge is learnable without a hypothesized advantage.If the model has no substantive advantages over humans, the result can generalize to the hypothesized human scenario.
  • 1 Introduction: Ablation experiments remove a hypothesized advantage from the input or learner and test whether the model still acquires the target knowledge.The toy example removes triply embedded clauses and evaluates agreement-error judgments under a grounded, non-hierarchical human scenario.
  • 1 Introduction: Positive model results are more meaningful than negative results because success without an advantage can generalize without reproducing every human advantage.Negative results require a model with at least as many relevant advantages as humans, including grounding when it could accelerate learning.
  • 1 Introduction: Ablations can test poverty-of-the-stimulus claims and challenge proposals that language-specific innate biases are necessary for human language learning.The paper frames these experiments as causal tests of the necessity of hypothesized advantages.
  • 1 Introduction: Closing the human–model data-efficiency gap may require replacing some text-only input with multimodal stimuli and interaction with other agents.Attempts to reduce models’ massive linguistic-data advantage have produced worse performance.

2 Evidence from Model Learners

Model learners can inform human learnability questions when their advantages and disadvantages are explicitly compared with those hypothesized for humans. Positive results are generally easier to generalize, but failures can be confounded by model disadvantages and by uncertainty about human inductive bias.

  • 2 Evidence from Model Learners: Model results generalize to humans only when differences between model and human learning scenarios do not undermine the intended conclusion.Differences may be irrelevant, strengthen the inference, or introduce confounds depending on whether the model succeeds or fails.
  • 2 Evidence from Model Learners: A model’s success while operating at a strict disadvantage can establish that the hypothesized human scenario is sufficient for learning the target.The converse case—failure despite a strict model advantage—can establish insufficiency, but is less practical.
  • 2 Evidence from Model Learners: A model failure after ablation is difficult to interpret when the model has an additional disadvantage that humans may not share.Agreement errors could confound a failure caused by removing irregular-plural evidence.
  • 2 Evidence from Model Learners: Model learners need not perfectly reproduce humans: undershooting human advantages can make successful learning results especially informative.The paper recommends impoverished learners, then enriching them with sensorimotor input or interaction if they fail for uncertain reasons.
  • 2 Evidence from Model Learners: Ablation studies are better suited to supporting empiricist claims than nativist claims because positive learnability results are more practically generalizable.Strong nativist evidence would require failure in a model scenario without significant disadvantages relative to a typical human.

3 Tests of Human-Like Linguistic Knowledge

Tests of human-like linguistic knowledge primarily measure observable performance, while using controlled supervised and unsupervised evaluations to probe grammatical generalization. These include likelihood-based behavioral tasks, compositional generalization, and Poverty of the Stimulus designs.

  • 3.1 Testing for Competence vs. Performance: Behavioral tests are practical and theory-neutral measures of performance, whereas competence remains a theoretical construct even for humans.
  • 3.2 Unsupervised Tests: Unsupervised evaluations reveal linguistic knowledge from language-model training without additional task-specific supervision.
  • 3.2.1 Acceptability Judgments, Minimal Pairs, BLiMP: Acceptability evaluation compares language-model likelihoods for minimally different grammatical and ungrammatical sentences, with minimal pairs isolating decision boundaries and gradient contrasts.
  • 3.2.1 Acceptability Judgments, Minimal Pairs, BLiMP: BLiMP scales minimal-pair evaluation across 67 English pair types in 12 broad categories spanning morphology, syntax, and semantics.
  • 3.2 Unsupervised Tests: Reading-time and age-of-acquisition predictions provide additional behavioral comparisons between language models and humans.
  • 3.3 Supervised Tests: Supervised probing includes compositional-generalization and Poverty of the Stimulus tests that examine whether models form high-level rules from controlled training and test distributions.

4 The Learning Environment

Human and model learning environments differ in data quantity, modality, grounding, conceptual support, and interaction, weakening direct generalization from models to children. Human-scale and multimodal approaches remain promising but currently underperform or lack sufficient evidence for human-like language learning.

  • 4.1 Data Quantity: Popular language models receive far more linguistic input than children, ranging from 1B words for ELMo to about 200B for GPT-3.These quantities correspond to roughly 300 to 20,000 human years of linguistic experience.
  • 4.1 Data Quantity: Models trained on 100M words reach near-human performance in at most 2 BLiMP categories, while 10M-word models reach it in only 1.RoBERTa models trained on about 30B words can achieve near-human performance, but performance falls sharply at human-scale data quantities.
  • 4.3 Prosody: Transcribed child-directed speech and other speech corpora improve ecological validity, but text omits prosody while punctuation and whitespace give models boundary information.CHILDES contains transcribed parent-child discourse, while COCA provides 83M words of transcribed unscripted speech.
  • 4.3 Prosody: Human-like input should include spoken or signed communication, but current audio-trained models achieve only 58% accuracy on modified audio BLiMP.This is 8 percentage points above chance, compared with over 79% for RoBERTa trained on 100M words.
  • 4.4 Multimodal and Interactive Input: Text-only models must acquire conceptual scaffolding from text, and their limited world knowledge is unlikely to support language acquisition from human-scale data.Strong performance on limited world-knowledge benchmarks requires training on the order of billions of words.
  • 4.4 Multimodal and Interactive Input: Multimodal and interactive learning have not yet closed the human-model gap: visual enrichment rarely improves language evaluations, and interaction can produce unintelligible language drift.SAYCam is ecologically close to children’s experience but currently contains only an estimated 1–2M words; multimodal datasets also often lack extended discourse.

5 The Learner

An appropriate learner should minimize built-in advantages over humans so that positive learning results can generalize to hypothesized human scenarios. Constructing such a learner remains difficult because inductive biases are hard to identify, control, and compare with human biases.

  • 5 The Learner: The learner’s architecture and learning algorithm determine built-in advantages that can affect whether model results generalize to humans.The less advantage a model has over humans independent of the experimental manipulation, the stronger the basis for generalization.
  • 5 The Learner: Building an ideal learner is difficult because inductive bias is theoretically contested, empirically challenging to probe, and only weakly controllable.Assessing whether a model’s bias is no more advantageous than a human’s also requires understanding human innate advantages.

5.1 Formalizing Innate Advantage

The paper formalizes innate advantage as how strongly a learner favors acceptable target generalizations across a specified distribution of learning environments.

  • 5.1 Formalizing Innate Advantage: A learner’s inductive bias determines which generalization it selects from finite examples, with an advantage arising when it favors the right generalizations.
  • 5.1 Formalizing Innate Advantage: For deterministic learners, the environment maps to one hypothesis, allowing advantage to be expressed as the probability of selecting the target hypothesis.
  • 5.1 Formalizing Innate Advantage: The formalization requires choosing a realistic environment distribution that supports related hypotheses while assigning low probability to adversarial environments.
  • 5.1 Formalizing Innate Advantage: The target may be a set of acceptable generalizations rather than a single function, accommodating error tolerance and individual variation.
  • 5.1 Formalizing Innate Advantage: Innate advantage is the probability that a learner converges on an acceptable target generalization in a typical learning environment.The quantity depends on a prior over environments, so success in contrived environments alone does not establish an advantage.

5.2 A Lower Bound on Human Inductive Bias

A lower-bound argument requires a model learner whose biases are no stronger than humans’ while still allowing domain-general biases and expressive representations.

  • 5.2 A Lower Bound on Human Inductive Bias: In practice, available learners constrain model choice, and theoretical lower bounds on human inductive bias are difficult to achieve tightly.
  • 5.2 A Lower Bound on Human Inductive Bias: Comparing model and human inductive biases may require empirical work more intensive than the ablation experiments used to test learnability.
  • 5.2 A Lower Bound on Human Inductive Bias: A model need not be an unbiased tabula rasa; it must only lack language-specific bias while retaining domain-general biases no stronger than humans’.
  • 5.2 A Lower Bound on Human Inductive Bias: Inductive bias may be better compared by the generalizations learners favor than by whether a bias is labeled language-specific.
  • 5.2 A Lower Bound on Human Inductive Bias: Greater expressive capacity does not guarantee greater advantage, because restricting hypotheses can shift probability toward the target generalizations.

5.3 Achieving a Lower Bound on Human Inductive Bias in Practice

Current neural architectures are imperfect model learners, but evidence about their inductive biases supports cautious relevance to human acquisition while leaving substantial uncertainty.

  • 5.3 Achieving a Lower Bound on Human Inductive Bias in Practice: Transformers dominate modern NLP applications, while RNNs remain widely studied in language-model probing.
  • 5.3 Achieving a Lower Bound on Human Inductive Bias in Practice: Neural networks do not clearly provide a strict lower bound on human inductive bias, because their empirical success may reflect advantageous biases as well as training efficiency.
  • 5.3 Achieving a Lower Bound on Human Inductive Bias in Practice: Evidence that untrained ANNs lack several human-like biases makes them plausible model learners, but stronger evidence is needed for robust existence proofs.
  • 5.3 Achieving a Lower Bound on Human Inductive Bias in Practice: RNNs encode locality through sequential processing, whereas Transformers must learn locality from positional embeddings.
  • 5.3 Achieving a Lower Bound on Human Inductive Bias in Practice: LSTM and Transformer performance can nevertheless be closely related, weakening a simple architecture-based distinction in grammar learning.

5.4 Summary

Neural networks may provide relevant evidence about human language acquisition despite imperfect model fidelity. However, uncertainty about model and human biases limits how strongly such evidence can support lower-bound claims.

  • 5.4 Summary: Randomly initialized LSTMs and Transformers generally have weak language-specific inductive biases, making their learnability results likely relevant to human acquisition.
  • 5.4 Summary: This relevance remains provisional because model advantages, human and model bias measurements, and architectural control are all limited.

6 Discussion

Artificial learners offer scalable, ethically feasible experiments for studying human language acquisition, but current models remain far from the ecologically valid best-case scenario. Progress requires more plausible learners and environments, while imperfect models can still provide converging evidence about learnability.

  • 6 Discussion: Although current model learners are far from the stringent no-advantage ideal, increasingly plausible learners can still contribute converging evidence about human learnability.The authors do not treat the absence of incontrovertible proof as grounds for abandoning artificial-learner research.
  • 6.1 The Case for Model Learners: Artificial learners are scalable: simulating an entire human language-acquisition period can take about one week on a single computer.Parallelization can make these simulations faster; the estimate is based on reproducing Warstadt et al.’s pretraining procedure.
  • 6.1 The Case for Model Learners: Model learners enable acquisition ablations and experimental manipulations that are infeasible or unethical with human subjects.Researchers can alter learning algorithms, neural architectures, internal features, and environmental distributions, including syntactic phenomena, word types, noise, and counterfactuals.
  • 6 Discussion: The rapid progress of machine learning makes artificial learners an increasingly salient possibility for research on human language acquisition.The paper links this opportunity to advances in artificial language processing and access to capable models.
  • 6.2 The Future of Model Learners: More ecologically valid learners require deliberate model-building objectives distinct from simply optimizing state-of-the-art NLP performance.Suggested mechanisms include benchmarks and competitions that constrain the quantity and nature of pretraining data.
Loading 2208.07998v2…