PaperlayerView exact revision on arXiv

Source-linked AI summary

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure

Lukas Galke, Yoav Ram, Limor Raviv

arXiv:2302.12239v4cs.CL

TL;DR

The paper asks whether the human advantage for learning compositional languages also appears in deep neural networks. It tests GPT-3.5 and recurrent neural networks on languages varying in compositional structure, finding more systematic, consistent, and human-like generalization for more structured input.

  • Problem

    It was unknown whether deep neural networks, unlike humans, benefit from compositional structure when learning and generalizing a new language.

    Source: Discussion · pp. 12, 15

  • Method

    The study compares GPT-3.5 and recurrent neural networks with humans while they learn artificial languages differing in compositional structure.

    Source: Discussion · p. 13, Methods · pp. 17–18

  • Results

    More structured languages produced more systematic generalization, greater agreement among neural agents, and greater similarity between neural and human learners across humans, RNNs, and GPT-3.5.

    Source: Discussion · p. 13

  • Takeaways & Limitations

    Compositional structure is associated with more transparent and human-like generalization in both biological and artificial learning systems.

    Source: Discussion · pp. 13, 15

    The amount of training needed for neural networks to match humans depends on data quantity, model parameters, and optimization steps.

    Source: Discussion · p. 14

Abstract

from arXiv · show

Deep neural networks drive the success of natural language processing. A fundamental property of language is its compositional structure, allowing humans to systematically produce forms for new meanings. For humans, languages with more compositional and transparent structures are typically easier to learn than those with opaque and irregular structures. However, this learnability advantage has not yet been shown for deep neural networks, limiting their use as models for human language learning. Here, we directly test how neural networks compare to humans in learning and generalizing different languages that vary in their degree of compositional structure. We evaluate the memorization and generalization capabilities of a large language model and recurrent neural networks, and show that both deep neural networks exhibit a learnability advantage for more structured linguistic input: neural networks exposed to more compositional languages show more systematic generalization, greater agreement between different agents, and greater similarity to human learners.

Results

Across humans, GPT-3.5, and recurrent neural networks, more compositional input languages produced more systematic, human-like, and convergent generalizations.

More compositional structure leads to higher similarity to humans and more systematic generalization of large language models

  • GPT-3.5 produced more systematic generalizations and outputs more similar to human learners when the input language was more compositional.This pattern was measured using systematicity and character-level length-normalized production similarity during generalization.Source: More compositional structure leads to… · p. 7

More compositional structure leads to higher similarity to humans and more systematic generalization with recurrent neural networks

  • Across humans, GPT-3.5, and recurrent neural networks, errors on structured languages were more similar to the correct labels.The memorization-error pattern was consistent across biological and artificial learners, although the slope differed between humans and RNNs.Source: More compositional structure leads to… · pp. 8, 9, 12
  • More structured languages were learned better during RNN memorization, although all languages reached production similarity of at least 0.8 by round 60.The initial memorization advantage for structured languages diminished over training rounds, while similarity to human learners increased with structure.Source: More compositional structure leads to… · p. 11
  • For neural network agents, more structured input languages increased generalization scores, similarity to humans, and agreement between independently learning agents during generalization.Generalization score increased with structure (β = 0.088, p < 0.001), as did similarity to humans (β = 0.132, p < 0.001) and agreement between agents (β = 0.043, p < 0.001).Source: More compositional structure leads to… · p. 12

Results

Discussion

Deep neural networks, like humans, learn and generalize more systematically from more structured languages, with greater agreement and human-like behavior. The discussion connects this finding to language evolution, machine learning, and limitations involving training and input representations.

Discussion

  • More structured languages produce more systematic, consistent, and human-like generalization across recurrent networks, transformer-based models, and humans.These systems show the same broad bias in systematic generalization and memorization errors, although humans and recurrent networks differ in the slope of memorization error.Source: Discussion · p. 13
  • Higher linguistic structure increases agreement between neural agents and between neural agents and humans because available generalization patterns are more transparent.Lower-structured languages offer fewer transparent patterns, making multiple generalizations similarly likely and increasing production variation.Source: Discussion · p. 13
  • Mid- and mid-low-structured languages can learn quickly because reduced variation simplifies memorization, but label reuse can harm full differentiation and systematic generalization.One example uses the same marker for two diagonal directions, trading expressiveness for easier initial learning.Source: Discussion · p. 13
  • The findings support learning grammatical structure directly from grounded linguistic input and predict that children may likewise benefit from systematic compositional structure.The child-learning prediction is presented as future work being tested.Source: Discussion · pp. 13–14
  • Structured inputs can improve systematic generalization to unseen combinations, and natural-language evidence shows Wug Test accuracy correlates negatively with irregularity.The reported correlations are Spearman’s ρ = −0.96, p < 10^-15, and Kendall’s τ = −0.86, p < 10^-14.Source: Discussion · p. 14
  • Low-resource languages may pose a double challenge for neural language modeling because they typically combine less structure with limited training data.The passage also suggests that highly structured programming languages may contribute beneficially to large-language-model training data.Source: Discussion · p. 14
  • These results support adding learnability pressure to emergent-communication models, bringing neural simulations closer to language-evolution research.The authors argue that a lack of cognitive constraints may remove the pressure underlying human language evolution.Source: Discussion · p. 14
  • The amount of training needed by neural networks relative to humans remains difficult to predict because it depends on data, optimized parameters, and optimization steps.The study identifies the equalization point from recurrent-network learning trajectories, while final analyses use end-of-training productions.Source: Discussion · pp. 14–15

Methods

The study uses ten artificial languages varying in compositional structure, compares GPT-3.5, recurrent networks, and humans, and evaluates memorization, generalization, convergence, and human similarity.

Input Languages

  • Ten input languages contain shape–angle scene-label mappings and vary in structure scores from 0.09 to 0.85.Their structure is quantified by topographic similarity between semantic differences and normalized edit distances.Source: Methods · pp. 15–16

Large Language Models

  • GPT-3.5 receives 23 shape–angle examples and predicts labels for memorized or unseen combinations through deterministic in-context completion.Sampling temperature is zero, and the model is not given an explicit task description.Source: Methods · p. 16

Custom Recurrent Neural Network Architecture

  • The recurrent architecture combines a generative Writer with a contrastive Reader, both linked to scene representations produced by an Encoder.The Writer generates labels letter by letter, while the Reader aligns matching scene and label representations.Source: Methods · p. 17

Input Representation

  • The recurrent networks use a six-dimensional shape-plus-sine/cosine angle representation to focus evaluation on systematic generalization rather than visual disentangling.This simplified representation makes nearby angles similar while preserving angle identity.Source: Methods · p. 17

Implementation and Robustness

  • The models use Adam with a default learning rate of 10^-3, and supplementary analyses test robustness to architecture and hyperparameter choices.Larger hidden sizes increase the speed of memorization and generalization, while broader hyperparameter results are reported as robust.Source: Methods · pp. 18–19

Training Procedure

  • Training mirrors the human procedure through exposure, guessing, and production blocks implemented with generative and contrastive losses.The main exposure and guessing objective is L = Lgen + αconLcon with αcon = 0.1, whereas production uses L = Lgen.Source: Methods · pp. 18–19

Measures

  • Production similarity is one minus normalized edit distance, while generalization and convergence quantify systematicity and agreement on unseen scenes.A convergence score of 1 means agents produce identical labels for each test scene, while 0 indicates completely different labels.Source: Methods · pp. 19–20

Statistical Analyses

  • The analysis trains 100 randomly initialized agents per language for 100 rounds and models production similarity, generalization, convergence, and human similarity.Linear mixed-effects models use structure score and logarithmized round number as fixed effects, with seed and scene as random effects.Source: Methods · p. 20

Production Similarity to Ground Truth during Memorization

Figure 6 plots production similarity to the input language’s ground-truth labels across training rounds, with darker colors indicating greater structure and stars marking performance above humans.

Production Similarity to Ground Truth during Memorization

  • Production similarity to ground truth is shown as a function of round number, with color encoding input-language structure and stars marking neural agents above human performance.The figure supports comparison of memorization trajectories across languages with different structural scores.Source: Figure 6 · p. 31

Systematicity during Generalization

The supplied figures present production similarity to humans, generalization score, and convergence across rounds, with color indicating language structure.

Systematicity during Generalization

  • Table 3 lists the structure scores of the input languages used in the analyses.The supplied table caption identifies its scope but does not provide the individual values.Source: Table 3 · p. 32
  • Figure 8 plots generalization score across rounds, while Figures 9 and 7 show convergence and production similarity to human learners, respectively.The captions identify structure as the color dimension and, for Figures 8 and 9, mark cases where neural agents exceed human performance.Source: Figure 7 · p. 32, Figure 8 · p. 33, Figure 9 · p. 34

Binary Accuracy with respect to Ground Truth during Memorization

The supplied material covers mixed-effects analyses, multiple sensitivity checks, and example-data tables for memorization, human similarity, systematicity, convergence, and generalization.

Statistical Analyses of the Results

  • Table 4 reports results from six linear mixed-effects models, including production similarity to ground truth during memorization, production similarity to humans during generalization, and generalization score; all tests are two-sided.Source: Statistical Analyses of the Results · p. 34, Table 4 · p. 37

LME 1: Production Similarity during Memorization

Sensitivity to Hyperparameters

Sensitivity to the Scaling Factor for the Contrastive Loss Term

  • A contrastive-loss scaling factor of 0.1 gives the best learning speed, with little difference from 0.2 for human-similarity outcomes.From step 1,300, generalization score increases faster with 0.1 than with the other tested scaling factors.Source: Sensitivity to the Scaling Factor for… · p. 44

Structure effect does not depend on specific hyperparameter choices

Example Data

Loading 2302.12239v4…