PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Locality as an Inductive Bias for Neural Language Models

Taiga Someya, Anej Svete, Brian DuSell, Timothy J. O'Donnell, Mario Giulianelli, Ryan Cotterell

arXiv:2506.05136v1cs.CL

TL;DR

The paper addresses whether neural language-model inductive biases align with human processing constraints and introduces m-local entropy as a quantitative measure for testing this question. Across perturbed natural corpora and PFSA-generated languages, higher m-local entropy is associated with poorer learning by LSTM and Transformer models. The findings indicate sensitivity to local statistical structure, while the analysis remains correlational and limited by the expressiveness of PFSAs and the possible influence of other inductive biases.

  • Problem

    Prior work showed that disrupting local structure affects neural language-model learnability but did not isolate the specific property responsible.

  • Method

    The paper introduces m-local entropy, derived from lossy-context surprisal, and evaluates it using perturbed natural-language corpora and PFSA-generated languages.

  • Results

    Higher m-local entropy is associated with less effective learning by both LSTM and Transformer language models across natural-corpus perturbations and PFSA experiments.

  • Takeaways & Limitations

    Neural language models are strongly sensitive to local statistical structure, suggesting a shared inductive bias for information locality with human learners.

  • Takeaways & Limitations

    The analysis is correlational, uses the limited family of PFSA-generated languages, and does not disentangle information locality from other inductive biases.

Abstract

from arXiv · show

Inductive biases are inherent in every machine learning system, shaping how models generalize from finite data. In the case of neural language models (LMs), debates persist as to whether these biases align with or diverge from human processing constraints. To address this issue, we propose a quantitative framework that allows for controlled investigations into the nature of these biases. Within our framework, we introduce $m$-local entropy$\unicode{x2013}$an information-theoretic measure derived from average lossy-context surprisal$\unicode{x2013}$that captures the local uncertainty of a language by quantifying how effectively the $m-1$ preceding symbols disambiguate the next symbol. In experiments on both perturbed natural language corpora and languages defined by probabilistic finite-state automata (PFSAs), we show that languages with higher $m$-local entropy are more difficult for Transformer and LSTM LMs to learn. These results suggest that neural LMs, much like humans, are highly sensitive to the local statistical structure of a language.

1 Introduction

The paper asks whether neural language models share human-relevant inductive biases by testing whether local statistical structure predicts learning difficulty. It introduces m-local entropy and finds that higher local entropy makes languages harder for LSTM and Transformer models to learn.

  • 1 Introduction: The work targets a gap left by prior perturbation studies, which showed altered sequential structure affects learnability without isolating the responsible property.
  • 1 Introduction: Information locality links the study to human memory constraints, because long-range dependencies are difficult for human processors to integrate.
  • 1 Introduction: m-local entropy quantifies local predictability, enabling controlled tests of whether neural language models are sensitive to information locality.The measure is derived from lossy-context surprisal and reflects how preceding symbols constrain the next symbol.
  • 1 Introduction: Perturbing natural-language corpora showed systematic performance degradation for LSTM and Transformer models as m-local entropy increased, even with global and next-symbol entropy held constant.
  • 1 Introduction: PFSA experiments showed that the observed relationship between local entropy and learnability was not an artifact of the corpora or perturbation functions used.

2 Formal Background

The section formalizes language models over finite-symbol strings, defines global and m-local entropy, and introduces PFSAs for controlled, exact analyses of language structure and learnability. It distinguishes local uncertainty from entropy measures invariant under the perturbations used in the experiments.

  • 2.1 Languages and Language Models: A language model is a probability distribution over finite strings, with prefix and conditional prefix probabilities supporting next-symbol prediction and string-probability factorization.Modern neural LMs directly parameterize the next-symbol distribution, so training maximizes the corresponding log-likelihood.
  • 2.1 Languages and Language Models: Finite expected string length is assumed, while additional finiteness conditions govern normalization of prefix probabilities and related sums.The paper assumes finite mean length and notes that a stronger second-moment condition ensures finiteness of another normalization quantity.
  • 2.2.1 Global Entropy: Global entropy measures uncertainty over complete strings, whereas next-symbol entropy averages conditional next-symbol uncertainty across contexts and is proportional to global entropy.Under the stated identity, the two measures would have identical predictive power in a linear model.
  • 2.2.1 Global Entropy: Global and next-symbol entropy cannot isolate the targeted local structure because they remain unchanged under the length-preserving bijective perturbations used to construct counterfactual corpora.The perturbations preserve global and next-symbol entropy while changing m-local entropy.
  • 2.2.2 M-local Entropy: m-local entropy quantifies next-symbol unpredictability after observing exactly m−1 preceding symbols, averaging context-specific uncertainty using context probabilities.It represents fixed-context uncertainty rather than uncertainty conditioned on the full available context.
  • 2.2.2 M-local Entropy: Unlike global entropy, m-local entropy can differ substantially between languages with identical global entropy and can change under bijective transformations of strings.This property enables experiments testing how disrupted local statistical structure relates to neural language-model learnability.
  • 2.2.2 M-local Entropy: m-local entropy coincides with average lossy-context surprisal when the memory representation retains the preceding m−1 symbols and contexts are weighted by infix probabilities.The paper presents this as a direct connection between lossy-context surprisal and language-model learnability.
  • 2.3 Probabilistic Finite-State Automata: A PFSA is a finite weighted state-transition system with initial and final weighting functions that induces a language model over strings.PFSAs provide controlled experimental languages and permit exact computation of string, prefix, infix, and m-local-entropy quantities.

3 Experiment 1: LM Performance along the M-local Entropy Continuum

Experiment 1 creates a controlled continuum of local complexity by bijectively perturbing a natural-language corpus while preserving global and next-symbol entropy, then tests whether m-local entropy predicts neural-LM learnability.

  • 3 Experiment 1: LM Performance along the M-local Entropy Continuum: The experiment applies length-preserving bijective perturbations to natural language, producing corpora with different m-local entropy but identical global and next-symbol entropy.This design creates a measurable continuum of local complexity rather than a qualitative notion of possible language variation.
  • 3.2 Estimating the M-local Entropy: The m-local entropy estimator uses an order-(m − 1) n-gram model trained on the concatenated training, validation, and test corpora.The estimator is treated as a practical proxy for the target quantity.
  • 3.2.1 Validating M-local Entropy Estimation: Across 16 PFSAs, 200K strings yield relative estimation error below 1% through m = 4 and below 8% at m = 5.Increasing corpus size from 50K to 200K consistently reduces absolute and relative error.
  • 3.3.1 Neural Language Models: The neural-LM evaluation uses a single-layer 512-dimensional LSTM and a 4-layer causally masked Transformer, each trained with five random seeds.Both models are trained with the standard language-modeling objective.
  • 3.3.2 Dataset: The natural-language dataset is a BLLIP subset of roughly 200K sentences and 5M tokens, expanded to 124 corpora through perturbations and random seeds.K-LOCALDETERMINISTICSHUFFLE uses k values from 3 through 7.
  • 3.3.3 Evaluating Learning Difficulty: Learning difficulty is measured by test-set next-symbol cross-entropy, with equal inherent global entropy enabling comparisons across transformed languages.The evaluation accounts for each target language’s entropy so cross-entropy differences reflect model approximation rather than differing global uncertainty.
  • 3.4 Results: DETERMINISTICSHUFFLE produces the highest m-local entropies, while K-LOCALDETERMINISTICSHUFFLE increases entropy progressively with window size k.REVERSE barely changes m-local entropy, whereas EVENODDSHUFFLE and ODDEVENSHUFFLE increase it more moderately.
  • 3.4 Results: Higher m-local entropy strongly predicts higher next-symbol cross-entropy for both LSTM and Transformer LMs; at m = 4, R2 reaches 0.922 and 0.915, respectively.Because the transformations preserve global and global next-symbol entropy, the association isolates local entropy as a key correlate of learning difficulty.

4 Experiment 2: Controlled Learnability Tests with PFSAs

Experiment 2 tests the local-entropy hypothesis on randomly generated PFSAs, where relevant entropy quantities can be computed exactly and global entropy is controlled.

  • 4.1 Experimental Setup: The PFSA experiment reuses the neural-LM architectures and training configuration from Experiment 1 while controlling for global entropy.PFSAs permit exact computation of quantities including m-local entropy.
  • 4.1.1 Generating Datasets Using PFSAs: Random PFSAs vary alphabet sizes across {32, 48, 64} and state counts across {16, 24, 32}, with 25 automata generated for each of nine configurations.Each PFSA supplies 20K training, 5K validation, and 5K test strings.
  • 4.1.2 Evaluating Learning Difficulty: Learning difficulty is evaluated with estimated KL divergence, obtained by subtracting PFSA next-symbol entropy from the model’s estimated next-symbol cross-entropy.The resulting quantity is evaluated on the test set.
  • 4.2 M-local Entropy and LM Performance: Across both architectures and all m values from 2 through 5, higher PFSA m-local entropy correlates positively with higher test-set KL divergence.The result indicates that neural LMs find distributions with greater local uncertainty more difficult to model.
  • 4.2 M-local Entropy and LM Performance: Within each topological cluster, the Transformer consistently has higher KL divergence than the LSTM, indicating better modeling of these distributions by the LSTM.The comparison is reported for the controlled PFSA setting.
  • 4.2 M-local Entropy and LM Performance: When alphabet size is held constant, PFSAs with more states have higher KL divergence.This pattern is reported as consistent with Borenstein et al. (2024).

5 Discussion and Conclusion

The paper argues that m-local entropy provides an information-theoretic account of LM learning difficulty and connects neural LM biases with information locality. Across natural-language perturbations and PFSA experiments, both architectures more effectively model languages with lower m-local entropy, motivating further work on cognitively plausible evaluation and training.

  • 5 Discussion and Conclusion: m-local entropy connects LM inductive biases with statistical properties of language associated with human functional pressures.The measure is grounded in lossy-context surprisal theory and information locality principles.
  • 5 Discussion and Conclusion: Figure 3 plots symbol-level KL divergence against m-local entropy across m = 2, 3, 4, 5 for LSTM and Transformer models and PFSA topological clusters.The dashed lines are cluster-specific linear fits.
  • 5 Discussion and Conclusion: Both LSTM and Transformer LMs consistently model languages with lower m-local entropy more effectively across perturbed natural-language and PFSA experiments.This shared sensitivity suggests a common inductive bias related to incremental language processing.
  • 5 Discussion and Conclusion: Future research could examine inductive biases beyond information locality and use local entropy in model evaluation or as a training regularization signal.Suggested directions include hierarchical structure and structure dependence.
  • 5 Discussion and Conclusion: The study presents evidence that neural LMs are strongly sensitive to local statistical structure, supporting future work on alignment between artificial and human language processors.

Limitations

The paper’s evidence is correlational and its controlled PFSA experiments cover only a limited class of language models. Other inductive biases may also influence neural LM learning, so the observed relationships may differ with more expressive formalisms or additional factors.

  • Limitations: The analysis establishes correlation between m-local entropy and LM performance but does not identify the mechanisms linking local uncertainty to learning dynamics.
  • Limitations: PFSA-based experiments provide tractability but represent only a limited set of possible language models, so more expressive formalisms could reveal different relationships.The paper specifically mentions pushdown automata and more powerful models as possible alternatives.
  • Limitations: Information locality may coexist with other influential inductive biases, whose individual and combined effects remain to be disentangled.

Ethical considerations

The appendix documents AI-use compliance and develops PFSA definitions and computational properties needed to calculate probabilities, next-symbol distributions, and m-local entropy. These results support efficient exact analysis of PFSA-induced language models.

  • Ethical considerations: The authors report using ChatGPT and GitHub Copilot for writing and coding assistance in compliance with ACL policy.
  • A Probabilistic Finite-State Automata: A PFSA is represented with transition, symbol-specific transition, and emission matrices, with string transitions obtained by multiplying symbol-specific matrices.
  • A Probabilistic Finite-State Automata: The appendix defines path-prefix and related path distributions as analogues of prefix and infix string probabilities.
  • A.1 Useful Properties of Probabilistic Finite-State Automata: PFSAs support efficient exact computation of quantities including string, prefix, infix, and next-symbol probabilities.
  • A.1 Useful Properties of Probabilistic Finite-State Automata: For deterministic PFSAs, m-local entropy can be computed by evaluating context probabilities and conditional next-symbol entropies for each c ∈Σ^(m−1).

B Generating Random PFSAs

The experiments generate random PFSAs by sampling topology and weights under specified state, alphabet, and length controls, then evaluate Transformer and LSTM baselines. The appendix also specifies the architectures and outgoing-symbol construction used in these experiments.

  • B Generating Random PFSAs: Random PFSA generation takes the number of states, alphabet size, target mean string length, and topology and weight random generators as inputs.The output has randomly assigned transitions and normalized weights.
  • B Generating Random PFSAs: The implementation initializes symbol-specific transition matrices and final weights while normalizing outgoing weights for each state.
  • C.1 Transformer: The Transformer uses four causally masked layers, 768-dimensional embeddings and feedforward layers, 12 attention heads, sinusoidal positional encodings, pre-normalization, and dropout.
  • B Generating Random PFSAs: The outgoing-symbol procedure assigns each state at least a minimum number of symbols, ensures every symbol appears in some state, and adds random transitions.
  • C.2 LSTM: The LSTM is a single-layer PyTorch model with 512-dimensional hidden units and modifications based on Butoi et al. (2025).

D Hyperparameters for Neural Language Model Training

Training uses regularized optimization, validation-based checkpoint selection, adaptive learning rates, and early stopping. Model parameters and batching are initialized and constrained using fixed procedures.

  • Models use dropout rate 0.1 where applicable, layer-normalization weights initialized to 1, biases to 0, and other parameters sampled uniformly from [−0.1, 0.1].
  • Each epoch shuffles the training data, groups similarly sized strings, and limits batches to 2,048 symbols including padding, BOS, and EOS symbols.
  • Training minimizes validation cross-entropy with Adam, clips gradients at 5, evaluates every 10K examples, reduces learning rates after five stagnant checkpoints, and stops after ten.
  • Reported results use the checkpoint with lowest validation cross-entropy, with training capped at 1K epochs.

E Pearson Correlation Coefficients between M-local Entropy and KL Divergence

This section reports Pearson correlations between PFSA m-local entropy and estimated KL divergence across architectures and PFSA configurations. Table 3 organizes these coefficients by architecture, state count, and alphabet size.

  • Pearson correlations relate PFSA m-local entropy to estimated KL divergence across the evaluated neural-language-model settings.
  • Table 3 breaks out the correlations by model architecture, number of states |Q|, and alphabet size |Σ|.

F Additional Experiments with BabyLM Corpus

Additional BabyLM experiments reproduce the main experimental trends, with a slight difference for the REVERSE language. The experiments used substantial computational resources across two GPU models.

  • F Additional Experiments with BabyLM Corpus: The BabyLM experiments show the same trends as the main experiment, although REVERSE exhibits slightly different tendencies.
  • G Computational Resources: The full set of experiments consumed approximately 717.5 GPU hours on NVIDIA GeForce RTX 4090 24GB and Quadro RTX 6000 24GB GPUs.
Loading 2506.05136v1…