PaperlayerView exact revision on arXiv

Source-linked AI summary

Informational Antilocality and the Locality Bias in LLMs

Andrew McInnerney, Shane Storks, Steven Abney, Richard L. Lewis

arXiv:2608.27760v1cs.CL

TL;DR

The paper asks whether LLMs’ apparent information-locality bias reflects a genuine constraint on learning non-local languages or another correlated inductive bias. It defines and constructs controlled k-antilocal languages, trains GPT-2 and DeBERTa models on them, and finds comparable final cross-entropy but slower convergence as antilocality increases. The results support a bias in learning speed rather than achievable performance, while the simple datasets and limited architectural coverage constrain interpretation.

  • Problem

    The paper asks whether LLMs’ apparent information-locality bias reflects a genuine constraint on learning non-local languages or another correlated inductive bias.

  • Method

    The authors define k-antilocality, construct controlled k-back and paired control languages with varying k, and train GPT-2 and DeBERTa models on them.

  • Results

    Comparable final cross-entropy is achieved across k and language type, while more antilocal k-back languages converge more slowly than paired controls.

  • Takeaways & Limitations

    The results support a GPT-2 learning bias toward information locality that appears in learning speed rather than achievable performance.

  • Takeaways & Limitations

    The simple languages may not generalize to human language, and k-back and control languages differ in additional properties such as crossing dependencies.

Abstract

from arXiv · show

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of $k$ contiguous symbols. We construct such languages with increasing $k$, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

1 Introduction

The paper situates its study in debates over whether LLMs share human-like linguistic biases, focusing on information locality. It asks whether reduced locality affects LLM learning success, learning speed, or both.

  • Motivation: LLMs’ ability to learn unnatural languages is central to debates about their similarity to human linguistic cognition.Prior studies often measure learning success using cross-entropy loss or KL divergence.
  • Motivation: Information locality has been proposed as a bias shared by humans and LLMs, making languages with less local statistical structure harder to learn.Human memory degradation provides a natural source of this bias in humans.
  • Motivation: Because transformers access all input tokens simultaneously and use learned positional encodings, the source of an LLM locality bias is unclear.The observed bias might reflect inductive biases correlated with, but dissociable from, information locality.
  • Contribution: The paper defines antilocality, constructs languages with varying antilocality, and trains LLMs on them.The study tests whether antilocality affects achieved cross-entropy loss and convergence speed.
  • Contribution: Antilocality does not limit achieved cross-entropy loss in the studied languages, but it markedly slows convergence; the effect may depend on autoregression.A masked LLM reportedly learns the antilocal languages as easily as matched controls.

2 Related Work

Related work finds mixed evidence about whether LLMs distinguish natural from impossible languages. Across these studies, learning speed and local statistical structure emerge as important dimensions alongside final performance.

  • Learning impossible languages: Kallini et al. found that LLMs struggled with perturbed languages in both minimum achieved perplexity and learning speed.The result was interpreted as evidence that data-driven LLMs instantiate some human-relevant language-learning biases.
  • Learning impossible languages: Subsequent studies reported that LLMs often failed to distinguish attested and impossible languages, while some impossible forms were easier than attested forms across languages.These findings weakened a simple correspondence between typological status and ease of learning.
  • Learning speed: Xu et al. found similar perplexity but slower learning for perturbed word-order-harmony languages, suggesting partial rather than global typological alignment.Their results separate learning success from learning speed.
  • Information locality: Studies manipulating local statistical structure suggest information locality as an LLM learning bias, though the role of dependency-length distributions remains debated.Someya et al. reported lower success for languages with higher local entropy when global entropy was held constant.

3 Antilocal Languages

The paper defines k-antilocality through mutual information, then constructs controlled k-back languages by interleaving components generated from rotation classes. Paired controls preserve global entropy while differing in dependency organization.

  • Antilocality definition: A language is k-antilocal when a symbol has zero mutual information with every contiguous context of length n ≤ k.The construction therefore places a lower bound of k on dependency length.
  • Antilocality definition: For the studied languages, n-gram distributions are uniform for all n ≤ k + 1, and the paper shows that such languages are k-antilocal.This provides a simple operational family for varying antilocality.
  • Language construction: The authors construct eight pairs of k-back and control languages from simple trigram languages partitioned into rotation classes.Rotation classes group trigrams by cyclic symbol-transition distance and provide the language generators.
  • Language construction: Each k-back language uses k + 1 rotation classes, samples m trigrams per class, concatenates them into base components, and interleaves those components.Any window of size n ≤ k + 1 contains at most one symbol from each generator, producing uniform n-gram distributions.
  • Language construction: Control languages use the same generated components but concatenate them instead of interleaving them.Target and control sentences are permutations of one another, so corresponding languages match in global entropy.
  • Experimental control: The k-back construction fixes dependency length at k + 1, reducing confounding from uneven dependency-length distributions in natural-language datasets.This makes performance differences less attributable to having more short-dependency examples in training data.

4 Experiments

The experiments train GPT-2 on k-back and matched control languages across eight k values, measuring validation cross-entropy over training. Models reach nearly identical final losses, but increasingly antilocal languages converge more slowly.

  • Experimental setup: GPT-2 models were trained on k-back languages for k from 1 through 8, using approximately 100 million tokens per language.Each corpus contained 80,000 training, 10,000 validation, and 10,000 unused test sentences; models were trained across three random seeds.
  • Learning success: Models converged to approximately the same cross-entropy loss across k values and between k-back and control languages.Minimum losses fell within 1% of the approximately 0.596 minimum possible cross-entropy.
  • Learning success: Minimum losses consistently fell within 1% of the minimum possible cross-entropy, approximately 0.596 for all k.The minimum possible loss is the average per-token entropy of the k-back languages’ true probability distribution.
  • Learning speed: During the first 5,000 training steps, k-back models had much larger validation losses than models trained on paired controls.The loss divergence occurred for all k values.
  • Learning speed: As k increased, the period of loss divergence generally lengthened because k-back languages took longer to converge.The results indicate that greater antilocality affects convergence speed rather than final learning success.

5 Discussion

The results support a locality bias in GPT-2’s learning speed: more local languages are learned faster, while antilocal languages eventually reach comparable performance. The discussion contrasts this with prior effects on achievable loss and considers autoregression as a possible mechanism.

  • Locality bias: GPT-2 learned control languages fairly quickly, while antilocal languages reached comparable performance markedly more slowly, with delay increasing with k.The authors interpret this pattern as evidence that more antilocal languages are more difficult to learn in terms of speed.
  • Learning success: Unlike some prior work, the study found no effect of locality on achievable cross-entropy loss or, by proxy, KL divergence.All models achieved approximately optimal cross-entropy loss, corresponding to KL divergence of approximately 0, regardless of k.
  • Interpretation: The authors suggest that prior differences in achieved loss may reflect factors other than information locality, such as skewed dependency-length distributions.Their datasets held dependency length fixed, whereas prior training corpora may contain more short dependencies than long ones.
  • Possible mechanism: The mechanism behind GPT-2’s locality bias remains unclear, though it may depend on GPT’s autoregressive, incremental design.A masked DeBERTa model reportedly learned k-antilocal languages as easily as matched controls, despite architectural differences.

Limitations

The study’s conclusions are bounded by its narrow language constructions, simple controlled datasets, limited parameter settings, and restricted architectural comparisons. These choices leave broader language implications and fine-grained architectural effects unresolved.

  • Language constructions: The study explores only one type of antilocality, produced by interleaving trigrams from distinct generators.Other methods of constructing antilocal languages may yield additional insights.
  • Experimental scope: Increasing k further or varying generation parameters could produce different effects, but longer sequences would make training substantially more compute-intensive.The study used trigrams, six-token vocabularies, single equivalence-class generators, and approximately 1,000-token sentences.
  • External validity: The simple languages may have limited implications for LLM learning of much more complex human language.The authors also caution that k-back and control languages differ in other properties, including crossing dependencies.
  • Architectural scope: The work trains only one GPT-2 and one comparably sized DeBERTa architecture, limiting conclusions about scaling and architectural properties.It cannot answer fine-grained questions about how features such as positional encoding influence locality bias.
  • Scope: The study intentionally focuses on a highly controlled and structured setting, leaving broader variations for future work.The authors frame the work as a first step toward understanding locality effects on LLM learning.

A Antilocality and N-gram Uniformity

The paper defines k-antilocality through zero mutual information between a symbol and any context of length n ≤ k, and relates this condition to uniform n-gram distributions.

  • Definition: A language is k-antilocal when I(W; Cn) = 0 for every context length n ≤ k.W denotes a symbol and Cn an n-length context window.
  • N-gram Uniformity: Uniform n-gram distributions for n ≤ k + 1 are used to construct languages satisfying k-antilocality.The paper demonstrates the implication algebraically through equivalent mutual-information and entropy conditions.
  • Information-Theoretic Equivalence: Zero mutual information is equivalent to additive joint entropy: H(W, Cn) = H(W) + H(Cn).The derivation proceeds by showing equal conditional and marginal probabilities, then taking expectations and negating both sides.
  • Construction Basis: The constructed languages use trigrams over the six-symbol vocabulary V = {a, b, c, d, e, f}.Table 1 summarizes the features of these k-back languages.

B Language Generation

The authors generate k-back languages by combining randomly sampled trigram components from rotation-class generators, with k controlling the number of interleaved components.

  • Language Generation: The study generates k-back languages for 8 values of k.Each language is designed to vary antilocality while retaining a common construction framework.
  • Sampling: Each corpus contains 100,000 sentences of approximately 1,000 tokens, totaling about 100 million tokens.For each k, k + 1 generators are selected from rotation classes over V = {a, b, c, d, e, f}.
  • Sentence Length: The value of m is chosen to produce sentences of length L_sentence = 1,000, using a floor operation so m is an integer.Start and end symbols contribute an additional factor of 2 in the sentence-length formula.
  • Entropy Control: All k-back languages have approximately equal entropy because their sentences are equiprobable and m(k + 1) is approximately constant.The resulting entropy is roughly 3 log |V| across languages.
  • Measurement Caveat: The reported per-token cross-entropy is slightly unstable because sentence lengths vary with k due to the floor operation in m.The ideal value is H(L)/1000 ≈ 0.596, but actual sentence lengths differ slightly.

C Model Training Details

The experiments train GPT-2 models with a fixed optimization setup and evaluate validation cross-entropy repeatedly throughout training, using early stopping near analytical optima.

  • Training Configuration: GPT-2 models are trained on a single NVIDIA A40 GPU with batch size 16, maximum learning rate 1 × 10^-5, and sequence length 1,024.The learning-rate schedule uses constant learning rate with linear warmup for the first 1,000 steps.
  • Training Configuration: Models train for a maximum of 4 epochs, a duration found sufficient for convergence on the hardest k-back language in preliminary tests.The setup follows the stated implementation configuration for the GPT-2 experiments.
  • Evaluation: Validation uses 10,000 held-out examples evaluated every 100 training steps, yielding 200 evaluations.When early stopping occurs, later losses are held constant at the last recorded value.

D Supplementary DeBERTa Results

DeBERTa-v3 reaches comparable final cross-entropy across antilocal and control languages, while its convergence speed shows no clear dependence on antilocality; higher k instead slows both language types.

  • Model and Evaluation: DeBERTa-v3 is a 184M-parameter masked transformer evaluated with a 15% mask probability.Its early-stopping target is within 0.0150 of the optimal masked-token loss, 0.0403.
  • Results: DeBERTa-v3 achieves comparable cross-entropy loss across all languages regardless of antilocality.Figure 6 states that all models eventually achieve optimal performance on all languages.
  • Results: Higher k slows learning for both k-back languages and matched controls, so the slowdown cannot be attributed to antilocality.The authors attribute the global slowdown to the greater number of distinct generators and patterns at higher k.
  • Convergence Speed: DeBERTa learns k-back languages about as quickly as corresponding controls, with no clear trend toward slower learning for more antilocal languages.For some cases, such as k = 6, controls were slightly faster on average.
  • Convergence Dynamics: Unlike GPT-2, DeBERTa shows no distinct random-guessing plateaus for k-back languages compared with controls.The authors therefore suggest that any information-locality bias in DeBERTa is quite weak.
  • Model Choice: Earlier BERT and RoBERTa models did not converge on these languages, reaching only the loss equivalent to a uniform distribution over six content tokens.DeBERTa was selected over those earlier masked language models for this reason.
  • Results: Figure 5 reports validation cross-entropy averaged across three seeds, with error ribbons showing standard deviation across seeds.The figure compares DeBERTa-v3 models trained on k-antilocal and matched control languages.
Loading 2608.27760v1…