Source-linked AI summary
Examining the Inductive Bias of Neural Language Models with Artificial Languages
Jennifer C. White, Ryan Cotterell
TL;DR
The paper asks whether neural language-model architectures have inductive biases toward particular language types, a question complicated by typological and corpus confounds. It constructs controlled artificial languages that vary only selected grammatical features, then compares LSTMs and transformers. LSTMs show little word-order preference, whereas transformers vary substantially, and neither architecture’s bias matches attested natural-language tendencies.
Problem
Typological and corpus differences make it difficult to determine whether language-model performance reflects a specific linguistic feature such as word order.
Method
The authors engineer simple grammars with switches that generate parallel artificial languages differing only in controlled typological parameters.
Results
LSTMs show little preference across word orders, while transformers display substantial variation across artificial languages; neither bias reflects attested natural-language tendencies.
Takeaways & Limitations
Controlled grammar engineering provides a causal framework for revealing architecture-specific inductive biases in neural language models.
Takeaways & Limitations
The simple artificial grammars do not cover all constructions or all word orders found in natural languages.
Abstract
from arXiv · showhide
Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These languages are constructed to allow us to create parallel corpora across languages that differ only in the typological feature being investigated, such as word order. We then use them to train and test language models. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models. Using this method, we find that commonly used neural architectures exhibit different inductive biases: LSTMs display little preference with respect to word ordering, while transformers display a clear preference for some orderings over others. Further, we find that neither the inductive bias of the LSTM nor that of the transformer appears to reflect any tendencies that we see in attested natural languages.
1 Introduction
The paper introduces a controlled artificial-language framework for testing whether neural language models have inductive biases toward particular typological patterns. Applying it to word order, the authors find contrasting architecture-specific preferences that do not match attested natural-language tendencies.
- The framework addresses the black-box nature of modern architectures by testing inductive bias directly rather than inferring it from model structure.The motivating question is whether architecture contributes to making some language types easier to learn.
- Natural-language comparisons confound word order with other dimensions, including subject marking, agreement, adposition type, and sentence content or style.These confounds make performance differences difficult to attribute to word order alone.
- Artificial languages isolate a typological parameter by varying grammar switches while holding other linguistic and corpus factors controlled.The grammars generate parallel languages differing in features such as constituent ordering, enabling causal attribution.
- Transformers show substantial variation in perplexity across artificial word orders, whereas LSTMs perform similarly across the tested grammar variations.This contrast indicates distinct inductive biases between the two architectures.
2 Why Artificial Languages?
Prior studies struggled to identify typological effects because natural languages differ along many dimensions, while artificial-language studies offered varying compromises between realism and control. This paper argues for maximizing control with simple, extensible grammars so observed differences can support causal claims.
- 2.1 Previous Work: Artificial-language research has tested neural models on hierarchical generalization, systematic composition, and compositional structure, establishing a precedent for controlled evaluation.These studies include McCoy et al. (2018), Lake and Baroni (2018), and Bowman et al. (2015).
- 2.1 Previous Work: Ravfogel et al. (2019) found better performance on an SVO artificial English variant, but retained English constituent-internal ordering that could confound the result.The present approach removes such inherited English ordering by constructing fully artificial languages.
- 2.1 Previous Work: Correlational studies using attested languages reported negative findings, but typological confounds and limited language samples weakened strong causal conclusions.Natural languages differ across multiple dimensions, and the available set may be too small for robust correlational claims.
- 2.2 The Necessity of Artificial Languages: Natural languages maximize realism but reduce control, whereas modified natural languages provide an intermediate compromise without maximizing either property.This trade-off motivates the paper’s fully artificial alternative.
- 2.2 The Necessity of Artificial Languages: The study maximizes control with simple grammars, accepting incomplete natural-language coverage so differences between tested languages can be interpreted causally.The authors note that the base grammars can later be expanded to test additional hypotheses.
3 Constructing Controlled Languages
The paper constructs artificial languages whose parallel sentences differ only in controlled constituent-order switches, enabling causal analysis of typological effects on language models. The grammars combine simple context-free structures, morphology, and six binary ordering choices to represent a controlled subset of natural-language patterns.
- 3.1 A Fully Controlled Experiment: Parallel corpora differ only in switch-configured constituent order, allowing performance differences to be attributed to the switches rather than content, style, or sentence complexity.The same generated sentences are replicated across every switch combination and reordered as required.
- 3.2 Our Context-Free Grammar: The grammar uses pseudowords, lexical categories, noun and verb morphology, and subject–verb number agreement to provide structured but simplified language-learning environments.Agreement requires models to retain the subject’s number when predicting a verb form.
- 3.1 A Fully Controlled Experiment: The artificial languages use probabilistic context-free grammars whose switch vector reverses selected production-rule orders, producing 2^K grammars from K binary switches.For example, the S switch changes S →NP VP into S →VP NP.
- 3.3 Controlled Typological Variation: Six switches independently vary subject–verb-phrase order, verb–object order, complementizer position, prepositional-phrase order, adjective position, and relative-clause position.The S and VP switches jointly produce SOV, SVO, VOS, and OVS patterns.
- 3.3 Controlled Typological Variation: The configurations approximate English, Japanese, and Spanish, but cover only a subset of attested word orders and omit patterns such as VSO.Natural-language word order can also follow more complex or flexible rules.
4 Experiments
The experiments compare transformer and LSTM language models across matched artificial grammars and evaluate their held-out-set perplexity. Six binary switches yield 64 grammars, with repeated train–dev–test splits and averaging across runs to reduce training variability.
- 4 Experiments: The study compares transformer-based and LSTM-based language models implemented in Fairseq using average test-set perplexity as the evaluation measure.Lower perplexity indicates a better fit to held-out data.
- 4 Experiments: Six binary switches produce 64 grammars, each with 100,000 sentences whose cross-grammar counterparts differ only in constituent ordering.Equivalent sentences occupy equivalent data splits across grammars, enabling direct perplexity comparisons.
- 4 Experiments: Each architecture is trained separately on every grammar, with ten models per grammar evaluated on held-out test sentences and averaged across splits.Averaging across ten splits helps account for variability between individual training runs.
5 Results and Analysis
The analysis evaluates grammar-wise perplexity and models switch effects with mixed-effects regression while accounting for paired sentences and sentence-specific difficulty.
- 5.1 Perplexity Evaluation: Average perplexity was computed for each grammar, with lower values indicating better fit to held-out data.The averages aggregate evaluations across test splits.
- 5.2 Mixed-Effects Modeling: The mixed-effects model estimates switch and switch-interaction effects on perplexity while controlling for related parallel sentences through sentence-level random effects.Each unique sentence receives a random difficulty term, and sentence–grammar noise is modeled separately.
- 5.2 Mixed-Effects Modeling: A positive switch coefficient indicates worse performance with head-initial ordering, whereas interaction signs indicate whether shared head-directionality worsens or improves performance.Negative interaction coefficients would be expected if model biases reflected correlated headedness in natural languages.
6 Discussion
LSTMs are largely insensitive to the tested word-order switches, whereas transformers vary substantially across grammars and do not mirror word-order or headedness tendencies in natural languages.
- Differences Between Architectures: LSTMs show similar perplexities across switch configurations, while transformer perplexities vary considerably between grammars.The corresponding mixed-effects coefficients are small for LSTMs but much larger for transformers.
- Correlated Switches: Transformer single-switch coefficients favor head-final ordering, but interaction effects vary across constituent pairs rather than consistently favoring shared head-directionality.Some interactions improve performance with shared directionality, while others worsen it.
- Tendencies in Attested Natural Languages: Transformer performance does not correlate with attested word-order prevalence, performing similarly on common SOV and rare OVS languages.The analysis also finds no consistent preference for the shared headedness correlations observed in natural languages.
7 Conclusion
The paper introduces controlled artificial languages to isolate typological effects and finds sharply different word-order biases in LSTMs and transformers.
- 7 Conclusion: Controlled artificial languages eliminate corpus differences and restrict variation between grammars to selected typological features.The methodology is used to investigate LSTM and transformer language models.
- 7 Conclusion: LSTMs show little performance variation across word orders, whereas transformers vary significantly across artificial languages.These findings establish starkly different inductive biases with respect to word order.
Ethical Considerations
The authors report no ethical concerns associated with the research presented.
- The authors foresee no ethical concerns with the research presented in this paper.