PaperlayerView exact revision on arXiv

Source-linked AI summary

Typological Alignment of Stack-Based Language Models on Mildly Context-Sensitive Artificial Languages

Nadine El-Naggar, Tatsuki Kuribayashi, Ted Briscoe

arXiv:2610.02040v1cs.CL

TL;DR

Natural languages have skewed word-order distributions, and learning biases are one proposed explanation; whether stack-based models generalize to complex cross-serial dependencies remains an open question. The paper compares LM architectures and stack representations on artificial languages and assesses their alignment with typological patterns. Models with limited working memory, including RNNSUP, perform most effectively and are more typologically aligned with common word orders.

  • Problem

    Natural languages show skewed word-order frequencies, while prior SLM evaluations have not established their generalization to cross-serial dependencies in more realistic artificial languages.

    Source: §1 · p. 1

  • Method

    The study compares LM architectures and stack representations on artificial languages and assesses how their learning aligns with attested word-order distributions.

    Source: §1 · p. 2

  • Results

    Models with limited working memory, including RNNSUP, performed most effectively and were more typologically aligned with commonly occurring word orders.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The findings are consistent with the less-is-more hypothesis based on human working-memory limitations.

    Source: §7 · p. 9, §6 · p. 9

    The artificial languages simplify or exclude features found in natural languages.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Some properties of languages, e.g., subject-object-verb (SOV) word order, are more prevalent than others among the thousands of attested natural languages (NLs). Such typological commonality is often attributed to learning biases. Computational simulations, recently with language models (LMs), have facilitated the exploration of this theory. In this paper, we extend existing analyses of the relationship between LMs' learning biases and typological commonality on both data and model sides, focusing on: (i) cross-serial dependencies, the upper limit of attested syntactic complexity, and (ii) stack-based LMs (SLMs), potentially facilitating learning of hierarchical patterns. We first evaluate generalization of SLMs on cross-serial dependencies across diverse artificial languages and confirm that they struggle with such constructions. However, SLMs with limited working memory generalize better suggesting a possible basis for such inductive bias and thus the typological commonality of some word order configurations.

1 Introduction

Natural languages show skewed word-order frequencies, and learning biases have been proposed as one explanation. This study examines whether stack-based models generalize to cross-serial dependencies and how their biases align with natural-language word-order distributions.

1 Introduction

  • SOV is substantially more common than OSV among natural languages, whose typological features are not randomly distributed.Source: §1 · p. 1
  • Theories of learning bias propose that easier-to-learn grammars are more likely to survive successive (re-)learning.Source: §1 · p. 1
  • Artificial-language studies of typological word order have generally lacked cross-serial dependencies, which reflect a high level of syntactic complexity.Source: §1 · p. 1
  • Prior stack-based LM research leaves open whether these models generalize to cross-serial dependencies in more realistic artificial languages beyond binary mirror languages.Source: §1 · p. 1
  • The study asks how architectures and stack representations affect cross-serial generalization and whether model biases align with natural-language word-order distributions.Source: §1 · p. 2

2 Background

Artificial languages allow controlled study of specific properties, while typological alignment research asks how LM learning relates to natural-language patterns. Stack-based LMs add differentiable stack memory, but their generalization to complex artificial languages and typological alignment remain underexplored.

2.1 Artificial Languages

  • Grammar-generated artificial languages permit controlled experiments but may omit realistic natural-language properties, unlike counterfactual corpus modifications that can retain heterogeneous properties and errors.Source: §2.1 · p. 2

2.2 Typological Alignment of LMs

  • Typological-alignment research uses LM performance and databases such as WALS to examine whether learning biases align with common natural-language patterns.Source: §2.2 · p. 2

2.3 Stack-Based LMs (SLMs)

  • Stack-based LMs use differentiable stack memory to emulate a push-down automaton; SUP and ND variants differ in how they represent possible stack actions.ND stacks can theoretically recognize all context-free languages, whereas SUP stacks have no such guarantee.Source: §2.3 · pp. 2–3
  • Prior SLM evaluations focused on simplified artificial languages, leaving their generalization to more complex constructions and typological alignment largely untested.Source: §2.3 · p. 3

3 Artificial Language Datasets

The paper builds grammar-based artificial languages with controlled word orders and added cross-serial dependencies, then evaluates models on increasingly complex constructions.

3.1 Generalized Categorial Grammar

  • Generalized categorial grammar assigns categories to words and combines them through application, composition, coordination, and permutation operations.Slash notation encodes the position of an argument relative to the resulting category.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 GCG-based ALs

  • The grammar-based artificial languages use shared word forms across word orders, case-marked lexical items, and parser-filtered sentences without lexical category ambiguity.The modified parser enables crossed composition to parse cross-serial dependencies and their extensions.Source: §3.2 · pp. 4–5

3.3 Modifications to the ALs

  • The dataset modifications add dative and causative forms, extend long test data, and restrict base orders to SOV and VOS, yielding 32 configurations.The 32 artificial languages vary across five word-order parameters.Source: §3.3 · pp. 5–6

3.4 CSD-Focused Generalization Test Sets

  • CSD-focused tests cover relative-clause, coordinated-relative-clause, and nested-relative-clause extensions, with 20K sentences in each test set.The test constructions expand the base 3-NP 2-VP and 4-NP 3-VP patterns.Source: §3.4 · p. 6

4 Experimental Settings

The experiments compare RNN, LSTM, and Transformer models with and without stack modules. Models are trained on short artificial-language data and evaluated across word orders and test-set lengths.

4.1 Models

  • The study tests RNN, LSTM, and Transformer architectures with SUP and ND stacks, keeping model sizes near 200K parameters.The six stack-model variants combine each of the three architectures with either stack type.Source: §4.1 · p. 6
  • Model-specific searches select minibatch size and initial learning rate using validation cross-entropy, with early stopping after three unimproved epochs.Training averages 18 epochs.Source: §4.1 · p. 6

4.2 Evaluation

  • For each model and word order, training uses SHORT data, and perplexity is averaged across five random seeds on six evaluation datasets.Typological alignment is assessed using word-order counts from WALS.Source: §4.2 · p. 6

5 Experiments

Across 32 artificial languages, stack modules do not consistently improve perplexity, though RNNs with superposition stacks outperform base RNNs across evaluation sets. Typological alignment is stronger under out-of-distribution evaluation, and regression analysis identifies superposition stacks as enhancing that alignment.

5 Experiments

  • The experiments train models on short artificial-language sets and evaluate perplexity across word-order configurations, including the MEDIUM test set.Figure 2 shows MEDIUM-set perplexity distributions, with standard deviations across five runs using different random seeds.Source: §5 · pp. 6–7, Figure 2 · p. 7

5.1 Averaged PPLs

  • RNNs with superposition stacks yield consistently better perplexity than base RNNs across evaluation sets, while stack modules do not always improve perplexity.Under fixed training resources, Transformer-based SLMs and SLMs with nondeterministic stacks are not the best-generalizing architectures on these more realistic artificial languages.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Typological Alignment

  • Typological alignment is more enhanced when evaluation targets out-of-distribution generalization than in-distribution fit.On targeted cross-serial-dependency data, non-Transformer LMs with superposition stacks typically align better than Transformer-based SLMs.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Regression Analysis

  • A statistically significant negative frequency-by-stack coefficient for superposition stacks indicates they enhance alignment between model preferences and attested word-order frequencies.The regression predicts perplexity from word order, its typological frequency, architecture, stack representation, and their interactions.Source: §5.3 · pp. 8–9, Table 5 · p. 8

6 Discussion

The constrained RNNSUP model achieved the best PPLs and typological alignment, while the more expressive ND stacks showed greater training instability. The authors suggest limited working memory may regularize learning, but note that instability may also explain architectural differences.

6 Discussion

  • RNNSUP achieved the best PPLs and typological alignment despite the greater expressive power of ND stacks and Transformers.The authors interpret its constraint as a possible regularizer that guides successful generalization.Source: §6 · p. 9
  • Limited working memory may support successful generalization, consistent with the less-is-more hypothesis and prior findings on grammar acquisition.The resource limitation in this study is not dynamically relaxed during training.Source: §6 · p. 9
  • ND SLMs showed greater instability than SUP SLMs, so unstable convergence may contribute to their misalignment and the study cannot yet demonstrate ND stacks’ effectiveness statistically.Training was stopped with early stopping; the reported seed variance reflects inherent training instability.Source: §6 · p. 9

7 Conclusion

The paper evaluates RNN, LSTM, and Transformer models on artificial languages with varied word orders, focusing on cross-serial generalization and typological alignment. Models with limited working memory, including RNNSUP, performed most effectively and aligned more closely with common natural-language word orders.

7 Conclusion

  • The study evaluates RNN, LSTM, and Transformer models on artificial languages with differing word orders, focusing on test data containing cross-serial constructions.Source: §7 · p. 9
  • Models with limited working memory, including RNNSUP SLM, performed most effectively and aligned more closely with commonly occurring natural-language word orders.Source: §7 · p. 9
  • The authors describe these findings as consistent with the less-is-more hypothesis based on human working-memory limitations.Source: §7 · p. 9

Limitations

The experiments use controlled artificial languages, but these simplify or exclude features of natural languages. The authors identify omitted ambiguity and limited construction coverage, and call for further empirical and theoretical work.

Limitations

  • The artificial-language sentences mostly use English word forms to abstract away from tokenization differences, but this controlled design simplifies or excludes natural-language features.Source: Limitations · p. 9
  • The current languages exclude lexical category ambiguity and nearly all structural attachment ambiguity, motivating experiments that add ambiguity and broaden construction types.Source: Limitations · p. 9
  • The authors identify higher training and performance variance in ND-stack models and call for work on its causes, LSTM differences, hyperparameters, and theoretical understanding of SLM learning.Source: Limitations · p. 9

Ethical Statement

The paper reports no ethical concerns or content-related risks.

Ethical Statement

AI Writing/Coding Assistance Policy

The appendix documents language-only AI writing assistance, model and data settings, and the artifacts and linguistic resources used.

AI Writing/Coding Assistance Policy

A.1 Model Details

  • Models followed prior configurations, with about 200K parameters, architecture-specific layer counts, and tuned minibatch sizes and learning rates.The models used 0.1 dropout; hyperparameters were selected after five-epoch searches across ten runs.Source: §A.1 · p. 12
  • SHORT training, validation, and test sets contain approximately 70K, 10K, and 20K sentences, while each other test set contains approximately 20K.Training used four NVIDIA RTX 6000 Ada GPUs for about 140 GPU hours in total.Source: §A.1 · p. 12

A.2 Artifacts

  • The appendix lists the paper’s artifacts and the WALS and Grambank chapters used to estimate word-order frequencies.Source: §A.2 · p. 12

B SLM Variants and Ablations

The appendix reports ablation settings and seed-to-seed stability; stacked RNNs are at least as stable as base RNNs, whereas stacked LSTMs and Transformers tend to be less stable.

B SLM Variants and Ablations

C Seed variation

  • Stack-based RNNs have comparable or better PPL stability than base RNNs, while stack-based LSTMs and Transformers tend to have higher CVs than their base models.Table 10 averages CVs over 32 languages for each model and evaluation-data setting.Source: C Seed variation · p. 12, Table 10 · p. 14

Related papers

Loading 2610.02040v1…