Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Introduction

• The study asks whether LMs transfer hierarchical structure from L1, since scaling and non-linguistic factors such as position embeddings could also explain cross-lingual gains.The authors argue that controlled counterfactual ablations are needed to study the causal effect of L1-to-L2 transfer.Source: §1 · p. 2
• The CPOS design makes L2 ambiguous between linear and hierarchical rules while varying the L1 evidence for hierarchical generalization.The paper controls direct evidence from agreement constructions and indirect evidence from deeply nested constructions.Source: §1 · p. 2
• Across English, German, Russian, and Japanese as L1, positive hierarchical evidence improves L2 English generalization, but transfer is imperfect and varies by language pair.German as L1 supports more successful English generalization than Japanese as L1.Source: §1 · p. 2
• In our analysis, eliminating vocabulary overlap—even when the same English corpora are used for L1 and L2 but encoded with disjoint token IDs—leads to clear transfer failure, making overlap a primary condition.Source: §1 · p. 2

2.1 Cross-lingual Transfer and the L2-LM Paradigm

• Cross-lingual transfer research treats L1-to-L2 continued pretraining as both a practical method and a probe of what knowledge transfers, with L1 variation and fixed L2 helping isolate source-language effects.Prior studies also examine typological distance, inductive-bias transfer, and crosslinguistic priming (Yadavalli et al., 2023; Oba et al., 2023; Papadimitriou and Jurafsky, 2020; Arnett et al., 2025).Source: §2.1 · p. 2
• Prior work reports that shared subword vocabulary drives cross-lingual transfer and embedding-space alignment (Kallini et al., 2025; Škrjanec et al., 2026).The paper adopts token-ID offsetting to control vocabulary overlap.Source: §2.1 · p. 3

2.2 Hierarchical Generalization and Poverty-of-Stimulus Design

• Poverty-of-stimulus tests contrast linear and hierarchical rules using ambiguous training data, then evaluate held-out cases where the rules predict differently.With an intervening attractor, linear agreement follows the nearest noun, whereas hierarchical agreement follows the grammatical subject (Gulordava et al., 2018).Source: §2.2 · p. 3
• Earlier hierarchical-generalization studies vary evidence, embedding depth, objectives, training duration, and modalities, but their monolingual designs leave cross-language transfer unresolved.Some studies using small, developmentally plausible inputs report failure to acquire hierarchical rules (Yedetore et al., 2023; Padovani et al., 2025).Source: §2.2 · p. 3

3.1 Cross-lingual Poverty-of-Stimulus (CPOS) Setting

• CPOS varies positive hierarchical evidence in L1, then tests whether LMs use it to generalize in L2; the study adopts POS experimental settings without claiming implications for the original POS debate.The design trains on evidence-bearing L1 data and continues training on L2 data without that evidence.Source: §3.1 · pp. 3–4

3.2 Evaluation Task (L2)

• L2 subject-verb agreement distinguishes linear from hierarchical generalization on attractor examples, where the nearest noun and grammatical subject can predict different agreement.Agreement was selected because it supports clear positive-evidence definitions and is attested across many languages.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Positive Evidence for Hierarchical Generalization (L1)

• L1 positive evidence includes intervening noun phrases with number mismatches and deeply embedded clauses, which can cue hierarchical agreement and sentence structure.The study conservatively removes intervening noun phrases regardless of their morphological features and filters sentences above an embedding-depth threshold N.Source: §3.3 · pp. 4–5

3.4 L1 Training Data Creation

• For English, German, Russian, and Japanese, the authors create equal-token-count Ambiguous and Disambiguating Wikipedia corpora by parsing and classifying sentences for positive evidence.The Disambiguating version replaces sentences without positive evidence with positive-evidence sentences, isolating evidence availability in L1 training.Source: §3.4 · p. 5

4 Experimental Settings

• The experimental settings test hierarchical generalization by varying L1 evidence exposure and source language.The overall design crosses HIER →LIN and LIN →LIN with four L1 languages.Source: §4 · p. 5, §4.2 · p. 5

4.1 Condition 1: Exposure to Positive Evidence in L1

• L1 training varies exposure to disambiguating positive-evidence data, while L2 training uses ambiguous data.The two L1 regimes differ only in whether positive evidence is available.Source: §4.1 · p. 5

4.2 Condition 2: Choice of L1

• The study varies L1 across English, German, Russian, and Japanese while keeping L2 fixed at English.The language set spans typological properties relevant to subject–verb agreement and attractor configurations.Source: §4.2 · p. 5

4.3 Evaluation Metrics

• Accuracy measures whether a model assigns lower sentence-level perplexity to a grammatical sentence than to its minimally differing ungrammatical counterpart.CLAMS is the primary benchmark, while MultiBLiMP serves as a directional check.Source: §4.3 · pp. 5–6

4.4 Detailed Setup

• Models are trained on L1 and then continue pretraining on L2, with three random-seed runs averaged for each setting.The setup uses GPT-2-small; the reported full-overlap accuracies exclude OOV items.Source: §4.4 · p. 6, Table 1 · p. 6

5 Experimental Results

• On CLAMS, reported transfer effects (∆acc) decrease monotonically from English (+5.45) through German (+1.68) and Russian (+1.04) to Japanese (+0.53).Under LIN →LIN, accuracy is around chance (51.1–56.0%) for all four L1 languages; HIER →LIN yields ∆acc > 0 for every L1.Source: §5 · pp. 6–7
• On MultiBLiMP, ∆acc is English +8.53, German −2.54, Russian +0.57, and Japanese +1.98.The authors note that MultiBLiMP has far fewer items than CLAMS.Source: §5 · p. 7
• On MultiBLiMP, transfer effects are not pronounced at a particular L2-training epoch and become stable after several epochs.Figure 3 plots seed means and ±1 SD across three seeds under full vocabulary overlap.Source: §5 · p. 7, Figure 3 · p. 7

Controlling Vocabulary Overlap

• The study compares fully shared vocabularies, a 1,000-token shared bridge, and fully disjoint vocabularies by offsetting shared token IDs.For En→En, the no-overlap condition re-tokenizes the L2 English corpus with new IDs; the paper describes this as an extreme counterfactual baseline.Source: §6 · p. 7

Statistical Analysis

• A Bayesian Bernoulli-logit mixed-effects model estimates the HIER→LIN minus LIN→LIN contrast within each overlap and L1-language combination.The model uses item-level correctness across 72 runs; estimates are posterior medians with equal-tailed 95% credible intervals on the log-odds scale.Source: §6 · p. 7

Results

• Table 2 reports final-checkpoint accuracy for CLAMS and MultiBLiMP, while Figure 4 shows Δacc by L1, overlap condition, and L2-training epoch.Figure 4 lines show seed means and shaded bands show ±1 SD across three random seeds.Source: §6 · p. 7, Table 2 · p. 8, Figure 4 · p. 8
• On CLAMS, the overall transfer contrast is Δhier = 0.121 [0.092, 0.150], with a credible effect under overlap but not without it.Under overlap, Δhier = 0.144 [0.083, 0.205]; under no overlap, Δhier = 0.025 [−0.025, 0.074].Source: §6 · p. 8
• On MultiBLiMP, the En→En contrast is credible under FULL OVERLAP and 1K OVERLAP, while all non-English credible intervals include zero.The reported dependence on vocabulary overlap is not specific to CLAMS, but is observed across several datasets.Source: §6 · p. 8

• Transfer is tightly constrained by shared vocabulary, and even minimal overlap can serve as an anchor for transfer.Source: §7 · p. 8

Conclusion

• Across four typologically diverse L1s, exposure to disambiguating L1 data yielded above-chance L2 hierarchical accuracy despite no positive evidence in L2 and a frozen Transformer.The magnitude of transfer varied systematically across L1 languages.Source: §8 · p. 8
• Figure 5 plots Δacc on CLAMS and MultiBLiMP for En→En under extended training, from 8 epochs in L1 to 8 epochs in L2.Figure 6 presents CLAMS transfer contrasts as posterior medians of Δhier with 95% credible intervals.Source: Figure 5 · p. 9, Figure 6 · p. 9
• Making L1 and L2 vocabularies fully disjoint nearly eliminated transfer, whereas 1,000 shared tokens sufficed for late-training recovery to fully shared levels.Source: §8 · p. 9
• The findings indicate that hierarchical generalization acquired during pretraining is not tied to one language, but its deployment in a new language is conditional on a lexical bridge.The conclusion connects this condition to the design of multilingual tokenizers when cross-lingual reuse of syntactic abstractions is at stake.Source: §8 · p. 9

Limitations

• The experiments use GPT-2-small, four L1 languages, and English as the fixed L2; generalization beyond these choices, particularly outside Indo-European, remains untested.Varying L2 would be needed to disentangle L1/L2 status from pair-specific lexical and structural proximity.Source: Limitations · p. 9
• Freezing the Transformer during L2 training isolates transfer from embedding-layer adaptation, but findings do not directly extend to standard multilingual pretraining or joint training.Whether vocabulary-overlap dependence survives more flexible fine-tuning remains open; the study tests attractor-based subject–verb agreement and clausal embedding, not other hierarchical phenomena.Source: Limitations · p. 9
• The language gradient appears on CLAMS, but MultiBLiMP shows a credible effect only for English and a reversed German point estimate, leaving the explanations unresolved.The authors identify possible synthetic-data regularities, naturalistic-data biases, and limited benchmark items and seeds; they call for larger naturalistic benchmarks and more replications.Source: Limitations · p. 9

Ethical Statement

• The authors used ChatGPT, Gemini, and Claude for proofreading and writing assistance, manually reviewed generated text, and retained responsibility for the paper’s accuracy and integrity.They report no intent to generate or promote harmful content in the experiments.Source: Ethical Statement · p. 10

A.1 Corpus Construction Details

• Training corpora come from Wikipedia/Wikimedia for parser-domain alignment and comparable scale, and parsed nsubj sentences are classified by attractors and clausal embedding depth.Positive-evidence sentences contain attractors and have embedding depth of at least one; non-evidential sentences have no attractors and depth zero.Source: §A.1 · p. 12

A.2 Training Corpus Statistics

• Table 3 reports sentence counts and average sentence lengths for ambiguous and disambiguating partitions, while Figures 7 and 8 show extended-training accuracy changes for De→En and Ru→En.The figures use eight L1 and eight L2 epochs; lines are three-seed means and shaded bands show ±1 SD.Source: §A.2 · p. 12, Figure 7 · p. 13, Figure 8 · p. 13

A.3 Evaluation Benchmark Coverage

• Table 5 lists the hierarchical phenomena evaluated in CLAMS and MultiBLiMP and the number of minimal-pair items for each.These items are aggregated into the accuracy reported in Section 5.Source: §A.3 · p. 13

B.1 Implementation of Vocabulary Overlap.

• Vocabulary overlap is manipulated by retaining a subset of native shared tokens with identical IDs and offsetting the remaining shared-token IDs on the L2 side.FULL OVERLAP retains all shared tokens, NO OVERLAP offsets all of them, and 1K OVERLAP retains 1,000 shared tokens.Source: §B.1 · p. 13, Figure 9 · p. 13

B.2 Vocabulary Overlap and Parameter Counts

• Table 6 reports effective L1- and L2-specific vocabulary sizes and total parameter counts for each language-pair and overlap-setting combination.It separates the L1 LIN and HIER training corpora.Source: §B.2 · p. 13

B.3 Coverage of Overlap vocabulary in the Training Corpus

• Table 7 reports per-corpus token coverage across shared, disjointed, and language-specific vocabulary categories for each overlap condition.Within each pair and hierarchy condition, the three category percentages sum to approximately 100% on each side.Source: §B.3 · p. 13

C Hyperparameters

• The experiments use the same hyperparameter configuration for L1 and L2 training, except that the Transformer layers are frozen during L2 training.Table 8 reports the settings used across experiments.Source: C Hyperparameters · pp. 13–14

D Details of the Statistical Analysis

• The statistical analysis fits a separate Bayesian Bernoulli-logit mixed-effects model for each benchmark, with item and run intercepts accounting for repeated items and within-run dependence.It uses 72 runs and four chains with 4,000 posterior draws each; all R-hat values were at most 1.01, with no divergent transitions.Source: D Details of the Statistical Analysis · p. 14

E Unfreezing Experiment

• The unfrozen-Transformer ablation updates all Transformer layers during L2 training and compares FULL OVERLAP with NO OVERLAP while keeping other settings unchanged.Figure 10 presents results for both overlap conditions.Source: E Unfreezing Experiment · p. 14, Figure 10 · p. 15

F Computational Resources

• The experiments used approximately 1,116 GPU hours on NVIDIA A100 80GB and 40GB GPUs.Source: F Computational Resources · p. 15
• For English, the overlap-presence interaction estimate was +0.545 [0.047, 1.048] on the log-odds scale, with its 95% credible interval entirely above zero.The contrast is Present −No, defined as the mean of FULL and 1K overlap estimates minus the NO overlap estimate.Source: F Computational Resources · p. 17, Table 9 · p. 17