PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

Nikitas Theodoropoulos, Maria Lymperaiou, Giorgos Filandrianos

arXiv:2609.37121v1cs.CL

TL;DR

Evidence on cross-linguistic influence varies across studies, and orthographic model input differs from the speech children primarily acquire. This paper compares matched bilingual BabyLMs across orthographic and phonemic input and four English-L2 pairings. Phonemic models show clearer L1-related variation, with syntactic similarity aligning with persistent grammatical differences and phoneme-inventory similarity with an early lexical advantage that disappears after English exposure.

  • Problem

    Cross-linguistic influence varies across studies, while language models’ orthographic input differs from the spoken input children primarily acquire.

    Source: §1 · pp. 1–2

  • Method

    The study compares matched bilingual BabyLMs across orthographic and phonemic input, keeping English as L2 while varying L1 across four languages.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    Phoneme LMs show clearer L1-related variation: syntactic similarity aligns with persistent grammatical differences, while phoneme-inventory similarity aligns with an early lexical advantage that disappears after English exposure.

    Source: §6 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    Input modality shapes model performance and cross-linguistic influence during learning, with phonemic input showing stronger and clearer L1-related variation than orthographic input.

    Source: §6 · p. 9, §5 · p. 9

    Phoneme input remains an intermediate representation that omits prosody, timing, and stress, so the authors call for investigating cross-linguistic effects with raw speech.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Cross-linguistic effects are a central topic in bilingual first-language acquisition. Artificial learners can help investigate L1-L2 interactions by enabling controlled comparisons across language combinations and learning conditions. Recent work explores this direction by training bilingual language models under developmentally plausible constraints. However, human and model learners still diverge in fundamental ways, with one major difference being input modality: children learn primarily from spoken input, whereas language models are typically trained on orthographic text. To reduce this gap, researchers have trained models on phonemic representations of speech. In this work, we combine these research directions to train bilingual BabyLMs with phonemic input. We keep English fixed as the L2 and vary the L1 across German, Swedish, Persian, and Basque, selected to represent contrasting combinations of syntactic and phoneme-inventory distance from English. Our results show stronger L1-related variation in grammatical learning trajectories under phonemic than orthographic input, while early lexical differences align with phoneme-inventory similarity.

1 Introduction

Cross-linguistic influence in bilingual learning varies across studies, and artificial learners offer a way to compare language pairs and learning conditions systematically. This work compares matched bilingual models trained on orthographic or phonemic input to examine how modality and typological distance relate to English grammatical learning.

1 Introduction

  • Cross-linguistic influence can be asymmetric and often involves L1 effects on the weaker or later-acquired L2, particularly in morphosyntax.Its conditions and strength remain unclear, with variation across studies even for the same languages and phenomena.Source: §1 · p. 1
  • Phonemic models offer a practical intermediate representation that more closely approximates spoken input while retaining text-based modeling’s efficiency, control, and data availability.Children primarily acquire language through speech, while artificial learners have generally relied on orthographic input.Source: §1 · pp. 1–2
  • The study tests whether L1 effects on English grammatical learning differ between orthographic and phonemic input and relate to syntactic or phoneme-inventory distance.It keeps English as L2 and varies L1 across German, Swedish, Persian, and Basque.Source: §1 · p. 2
  • The work introduces a controlled framework and trains matched bilingual models with English as L2 and four L1s spanning distinct syntactic and phoneme-inventory distances.It also releases bilingual corpora in orthographic and phonemic form, the processing pipeline, trained models, and code.Source: §1 · p. 2

2 Related work

Prior work documents selective cross-linguistic influence in bilingual development and language models, while BabyLM and phoneme-based models offer complementary tools for studying acquisition. This paper extends those directions by examining L1 effects across orthographic and phonemic bilingual models.

Cross-linguistic effects

  • Bilingual children develop distinct but interacting linguistic systems, with cross-linguistic effects linked to structural overlap and varying substantially across studies.Related language-model work finds that shared training can produce positive or negative interference depending on data balance, capacity, and linguistic similarity.Source: §2 · p. 2

Bilingual BabyLMs

  • Bilingual BabyLM studies have examined effects of bilingualism, exposure onset, language dominance, similarity, and child-directed speech.The cited studies report no strong general bilingualism effect and find that dominance and similarity contribute to cross-linguistic effects.Source: §2 · p. 2
  • This work extends prior bilingual and second-language research by comparing L1-induced effects across orthographic and phonemic input modalities.Related studies examine typological similarity, learner errors, sentence processing, and L1 influence in L2 production.Source: §2 · p. 3

Phoneme BabyLMs

  • Phoneme-based BabyLM research finds substantial linguistic ability and cross-linguistic phonological learning, motivating tests of L1 influence in bilingual phoneme models.Prior comparisons report modest reductions on conventional tasks but advantages for phonological analysis.Source: §2 · p. 3

3 Methods

The methods compare matched bilingual learners across input modalities while varying L1 typological distance from English. Models are trained on paired corpora and evaluated with grammatical minimal-pair tasks and phoneme-specific diagnostics.

3 Methods

  • The controlled design compares bilingual orthographic and phoneme learners, defining cross-linguistic effects as L2 ability differences resulting from L1 influence.The same language pairs, source data, training schedule, and optimization settings are held constant where possible, with modality-specific tokenizers differing in vocabulary and embedding size.Source: §3 · p. 3

3.1 Simulating bilingual acquisition

  • Matched GPT-2 models use 85M non-embedding parameters, with paired corpora built from 10M L1 and 10M English words and phonemized using G2P+.Training begins with 10k L1-only steps, followed by 20k steps of interleaved L1 and English input; three random seeds are used.Source: §3.1 · pp. 3–4

3.2 Language selection

  • German, Swedish, Persian, and Basque sample four combinations of low or high syntactic and phoneme-inventory distance from English.English is fixed as L2, and candidate L1s need at least 10M training words; morphological and phonological distance measures were excluded for insufficient candidate coverage.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 Evaluation

  • Evaluation uses BLiMP and MultiBLiMP minimal pairs, plus BabySLM syntactic and lexical tests for targeted grammatical and phoneme-model diagnostics.BLiMP covers 67 datasets, MultiBLiMP targets subject–verb agreement across 101 languages, and BabySLM Lexical is available only for phoneme models.Source: §3.3 · p. 5, Table 1 · p. 5

4 Results

Orthographic models achieved higher grammatical accuracy overall, while phoneme models showed clearer L1-related grammatical patterns. Phoneme-inventory similarity aligned with an early lexical advantage that disappeared with English exposure; training trajectories also revealed differences that final scores can conceal.

4.1 Minimal-Pair Evaluation

  • Orthographic LMs achieved higher overall grammatical scores, but phoneme LMs showed a clearer relationship between L1 identity and grammatical performance.Figure 3 compares English evaluation accuracy by L1–English syntactic distance across phoneme and orthographic models.Source: Figure 3 · p. 6, §4.1 · pp. 6–7
  • Orthographic LMs showed only a weak, benchmark-dependent association between syntactic distance and grammatical performance, without a persistent ordering in training trajectories.Figure 5 presents orthographic-model accuracy over training.Source: §4.1 · p. 6, §4.2 · p. 7, Figure 5 · p. 8
  • German and Swedish phoneme LMs consistently outperformed Persian and Basque across grammatical benchmarks, aligning with their closer syntactic relationship to English.The authors caution that language-specific properties and the single language representing each distance configuration prevent attributing the ordering independently to syntactic distance.Source: §4.1 · p. 6
  • The reported syntactic-distance correlations for phoneme LMs were −0.94, −0.99, and −0.95 for BLiMP, MultiBLiMP, and BabySLM Syntactic, respectively, versus −0.47, +0.13, and −0.46 for orthographic LMs.With only four L1s, the authors characterize these correlations as descriptive, not evidence of a predictive relationship.Source: Figure 3 · p. 6, §4.1 · p. 7

4.2 Training Dynamics

  • After English was introduced, phoneme LMs showed sustained grammatical separation by L1, most clearly on BabySLM Syntactic, with German generally highest and Swedish above Persian and Basque.The separation was less pronounced on phonemic BLiMP and English MultiBLiMP; benchmark composition may contribute to differences in sensitivity.Source: Figure 4 · p. 7, §4.2 · p. 8
  • During L1-only training, German and Persian phoneme LMs began with higher English lexical accuracy, but all four conditions converged to near 88% after the first 1,000 steps of English exposure.Final BabySLM lexical scores were 86.8–87.3 for bilingual phoneme LMs, close to the monolingual score of 88.6.Source: §4.1 · p. 6, §4.2 · p. 8
  • The lexical trajectories suggest L1-related differences can affect learning rate rather than final performance, so final-checkpoint scores can conceal when linguistic knowledge emerges.Checkpoints were collected throughout training, including logarithmically spaced checkpoints immediately after English was introduced.Source: §4.1 · p. 6, §4.2 · pp. 7–8

5 Discussion

The findings suggest that cross-linguistic influence varies by linguistic level and learning stage, with stronger patterns under phonemic than orthographic input. The authors also identify scope limits that make these interpretations exploratory.

  • Phoneme-inventory similarity aligned with an early lexical advantage that disappeared after sustained English exposure, while syntactic similarity aligned with more persistent grammatical differences.The same broad pattern was substantially weaker and less consistent in orthographic text models.Source: §5 · p. 9
  • Orthographic input yielded higher grammatical accuracy overall, whereas phonemic input produced clearer L1-related ordering; lexical trajectories suggest influence may affect learning rate rather than final performance.This highlights a limitation of evaluating learning only at the final checkpoint.Source: §5 · p. 9
  • The interpretations remain exploratory because each distance configuration is represented by a single language and the L1 corpora differ substantially in composition.Source: §5 · p. 9

6 Conclusion

The study compared bilingual BabyLMs trained on orthographic and phonemic input. Orthographic models achieved higher grammatical accuracy overall, while phoneme models showed clearer L1-related grammatical variation and an early lexical advantage associated with phoneme-inventory similarity.

  • Orthographic LMs achieved higher overall grammatical accuracy than phoneme LMs.Source: §6 · p. 9
  • Phoneme LMs showed clearer L1-related grammatical variation, with syntactic similarity aligning with more persistent differences.Source: §6 · p. 9
  • Phoneme-inventory similarity aligned with an early lexical advantage that disappeared after English exposure.Source: §6 · p. 9

Limitations

The study’s cross-linguistic evidence is limited to four L1s paired with English as the sole L2. Its phoneme input and training setup also leave important aspects of child learning and alternative modeling conditions untested.

Limitations

  • The cross-linguistic analysis covers only German, Swedish, Persian, and Basque as L1s, with English as the sole L2.The authors recommend expanding to more typologically diverse language pairs.Source: Limitations · p. 9
  • Only GPT-2 with Goldfish-suite hyperparameters and sequential-interleaved training was tested.Other data-mixing regimes, including pure sequential or interleaved training, remain to be disentangled for phoneme LMs.Source: Limitations · p. 9
  • Phoneme input omits prosody, timing, and stress, so the authors call for investigating cross-linguistic effects with raw speech.Phonemes are an intermediate step toward children’s actual input.Source: Limitations · p. 9

A Training Details

The models use distinct tokenization setups for orthographic and phoneme input, while keeping the optimizer and learning-rate schedule fixed across training phases. Phoneme streams fit less content into the same 512-token sequence length.

Training Details

  • Orthographic LMs use a 30,000-token BPE vocabulary, while phoneme LMs use language-pair-specific tokenizers built from phonemes occurring more than 10 times.The tokenizer accounts for diverging embedding sizes of 23M versus 500k.Source: A Training Details · pp. 11–12
  • Both training phases use the same optimizer and learning-rate scheduler, with only the data source changing.Models use sequence length 512 and mixed precision (bf16).Source: A Training Details · pp. 11–12
  • At sequence length 512, the uncompressed phoneme stream fits less content per block than BPE, reducing effective context.Training took approximately 1 hour per model on one RTX 5090 GPU; three seeds across both modalities required 30 hours total.Source: A Training Details · pp. 11–12

B Detailed BLiMP Performance

BLiMP learning trajectories vary by phenomenon, and phoneme LMs show L1-related differences on selected grammatical categories where orthographic LMs show few strong cross-linguistic effects. Across modalities, scores are similar except in semantics, with some phoneme-LM phenomena converging more slowly.

Detailed BLiMP Performance

  • Trajectories differed by phenomenon: filler-gap dependencies and control/raising saturated early, determiner-noun agreement and ellipsis improved, while island effects and NPI licensing stayed near chance.English monolingual LMs generally outperformed bilingual models, particularly in morphology.Source: B Detailed BLiMP Performance · p. 12
  • Models scored highest in morphology and lowest in semantics across the linguistic fields summarized in Table 3.The table reports BLiMP accuracy across linguistic fields.Source: B Detailed BLiMP Performance · p. 12, Table 3 · p. 12
  • Phoneme LMs from German and Swedish scored higher on determiner-noun agreement, subject-verb agreement, and ellipsis, while orthographic LMs showed no strong cross-linguistic effects apart from occasional reduced Basque performance.The Basque reductions occurred for subject-verb agreement and quantifiers.Source: B Detailed BLiMP Performance · p. 12
  • Across modalities, phoneme LMs scored similarly to orthographic LMs except in semantics, where reduced phoneme-LM context may hinder long-range dependencies.Phoneme LMs also took longer to converge for irregular forms, anaphor agreement, and quantifiers.Source: B Detailed BLiMP Performance · p. 12

C Data Distribution

The data distribution is shown across languages, with the 100M-word-tier languages downsampled to 10M equivalent English words and adjusted using byte premiums. The authors chose developmentally plausible rather than parallel or category-matched datasets.

Data Distribution

  • German, Persian, and English, each in the 100M-word tier, were downsampled to 10M equivalent English words using byte premiums.The adjustment aimed to approximate English’s data distribution.Source: C Data Distribution · p. 12
  • Figure 6 shows data-category distributions across languages, while the study prioritizes developmentally plausible corpora over parallel or category-matched datasets.The authors identify data source as a factor that may influence performance, particularly for phoneme LMs.Source: C Data Distribution · p. 12
Loading 2609.37121v1…