Source-linked AI summary
Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence
Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kianté Brantley
TL;DR
This paper asks whether code-switched training can align multilingual representations under data constraints. It compares code-switched and monolingual corpora, finding parallel-sentence alignment and a BabyLM-suite gain under curriculum ordering.
Problem
The paper asks whether training on code-switched data can align parallel-text representations in multilingual language models.
Source: A Examples of each data type · p. 2Method
Results
Takeaways & Limitations
The findings support code-switching curricula as a data-augmentation approach for multilingual pretraining within the tested setting.
Source: §8 · p. 7LLM-generated code-switching is expensive to scale and may introduce text-distribution confounds; results cover one approximately 98M-parameter architecture at a 100M-word budget.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.
1 Introduction
The paper asks whether code-switching can improve multilingual learning under BabyLM’s 100M-word constraint. It studies code-switched pretraining for downstream performance and cross-lingual representation alignment.
1 Introduction
- The study targets data-constrained multilingual modeling: the BabyLM multilingual track allocates a total 100M-word budget across English, Dutch, and Chinese.The motivation notes that children communicate fluently, often in multiple languages, after exposure equivalent to under 100 million English words.Source: §1 · p. 1
A Examples of each data type
- Code-switching is motivated by multilingual children’s use of multiple languages within an utterance, prompting the question of whether training on such data aligns parallel-text representations.The introduction situates this question alongside prior multilingual pretraining and fine-tuning approaches.Source: A Examples of each data type · pp. 1–2
- The authors adapt code-switching curriculum learning to multilingual pretraining and compare code-switched data with baselines using the same documents in their original languages.Their questions address downstream BabyLM performance, cross-language representation alignment, and whether words embedded in code-switched contexts are represented differently.Source: A Examples of each data type · p. 2
- Code-switched models improve BabyLM-suite performance under curriculum ordering, while shuffled ordering does not; parallel-text representations align and embedded words align more closely to translations.The curriculum proceeds from word-level to sentence-level code-switching and then monolingual documents.Source: A Examples of each data type · p. 2
2 Related Work
This work connects data-constrained multilingual pretraining, synthetic code-switching as data augmentation, and research on cross-lingual representation alignment.
Developmentally plausible language modeling
- The study adopts the multilingual BabyLM challenge’s constraints and evaluation suite, based on the BabyBabelLM corpora.Prior developmentally plausible language modeling work includes a negative finding for cognitively inspired curriculum learning and studies of synthetic data mixtures.Source: §2 · p. 2
CS as data augmentation for multilingual training
- Prior code-switching augmentation approaches use bilingual dictionaries or LLM prompting, while this study most closely follows curriculum learning but focuses on pretraining rather than fine-tuning.The cited literature also includes work questioning whether code-switched data is more important than parallel data for translation; both studies stress fine-grained lexical alignment.Source: §2 · pp. 2–3
Cross-lingual representation alignment
- Earlier studies examine shared cross-language components, alignment during training, token overlap, and typological similarity; this work adds a controlled study of code-switching curricula’s effects on alignment.The cited work also includes a survey of cross-lingual alignment.Source: §2 · p. 3
3 Synthetic Code-Switched Data Generation
The authors construct matched-budget code-switched and monolingual corpora from English, Dutch, and Chinese BabyBabelLM data, plus controls that vary the type of code-switching.
3 Synthetic Code-Switched Data Generation
- The base non-code-switched corpus combines sampled monolingual documents from the English, Dutch, and Chinese BabyBabelLM datasets.The paper also constructs four control corpora to isolate experimental factors.Source: §3 · p. 3
- For each language subset, the code-switched corpus allocates equal thirds to monolingual documents, word-level switching, and sentence-level switching.An instruction-tuned LLM translates selected words or roughly half the sentences into an embedded language; the authors check generated text programmatically and manually with bilingual speakers.Source: §3 · p. 3
- The controls isolate word-level or sentence-level switching, replace or insert sampled embedded-language words in a word-salad corpus, or append LLM translations to documents.The document-translation control discards a third of its documents to keep the word budget consistent.Source: §3 · p. 3
4 Experimental Setup
Across the tested setups, code-switched training with a three-stage curriculum ranks highest on average, while evidence for its advantage over other orderings is limited. The experiments also compare downstream evaluation outcomes and cross-language retrieval.
4 Experimental Setup
- The main experiments compare code-switched and non-code-switched corpora under shuffled and three-stage curriculum orderings, with eight random seeds per setup.Models use GPT-2-small architecture and a byte-level BPE tokenizer of vocabulary size 16,384.Source: §4 · p. 4
- The curriculum trains for 10 epochs each on word-level code-switching, sentence-level code-switching, and monolingual documents; evaluation averages per-language scores across the BabyLM task suites.The suites test grammatical fluency, cognitive plausibility, and adaptability through fine-tuning.Source: §4 · p. 4
5 Results
- CS/curriculum scores highest on average (46.72), and its improvement over non-CS/curriculum is significant (p = 0.016), but its gap over non-CS/shuffled is not (p = 0.41).Each setup exceeds the BabyLM baseline, which scores 45.94; other pairwise comparisons are not significant.Source: §5 · pp. 4–5
- Under curriculum ordering, most of the code-switched models’ advantage over non-code-switched models comes from fine-tuning tasks; under shuffled ordering, their zero-shot and fine-tuning averages are within noise.Figure 2 reports layerwise bitext retrieval precision@1 across English, Dutch, and Chinese, and says non-CS retrieval fails for language pairs with different scripts.Source: §5 · p. 5, Figure 2 · p. 5
6 Cross-Lingual Representation Alignment
This section tests whether code-switched (CS) training aligns sentence representations with their translations, using bitext retrieval. CS models show stronger alignment than non-CS models; alignment persists through monolingual training, but incoherent switching does not induce the same alignment.
6 Cross-Lingual Representation Alignment
- Bitext retrieval measures whether a sentence’s translation is its nearest neighbor by representation similarity, averaged across sentences in both directions.Sentence representations are mean-pooled residual-stream vectors, and cosine similarities receive CSLS adjustment.Source: §6 · p. 5
- 61.4% versus 5.5%: under curriculum ordering, the CS model retrieved English–Chinese translations more often than the non-CS baseline across layers 6–11.The comparison is an eight-seed mean; other measures also corroborate alignment for cross-script language pairs.Source: §6 · pp. 5–6
- The incoherent word-salad control did not induce the same alignment, suggesting grammatically and semantically coherent CS matters.Source: §6 · p. 6
- Alignment improved during word- and sentence-level CS training and was maintained during subsequent monolingual-document training.Figure 3 tracks sentence-level alignment across curriculum stages; line color indicates the current CS type.Source: §6 · p. 6, Figure 3 · p. 6
7 Representations of Code-Switched Words
This section examines whether CS training aligns word representations with translations, including words never embedded during word-level switching. Translation rankings improve for both word groups, with alignment especially evident across distinct scripts.
7 Representations of Code-Switched Words
- Embedded words were inserted as foreign-language words during word-level CS; never-embedded words were neither inserted nor replaced.Representations are computed from word contexts, averaging token positions, sentences, and layers 6–11.Source: §7 · p. 6
- The translation-ranking measure uses CSLS-adjusted similarity across candidate translations; Figure 4 reports medians and pair counts by language pair.The reported set contains 1,102 word pairs across four language directions with accessible bilingual dictionaries.Source: §7 · p. 6, Figure 4 · p. 7
- +0.064 for embedded words and +0.045 for never-embedded words: CS raises median translation percentile rank over the non-CS baseline, especially across distinct scripts.English–Dutch translations were already closely aligned in both models.Source: §7 · pp. 6–7, Figure 4 · p. 7
8 Conclusion
Under the three-stage code-switching curriculum, models gained on the BabyLM evaluation suite and showed cross-lingual alignment; the study is limited to one model scale and three languages. The authors also identify a possible corpus-distribution confound in the synthetic data.
8 Conclusion
- A three-stage curriculum produced a 0.36-point average BabyLM evaluation-suite gain across eight seeds, while shuffled ordering showed no statistically significant improvement.The sequence was word-level code-switching, sentence-level code-switching, then monolingual documents, compared with a matched monolingual corpus.Source: §8 · p. 7
- Code-switching aligned representations of parallel sentences, and embedded words aligned more closely with their translations than words never seen in a code-switched context.Under curriculum ordering, parallel-sentence alignment persisted through subsequent training on monolingual documents.Source: §8 · p. 7
Limitations
- The study tests only English, Dutch, and Chinese; generalization to more distant or lower-resource pairs and to more than three languages remains untested.Source: Limitations · p. 7
- The synthetic code-switching generation method is costly to scale, and text-distribution differences may confound attribution of performance gains to code-switching.The concern applies to sentence-level code-switching data and the document-translation control relative to BabyBabelLM.Source: Limitations · p. 7
- The results cover only a ∼98M-parameter GPT-2 at a 100M-word budget; whether benefits persist at larger scales is left for future work.Source: Limitations · p. 7
A Data Generation Details
The appendix documents how the multilingual corpora were assembled, quality-filtered, and budget-checked, and how the generation model and prompts were selected for different switching tasks.
A Data Generation Details
- The data use BabyBabelLM English, Dutch, and Chinese sources, with word counts based on released dataset token counts and a script-aware approximation for Chinese.Chinese counts are approximated as 0.711 units per character; short passages and Chinese passages dominated by Latin rather than Han characters are dropped.Source: A Data Generation Details · pp. 11–12
- The production system generated word- and sentence-level switching with DeepSeek-V4-Flash, while full-passage translation controls used Claude Haiku 4.5.Each switching type covered the same 413,302 passages, and total API cost was approximately $284.Source: A Data Generation Details · p. 12
- Generation outputs were filtered for presence of both languages, actual switching, and plausible length; sentence-level acceptance was lower for Chinese-matrix text.Acceptance was 0.47–0.56 for sentence-level switching into a Chinese matrix, versus 0.79–0.87 into English or Dutch matrices.Source: A Data Generation Details · p. 12
- Accepted passages were reassembled in document order, and the non-code-switched baseline used matching source passages and gaps.The code-switched and baseline corpora were near-identical in volume by the reported budget checks: approximately 96.6M versus 95.7M words.Source: A Data Generation Details · p. 12
B Generation Model and Prompts
- DeepSeek-V4-Flash had the highest pass rate in the model-selection sample and was chosen after manual inspection across switching types.Its pass rate was 0.72, compared with 0.67 for Claude Haiku 4.5, 0.33 for Gemini 2.5 Flash-Lite, and 0.19 for Qwen3-30B-A3B.Source: B Generation Model and Prompts · pp. 12–13
- Figure 5 organizes verbatim examples by matrix language in rows and switch type in columns, with embedded-language spans colored by language.Each row’s non-code-switched cell is the true source document for its word-level example.Source: Figure 5 · p. 13
- The prompts keep the matrix language grammatically dominant for word-level switching, while sentence-level prompts translate roughly half the sentences and leave the rest in the matrix language.The document-translation control instead requests a faithful translation of the entire passage into the embedded language.Source: B Generation Model and Prompts · p. 13
C Evaluation Suite
The evaluation uses the official multilingual BabyLM pipeline across zero-shot and fine-tuning tasks in English, Dutch, and Chinese where available. Its task-level scores and held-out language-modeling measure provide complementary evaluation views.
C Evaluation Suite
- The official BabyLM pipeline scores each available task in each of the three languages.Source: C Evaluation Suite · p. 13
- Zero-shot evaluations include grammar and commonsense reasoning tasks scored by comparing candidate log-probabilities; two Chinese minimal-pair tasks are hidden and excluded from averages.Source: C Evaluation Suite · pp. 13, 15
- Table 3 reports per-task, per-language accuracy as mean ± SD across eight seeds, while Table 4 reports held-out bits per UTF-8 byte, with lower values better.Source: Table 3 · p. 14, Table 4 · p. 15
- Fine-tuning evaluations cover classification and cross-lingual POS tagging, with classification trained and evaluated within one language and POS tagging jointly trained across all three.Source: C Evaluation Suite · p. 15
D Details on Bitext Retrieval
The protocol evaluates sentence alignment with bidirectional bitext retrieval on FLORES+ representations, using CSLS to reduce hubness. It also reports complementary alignment measures and a chance reference.
D Details on Bitext Retrieval
- FLORES+ dev sentences in English, Dutch, and Chinese are truncated to 128 tokens, and sentence representations average token residual-stream vectors without padding.The protocol constructs pairwise cosine similarities between source- and target-language sentence representations.Source: D Details on Bitext Retrieval · p. 15
- Beyond retrieval, the curriculum comparison reports linear CKA, cosine gap, centroid distance, and per-language MEXA at the CS model’s peak-retrieval layer.The table gives mean CS/non-CS values; lower centroid distance indicates better alignment.Source: Table 5 · p. 15
- CSLS adjusts cosine similarities using each source sentence’s nearest targets and each target’s nearest sources, mitigating hubness in nearest-neighbor retrieval.The protocol uses k=10 for this local scaling.Source: D Details on Bitext Retrieval · p. 15
- P@1 averages the fraction of correct top-ranked translations in both retrieval directions; chance is approximately 0.1%.A retrieval is correct when a sentence’s translation ranks first under the adjusted similarity matrix.Source: D Details on Bitext Retrieval · p. 15
E Details on Word-Level Alignment
This section describes how word pairs are selected for representation comparisons and reports layerwise representational differences between code-switched and non-code-switched models. It also points to figures and a final-checkpoint comparison across eight conditions.
E Details on Word-Level Alignment
- Word candidate sets group words by part of speech and subword-token count, then match embedded and never-embedded words by log10 monolingual frequency.The Hungarian algorithm makes the one-to-one matches; translation labels come from MUSE dictionaries and corpus-generation substitutions.Source: E Details on Word-Level Alignment · p. 15
- Non-CS models rank the correct translation at median rank 105 for English–Chinese and 121 for Dutch–Chinese, versus chance at 499.These are averages over layers 6–11, out of 997 candidates; the passage interprets this as general alignment that code-switching sharpens rather than creates from scratch.Source: Figure 6 · p. 16
- Figure 7 plots layerwise bitext retrieval P@1 across eight conditions, with eight-seed means, standard-deviation bands, chance, and layers 6–11 marked.The four main conditions vary corpus type and ordering; four additional conditions are shuffled-ordering controls.Source: Figure 7 · p. 16
- Table 6 compares all eight conditions at final checkpoints using layer-averaged FLORES+ P@1 by language pair and the leaderboard score.The passage identifies the table’s measures and scope but does not give cell values.Source: Table 6 · p. 17
- At layer zero, CS and non-CS word representations have high cosine similarity; early layers are less similar, while later-layer representations become more similar again.The comparison covers 2,204 words in 1,102 matched pairs.Source: Figure 8 · p. 16, E Details on Word-Level Alignment · p. 17