Source-linked AI summary
When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM
Hanyu Liao, Yijing Chen, Tao Ji
TL;DR
Multilingual pretraining commonly fixes sampling proportions despite changing proficiency and competition for shared capacity. ARMOR-DP diagnoses sources of weakness and adjusts sampling from per-language validation loss; experiments find that dynamic protection tracks changing bottlenecks and improves on fixed protection. The paper concludes that multilingual weakness is a training-time state, not a fixed language property.
Problem
Static multilingual sampling proportions are agnostic to evolving proficiency, although constrained models must allocate shared capacity among languages.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Multilingual weakness is an evolving training state, so protection can follow the current learning state rather than a predetermined schedule.
Source: §6 · pp. 7–8The 100M capacity frontier is single-seed, and leaderboard-only tasks were scored only after the final d768 checkpoint was submitted, not used for model selection.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training. We present ARMOR-DP, a modular framework that diagnoses imbalance arising from representation, exposure, capacity, and objective pressure, and uses per-language validation loss to update sampling. Controlled experiments show that equal character budgets create a severe representation artifact for Chinese; byte-level encoding removes much of it, while static Chinese protection can subsequently make English the weakest language. The dynamic controller tracks these transitions and achieves a better score--balance tradeoff than fixed protection. On official evaluation, curriculum-dynamic checkpoints outperform a matched static Chinese-protective baseline while using fewer Chinese epochs; scaling capacity yields the strongest internal ARMOR-DP system. These results support a simple conclusion: multilingual weakness is a training-time state rather than a fixed property of a language. ARMOR-DP therefore separates diagnosis from intervention, allowing protection to follow the current bottleneck instead of a predetermined language label.
1 Introduction
Multilingual BabyLM's fixed sampling proportions can underserve whichever language currently lags, and diagnosing weakness requires looking beyond exposure. ARMOR-DP separates possible sources of weakness and uses validation loss to guide responsive sampling.
1 Introduction
- The BabyLM challenge studies language acquisition under a developmentally plausible data budget, and the 2026 BabyBabelLM track extends it to multilingual learning.Changing child-directed speech motivates responsiveness as an analogy, not as a claim that caregivers optimize language proportions.Source: §1 · p. 1
- Fixed sampling temperatures and predetermined corpus proportions ignore evolving proficiency, potentially underserving the language that currently lags.Diagnosing that weakness cannot be done from exposure alone.Source: §1 · p. 1
- ARMOR-DP separates four possible sources of weakness—representation, exposure, capacity, and objective pressure—in a modular diagnostic and intervention framework.The proposed framework turns per-language validation loss into an online sampling signal.Source: §1 · pp. 1–2
- Official evaluations show curriculum-dynamic training improves on a matched static Chinese-protective system within the challenge budget.The paper's contributions also report that the identity of the weakest language changes as interventions take effect.Source: §1 · p. 2
2 ARMOR-DP Framework
ARMOR-DP is a modular toolkit that diagnoses different sources of multilingual weakness and pairs them with probes or interventions. Its controller uses validation-loss deviations to adjust sampling as learning pressure changes.
2 ARMOR-DP Framework
- ARMOR-DP pairs four sources of weakness with corresponding probes or interventions, and components can be enabled or omitted based on empirical evidence.The controller smooths sampling updates with momentum and clips probabilities to 18%–50%, preventing starvation and constraining any language's share.Source: §2 · pp. 2–3
- UTF-8 byte units provide script-fair representation by removing the comparison artifact of sparse Chinese character symbols versus reused Latin alphabets.This is based on tokenization-free and byte-level models (Xue et al., 2022; Clark et al., 2022; Tay et al., 2022).Source: §2 · p. 2
- Exposure experiments compare balanced sampling with Chinese-protective 25/25/50 English/Dutch/Chinese ratios, including a schedule that switches halfway through.The schedules relate to temperature-based multilingual sampling and curriculum learning (Conneau et al., 2020).Source: §2 · p. 2
- Capacity interventions test language adapters and a language-conditioned attention-head routing gate, used to study allocation rather than introduce a new routing mechanism.The work situates these mechanisms alongside prior adapters and routing methods (Houlsby et al., 2019; Pfeiffer et al., 2020; Fedus et al., 2022).Source: §2 · p. 2
- Objective experiments compare rare-token weighting with pressure-aware weighting, which combines excess loss, script pressure, and exposure.Rare-token weighting is diagnostic-only and can over-protect Chinese after exposure has helped it; pressure-aware weighting reduces this failure. The final system omits the objective layer.Source: §2 · p. 2
- The controller estimates each language's relative weakness from its validation-loss deviation and reallocates sampling probability over training.Its score combines loss gap, an optional script-complexity prior, and negative feedback from current sampling; final checkpoints disable the prior.Source: §2 · p. 2
3 Experimental Setup
The experimental setup uses small diagnostic probes to isolate mechanisms, then trains submission-facing checkpoints with selected components and evaluates them using official pipelines.
3 Experimental Setup
- Diagnostic probes use equal-length 128-character windows and equal sample counts per language; the focused setup uses 180 windows per language, 64-token sequences, a 2-layer width-56 Transformer, 240 steps, and five seeds.These short runs isolate mechanisms rather than estimate full-scale competence; benchmark claims instead use 1M- and 100M-budget checkpoints.Source: §3 · p. 3
- Submission-facing checkpoints combine byte-level representation, exposure control, and dynamic sampling, while objective armor is evaluated separately as an optional diagnostic module.UTF-8 bytes map to training ids with a two-token offset reserved for padding and EOS, consistently across training and evaluation export.Source: §3 · p. 3
- The 100M frontier compares curriculum-dynamic d64, d256, d512, and d768 checkpoints with a matched d512 static Chinese-protective baseline under a ten-epoch maximum.Checkpoints use context length 512; official evaluation uses public zero-shot and fine-tuning pipelines, with leaderboard-only tasks scored after final submission.Source: §3 · p. 3
4 Results
ARMOR-DP experiments show that multilingual weakness shifts as representation, exposure, capacity, and objective interventions change training pressure. Curriculum-dynamic protection improves the matched-capacity public comparison over static Chinese protection, while larger capacity strengthens the internal system.
4.1 Balanced input is not balanced pressure
- The character-level baseline has mean loss 3.066 and gap 1.858; byte units reduce these to 2.362 and 0.185, addressing Chinese’s larger byte and character inventory.With equal 23,040-character budgets, Chinese contains 66,228 UTF-8 bytes and 1,535 unique characters, versus about 23,000 bytes and fewer than 60 unique characters for English and Dutch.Source: §4.1 · p. 3
4.2 Protection moves the bottleneck
- Exposure and capacity gains can shift the weakest language to English; static Chinese-protective adapters improve mean loss but widen the language gap.Figure 2 plots final mean validation loss against final language gap, with lower values better on both axes.Source: §4.2 · p. 4, Figure 2 · p. 4
4.3 Dynamic control turns movement into signal
- Across three seeds, adaptive exposure records two Dutch-to-Chinese and three Chinese-to-English weakest-language transitions; dynamic control keeps balance near the balanced-training level while slightly lowering mean loss.The controller evaluates per-language losses every 24 steps and updates sampling ratios with capped adaptive protection.Source: §4.3 · p. 4
4.4 Objectives must not repeat exposure
- Scheduled token-level pressure at strength 0.20 reduces local 1M gap from 0.253 to 0.240 with mean loss essentially unchanged, but public accuracy falls from 0.4634 to 0.4612.The authors retain curriculum-dynamic as the scorer; pressure-aware weighting can improve balance, but its strength requires scheduling and should not duplicate exposure protection.Source: §4.4 · p. 5
4.5 Public evaluation separates score and diagnosis
- Curriculum-dynamic achieves the highest fine-tuning mean and best average of zero-shot and fine-tuning performance, with Chinese accuracy 0.244 versus 0.210 for balanced adapters and 0.207 for static ZH50.Pressure-aware variants prioritize balance instead, with curriculum-dynamic+pressure attaining the smallest fine-tuning gap while shifting the weakest language from Chinese to Dutch.Source: §4.5 · p. 6
4.6 Focused 1M grid
- At 1M predicted tokens, curriculum-dynamic gives up 0.015 mean loss versus static ZH50 while reducing the language gap from 0.389 to 0.253; all runs remain below the ten-epoch limit.Static ZH50 makes English the weakest language; pressure-aware variants improve balance but reduce overall mean score relative to curriculum-dynamic alone.Source: §4.6 · p. 6
4.7 Capacity scaling makes dynamic protection competitive
- Scaling curriculum-dynamic capacity from d64 to d768 raises public combined score from 0.333 to 0.387, while d256 and d512 score 0.371 and 0.381.The 100M d768 checkpoint is the strongest ARMOR-DP public scorer; the official submission returned a multilingual average of 38.32.Source: §4.7 · pp. 6–7
- 0.387 public combined score for d768 curriculum-dynamic exceeds the matched d512 static ZH50 score of 0.359; d512 gaps are 0.079 versus 0.495, with fewer than 6.61 versus 9.78 Chinese epochs.The released equal-trilingual GPT-2 reference scores 0.410; the d768 checkpoint is comparable on fine-tuning, but the paper does not claim leaderboard dominance.Source: §4.7 · p. 6
5 Related Work
ARMOR-DP draws on work in sample-efficient pretraining, byte-level representation, and multilingual allocation tools, combining them around the changing identity of the weakest language.
5 Related Work
- Byte- and token-free models motivate using bytes as a diagnostic representation, not as a claim that bytes are universally optimal.The passage cites Xue et al. (2022), Clark et al. (2022), and Tay et al. (2022).Source: §5 · p. 7
- BabyLM studies sample-efficient pretraining under developmentally plausible budgets; this setting adds multilingual constraints involving script and morphology.The passage cites BabyLM Organizers (2026b,a) and Warstadt et al. (2025).Source: §5 · p. 7
- The paper combines existing tools for allocating exposure, capacity, and gradient pressure around the moving identity of the weakest language.These tools include multilingual sampling, adapters, curriculum learning, sparse routing, and loss reweighting.Source: §5 · p. 7
6 Discussion
The discussion frames multilingual imbalance as a changing training state, with weaknesses shifting across representation, exposure, and capacity. ARMOR-DP separates these factors so interventions can follow the current learning state.
6 Discussion
- Multilingual imbalance evolves as the dominant weakness shifts from representation to exposure and then capacity, making static protection progressively less effective.ARMOR-DP separates these factors and their interventions so adaptation can follow the current learning state rather than a predetermined schedule.Source: §6 · pp. 7–8
- Byte-level representation, exposure control, and dynamic control form the final BabyLM system, while objective weighting remains optional because it reduces cross-lingual gaps but modestly lowers mean score.The modular separation is intended to make interactions and future combinations easier to analyze.Source: §6 · p. 8
7 Submission Choice
The final submission is the curriculum-dynamic d768 checkpoint, selected for its overall public performance and competitive cross-lingual balance. The matched d512 checkpoint serves as the diagnostic control against static ZH50.
7 Submission Choice
- The curriculum-dynamic d768 checkpoint is the final submission, with the best overall public performance and competitive cross-lingual balance.Source: §7 · p. 8
- The matched d512 checkpoint is retained as the principal diagnostic control against static ZH50.Source: §7 · p. 8
- The submission choice distinguishes the best overall public performer from the checkpoint kept for a matched static-ZH50 comparison.Source: §7 · p. 8
8 Conclusion
The conclusion presents ARMOR-DP as protection for the language under greatest developmental pressure, with modular interventions that track changing bottlenecks. It frames the toolkit as a flexible basis for future multilingual training.
8 Conclusion
- Representation establishes a script-fair foundation, while exposure and capacity improve performance and dynamic control tracks bottleneck transitions that fixed protection misses.Source: §8 · p. 8
- The paper frames multilingual BabyLM as protecting the language currently under greatest developmental pressure.Source: §8 · p. 8
- ARMOR-DP separates diagnosis from modular, empirically instantiable interventions, offering a flexible basis for future multilingual training.Source: §8 · p. 8
Limitations
The paper’s evidence is bounded by small diagnostic runs, limited controller testing, and a single-seed large-capacity frontier. Reproduction details also clarify how evaluation and training budgets were accounted for.
Limitations
- Large-scale score variance is unmeasured because the 100M capacity frontier uses a single seed; controller results cover only three languages and one configuration.Leaderboard-only tasks were scored after the final d768 submission, not used for model selection; actual predicted non-padding tokens in 100M runs are below the maximum-token budget.Source: Limitations · p. 8
B Hyperparameters and Controller Settings
- Final checkpoints disable the script-complexity prior because it over-corrected Chinese later in training; whether a decaying prior can retain its early benefit remains open.The controller sensitivity discussion also notes that bounds and updates with substantially more than three languages remain untested.Source: Limitations · p. 9, B Hyperparameters and Controller… · p. 11
A Reproducibility Details
- Reproduction accounting uses predicted non-padding tokens, records actual consumed epochs per language, and evaluates public zero-shot and supervised tasks through separate HuggingFace and official pipelines.All reported competition-facing runs remain below the 10-epoch limit; the 100M frontier’s public fine-tuning pass contains 22 task-language pairs.Source: A Reproducibility Details · p. 10, Table 9 · p. 10, Table 10 · p. 10
C Additional Public Evaluation Details
Additional public evaluation reports a combined-score advantage for curriculum-dynamic over static ZH50 at 5M, with the same direction at 100M. The accompanying fine-tuning table identifies the public task groups underlying the final 100M d768 mean.
C Additional Public Evaluation Details
- At 5M, curriculum-dynamic beats static ZH50 on combined score, mostly because static ZH50 loses more fine-tuning accuracy; the same direction holds at 100M.Combined is the unweighted average of zero-shot and fine-tuning means.Source: C Additional Public Evaluation Details · p. 11, Table 11 · p. 11
- Table 12 reports public fine-tuning task accuracies for the final 100M d768 curriculum-dynamic checkpoint, with dashes marking language-task pairs absent from the local public suite.The expanded task table is included to identify the public task groups underlying the reported fine-tuning mean, not to tune main-paper claims.Source: Table 12 · p. 11, C Additional Public Evaluation Details · p. 11
D Negative Results and Deferred Directions
Negative results show that simple frequency-based weighting can worsen balance, while pressure-aware weighting has a balance benefit but is omitted from the final scorer because it lowers mean public score. Capacity routing remains diagnostic rather than the main scoring choice.
D Negative Results and Deferred Directions
- Language-conditioned routing gates offer an interpretable capacity diagnostic but did not beat adapters as the main scorer, so final competition-facing runs use adapters.Static ZH50 is retained as the matched baseline for the dynamic comparison.Source: D Negative Results and Deferred… · p. 11
- ZH50+rare reaches a gap of 0.481, versus 0.348 for ZH50 and 0.118 for adaptive exposure, as rare-token weighting makes English the weakest language.The weighting lowers Chinese loss after protective exposure has helped, but widens the balance gap.Source: D Negative Results and Deferred… · p. 11
- Pressure-aware weighting gives the smallest public zero-shot gap in the 1M suite and smallest public fine-tuning gap with curriculum-dynamic exposure, but lowers mean public score and is omitted from the final scorer.Pressure objectives remain a fourth-layer research direction.Source: D Negative Results and Deferred… · p. 11