Source-linked AI summary
Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Adam Zachary Wasserman, David Beauchemin
TL;DR
BabyLM submissions through 2025 were English-only, leaving open whether child-scale findings generalize beyond English; native French evaluation also matters because translated benchmarks can confound competence with translation quality and lexical frequency. MéTRON-FR is a 125M-parameter GPT-2 pretrained five epochs on French-only MéTRON-FR Strict with Haitian-Creole lexical-oracle oversampling, evaluated on BabyLM 2025 and Quebec French benchmarks (Beauchemin et al., 2026b; Beauchemin and Khoury, 2025). 85.97% on QFrBLiMP exceeds the best native-BLiMP score posted in any other official 2026 track language; translated-data D+C helps short-input relational tasks, while inference-time-axiom B does not.
Problem
BabyLM submissions through 2025 were English-only, leaving open whether child-scale findings generalize beyond English; native French evaluation also matters because translated benchmarks can confound competence with translation quality and lexical frequency.
Source: §1 · p. 1Method
MéTRON-FR is a 125M-parameter GPT-2 pretrained five epochs on French-only MéTRON-FR Strict with Haitian-Creole lexical-oracle oversampling, evaluated on BabyLM 2025 and Quebec French benchmarks (Beauchemin et al., 2026b; Beauchemin and Khoury, 2025).
Source: §1 · p. 1Results
85.97% on QFrBLiMP exceeds the best native-BLiMP score posted in any other official 2026 track language; translated-data D+C helps short-input relational tasks, while inference-time-axiom B does not.
Source: §1 · p. 1, Limitations · p. 9Takeaways & Limitations
Native-language minimal-pair benchmarks and protocol-level diagnostics belong in standard reporting for child-scale submissions relying on single-token zero-shot scoring.
Source: §6 · p. 9A 4.90-percentage-point gap between nominally identical GLUE-axiomatic rebuilds bounds single-seed claims on that benchmark to approximately 5pp at this scale.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
We submit M\'eTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-language minimal-pair benchmarks as standard diagnostics.
1 Introduction
The 2026 BabyLM edition opens child-scale evaluation to languages beyond English, making it possible to test whether earlier lessons generalize beyond English. MéTRON-FR brings French-only pretraining to that setting and pairs native grammatical evaluation with cross-lingual diagnostics.
1 Introduction
- The Strict track’s 100M-word limit makes data curation and architectural choices central to performance and supports tractable studies of acquisition and efficiency.Submissions are evaluated on grammatical, world-knowledge, and downstream-task probes.Source: §1 · p. 1
- Earlier Strict-track submissions through 2025 were English-only, leaving open whether child-scale findings reflect language acquisition generally or English specifically.The 2026 edition lifts the English-only restriction and includes French in the 15-language BabyBabelLM corpus.Source: §1 · p. 1
- MéTRON-FR is a 125M-parameter GPT-2 trained for five epochs on a French-only corpus, evaluated with the official BabyLM pipeline and Quebec-French benchmarks.The submission uses sentence-level oversampling driven by a Haitian-Creole lexical oracle.Source: §1 · p. 1
- 85.97% on QFrBLiMP exceeds the best native-BLiMP score posted in any other official language of the 2026 multilingual track.The compared scores are 80.75% on English BLiMP, 81.70% on Dutch BLiMP-NL, and 78.60% on Chinese ZhoBLiMP; each is a separate monolingual native-language instrument.Source: §1 · p. 1
- The paper presents cross-lingual GLUE, Bilingual Lexicon Induction, and forensic ablations as evidence for its central message and reporting diagnostics.Its stated contributions include a French task-data translation and rank-16 LoRA protocol and a child-scale geometric diagnostic of cross-lingual structural alignment.Source: §1 · pp. 1–2
2 Related Work
Prior child-scale BabyLM work was English-only, while multilingual resources and native-language benchmarks now enable non-English evaluation. Related work also supplies complementary methods for measuring cross-lingual alignment and downstream transfer.
Non-English and multilingual child-scale LMs
- Strict-track BabyLM entries from 2023 to 2025 were English-only and advanced mainly through architectural innovation, while the 2026 multilingual track uses a 15-language corpus including French.Matched-architecture studies report large training-efficiency differences across typologically diverse languages, with morphologically rich languages reaching grammatical competence with orders-of-magnitude fewer tokens than English.Source: §2 · p. 2
Native non-English minimal-pair and acceptability benchmarks
- Translated BLiMP can confound model competence with translation quality and lexical frequency, motivating native-language targeted-syntax and acceptability benchmarks such as QFrBLiMP and QFrCoLA.COLE bundles grammatical probes with lexical and cultural-knowledge tasks.Source: §2 · p. 2
CL transfer and structural alignment
- Bilingual Lexicon Induction measures geometric cross-lingual alignment without translation, while Low-Rank Adaptation supports downstream transfer experiments while preserving the pretrained base.The cited work notes that BLI had not been adapted as a child-scale diagnostic linked to formal-versus-functional linguistic competence.Source: §2 · p. 2
3 Methodology
The methodology uses one submitted checkpoint to study native grammatical competence, cross-lingual transfer, and benchmark-scoring confounds. It specifies the model and corpus, describes its sampling choices, and defines the evaluation instruments and metrics.
3 Methodology
- Experiments address native grammatical competence, cross-lingual transfer through BLI and GLUE, and forensic ablations of tokenizer, prompt-template, and lexical-frequency confounds using one shared checkpoint.No checkpoint is selected separately for each experiment.Source: §3 · p. 2
3.1 Architecture and Training
- MéTRON-FR is a 125M-parameter GPT-2 pretrained for five epochs, with a five-seed reproduction at matched corpus, tokenizer, and hyperparameters used to estimate variance.Reported section 4 results give mean ± standard deviation across five seeds alongside submitted-checkpoint headline values from epoch 3, seed 42.Source: §3.1 · p. 2
3.2 Tokenizer
- The model uses a 50,000-token BPE tokenizer trained on the French training corpus; among the compared 16K, 50K, and 64K CDS-trained alternatives, 50K had the lowest perplexity and best downstream stability.The tokenizer is not treated as a tuning parameter, though a later forensic ablation swaps it while holding other listed training conditions constant.Source: §3.2 · p. 3
3.3 Corpus Composition
- The French-only corpus contains 92,469,402 words from CHILDES French, BabyBabelLM French text, and filtered French Wikipedia.The 73-entry French–English lemma bridge, expanded to 715 form pairs, supplies BLI’s seed dictionary but is not part of training; the corpus is pan-French rather than Quebec-specific.Source: §3.3 · p. 3
3.4 The Haitian-Creole Lexical Oracle
- Sentence sampling oversamples French sentences rich in lemmas whose cognates survived into Haitian Creole, using Haitian Creole as a lexical filter rather than a training source.The authors present this as an exploratory heuristic, not a validated contribution: its effect is not isolated against an unweighted baseline, which they identify as follow-up work.Source: §3.4 · p. 3
3.5 Evaluation Suite
- Evaluation combines English zero-shot probes and fine-tuned tasks with native Quebec-French benchmarks, plus BLI and cross-lingual GLUE measurements.QFrBLiMP and QFrCoLA are native Quebec-French instruments; QFrBLiMP encodes the OQLF Quebec norm, distinct from the metropolitan French target of MultiBLiMP.Source: §3.5 · pp. 3–4
3.6 Evaluation Metrics
- Minimal-pair benchmarks use zero-shot item accuracy, QFrCoLA uses fine-tuned accuracy and MCC, CL GLUE reports per-task accuracy and a five-task mean, and BLI reports held-out word-translation precision@k.CL GLUE uses rank-16 LoRA on French-translated task data; BLI uses an orthogonal Procrustes fit.Source: §3.6 · p. 4
4 Results
MéTRON-FR reaches 85.97 ± 0.17% on native Quebec-French grammatical minimal pairs, while cross-lingual evaluations show both alignment with a grammatically competent English model and task-dependent adaptation gains. The results also expose limits in translated evaluations and distinguish grammatical competence from continued loss reduction.
4.1 Native Quebec-French Evaluation
- QFrBLiMP’s anglicism bucket probes both French phonotactic and morphological regularities and sensitivity to the prescriptive OQLF norm.The authors caution that metropolitan-French benchmark results may differ on anglicism-sensitive items, while morphological and syntactic sub-scores are expected to generalize.Source: §4.1 · pp. 4–5
- 85.97 ± 0.17% overall on QFrBLiMP at epoch 3; accuracy remains within approximately 0.40pp across epochs as training loss continues to decline.The epoch-1 reference checkpoint scores 83.53% overall.Source: §4.1 · p. 4, Table 1 · p. 5, Figure 1 · p. 5
4.2 Official BabyLM Suite
- Translated BLiMP and EWoK results have important interpretive limits: translated BLiMP is not directly comparable to English-native evaluation, and EWoK remains within chance intervals.The translated BLiMP Supplement is below chance at 34.00%; EWoK is within [47.90%, 52.10%] across all four tested interventions.Source: §4.2 · p. 5
4.3 CL Structural Alignment via BLI
- 68.84 ± 8.61% p@1 alignment to GPT-2 contrasts with 25.00% for the matched-architecture English model that failed to acquire English grammar.The held-out word-pair alignment uses a frozen orthogonal map; the authors interpret the contrast as tracking acquired grammatical competence rather than training duration.Source: Table 2 · p. 6, §4.3 · p. 6
4.2 Official BabyLM Suite
- 62.80% on the BabyLM-weighted metric, at or above all Strict-track reference points re-evaluated on the 2025 pipeline.Translated BLiMP scores 76.28%, while the translated BLiMP Supplement scores 34.00%; EWoK falls within the 95% chance interval across four interventions.Source: §4.2 · pp. 5–6
4.4 CL GLUE Experiment Grid
- French-translated task data with rank-16 LoRA yields gains on BoolQ (+3.45pp), RTE (+5.13pp), and MRPC (+2.89pp), but MNLI regresses (−11.10pp).The mean across five tasks is approximately flat (−0.31pp); the WSC delta is statistically inconclusive (−1.92pp with σ = 8.88pp). The rank-16 English-LoRA control shows no consistent gain over the rank-8 baseline.Source: §4.4 · p. 6, Table 3 · p. 7
- LoRA leaves the epoch-1 base checkpoint bit-identical, preserving its 83.53% QFrBLiMP score; full fine-tuning on BoolQ shifts it to 83.36%.The authors select LoRA for its construction-time preservation guarantee.Source: §4.4 · p. 7
5 Methodological Findings and Recommendations
At child scale, single-token zero-shot scores respond principally to tokenizer and answer-template properties, with task-level competence a secondary signal at best. The ablations motivate controls for tokenizer and prompt artifacts and preference for native-language evaluation.
5 Methodological Findings and Recommendations
- At 125M parameters and 100M words, single-token log-probability scores on short-sequence benchmarks respond principally to tokenizer and answer-template properties.The authors describe these unregistered analyses as forensic diagnostics, not confirmatory tests.Source: §5 · p. 7
5.1 Tokenizer-Swap on GLUE-Axiomatic
- Only v3d’s 16K CDS tokenizer swap reproduces the v2 GLUE-axiomatic collapse, while the format-targeted instructions and EWoK-targeted Wikipedia do not regress it individually.The Table 4 ablation isolates interventions against the v2 collapse.Source: §5.1 · p. 7, Table 4 · p. 7
5.2 Placebo Control on Dictionary-Axiom Prompting
- Targeted, placebo, and tautology axioms yield similar gains under the same French answer template, so the targeted-minus-placebo effect is −0.27 pp.The authors interpret the content-independent gain as structural prompting noise, not evidence of cross-lingual lexical transfer.Source: §5.2 · p. 8, Table 5 · p. 8
5.3 Translated BLiMP Supplement and EWoK at Child Scale
- The translated BLiMP Supplement’s 34% score reflects translation-specific lexical frequency rather than the pragmatic inference phenomena probed by the English original.EWoK at n=2200 falls within the 95% chance interval [47.90%, 52.10%] across all four tested interventions.Source: §5.3 · p. 8
5.4 Recommendations
- The authors recommend native-language minimal-pair benchmarks where available, or placebo-controlled translation otherwise, and two diagnostics for single-token scoring: tokenizer-swap sensitivity and matched-length unrelated prompts.The seven-decision mapping in Table 6 is backed by experiments in the paper; every tested wrong-tool entry produced a null or negative result.Source: §5.4 · p. 8
6 Conclusion
MéTRON-FR demonstrates strong French-only child-scale results on QFrBLiMP and the BabyLM-weighted metric, while exposing measurement gaps in the suite. The authors test pretraining-language choice as a distinct axis and recommend native-language benchmarks and protocol diagnostics.
6 Conclusion
- 85.97% on QFrBLiMP exceeds native-BLiMP scores in the other official 2026 languages, while 62.80% on the BabyLM-weighted metric is at or above re-evaluated Strict-track reference points.Source: §6 · pp. 8–9
- The authors frame the results as exposing measurement gaps within the BabyLM suite.Source: §6 · p. 9
- The submission tests pretraining-language choice rather than superseding architectural work, and the authors recommend native-language minimal-pair benchmarks and protocol-level diagnostics for single-token zero-shot scoring.Source: §6 · p. 9
Limitations
The reported findings have boundaries in training-time noise, Quebec-French normative scope, and the single-language design. The authors also flag post hoc diagnostics and missing controls for interpreting cross-lingual GLUE results.
Training-time noise
- A 4.90pp rebuild gap bounds single-seed GLUE-axiomatic claims to approximately 5pp at this scale.Several per-task estimates and relative differences between rebuild variants sit within or near that noise floor, although qualitative conclusions survive it.Source: Limitations · p. 9
Quebec-French scope
- Anglicism-sensitive QFrBLiMP and QFrCoLA findings do not generalize to metropolitan French without revalidation, while morphological and syntactic findings are expected to generalize.The stated boundary reflects the more permissive metropolitan norm on lexical anglicisms.Source: Limitations · p. 9
Single-language scope
- A single-language design cannot determine whether the observed benchmark gap, BLI contrast, and tokenizer and placebo sensitivities generalize beyond French.The authors propose replication across BabyBabelLM languages with native minimal-pair instruments as a test.Source: Limitations · p. 9
Diagnostic and oracle scope
- Section 5 diagnostics are post hoc rather than preregistered, and the Haitian-Creole oracle’s contribution is not isolated against an unweighted baseline.The clean-rebuild result bounds any oracle-specific contribution to within the rebuild noise floor.Source: Limitations · p. 9
CL GLUE controls
- The CL GLUE results are not directly comparable to English-trained baselines, and the protocol lacks an English-native child-scale model control.Rank-16 English-LoRA controls isolate rank effects, but the missing control would test whether translated-French LoRA benefits any model or specifically French-trained representations.Source: Limitations · p. 9
Ethical Considerations
The study uses French research-licensed or public-domain sources and describes the limits of its Haitian-Creole curation heuristic. It frames the released model as a research artifact and identifies the Quebec-French scope of its benchmark judgments.
Data and privacy
- The French corpus combines CHILDES, French Wikipedia, and BabyBabelLM sources; the study collects no human-subjects data or adds individually identifying content beyond published source corpora.CHILDES transcripts are used within their documented research-release terms.Source: Ethical Considerations · p. 10
Curation scope
- The Haitian-Creole lexical oracle filters and oversamples French material only; no Haitian-Creole text enters training, and the procedure makes no claims about Haitian Creole or its speakers.The authors note the asymmetry of using a French-derived creole as an analytic tool for a French-only model.Source: Ethical Considerations · p. 10
Benchmark scope
- QFrBLiMP encodes the OQLF prescriptive norm, which differs from metropolitan French on probed anglicisms, so submissions should report this scope with the score.The OQLF is described as an autonomous regulatory body.Source: Ethical Considerations · p. 10
Release and use
- The 125M-parameter model trained on 92.47M words has limited generation quality and is not appropriate for user-facing applications.It is released as a research artifact for the BabyLM community, not as a deployment-ready system.Source: Ethical Considerations · p. 10
Use of AI Assistants
The authors describe generative-AI use as limited to editing author-written prose and reviewing author-written code. The paper also summarizes evaluation choices as experimentally backed mappings between decisions and tools.
Use of AI Assistants
- Generative-AI assistants were used for copy-editing author-written prose and reviewing scripts, not to generate research ideas, run experiments, analyze results, or write scientific argumentation.The authors state that they take full responsibility for the content.Source: Use of AI Assistants · p. 10
A Right Tool, Right Job Summary
- Table 6 maps evaluation decisions to experimentally backed tools, and every tested wrong-tool entry produced a null or negative result.The passage describes seven pipeline decisions.Source: Table 6 · p. 12
B Evaluation Metric Details
The appendices specify evaluation metrics, curation and adaptation procedures, reproduction choices, and supplementary benchmark and alignment findings. They include a translated-BLiMP paradigm breakdown and qualitative held-out BLI retrieval examples.
B Evaluation Metric Details
- Minimal-pair benchmarks use zero-shot item-level accuracy, QFrCoLA uses fine-tuned accuracy and MCC, CL GLUE uses rank-16 LoRA accuracy, and BLI reports Procrustes fit and held-out precision@k.The passage also gives chance MCC as 0.00 and English CoLA BERT-base MCC 0.52 as an upper-bound reference.Source: B Evaluation Metric Details · p. 12
C Haitian-Creole Oracle Procedure
- French sentences are oversampled at weight 3.00 when dense in French cognates of high-frequency Haitian-Creole lemmas; no Haitian-Creole text enters training.The mapped list uses a 333-lemma French cognate set, and unit weight is the default.Source: C Haitian-Creole Oracle Procedure · p. 12
D CL GLUE Lever Specification
- The four GLUE levers are rank-8 English-LoRA baseline, five-epoch baseline, tuned rank-16 English-LoRA, and translated-French data with rank-16 LoRA; lever B is inference-time vocabulary axioms.Lever B is placebo-tested in Section 5.2.Source: D CL GLUE Lever Specification · p. 12
E Reproduction Checklist
- Each of five reproduction seeds is evaluated at every epoch on QFrBLiMP, and its best-accuracy epoch supplies downstream evaluations; the submitted checkpoint is epoch 3, seed 42.The per-seed checkpoint is selected by argmax of QFrBLiMP overall accuracy.Source: E Reproduction Checklist · p. 12
F Per-paradigm Translated BLiMP
- 15 translated-BLiMP paradigms score above 80%, while the lowest-scoring paradigms cluster in Principle A and gender agreement.Each of the 38 paradigms contains 200 unique minimal pairs; the passage attributes the low-score clusters to French–English grammatical differences that make translation lossiest there.Source: F Per-paradigm Translated BLiMP · p. 12
G Strict-track Reference Comparison
- Table 10 reproduces Strict-track reference points under the 2025 evaluation pipeline, drawn from Charpentier et al. (2025, Table 3).These are the reference points cited in Section 4.2.Source: G Strict-track Reference Comparison · p. 13
H BLI Method Details
- BLI learns an orthogonal map by Procrustes from 194 single-token bilingual pairs and evaluates precision@k on 48 held-out pairs.The 242 retained pairs follow morphology-aware normalization and filtering for forms that are single-token in both embedding matrices.Source: H BLI Method Details · p. 13
I BLI Alignment Qualitative Examples
- Content-word mappings reach rank 1 in most held-out BLI examples, while function-word and morphologically complex inflections frequently rank 2–5.The examples are French-to-English retrievals under the learned orthogonal map aligning MéTRON-FR embeddings to GPT-2.Source: I BLI Alignment Qualitative Examples · p. 13
J Tokenizer Ablation Full Protocol
The tokenizer ablation holds training conditions constant while varying tokenizer configurations, and its evaluation protocol includes a rebuild-noise positive control. The accompanying discussion clarifies that QFrBLiMP measures the OQLF norm and describes LoRA’s preservation of the base-model score.
J Tokenizer Ablation Full Protocol
- The v3a–v3d models use the v1 architecture and training settings, run one epoch on their respective corpora, and are evaluated at chck_92M-equivalent checkpoints.The shared settings include batch size, sequence length, optimizer, learning-rate schedule, and seed 42.Source: J Tokenizer Ablation Full Protocol · p. 13
- The tokenizer comparison includes 50K-FR BPE trained on the 92.47M-word v1 corpus and 16K-CDS BPE trained on its 25M-word CHILDES and babylm-fra component.The GLUE-axiomatic set uses five translated tasks and the lowest-loss-token-among-template-options heuristic.Source: J Tokenizer Ablation Full Protocol · p. 13
- 4.90pp of v1–v3a rebuild noise is documented as a positive control for the tokenizer ablation.The rebuild uses the same recipe with different pre-tokenization-shuffle seeds.Source: J Tokenizer Ablation Full Protocol · p. 13
K Quebec French Normative Framework
- 80.15% on the anglicism sub-score reflects both French phonotactic and morphological regularities and sensitivity to the OQLF norm.QFrBLiMP tests internalization of the OQLF norm, not French grammaticality in a language-neutral sense.Source: K Quebec French Normative Framework · p. 14
- QFrBLiMP performance does not translate one-to-one to metropolitan French, where informal usage admits anglicisms that the OQLF norm disfavors.The cited contrasts include checker and matcher, which metropolitan informal usage admits, and courriel, which the OQLF prefers over email.Source: K Quebec French Normative Framework · p. 14
L LoRA Preservation Details
- Every LoRA adapter preserves the 83.53% QFrBLiMP baseline because the base parameters are bit-identical and adapter weights do not enter the autoregressive scoring head.Full fine-tuning on BoolQ shifts QFrBLiMP by −0.17pp; the tuned LoRA uses 1.29% of total parameters.Source: L LoRA Preservation Details · p. 15
M Additional Methodological Details
The additional analyses link a GLUE-axiomatic regression to output calibration and tokenizer choice, show that EWoK results remain within chance intervals, and test prompting with a matched-template placebo. They also report sample BLI retrievals and QFrBLiMP preservation after LoRA.
M Additional Methodological Details
- v2 predicted “non” for 97.60% of BoolQ items, versus v1’s “oui” for 66.60% and a gold yes-rate of 62.20%; the regression was attributed to token calibration.A single-intervention v3 ablation localized the calibration shift to the 16K CDS-trained tokenizer.Source: M Additional Methodological Details · p. 15
- With the French answer template held constant, the structural prompting effect is approximately 2 pp, while the translation-specific effect is −0.27 pp.The original 5–9pp axiom effect compared an English template without axioms against a French template with axioms.Source: M Additional Methodological Details · p. 15
- In sample BLI retrievals, content verbs and inflected past participles align at rank 1, while function words rank within the top 5.Source: Table 11 · p. 18
- Every LoRA adapter preserves the 83.53% QFrBLiMP score, while full fine-tuning shifts it by 0.17pp.Source: Table 12 · p. 18
- 7.70pp GLUE-axiomatic regression in v2 motivated the v3 ablation.The v2 model regressed on every metric in the corpus-reallocation comparison.Source: M Additional Methodological Details · p. 15, Table 13 · p. 18
- All five EWoK conditions fall inside the 95% chance interval [47.90%, 52.10%] (n=2200), and no domain shows statistically significant separation from chance.Source: M Additional Methodological Details · p. 15, Table 14 · p. 19