PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models

Zhanyu Chen, Jaap Jumelet

arXiv:2610.00540v1cs.CL

TL;DR

Evidence on multilingual grammatical competence has not systematically addressed how evaluation paradigm, post-training, and language resource availability interact. The paper evaluates base and post-trained models across 101 languages using four methods, finding that post-training costs are higher for low-resource languages and native-language prompting can expose otherwise weakly expressed competence.

  • Problem

    How grammatical competence is measured varies, and the interaction of evaluation method, post-training, and language resource availability has not been systematically examined.

    Source: §1 · p. 1

  • Method

    The study evaluates base and post-trained models from six families on MultiBLiMP across 101 languages using four evaluation methods.

    Source: §1 · p. 1

  • Results

    Post-training carries a higher grammatical-competence cost for low-resource languages, while native-language prompting yields disproportionate gains and exposes weakly expressed knowledge.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Multilingual grammatical evaluation should use language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.

    Source: §1 · p. 2

    MultiBLiMP covers agreement phenomena only, so the reported patterns may not generalize to grammar more broadly; low-resource estimates are also less certain because those languages have fewer minimal pairs.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Claims about the grammatical competence of multilingual language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and language resource availability has not been systematically examined. We evaluate base and post-trained models from six families on MultiBLiMP, a syntactic minimal-pair benchmark covering 101 languages, using four evaluation methods. We report three principal findings. First, post-training degrades grammatical competence, but the magnitude of this effect is reduced unevenly by model scale, while low-resource languages bear the highest cost. Second, post-trained models retain grammatical knowledge they cannot articulate through explicit prompting, yet this is measurable only in high-resource languages, because near-chance baselines in low-resource settings leave little knowledge to hide. Third, native-language prompting recovers otherwise hidden competence on low-resource languages, demonstrating that only high-resource languages can be probed directly from unprompted probabilities. We conclude that multilingual grammatical evaluation must adopt language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.

1 Introduction

Research on multilingual LLMs has not systematically examined how evaluation method and post-training interact with language resource availability. Across 101 languages, the study finds post-training costs are larger for low-resource languages, explicit prompting misses some knowledge, and native-language prompts recover competence that direct probabilities may understate.

Motivation

  • MultiBLiMP covers 101 languages, enabling analysis of linguistic ability across resource levels where its interaction with evaluation method remains underexamined.The study measures performance across multiple metrics because the best way to assess LLM linguistic ability remains contested.Source: §1 · p. 1

Findings

  • Post-training imposes an alignment tax that is substantially higher for low-resource languages, while its variation across families is not determined by model scale.The authors point to post-training recipe factors, whose specific effects require further investigation.Source: §1 · p. 1
  • The inarticulate gap—retained grammatical knowledge not expressed through explicit prompting—is measurable only in high-resource languages, because low-resource baselines are near chance.The alignment tax and inarticulate gap are moderately negatively correlated in four of six model families.Source: §1 · pp. 1–2
  • Native-language prompting yields disproportionate gains for low-resource languages, revealing competence weakly expressed by direct probability assessment.The authors argue that multilingual grammatical evaluation should use language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.Source: §1 · p. 2

2 Related Work

Prior work finds that explicit grammaticality questions can perform near chance even when probability comparisons align more strongly with human judgments. Cross-lingual studies have largely relied on log-probability, and the interaction of multiple evaluation methods with language diversity had remained unexamined.

2.1 Evaluating Linguistic Ability

  • Metalinguistic prompting can yield near-chance accuracy and a yes-response bias, while log-probability comparisons have shown stronger alignment with human judgments.Prior work argues that metalinguistic failures may reflect instruction-following demands rather than grammatical incompetence.Source: §2.1 · p. 2
  • The inarticulate gap—knowledge retained by post-trained models but not articulated through chat—had been studied only in English.Song et al. (2025) compared minimal-pair and metalinguistic evaluation on the same items.Source: §2.1 · p. 2

2.2 Multilingual Evaluation

  • Prior multilingual minimal-pair evaluations used log-probability alone, while no existing work applied multiple methods across a typologically diverse language set.The paper addresses this gap by comparing four methods across 101 languages spanning four resource tiers.Source: §2.2 · p. 2

3 Methods

The study compares four evaluation strategies on multilingual minimal pairs, using equal-weighted language accuracy across a broad set of languages and model families. Its methods distinguish probability-based judgments from explicit instruction-following, and assess both resource tiers and paired base and post-trained models.

3.1 Evaluation Methods

  • Four evaluation strategies compare discriminative sequence scoring with explicit meta-linguistic instruction following using the lm-eval framework.The meta task asks the model to select the grammatical sentence in a chat-style question-answer format and tests explicit articulation and formatting adherence.Source: §3.1 · p. 2, Discriminative Evaluation… · p. 3

Discriminative Evaluation (Likelihood-based)

  • Likelihood-based evaluation compares grammatical and ungrammatical sentence probabilities, using direct, English-prompted, or native-prompted contexts.Direct probability uses raw text for base models and each post-trained model's own chat template; native prompts use target-language instructions, with shortened or attested cues for 14 languages.Source: Discriminative Evaluation… · pp. 2–3

3.2 Data

  • MultiBLiMP tests 121,305 minimal pairs across 101 languages, focusing on subject-verb and subject-participle agreement; language accuracy is averaged equally.Resource tiers use Common Crawl frequency as a proxy for web-scale language data because the evaluated models' training data are not released.Source: §3.2 · pp. 3–4, Discriminative Evaluation… · p. 3

3.3 Models

  • The study evaluates six open-weight model families with base and post-trained checkpoints, including TinyAya, designed with emphasis on low-resource language representation.The model set includes dense and mixture-of-experts architectures and spans multiple parameter scales.Source: §3.3 · p. 4

4 Experiments Results

Across six model families and 101 languages, grammatical-competence results vary with evaluation method, model variant, and language resource availability. Post-training generally degrades direct judgments, while native-language prompting and scale affect how much competence is expressed or measured.

4.1 General Model Performance

  • Base Gemma 4 with native prompting reaches 88.36% accuracy, and native prompting with base models has the highest average accuracy across models.Discriminative-method averages range from 78.32% for direct PT to 83.98% for native prompt base; meta averages are 69.6% for PT and 70.06% for base.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2 Direct Evaluation

  • Direct accuracy rises with resource availability, base models outperform PT models in every tier, and Super High versus Low/Zero accuracy differs by about 30–35 percentage points.Base Gemma 3 and Gemma 4 reach 98.60% and 98.86% on Super High languages; Low/Zero scores cluster at 60–69%.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 The Linguistic Alignment Tax

  • Post-training produces a linguistic alignment tax, and scaling reduces it unevenly: Low/Zero languages retain the highest tax at the largest scale.The normalized tax excludes languages with base accuracy at or below chance. Across families, tax ranges from 1.4% for TinyAya to 30.9% for Gemma 4, despite comparable base accuracies.Source: §4.3 · p. 6

4.4 The Inarticulate Gap

  • The inarticulate gap is the difference between Direct and Meta PT accuracy; base models also show positive gaps, providing a baseline for interpreting post-training.For Gemma 3, the Super High gap narrows from 0.473 at 1B to 0.070 at 27B.Source: §4.4 · p. 7, Base Models as a Format Mismatch… · p. 7

Base Models as a Format Mismatch Baseline

  • Direct accuracy exceeds Meta accuracy for most PT language-size pairs, but the gap is primarily a high-resource phenomenon because Low/Zero Direct accuracy is near chance.The share of pairs with Direct accuracy above Meta ranges from 66.7% in Low/Zero to 92.5% in Mid; scaling leaves the Low/Zero gap essentially flat (0.035 to 0.047).Source: Base Models as a Format Mismatch… · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
  • Alignment tax and inarticulate gap move in opposite directions in four of six families, while the base gap predicts the PT gap across families.The base-to-PT gap correlation is ρ = 0.603 (p < 0.001); the alignment-tax and inarticulate-gap correlation is absent in Qwen 3 and Qwen 3.5.Source: Base Models as a Format Mismatch… · pp. 7–8, Figure 6 · p. 8

4.5 Language-specific Prompting

  • Native-language prompts yield larger gains for low-resource languages, where they help express grammatical knowledge that is weakly reflected in direct probabilities.High-resource gains approach zero; smaller models also benefit from explicit prompting in mid- to high-resource languages.Source: §4.5 · p. 8

4.6 Item-level Analysis of the Alignment Tax

  • For Gemma 4 (31B), post-training losses outnumber gains 14.5% to 2.9%, and Low/Zero items remain more likely to be lost than Super High items at equal margin.At equal margin, Low/Zero loss odds exceed Super High odds by a factor of 2.3–4.3 in every family; less frequent agreement features also have higher raw loss rates, including 37.4% for dual versus 11.7% for third person.Source: §4.6 · pp. 8–9

5 Discussion

Post-training filters multilingual grammatical knowledge, with losses shaped by language-resource availability and evaluation method. Native-language prompting improves low-resource performance, while the effects of chain-of-thought reasoning remain open.

5 Discussion

  • Post-training degrades grammatical competence across languages, with losses amplified in low-resource languages and the inarticulate gap driven by language-level properties.The findings indicate that post-training filters pretraining knowledge according to resource availability and evaluation method.Source: §5 · p. 9
  • Metalinguistic accuracy ranges from 55% for Llama 3 to 84% for Gemma 4, conflating grammatical knowledge with instruction-following ability across model families.The authors recommend reporting multiple evaluation paradigms rather than treating any single method as definitive.Source: §5 · p. 9
  • Providing evaluation instructions in the target language consistently improves accuracy for low-resource languages, suggesting direct evaluation may underestimate their performance.Source: §5 · p. 9
  • The protocol disables thinking mode where supported, leaving open how chain-of-thought reasoning affects grammatical articulation.Source: §5 · p. 9

Limitations

The paper limits its conclusions to agreement processing and cautions that resource-tier comparisons may not reflect models’ actual language exposure. It also notes that post-training differences cannot be attributed to specific recipe components.

Limitations

  • MultiBLiMP covers only subject-verb and subject-participle agreement, so the reported patterns do not generalize to grammar broadly.Low-resource languages also have fewer minimal pairs, making their performance estimates more uncertain.Source: Limitations · p. 10
  • Resource tiers use Common Crawl counts, while unreleased training data leave actual per-language pre-training exposure uncertain across model families.The authors therefore caution against cross-family tier-level comparisons.Source: Limitations · p. 10
  • Post-training procedures are undocumented, so the order-of-magnitude variation in alignment tax cannot be attributed to a specific recipe component.Tokenizer segmentation, chat-template formatting, and architecture could also contribute.Source: Limitations · p. 10

A Language List

  • Table 3 organizes 101 MultiBLiMP languages by resource tier and reports each language’s number of minimal pairs.Its tier labels use CC-token thresholds, from Super High (>10^11) to Low/Zero (<10^7).Source: Table 3 · p. 13

B Average Model Accuracy

Table 4 reports average model accuracy across four evaluation methods, with results organized by model size and base or post-trained variant. TinyAya’s performance is skewed toward Super High-resource languages.

B Average Model Accuracy

  • Table 4 reports average model accuracy by evaluation method, model size, and base or post-trained variant.The accompanying text identifies four evaluation methods but supplies no accuracy values here.Source: B Average Model Accuracy · p. 14, Table 4 · p. 14

C TinyAya Performance

  • TinyAya underperforms models up to 9B parameters on low- and mid-resource languages while showing a strong skew toward Super High-resource languages.Figure 9 marks TinyAya with a blue star.Source: Figure 9 · p. 15
  • Figure 9 compares TinyAya with all other models up to 9B parameters.The figure caption highlights TinyAya’s resource-tier skew and underperformance on low- and mid-resource languages.Source: Figure 9 · p. 15

D Cross-Family Results

The cross-family results section reports per-language Mean Direct accuracy for model variants. Its CC-100 analysis uses within-family regressions and a cross-family mixed-effects model.

D Cross-Family Results

D.1 CC-100 Frequency and Base Accuracy

  • Table 7 reports within-family OLS regressions predicting base accuracy from log-transformed CC-100 corpus size and model scale, plus a cross-family mixed-effects model.The mixed-effects model includes model as a random intercept.Source: §D.1 · pp. 17–18, Table 7 · p. 18

D.2 Log-Probability Differences

For Gemma 3 (27B, base), mean grammatical-versus-ungrammatical log-probability differences decline across resource tiers, and some Low/Zero languages have negative means. At the item level, the lower end differs more across tiers than the upper end.

D.2 Log-Probability Differences

  • 9.18, 8.02, 6.53, and 2.85 natural-log units are the tier-averaged log-probability differences from Super High to Low/Zero for Gemma 3 (27B, base).Five Low/Zero languages have negative means; Tupinambá is −6.06 across 9 items and Warlpiri is −5.50 across 12 items.Source: §D.2 · p. 18, Figure 10 · p. 18
  • Five Low/Zero languages have negative mean log-probability differences, indicating higher average probability for the ungrammatical sentence in those languages.The smallest means reported are Tupinambá’s −6.06 (9 items) and Warlpiri’s −5.50 (12 items).Source: §D.2 · p. 18
  • The p90 values are 15.0, 14.0, 14.0, and 14.0 across resource tiers, while p10 falls from 3.5 to −5.0.The share of items separated by at most one natural-log unit rises from 2.6% to 14.2% across tiers.Source: §D.2 · pp. 18–19, Table 8 · p. 19

D.3 Raw Alignment Tax

Raw percentage-point alignment tax does not consistently peak in the Low/Zero tier. Its near-chance base accuracy leaves fewer points to lose, which normalization accounts for.

D.3 Raw Alignment Tax

  • Raw percentage-point tax is not systematically largest in Low/Zero: it peaks in Mid for Gemma 4 and is smallest in Low/Zero for TinyAya.Table 9 reports raw tax for the largest model in each family, averaging equally across languages within each tier.Source: §D.3 · p. 19, Table 9 · p. 19
  • Near-chance Low/Zero base accuracy leaves few percentage points available to lose, so raw tax does not make the degradation comparable across tiers.The normalized tax divides by Accbase −0.5 to account for this difference.Source: §D.3 · p. 19
  • Table 9 reports raw Accbase − AccPT percentage-point differences by tier and overall, without excluding languages.Values average equally across languages in each tier and across all 101 languages in the final column.Source: Table 9 · p. 19

D.4 Small Test Sets in the Low/Zero Tier

The Low/Zero tax analysis tests sensitivity to small language test sets by progressively removing the smallest sets. The accompanying inarticulate-gap table reports means by resource tier and model family for the largest models.

D.4 Small Test Sets in the Low/Zero Tier

  • At 27B, Low/Zero remains the highest-tax tier across cutoffs, with tax between 17.2% and 20.6%, though its confidence interval overlaps Mid throughout.The other tiers remain at 7.1%, 9.3%, and 16.1%; at 12B the estimate is unstable and includes zero at every cutoff.Source: §D.4 · p. 20
  • For Gemma 3 (27B), Low/Zero Direct accuracy rises from 66.7/61.7 across all 36 languages to 72.0/67.9 across the 12 largest, while base exceeds post-trained accuracy at every cutoff for all six families.The 4B tax rises from 33.0% to 43.1% as the smallest test sets are dropped; the 12B-to-27B rebound remains non-significant under every cutoff.Source: §D.4 · p. 20

D.5 Gap by Resource Tier and Family

  • Table 11 reports mean inarticulate gaps by resource tier and model family for each family’s largest model.The supplied passages specify the table’s scope but not its cell values or comparative outcomes.Source: §D.5 · p. 20, Table 11 · p. 20

D.6 Substitute Native Cues and the Prompt Gain

The analysis tests whether non-translated native cues explain Low/Zero Prompt Gain. The substitute-cue group has lower raw gains, but accuracy-controlled comparisons show that these cues understate rather than create the reported gradient.

D.6 Substitute Native Cues and the Prompt Gain

  • Across 828 Low/Zero language–model pairs, substitute cues average +0.8 percentage points Prompt Gain versus +2.8 for translated cues, with substitute-group median exactly zero.The groups comprise 13 substitute-cue and 23 translated-cue languages across 23 post-trained models.Source: §D.6 · p. 20
  • Excluding the 13 substitute-cue languages raises Low/Zero’s mean Prompt Gain from +2.1 to +2.8 points, above Mid’s +2.4.The analysis was motivated by the possibility that non-translated cues among 14 languages, 13 in Low/Zero, could affect that tier’s reported gain.Source: §D.6 · pp. 20, 22
  • After controlling for Direct accuracy, substitute cues have a lower Prompt Gain coefficient than translated cues, and the ordering holds in 22 of 23 models.The coefficient grows in magnitude from −2.0 to −4.0 percentage points (p < 0.001); the ordering also holds below and above 0.6 Direct accuracy.Source: §D.6 · p. 22
  • Native cues are direct translations for 87 languages; among 14 exceptions, 12 use shortened language-naming cues and two use attested phrases.The table identifies these as cases where the full English-template wording could not be rendered.Source: §D.6 · p. 23
Loading 2610.00540v1…