PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilingual GSM-Symbolic: What determines capability transfer across languages?

Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard, Dan Saattrup Smart, Simon Enni, Isaac Chung, Sofie Bruun, Ayush Sunil Munot, Max Müller-Eberstein, Adnan El-Assadi, Elisa Bassignana, Gianluca Barmina, Hafsteinn Einarsson, Iben Nyholm Debess, Linda Freienthal, Lukas Galke Poech, Mike Zhang, Nicolas Legrand, Vladimir Salnikov, Yevhen Kostiuk, Zafar Hussain, Sagandeep Kaur, Agnes Toftgård, Marie Mattson, Kristoffer Nielbo

arXiv:2610.03367v1cs.AIcs.CL

TL;DR

Capability transfer across languages is understudied, and existing mathematical evaluations can confound language differences with benchmark variation. This paper introduces an extensible multilingual template benchmark to study transfer and finds that model size and reasoning reduce the low- versus high-resource language gap, but have little effect on typologically distant languages.

  • Problem

    Capability transfer across languages is understudied, and different mathematical benchmarks vary in formatting, difficulty, and subdomains, complicating comparable evaluation.

    Source: §1 · p. 1

  • Method

    The paper introduces a verified, extensible benchmark of item-matched mathematical word-problem templates across 15 languages.

    Source: §1 · p. 2

  • Results

    Model size (β = −0.25) and reasoning (β = −0.20) reduce the low- versus high-resource language gap, but have little or no effect on typologically distant languages.

    Source: CONCLUSION · p. 10

  • Takeaways & Limitations

    Performance on an unseen language can be predicted within 5.98pp, or within 4.19pp with measurements from 10 target-language templates.

    Source: CONCLUSION · p. 10

    The benchmark covers grade-school arithmetic, so strong results do not establish general capability or usability in a language.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datasets and rarely examine its determinants jointly. Identifying what predicts transfer would let us avoid exhaustive evaluation across all language pairs and let developers target the factors that limit performance in low-resource languages. To evaluate cross-lingual capability transfer, we introduce Multilingual GSM-Symbolic, an extensible multilingual mathematical dataset covering 30,000 item-matched question-answer pairs and spanning 15 languages. It utilises symbolic templates to prevent overfitting and ensure generalisation by allowing generation of millions of high-quality variations from a single sample. Using Multilingual GSM-Symbolic, we quantify the largest determinants of capability as model size ($\beta = 1.77$), language resource level ($\beta = 0.77$), reasoning ($\beta = 0.67$) and typological distance ($\beta = -0.25$). This joint estimation allows these determinants to be expressed in terms of one another: a 32B model evaluated in Marathi performs like a 10B model in English. Our findings have important implications for model developers, showing that model size and reasoning narrow the performance gap between low- and high-resource languages ($\beta = -0.27$ and $\beta = -0.20$, respectively), while similar levers have little or no effect on typologically distant languages. Overall, our analysis framework explains 92% of between-language variation, but only 23% of the model-by-language variation, and predicts a model's performance on an unseen language within 6.0pp (r=.96). Incorporating measurements from just 10 templates in the target language reduces this to 4.19pp, enabling reasonable estimates of performance with little or no downstream dataset.

1 INTRODUCTION

Cross-language capability transfer is understudied, and comparing different mathematical benchmarks can confound language effects with other differences. The paper introduces an extensible, item-matched multilingual benchmark and uses it to study transfer determinants and performance prediction.

  • Item-matched mathematical samples are needed to compare capability across languages because different benchmarks vary in formatting, difficulty, and subdomains.Source: §1 · p. 2
  • MULTILINGUAL GSM-SYMBOLIC provides verified, manually reviewed word-problem templates across 15 languages, with code that supports extension and resampling.The overview reports 1,500 localized templates and 30,000 sampled instances.Source: §1 · p. 2, Figure 1 · p. 2
  • The study jointly models model-level and language-level features and their interactions to examine what determines cross-language transfer.Source: §1 · p. 2
  • The authors report that performance on an unseen language can be predicted reasonably, with predictions improved by measurements from just 10 target-language templates.Source: §1 · p. 2

2 RELATED WORK

Prior work uses mathematical reasoning benchmarks to study LLM capabilities and has examined both model and language factors in transfer. Static or otherwise limited multilingual benchmarks and the lack of joint modeling leave a gap this paper addresses.

2.1 MULTILINGUAL MATHEMATICAL REASONING

  • GSM8K and related benchmarks use grade-school arithmetic to study multi-step reasoning, but fixed problem wording and values make reasoning difficult to separate from surface-form sensitivity or contamination.Source: §2.1 · p. 3
  • Existing multilingual resources have limitations: MGSM is a static 250-question translation, while MGSM-Pro and mGSM-Symbolic have constraints on template release, validation, or staticness.Source: §2.1 · p. 3

2.2 WHAT DETERMINES TRANSFER?

  • Prior work links transfer to model scale and target-language data, and associates target-language performance with pretraining frequency and similarity to English.Source: §2.2 · pp. 3–4
  • Although prior studies examine model and language factors, these factors and their interactions have rarely been modeled jointly.Source: §2.2 · p. 4

3 METHOD

The method builds and validates a localized, item-matched template benchmark, then models solved instances with language and model features and their interactions. The study selects models to estimate these effects and evaluates them with zero-shot translated prompts.

3.1 DATASET CONSTRUCTION

  • The dataset extends GSM-Symbolic with a multilingual parser, diverse language selection, native-speaker translation and localization, and template-level human review and automated validation.Source: §3.1 · p. 4

3.2 MODELING CAPABILITY TRANSFER

  • A crossed random-effects logistic model uses whether each matched instance is solved as its response, separating language effects from difficulty and accounting for family, model, variant, template, language, and model-by-language variation.Source: §3.2 · pp. 4–5

3.3 FEATURES

  • Language features include Common Crawl resource level, URIEL-based typological distance from English, and within-language tokenizer fertility; model size and reasoning interact with language features.Source: §3.3 · p. 5

3.4 MODEL SELECTION

  • Models are selected across family-by-size grids to estimate scale effects within families; Figure 3 indicates language differences emerge later in solution processing rather than during initial parsing.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5 EVALUATION

  • Evaluation uses zero-shot prompting with human-translated English instructions, and responses without a parseable answer are scored incorrect.Source: §3.5 · p. 6

4 RESULTS AND DISCUSSION

Model size, reasoning, and language resource level are associated with capability differences, while their effects vary across language conditions. Held-out-language estimates are useful overall but leave some model–language outcomes difficult to predict; symbolic variants are harder, and multilingual gaps accumulate throughout solution stages.

4.1 WHAT DETERMINES CAPABILITIES?

  • Larger models (βsize = 1.88, p < 0.001) and reasoning (βreasoning = 0.67, p < 0.001) improve capability, though the size effect declines with scale (βsize×reasoning = −0.34, p < 0.001).Higher-resource languages are substantially easier (βresource = 0.77, p < 0.001).Source: §4.1 · pp. 6–7

4.1.1 INTERACTIONS: HOW DO WE CLOSE THE CAPABILITY GAPS?

  • Model size and reasoning narrow the resource gap (βresource×size = −0.27, p < 0.001; βresource×reasoning = −0.20, p < .001), but size has no detected effect on typological distance.Reasoning has a small effect on distant languages (βdistance×reasoning = 0.11, p = .03), which does not reproduce under model ablations.Source: §4.1.1 · p. 7, Figure 4 · p. 7

4.2 PREDICTING PERFORMANCE ON AN UNSEEN LANGUAGE

  • A language’s average difficulty is predicted with 3.2pp error (Pearson r = 0.93), and specific model–language performance with 5.98pp error (Pearson r = 0.96), improving to 2.38pp and 4.19pp using 10 templates.Language features remove 92% of between-language variance but only 23% of model-by-language variance; model–language idiosyncrasies remain the largest held-out failures.Source: §4.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.3 SYMBOLIC VARIANTS ARE HARDER, BUT HOW MUCH IS UNSYSTEMATIC

  • Across 705 model–language pairs, symbolic instances score 3.14pp below original GSM8K questions, and this penalty is unrelated to models’ accuracy levels (r = −0.02).The penalty varies by language, from 1.7pp in English to 5.5pp in Russian, but language-specific intervals are wide; transfer-predicting features do not predict symbolic fragility.Source: §4.3 · p. 8

4.4 WHERE IN THE SOLUTION DOES THE GAP APPEAR?

  • Multilingual performance gaps accumulate across operand reads, intermediate computations, and final answers, rather than arising only from parsing.Resource effects rise across stages (βoperand = 0.32, βintermediate = 0.45, βfinal = 0.77); typological-distance effects are significant at intermediate and final stages, but not operand reads.Source: §4.4 · pp. 8–9

LIMITATIONS

The study’s estimates are bounded by its language and template coverage, grade-school math setting, model selection, and imperfect language-feature measures. These constraints qualify how broadly its transfer findings should be applied.

Size and coverage

  • 100 templates across 15 languages limit coverage; adding languages may increase the dataset’s utility, and 400 templates would be needed to halve the 2.29pp standard error for a model-language prediction.At 50 samples, the reported rank correlation is 0.99, while further sampling yields only minor gains.Source: LIMITATIONS · p. 9

Saturation

  • The grade-school problems are relatively simple, so saturation may affect estimates, although the reported finding remains stable across tested performance thresholds.Source: LIMITATIONS · p. 9

On the generalisation of our results

  • Because the study uses grade-school mathematics, its transfer determinants and effect sizes may differ in other domains.Source: LIMITATIONS · p. 9

Model selection

  • Estimates characterize instruction-aligned, non-base models and exclude Mixture of Experts models; generalization across training stages and architectures remains to be explored.Source: LIMITATIONS · p. 9

Feature specifications

  • Typological distance also encodes non-relative fertility because of its correlation with fertility, so its estimated effects require cautious interpretation.Source: LIMITATIONS · p. 9
  • Language resource is multidimensional, and Common Crawl is an imperfect proxy whose usable-text share and misclassification rates vary by resource level; model-specific training-data measures could improve fit.Comparisons across model families are difficult to interpret because family, post-training recipe, and recency are conflated.Source: LIMITATIONS · p. 9

CONCLUSION

The conclusion presents a joint account of cross-language capability transfer and its implications for model development and evaluation. It reports estimated determinants, language-gap patterns, and prediction accuracy for unseen languages.

CONCLUSION

  • Model size (β = 1.77), language resource level (β = 0.77), reasoning (β = 0.67), and typological distance (β = −0.25) are the largest estimated capability determinants.Source: CONCLUSION · p. 9
  • β = −0.25 for size and β = −0.20 for reasoning describe reductions in the gap between low- and high-resource languages, while similar levers have little to no effect on typologically distant languages.Source: CONCLUSION · p. 10
  • 5.98pp is the reported unseen-language performance prediction error, reduced to 4.19pp with 10 target-language templates; average language difficulty is predicted within 3.2pp, or 2.38pp with 10 templates.These estimates support reasonable performance estimates with little or no downstream data.Source: CONCLUSION · p. 10

ETHICS STATEMENT

The statement describes translator participation and dataset provenance, then sets boundaries on interpreting the benchmark and its resistance to contamination. It also reports the experiments’ energy and reproducibility details.

Human participants and compensation

  • Native speakers with technical expertise translated and validated the dataset; participation was voluntary, translators could withdraw, and substantive contributors were listed as authors when meeting co-authorship standards.Source: ETHICS STATEMENT · p. 10

Data provenance and licensing

  • Multilingual GSM-Symbolic derives from GSM8K and GSM-Symbolic templates, is released under the MIT license, and contains no personally identifiable information.Source: ETHICS STATEMENT · p. 10

Representation and the risk of misreading the benchmark

  • Grade-school arithmetic is a narrow, culturally light slice of language use, so strong benchmark performance is not evidence of general capability or usability in a language.The authors encourage per-language reporting rather than a single multilingual average.Source: ETHICS STATEMENT · p. 10

Contamination and benchmark lifetime

  • Symbolic templates permit millions of unseen surface variants, mitigating but not eliminating contamination risk because template-level memorization remains possible.The released generation procedure allows fresh, previously unpublished variants to be drawn at evaluation time.Source: ETHICS STATEMENT · p. 10

Environmental Impact

  • Experiments used 180 GPU hours on NVIDIA Blackwell B200 GPUs; the authors estimate 84% renewable energy and release per-model, per-sample logs for reproducibility and extension.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

Translators used LLM assistance followed by manual review, and none of those models were evaluated in the study. The authors also report using LLM assistance for code and manuscript work while verifying the reported results and claims.

AI USE STATEMENT

  • Translators used LLM assistance followed by manual review and correction, and none of the assistance models were among those evaluated.Source: AI USE STATEMENT · p. 10
  • Each template records its translation source language, assistance model, and annotator-reported proficiency, making item provenance inspectable.Source: AI USE STATEMENT · p. 10
  • The authors used LLM assistance for code and manuscript editing and reviewing, while producing and verifying the reported results, analyses, and claims.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The robustness checks find that the distance effect is not explained by script, while translation quality and unit choices are separately tested. The paper also provides versioned data and code to support reproducibility and extension.

REPRODUCIBILITY STATEMENT

  • Versioned code, dataset, and test suite support reproducibility, traceability, and future dataset extensions; the dataset is also integrated with EuroEval and Inspect AI.Exact model-by-sample logs are provided for further analysis.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 ALTERNATIVE GLMM SPECIFICATIONS

  • Adding script barely changes the typological-distance estimate (-0.29 to -0.28), while resource-by-distance is nonsignificant (β = 0.02, p = .61).The script effect is also nonsignificant (β = −0.03, p = .90).Source: §A.1 · p. 16
  • The template-by-language effect has sd = 0.92, but adding it leaves significant coefficients' signs and significance unchanged and language-level estimates unaffected.The model explains 91% of between-language and 16% of model-by-language variance with this term, versus 89% and 20% without it.Source: §A.1 · p. 16

A.2 MACHINE VS HUMAN-TRANSLATION

  • Machine-translated and human-verified Icelandic scores were not equivalent within the 1pp margin (p > 0.999), motivating analysis with human-validated languages.Initial machine translations are shared for further validation, but the authors caution that they should be used carefully.Source: §A.2 · p. 16

A.3 ENGLISH IMPERIAL VS ENGLISH METRIC

  • English scores using imperial and metric units were equivalent within the 1pp margin (p < 0.01), supporting localization of units rather than attributing language gaps to unit systems.The authors also report no evidence that metric units ease model computation.Source: §A.3 · p. 17

A.4 SAMPLE SIZE

The sample-size analysis finds that model rankings stabilize with far fewer than 100 templates, while individual accuracy estimates remain less precise. Increasing instances per template yields diminishing returns because most variance is between templates.

A.4 SAMPLE SIZE

  • Model rankings were stable well below the full set: 10 symbolic templates reproduced the full ordering at ρ = 0.96, and 50 templates did so at ρ = 0.99.Source: §A.4 · p. 17
  • A single model-language accuracy still has a 2.29-point standard error at 100 templates; halving that error would require roughly 400 templates.The error falls as 1/√k.Source: §A.4 · p. 17
  • At every budget, symbolic templates are 1.46 times more precise in standard error; about 213 original templates would match the precision of 100 symbolic templates.Source: §A.4 · p. 18
  • At the full template set, increasing instances from 5 to 10 removes 6% of standard error, while increasing from 10 to 20 removes only 3%.At 20 instances, the template-level variance is already 93% of the total, so the binding constraint is template count for this generator.Source: §A.4 · p. 18

D SYNTHETIC VS ORIGINAL VARIANTS

The symbolic-versus-original analysis estimates an average symbolic penalty and tests whether it varies by language. Separate reasoning analyses report smaller transfer gaps with reasoning enabled and under compute constraints.

D.1 MODELING THE SYMBOLIC PENALTY

  • The symbolic penalty is estimated jointly across both splits, avoiding a paired test that would treat models within a language as independent observations of language-level fragility.The model includes an average penalty, language-feature predictors of that penalty, and residual language-specific variation.Source: §D.1 · p. 20
  • The symbolic split carries an average penalty of −0.297 log-odds (p < 10^-15), with language-specific penalty variation (τ = 0.059 log-odds, p < 0.001).Resource and typological distance do not predict the penalty (p = 0.64 and p = 0.85).Source: Figure 10 · p. 20, §D.1 · p. 21

E.1 CAPABILITY TRANSFER GAP

  • Across Qwen model sizes, enabling reasoning reduces the transfer gap.Figure 12 compares the gap with reasoning enabled and disabled across model sizes.Source: §E.1 · p. 21, Figure 12 · p. 22

E.2 UNDER COMPUTE CONSTRAINT

  • Under inference-time compute constraints, larger models can often be preferable for reducing the language gap, while reasoning gives the largest possible absolute reduction.The comparison is limited to models that allow reasoning to be toggled.Source: §E.2 · p. 22, Figure 13 · p. 22

F LEAVE-ONE-LANGUAGE-OUT PREDICTION

The paper tests leave-one-language-out prediction, estimates the model-size cost of low-resource languages, and analyzes where solution errors arise. The supplied passages describe these methods and selected findings, but not the prediction-error results.

F LEAVE-ONE-LANGUAGE-OUT PREDICTION

  • Leave-one-language-out prediction refits the model on other languages, setting language and model-by-language random effects to zero for the unseen language.The baseline predicts using mean accuracy over the training set.Source: F LEAVE-ONE-LANGUAGE-OUT PREDICTION · p. 22

G RESOURCE COST IN MODEL SIZE

H FEATURE DESIGN SPACE

I SOLUTION GAP ANALYSIS

  • The solution-gap analysis distinguishes prompt information-extraction failures from failures applying reasoning after extracting numbers.The former may indicate language understanding limits, while the latter reflects reasoning instability across languages.Source: I SOLUTION GAP ANALYSIS · pp. 24–25

I.1 OUTCOME EXTRACTION

  • The analysis extracts annotated operands and intermediate computations from responses, then assesses final answers separately.Gold operand counts average 5.31–5.45 per item and intermediate counts 3.09–3.17 across languages; unannotated items are excluded from the intermediate model.Source: §I.1 · p. 25

I.2 GLMM ESTIMATES

  • Each solution stage is fitted with Equation 1's specification, excluding reasoning and its related random intercepts while controlling for model variants.Table 6 reports the fixed effects and variance components by stage.Source: §I.2 · p. 25, Table 6 · p. 25

I.3 IS THE SIZE EFFECT A CEILING ARTIFACT?

The resource-by-size interaction persists after saturated cells are removed, while the distance-by-size interaction remains null. The accompanying appendices document model and language materials, comparisons, and benchmark results.

I.3 IS THE SIZE EFFECT A CEILING ARTIFACT

  • −0.188 (p = 1.4 × 10−5) resource × size persists with saturation eliminated, while distance × size is null in every specification.The refits progressively attenuate the resource × size estimate from −0.265 to −0.188; only three paired models remain for testing reasoning interactions, so those estimates are not interpreted.Source: Table 7 · p. 26, §I.3 · p. 26

J MODEL IDENTIFIERS AND REVISIONS

  • Table 8 provides the Hugging Face repository identifiers and revision commit hashes used in the evaluation.The supplied passages identify the table’s contents but do not report its entries.Source: Table 8 · p. 27

K.1 INCLUDED LANGUAGES

  • The language overview covers 102 languages, while only verified subsets are used in the paper’s analysis.Languages marked with parenthesized check marks are still undergoing validation and are excluded from the analysis.Source: Table 9 · p. 28, §K.1 · p. 28

K.2 LANGUAGE ANNOTATION OVERVIEW

  • The annotation overview tracks human verification and review progress, alongside computational validation and error-analysis status.The table notes that computational validation is enforced by CI and marks error analysis only when complete for all active templates.Source: §K.2 · p. 28, Table 10 · p. 29

L MACHINE VS HUMAN-TRANSLATION FULL COMPARISON

M ENGLISH STANDARD VS HUMAN-TRANSLATION FULL COMPARISON

N MODEL RESULTS

  • The model-results section provides both a simple overview and detailed per-language results; Table 11 includes every run, including two marked with an asterisk.The table caption says those two models are excluded from the model in Equation 1.Source: N MODEL RESULTS · p. 32, §N.1 · p. 32, Table 11 · p. 32

N.1 OVERVIEW

  • Table 11 reports exact-answer accuracy averaged over 15 languages, the English-minus-other-languages gap, and each model’s lowest-scoring language.The caption gives approximate sampling error of ±2.3 points for symbolic scores, ±3.4 for original scores, and ±1.4 for the 15-language averages.Source: Table 11 · p. 32

N.2 MODEL RESULTS BY LANGUAGE

The per-language results report original and synthetic exact-answer accuracy across the evaluated models and languages, with confidence intervals; response logs are also distributed for further analysis.

N.2 MODEL RESULTS BY LANGUAGE

  • Response logs are distributed with the dataset to support further analysis of the per-language results.Source: §N.2 · p. 33
  • Table 12 reports original and synthetic exact-answer accuracy by model and language for all 48 models, with 95% confidence intervals.The intervals are computed using Wilson score intervals over scored examples within each split.Source: Table 12 · p. 33

Related papers

Loading 2610.03367v1…