Source-linked AI summary
Skill Issue: Are Skills Language-Invariant in LLMs?
Bobby Cheng, Adam Gaber, Zhengyuan Liu, Catherine Arnett, Omer Goldman, Cheston Tan, Leshem Choshen
TL;DR
The paper investigates whether the same multilingual model realizes different skills through different language interfaces, beyond unequal knowledge access. It uses controlled self-play in a multilingual TextArena extension, finding substantial within-model variation in playing strength and behavior across languages. Stronger-language reasoning recovers performance in some games, but the effect is limited in others and model scale and closed training data constrain interpretation.
Problem
Existing cross-lingual inconsistency research mainly examines knowledge accessibility, leaving whether models express different interactive skills across languages less understood.
Method
The study uses multilingual self-play, keeping models, opponents, game states, rules, and actions fixed while assigning different language interfaces, across six games and eight languages.
Results
Across three models, eight languages, and six games, playing strength, strategic behavior, spatial features, and access to known strategies differ within the same model across languages.
Takeaways & Limitations
Multilingual evaluation should test behavioral consistency across language interfaces, not only understanding of equivalent inputs.
Takeaways & Limitations
The evaluation covers only 3B–4B parameter models, so cross-lingual skill inconsistency may differ for larger models.
Abstract
from arXiv · showhide
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-lingual skill inconsistency orthogonally from knowledge and general benchmark performance. We do this via multilingual self-play: two instances of the same model compete in a text-based game, each interacting through a different language interface. Since the model, opponent, rules, state space, and available actions remain fixed, this setting isolates the effect of language on the model's realized behavior. We build a multilingual extension to TextArena and evaluate three open-weight models across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction. We find that the same model can exhibit markedly different playing strength across languages, with systematic variation in win--loss margins, invalid actions, and strategic tendencies. Detailed analyses reveal language-specific failures in spatial reasoning, card-conditioned decisions, and optimal move selection. In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process. These results show that skill discrepancies are a measurable major roadblock in the development of truly multilingual models. Better understanding these discrepancies can help us design models that perform more equitably across languages.
1 Introduction
The paper asks whether multilingual models differ in their realized skills across language interfaces, beyond differences in knowledge access. Controlled multilingual self-play reveals substantial behavioral variation across languages, including spatial, strategic, and strategy-retrieval differences.
- Motivation: Cross-lingual skill inconsistency asks whether the same model exhibits different skills depending on its interaction language, beyond retrieving different knowledge.Prior work mainly examined unequal knowledge accessibility and limited cross-lingual transfer.
- Approach: Multilingual self-play holds the model, opponent, rules, board states, cards, numerical information, and action spaces fixed while varying language interfaces.Equal skills would produce equal playing strength and randomly distributed wins and losses, as in same-language self-play.
- Scope: The study extends TextArena to 65 games in 193 languages and evaluates manually verified translations of six games in eight languages with three open-weight models.The evaluated models are Gemma 4, Ministral 3, and Qwen 3, all 4B-sized.
- Findings: Models are behaviorally inconsistent across languages, with differences in playing strength, spatial reasoning failures, strategic risk profiles, error rates, and access to known strategies.The reported differences include language-specific variation in spatial failure axes and reliable execution of known strategies.
- Findings: Reasoning in a stronger language recovers substantial performance in some games, suggesting language sensitivity may arise during reasoning, state interpretation, or both.The paper also reports that multilingual benchmarks and uneven language availability in training data partly explain the differences.
2 Multilingual TextArena
Multilingual TextArena extends an open-source competitive-game benchmark so observations can be rendered in different languages without changing game mechanics. The paper combines tiered translation and verification workflows, with experiments focused on eight manually verified languages.
- Benchmark extension: Multilingual TextArena renders player observations in different languages while preserving rules, legal actions, rewards, and transition dynamics.The extension builds on TextArena’s open-source collection of competitive single-, two-, and multi-player text-based games.
- Tier-A languages: The Tier-A workflow selects English, Arabic, German, Spanish, French, Hebrew, Malay, and Chinese for typological, script, cultural, and speaker diversity.Six games were translated into these languages and manually verified by native speakers.
- Tiered translation: Tier B covers 42 additional high- and mid-resource languages using automatic translation, back-translation, and model-based fidelity judgments.The pipeline uses Llama-3.1-405B and Qwen2.5-72B for translation and verification.
- Tiered translation: For 142 remaining languages, NLLB-200 provides translation and back-translation, while two models independently assess fidelity for tier assignment.Agreement that back-translation is faithful at least 85% of the time determines Tier C assignment; otherwise the language is Tier E.
3 Experiment Setup
The experiments compare three similarly sized open-weight models in six two-player games spanning reasoning, planning, allocation, bluffing, cooperation, and adaptation. Each language interface is evaluated through controlled multi-turn self-play with balanced roles and extensive sampling.
- Models: The study evaluates Gemma-4-E4B-it, Qwen3-4B, and Ministral3-3B-Instruct-2512 as comparably sized open-weight models.General multilingual capability is separately assessed with Global-MMLU and Belebele.
- Games: The six games cover perfect-information play, resource allocation, imperfect information, and repeated social interaction.They are TicTacToe, Nim, SimpleTak, Colonel Blotto, Kuhn Poker, and Iterated Prisoner’s Dilemma.
- Interaction protocol: Each multi-turn environment exposes a player-visible observation, receives an action, transitions to a new state, and ends in a win, loss, or draw.Invalid actions can be corrected once; otherwise they cause an immediate loss.
- Language assignment: Each player receives instructions and observations in an assigned language from eight languages, while relevant opponent messages remain in their original language.Language-independent board symbols, coordinates, cards, numbers, and fixed action syntax preserve shared mechanics and formats.
- Evaluation: The protocol evaluates every language pair in both player-role assignments using 400 self-play games per direction.This produces 28,800 games per model-game pair and balances language effects against structural role advantages.
4 Metrics
The paper measures language-conditioned strength using role-pooled pairwise win–loss margins, then averages these margins across languages and games.
- Role-pooled margins compare two languages across both player-role assignments, with wins and losses counted from one language’s perspective.The margin ranges from −1 to 1; positive values favor language A, negative values favor language B, and draws contribute zero.
- Mean language margin averages a language’s pairwise margin against every other language within a model and game.Higher values indicate stronger average self-play performance through that language.
- Model-level mean language margin macro-averages each language’s mean margin across games to summarize overall language strength.
5 Results
Across multilingual self-play, models express different strengths and failure modes through different language interfaces, even when games retain the same underlying information and actions. Stronger reasoning languages can recover performance in some settings, while benchmark competence, data availability, and script explain only part of the variation.
- 5.1 Capabilities Differ Across Languages: The same model expresses substantially different capabilities across language interfaces despite unchanged boards, cards, numerical information, legal actions, and strategies.English is strongest on average across all three models, Hebrew is consistently among the weakest, Gemma is comparatively stable, and Qwen shows the sharpest hierarchy.
- 5.1 Capabilities Differ Across Languages: Colonel Blotto has the largest language gap across all three models, whereas Kuhn Poker is consistently among the least sensitive.Sensitivity in the remaining games depends more on the model; Iterated Prisoner’s Dilemma is relatively stable for Gemma and Qwen but more sensitive for Ministral.
- 5.2 Language-Conditioned Skills Difference: Spatial-game failures vary by language, with Arabic and Hebrew skewing toward column and diagonal defeats rather than the more balanced English loss distribution.Game trajectories frequently reveal cell-mislabeling errors in non-Latin-script interfaces.
- 5.2 Language-Conditioned Skills Difference: Nim strategy mentions, optimal first-move execution, and win rates drop sharply in Arabic and Hebrew for Qwen and Ministral.Many remaining strategy mentions in these languages come from games where the model spontaneously switched into a Latin script during reasoning, indicating unreliable access to a known strategy.
- 5.3 Stronger Languages Enable Recovery: 89.4% of Gemma’s reachable TicTacToe gap is recovered by switching reasoning from German to English while keeping the German interface fixed.In SimpleTak, the margin improves from µ = −0.21 to µ = +0.05, recovering 60.5%; Kuhn Poker recovery is limited and non-monotonic.
- 5.4 Explaining Outcome Differences: Mean language margins correlate with Global MMLU at r between 0.73 and 0.92 and with Belebele at r between 0.71 and 0.79, but benchmark level does not predict cross-language stability.Gemma has the lowest Global MMLU mean but the smallest spread, while Qwen has the highest mean and largest spread.
- 5.4 Explaining Outcome Differences: Malay outperforms Hebrew for every model despite having less available web text by the proxy, while Chinese reaches near-English margins for Qwen and Ministral despite roughly 20× less text than English.Data quantity and script each explain part of the hierarchy, but remaining variation is model-specific.
6 Related Works
Prior multilingual research emphasizes language understanding, knowledge access, transfer, and static benchmark accuracy, while this work studies language-conditioned skill expression during evolving interactions.
- Multilingual benchmarks increasingly cover understanding, reasoning, knowledge-intensive tasks, code generation, instruction following, and culturally situated knowledge.
- Static benchmarks use fixed inputs and predefined answers, revealing cross-language accuracy differences but not equivalent policies over evolving interactions.The paper uses Belebele and Global-MMLU as references while Multilingual TextArena tests interactive skills under different language interfaces.
- Prior work studies factual compartmentalization and post-hoc multilingual routing, including cases where information is irretrievable or stored separately across languages.
- Cross-lingual skill-transfer research examines whether competence learned through one or few languages generalizes to others, including the effect of fine-tuning source language.
- This work instead studies language-conditioned skill access and expression rather than acquiring a skill through one language.
- Interactive benchmarks evaluate state interpretation, valid action selection, adaptation across turns, planning, decision-making, and social reasoning.
- Unlike prior comparisons using a shared language interface, this work extends TextArena to study language-conditioned variation within a fixed model.
7 Conclusion
The paper uses controlled multilingual self-play to test whether LLM skills remain consistent across language interfaces. Across models, languages, and games, it finds language-dependent differences in playing strength, strategy, spatial behavior, and reasoning-stage performance.
- Controlled self-play across three models, eight languages, and six games revealed differences in playing strength, strategic behavior, and spatial features within the same model.
- Reasoning in a stronger language recovered substantial performance in some games, but the effect was limited in others.This suggests language can affect multiple interaction stages, including state interpretation, reasoning, knowledge retrieval, and action selection.
- Multilingual evaluation should assess both understanding of equivalent inputs and behavioral consistency across languages.
Limitations
The evaluation is constrained by limited transparency about model training data and by its focus on 3B–4B-parameter models. These boundaries limit explanations of model-specific language differences and prevent conclusions about scale-invariant effects.
- Closed-data models: Training data for the evaluated models remains closed, making the causes of per-model language-performance differences difficult to identify.Explaining the results therefore required assumptions and estimates, such as multilingual-data availability.
- Model scale: The evaluation is limited to models in the 3B to 4B parameter range.This scale enabled controlled and cost-efficient evaluation over more than half a million games.
- Model scale: The results should not be interpreted as establishing scale-invariant language effects.
- Experimental prompts: The study uses model-specific chat wrappers while keeping task instructions as consistent as possible across models.The scientifically relevant prompt variants are the default action prompt and the language-conditioned reasoning prompt.
- Experimental prompts: The default action prompt omits an explicit instruction to reason in the provided language.
- Experimental prompts: The language-conditioned reasoning prompt explicitly instructs models to reason in the language provided by the environment interface.
B Experimental Setup
The experiments evaluate multilingual gameplay through inference-only rollouts using TextArena environments, translated game prompts, and controlled self-play. The setup spans several games with fixed rules, structured action formats, and multilingual interfaces.
- Infrastructure: The experiments use inference only, with no model training or hyperparameter search.Models are served with vLLM and distributed rollout collection is orchestrated with Ray.
- Evaluation scale: A primary model–game run contains 28,800 self-play games, totaling 518,400 games across three models and six games.Each primary run uses two NVIDIA H200 GPUs; excluding Iterated Prisoner’s Dilemma, it requires approximately 6 H200 GPU-hours.
- Translation pipeline: The multilingual extension uses a one-shot prompting process to generate multilingual implementations and English templates from monolingual environments.
- Translation pipeline: The resulting English templates are translated with GPT-5.2, back-translated with Google Translate, and reviewed by three native speakers for fidelity.Manual review rarely identified substantive issues.
- Multilingual interface: Each player’s language is specified through a language mapping passed to the environment’s reset method.
- Game environments: Nim, Tic Tac Toe, Colonel Blotto, Kuhn Poker, and Simple Tak use fixed game rules with structured action formats tailored to each environment.Actions include pile-and-quantity moves, indexed cells, troop allocations summing to 20, legal poker actions, and indexed cell placement.
E Multilingual UI Localization
The multilingual TextArena resource uses tiered translation and verification pipelines that preserve structural playability while scaling across languages. Experimental languages expose measured fidelity and confidence rather than silently shipping known-wrong strings.
- Tiered pipelines: Two coordinated pipelines share verifiability requirements but switch translators and verifiers according to language-resource level.Higher- and mid-resource languages use gateway models and layered checks, while low-resource languages use NLLB-200 and reader-free verification.
- Low-resource verification: Reader-free back-translation over-flags low-resource languages, whereas direct judging reaches 100% sensitivity and specificity on a human-known control.For Hausa, Yoruba, and Twi, back-translation reported 108, 162, and 317 divergences despite essentially none being real.
- Quality labeling: Known-wrong strings are reverted to English, while languages are labeled certified-flagged at measured fidelity ≥85% or experimental below that threshold.Each shipped language publishes fidelity and target-language coverage, and runtime warnings identify non-certified locales.
- Scale and cost: Localization compute requires tens of GPU-hours, but trustworthy verification across languages the authors cannot read is the binding constraint.The released resource includes locale files and per-language confidence records, while reproducibility tooling remains on a separate branch.
G.1 Differences in Spatial Reasoning
Language interfaces produce distinct spatial failure patterns rather than merely changing aggregate playing strength. Gemma and Ministral show more column and diagonal defeats in several non-English, low-resource, or non-Latin interfaces, while Qwen is more stable across languages.
- TicTacToe: Non-English interfaces, especially low-resource or non-Latin ones, skew Gemma and Ministral TicTacToe defeats toward columns and diagonals relative to English.Qwen3-4B instead shows a more stable defeat pattern across languages.
- TicTacToe: Gemma’s English TicTacToe defeats are 28.5% rows, 34.8% columns, and 36.7% diagonals, while Arabic reaches 45.3% diagonal defeats.Arabic and Hebrew also show qualitative cell-sequence mislabeling as columns or diagonals.
- Interpretation: Loss-type distributions measure which rows, columns, or diagonals each model fails to track under different language interfaces.The accompanying tables report defeat percentages by language and model for TicTacToe and by language for SimpleTak.
- SimpleTak: Gemma’s SimpleTak wins use straight lines in over 90% of cases, while English losses are 53.2% rows and 46.8% columns.Some non-Latin and low-resource languages skew toward column losses compared with English.
- Strategic comparison: Kuhn Poker tests card-conditioned strategic behavior through betting, checking, calling, and folding decisions tied to private cards.Betting with J represents bluffing, betting with K seeks extra payoff, and folding K after a bet is a severe error.
- Strategic comparison: Gemma’s largest cross-language betting variation concerns Q, ranging from 41.6% in Malay to 63.3% in Hebrew.Its decisions for clearly weak or strong cards are more stable, although near-zero bluffing reflects conservatism rather than necessarily optimal play.
G.3 Differences in pre-existing knowledge
The paper separates knowledge of an optimal Nim strategy from the ability to execute it. Strategy mentions and performance vary across languages, and changing processing language can retrieve knowledge that otherwise remains inaccessible.
- Strategy knowledge: Nim strategy mentions vary substantially across languages, with Qwen3-4B and Ministral3-3B showing rough alignment between mentions and win rate.Languages with more optimal-strategy mentions generally perform better than languages with fewer mentions for these models.
- Knowledge versus execution: Optimal-strategy knowledge does not necessarily yield execution because nim-sum play also requires mathematical skill.The analysis therefore examines first-player opening moves for which the board allows only one optimal action.
- Knowledge versus execution: Qwen3-4B executes the optimal first move 80.8% of the time in English but only 24.6% in French despite similar strategy-mention counts.Comparable gaps appear across languages for both Qwen3-4B and Ministral3-3B.
- Processing language: For Ministral3-3B, 70% of Arabic and 50% of Hebrew optimal-strategy mentions came from logs where the model switched into Latin script.Such language switching occurred in only 3.7% of Arabic logs and 1% of Hebrew logs.
- Processing language: Switching processing language can retrieve crucial knowledge that would otherwise be lost under the same game interface.This finding connects language-conditioned strategy access to the paper’s recovery analysis.