Source-linked AI summary

Disentangling Language Roles in Multilingual LLM Task Execution

Qishi Zhan, Minxuan Hu, Seoyeon Jang, Lei Zhao, Ziheng Chen, Man Liang, Xinyue Xiang, Jiaxin Liu, Guansu Wang, Liang He

arXiv:2605.27649v1cs.CLcs.LG

TL;DR

Existing multilingual benchmarks often do not independently control instruction, content, and response languages, obscuring where task-execution failures arise. MTM-Bench crosses those roles and finds response language is the main source of variation, while mismatch count alone is not monotonic.

  • Problem

    Existing multilingual benchmarks often do not independently control instruction, source-content, and required-response languages, leaving the source of task-execution failures unclear.

    Source: §1 · p. 2
  • Method

    MTM-Bench fully crosses English, Spanish, and Chinese language roles across three task families and scores decomposed outcomes with a targeted human audit.

    Source: §1 · pp. 1–2
  • Results

    Across 20 models and 27 triplets, response language is the main source of variation; response-only versus full-mismatch results suggest mismatch count is not monotonic, with ordering varying across models.

    Source: §8 · p. 8
  • Takeaways & Limitations

    Multilingual evaluation should track each language’s task role because aggregate scores can obscure distinct semantic, language-control, and format failures.

    Source: §8 · p. 8

    The response-role finding is controlled evidence within the EN/ES/ZH setting, not a universal claim about multilingual task execution.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet \((L_{\text{instr}}, L_{\text{content}}, L_{\text{resp}})\). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2{,}430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.

1 Introduction

MTM-Bench treats multilingual task execution as an interaction among instruction, content, and response languages that may differ. Its fully crossed design reveals role-specific effects that aggregate language configurations cannot isolate.

1 Introduction

  • Existing benchmarks often leave unclear whether failures arise from task misunderstanding, content processing, target-language realization, constraint following, or their interaction.They often do not independently control instruction, source-content, and required-response languages in one design.Source: §1 · p. 1
  • MTM-Bench enumerates all 27 English, Spanish, and Chinese language-role triplets and contains 2,430 instances per model across three task families.The families are semantic reversal, final-state extraction, and language purity with update realization.Source: §1 · p. 1
  • Response-language variation is much larger than instruction- or content-language variation, while mismatch count is not a monotonic predictor of difficulty.The response-only versus full-mismatch comparison varies across models.Source: §1 · p. 2
  • The benchmark scores semantic correctness, target-language adherence, constraint satisfaction, contamination, and joint success, with scoring checked by a targeted human audit.Source: §1 · p. 2

2 Related Work

Prior multilingual instruction-following work expands language coverage and evaluates transfer, constraints, and multilingual outputs. MTM-Bench complements it by varying instruction, content, and response language roles independently.

2 Related Work

3 Framework

The framework represents each evaluation as an instruction, source content, target condition, and three language-role assignments. It scores semantic, language, and constraint requirements separately, alongside contamination and joint success.

3 Framework

  • Each instance packages an instruction, source content, reference answer or target condition, and the instruction, content, and required response languages.Given this instance, the model produces an output to evaluate.Source: §3 · p. 3
  • Semantic correctness, target-language adherence, and explicit constraint satisfaction are scored as three binary dimensions.Semantic correctness measures target meaning; language correctness checks the required response language; constraint correctness checks output requirements.Source: §3 · p. 3
  • The contamination ratio is a continuous measure of non-target-language material, while language correctness is a binary admissibility judgment.The ratio distinguishes isolated carryover from substantial contamination.Source: §3 · p. 3
  • Joint success multiplies semantic, language, and constraint correctness, treating all three requirements as necessary rather than compensatory.A correct meaning cannot compensate for the wrong response language or a violated output format.Source: §3 · p. 3

4 Benchmark

MTM-Bench uses controlled, short tasks across English, Spanish, and Chinese to compare language roles while holding task semantics and scoring rules stable. Its three task families target distinct execution challenges.

4 Benchmark

  • The benchmark uses a compact EN/ES/ZH testbed and short, verifiable tasks to keep semantic portability, language-control failures, and format compliance observable.English, Spanish, and Chinese provide high-resource, same-script, and non-Latin-script contrasts while keeping construction and scoring validation controlled.Source: §4 · p. 3

4.1 Benchmark Construction

  • Items are rewritten and localized rather than directly translated, then instantiated across language-role triplets while preserving each task’s semantic target and fixed task-defining values.Manual checks remove or revise ambiguous items; 30 base items per family yield 810 instances per family and 2,430 total per model.Source: §4.1 · p. 4

4.2 Task Families

  • The task families test recovery of intended meaning, extraction of the final valid value, and localized realization of an updated state.Semantic reversal counters misleading surface wording; final-state extraction resolves corrections or reassignment. Language-purity tasks require ordinary lexical and functional material in the response language, while allowing specified names and numeric expressions to remain unchanged; non-localized structural carryover fails the task’s criterion, which does not judge code-switching when users permit it.Source: §4.2 · p. 4

5 Evaluation Protocol

The evaluation protocol separates semantic correctness from language and constraint compliance, with task-specific semantic scoring and a human audit of automatic labels.

5 Evaluation Protocol

  • The protocol tracks SEMANTICCORRECT, LANGCORRECT, CONSTRAINTCORRECT, CONTAMINATIONRATIO, and JOINTSUCCESS as separate evaluation metrics.Language and constraint correctness are evaluated separately from semantic correctness.Source: §5 · pp. 4–5
  • Semantic scoring requires stance recovery for Family A, the final valid value for Family B, and a non-contradictory account of the main update for Family C.Family B accepts format-equivalent or cross-linguistic variants referring to the same final value.Source: §5 · p. 4
  • LANGCORRECT fails on non-target lexical items or structural frames unless they are allowlisted; pure numerals and specified names are language-neutral.For example, “327” passes for a ZH target, but “Room 327” fails because “Room” is not allowlisted.Source: §5 · p. 5
  • CONSTRAINTCORRECT checks five recurring requirements: brief-reason yes/no, output-only, single-sentence, banned-word, and starts-with-prefix formats.Output-only explanations violate the requirement, while minor formatting variation is allowed if the final value is unchanged.Source: §5 · p. 5
  • CONTAMINATIONRATIO counts non-target scored units after applying the allowlist, with token categories specified separately for ZH and ES targets.For ZH, non-allowlisted Latin tokens count; for ES, specified English structural tokens, venue words, month names, and common function words count.Source: §5 · p. 5
  • The human audit covered 1,022 targeted boundary cases and 300 unflagged routine outputs; component-level overrides were 18.2% and 7.3%, respectively.The JOINTSUCCESS-level targeted-set override rate was 15.4%; some component corrections did not change the joint decision.Source: §5 · p. 5

6 Experimental Setup

The experiments evaluate 20 frontier and open-weight models on the full benchmark in a zero-shot setting, with main results organized by JOINTSUCCESS.

6 Experimental Setup

  • The 20 evaluated models include frontier commercial systems, open-weight models, and models with varying degrees of multilingual specialization.The set spans systems from Anthropic, OpenAI, Google DeepMind, DeepSeek, Alibaba Cloud, MiniMax, and Meta.Source: §6 · p. 5
  • 2,430 instances per model span three task families, 30 base items, and all 27 language triplets in a zero-shot evaluation.Models receive one response per instance using a fixed prompt template; temperature is set to 0 when supported.Source: §6 · p. 5
  • Table 1 presents main results for 20 models, with top-three and bottom-three values marked per column and models sorted by JOINTSUCCESS.Source: Table 1 · p. 6

7 Results

Across 20 models, joint success trails component scores in isolation, and response language is the strongest role-level axis. Mismatch-count patterns and task-family profiles show that aggregate success masks distinct execution failures.

7.1 Overall Performance

  • Across 20 models, JOINTSUCCESS ranges 0.537–0.840 (mean 0.745), below mean LANGCORRECT 0.905, SEMANTICCORRECT 0.893, and CONSTRAINTCORRECT 0.880.Qwen3-Max leads JOINTSUCCESS at 0.840, followed by DeepSeek-R1 at 0.822 and Qwen3.5-Plus at 0.820.Source: §7.1 · p. 6

7.3 Role Position Beyond Mismatch Count

  • Response-only JOINTSUCCESS is 0.694 versus 0.705 for full mismatch (full-minus-response-only 0.011; 95% interval [0.001, 0.022]), contrary to monotonic mismatch-count difficulty; ordering varies by model.Same, instruction-only, and content-only scores are 0.829, 0.795, and 0.746, respectively.Source: Table 2 · p. 6, §7.3 · p. 7

7.2 Response Language Is the Dominant Axis

  • 0.113 is the response-axis JOINTSUCCESS range (EN 0.819, ES 0.711, ZH 0.706), versus 0.029 for content and 0.018 for instruction.The ordering remains under clustered bootstrap resampling and controls for model identity and base-item difficulty.Source: §7.2 · pp. 6–7, Figure 2 · p. 7, Table 3 · p. 7
  • The response-axis ranges are 0.091 for SEMANTICCORRECT and 0.105 for LANGCORRECT; ZH-response contamination averages 0.067 versus 0.023 for EN.Content-slot effects are smaller and not directionally consistent across ES and ZH.Source: §7.2 · p. 7

7.4 Task Families Expose Distinct Bottlenecks

  • Extraction alone avoids the response-language decline (ZH JOINTSUCCESS 0.746 vs EN 0.743); update tasks retain English structural tokens on ES responses and have lowest JOINTSUCCESS on ZH.Source: Figure 3 · p. 8, §7.4 · p. 8
  • Despite similar family JOINTSUCCESS (0.722–0.786), reversal has SEMANTICCORRECT 0.925; extraction pairs LANGCORRECT 0.948 with CONSTRAINTCORRECT 0.796; update realization pairs CONSTRAINTCORRECT 0.936 with SEMANTICCORRECT 0.816.Source: §7.4 · p. 8
  • Failure components differ: extraction is constraint-limited; semantic reversal with ZH is language-limited, while language-purity ZH is semantic-limited and ES combines semantic and language-control failures.Source: Figure 4 · p. 8, §7.4 · p. 8

8 Conclusion

Across 20 models and 27 triplets, response language is the main source of variation, while task families expose distinct failure channels. The comparison also argues against mismatch count alone as a monotonic account of difficulty.

8 Conclusion

  • Response language is the main source of variation across 20 models and 27 triplets, while instruction and content languages play smaller, less consistent roles.The response-only versus full-mismatch comparison varies across models, so the authors do not claim full mismatch is directionally easier.Source: §8 · p. 8
  • Mismatch count is not a monotonic predictor of difficulty: once the answer language departs from preceding context, extra mismatches add little explanatory power, with ordering varying by model.Source: §8 · p. 8
  • Similar end-to-end scores can conceal distinct failures, including correct meanings in the wrong language and target-language outputs that violate format requirements.The paper argues that evaluation should track language roles rather than collapse configurations into aggregate scores.Source: §8 · p. 8

Limitations

The findings are bounded by the benchmark’s language and task coverage, small item set, and residual scoring uncertainty. The authors characterize MTM-Bench as a controlled diagnostic probe, not a comprehensive multilingual leaderboard.

Limitations

  • The response-role finding is controlled to EN, ES, and ZH; broader language coverage and task families are needed to test whether it extends to more typologically distant or lower-resource languages.Source: Limitations · p. 9
  • With 30 base items per family, MTM-Bench is better suited to controlled role-specific diagnosis than broad domain-level performance estimates.Language-purity results concern short administrative updates, not update-generation tasks generally.Source: Limitations · p. 9
  • Automated scoring can err on short extractions, mixed scripts, paraphrases, aliases, and localized names; semantic labels for Families A and C rely on a single LLM-assisted judge.The targeted human audit and sanity check reduce but do not eliminate residual boundary-case errors; expanded human annotation and multi-judge analysis would further characterize the risk.Source: Limitations · p. 9
  • The authors position MTM-Bench as a controlled diagnostic probe rather than a comprehensive multilingual leaderboard.Benchmark data, evaluation code, scored outputs, and bootstrap scripts are included in the supplementary material.Source: Limitations · p. 9

Ethical Considerations

These supplementary sections report additional model and triplet analyses, benchmark procedures, and task-specific scoring details. They also describe the ethical scope and controlled-use setting of MTM-Bench.

Ethical Considerations

  • MTM-Bench uses synthetic scenarios and is intended for controlled evaluation, not high-stakes deployment; its audit was conducted internally without recruited external participants or crowdworkers.Released research artifacts include synthetic instances, evaluation scripts, aggregate analyses, and scored model outputs; model weights and provider artifacts are not redistributed.Source: Ethical Considerations · p. 9

B Additional Model-Level Results

B.1 Model-level Heterogeneity in Response-only and Full-mismatch Triplets

  • 12 models perform worse on response-only than full-mismatch triplets, 7 show the reverse, and 1 is tied or nearly tied; the aggregate full-minus-response-only difference is 0.011.The authors interpret this as evidence against mismatch count as a monotonic difficulty predictor, not as a claim that full mismatch is intrinsically easier.Source: §B.1 · p. 13

B Additional Model-Level Results

D.1 LangCorrect Edge Cases

D.2 Task-Specific Semantic Rubrics

  • The semantic rubrics require stance polarity and target preservation for reversal, final valid values for extraction, and an accurate, non-contradictory core update for update realization.Paraphrase is allowed for semantic reversal and update realization, while earlier intermediate values receive no credit in extraction.Source: §D.2 · pp. 14–15

C Additional Triplet-Level Visualizations

E LLM-Assisted Semantic Judging Details

  • Families A and C use LLM-assisted semantic judging, while Family B uses a deterministic final-state checker; language adherence and format constraints are scored separately.The semantic judge is blinded to evaluated-model identity and handles semantic content only.Source: E LLM-Assisted Semantic Judging Details · p. 15

F Human Scoring Audit

The human audit checks component labels and joint success, finding that disagreements cluster around specific boundary cases. It supports decomposed scoring while warranting caution with closely ranked models.

F Human Scoring Audit

  • The audit reviews the benchmark item, model response, and automatic component labels, recording agreements and human overrides for semantic, language, and constraint labels.Overrides are adjudicated using the Appendix D.1 edge-case policy.Source: F Human Scoring Audit · p. 16
  • Table 6 summarizes audit subset sizes and component-level and JOINTSUCCESS override rates.The audit includes 1,022 targeted cases and an unflagged sanity check of 300 outputs.Source: F Human Scoring Audit · p. 16
  • Agreement varies by response language most in semantic and constraint labels, while LANGCORRECT agreement remains stable across all three languages.Figure 8 illustrates representative overrides involving output form, punctuation constraints, paraphrases, and old-to-new transition framing.Source: Table 7 · p. 16, F Human Scoring Audit · p. 16
  • Routine unflagged cases have lower override rates than the targeted audit set; overrides cluster around strict output-only constraints, paraphrastic matches, and target-language boundary cases.Source: F Human Scoring Audit · p. 16
  • JOINTSUCCESS overrides are lower than component-level overrides because some component corrections do not change the final joint decision.For example, a corrected language label cannot change JOINTSUCCESS when the response already violates a format constraint.Source: F Human Scoring Audit · p. 16

G Statistical Uncertainty Analysis

Clustered uncertainty analysis supports response language as the dominant source of JOINTSUCCESS variation, while response-only versus full-mismatch results do not support a monotonic mismatch-count account. Model- and item-controlled regression reinforces the response-language pattern, and representative outputs show that semantic correctness, language adherence, and output constraints can diverge.

G Statistical Uncertainty Analysis

  • Clustered bootstrap resamples base items with replacement and retains all associated triplets and model outputs when recomputing aggregate estimates.This avoids treating triplet variants of the same item as independent observations; reported 95% intervals are percentile intervals over replicates.Source: G Statistical Uncertainty Analysis · pp. 16–17
  • 0.113 is the JOINTSUCCESS response-axis range (95% clustered bootstrap interval [0.083, 0.156]), versus 0.029 [0.012, 0.049] for content and 0.018 [0.006, 0.033] for instruction.The separated intervals support response-language dominance beyond item-level sampling variance.Source: G Statistical Uncertainty Analysis · p. 17, Table 8 · p. 17
  • 0.011 is the full-minus-response-only JOINTSUCCESS difference (95% clustered bootstrap interval [0.001, 0.022]); mismatch count is not a monotonic difficulty predictor.Observed means were 0.705 for full mismatch and 0.694 for response-only; model-level ordering varies.Source: G Statistical Uncertainty Analysis · pp. 17–18

H Regression Analysis with Model and Item Controls

  • The fixed-effect linear probability model estimates JOINTSUCCESS with model and base-item fixed effects, base-item-clustered standard errors, and English as each language slot’s reference.The regression checks whether the response-language effect is driven solely by model differences or base-item difficulty.Source: H Regression Analysis with Model and… · p. 18
  • 0.108 and 0.113 are the JOINTSUCCESS decreases associated with requiring Spanish and Chinese rather than English responses, respectively, after model- and item-level controls.Instruction-language coefficients are close to zero, while content-language effects are smaller and not directionally consistent across Spanish and Chinese.Source: H Regression Analysis with Model and… · p. 18, Table 9 · p. 19

I Benchmark Items and Representative Output Cases

Loading 2605.27649v1…