PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation

Zahraa Al Sahili, Ioannis Patras, Matthew Purver

arXiv:2610.20221v1cs.CLcs.AI

TL;DR

Safety and reasoning capabilities are still evaluated predominantly in English, and translated benchmarks must preserve task-specific meaning for cross-language comparisons. The paper builds aligned multilingual evaluations with reasoning-guided translation, validation, and repair; evaluations reveal contrasting refusal patterns and lower-resource reasoning degradation.

  • Problem

    Because model performance can vary across languages, comparable evaluation instances are needed to distinguish capability differences from differences in benchmark construction.

    Source: §1 · p. 1

  • Method

    The framework uses reasoning-guided configuration selection, task-aware translation constraints, source-aware validation and repair, and translator-independent screening to build aligned multilingual resources.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Outside English, the models show contrasting refusal errors, while MATH-500 accuracy falls by up to 20 and 15 percentage points for Llama-3.1-8B and Qwen2.5-7B, respectively, in lower-resource languages.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Multilingual evaluation can expose weaknesses missed by English-only testing, and harmful-request handling, over-refusal, and reasoning emerge as separate dimensions of model behaviour.

    Source: §5 · p. 6

    Downstream evaluation covers only two 7–8B open-source models under fixed settings, and non-English response truncation may affect comparisons.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in English. Translating existing benchmarks offers a scalable approach to multilingual evaluation, but preserving task-specific meaning across languages presents significant challenges, particularly for harmful requests, mathematical reasoning, and conversational interactions. We introduce a reasoning-guided translation framework for constructing multilingual evaluations designed to preserve the structure and intended meaning of diverse benchmark tasks. Our approach separates translation strategy selection from large-scale generation, combining reasoning-guided configuration selection with task-aware validation, targeted repair, and independent quality screening. Using this framework, we develop a source-aligned evaluation collection spanning seven safety, conversational, and mathematical reasoning resources across 11 languages. The resulting collection achieves 99.94\% coverage of intended item--language pairs in the two subsets requiring repair. Evaluations of open-source LLMs reveal distinct language-dependent vulnerabilities, including inconsistent refusal behaviour and substantial reasoning degradation in lower-resource languages. These findings highlight the importance of task-sensitive benchmark construction and systematic quality assessment for understanding multilingual model behaviour beyond English.

1 Introduction

English-centric evaluation leaves multilingual safety and reasoning capabilities unevenly measured, while translation can alter the task properties benchmarks are meant to test. The paper introduces a task-aware, reasoning-guided framework and aligned resources to support cross-language evaluation.

1 Introduction

  • Translation must preserve harmful intent, benign-versus-unsafe distinctions, mathematical constraints, and conversational structure; otherwise observed gaps may reflect benchmark construction rather than model capability.Translation-model refusals or altered meanings can also bias evaluation if unsuccessful items are silently excluded.Source: §1 · p. 1
  • The framework separates translation-strategy selection from large-scale generation and integrates task-aware validation, targeted repair, and translator-independent screening.It is designed to identify potential failures while retaining traceability to original instances.Source: §1 · pp. 1–2
  • The collection aligns seven English-language resources across 11 target languages, preserving source identifiers and task annotations for item-level comparisons.Its resources span harmful requests, adversarial and benign safety prompts, over-refusal, naturalistic conversations, and mathematical reasoning.Source: §1 · p. 2
  • 99.94% of intended item–language pairs in the two subsets requiring repair met operational validation criteria, while evaluations revealed model-specific refusal patterns and lower-resource reasoning degradation.The findings underscore the value of aligned, systematically validated resources for multilingual evaluation.Source: §1 · p. 2

2 Related Work

Prior work establishes multilingual safety and mathematical benchmarks, as well as automated translation and prompt-selection methods. This paper combines complementary safety, conversational, and reasoning resources while using reasoning-guided configuration selection and separate structural and semantic assessment.

Multilingual safety evaluation

  • Existing safety studies motivate language-aligned evaluation, but the cited benchmarks do not jointly cover safety, over-refusal, and conversational sources.XSafety measures harmful-response behavior across ten languages, while Deng et al. (2024) report multilingual prompting vulnerabilities [Deng et al., 2024].Source: §2 · p. 2

Multilingual mathematical reasoning

  • Prior multilingual math resources include manually translated grade-school problems and translated competition-level tasks; the paper does not claim to be the first to translate these tasks.MGSM [Shi et al., 2023], MCLM [Son et al., 2025], and MMATH [Luo et al., 2025] are cited as related work.Source: §2 · p. 2

Benchmark translation and automated selection

  • Building on LLM-based prompt proposal and translation-quality judging, the framework selects a configuration on a small development set, uses a lighter production model, and distinguishes format checks from semantic validation.The related work includes Automatic Prompt Engineer and ProTeGi [Zhou et al., 2023; Pryzant et al., 2023] and M-MAD [Feng et al., 2025].Source: §2 · pp. 2–3

3 Benchmark Construction

The collection combines seven English-language resources with distinct safety, conversational, and mathematical evaluation objectives. Translation and validation requirements are tailored to each task, while structural compliance is not treated as proof of semantic equivalence.

3 Benchmark Construction

  • The seven resources cover harmful requests, adversarial and benign safety prompts, over-refusal, naturalistic conversations, and mathematical reasoning.They include StrongREJECT, JBB-Behaviors, WildJailbreak, XSTest, WildChat, MATH-500, and AIME 2024.Source: §3 · p. 3
  • The collection covers 11 target languages; WildChat contributes 370 sampled conversations and WildJailbreak 250 prompts, while the other five resources are translated in full.The full resources are StrongREJECT (313), JBB-Behaviors (100 harmful and 100 benign), XSTest (450), MATH-500 (500), and AIME 2024 (30).Source: §3 · p. 3
  • Task-specific instructions preserve harmful intent, benign-versus-unsafe distinctions, mathematical quantities and constraints, and conversational roles, indices, boundaries, and ordering.Source identifiers, metadata, mathematical notation, reference answers, and code are retained for alignment.Source: §3 · p. 3
  • Meeting structural requirements alone does not establish semantic equivalence between source and translated instances.Source: §3 · p. 3

4 Translation Framework

The translation framework selects a configuration on a small development set before large-scale generation, then validates outputs and repairs structural failures. It distinguishes operational validity from semantic fidelity and records unresolved cases.

4 Translation Framework

  • A reasoning model proposes three translation instructions, which are tested with five candidate translators on a 70-item Thai development set across 15 configurations.The pilot assesses API and JSON success, structure, mathematical numbers, and refusal-like outputs; these are indicators of operational reliability, not direct semantic-fidelity measures.Source: §4 · p. 4
  • GPT-5-mini with P2 is selected for production as a compromise between structural validity and monetary cost, although GPT-5.1 with P2 has higher pilot structural validity.Source: §4 · p. 4
  • Production validation checks parseability, nonempty content, and source-specific structure, treating these operational criteria separately from semantic fidelity.Failed WildChat and WildJailbreak records undergo targeted regeneration from their English sources using stricter instructions.Source: §4 · p. 4
  • Unresolved invalid records are excluded and recorded as missing, so failures appear in coverage statistics rather than being silently discarded.WildChat checks message indices and ordering; WildJailbreak checks nonempty prompts and preserved source labels.Source: §4 · p. 4

5 Results and Discussion

After repair, the collection achieved 99.94% operational coverage of intended WildChat and WildJailbreak item–language pairs, while downstream tests found language- and model-specific refusal patterns and lower-resource reasoning degradation. Screening and validation results also show that structural acceptance is distinct from semantic fidelity.

Translation configuration analysis

  • On the 70-item Thai pilot, P2 yielded 70/70 valid GPT-5.1 outputs and 68/70 (97.1%) for GPT-5-mini, while Gemini 2.5 Flash reached full validity under P1.Structural validity varied across translators and instructions, so the most effective instruction depended on the model.Source: §5 · pp. 4–5, Table 2 · p. 5
  • GPT-5-mini’s selected configuration took a median 10.9 s per item versus 3.4 s for GPT-5.1, while lowering projected collection-wide cost from approximately $145 to $97.The choice prioritized affordability while maintaining high structural validity, not latency, peak pilot validity, or semantic-quality optimality.Source: §5 · p. 5

Failure recovery and collection coverage

  • 978 translations required repair; subsequent attempts left four unresolved cases, all in WildChat, where messages were missing despite otherwise valid JSON.The first GPT-4o repair pass recovered 392 records, leaving 586 unresolved; later recoveries were not logged by individual pass.Source: §5 · p. 5
  • 99.94% of intended WildChat and WildJailbreak item–language pairs met operational validation after repair, with four WildChat translations unresolved.Table 3 covers 370 WildChat conversations and 250 WildJailbreak prompts across 11 languages; all 6,816 retained records passed the specified structural checks.Source: Table 3 · p. 5, §5 · p. 5

Limitations

  • Structural checks and regeneration do not guarantee preserved meaning: numerical recall cannot establish mathematical equivalence, and a single-language pilot limits conclusions about configuration suitability across target languages.Automatic validation and fidelity screening also do not establish equivalent task difficulty across languages; further bilingual assessment is needed.Source: §5 · pp. 5–6

Independent fidelity screening

  • 1.2% of audited item–language pairs remained in English despite structural validation, while a separate model judge rated 99.6% of assessed translations as meaning-preserving; these are automatic signals, not verified bilingual fidelity.Pairs retaining English content were flagged and excluded from cross-language comparisons.Source: §5 · pp. 5–6

Downstream multilingual evaluation

  • Outside English, Llama-3.1-8B under-refused harmful requests, while Qwen2.5-7B over-refused benign prompts across all 11 target languages; MATH-500 accuracy fell by up to 20 and 15 percentage points, respectively.The results indicate that refusal patterns differ by model and that multilingual safety performance does not degrade uniformly.Source: §5 · p. 6

Implications for multilingual evaluation

  • Multilingual benchmark studies should report language-specific coverage and unsuccessful translations, compare shared source instances where appropriate, and retain separate measures for distinct task objectives.Harmful-request handling, over-refusal, conversational behaviour, and mathematical reasoning cannot be adequately represented by one aggregate metric.Source: §5 · p. 6

6 Conclusion

The paper presents a reasoning-guided framework and source-aligned multilingual evaluation collection spanning seven resources and 11 languages. Its aligned evaluations reveal language-dependent safety and reasoning vulnerabilities, while automatic screening cannot replace bilingual verification of semantic fidelity.

6 Conclusion

  • 99.94% structural coverage was achieved for intended WildChat and WildJailbreak item–language pairs.Independent quality screening also identified residual translation failures that structural checks alone could not detect.Source: §6 · p. 6
  • The framework separates translation-configuration selection from large-scale generation and uses task-aware validation and targeted repair.The resulting workflow is described as scalable and auditable.Source: §6 · p. 6
  • Aligned evaluations revealed under-refusal, over-refusal, and substantial reasoning degradation across languages.The paper concludes that multilingual capabilities should be evaluated across complementary tasks rather than with English-centric assessments or aggregate scores.Source: §6 · p. 6
  • Automatic screening cannot replace bilingual verification of semantic fidelity.The study calls for task-sensitive validation and transparent quality assessment in multilingual benchmark construction.Source: §6 · p. 6

Limitations

The study’s configuration pilot, fidelity assessment, and downstream evaluation each have scope limits. These constrain how broadly its translation strategy and observed model differences can be generalized.

Limitations

  • The configuration pilot uses only 70 Thai instances per configuration, limiting generalizability across languages, and GPT-5.4 Thinking both proposes and assesses configurations, potentially introducing selection bias.Latency and cost estimates use a separate 14-item-per-model sample.Source: Limitations · p. 6
  • Cost projections use Thai tokenisation and published API prices, exclude configuration selection and repair, and therefore do not establish a globally optimal translation configuration.Latency and cost estimates also rely on a separate 14-item-per-model sample.Source: Limitations · pp. 6–7
  • Detailed repair provenance is available only for WildChat and WildJailbreak, and intermediate repair attempts were not consistently logged.Automatic indicators and an LLM-based judge may miss subtle semantic changes, cultural differences, or shifts in task difficulty; bilingual human evaluation remains necessary, but verified human assessments are unavailable.Source: Limitations · p. 7
  • Downstream evaluation covers two 7–8B open-source models, greedy decoding, an LLM-based safety judge, and a fixed generation budget, limiting generalization to other models and settings.The budget causes greater truncation of non-English responses and may affect cross-language comparisons; original benchmarks may also reflect cultural and sampling biases, with redistribution subject to licensing, access, and privacy constraints.Source: Limitations · p. 7

A Automatic Validation of Translation Fidelity

The authors audit translated items with multiple automatic signals, including checks for untranslated content, numeral retention, translation similarity, and safety-label preservation. These signals help screen for failures but do not establish human-rated fidelity.

A Automatic Validation of Translation Fidelity

  • Four translator-independent signals audit structurally accepted pairs, but these automatic proxies prioritize review rather than replace bilingual human assessment.The signals are reference-free quality estimation, cross-lingual embedding similarity, independent-machine-translation back-translation, and an LLM judge from another model family.Source: A Automatic Validation of Translation… · p. 8
  • 274 item–language pairs (1.2% of audited pairs) remained in English despite structural validation and were excluded from cross-lingual comparisons.Most were short XSTest prompts (179 pairs; 3.6% of its 4,950 pairs) or WildChat user turns (79); the remainder were spread across other sources.Source: A Automatic Validation of Translation… · p. 8
  • Numeral recall was at least 98.6% for every language in both MATH-500 and AIME 2024, but detects altered quantities rather than changes to conditions or wording.Numeral recall counts source numbers retained after native digit systems are normalized.Source: A Automatic Validation of Translation… · p. 9
  • CometKiwi averaged 0.84 across six sources and 11 languages, but scores are used only to rank items within each language because they are not calibrated across languages.WildChat is excluded because its long, code-heavy turns fall outside the model’s sentence-level setting.Source: A Automatic Validation of Translation… · p. 9, Figure 2 · p. 10
  • LaBSE source–translation similarity averaged 0.84–0.92 per source, while back-translation chrF++ ranged from 56.1 for WildChat to 70.6 for JBB-Behaviors.Back-translation scores may underestimate direct fidelity because they include errors from a second system.Source: A Automatic Validation of Translation… · p. 9
  • The independent judge matched English and translated safety labels on 92.6% of items, but its 88.7% agreement with English gold labels prevents isolating translation errors from judge errors.Agreement was 99.2% for StrongREJECT and lowest for JBB-Behaviors (89.7%) and XSTest (90.2%).Source: A Automatic Validation of Translation… · p. 9, Table 5 · p. 9
  • Paired fidelity judgments scored meaning preservation at 4.98 on average and judged 99.6% of translations to preserve intent or mathematical content, but the authors treat these near-ceiling scores as screening signals because LLM judges may be lenient.The judge marked 0.3% of safety and conversational items as softened, rising to 0.6% for WildChat.Source: A Automatic Validation of Translation… · p. 10

B Downstream Evaluation

Evaluations on identical translated items show distinct language-dependent patterns in refusal and mathematical reasoning. Interpretation is qualified by response truncation, and the configuration pilot uses a small Thai development set.

B Downstream Evaluation

  • The evaluation compares two open instruction-tuned models on English and 11 target languages, using the same sampled items and paired tests against English.Prompts use a single user turn without a system prompt; decoding is greedy, with up to 768 new tokens for safety and 3,072 for MATH-500. Both models answer in the prompt language for at least 94.6% of safety items in every language.Source: B Downstream Evaluation · p. 10
  • Safety responses are judged against the English prompt with a different-family model, and mathematical answers are checked symbolically against reference answers.Untranslated pairs are excluded, and language comparisons use paired bootstrap tests over shared items.Source: B Downstream Evaluation · p. 10
  • Llama-3.1-8B under-refuses in several languages, while Qwen2.5-7B over-refuses safe XSTest prompts across all 11 target languages.For Llama, unsafe XSTest refusal falls from 97.5% in English to 69.5% in Turkish; Qwen’s safe-prompt over-refusal rises from 6.4% to 13.0–23.7%.Source: B Downstream Evaluation · pp. 10–11, Table 6 · p. 11, Figure 3 · p. 11
  • MATH-500 accuracy drops significantly for both models in Bengali, Persian, Hindi, and Turkish; the largest reported drops are 44.0% to 24.0% for Llama in Arabic and 74.0% to 58.6% for Qwen in Turkish.Spanish and French remain within four points of English for both models; AIME 2024 shows no significant differences with 30 problems and accuracies at most 14.3%.Source: B Downstream Evaluation · p. 11, Table 6 · p. 11
  • On WildJailbreak, refusal increases in several languages, while harmfulness scores decrease in every language.The judge rates 99.7% of translated prompts as intent-preserving; the authors interpret the pattern as weaker performance following long role-play framings, not safer behavior.Source: B Downstream Evaluation · pp. 11–12
  • The 768-token budget truncates more non-English responses, biasing harmfulness scores downward; this makes reported increases for direct requests conservative, while some decreases may partly reflect truncation.Refusal decisions usually occur in the first sentences and are largely unaffected; the authors recommend reporting adversarial and direct harmful requests separately.Source: B Downstream Evaluation · p. 12

C Prompt Design and Calibration

  • The production translation prompt is language-specific P2, selected from three prompts and five translators using a 1,050-call Thai pilot.P2 emphasizes safety intent and mathematical constraints; because GPT-5.4 Thinking proposes and helps assess configurations, selection is a screened engineering choice rather than an independent semantic-fidelity assessment.Source: C Prompt Design and Calibration · p. 12

D Repair Algorithm and Validation

The section documents a source-specific repair workflow and its observed outcomes for WildChat and WildJailbreak, while noting that comparable repair logs are unavailable for the other five sources. It also describes provenance and the translation and repair prompt contracts.

D Repair Algorithm and Validation

  • Repair reconstructs failed records from the original English input, then validates source-specific output schemas while preserving conversation structure or benchmark metadata.The queue is deduplicated by benchmark, source UID, and target language; WildChat message indices must match the source exactly.Source: D Repair Algorithm and Validation · p. 12
  • 4,066/4,070 WildChat pairs and 2,750/2,750 WildJailbreak pairs were accepted in the final outputs, leaving four WildChat cases unresolved.The first repair pass accepted 392 of 978 queued failures (40.1%); the authors do not attribute final counts to undocumented intervening operations.Source: D Repair Algorithm and Validation · p. 12

E Data Provenance and Release Considerations

  • The record-level repair audit covers only WildChat and WildJailbreak; the other five sources have accepted outputs and automatic screening but no comparable repair log.The authors recommend retaining dataset versions, item identifiers, target languages, validation statuses, repair provenance, and unresolved pairs in a reproducible release.Source: D Repair Algorithm and Validation · p. 13, E Data Provenance and Release… · p. 13

F Full Translation Prompts

  • The selected P2 translation prompt requires valid JSON and preservation of benchmark-specific intent, including safety intent and mathematical quantities or constraints.The templates also instruct translators not to answer or add explanations.Source: F Full Translation Prompts · p. 13

G Full Failure-Repair Prompts

  • Repair prompts retranslate original English content rather than partial translations, with WildChat requiring one indexed output per input message and forbidding omissions.The prompt also treats the content as benchmark data, not a request to answer, refuse, or add commentary.Source: G Full Failure-Repair Prompts · p. 14

H Speed and Cost of Translation Configurations

The deployed GPT-5-mini configuration trades cost and validity against speed. Estimates are based on a small, single-language sample and do not measure controlled throughput.

H Speed and Cost of Translation Configurations

  • $4.20 per 1,000 items and about $97 for the full collection are projected for GPT-5-mini, whose validity was 97.1% versus 84.3% for Gemini 2.5 Flash-Lite.The comparison is under P2; the table covers speed, cost, and structural validity for 14 Thai items per translator.Source: Table 8 · p. 14, H Speed and Cost of Translation… · p. 14
  • 10.9 s is GPT-5-mini’s median latency per item, compared with 2.4–4.3 s for the other full-size models; a WildChat conversation reached 42.7 s.These are single-run per-item observations, not controlled throughput measurements.Source: H Speed and Cost of Translation… · p. 14
  • GPT-5.1 has the highest validity at a moderate cost premium, while Gemini 2.5 Flash offers similar cost and validity at lower latency.The cost and latency estimates use a small single-language sample, reflect prices at measurement time, and exclude repair passes and configuration-selection costs.Source: H Speed and Cost of Translation… · p. 15

I Human Evaluation Protocol

The protocol proposes bilingual assessment of translation fidelity and task-critical meaning using a stratified, source-aligned sample. Annotation results are ongoing and are not included in the submission.

I Human Evaluation Protocol

  • The sample contains 15 source-aligned items per target language across five benchmarks, yielding 165 translation pairs; WildChat and WildJailbreak are excluded.The same English items are used across all 11 target languages.Source: I Human Evaluation Protocol · p. 15
  • Annotators compare each English item with its translation, rating semantic fidelity and task-critical preservation and identifying meaningful errors; two independent reviews are sought where available.For mathematical items, reviewers assess quantities, notation, and constraints without needing to solve the problems.Source: I Human Evaluation Protocol · p. 15
  • Structural acceptance is not verified semantic equivalence, and human-evaluation results are absent from the submission while annotation coverage may vary by language.Unavailable annotations are not counted as successful assessments.Source: I Human Evaluation Protocol · p. 15
Loading 2610.20221v1…