Source-linked AI summary
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Hanna Yukhymenko, Anton Alexandrov, Martin Vechev
TL;DR
Multilingual LLM evaluation is limited by unreliable translated benchmarks, whose flaws include contextual mismatches and linguistic answer leakage. The paper introduces a configurable automated framework combining candidate generation, self-improvement, and ranking methods, and reports improved translation and evaluation quality over existing resources. Its scope is eight Eastern and Southern European languages, with broader low-resource generalization left for future study.
Problem
Existing multilingual benchmarks receive limited translation-quality scrutiny and can contain contextual mismatches or linguistic answer leakage that compromise evaluation integrity.
Method
The paper presents a configurable automated framework using one-prompt translation, Best-of-N, USI, and multi-round T-RANK ranking across datasets, benchmarks, and model types.
Results
The integrated methods improve WMT24++ and FLORES performance, while COMET-based and LLM-as-a-judge evaluations show improvements over existing translations.
Takeaways & Limitations
The framework provides a practical approach for producing scalable multilingual evaluation resources that balance translation accuracy, computational efficiency, and scalability.
Takeaways & Limitations
The framework applies uniform methods without automatically estimating input difficulty, and its generalizability beyond Eastern and Southern European languages remains unvalidated.
Abstract
from arXiv · showhide
The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.
1 Introduction
The paper frames multilingual benchmark translation quality as an underexplored reliability challenge and introduces an automated, configurable framework using test-time scaling methods. Evaluations across mid-resource languages and translated benchmarks report higher-quality translations and more accurate multilingual evaluation.
- Motivation: Existing multilingual benchmarks remain unreliable because many were translated with older models or classical machine-translation tools, with limited attention to translation quality.The paper identifies benchmark quality as a significant research gap despite benchmarks’ role in assessing model quality.
- Motivation: Separately translating questions and answers can create contextual and grammatical mismatches, especially in languages with complex grammatical features.The study focuses on Eastern and Southern European mid-resource languages and translates MMLU, Hellaswag, ARC, and Winogrande into eight languages.
- Framework: The framework supports four configurable approaches: one-prompt translation with self-correction, Best-of-N, USI, and T-RANK.These methods balance translation quality, cost, and time across model types, including open-weight models.
- Framework: T-RANK samples multiple candidates and uses multi-round competitive ranking to improve error detection and translation quality.The framework also includes USI, which unifies sampled translation candidates into refined outputs.
- Evaluation: Evaluations across several mid-resource languages show that the integrated methods produce higher-quality translations, more accurate evaluation results, and improved text quality.The methods also improve performance on WMT24++ and FLORES machine-translation benchmarks.
- Impact: The released translated benchmarks and framework support more efficient multilingual model development by improving data and benchmark acquisition.The released resources cover Ukrainian, Romanian, Slovak, Lithuanian, Bulgarian, Turkish, Greek, and Estonian.
2 Background and Related Work
Recent work uses adaptive prompting, self-refinement, and test-time compute scaling to improve LLM-based translation, but multilingual benchmark translations remain vulnerable to contextual, grammatical, and quality-control problems.
- Translation Methods: AFSP retrieves similar demonstrations and reranks candidate translations to address prompt sensitivity and contextual appropriateness.The framework dynamically selects examples from parallel corpora using LLM-generated embeddings.
- Translation Methods: TEaR iteratively translates, estimates quality, and refines outputs, with experiments showing that general-purpose LLMs can perform translation and evaluation.Its self-refinement process targets errors identified during quality estimation.
- Translation Methods: Test-time scaling methods generate or combine multiple translation candidates to improve quality beyond single-output translation.Best-of-N selects among candidates, while Fusion-of-N synthesizes information from multiple outputs; related studies report multilingual gains.
- Benchmark Limitations: Existing multilingual resources face quality limitations from automated translation, incomplete human verification, language-specific grammatical phenomena, and context-agnostic question–answer translation.Examples include MuBench’s predominantly automated process, Global-MMLU’s separate translation of questions and answers, and Okapi’s limited handling of grammatical features.
- Benchmark Limitations: Gender-specific answer-option morphology can leak answers or mislead models in sentence-completion benchmarks, threatening evaluation integrity.The issue is reported for Winogrande, MMLU, and Hellaswag in languages with gender-specific adjectives.
- Benchmark Limitations: Translating questions and answer options within the same prompt context preserves semantic relationships and prevents contextual mistakes during evaluation.This addresses inconsistencies caused by translating benchmark components separately.
3 An Efficient Automated Translation Framework
The framework automates translation for datasets and benchmarks while preserving benchmark question–answer structure and allowing configurable methods for different language-resource conditions. It implements lightweight translation, candidate selection or combination, and T-RANK’s comparative ranking with multi-round positional control.
- Framework Design: Users can configure translation approaches for varying language-resource availability and language-specific grammatical phenomena.The framework targets cost- and time-effectiveness while supporting language-specific prompts, examples, and verification stages.
- Framework Design: The framework supports dataset and benchmark modes with separate prompts and data handling because benchmark question–answer relationships require preservation.Dataset translation is more straightforward, whereas benchmark translation must maintain connections between questions and answers.
- Translation Methods: USI samples multiple translations and asks an evaluator LLM to combine them into a best version according to predefined criteria.The method adapts self-improvement and Fusion-of-N ideas to machine translation.
- Translation Methods: SC performs zero-shot translation with an optional independent self-check, providing a lower-cost method suited to large texts and high-resource languages.The self-check evaluates and corrects the translation against the original content.
- Translation Methods: Best-of-N samples diverse candidates at temperature 0.7, scores them from 1–10, and selects the highest-scoring translation.The implementation remains training-free and language-agnostic, but the passage reports lower quality than T-RANK and USI.
- Translation Methods: T-RANK samples diverse translations, comparatively ranks them against quality criteria, selects the top candidate, and can refine it.The criteria include general quality, domain consistency, and preservation of the original question idea.
- Translation Methods: Multi-round ranking moves each candidate through every possible position once to reduce positional bias in LLM evaluation.For N candidates, the procedure uses N rounds before aggregating the ranking information.
- Experimental Scope: The framework translates widely used benchmarks into Eastern and Southern European mid-resource languages, including Ukrainian, Romanian, Slovak, Lithuanian, Greek, Bulgarian, Turkish, and Estonian.The selected benchmarks include MMLU, Hellaswag, ARC, and Winogrande.
4 Evaluation Results
The framework is evaluated on standard machine-translation benchmarks and multilingual benchmark translations using reference-based metrics, quality estimation, and LLM-as-a-judge comparisons. Results generally favor USI and T-RANK, while evaluation reliability and method choice depend on text complexity and assessment setup.
- Machine Translation Benchmarks: FLORES and WMT24++ provide standard multilingual evaluation settings with broad language coverage and human translations.FLORES contains professionally translated multilingual sentences, while WMT24++ spans 55 languages and dialects across four domains.
- Machine Translation Benchmarks: COMET evaluates English-to-Ukrainian translations by comparing source, hypothesis, and reference texts, with system-level scores reported for WMT24++ and FLORES.The evaluation uses Unbabel/XCOMET-XL for reference-based quality estimation.
- Machine Translation Benchmarks: USI and T-RANK show clear advantages over other methods, but their relative ranking remains uncertain and may depend on translation cost and effort.The paper motivates additional reference-free quality-estimation and other comparative tests because COMET is not entirely reliable.
- Evaluation Reliability: COMET scores can be affected by domain bias, technical setup, and discrepancies with human judgment, so automated evaluation should be complemented by human or LLM-based assessment.These limitations motivate using multiple evaluation paradigms rather than relying on COMET alone.
- Multilingual Benchmark Translation Quality: Our translations win significantly more LLM-as-a-judge comparisons than Global-MMLU across all evaluated languages.Gemini-2.5-Flash judges translations produced with T-RANK and USI, addressing limitations of quality-estimation models on longer benchmark texts.
5 Conclusion
The paper presents an automated framework for rapid, high-quality translation of datasets and benchmarks with minimal human intervention. Its test-time compute methods, especially T-RANK and USI, improve translation evaluation results and support practical multilingual benchmark creation.
- Conclusion: The framework enables rapid, high-quality dataset and benchmark translation while minimizing human intervention.It is designed as an automated approach for producing multilingual evaluation resources.
- Conclusion: T-RANK and USI achieve the strongest performance through test-time compute strategies on WMT24++ and FLORES.The framework's translation quality is additionally supported by quality-estimation and LLM-as-a-judge evaluations.
- Conclusion: Mid-sized models receive higher evaluation scores on the improved translations than on existing benchmark translations.The reported comparisons include Gemma 3, Qwen 3, and Llama 3.1.
Limitations
The study identifies limitations involving candidate ranking, uniform method selection, model availability, and language coverage. These constraints leave adaptive selection, broader open-weight evaluation, and testing beyond European languages for future work.
- Limitations: LLM-based scoring is used for Best-of-N selection instead of dedicated translation-quality models, which may reduce candidate-ranking reliability.The limitation concerns the reliability of the selection process rather than the translation framework as a whole.
- Limitations: Uniform methods are applied across entries without automatically estimating translation difficulty, although shorter non-question-answering texts may not benefit from the most expensive methods.Users must select methods according to their objectives and resource constraints.
- Limitations: The evaluation relies primarily on closed-source translation models and includes limited testing of open-weight alternatives.The authors hypothesize benefits for open-weight models but state that comprehensive validation is still required.
- Limitations: The framework is evaluated mainly on Eastern and Southern European languages, leaving its generalizability to broader low-resource languages unresolved.Further research is needed across languages with diverse linguistic characteristics.
Ethics Statement
The paper frames improved multilingual benchmark quality as a means to support more equitable evaluation across languages. It also acknowledges reproducibility and representation concerns and states that the code and translated benchmarks will be openly released.
- Ethical Considerations: The work aims to improve multilingual benchmark quality for more equitable language-model evaluation across diverse languages.The stated motivation is to address uneven multilingual evaluation coverage.
- Ethical Considerations: Reliance on closed-source models creates reproducibility concerns, motivating planned evaluation of open-weight alternatives.The authors connect broader model accessibility with future work on reproducible evaluation.
- Ethical Considerations: The focus on Eastern European languages leaves many lower-resource languages underrepresented.The paper encourages extending multilingual evaluation to those languages.
- Open Release: The code and translated benchmarks will be released under permissible open licenses to enable community validation and reproducibility.Generative AI was used for language assistance and boilerplate code, while the core research contributions and findings are attributed to the authors.
A.1 Translation Examples
The examples illustrate how T-RANK and related translation procedures address contextual, grammatical, and ranking issues in multilingual benchmark translation. They also expose a residual positional bias in T-RANK.
- T-RANK correction: T-RANK identifies and corrects translation errors by having a judge model evaluate candidates competitively.The competitive ranking setup improves the judge model’s ability to notice translation flaws.
- Context preservation: Translating questions and answers separately can create grammatical answer leakage, whereas preserving context masks the answer-revealing morphology.The issue is especially relevant when answer options use gender-specific grammatical forms.
- Existing translation errors: Existing MMLU and Hellaswag translations contain recurring errors, including unnatural phrasing, grammatical mistakes, and broken cohesion.These problems arise when shared narrative or question context is not maintained across translated components.
- T-RANK limitation: T-RANK retains a positional bias toward the candidate shown at input position 2 despite rotating candidates across rounds.Position 2 receives the lowest average rank and rank 1 in all five most frequent rank combinations.
A.2 Benchmark Translation Statistics
This section reports the statistics, data splits, sample counts, models, and methods used for the selected benchmark translations.
- Translation statistics: The authors provide statistics for the selected benchmark translations.
- Dataset composition: The section documents the benchmark splits and total number of samples used.
- Models and methods: It also records the models and methods used for final translation.
A.3 Detailed Evaluation Results
The evaluation compares the authors’ translations with existing resources across eight languages and multiple translation benchmarks. Results show broad improvements, while some cases still favor professional translation and method choice depends on the model and use case.
- Benchmark evaluations: The authors report consistent improvements of their translations over existing benchmarks across Ukrainian, Romanian, Slovak, Lithuanian, Bulgarian, Turkish, Greek, and Estonian.The comparisons include Okapi, MuBench, Global-MMLU, and, for Bulgarian, a professionally translated benchmark.
- Benchmark evaluations: Professionally translated Bulgarian Winogrande performs better than the authors’ automated translation, indicating that human intervention remains necessary for some languages and benchmarks.
- Machine translation evaluation: The translation benchmarks are evaluated with reference-based and reference-free COMET setups on WMT and FLORES using GPT-4o-mini and Gemini-2.0-Flash.
- Machine translation evaluation: USI gives the best results on both datasets and evaluation types, while sampling candidates from multiple English- and target-language prompts further improves quality over single-prompt generation.
- Method comparison: For Gemini-2.0-Flash, T-RANK’s multi-prompt method slightly outperforms USI on most language pairs.The authors associate method suitability with model and use-case differences, recommending USI for shorter texts and observing improved T-RANK quality for benchmark translation.