Source-linked AI summary
Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Pinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu, David Tan, Doreen Osmelak, Ona de Gibert, Ariun-Erdene Tumurchuluun, Ashok Urlana, Fedor Sizov, Hale Sirin, Jesujoba Alabi, Karrar Talib Abed, Mateusz Klimaszewski, Nikolay Bogoychev, Niyati Bafna, Patricia Schmidtova, Preksha Manjunath Shanbhag, Sherrie Shen, Vilem Zouhar, Vivek Iyer, Yasser Hamidullah, Yusser Al Ghussin, Zheng Zhao
TL;DR
Existing English-anchored benchmarks are vulnerable to contamination and overlook locale-specific cultural variation. The paper introduces Cultivar, a source-contrastive localisation of FLORES, and evaluates 32 open-weight models to measure robustness and contamination-related discrepancies. MT-specialised models are less robust, some models potentially overfit FLORES, and models tend to translate US-grounded content better than content from native regions.
Problem
English-anchored multilingual benchmarks are vulnerable to contamination and language-oriented evaluation overlooks locale-specific cultural variation.
Method
Cultivar creates locale-grounded, source-contrastive FLORES pairs and compares original and localised performance across 32 open-weight models.
Results
MT-specialised and smaller systems perform worse on localised content, while contrastive gaps reveal potential FLORES overfitting and a US-centric performance bias.
Takeaways & Limitations
Locale-aware evaluation is needed to assess multilingual translation robustness and better reflect real-world use cases.
Takeaways & Limitations
Locale difficulty may reflect both model artifacts and data artifacts because localisations were produced by an LLM and edited by individual annotators.
Abstract
from arXiv · showhide
Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlooks locale and cultural considerations. We therefore advocate for source-contrastive evaluation and instantiate it with Cultivar, a localised subset of FLORES, which enables locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancy allows the probing of data contamination and localisation robustness. We benchmark 32 open-weight models and find that MT-specialised models are less robust, a few models potentially overfit FLORES, and models tend to translate US content better than that of other locales, regardless of language.
1 Introduction
Existing multilingual benchmarks often derive every instance from English, exposing them to contamination and mismatching native source-content use. Cultivar addresses these issues with source-contrastive, locale-oriented evaluation across 27 locales and 32 models.
- English-anchored multilingual test suites facilitate parallel evaluation but risk contamination and create a language-content mismatch for translation into English.
- Language-oriented tests can miss regional variation because one language may span multiple locales and diaspora communities.
- Cultivar uses paired instances that preserve sentence structure while varying locale-specific content, enabling discrepancy-based analyses of contamination and localisation robustness.
- Cultivar is a locale-grounded multilingual test set produced through human-LLM co-creation while preserving comparability and incorporating native source content.
- The study evaluates 32 models with locale-level scores and contrastive comparisons against FLORES, examining localisation robustness and potential overfitting.
- MT-specialised and smaller models tend to perform worse on localised content, while gaps between original and localised tests indicate potential FLORES overfitting and stronger performance on US-grounded content.
2 Cultivar
Cultivar localises selected FLORES instances through LLM paraphrasing and human review, creating comparable source-target pairs grounded in specific locations. The resulting benchmark contains 27 locales and is predominantly newly localised content.
- Cultivar’s three-stage pipeline selects named-entity-rich instances, generates LLM-assisted paraphrases, and applies human post-editing for quality.
- The localisation design preserves original sentence structure while adapting cultural content, keeping sentence pairs directly comparable.
- 658 FLORES development-set sentences contain at least one named entity, and 200 with at least two named entities are selected for localisation.
- Annotators must be proficient in both languages and live or have lived in the target location, supporting linguistic competence and cultural familiarity.
- Annotators accept, edit, or revert each LLM paraphrase after inspecting original and localised pairs in a side-by-side interface.
- The final benchmark contains 27 language-script-location combinations, covering 21 languages or dialects, 8 scripts, and 21 countries or regions, with 200 instances per locale.
- On average, annotators keep 70% of LLM localisations, post-edit roughly 28%, and revert fewer than 2%, leaving over 98% as newly localised content.
3 Experimental Setup
The experiments evaluate 32 open-weight models with standard translation metrics and discrepancy measures comparing original FLORES with localised Cultivar. They also introduce lexical analyses to distinguish retained original content from missing locale-specific content.
- The study evaluates 32 open-weight models selected across translation specialisation, size, language focus, and geographic orientation.
- BLEU and chrF measure translation quality using subword- and character-level evaluation implemented in sacrebleu.
- Performance discrepancy compares localised and original test results to quantify robustness rather than identify the highest-quality translation models.
- More negative ∆ indicates greater degradation on localised source texts, while ∆→0 indicates consistent behaviour regardless of absolute translation quality.
- Mean pairwise ∆metric aggregates localisation effects across locales and can also be averaged across models without pairing.
- Locale-specific word recall compares translations with original and localised references to focus on content changed by localisation.
- False positives indicate retained original terms, whereas false negatives indicate absent newly introduced terms; false negatives are an upper bound rather than an exact failure rate.
- Spearman’s rs and Kendall’s τb measure alignment between 32-model rankings on FLORES and Cultivar for BLEU and chrF.
4 Empirical Results and Analysis
Across 32 models, Cultivar reveals how localisation changes translation performance relative to FLORES, with robustness varying by model type, size, locale, and evidence of possible overfitting.
- Model rankings: Cultivar and FLORES induce highly consistent model rankings, with rs ≥0.93 and τb ≥0.80 across all translation directions.Average correlations reach rs ≥0.98 and τb ≥0.89.
- Overall localisation effects: Mean localisation differences are −0.49 for ∆BLEU and −1.1 for ∆chrF, while most models fall within −2 to +2 and −2 to +0.7, respectively.Two models show substantially larger declines, exceeding 8 BLEU and 5 chrF points.
- Model type: MT-optimised models consistently cluster in the negative ∆ region, including NLLB, MADLAD, Hy-MT2, and Seed-X-PPO-7B.This pattern occurs for both pre-trained translation models and LLMs further optimised for translation.
- Model size: ∆BLEU and ∆chrF generally increase with model size in 6 of 8 model families, though Hy-MT2 declines and Llama 3.X peaks at 8B before dropping at 70B.The authors interpret increasing within-family divergence between FLORES and Cultivar with reservations, noting that FLORES may underestimate stronger models’ multilingual capabilities.
- Locale effects: Locale effects vary: US localisations produce positive ∆s for Hindi and Chinese, whereas Germany, China, and Singapore produce negative ∆s, with Singapore Chinese especially low.Across locale averages, most ∆BLEU values range from −2 to +2 and most ∆chrF values from −2 to +1; locale difficulty may also reflect LLM and annotator data artefacts.
5 Manual Error Analysis
Manual inspection categorises translation errors into locale-specific and general types, revealing that locale and model size shape both error frequency and error character. Named entities dominate failures, while regional content introduces distinct challenges such as Pinyin, idioms, and measurement conversions.
- Error taxonomy: Manual analysis defines nine error types, separating four locale-specific errors from five general translation errors.Locale-specific types cover named entities, idioms and cultural expressions, numbers or units, and Pinyin; general types include mistranslation, omissions, additions, retained source language, and unwanted tokens.
- Error taxonomy: Errors are classified by manifestation rather than cause, with repeated errors of one type counted once per instance.An observed error can reflect either locale-specific or general difficulty, while different error types in the same instance are counted separately.
- Locale and model effects: US-grounded Chinese content produces the fewest errors across regional variants, especially among smaller models with higher overall error rates.The manual findings qualitatively support earlier BLEU, chrF, and ∆-score patterns.
- Locale and model effects: Locale-specific and general errors decrease as total parameter count increases, whereas active parameter count has a minor role.In Qwen3.5 mixture-of-experts models, the 35B-A3B and 122B-A10B systems have substantially fewer errors than the 2B, 4B, and 9B dense models despite activating fewer or comparable parameters.
- Error patterns: Named-entity errors exceed 80% of locale-specific errors and more than 50% of all recorded errors across models and locations.This remains the primary error source even when general translation succeeds.
- Error patterns: Regional variants produce distinct error profiles: Pinyin errors occur only in China, idiom and culture errors are more frequent there, and number-unit errors are concentrated in the UK and China.The UK and China splits contain 22 and 14 number-unit errors, respectively, versus 5 in the US split; examples include jin, British pounds, and chronological conversions.
6 Conclusion
Cultivar is a source-contrastive, locale-oriented test set designed to complement massive multilingual evaluation. Across 32 models, it finds weaker performance on localized content for smaller and MT-specialized systems, while exposing potential FLORES overfitting and US-centric bias.
- Conclusion: Cultivar isolates robustness to cultural content from general translation capability and provides a diagnostic for data contamination.Its source-contrastive, locale-oriented design complements existing massive multilingual translation evaluation.
- Conclusion: Across 32 models, smaller and MT-specialized systems perform worse on localized content.The benchmark also exposes potential FLORES overfitting and a pervasive US-centric bias in model performance.
A LLM Localisation Prompts
The localization prompt asks an LLM to localize both sentences in a parallel pair to a specified location by rephrasing named entities. Figure 4 presents the system and user prompts used for this process.
- Prompt design: The system prompt instructs the LLM to localize both sentences in a source-target parallel pair to a specified location.The stated operation is rephrasing named entities, including culturally and historically relevant content.
- Prompt design: Figure 4 documents the system and user prompts used for LLM localization.
B Instructions Presented to Annotators
The annotation instructions frame the task as creating source-language-grounded variants of FLORES+ by replacing named entities with culturally, historically, geographically, or politically appropriate alternatives. Figure 5 presents the instructions given to human annotators.
- Background and motivation: FLORES+ is English-centric because it translates English material into hundreds of languages for multiway parallelism.The instructions motivate localization because non-English translation sources would rarely be English-native in practical use.
- Annotation task: The project creates source-language-grounded FLORES+ variants by replacing named entities with locally appropriate alternatives.The intended grounding can be cultural, historical, geographical, or political.
- Annotation task: Figure 5 presents the annotation instructions given to human annotators.
C ∆Breakdown by Model and Locale
Figure 6 maps model- and locale-level changes in BLEU and chrF across the evaluation.
- The heatmaps show ∆BLEU on top and ∆chrF on the bottom, with models on the y-axis and locales on the x-axis.Models are sorted by mean ∆ across locales, locales by mean ∆ across models, and more negative ∆ values appear toward the bottom and right.