Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

• English-anchored multilingual suites risk contamination and input-content mismatch, while naturally written, culturally grounded tests improve authenticity but sacrifice comparability.Source: §1 · p. 1
• Because languages can span multiple regions and diaspora communities, one test per language can miss other locales or produce an arbitrary mixed-origin result.Source: §1 · pp. 1–2
• Cultivar provides a 27-locale test set created through human-LLM co-creation from FLORES while preserving comparability and incorporating native source content.Source: §1 · p. 2
• Paired instances preserve sentence structure while varying locale-specific content, enabling performance discrepancies to probe contamination and quantify localisation robustness.Source: §1 · p. 2
• The evaluation covers 32 models with BLEU, chrF, and word recall, reporting locale-level scores and contrasting Cultivar with FLORES to examine robustness and overfitting.Source: §1 · p. 2
• MT-specialised and smaller models tend to lose performance on localised content; original/localised gaps suggest potential FLORES overfitting, and models favor US-grounded content over regional official-language content.Source: §1 · p. 2

2 Cultivar

• The three-stage pipeline selects named-entity-rich instances, uses LLM-assisted paraphrasing, then applies human post-editing; it can also be applied to other benchmarks.Localisation is creative paraphrasing that preserves original sentence structure while adapting cultural content; this study covers only any-to-English directions, whose targets remain comparable despite differing across locales.Source: §2 · p. 2

2.1 Named entity-rich instance identification

• Named-entity annotation found 658 English FLORES dev sentences, about 65% of the set, and researchers selected 200 with at least two entities for localisation.They excluded examples dominated by numerals or globally shared entities in medicine and science.Source: §2.1 · p. 2

2.2 LLM paraphrasing

• For each selected instance, GPT-5.5 generates a localised source-English pair from the original pair and target location, preserving structure where possible while replacing content with local alternatives.The workflow uses LLM assistance because localisation requires creativity and diversity beyond translation; human post-editing supports quality and cultural appropriateness.Source: §2.2 · p. 2, §2.3 · p. 3

2.3 Human annotation

• Annotators proficient in both languages and familiar with the location review each pair, then accept, post-edit, or revert the LLM paraphrase.A custom interface displays original and localised pairs side by side, highlights changed spans, and initializes the edit box with the LLM version.Source: §2.3 · pp. 3–4

2.4 Annotated data statistics

• Cultivar contains 27 language-script-location locales, with 21 languages or dialects, 8 writing scripts, 21 countries or regions, and 200 instances per locale.Each locale’s instances were contributed by a single annotator.Source: §2.4 · p. 4
• Annotators accept 70% of LLM localisations on average, post-edit roughly 28%, and revert fewer than 2%, leaving over 98% as newly localised content.Some locales are outliers: nearly all Hindi-to-Germany paraphrases are edited, while only 7 Filipino and 13 Malagasy instances are changed.Source: §2.4 · p. 4

3.1 Models

• The benchmark covers 32 open-weight models deployable on at most two NVIDIA A100 80GB GPUs, selected across four dimensions and grouped as MT, LLM+MT, and LLM models.The dimensions are translation-specific versus general-purpose, 0.8–122B size, English-focused versus multilingual, and models made in or targeting diverse locations.Source: §3.1 · p. 5

3.2 Evaluation

• BLEU and chrF use spBLEU with the flores200 tokenizer and chrF with zero word order, respectively, as implemented in sacrebleu.These metrics operate on subwords and characters without tokenization complications for certain languages.Source: §3.2 · p. 5
• Robustness is measured as the performance discrepancy between localised and original tests, with increasingly negative Δ indicating greater degradation and Δ near zero indicating consistent performance regardless of translation quality.The mean pairwise Δmetric aggregates locale-language pairs, and may also be averaged across models without pairing.Source: §3.2 · p. 5
• Locale-specific word recall compares localised translations with original FLORES and localised Cultivar references to track obsolete original terms and omitted newly introduced terms.False positives suggest continued generation of removed FLORES terms and possible memorization; false negatives indicate missing locale-specific content but are an upper bound on localisation failures because multiple translations may be valid.Source: §3.2 · p. 5
• Table 2 reports Spearman’s rs and Kendall’s τb alignment between 32-model rankings on original FLORES and localised Cultivar when translating into English.Source: Table 2 · p. 6

4.1 Do Cultivar and FLORES yield consistent model rankings?

• Cultivar and FLORES yield highly consistent model rankings, with Spearman rs ≥0.93 and Kendall τb ≥0.80 in every direction, and averages of rs ≥0.98 and τb ≥0.89.Source: §4.1 · p. 6

4.2 Do models perform better or worse with localised source content?

• Most models’ false negatives are 10–30%; treating weaker low-BLEU, low-chrF models with FN >44% as random baselines, the authors conclude most models adapt to localized sources to some extent.MT-optimised models rank lower by ∆ scores and cluster consistently in the negative region, including NLLB, MADLAD, Hy-MT2 and Seed-X-PPO-7B.Source: §4.2 · p. 6
• Delta BLEU and chrF rise with model size in 6 of 8 families, but Hy-MT2 dips and Llama 3.X peaks at 8B before falling at 70B, with version differences a caveat.The authors suggest this divergence between FLORES and Cultivar may mean FLORES underestimates stronger models’ multilingual translation capabilities.Source: §4.2 · p. 6
• Mean pairwise deltas average −0.49 BLEU and −1.1 chrF; most models fall within −2 to +2 BLEU and −2 to +0.7 chrF, while two bottom models drop over 8 BLEU and 5 chrF.SmolLM-1.7B-Instruct has near-zero deltas but is described as a weak-translation random baseline.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Are models FLORES contaminated?

• Two strong FLORES models, aya-expanse-8b and Seed-X-PPO-7B, drop over 8 BLEU on Cultivar, but false positives show no direct evidence of exact FLORES-text memorisation.Their declines occur across most source-to-English directions; the authors say the pattern may instead indicate weak localisation robustness or some FLORES overfitting.Source: §4.3 · p. 7

4.5 Same language, different localisations

• Models translate US-localised Hindi and Chinese better than FLORES, but Chinese content about China and Singapore worse, suggesting a US-centric training-data bias.The authors describe Cultivar’s US localisation as easier for models and the negative deltas elsewhere as reduced capability on source-localised content.Source: §4.5 · pp. 8–9

4.4 Which locales are harder?

• BLEU improves for Madagascar and India localisations, while chrF improves for US Hindi and Chinese plus Madagascar; Chinese Singapore and five other test sets show declines around −2.The authors note that locale difficulty can change after localisation, but attribute it potentially to both model and data artifacts, including annotator biases.Source: §4.4 · pp. 7–8, Table 4 · p. 9

5.1 Setup

• Manual analysis defines nine error types in locale-specific and general categories, classifying errors by manifestation rather than underlying cause.A Chinese native speaker fluent in English inspected selected Chinese-to-English outputs with sources and references. Locale-specific types are named entities, idioms and cultural expressions, numbers or units, and Pinyin; general types include semantic mistranslation, omissions, additions or hallucinations, retained source language, and unwanted reasoning or formatting tokens. Repeated instances of one type count once, while different types count separately.Source: §5.1 · p. 9

5.2 Observations corroborate earlier findings

• Locale-specific and general errors decline as total parameter count increases, while active parameter count plays a minor role.Qwen3.5 mixture-of-experts models 35B-A3B and 122B-A10B have substantially fewer errors than dense 2B, 4B, and 9B models despite activating fewer or comparable parameters; the pattern qualitatively supports earlier BLEU, chrF, and ∆ findings.Source: §5.2 · p. 9
• Seed-X-PPO-7B’s Chinese BLEU–error correlation is −0.64 including it and −0.86 excluding it, yet its BLEU exceeds other models despite comparable or higher error counts.Its US-split BLEU is much higher than its UK and China scores; this mismatch may indicate FLORES-like contamination inflating automatic scores.Source: §5.2 · p. 9
• Across the three regional variants, US-grounded Chinese content yields the fewest translation errors, especially among smaller models with higher error rates.Table 5 reports error counts by model and country, alongside BLEU for reference.Source: §5.2 · p. 9, Table 5 · p. 10

5.3 Locale-specific error patterns

• Named entities account for > 80% of locale-specific errors and > 50% of all errors, while other error types vary by locale.Idioms and culturally specific expressions are more frequent in China; Pinyin errors occur only there and make up 20% of all errors, reflecting Chinese-to-Latin romanisation demands. Number-and-unit errors total 22 in the UK, 14 in China, and 5 in the US, involving region-specific measurements, formats, and conversions. These patterns show how localization changes translation difficulty and motivate locale-oriented testing.Source: §5.3 · p. 10

6 Conclusion

• Across 32 models, smaller and MT-specialised systems perform worse on localised content, while the contrastive design exposes potential FLORES overfitting and better translation of US-grounded than native-region content.The latter pattern is described as a pervasive US-centric bias.Source: §6 · p. 10
• Cultivar complements multilingual translation evaluation by isolating robustness to cultural content from general translation capability and serving as a diagnostic tool for data contamination.Source: §6 · p. 10
• The conclusion argues that true multilingual proficiency requires locale-aware evaluation to reflect real-world use cases accurately.Source: §6 · p. 10

• Figure 4 shows the system and user prompts used for LLM localisation.Source: Figure 4 · p. 13

• Figure 5 presents the annotation instruction given to human annotators.Source: Figure 5 · p. 13

• Figure 6 heatmaps show ∆BLEU and ∆chrF by model and locale, sorting models and locales by their mean ∆, with more negative values toward the bottom and right.Source: Figure 6 · p. 14