Generating Difficult-to-Translate Texts

• Selection methods fall short because naturally occurring challenging examples are insufficient (Proietti et al., 2025).Source: §1 · p. 1
• Zero-shot from-scratch generation is reported inadequate for difficulty and diversity (Pombal et al., 2025).Source: §1 · p. 1
• Human practitioners probe model weaknesses by iteratively revising inputs in response to faulty MT outputs, rather than only selecting or synthesizing examples.Source: §1 · p. 1
• MT-breaker iteratively modifies a seed using the MT model’s translation as feedback, then selects the source that produced the worst translation.Source: §1 · pp. 1–2
• MT-breaker increases generated-example difficulty while preserving source diversity and naturalness, with difficulty also transferring to other models and languages.Source: §1 · p. 2

• The seeds baseline uses existing dataset texts without creating new examples.Source: §2 · p. 2
• Zero-shot benchmarking prompts an LLM to generate benchmark examples, optionally steering topics, domains, or difficulty; its history variant includes prior generations to avoid repetition.Source: §2 · p. 2

MT-breaker

• MT-breaker starts with a seed, feeds its translation and the seed to an LLM for iterative modification, then selects the most difficult source using quality-estimation scores.Source: §2 · p. 2

3.1 Increasing Difficulty

• Translation quality declines with each MT-breaker step, plateauing around step 10; quality-estimation guidance does not help seeded generation and worsens seedless generation.For seedless generation, the per-step averages are the same, but the quality-estimation version has lower variance (σ2=297 vs. 325), which can raise its minimum score. Seeded methods are less difficult than seedless methods.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Data Quality

• Methods without explicit diversity treatment produce less diverse but more difficult texts; MT-breaker seeded offers a balance when diversity matters.MT-breaker seedless is an apt choice when difficulty alone is the priority.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

3.3 Transfer Between MT Models

• Targeted MT-breaker examples most strongly challenge the target model, while difficulty transfers to other models but is generally on par with or slightly below Zeroshot.Targeting Google Translate generally yields lower difficulty for other models, though this is not consistent when targeting more performant models.Source: §3.3 · p. 5, Table 4 · p. 6
• Multi-model MT-breaker is at worst second best for individual models, outperforms transfer from an individual target, and raises difficulty for unseen o4-mini.The authors suggest this indicates a more general notion of difficulty than model-specific optimization.Source: §3.3 · p. 5

3.4 Transfer between LLMs

• LLM choice yields systematically different outputs without discernible bias; Gemini 2.0 optimization appears more difficult but substantially less diverse.This pattern is consistent with the reported difficulty–diversity trade-off.Source: §3.4 · pp. 5–6, Table 3 · p. 5

3.5 Transfer Between Languages

• Examples are most difficult in their targeted language direction, but some difficulty transfers; seedless MT-breaker remains more difficult than Zeroshot (min) on non-target languages.Translations from other languages into English show similar method rankings.Source: §3.5 · p. 6, Table 5 · p. 6

3.6 Human Translations and Annotations

• Human evaluations show seedless examples are most difficult, followed by Zeroshot and seeded MT-breaker, with little difference between human and automatic translations.Some Zeroshot and seedless segments were marked “untranslateable”; Mistranslation and Creative reinterpretation were more common with MT-breaker and Zeroshot than with Seeds.Source: §3.6 · pp. 6–7, Table 6 · p. 7

3.7 Interview with Experts

• Experts iteratively edit inputs that trigger minor errors to provoke larger failures, sometimes exploring non-standard inputs; MT-breaker adopts iterative exploitation but restricts edits to grammatical changes.The interview study involved seven experienced MT practitioners; MT-breaker exploits an input for up to 10 steps.Source: §3.7 · p. 7

Selecting examples

• Selection methods use rarity, complexity, or predicted difficulty, but are limited by corpus diversity and may miss novel failure modes.Chen et al. (2023) use lexical rarity, sentence length, low-frequency parse trees, and translation entropy; Proietti et al. (2025) predict difficulty, while Zouhar et al. (2025) find error-rich samples more useful for model comparison.Source: §4 · p. 7

Generating examples

• From-scratch generation ranges from manually or rule-constructed examples to zero-shot LLM benchmarking, which can lack difficulty and diversity for state-of-the-art models.Examples include Isabelle et al. (2017), Manakhimova et al. (2023), and Pombal et al. (2025).Source: §4 · p. 8
• MT-breaker combines scalable LLM generation with seed texts and iterative feedback to target difficulty while preserving diversity.This contrasts with zero-shot generation from scratch, which may lack sufficient difficulty and diversity for state-of-the-art models (Pombal et al., 2025).Source: §4 · p. 8

Adversarial examples

• Adversarial methods seek model failure through minimal perturbations to otherwise easy inputs, including reinforcement-learning token edits and heuristic edits.Zou et al. (2020) preserve source meaning while degrading translation quality; Zhang et al. (2021) probe brittleness to superficial changes.Source: §4 · p. 8

Interactive discovery

• Interactive discovery methods use one model to reveal another’s capabilities or features, but prior examples did not use machine translation’s accurate quality-estimation metrics.Lu et al. (2025) generate open-ended tasks, while Tjuatja and Neubig (2025) discover interpretable comparative features.Source: §4 · p. 8

5 Conclusion

• The dataset is tailored to the target model’s vulnerabilities, so evaluating other models on it would be misleading and unfair.The authors suggest using a model’s own generated set to find weaknesses for hillclimbing, or generating separate sets to compare models’ worst-case performance.Source: §5 · p. 8
• MT-breaker increases translation difficulty for targeted models while maintaining seed-text naturalness and diversity, with difficulty also transferring to other models and languages.The increase is confirmed by expert human evaluation.Source: §5 · p. 8
• Difficulty transfers beyond the target, but is most pronounced for the targeted model and language.Source: §5 · p. 8
• The authors hypothesize that some generated texts may be difficult for human translators rather than machine translation models.They suggest this may reflect internet training data discussing difficulty from a human translator’s perspective.Source: §5 · p. 8

Limitations

• Quality estimation serves as both the optimization objective and a quantity of interest, although expert MQM evaluation also confirms the findings.The authors state that they rely heavily on this metric.Source: Limitations · p. 8
• The Zeroshot baseline omits domain conditioning, which could yield more diverse outputs, to make its implementation more comparable with other methods.Source: Limitations · p. 8
• Generated texts fall outside the expected user distribution because the study targets less likely, more difficult examples that reveal worst-case model performance.Source: Limitations · p. 8

Ethics Statement

• The human-study data came from the WMT 2024 test set, already screened for disturbing content.Source: Ethics Statement · p. 9
• Translators and annotators were fairly compensated and consented to participate in the research study.Source: Ethics Statement · p. 9
• The study collected no personal data.Source: Ethics Statement · p. 9

A Source-only MetricX

• Source-only MetricX estimates difficulty using only the source text, without the translation model’s output.Source: A Source-only MetricX · p. 11
• Its training target is the average human assessment score aggregated across translations of the same source.For example, scores of 76 and 98 yield a target of 87.Source: A Source-only MetricX · p. 11
• The rest of its training follows the MetricX-23 procedure.Source: A Source-only MetricX · p. 11

B Prompts

• The first prompt is the same across Zeroshot and MT-breaker.Source: B Prompts · p. 11
• The follow-up asks the model to make translation more difficult by modifying at most 75% of the best source, defined as the one with the lowest score.Source: B Prompts · p. 11
• Source analysis asks for grammaticality, naturalness, word rarity, syntax complexity, and one to five topics.Source: B Prompts · p. 11
• Target analysis asks for possible translation-error reasons, including idioms, ambiguity, named entities, dates, syntax, and semantics.Source: B Prompts · p. 11

C Data Examples

• Table 7 presents English generated-source examples from Zeroshot, Seeds, and seeded MT-breaker.It extends Table 1 from the main text.Source: Table 7 · p. 12
• In the English→Czech run, step 3 is selected as the final candidate because it has the lowest average score.The table shows one run across 10 steps and underlines major errors.Source: Table 8 · p. 13
• In the English→German run, step 5 is selected as the final candidate because it has the lowest average score.The table shows one run across 10 steps and underlines major errors.Source: Table 9 · p. 14

D Human Evaluation

• Table 10 shows human translations and annotations, with QE data limited to segments translated by all humans.It distinguishes segments not translated by human translators as “NS.”Source: Table 10 · p. 15
• Table 10 splits results by English→Czech and English→German directions and also reports averages.The MQM score is converted to 100+4·MQM to place it on a scale similar to MetricX.Source: Table 10 · p. 15
• Table 11 reports MQM error-category distributions only for categories occurring at least 10 times overall.Percentages are normalized independently for each table column, and only segments with human translations are included.Source: Table 11 · p. 16