PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmenting Text to Increase Translation Difficulty

William Kalikman, Šimon Sukup, Michal Tešnar, Vilém Zouhar

arXiv:2608.15932v1cs.AI

TL;DR

Standard machine-translation benchmarks are saturating, making it difficult to distinguish models of varying quality. This paper introduces gradient-based Adversarial Translation Optimization to create harder texts, and finds that both variants consistently reduce translation quality versus baselines, with a naturalness tradeoff.

  • Problem

    Standard machine-translation benchmarks are saturating, creating a need for more challenging evaluation data to distinguish models of varying quality.

  • Method

    Adversarial Translation Optimization combines gradient-based difficulty minimization, fluency signals, and beam search to iteratively modify source texts.

  • Results

    Both ATO variants consistently produce harder-to-translate texts than paraphrasing and zero-shot baselines across automatic metrics and human translation-quality judgments.

  • Takeaways & Limitations

    ATO provides a scalable, gradient-based way to construct challenging translation benchmarks without LLM prompting, human intervention, or task-specific training.

  • Takeaways & Limitations

    Subword-level optimization can produce combinations of subword fragments that do not form coherent words, reflecting a fluency limitation.

Abstract

from arXiv · show

As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality. We propose augmenting existing benchmarks to increase translation difficulty by combining adversarial optimization with a differentiable translation difficulty estimator. Our Adversarial Translation Optimization (ATO) uses gradients from a combined difficulty and fluency objective to iteratively replace tokens. Because each step branches over candidate substitutions at every position, optimization becomes a tree search problem, which we address with Beam Search. ATO offers a gradient-based alternative to LLM-based dataset creation without LLM prompting, expensive human curation, or task-specific model training. Our ATO-modified benchmark lowers average translation quality (xCOMET) from 0.93 to 0.82, compared to 0.88 for paraphrasing and 0.86 for a zero-shot baseline. Human evaluation shows the modified texts are somewhat less natural than the baselines but remain reasonably grammatical and plausible while being substantially harder to translate. We release two datasets of 350 English texts each, generated by our methods, as well as the code.

1 Introduction

Standard machine translation benchmarks are saturated, motivating harder evaluation data. The paper introduces gradient-based Adversarial Translation Optimization (ATO), which increases translation difficulty while preserving fluency and produces augmented datasets without LLM prompting, human curation, or task-specific training.

  • Near-identical state-of-the-art scores on standard benchmarks make it difficult to distinguish translation models of varying quality.
  • ATO uses gradients from a differentiable translation-difficulty estimator to iteratively modify source text toward harder-to-translate inputs.The estimator is instantiated with Sentinel, which predicts expected translation quality from source text alone.
  • ATO-Direct enforces fluency through whole-word substitutions, whereas ATO-TwoPhase optimizes subwords before recovering fluency in a second phase.
  • 0.82 average xCOMET for ATO-TwoPhase is lower than 0.93 for the base, 0.88 for paraphrasing, and 0.86 for the zero-shot baseline.The evaluation uses 200 seed texts translated into five target languages by three translation models of varying capability.
  • ATO provides a gradient-based method for increasing benchmark difficulty without LLM prompting, human curation, or task-specific model training, alongside two datasets of 350 augmented English texts each.The datasets are produced by ATO-Direct and ATO-TwoPhase.

2 Related Work

Prior adversarial text optimization methods include reinforcement learning, generative adversarial networks, and embedding-space continuous relaxations, while fluency-constrained searches improve substitution quality. Translation challenge-set construction ranges from expensive expert curation and limited filtering to black-box LLM generation without per-position objective gradients.

  • Adversarial optimization: Reinforcement learning, generative adversarial networks, and embedding-space continuous relaxations have been applied to adversarial text optimization but offer limited control over individual token substitutions.HotFlip introduced gradient-based token substitution, according to the passage.
  • Fluency-constrained adversarial search: BeamAttack combines gradient-guided substitution with Beam Search to explore locally suboptimal candidates that may yield more fluent texts later.The method addresses fluency limitations in adversarial search.
  • Fluency-constrained adversarial search: BESA proposes replacements with a masked language model and combines fluency and adversarial effectiveness with energy-based annealing to escape local minima.The passage describes BESA as a fluency-constrained adversarial search method.
  • Dataset curation and generation: Traditional challenge sets use targeted expert curation that is expensive and difficult to scale, while filtering depends on naturally difficult text in existing web corpora.These approaches address how difficult translation benchmarks are sourced.
  • Dataset curation and generation: LLM-based zero-shot or iterative prompting generates difficult texts as a black-box process without per-position objective gradients, precluding gradient-based optimization with multiple differentiable losses.The passage contrasts black-box LLM generation with gradient-based methods.

3 Methods

The method uses gradient-guided token substitutions to optimize complete source texts for translation difficulty while preserving fluency. Beam Search and two complementary ATO variants balance broader search against grammaticality and coherence constraints.

  • Method design: ATO optimizes full source texts for translation difficulty rather than adversarial suffixes or LLM jailbreaking, while incorporating differentiable grammaticality signals.Its objective targets difficult yet complete, fluent texts instead of relying on post-hoc fluency filtering.
  • Difficulty estimation: Lower values of the differentiable difficulty score D(t) indicate greater translation difficulty, and Sentinel-src-25 instantiates it using source-only regression from human translation-quality judgments.Sentinel-src-25 is built on XLM-RoBERTa-large and predicts expected translation quality from the source text alone.
  • Gradient-guided search: Gradients over the L×|V| one-hot input estimate each token replacement’s effect on D and guide candidate selection.The gradient entry at position i and vocabulary token v provides a first-order approximation of the score change for that substitution.
  • Beam Search: Beam Search selects the top K=200 gradient-ranked position-token pairs, samples N=64 mutations per beam member, retains the B lowest-loss candidates, and injects 15 random candidates.Punctuation and recently changed positions are excluded or frozen to reduce cycling, while random injection prevents premature convergence.
  • ATO variants: ATO-Direct restricts substitutions to whole English words and chooses the lowest-perplexity output, whereas ATO-TwoPhase permits subwords before a second fluency-optimization phase.ATO-TwoPhase can reach lower Sentinel scores through its broader first-phase vocabulary, but risks subword-fragment combinations that harm text quality.
  • Optimization objective: The combined objective uses λ_f=20 and λ_c=50 so gradients jointly steer substitutions toward translation difficulty, grammatical naturalness, and cross-beam consistency.The objective combines Sentinel difficulty, CoLA fluency, and cosine cohesion, with all terms differentiable with respect to the token representation.

4 Experiments

Experiments evaluate whether ATO produces linguistically acceptable source texts that are harder to translate, using independent reference-free metrics, multiple translation models and languages, and human ratings. ATO-TwoPhase most strongly reduces translation quality, while ATO-Direct better preserves human-rated grammaticality and plausibility.

  • Evaluation metrics: Independent reference-free metrics assess translation quality because reference translations are unavailable and Sentinel-src cannot evaluate its own optimization objectively.MetricX is reported as 1 − MetricX/25, while xCOMET ranges from 0 to 1 with lower scores indicating worse translations.
  • Experimental setup: The evaluation compares five variants across 200 automatic-evaluation seed texts and 25 human-evaluation texts, translated into German, Spanish, Russian, Czech, and Icelandic.Variants include Base, DIPPER Paraphrasing, Qwen Zeroshot, ATO-Direct, and ATO-TwoPhase.
  • Translation quality: 0.82 xCOMET is achieved by ATO-TwoPhase, down from 0.93 for Base, compared with 0.88 for Paraphrasing and 0.86 for Zeroshot.Scores are averaged across five target languages and three translation models.
  • Variation across models: 0.13 points is the largest ATO-TwoPhase xCOMET drop for NLLB-200-3.3B, compared with 0.11 for Gemini-3-Flash and 0.09 for TranslateGemma.The effect is therefore observed across translation models with different capability levels.
  • ATO variants: ATO-TwoPhase consistently lowers translation quality more than ATO-Direct across model–language pairs, while its perplexity is lower because of explicit fluency optimization.ATO-Direct’s whole-word constraint yields texts judged more grammatical and plausible by human raters.
  • Human evaluation: ATO-generated texts receive lower translation-quality ratings than baselines, alongside lower grammaticality and plausibility, while ATO-Direct retains higher human-rated fluency than ATO-TwoPhase.The human evaluation recruited 13 annotators who rated English texts and translations on 1–5 scales.

5 Discussion

ATO consistently makes texts harder to translate than zero-shot and paraphrasing baselines, with ATO-TwoPhase producing the largest difficulty increase. The methods expose distinct translation failure modes while requiring a tradeoff between translation difficulty and human-perceived naturalness.

  • Overall findings: ATO variants consistently lower translation quality relative to zero-shot and paraphrasing baselines across automatic metrics and human judgments.ATO-TwoPhase achieves the largest difficulty increase, while naturalness proxies trade off against translation quality.
  • ATO-TwoPhase: Phase 1 maximizes translation difficulty through token replacement, whereas Phase 2 optimizes perplexity to recover fluency while implicitly preserving difficulty.Phase 1 can produce ungrammatical or nonsensical text; the resulting output is not human-readable before Phase 2 refinement.
  • Qualitative evaluation: ATO-Direct can cause NLLB-200-3.3B to omit contextually unlikely words and mishandle uncommon phrases introduced through unexpected word combinations.These texts are difficult but not necessarily untranslatable.
  • Qualitative evaluation: ATO-TwoPhase can lead NLLB to produce fluent but incorrect translations by filling in contextually plausible meanings instead of faithfully translating the source.Examples include rendering “methods mince to people” as “nicht hilfreich sind” and worsening performance through an out-of-context word.

6 Conclusion

ATO is a gradient-based method for augmenting source texts to increase translation difficulty while preserving grammaticality through fluency-aware optimization. Experiments show both ATO variants create substantially harder translation benchmarks, and the released datasets support scalable evaluation without LLM prompting or human intervention.

  • ATO combines Greedy Coordinate Gradient, Beam Search, and a differentiable fluency signal to iteratively modify source texts toward lower Sentinel-src scores.ATO-Direct restricts substitutions to whole-word tokens, while ATO-TwoPhase operates over subword vocabularies before recovering fluency through perplexity-guided optimization.
  • Both ATO variants produced substantially harder-to-translate text than paraphrasing and zero-shot baselines across MetricX and xCOMET evaluations.The experiments used 200 seed texts translated into five languages by three models of varying capability.
  • Human evaluators found ATO-generated translations lower quality, while rating the texts less grammatical and plausible than baseline texts.
  • ATO requires no LLM prompting, human intervention, or hand-crafted datasets, and its per-token gradient signal supports interpretable optimization and direct fluency constraints.Given a differentiable difficulty estimator, the method can be applied to any source text for scalable construction of challenging translation benchmarks.
  • 350 augmented texts were released in each of two datasets, one produced by ATO-Direct and one by ATO-TwoPhase.

Sustainability Statement

Dataset creation required approximately 204 RTX Pro 6000 GPU hours, while local translation evaluation took about 2 hours on an RTX 3090. Using Swiss electricity emissions estimates, the authors calculate total emissions of approximately 5.57 kg CO2.

  • Dataset creation: ∼204 RTX Pro 6000 GPU hours were required for dataset creation across both datasets.Phase 1 took ∼10 hours, Phase 2 ∼95.5 wall-clock hours across 2x RTX Pro 6000 GPUs, and Qwen 72B perplexity scoring ∼3 hours on 2x RTX Pro 6000 GPUs.
  • Evaluation: ∼2 hours on a GeForce RTX 3090 were required for local translation evaluation.The authors could not reliably estimate the impact of API usage but expected it to represent a very small share relative to dataset creation.
  • Emissions calculation: 5.57 kg CO2 was the calculated total emissions, comprising 5.51 kg from dataset creation and 0.06 kg from evaluation.Experiments ran on private infrastructure in Switzerland, using an emissions factor of 0.09 kgCO2e/kWh and an RTX A6000 approximation for RTX Pro 6000 usage.

A Additional Results · A.1 xCOMET

The appendix presents additional results from automatic evaluation, including per-language and per-model xCOMET scores with 95% confidence intervals across augmentation methods.

  • A Additional Results: The appendix reports additional results from automatic evaluation.
  • A.1 xCOMET: Table 7 reports xCOMET scores for each language.
  • A.1 xCOMET: Table 7 reports xCOMET scores for each model.
  • A.1 xCOMET: The reported scores are organized by language and model.
  • A.1 xCOMET: The table includes 95% confidence intervals for the xCOMET scores.
  • A.1 xCOMET: The xCOMET results cover all augmentation methods.

A.2 MetricX

Table 8 reports MetricX scores by language and model, including 95% confidence intervals across all augmentation methods.

  • MetricX: Table 8 presents MetricX scores across all augmentation methods.The table reports results by language and model, with 95% confidence intervals.
  • MetricX: The MetricX results are broken down by language and model.
  • MetricX: Each reported MetricX score includes a 95% confidence interval.

A.3 Data Metrics … C.3 Inter-Annotator Agreement

The paper evaluates data quality, documents translation and baseline prompts, specifies replacement and paraphrasing procedures, and reports human-evaluation agreement. These analyses indicate stable word diversity, controlled perturbations, and relative annotator agreement despite variation in absolute agreement.

  • A.3 Data Metrics: Word diversity remains stable across methods, while word count increases steadily and word length shows no clear trend.Word diversity is normalized by total word count; the analysis tests whether optimization collapses onto a small token set.
  • B.1 Translation Prompt: Translation models were prompted to preserve list order, return only a Python-style list, and avoid skipping difficult sequences.Translation strings were batched by 10 for throughput.
  • B.2 Zeroshot Baseline Prompt: Qwen 2.5 72B and DeepSeek-V4-Flash served as zero-shot replacement baselines using the Figure 4 prompt.DeepSeek-V4-Flash was accessed through the OpenRouter API as deepseek/deepseek-v4-flash on 06-05-2026.
  • B.3 DIPPER Paraphrasing Baseline Details: DIPPER paraphrasing used zero order diversity and lexical diversity 20 to match the two-phase method’s edit-distance distribution.DIPPER’s lexical diversity controls how aggressively words change, while order diversity controls word reordering.
  • B.4 Random Replacement Baseline Details: Random replacement changes exactly 2 words or subwords per seed sentence, with whole-word candidates split into common and rare frequency buckets.Candidates require Zipf frequency greater than 2.5; the median split is Zipf ≥4.09 for common and 2.5 < Zipf < 4.09 for rare.
  • C.1 Grammaticality and Plausibility: Human ratings assessed grammaticality on a 1–5 scale ranging from completely ungrammatical to fully grammatical.The supplied instructions define intermediate levels for major problems, noticeable issues, and minor imperfections.
  • C.2 Translation Quality: Translation quality was evaluated using the instructions shown in Figure 6.The supplied figure is identified as the instructions for rating translation quality.
  • C.3 Inter-Annotator Agreement: Absolute Krippendorff’s 𝛼 ranged from 0.18–0.79 across metrics, whereas relative 𝛼 ranged from 0.41–0.51.The relative range suggests per-rater scale bias contributes to absolute variation rather than disagreement about relative ordering.

C.4 Evaluation interface

The evaluation uses Pearmut, an open-source translation evaluation tool, making the assessments reproducible from the evaluated data. Separate interfaces support translation-quality evaluation across languages and English-text quality evaluation.

  • Implementation: Pearmut is an open-source translation evaluation tool used to make evaluations reproducible given the data being evaluated.The tool is attributed to Zouhar and Kocmi (2026).
  • Translation quality: The translation-quality evaluation interface is shown for Icelandic, with separate interfaces for each other language.
  • English text quality: A separate interface evaluates English text quality.
Loading 2608.15932v1…