Augmenting Text to Increase Translation Difficulty

1 Introduction

• Both ATO variants produce harder-to-translate text than the baselines; average xCOMET drops from 0.93 (base) to 0.82 (ATO-TwoPhase), versus 0.88 for paraphrasing and 0.86 for the zero-shot baseline.The evaluation used 200 seed texts translated into five target languages by three models; human annotations confirmed lower translation quality.Source: Figure 1 · p. 1, §1 · p. 2
• The approach targets limits of expert-crafted challenge sets, naturally difficult web text, and LLM rewriting, which lacks per-position gradients for directly constraining difficulty and fluency.The introduction describes expert-crafted sets as expensive and difficult to scale, and web mining as limited by the availability of naturally difficult content.Source: §1 · p. 1
• ATO combines gradient-guided candidate selection, Beam Search, and a differentiable grammaticality signal to seek difficult-to-translate, well-formed texts.It uses Sentinel, a regression model that predicts expected translation quality from source text alone, and minimizes its score; following the difficulty gradient alone can produce nonsensical text.Source: §1 · pp. 1–2
• ATO-Direct restricts substitutions to whole-word tokens and selects outputs by perplexity, while ATO-TwoPhase first optimizes over subwords and then recovers fluency using another language model.ATO-Direct enforces fluency by construction; ATO-TwoPhase optimizes for perplexity in its second phase.Source: §1 · p. 2
• The contributions are gradient-based ATO without LLM prompting, human curation, or task-specific model training, plus two datasets of 350 augmented English texts each.The datasets were produced by ATO-Direct and ATO-TwoPhase.Source: §1 · p. 2

Adversarial optimization

• Gradient-based token attacks improve substitution control, but Greedy Coordinate Gradient (Zou et al., 2023) can produce disfluent or nonsensical text without fluency constraints.Earlier work used reinforcement learning, generative adversarial networks, and embedding-space relaxations (Sadrizadeh et al., 2023); HotFlip uses one-hot gradients to identify single-token substitutions.Source: §2 · p. 2

Fluency-constrained adversarial search

• BeamAttack, BESA, and AutoDAN address fluency, but target adversarial suffixes for LLM safety.BeamAttack uses Beam Search to explore locally suboptimal candidates, BESA proposes fluent replacements with a masked language model and energy-based annealing, and AutoDAN uses a hierarchical genetic algorithm with LLM-based fitness metrics.Source: §2 · p. 2

Dataset curation and generation

• Benchmark sourcing ranges from costly expert curation to limited web filtering and black-box LLM generation, which lacks gradients for optimizing combined differentiable losses.Expert curation is targeted but difficult to scale; filtering searches for naturally difficult web text (Proietti et al., 2025; Xu et al., 2025), while zero-shot or iterative LLM prompting generates texts without per-position gradients (Pombal et al., 2025; Zouhar et al., 2025).Source: §2 · pp. 2–3

3.1 Optimization Objective

• Sentinel-src-25 provides a differentiable, source-only estimate of translation quality, with lower scores indicating texts expected to be harder to translate.Gradients through a one-hot token representation estimate how substitutions affect the difficulty score and guide candidate selection.Source: §3.1 · p. 3

3.2 Beam Search with Greedy Coordinate Gradient

• ATO seeds search with real text, maintains a beam of 50 candidates, samples 64 mutations per member from a gradient-ranked shortlist of 200 position-token pairs, retains globally the lowest-loss candidates, and injects 15 random candidates.Source: §3.2 · pp. 3–4

3.3 ATO-Direct

• ATO-Direct performs one beam-search phase using whole-word substitutions and combines Sentinel difficulty, CoLA grammaticality, and beam-level cosine cohesion in its objective.It uses λ_f = 20 and λ_c = 50, then selects the candidate with the lowest Qwen 2.5-72B perplexity; split-word positions are frozen.Source: §3.3 · pp. 4–5

3.4 ATO-TwoPhase

• ATO-TwoPhase first searches a broader subword vocabulary for lower Sentinel scores, then optimizes fluency with Qwen perplexity, trading broader search for possible incoherent word fragments.The first phase uses the same Sentinel, CoLA, and cosine-cohesion objective as ATO-Direct; the final output is the lowest-perplexity candidate encountered in Phase 2.Source: §3.4 · pp. 5–6

3.5 Baselines

• Baselines include two-word edits prompted from Qwen2.5-72B-Instruct or DeepSeek-V4-Flash, DIPPER paraphrases, and random token replacements.ATO-Direct replaces one to three words of the original text as the point of comparison for the prompted LLM edits.Source: §3.5 · pp. 6–7

4 Experiments

• The evaluation uses 200 automatic and 25 human English seeds from FLORES200 and WMT22–24, comparing five variants translated into five languages by three models.Seeds take the first 50 tokens, truncated at sentence-ending punctuation; resulting segments are typically 10–20 words and may be fragments, titles, or full sentences. Difficulty is assessed with reference-free metrics and human judgments rather than reference-based metrics or the optimized Sentinel-src objective.Source: §4 · p. 7

4.1 Automatic Evaluation

• Both ATO variants substantially reduce Sentinel-src scores, with TwoPhase lowest; perplexity rises moderately for Direct but decreases for TwoPhase, while baselines leave Sentinel scores approximately unchanged.The TwoPhase perplexity decrease reflects Phase 2’s explicit perplexity optimization; by this measure its texts are no less fluent than the originals.Source: §4.1 · pp. 7–8
• On xCOMET, ATO-TwoPhase lowers scores from 0.93 to 0.82, versus 0.88 for Paraphrasing and 0.86 for Zeroshot; it achieves the largest decrease on both metrics.Scores are averaged across five target languages and three translation models; the differences from both baselines exceed the 95% confidence intervals. ATO-Direct scores 0.86 on xCOMET and 0.85 on MetricX, versus 0.87 for Zeroshot on MetricX, with overlapping CIs.Source: §4.1 · p. 8, Table 3 · p. 8
• ATO-TwoPhase lowers xCOMET by 0.13 for NLLB-200-3.3B, 0.11 for Gemini-3-Flash, and 0.09 for TranslateGemma; drops occur in all five languages.Drops are largest for Czech, Russian, and Icelandic (0.12–0.14), compared with 0.10 for Spanish and 0.07 for German.Source: §4.1 · p. 8
• ATO-TwoPhase scores lower on translation quality than ATO-Direct across all model–language pairs and has lower perplexity, while Direct’s whole-word constraint yields more grammatical and plausible texts.The paper notes that perplexity alone does not capture all aspects of naturalness.Source: §4.1 · p. 8

4.2 Human Evaluation

• Both ATO variants lower human-rated translation quality versus the baselines, but ATO-Direct achieves a comparable reduction while retaining higher grammaticality and plausibility.Thirteen annotators rated English text grammaticality and plausibility on 1–5 scales, and rated translations of all five variants into five languages.Source: §4.2 · pp. 8–9

Discussion

• Across automatic metrics and human judgments, both ATO variants produce harder-to-translate texts than zero-shot and paraphrasing baselines, with ATO-TwoPhase achieving the largest difficulty increase, at a naturalness-proxy tradeoff.The cited tradeoff concerns grammaticality and plausibility versus translation quality.Source: §5 · p. 9
• Phase 1 replaces tokens to maximize translation difficulty, while Phase 2 optimizes perplexity to recover fluency while implicitly preserving difficulty.In the example, Phase 1's output was not human-readable, whereas the final text was fluent and grammatical and remained harder to translate than the base text.Source: §5 · p. 9

Qualitative evaluation

• In qualitative examples translated by NLLB-200-3.3B, ATO-Direct can cause omission of a contextually unlikely but directly translatable word; in one case, “run” could enter “die Laufinteressen” but was omitted.The passage says these translations contain unexpected word combinations, but that this does not mean they are untranslatable.Source: §5 · p. 9
• ATO-Direct’s uncommon or awkward phrases can prompt NLLB-200-3.3B to distort syntactic structure or add explanatory meaning absent from the source.Examples include translating “it was my cable to assist” around “cable” and elaborating “sorry investment” into an investment that initially seemed like a failure.Source: §5 · p. 9, Table 5 · p. 10

6 Conclusion

• ATO combines Greedy Coordinate Gradient, Beam Search, and a differentiable fluency signal to iteratively modify source texts toward lower Sentinel-src scores while preserving grammaticality.ATO-Direct restricts substitutions to whole-word tokens.Source: §6 · p. 10
• ATO variants produced substantially harder-to-translate text than paraphrasing and zero-shot baselines on MetricX and xCOMET; human evaluators found lower translation quality but reduced grammaticality and plausibility.The authors suggest stronger fluency constraints or alternative optimization objectives as future work to better reconcile difficulty with naturalness.Source: §6 · p. 10
• ATO requires no LLM prompting, human intervention, or hand-crafted datasets, and its per-token gradients support interpretable optimization and direct fluency constraints.With a differentiable difficulty estimator, it can be applied to any source text; the authors release two 350-text datasets, one each from ATO-Direct and ATO-TwoPhase.Source: §6 · p. 10

• ∼204 RTX Pro 6000 GPU hours were used for dataset creation across both datasets.Phase 1 took ∼10 hours, Phase 2 took ∼95.5 wall-clock hours across 2x RTX Pro 6000 GPUs, and Qwen 72B perplexity scoring took ∼3 wall-clock hours on 2x RTX Pro 6000 GPUs.Source: Sustainability Statement · p. 11
• Local translation evaluation took ∼2 hours on a GeForce RTX 3090, while API usage impact could not be reliably estimated.The authors expect API usage to account for a very small share of total impact relative to dataset creation.Source: Sustainability Statement · p. 11
• 5.51 kg CO2 for dataset creation and 0.06 kg for evaluation yield an estimated total of ≈5.57 kg CO2.The estimate uses a Swiss electricity emissions factor of 0.09 kgCO2e/kWh and approximates RTX Pro 6000 use with an RTX A6000 in the Machine Learning CO2 Impact Calculator.Source: Sustainability Statement · p. 11

A Additional Results

• The appendix presents additional results from the automatic evaluation.Source: A Additional Results · p. 14

A.1 xCOMET

• Table 7 reports per-language, per-model xCOMET scores with 95% confidence intervals across all augmentation methods.Source: Table 7 · p. 14

A.2 MetricX

• Table 8 reports per-language, per-model MetricX scores with 95% confidence intervals across all augmentation methods.Source: Table 8 · p. 15

A.3 Data Metrics

• Word counts rise steadily across generation methods, while word diversity remains stable and word length shows no clear trend, indicating no fixed-token collapse.The analysis measures word diversity as unique words normalized by total word count, alongside average word length and word count.Source: §A.3 · pp. 15–16, Table 9 · p. 16

B.1 Translation Prompt

• Gemini-3-Flash and DeepSeek-V4-Flash translations used the Figure 3 prompt, with inputs batched by 10.Gemini was accessed through the official Google API; DeepSeek was accessed through OpenRouter.Source: §B.1 · p. 16, Figure 3 · p. 16

B.2 Zeroshot Baseline Prompt

• Qwen 2.5 72B and DeepSeek-V4-Flash served as zero-shot replacement baselines using the prompt shown in Figure 4.DeepSeek-V4-Flash was accessed through OpenRouter.Source: §B.2 · p. 16, Figure 4 · p. 16

B.3 DIPPER Paraphrasing Baseline Details

• DIPPER paraphrasing fixed order diversity at zero and set lexical diversity to 20, matching its edit-distance distribution to the two-phase method.Lexical diversity was selected by comparing edit-distance distributions across the 0–100 range.Source: §B.3 · pp. 16–17

B.4 Random Replacement Baseline Details

• The random-replacement baseline substitutes exactly two whole words or subwords per seed sentence and divides eligible whole words into common and rare buckets.It filters out words with Zipf frequency ≤2.5; common words have Zipf ≥4.09, and rare words have 2.5 < Zipf < 4.09.Source: §B.4 · p. 17

C.1 Grammaticality and Plausibility

• Figure 5 provides instructions for rating grammaticality and plausibility.Source: Figure 5 · p. 17

C.2 Translation Quality

• Figure 6 provides instructions for rating translation quality.Source: Figure 6 · p. 17

C.3 Inter-Annotator Agreement

• Relative Krippendorff’s α ranges from 0.41 to 0.51, while absolute α varies from 0.18 to 0.79 across human-evaluation metrics.The table attributes the absolute variation to per-rater scale bias rather than disagreement on relative item ordering.Source: Table 10 · p. 17

• Pearmut (Zouhar and Kocmi, 2026) is an open-source translation evaluation tool, making evaluations reproducible given the data to be evaluated.Source: §C.4 · p. 18
• The translation-quality interface is shown for Icelandic, with a separate interface for each other language.Source: Figure 7 · p. 18
• A separate interface is shown for evaluating English text quality.Source: Figure 8 · p. 18