PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Adversarial Texts for Machine Translation via GRPO

Florian Zogaj, Jakob Hütteneder, Giovanni De Muri, Federico Villa, Aryan Sood, Vilém Zouhar

arXiv:2609.06048v1cs.CL

TL;DR

Improving MT systems have made standard benchmarks less informative for exposing weaknesses. The paper fine-tunes an LLM with GRPO to rewrite existing texts, and reports lower translation quality across WMT evaluations while preserving much linguistic quality.

  • Problem

    As MT systems improve, standard benchmarks become less informative for exposing their remaining weaknesses.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes an LLM with GRPO, rewarding increased translation difficulty while adding rewards to preserve rewritten-text quality.

    Source: §3.3 · p. 3, §5 · p. 9

  • Results

    On WMT25, GRPO fine-tuning substantially lowers translation quality without simply producing noisy or ungrammatical text, and the effect transfers to WMT19–WMT24.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Reinforcement-learning-based adversarial rewriting may become a practical component of MT evaluation for revealing weaknesses standard benchmarks increasingly fail to capture.

    Source: §5 · p. 9

    Difficulty estimators may be exploitable and imperfectly reflect human judgments; training and automatic evaluation focus on English→Italian, with cross-target transfer assessed only on English→German.

    Source: Limitations · p. 10

Abstract

from arXiv · show

As machine translation (MT) systems continue to improve, standard benchmarks become less informative for exposing remaining weaknesses. Traditional methods for creating challenging test sets rely on expensive manual creation or curation, while automated approaches struggle to produce sets with the necessary translation difficulty and linguistic diversity. We propose a scalable reinforcement-learning-based approach for rewriting existing source texts into instances that are more difficult to translate for MT systems. We fine-tune a large language model with Group Relative Policy Optimization (GRPO), using reward signals based on translation difficulty together with constraints for semantic similarity, grammaticality, and approximate length preservation. On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability, whereas the base model remains at 0.64. Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data, and human evaluation further shows that the rewrites substantially lower translation quality while incurring a moderate drop in naturalness and only a small change in grammaticality. We release our code to support reproducibility.

1 Introduction

Standard MT benchmarks have become less informative as systems improve, motivating more challenging test sets. Existing construction approaches face cost, scalability, difficulty, or diversity constraints.

1 Introduction

  • Standard benchmarks are increasingly insufficient to expose weaknesses or distinguish MT-system capabilities.The paper frames this as a need for more challenging test sets to guide research and deployment decisions.Source: §1 · p. 1
  • On WMT23–WMT25, fine-tuned rewrites lower translation quality relative to the base model and original data while maintaining grammaticality.Source: Figure 1 · p. 1
  • Manual curation of linguistically complex challenge examples is expensive, unscalable, and dependent on predefined design choices.Source: §1 · p. 1
  • Filtering natural datasets for difficult examples requires massive scale because naturally occurring sentences are usually simple to translate.Source: §1 · p. 1
  • Zero-shot LLM generation is scalable but struggles to produce sufficient translation difficulty and diversity.Source: §1 · p. 1

Training

The available fragment consists of a short list of text snippets.

Training

MT QE

The approach trains an LLM with GRPO using translation-difficulty signals, then generates difficult-to-translate rewrites directly at inference. Reported evaluations show reduced translation quality while maintaining grammaticality and naturalness.

MT QE

  • During training, multiple rewrites are evaluated by MT and QE models, and their translation-quality difference supplies the GRPO optimization signal.At inference, the model directly produces difficult-to-translate rewrites.Source: Figure 2 · p. 2
  • The method fine-tunes an LLM with GRPO to rewrite English text, using difficulty-estimation rewards to reduce expected translation quality.Source: MT QE · p. 2
  • Fine-tuned rewrites substantially reduce translation quality on WMT23–WMT25 while maintaining grammaticality.Figure 1 compares the fine-tuned model with the base model and original datasets.Source: MT QE · p. 2
  • Human evaluation confirms that the generated rewrites remain natural while reducing translation quality.Source: MT QE · p. 2

2 Related Work

Prior challenge-set work includes manual construction, corpus mining, and text generation; reinforcement learning has also been applied to adversarial text. Translation difficulty can be estimated through MT+QE or source-only Sentinel approaches.

2 Related Work

  • Challenge sets probe MT weaknesses missed by standard benchmarks, but traditional manual or rule-based construction is limited in scalability.ACES contains 68 linguistic error features.Source: §2 · p. 2
  • Corpus-mining methods search large-scale natural data for difficult examples, while naturally occurring sentences motivate the need for massive scale.Xu et al. (2025) frame discovery as a multi-armed bandit problem for finding seed topics.Source: §2 · p. 2
  • Generation-based work modifies source sentences iteratively, whereas this paper trains a generator to produce difficult rewrites directly at inference.Source: §2 · p. 2
  • Prior reinforcement-learning work uses minimal semantic-preserving perturbations for classification or token edits for MT, and QE-guided beam search for translation difficulty.Source: §2 · p. 2
  • MT+QE estimates difficulty from a source and candidate translation, whereas Sentinel estimates it from the source alone.CometKiwi implements MT+QE without a human reference; Sentinel needs neither candidate nor reference translation.Source: §2 · pp. 2–3

3 Methods

The methods fine-tune an LLM to rewrite source sentences for greater translation difficulty while preserving linguistic quality. GRPO uses rewards for translation difficulty alongside constraints on grammaticality, length, and semantic similarity.

3.1 Generative Model and Prompting

  • The rewriting model is Llama-3.1-8B-Instruct fine-tuned with LoRA, prompted to increase translation difficulty while preserving meaning, grammaticality, and approximate length.Source: §3.1 · p. 3

3.2 GRPO Fine-Tuning

  • GRPO compares multiple rewrites of each prompted input and updates the policy to favor candidates with higher weighted rewards.The reward is computed separately for each candidate completion.Source: §3.2 · p. 3

3.3 Reward Design

  • The weighted reward combines difficulty with CoLA and LanguageTool grammaticality signals, approximate-length preservation, and semantic similarity constraints.The reported weights are λtd = 1.0, λlen = 0.4, λsem = 0.8, λcola = 0.5, and λlt = 0.2.Source: §3.3 · pp. 3–4
  • The translation-difficulty reward favors rewrites whose estimator score is lower than the original’s, with differences scaled and clipped.Training uses either Sentinel or a COMET-based score as the estimator.Source: §3.3 · p. 3
  • The semantic-similarity reward penalizes scores below 0.70 and rewards scores at or above that threshold, avoiding pressure to nearly copy the input.Earlier shaped rewards encouraged minimal edits and empirically led to earlier collapse.Source: §3.3 · p. 4
  • Reward weights and design hyperparameters were chosen in preliminary experiments, without exhaustive tuning; systematic optimization is left for future work.Source: §3.3 · p. 4

4 Experiments

The experiments compare reward formulations and test whether GRPO rewrites make WMT benchmarks harder to translate while retaining linguistic quality. COMET-Average performs best across COMET evaluations, transfers to WMT24, and is evaluated by human raters and across training checkpoints.

4 Experiments

  • The experiments compare Sentinel and COMET rewards, evaluate four reward-trained variants across four difficulty metrics, and test WMT25 training against unseen WMT24 and earlier benchmarks.Training rewards use English→Italian translations from NLLB and Helsinki; human evaluation separately tests English→German transfer.Source: §4 · p. 4

4.2 Analysis of WMT25 Results

  • COMET-based rewards lower translation quality across metrics, whereas the Sentinel-trained variant performs best only on Sentinel and does not generalize to COMET evaluations.COMET-Average is the strongest overall variant; averaging reward signals across MT systems provides a more stable training objective than using a single estimator alone.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • COMET-Average achieves the best scores under all COMET-based WMT25 evaluations, remains competitive on Sentinel, and reduces translation quality without added grammatical errors.Its naturalness remains above the original baseline but below the non-fine-tuned base model; average output length stays close to the original, and diversity remains substantial.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 Generalization to WMT24

  • On unseen WMT24, COMET-Average lowers COMET-Average from 0.74 to 0.54 and Sentinel from -0.54 to -1.21, while the base model leaves COMET scores essentially unchanged.The fine-tuned model also keeps grammatical errors below the original and maintains similar length, Self-chrF, and naturalness.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Training Dynamics and Checkpoint Selection

  • At step 2000, translation quality has fallen substantially while grammaticality, naturalness, length, and diversity remain favorable; later checkpoints show degeneration and repeated constructions.Table 1 shows the selected checkpoint's common bigram distribution remains similar to the original, unlike the later collapsed checkpoint.Source: Table 1 · p. 5, §4.4 · p. 7, Figure 3 · p. 8

4.5 Human Evaluation

  • The human study uses 50 WMT25 source segments and 10 annotators, with English→German translations testing transfer beyond the English→Italian training rewards.Annotators rated original, base-model, and fine-tuned sentences; the three variants were randomly shuffled within each instance.Source: §4.5 · pp. 7–8
  • Human-rated translation quality falls from 56.51 for original sources to 41.29 for fine-tuned rewrites, while base-model rewrites score 57.56.The table characterizes the accompanying naturalness drop as moderate and grammaticality change as small.Source: §4.5 · pp. 8–9, Table 4 · p. 9
  • The 15.2-point human-rated translation-quality reduction exceeds the 7.1-point naturalness and 2.6-point grammaticality reductions, and translation quality does not correlate significantly with either fluency rating for the rewrites.The authors treat LLM-based naturalness and complexity scores as diagnostic rather than calibrated estimates of human judgment.Source: §4.5 · p. 9

5 Conclusion

The paper presents GRPO-based rewriting that lowers translation quality while retaining readability, and reports transfer beyond training data. It also identifies computational, estimator, and language-pair limits.

5 Conclusion

  • The approach uses GRPO to rewrite existing datasets into harder-to-translate variants while preserving naturalness, grammaticality, and approximate meaning.The conclusion identifies COMET-Average as providing the strongest trade-off between adversarial strength, readability, and diversity.Source: §5 · p. 9
  • GRPO fine-tuning substantially lowers translation quality without simply producing noisy or ungrammatical text; results transfer to WMT19–WMT24, and human evaluation confirms the reduction.Human evaluation finds a moderate drop in naturalness and only a small change in grammaticality.Source: §5 · p. 9
  • The authors suggest adversarial rewriting could become a practical component of MT evaluation, helping reveal weaknesses standard benchmarks increasingly fail to capture.Source: §5 · p. 9

Limitations

  • Reinforcement-learning fine-tuning is more computationally expensive and complex than zero-shot generation or filtering, though its cost can be amortized across future datasets.Source: Limitations · p. 10
  • Difficulty estimators may be exploited and may not reflect human judgments; LLM-based diagnostics are uncalibrated and can overestimate naturalness.Source: Limitations · p. 10
  • COMET-based training rewards and automatic evaluation cover English→Italian, while cross-target transfer is evaluated only on English→German.The authors say this does not establish generalization across broader language pairs, particularly low-resource or typologically distant ones.Source: Limitations · p. 10

A Prompts

  • The rewriting prompt asks for fluent, grammatical text that preserves meaning, and suggests idioms, ambiguity, nested clauses, and unusual but natural collocations to raise translation difficulty.Source: A Prompts · p. 12

B Collapse Pattern

Rewrites initially become more challenging while remaining fluent and interpretable, but after step 2000 they collapse toward repeated preferred constructions. GRPO updates the policy according to each completion’s reward relative to other samples for the same prompt.

B Collapse Pattern

  • After step 2000, rewrites converge on repeated constructions, culminating in repeated “high-five” and a collapsed checkpoint dominated by bigrams containing “high.”The example progresses from “high-five to that” at step 2500 to repeated “high-five” at step 3000.Source: B Collapse Pattern · p. 12
  • Earlier COMET-Average checkpoints show changing scores: step 500 Sentinel: -0.82, NLLB: 0.62, Helsinki: 0.67; step 1000 Sentinel: -0.81, NLLB: 0.56, Helsinki: 0.64.At step 1500, scores are Sentinel: -0.29, NLLB: 0.62, and Helsinki: 0.55; at step 2000, they are Sentinel: -0.95, NLLB: 0.40, and Helsinki: 0.59.Source: B Collapse Pattern · p. 12
  • COMET-Average step 3000 scores are Sentinel: -0.17, NLLB: 0.30, and Helsinki: 0.23.At step 2500, the corresponding scores are Sentinel: -0.89, NLLB: 0.40, and Helsinki: 0.31.Source: B Collapse Pattern · p. 12

C GRPO Objective

  • GRPO increases the probability of completions with higher rewards than other samples for the same prompt and discourages lower-reward completions.The old-policy samples remain fixed for L optimization steps while the current policy is updated.Source: C GRPO Objective · p. 13

D Evaluation metrics on WMT19–WMT23

The section compares the original datasets, a non-fine-tuned base model, and a selected-checkpoint COMET-Average model across WMT19–WMT23. The reported COMET scores concern English→Italian translations.

D Evaluation metrics on WMT19–WMT23

E Model Evaluations on WMT25

The section tracks translation difficulty during WMT25 training across four reward configurations. Its figures show COMET and Sentinel scores and mark the checkpoint selected for the main-paper evaluation.

E Model Evaluations on WMT25

F Sentinel Rewrites

Sentinel-based rewrites at the selected step 3500 checkpoint tend to become single, very long sentences from early training onward. The appendix also identifies the models and software used to support reproducibility.

F Sentinel Rewrites

  • Sentinel-optimized rewrites tend to become single, very long sentences from early training stages onward, unlike COMET-based rewrites.Source: F Sentinel Rewrites · p. 15
  • A T9-keyboard query is rewritten as one long sentence retaining its questions, while another example expands a short headline into a sentence.Source: F Sentinel Rewrites · p. 16

G Models Used

  • Llama-3.1-8B-Instruct serves as the LoRA-fine-tuned rewriting policy and the unmodified LLM judge; the complete software environment is released with the code.The listed software includes Python 3.10.13, torch 2.10.0, transformers 4.57.2, and trl 0.23.0.Source: Table 10 · p. 16, G Models Used · p. 16
Loading 2609.06048v1…