Source-linked AI summary
A Relaxed Optimization Approach for Adversarial Attacks against Neural Machine Translation Models
Sahar Sadrizadeh, Clément Barbier, Ljiljana Dolamic, Pascal Frossard
TL;DR
NMT adversarial attacks aim to damage translation while keeping the input sentence semantically similar, but discrete text makes optimization difficult. The paper relaxes token choices into distributions, optimizes them, and samples adversarial examples. It reports strong degradation and better performance than prior attacks across white-box and black-box settings.
Problem
Textual inputs are discrete, making optimization-based attacks difficult, while NMT adversarial examples must preserve source meaning and degrade translation quality.
Source: §2.1 · p. 2Method
The attack optimizes token probability distributions through a continuous relaxation, samples adversarial examples, and extends sampling to a reference model for black-box attacks.
Source: §1 · p. 2, §2.2 · p. 3Results
Takeaways & Limitations
The experiments support the attack’s effectiveness across different NMT models and translation tasks, including black-box settings.
Source: §4 · p. 6The code for the untargeted NMT attack in [6] was not publicly available.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
In this paper, we propose an optimization-based adversarial attack against Neural Machine Translation (NMT) models. First, we propose an optimization problem to generate adversarial examples that are semantically similar to the original sentences but destroy the translation generated by the target NMT model. This optimization problem is discrete, and we propose a continuous relaxation to solve it. With this relaxation, we find a probability distribution for each token in the adversarial example, and then we can generate multiple adversarial examples by sampling from these distributions. Experimental results show that our attack significantly degrades the translation quality of multiple NMT models while maintaining the semantic similarity between the original and adversarial sentences. Furthermore, our attack outperforms the baselines in terms of success rate, similarity preservation, effect on translation quality, and token error rate. Finally, we propose a black-box extension of our attack by sampling from an optimized probability distribution for a reference model whose gradients are accessible.
1 Introduction
NMT attacks seek to lower translation quality while preserving source-language meaning, but text’s discrete nature makes optimization difficult. The paper proposes an optimization-based attack and reports degradation with preserved similarity and stronger baseline comparisons.
1 Introduction
- The study of adversarial examples is presented as a prerequisite for making NMT systems robust and reliable.The paper frames adversarial examples as revealing NMT vulnerabilities.Source: §1 · p. 1
- Text attacks must preserve meaning while disrupting translation, yet discrete textual inputs make optimization-based methods difficult to apply.The stated goal is a source-language meaning-preserving example that decreases target-language translation quality.Source: §1 · p. 1
- Prior word-replacement attacks use heuristic word selection and substitution, while character-level attacks can be easily detected.The paper also notes that embedding-space similarity in Cheng et al. may not ensure semantic similarity to the original sentence.Source: §1 · pp. 1–2
- The attack optimizes adversarial examples to preserve source meaning while degrading NMT translation, outperforming prior attacks across several reported criteria.The paper reports that for more than 65% of sentences, its white-box attack reduces BLEU by more than half while maintaining semantic similarity.Source: §1 · p. 2
2 Proposed Adversarial Attack
The attack combines a translation-disruption objective with a source-sentence similarity constraint. It relaxes the discrete token choices into probability distributions, enabling differentiable optimization and sampling of multiple adversarial examples.
2.1 Discrete Optimization Problem
- The discrete objective seeks a same-length adversarial sentence that moves the target model’s translation away from the reference while remaining semantically similar to the source.The attack loss maximizes the NMT training loss; the weighted objective combines it with a similarity loss, whose relative importance is controlled by α.Source: §2.1 · pp. 2–3
- The similarity loss uses contextualized language-model embeddings and cosine similarity, with inverse-document-frequency scores weighting token importance.The metric adapts BERTScore [24] to same-length sentences; rare tokens receive greater similarity importance [21].Source: §2.1 · pp. 2–3
2.2 Continuous Relaxation
- The discrete token choices are represented by one-hot vectors, and the continuous relaxation instead optimizes a categorical probability distribution for each token using Gumbel-Softmax sampling.Soft samples form embedding combinations for the models; stochastic optimization samples batches, then multiple adversarial examples can be drawn from the optimized distributions.Source: §2.2 · p. 3
- The relaxed objective is optimized stochastically with Gumbel-Softmax samples, after which repeated sampling can produce candidates for selecting a stronger translation-degrading example.The paper describes the resulting optimization problem as continuous.Source: §2.2 · p. 3
3 Experimental Results
The experiments evaluate the attack against multiple NMT models and datasets, comparing white-box performance with existing attacks and testing a transfer-based black-box extension. The authors report strong translation degradation while preserving source similarity, with competitive results in the black-box setting.
3.1 Setup
- Tests use WMT14 and OPUS datasets with Marian and mBART50 models, evaluating similarity, fluency, translation degradation, attack success, and token error rate against kNN and adapted Seq2Sick.The setup also uses a GPT-2-structure language model trained on WikiText-103 for contextual embeddings.Source: §3.1 · pp. 3–4
3.2 White-box Attack Results
- Across tested NMT models, the attack reduces BLEU by half on average while keeping semantic similarity above 0.85, and outperforms baselines on several metrics.More than 60% of input sentences have at least one successful sample; translation-quality effects exceed baselines in BLEU and chrF, while perplexity is competitive with Seq2Sick.Source: Table 1 · p. 4, §3.2 · p. 4
- For Marian NMT (En-Fr), distribution optimization takes 2.91 seconds and generating 100 samples takes 4.75 seconds, versus 38.85 seconds for Seq2Sick.The measurements use an NVIDIA A100 GPU; kNN takes 1.45 seconds per example but is less effective in success rate.Source: §3.2 · p. 4
- Lower α produces more aggressive adversarial sentences and greater BLEU degradation, while increasing α decreases attack success rate.At α = 45, relative BLEU decrease exceeds 0.3 for more than 90% of generated examples.Source: §3.2 · pp. 4–5, Figure 1 · p. 5, Figure 2 · p. 5
- White-box performance is consistent across the tested OPUS datasets, although OPUS Books has lower semantic similarity and higher perplexity.The authors attribute this difference to the distance between book passages and WikiText-103, which makes the contextual-embedding similarity loss less efficient.Source: Table 3 · p. 5, §3.2 · p. 5
3.3 Extension as a Black-box Attack
- In black-box transfer to mBART50, the attack's performance is almost unchanged from white-box results and it outperforms baselines in success rate, token error rate, and RDBLEU.Semantic similarity and RDchrF are comparable to WSLS; the authors also report stronger meaning preservation with fewer perturbed tokens while highly affecting translation quality.Source: Table 4 · p. 5, §3.3 · pp. 5–6
- The black-box extension optimizes token distributions against a gradient-accessible Marian reference model, then samples and tests them against mBART50.For English-to-French, optimizing takes 2.91 seconds and sampling and testing 100 examples takes 16.56 seconds; WSLS takes 1904.98 seconds and requires many target-model queries.Source: §3.3 · p. 5
4 Conclusion
The paper proposes an optimization-based adversarial attack in the discrete input-token space, with a continuous relaxation that supports differentiable sampling and a black-box extension. Experiments show effectiveness across NMT models and translation tasks, with the attack outperforming other methods on several measures.
4 Conclusion
- The attack outperforms other methods in white-box and black-box settings on success rate, source-language semantic similarity, and token error rate.The experiments also demonstrate effectiveness against different NMT models and translation tasks.Source: §4 · p. 6
- The method formulates an optimization problem over discrete input tokens and relaxes it using per-token probability distributions and the Gumbel-Softmax trick.Source: §4 · p. 6
- Sampling from the optimized probability distributions successfully attacks target NMT models in the black-box setting.Source: §4 · p. 6
- The attack helps expose NMT vulnerabilities and underscores the need for more robust systems in real-time scenarios.Source: §4 · p. 6