Source-linked AI summary
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, Cho-Jui Hsieh
TL;DR
The paper asks whether seq2seq models can be effectively attacked despite discrete inputs and almost infinite sequence outputs, and whether they are more robust than CNN classifiers. It proposes Seq2Sick, combining projected gradient methods, group lasso, gradient regularization, and specialized attack losses. Experiments show high-success attacks with small distortions, while seq2seq models remain more robust than CNN-based classifiers.
Problem
Adversarial attacks on seq2seq models are difficult to evaluate because their inputs are discrete text strings and their outputs have almost infinite possibilities.
Method
Seq2Sick combines projected gradient descent, group lasso, gradient regularization, and novel loss functions for non-overlapping and targeted keyword attacks.
Results
Seq2seq attacks achieve high success rates with relatively small distortions, including changing 2 or 3 words on average for non-overlapping attacks and generating different outputs for more than 80% of sentences.
Takeaways & Limitations
Seq2seq models are intrinsically more robust than CNN-based classifiers, though their adversarial examples are more perceptible.
Takeaways & Limitations
Seq2seq adversarial examples generally require larger, more perceptible distortions than image-classifier adversarial examples.
Abstract
from arXiv · showhide
Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input space is continuous and output space is finite. In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities. To address the challenges caused by the discrete input space, we propose a projected gradient method combined with group lasso and gradient regularization. To handle the almost infinite output space, we design some novel loss functions to conduct non-overlapping attack and targeted keyword attack. We apply our algorithm to machine translation and text summarization tasks, and verify the effectiveness of the proposed algorithm: by changing less than 3 words, we can make seq2seq model to produce desired outputs with high success rates. On the other hand, we recognize that, compared with the well-evaluated CNN-based classifiers, seq2seq models are intrinsically more robust to adversarial attacks.
Introduction
The paper examines adversarial attacks on seq2seq models, where discrete inputs and nearly unbounded sequence outputs make attacks harder than image classification. Seq2Sick shows that small input changes can substantially alter outputs, while seq2seq models remain intrinsically more robust than CNN classifiers.
- Motivation: Seq2seq attacks are challenging because inputs are discrete and outputs have an almost infinite number of possible sequences.Word-level changes can alter meaning, requiring sparse modifications, while sequence outputs are far less constrained than finite classification labels.
- Approach: Seq2Sick is an optimization-based framework for learning an input sequence close to the original while changing its output substantially.The framework addresses adversarial example construction for sequence-to-sequence models.
- Approach: The framework combines projected gradient descent, group lasso, and gradient regularization to address discrete inputs and sparse word-level changes.It also uses novel loss functions for non-overlapping and targeted keyword attacks.
- Results: Seq2Sick achieves high success rates in non-overlapping and targeted keyword attacks while changing fewer than 3 words.The paper applies the method to machine translation and text summarization tasks.
- Robustness: Seq2seq models are intrinsically more robust than CNN-based image classifiers because their inputs are discrete and their output space is exponentially large.Adversarial examples for seq2seq models generally have larger and more perceptible distortions.
Related work and Background
Prior text attacks commonly use greedy replacement search, while seq2seq models map input vectors through an encoder-decoder architecture to generate sequence outputs. Seq2Sick instead searches replacement positions jointly and models the encoder context used by the decoder.
- Related Work: Earlier text-adversarial methods targeted classification models using FGSM, reinforcement learning, typos, synonyms, or black-box scoring functions.These methods addressed sentiment or other sequence-classification settings rather than the full seq2seq attack problem.
- Related Work: Most previous methods use greedy search to select the best replacement word and position at each step.The resulting search space grows rapidly as input sequences become longer.
- Seq2Sick: Seq2Sick jointly searches replacement positions using group lasso, projected gradient descent, and gradient regularization.The approach is presented as a response to the rapidly growing search space of greedy methods.
- Seq2seq Background: A seq2seq model maps an input sequence of vectors X = (x1, . . . , xN) to an output sequence Y = {y1, . . . , yM} with an encoder-decoder architecture.The encoder computes hidden states and a context vector from the input sequence; the decoder uses that context and prior outputs to predict the next word.
- Seq2seq Background: The decoder’s output logits represent scores for every possible word in the output vocabulary.This vocabulary-level prediction contributes to the large output space that distinguishes seq2seq models from finite-label classifiers.
Seq2Sick: Proposed Framework
Seq2Sick attacks seq2seq models through constrained optimization that handles discrete inputs and nearly unlimited sequence outputs. It combines projected gradient descent, sparsity and embedding-space regularization with specialized losses for non-overlapping and targeted-keyword attacks.
- Optimization framework: Seq2Sick formulates seq2seq attacks as optimizing attack loss and distortion regularization, with λ balancing attack success against distortion.Smaller λ favors attack success but produces larger distortion.
- Output attacks: Non-overlapping attack requires the adversarial output to share no words with the original output, making it harder than ordinary untargeted attack.Untargeted attack only requires the output to differ from the original.
- Output attacks: Targeted-keyword attack requires every specified keyword to appear in the output, without fixing keyword positions.The loss lets keywords become top-1 predictions at any positions and becomes harder as more keywords are added.
- Output attacks: A mask function resolves keyword collisions by excluding positions already occupied by targeted keywords from placement of remaining keywords.This prevents multiple keywords from competing for the same output position.
- Discrete input handling: Projected gradient descent projects each perturbed embedding back into the input vocabulary, ensuring adversarial inputs map to specific words.This addresses the failure of continuous optimization followed by nearest-neighbor search, which failed all 100 targeted-keyword attacks on Gigaword.
- Discrete input handling: Group lasso enforces word-level sparsity, while gradient regularization keeps perturbed embeddings close to the word-embedding space.The method optimizes group lasso with a proximal operator and computes attack-loss gradients through back-propagation.
Experiments
Experiments evaluate Seq2Sick on text summarization and machine translation using non-overlapping and targeted-keyword attacks. The method achieves high attack effectiveness with sparse input changes while largely preserving semantic properties.
- Experimental setup: Experiments cover text summarization and machine translation, using DUC2003, DUC2004, Gigaword, and WMT’16 Multimodal Translation samples.The summarization and translation models are implemented with OpenNMT-py.
- Text summarization: Non-overlapping attacks change 2 or 3 words on average and produce entirely different outputs for more than 80% of sentences.Success requires no common word at any position between adversarial and original output sequences.
- Text summarization: Changing one word can make the output sequence look completely different and completely change the sentence’s meaning.
- Machine translation: The framework evaluates both non-overlapping and targeted-keyword attacks in English-German machine translation.Reported metrics include success rates, BLEU scores, and average numbers of changed words.
- Text summarization: Targeted-keyword attacks achieve high success rates for 1 or 2 keywords with relatively small input changes, while 3 keywords are more challenging.High BLEU scores and low average numbers of changed words indicate similarity between crafted and original inputs.
- Analysis of syntactic structure and semantic meaning preservation: Only 2.2% of adversarial examples have semantic meanings different from the originals, supporting preservation of semantic classification.The method constrains changed-word counts and encourages changed words to remain close in embedding space.
Analysis and Discussions
Seq2seq models can be attacked successfully, but their discrete inputs and expansive output space make them more robust than CNN-based classifiers.
- Analysis and Discussions: The targeted keyword attack reveals that changed words can co-occur with target keywords in training data, while attention may fail to select the appropriate focus.The authors suggest self-attention as a possible response to this attention-related weakness.
- Analysis and Discussions: 17% lower success rate and 0.13 lower BLEU score occur for targeted keyword attacks on subword transformer models, indicating greater adversarial robustness.This comparison is reported for the subword transformer model relative to the attacked model discussed previously.
- Analysis and Discussions: Seq2seq models achieve 84%–100% success rates in attacks using 1 or 2 keywords, yet fully forcing a particular output sentence remains difficult.Some sentences are impossible for seq2seq models to generate, and input changes remain easier for humans to detect than imperceptible image perturbations.
Conclusion
Seq2Sick generates adversarial examples for sequence-to-sequence models using optimization and sparsity techniques tailored to discrete inputs. The paper demonstrates non-overlapping and targeted-keyword attacks in machine translation and text summarization, including major output changes from minimal input edits.
- Seq2Sick generates adversarial examples for sequence-to-sequence neural network models.
- A projected gradient method addresses discrete inputs, while group lasso enforces sparse distortion and regularization improves attack success.
- Machine translation examples cover non-overlapping attacks and targeted keyword attacks such as “Hund sitzt”.
- Changing only one word can make a text summarization output sequence completely different under the non-overlapping method.
- Text summarization examples also demonstrate targeted keywords such as “police arrest”.