Crafting Adversarial Input Sequences for Recurrent Neural Networks

Nicolas Papernot; Patrick McDaniel; Ananthram Swami; Richard Harang

Source-linked AI summary

Crafting Adversarial Input Sequences for Recurrent Neural Networks

Nicolas Papernot, Patrick McDaniel, Ananthram Swami, Richard Harang

arXiv:1604.08275v1Apr 28, 2016cs.CRcs.LGcs.NE

TL;DR

Neural networks have been shown vulnerable to small adversarial input perturbations, while attacks on RNNs handling sequential data were less developed. This paper formalizes adversarial sequences and adapts forward-derivative methods to recurrent computation, demonstrating misleading categorical and sequential outputs. The authors report 100% wrong-class predictions after changing an average of 9.18 words per movie review, and identify model-architecture and parameter access as a threat-model assumption.

Problem
Prior adversarial-sample research primarily studied classification models, especially computer-vision systems, leaving RNNs with sequential inputs and outputs less studied despite their vulnerability to crafted perturbations.
Method
The paper formalizes adversarial sequences and adapts forward-derivative crafting methods to RNNs by unfolding their cyclical computational graphs.
Results
The experiments produced adversarial inputs that misled both categorical and sequential RNNs, including 100% wrong-class predictions after changing an average of 9.18 words per movie review.
Takeaways & Limitations
Adversarial-sample algorithms developed for feed-forward classifiers can be adapted to manipulate RNN classification predictions and sequence outputs at test time.
Takeaways & Limitations
The threat model assumes adversary access to the RNN architecture and parameters learned during training.

Abstract

from arXiv · show

Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors. Previous efforts have shown that numerous machine learning models were vulnerable to adversarial manipulations of their inputs taking the form of adversarial samples. Such inputs are crafted by adding carefully selected perturbations to legitimate inputs so as to force the machine learning model to misbehave, for instance by outputting a wrong class if the machine learning task of interest is classification. In fact, to the best of our knowledge, all previous work on adversarial samples crafting for neural network considered models used to solve classification tasks, most frequently in computer vision applications. In this paper, we contribute to the field of adversarial machine learning by investigating adversarial input sequences for recurrent neural networks processing sequential data. We show that the classes of algorithms introduced previously to craft adversarial samples misclassified by feed-forward neural networks can be adapted to recurrent neural networks. In a experiment, we show that adversaries can craft adversarial sequences misleading both categorical and sequential recurrent neural networks.

I. INTRODUCTION

Prior work established that neural networks are vulnerable to carefully crafted input perturbations, but adversarial-sequence attacks on RNNs processing sequential data remained insufficiently studied. This paper adapts adversarial-sample crafting methods to RNNs and demonstrates attacks against both categorical and sequential predictions.

Motivation: Small, carefully selected perturbations can substantially alter neural-network outputs while remaining difficult for humans to distinguish.Existing defenses improve robustness, but models remain largely vulnerable.
Research gap: Previous adversarial-sample methods primarily targeted image-classification models with linear, differentiable preprocessing.Sequential models instead may use nonlinear and nondifferentiable preprocessing, such as mappings from words to embeddings.
Research gap: RNNs support sequential applications including sentiment analysis, malware classification, and financial trend prediction.Their relevance motivates studying adversarial manipulation of sequential inputs and outputs.
Contributions: The paper formalizes adversarial sequences and adapts forward-derivative methods to RNNs through computation over cyclical computational graphs.The contribution includes computing forward derivatives for recurrent architectures.
Results: 9 words changed in a 71-word movie review were sufficient on average to cause 100% wrong-class predictions for a categorical RNN.The authors also show Jacobian-crafted sequences can perturb sequential outputs of another RNN.

II. ABOUT RECURRENT NEURAL NETWORKS

The paper introduces machine learning and neural-network concepts before focusing on RNNs, which model sequential data by carrying information through recurrent state updates. A recurrent architecture combines the current input with the previous state to produce each new state and output.

Machine Learning: Machine learning methods analyze large datasets across supervised, unsupervised, and reinforcement-learning tasks.Supervised outputs support tasks such as classification, pattern recognition, and regression.
Neural Networks: Neural networks use neurons that apply activation functions to inputs and pass outputs through interconnected layers.Weights parameterize links between neurons and encode learned model knowledge.
Recurrent Neural Networks: At time t, an RNN state incorporates the current input, the previous state, and a bias before applying an activation function.The state transfers aspects of earlier sequence steps to later steps, and the output is computed from the current hidden state.
Recurrent Neural Networks: RNNs are neural-network variants adapted to modeling sequential data by introducing cycles into the computational graph.These cycles also create parameter sharing across time steps.

III. CRAFTING ADVERSARIAL SEQUENCES

The paper formalizes adversarial sequences and adapts neural-network classifier attacks to recurrent neural networks processing sequential inputs and outputs.

III. CRAFTING ADVERSARIAL SEQUENCES: The approach treats adversarial-sequence crafting as an adaptation of techniques originally designed for neural-network classifiers.The method targets the specific computational structure of recurrent models.

A. Adversarial Samples and Sequences

Adversarial samples minimize input perturbation while changing a classifier’s output; adversarial sequences generalize this objective to make an RNN output a desired target sequence within an acceptable error.

Adversarial Samples: An adversarial sample adds the smallest suitable perturbation to a legitimate input while forcing a different or adversarially chosen class.Exact optimization can be difficult for deep networks because of non-convexity and nonlinearity.
Adversarial Sequences: For sequential outputs, the optimization target is a vector y* that the RNN should match as closely as possible after perturbing its input sequence.The categorical misclassification formulation does not directly apply when outputs are sequences.
Adversarial Sequences: The generalized objective measures input or output discrepancies with an appropriate norm and permits an acceptable error ∆ from the target.For input sequences, one possible norm counts the number of perturbed sequence steps.

B. Using the Fast Gradient Sign Method

The fast gradient sign method linearizes a differentiable model’s cost around its input and selects a perturbation from the input gradient. For recurrent models, it remains applicable when inputs are continuous-valued.

The fast gradient sign method approximates adversarial optimization by linearizing the model’s cost function around its input.It computes gradients with respect to the input rather than model parameters.
The method selects a perturbation using the gradient of the cost function with respect to the input itself.
Increasing ε makes misclassification more likely but also increases perturbation magnitude and distinguishability.
The method applies to recurrent connections when the model is differentiable and its inputs are continuous-valued.

C. Using the Forward Derivative

The forward derivative uses the model Jacobian to select perturbations, and computational graph unfolding adapts this approach to recurrent networks with cycles. The resulting method targets both categorical and sequential outputs.

The forward derivative is the model’s Jacobian and can target a chosen adversarial class or any class different from the original.
The Jacobian quantifies how each input component affects each output component.For J_f[i,j], x_i is an input component and f_j is an output component.
Computational graph unfolding recursively expands recurrent cycles so the RNN graph becomes acyclic and its Jacobian can be computed.
The forward-derivative approach is applied to categorical and sequential RNNs, with binary classification simplifying perturbation selection.
For sequential outputs, perturbing input components according to sgn(Jf[i,j]) × sgn(y*_j) moves output step j toward the desired adversarial component.

IV. EVALUATION

The evaluation crafts adversarial sequences for both categorical and sequential RNNs. It alters words to change sentiment classifications and uses a Jacobian-based attack to manipulate sequential outputs.

The evaluation covers categorical and sequential RNNs using Jacobian-based adversarial sequence attacks.
The categorical RNN performs sentiment analysis on movie reviews, while the sequential RNN maps synthetic input sequences to output sequences.
The sequential attack alters model outputs by identifying the contribution of each input sequence step.

A. Recurrent Neural Networks with Categorical Output

The categorical experiment attacks an LSTM movie-review classifier by replacing words with dictionary entries selected from Jacobian directions. The attack achieves complete training-set misclassification with sparse word changes.

100% training-set error was achieved by changing an average of 9.18 words in each 2,000 movie reviews averaging 71.06 words.
The LSTM architecture contains input, LSTM, mean-pooling, and softmax layers for classifying reviews as negative or positive.
The trained model reached 100% training accuracy and 78.21% testing accuracy before adversarial crafting.
The attack iteratively modifies words until the LSTM’s predicted class changes.
For each word, the Jacobian sign indicates an embedding direction that reduces the probability of the current class.
Because legitimate word embeddings form a finite dictionary, the attack selects the dictionary word whose embedding difference most closely matches the Jacobian direction.

B. Recurrent Neural Networks with Sequential Output

The sequential RNN is trained on synthetic sequence pairs, and its Jacobian is used to craft input changes that target selected output steps. Figure 4 illustrates how sparse input modifications alter corresponding sequential predictions.

Experimental Setup: The experiment trains a sequential RNN on 100 synthetic input-output sequence pairs, with 5 input values and 3 output values per step across 10 steps.Inputs use standard-normal samples, while outputs are generated with correlations to preceding input steps.
Adversarial Sequences: Figure 4 distinguishes legitimate and adversarial inputs from target outputs and model predictions using solid, dashed, and dotted lines.Solid input lines are legitimate, dashed input lines are adversarial, solid output lines are targets, dotted lines are ordinary predictions, and dashed output lines are adversarial predictions.
Adversarial Sequences: The model’s Jacobian identifies which input sequence steps contribute most strongly to selected output sequence steps.Inputs with high Jf[i, j] and low Jf[i, k] for k ≠ j are altered to target output step j.
Adversarial Sequences: The example changes selected input values at steps 4 and 6 to modify output value 0 at step 5 and value 2 at step 8.The passage presents this as a qualitative example; detailed quantitative evaluation was left for future work.

V. DISCUSSION AND RELATED WORK

The discussion places the approach within adversarial learning while emphasizing both its broad theoretical applicability and important threat-model and validation constraints.

Discussion: The theoretical approach applies to neural networks with recurrent components regardless of output data type.Experiments used an LSTM with categorical outputs and a low-dimensional vanilla RNN with sequential outputs as preliminary validation.
Threat Model: The study assumes adversaries can access the model architecture and learned parameter values.The authors note that acquiring such parameter knowledge is not always realistic without access to the hosting system.
Threat Model: Adapting black-box attacks based on substitute models and adversarial transferability to RNNs remains future work.The discussion identifies weaker threat models as a direction for understanding vulnerabilities and defenses.

VI. CONCLUSIONS

The paper concludes that RNNs remain vulnerable to carefully crafted input perturbations and that feed-forward attack techniques can be adapted to sequential settings. Its validation covers both classification and sequence-to-sequence predictions, while broader data types and stronger semantic constraints remain open directions.

VI. CONCLUSIONS: RNNs are vulnerable to adversarially selected input perturbations, extending vulnerabilities previously identified in feed-forward neural networks.The conclusion frames this vulnerability across recurrent models rather than only computer-vision classifiers.
VI. CONCLUSIONS: Computational graph unfolding adapts established adversarial-sample techniques to craft sequential adversarial inputs for RNNs.The paper formalizes the sequential optimization problem and demonstrates the adaptation experimentally.
VI. CONCLUSIONS: The experiments validate adversarial inputs against models making both classification and sequence-to-sequence predictions.The conclusion reports these as the two demonstrated output settings.
VI. CONCLUSIONS: Future work should examine different data types and weaker threat models, while improving adversarial-sequence grammar and human indistinguishability.The authors state that natural-language preprocessing introduced difficulties compared with computer vision.