Source-linked AI summary
Black-Box Attacks against RNN based Malware Detection Algorithms
Weiwei Hu, Ying Tan
TL;DR
Adversarial attacks have primarily targeted fixed-dimensional models, while the security of sequential RNN-based malware detectors requires attacks that modify API sequences. The paper trains a substitute RNN and uses a generative RNN with Gumbel-Softmax to insert APIs, causing most generated adversarial examples to bypass several black-box victim RNNs.
Problem
Adversarial-example research focused mainly on fixed-dimensional inputs, while sequential API-based malware detectors require attacks that handle discrete, variable-length sequences.
Method
The paper trains a substitute RNN to fit a black-box victim and uses a generative RNN with Gumbel-Softmax to generate and insert APIs into malware sequences.
Results
Most generated adversarial examples bypassed several black-box victim RNNs, with adversarial detection rates ranging from 0.44% to 12.10% versus 90.74% to 93.87% before attack.
Takeaways & Limitations
Sequential RNN-based malware detectors are also vulnerable to adversarial attacks, and the generated examples transfer across different victim models and training sets.
Takeaways & Limitations
The attack assumes malware authors can work without victim structures or weights by approximating the victim with a substitute RNN, while earlier sequential attacks could not insert or delete elements.
Abstract
from arXiv · showhide
Recent researches have shown that machine learning based malware detection algorithms are very vulnerable under the attacks of adversarial examples. These works mainly focused on the detection algorithms which use features with fixed dimension, while some researchers have begun to use recurrent neural networks (RNN) to detect malware based on sequential API features. This paper proposes a novel algorithm to generate sequential adversarial examples, which are used to attack a RNN based malware detection system. It is usually hard for malicious attackers to know the exact structures and weights of the victim RNN. A substitute RNN is trained to approximate the victim RNN. Then we propose a generative RNN to output sequential adversarial examples from the original sequential malware inputs. Experimental results showed that RNN based malware detection algorithms fail to detect most of the generated malicious adversarial examples, which means the proposed model is able to effectively bypass the detection algorithms.
1 Introduction
Adversarial examples expose weaknesses in malware detectors, while RNN-based detectors introduce sequential API inputs that require a different attack strategy. This paper proposes generative and substitute RNNs to insert irrelevant APIs and create sequential adversarial examples.
- Adversarial perturbations can cause machine-learning classifiers to misclassify otherwise correctly classified samples.
- Existing malware detectors commonly represent programs as fixed-dimensional feature vectors, including binary vectors indicating system-API presence.
- RNN malware detectors instead process invoked API sequences to predict whether programs are benign or malicious.
- API sequences contain discrete symbols and variable lengths, making their adversarial-example generation different from image attacks.
- The proposed approach uses a generative RNN to insert irrelevant APIs, while a substitute RNN approximates the victim and Gumbel-Softmax propagates gradients between them.
2 Adversarial Examples
Earlier adversarial-example research largely targeted fixed-dimensional or feed-forward models, leaving sequential modifications insufficiently addressed. Black-box substitute models enabled attacks without direct access to victim internals, but existing RNN adaptations could not truly insert or delete sequence elements.
- Prior adversarial-example methods mainly targeted images and feed-forward neural networks with fixed-dimensional inputs.
- Feature-based Android malware attacks made about 60% to 70% of malware undetected for the three best-performing detection models.
- Existing RNN attack adaptations replaced words but could not insert or delete sequence elements, so their perturbations were not truly sequential.
- Substitute networks allow black-box attacks by fitting a victim model from its outputs on training data.
- Other black-box strategies include greedy local search and ensemble-based generation, with transferability enabling attacks against different models.
- Feature selection, defensive distillation, and retraining have limited effectiveness, especially under repeated attacks.
3 RNN for Malware Detection
RNN malware detection treats API traces as sequential classification inputs and can use multiple sequence representations. The models may process sequences forward or bidirectionally, use pooling or attention, and remain largely opaque to malware authors.
- RNNs classify API sequences as benign or malware, treating malware detection as a sequential classification problem.
- Victim malware-detection RNNs are typically black boxes, so attackers must account for variants such as bidirectionality, pooling, and attention.
- Each API is encoded as an M-dimensional one-hot vector with exactly one active coordinate.
- An API sequence is represented as x1, x2, ..., xT, and the RNN produces corresponding hidden states h1, h2, ..., hT.
- The basic RNN uses the final hidden state hT to compute class probabilities and applies cross entropy for classification.
- Average pooling represents a sequence using the average of hidden states rather than only hT.
- Attention represents a sequence with a weighted average of hidden states, where an attention function assigns importance to each time step.
- Bidirectional RNNs process both the original and reversed sequences, concatenating hidden states from both directions for output prediction.
4 Attacking RNN based Malware Detection Algorithms
The paper attacks RNN malware detectors with sequential adversarial examples generated by a generative RNN that inserts APIs into malware sequences. A substitute RNN approximates the black-box victim and enables gradient-based training through Gumbel-Softmax outputs.
- 4.1 The Generative RNN: The proposed approach generates sequential adversarial examples that insert irrelevant APIs into original malware API sequences.Unlike methods that replace existing sequence elements, the approach mines vulnerabilities in sequential patterns by inserting generated API subsequences.
- 4.1 The Generative RNN: The generative RNN receives a malware API sequence and generates a short API subsequence after each original API for possible insertion.At each time step, decoder outputs of length L are generated from the corresponding hidden state.
- 4.1 The Generative RNN: A special null API controls sequence length by allowing the generator to insert no API at a decoding step, while null outputs are removed from the final one-hot adversarial sequence.The corresponding Gumbel-Softmax null outputs are retained during training to preserve gradient connections.
- 4.1 The Generative RNN: Gumbel-Softmax replaces discrete sampled APIs with differentiable representations so gradients can flow from the substitute RNN to the generative RNN.Directly passing sampled one-hot APIs prevents gradient propagation through the substitute RNN.
- 4.2 The Substitute RNN: The substitute RNN is trained to fit the black-box victim using benign sequences and Gumbel-Softmax malware outputs, with bidirectional recurrence and attention providing strong representation capacity.Victim outputs serve as target labels, while the victim classifies one-hot adversarial examples and the substitute estimates malicious probability on differentiable outputs.
- 4.3 Training: The generative RNN minimizes predicted malicious probability while regularization limits inserted APIs by maximizing the expected probability of the null API.Training alternates substitute-RNN updates on benign and malware-related data with generative-RNN updates on malware data.
5 Experiments
The experiments evaluate victim RNN architectures and show that the proposed attack sharply reduces malware detection, generalizes to unseen malware, and transfers across models and training sets.
- Experimental Setup: The experiments use LSTM-based RNNs trained with Adam on 180 programs, 70% of which are malware, with separate data partitions for attack and victim models.The attack and victim models use different training sets to reflect real-world conditions.
- Victim RNNs: Table 1 compares victim RNN AUC before attacks across LSTM, BiLSTM, average-pooling, and attention-based representations.Attention and pooling represent the whole sequence, whereas the first LSTM models use the last hidden state.
- Victim RNNs: Attention-based representations outperform non-attention approaches, while bidirectionality improves last-hidden-state models but not average-pooling or attention models.The paper attributes the latter pattern to the ability of pooling and attention to capture whole-sequence information and to the greater difficulty of training larger bidirectional models.
- Attack Results: After attacks, victim-RNN detection rates fall from 90.74%–93.87% on original malware to 0.44%–12.10% on adversarial examples.The reported result means about 90% of malware bypasses detection under the proposed attack model.
- Attack Results: Except for LSTM, all victim models have adversarial-example detection rates below 3.03%, while LSTM reaches 12.10% on training data and 11.95% on test data.The paper relates LSTM’s higher rates to a substitute RNN that did not fit that victim model well on the training data.
- Generalization and Transferability: Small training–test differences indicate strong generalization to unseen malware, and adversarial examples transfer across different victim models and training sets.The paper states that this transferability makes attacks easier for malware authors.
6 Conclusions and Future Works
The paper proposes sequential adversarial-example generation for malware and concludes that the method bypasses black-box victim RNNs with different structures. It also identifies robust defenses and cross-architecture transfer as future concerns.
- Conclusions: The proposed sequence-to-sequence generative model uses a substitute RNN and Gumbel-Softmax to generate discrete API perturbations that bypass black-box victim RNNs.Gumbel-Softmax propagates gradients from the substitute RNN to the generative RNN.
- Conclusions: Sequential machine models, including malware detectors, are vulnerable to adversarial attacks, motivating the development of robust defensive models.The paper contrasts this setting with earlier adversarial-example research focused mainly on fixed-dimension images.
- Future Works: Future work will test attacks against CNN-based malware detectors and study transferability between RNN and CNN models.The paper specifically proposes evaluating whether substitute RNNs can fit victim CNNs and vice versa.