Evaluating Explanation Methods for Deep Learning in Security

1. Introduction

• Deep learning is used in security tasks, but opaque decisions make systems difficult to audit and protect from attacks.Although explanation methods have been developed for images and extended to text and sequences, security application has received little attention and only one technique had been investigated there.Source: §1 · p. 1

General evaluation criteria.

• The evaluation compares descriptive accuracy and sparsity with security-focused criteria of completeness, stability, robustness, and efficiency.Sparsity matters because practitioners cannot inspect large feature sets at once; attackers may also use pathological inputs to mislead, corrupt, or slow explanations.Source: General evaluation criteria · p. 1, Security evaluation criteria · p. 1

Security evaluation criteria.

• IG and LRP comply best with the evaluated security criteria, while methods differ in sparsity, ability to handle structured security data, and output stability.The best fit still depends on the particular task.Source: Security evaluation criteria · p. 2
• The qualitative VulDeePecker example shows explanations can differ: one is nuanced, another unsharp, and a third contradicts the first.The figure orders the displays as original code, LRP, LEMNA, and LIME.Source: Security evaluation criteria · p. 2, Figure 1 · p. 2
• Across all four security tasks, explanations revealed features unrelated to security that nevertheless contributed strongly to predictions.The authors argue explanations can help identify such dataset artifacts and keep learning focused on the security task.Source: Security evaluation criteria · p. 2

2.1. Neural Network Architectures

• Security commonly uses MLP, CNN, and RNN architectures, which are the network types the study focuses on.The paper points readers to broader network-architecture references for detail.Source: §2.1 · p. 2, Figure 2 · p. 2

Multilayer Perceptrons (MLPs).

• MLPs are fully connected feedforward networks used in security, but multiple neurons can influence a decision across layers.They have input and output layers at the ends of a sequence of fully connected layers.Source: Multilayer Perceptrons (MLPs) · p. 2

Convolutional Neural Networks (CNNs).

• CNNs use local, overlapping receptive fields to identify spatial structure, but their decisions are difficult to explain because outputs must be unfolded and unpooled.The paper notes their use in detecting malicious patterns in Android application bytecode.Source: Convolutional Neural Networks (CNNs) · p. 2

2.1. Neural Network Architectures

• RNNs process sequences through recurrent connections that support memory, but a feature’s relevance depends on previously processed features.The paper cites security uses including native-code function recognition and software vulnerability discovery.Source: Recurrent Neural Networks (RNNs) · p. 3

2. Explainable Deep Learning

• The paper focuses on explaining individual neural-network predictions because adequately decoding the complete decision process remains challenging.An explanation assigns relevance values to input dimensions, which can be overlaid as a heatmap; blue indicates importance toward a prediction and orange against it.Source: §2.2 · p. 3

2.2. Explanation Strategies

• Black-box explanations approximate a model without its parameters, whereas white-box explanations use known parameters and network structure directly.Black-box methods can omit information available through architecture and parameters; some white-box methods are limited to specific layouts and may not apply to all studied architectures.Source: §2.2 · p. 3

Gradients and IG.

• Gradient explanations use the output’s derivative with respect to each input feature, while Integrated Gradients accumulates gradients along a path from a baseline input.Both methods support the relevant network architectures and are included in the comparison.Source: Gradients and IG · pp. 3–4

LRP and DeepLift.

• LRP propagates relevance backward under a conservation property, while DeepLift uses a reference activation and conserves the output difference across feature relevances.Because DeepLift and IG are closely related, the study evaluates ϵ-LRP instead of DeepLift.Source: LRP and DeepLift · p. 4

LIME and SHAP.

• LIME and SHAP approximate a model’s local behavior using randomly perturbed inputs; LIME fits weighted linear regression, while SHAP uses a kernel yielding Shapley values.Both can be applied to any learning model.Source: LIME and SHAP · p. 4

LEMNA.

• LEMNA is a security-oriented black-box method that approximates a decision boundary with a weighted mixture of linear models.Its mixture components are local linear approximations near the prediction.Source: LEMNA · p. 4

3.2. Security Systems

• The four systems span the three major network types and security tasks, while differing in input scale and representation.Drebin+ uses thousands of dimensions, Mimicus+ uses a small set of engineered features, DAMD learns from large inputs, and VulDeePecker analyzes program code.Source: §3.2 · p. 4, VulDeePecker · p. 5
• Drebin+ and Mimicus+ are MLP malware detectors; Drebin+ analyzes Android application features, while Mimicus+ uses 135 binary PDF features.The Mimicus+ reimplementation comes close to perfect detection on its original dataset.Source: Drebin+ · p. 4, Mimicus+ · pp. 4–5
• DAMD applies a six-layer CNN to raw Dalvik bytecode for Android malware detection and produces results similar to the original publication.Its Malware Genome Project dataset contains 2,123 applications, split 75% for training and 25% for testing.Source: DAMD · p. 5
• VulDeePecker uses an RNN with 300 LSTM cells and word2vec embeddings to identify vulnerabilities in C/C++ code gadgets.The CWE-119 dataset contains 39,757 gadgets, and the reimplementation attains comparable accuracy using an 80%/20% train-test split.Source: VulDeePecker · p. 5

4. Evaluation Criteria

• Across four datasets, explanation methods’ top features differ considerably—indeed, for VulDeePecker all methods select different top-10 features—so the methods cannot be simply interchanged; Figure 3 compares their top-feature overlap.The comparison uses k = 10, except for DAMD, where k = 50 because the dataset contains long opcode sequences.Source: §4 · p. 5, Figure 3 · p. 5

4.1. General Criteria: Descriptive Accuracy

• Descriptive accuracy evaluates explanations by removing their most relevant features and measuring how the original prediction changes.A steeper decline indicates a better explanation because the removed features capture more prediction context.Source: §4.1 · pp. 5–6

4.2. General Criteria: Descriptive Sparsity

• Descriptive sparsity measures whether explanations concentrate relevance on a small, inspectable set of features.For DAMD features above 0.2, LRP returns 14 relevant features while LEMNA returns 2,048; accuracy and sparsity are not correlated and both matter.Source: §4.2 · p. 6

4.3. Security Criteria: Completeness

• Completeness requires non-degenerated explanations for every possible input, since perturbation-based black-box methods may fail to estimate the prediction function.For Drebin+, SHAP returned an all-zero explanation when random perturbations failed to switch a benign sample’s label.Source: §4.3 · p. 7

4.4. Security Criteria: Stability

• Stability requires similar explanations across repeated runs, but random perturbations in black-box methods can produce substantially different outputs.LEMNA assigns very different relevance to some features across two runs on the same Mimicus+ PDF sample.Source: §4.4 · p. 7, Table 6 · p. 7

4.5. Security Criteria: Efficiency

• Efficiency means producing explanations without delaying an expert’s workflow, which depends on the security task.LEMNA takes about one hour on the largest DAMD sample, which has 530,000 features.Source: §4.5 · pp. 7–8

4.6. Security Criteria: Robustness

• Robustness requires that adversarial perturbations cannot decouple an explanation’s relevance vector from the model prediction, though this property remains poorly understood.The paper assesses robustness based on existing literature because guarantees and strong defenses have not been established.Source: §4.6 · p. 8

5.2. Descriptive Accuracy

• IG is best across datasets on descriptive-accuracy AUC, followed by LRP; IG is up to 48% better on average than other methods.Lower AUC indicates better explanations; on VulDeePecker, LIME can provide accuracy comparable to IG and LRP.Source: §5.2 · p. 8, Table 7 · p. 8, Figure 5 · p. 9

5.3. Descriptive Sparsity

• White-box methods are up to 50% sparser; on DAMD, IG marks nearly all features irrelevant while retaining very good accuracy, useful for samples with up to 520,000 features.For VulDeePecker, LEMNA assigns high relevance to many tokens, consistent with its Fused Lasso design.Source: Table 7 · p. 8, §5.3 · p. 9

5.4. Completeness of Explanations

• Only 31% of Drebin+ benign samples reached p=5%, and feature imbalance often leaves benign applications with empty explanations, prompting caution with perturbation-based methods.The authors recommend using these methods only where incomplete explanations can be compensated by other means.Source: §5.4 · pp. 9–10

5.5. Stability of Explanations

• Deterministic Gradients, IG, and LRP score 1.0; perturbation methods never exceed 0.5, leaving half of top features nonoverlapping and trailing by at least 2.5×.The comparison uses top-10 features except for DAMD, where it uses top-50.Source: §5.5 · p. 10, Table 8 · p. 10

5.6. Efficiency of Explanations

• Gradients and LRP are fastest overall via a single backward pass; batching gives more than 16,000× speedup on Mimicus+ over the fastest black-box method, with GPU adding up to 3×.Black-box runtime increases on high-dimensional datasets, and the reported speedups exclude perturbation creation and classification.Source: §5.6 · p. 10, Table 9 · p. 10

5.7. Robustness of Explanations

• Attacks threaten explanation robustness, but access hurdles and discrete inputs, including potentially broken code, make transfer difficult and assessment uncertain.The authors state that adapting attacks to structured domains requires further research.Source: §5.7 · p. 11

5.8. Summary

• Across all six criteria, white-box methods rank above black-box methods; IG and LRP are best overall, with benchmark runtimes under 50 ms and deterministic, complete explanations.When white-box access is unavailable, the authors recommend LIME or model stealing to enable white-box explanations.Source: §5.8 · p. 11, Table 10 · p. 11

5.9. Model Stealing for White-Box Explanations

• Stolen-model LRP explanations remain similar but not globally transferable: Drebin+ original-to-surrogate overlap is about 0.7 versus about 0.85 between surrogates; Mimicus+ overlap is about 0.55.The experiments vary surrogate depth while keeping 200 units per layer.Source: §5.9 · p. 12, Figure 7 · p. 12

6.1. Insights on Mimicus+

• Mimicus+ assigns strong importance to JavaScript, but trailer-section and US-letter-box counts appear as dataset artifacts rather than security-related signals.JavaScript is frequently used in malicious PDFs, whereas count_trailer and count_box_letter are described as difficult to relate to security.Source: §6.1 · p. 12
• In Mimicus+, JavaScript appears in 88% of malicious documents versus about 6% of benign samples, making it extremely discriminating; attackers could potentially evade detection by omitting or obfuscating it.Source: §6.1 · p. 12

6.2. Insights on Drebin+

• In Drebin+, touchscreen, launcher, and INTERNET features together indicate benign applications despite occurring in both benign and malicious samples.For malware, high-scoring SEND_SMS, READ_PHONE_STATE, and getSimCountryIso features align with malicious functionality, including FakeInstaller.Source: §6.2 · pp. 12–13

6.3. Insights on VulDeePecker

• VulDeePecker explanations reflect token-level structure: adjacent tokens often share colors, while uncolored tokens separate positive and negative relevance.This contrasts with interpreting the source code directly, which can create misleading relationships between highlighted tokens.Source: §6.3 · p. 13
• VulDeePecker’s highlighted tokens can mislead analysts because source-level relationships may be absent from the extracted input and essential code is truncated.The authors suggest longer sequences and training-data cleansing to address truncated slices and apparently irrelevant tokens; highlighted punctuation also seems to indicate overfitting.Source: §6.3 · p. 13

6.4. Insights on DAMD

• For DAMD, LRP and IG identify identical opcode sequences across members of each examined malware family, and the sequences connect to malicious functionality.The analysis covers GoldDream, DroidKungFu, and DroidDream; the GoldDream sequence occurs in code that intercepts communications and stores information before exfiltration.Source: §6.4 · p. 13

7. Conclusion

• Integrated Gradients and LRP comply best with the proposed requirements, though the importance of criteria depends on the security task.The authors generally recommend both methods for explaining predictions in security systems.Source: §7 · p. 13
• Across all considered systems, explanations reveal prediction-driving artifacts unrelated to security, which may reflect networks overfitting data peculiarities.The authors conclude explanations should help identify training-data artifacts and keep learning focused on the target security problem.Source: §7 · p. 13
• The authors describe the study as a first step toward explainable security systems and make implementations and datasets publicly available.They intend these resources to support further research extending explanation methods.Source: §7 · p. 13

Adversarial examples

• Adversarial perturbations change a prediction but do not explain why the original input received its label, despite their similarity to gradient-based explanations.The paper says adversarial-example algorithms are insufficient for computing reasonable explanations.Source: Adversarial examples · p. 16, §1 · p. 16

Adversarial examples

• The study does not evaluate adversarial-example attacks or defenses; securely augmenting those techniques with explanations remains an open question.The authors note that recent explanation-based defenses for adversarial examples have proven ineffective.Source: Adversarial examples · p. 16

• Feature selection reduces a learning problem’s dimensionality by selecting a subset of discriminative features.The selected features can often be investigated and may capture characteristics of the underlying data.Source: Feature selection · p. 16
• Selected features may appear suitable for explanations because they can be investigated and often capture characteristics of the data.Source: Feature selection · p. 16
• Because feature selection is independent of a particular learning model, it cannot directly explain a neural network’s decisions.Source: Feature selection · p. 16

• Methods that do not support common neural-network architectures used in security are excluded from the evaluation.The paper refers to Table 1 for the architecture incompatibilities.Source: §2 · p. 16
• PatternNet determines gradients and replaces network weights with informative directions, while PatternAttribution builds on LRP and computes explanations relative to root points with output 0.Source: PatternNet and PatternAttribution · p. 16
• PatternNet and PatternAttribution support feed-forward and convolutional networks, but not recurrent neural networks.Source: PatternNet and PatternAttribution · p. 16

DeConvNet and GuidedBackProp.

• DeConvNet and GuidedBackProp reconstruct an input from an output by reversing convolution, ReLU, and max-pooling computations, but are restricted to convolutional networks.Source: DeConvNet and GuidedBackProp · p. 16

CAM, GradCAM, and GradCAM++.

• CAM, GradCAM, and GradCAM++ use activations from the last convolutional layer with global average pooling; they require a final convolutional layer, uncommon in security applications.GradCAM and GradCAM++ include specific gradients when calculating weights.Source: CAM, GradCAM, and GradCAM++ · p. 16

RTIS and MASK.

• RTIS and MASK optimize a mask to identify sparse relevant features, adding regularization and blurring to reduce noisy results, but these approaches apply only to images.Source: RTIS and MASK · p. 16

Quantitative Input Influence.

• For binary features, Quantitative Input Influence can assign many features zero relevance, which the authors regard as meaningless; its set-based extension does not solve the problem.Source: Quantitative Input Influence · pp. 16–17

3. Completeness of Datasets: Example calculation

• 29% of samples cannot be explained well on average; sufficient opposite-class perturbations are achieved for only 32.6% of benign Drebin+ samples and 2.8% of benign DAMD samples.The table reports samples remaining when enforcing at least 5% opposite-class perturbations.Source: Table 12 · p. 17, §3 · p. 17