Source-linked AI summary
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana
TL;DR
DL systems require systematic testing for rare corner cases, but existing approaches depend heavily on manual labels and leave much of the learned logic unexplored. DeepXplore combines neuron coverage, differential testing across similar models, and gradient-based joint optimization to generate tests. It found thousands of incorrect behaviors across 15 models and five datasets, while its tests supported up to 3% accuracy improvement through retraining.
Problem
Existing DL testing relies heavily on manual labels, while random and adversarial tests may leave rare learned behaviors and corner cases unexplored.
Method
DeepXplore introduces neuron coverage, uses similar DL systems as cross-referencing oracles, and solves a joint coverage-and-differential-behavior objective with gradient-based search.
Results
DeepXplore found thousands of incorrect corner-case behaviors in 15 state-of-the-art DNNs trained on five real-world datasets, and retraining on its tests improved classification accuracy by up to 3%.
Takeaways & Limitations
The generated tests provide a way to expose erroneous DL behavior without manual labels and can support retraining and identification of potentially polluted training data.
Takeaways & Limitations
Unusual erroneous behaviors can remain unexplored when tests do not activate unlikely combinations of neurons, and labeling complex real-world inputs is expensive.
Abstract
from arXiv · showhide
Deep learning (DL) systems are increasingly deployed in safety- and security-critical domains including self-driving cars and malware detection, where the correctness and predictability of a system's behavior for corner case inputs are of great importance. Existing DL testing depends heavily on manually labeled data and therefore often fails to expose erroneous behaviors for rare inputs. We design, implement, and evaluate DeepXplore, the first whitebox framework for systematically testing real-world DL systems. First, we introduce neuron coverage for systematically measuring the parts of a DL system exercised by test inputs. Next, we leverage multiple DL systems with similar functionality as cross-referencing oracles to avoid manual checking. Finally, we demonstrate how finding inputs for DL systems that both trigger many differential behaviors and achieve high neuron coverage can be represented as a joint optimization problem and solved efficiently using gradient-based search techniques. DeepXplore efficiently finds thousands of incorrect corner case behaviors (e.g., self-driving cars crashing into guard rails and malware masquerading as benign software) in state-of-the-art DL models with thousands of neurons trained on five popular datasets including ImageNet and Udacity self-driving challenge data. For all tested DL models, on average, DeepXplore generated one test input demonstrating incorrect behavior within one second while running only on a commodity laptop. We further show that the test inputs generated by DeepXplore can also be used to retrain the corresponding DL model to improve the model's accuracy by up to 3%.
1 INTRODUCTION
DeepXplore addresses the difficulty of systematically testing safety- and security-critical DL systems, whose large input spaces and reliance on manual labels leave rare erroneous behaviors underexplored. It combines neuron coverage, differential testing, and gradient-based joint optimization to find and help repair incorrect corner cases.
- Motivation: DL systems deployed in safety- and security-critical domains can exhibit unexpected corner-case behaviors with serious consequences.Examples include self-driving cars and malware detection systems.
- Limitations of Existing Testing: Manual labels, unguided simulation, and tiny adversarial perturbations cover only a small fraction of large real-world DL input spaces.These approaches therefore struggle to expose rare erroneous behaviors without substantial human inspection.
- DeepXplore Approach: Neuron coverage measures the parts of DL logic exercised by test inputs through the neurons activated above a threshold.For most tested systems, one random input achieved 100% code coverage but less than 10% neuron coverage.
- DeepXplore Approach: DeepXplore uses similar DL systems as cross-referencing oracles to identify differential behaviors without manual checking.Disagreement among systems performing the same task flags potentially erroneous corner cases.
- DeepXplore Approach: DeepXplore formulates maximizing neuron coverage and differential behaviors as a joint optimization problem solved with gradients calculated through whitebox model access.User-provided constraints can simulate realistic inputs such as lighting and occlusion.
- Results: Thousands of incorrect corner-case behaviors were found across 15 state-of-the-art models trained on five real-world datasets.The generated tests also produced 34.4% and 33.2% higher average neuron coverage than random and adversarial inputs, respectively, and enabled up to 3% accuracy improvement through retraining.
2 BACKGROUND
DL systems learn rules from data rather than having their logic directly specified, making their internal behavior difficult to inspect and test. Low-coverage random testing and expensive labeling leave unusual neuron combinations and corresponding erroneous behaviors unexplored.
- 2.1 DL Systems: DL systems may combine DNN components with traditional software, while DNN rules are learned indirectly through data, features, and architecture.This makes much of the learned logic unknown even to developers.
- 2.2 Deep Neural Networks: DNNs consist of layered neurons that transform inputs into progressively higher-level representations for final predictions.Connections carry learned weights, and common activation functions include sigmoid, hyperbolic tangent, and ReLU.
- 2.3 Limitations of Existing DNN Testing: Existing DNN testing requires expensive human effort to provide correct labels or actions for complex, high-dimensional real-world inputs.Determining whether executable files are malicious may require sandbox execution and monitoring, which adds performance overhead.
- 2.3 Limitations of Existing DNN Testing: Random train-test splits may exercise only a small subset of the rules learned by a DNN.Adversarial testing also focuses on tiny perturbations and does not maximize coverage.
- 2.3 Limitations of Existing DNN Testing: Random inputs rarely activate unlikely combinations of neurons, leaving many incorrect DNN behaviors unexplored.A rare combination such as high activation for “Nose” and “Red” can produce a misclassification that regular testing misses.
3 OVERVIEW
DeepXplore generates unlabeled test inputs that jointly increase neuron coverage and induce disagreement among similar DNNs. It uses gradient-guided search from seed inputs while enforcing constraints so the resulting tests remain realistic.
- Workflow: DeepXplore starts from unlabeled seed inputs and jointly maximizes neuron coverage and differential behavior across tested DNNs.The framework activates neurons above a customizable threshold while making similar models behave differently.
- Workflow: Domain-specific constraints keep generated inputs valid and realistic, such as restricting image pixels to the range 0–255.Users can specify constraints during optimization, including limits on pixel values.
- Differential behavior: The resulting tests expose differential outputs, such as one DNN classifying an input as a car while another classifies it as a face.These inputs are constructed to reveal disagreement between models intended to perform the same task.
- Gradient-guided search: DeepXplore uses gradient ascent to modify each seed input toward regions where similar DNNs cross different decision boundaries.The search direction is gradient-guided, although it does not guarantee the fastest convergence.
4 METHODOLOGY
The methodology defines neuron coverage and input gradients, then formulates test generation as a joint optimization over differential behavior and coverage. Gradient ascent modifies inputs while constraint handling preserves domain requirements.
- 4.1 Definitions: Neuron coverage is the ratio of uniquely activated neurons across test inputs to the DNN’s total neurons.A neuron counts as activated when its output exceeds a threshold, such as 0.
- 4.1 Definitions: DeepXplore computes gradients of neuron outputs with respect to the input using the chain rule, with gradient dimensionality matching the input.The model parameters remain fixed while the input is modified during testing.
- 4.2 DeepXplore algorithm: The algorithm maximizes differential behavior by changing a seed input so at least one similar DNN predicts a different class.It randomly selects one network and balances its divergent prediction against the others’ original class outputs.
- 4.2 DeepXplore algorithm: The joint objective combines differential behavior with activation of an inactivated neuron, weighted by λ2, and is optimized through gradient ascent.Algorithm 1 repeatedly updates the input, checks for disagreement, records successful tests, and tracks activated neurons.
- 4.2 DeepXplore algorithm: A rule-based constraint handler modifies each gradient update so the generated input continues satisfying user-specified domain constraints.Most constraints were not easily embedded directly through Lagrange multipliers; discrete features may be rounded to integers.
5 IMPLEMENTATION
DeepXplore is implemented in Python on TensorFlow and Keras, using TensorFlow’s gradient computation and sub-DNN support without modifying either framework.
- Implementation: DeepXplore consists of around 7,086 lines of Python code built on TensorFlow 1.0.1 and Keras 2.0.3.The implementation does not require modifications to the underlying frameworks.
- Implementation: The implementation leverages TensorFlow for efficient gradients and for treating arbitrary neuron outputs as sub-DNN outputs.This supports the joint optimization process and access to intermediate neuron values.
6 EXPERIMENTAL SETUP
DeepXplore is evaluated across five public datasets and fifteen DNNs, while domain-specific constraints keep generated inputs realistic for images, PDFs, and Android applications.
- Test datasets and DNNs: Five public datasets—MNIST, ImageNet, Driving, Contagio/VirusTotal, and Drebin—provide varied image, driving, PDF-malware, and Android-malware inputs.
- Test datasets and DNNs: Fifteen DNNs are evaluated, using three different architectures for each dataset.The models are either pretrained or trained using public real-world architectures with comparable state-of-the-art performance for their datasets.
- Domain-specific constraints: Generated tests are constrained to remain valid and realistic, including physically producible images and specification-compliant PDFs.
- Domain-specific constraints: Image brightness constraints uniformly increase or decrease all pixel values according to the mean gradient during gradient ascent.Figure 8 includes examples of difference-inducing inputs generated under these constraints.
- Domain-specific constraints: Occlusion constraints apply gradient-guided modifications within a freely positioned rectangular image region.
- Domain-specific constraints: Android-manifest constraints preserve application code by allowing only feature additions, while Contagio/VirusTotal modifications follow feature-specific restrictions.
7 RESULTS
DeepXplore’s evaluation shows that neuron coverage improves the comprehensiveness and diversity of generated tests while enabling efficient discovery of differential behaviors across diverse DNN tasks. The results also identify practical limitations when models have very similar decision boundaries.
- Evaluation setup: DeepXplore generated difference-inducing inputs across vision and malware tasks, using domain-specific constraints for realistic perturbations.The experiments covered MNIST, ImageNet, driving data, Android malware, and PDF malware.
- Benefits of neuron coverage: 100% code coverage could coexist with neuron coverage below 34%, showing that code coverage poorly reflects the DNN logic exercised by test inputs.Ten randomly selected inputs achieved 100% code coverage for all tested DNNs, while neuron coverage never exceeded 34%.
- Benefits of neuron coverage: Neuron coverage increased generated-test diversity, even when its absolute increase was only 1–2 percentage points.Diversity was measured using average L1 distance from the original seed, and higher neuron coverage redirected optimization toward diverse differences rather than repeated root causes.
- Performance: 34.4% and 33.2% more neurons were covered by DeepXplore than by random and adversarial testing, respectively.Higher activation thresholds reduced coverage for all approaches.
- Performance: DeepXplore generally found differential behaviors efficiently, but required more iterations as DNN decision boundaries became increasingly similar.The evaluation reports only one failure among models with minute variations, while very similar decision boundaries could prevent finding an input within a reasonable time.
- Improving DNNs with DeepXplore: DeepXplore-generated inputs correctly identified 95.6% of polluted training samples in the evaluated LeNet-5 setting.The procedure searched for structurally similar training samples near generated inputs and treated those samples as polluted data.
8 DISCUSSION
DeepXplore’s differential testing depends on comparing multiple DNNs with the same functionality, and its effectiveness varies with how different their decision boundaries are. The approach is generally practical because independently trained models are often available, but it cannot detect errors shared by all tested models.
- Differential testing requires at least two DNNs with the same functionality, and finding inputs takes longer when their models differ only slightly.The tested models’ decision boundaries reflect differences in training data, architecture, and hyperparameters.
- Most cases provide multiple suitable DNNs because developers independently train customized models for improved accuracy.
- DeepXplore cannot detect an erroneous behavior when all tested DNNs make the same mistake.The authors report this is usually uncommon because the models are independently constructed and trained.
9 RELATED WORK
Related approaches expose specific DNN failures or seek formal safety guarantees, whereas DeepXplore combines differential testing, neuron coverage, and gradient-based search to explore broader erroneous behaviors in large models.
- Adversarial-image methods expose errors using minute, visually imperceptible perturbations but have low neuron coverage and require manual inspection for visible changes.
- Formal DNN-verification techniques do not scale well to interesting safety-property violations in real-world models, while DeepXplore does not guarantee satisfaction of a specified safety property.
- DeepXplore formulates difference-inducing input generation and neuron-coverage maximization as a joint optimization problem solved efficiently with gradient ascent.
10 CONCLUSION
DeepXplore is a whitebox framework that combines neuron coverage, differential testing, and gradient ascent to identify erroneous behaviors without manual labels. It found thousands of such behaviors across fifteen state-of-the-art DNNs and five real-world datasets.
- DeepXplore automatically identifies erroneous behaviors without manual labels while measuring exercised DNN rules with neuron coverage.
- DeepXplore uses gradient ascent to maximize neuron coverage and potentially erroneous behaviors jointly.
- Thousands of erroneous behaviors were found in fifteen state-of-the-art DNNs trained on five real-world datasets.