Source-linked AI summary
TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
Augustus Odena, Ian Goodfellow
TL;DR
Neural networks are difficult to debug, especially when errors arise only for rare inputs. The paper adapts coverage-guided fuzzing using approximate nearest-neighbor activation coverage and implements it in TensorFuzz. TensorFuzz is applied to numerical errors, quantization disagreements, and undesirable language-model behavior, while the paper notes limitations in existing coverage metrics and an approximate coverage optimization.
Problem
Neural networks are difficult to debug, and numerical issues may be triggered only by rarely encountered inputs.
Method
The paper adapts coverage-guided fuzzing to neural networks, using approximate nearest neighbors over activations as a coverage checker and implementing the approach in TensorFuzz.
Results
TensorFuzz found numerical errors, disagreements between neural networks and quantized versions, and undesirable behavior in character-level language models.
Takeaways & Limitations
TensorFuzz provides an open-source implementation that others can use to build on the approach and find real issues in neural networks.
Takeaways & Limitations
Neuron coverage can be easy to satisfy and specialized to ReLUs, while a distance-sensitive Bloom-filter optimization may miss newly covered inputs and remains future work.
Abstract
from arXiv · showhide
Machine learning models are notoriously difficult to interpret and debug. This is particularly true of neural networks. In this work, we introduce automated software testing techniques for neural networks that are well-suited to discovering errors which occur only for rare inputs. Specifically, we develop coverage-guided fuzzing (CGF) methods for neural networks. In CGF, random mutations of inputs to a neural network are guided by a coverage metric toward the goal of satisfying user-specified constraints. We describe how fast approximate nearest neighbor algorithms can provide this coverage metric. We then discuss the application of CGF to the following goals: finding numerical errors in trained neural networks, generating disagreements between neural networks and quantized versions of those networks, and surfacing undesirable behavior in character level language models. Finally, we release an open source library called TensorFuzz that implements the described techniques.
1 Introduction
Neural networks are difficult to debug despite their growing use in consequential applications. The paper adapts coverage-guided fuzzing to neural networks and applies it through TensorFuzz to several rare or undesirable behaviors.
- Neural networks are increasingly used in medical, transportation, judicial, aviation, and power-grid applications, making reliability essential.
- Machine learning models are difficult to debug and interpret, complicating robust experimental conclusions about their behavior.
- The paper adapts coverage-guided fuzzing from traditional software engineering to test neural networks.
- Fast approximate nearest-neighbor algorithms provide a general way to check neural-network coverage.
- TensorFuzz is an open-source library that applies the proposed techniques to numerical issues, quantization disagreements, and undesirable language-model behavior.
2 Background
This section motivates coverage-guided fuzzing for neural networks by contrasting it with traditional software fuzzing and reviewing existing neural-network coverage metrics. It proposes approximate-nearest-neighbor activation coverage as a simple, cheap metric applicable across architectures.
- Traditional coverage-guided fuzzing: Coverage-guided fuzzing mutates inputs from a maintained corpus and retains mutations that exercise new coverage.Traditional coverage commonly measures newly executed code regions or branches.
- Why neural networks require different coverage: Traditional code and branch coverage often remain unchanged across neural-network inputs, despite meaningful behavioral variation.Neural networks primarily apply matrix multiplications and elementwise operations, whose branching behavior is largely input-independent.
- Existing neural-network testing methods: Existing neural-network testing metrics include neuron, k-multisection, neuron-boundary, and sign-based coverage, alongside metamorphic, black-box, and concolic approaches.These methods target activation behavior, transformed inputs, image-specific operations, or symbolic execution in different ways.
- Limitations of existing metrics: Neuron coverage can be nearly saturated by only 25 randomly selected MNIST test images and is specialized to ReLU networks.Neuron-boundary coverage avoids the ReLU requirement but still treats neurons independently, making full coverage easy to achieve with few examples.
- Activation-based coverage: The proposed metric stores activation sets and uses approximate nearest neighbors to detect whether an input lies beyond a pre-specified activation-space distance.The goal is a coverage metric that is simple, cheap to compute, and applicable to diverse neural-network architectures.
3 The TensorFuzz library
TensorFuzz adapts coverage-guided fuzzing to TensorFlow graphs by using neural-network activations rather than traditional code coverage. Its procedure mutates valid inputs, retains inputs that produce novel coverage, and flags those satisfying an objective.
- Library architecture: TensorFuzz feeds valid inputs to arbitrary TensorFlow graphs and measures coverage from computation-graph activations instead of basic blocks or control flow.The library is designed for diverse neural-network inputs, including correctly shaped images and vocabulary-constrained character sequences.
- Fuzzing loop: The fuzzer repeatedly chooses corpus inputs, mutates them, evaluates coverage and objectives, and retains inputs that add coverage or satisfy the objective.Coverage arrays support novelty checking, while metadata arrays support objective evaluation.
- Input selection: A recency-weighted input chooser favors recently added corpus elements because their advantage for yielding new coverage decays over time.The selection probability is controlled by an exponential function of corpus-element age.
- Mutation: Image mutations add configurable white noise, while text mutations randomly delete, insert, or substitute characters.Image mutations can also constrain the total difference from the source input.
- Coverage analysis: The coverage analyzer uses approximate nearest neighbors to treat sufficiently distant activation vectors as new coverage, avoiding the weak guidance produced by treating every activation vector as novel.The implementation uses FLANN and can track selected activations such as logits or the layer before them.
- Coverage analysis: A distance-sensitive Bloom filter could accelerate novelty checks, but may incorrectly mark genuinely new coverage as not new and remains future work.This optimization is acceptable only when occasional missed coverage is tolerable.
4 Experimental results
The experiments apply TensorFuzz to numerical errors, quantization disagreements, and undesirable language-model behavior. Across these settings, it exposes behaviors that existing data or equal-budget random search did not reveal.
- Numerical errors: TensorFuzz found a non-finite element in every MNIST fuzzing run, whereas random search found none in 100,000-sample trials across 10 initializations.The classifier deliberately used a poorly implemented cross-entropy loss and achieved 98% validation accuracy.
- Quantization disagreements: The 32-bit and 16-bit MNIST models agreed on all 10,000 test images, but TensorFuzz found disagreements near 70% of fuzzed images within an infinity-norm radius of 0.4.Mutations were restricted near seed images to focus on inputs with relatively unambiguous class semantics.
- Quantization disagreements: Random search found no new quantization disagreements beyond those already present when given the same number of mutations as TensorFuzz.This comparison was intended to test whether coverage guidance added value in the quantization setting.
- Language-model behavior: For a character-level LSTM, TensorFuzz and random search both generated repeated words, but TensorFuzz generated six of ten blacklist words versus one for random search in 24 hours.The fuzzing goals were to surface repeated-word behavior and blacklist violations during language-model sampling.
5 Conclusion
The paper presents coverage-guided fuzzing for neural networks, demonstrates TensorFuzz across numerical, quantization, and recurrent-model behaviors, and releases its implementation for reuse.
- Conclusion: The authors introduce neural-network coverage-guided fuzzing, demonstrate TensorFuzz on three classes of behaviors, and release the implementation.The demonstrated applications are numerical errors, disagreements between models and quantized versions, and undesirable RNN behavior.