PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding Deep Learning System Testing using Surprise Adequacy

Jinhan Kim, Robert Feldt, Shin Yoo

arXiv:1808.08444v1cs.SEcs.NE

TL;DR

Existing neuron-activation coverage criteria are not sufficiently fine grained for subtle DL behaviours, and prior evaluations did not guide testing. SADL measures input surprise relative to training data and uses it to assess and select test inputs. Systematic surprise-based sampling improved adversarial-example retraining accuracy by up to 77.5%.

  • Problem

    Existing neuron-activation coverage criteria are not sufficiently fine grained for subtle DL behaviours, while prior evaluations focused on correlation rather than guiding actual testing.

  • Method

    SADL measures each input’s relative surprise compared with training data and uses Surprise Coverage to systematically diversify test inputs across surprise ranges.

  • Results

    Systematic sampling with broader SA values improved DL-system accuracy after adversarial-example retraining by up to 77.5%.

  • Takeaways & Limitations

    SA can accurately classify adversarial examples, while SC can guide selection of inputs for more effective adversarial-example retraining.

  • Takeaways & Limitations

    External validity is constrained by the number of studied models and input-generation techniques, and SADL may be less effective for other DL systems.

Abstract

from arXiv · show

Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

I. INTRODUCTION

DL systems used in safety- and security-critical domains can exhibit unexpected behaviour, while existing neuron-counting coverage criteria provide limited guidance about individual test inputs. SADL addresses this gap by measuring input surprise relative to training data and using it to guide testing.

  • DL systems are increasingly adopted in safety- and security-critical domains, creating a need to verify their correctness and predictability.
  • Traditional structural coverage is not very useful for DL systems because their behaviour is not explicitly encoded in control-flow structures.
  • Existing neuron-based coverage criteria aggregate activation conditions by counting, providing little information about the relative value of individual inputs.
  • SADL measures how much an input's behaviour differs from training data and supports systematic diversification from familiar to adversarial inputs.
  • 100% and 94.53% ROC-AUC were achieved for adversarial-example classification on MNIST and CIFAR-10, respectively.
  • Up to 77.5% accuracy improvement after retraining was obtained by sampling additional adversarial inputs with broader surprise-adequacy values.

II. SURPRISE ADEQUACY FOR DEEP LEARNING SYSTEMS

The paper motivates measuring DL-test diversity relative to training data and formalizes activation traces as representations of system behaviour. Surprise Adequacy compares new-input traces with training traces, optionally using class-specific training subsets.

  • Existing criteria measure diversity through thresholded or bucketed neuron activations, but these discretisations do not capture continuous differences between inputs.
  • An activation trace is the vector of activation values produced by an input over an ordered subset of neurons.
  • The approach assumes activation traces over all neurons fully capture DL-system behaviour because behaviour follows data-flow rather than control-flow.
  • Surprise Adequacy quantitatively measures a new input's novelty by comparing its activation trace with traces recorded from the training set.
  • For classification, surprise can be measured against activation traces from the subset of training inputs assigned to the new input's predicted class.

B. Likelihood-based Surprise Adequacy

Likelihood-based Surprise Adequacy estimates input rareness from activation-value densities, while Distance-based Surprise Adequacy uses Euclidean distances between activation traces. The latter relates surprise to proximity to class boundaries in the illustrated classification setting.

  • Likelihood-based Surprise Adequacy: Likelihood-based Surprise Adequacy uses KDE to estimate activation-value densities and derives surprise from the rarity of a new input.
  • Likelihood-based Surprise Adequacy: LSA restricts computation to a selected neuron layer and filters low-variance neurons to reduce dimensionality and computational cost.
  • Likelihood-based Surprise Adequacy: LSA is defined as the negative logarithm of estimated density, so lower density produces greater surprise.
  • Likelihood-based Surprise Adequacy: LSA can be computed per class for classifiers, using training inputs assigned to the relevant class as the reference set.
  • Distance-based Surprise Adequacy: In Figure 1, black dots denote training activation traces and grey dots denote new inputs x1 and x2, with x1 shown as more surprising relative to class c1.

C. Distance-based Surprise Adequacy

Distance-based Surprise Adequacy (DSA) measures how far a new input’s activation trace lies from training examples, relative to cross-class distances. This ratio captures surprise at the individual-input level.

  • DSA uses Euclidean distance between a new input’s activation trace and activation traces observed during training.
  • For input x, the reference point xa is the closest training neighbour sharing x’s predicted class cx.
  • The method then finds xb, the closest neighbour of xa from a class other than cx, and computes the corresponding cross-class distance.
  • DSA compares x’s distance to its own-class reference with the known distance between that class and other classes.
  • DSA is calculated as the ratio between dista and distb; more complicated formulations are left for future work.

D. Surprise Coverage

Surprise Coverage (SC) measures how broadly a test set spans surprise values, from training-like inputs to highly different ones. Its usefulness is bounded by domain relevance and a predefined surprise range.

  • SC discretises continuous LSA and DSA values into n buckets spanning (0, U], then counts the buckets represented by an input set.
  • High SC represents inputs ranging from similar to training data at low surprise to very different inputs at high surprise.
  • SC requires a predefined upper bound because arbitrarily surprising inputs may be irrelevant to the problem domain.
  • Unlike structural coverage, SC does not form a combinatorial set-cover problem because each input yields only one surprise value and bucket.
  • Research questions: The evaluation examines whether SC correlates with existing coverage criteria while controlling diversity by cumulatively adding generated inputs.
  • Research questions: Retraining selects four 100-image subsets from overlapping surprise ranges and measures accuracy for MNIST and CIFAR-10 or MSE for Dave-2.

IV. EXPERIMENTAL SETUP

The experimental evaluation tests SADL across original datasets, adversarial examples, and synthetic inputs generated for four deep-learning systems.

  • SADL is evaluated on four deep-learning systems using original test sets, five adversarial attack strategies, and DeepXplore- or DeepTest-generated synthetic inputs.

A. Datasets and DL Systems

The study combines image-classification systems with autonomous-driving models to evaluate SADL across standard and safety-critical settings. Inputs include adversarial examples and synthetic driving scenarios.

  • MNIST uses a five-layer ConvNet trained to 99.31% test accuracy, while CIFAR-10 uses a 12-layer ConvNet trained to 82.27%.
  • The autonomous-driving evaluation uses Udacity camera images, with model performance measured by mean squared error between actual and predicted steering angles.
  • Input generation: Adversarial examples are generated for MNIST and CIFAR-10 using five widely studied attack methods.
  • Input generation: Dave-2 and Chauffeur use DeepXplore and DeepTest to synthesise driving inputs intended to detect erroneous autonomous-driving behaviour.

C. Configurations

The experiments configure SA and coverage-criterion analyses across MNIST, CIFAR-10, and autonomous-driving systems, then evaluate surprise-based input selection and adversarial-example detection. Results indicate that higher-surprise inputs are harder to classify and that SA separates adversarial examples from original test inputs.

  • Configurations: LSA uses activation 2 for MNIST and activation 6 for CIFAR-10 because computing LSA over all neurons is computationally infeasible.The default LSA activation variance threshold is 10^-5, and KDE bandwidth uses Scott’s Rule.
  • Input surprise: Inputs with higher SA are harder to correctly classify, and adversarial examples can be classified based on SA.This conclusion combines the test-input accuracy analysis with adversarial-example detection analyses.
  • Input surprise: Original test inputs tend to have the lowest SA, whereas most adversarial examples are separated by higher SA values.The comparison uses sorted DSA and LSA values for MNIST and CIFAR-10.

B. Impact of Layer Selection (RQ2)

Layer selection affects SA-based adversarial-example classification. DSA generally benefits from deeper layers, whereas LSA shows no clear deepest-layer advantage, and the pattern varies by attack strategy.

  • MNIST: 100% ROC-AUC is explained by clear LSA separation between original test inputs and some MNIST attacks at activation 1.For C&W at activation 3, line crossing produces a low ROC-AUC of 37.96%.
  • CIFAR-10: For CIFAR-10, DSA’s deepest layer produces the most accurate classifier for three of five attack strategies, while BIM-A favors the second deepest layer.The attack strategies named are BIM-B, JSMA, and C&W for the deepest layer.
  • CIFAR-10: LSA has no strong evidence that the deepest layer produces the most accurate classifier for CIFAR-10.This contrasts with the deeper-layer pattern observed for DSA.
  • CIFAR-10: Per-layer DSA produces more accurate CIFAR-10 classification results than all-neuron DSA.The comparison uses the corresponding results from Tables III, IV, and V with identical models.
  • Impact of Layer Selection (RQ2): DSA is sensitive to layer selection and benefits from choosing a deeper layer.The reported layer sensitivity varies across adversarial-example generation strategies.

C. Correlation between SC and Other Criteria (RQ3)

The study compares how coverage criteria respond as input diversity increases. Most criteria rise with added inputs, but NC plateaus and the response depends on the added input type and system.

  • Correlation between SC and Other Criteria (RQ3): DSC cannot be computed for CIFAR-10 and Chauffeur because these systems are not classifiers.The table and figure analysis therefore excludes DSC for those two systems.
  • Correlation between SC and Other Criteria (RQ3): Most studied coverage criteria increase as additional inputs are added, while NC plateaus across many steps.The NC pattern is consistent with existing work cited by the authors.
  • Correlation between SC and Other Criteria (RQ3): SNAC, KMNC, and NBC increase substantially when BIM-B examples are added to CIFAR-10, whereas they change little with C&W inputs.The result demonstrates that criterion responses depend on the type of added inputs.
  • Correlation between SC and Other Criteria (RQ3): For Chauffeur, SNAC and NBC increase similarly with input Set 1, while KMNC increases more steadily.The comparison concerns how criteria respond to increasing input diversity.
  • Correlation between SC and Other Criteria (RQ3): Except for NC, SC is correlated with the other previously introduced coverage criteria.Table VI summarizes changes in coverage values as more inputs are added to the original test set.

D. Retraining Guidance (RQ4)

SA-based sampling guides retraining by selecting inputs from increasingly wider surprise ranges. Across studied configurations, broader ranges generally improve retraining performance, with the largest reported gain reaching 77.5%.

  • Retraining Guidance (RQ4): SA guidance samples additional training inputs from four increasingly wider ranges of surprise.The ranges span progressively broader portions of the SA interval, and retraining configurations evaluate their mean and standard deviation.
  • Retraining Guidance (RQ4): 13 configurations achieve their best retraining performance using the widest SA range, more than any narrower range.The next-best ranges account for 5, 3, and 3 configurations, respectively.
  • Retraining Guidance (RQ4): MNIST retraining against BIM-B using DSA rises from 9.40% to 40.94%, but the authors suspect this result is an outlier.The corresponding accuracies for the narrower ranges are significantly smaller than in other configurations.
  • Retraining Guidance (RQ4): The guidance conclusion is limited to the DL systems and input-generation techniques studied.The authors identify the correctness of the studied implementations and SA computation as threats to internal validity, while model and technique coverage constrain external validity.

VII. RELATED WORK

SADL extends DL testing beyond neuron-count coverage by quantifying input surprise relative to training data. Its SA and SC measures support adversarial-example classification and input selection for retraining.

  • VII. RELATED WORK: Existing DL testing techniques commonly assume that similar inputs should produce similar outputs or that more diverse inputs improve testing effectiveness.Examples include metamorphic testing, disagreement among independently trained systems, and neuron coverage.
  • VII. RELATED WORK: SADL differs from coverage-counting approaches by measuring the amount of surprise instead of merely detecting surprise through increased coverage.This distinction is stated in comparison with DeepCT and related coverage criteria.
  • VII. RELATED WORK: SADL quantifies each input's relative surprise with respect to training data through Surprise Adequacy (SA).It also defines Surprise Coverage (SC) over discretised surprise ranges rather than neuron activation traits.
  • VII. RELATED WORK: SA is correlated with DL-system difficulty on inputs and can accurately classify adversarial examples, while SC guides retraining-input selection.The reported uses include adversarial examples and inputs synthesized by DeepXplore.
Loading 1808.08444v1…