PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding Deep Learning System Testing using Surprise Adequacy

Jinhan Kim, Robert Feldt, Shin Yoo

arXiv:1808.08444v1cs.SEcs.NE

TL;DR

Existing neuron-activation coverage criteria are not sufficiently fine grained for subtle DL behaviours, and prior evaluations did not guide testing. SADL measures input surprise relative to training data and uses it to assess and select test inputs. Systematic surprise-based sampling improved adversarial-example retraining accuracy by up to 77.5%.

  • Problem

    Existing neuron-activation coverage criteria are not sufficiently fine grained for subtle DL behaviours, while prior evaluations focused on correlation rather than guiding actual testing.

    Source: §I · pp. 1–2

  • Method

    SADL measures each input’s relative surprise compared with training data and uses Surprise Coverage to systematically diversify test inputs across surprise ranges.

    Source: §I · p. 2, D. Surprise Coverage · p. 4, §VIII · p. 10

  • Results

    Systematic sampling with broader SA values improved DL-system accuracy after adversarial-example retraining by up to 77.5%.

    Source: §I · p. 2

  • Takeaways & Limitations

    SA can accurately classify adversarial examples, while SC can guide selection of inputs for more effective adversarial-example retraining.

    Source: §I · p. 2, §VIII · p. 10

    External validity is constrained by the number of studied models and input-generation techniques, and SADL may be less effective for other DL systems.

    Source: §VI · p. 10

Abstract

from arXiv · show

Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

I. INTRODUCTION

DL systems used in safety- and security-critical domains can exhibit unexpected behaviour, while existing neuron-counting coverage criteria provide limited guidance about individual test inputs. SADL addresses this gap by measuring input surprise relative to training data and using it to guide testing.

  • DL systems are increasingly adopted in safety- and security-critical domains, creating a need to verify their correctness and predictability.Source: §I · p. 1
  • Traditional structural coverage is not very useful for DL systems because their behaviour is not explicitly encoded in control-flow structures.Source: §I · p. 1
  • Existing neuron-based coverage criteria aggregate activation conditions by counting, providing little information about the relative value of individual inputs.Source: §I · pp. 1–2
  • SADL measures how much an input's behaviour differs from training data and supports systematic diversification from familiar to adversarial inputs.Source: §I · p. 2
  • 100% and 94.53% ROC-AUC were achieved for adversarial-example classification on MNIST and CIFAR-10, respectively.Source: §I · p. 2
  • Up to 77.5% accuracy improvement after retraining was obtained by sampling additional adversarial inputs with broader surprise-adequacy values.Source: §I · p. 2

II. SURPRISE ADEQUACY FOR DEEP LEARNING SYSTEMS

The paper motivates measuring DL-test diversity relative to training data and formalizes activation traces as representations of system behaviour. Surprise Adequacy compares new-input traces with training traces, optionally using class-specific training subsets.

  • Existing criteria measure diversity through thresholded or bucketed neuron activations, but these discretisations do not capture continuous differences between inputs.Source: §II · p. 2
  • An activation trace is the vector of activation values produced by an input over an ordered subset of neurons.Source: A. Activation Trace and Surprise… · p. 2
  • The approach assumes activation traces over all neurons fully capture DL-system behaviour because behaviour follows data-flow rather than control-flow.Source: A. Activation Trace and Surprise… · pp. 2–3
  • Surprise Adequacy quantitatively measures a new input's novelty by comparing its activation trace with traces recorded from the training set.Source: A. Activation Trace and Surprise… · p. 3
  • For classification, surprise can be measured against activation traces from the subset of training inputs assigned to the new input's predicted class.Source: A. Activation Trace and Surprise… · p. 3

B. Likelihood-based Surprise Adequacy

Likelihood-based Surprise Adequacy estimates input rareness from activation-value densities, while Distance-based Surprise Adequacy uses Euclidean distances between activation traces. The latter relates surprise to proximity to class boundaries in the illustrated classification setting.

Likelihood-based Surprise Adequacy

Distance-based Surprise Adequacy

C. Distance-based Surprise Adequacy

Distance-based Surprise Adequacy (DSA) measures how far a new input’s activation trace lies from training examples, relative to cross-class distances. This ratio captures surprise at the individual-input level.

D. Surprise Coverage

Surprise Coverage (SC) measures how broadly a test set spans surprise values, from training-like inputs to highly different ones. Its usefulness is bounded by domain relevance and a predefined surprise range.

  • SC discretises continuous LSA and DSA values into n buckets spanning (0, U], then counts the buckets represented by an input set.Source: D. Surprise Coverage · p. 4
  • High SC represents inputs ranging from similar to training data at low surprise to very different inputs at high surprise.Source: D. Surprise Coverage · p. 4
  • SC requires a predefined upper bound because arbitrarily surprising inputs may be irrelevant to the problem domain.Source: D. Surprise Coverage · p. 4
  • Unlike structural coverage, SC does not form a combinatorial set-cover problem because each input yields only one surprise value and bucket.Source: D. Surprise Coverage · p. 4

Research questions

  • The evaluation examines whether SC correlates with existing coverage criteria while controlling diversity by cumulatively adding generated inputs.Source: §III · p. 5
  • Retraining selects four 100-image subsets from overlapping surprise ranges and measures accuracy for MNIST and CIFAR-10 or MSE for Dave-2.Source: §III · p. 5

IV. EXPERIMENTAL SETUP

The experimental evaluation tests SADL across original datasets, adversarial examples, and synthetic inputs generated for four deep-learning systems.

  • SADL is evaluated on four deep-learning systems using original test sets, five adversarial attack strategies, and DeepXplore- or DeepTest-generated synthetic inputs.Source: §IV · p. 5

A. Datasets and DL Systems

The study combines image-classification systems with autonomous-driving models to evaluate SADL across standard and safety-critical settings. Inputs include adversarial examples and synthetic driving scenarios.

  • MNIST uses a five-layer ConvNet trained to 99.31% test accuracy, while CIFAR-10 uses a 12-layer ConvNet trained to 82.27%.Source: A. Datasets and DL Systems · p. 5
  • The autonomous-driving evaluation uses Udacity camera images, with model performance measured by mean squared error between actual and predicted steering angles.Source: A. Datasets and DL Systems · p. 5

Input generation

C. Configurations

The experiments configure SA and coverage-criterion analyses across MNIST, CIFAR-10, and autonomous-driving systems, then evaluate surprise-based input selection and adversarial-example detection. Results indicate that higher-surprise inputs are harder to classify and that SA separates adversarial examples from original test inputs.

Configurations

  • LSA uses activation 2 for MNIST and activation 6 for CIFAR-10 because computing LSA over all neurons is computationally infeasible.The default LSA activation variance threshold is 10^-5, and KDE bandwidth uses Scott’s Rule.Source: C. Configurations · p. 6

Input surprise

  • Inputs with higher SA are harder to correctly classify, and adversarial examples can be classified based on SA.This conclusion combines the test-input accuracy analysis with adversarial-example detection analyses.Source: A. Input Surprise (RQ1) · p. 7
  • Original test inputs tend to have the lowest SA, whereas most adversarial examples are separated by higher SA values.The comparison uses sorted DSA and LSA values for MNIST and CIFAR-10.Source: A. Input Surprise (RQ1) · p. 7

B. Impact of Layer Selection (RQ2)

Layer selection affects SA-based adversarial-example classification. DSA generally benefits from deeper layers, whereas LSA shows no clear deepest-layer advantage, and the pattern varies by attack strategy.

MNIST

  • 100% ROC-AUC is explained by clear LSA separation between original test inputs and some MNIST attacks at activation 1.For C&W at activation 3, line crossing produces a low ROC-AUC of 37.96%.Source: B. Impact of Layer Selection (RQ2) · p. 8

CIFAR-10

  • For CIFAR-10, DSA’s deepest layer produces the most accurate classifier for three of five attack strategies, while BIM-A favors the second deepest layer.The attack strategies named are BIM-B, JSMA, and C&W for the deepest layer.Source: B. Impact of Layer Selection (RQ2) · p. 8
  • LSA has no strong evidence that the deepest layer produces the most accurate classifier for CIFAR-10.This contrasts with the deeper-layer pattern observed for DSA.Source: B. Impact of Layer Selection (RQ2) · p. 8
  • Per-layer DSA produces more accurate CIFAR-10 classification results than all-neuron DSA.The comparison uses the corresponding results from Tables III, IV, and V with identical models.Source: B. Impact of Layer Selection (RQ2) · p. 8

Impact of Layer Selection (RQ2)

  • DSA is sensitive to layer selection and benefits from choosing a deeper layer.The reported layer sensitivity varies across adversarial-example generation strategies.Source: B. Impact of Layer Selection (RQ2) · p. 8

C. Correlation between SC and Other Criteria (RQ3)

The study compares how coverage criteria respond as input diversity increases. Most criteria rise with added inputs, but NC plateaus and the response depends on the added input type and system.

Correlation between SC and Other Criteria (RQ3)

D. Retraining Guidance (RQ4)

SA-based sampling guides retraining by selecting inputs from increasingly wider surprise ranges. Across studied configurations, broader ranges generally improve retraining performance, with the largest reported gain reaching 77.5%.

Retraining Guidance (RQ4)

  • SA guidance samples additional training inputs from four increasingly wider ranges of surprise.The ranges span progressively broader portions of the SA interval, and retraining configurations evaluate their mean and standard deviation.Source: D. Retraining Guidance (RQ4) · p. 9, Table VII · p. 10
  • 13 configurations achieve their best retraining performance using the widest SA range, more than any narrower range.The next-best ranges account for 5, 3, and 3 configurations, respectively.Source: D. Retraining Guidance (RQ4) · p. 9
  • MNIST retraining against BIM-B using DSA rises from 9.40% to 40.94%, but the authors suspect this result is an outlier.The corresponding accuracies for the narrower ranges are significantly smaller than in other configurations.Source: D. Retraining Guidance (RQ4) · p. 9
  • The guidance conclusion is limited to the DL systems and input-generation techniques studied.The authors identify the correctness of the studied implementations and SA computation as threats to internal validity, while model and technique coverage constrain external validity.Source: D. Retraining Guidance (RQ4) · p. 9, §VI · pp. 9–10

VII. RELATED WORK

SADL extends DL testing beyond neuron-count coverage by quantifying input surprise relative to training data. Its SA and SC measures support adversarial-example classification and input selection for retraining.

VII. RELATED WORK

  • Existing DL testing techniques commonly assume that similar inputs should produce similar outputs or that more diverse inputs improve testing effectiveness.Examples include metamorphic testing, disagreement among independently trained systems, and neuron coverage.Source: §VII · p. 10
  • SADL differs from coverage-counting approaches by measuring the amount of surprise instead of merely detecting surprise through increased coverage.This distinction is stated in comparison with DeepCT and related coverage criteria.Source: §VII · p. 10, §VIII · p. 10
  • SADL quantifies each input's relative surprise with respect to training data through Surprise Adequacy (SA).It also defines Surprise Coverage (SC) over discretised surprise ranges rather than neuron activation traits.Source: §VIII · p. 10
  • SA is correlated with DL-system difficulty on inputs and can accurately classify adversarial examples, while SC guides retraining-input selection.The reported uses include adversarial examples and inputs synthesized by DeepXplore.Source: §VIII · p. 10

Related papers

Loading 1808.08444v1…