PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing

Vincenzo Riccio, Paolo Tonella

arXiv:2007.02787v1cs.SEcs.AIcs.LG

TL;DR

DL testing must determine whether system misbehaviour intersects the validity domain, especially for critical applications. The paper introduces frontier-of-behaviours analysis and DeepJanus, a model-based search tool, and finds that high-quality systems misbehave mainly on unrealistic or human-challenging inputs, unlike poorly trained systems.

  • Problem

    Testing DL systems is critical because they produce outputs even for inputs outside their validity domains, making the intersection of valid inputs and misbehaviour regions central to quality assessment.

  • Method

    DeepJanus uses model-based multi-objective evolutionary search to generate similar input pairs that cross a DL system’s frontier of behaviours.

  • Results

    High-quality systems have frontiers with smaller validity-domain intersections and, in self-driving tests, larger radii and mostly unrealistic roads, while poorly trained systems include many valid problematic inputs.

  • Takeaways & Limitations

    Frontier inputs provide quantitative and qualitative information for assessing DL quality and identifying valid inputs that systems cannot handle properly.

  • Takeaways & Limitations

    The findings may not generalise to other DL systems because the empirical study used only two subject systems.

Abstract

from arXiv · show

With the increasing adoption of Deep Learning (DL) for critical tasks, such as autonomous driving, the evaluation of the quality of systems that rely on DL has become crucial. Once trained, DL systems produce an output for any arbitrary numeric vector provided as input, regardless of whether it is within or outside the validity domain of the system under test. Hence, the quality of such systems is determined by the intersection between their validity domain and the regions where their outputs exhibit a misbehaviour. In this paper, we introduce the notion of frontier of behaviours, i.e., the inputs at which the DL system starts to misbehave. If the frontier of misbehaviours is outside the validity domain of the system, the quality check is passed. Otherwise, the inputs at the intersection represent quality deficiencies of the system. We developed DeepJanus, a search-based tool that generates frontier inputs for DL systems. The experimental results obtained for the lane keeping component of a self-driving car show that the frontier of a well trained system contains almost exclusively unrealistic roads that violate the best practices of civil engineering, while the frontier of a poorly trained one includes many valid inputs that point to serious deficiencies of the system.

1 INTRODUCTION

Testing DL systems is critical because their behaviour must be assessed within the validity domain of applications such as autonomous driving. The paper introduces frontiers of behaviours and DeepJanus to generate realistic boundary cases and assess whether misbehaviours affect valid inputs.

  • Existing DL test-generation methods manipulate raw inputs or derive inputs from domain models, but model-based methods do not thoroughly characterise misbehaviour regions.
  • The frontier of behaviours consists of similar input pairs that trigger different system behaviours, revealing where expected behaviour begins to fail.A low-quality system may misbehave on inputs close to valid cases, whereas a high-quality system begins misbehaving only outside the validity domain.
  • DeepJanus uses a model-based multi-objective evolutionary search to explore the frontier, minimise within-pair distance, and promote broad frontier coverage.Its outputs can be interpreted against the system’s validity domain and compared using frontier-size metrics.
  • The evaluation covers handwritten-digit classification and self-driving-car steering regression, using human assessment and highway-design conformance to examine frontier realism and quality.
  • DeepJanus generates less concentrated and more realistic corner cases than DLFuzz, whose boundary adversarial inputs are produced by pixel manipulation.

2 BACKGROUND

DL systems combine neural-network software with training data to transform numeric inputs into predictions, and evolutionary search provides a way to optimise candidate tests. These concepts underpin model-based exploration of DL behaviour.

  • 2.1 Deep Learning Systems: A DL system includes one or more DNNs whose behaviour depends on both implementation code and the data used to train their components.
  • 2.1 Deep Learning Systems: DNNs transform numeric inputs into outputs, producing continuous values for regression or discrete classes for classification.
  • 2.1 Deep Learning Systems: DNNs are layered collections of weighted neurons trained iteratively on labelled data, with epochs and learning rate influencing fitting and generalisation.
  • 2.2 Evolutionary Algorithms: Evolutionary algorithms evolve candidate solutions through mutation, crossover, fitness-based selection, and, for multiple objectives, Pareto-front analysis.
  • 2.2 Evolutionary Algorithms: Novelty search rewards diversity to prevent solutions from concentrating in a small region of the input space.

3 MOTIVATING EXAMPLE

The motivating example compares frontiers of high- and low-quality handwritten-digit classifiers using similar inputs that receive different classifications. Low-quality frontiers remain near recognisable valid digits, whereas high-quality frontiers contain cases difficult even for humans.

  • The frontier of a classifier comprises similar digit images that are classified differently, with one classified correctly and the other incorrectly.
  • The example starts from correctly classified MNIST fives and manipulates model control points to generate frontier pairs for low- and high-quality systems.
  • Low-quality frontier inputs remain very similar to original samples and recognisable as fives, while high-quality frontier inputs become ambiguous or unclassifiable to humans.
  • A low-quality system’s frontier lies near inputs it should classify correctly, whereas a high-quality system’s frontier contains cases outside the validity domain.

4 MODEL-BASED INPUT REPRESENTATION

DeepJanus represents inputs with domain-specific models so generated frontier cases remain realistic and satisfy constraints. It uses SVG Bézier curves for digits and spline-based road models rendered in BeamNG.

  • 4 MODEL-BASED INPUT REPRESENTATION: DeepJanus generates realistic frontier inputs from abstract domain models while enforcing domain-specific constraints during concrete rendering.The approach transforms concrete inputs into manipulable models when needed and always transforms models back into concrete test inputs.
  • 4.1 Image Classification: For handwritten digits, SVG Bézier-curve models preserve smoothness and curvature during parameter manipulation before rasterization to 28 x 28 grayscale images.Potrace converts bitmap inputs into smooth vector representations, while LibRsvg and Cairo perform the reverse rasterization.
  • 4.2 Steering Angle Prediction: For steering prediction, a test scenario combines roads, driving-task parameters, weather and lighting, and the car within the BeamNG simulator.The study simplifies scenarios to two-lane asphalt roads with fixed markings and clear weather.
  • 4.2 Steering Angle Prediction: Roads are modeled as interpolated point sequences from Catmull-Rom splines whose center-line control points determine smooth, realistic geometry.BeamNG renders the resulting road points from a JSON scenario description.
  • 4.2 Steering Angle Prediction: The road transformation enforces distinct task endpoints, a fixed square bounding box, and no road self-intersections.

5 THE DEEPJANUS TECHNIQUE

DeepJanus searches for similar input pairs that straddle a DL system’s behavioural frontier while exploring that frontier broadly. Its multi-objective evolutionary search combines frontier proximity, pair similarity, diversity, and archival memory.

  • 5 THE DEEPJANUS TECHNIQUE: DeepJanus finds pairs containing one expected-behaviour input and one similar misbehaving input, placing failures close to the validity domain.This pair-based design targets likely valid corner cases rather than isolated misbehaviour-triggering inputs.
  • 5 THE DEEPJANUS TECHNIQUE: The search maximises closeness to the frontier and diversity across solutions while maintaining similarity within each input pair.Its fitness functions separately measure individual quality and frontier closeness.
  • 5 THE DEEPJANUS TECHNIQUE: DeepJanus extends NSGA-II with novelty-based sparseness and an archive to avoid cycling and promote broad frontier exploration.The implementation is provided in Python using the DEAP evolutionary-computation framework.
  • 5.1 Fitness Functions: The within-pair distance is domain-specific, using Euclidean pixel distance for classification and weighted Levenshtein distance for regression inputs.Pair distance is computed by taking the smaller of the two possible member alignments.
  • 5.1.2 Closeness to the frontier: The frontier-closeness objective evaluates both pair members and assigns positive values to expected behaviour and negative values to misbehaviour.The domain-specific eval function measures output quality during execution, using classification confidence or lane-centering behaviour.

5.2 Initial Population

DeepJanus initializes its population by mutating correctly handled seeds into paired inputs. Seeds come from correctly classified MNIST samples or valid roads on which the car remains in its lane.

  • 5.2 Initial Population: Each initial individual copies one correct seed twice and mutates one copy to create a pair for frontier exploration.
  • 5.2 Initial Population: Classification seeds are correctly recognized MNIST samples, whereas regression seeds are valid roads producing positive steering behaviour.For road scenarios, seeds are retained when the car does not depart from the lane.

5.3 Archive of Solutions

DeepJanus archives nondominated frontier individuals to preserve explored regions and control the granularity of the final frontier. A distance threshold determines whether new frontier pairs are retained.

  • 5.3 Archive of Solutions: The archive stores the best nondominated individuals, preventing cycling and preserving the final explored solution.
  • 5.3 Archive of Solutions: A new frontier pair enters the archive only when its distance from the nearest archived neighbor exceeds the threshold t_a.
  • 5.3 Archive of Solutions: Lowering t_a increases frontier granularity by admitting more similar individuals, while raising it makes archive admission more selective.The tester can adjust t_a to obtain a frontier of the desired size.

5.4 Selection Operator

DeepJanus selects and mutates model-based inputs while preserving domain validity and pairwise distinctness, using repopulation to maintain exploration diversity.

  • 6.1 Selection Operator: NSGA-II tournament selection favors lower non-domination rank and, within a Pareto front, higher crowding distance to promote diversity.
  • 6.1 Selection Operator: Mutations perturb control parameters of one pair member within a configurable range, with domain and distinctness checks repeated until valid.
  • 6.1 Selection Operator: Classification mutations displace SVG points, while regression mutations perturb control points defining road shapes.
  • 6.1 Selection Operator: Repopulation replaces dominated individuals with seed-derived inputs to mitigate local optima and increase population variation.

6 EXPERIMENTAL EVALUATION

The evaluation tests DeepJanus on handwritten-digit classification and self-driving regression, comparing high- and low-quality systems, validity-domain intersections, human judgments, and DLFuzz.

  • 6.1 Subject Systems: DeepJanus evaluates MNIST digit classification and BeamNG steering-angle regression, each with high-quality and low-quality system versions.
  • 6.2 Research Questions: The main evaluation asks whether frontier inputs intersect the system’s validity domain, where intersection indicates issues that may affect real executions.
  • 6.3 Experimental Procedure: Validity is assessed through human digit recognition for MNIST and AASHTO road-design guidelines for BeamNG.
  • 6.2 Research Questions: The study compares high- versus low-quality frontiers using frontier size, human pairwise judgments, and qualitative input assessment.
  • 6.2 Research Questions: DeepJanus is compared with DLFuzz using frontier radius to assess whether it characterises behaviours better than pixel-level boundary generation.
  • 6.3 Experimental Procedure: The procedure generates frontiers, computes radius, compares DLFuzz boundary inputs, and conducts digit-recognition and pairwise image-comparison surveys.

7 RESULTS

DeepJanus identifies frontier inputs that distinguish high- and low-quality systems, expose validity-domain intersections, and explore behavioural boundaries more thoroughly than DLFuzz. Across MNIST and BeamNG, high-quality systems show fewer or more difficult valid frontier inputs, while human judgments and frontier radii discriminate quality.

  • 7.1 RQ1 (Effectiveness): High-quality systems had fewer valid frontier inputs: MNIST HQ had 21 incorrectly recognised images among 90, whereas MNIST LQ had 82 correctly recognised frontier inputs; BeamNG HQ had only one valid frontier road.The MNIST difference was statistically significant, and the HQ BeamNG road exceeded the AASHTO curvature threshold by only 0.371 feet.
  • 7.2 RQ2 (Discrimination): DeepJanus frontier radii significantly distinguished HQ from LQ systems in both MNIST and BeamNG, with larger radii for the higher-quality systems.For MNIST and BeamNG, the LQ radius was significantly smaller, with large or medium effect sizes reported for BeamNG inner and outer radii.
  • 7.2 RQ2 (Discrimination): 95% of MNIST and 89% of BeamNG comparisons judged LQ frontier inputs easier to classify or drive than HQ frontier inputs.The corresponding binomial tests yielded very low p-values, indicating that these choices were unlikely to be random and uniform.
  • 7.3 RQ3 (Comparison): DeepJanus explored significantly larger frontiers than DLFuzz, whose boundary-input radii were smaller for both HQ and LQ systems.DLFuzz did not significantly distinguish HQ from LQ, whereas DeepJanus’s larger radii reflected more thorough exploration.
  • 7.3 RQ3 (Comparison): DeepJanus generated more realistic MNIST images than DLFuzz because it manipulates modelled digit shapes and promotes diversity, whereas DLFuzz applies small pixel perturbations.The qualitative comparison reports that DLFuzz images were more concentrated and less realistic.
  • 7.4 Threats to Validity: The findings may not generalise to other DL systems because the evaluation used only two subject systems, despite covering classification and regression.The authors identify the choice of subject systems as an external-validity threat and call for studies with a wider set of systems.

8 RELATED WORK

Prior DL test-generation methods either perturb raw data or generate scenarios from input models, but they do not fully delimit misbehaviour frontiers. DeepJanus addresses this gap with model-based sampling of realistic inputs and thorough frontier coverage.

  • Existing generators either perturb available data or derive inputs from models, targeting inconsistencies, failures, or proxy criteria such as neuron coverage and surprise adequacy.
  • Raw-data manipulation can produce unrealistic inputs, whereas model-based generation constrains data to the input model’s validity requirements.
  • DeepJanus differs by sampling the entire frontier of misbehaviours rather than merely sampling failures at arbitrary distances from nominal behaviour.

9 CONCLUSIONS AND FUTURE WORK

DeepJanus characterises DL-system quality through frontiers of behaviours and provides quantitative and qualitative evidence for assessing system quality. The authors report greater effectiveness than DLFuzz and plan broader validation across more complex systems.

  • DeepJanus characterises DL-system quality using pairs of similar inputs that trigger different behaviours and are far from each other.
  • Frontier inputs provide quantitative and qualitative information for assessing quality and identifying valid inputs that the system cannot handle properly.
  • DeepJanus is reported to be more effective than DLFuzz for characterising the frontier of behaviours.
  • Future work will evaluate a wider range of DL systems with increasingly complex input domains, including industrial systems.
Loading 2007.02787v1…