PaperlayerView exact revision on arXiv

Source-linked AI summary

Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

Kyle D. Julian, Ritchie Lee, Mykel J. Kochenderfer

arXiv:2003.02381v1cs.ROcs.SYeess.SY

TL;DR

Closed-loop image-based neural-network controllers are difficult to verify because individual-image robustness does not establish multi-step safety in complex black-box environments. The paper combines adaptive stress testing with neural-network verification to search for likely failure-inducing image-disturbance sequences. In an aircraft taxiway application, the method found failures and revealed explainable weaknesses involving taxiway imagery and centerline-dash gaps.

  • Problem

    Individual-image robustness does not address multi-step safety for image-based neural-network controllers operating in complex black-box environments.

    Source: §I · p. 1, §VI · p. 6

  • Method

    The method combines adaptive stress testing with neural-network verification to search efficiently for failure-inducing sequences of image disturbances.

    Source: §I · p. 1, A. Adaptive Stress Testing · pp. 2–3

  • Results

    AST found image disturbances that guided the aircraft off the taxiway and revealed that gaps between centerline dashes are more susceptible to adversarial perturbations.

    Source: §VI · p. 6, §V · p. 6

  • Takeaways & Limitations

    Analyzing AST failure sequences can expose explainable neural-network weaknesses and problematic scenarios to address.

    Source: §I · p. 1, §V · p. 6

Abstract

from arXiv · show

Neural networks have become state-of-the-art for computer vision problems because of their ability to efficiently model complex functions from large amounts of data. While neural networks can be shown to perform well empirically for a variety of tasks, their performance is difficult to guarantee. Neural network verification tools have been developed that can certify robustness with respect to a given input image; however, for neural network systems used in closed-loop controllers, robustness with respect to individual images does not address multi-step properties of the neural network controller and its environment. Furthermore, neural network systems interacting in the physical world and using natural images are operating in a black-box environment, making formal verification intractable. This work combines the adaptive stress testing (AST) framework with neural network verification tools to search for the most likely sequence of image disturbances that cause the neural network controlled system to reach a failure. An autonomous aircraft taxi application is presented, and results show that the AST method finds failures with more likely image disturbances than baseline methods. Further analysis of AST results revealed an explainable cause of the failure, giving insight into the problematic scenarios that should be addressed.

I. INTRODUCTION

Image-based neural-network controllers raise safety concerns because local input-output robustness does not establish multi-step closed-loop safety in complex environments. The paper combines adaptive stress testing with neural-network verification to search black-box simulators for failure-inducing image disturbances.

I. INTRODUCTION

  • Neural networks control vehicles and other autonomous systems, making failures such as steering off roads or into other vehicles potentially catastrophic.The paper situates image-based controllers across cars, aircraft, and quadrotors, alongside other image-based decision tasks.Source: §I · p. 1
  • Closed-loop safety cannot be established by verifying individual image input-output properties when environments are complex and image inputs are high-dimensional.Existing multi-step verification approaches require low-dimensional inputs and adequate environmental models, while local robustness remains an input-output property.Source: §I · p. 1
  • The method combines local robustness verification with adaptive stress testing to search efficiently for image-disturbance sequences that cause failure in a black-box simulator.Verification reduces the search space below the full image dimension while avoiding assumptions about the environment.Source: §I · p. 1
  • An aircraft taxiway application demonstrates that the method can find disturbances guiding the aircraft off the taxiway and expose explainable neural-network weaknesses.The application uses the X-Plane 11 photo-realistic flight simulator.Source: §I · p. 1

A. Markov Decision Process

The paper formulates sequential simulator interaction with Markov decision processes and uses Monte Carlo tree search to optimize actions when transitions are difficult to model. MCTS expands and evaluates a search tree through search, expansion, rollout, and value propagation.

A. Markov Decision Process

  • An MDP represents sequential decisions with states, actions, rewards, and transitions, with a policy selected to maximize expected action value.The framework uses the tuple (S, A, R, T).Source: A. Markov Decision Process · pp. 1–2
  • Reinforcement learning is used when the transition function is unknown or difficult to model, as in image-based navigation.It estimates the optimal action-value function through simulation rather than requiring an explicit transition model.Source: A. Markov Decision Process · p. 2

B. Monte Carlo Tree Search

  • MCTS builds a tree from an initial state by selecting actions, expanding a new state, rolling out a default policy, and propagating estimated values upward.The method assumes deterministic transitions, while rollout depth is finite and therefore uses γ = 1.Source: B. Monte Carlo Tree Search · p. 2

C. Deep Q-Learning

Deep Q-learning complements tree search by learning a global functional representation of action values that generalizes across nearby states. The implementation updates a neural network through temporal-difference loss and prioritized experience replay.

C. Deep Q-Learning

  • DQN generalizes action-value estimates across states because one neural network represents Q(s, a) globally.Updating the value at one state also updates values for nearby states, unlike MCTS's state-specific tree estimates.Source: C. Deep Q-Learning · p. 2
  • The DQN parameters are trained by gradient descent on temporal-difference error using prioritized experience replay.The implementation is based on OpenAI Baselines and prioritized replay,.Source: C. Deep Q-Learning · p. 2

D. Neural Network Verification

Neural-network verification characterizes local image robustness, but that property alone cannot determine safe perturbation levels for closed-loop control. Adaptive stress testing instead treats image disturbances as actions and uses model-free reinforcement learning against a black-box simulator to find likely failures.

D. Neural Network Verification

  • Verification tools test neural-network input-output properties over convex input and output regions, returning either a guarantee or a satisfying counterexample.For image inputs, they can assess robustness to noise around a given image.Source: D. Neural Network Verification · p. 2
  • Local robustness limits pixel changes by δ and output changes by ϵ, but robustness around validation images cannot determine what perturbation level is safely tolerable.This limitation motivates combining verification with a sequential failure-search method.Source: D. Neural Network Verification · p. 2

A. Adaptive Stress Testing

  • Adaptive stress testing is model-free reinforcement learning that optimizes simulator-controlled environmental factors to cause a learned agent to fail.For image-based controllers, those actions are disturbances to the controller's input images.Source: A. Adaptive Stress Testing · pp. 2–3
  • AST interacts with a black-box simulator through initialization, stepping, terminal-state checks, and failure-state checks, with rewards encouraging likely failure sequences.This interface avoids requiring an explicit environmental model.Source: A. Adaptive Stress Testing · p. 3

B. Reward Function

AST rewards action sequences that reach a failure while favoring actions with higher likelihood, with penalties discouraging non-failure terminal states.

B. Reward Function

  • AST seeks the most likely action sequence that reaches a failure state from the initial state.The sequence evolves through simulator transitions until the terminal state satisfies the failure condition.Source: B. Reward Function · p. 3
  • The reward function uses log p(a) for nonterminal, nonfailure steps to optimize the likelihood of the selected actions.Source: B. Reward Function · p. 3
  • Large α and β penalties discourage reaching terminal states without failure before action-sequence likelihood is optimized.Dist(s) measures simulator closeness to failure, while α and β scale the terminal penalty.Source: B. Reward Function · p. 3

C. Action Space

The image-based action space is reduced from pixel dimensionality to a one-dimensional disturbance bound, enabling AST to search high-dimensional image disturbances with a likelihood model favoring smaller changes.

C. Action Space

  • High-dimensional image actions make conventional AST inefficient because the action space grows exponentially with image dimensionality.Source: C. Action Space · p. 3
  • Neural network verification computes a disturbance that maximizes output change within a per-pixel δ-neighborhood of the input image.The approach can also compute disturbances that minimize network output.Source: C. Action Space · p. 3
  • Algorithm 1 parallelizes satisfiability queries to find the largest output-change threshold with a corresponding image disturbance.It updates the SAT and UNSAT bounds until the specified tolerance is reached.Source: C. Action Space · p. 3
  • The resulting AST action is represented by the scalar disturbance bound δ rather than the full image disturbance dimensionality.The constraint ∥a∥∞≤δ supports defining action likelihood as a function of δ.Source: C. Action Space · p. 3
  • A zero-mean univariate normal likelihood makes smaller image disturbances more likely than larger ones.This likelihood construction permits AST to operate on image-based control problems.Source: C. Action Space · p. 3

IV. AIRCRAFT TAXI APPLICATION

The aircraft taxi application uses a simulated Cessna 208B controlled from once-per-second runway images, with a neural network estimating cross-track position and heading for taxiway following.

IV. AIRCRAFT TAXI APPLICATION

  • A simulated Cessna 208B taxis at 5 m/s on runway 04 and must remain on the taxiway using right-wing camera images captured once per second.The aircraft is simulated in X-Plane 11 at Grant County International Airport.Source: §IV · p. 4
  • The neural network maps runway images to cross-track position d and heading angle θ, which are used to control the aircraft.The taxiway center and heading are defined as d = 0 m and θ = 0°.Source: §IV · p. 4

A. Neural Network Design

The controller compresses runway images into a small grayscale representation and uses a compact neural network whose outputs are combined into a rudder command.

A. Neural Network Design

  • Downsampling preserves important runway information while reducing network size and potentially improving robustness to pixel perturbations and verification speed.Source: A. Neural Network Design · p. 4
  • Runway images are reduced from 200 × 360 RGB pixels to 8 × 16 grayscale pixels through cropping, resizing, box averaging, and mean-value biasing.Biasing the mean pixel value to 0.5 is intended to improve lighting generalization and constrain adversarial images.Source: A. Neural Network Design · p. 4
  • The compact network has three hidden layers and 32 ReLU activations, using convolutional and fully connected layers to estimate d and θ.Source: A. Neural Network Design · p. 4
  • The estimated cross-track position and heading are combined through proportional control into rudder command r.Because the final network layer and control law are linear, they can be merged into the network's final layer.Source: A. Neural Network Design · p. 4

B. Preliminary Validation

Nominal simulations indicate that the aircraft controller maintains accurate taxiing across varied initial states and cloud conditions, motivating stress testing under image perturbations. The experiments compare AST configurations using Marabou-generated or randomly sampled disturbances, with failure defined by large cross-track error.

B. Preliminary Validation

  • After 20 seconds, the average absolute crosstrack position was 0.287 m, with 0.534 m standard deviation and 1.015 m maximum, while heading stayed under 2° in all 400 simulations.These tests covered four cloud conditions, random initial crosstrack positions and headings, and random simulated times from 9am to 3pm.Source: B. Preliminary Validation · p. 4
  • Nominal success does not establish robustness because real taxiway images can contain skid marks, reflective puddles, and paint spots.The proposed method targets these perturbed-image scenarios to expose network shortcomings and failure modes.Source: B. Preliminary Validation · p. 4

C. Experimental Setup

  • The experiments compare MCTS with Marabou, DQN with Marabou, and MCTS with random samples using discrete disturbances bounded by δmax.Failure states satisfy |d| > 10 m, terminal states exceed 200 m downtrack, and the random baseline selects the perturbation producing the largest network-output change from 5000 samples.Source: C. Experimental Setup · p. 4
  • MCTS search trees were examined at δmax values of 0.02, 0.035, and 0.04 using two actions.The setup also used parallel Marabou queries and pre-computed rollouts with linear interpolation to accelerate MCTS evaluation.Source: Figure 2 · p. 5, C. Experimental Setup · p. 5

V. RESULTS

AST with neural-network verification finds failure-inducing image-disturbance sequences and reveals distinct mechanisms by which the aircraft leaves the taxiway. MCTS, DQN, and Marabou-based experiments show trade-offs between action resolution, generalization, computation, and disturbance selection.

MCTS with Marabou

  • Increasing MCTS action count makes failure sequences longer, with lower log-likelihood and lower δmean, because weak disturbances are also explored.The six-action sequence oscillates the aircraft across the centerline until it gains enough momentum to leave the taxiway.Source: §V · p. 5
  • At δmax = 0.035, AST finds a nonobvious failure by first turning right through reduced network output, then sharply turning left off the taxiway.Always maximizing the network output is insufficient at this disturbance magnitude, whereas the adaptive sequence succeeds.Source: §V · p. 5

DQN

  • DQN performs worse than MCTS with few actions but scales better to many actions and generalizes its learned policy across initial states.MCTS runs required 1–6 hours, whereas DQN training required 24–36 hours.Source: §V · p. 5, Figure 4 · p. 5

Random sampling

  • Random image sampling at δmax = 0.035 never drives crosstrack position beyond 5 m, supporting Marabou’s importance for high-dimensional disturbance search.Both MCTS and DQN with Marabou return disturbance sequences that lead the aircraft off the taxiway.Source: §V · pp. 5–6

Failure analysis

  • AST reconstructions show that darkening taxiway edge lines makes the thin boundaries difficult to discern, identifying those lines as important for accurate predictions.The analysis uses original, downsampled, perturbed, and reconstructed images along the two-action MCTS policy.Source: §V · p. 6, Figure 5 · p. 6
  • Trajectories crossing the centerline between dashes reach the taxiway edge, while other initial positions remain on the taxiway.The corresponding times exhibit larger ϵ values, suggesting that travel between centerline dashes is easier for the network to fool.Source: §V · p. 6, Figure 6 · p. 6

VI. CONCLUSIONS

The paper presents adaptive stress testing for validating image-based neural network controllers in black-box, safety-critical settings. In an aircraft taxiway application, the method finds failure-inducing image disturbances and identifies gaps between centerline dashes as especially vulnerable scenarios.

VI. CONCLUSIONS

  • Adaptive stress testing uses reinforcement learning to find likely failure modes for image-based neural network controllers in settings where formal verification is intractable.The approach is designed for high-dimensional inputs and integrates with a black-box simulator.Source: §VI · p. 6
  • The aircraft taxiway demonstration shows that image disturbances can cause the neural network to guide the aircraft off the runway.Source: §VI · p. 6
  • Gaps between centerline dashes are more susceptible to adversarial perturbations than other tested trajectory conditions.Trajectories crossing the centerline between dashes reach the taxiway edge, while other trajectories remain on the taxiway.Source: §VI · p. 6
  • Future work includes improving robustness, incorporating continuous actions, considering non-pixel perturbations, adding another camera, and validating the simulator against a real aircraft.Source: §VI · p. 6
Loading 2003.02381v1…