Source-linked AI summary
Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles
Ziyuan Zhong, Gail Kaiser, Baishakhi Ray
TL;DR
Real-world and simulation testing can miss or poorly handle complex AV corner cases, while unique violation-producing scenarios are sparse. AutoFuzz uses simulator API grammars and neural-network-guided evolutionary search to generate valid scenarios and seek distinct violations. It found more unique violations than the best-performing baseline, and fine-tuning reduced violations for the learning-based controller.
Problem
Testing AV controllers against complex corner cases is difficult, and unique violation-producing inputs are sparse in a large scenario space.
Source: §1 · p. 2Method
AutoFuzz searches simulator API grammars for valid initial scenes and uses learned seed selection and gradient-guided mutation to find distinct traffic violations.
Source: §1 · p. 2Results
AutoFuzz found 10-39% more unique traffic violations per scenario than the best-performing baseline method, while fine-tuning reduced violations 75-100% for the learning-based controller.
Source: §1 · p. 2Takeaways & Limitations
The prototype found hundreds of realistic unique traffic violations resembling complex crashes and other driving offenses.
Source: §10 · p. 13The tested road infrastructure is limited to default simulator maps, mostly modeled on current U.S. roads.
Source: §9 · p. 13
Abstract
from arXiv · showhide
Self-driving cars and trucks, autonomous vehicles (AVs), should not be accepted by regulatory bodies and the public until they have much higher confidence in their safety and reliability -- which can most practically and convincingly be achieved by testing. But existing testing methods are inadequate for checking the end-to-end behaviors of AV controllers against complex, real-world corner cases involving interactions with multiple independent agents such as pedestrians and human-driven vehicles. While test-driving AVs on streets and highways fails to capture many rare events, existing simulation-based testing methods mainly focus on simple scenarios and do not scale well for complex driving situations that require sophisticated awareness of the surroundings. To address these limitations, we propose a new fuzz testing technique, called AutoFuzz, which can leverage widely-used AV simulators' API grammars to generate semantically and temporally valid complex driving scenarios (sequences of scenes). To efficiently search for traffic violations-inducing scenarios in a large search space, we propose a constrained neural network (NN) evolutionary search method to optimize AutoFuzz. Evaluation of our prototype on one state-of-the-art learning-based controller, two rule-based controllers, and one industrial-grade controller in five scenarios shows that AutoFuzz efficiently finds hundreds of traffic violations in high-fidelity simulation environments. For each scenario, AutoFuzz can find on average 10-39% more unique traffic violations than the best-performing baseline method. Further, fine-tuning the learning-based controller with the traffic violations found by AutoFuzz successfully reduced the traffic violations found in the new version of the AV controller software.
1 INTRODUCTION
AutoFuzz targets diverse traffic violations in complex AV scenarios, which are difficult to test in the real world and costly to generate through invalid simulation inputs. It combines simulator-grammar-based scenario generation with learning-guided evolutionary search; its evaluation reports more unique violations than the best-performing baseline and fewer violations after fine-tuning a learning-based controller.
1 INTRODUCTION
- Complex crash scenarios involve interactions among roads, weather, stationary objects, and moving agents, while arbitrary mutations can yield invalid scenes and waste testing effort.The framework also aims to find diverse violations rather than repeated variants of essentially the same situation.Source: §1 · pp. 1–2
- AutoFuzz uses simulator API grammar to generate semantically valid initial scenes, from which the simulator produces sequences of valid scenes.The simulator is treated as a black box during initial-scene generation.Source: §1 · p. 2
- An incremental neural-network classifier ranks candidate seeds by their predicted likelihood of producing a unique traffic violation.The classifier learns from previous test runs and selects the highest-ranked candidates.Source: §1 · p. 2
- Gradient-guided mutations are projected back into a feasible region so they obey grammar constraints while staying close to the proposed mutation.The constraints are expressed as linear equations in a gradient-guided linear regression.Source: §1 · p. 2
- The authors present AutoFuzz as an open-source general framework for comparing AV fuzzing methods across controllers, scenarios, and high-fidelity simulators.They state that the framework lets researchers test systems under user-specified scenarios in popular open-source simulators.Source: §1 · pp. 2–3
- AutoFuzz found 10-39% more unique traffic violations per scenario than the best-performing baseline method.The prototype was evaluated on four AV controllers.Source: §1 · p. 2
- Fine-tuning the learning-based controller with violation-producing test cases reduced traffic violations by 75-100%.Source: §1 · p. 2
2 BACKGROUND
The background distinguishes scenes from their temporal scenarios and reviews the difficulty of testing AV controllers against rare, complex interactions. AutoFuzz is presented as a framework for testing varied controllers and scenarios, illustrated by a pedestrian-related crash sequence.
2.1 Definitions
- A scene is one simulation frame, while a scenario is the temporal development across a sequence of scenes; functional, logical, and specific scenarios describe progressively more concrete forms.A logical scenario bounds the parameter space for fuzzing, and a specific scenario is one concrete instance in that space.Source: §2.1 · p. 3
2.2 Testing Autonomous Vehicle Controllers
- Real-world testing struggles to target rare corner cases, while prior simulation approaches often limit scenario variation, agent interactions, violation types, or system coverage.AutoFuzz is presented as generalized across AV systems and scenarios, with seed-selection and mutation strategies intended to find more unique violations than existing methods.Source: §2.2 · p. 3
2.3 Motivating Example
- In the motivating crash, a leading car brakes for a crossing pedestrian and the turning ego car then collides with it; AutoFuzz searches agent behavior and driving conditions to simulate the sequence.The text also identifies pedestrian occlusion and nighttime heavy rain as challenging specific scenarios.Source: §2.3 · p. 3, Figure 2 · p. 4
3 API GRAMMAR
AutoFuzz derives a test-generation grammar from CARLA’s API specifications. The grammar represents scene components and can include user-defined search ranges and constraints.
3 API GRAMMAR
- The prototype uses CARLA APIs to represent a scene with a route map, the ego car, static and dynamic objects, and background conditions such as weather and road conditions.The API specifications are provided as Python APIs.Source: §3 · p. 4
- AutoFuzz constructs a JSON-encoded test-generation grammar from CARLA’s API specifications, with fields for map, ego car, objects, and background.Users can add search-region restrictions and constraints, such as a minimum distance between the ego car and leading car.Source: §3 · p. 4
- The resulting grammar can include optional search ranges and constraints that bound the fuzzing search.Further details of those ranges and constraints are given in supplementary material.Source: §3 · p. 4
4 METHODOLOGY
AutoFuzz defines diversity in traffic violations, generates grammatically valid inputs, and uses evolutionary search to find violations. Its search combines weighted objectives with learned seed ranking and constrained gradient-guided mutation.
4 METHODOLOGY
- The methodology addresses how to define unique violations, generate semantically valid scenes, and search for more valid unique violations.The prototype fuzzes over background and objects using a simplified description of CARLA’s APIs.Source: Figure 3 · p. 4, §4 · p. 4
4.1 Diverse Traffic Violations
- Two violation-producing scenarios count as unique when at least a configurable percentage of changeable fields differs, limiting distinctions caused by irrelevant input changes.For continuous fields, normalized values must differ by at least a user-defined threshold; the definition supports more exploration than Pareto-front search, including less severe violations.Source: §4.1 · pp. 4–5
4.2 Fuzzing with API Grammar
- Users provide an API grammar, route map, starting scene, and optionally a customized search region and constraints; sampled seeds obey the grammar’s constraints.AutoFuzz iteratively simulates seeds, uses prior-run feedback to generate and mutate new seeds while preserving semantic validity, and repeats until its budget expires.Source: §4.2 · p. 5
4.3 Fuzzing under Evolutionary Framework
- Evolutionary fuzzing searches valid input scenes under a fixed resource budget, selecting and mutating inputs to focus on promising regions rather than exhaustively searching the space.The objective is to maximize the number of unique traffic violations found through simulation.Source: §4.3 · p. 5
4.3.1 Objective Function.
- For collisions and out-of-road violations, AutoFuzz uses weighted smooth objectives to guide searches toward violations, formulated as constrained weighted-sum optimization.Collision objectives promote severity, chance of collision, and visibility; out-of-road objectives include wrong-lane, off-road, and lane-deviation terms.Source: §4.3.1 · p. 6
4.3.2 Seed Selection.
- At each generation, a neural network trained on previously executed seeds ranks candidates by confidence that they will yield a unique violation, and AutoFuzz selects the top configurable number.The seed-selection strategy is intended to avoid spending limited resources on unproductive seeds.Source: §4.3.2 · p. 6
4.3.3 Mutation.
- A constrained gradient-guided perturbation mutates lower-confidence selected seeds toward higher confidence by iteratively back-propagating the input gradient with respect to the network’s prediction.This mutation step is applied after the neural network ranks and selects candidate seeds.Source: §4.3.3 · p. 6, §4.3.2 · p. 6
5 IMPLEMENTATION DETAILS
AutoFuzz combines grammar-constrained sampling and evolutionary search with incremental neural-network seed selection and constrained gradient-guided mutation. The mutation procedure keeps perturbations within input bounds and grammar constraints while seeking test cases likely to produce violations.
Sampling
- Seed cases are sampled randomly or with a genetic algorithm under API-grammar constraints, then filtered for similarity to previously found violations.Variables follow user-specified distributions, search ranges, and constraints; unsatisfied constraints trigger resampling.Source: §5 · pp. 6–7
Evolutionary search
- At each generation, genetic-algorithm parents are selected by binary tournament, then produce offspring through simulated binary crossover and polynomial mutation.Crossover uses η = 5 and probability 0.8; mutation magnitude ηm is set to 5.Source: §5 · p. 7
Seed selection
- An incrementally trained shallow neural network ranks candidate seeds by predicted likelihood of producing a traffic violation, and the highest-ranked are selected.The network is trained on results from previous runs and uses binary classification rather than regression over multiple objectives.Source: §5 · p. 7
Constrained gradient-guided mutation
- For selected cases, gradient-guided mutation iteratively perturbs inputs toward higher predicted violation likelihood, stopping when a confidence threshold is reached or a similar known violation is encountered.Perturbations are clipped to allowable ranges and bounds, then checked against grammar constraints; linear regression projects constraint-violating perturbations back into the constrained region.Source: §5 · pp. 7–8
6 EXPERIMENTAL DESIGN
The experiments evaluate AutoFuzz across five logical scenarios, multiple controllers, and CARLA, with an additional APOLLO6.0 study in SVL. Search quality is measured by unique violations per simulation and compared with evolutionary-search baselines and variants.
Environment and scenarios
- The primary evaluation uses CARLA 0.9.9 and five NHTSA-inspired logical scenarios; a generalizability study uses SVL 2021.3.The first three scenarios cover six frequent pre-crash functional scenarios, while the other two are also common pre-crash scenarios.Source: §6 · p. 8
Controllers
- The tested controllers are two rule-based PID controllers, the end-to-end vision-based lbc controller, and the modular industrial-grade APOLLO6.0 controller.PID controllers assume access to other objects' states and a trajectory to follow.Source: §6 · p. 8
Metrics
- Search quality is measured as unique traffic violations per simulation, with collision-counting rules tailored to each controller's view and the AV-FUZZER comparison including seed-collection simulations.Each simulation lasts until a violation, arrival at the destination, or a 50-second timeout; an average simulation takes about 40 seconds.Source: §6 · p. 8
Baselines and variants
- AutoFuzz is compared with NSGA2-DT, NSGA2-SM, NSGA2-UN-SM-A, and AV-FUZZER using shared objectives and initial-population sampling with uniqueness filtering.NSGA2-DT and NSGA2-SM are multi-objective genetic algorithms; AV-FUZZER is single-objective. The study also compares AutoFuzz design variants.Source: §6 · pp. 8–9
Comparison scope
- FITEST, Asfault, and FusionFuzz are not directly compared because they target integration testing, road-network variation, or sensor fusion rather than black-box end-to-end testing on a predefined map.The study's search mutates scenario elements such as weather, agents, their positions, and behaviors.Source: §6 · p. 9
7 RESULTS
AutoFuzz finds more unique traffic violations than the evaluated baselines, including in a separate APOLLO6.0/SVL study, and its components contribute to search performance. The results also report successful repair of many previously failing scenarios after fine-tuning lbc with violation-derived data.
RQ1: Evaluating Performance
- Across 700 simulations targeting collision traffic violations in the town05 logical scenario, AutoFuzz found 725 unique traffic violations: 575 for lbc, 80 for pid-1, and 70 for pid-2.Source: §7 · p. 9
- In one example, changing a leading vehicle's speed from 3 m/s to 4 m/s or 2 m/s changed whether the ego car collided with a pedestrian hidden behind it.The 3 m/s case collides; the 4 m/s and 2 m/s cases avoid the collision, and the paper notes that braking at first sight of the pedestrian could also prevent the collision.Source: Figure 6 · p. 10, §7 · p. 10
- 10%-39% more unique traffic violations than the best baseline across four logical scenarios; the reported gain is statistically significant (p-value 3.95e−3).The method found 41, 51, 135, and 111 more unique violations than the second-best method in the four scenarios.Source: Figure 7 · p. 10, §7 · p. 10
- GRAD and NSGA2-UN-SM-A each had 100% unique violations, compared with 42%, 22%, and 10% for the other three methods; both include duplicate elimination in their search strategies.The authors report that NSGA2-SM, NSGA2-DT, and AV-FUZZER waste resources by running similar violation-producing scenarios.Source: §7 · p. 10
- 99, 139, and 121 more unique violations than the baselines under three different initial-seed settings.The comparison uses 700 search simulations after initial seed collection.Source: §7 · p. 11
RQ2: Evaluating Design Choices
- Ablations show that gradient-guided mutation and neural-network seed selection each improve unique-violation search; GA-UN also finds slightly more unique violations than GA.A larger perturbation bound (ϵ = 1 rather than 0.3) detects slightly more violations, and GA-UN-NN-GRAD outperforms RANDOM-UN-NN-GRAD.Source: §7 · p. 11
- Under seven of nine uniqueness-threshold settings, GA-UN-NN-GRAD finds at least 10-30% more unique violations than the second-best baseline; none of the methods finds new violations at (10, 75) or (20, 75).The paper attributes the latter result to a uniqueness constraint too stringent for the sampling component to satisfy.Source: Table 4 · p. 11, §7 · p. 11
RQ3: Evaluating Impact on Repair
- After fine-tuning lbc with camera data from 100 training scenarios, the retrained controller succeeds in over 75% of 100 held-out scenarios that had previously failed.The retraining scenarios were selected from collisions found by GA-UN-NN-GRAD across four logical scenarios.Source: Table 5 · p. 11, §7 · p. 11
8 RELATED WORK
The related work discusses grammar-based fuzzing for constrained inputs and simulation-based end-to-end AV testing. It distinguishes AutoFuzz's parameterized scenario search from approaches that do not support variations of constructed scenarios or an easily fuzzed search space.
Grammar-based fuzzing
- Grammar-based fuzzing is used for complex input formats, including APIs and compilers, to obey domain-specific constraints and narrow the search space.The paper contrasts this with fuzzing simpler formats such as images or audio.Source: §8 · p. 12
Scenario specification
- OpenScenario and GeoScenario describe driving scenarios, but the paper says neither provides a parametric search space that can be easily fuzzed.AutoFuzz parameterizes functional scenarios so users can specify parameter ranges, distributions, and constraints for finding violations.Source: §8 · p. 12
9 DISCUSSION & THREATS TO VALIDITY
The discussion limits the realism and scope of the evaluation, while arguing that current-road testing and configurable violation uniqueness remain useful within those boundaries.
Realism
- The simulator bounds realism: violations may reflect simulator-controller interactions, and temporal development is limited to one behavior in CARLA and initial states in SVL.The authors use two simulators and four controllers to mitigate the internal-validity threat.Source: §9 · pp. 12–13
Road Infrastructures
- The study uses default simulator maps mostly modeled on U.S. roads; CAV-supporting infrastructure is left for future work.The authors frame individual-AV testing on current public roads as necessary while CAV-ready public infrastructure remains unavailable.Source: §9 · p. 13
Unique Violations
- Traffic-violation uniqueness is configurable but not based on identifying root causes, which the authors say is infeasible in their black-box setting.Feature interactions and the absence of consensus on whether similar failures share a cause complicate alternative definitions.Source: §9 · p. 13
Policy Implication
- The authors urge public officials to establish comprehensive AV safety testing standards and require acceptable violation levels before public-road deployment.The policy discussion calls for AVs to satisfy safety requirements in the necessary testing process.Source: §9 · p. 13
10 CONCLUSION
AutoFuzz finds realistic unique traffic violations across four controllers and outperforms baseline methods; the conclusion also reports using discovered violations to improve a learning-based controller. The accompanying appendices detail its fuzzing algorithm and provide examples and additional results.
10 CONCLUSION
- AutoFuzz found hundreds of realistic unique violations across four controllers and outperformed baseline methods; the violations were also used to improve a learning-based controller on similar cases.The conclusion describes the violations as resembling complex real-world crashes and other driving offenses.Source: §10 · p. 13
APPENDIX A DETAILS OF AutoFuzz GA-BASED FUZZING
- Algorithm 2 gives the detailed implementation of GA-based AutoFuzz fuzzing, highlighting its neural-network and gradient-based mutation components.The listed algorithm inputs include sampling, evaluation, generation limits, and population size.Source: Appendix A · p. 16, NN-GRAD) · p. 16
NN-GRAD)
- The NN-GRAD procedure ranks candidate offspring by confidence, applies constrained gradient-guided mutation, and selects survivors for the next population.The pseudocode also includes mutation, filtering, and population survival steps.Source: NN-GRAD) · p. 16
APPENDIX B ADDITIONAL TRAFFIC VIOLATIONS EXAMPLES
- Additional examples show LBC collisions with fencing, a truck, and pedestrians, as well as wrong-lane driving across several towns.The examples include a cyclist collision in town07.Source: Figure 13 · p. 16, Appendix B · p. 16
APPENDIX C ADDITIONAL FIGURES
- Figure 14 reports the unique collision and out-of-road violations found by AutoFuzz in town05 over 700 simulations for each controller.The passage describes the figure's scope but does not provide its plotted values.Source: Appendix C · p. 16
APPENDIX D DETAILS ON PARAMETERIZING THE FUZZABLE FIELDS
AutoFuzz constrains fuzzable fields to reduce unrealistic or excessive search, and defines distance-based objectives for wrong-lane and off-road behavior.
APPENDIX D DETAILS ON PARAMETERIZING THE FUZZABLE FIELDS
- Fuzzable fields can be bounded by minimum and maximum values, with optional clipped distributions and map-waypoint updates for sampled locations.The example uses a bounded normal distribution for a pedestrian location.Source: Appendix D · p. 16, Figure 14 · p. 17
- User-specified inequalities further constrain field combinations, using coefficients, selected field labels, and a right-hand-side value.The constraint is encoded as coefficient[1]×label[1]−coefficient[2]×label[2] ≤ value.Source: Appendix D · p. 17
APPENDIX E DETAILS AND ILLUSTRATION OF OBJECTIVES FOR OUT-OF-ROAD VIOLATONS
- The out-of-road objectives combine lane deviation angle and distance, and measure wrong-lane or off-road status by distance to the relevant road region.Fdeviation = Θ × ddev; Fwronglane uses the minimum of distances in several directions to an oppositely directed road.Source: Appendix E · p. 17, Figure 15 · p. 17
APPENDIX F DETAILED RESULTS ON PERCENTAGE OF TRAFFIC
The appendix identifies a uniqueness-percentage table and describes a limitation in the SVL APOLLO6.0 configuration.
VIOLATIONS BEING UNIQUE
- Table 6 reports the percentage of found traffic violations that are unique, but the supplied passage gives no values.Source: Table 6 · p. 17
APPENDIX G MORE DETAILS ON APOLLO6.0 IN SVL
- The SVL APOLLO6.0 setup tests all modules except the traffic-light module, whose activation fails; ground-truth traffic-light information is used instead.The perception and camera modules were activated in the authors' modified version.Source: Appendix G · p. 17
A LOGICAL SCENARIO EXAMPLE
The “Turning right while leading car slows down/stops” scenario defines a 26-dimensional search space while keeping the leading vehicle and pedestrian present. Its parameters specify how those agents move and how their locations relate to the ego car’s route.
A LOGICAL SCENARIO EXAMPLE
- The scenario’s 26-dimensional search space varies weather and lighting and agent attributes, while always including a leading vehicle and pedestrian and excluding influential static objects.The number of pedestrians, vehicles, and static objects is not searched for this scenario.Source: A LOGICAL SCENARIO EXAMPLE · p. 17, Table 7 · p. 18
- Pedestrians are parameterized by type, initial position and orientation, trigger distance, target speed, and travel distance.A pedestrian starts moving when the ego car comes within the trigger distance.Source: A LOGICAL SCENARIO EXAMPLE · pp. 17–18
- NPC vehicles have pedestrian-like movement parameters plus color, waypoint-following, and collision-avoidance settings.Waypoint following directs a vehicle to a target location; collision avoidance makes it stop when a collision is likely given current velocities.Source: A LOGICAL SCENARIO EXAMPLE · p. 18
- The ego car’s start and end locations are hard-coded, and agent locations use coordinates relative to the route midpoint.The route midpoint between the ego car’s start and end locations serves as the coordinate center for the pedestrian and vehicle.Source: A LOGICAL SCENARIO EXAMPLE · p. 18