PaperlayerView exact revision on arXiv

Source-linked AI summary

FaultGene: LLM-Assisted Fault-Augmented Test Generation for Hardware-in-the-Loop Validation of Autonomous Driving Systems

Chaoran Yuan, Fadi Kurdahi

arXiv:2610.29969v1cs.SEcs.AI

TL;DR

Preparing fault-augmented scenarios for HIL validation is labor-intensive, while simulation alone cannot fully capture hardware-dependent effects. FaultGene generates tests from baseline scenarios and structured fault specifications, using offline HIL feedback for refinement and fine-tuning. On held-out combinations, it improves semantic alignment, compilation, and execution success over one-shot generation while preserving main HIL fault-sensitivity trends.

  • Problem

    Preparing fault-augmented scenarios for HIL testing remains largely manual, while simulation cannot fully capture hardware-dependent behavior and fault propagation.

    Source: §I · p. 1

  • Method

    FaultGene generates fault-augmented tests from baseline Scenic/CARLA scenarios and structured fault specifications, using offline execution feedback and semantic checks to refine and fine-tune the generator.

    Source: §I · p. 1, D. Offline Iterative Refinement and… · p. 5

  • Results

    FaultGene improves semantic alignment, compilation, and execution success over one-shot generation while preserving the main HIL sensor- and compute-fault sensitivity trends.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The tuned generator produces fault-augmented tests at deployment without manually engineered reference executions or iterative refinement.

    Source: §I · p. 2, §VII · p. 8

    Timing effects are evaluated after generation rather than incorporated into the generation objective.

    Source: §VII · p. 8

Abstract

from arXiv · show

Reliable validation of autonomous driving cyber-physical systems requires testing not only nominal driving behavior but also failures arising from faults in sensing and embedded computation. Physical road testing provides realistic system behavior but is costly and difficult to reproduce, while simulation enables scalable evaluation but cannot fully capture hardware-dependent behavior and fault propagation. Hardware-in-the-loop (HIL) validation bridges this gap by executing the driving stack on real embedded hardware within a simulated environment. However, preparing fault-augmented scenarios for HIL testing remains largely a manual process, limiting test coverage, scalability, and efficiency. We present FaultGene, an LLM-assisted framework for automated generation of executable fault-augmented tests for HIL fault-resilience validation. Given a baseline Scenic/CARLA scenario and a structured fault specification, FaultGene generates a fault-augmented test case that captures the effects of HIL-level fault injection. During offline refinement, semantic-consistency checks and execution-level feedback are used to correct candidates that fail to preserve the intended scenario context or produce the desired fault-induced degradation, and to construct high-quality supervision for fine-tuning. The resulting generator then produces fault-augmented tests directly at deployment time without requiring manually engineered reference executions or repeated refinement loops.

I. INTRODUCTION

FaultGene generates fault-augmented HIL tests from baseline scenarios and structured fault specifications, using offline execution feedback to refine the generator. On held-out scenario–fault combinations, it improves validity and executability over one-shot generation and reduces manual fault-engineering effort.

I. INTRODUCTION

  • Road testing is costly and difficult to reproduce, while simulation alone cannot fully capture hardware-dependent effects such as timing variation and scheduling jitter.Manually constructing HIL fault scenarios across fault types, activation times, and severities is labor-intensive and difficult to scale.Source: §I · p. 1
  • One-shot LLM generation may violate the intended context, fail to execute, or fail to expose the specified fault meaningfully.These limitations motivate execution-guided refinement grounded in HIL behavior.Source: §I · p. 1
  • FaultGene generates executable fault-augmented tests from a baseline Scenic/CARLA scenario and a structured fault specification.The specification includes fault type, activation timing, and severity.Source: §I · pp. 1–2
  • Offline HIL execution feedback supports refinement and fine-tuning, while deployment produces tests in one pass without reference executions or iterative refinement.The feedback addresses semantic consistency and fault-induced system behavior.Source: §I · pp. 1–2
  • 72× less fault-engineering effort per fault type than conventional manual fault-scenario engineering.The evaluation also reports improved scenario validity and executability over one-shot LLM generation.Source: §I · p. 2

II. RELATED WORKS

Prior work spans digital-twin HIL validation, fault injection, and driving-scenario generation, but existing approaches do not jointly address fault-oriented scenario generation and realistic embedded execution. ADDT provides a CARLA-based HIL environment in which the simulator and driving stack execute in a closed loop.

II. RELATED WORKS

  • Scenario-generation research includes Scenic probabilistic and constraint-based specification, VerifAI simulation-based verification and falsification, search and optimization, and learning scenario representations from videos.Scenic and VerifAI have also been combined to identify challenging scenarios.Source: §II · p. 2
  • Existing fault-injection approaches generally do not jointly address fault-oriented scenario generation and realistic embedded execution.Prior work includes component-level fault injection and LLM-based scenario generation, but these typically rely on predefined scenario spaces or text-/simulation-level generation.Source: §II · p. 2

A. Hardware-in-the-Loop Setup

  • ADDT connects a CARLA simulator and an autonomous-driving computing system over Ethernet for closed-loop execution in a virtual environment.The simulator streams sensor data to the driving system and receives its control commands.Source: A. Hardware-in-the-Loop Setup · p. 2, §III · p. 2

1) Digital-Twin Simulation System:

  • The digital-twin simulation system uses CARLA, with OpenSCENARIO and Scenic specifying and parameterizing traffic scenarios.The simulator runs on a workstation with an Intel i7-9800X CPU and NVIDIA RTX 2080 Ti GPU.Source: 1) Digital-Twin Simulation System · p. 2

2) AD Computing System:

ADDT evaluates fault resilience by injecting faults into sensing and computing paths. The evaluation uses representative driving scenarios and models temporal, spatial, CPU, and GPU faults.

2) AD Computing System:

  • The AD stack runs on Jetson Xavier or Orin hardware, processing camera streams at 10 Hz and producing control commands at 100 Hz.The hardware processes six 480p or three 720p camera streams.Source: 2) AD Computing System · p. 2

3) Testing Scenarios:

  • Evaluation covers vehicle following, large-curvature turning, and overtaking, with configurations including vehicle speeds and inter-vehicle distances.Fig. 2 shows the scenario configurations and parameters.Source: 3) Testing Scenarios · p. 2, Figure 2 · p. 3

B. Fault Injection Mechanism

  • Temporal sensor faults introduce timestamp delays, with delays beyond the perception model’s out-of-order tolerance treated as frame drops.Spatial faults perturb ground-truth extrinsic position and orientation parameters to emulate sensor misalignment with controllable severity.Source: B. Fault Injection Mechanism · p. 2
  • CPU and GPU faults inject register-level bit flips at specified locations and values, allowing errors to propagate through subsequent AD computations.ADDT uses LLFI for CPU faults and NVBitFI for GPU faults.Source: B. Fault Injection Mechanism · p. 3

IV. FAULTGENE: AUTOMATED FAULT-AUGMENTED TEST GENERATION

FaultGene generates constrained fault-augmented scenario variants from baseline Scenic/CARLA scenarios and structured fault specifications. Offline execution comparisons with manually engineered HIL references guide refinement and fine-tuning; ScenarioFeat supplies semantic features for those comparisons.

IV. FAULTGENE: AUTOMATED FAULT-AUGMENTED TEST GENERATION

  • FaultGene has offline construction and deployment phases, with the tuned generator producing tests in one pass without references, execution feedback, or refinement loops.During offline construction, generated executions are compared with manually engineered ADDT HIL references to guide prompt refinement and supervision construction.Source: §IV · p. 3, Figure 3 · p. 4

A. ScenarioFeat: Video-Based Scenario Feature Extraction

  • ScenarioFeat extracts semantic features from execution videos to compare generated executions with manually engineered references.Its predefined features describe driving context and safety-relevant behavior, including road environment, object interactions, obstacle relationships, and vehicle maneuvers.Source: A. ScenarioFeat: Video-Based Scenario… · p. 3

B. FaultGene: Prompt-Based Scenic Generation

FaultGene transforms a baseline scenario and structured fault specification into a fault-augmented Scenic program while preserving the original driving context. Reference-guided semantic and execution checks assess whether generated scenarios retain that context and produce meaningful fault effects.

B. FaultGene: Prompt-Based Scenic Generation

  • The generator takes a baseline scenario and fault specification as input and produces a fault-augmented Scenic program.The fault specification defines fault type, activation condition, and severity.Source: B. FaultGene: Prompt-Based Scenic… · p. 4
  • The generated scenario must preserve the original road layout, actor identities, and intended maneuver semantics while satisfying Scenic syntax constraints.Structured Scenic output can be syntax-checked and executed in CARLA or CARLA-based HIL setups such as ADDT.Source: B. FaultGene: Prompt-Based Scenic… · p. 4

C. Reference-Guided Similarity and Execution Checks

  • Each generated scenario is executed through the HIL loop and compared with its corresponding manually fault-engineered ADDT HIL reference.These references provide offline supervision grounding and are not required at deployment.Source: C. Reference-Guided Similarity and… · p. 4
  • ScenarioFeat extracts semantic feature vectors from the reference and generated execution videos for comparison.The mismatch vector records feature-wise differences between the two executions.Source: C. Reference-Guided Similarity and… · p. 4
  • Semantic inconsistency is assessed against feature-specific thresholds that trade off context preservation and scenario diversity during offline refinement.The threshold is used only during offline refinement.Source: C. Reference-Guided Similarity and… · p. 4
  • Fault relevance is evaluated using near-collisions, inter-vehicle clearance, maneuver interruption, collision outcome, and task completion.The scenario must exhibit safety-relevant degradation without trivial invalidity such as deadlock, early termination, or failure to trigger the fault.Source: C. Reference-Guided Similarity and… · p. 4
  • Together, the checks test whether a scenario preserves its intended context, implements the requested fault, and lets that fault manifest during execution.Source: C. Reference-Guided Similarity and… · p. 4

D. Offline Iterative Refinement and Supervised FineTuning

Candidates that fail semantic, fault-relevance, compilation, or execution checks are refined offline using structured feedback. The refined Scenic programs then serve as targets for supervised fine-tuning, and generalization is tested on held-out scenario–fault combinations.

D. Offline Iterative Refinement and Supervised FineTuning

  • Candidates failing semantic or fault-relevance checks, compilation, or execution are refined using structured feedback on the identified mismatch.Feedback can request corrected actor placement, changed activation timing or fault severity, or removal of deadlocks and early termination.Source: D. Offline Iterative Refinement and… · p. 4
  • Refinement uses rule-guided LLM regeneration conditioned on the baseline scenario, fault specification, current candidate, and structured feedback.It is not a reinforcement-learning policy or an external optimizer that directly edits Scenic code.Source: D. Offline Iterative Refinement and… · p. 5
  • Teacher-forced causal language modeling conditions on the baseline scenario, fault specification, and preceding program tokens while computing loss over the target Scenic program.Source: D. Offline Iterative Refinement and… · p. 5
  • Across training examples, shared parameters learn reusable transformations for preserving scenario elements, encoding faults, and adjusting actor timing and interactions.The stated objective is to learn the conditional mapping from scenario–fault pairs to valid, fault-relevant Scenic programs.Source: D. Offline Iterative Refinement and… · p. 5
  • The refined Scenic program is used as the supervised target for its baseline-scenario and fault-specification input pair.This target turns execution-level behavioral supervision into an input-aligned program target rather than using the manually engineered HIL reference itself.Source: D. Offline Iterative Refinement and… · p. 5
  • Fine-tuning absorbs offline corrections into trainable LoRA parameters, with generalization evaluated on scenario–fault combinations excluded from refinement and fine-tuning.Source: D. Offline Iterative Refinement and… · p. 5

E. Deployment-Time Fault-Augmented Test Generation

At deployment, the fine-tuned generator maps a new baseline scenario and fault specification to a Scenic program for execution. Because offline corrections are absorbed through fine-tuning, generation requires neither reference comparisons nor repeated refinement.

E. Deployment-Time Fault-Augmented Test Generation

V. EXPERIMENTAL SETUP

The evaluation uses CARLA and Scenic, and tests FaultGene on modular and end-to-end driving architectures. The setup includes a disjoint held-out split, QLoRA fine-tuning, and measures of scenario validity and fault sensitivity.

V. EXPERIMENTAL SETUP

  • CARLA provides the simulated driving environment, while Scenic specifies structured scenarios and integrates with CARLA.CARLA's physics engine and environmental rendering support reproducible driving scenarios; Scenic represents vehicle configurations, traffic interactions, and environmental conditions.Source: §V · p. 5
  • FaultGene is evaluated on a modular architecture and an end-to-end architecture to cover different system organizations and fault-propagation paths.The modular stack combines FastBEV perception, an MLP planner, and a PID controller; PilotNet maps front-camera images to steering commands.Source: §V · p. 5
  • The refinement dataset contains 600 baseline scenario–fault combinations, with 200 disjoint combinations reserved for held-out evaluation.Held-out combinations are excluded from refinement and fine-tuning.Source: §V · p. 5
  • Table I reports semantic alignment, compilation success, and execution success for modular and end-to-end systems.Semantic alignment is the average feature agreement between generated and reference executions.Source: Table I · p. 6, §V · p. 6
  • Table II compares ADDT and FaultGene sensor-fault sensitivity trends across camera frame-drop rates using mission success rate.Source: Table II · p. 6
  • Fine-tuning uses QLoRA-based supervised learning on the 600 scenario–fault pairs and their refined Scenic programs.The pretrained Qwen3-14B model is frozen in 4-bit form while LoRA adapters are optimized.Source: §V · p. 6

VI. RESULTS AND EVALUATION

FaultGene is evaluated on held-out scenario–fault combinations across modular and end-to-end systems, with results covering validity, executability, and preservation of intended scenario behavior.

A. Scenario Validity, Executability, and the Effect of Iterative Refinement and Fine-Tuning

  • For the modular stack, one-shot generation without refinement achieves 68% compilation success; with fine-tuned FaultGene, compilation success rises to 96% and execution success to 93%, while 81% complete closed-loop execution is reported.Remaining execution failures primarily result from physically infeasible initializations, including overlapping vehicles and deadlocked traffic configurations.Source: A. Scenario Validity, Executability,… · p. 6
  • For the end-to-end system, compilation and execution success increase from 71% and 74% to 94% and 90%, respectively.The results indicate that refinement benefits both the modular and end-to-end systems.Source: A. Scenario Validity, Executability,… · p. 6
  • Semantic alignment scores rise from 0.45 to 0.89 for the modular system and from 0.43 to 0.86 for the end-to-end system.ScenarioFeat scores whether generated scenarios preserve predefined feature categories.Source: A. Scenario Validity, Executability,… · p. 6

B. Consistency with ADDT Fault Trends

FaultGene-generated tests are assessed for whether they preserve relative sensor- and compute-fault vulnerability patterns observed in ADDT, rather than identical mission-success values.

B. Consistency with ADDT Fault Trends

  • Under increasing camera frame-drop rates, FaultGene preserves ADDT’s sensor-fault ordering: vehicle following is most robust, large-curve turning degrades moderately, and overtaking is most sensitive.Overtaking degrades more rapidly than the other two scenarios; the end-to-end evaluation also follows ADDT trends for lane keeping and large-curve turning.Source: B. Consistency with ADDT Fault Trends · pp. 6–7
  • Across modular scenarios, later pipeline-stage faults cause greater mission-success degradation than perception faults, with control-node faults especially damaging in turning and overtaking.Fig. 4 compares mission success across three scenarios, perception, planning, and control nodes, and 1, 3, or 5 injected bit-flip faults.Source: B. Consistency with ADDT Fault Trends · p. 7, Figure 4 · p. 7
  • Scenario-level sensitivity also matches ADDT: overtaking is more vulnerable to planning faults, while large-curvature turning is particularly vulnerable to control faults.Vehicle following is more stable overall but retains the same module-level ranking.Source: B. Consistency with ADDT Fault Trends · p. 7

C. Testing Efficiency

FaultGene reduces fault-scenario preparation effort while retaining the same HIL platform and execution time assumptions. The conclusion notes that its timing effects are evaluated after generation, not incorporated into the generation objective.

C. Testing Efficiency

  • For 20 fault types, preparation falls from 12 h to 10 min per type, a 72× speedup; the modeled 1,000-run campaign falls from 440 h to 203.33 h.The estimated total campaign-time reduction is 53.8%, or a 2.16× speedup.Source: C. Testing Efficiency · pp. 7–8
  • The same HIL setup is estimated to cost below $5,000, retaining ADDT’s approximately 10× hardware-cost advantage over a physical test vehicle.The comparison excludes the vehicle’s insurance, maintenance, calibration, and safety-personnel costs.Source: C. Testing Efficiency · p. 8

VII. CONCLUSION

  • FaultGene can be adapted to other cyber-physical systems by replacing the scenario representation, execution metrics, and fault-injection interfaces.The framework is also not tied to LLFI and NVBitFI and can incorporate other fault-injection backends.Source: §VII · p. 8
  • A current limitation is that timing effects are evaluated after generation rather than incorporated into the generation objective.Future work will integrate HIL timing measurements and broader fault-injection backends into feedback.Source: §VII · p. 8
Loading 2610.29969v1…