PaperlayerView exact revision on arXiv

Source-linked AI summary

Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation

Zaid Ghazal, Khouloud Gaaloul, Bruce Maxim

arXiv:2609.27296v1cs.SEcs.AIcs.MAcs.SYeess.SY

TL;DR

ADS simulation testing must find rare safety-critical failures while producing executable, diverse scenarios, yet current LLM approaches are largely open-loop. Teach-to-Crash uses a constrained scenario representation and a Student–Teacher loop; in its CARLA case study, it leads on several reported measures, with PAFOT retaining a higher mean collision discovery rate.

  • Problem

    Fixed-budget ADS testing must search a high-dimensional space for rare safety-critical interactions, while open-loop LLM generation lacks execution feedback to assess whether scenarios improve collision outcomes.

    Source: §1 · p. 1

  • Method

    Teach-to-Crash uses a constrained ego-centric scenario representation, a Student LLM to generate executable scenarios, and a Teacher that intervenes when collision-rate and TTC progress stagnates.

    Source: §1 · p. 1, §4.1 · p. 4, §4.3 · p. 6

  • Results

    Teach-to-Crash reports a 90.79% mean Collision Hit Rate and 18.31 s mean Time-to-Collision; PAFOT has a higher mean Collision Discovery Rate of 179.44 versus 136.21, with substantially larger variance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Within the evaluated CARLA scope, the results support closed-loop dual-LLM reasoning as adaptive search control for generating failures that are frequent, diverse, and assessed as more frequently avoidable.

    Source: §8 · p. 10

    The evaluation is limited to CARLA and omits same-model or stronger-LLM closed-loop comparisons, diversity-aware baselines, and wall-clock, latency, or token-cost measurements.

    Source: §7 · p. 10, §8 · p. 10

Abstract

from arXiv · show

Validating Autonomous Driving Systems (ADS) in simulation requires testing architectures that can discover rare, safety-critical failures while generating scenarios that are executable, diverse, and useful for downstream failure analysis. We introduce Teach-to-Crash, a closed-loop testing framework that combines a constrained ego-centric scenario representation, stagnation-aware search control, and a dual-LLM architecture for adaptive failure discovery. A high-reasoning Teacher LLM acts as an adaptive search controller, while a low-reasoning Student LLM emits simulator-executable scenarios in a strict JSON schema. The Teacher intervenes only when rolling collision rate and time-to-collision metrics stagnate, providing strategic guidance to redirect the search. In a CARLA case study with two experimental setups that vary the ego vehicle's speed policy, Teach-to-Crash achieves the highest Collision Hit Rate (90.79%), the shortest mean Time-to-Collision (18.31 s), and a competitive Collision Discovery Rate (136.21). PAFOT attains a higher mean CDR (179.44), but with substantially larger variance. Teach-to-Crash also yields the highest diversity (0.547) and, averaged across both setups on the CARLA Traffic Manager controller, the highest avoidability-based usefulness proxy (60.04%) among the compared methods. These results, within the evaluated CARLA scope, provide evidence that closed-loop dual-LLM reasoning can steer adversarial simulation-based testing over a constrained executable program space, generating failures that are frequent, structurally diverse, and assessed as more frequently avoidable.

1 Introduction

Teach-to-Crash addresses limits of fixed-budget ADS scenario search with a closed-loop Student–Teacher framework and a constrained ego-centric representation. In the CARLA evaluation, it leads on collision hit rate, time-to-collision, diversity, and avoidability-based usefulness, while PAFOT has a higher mean collision discovery rate with greater variance.

1 Introduction

  • Fixed-budget testing remains difficult because safety-critical multi-agent behaviors occupy a high-dimensional space, while existing LLM generators are mostly open-loop and cannot use execution feedback to assess collision-rate progress.The paper identifies wasted executions on behaviors that never become ego-relevant and stagnation in near-duplicate interactions as persistent bottlenecks.Source: §1 · p. 1
  • Teach-to-Crash combines a closed-loop Student–Teacher architecture with an ego-centric, position-based scenario representation to generate failure-inducing ADS tests.A validator enforces structural constraints on LLM outputs before execution.Source: §1 · pp. 1–2
  • A higher-reasoning Teacher intervenes when collision-rate and time-to-collision progress stalls, guiding the Student toward higher-collision-rate scenarios.The Student is a lower-capability tier by design; the Teacher’s role separation is not isolated from model capability in the evaluation.Source: §1 · p. 2
  • 90.79% mean Collision Hit Rate and 18.31 s mean Time-to-Collision are the best reported across the compared methods; diversity is 0.547 and avoidability-based usefulness is 60.04%.Collision Discovery Rate averages 136.21; PAFOT averages 179.44 with substantially larger variance.Source: §1 · p. 2
  • The evaluation compares Teach-to-Crash with representative baselines across two experimental setups and the CARLA Traffic Manager and TransFuser controllers.The case study is conducted in CARLA.Source: §1 · p. 2

2 Motivating Example and Problem Definition

The problem is to select executable ADS scenarios within a fixed testing budget that induce failures, expose critical interactions, and remain diverse. Teach-to-Crash uses an ego-centered position grid and closed-loop guidance, while diversity and richer criticality measures are assessed offline rather than serving as live control signals.

2 Motivating Example and Problem Definition

  • A single open-loop generator can stagnate on repeated near-collisions; the Teacher may instead guide a cut-in and blocker interaction toward earlier, more frequent collisions.The motivating example begins with a squeeze-and-cut-in that becomes a near-miss as the ego brakes.Source: §2 · pp. 2–3
  • An ego-centered grid encodes NPC positions as discrete regions and decodes each region into a continuous target waypoint; the illustrated configuration uses nine regions, with position 9 reserved for the ego.Each NPC’s scenario is a sequence of position and target-speed pairs that is compiled into an executable program.Source: §2 · pp. 2–3
  • Scenario diversity is the mean pairwise Jaccard distance between scenario fingerprints, with target speeds placed into bins so minor speed changes do not count as distinct behavior.Larger diversity values indicate coverage of more structurally distinct interaction patterns.Source: §2 · p. 3
  • Avoidable collision is a usefulness proxy based on whether the ego had lateral space or sufficient stopping distance in the two seconds before impact.Its thresholds are heuristic, and the proxy is not a substitute for reachability analysis or controller-specific reasoning.Source: §2 · p. 3
  • Under budget B, scenario generation seeks to maximize the fraction of failure-inducing scenarios, their criticality, and their diversity.The search selects and executes scenarios from the space induced by an ego-centered position grid.Source: §2 · p. 3
  • Online control uses rolling collision rate and TTC rather than the richer Crit score, while diversity is encouraged by prompts and evaluated offline over the archive.The archive’s Crit score aggregates four surrogate risk metrics; it is not optimized directly by the online loop.Source: §2 · p. 3

3 Related Works

Prior work generates critical ADS scenarios through search, learned agents, or LLMs, but differs in its representations, search controls, and use of diversity. Teach-to-Crash places LLM reasoning inside the testing loop as stagnation-aware adaptive search control over an executable scenario space.

Search-Based Critical Scenario Generation

  • PAFOT [9] uses the same nine-position ego-centric grid with a genetic algorithm, whereas Teach-to-Crash uses LLM generation and Teacher-guided rewriting; PAFOT is its primary non-LLM baseline.AVFuzzer searches ego-relative interactions through fuzzing, and BehAVExplor [7] and DoFuzz explicitly steer search using diversity.Source: §3 · p. 4

Deep Learning for Safety-Critical Scenario Generation

  • Deep-learning approaches train adversarial or generative agents or learn priors over risky interactions; Feng et al. use dense importance sampling rather than a fixed discrete grid.These approaches typically require substantial simulation budgets.Source: §3 · p. 4

LLM-Driven Critical Scenario Generation

  • LLM methods synthesize, rewrite, or diversify scenarios, but the described approaches generally use single-stage generation with separate validation or regeneration rather than a stagnation-triggered online controller.ChatScene [40] is the leading open-loop LLM baseline; LeGEND [36] uses retrieval and refinement, while TARGET derives its representation from traffic-rule text.Source: §3 · p. 4

4 Methodology

Teach-to-Crash uses a constrained Student–Teacher loop to generate, validate, and execute ego-centric scenarios, with Teacher guidance triggered when rolling collision-rate and time-to-collision progress stagnates. Its representation and search objectives balance executable scenario generation, criticality, diversity, and downstream analysis within a bounded CARLA interaction scope.

4 Methodology

  • Each run executes up to 45 iterations of 10 scenarios, using a 10-record rolling window and targets of 90% collision rate and 12-second TTC.Runs are averaged over 10 independent executions per method and setup; reaching the targets gates Teacher intervention but does not terminate a run early.Source: §4 · p. 4

4.1 Framework Overview

  • The framework iterates Student generation, validation, simulation, and outcome logging; the Teacher guides the next generation only when rolling collision-rate and time-to-collision progress plateaus.The ADS is treated as a black box, requiring simulator execution and outcome logging.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
  • The Student emits strict-JSON scenario batches from recent outcomes, bounds, and any guidance; the higher-reasoning Teacher reads rolling metrics and targets to issue strategic generation directives.The Teacher’s guidance can address position trajectories, speed profiles, NPC roles, and validity reminders.Source: §4.1 · pp. 4–5

4.2 Operating Environment

  • CARLA episodes run for up to 60 seconds, but NPC–NPC contacts are not separately sensed and rear-side ego collisions may be excluded by configuration.Unparseable or invalid scenarios are discarded and are not added to outcome history.Source: §4.2 · p. 6

4.3 Scenario Representation

  • Each scenario is a two-NPC, six-step program of target-speed and discrete ego-centric position pairs, using a compact grammar designed to avoid map-specific coordinates.The evaluation fixes six 10-second segments without sweeping alternative sequence lengths, and covers highway-lane interactions rather than intersections, turns, or general urban traffic-rule interactions.Source: §4.3 · p. 6

4.4 Search Objectives

  • Collision rate and TTC are monitored over recent scenarios, while fingerprint-based diversity is measured offline; stagnation is judged against the last improving checkpoint.When progress fails the configured improvement tests, the Teacher biases the next Student call toward revised positions, timing, and speed evolution.Source: §4.4 · pp. 6–7

5 Evaluation Setup

The evaluation uses CARLA Town06 with a fixed vehicle configuration and two ego speed policies, then tests the setup across repeated runs. Traffic Manager is the primary controller for effectiveness, timing, and diversity evaluation.

5 Evaluation Setup

  • The CARLA evaluation uses Town06, synchronous simulation, a 0.05-second time step, and a 60-second horizon.The map includes highway and urban segments, including junctions, straightaways, intersections, merging and diverging lanes, and Michigan Left turns.Source: §5 · p. 7
  • The primary evaluation uses CARLA Traffic Manager, with two Nissan Micra NPCs; Setup A uses a 60-mph ego target speed and Setup B reduces it by 20% to about 48 mph.Both setups use a 5.0-meter global following distance; TransFuser is additionally evaluated for usefulness.Source: §5 · p. 7
  • The evaluation uses gpt-5-nano as Student and gpt-5-mini as Teacher across 10 independent runs of 45 iterations, with 10 scenarios generated per iteration.Teacher guidance uses statistics from the most recent 10 records.Source: §5 · p. 7

6 Evaluation

The evaluation compares collision finding, timing, diversity, and avoidability-based usefulness across CARLA setups, using PAFOT and ChatScene as baselines.

6.1 Baseline Methods

  • Teach-to-Crash is compared with PAFOT, a genetic algorithm searching a fixed ego-centric grid, and ChatScene, an open-loop LLM generator of executable Scenic scripts.The comparison uses the same CARLA map, ego controller, NPC types, simulator configuration, 60 s horizon, and collision logging.Source: §6.1 · p. 7

6.2 Research Questions

  • The four research questions assess collision-finding effectiveness, collision timing, scenario diversity, and avoidability-based usefulness across 10 runs.Metrics include CHR and CDR; TTC; Jaccard novelty, failure clusters, and unique-failure ratio; and avoidable collision measures.Source: §6.2 · p. 7

6.3 RQ1: Collision-Finding Effectiveness

  • 90.79% mean Collision Hit Rate significantly exceeds PAFOT (79.09%) and ChatScene (64.03%); PAFOT has higher mean Collision Discovery Rate (179.44 col/hr vs. 136.21).Teacher-guided scenarios account for over 77% of collisions in the reported Student-only and Teacher-guided partition, but the ablation does not isolate intervention timing or causal contribution.Source: §6.3 · p. 8, Table 1 · p. 8

6.4 RQ2: Collision Timing and Criticality

  • 18.31 s mean Time-to-Collision is shorter than PAFOT (23.71 s) and ChatScene (30.32 s), indicating earlier collisions within the rollout horizon, not faster testing.The timing comparison is summarized in Figure 5.Source: §6.4 · p. 8, Figure 5 · p. 9

6.5 RQ3: Diversity of Collision Scenarios

  • 0.547 average Jaccard novelty and 404.4 failure clusters per run lead both baselines; the 0.900 unique-failure ratio indicates fewer redundant collision signatures.Figure 6 illustrates three distinct interaction types: right-rear, right-front, and left-front collisions; these examples do not establish overlap with baseline patterns.Source: §6.5 · p. 9, Figure 6 · p. 9

6.6 RQ4: Avoidability-Based Usefulness Proxy

  • Teach-to-Crash has the highest avoidable collision rate under the proxy in all evaluated configurations: 39.80% with Traffic Manager on Setup A, 84.06% with TraverseV4, and 80.27% on Setup B.TraverseV4 was tested only for Teach-to-Crash, so it provides no direct baseline comparison; Table 2 also reports avoidable collisions per run and per 100 scenarios.Source: §6.6 · pp. 9–10, Table 2 · p. 10

7 Threats to Validity

The evaluation is bounded by potential implementation bias and residual generation variability, a narrow CARLA scenario and controller scope, proxy-based construct measures, and limited baseline coverage.

Internal Validity

  • ChatScene adaptation may bias comparisons, and repeated runs with fixed prompts reduce but do not eliminate stochastic variability.All methods share Town06, controllers, NPC types, and simulator configuration; effectiveness and diversity conclusions use CHR, TTC, and fingerprint novelty, independent of LLM inference overhead.Source: §7 · p. 10

External Validity

  • The case study covers one map, a fixed ego-relative grid, two NPCs, highway lane-following, and two controllers, not intersections, urban traffic rules, Apollo, or Autoware.Two NPCs keep the search tractable but miss denser traffic, and N and K were not swept.Source: §7 · p. 10

Construct Validity

  • Diversity uses Jaccard novelty and failure clusters, while usefulness uses a simplified kinematic avoidable-collision proxy, not reachability analysis or demonstrated debugging benefit.Per-run results do not establish whether methods discover overlapping or disjoint failure clusters.Source: §7 · p. 10

Scope and Baseline Coverage

  • Comparisons omit a same-model or stronger-LLM closed loop and diversity-aware baselines, so dual-LLM superiority over untested alternatives is not claimed.The study also reports no wall-clock time, LLM latency, or token cost and makes no efficiency claims.Source: §7 · p. 10

8 Conclusion and Future Work

Teach-to-Crash uses execution feedback in a closed-loop dual-LLM architecture to steer collision-inducing scenario generation. In the evaluated CARLA case study, it leads the compared methods on collision rates, mean time-to-collision, failure-archive diversity, and the avoidability-based usefulness proxy, while future work targets broader evaluation and measurement.

Conclusion

  • In the CARLA case study, Teach-to-Crash achieves the highest collision rates, shortest mean Time-to-Collision, most diverse failure archive, and highest avoidability-based usefulness proxy among compared methods on both controllers.The evaluation covers a single map, highway lane-following, and two NPCs.Source: §8 · p. 10
  • The ablation links Teacher-guided scenarios to most discovered collisions but does not isolate intervention timing or model capability.The conclusion describes adaptive search control as trading CHR for CDR relative to PAFOT, with TTC treated as a criticality signal rather than measured end-to-end efficiency.Source: §8 · p. 10

Future Work

  • Future work includes sensitivity analysis, richer severity and diversity objectives, broader maps and ADS stacks, additional baselines, token-cost measurement, and reachability-based avoidability analysis.Source: §8 · p. 10
Loading 2609.27296v1…