Source-linked AI summary
Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation
Hanlin Tian, Kethan Reddy, Yuxiang Feng, Mohammed Quddus, Yiannis Demiris, Panagiotis Angeloudis
TL;DR
RL-based AV training can be limited by narrow scenario coverage and the challenge of generating diverse, critical situations. CRITICAL closes a feedback loop between scenario generation and training, using real-world traffic dynamics, safety metrics, and optional LLM analysis. In the reported evaluation, LLM-enhanced critical case generation achieved higher reward, longer average episodes, and fewer crashes than the baseline PPO model.
Problem
RL-based AV training needs diverse, realistic scenarios, while interpretable frameworks for generating safety-critical situations remain limited.
Source: §I · p. 1, C. Critical Scenarios · p. 3Method
Results
LLM-enhanced critical case generation achieved a reward of 76.886, an average episode length of 93.019, and 61 crashes.
Source: D. Evaluation and Results · p. 6Takeaways & Limitations
The authors report that continual feedback between scenario generation and RL training enhances learning rate, performance, and robustness.
Source: §V · p. 6
Abstract
from arXiv · showhide
This paper introduces CRITICAL, a novel closed-loop framework for autonomous vehicle (AV) training and testing. CRITICAL stands out for its ability to generate diverse scenarios, focusing on critical driving situations that target specific learning and performance gaps identified in the Reinforcement Learning (RL) agent. The framework achieves this by integrating real-world traffic dynamics, driving behavior analysis, surrogate safety measures, and an optional Large Language Model (LLM) component. It is proven that the establishment of a closed feedback loop between the data generation pipeline and the training process can enhance the learning rate during training, elevate overall system performance, and augment safety resilience. Our evaluations, conducted using the Proximal Policy Optimization (PPO) and the HighwayEnv simulation environment, demonstrate noticeable performance improvements with the integration of critical case generation and LLM analysis, indicating CRITICAL's potential to improve the robustness of AV systems and streamline the generation of critical scenarios. This ultimately serves to hasten the development of AV agents, expand the general scope of RL training, and ameliorate validation efforts for AV safety.
I. INTRODUCTION
The introduction identifies limited scenario diversity and interpretability as challenges for RL-based AV training. It presents CRITICAL as a closed-loop framework that uses real-world driving data and optional LLM analysis to generate critical scenarios.
I. INTRODUCTION
- Conventional controlled, repeatable scenarios may not cover real-world complexity, leaving agents inadequately prepared for unpredictable traffic.The introduction also identifies a lack of interpretable frameworks for generating diverse, safety-critical scenarios.Source: §I · p. 1
- CRITICAL can use episode outputs directly to generate scenarios or send a parsed prompt to an LLM for an alternative environment configuration.The approach is described as combining data-driven and knowledge-driven methods rather than relying exclusively on either.Source: Figure 1 · p. 1, §I · p. 2
- CRITICAL combines scenario generation with RL training feedback to target critical driving situations and improve AV training and evaluation.The framework is designed to expose agents to varied scenarios and use training-episode analysis to refine scenarios.Source: §I · p. 2
- The framework uses highD traffic data, driving-behavior analysis, and risk metrics to create high-fidelity critical scenarios for training.Its closed-loop feedback is reported to elevate learning rate, performance, and adaptability to safety-critical situations.Source: §I · p. 2
II. RELATED WORK
Related work spans data-driven, adversarial, and knowledge-based approaches to generating autonomous-vehicle scenarios.
A. Scenario Generation for Autonomous Vehicle
- Data-driven scenario generation ranges from direct sampling and clustering to probabilistic modeling and deep generative models.The cited examples include Traffic Primitives and random perturbation for scenario augmentation.Source: A. Scenario Generation for Autonomous… · p. 2
- Adversarial methods generate high-risk scenarios through challenging interactions, using generators and victim models or perturbations to static-scene data.The passage distinguishes dynamic scenario generation as important for evaluating planning and control modules.Source: A. Scenario Generation for Autonomous… · p. 2
- Knowledge-based generation uses domain expertise and predefined rules to create realistic scenarios consistent with traffic laws and physical principles.Examples include optimizing object behavior and environmental conditions or using RL for adversarial policy development.Source: A. Scenario Generation for Autonomous… · p. 2
B. Large Language Models in Autonomous Vehicles
The paper reviews LLM capabilities and their use in autonomous-driving research, then motivates CRITICAL as a framework for augmenting RL training with critical scenarios. It distinguishes scenario rarity from safety-boundary criticality.
B. Large Language Models in Autonomous Vehicles
- CRITICAL’s architecture uses highD-based configurations to create HighwayEnv scenarios, then uses episode risk assessments to select direct or LLM-assisted scenario generation.The framework collects failure reports, risk metrics, and rewards across episodes.Source: Figure 3 · p. 3
- LLM research includes reasoning-oriented question answering and prompt tuning, while AV applications include interpreting driving environments and joint GraphVQA and end-to-end driving.DriveLM is described as a nuScenes- and CARLA-based dataset with a VLM baseline for those tasks.Source: B. Large Language Models in Autonomous… · pp. 2–3
C. Critical Scenarios
- Edge cases are infrequent or extreme scenarios, while boundary scenarios lie near the safe–unsafe boundary; critical scenarios combine the two categories.The paper frames this distinction as separating scarcity from safety in scenario categorization.Source: C. Critical Scenarios · p. 3
- The paper identifies LLM-enhanced generation of complex, critical driving situations as underexplored and calls for an open-source framework that augments RL training data.The stated motivation is adaptive generation of diverse and challenging scenarios.Source: C. Critical Scenarios · p. 3
III. METHODOLOGY
The methodology uses HighwayEnv as a configurable simulation environment and PPO for AV path planning and prediction.
A. Reinforcement Learning for Autonomous Vehicle Environment
- HighwayEnv parameters such as traffic dynamics and vehicle behavior change the scenarios presented to the RL agent during training.The environment configuration is adjusted to diversify training scenarios.Source: A. Reinforcement Learning for… · p. 3
- The AV path-planning and prediction models use PPO, selected for balancing performance enhancement with training stability.The passage attributes this balance to PPO’s objective function.Source: A. Reinforcement Learning for… · p. 3
- PPO’s epsilon-clipped objective regulates update size to support gradual, stable learning improvements.The passage identifies the probability ratio and advantage estimate as terms in the objective.Source: A. Reinforcement Learning for… · p. 3
B. highD Dataset for Realistic Traffic Simulation
CRITICAL uses highD traffic data and driving-behavior knowledge to ground simulated scenarios in real-world traffic. TTC and a unified risk index identify configurations that can supply safety-critical challenges.
B. highD Dataset for Realistic Traffic Simulation
- The highD dataset supplies natural highway trajectories and human-driving behavior knowledge for realistic traffic simulation and scenario diversification.The paper describes highD recordings of German highways and uses real-world knowledge or LLM priors when changing environment configurations after training episodes.Source: B. highD Dataset for Realistic Traffic… · pp. 3–4
C. Risk Metrics to Measure Safety-Criticality
- TTC uses relative distance and relative speed; lower TTC indicates higher collision risk, and crossing its threshold increments a near-miss count.The framework also uses a unified risk index based on longitudinal and lateral risk indices; both risk propensity parameters β and γ are set to 1.Source: C. Risk Metrics to Measure… · p. 4
- The unified risk index combines longitudinal and lateral risk, and configurations exceeding its predefined threshold contribute to criticality assessment.The paper describes these metrics as surrogate measures for evaluating the safety-criticality of ego-vehicle maneuvers.Source: C. Risk Metrics to Measure… · p. 4
- Configurations repeatedly exceeding TTC or unified-risk thresholds are treated as boundary scenarios, while high-criticality, low-occurrence configurations are treated as edge cases for training or LLM-inspired generation.The framework uses these identified configurations to present the RL agent with safety-critical challenges.Source: C. Risk Metrics to Measure… · p. 4
A. Transforming highD Data for Scenario Generation
The paper converts highD observations into behavior categories and simulated traffic configurations, then selects high-risk vehicle interactions to recreate critical scenarios.
A. Transforming highD Data for Scenario Generation
- KPrototypes clusters vehicles using speed, acceleration, and lane-changing behavior to generate behavior categories for HighwayEnv traffic scenarios.The experiment labels driving behaviors as aggressive, defensive, or normal and models them with parameters such as politeness and preferred following distance.Source: A. Transforming highD Data for… · p. 4
- The Risk Perception metric ranks vehicle pairs using Time Headway and Time To Collision, and the highest-RP instances are selected for critical-scenario recreation.The passage sets A to 1 and B to 4 in its RP calculation.Source: A. Transforming highD Data for… · p. 5
- The scenario database records 60 highD traffic configurations, including vehicle types, counts, dynamic behaviors, and critical vehicle pairs.Table I is identified as the summary of scenario features.Source: A. Transforming highD Data for… · p. 5, Table I · p. 5
B. Prompting the LLM
CRITICAL uses LangChain and LLM analysis to suggest simulation-parameter changes from traffic data, risk counts, and prior run outcomes. Parameter ranges and JSON formatting constrain the resulting scenario modifications.
B. Prompting the LLM
- CRITICAL supplies the LLM with highD traffic patterns, TTC near-miss counts, and unified-risk threshold counts to inform realistic configuration suggestions.The pipeline uses LangChain for language-model scenario analysis and generation.Source: B. Prompting the LLM · p. 5
- The LLM uses recorded configurations, outcomes, and vehicle failure types alongside real-world driving behaviors to suggest changes to simulation parameters.The run history is used to identify trends and patterns.Source: B. Prompting the LLM · p. 5
- Scenario parameters are represented as JSON dictionaries with predefined realistic value limits, enabling parsing of LLM outputs while keeping adaptations controlled.The parameter dictionaries are detailed in Table I.Source: B. Prompting the LLM · p. 5
C. Computational Setup
The reported computational setup uses an Intel Xeon w5-2455X processor, an NVIDIA A6000, and 256GB of RAM. The accompanying figure compares training losses across three PPO configurations.
C. Computational Setup
- The experiments ran on an Intel(R) Xeon(R) w5-2455X processor, an NVIDIA A6000 graphics card, and 256GB of RAM.Source: C. Computational Setup · p. 5
- Fig. 4 compares training loss for Vanilla PPO, PPO with Critical Case Generation, and PPO with Critical Case Generation and LLM Analysis.Source: Figure 4 · p. 5
D. Evaluation and Results
Across PPO-only, critical-case, and LLM-enhanced evaluations, adding critical scenarios corresponded to higher reported reward and longer episodes with fewer crashes; LLM-enhanced training also yielded the lowest, most stable loss.
D. Evaluation and Results
- The comparison included PPO-only baseline, critical case generation, and LLM-enhanced generation using Mistral-7B-Instruct.The LLM was selected for its ability to understand and generate instructional content for varied, intricate scenarios.Source: D. Evaluation and Results · p. 5
- Critical-scenario training reduced loss versus baseline, while LLM-enhanced training showed the lowest and most stable loss.The paper presents these loss-curve patterns as evidence of accelerated learning and stronger learning efficacy.Source: D. Evaluation and Results · p. 5
- Models trained on 50 scenario configurations were evaluated on a separate set of 10 configurations, with each test configuration run 10 times.The reported final performance was averaged across those runs.Source: D. Evaluation and Results · p. 5
- Reward rose from 50.068 (baseline) to 61.527 (critical case generation) and 76.886 (LLM-enhanced); average episode length rose 55.760, 63.319, 93.019, while crashes fell 89, 81, 61.Table II reports these testing results averaged over 10 runs.Source: D. Evaluation and Results · pp. 5–6, Table II · p. 6
- Figure 5 encodes unified-risk-index threshold counts in blue on the left and TTC near-miss counts in green on the right; its rows show baseline, non-LLM, and LLM-enhanced configurations.The baseline is on the top row, critical case generation without LLM in the middle, and LLM-enhanced generation on the bottom.Source: Figure 5 · p. 6
E. Critical Scenario Generation: Affects on RL-Agent Performance & Criticality Measurements
The reported risk-measure distributions show lower criticality as CRITICAL training improves, and the authors describe a feedback loop that exposes the PPO agent to diverse challenging scenarios. Validation on additional datasets and RL algorithms remains future work.
E. Critical Scenario Generation: Affects on RL-Agent Performance & Criticality Measurements
- LLM-enhanced configurations show a marked decrease in mean r-threshold counts, which the authors interpret as superior agent performance across scenarios.Criticality scores and risk metrics, including unified risk index r and TTC near-miss, were collected throughout training.Source: E. Critical Scenario Generation:… · p. 6
- The authors describe LLM analysis as refining training and preparing AVs for varied real-world conditions, with safety and performance benefits.They characterize these findings as evidence of the efficacy of the approach, particularly the LLM integration.Source: E. Critical Scenario Generation:… · p. 6
V. CONCLUSIONS AND FUTURE WORK
- CRITICAL dynamically exposes the PPO agent to diverse challenging scenarios; the authors report this feedback loop improves performance, learning rate, and robustness.The framework incorporates highD traffic dynamics and surrogate safety measures to assess scenario criticality.Source: §V · p. 6
- TTC and unified risk index measures track scenario risk; mean configuration criticality declines as AV performance improves.The authors present this pattern as evidence of CRITICAL's effectiveness for training RL-based AV algorithms.Source: §V · p. 6
- Future validation will extend beyond highD and assess additional RL algorithms to examine generalizability across traffic conditions and road layouts.The authors also identify algorithm versatility and optimization across training paradigms as future aims.Source: §V · p. 6