PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

Jiawei Zhang, Chejian Xu, Bo Li

arXiv:2405.14062v1cs.AIcs.LG

TL;DR

Autonomous-vehicle testing needs broader safety-critical scenarios than existing approaches provide, while direct LLM-to-code generation can produce invalid Scenic programs. ChatScene retrieves and assembles Scenic snippets from LLM-generated scenario descriptions to execute diverse simulations in CARLA, increasing collision rates by 15% over SOTA baselines and reducing them by an additional 9% after adversarial fine-tuning.

  • Problem

    Existing methods cover a narrow range of safety-critical scenarios, while exhaustive real-world testing is expensive and data-intensive for autonomous vehicles.

  • Method

    ChatScene generates scenario descriptions with an LLM, decomposes them into components, retrieves Scenic code snippets, and assembles executable CARLA simulations.

  • Results

    15% higher collision rates were achieved by ChatScene scenarios versus state-of-the-art baselines, while adversarial fine-tuning reduced collision rates by an additional 9% versus SOTA.

  • Takeaways & Limitations

    ChatScene generates more challenging safety-critical scenarios and supports fine-tuning that improves autonomous-vehicle robustness against collisions.

  • Takeaways & Limitations

    Prior methods and scenario coverage remain limited to a narrow range of safety-critical situations that may not encompass real-world complexity.

Abstract

from arXiv · show

We present ChatScene, a Large Language Model (LLM)-based agent that leverages the capabilities of LLMs to generate safety-critical scenarios for autonomous vehicles. Given unstructured language instructions, the agent first generates textually described traffic scenarios using LLMs. These scenario descriptions are subsequently broken down into several sub-descriptions for specified details such as behaviors and locations of vehicles. The agent then distinctively transforms the textually described sub-scenarios into domain-specific languages, which then generate actual code for prediction and control in simulators, facilitating the creation of diverse and complex scenarios within the CARLA simulation environment. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets by training a knowledge database containing the scenario description and code pairs. Extensive experimental results underscore the efficacy of ChatScene in improving the safety of autonomous vehicles. For instance, the scenarios generated by ChatScene show a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, we show that by using our generated safety-critical scenarios to fine-tune different RL-based autonomous driving models, they can achieve a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs.

1. Introduction

ChatScene addresses the need for broader, efficient safety-critical AV testing by converting language descriptions into executable CARLA simulations. It combines LLM-generated scenarios with retrieval of Scenic code snippets to produce diverse adversarial scenes.

  • Motivation: Real-world testing is expensive and data-intensive, motivating simulated safety-critical scenarios as a cost-effective alternative.Safety-critical events are rare in collected driving data, while exhaustive testing is needed before deployment.
  • Motivation: Existing adversarial-scenario methods cover only a narrow range of situations and may not capture real-world complexity.The cited limitation motivates broader scenario generation for autonomous-driving evaluation.
  • Research question: ChatScene asks whether LLMs can generate broader safety-critical descriptions and automatically convert them into simulations for AV testing.These questions directly frame the paper’s problem.
  • Approach: The agent generates a scenario description, extracts component characteristics, retrieves corresponding Scenic snippets, and assembles an executable CARLA script.The pipeline covers behaviors and other scenario components before rendering the simulation.
  • Contribution: ChatScene introduces an LLM-based agent that transforms textual safety-critical scenarios into executable CARLA simulations.Its retrieval database supports the translation from descriptions to Scenic code.
  • Results: In Safebench evaluations, ChatScene’s adversarial scenes increased collision rates by 15% compared with four state-of-the-art baselines.The result indicates that the generated scenarios provide more challenging safety tests.

2. Related Work

Prior AV scenario-generation approaches rely on data, learned adversaries, or manually encoded knowledge, each with limitations in cost, diversity, or coverage. ChatScene combines LLM-derived knowledge with Scenic-based environmental optimization.

  • Scenario generation: Data-driven generation uses real-world data but suffers from scarce, costly data and the rarity of genuinely risky scenarios.Safety-critical events lie in the long-tail of real-world scenario distributions.
  • Scenario generation: Knowledge-based generation is systematic and potentially diverse, but manually encoding rules is complex and may miss safety-critical situations.Manual rules can therefore produce fewer risky scenarios than desired.
  • ChatScene: ChatScene integrates diverse real-world knowledge from LLMs with Scenic optimization of surrounding-agent parameters to increase scenario risk and complexity.The optimized parameters include nearby pedestrian and vehicle speeds.
  • LLMs for autonomous driving: Unlike related LLM driving studies focused on descriptions, actions, or interpretation, ChatScene primarily generates safety-critical scenarios for CARLA evaluation.Its scenarios can support training or testing reinforcement-learning-controlled ego vehicles.

3. Methodology

ChatScene uses retrieval-based Scenic code assembly rather than relying on direct LLM code generation. It builds component-specific databases, extracts structured descriptions, retrieves snippets, and renders parameterized CARLA simulations.

  • Motivation and Notations: Directly prompting LLMs for Scenic code can produce non-compilable code or nonexistent API calls because Scenic examples are scarce.ChatScene therefore uses an indirect retrieval-based approach.
  • Motivation and Notations: Scenic code is decomposed into fixed settings, adversarial behavior, road geometry, and surrounding-vehicle spawn positions.The ego vehicle’s behavior is supplied by a reinforcement-learning controller rather than defined in the scenario script.
  • Construction of the Retrieval Database: The retrieval database pairs descriptions with Scenic snippets for adversarial behavior, geometry, and relative spawn positions.Initial examples are sourced from Scenic and manually decomposed into description-snippet pairs.
  • Safety-Critical Scenario Generation: ChatScene generates a scenario description, extracts Behavior, Geometry, and Spawn Position sub-descriptions, encodes them, retrieves matching snippets, and assembles a CARLA script.Few-shot prompting structures the component extraction before retrieval.
  • Safety-Critical Scenario Generation: The system samples parameters such as adversary speed and distance, records vehicle statistics, and narrows sampling ranges around collision-associated values.Collision-prone parameters are modeled as approximately Gaussian and sampled within [μ − σ, μ + σ].
  • Evaluation: Scenario-generation evaluation reports collision rate, overall score, and average displacement error across three differently trained ego vehicles.The table summarizes mean performance over scenes for each base scenario and averages across base scenarios.

4. Experiment

The experiments evaluate ChatScene’s scenario-generation and adversarial-finetuning capabilities across diverse CARLA-based traffic scenarios. ChatScene produces more challenging and diverse scenarios than baselines and improves ego-vehicle robustness after finetuning.

  • Evaluation setup: The evaluation measures scenario safety-criticality through collision rate and overall performance, while also assessing trajectory diversity with ADE across base traffic scenarios.Experiments use three ego-vehicle training paradigms and eight base traffic scenarios.
  • Scenario generation: ChatScene generates five unique scenario descriptions per base scenario and maps them into Scenic scripts for simulation.The approach contrasts with baselines that provide one scenario per base scenario.
  • Safety-critical scenario generation: 15% higher collision rates than the most competitive baselines show that ChatScene creates more safety-critical scenarios.The scenarios challenge autonomous-driving systems in more adversarial environments.
  • Safety-critical scenario generation: 16% greater relative reduction in overall score than the leading baseline indicates that ChatScene produces more challenging scenarios.The scenarios also reduce route completion and require higher acceleration, yaw velocity, and more frequent lane invasions to avoid collisions.
  • Scenario diversity: ChatScene achieves the highest ADE and outperforms existing benchmarks across all reported metrics for each base scenario.The ADE result reflects greater variability in adversarial-object trajectories and supports the diversity of the generated scenarios.
  • Implications: The experiments conclude that challenging and diverse generated scenarios can support more robust autonomous-driving algorithms and safer evaluation.The paper reports elevated collision rates during testing and improved collision avoidance after finetuning.
  • Adversarial training: After adversarial finetuning, collision rates fall 9% below SOTA and 51% below the original ego vehicle, while overall score improves 43% relatively.The finetuned ego vehicle surpasses agents trained with alternative approaches in most base scenarios.

5. Conclusion

ChatScene is an LLM-based agent that generates safety-critical scenario descriptions, retrieves corresponding Scenic code, and compiles simulations in CARLA. Its generated scenarios are more challenging and improve ego-vehicle collision avoidance through finetuning.

  • Method: ChatScene automatically generates scenario descriptions, decomposes them, retrieves Scenic code, and compiles simulations within CARLA.The pipeline connects language-based scenario generation with executable simulation scenes.
  • Scenario generation: ChatScene scenarios produce higher ego-vehicle collision rates than other methods under the same scenarios.This indicates greater challenge for safety-critical testing.
  • Adversarial training: ChatScene-generated scenarios improve ego vehicles’ collision avoidance during safety-critical finetuning.The conclusion presents this as evidence of utility for improving autonomous-vehicle robustness.

A. Few-Shot Prompts

The appendix describes prompts for constructing a Scenic-snippet retrieval database and extracting component-specific descriptions from complete scenario descriptions.

  • A.1. Prompt for generating snippets: The snippet-collection prompt targets adversarial behaviors of surrounding vehicles in a “Straight Obstacle” base scenario.The example illustrates how scenario components are represented as description-snippet pairs.
  • A.1. Prompt for generating snippets: GPT-4.0 responses are reviewed for adversarial behavior, with manual verification and correction of errors.This review is part of preparing the description-snippet pairs.
  • A.1. Prompt for generating snippets: The snippet prompt supports collecting Scenic code fragments for fundamental driving-scenario elements.The broader method uses these fragments in a retrieval database.
  • A.2. Prompt for extracting descriptions: The appendix separates scenario-component snippet collection from extraction of component descriptions.Together, the prompts support the decomposition and retrieval stages of ChatScene.
  • A.2. Prompt for extracting descriptions: A few-shot prompt extracts component-specific descriptions from a comprehensive input scenario description.The appendix presents this as a separate prompt from snippet collection.
  • A.2. Prompt for extracting descriptions: Regular expressions are used to extract the specified descriptions for each component.This provides the mechanical extraction step after prompting.

B. Detailed Scenario descriptions

This section provides detailed descriptions of the scenarios generated by the method for each base scenario, with complete descriptions presented in Tables 6 and 7.

  • Complete descriptions of the generated scenarios for each base scenario are presented in Tables 6 and 7.

C. Additional Experiment Details and Results

Experiments use one NVIDIA RTX A6000 and GPT-4 as the underlying language model, with online monitoring and API-based automation available.

  • Experiments run on one NVIDIA RTX A6000.
  • GPT-4 is used as the underlying LLM for manually monitoring generation quality and can be adapted to the gpt-4-1106-preview API.

C.1. Detailed Metric

The evaluation adopts Safebench metrics covering collision avoidance, traffic-rule compliance, roadway adherence, route completion, efficiency, vehicle dynamics, and lane keeping, with OS aggregating them.

  • CR measures collision frequency, while RR and SS measure running red lights and failing to stop at stop signs.
  • OR measures average distance driven out of the roadway, and RF measures route-following stability.
  • Comp and TS measure route completion and completion time, while ACC and YV characterize acceleration and turning behavior.
  • LI measures lane-invasion frequency, while OS aggregates all metrics using Safebench’s weights.

C.2. Detailed Performance for Differently Trained ego vehicles

ChatScene’s generated scenes generalize across ego vehicles trained with SAC, PPO, and TD3, achieving the best average adversarial performance.

  • ChatScene consistently achieves the best average performance across ego vehicles trained with SAC, PPO, and TD3.Collision-rate results are reported in Table 8, and overall scores in Table 9.
  • The evaluation assesses adversarial performance separately for each test ego vehicle and reports detailed collision-rate and overall-score statistics.

C.3. Adversarial Finetuning Details

The finetuning and scenario-generation details combine adversarial behavior prompts, component-level decomposition, Scenic code snippets, and evaluation procedures. The experiments also note a stopping-strategy issue under prolonged training and assess performance across multiple scenario-generation methods and RL-trained ego vehicles.

  • Adversarial Finetuning Details: Adversarial finetuning uses a pretrained SAC-trained model, 500 epochs, learning rates of 0.0001, separate collision and non-collision replay buffers, and batch size 512.
  • Adversarial Finetuning Details: Prolonged training epochs caused the model to adopt a stopping strategy.
  • Adversarial Finetuning Details: The preferred checkpoint is selected after epoch 100 at 50-epoch intervals using the lowest collision rate while maintaining route completion above 0.3.
  • Adversarial Scenario Prompts: The prompt examples generate safety-critical behaviors for pedestrians, cars, cyclists, and motorcycles while requiring Scenic-compatible code without new APIs.
  • Scenario Decomposition and Retrieval: ChatScene decomposes scenarios into behavior, geometry, and spawn position, then retrieves and assembles corresponding Scenic snippets into simulations.Examples include sudden braking, crossing pedestrians, oncoming turns, and rear approaches.
  • Evaluation Materials: Tables 4–9 document behavior-generation and component-extraction prompts, generated scenario descriptions, collision-rate comparisons, and overall-score comparisons.
Loading 2405.14062v1…