PaperlayerView exact revision on arXiv

Source-linked AI summary

Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving

Yuan Gao, Wenting Miao, Mattia Piccinini, Haoyu Wang, Qunying Song, Johannes Betz

arXiv:2607.14387v1cs.AIcs.RO

TL;DR

Generating diverse, regulation-compliant autonomous-driving scenarios as executable DSL scripts remains difficult because existing retrieval-based methods trade compilation reliability against generalizability. Chat2Scenic addresses this with an iterative retrieval-augmented framework and regulation-grounded benchmark, achieving 76.42% CSR and 58.17% FA while outperforming retrieval-assemble and full-script generation approaches.

  • Problem

    Existing methods struggle to generate complex, regulation-grounded ADS scenarios while balancing compilation reliability and generalizability.

  • Method

    Chat2Scenic combines interactive refinement, retrieval-augmented regulatory and DSL knowledge, and iterative component-wise Scenic code generation.

  • Results

    76.42% CSR and 58.17% FA were achieved, outperforming retrieval-assemble and full-script generation approaches.

  • Takeaways & Limitations

    The regulation-grounded 123-scenario benchmark and evaluation metrics support systematic assessment of ADS scenario-generation frameworks.

Abstract

from arXiv · show

Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. Yet automatically generating such scripts from regulatory descriptions remains an open challenge, and existing approaches face fundamental trade-offs. Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, whereas retrieval-based full-script generation suffers from low compilation success rates. We present Chat2Scenic, the first iterative retrieval-augmented framework to generate scenario scripts in Domain Specific Language (DSL). Specifically, Chat2Scenic provides a chatbot interface that supports interactive scenario refinement and integrates Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax. Furthermore, we propose an open benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, as well as other sources. Extensive evaluation with State-of-the-Art (SOTA) Large Language Models (LLMs) demonstrates that Chat2Scenic achieves 76.42% Compilation Success Rate (CSR) and 58.17% Framework Accuracy (FA), outperforming existing methods (Retrieval Assemble with 30.08% CSR, 11.03% FA and Retrieval full script generation with 16.26% CSR, 10.86% FA). To facilitate future research, we release our code as open source at https://github.com/TUM-AVS/chat2scenic.

I. INTRODUCTION · A. Related Work · 1) Classical Scenario Generation:

The paper motivates regulation-grounded ADS scenario generation by highlighting the limitations of physical testing and current automated DSL-generation approaches. It situates Chat2Scenic within classical scenario-generation research spanning data-driven, adversarial, knowledge-based, and multi-abstraction methods.

  • I. INTRODUCTION: Physical testing alone cannot adequately capture the diversity of driving situations needed to validate increasingly capable ADS.The introduction links this challenge to the maturation of autonomous driving and deployment of SAE Level 4 robotaxis within specific ODDs.
  • I. INTRODUCTION: LLM-based generation of diverse, regulation-grounded scenarios remains challenging.This limitation is explicitly identified in the introduction as an unresolved difficulty.
  • I. INTRODUCTION: The authors are affiliated with the Professorship of Autonomous Vehicle Systems at the Technical University of Munich and the Munich Institute of Robotics and Machine Intelligence.The affiliation passage identifies Y. Gao, W. Miao, M. Piccinini, H. Wang, and J. Betz.
  • I. INTRODUCTION: Q. Song is affiliated with University College London, and the listed authors have equal contribution marked by an asterisk.The passage specifies London, United Kingdom, for the affiliation.
  • I. INTRODUCTION: Chat2Scenic iteratively generates DSL blocks that interact with a database to create complex driving scenarios.The figure contrasts this process with assembling existing DSL components or directly generating complete DSL programs.
  • I. INTRODUCTION: Retrieval-assemble methods achieve reasonable compilation rates but lack generalizability, whereas direct DSL generation offers flexibility but has low compilation success rates.These trade-offs motivate the paper’s iterative generation approach.
  • 1) Classical Scenario Generation:: Scenario-generation research is commonly categorized into data-driven, adversarial, and knowledge-based approaches.Data-driven methods use real-world datasets, adversarial methods synthesize rare high-risk situations, and knowledge-based methods integrate domain expertise.
  • 1) Classical Scenario Generation:: Classical scenario-generation work also examines functional, logical, and concrete abstraction levels using learning-, optimization-, data-extraction-, and manual-design methods.Schutt et al. examine these dimensions in the context of the SOTIF standard.

2) LLM-based Scenario Generation:

LLMs generate autonomous-driving scenarios from natural-language inputs across vehicle- and traffic-centric simulators, using DSLs, APIs, retrieval-assemble pipelines, or direct generation. Existing approaches improve realism or executability but face limitations in expressiveness, generalizability, or generation complexity.

  • LLM-based Scenario Generation:: LLM-based scenario generation targets vehicle-centric simulators such as CARLA and traffic-centric simulators such as SUMO.ChatSUMO and OmniTester demonstrate scalable language-driven generation for realistic urban traffic simulations.
  • LLM-based Scenario Generation:: Recent work generates CARLA scenarios through Scenic or OpenScenario DSLs and simulator APIs.API-based methods rely on large databases and mapping rules, limiting expressiveness, whereas DSLs offer formal, human-readable parameterized specifications.
  • LLM-based Scenario Generation:: Retrieval-assemble methods parse traffic rules or behaviors, retrieve code or routes, and assemble executable scenario scripts.TARGET, ChatScene, and Text2Scenario represent this family using multi-stage prompting, RAG, hierarchical retrieval, or priority-based DSL assembly.
  • LLM-based Scenario Generation:: Retrieval-assemble methods lack generalizability to novel scenarios beyond their predefined code databases.This limitation constrains their applicability when required scenarios are not represented in the available retrieval resources.
  • LLM-based Scenario Generation:: Direct-generation methods include multimodal prompting, template-based two-stage processing, constrained decoding, and compiler-feedback refinement.LEADE, LeGEND, and ScenicNL exemplify these strategies for producing executable scenario DSL or concrete scripts.

B. Critical Summary · C. Contribution · II. METHODOLOGY

The paper addresses limitations in existing DSL scenario-generation methods by introducing Chat2Scenic, an iterative chatbot framework combining interactive refinement, retrieval-augmented generation, and component-wise code synthesis. It also contributes a broad benchmark and evaluates the framework across diverse LLMs and prompting settings.

  • B. Critical Summary: Existing retrieval-based assembly methods construct DSL programs from stored snippets, limiting generalization and scenario diversity.
  • B. Critical Summary: Direct DSL-generation methods face compilation-reliability challenges and lack comprehensive compilation-success evaluation.
  • C. Contribution: Chat2Scenic uses iterative component-wise generation, advanced prompting, and RAG to outperform SOTA methods across compilation success, semantic accuracy, and framework accuracy.
  • C. Contribution: The authors curate 120+ autonomous-driving scenario descriptions and propose dedicated metrics for evaluating the bot.
  • C. Contribution: The evaluation spans closed- and open-source LLMs, includes prompting ablations, and benchmarks Chat2Scenic against SOTA methods.
  • II. METHODOLOGY: Chat2Scenic comprises interactive, RAG, and generation modules followed by a simulation interface.
  • II. METHODOLOGY: The interactive module parses descriptions into logical structured representations and supports interactive refinement.
  • II. METHODOLOGY: The RAG module retrieves domain-specific knowledge, while the generation module iteratively produces Scenic components until a complete program is produced.

A. Interactive Module · 1) Logical Structure Schema: · 2) Interpreter:

Chat2Scenic combines a chatbot interface for regulation-grounded, iterative scenario refinement with a modular logical representation and an LLM-based interpreter. The interpreter decomposes descriptions into DSL-aligned component abstractions that support downstream retrieval and code generation.

  • A. Interactive Module: Chat2Scenic provides a Gradio-supported chatbot interface for describing scenarios, querying regulation-grounded retrieved scenarios, and iteratively refining descriptions.The interface is illustrated in Fig. 3.
  • 1) Logical Structure Schema:: The framework represents each user request as a logical structure S built from the DSL’s building blocks for controllable, modular scenario generation.The representation is defined as a set of global configuration and scenario components.
  • 1) Logical Structure Schema:: The logical structure S comprises global configuration G and scenario components S_int.This decomposition separates global settings from scenario-specific elements.
  • 1) Logical Structure Schema:: Global configuration G covers the map, weather, and vehicle models and is independently extracted by the Global Configuration Generator.The passage identifies this generator as part of Section II-C.1.
  • 1) Logical Structure Schema:: The Interpreter extracts S_int = {R, E, O, T}, representing spatial relations, ego behavior, objects, and restrictions.Objects are represented as O = {O_1, ..., O_N}, while restrictions include initial conditions in the supplied passage.
  • 2) Interpreter:: An LLM-based interpreter uses explicit instructions and few-shot examples to extract S_int from scenario descriptions.Each extracted component is abstracted into a single-sentence description.
  • 2) Interpreter:: The interpreter’s single-sentence component abstractions serve as semantic queries for component code generation.This aligns interpretation with database code snippets and Scenic syntax, facilitating retrieval and generation.

B. RAG Module · 1) Code Snippet Database: · 2) Documentation Database:

Chat2Scenic integrates LangChain-supported RAG to retrieve external knowledge during scenario generation, improving accuracy and reducing hallucinations. Its RAG module combines specialized retrieval over complementary code-snippet and documentation databases.

  • B. RAG Module: RAG retrieves relevant external knowledge during scenario generation to improve accuracy and reduce hallucinations.The framework integrates RAG supported by LangChain.
  • B. RAG Module: The module constructs complementary code snippet and documentation databases with specialized retrievers for each.These databases ground generation in both executable DSL code and supporting domain documentation.
  • 1) Code Snippet Database:: The code database collects official Scenic code and decomposes scenarios into component-level units covering Sint components R, E, O, and T.Each scenario is represented through independent component-level code units.
  • 1) Code Snippet Database:: Each code-database entry pairs an LLM-generated one-sentence natural-language description with its corresponding Scenic code snippet.An example description is “A car travels forward and makes a right turn.”
  • 2) Documentation Database:: The documentation database aggregates official Scenic and regulatory documents, including UN Vehicle Regulations, then cleans and hierarchically chunks them while preserving document structure.Documents are collected through web crawling and split using structure-aware hierarchical separators.
  • 2) Documentation Database:: Documentation chunks are embedded and indexed with BM25, enabling hybrid retrieval that combines semantic matching with keyword-based search.The passage specifies Best Match 25 (BM25) for keyword-based retrieval.

3) Dual Retriever Architecture: · C. Generation Module · 1) Global Configuration Generator:

Chat2Scenic generates executable Scenic programs through a dual-retriever, multi-stage pipeline that combines code and documentation context with global configuration and dependency-ordered component generation. The global configuration stage detects scenario settings, applies confidence-based defaults, and produces an executable Scenic header.

  • 3) Dual Retriever Architecture:: The dual-retriever architecture supplies code-level and language-level domain-specific context during scenario generation.The code retriever uses semantic search over code snippets, while the documentation retriever combines keyword and embedding retrieval before Reciprocal Rank Fusion.
  • C. Generation Module: The pipeline extracts logical structure, detects settings, generates global configuration, and produces spatial, ego, object, and restriction code in sequence.Algorithm 1 represents the generated program as Scode = {Gcode, Rcode, Ecode, O1:N,code, Tcode}.
  • 3) Dual Retriever Architecture:: The code retriever returns the top-3 semantically similar snippets after filtering results by component type with COSINE similarity.Retrieval is conditioned on a component description, such as the ego vehicle traveling forward.
  • 3) Dual Retriever Architecture:: The documentation retriever uses BM25 for explicit Scenic API keywords and embeddings for related behavioral descriptions, then merges rankings with RRF.For lane-following behavior, BM25 can retrieve FollowLaneBehavior definitions while embeddings retrieve semantically related descriptions.
  • C. Generation Module: The generation module transforms logical representation S into executable Scenic code through global configuration, iterative component generation with accumulated context, and concatenation.The framework uses prompting with pre-trained LLMs rather than fine-tuning.
  • 1) Global Configuration Generator:: The Global Configuration Generator detects and formalizes scenario settings through a two-stage SettingsDetector and HeaderGenerator pipeline.SettingsDetector uses rule-based keyword matching and example-based reasoning, while HeaderGenerator converts detected settings into executable Scenic configuration.
  • 1) Global Configuration Generator:: Only detections with confidence ≥0.6 are accepted; low-confidence or null results default to Town05, ClearNoon, and Lincoln mkz.The generated Scenic header covers CARLA map parameters, weather, and vehicle models.
  • 1) Global Configuration Generator:: Contextual Prompting injects syntax constraints, hierarchical type systems, and available operators to strengthen domain understanding while allowing flexible knowledge additions.An example hierarchy is NetworkElement → LinearElement → {Road, Lane} for GeneratorR.

2) Iterative Component Generator:

The Iterative Component Generator combines structured Chain-of-Thought reasoning with example-based and dynamically retrieved in-context learning. Together, these techniques incrementally guide component generation while adapting examples and documentation to each query.

  • Iterative Component Generator: Chain-of-Thought structures each component’s generation into explicit, tailored reasoning steps, improving understanding and generation accuracy.For Generator E, the steps include understanding requirements, examining context, selecting types, defining parameters and behavior, instantiating, and validating.
  • Iterative Component Generator: In-Context Learning uses positive implementation examples and negative error-correction pairs to anchor syntax, reduce ambiguity, and prevent common pitfalls.The model learns directly from few-shot demonstrations included in the prompt context.
  • Iterative Component Generator: Retrieval-Augmented In-Context Learning dynamically retrieves query-relevant context through CodeICL and DocICL, supporting flexible adaptation and extension to new domains.CodeICL retrieves the top-3 semantically similar code snippets, while DocICL retrieves relevant documentation chunks; the database can be extended independently.

III. RESULTS & DISCUSSION · A. Experimental Setup · 1) Task Definition and Benchmark:

The evaluation uses CARLA and Scenic on specified hardware, and introduces a 123-scenario regulation-grounded benchmark spanning multiple sources. It evaluates proprietary and open-source LLMs across several model families and configurations.

  • A. Experimental Setup: Experiments use the 3D simulator CARLA 0.9.15 with the compatible DSL Scenic 3.1.0.All experiments run on a Dell Alienware R15 with an NVIDIA RTX 4090 GPU and 24GB VRAM.
  • 1) Task Definition and Benchmark:: The benchmark contains 123 regulation-grounded scenario specifications from multiple sources.Sources include CARLA Leaderboard5, NHTSA crash and pre-crash datasets, and United Nations Vehicle Regulations.
  • 1) Task Definition and Benchmark:: 24 scenarios come from the CARLA Leaderboard5.This source is one component of the benchmark’s 123 scenario specifications.
  • 1) Task Definition and Benchmark:: 47 scenarios come from NHTSA crash and pre-crash datasets.The benchmark draws these scenarios from NHTSA sources.
  • 1) Task Definition and Benchmark:: 52 scenarios come from United Nations Vehicle Regulations, including UN R152, R157, and R171.The passage identifies these regulations as part of the benchmark’s source distribution.
  • 1) Task Definition and Benchmark:: The evaluation covers proprietary LLMs including Qwen, DeepSeek-V3.2, Google Gemini-3, and Qwen3-Coder configurations.Evaluated variants include Qwen Flash and Plus, DeepSeek Chat and Reasoner, Gemini Pro and Flash, and Qwen3-Coder Plus and Flash.
  • 1) Task Definition and Benchmark:: Open-source evaluation uses Ollama with GPT-OSS 20B, Mistral-Small-3.2, Gemma-3-27B, Qwen3:30B, and Qwen3-Coder:30B.The models include both general-purpose and code-specialized systems.

2) Evaluated Large Language Models: · 3) Evaluation Metrics: · B. Experiments and Results

The evaluation uses framework-performance and scenario-generation-accuracy metrics, complemented by qualitative examples, ablation studies, and cross-model comparisons on the proposed benchmark.

  • 2) Evaluated Large Language Models:: The framework is comprehensively evaluated using two categories of metrics.These categories assess framework performance and scenario-generation accuracy.
  • 3) Evaluation Metrics:: Compilation Success Rate (CSR) measures the percentage of generated Scenic programs that successfully execute in the simulator.
  • B. Experiments and Results: Fig. 5 presents CARLA-rendered examples from the CARLA Leaderboard, UN R171, and NHTSA Crash benchmark sources.The examples depict an unsignalized intersection, obstacle avoidance by a swerving lead car, and an undetected pedestrian crossing.
  • 3) Evaluation Metrics:: Response Time (RT) is the average generation time per scenario from description to code.
  • 3) Evaluation Metrics:: Token Usage (Tokens) is the average number of tokens consumed per scenario.
  • 3) Evaluation Metrics:: For successfully compiling scenarios, human evaluation measures how accurately generated executable code reflects the original description across semantically distinct driving-scenario layers.BLEU and ROUGE-L are not applicable because the output is executable code.
  • B. Experiments and Results: The experiments additionally include comprehensive ablation studies and cross-model comparisons on the proposed benchmark.

1) Ablation Study:

The ablation study evaluates 12 prompting configurations and shows that progressively adding contextual, reasoning, and in-context techniques improves scenario-generation performance. The strongest reported configuration reaches 58.17% Framework Accuracy, while DocICL adds cost without improving performance.

  • Configuration Design: The study evaluates 12 configurations across zero/few-shot, contextual, thinking, and extended tiers using Gemini 3 Flash.The techniques include CP, CoT, ICL, and RAG-ICL.
  • Performance Progression: Adding CP to the zero-shot baseline raises CSR from 0% to 12.20% and FA from 0% to 8.51%.C1 is the zero-shot baseline, whereas C4 adds CP alone.
  • Performance Progression: 47.15% CSR and 32.62% FA are achieved by combining CP with ICL, compared with 12.20% CSR and 8.51% FA for CP alone.This comparison shows the reported gain from adding ICL to CP.
  • Performance Progression: 54.47% CSR and 39.09% FA are achieved by adding CoT to CP, creating the reported thinking-tier configuration C9.The passage describes further improvement from the CP+ICL configuration when CoT is added.
  • Best Configuration: 58.17% FA is reported for C11, with a response time of 222.11 seconds.C11 incorporates CP, CoT, ICL, and CodeICL.
  • Extended Configuration: Adding DocICL in C12 does not improve performance and incurs higher time and token costs because Scenic documentation is better suited to interactive code-concept chats than code generation.The passage also states that CodeICL provides more code examples and improves generation efficiency.

2) Comparison across Models and SOTA Methods: · IV. CONCLUSION AND FUTURE WORK

Chat2Scenic achieves the strongest reported performance in cross-model and SOTA comparisons, led by Gemini-3-Flash and iterative component-wise generation. The framework is presented with a 123-scenario benchmark and advanced prompting results of 76.42% CSR, 76.12% SQ, and 58.17% FA.

  • 2) Comparison across Models and SOTA Methods:: Gemini-3 family models outperform other proprietary models, likely benefiting from powerful capabilities and long context windows.
  • 2) Comparison across Models and SOTA Methods:: 76.42% CSR and 58.17% FA make Gemini-3-Flash the best overall model evaluated.Gemini-3-Flash outperforms Gemini-3-Pro, which achieves 60.16% CSR and 44.22% FA.
  • 2) Comparison across Models and SOTA Methods:: 0%–1.62% CSR is achieved by most open-source models, whose smaller scale limits general capabilities.
  • 2) Comparison across Models and SOTA Methods:: 30.08% CSR makes ChatScene stronger than NL2Scenic among the compared SOTA methods.ChatScene uses a pre-built CARLA Leaderboard scenario code database for retrieval-based assembly.
  • 2) Comparison across Models and SOTA Methods:: 76.42% CSR lets Chat2Scenic significantly outperform ChatScene and NL2Scenic.The framework uses iterative component-wise generation and diverse prompting strategies, while NL2Scenic’s complete-script retrieval struggles with complex scenarios.
  • IV. CONCLUSION AND FUTURE WORK: Chat2Scenic is presented as the first iterative retrieval-augmented framework for autonomous-driving scenario generation.
  • IV. CONCLUSION AND FUTURE WORK: 123 regulation-grounded scenarios from multiple sources form the benchmark introduced for systematic evaluation.The benchmark is accompanied by comprehensive evaluation metrics.
  • IV. CONCLUSION AND FUTURE WORK: 76.42% CSR, 76.12% SQ, and 58.17% FA are achieved by Gemini-3-Flash with CP+CoT+ICL+CodeICL prompting.These results were demonstrated through ablation studies and cross-model comparisons.
Loading 2607.14387v1…