PaperlayerView exact revision on arXiv

Source-linked AI summary

Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model

Bo-Kai Ruan, Hao-Tang Tsui, Yung-Hui Li, Hong-Han Shuai

arXiv:2409.09575v4cs.RO

TL;DR

Randomized and replay-based simulator scenarios have limits in intentional control and novel-scene coverage, while prior prompt-driven work has gaps in free-form layout generation and environmental control. TTSG uses language-model prompt analysis, road retrieval, agent planning, and plan-aware road ranking to generate traffic scenes. Experiments report improved agent safety and action reasoning, with limitations from language-model errors and the available CARLA map conditions.

  • Problem

    Random sampling lacks intentional control and replay is limited by logged-data distributions, while instruction-driven methods have gaps in free-form layout generation, spawn-point independence, and environmental conditions.

    Source: §1 · p. 1

  • Method

    TTSG parses natural-language scene requirements, retrieves roads, plans agents, and ranks candidate roads for compatibility with environmental constraints and planned maneuvers.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Experiments on critical-scenario training and driving narration and reasoning report improved agent safety and enhanced action reasoning capabilities.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The framework supports routine and critical scenarios, sequential events, and natural-language scene generation without predefined routes or spawn points.

    Source: §1 · p. 2, §6 · p. 8

    LLMs may misinterpret prompts or hallucinate elements, and the built-in CARLA map limits retrieval of complex or novel signal and object conditions.

    Source: J. Limitation · p. 15

Abstract

from arXiv · show

Generating realistic and controllable traffic scenes from natural language can greatly enhance the development and evaluation of autonomous driving systems. However, this task poses unique challenges: (1) grounding free-form text into spatially valid and semantically coherent layouts, (2) composing scenarios without predefined locations, and (3) planning multi-agent behaviors and selecting roads that respect agents' configurations. To address these, we propose a modular framework, TTSG, comprising prompt analysis, road retrieval, agent planning, and a novel plan-aware road ranking algorithm to solve these challenges. While large language models (LLMs) are used as general planners, our design integrates them into a tightly controlled pipeline that enforces structure, feasibility, and scene diversity. Notably, our ranking strategy ensures consistency between agent actions and road geometry, enabling scene generation without predefined routes or spawn points. The framework supports both routine and safety-critical scenarios, as well as multi-stage event composition. Experiments on SafeBench demonstrate that our method achieves the lowest average collision rate (3.5\%) across three critical scenarios. Moreover, driving captioning models trained on our generated scenes improve action reasoning by over 30 CIDEr points. These results underscore our proposed framework for flexible, interpretable, and safety-oriented simulation.

1. Introduction

Simulator-based generation offers safe, repeatable testing, but random sampling lacks intentional control and replay is constrained by logged-data distributions. Existing prompt-driven approaches also struggle with free-form descriptions, predefined locations, or environmental conditions; TTSG addresses these gaps with natural-language scene and agent planning.

Motivation

  • Random sampling diversifies scenes but lacks intentional control, while replay remains limited by the distribution of collected data.These constraints make it difficult to systematically evaluate specific failure modes, edge cases, or unseen scenarios.Source: §1 · p. 1
  • Prior instruction-driven methods rely on structured inputs, require manually specified spawn points, or omit environmental conditions such as signals, objects, and weather.The cited approaches include LCTGen, ProSim, CTG++, and ChatScene.Source: §1 · p. 1

Contribution

  • TTSG interprets free-form language to generate layouts and plan agents without rigid prompt formats or predefined spawn points.It also supports conditional generation based on road signals and objects.Source: §1 · p. 2
  • The framework parses scene elements, retrieves candidate roads, plans multi-agent behavior, ranks roads for compatibility, and renders executable scenes.Its road-ranking algorithm aligns candidate roads with environmental constraints and planned maneuvers.Source: §1 · p. 2
  • The framework supports sequential event simulation and reports trained-agent collision reduction and improved captioning-model action reasoning.The introduction identifies these as contributions but supplies no quantitative values for the improvements.Source: §1 · p. 2

2. Related Work

Prior work uses learning-based methods and language models for scene generation, agent control, and driving-related reasoning. TTSG is positioned as a training-free pipeline that generates detailed layouts and plans directly from natural language without predefined routes or spawn points.

2.1. Scene Generation with LMs

  • Scene-generation research schedules trajectories on scenario maps, retrieves prior scenarios, or uses language models to structure prompts, guide diffusion losses, predict policies, and coordinate critical scenarios.The cited examples include [12] [39], [8], LCTGen [25], CTG++ [38], ProSim [26], and ChatScene [36].Source: §2.1 · p. 2
  • Unlike these prior approaches, TTSG uses a training-free pipeline for detailed scene layout and agent planning from natural language, without predefined routes or spawn points.It also supports sequential-event design and flexible environmental control.Source: §2.1 · p. 2

2.2. Autonomous Driving with LMs

  • Driving-language work applies language models to captioning, decision reasoning, vehicle action planning, safety training, and multi-agent interaction.Examples include ADAPT [17], [5] [23], LLM-Attacker [20], [21], and [14].Source: §2.2 · p. 2

3. Text-to-Traffic Scene Generation

TTSG turns a natural-language description into a rendered traffic scene through road-graph construction, prompt analysis, retrieval, agent planning, and plan-aware ranking. Its applications include multi-agent, critical, and sequential scenarios.

3. Text-to-Traffic Scene Generation

  • The framework decomposes scene descriptions, retrieves road candidates, plans agents, ranks roads for compatibility, and renders the selected road and plans into a traffic scene.Figure 1 depicts the five stages: analysis, retrieval, planning, ranking, and generation.Source: §3 · p. 2, Figure 1 · p. 3

3.1. Construction of Road Graph and Agent Set

  • Each CARLA map is converted to an OpenDRIVE road graph encoding connectivity, signals, objects, junctions, and lane configurations for road filtering and automatic spawn selection.The agent set categorizes supported types, including regular vehicles, emergency vehicles, and pedestrians.Source: §3.1 · p. 3

3.2. Retrieval and Planning with LLM

  • Format verification checks each stage's outputs, and detected discrepancies trigger prompt resubmission for automatic correction, with occasional computational overhead.The rendering module produces scene frames with images, actions, and trajectories, and supports closed-loop interactions in blocking scenarios.Source: §3.2 · pp. 3, 5
  • Prompt analysis decomposes requirements into signals, objects, and agent configurations, then retrieval filters graph roads using conditions such as lanes, signals, and road objects.The analysis-based approach is reported as comparable to CoT while using significantly fewer tokens.Source: §3.2 · pp. 3–4
  • Agent plans specify types, actions, positions, orientations, and relative distances, enabling coordinated multi-agent setups from explicit or implicit descriptions.Agents can be positioned relative to the ego vehicle or to one another.Source: §3.2 · p. 4
  • Road ranking scores candidate roads against planned agent requirements and selects a highest-scoring road, randomizing ties to support selection diversity.Road properties include directional options, road types, and length needed to accommodate the agents.Source: §3.2 · p. 4

3.3. Applications with TTSG

  • Applications include implicit multi-agent scenes, priority for emergency vehicles, and critical scenarios with controlled agent positions and environmental conditions such as weather.Figure 2 demonstrates multi-agent planning and a critical scenario; Figure 3 illustrates special vehicles and weather control.Source: Figure 2 · p. 4, §3.3 · p. 5, Figure 3 · p. 5
  • Sequential events reuse the final positions from an initial scene as starting points, allowing later stages to be planned with prompt analysis and agent planning.The example combines an unprotected left turn with a later blocking event while maintaining coherent transitions and temporal consistency.Source: §3.3 · p. 6

4. Experiments

The experiments evaluate TTSG in critical-scenario training and test its planning and road-ranking components. The pipeline lowers average collision rate, while ablations show benefits from analysis and plan-aware road ranking.

4.1. Environment and Pipeline Setup

  • The CARLA setup includes nine agent types, twelve ego-relative position configurations, and seven basic actions, with GPT-4o used for prompt analysis, retrieval, and planning.The experiments evaluate collision rate and driving score, considering results valid only when route completion exceeds 30%.Source: §4.1 · p. 6, §4.2 · p. 6

4.2. Generation for Critical Scenarios

  • Average collision rate falls from 0.08 to 0.035, while driving scores are slightly lower in some scenarios despite improved average performance.The evaluation covers straight-obstacle, lane-changing, and unprotected-left-turn scenarios; results are reported after 800 training episodes.Source: §4.2 · p. 6

4.3. Ablation Study

  • The analysis-based strategy improves agent and road planning accuracy over the variant without analysis, while the hybrid with CoT yields stronger results.TTSG achieves planning quality comparable to CoT-based prompting while significantly reducing token usage.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • Road ranking consistently improves scene accuracy by selecting roads aligned with agent plans rather than choosing roads randomly.Without ranking, road selection can overlook turn permissions or suitable spawning points.Source: §4.3 · p. 7, Table 2 · p. 7

5. Discussion

The discussion tests generated scenes for critical-scenario captioning, LLM flexibility, and scene diversity. Captioning improves after fine-tuning on generated scenes, and diversity varies with how specifically prompts constrain agents.

5.1. Generation for Narration and Reasoning

  • Fine-tuning ADAPT on generated critical scenes improves action reasoning and produces more precise action descriptions.The training and evaluation sets each contain 20 scenes; the ego agent is set to cautious behavior, and collision cases are resampled.Source: §5.1 · p. 7, Table 5 · p. 8

5.2. Performance Across Different LLMs

  • TTSG performs strongly across tested LLMs, with Gemma3-12B competitive in several categories and Gemini-2.5-Flash and Claude-Sonnet-3.5 near-perfect in agent and road planning.The evaluated models include open-source and API-based systems.Source: §5.2 · pp. 7–8

5.3. Diversity Test

  • Road selections are nearly unique across scenarios, while agent diversity is lower for specific agent descriptions and higher for less-specific prompts.Scene accuracy is lower in cases with precise timing or restrictive vehicle conditions, though the rendering interface produces most scenarios accurately.Source: §5.3 · p. 8, Table 4 · p. 8

6. Conclusion and Future Work

TTSG generates traffic layouts and plans agent behavior from natural-language descriptions, using plan-aware road ranking to support routine, critical, and sequential scenarios. Experiments demonstrate improvements in agent safety and action reasoning; future work targets new traffic objects and LLM-controlled multi-agent actions.

6. Conclusion and Future Work

  • TTSG interprets scenario requirements, retrieves road candidates, plans agent behavior, and ranks roads according to agent configurations.The framework supports routine and critical scenarios as well as sequential events.Source: §6 · p. 8
  • Experiments on critical-scenario training and driving narration and reasoning demonstrate improvements in agent safety and action reasoning.Source: §6 · p. 8
  • Future work will extend generation to entirely new traffic objects and use LLMs to control multi-agent actions.Source: §6 · p. 8

Supplementary Material

The supplementary material lists supported scene elements, describes the LLM system prompt and stage-specific formats, and illustrates how road ranking scores candidate roads against agent plans.

A. Environment Setup

  • The pipeline supports varied agent types, actions, behaviors, ego-relative positions, road segments, signals, road objects, and weather conditions.Agent behaviors include cautious, normal, and aggressive; weather combines a condition with a time of day.Source: A. Environment Setup · p. 11

B. System Prompt

  • The system prompt assigns the LLM analysis, road retrieval, or planning based on an input prefix and requires outputs to follow specified formats.The prompt describes the road network as a graph whose nodes represent road IDs and whose edges represent connections.Source: B. System Prompt · p. 11

C. Prompt Structure

  • Stage-specific prompt formats represent extracted objects, signals, and agents, road retrieval conditions, and environment and agent-planning fields.Agent planning fields include type, ego status, action, behavior, position ID, road type, and relative position.Source: C. Prompt Structure · p. 12

D. Road Ranking Example

  • Candidate roads receive one point for each fulfilled agent-configuration condition, and Road A scores highest in the example.The ranking checks for an adjacent straight segment when an agent on a left-turn lane is planned to turn left.Source: D. Road Ranking Example · p. 12

E. Criticality of Generated Scenario

The framework generates critical scenarios through a planning-based pipeline, with criticality adjustable at test time. It produces meaningful risky scenes without adversarial training or gradient-based optimization, though it does not attain the highest collision rate among all baselines.

E. Criticality of Generated Scenario

  • Criticality is evaluated on crossing negotiation, red-light running, and lane changing, with 100 evaluations per scenario across ten routes.Higher collision rates indicate stronger criticality in this evaluation.Source: E. Criticality of Generated Scenario · p. 12
  • Scene parameters such as surrounding-vehicle speed and threat timing can be changed at test time to produce three criticality levels without retraining.These variations apply only during evaluation; ego-agent training uses fixed scene parameters.Source: E. Criticality of Generated Scenario · p. 12
  • The method does not achieve the highest collision rate among baselines, but outperforms several training-based methods through structured reasoning rather than adversarial training or gradient optimization.The resulting risk is described as natural and interpretable, grounded in agent intent, spatial relationships, and scene semantics.Source: E. Criticality of Generated Scenario · pp. 12–13, Table 7 · p. 13

F. Failure Rate

The paper measures pipeline failure across prompt analysis, road retrieval, and agent planning, and describes six SafeBench scenarios used for training or criticality evaluation.

F. Failure Rate

  • Closed-source models achieve near-perfect success rates, while open-source models reach up to 93% across pipeline stages.Failure rates are computed across prompt analysis, road retrieval, and agent planning; each model is tested on 40 runs.Source: F. Failure Rate · p. 13
  • When parsing fails or unsupported options appear, the prompt is resubmitted to the LLM, which may add token usage and computational overhead.Such failures are reported as rare for both open- and closed-source models.Source: F. Failure Rate · p. 13

G. Details for Scenarios on SafeBench

  • The six scenarios are divided between ego-agent training and scene-criticality evaluation, with collision rate, route incompleteness, and driving score among the defined metrics.Driving score combines collision rate, rule violation, route following, smoothness, and speed.Source: G. Details for Scenarios on SafeBench · pp. 13–14
  • Straight-obstacle scenes place an agent suddenly ahead of the ego vehicle, while lane-changing scenes involve unexpected maneuvers from different directions.The ego agent should swerve or slow for an obstacle and proactively avoid dangerous lane changes.Source: G. Details for Scenarios on SafeBench · p. 14
  • Unprotected-left-turn scenes involve simultaneous intersection arrival and possible trajectory overlap; crossing negotiation scenes require timing a safe passage without traffic lights.The ego agent must yield or adjust movement in the former and negotiate when to proceed in the latter.Source: G. Details for Scenarios on SafeBench · p. 14
  • Red-light-running scenes pit the ego vehicle against a perpendicular rule violator, while right-turn scenes involve another vehicle entering the ego vehicle’s intended lane.The ego agent must anticipate the violation or complete its turn cautiously around surrounding traffic.Source: G. Details for Scenarios on SafeBench · p. 14

H. Prompts for Ablation Study

The ablation study uses prompts covering normal, critical, and conditional traffic situations to evaluate pipeline designs across scenario types.

Normal

  • Normal prompts range from ordinary straight driving to mixed vehicle actions at an intersection and a pedestrian blocking a left-turn destination.Examples include cars turning in different directions and a bus and truck approaching from different roads.Source: H. Prompts for Ablation Study · p. 14

Critical

  • Critical prompts describe a dangerous motorcycle turning across the ego vehicle, agents blocking or crossing ahead, and oncoming cars during a left turn..Source: H. Prompts for Ablation Study · p. 14

Conditional

  • Conditional prompts vary road context through an uncontrolled left turn, a signalized intersection with puddles, or a parallel open crosswalk.The prompts combine these conditions with other vehicles or a pedestrian crossing.Source: H. Prompts for Ablation Study · p. 14

I. Metrics Definition

The evaluation metrics quantify agent and road accuracy as well as diversity across generated scenes.

Agent Accuracy

  • Agent Accuracy measures correctly matched agents while accounting for missing and hallucinated agents.Matches use attributes such as type, position, and action; each predicted agent can be matched only once, and the denominator penalizes both over- and under-generation.Source: §I · p. 14

Road Accuracy

  • Road Accuracy measures the proportion of correctly predicted road properties, including structural or semantic attributes such as lane counts, signals, and static objects.A predicted property must match an unmatched ground-truth property, and normalization by max(m, n) penalizes missing and extra properties.Source: §I · pp. 14–15

Agent Diversity

  • Agent Diversity is the proportion of unique generated agent configurations, based on combinations of agent type, action, and relative position..Source: §I · p. 15

Road Diversity

  • Road Diversity is the proportion of unique roads selected for generation, with uniqueness determined by simulator-assigned road ID..Source: §I · p. 15

J. Limitation

The framework has two stated limitations: LLM language errors can affect planning, and CARLA’s built-in map constrains which signals, objects, and conditions can be retrieved.

J. Limitation

  • LLM planning may misread a blocking action or hallucinate unspecified elements such as traffic lights.The authors suggest more capable reasoning models or expert-guided fine-tuning as possible mitigations.Source: J. Limitation · p. 15
  • CARLA’s built-in map prevents retrieval of complex signal-object combinations and novel or unsupported conditions.Examples include a road allowing all directions without traffic lights or stop signs and a requested speed limit of “200.”Source: J. Limitation · p. 15
  • The authors plan to support custom OpenDRIVE code generation to enable more flexible scene customization.Source: J. Limitation · p. 15
Loading 2409.09575v4…