Source-linked AI summary
Language Conditioned Traffic Generation
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, Philipp Kraehenbuehl
TL;DR
Driving simulators need realistic, scalable, and interesting traffic scenarios, but language-conditioned generation lacks shared representations and paired language-traffic data. LCTGen combines an LLM Interpreter with a Transformer Generator to create traffic from descriptions; it reports stronger realism and controllability than prior work, with applications in editing and policy evaluation.
Problem
Simulators need realistic, scalable, and interesting traffic, while language-conditioned traffic generation lacks a shared language-traffic representation and paired datasets.
Source: §1 · p. 1Method
LCTGen uses an LLM-powered Interpreter to structure a language query and retrieve a matching map, then a query-based Transformer Generator produces the traffic scenario.
Source: §1 · p. 1Results
LCTGen exhibits superior realism and controllability over prior work, and human evaluators preferred its outputs to each baseline at least 90% of the time.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
LCTGen is also applied to instructional traffic scenario editing and controllable self-driving policy evaluation.
Source: §6 · p. 8The Interpreter cannot produce perfect agent placements and trajectories because it lacks direct access to detailed lane information from the map.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Simulation forms the backbone of modern self-driving development. Simulators help develop, test, and improve driving systems without putting humans, vehicles, or their environment at risk. However, simulators face a major challenge: They rely on realistic, scalable, yet interesting content. While recent advances in rendering and scene reconstruction make great strides in creating static scene assets, modeling their layout, dynamics, and behaviors remains challenging. In this work, we turn to language as a source of supervision for dynamic traffic scene generation. Our model, LCTGen, combines a large language model with a transformer-based decoder architecture that selects likely map locations from a dataset of maps, and produces an initial traffic distribution, as well as the dynamics of each vehicle. LCTGen outperforms prior work in both unconditional and conditional traffic scene generation in terms of realism and fidelity. Code and video will be available at https://ariostgx.github.io/lctgen.
1 Introduction
Driving simulators need realistic, scalable, and interesting traffic scenarios. LCTGen uses language, an LLM-based Interpreter, and a query-based Transformer Generator to create language-conditioned traffic, with reported gains in realism and controllability over prior work.
1 Introduction
- Driving simulators need realistic, scalable scenarios that also produce interesting traffic situations for testing self-driving agents.Source: §1 · p. 1
- LCTGen takes a natural-language traffic description and outputs actors’ initial states and motions on a compatible map, including scenarios described in detailed crash reports.Source: §1 · p. 1
- LCTGen uses an LLM-powered Interpreter to structure user queries and retrieve a matching map, then a query-based Transformer Generator creates the specified traffic scenario.The Generator produces the full scenario in a single pass.Source: §1 · p. 1
- LCTGen is introduced as a language-conditioned traffic-generation model and a way to use LLMs despite the absence of paired language-scene data.Source: §1 · p. 1
- LCTGen exhibits superior realism and controllability over prior work and supports instructional traffic editing and controllable self-driving policy evaluation.Source: §1 · p. 2
2 Related Work
Prior traffic generation approaches use expert-defined rules or learn from real-world data, but the described learning-based methods lack user-preference control. LCTGen applies language conditioning without paired text-traffic training data.
2 Related Work
- Rule-based traffic generation can lack realism relative to real-world traffic distributions and requires substantial expert effort to scale [8] [11] [12] [13].LCTGen instead learns real-world traffic distributions and uses language descriptions to reduce the requirement for human experts.Source: §2 · p. 2
- SceneGen [14], TrafficGen [15], and BITS [16] learn realistic traffic generation from real-world data, but lack control toward users’ preferences.LCTGen adds natural-language controllability and is reported to outperform prior work in unconditional reconstruction.Source: §2 · p. 2
- Text-conditioned generation spans image [17], audio [18], motion [19], and 3D object [20] creation, with these methods typically requiring large-scale paired content-text data.LCTGen uses an LLM and structured representation for text-conditioned traffic generation without text-traffic paired data.Source: §2 · p. 2
- The paper adapts GPT-4 with in-context learning and chain-of-thought prompting [26] as its Interpreter.Source: §2 · p. 2
3 Preliminaries
The preliminaries define a traffic scenario as a map and a sequence of vehicle states, then describe the lane-based map representation and vehicle state elements used in scenes.
3 Preliminaries
- A traffic scenario τ consists of a map region m and vehicle states across T timesteps, s1:T.Source: §3 · p. 2
- Each map region is represented by lane segments carrying endpoints, lane type, and traffic-light control state.Source: §3 · p. 3
- Each vehicle is modeled by position, heading, velocity, and size, with the vehicle centered in the first frame designated the ego-vehicle.The ego-vehicle represents the self-driving vehicle in simulation platforms.Source: §3 · p. 3
4 LCTGen: Language-Conditioned Traffic Generation
LCTGen combines language interpretation, map retrieval, and traffic generation to produce scenarios from text and map data. Its components are stochastic, and only the Generator is trained on real-world scenarios.
4 LCTGen: Language-Conditioned Traffic Generation
- LCTGen samples a map region matching the interpreted description, then generates a traffic scenario from that map and structured representation.The system is trained using a real-world scenario-only driving dataset.Source: §4 · p. 3
4.1 Interpreter
- GPT-4 translates each traffic description into a YAML-like structured representation using in-context learning and Chain-of-Thought prompting.The representation encodes map properties and each agent’s position relative to the ego vehicle, orientation, speed range, and action.Source: §4.1 · p. 3
4.2 Retrieval
- Map retrieval ranks preprocessed regions by distance from the requested map representation and randomly samples from the top-K closest regions.Regions are centered on automated-vehicle locations in offline driving traces.Source: §4.2 · pp. 3–4
4.3 Generator
- The Generator uses a map encoder, agent-query generator, generative transformer, and scene decoder to model agent-agent and agent-map interactions.The map encoder uses multi-context gating blocks to fuse lane information; the transformer processes agent queries against map features.Source: §4.3 · p. 4
- For each agent, the decoder predicts attributes with a Gaussian mixture model and outputs multiple possible future trajectories with probabilities.At inference, the model samples predicted position, attribute, and motion distributions to form the final scenario.Source: §4.3 · p. 5
4.4 Training
- An Encoder converts real-world scenarios into structured agent codes, allowing the Generator to train by reconstructing the original scenarios.The objective combines position cross-entropy, attribute negative log-likelihood, and motion mean-squared error.Source: §4.4 · p. 5
5 Experiments
Experiments compare LCTGen with prior methods on scene realism and text alignment, then examine instructional editing and component ablations. The reported results favor LCTGen across these evaluations.
5 Experiments
- The Waymo Open Dataset supplies 68k training scenarios and 2.5k test scenarios for experiments on five-second traffic scenes.The scenes are represented at 10 fps, with up to 384 lanes and 32 vehicles.Source: §5 · p. 5
5.1 Scene Reconstruction Evaluation
- Conditioned LCTGen improves Position, Heading, and Speed MMD by >2× over TrafficGen and MotionCLIP, while motion mADE and mFDE are 7–8x smaller than baselines.The table evaluates traffic scenario generation realism, with lower values better.Source: Table 1 · p. 6, §5.1 · p. 6
5.2 Language-conditioned Simulation Evaluation
- The text-alignment evaluation uses 38 Crash Report cases and 40 Attribute Description cases, with descriptions covering scenario attributes and combinations of two to four attributes.Examples pair each input text with its generated scenario; Figure 4 shows Crash Report examples on the left and Attribute Description examples on the right.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
- In human A/B tests, evaluators chose LCTGen at least 90% of the time and consistently gave it higher text-matching scores than other methods.The study used 12 evaluators, who contributed 1,872 scores for each model.Source: §5.2 · p. 7, Table 2 · p. 7
5.3 Application: Instructional Traffic Scenario Editing
- LCTGen edits real or generated scenarios from instructions, supporting vehicle removal, addition, and action changes, and is also applied to controllable self-driving policy evaluation.The Figure 5 example shows consecutive edits to a real-world scenario.Source: §5.3 · p. 8, Figure 5 · p. 8
5.4 Ablation study
- Masking quadrant index, distance, or orientation from the structured representation significantly degrades scene initialization, especially Position and Heading prediction.Table 3 reports the scene reconstruction ablation on Waymo Open Dataset.Source: §5.4 · p. 8, Table 3 · p. 8
- Masking speed range or action description significantly degrades motion metrics, and using TrafficGen for motion after LCTGen initialization performs worse than LCTGen’s one-shot generation.The authors report that the end-to-end design can lead to better performance.Source: §5.4 · p. 8, Table 3 · p. 8
6 Conclusion
This section summarizes LCTGen’s language-conditioned traffic generation and controllable evaluation capabilities, notes a limitation related to lane information, and describes instructional editing of traffic scenarios.
6 Conclusion
- LCTGen is a first-of-its-kind language-conditioned traffic generation method whose realism exceeds prior state-of-the-art methods, with applications in scenario editing and driving-policy evaluation.The conclusion also describes its generated scenarios as realistic and interesting.Source: §6 · p. 8
- The Interpreter cannot reliably produce perfect agent placements and trajectories because it lacks detailed lane information; future work proposes map and math APIs to retrieve precise map data.The proposed APIs are intended to support more comprehensive traffic scenarios.Source: §6 · p. 9
A.1 Structured representation details
- The structured representation encodes maps with 6 integer dimensions and each agent with 8, including relative position, distance, orientation, speed, and actions over the next four seconds.Map dimensions cover lane counts, distance to the nearest intersection, and ego lane; agent actions are discretized at one-second intervals.Source: §A.1 · p. 12
A.3 Instructional editing prompts
- For instructional editing, the Interpreter uses a 5-meter agent-distance interval instead of 20 meters to keep unedited agents in the same region before and after edits.Source: §A.3 · p. 14
- The editing prompt treats V1 as the ego vehicle and asks for a modified fixed-form scenario based on the scenario, map description, and natural-language instruction.It first identifies which vehicle or vehicles the instruction targets.Source: §A.3 · p. 15
- For an instruction to move the vehicle behind the ego vehicle to the opposite lane and accelerate, the example modifies V2’s direction, position, and speed.V2 is assigned the opposite direction and left-back position, with speed set to 10 (25 m/s).Source: §A.3 · p. 16
- For an instruction to remove vehicles ahead of the ego car that travel in the same direction, the example removes V3, V4, and V6.The example distinguishes the vehicles ahead from those moving in the ego vehicle’s direction before removing them.Source: §A.3 · p. 16
B.1 Training objectives
The appendix specifies how LCTGen’s training losses supervise lane placement, agent attributes, and future motion, then describes the MotionCLIP baseline and the MMD metric.
B.1 Training objectives
- Lane-position predictions use cross-entropy against each ground-truth agent’s lane-segment id.The target id is the lane segment occupied by that agent.Source: §B.1 · p. 16
- Agent heading, velocity, size, and position-shift attributes use negative log-likelihood under predicted Gaussian mixture models.The GMM parameters are predicted by an MLP for each attribute.Source: §B.1 · p. 17
- For motion, the model predicts multiple future trajectories and uses the closest prediction, encouraging its probability while reducing its distance from ground truth.The motion loss sums across predicted agents.Source: §B.1 · p. 17
C.1 Baseline implementation
- MotionCLIP aligns scenario and text embeddings, and its training pairs are made by converting real scenarios into attribute-based descriptions.At inference, its text encoder supplies a latent code that the decoder uses with the map to generate a scenario.Source: §C.1 · pp. 17–18
C.2 Metric
- MMD measures the distance between generated and real distributions, using a Gaussian kernel and computing differences per attribute.The metric passage describes comparisons for each real/generated scenario pair.Source: §C.2 · p. 18
C.3 Dataset
The dataset appendix describes crash-report and attribute-based text inputs, then outlines a human evaluation of how well generated scenarios match those inputs.
C.3 Dataset
- The crash-report set contains 38 CIREN cases, each with a long scenario description and a PDF diagram shown to evaluators as a reference.The diagrams supplement the lengthy texts during the human study.Source: §C.3 · p. 18
- Attribute descriptions cover traffic sparsity, vehicle position, speed, and ego-vehicle motion, with sentences for individual attributes and combinations of two to four.The appendix reports 40 cases created for human evaluation.Source: §C.3 · p. 19
C.4 Human study
- Twelve evaluators each assessed all 78 cases, with the study taking about 80 minutes per evaluator on average.Evaluators were not told how the models performed on the two datasets.Source: §C.4 · p. 20
C.5 Qualitative result full texts
The appendix supplies full input texts for qualitative crash-report examples; the figure summaries shorten those inputs because of space constraints.
C.5 Qualitative result full texts
- The displayed summaries for five Crash Report examples in Figures 4 and A2 are shortened versions of the full input texts.The examples are described as model outputs, while the accompanying texts are summaries from the Interpreter.Source: §C.5 · p. 20
- One full input describes a signalized, multi-lane intersection crash in which a left-turning vehicle strikes an oncoming car.The account includes the vehicles’ post-impact rotation and final positions in the intersection.Source: §C.5 · p. 20
- Another input describes a construction-zone rear-end crash after a vehicle stopped in the travel lane to collect debris.The following driver braked and steered shortly before impact.Source: §C.5 · p. 20
- Other full inputs include a head-on offset collision after a vehicle crossed into oncoming traffic and an intersection collision involving a vehicle proceeding from a stop sign.The latter account says both vehicles moved off the intersection’s northwest corner after impact.Source: §C.5 · pp. 20–21
D.1 Controllable self-driving policy evaluation
The appendix covers controllable self-driving policy evaluation and qualitative text-conditioned simulation examples from two datasets.
D.1 Controllable self-driving policy evaluation
- The section frames the evaluation as testing controllability in self-driving policies.The table is captioned as a controllable self-driving policy evaluation.Source: §D.1 · p. 21, Table A1 · p. 21
- Table A1 organizes policy evaluation around real test data, Crash Report generation, and LCTGen variants labeled sparse, medium, and dense.The supplied passages identify these comparison categories but do not report table values.Source: §D.1 · p. 21, Table A1 · p. 21
D.2 Text-conditioned simulation qualitative results
- Qualitative text-conditioned simulation examples include three Crash Report cases and three Attribute Description cases.The appendix points to Figure A2 for these examples.Source: §D.2 · p. 21
D.3 Human study statistics
Human evaluations favor LCTGen in scenario-text alignment, while the appendix also presents Interpreter input-output examples that translate crash descriptions into structured scenario representations.
D.3 Human study statistics
- Human evaluators assign LCTGen significantly better scores across the two datasets.Figure A3 presents the score distributions.Source: §D.3 · p. 21, Figure A3 · p. 22
- LCTGen is chosen as the better text match at least 90% of the time and consistently receives higher evaluator matching scores than other methods.The preference score measures how often LCTGen is selected over each baseline.Source: §D.3 · p. 23
- Average per-case evaluator score standard deviations are used to characterize subjective scoring differences, and the authors conclude their model outperforms compared methods with high confidence.The standard deviations are averaged across cases and reported as “Human Std.”Source: §D.3 · p. 23, Table A2 · p. 23
D.4 Interpreter input-output examples
- Interpreter examples show crash-report text summarized into a structured scenario description, including vehicles, their collision, and roadway context.One example describes Vehicle 1 striking the rear of Vehicle 2 at an urban intersection.Source: §D.4 · p. 24
D.5 Attribute Description Result Split
The Attribute Description results are split by attribute, with LCTGen showing nearly identical performance across them and different baselines leading on two attributes.
D.5 Attribute Description Result Split
- The analysis separates matching scores by attribute within the full Attribute Description dataset.Table A3 is identified as the human-study result split on Attribute Description scores.Source: Table A3 · p. 26, §D.5 · p. 26
- LCTGen has nearly identical matching-score performance across the evaluated attributes.The split analysis is reported for the Attribute Description dataset.Source: §D.5 · p. 26, Table A3 · p. 26
- TrafficGen achieves the best result for Density, while MotionCLIP performs best for Position.The table presents the attribute-level score split.Source: §D.5 · p. 26, Table A3 · p. 26
D.6 Full Ablation Study
The appendix provides full ablation results and additional instructional editing examples, including edits made directly from an input scenario.
D.6 Full Ablation Study
- Table A4 reports the full ablation study and adds variants using the learnable query xi and GMM prediction for attributes.The provided text names these variants but gives no result values.Source: §D.6 · p. 26, Table A4 · p. 26
D.7 Instructional traffic scenario editing
- Figure A5 shows additional instructional traffic scenario editing examples, with each example edited from its input scenario.The appendix distinguishes these from the compound editing in Figure 5 of the main paper.Source: §D.7 · p. 27, Figure A5 · p. 27
- The instructional editing examples are presented separately from the full ablation results.The supplied passage identifies Figure A5 as additional scenario-editing examples.Source: Table A4 · p. 26, §D.7 · p. 27