Source-linked AI summary
Language-Guided Traffic Simulation via Scene-Level Diffusion
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, Baishakhi Ray
TL;DR
CTG++ targets the difficulty of producing realistic, controllable traffic without requiring substantial domain expertise. It jointly models scenes with conditional diffusion and uses GPT4 to convert language queries into differentiable guidance losses, with evaluations reporting realistic, query-compliant generation and strong baseline comparisons.
Problem
Traffic simulators need realistic and controllable behavior, but existing learning-based control approaches require substantial domain expertise and lack an easy language interface.
Method
CTG++ jointly generates multi-agent traffic with a scene-level conditional diffusion model and uses GPT4 to translate user queries into differentiable losses for denoising guidance.
Results
CTG++ generates realistic trajectories that follow user-specified language commands and outperforms baselines across stability, rule satisfaction, and realism evaluations.
Takeaways & Limitations
The approach provides a language-guided interface for specifying traffic behaviors while retaining scene-level modeling of multi-agent interactions.
Takeaways & Limitations
CTG++ does not support complex commands involving many map interactions and lacks automatic detection and repair of erroneous GPT4-generated loss functions.
Abstract
from arXiv · showhide
Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. However, current approaches for controlling learning-based traffic models require significant domain expertise and are difficult for practitioners to use. To remedy this, we present CTG++, a scene-level conditional diffusion model that can be guided by language instructions. Developing this requires tackling two challenges: the need for a realistic and controllable traffic model backbone, and an effective method to interface with a traffic model using language. To address these challenges, we first propose a scene-level diffusion model equipped with a spatio-temporal transformer backbone, which generates realistic and controllable traffic. We then harness a large language model (LLM) to convert a user's query into a loss function, guiding the diffusion model towards query-compliant generation. Through comprehensive evaluation, we demonstrate the effectiveness of our proposed method in generating realistic, query-compliant traffic simulations.
1 Introduction
CTG++ addresses the need for realistic, controllable, and user-friendly traffic simulation by combining scene-level diffusion with language-guided loss functions. Its LLM interface converts user queries into losses that guide query-compliant trajectories.
- Motivation: Realistic, controllable traffic simulation supports safer and more transferable AV development by enabling targeted testing of specific behaviors.The introduction identifies realism, controllability, and usability as requirements, including dangerous scenarios such as vehicle collisions.
- Challenges: Existing simulators and data-driven methods lack either realistic multi-agent behavior, expressive control, or both.Rule-based and replay-based systems lack realism or expressiveness, while data-driven approaches primarily reflect training distributions.
- Challenges: Language is difficult to connect to traffic trajectories because abstract commands such as “traffic jam” or “following” must control low-level agent behavior.The paper identifies language grounding as a second challenge distinct from constructing the traffic model itself.
- CTG++: CTG++ uses a scene-level conditional diffusion model with a spatial-temporal transformer to model multi-agent interactions and generate controllable traffic.Alternating temporal and spatial attention captures interactions over time, while the model generates trajectories for the whole scene.
- CTG++: GPT4 converts a user query into a differentiable loss that guides CTG++ toward command-compliant trajectories.The loss provides an intermediate representation between natural-language intent and traffic generation.
- Evaluation: Evaluation on nuScenes reports language-command following, realistic trajectories, and superiority over state-of-the-art baselines in high-quality scenario generation.The contribution summary describes both language compliance and realistic generation, alongside broad baseline comparisons.
2 Related Work
Prior traffic simulators offer either controllability or realism, while diffusion guidance can require expert-designed objectives. CTG++ instead uses an LLM to translate natural-language properties into loss functions for jointly controlled traffic.
- Traffic simulation: Rule-based traffic simulation provides user-friendly route control but limited behavioral expressiveness, whereas learning-based methods aim to imitate authentic driving data.The related work divides traffic simulation into rule-based and learning-based approaches with different realism-control trade-offs.
- Diffusion models: Conditional diffusion methods guide generation with trained classifiers or expert-designed losses to enforce user-desired properties.These approaches apply guidance during denoising to steer generated trajectories.
- Diffusion models: CTG manually specifies Signal Temporal Logic losses, so adapting it to each new property requires domain expertise.The paper contrasts this manual objective construction with its language-based alternative.
- Large language models: LLM-based robotics systems use high-level function libraries to map language instructions into executable behavior plans.The approach motivates using an intermediate interface rather than direct text-to-traffic training.
3 Methodology
CTG++ formulates traffic generation as query-satisfying imitation learning and jointly denoises all agents with a scene-level spatial-temporal transformer. During inference, GPT4-generated differentiable losses guide denoising toward the requested behavior.
- Pipeline: The training pipeline learns a scene-level conditional diffusion model from real driving data, then uses query-derived losses during inference.The model captures diverse behaviors during training and applies language-derived guidance at generation time.
- Problem formulation: Traffic generation conditions on maps and recent states, then produces future multi-agent trajectories that maximize a user-derived rule-satisfaction function.The formulation includes a dynamics transition, decision context, and a reward-like function over future states and actions.
- Scene-level diffusion: Unlike single-agent diffusion, CTG++ jointly models all traffic participants to capture significant interactions within a scene.Scene-level sampling predicts outcomes for all agents simultaneously rather than independently.
- Trajectory representation: The model predicts action trajectories and rolls them through known dynamics to obtain physically feasible state trajectories.State trajectories are consistently defined as the result of applying the dynamics function to the predicted actions.
- Scene-level spatial-temporal transformer: Its spatial-temporal transformer alternates temporal modeling with spatial interaction modeling, using agent-centric coordinates and relative information between agents.Agent-centric coordinates improve robustness to scene transformations, while spatial attention restores relative interaction information.
- Scene-level spatial-temporal transformer: At inference, future relative information is estimated with a constant-velocity model before being incorporated into spatial attention.Ground-truth relative information is available during training but unavailable for future inference steps.
- Language-guided generation: The final predicted action trajectory is converted through dynamics into the generated state trajectory after iterative differentiable guidance.This connects the language-derived objective to the model’s trajectory output.
- Language-guided generation: GPT4 receives trajectory-manipulation helpers, query-to-loss examples, and the user query, then returns a guidance function used during denoising.The generated loss perturbs the predicted diffusion mean through its gradient, steering trajectories toward the requested property.
4 Experiments
Experiments evaluate CTG++ on nuScenes using query-compliance, stability, controllability, and realism measures, alongside qualitative case studies and feature ablations. CTG++ generates query-compliant trajectories while maintaining realistic, stable multi-agent behavior and outperforms or improves upon the compared baselines across the reported evaluations.
- Experimental Setup: CTG++ is evaluated on 100 nuScenes validation scenes using GPT-generated and STL rules, with stability, controllability, and realism as evaluation dimensions.The baselines include CTG, BITS, and BITS+opt.
- Case Study of Language Interface: CTG++ generates trajectories satisfying distance-following and collision queries while preserving smooth motion and avoiding unintended collisions or off-road incidents.The two case studies cover vehicle following and deliberate collision generation.
- Evaluation of Traffic Model: CTG++ secures the lowest failure rate and scene-level realism deviation in 7 of 8 rule settings, balancing stability, rule satisfaction, and realism.The evaluation includes two GPT-generated rules and six STL rules.
- Evaluation of Traffic Model: Compared with CTG, CTG++ produces more realistic rule-compliant trajectories with fewer collisions and off-road incidents.The qualitative comparison uses the same language-generated loss functions and scenes.
- Ablation Study: Removing edge information increases failure rates by reducing interaction awareness, while scene-centric coordinates inflate failure rates and realism deviations during rollout.Without edge information, the model attends arbitrarily and can collide with the vehicle ahead.
5 Conclusion
CTG++ combines an LLM-based language interface with a scene-level conditional diffusion model to generate realistic, query-compliant traffic. The paper reports effective language-guided simulation, while identifying support for complex map interactions, automatic loss-function repair, and generation speed as limitations.
- Conclusion: CTG++ uses an LLM to translate user queries into differentiable loss functions that guide a scene-level conditional diffusion model.The diffusion model uses a spatial-temporal transformer architecture to generate realistic, query-compliant trajectories.
- Conclusion: Extensive evaluation demonstrates CTG++’s effectiveness in generating realistic, query-compliant traffic simulations.
- Limitations and Future Work: CTG++ does not support complex commands involving many interactions with maps, and its GPT-4-generated loss functions lack automatic error detection and repair.The paper also reports that generating each simulated scenario takes about one minute.
A Algorithm of Training and Sampling in Details
The paper follows the training and sampling procedures from prior work and provides detailed algorithms for both processes.
- Algorithm of Training and Sampling in Details: CTG++ largely follows prior training and sampling procedures, with detailed algorithms provided for each.
A.1 Training
CTG++ trains a scene-level conditional diffusion model on real-world trajectories and guides denoising with an objective selected during inference. Its scene-level formulation jointly predicts agents and uses iterative gradient-based guidance for rule satisfaction.
- Training: Scene-level trajectory sampling jointly predicts all agents, unlike CTG’s agent-level sampling, to capture interactions inherently.The model samples trajectories from real-world driving data during training.
- Training: Training corrupts action trajectories at a randomly selected diffusion step and predicts the original clean trajectory from noisy input.The simplified objective updates the model using the discrepancy between ground-truth and predicted clean trajectories.
- Training: CTG++ uses a cosine variance schedule with K = 100 diffusion steps.
- Sampling: During guided sampling, gradients of a specified objective perturb predicted means, with updates clipped before continuing denoising.The algorithm executes the first l actions after denoising and replans iteratively.
- Sampling: The sampler guides several diffusion samples and selects the one with the best rule satisfaction according to J.
B.1 Detailed Architecture
The architecture combines temporal and social attention with gated environmental-feature fusion to model multi-agent traffic during denoising. Its inputs encode agent histories and predicted futures, while attention updates representations across time and neighboring agents.
- Architecture: Figure A1 presents the test-time denoising data flow, including position encoding, rFFN, and the explicit guidance module.
- Temporal Attention: The model projects each agent-timestep element from dimension ds + da into latent dimension dh before temporal attention.The denoising step is injected through sinusoidal positional encoding.
- Gated Attention: A gated attention block fuses environmental features with central-agent features, giving feature updates additional control.
- Gated Attention: The social attention layer uses neighboring agents within a social radius and learnable matrices to form query, key, and value inputs.The neighborhood excludes the central agent itself.
C Qualitative Comparison under STL rules
Across qualitative STL-rule comparisons, CTG++ generally preserves realistic trajectories while satisfying rules, whereas baselines often introduce collisions, off-road behavior, or implausible paths.
- Overall Comparison: CTG++ generally generates realistic, rule-satisfying trajectories, while the strongest baseline often sacrifices realism, safety, or interaction quality.Observed baseline failures include curvy trajectories, off-road accidents, and collisions.
- Speed Limit: CTG++ achieves lower speed-limit rule violation than CTG, which also produces a collision between the blue and green vehicles.
- Target Speed: CTG slightly improves target-speed satisfaction but causes a collision with crossing vehicles and then goes off-road.
- No Collision: Both CTG++ and BITS+opt satisfy the no-collision rule perfectly, but BITS+opt uses highly curvy, unrealistic trajectories.
- No Off-Road: Under no off-road control, both methods satisfy the rule, but CTG produces multiple collisions involving the pink vehicle and stationary vehicles.
D.1 Training Hyperparameters
CTG++ training uses a fixed hardware and optimizer configuration, while selected evaluation scenes are filtered by motion, distance, and orientation criteria before query-based comparisons.
- Training Hyperparameters: CTG++ trains for 50K iterations in approximately 10 hours using an Intel i9 12900, NVIDIA GeForce RTX 3090, and Adam learning rate 1e-4.
- Pair Selection Criteria: Evaluation selects vehicle pairs with speeds above 2m/s and distances between 10m and 30m at 0s and 2s.
- Qualitative Comparison: In the stop-sign and no-off-road comparison, CTG++ satisfies both rules, whereas CTG satisfies only no-off-road and collides.
- Qualitative Comparison: For goal waypoint and target-speed rules, CTG++ satisfies both better than CTG, which ends with a collision between two orange vehicles.
- Pair Selection Criteria: Pair selection also requires orientation differences below 108 degrees for GPT collision and 36 degrees for GPT keep distance.
- Pair Selection Criteria: The criteria filter scenes likely to contain keep-distance or collision interactions, selecting the closest qualifying pair and skipping scenes without one.
- Pair Selection Criteria: After filtering 100 validation scenes, 50 remain for GPT collision and 40 for GPT keep distance.
E Experiment Details
CTG++ is evaluated with rule-violation metrics and language-interface examples that show how GPT-generated losses and helper APIs encode traffic constraints over trajectories.
- Evaluation Metrics: Rule-violation metrics are averaged over validation scenes, with smaller values indicating better rule satisfaction.The evaluation includes GPT-generated and STL rules such as keeping distance, avoiding collisions, respecting speed limits, and reaching waypoints.
- Evaluation Metrics: GPT Keep Distance measures a selected following vehicle’s average L2 distance deviation from a specified range.
- Evaluation Metrics: Collision, speed, off-road, waypoint, and stop-sign rules are evaluated using scene-level rates or deviations tailored to each constraint.The metrics include collision rate, speed deviation, off-road rate, waypoint distance deviation, and minimum speed in stop-sign regions.
- Language Interface: The language interface uses vehicle indices rather than illustrative colors, allowing users to specify controlled vehicles in queries.For example, a query can direct vehicle 1 to collide with vehicle 2.
- Language Interface: GPT4 receives API functions, helper functions, and paired examples to generate standardized differentiable loss classes over trajectory dimensions.The interface specifies trajectory tensor shape and the expected loss-class forward function, while examples demonstrate acceleration limits and relative vehicle positioning.
- Language Interface: The appendices provide a complete GPT Collision query-response example and four additional success examples using the same system messages.
F.3.1 Success Example: GPT Collision
This example shows GPT4 converting a collision-related query into a syntactically correct guidance-loss program that uses trajectory and coordinate-manipulation APIs. The resulting loss captures collision behavior and illustrates GPT4’s ability to generalize beyond the paired examples.
- Prompt and API setup: GPT4 receives a user query, loss-function API, helper-function API, and paired examples before returning an implemented loss function.The interface exposes trajectory tensors, coordinate transformations, lane projections, agent selection, and other helpers.
- Coordinate handling: The interface supports trajectory-coordinate transformations between agent and world frames, enabling relational constraints between selected vehicles.The APIs transform predictions into world coordinates and into another agent’s coordinate frame before selecting relevant agents.
- Generated program: The generated program follows the provided template and is syntactically correct.The paper identifies template adherence and syntactic correctness as properties of the returned program.
- Collision guidance: GPT4 interprets “collision” and integrates that concept with trajectory inputs through helper functions, despite collision being absent from the paired examples.The resulting loss encourages trajectory collision between two vehicles.
- Generalization across queries: GPT4 also demonstrates understanding of distance, orientation, multi-vehicle, and lane-following queries beyond the collision example.The examples report generated losses for keeping distance, matching direction, involving three vehicles, and following current lanes.
- Limitations: The language interface remains limited for complex vehicle–map interactions and can misunderstand concepts such as “cut in.”The paper gives rightmost-lane/intersection-turning and lateral cut-in behavior as failure cases, while suggesting additional map helpers and examples as possible extensions.
G Quantitative Evaluation Results with Additional Runs
Additional experiments compare CTG++ with CTG under GPT-generated and STL rules across three random-seed runs. Results are summarized using means and standard deviations, with wins highlighted only when statistically separated by the stated criterion.
- CTG++ is compared against CTG, identified as the strongest baseline, under GPT-generated and STL rules.
- The experiments use three distinct runs with varying random seeds and report the average and standard deviation.
- A method is highlighted as winning only when its value differs from the other by at least the sum of their standard deviations.