Source-linked AI summary
Guided Conditional Diffusion for Controllable Traffic Simulation
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, Marco Pavone
TL;DR
Realistic traffic models are difficult to control for targeted AV testing, while controllable models may sacrifice realism. CTG addresses this gap with dynamics-enforced conditional diffusion and inference-time STL guidance, outperforming strong baselines in the controllability-realism tradeoff across nuScenes evaluations.
Problem
Traffic simulation requires both realistic behavior for real-world transfer and controllability for constructing targeted AV-testing scenarios, but existing approaches do not provide both.
Method
CTG trains a conditional diffusion model with enforced vehicle dynamics and uses inference-time STL guidance to satisfy user-specified trajectory rules.
Results
Across nuScenes evaluations, CTG generally achieves lower rule violation, realism deviation, and failure rates than baselines, with the lowest rule violation in three of five settings.
Takeaways & Limitations
CTG demonstrates a controllability-realism tradeoff for diverse and composite traffic rules while maintaining physically feasible trajectory generation.
Abstract
from arXiv · showhide
Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Typical heuristic-based traffic models offer flexible control to make vehicles follow specific trajectories and traffic rules. On the other hand, data-driven approaches generate realistic and human-like behaviors, improving transfer from simulated to real-world traffic. However, to the best of our knowledge, no traffic model offers both controllability and realism. In this work, we develop a conditional diffusion model for controllable traffic generation (CTG) that allows users to control desired properties of trajectories at test time (e.g., reach a goal or follow a speed limit) while maintaining realism and physical feasibility through enforced dynamics. The key technical idea is to leverage recent advances from diffusion modeling and differentiable logic to guide generated trajectories to meet rules defined using signal temporal logic (STL). We further extend guidance to multi-agent settings and enable interaction-based rules like collision avoidance. CTG is extensively evaluated on the nuScenes dataset for diverse and composite rules, demonstrating improvement over strong baselines in terms of the controllability-realism tradeoff.
I. INTRODUCTION
Traffic simulation needs both realism for transfer to the real world and controllability for constructing targeted AV-testing scenarios. CTG combines conditional diffusion with inference-time STL guidance to generate realistic, physically feasible trajectories satisfying user objectives.
- Realistic traffic supports transfer from simulation to real-world driving, while controllability enables fine-grained scenarios for analyzing specific AV behavior.
- Existing simulators use replayed logs or heuristic controllers that provide control but can lack realistic, human-like interaction.
- Learned traffic models generate reasonable behaviors from driving data but lack mechanisms to control generated traffic toward novel objectives.
- CTG uses a conditional agent-centric diffusion model conditioned on roadmaps and neighboring agents, with enforced vehicle dynamics for physical feasibility.
- At inference time, STL guidance steers diffusion sampling toward user-specified objectives such as collision avoidance, goal reaching, and road rules.
II. RELATED WORK AND BACKGROUND
Prior traffic models span rule-based controllers, learned behavior models, and optimization-based scenario generation, but controllable diffusion offers a different inference-time approach. CTG uses STL robustness to guide multi-agent generation under contextual traffic information.
- Rule-based approaches use analytical models with fixed routes and separated longitudinal and lateral motions, limiting behavioral expressiveness.
- Learning-based approaches imitate real-world driving behavior from trajectory datasets, including scene-level simulation with TrafficSim.
- Adversarial scenario methods generate challenging AV interactions through latent-space search, Bayesian optimization, or reinforcement learning.
- Earlier controllable diffusion methods guide image, video, language, robot, or pedestrian generation, whereas CTG conditions traffic generation on maps and neighboring agents.
- STL formulas express spatio-temporal constraints over signals using predicates, logical connectives, and temporal operators.
- CTG uses STL robustness as an inference-time guidance function and extends guidance to multiple interacting agents.
III. CONTROLLABLE TRAFFIC GENERATION
CTG formulates controllable traffic generation as producing a future target-agent trajectory from contextual traffic information while balancing rule satisfaction with realism. The formulation includes vehicle dynamics and explicitly recognizes this tradeoff.
- Problem formulation: The target vehicle is represented by states including location, speed, and yaw, and actions consisting of acceleration and yaw rate.
- Problem formulation: Decision-relevant context combines an agent-centric semantic map with recent states of the target vehicle and neighboring agents.
- Problem formulation: A transition function f, implemented as a unicycle dynamics model, computes future vehicle states from previous states and controls.
- Problem formulation: The objective is to generate a realistic target-agent trajectory that maximizes rule satisfaction over the next T time steps.
- Problem formulation: Rule satisfaction and realism can conflict, so CTG trains a rule-agnostic model on real-world data and applies rule-specific guidance during inference.
B. Conditional Diffusion for Traffic Modeling
The model conditionally generates future action and state trajectories through iterative denoising, while known dynamics roll out physically feasible states from predicted actions.
- B. Conditional Diffusion for Traffic Modeling: CTG applies a conditional diffusion model iteratively from Gaussian noise to predict future trajectories of states and actions using context at every denoising step.The context includes agent-centric map information and past trajectories encoded before entering the diffusion model.
- B. Conditional Diffusion for Traffic Modeling: The model predicts only actions, then uses the known dynamics function f to roll out states from the initial state, ensuring physical feasibility throughout denoising.State trajectories are defined as τs = f(s0, τa).
- B. Conditional Diffusion for Traffic Modeling: The forward process gradually adds noise according to a fixed variance schedule, while the learned reverse process predicts denoised trajectories conditioned on context.The reverse process begins from sampled Gaussian noise and is applied iteratively.
- B. Conditional Diffusion for Traffic Modeling: Training samples clean trajectories and context from real-world driving data, corrupts the trajectory at a uniformly sampled diffusion step, and supervises denoising.The loss supervises both action and state trajectories, with state information improving generation quality.
- B. Conditional Diffusion for Traffic Modeling: At test time, each denoising step predicts the next-step distribution mean, which is perturbed according to the desired guidance function.The guidance function modifies the predicted mean rather than replacing the diffusion transition.
C. Guided Generation with Signal Temporal Logic (STL)
The guided sampler modifies conditional diffusion predictions using iterative, clipped gradients of STL-based robustness functions to satisfy user-specified trajectory rules.
- C. Guided Generation with Signal Temporal Logic (STL): CTG guides conditional diffusion by perturbing predicted transition means with gradients of a satisfaction-based objective.The guide J is constructed from the robustness of an STL formula.
- C. Guided Generation with Signal Temporal Logic (STL): Algorithm 1 initializes noisy trajectories, repeatedly denoises them with the conditional model, and applies inner gradient updates clipped by the diffusion noise scale.The algorithm uses encoder Fθ, transition function f, guide J, scale α, covariances Σk, and M inner updates.
- C. Guided Generation with Signal Temporal Logic (STL): Iterative inner optimization with clipping allows a controllability-realism tradeoff through the learning rate and number of optimization steps.This replaces the single-step gradient update used by Diffuser.
- C. Guided Generation with Signal Temporal Logic (STL): STL formulas analytically specify trajectory constraints, and their robustness values serve as guidance functions without training a separate classifier or reward model.The formulas can describe complex behavior despite relatively simple rule expressions.
- C. Guided Generation with Signal Temporal Logic (STL): The sampler guides several diffusion samples and filters them by selecting the trajectory with the best rule satisfaction after denoising.This selection procedure is called filtration.
IV. EXPERIMENTS
Experiments test whether CTG satisfies user-specified rules and preserves realism, comparing single-rule and multiple-rule performance against strong baselines.
- IV. EXPERIMENTS: The experiments evaluate controllable traffic generation by measuring user-specified rule satisfaction and realism relative to strong baselines.The study includes quantitative and qualitative comparisons in single-rule and multirule settings, followed by ablation.
- IV. EXPERIMENTS: Table I pairs each rule’s STL formula with its corresponding evaluation metric.The table defines the rule specifications used in the experiments.
- IV. EXPERIMENTS: Table II reports quantitative single-rule results and highlights the top two methods for each metric.The table is used to compare methods across rule-compliance and realism-related measures.
- IV. EXPERIMENTS: Table III reports quantitative results for the multiple-rules setting.This setting evaluates combined rule requirements rather than isolated rules.
A. Experimental Setup
The evaluation trains on nuScenes driving scenes and measures controllability, realism, and stability, with the study limited to vehicle simulation.
- A. Experimental Setup: nuScenes provides 5.5 hours of accurate trajectories across two cities with diverse scenarios and dense traffic.Models train on the train split and are evaluated on 100 randomly sampled validation scenes.
- A. Experimental Setup: The evaluation focuses on controllability, realism, and stability, including collision and off-road failure behavior.Stability is assessed through failure rate, defined as the average fraction of agents experiencing a critical failure in a scene.
- A. Experimental Setup: The study focuses only on vehicle simulation and defers pedestrians and cyclists to future work.This defines the evaluation’s agent-type scope.
- A. Experimental Setup: Realism metrics include acceleration magnitude, latitudinal acceleration magnitude, and jerk.These metrics quantify motion characteristics of generated trajectories.
- A. Experimental Setup: Baselines augment SimNet, TrafficSim, and BITS with test-time rule optimization or filtration using CTG’s guidance loss.Variants include optimization of output action trajectories, latent-space optimization, and sampling-rank adaptation.
B. Single Rule Evaluation
Single-rule evaluation tests five STL constraints, including target speed and goal waypoint, and finds CTG generally improves rule compliance, realism, and failure rate over baselines.
- Single-rule settings: The evaluation applies five STL rules: speed limit, target speed, goal waypoint, no collision, and no off-road.Goal waypoints are set 15 seconds along ground-truth trajectories, while target speed is set to 50% of the ground-truth speed.
- Quantitative results: CTG achieves lower rule violation, realism deviation, and failure rate than the baselines in general.Across five settings, CTG has the lowest rule violation in three and is competitive in the others; it is usually among the top two for realism deviation and failure rate.
- Qualitative comparison: For target speed, CTG has the lowest rule violation while producing more realistic trajectories than TrafficSim+opt and BC+opt.These are identified as the strongest baselines for the target-speed comparison.
- Qualitative comparison: For the waypoint example, BC+opt achieves better rule satisfaction, but its curvy trajectories and those of BITS+opt produce multiple collisions.The figure reports rule violations in m/s for target speed and meters for goal waypoint.
C. Multiple Rules Evaluation
Multiple-rule experiments evaluate composite traffic constraints, including stop-sign behavior, off-road avoidance, waypoint reaching, and target-speed tracking, with CTG variants balancing compliance and realism.
- Multiple-rule settings: Composite rules test stopping at stop signs while avoiding off-road driving, and reaching waypoints while following specified target speeds.Stop regions are 20×20 m boxes positioned 5 seconds along ground-truth trajectories; waypoints are positioned 10 seconds along them.
- Results: CTG variants achieve the top two lowest rule violation and realism deviation in both multiple-rule settings, with only slightly higher failure rates.The comparison results are reported in Table III.
- Ablation context: The ablation table identifies dynamics enforcement, filtration, guidance, action postoptimization, and inner optimization steps as CTG features under study.These features are abbreviated dyn, f, g, a, and op, respectively.
D. Ablation Study.
The ablation study shows that guidance is central to rule compliance, while combining or intensifying optimization can trade improved compliance for worse realism and failure rates.
- Guidance and filtration: Without guidance, rule violation increases greatly under the speed-limit setting.Filtration is more effective when paired with guidance than when used alone because it selects among guided samples with differing rule satisfaction.
- Optimization variants: Replacing guidance with action optimization worsens all metrics, while combining both reduces rule violation at the cost of higher realism deviation and failure rate.The comparison evaluates an additional output action optimization component.
- Optimization variants: More inner optimization steps improve rule violation but worsen realism and failure rate.This result exposes a controllability-realism-stability trade-off within guidance.
- State and dynamics design: Supervising only action trajectories gives more faithful accelerations and lower realism deviation, but fails to preserve the other benefits described for the full design.The supplied passage ends before specifying the remaining effects of this variant and dynamics removal.
V. CONCLUSION
The conclusion presents CTG as a conditional diffusion model for controllable traffic simulation and identifies broader agent coverage and long-term robust simulation as future directions.
- Conclusion: CTG is proposed for controllable traffic simulation and is positioned as opening several future research directions.The conclusion frames the model as the paper’s central proposal.
- Future directions: Current work models vehicles only, leaving cyclists and pedestrians as important agents for future AV-interaction simulation.The scope boundary is stated explicitly in the conclusion.
- Future directions: Collision and off-road guidance are identified as an important application for very long-term, robust traffic simulation.The conclusion also suggests that the guidance framework may extend beyond AV applications to learned models that must react online to novel objectives.