PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis

Yuan Gao, Mattia Piccinini, Yuchen Zhang, Dingrui Wang, Korbinian Moller, Roberto Brusnicki, Baha Zarrouki, Alessio Gambi, Jan Frederik Totz, Kai Storms, Steven Peters, Andrea Stocco, Bassam Alrifaee, Marco Pavone, Johannes Betz

arXiv:2506.11526v4cs.ROcs.AI

TL;DR

Autonomous-driving validation needs diverse, controllable scenarios, including rare safety-critical cases that real-world datasets often miss. This survey examines foundation models for scenario generation and analysis, finding them promising for improving the realism, diversity, scalability, and efficiency of scenario-based testing.

  • Problem

    Scenario-based testing requires systematic generation and analysis of diverse, controllable, and safety-critical driving situations, including rare corner cases often absent from real-world datasets.

  • Method

    The survey examines foundation models for autonomous-driving scenario generation and analysis, covering LLMs, VLMs, MLLMs, diffusion models, and world models.

  • Results

    Foundation models have emerged as promising tools for enhancing the realism, diversity, and scalability of scenario-based testing in autonomous driving.

  • Takeaways & Limitations

    Their ability to support both scenario generation and scenario analysis positions foundation models as enablers of robust and efficient autonomous-driving validation frameworks.

  • Takeaways & Limitations

    A persistent gap remains between simulation-based scenario generation and real-world validation, motivating alignment with practical safety standards such as SOTIF.

Abstract

from arXiv · show

For autonomous vehicles, safe navigation in complex environments depends on handling a broad range of diverse and rare driving scenarios. Simulation- and scenario-based testing have emerged as key approaches to development and validation of autonomous driving systems. Traditional scenario generation relies on rule-based systems, knowledge-driven models, and data-driven synthesis, often producing limited diversity and unrealistic safety-critical cases. With the emergence of foundation models, which represent a new generation of pre-trained, general-purpose AI models, developers can process heterogeneous inputs (e.g., natural language, sensor data, HD maps, and control actions), enabling the synthesis and interpretation of complex driving scenarios. In this paper, we conduct a survey about the application of foundation models for scenario generation and scenario analysis in autonomous driving (as of May 2025). Our survey presents a unified taxonomy that includes large language models, vision-language models, multimodal large language models, diffusion models, and world models for the generation and analysis of autonomous driving scenarios. In addition, we review the methodologies, open-source datasets, simulation platforms, and benchmark challenges, and we examine the evaluation metrics tailored explicitly to scenario generation and analysis. Finally, the survey concludes by highlighting the open challenges and research questions, and outlining promising future research directions. All reviewed papers are listed in a continuously maintained repository, which contains supplementary materials and is available at https://github.com/TUM-AVS/FM-for-Scenario-Generation-Analysis.

I. Introduction

Scenario-based simulation is central to validating autonomous driving safety, while foundation models broaden scenario generation and analysis across heterogeneous modalities and model families. This survey addresses a gap in prior reviews by systematically examining these applications.

  • I. Introduction: Simulation-based testing enables realistic, reproducible, controllable evaluation of safety-critical and rare autonomous-driving scenarios.Scenario generation creates diverse, controllable situations, while scenario analysis evaluates safety, risk, and behavior.
  • I. Introduction: The survey examines foundation models for autonomous-driving scenario generation and analysis using keyword-based literature searches covering peer-reviewed papers and preprints.Its review period extends from October 2022 to May 2025, with emphasis on autonomous driving, computer vision, machine learning, and robotics venues.
  • I. Introduction: The survey structure covers foundation-model concepts, language-based models, and their applications to autonomous-driving scenario generation and analysis.Its organization is summarized in Figure 3 and spans LLMs, VLMs, and MLLMs alongside broader foundation-model categories.
  • I. Introduction: Foundation models are general-purpose models trained on large-scale unlabeled data and adaptable across applications and modalities.Their development spans language, visual, multimodal, diffusion, and world-model families.
  • I. Introduction: Earlier autonomous-driving surveys covered broad model applications but did not explicitly focus on foundation-model roles in scenario generation or analysis.This survey aims to fill that gap through a dedicated review.

C. Scenario Generation in Autonomous Driving

Autonomous-driving scenarios appear in diverse annotated, sensor-based, map-based, simulated, and traffic-level formats that support testing across perception, planning, control, and analysis. Existing surveys organize classical methods and selected foundation-model work unevenly, leaving gaps in task, modality, and evaluation coverage.

  • C. Scenario Generation in Autonomous Driving: Scenario formats range from annotated sensor streams and multi-camera data to map layouts, simulated urban scenes, and traffic-level environments.These formats support scenario-based testing across different autonomous-driving components.
  • C. Scenario Generation in Autonomous Driving: Classical scenario-generation reviews cover knowledge-driven, data-driven, adversarial, optimization-based, extracted, and manually designed approaches across multiple abstraction levels.The reviewed methods include functional, logical, and concrete scenario representations.
  • C. Scenario Generation in Autonomous Driving: Existing foundation-model surveys provide limited scenario-generation coverage and often omit detailed techniques, controllability, or evaluation strategies.Related VLM, diffusion, and world-model reviews also leave model distinctions and task-specific evaluation insufficiently organized.
  • C. Scenario Generation in Autonomous Driving: Scenario analysis includes evaluation, scene understanding, risk assessment, anomaly detection, accident prediction, and identification of safety-critical situations.It supports robustness assessment and decision-making in simulation and real-world environments.
  • C. Scenario Generation in Autonomous Driving: The reviewed survey comparison distinguishes generation and analysis by scenario category, modality, datasets, models, techniques, and metrics.Examples include realism for generation and accuracy or language-generation quality for analysis.
  • C. Scenario Generation in Autonomous Driving: 345 papers are categorized into 93 on scenario generation, 56 on scenario analysis, 57 on datasets, 21 on simulators, 25 on benchmark challenges, and 93 on other topics.The categories cover the survey’s broader foundation-model and scenario-testing landscape.

E. Critical Summary

The survey frames foundation-model research for autonomous-driving scenarios as a broad but still incomplete field requiring structured coverage of models, methods, datasets, simulators, and evaluation. It also reviews LLM adaptation techniques and related scenario-analysis tasks.

  • E. Critical Summary: Prior reviews lacked a dedicated, comprehensive focus on foundation-model scenario generation and analysis in autonomous driving.The survey responds with a structured review intended to cover this gap.
  • E. Critical Summary: The survey classifies LLMs, VLMs, MLLMs, diffusion models, and world models alongside scenario categories, modalities, datasets, techniques, and metrics.It also reviews datasets, simulators, and benchmarking competitions.
  • E. Critical Summary: LLM-related scenario work spans generation, safety-critical cases, scene synthesis, policy evaluation, closed-loop simulation, ADAS testing, question answering, understanding, and evaluation.The reviewed LLM section covers both scenario creation and analysis.
  • E. Critical Summary: Prompt engineering guides pretrained language models through structured inputs without modifying their internal parameters.The surveyed techniques include contextual prompting, chain-of-thought, in-context learning, self-consistency, retrieval augmentation, fine-tuning, and multi-LLM systems.
  • E. Critical Summary: Fine-tuning adapts models using domain-specific data, while parameter-efficient methods update only a small portion of parameters and keep most of the model frozen.Full fine-tuning updates all parameters but requires substantial computation and has limited scalability.
  • E. Critical Summary: Multi-stage prompting and multi-LLM agent systems coordinate multiple interacting language models for complex tasks.Tooling frameworks can support modular, agent-based architectures beyond single-prompt interactions.

B. LLM-Based Scenario Generation

LLM-based scenario generation spans policy testing, closed-loop interaction, safety-critical cases, real-world replication, and ADAS test conversion. These approaches increasingly combine retrieval, prompting, simulation, and adaptive generation, but remain limited in generalizability and risk coverage.

  • Scope and taxonomy: LLM-based generation is organized by objectives including driving-policy testing, safety-critical scenarios, real-world replication, and ADAS test scenarios.The reviewed literature is classified into objective-specific categories, with representative studies summarized in Table 2.
  • Safety-critical scenario generation: Safety-critical methods synthesize rare trajectories and scene configurations to stress-test autonomous-driving robustness beyond nominal conditions.Examples use critical demonstrations, chain-of-thought, self-consistency, retrieval, and domain-specific scenario languages.
  • Limitations and future directions: Current methods often operate offline or cover limited risk types, constraining generalization to complex multi-agent contexts.The survey identifies interactive generation, stronger simulation safety verification, and VLM-based plausibility assessment as future directions.
  • Real-world scenario replication: Real-world replication combines crash reports, recorded datasets, retrieval, and map grounding to reconstruct realistic executable scenarios.LCTGen converts crash reports into structured descriptions, retrieves relevant maps, and generates scenarios for simulation.
  • Closed-loop scenario generation: Closed-loop generation validates multi-agent interactions and ego-reactive behaviors, with ProSim enabling controllable rollouts and LLM-Attacker targeting adversarial testing.These systems use prompts, fine-tuned policy tokens, and simulator rollouts to generate interactive behaviors.

C. LLM-based Scenario Analysis

LLM-based scenario analysis addresses the mismatch between structured driving representations and language-model inputs through question answering, scenario understanding, and evaluation. Prompt dependence, preprocessing overhead, and limited datasets remain central barriers to efficient deployment.

  • Analysis scope: LLM-based scenario analysis must bridge structured DSL, sensor, and simulator representations with models primarily designed for natural-language input.The survey organizes existing work around question answering, scenario understanding, and scenario evaluation.
  • Question answering: Driving-scenario QA datasets support interpretation of structured contexts for downstream trajectory planning and decision-making.One approach converts vectorized agent states, speeds, and distances into natural-language QA pairs using driving rules.
  • Scenario understanding: Scenario-understanding systems transform agent states, road layouts, traffic signals, and other structured data into captions or narratives for reasoning.SenseRAG combines VLM descriptions, structured databases, SQL retrieval, and chain-of-thought reasoning.
  • Scenario evaluation: LLMs are used to evaluate anomalies, realism, safety-criticality, and driving behavior from structured simulation data or scenario images converted into language.Existing evaluation frameworks commonly rely on chain-of-thought and in-context learning.
  • Limitations and future directions: Analysis remains constrained by token-heavy, manually crafted prompting and by the latency and complexity of converting LiDAR, images, and radar into text.Fine-tuning could reduce elaborate prompting, but large high-quality scenario QA datasets and evaluation benchmarks remain scarce.

B. VLM-based Scenario Generation

VLM-based scenario generation combines visual inputs with language to create safety-critical, realistic, regulatory, and tailored dataset scenarios. Reported systems show promise for photorealistic synthesis and perception robustness, while interactivity, realism, and hybrid training remain underdeveloped.

  • Taxonomy: VLM scenario-generation research is grouped into safety-critical generation, real-world replication, ADAS testing, and dataset generation.The taxonomy emphasizes joint use of visual content and language for scenario synthesis.
  • Safety-critical scenario generation: VLMs can identify abnormal or near-failure behaviors and generate targeted, interpretable safety-critical scenarios.CurricuVLM analyzes BEV images and task descriptions, then guides trajectory generation through curriculum learning and reinforcement learning.
  • Dataset generation: WEDGE synthesizes images depicting 16 diverse extreme-weather conditions, and detectors trained on them show improved performance on real-world data.The dataset includes manually annotated 2D bounding boxes and targets perception robustness in adverse conditions.
  • ADAS test scenario generation: VLMs ground regulatory descriptions, test specifications, and crash reports in visual content to produce executable, reproducible ADAS scenarios.TRACE extracts road and environmental details from multimodal crash reports and converts generated paths into simulator-compatible DSL scenarios.
  • Limitations and future directions: Current VLM approaches lack online interactive editing, real-world log replay, and sufficiently explored hybrid training with real and synthetic data.These gaps limit flexible refinement, realism, and coverage of rare events such as crashes, occlusions, and anomalies.

C. VLM-based Scenario Analysis

VLM-based scenario analysis covers visual question answering, scene tagging, captioning, motion prediction, and higher-level reasoning across autonomous-driving tasks. Benchmarks demonstrate strong high-level reasoning in some settings, but fine-grained spatial reasoning, traffic-rule grounding, and deployment efficiency remain open challenges.

  • Scope and benchmarks: VLM scenario-analysis research focuses on foundation models with cross-domain generalization for VQA, scene understanding, benchmarking, and risk assessment.The survey distinguishes foundation VLMs from narrower scenario-analysis models and organizes applications around these analysis capabilities.
  • Visual question answering: VQA datasets increasingly cover perception, prediction, planning, and reasoning using images, LiDAR, video, counterfactuals, and standardized distractors.DriveLMM-o1 adds step-by-step reasoning data, while AutoDrive-QA standardizes evaluation with plausible incorrect choices.
  • Limitations and future directions: Most VQA systems overlook traffic rules and driving conventions, while onboard deployment still requires lower inference latency and resource demands.Suggested directions include rule-aware QA, model compression, efficient prompting, and lightweight architectures.
  • Scene understanding: Scene understanding progresses from categorical tagging to captioning, motion-context extraction, and reasoning about agent interactions, intent, causality, and situational context.VLMs are applied to risk assessment, trajectory prediction, traffic-scene classification, and causal reasoning.
  • Evaluation findings: ViLA shows strong performance on high-level VQA tasks but struggles with fine-grained spatial reasoning.Evaluation covers lane counting, intersection detection, scene classification, and point-cloud quality assessment on MAPLM-QA.

D. Limitations and Future Directions

VLM-based scenario generation remains underexplored, while scenario analysis is split between task-specific systems with external vision modules and tailored VQA fine-tuning. Key gaps include limited standardization, insufficient benchmarks, and the need to assess end-to-end multimodal reasoning.

  • VLM-based Scenario Generation: VLM-based scenario generation remains underexplored for driving-policy training and closed-loop generation.VLMs could also serve as auxiliary analysis modules that improve interpretability, fidelity, and iterative scenario refinement.
  • VLM-based Scenario Analysis: VLM-based scenario analysis follows task-specific frameworks with external computer-vision modules or tailored VQA fine-tuning.External modules support functions such as 3D grounding and hazard detection, while VQA adapts models for task-oriented performance.
  • VLM-based Scenario Analysis: End-to-end VLMs are being investigated as a way to reduce system complexity while preserving or enhancing analytical performance.The open question is whether general-purpose pretrained VLMs can handle scenario analysis without object detectors, depth estimators, or 3D grounders.
  • VLM-based Scenario Analysis: Scenario analysis lacks benchmark datasets, consistent VQA annotation frameworks, and unified evaluation metrics tailored to autonomous driving.These gaps constrain the standardization of a complex analysis pipeline.

V. Multimodal Large Language Models (MLLMs)

MLLMs extend language models with multiple driving-relevant modalities and architectural components for alignment, fusion, and task adaptation. Their adaptation combines modality bridging with instruction tuning and either parameter-efficient or full fine-tuning.

  • Architecture: MLLMs integrate three or more modalities, while autonomous-driving systems add video, LiDAR, BEV maps, and HD semantic features for spatial and temporal grounding.These modalities support reasoning over complex scenes and multi-agent dynamics.
  • Architecture: MLLM architectures typically combine modality-specific encoders, projection modules, and task-specific training strategies.Projectors map modality features into the LLM embedding space, enabling cross-modal alignment.
  • Modality Alignment: Alignment and fusion components include linear or MLP projectors, cross-attention, Q-Formers, fusion transformers, and structure-aware encoders.These modules connect visual, spatial, temporal, and structured perception inputs to language-based reasoning.
  • Adaptation: Multimodal fine-tuning uses paired instruction data, with PEFT updating limited parameters and FFT updating all model parameters.PEFT can train modality-alignment modules while freezing the LLM, whereas FFT includes vision, spatial, and language components.
  • Adaptation: FFT typically yields the highest task-specific performance but is computationally intensive, motivating applications to smaller models such as Qwen2-0.5B.This trade-off makes parameter and model size important adaptation choices.

B. MLLM-based Scenario Generation

MLLM-based scenario generation currently covers safety-critical corner cases and ADAS testing, while scenario analysis spans multimodal VQA, scene understanding, and temporal scenario understanding. The survey identifies only two generation studies and several unresolved robustness and validation gaps.

  • Safety-critical Scenario Generation: Safety-critical generation combines videos, GPS traces, crash reports, and other heterogeneous inputs to reconstruct rare, high-risk driving situations.AutoScenario uses GPT-4o to create structured descriptions, generate SUMO road networks and CARLA behaviors, and refine outputs against real crash data.
  • ADAS Testing Scenario Generation: ADAS-testing generation derives executable and reproducible test cases from real-traffic videos using multimodal prompting and scenario conversion.LEADE uses key frames from the HDD dataset with GPT-4V to create abstract scenarios for function-level validation.
  • MLLM-based Scenario Analysis: MLLM scenario analysis covers VQA, static scene understanding, and temporal scenario understanding involving agent interactions and causal events.The surveyed datasets include general AD tasks, spatio-temporal reasoning, risk-aware reasoning, and multimodal LiDAR extensions.
  • Limitations: Current VQA datasets primarily use common scenarios, leaving robustness on rare events, unfamiliar objects, and difficult conditions insufficiently tested.The survey calls for benchmarks that explicitly evaluate out-of-distribution generalization.
  • Limitations: Existing scenario-understanding work still lacks validation in realistic settings and needs long-range temporal modeling, causal inference, and robust out-of-distribution handling.These needs distinguish temporal scenario understanding from static image-based scene understanding.

D. Limitations and Future Directions

The survey highlights major limitations in multimodal foundation models and diffusion-based scenario generation, including scarce AD-specific pretraining, limited generation coverage, multimodal alignment challenges, and fidelity gaps. Future directions emphasize richer datasets, retrieval, controllability, and more realistic simulation outputs.

  • MLLM Limitations and Directions: No pretrained MLLM is specifically designed for autonomous driving with complementary LiDAR, camera, and radar modalities.The survey therefore highlights the need for large-scale multimodal datasets and AD-tailored pretrained MLLMs.
  • MLLM Limitations and Directions: MLLM-based scenario generation remains limited to two studies, leaving policy evaluation, closed-loop generation, and complex event reconstruction largely unexplored.This indicates a substantial research gap in applying MLLMs to scenario synthesis.
  • MLLM Limitations and Directions: Multimodal retrieval-augmented generation could combine maps, annotated traffic videos, and LiDAR point clouds to provide richer scenario context.The proposed direction targets more diverse, realistic, and situation-aware generation pipelines.
  • MLLM Limitations and Directions: Current pretrained MLLMs struggle to align and process diverse multimodal inputs for complex driving scenario analysis, often requiring instruction tuning or adapter-based adaptation.The survey also identifies high-fidelity sensor simulation, interaction modeling, edge deployment, and human-machine collaboration as open opportunities.
  • Diffusion Models: Diffusion models generate scenarios through a forward noise-addition process followed by iterative reverse denoising, supporting controllable synthesis rather than analysis.The survey covers their use for traffic flow, road layouts, images, and videos.
  • Diffusion Models: Diffusion-based traffic-flow generators still depend on manually crafted controls, tuned objective weights, or predefined token and mask schemas.Adapting these systems to new constraints can require costly retraining or extensive fine-tuning.
  • Diffusion Models: Static-scene diffusion generators often omit traffic signs, signals, and lane markings and may place agents unrealistically on unseen road geometries.These limitations reduce map fidelity and generalization beyond the training corpus.
  • Diffusion Models: Generated driving images may omit visual corner cases and camera artifacts such as rolling-shutter distortion, lens flare, and sensor noise.The resulting photometric gap limits direct use of synthetic frames for training perception systems.

C. Limitations and Future Directions

Diffusion models still face rigid conditioning and limited physical plausibility, while language models could provide richer knowledge for controllable, realistic, and interpretable scenario generation.

  • Conditioning: Diffusion-model conditioning often depends on manual tuning, predefined schemas, or task-specific retraining, limiting flexibility and scalability.Future work targets generalizable conditioning that integrates diverse inputs without substantial architectural changes or retraining.
  • Physical plausibility: Generated trajectories and scenes can achieve statistical realism while lacking fine-grained physical plausibility.Reported issues include implausible inertial dynamics, unnatural agent reactions, and inadequate modeling of occlusions or causal dependencies.
  • Physical plausibility: Physics-informed models are proposed to improve adherence to real-world physical laws and overall output realism.The survey presents this as a future direction for addressing physical inconsistencies in diffusion-generated scenarios.
  • Language-model integration: LLMs could serve as embedded knowledge sources encoding physical dynamics, semantic scene structure, and normative driving behavior.This role extends beyond translating natural language into guidance signals and may improve controllability, realism, and interpretability in complex or ambiguous environments.

VII. World Models (WMs)

World models learn compressed spatial-temporal representations to predict future states and generate driving scenarios. Their development spans encoder-predictor architectures, latent-space dreaming, and increasingly hybrid multimodal and diffusion-based designs.

  • World-model foundations: World models learn compressed spatial and temporal environment representations for future-state prediction involving dynamic agents and static objects.The survey categorizes world-model scenario generation into visual, 3D occupancy, and multimodal generation.
  • Architecture and evolution: A typical architecture encodes multimodal inputs into latent z_t and predicts future latent representation ˜z_t+1 using past information and a control action.Early systems used VAEs for encoding and recurrent networks such as LSTMs or GRUs for temporal prediction.
  • Scenario-generation applications: Recent world models combine multimodal semantic understanding with diffusion-based generative backbones to produce coherent, controllable, and semantically grounded driving simulations.Examples integrate VLM or MLLM capabilities with diffusion generation and conditioning such as maps, layouts, text, or actions.
  • Architecture and evolution: World-model architectures have progressed from recurrent predictors to transformers, latent diffusion models, diffusion transformers, and video diffusion systems.These architectures support future prediction and generation within increasingly advanced world-model designs.
  • World-model functions: World-model research in autonomous driving serves two broad functions: future motion prediction and scenario generation.The survey focuses primarily on scenario generation and summarizes datasets and code availability in its comparison table.
  • Scenario generation with world-model dreaming: World-model dreaming generates new scenarios by autoregressively sampling learned latent variables without additional real-world inputs.The resulting combinations can follow learned physical and logical patterns while introducing conditions not seen during training.

C. Limitations and Future Directions

The survey identifies computational intensity and weak physical realism as major world-model limitations, and organizes evaluation around framework performance, content quality, and application-specific safety and controllability.

  • World-model limitations: World models for 3D occupancy generation remain computationally intensive despite promising volumetric prediction capabilities.Future work should develop lightweight architectures and finer-grained occupancy representations.
  • World-model limitations: Current implementations struggle with physics-consistent modeling of complex multi-agent interactions, vehicle dynamics, tire-road friction, collision forces, weather, and traffic rules.The limitation also applies to general-purpose world models that are not tailored to autonomous driving.
  • Evaluation framework: Evaluation metrics are organized into framework performance, content quality, and application-specific categories.The categories cover computational efficiency and reliability, generated or analyzed content, and domain-relevant scenario properties.
  • Framework performance metrics: Framework efficiency uses response time and token usage, while effectiveness uses compile error rate and execution success rate.These measures assess computational cost and whether generated code or scenarios can be instantiated and executed successfully.
  • Content quality metrics: Content-quality evaluation includes trajectory accuracy, semantic correctness, and language quality alongside modality-specific realism measures.Examples include mADE and mFDE, accuracy or F1, BLEU and CIDEr, distributional distances, FID, FVD, mIoU, and Chamfer distance.
  • Application-specific metrics: Application-specific metrics assess safety-criticality and controllability through measures such as collision rate, TTC, risk score, CLIP alignment, and traffic-flow compliance.These metrics target safety properties and adherence to user-specified constraints such as speed, waypoints, lane assignments, vehicle counts, and scene type.

B. Datasets

Datasets underpin foundation-model scenario generation and analysis through multimodal sensors, rich annotations, and varied traffic conditions. The survey compares their suitability alongside simulator usage, with CARLA the most frequently used simulator.

  • Dataset requirements: Foundation-model workflows reconstruct real-world scenarios from trajectory data, sensor streams, and rich annotations, with model families using different modality combinations.LLMs typically use trajectories, VLMs and MLLMs add LiDAR, images, or video, diffusion models generate future scenes, and world models predict scene evolution.
  • Dataset evaluation: The dataset comparison considers sensor coverage, annotation depth, scene diversity, and potential for controllable generation.These properties are treated as fundamental to generalization because foundation models require large, diverse, and annotated data.
  • Dataset characteristics: Waymo and nuScenes provide multimodal RGB, LiDAR, and RADAR data, while detailed object, lane, and trajectory annotations support generation and safety analysis.Their coverage includes both sensor modalities and semantic-geometric labels relevant to perception, prediction, map-conditioned generation, and safety tasks.
  • Simulation platforms: Among simulators used in the surveyed papers, CARLA was used 8 times, followed by MetaDrive 4, LGSVL 3, and SUMO 3.Impact is measured by usage in included papers rather than citation count.
  • Dataset characteristics: Traffic-condition metadata spans time of day, environment type, and congestion, capturing variation in visibility, traffic flow, road layout, and driving behavior.This variation supports evaluating autonomous-driving performance across diverse scenarios.
  • Dataset adoption: Waymo Open and nuScenes are widespread because of their real-world fidelity, multisensor coverage, and comprehensive annotations.The survey also notes emerging visual question-answering datasets for language-based scenario analysis.

C. Simulators

Simulators support safe, reproducible evaluation and large-scale scenario generation for foundation-model-based autonomous-driving research. The survey distinguishes platforms by backend, perception realism, scenario customization, map source, API and DSL support, highlighting complementary simulator capabilities.

  • Simulator role: Simulators enable safe, reproducible testing, large-scale scenario generation, structured benchmarking, training-data creation, self-supervised pre-training, and sim-to-real validation.Foundation-model-based generation can use LLMs, VLMs, and MLLMs through APIs or domain-specific languages.
  • Simulator criteria: Backend choice trades rendering and vehicle-dynamics fidelity against lightweight large-scale planning and decision-making simulation.UE4 and Unity support high-fidelity rendering, whereas SUMO and Nocturne suit datasets where rendering realism is less critical.
  • Simulator criteria: Realistic perception provides physics-based camera, LiDAR, or radar outputs for training vision-language, sensor-fusion, and multimodal world models.The relevant capability is sensor emulation rather than visual rendering alone.
  • Simulator criteria: Custom scenario support enables evaluation and data generation through varied traffic participants, routes, vehicle behaviors, densities, weather, lighting, and scene appearance.CARLA provides APIs for scripting complex multi-agent interactions and adjusting these parameters.
  • Simulator criteria: Map sources, APIs, and DSLs determine geographic realism, environmental control, pipeline integration, and human-readable scenario specification.OpenStreetMap supports geographically realistic traffic, while Python, ROS 2, C++, and DSL interfaces support generation, interaction, robotics testing, and deployment workflows.
  • Impactful platforms: CARLA and SUMO are highlighted as impactful FM-research simulators because their capabilities complement different scenario-generation and evaluation needs.SUMO supports large-scale, interaction-heavy traffic modeling with real-world map integration, while CARLA supports richer scene and interaction customization.

X. Future Directions

The survey identifies research directions for improving foundation-model-based scenario generation and analysis, spanning realism, rare events, multimodal data, evaluation, efficiency, safety validation, and compliance. It also frames foundation models as promising tools for more scalable and robust autonomous-driving testing while noting persistent practical challenges.

  • Improve Realism: Improving scenario realism requires domain knowledge and hybrid physics-based and data-driven foundation-model approaches.The survey also points to world-model dreaming for high-fidelity sensor simulation.
  • Create Rare Events: Rare-event generation needs targeted datasets and causal or counterfactual reasoning to produce uncommon, high-risk scenarios.These methods are proposed to improve model accuracy on infrequent but critical situations.
  • Create Multimodal Datasets: Large-scale multimodal datasets should combine sensor, map, traffic-rule, control-action, human-feedback, and textual data.The survey also calls for architectures and training methods tailored to multimodal fusion.
  • Develop Metrics and KPIs for Comparison: Standardized benchmarks and metrics are needed to compare scenario realism, controllability, diversity, and safety-criticality objectively.The survey recommends broad community adoption and conference competitions to encourage standardization.
  • Reduce Computational Demands: Model distillation, pruning, and quantization are proposed to reduce computational demands without sacrificing performance.Computational efficiency and scalability are identified as major practical constraints for scenario generation and analysis.
  • FMs as Safe Data Flywheels: Foundation models could serve as safe data flywheels linking generated scenarios with testing, retraining, safety assessment, and performance monitoring.Future workflows must preserve scenario representativeness, balance real and synthetic data, and quantify the safety impact of generated edge cases.
  • Regulatory Compliance: Future development must address bias mitigation, data privacy, interpretability, real-world alignment, and out-of-distribution scenarios.The survey also identifies fine-grained controllability, robust realism, costly inference, and generalization as unresolved challenges.
  • Conclusion: The survey presents LLMs, VLMs, MLLMs, diffusion models, and world models as promising tools for improving scenario-based testing.Their heterogeneous-data capabilities support both scenario generation and scenario analysis, although robust validation remains necessary.
Loading 2506.11526v4…