Source-linked AI summary
A Survey on Simulators for Testing Self-Driving Cars
Prabhjot Kaur, Samira Taghavi, Zhaofeng Tian, Weisong Shi
TL;DR
Physical road testing is costly and can jeopardize public safety, while simulation can supplement testing. The paper identifies simulator requirements and compares six widely used tools. It concludes that CARLA and LGSVL best balance virtual-environment realism and vehicle calculations, while noting unresolved challenges such as the lack of common evaluation standards.
Problem
Method
Results
Takeaways & Limitations
Simulation can help design self-driving vehicles safely, cost-effectively, and promptly, provided simulated scenarios represent reality.
Source: §8 · p. 7Common standards for evaluating simulation results are lacking, and mileage reports do not capture the sophistication and diversity of miles collected through simulation.
Source: §6 · p. 7
Abstract
from arXiv · showhide
A rigorous and comprehensive testing plays a key role in training self-driving cars to handle variety of situations that they are expected to see on public roads. The physical testing on public roads is unsafe, costly, and not always reproducible. This is where testing in simulation helps fill the gap, however, the problem with simulation testing is that it is only as good as the simulator used for testing and how representative the simulated scenarios are of the real environment. In this paper, we identify key requirements that a good simulator must have. Further, we provide a comparison of commonly used simulators. Our analysis shows that CARLA and LGSVL simulators are the current state-of-the-art simulators for end to end testing of self-driving cars for the reasons mentioned in this paper. Finally, we also present current challenges that simulation testing continues to face as we march towards building fully autonomous cars.
1 Introduction
Physical road testing entails infrastructure costs and can jeopardize public safety, while simulation supplements it; the paper identifies simulator requirements and compares six tools.
1 Introduction
- The sophistication and diversity of accumulated miles shape product maturity, and simulation enables reproducible, scalable testing of scenarios restricted on public roads for safety.Physical testing requires additional infrastructure and can jeopardize public safety.Source: §1 · p. 1
- The comparison covers MATLAB/Simulink, CarSim, PreScan, Gazebo, CARLA, and LGSVL for perception, localization, vehicle control, and dynamic 3D environments.The paper also identifies requirements for an ideal self-driving-car simulator.Source: §1 · p. 1
- CARLA and LGSVL best balance detailed virtual environments with precise vehicle calculations, while other simulators serve specialized needs.Gazebo requires time-consuming manual scene creation; MATLAB/Simulink offers analysis tools, and CarSim specializes in vehicle dynamics.Source: §1 · p. 1
2 Motivation and Background
As automotive software and hardware grow more complex, testing must expand in scale and scope, combining physical and simulation testing across vehicle systems and autonomous-driving functions.
2 Motivation and Background
- Growing software and hardware complexity adds demands for testing autonomous-driving algorithms and software alongside traditional vehicle functions.The paper cites a 2020 Volvo vehicle with about 100 million lines of code as an example.Source: §2 · p. 2
- Vehicle testing spans multiple layers, from integrated circuits to overall vehicle behavior, and relies on both physical and simulation testing.As vehicle complexity and functionality grow, testing complexity, scale, and scope grow as well.Source: §2 · p. 2
- Automotive simulators have evolved beyond vehicle dynamics to support autonomous-driving requirements such as perception, localization, mapping, control, and path planning.The paper says simulator requirements must extend from physical car models to sensors, planning, and control.Source: §2 · p. 2
3 Methodology
The paper organizes simulator requirements around autonomous-driving functions, the simulated driving environment, data generation, and general software qualities.
3 Methodology
- The requirements framework groups criteria by functional architecture, driving infrastructure, secondary data-collection tasks, and general simulator qualities.Requirements 1–4 address vehicle functions; 5–7 support simulated driving infrastructure; 8 supports data collection; and 9 covers generic requirements.Source: §3 · p. 2
- Functional requirements cover realistic sensor inputs for perception, camera calibration for localization, map and algorithm interfaces for planning, and vehicle dynamics and programmable control.The paper discusses perception and SLAM as well as path-planning algorithms and closed-loop vehicle control.Source: §3 · pp. 2–3
- A realistic 3D environment should include static and dynamic objects, realistic object behavior, varied terrain, and weather; scene detail matters especially for testing perception.The paper describes game engines and real-environment HD maps as approaches to creating virtual environments.Source: §3 · p. 3
- Traffic infrastructure and scenario tools should model traffic aids and support replaying real events or creating varied what-if situations and dynamic agents.Flexible APIs can configure agents, pedestrian behavior, crashes, weather, sensor types, and traffic signs.Source: §3 · p. 3
- The framework also calls for 2D/3D ground-truth labels and general qualities including documentation, stability, flexibility, portability, scalability, and open-source access.The supplied table caption identifies testing requirements organized around SAE automation levels.Source: Table 1 · p. 2, §3 · pp. 3–4
4 Simulators
The paper surveys six simulators and describes their respective tools, models, environments, and integration features.
4.1 MATLAB/Simulink
- MATLAB/Simulink supports ADAS and automated-driving design and testing, including imported maps, sensor visualization, ground-truth labeling, HIL testing, and code generation.Its toolbox includes examples for functions such as adaptive cruise control and automatic emergency braking.Source: §4.1 · p. 4
4.2 CarSim
- The newest CarSim version supports moving objects and sensors for ADAS and autonomous-vehicle simulations, including up to 200 objects with independent locations and motions in addition to the simulated vehicle.When combined with sensors, objects can serve as detectable targets. Users can control vehicle motion with built-in options or external models, and CarSim interfaces with MATLAB and LabVIEW.Source: §4.2 · p. 4
4.3 PreScan
- PreScan offers traffic generation and customizable sensors and control logic, plus HIL simulation, replayable recorded data, and a real-vehicle VeHIL setup.VeHIL combines a test vehicle on a roller bench with wheeled robots representing other vehicles.Source: §4.3 · pp. 4–5
4.4 CARLA
- CARLA is a flexible, open-source Unreal Engine simulator with a customizable API for training and validating ADAS systems and driving policies.It uses OpenDRIVE for roads and urban settings; recent versions improved vehicle volumes and core physics and automated traffic-light and stop-sign placement.Source: §4.4 · p. 5
4.5 Gazebo
- Gazebo provides configurable physics, rendering, communication, and plugins; users define worlds and models in SDF files, and commonly use it with ROS.Users can create objects from scratch or draw on community-shared models and documentation.Source: §4.5 · pp. 5–6
4.6 LGSVL
- LGSVL connects autonomous-driving stacks through ROS, ROS2, and Cyber RT bridges and provides Unity-based environments, traffic and sensor simulation, and vehicle dynamics.It supports configurable sensors, external vehicle-dynamics models through FMI, Python control, and HD-map export.Source: §4.6 · p. 6
5 Comparison
The comparison identifies distinct strengths across the reviewed tools, with CARLA and LGSVL favored for end-to-end testing while other simulators suit more specialized needs.
5 Comparison
- MATLAB/Simulink supports simple scenarios, efficient computation and plotting, and vehicle-model co-simulation, while PreScan adds realistic environments and weather simulation.PreScan also interfaces with MATLAB/Simulink to make modeling more efficient.Source: §5 · p. 6
- Gazebo offers flexibility, ROS integration, and extensible sensor models, but users must manually build and configure 3D scenes, costing time and effort.Scene objects and their physics and positions are defined in XML.Source: §5 · p. 6
- CARLA has a built-in recorder and automated Unreal/OpenDRIVE map generation, whereas LGSVL has no built-in recorder and uses manually imported Unity map components.Both simulators offer common sensors such as depth cameras, lidar, and IMU, and allow custom sensors.Source: §5 · p. 6
- LGSVL and CARLA are most suited to end-to-end testing of perception, mapping, localization, and vehicle control because they provide many built-in automated features.Gazebo's time- and effort-intensive dynamic-scene creation makes it a less preferred choice for end-to-end systems testing.Source: §5 · p. 7
6 Challenges
The challenges include missing simulation-evaluation standards, lack of known simulators for connected-vehicle testing, and potential bugs in evolving projects.
6 Challenges
- Common standards for evaluating simulation results are lacking, and mileage reports do not record the sophistication or diversity of simulated miles.The paper argues that standards would make comparisons between research efforts more informative.Source: §6 · p. 7
- The authors are unaware of simulators capable of testing connected vehicles, where vehicles communicate with one another and infrastructure.They note that test beds are available, including examples in a US Department of Transportation report.Source: §6 · p. 7
- Ongoing projects such as CARLA and LGSVL may expose users to undocumented errors or bugs as they add recent technologies.The paper identifies community support as important for improving open-source simulators and ADAS tests.Source: §6 · p. 7
7 Related Work
The related-work discussion identifies additional simulators, reviews, and a distributed simulation platform beyond the works explicitly reviewed.
7 Related Work
- RoadView, which models traffic scenes from image sequences and road GIS data, is among the simulators not explicitly reviewed.Source: §7 · p. 7
- An in-depth review discusses CARLA and its use for autonomous-driving algorithm testing.Source: §7 · p. 7
- Other reviews cover automotive and robotic simulators [13],, and, while [37] discusses a distributed simulation platform for testing.Source: §7 · p. 7
8 Conclusion Remarks
The paper compares six simulators across self-driving functions, identifies simulator requirements and five observations, and notes continuing challenges to simulation testing.
8 Conclusion Remarks
- The paper compares MATLAB/Simulink, CarSim, PreScan, Gazebo, CARLA, and LGSVL for perception, mapping and localization, path planning, and vehicle control.Source: §8 · p. 7
- The analysis identifies key requirements for reliable simulator results and presents five observations about the compared simulators.Source: §8 · p. 7
- Simulation is presented as a safe, cost-effective, and timely aid to vehicle design, provided simulations represent reality; challenges such as missing common standards remain.Source: §8 · p. 7