Source-linked AI summary
CARLA: An Open Urban Driving Simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, Vladlen Koltun
TL;DR
Autonomous urban driving remains difficult to benchmark because existing environments lack the complexity or control needed for detailed evaluation. CARLA introduces an open simulator to evaluate modular, imitation-learning, and reinforcement-learning approaches, finding that performance declines with task difficulty and that reinforcement learning underperforms the other approaches.
Problem
Existing driving environments lack the urban complexity or detailed control needed to benchmark autonomous driving policies in densely populated settings.
Method
CARLA provides an open simulator with reusable urban assets, flexible sensor and environment configuration, and controlled evaluation of modular, imitation-learning, and reinforcement-learning approaches.
Results
Performance declines as tasks become more difficult; modular and imitation-learning methods perform similarly on most conditions, while reinforcement learning underperforms them.
Takeaways & Limitations
CARLA supports development, training, and detailed performance analysis of autonomous driving systems through controlled scenarios and simulator feedback.
Takeaways & Limitations
End-to-end approaches are susceptible to rare events, and significant robustness improvements may require advances in learning algorithms and model architectures.
Abstract
from arXiv · showhide
We introduce CARLA, an open-source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous urban driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. The simulation platform supports flexible specification of sensor suites and environmental conditions. We use CARLA to study the performance of three approaches to autonomous driving: a classic modular pipeline, an end-to-end model trained via imitation learning, and an end-to-end model trained via reinforcement learning. The approaches are evaluated in controlled scenarios of increasing difficulty, and their performance is examined via metrics provided by CARLA, illustrating the platform's utility for autonomous driving research. The supplementary video can be viewed at https://youtu.be/Hp8Dz-Zek2E
1 Introduction
Autonomous urban driving is difficult to research because dense multi-agent environments, costly physical-world infrastructure, and dangerous scenarios complicate training and validation. CARLA addresses these limitations as an open urban-driving simulator with free digital assets, configurable conditions and sensors, and controlled evaluation of multiple driving approaches.
- Densely populated urban driving is challenging because systems must handle complex intersection dynamics and respond to tens or hundreds of other actors.
- Physical-world research is hindered by the cost and logistical difficulty of operating robotic cars and collecting data covering numerous training and validation corner cases.
- Simulation can democratize autonomous-driving research, support system verification, and safely represent scenarios too dangerous to stage in the physical world.
- Existing platforms inadequately support urban-driving research because racing simulators omit key urban complications, while commercial games provide limited benchmarking control and sensor specification.
- CARLA is an open simulator developed for training, prototyping, and validation, and it evaluates modular, imitation-learning, and reinforcement-learning driving approaches in controlled scenarios of increasing difficulty.
2 Simulation Engine
CARLA is an open-source UE4-based simulation engine designed for flexible, realistic urban driving research. Its server-client architecture, configurable environments and NPCs, varied conditions, sensors, and traffic-rule measurements support agent interaction and evaluation.
- Simulation Engine: CARLA is an open-source layer over UE4 that provides realistic rendering and physics, basic NPC logic, and interoperable plugins.The engine is free for non-commercial use and designed for future community extensions.
- Simulation Engine: A server runs the simulation and renders scenes, while a Python client exchanges commands and sensor readings with the server through sockets.This interface connects autonomous agents to the dynamic simulated world.
- Environment: CARLA combines detailed urban assets with efficient rendering using low-weight geometry, textures, carefully crafted materials, and variable level of detail.Its environments include buildings, vegetation, signs, infrastructure, vehicles, and pedestrians; Town 1 contains 2.9 km of drivable roads for training, while Town 2 contains 1.4 km for testing.
- Non-player characters: NPC vehicles follow lanes, obey traffic lights and speed limits, and make intersection decisions, while pedestrians navigate town-specific maps and avoid one another and vehicles.NPC appearances are randomized through clothing, accessories, and vehicle materials to increase visual diversity.
- Conditions and sensors: CARLA supports midday and sunset lighting, nine weather conditions, configurable RGB cameras, ground-truth depth and semantic segmentation, and agent and traffic-rule measurements.The semantic segmentation pseudo-sensor provides 12 semantic classes, while measurements include location, orientation, speed, acceleration, collisions, and wrong-way-lane occupancy.
3 Autonomous Driving
CARLA is used to develop and analyze three autonomous-driving approaches: a modular pipeline, conditional imitation learning, and deep reinforcement learning. All methods operate on sensory observations and high-level topological-planner commands, while differing in their driving architecture and training signal.
- Approaches: CARLA evaluates a modular pipeline with dedicated perception, planning, and continuous-control subsystems, an end-to-end imitation-learning model, and an end-to-end reinforcement-learning model.The modular architecture follows common autonomous-driving systems, while the other approaches learn driving end-to-end.
- Common formulation: All methods map sensory observations and a high-level plan to three-dimensional steering, throttle, and brake actions.The topological planner uses the A∗ algorithm to advise left, right, or straight actions at intersections without providing a trajectory or geometric information.
- Modular pipeline: The modular pipeline estimates lanes, road limits, dynamic objects, hazards, and intersection proximity visually, then uses rule-based planning and PID control.Because no metric map is provided, local planning depends completely on the scene layout estimated by perception.
- Conditional imitation learning: Conditional imitation learning uses human driving traces containing observations, high-level commands, and actions to train an end-to-end driving model.The commands indicate driver intentions and are used alongside perceptual input.
- Deep reinforcement learning: Deep reinforcement learning uses A3C to train goal-directed navigation from environmental rewards, without human driving traces.Training used 10 parallel actor threads and 10 million simulation steps, with rewards based on speed, progress, collisions, sidewalk overlap, and opposite-lane overlap.
4 Experiments
The experiments compare modular, imitation-learning, and reinforcement-learning agents on four increasingly difficult goal-directed driving tasks across two towns and six weather conditions. Evaluation uses repeated episodes with a time-budgeted goal-reaching criterion while logging infractions such as sidewalk driving and collisions.
- Methods: Three methods are evaluated: modular pipeline, imitation learning, and reinforcement learning.The same agent for each approach is used across all four tasks without scenario-specific fine-tuning.
- Experimental setup: The methods are tested on four increasingly difficult goal-directed driving tasks in each of two towns and under six weather conditions.Agents start at an initialized location and must reach a destination point.
- Experimental setup: Town 1 is used for training, whereas Town 2 is reserved for testing; training uses four weather conditions, while testing uses cloudy daytime and soft rain at sunset.The test weather set is never used during training.
- Evaluation: Each task, town, and weather-set combination is tested over 25 episodes, with success defined as reaching the goal within a time budget based on an optimal 10 km/h route.Sidewalk driving and collisions do not terminate episodes but are logged and reported as infractions.
5 Results
CARLA’s evaluation shows that success rates decline as navigation tasks become harder, with new-town generalization substantially more difficult than new-weather generalization. The modular pipeline and imitation learning are generally comparable, while reinforcement learning performs worse overall despite substantially more training data.
- Overall results: Success rates decline as tasks become more difficult, and none of the methods achieves perfect performance even when driving straight under training conditions.Variability in sensory inputs and non-repeated test trajectories require robust generalization.
- Overall results: On populated-urban navigation, modular pipeline and imitation learning remain below 90% success in every condition and below 45% in the test town.The results indicate that performance is far from saturated and that new-environment generalization remains challenging.
- Generalization: Generalization to new weather is close to, and sometimes better than, training performance for modular pipeline and imitation learning, whereas a new town challenges all approaches.Table 1 compares training conditions with previously unseen weather and a previously unseen town.
- Modular pipeline vs end-to-end learning: Modular pipeline and imitation learning typically differ by less than 10% across testing conditions.The modular pipeline can perform better in new weather because the selected test weathers favor its perception system.
- Imitation learning vs reinforcement learning: Reinforcement learning performs significantly worse than imitation learning on all tasks despite training on 12 days of driving versus 14 hours for imitation learning.The discussion attributes this underperformance partly to reinforcement learning’s brittleness and the need for extensive task-specific hyperparameter search.
- Infraction analysis: All approaches perform better in the training town; imitation learning strays onto the opposite lane least frequently, while reinforcement learning is worst on that metric.Reinforcement learning collides with pedestrians least often, plausibly because such collisions incur a large negative reward.
6 Conclusion
CARLA is presented as an open simulator for autonomous driving, combining reusable digital assets with open-source code and protocols. The authors use it to test modular and end-to-end autonomous-driving approaches.
- 6 Conclusion: CARLA is an open simulator for autonomous driving with open-source code, protocols, and freely reusable digital assets.The digital assets were created specifically for autonomous-driving research.
- 6 Conclusion: The platform is used to test three approaches: a classic modular pipeline, end-to-end imitation learning, and end-to-end reinforcement learning.The end-to-end approaches use deep networks trained through imitation learning or reinforcement learning.
Supplementary Material · S.1 Simulator Technical Details · S.1.1 Client and Server Information Exchange
CARLA uses a client-server architecture in which the server renders the simulated world and the client controls the agent vehicle and selected simulation properties. The client sends vehicle and environment commands, while the server returns measurements and sensor-related state information.
- S.1.1 Client and Server Information Exchange: CARLA’s server runs and renders the world, while the client lets users control the agent vehicle and certain simulation properties.This client-server division defines the simulator’s basic information exchange.
- S.1.1 Client and Server Information Exchange: The client controls the agent vehicle through five command types.The command categories are steering, throttle, brake, hand brake, and reverse gear.
- S.1.1 Client and Server Information Exchange: Steering uses a real number from -1 to 1, whereas throttle and brake use real numbers from 0 to 1.For steering, -1 and 1 represent full left and full right.
- S.1.1 Client and Server Information Exchange: Hand brake and reverse gear are represented as boolean activation states.These commands indicate whether the hand brake or reverse gear is activated.
- S.1.1 Client and Server Information Exchange: Meta-commands let the client control the environment and server behavior.The listed controls include non-player vehicle count, pedestrian count, and weather or lighting preset selection.
- S.1.1 Client and Server Information Exchange: The client can set the number of non-player vehicles and pedestrians spawned in the city.Both quantities are specified as integer numbers.
- S.1.1 Client and Server Information Exchange: The client can select weather and lighting presets using a weather ID index.Supported presets include clear, cloudy, and multiple rain conditions across midday and sunset settings.
- S.1.1 Client and Server Information Exchange: The server sends the client measurements describing the world and the player’s state.These include player position, speed, collision impact, opposite-lane intersection, and sidewalk intersection.
S.1.2 Environment
CARLA provides two towns with mapped layouts and a diverse collection of urban assets, including cars and pedestrians.
- S.1.2 Environment: CARLA includes Town 1 and Town 2, alongside diverse assets such as cars and pedestrians.Maps and representative views of the towns are shown in Figure S.1, while Figure S.2 demonstrates asset diversity.
S.2 Driving Systems Technical Details
This section provides additional technical details about the autonomous driving systems tested in the study.
- S.2 Driving Systems Technical Details: The section presents additional technical details for the autonomous driving systems evaluated in the study.It focuses on the systems tested by the authors.
- S.2 Driving Systems Technical Details: Its purpose is to describe the autonomous driving systems used in the experiments.The passage identifies these systems as the systems tested in the study.
- S.2 Driving Systems Technical Details: The section supplements the main discussion with further system-level information.The passage characterizes the material as additional technical details.
S.2.1 Modular Pipeline … S.3.1 Types of Infractions
The supplementary methods specify implementations for modular, imitation-learning, and reinforcement-learning driving systems, then define infractions used to characterize their performance. The approaches are evaluated using distance traveled between infractions across five violation types.
- S.2.1 Modular Pipeline: The modular pipeline trains semantic segmentation with Adam at λ0 = 10−6 for 300 epochs, using a frozen ImageNet-pretrained ResNet and no augmentation.The network uses batch size 2.
- S.2.1 Modular Pipeline: Its intersection classifier uses 500 balanced images and Adam at λ0 = 10−3 for 500 epochs, without pre-training or data augmentation.Training uses batch size 5.
- S.2.1 Modular Pipeline: The modular local planner follows the ego-lane mask to maintain a fixed distance from the road’s right edge and computes center-directed waypoints for challenging left turns.Left turns are complicated by missing lane markings, longer target-lane distance, and limited forward-camera field of view.
- S.2.2 Imitation Learning: The imitation-learning network combines image perception, speed measurements, joint representation, and command-conditioned control branches to produce motor commands.Its training uses Adam, balanced mini-batches of 120 samples, an initial learning rate of 0.0002, and 294,000 iterations.
- S.2.2 Imitation Learning: Imitation-learning demonstrations combine 80% automated-agent data with 20% human-driver data, while steering perturbations improve policy robustness during collection.Perturbations occurred with pperturb = 0.1 per second, lasted 0.5 to 2 seconds, and had fixed intensity 0.15.
- S.3.1 Types of Infractions: Performance is characterized by average distance traveled between infractions covering opposite-lane driving, sidewalk occupancy, and collisions with static objects, cars, or pedestrians.Each violation lasts at most 2 seconds, so 10 seconds on the sidewalk counts as 5 violations.