Source-linked AI summary
Simulation-based reinforcement learning for real-world autonomous driving
Błażej Osiński, Adam Jakubowski, Piotr Miłoś, Paweł Zięcina, Christopher Galias, Silviu Homoceanu, Henryk Michalewski
TL;DR
The paper asks whether reinforcement learning in simulation can produce an end-to-end driving policy that operates a full-size real vehicle. It trains primarily with synthetic data, evaluates sim-to-real transfer, and analyzes how perception, control, and training choices affect performance.
Problem
The study investigates whether an end-to-end driving policy trained in simulation can perform real-world driving.
Method
The authors train reinforcement-learning policies in simulation from RGB images and semantic segmentation, using real-world labeled data only for segmentation training.
Results
The policies were deployed successfully on a full-size car in restricted real-world driving scenarios.
Takeaways & Limitations
The evaluation suggests that regularization, augmentation, semantic segmentation, and branching architectures are beneficial for real-world performance.
Takeaways & Limitations
The experiments cover restricted driving scenarios, and the simulated environment contains no moving cars or pedestrians.
Abstract
from arXiv · showhide
We use reinforcement learning in simulation to obtain a driving system controlling a full-size real-world vehicle. The driving policy takes RGB images from a single camera and their semantic segmentation as input. We use mostly synthetic data, with labelled real-world data appearing only in the training of the segmentation network. Using reinforcement learning in simulation and synthetic data is motivated by lowering costs and engineering effort. In real-world experiments we confirm that we achieved successful sim-to-real policy transfer. Based on the extensive evaluation, we analyze how design decisions about perception, control, and training impact the real-world performance.
I. Introduction
The work tests whether reinforcement learning in simulation can produce an end-to-end driving policy that transfers to a full-size real car. It uses mostly synthetic data to reduce development cost and engineering effort, while evaluating the resulting system across real-world scenarios.
- Real-world evaluation covers 9 driving scenarios totaling about 2.5 km, with more than 400 test drives across 10 models.
- The policy receives RGB images, semantic segmentation, car metrics, and a high-level navigation command, while only the segmentation network uses real-world training data.
- The study asks whether an end-to-end policy trained in simulation can perform real-world driving.
- Simulation and reinforcement learning provide rich experience, avoid real-world safety constraints, and reduce human engineering effort.
- The authors use CARLA and parallelized training to gather up to 100 years of simulated driving experience.
II. Related work
Prior work combines synthetic data, simulators, and reinforcement learning across real and simulated robotics domains. The cited studies include deployments on real robots, but span different platforms and training arrangements.
- Synthetic-data research spans real-world robotics experiments and simulated autonomous-driving systems built with game or physics engines.
- CARLA, AirSim, Unreal Engine 4, Unity, MuJoCo, and TORCS exemplify simulator-based research environments discussed in the related work.
- Prior studies used mostly synthetic or simulator-generated data to train policies for real-world robots, including vehicles, drones, and manipulators.
III. Environment and learning algorithm
The system uses CARLA to train an RL steering policy from a single camera’s RGB image and semantic segmentation, with randomized visual conditions and simulated vehicle dynamics. Its observations also include vehicle metrics and navigation commands.
- CARLA supplies customizable maps, weather, vehicles, and camera settings for autonomous-driving simulation.
- Training scenarios use routes through simulated maps resembling the real testing area, with approximately 50% of testing scenarios covered by developed maps.
- Training assumes static environments without moving cars or pedestrians, so safety-driver interventions remain unavoidable during real-world tests.
- The agent receives dense trajectory-following rewards, while real-world performance is measured by the percentage of distance driven autonomously.
- The action design explores steering control, with throttle handled by a PID controller and steering represented as a Gaussian policy during training.
- Semantic segmentation is trained separately with synthetic CARLA images, Mapillary data, and labeled real-world images from a similar environment.
- The policy combines a 134 × 84 RGB image from one front camera with semantic segmentation, speed, acceleration, and a high-level route command.
- Visual randomizations include weather, simulation quality, and camera-input augmentations to support transfer and policy robustness.
IV. Experiments
The experiments evaluate the learned models both in simulation and on the real vehicle, with substantially greater emphasis on real-world performance.
- The evaluation covers both simulation and reality, with much more focus on real-world experiments.
A. Experiment in simulation
Simulation experiments show that reduced randomization makes training easier and faster, but standard randomization improves generalization to unseen towns and weather.
- Performance summary: Figure 3 compares baseline models across nine real-world scenarios using mean and maximum autonomy.Autonomy is the percentage of distance driven autonomously, and models are sorted by mean performance.
- Trajectory deviation: Figure 4 measures models' average deviation from expert trajectories using GPS.
- Simulation randomization: Standard randomization produces better generalization to a holdout town and holdout weather than reduced randomization.The reduced variant used one weather setting, LOW quality, and no camera-input augmentation.
- Simulation randomization: The reduced-randomization variant is easier and faster to train in simulation.
B. Experiments in the real world
Real-world experiments evaluate autonomy across scenarios and compare training, perception, control, and regularization choices. The results show noisy sim-to-real evaluation, poor simulation-to-real score correlation, and benefits from randomization and suitable intermediate representations.
- Evaluation: Real-world experiments exhibit a high level of noise, making definitive conclusions difficult.
- Sim-to-real evaluation: Simulation scores are poor predictors of real-world performance among policies above the threshold for decent driving.The paper identifies this as an evident instance of the sim-to-real gap.
- Evaluation: Models are evaluated by the percentage of distance driven autonomously across all real-world scenarios.Figure 3 summarizes this metric for the tested models.
- Training choices: Reduced randomization produces the worst tested real-world model, while regularization improves observed behavior.The CONTINUOUS-LOW-RAND model performs significantly worse, whereas regularization nearly eliminates previously observed wobbling in one experiment.
- Control choices: Waypoint control uses a discrete action space and separate navigation-command heads, while its continuous version is weaker and wobbles even in simulation.The waypoint action space covers points within 5 meters, spaced every 5 degrees from −30 to 30.
- Perception and auxiliary tasks: Auxiliary depth prediction slightly speeds simulation training but does not improve real-world evaluation over baseline experiments.
- Perception and auxiliary tasks: Segmentation-only input performs only slightly weaker than the baseline, supporting semantic segmentation as a potentially useful transfer representation.
C. Selected failure cases
The paper introduces small implementation tweaks and potential pitfalls as illustrative practical considerations for researchers and practitioners.
- Selected failure cases: The paper presents two small tweaks and potential pitfalls as practical examples for researchers and practitioners.
1) Single-line versus double-line road markings:
Policies trained only on maps with double-line markings were not sensitive to single-line markings in real-world footage, but custom maps fixed this problem.
- 1) Single-line versus double-line road markings:: Policies trained on CARLA’s TOWN1 and TOWN2 maps were not sensitive to single-line road markings in real-world footage.
- 1) Single-line versus double-line road markings:: Introducing custom maps with single-line road markings fixed the observed real-world sensitivity problem.
- 1) Single-line versus double-line road markings:: The initial training maps featured only double-line road markings.
2) Bug in reward function resulting in driving over the curb:
A reward route built from sparse checkpoints connected by straight lines caused the car to reproduce a curb-crossing behavior in the real world.
- 2) Bug in reward function resulting in driving over the curb:: Sparse checkpoints created a connecting route line that crossed a curb on a road bend.
- 2) Bug in reward function resulting in driving over the curb:: The car exhibited the same curb-crossing behavior during real-world testing, transferring the reward-function idiosyncrasy.
- 2) Bug in reward function resulting in driving over the curb:: The car could drive on without touching the curb.
D. Offline models evaluation
Offline evaluation compares models’ steering against human reference drives using mean absolute error and relates it to autonomous driving performance.
- D. Offline models evaluation: The proxy metric compares model steering with human reference steering frame by frame using mean absolute error.
- D. Offline models evaluation: The autonomous-driving metric is the mean of the driven autonomously measure.
- D. Offline models evaluation: Models using waypoints are excluded because they have a different action space.
V. Conclusions and future work
The paper presents CARLA-trained end-to-end policies deployed on a full-size car, while identifying regularization, waypoint control, and offline proxy metrics as future directions.
- V. Conclusions and future work: CARLA-trained end-to-end policies were deployed and tested on a full-size car in restricted driving scenarios.
- V. Conclusions and future work: The authors identify more regularization, waypoint control, and offline proxy metrics as promising directions.
- V. Conclusions and future work: The paper discusses alternative replay-buffer training algorithms, asymmetric actor-critic architectures, generator-discriminator methods, and intermediate representations.