Source-linked AI summary
Virtual to Real Reinforcement Learning for Autonomous Driving
Xinlei Pan, Yurong You, Ziyan Wang, Cewu Lu
TL;DR
Real-world reinforcement-learning training for autonomous driving requires risky trial-and-error, while policies trained in simulation may not transfer across differing visual appearances. The paper translates virtual frames through scene parsing into realistic images and trains an A3C driving policy on them. Experiments report that this virtual-to-real reinforcement learning approach adapts to real-world driving data and outperforms virtual-only and domain-randomization reinforcement-learning baselines, while remaining below supervised learning overall.
Problem
Real-world reinforcement-learning training is unsafe because incorrect driving actions can damage vehicles and surroundings, while virtual-trained policies may not apply to real scenes with different visual appearances.
Method
The method uses an image-to-segmentation-to-image framework to convert virtual frames into realistic frames, then trains the driving policy with A3C.
Results
The proposed method performs better than virtual-only and domain-randomization reinforcement-learning baselines on real-world driving data, although supervised learning achieves the best overall performance.
Takeaways & Limitations
Training reinforcement learning with synthesized realistic images can produce a self-driving vehicle that is placed in the real world without training the policy directly on real driving interactions.
Takeaways & Limitations
The translation network does not generate all possible appearances for a segmentation map, so future work targets varied colors and textures to reduce training bias.
Abstract
from arXiv · showhide
Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.
1 Related Work
Prior work covers supervised driving-policy learning, reinforcement learning in simulation, virtual-to-real transfer, image translation, and scene parsing. These approaches address different parts of autonomous-driving learning but leave the practical transfer problem central.
- Supervised driving methods map visual inputs to actions or perception indicators, but typically require large labeled datasets or hand-engineered action rules.
- Deep reinforcement learning handles high-dimensional state and action spaces, yet requires environment interaction that is unsafe for real-world vehicle training.
- Simulation-based reinforcement learning can reduce training cost and risk, but transferring policies to real environments remains difficult and may still require real-world fine-tuning.
- Image-translation research uses generative approaches to predict images across modalities, including methods combining structure and style.
- The proposed framework builds on semantic image segmentation, using SegNet with encoder and decoder components.
2 Reinforcement Learning in the Wild
The method translates simulator frames through scene parsing into realistic images before reinforcement-learning training. It combines two image-translation networks with A3C and a reward that favors speed while penalizing track deviation and collisions.
- 2 Reinforcement Learning in the Wild: The realistic translation network addresses the visual gap by mapping virtual frames to scene parsing images and then to realistic frames.
- 2 Reinforcement Learning in the Wild: Scene parsing serves as an intermediate representation because virtual and real scenes share structures such as roads, vehicles, trees, and buildings.
- 2 Reinforcement Learning in the Wild: The translation networks use conditional GAN objectives with L1 regularization, where the generator minimizes against a discriminator and the regularization weight is λ.
- 2 Reinforcement Learning in the Wild: The first translator maps virtual images x to segmentations s, while the second maps segmentations s to realistic images y using separate noise terms.
- 2 Reinforcement Learning in the Wild: A3C trains the driving policy through multiple asynchronous actor-learner threads that explore separate environment copies and synchronize parameters.
- 2 Reinforcement Learning in the Wild: The reward encourages faster driving and collision avoidance using speed, heading alignment, and distance from the track center.
3 Experiments
The experiments evaluate virtual-to-real reinforcement learning using translated simulator images, real-world driving data, and transfer across virtual environments.
- 3 Experiments: The study compares virtual-to-real reinforcement learning with supervised learning and domain-randomization reinforcement learning across real-world and virtual-environment evaluations.The virtual simulator is TORCS, and experiments include real-world steering-action prediction and transfer between different virtual driving environments.
- 3 Experiments: The proposed pipeline translates simulator images into realistic images before feeding them as state inputs to an A3C driving policy.The translation network is trained using virtual images, segmentations, and real-image segmentations; the resulting realistic images train the reinforcement learning agent.
- 3 Experiments: The real-world dataset contains approximately 45k annotated images, with 15k used for supervised training and another 15k held out for testing.Virtual translation training uses 1,673 images collected across the Aalborg TORCS driving cycle.
- 3 Experiments: Real-world evaluation maps continuous steering angles to simulator actions: (−10,10) is straight, below −10 is left, and above 10 is right.Prediction accuracy is computed by comparing generated actions with ground-truth action labels after this mapping.
- 3.2 Transfer Learning in Virtual Driving Environments: For cross-environment transfer, the proposed method trains in E-track1 and evaluates in Cg-track2, while domain randomization trains across 10 virtual environments.Both translation networks use 15k segmented images from E-track1 and Cg-track2.
4 Results
The results show that segmentation preserves comparable scene structure across virtual and real images, while realistic translation supports stronger reinforcement-learning transfer.
- 4 Results: Virtual and real images have similar scene parsing results despite substantially different visual appearance, supporting scene parsing as an intermediate representation.The segmentation model was trained on Cityscapes and applied to both virtual and real images.
- 4 Results: The translation network synthesizes realistic images with appearance similar to the corresponding real-world images while preserving the virtual scene correspondence.Figure 4 compares virtual TORCS images in odd columns with translated realistic images in even columns.
- 4 Results: The proposed method performs better than baseline reinforcement learning trained only on virtual inputs, while supervised learning achieves the best overall performance using large amounts of labeled data.Table 1 reports action-prediction accuracy for the three methods.
- 4 Results: In cross-environment transfer, the proposed model outperforms domain randomization, although the same-environment Oracle achieves the best performance.The proposed model trains in E-track1 and observes translated images resembling the Cg-track2 test environment.
5 Conclusion
The proposed framework trains reinforcement learning agents on synthetic realistic images generated from virtual scenes, enabling application to real-world driving. Experiments support better real-environment generalization than pure virtual training or domain randomization, while future work targets translation diversity and stronger RL performance.
- Conclusion: Experiments show better real-environment generalization than training with pure virtual data or domain randomization.The authors report this as an experimental finding for reinforcement learning trained with synthetic real images.
- Conclusion: The authors identify better image-to-image translation and reinforcement-learning frameworks as next steps toward surpassing supervised learning.This is stated as a future performance goal rather than a demonstrated result.
- Conclusion: The framework translates virtual images through scene parsing into realistic images while preserving scene structure for reinforcement learning training.The parsing-to-realistic network can produce realistic appearances from shared semantic representations of virtual and real scenes.
- Conclusion: Future work aims to generate varied appearances from each segmentation map to reduce reinforcement-learning training bias.The authors specifically mention varying color and texture because a segmentation map does not uniquely determine appearance.
- Conclusion: The paper provides a first example of training a self-driving vehicle with reinforcement learning in a synthesized real environment and placing it in the real world.The proposed image-to-segmentation-to-image framework is used to create the synthesized real environment.