Source-linked AI summary
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
Xiaodan Liang, Tairui Wang, Luona Yang, Eric Xing
TL;DR
Autonomous urban driving needs policies that cope with complex multi-agent environments, while modular and supervised approaches have important limitations. CIRL combines human-demonstration initialization, controllable policies, and DDPG for vision-based driving. On CARLA, it achieves state-of-the-art performance across varied scenarios and generalizes to unseen environments, though further improvement is still needed.
Problem
Autonomous urban driving remains under-explored because learning policies for complex multi-agent dynamics is difficult, while modular and supervised approaches have limitations.
Method
CIRL combines imitation learning from human demonstrations with DDPG, controllable gating, and specialized steering-angle rewards for different driving commands.
Results
CIRL achieves state-of-the-art CARLA driving performance, surpasses prior modular, imitation-learning, and reinforcement-learning pipelines, and generalizes across environments and conditions.
Takeaways & Limitations
The results demonstrate a successful vision-based deep-reinforcement-learning pipeline for autonomous driving in a high-fidelity simulator.
Takeaways & Limitations
CIRL demonstrates strong generalization but still needs further improvements.
Abstract
from arXiv · showhide
Autonomous urban driving navigation with complex multi-agent dynamics is under-explored due to the difficulty of learning an optimal driving policy. The traditional modular pipeline heavily relies on hand-designed rules and the pre-processing perception system while the supervised learning-based models are limited by the accessibility of extensive human experience. We present a general and principled Controllable Imitative Reinforcement Learning (CIRL) approach which successfully makes the driving agent achieve higher success rates based on only vision inputs in a high-fidelity car simulator. To alleviate the low exploration efficiency for large continuous action space that often prohibits the use of classical RL on challenging real tasks, our CIRL explores over a reasonably constrained action space guided by encoded experiences that imitate human demonstrations, building upon Deep Deterministic Policy Gradient (DDPG). Moreover, we propose to specialize adaptive policies and steering-angle reward designs for different control signals (i.e. follow, straight, turn right, turn left) based on the shared representations to improve the model capability in tackling with diverse cases. Extensive experiments on CARLA driving benchmark demonstrate that CIRL substantially outperforms all previous methods in terms of the percentage of successfully completed episodes on a variety of goal-directed driving tasks. We also show its superior generalization capability in unseen environments. To our knowledge, this is the first successful case of the learned driving policy through reinforcement learning in the high-fidelity simulator, which performs better-than supervised imitation learning.
1 Introduction
Autonomous urban driving requires policies that handle complex visual, long-term, multi-agent interactions, but existing modular and supervised approaches remain limited. CIRL combines controllable imitation learning with DDPG to improve exploration and specialize policies across driving commands.
- Autonomous urban driving remains under-explored, especially in crowded environments requiring multi-agent reasoning, traffic-rule understanding, negotiation, and difficult turns.
- End-to-end imitation learning maps sensor inputs to controls but struggles to generalize beyond the coverage of human driving data.Existing models can fail on scenarios such as right and left turns when trained primarily for road following.
- Large continuous action spaces make reinforcement-learning exploration inefficient for autonomous driving.The relevant actions include sequences of continuous steering angles, brakes, and speeds.
- CIRL initializes DDPG actor networks with knowledge learned from human demonstrations, providing more reasonable exploration starting points.The imitation stage uses supervised learning on recorded human driving videos before reinforcement learning interacts with the simulator.
- CIRL uses controllable gating and distinct abnormal steering-angle rewards to specialize policies for different control signals.The control signals include follow, straight, turn right, and turn left.
- CIRL is reported to outperform previous modular and imitation-learning pipelines on diverse high-fidelity CARLA driving tasks.
2 Related Work
Prior autonomous-driving work largely uses modular perception-planning systems or end-to-end mappings from sensory input to controls. CIRL differs by combining imitation-derived initialization with reinforcement learning and controllable policies for continuous driving actions.
- Autonomous-driving methods are commonly organized as highly tuned modular pipelines or end-to-end approaches.The modular category combines visual perception algorithms with model-based planning and control.
- End-to-end Supervised Learning: Supervised imitation learning has been applied to navigation, off-road driving, and road following, but generally requires extensive data to generalize across environments.
- CIRL’s actor network uses a gating function to selectively activate branches for Straight, TurnLeft, TurnRight, and Follow commands.
- Reinforcement Learning for Autonomous Driving: Reinforcement learning uses trial-and-error without explicit human supervision, but large-scale continuous action spaces remain a critical challenge for applications.
- Unlike DQfD and DDPGfD, CIRL directly initializes actor parameters from imitation learning to guide reinforcement-learning exploration.The paper reports that this strategy is more efficient than DDPGfD in the autonomous-driving simulator.
3 Controllable Imitative Reinforcement Learning
CIRL combines controllable imitation learning with reinforcement learning to learn vision-based driving policies for diverse commands. It initializes exploration from human demonstrations, then uses DDPG, command-specific policies, and simulator-based rewards to improve robustness and generalization.
- Controllable Imitation Learning: CIRL uses supervised imitation learning to pretrain a deterministic policy from human driving videos, including observations, commands, speeds, and continuous vehicle actions.The action space includes steering, braking, and acceleration; commands select among Follow, Straight, TurnLeft, and TurnRight policy branches.
- Controllable Imitation Learning: The command-gating mechanism selectively activates four policy branches, allowing specialized action prediction for different driver intents and intersection maneuvers.The branches encode distinct hidden knowledge for lane following, straight driving, left turns, and right turns.
- Evaluation Benchmark: Experiments use forward-camera images, simulator speed, and planner commands, with Town 1 for training and Town 2 plus unseen weather settings for testing.The benchmark includes additional new settings to evaluate generalization beyond settings used in prior work.
- Imitative Reinforcement Learning: CIRL initializes DDPG’s actor with the pretrained imitation policy, producing reliable exploratory transitions while retaining noisy continuous-action exploration.The exploration noise follows an Ornstein-Uhlenbeck process, and target networks stabilize one-step off-policy evaluation.
- Imitative Reinforcement Learning: The critic learns from replay-buffer transitions using one-step returns, while the actor is updated through the critic’s action gradients from controllable imitative learning.Actor and critic are deep networks, and their detailed architectures are provided in the paper’s figures.
- Reward Module: CIRL’s reward combines command-dependent steering penalties and speed rewards with penalties for sidewalk overlap, opposite-lane overlap, and collisions.Sidewalk and opposite-lane violations receive -100 penalties; collision penalties are -100 for vehicles or pedestrians and -50 for other objects.
4 Experiments
Experiments on CARLA evaluate CIRL across goal-directed tasks, unseen towns, weather conditions, additional paths, and real-scene transfer. CIRL generally outperforms comparison methods and shows stronger generalization, while performance remains weak in heavy rain and unseen environments still require improvement.
- Comparisons with state-of-the-arts: CIRL substantially outperforms modular, imitation-learning, and conventional reinforcement-learning baselines across CARLA conditions.Performance is measured by the percentage of successfully completed episodes on four goal-directed navigation tasks.
- Comparisons with state-of-the-arts: 71% of episodes succeed for CIRL versus 59% for imitation learning and 12% for reinforcement learning in an unseen-town setting.The comparison concerns the reported unseen generalization setting.
- Comparisons with state-of-the-arts: CIRL combines imitation learning’s fast convergence with reinforcement learning’s robust long-term decision making using roughly 12 hours of simulator exploration.Imitation initialization provides a reasonable exploration point, while environmental reward feedback supports robustness and generalization.
- Generalization capability: 62% of Nav.dynamic episodes succeed for CIRL versus 44% for the modular pipeline and 42% for imitation learning in the New town/weather setting.This is reported as the most challenging navigation task and setting in the comparison.
- Generalization capability: Adapting CIRL to unseen towns can improve through training on a wider range of different scenes, indicating an explicit remaining generalization boundary.The paper reports reasonably robust performance on additional paths and weather settings but calls for further improvement in unseen-town adaptation.
- Generalization capability: CIRL remains weak under challenging rainy weather because heavy rain makes the road and surrounding dynamics difficult to perceive.Promising results are reported for good-visibility conditions, whereas rainy conditions produce very low success rates.
5 Conclusion
CIRL addresses vision-based autonomous driving in a high-fidelity simulator by combining controllable imitation learning, DDPG, and specialized steer-angle rewards. It achieves state-of-the-art CARLA performance and superior generalization across environments and conditions.
- CIRL combines controllable imitation learning with DDPG policy learning to address sample inefficiency in reinforcement learning for vision-based autonomous driving.The approach targets autonomous driving in a high-fidelity car simulator.
- Specialized steer-angle rewards enhance optimization of policy networks based on controllable imitation learning.
- CIRL achieves state-of-the-art driving performance on the CARLA benchmark, surpassing modular, imitation learning, and reinforcement learning pipelines.
- CIRL demonstrates superior generalization across a variety of different environments and conditions.