Source-linked AI summary
End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances
Marin Toromanoff, Emilie Wirbel, Fabien Moutarde
TL;DR
Urban driving remains difficult for RL because existing algorithms have not handled the complexity of traffic lights, intersections, pedestrians, and vehicles. The paper introduces implicit affordances, using a supervised visual encoder to provide RL state features, and reports a successful urban-driving agent that won the CARLA Challenge Camera Only track. The authors identify transfer to real images and extension to policy-based or actor-critic methods as future work.
Problem
Existing RL algorithms had not yet demonstrated successful urban driving with traffic-light handling and other complex interactions.
Method
Implicit affordances train a large encoder on driving-relevant affordances, then use its output features as the RL state instead of raw images.
Results
The agent successfully handled end-to-end urban driving from vision and won the Camera Only track of the CARLA Autonomous Driving Challenge.
Takeaways & Limitations
The method supports model-free RL for urban driving with traffic-light detection, intersection management, and avoidance of pedestrians and vehicles.
Takeaways & Limitations
The affordance encoder was not yet trained on real images, and applying the scheme to real cars was left for future work.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) aims at learning an optimal behavior policy from its own experiments and not rule-based control methods. However, there is no RL algorithm yet capable of handling a task as difficult as urban driving. We present a novel technique, coined implicit affordances, to effectively leverage RL for urban driving thus including lane keeping, pedestrians and vehicles avoidance, and traffic light detection. To our knowledge we are the first to present a successful RL agent handling such a complex task especially regarding the traffic light detection. Furthermore, we have demonstrated the effectiveness of our method by winning the Camera Only track of the CARLA challenge.
1. Introduction
Urban driving remains an open problem because intersections, traffic lights, pedestrians, and variable vehicle interactions are difficult for rule-based and existing RL approaches. The paper introduces implicit affordances to enable end-to-end vision-based RL and reports success in the CARLA challenge.
- Urban driving requires handling traffic lights, pedestrians, vehicles, and multiple intersection lanes, making it an open autonomous-driving problem.
- Imitation learning can exploit human-driving data but suffers from distribution mismatch and has mainly been limited to lane keeping and lateral control.
- Deep reinforcement learning learns from reward through trial and error, but typically requires substantially more data than supervised methods.
- Implicit affordances first train an encoder to predict traffic-light state and lane-related affordances, then use its features as the RL state instead of raw images.The features are stored directly in replay memory, requiring approximately 20 times less memory.
- The method targets lane keeping, traffic-light detection, pedestrian and vehicle avoidance, and intersection handling, and wins the Camera Only track of the CARLA challenge.
- The paper studies implicit-affordance parameters and reward shaping through extensive ablation experiments.
2. Related Work
Prior autonomous-driving RL work largely used simplified simulators or handled limited control tasks, while affordance methods and recent RL approaches addressed only parts of urban driving. The paper builds on these gaps with a vision-based urban-driving focus.
- TORCS-based RL studies optimized racing performance and did not handle intersections or traffic lights.
- CARLA introduced urban-driving scenarios, but its A3C RL baseline performed far behind imitation learning, while another RL method mainly fine-tuned imitation learning and omitted traffic lights.
- Real-car RL studies remained preliminary, scenario-specific, and focused mainly on steering or lane keeping rather than full throttle-and-steering control.
- Auxiliary-task methods improved data efficiency and performance in Atari and labyrinth exploration by predicting additional signals.
- Driving affordance methods predicted lane and obstacle information for rule-based controllers, with later work extending them toward more complicated urban scenarios.
- Related feature-based and model-based methods addressed navigation or collision prediction but did not combine model-free RL with traffic-light handling in urban driving.
3. The CARLA Challenge
The CARLA Challenge evaluates autonomous driving from sensors to control in unseen urban maps, requiring navigation, lane management, collision avoidance, and traffic-light compliance across US and EU layouts.
- The challenge tests driving in unseen maps from sensors to control while maintaining lanes and following right, left, and straight intersection orders.
- Its tasks include lane changes, pedestrian and vehicle avoidance, and simultaneous handling of US and EU traffic lights.
4. Method
The method uses value-based model-free RL with reward shaping and a large visual architecture adapted for urban driving. Implicit affordances address the training and replay-memory costs of high-resolution inputs.
- RL Setup: The approach uses value-based RL because it is considered more data-efficient than policy-based RL, while accepting discrete actions.
- Reward Shaping: The desired-speed reward peaks when speed matches a situation-dependent target that decreases near red lights and obstacles and returns to maximum after green lights.
- RL Setup: Rainbow-IQN Ape-X is adapted for distributed training because CARLA is too slow for conventional RL data collection.
- Reward Shaping: The waypoint API supplies lane geometry and intersection options used to compute the optimal trajectory and shape rewards.
- Reward Shaping: The position reward favors the lane center, reaches -1 at Dmax, and terminates episodes beyond Dmax; experiments set Dmax to 2 meters.
- Reward Shaping: A rotation reward penalizes angle differences from the nearest waypoint because lateral rewards alone allowed oscillations near the lane center.
- Network Architecture: The network uses four consecutive 288×288×3 frames and a ResNet-18 encoder because small 84×84 grayscale inputs cannot reliably show distant traffic lights.The architecture has 18 convolutional layers and approximately 30M parameters.
5. Challenges and Solutions to apply RL to Complex Autonomous Driving Tasks
The paper addresses RL’s difficulty with large visual inputs, replay memory, and complex urban-driving signals through implicit affordances, encoder pretraining, augmentation, and richer discrete-action handling.
- Training with high-complexity inputs: Large images make RL training and replay storage impractical, motivating alternatives to storing raw visual transitions.For 288 × 288 × 3 images, storing 1M transitions would require about 210GB.
- Implicit affordances: The encoder is pretrained on semantic and driving-relevant affordances, then its features are used as the RL state while the encoder remains frozen.Supervised targets include semantic segmentation, traffic-light state, intersection presence, and lane position information.
- Implicit affordances: Implicit affordances expose learned features rather than explicit predictions, allowing replay memory to store compact representations instead of raw images.The scheme is designed for replay-memory-based RL with larger networks and inputs.
- Encoder data augmentation: Viewpoint augmentation is mandatory because RL exploration deviates from autopilot lane-center trajectories, creating varied lane-marking positions.Training only on autopilot data produces poor performance when the agent departs from the lane center.
- Handling discrete actions: With discrete control, increasing steering resolution and averaging multiple predictions reduces oscillations and improves final behavior.Experiments use 9 or 27 steering values, combined with throttle and braking actions; reported results average three consecutive training snapshots.
6. Experiments and Ablation Studies
The experiments evaluate implicit affordances through controlled urban-driving scenarios, metric definitions, encoder and reward ablations, generalization tests, and benchmark comparisons. Results show that affordance pretraining and reward shaping are important for learning, while the method generalizes to unseen environments and compares strongly with prior baselines.
- Test scenarios: The evaluation uses Town05, changing weather, randomly spawned pedestrians, 10 urban scenarios, and 20M training iterations to create a challenging single-town setup.Each scenario contains 10 consecutive intersections; training uses 3 actors at 10 FPS.
- Metrics: The primary metric is the percentage of intersections successfully crossed, supplemented by traffic-light, pedestrian, and oscillation measures.Inters. is higher-is-better; TL and Ped. measure infraction-free passage, while Osc. is lower-is-better.
- Encoder ablations: Without affordance learning, agents cross less than 10% of intersections on average and larger image encoders require 50% or 200% more training time.The experiments were stopped after 10M steps, and full-image replay memory also required more memory.
- Encoder ablations: Removing traffic-light or semantic-segmentation losses harms performance, with segmentation removal particularly worsening pedestrian-collision performance.The authors associate the pedestrian result with difficulty detecting pedestrians and vehicles, whose information is contained in the semantic map.
- Reward and action ablations: With constant desired speed, the agent runs 70% of traffic lights and collides with 60% of pedestrians, indicating the importance of the speed reward component for braking behavior.The constant-speed agent fails to brake appropriately for red lights and pedestrian crossings.
- Reward, action, and generalization studies: Predicting steering derivatives increases oscillations, whereas using 9 or 27 steering values has no significant performance impact; training on multiple towns improves EU-town generalization.Single-US-town training performs poorly on the unseen EU town, while unseen-US-town performance is roughly similar across training settings.
- Benchmark comparison: On benchmark comparisons, LBC is the only method reported to outperform this RL agent on the hardest CoRL2017 task, while the proposed method surpasses the other RL baseline by a large margin.The paper also reports matching or outperforming imitation-learning approaches on the CARLA benchmark.
7. Conclusion
The paper presents a successful end-to-end vision-based RL agent for urban driving, using implicit affordances and a value-based Rainbow-IQN-Apex setup. It reports winning the CARLA Challenge Camera Only track and identifies real-image training and policy-based extensions as future work.
- The approach combines implicit affordances, adapted rewards, and a large conditional network with value-based Rainbow-IQN-Apex training.The agent is designed for end-to-end urban driving from vision, including traffic light detection.
- The reported evaluation included winning the Camera Only track in the CARLA challenge.
- Future work includes applying implicit affordances to policy-based or actor-critic methods and training the affordance encoder on real images.The stated real-image goal is application of the method on a real car.
A. Supplementary materials: Implementation details
The supplementary implementation section introduces the detailed hyperparameters and architectures used for both supervised and reinforcement-learning training.
- The section details hyperparameters for the supervised training phase.
- The section details hyperparameters for the reinforcement-learning training phase.
- The section also describes the architectures used in both training phases.
A.1. Supervised phase of affordances training: architecture and hyper-parameters
The supervised affordance-training phase uses a modified ResNet-18 encoder, weighted affordance losses, and a dataset of around 1M labeled simulator frames.
- Architecture: The encoder is based on ResNet-18, modified for 12-channel input from four stacked RGB frames and 2x2 downsampling kernels.
- Hyper-parameters: The traffic light state detection loss receives weight 10, while all other losses receive weight 1.
- Architecture: The semantic decoder uses nearest-neighbor upsampling followed by two batch-normalized convolutional layers.
- Dataset: The affordance encoder was trained on around 1M labeled frames collected mainly in CARLA Town05 and Town02.
A.2. Reinforcement Learning phase: architecture and hyper-parameters
The RL phase feeds frozen affordance-encoder features into a large gated network with multiple action heads. Stability experiments compare encoder-loss configurations, while training remains computationally expensive and limited in scope.
- Architecture: The RL state consists of 8162 features from a frozen affordance-trained image encoder.
- Architecture: The network maps encoder features through an 8162x1024 fully connected layer and concatenates four previous speed and steering values.
- Architecture: Six gated-network heads use separate weights and two fully connected layers with one hidden layer of size 512.
- Computational constraints: A 20M-step RL training run took more than one week on an Nvidia 1080 Ti, limiting extensive stability analysis across experiments.
- Stability study: At 20M steps, both No segmentation seeds performed substantially worse than both All affordances seeds.
B.2. Additional experiments
Additional experiments show that temporal information is important for the encoder, while predicting multiple segmentations and using skip connections has smaller effects on intersection performance. These findings support using temporal inputs and outputs without standard U-net skip connections, although the experiment used only one seed.
- Temporality: 64% of intersections were crossed with All Affordances using four frames, compared with 29% when using only one image.Removing temporal information from the input substantially reduced final performance.
- Temporality: Predicting one semantic segmentation instead of four had a marginal effect on intersections crossed but slightly reduced traffic-light and pedestrian performance.The comparison evaluates temporal supervision in the supervised encoder phase.
- Skip connections: U-net-like skip connections had a relatively small effect on the number of intersections crossed.The experiments also tested whether skip connections should be used in the semantic-segmentation decoder.
- Conclusions: The additional experiments supported adding temporality to encoder inputs and outputs and omitting standard U-net skip connections.These conclusions concern the supervised phase used to produce the RL state.
- Limitations: The reported effects may not be representative because only one random seed was used.The authors identify the single-seed evaluation as a limitation of these additional experiments.
B.4.1 Test weathers results (train and test town)
Under test weathers, the approach reached perfect scores across all reported tasks, but performed substantially worse than the LBC baseline on NoCrash because its frozen encoder misclassified sun reflections as moving obstacles. The authors attribute this boundary to weather-sensitive perception and leave improved supervised performance as future work.
- Evaluation scope: The benchmark reimplementation was completed only after acceptance, so test-weather results were reported in the supplementary materials.The original training setup was not reimplemented in time for the newer CARLA 0.9.6 benchmark conditions.
- Results: The approach reached a perfect score on all tasks under test weathers.Table 9 reports success-rate comparisons with multiple baselines on test weathers.
- Results: NoCrash performance under test weathers fell far behind the LBC baseline.The authors report that results were similar under train weathers but diverged under test weathers.
- Failure analysis: Sun reflections on the ground were predicted as moving obstacles, causing the RL agent to brake continuously and time out on NoCrash.The frozen encoder had been trained only on Town01 and train weathers.
- Training setting: The encoder was trained with around 500K labeled frames from Town01 and training weathers, followed by around 40M RL steps in the same town and weather conditions.Training used nine actors for the RL agent.