Source-linked AI summary
Conditional Affordance Learning for Driving in Urban Environments
Axel Sauer, Nikolay Savinov, Andreas Geiger
TL;DR
Existing autonomous-driving methods either build detailed environmental models or map images directly to controls, while direct perception had mainly been demonstrated in simple highway settings. This paper introduces Conditional Affordance Learning, which maps video and high-level directional commands to low-dimensional affordances for urban navigation. On CARLA, it reports up to 68 % better goal-directed navigation than the best baseline in a new-town training-weather condition, alongside traffic-rule handling and smoother driving in simulation.
Problem
Direct perception had been limited to highway driving, leaving urban requirements such as intersections, traffic lights, speed limits, and obstacles insufficiently addressed.
Method
Conditional Affordance Learning maps video and high-level navigation commands to low-dimensional affordances used by a controller for urban driving.
Results
68 % better goal-directed navigation than the best baseline was achieved by CAL in the new town under training weather conditions on CARLA.
Takeaways & Limitations
The approach handles traffic lights and speed signs from image-level labels and supports smoother, safer urban driving in simulation.
Takeaways & Limitations
The evaluation uses visual navigation from a single front-facing camera, and occasional right-turn sidewalk driving can cause collisions with static objects.
Abstract
from arXiv · showhide
Most existing approaches to autonomous driving fall into one of two categories: modular pipelines, that build an extensive model of the environment, and imitation learning approaches, that map images directly to control outputs. A recently proposed third paradigm, direct perception, aims to combine the advantages of both by using a neural network to learn appropriate low-dimensional intermediate representations. However, existing direct perception approaches are restricted to simple highway situations, lacking the ability to navigate intersections, stop at traffic lights or respect speed limits. In this work, we propose a direct perception approach which maps video input to intermediate representations suitable for autonomous navigation in complex urban environments given high-level directional inputs. Compared to state-of-the-art reinforcement and conditional imitation learning approaches, we achieve an improvement of up to 68 % in goal-directed navigation on the challenging CARLA simulation benchmark. In addition, our approach is the first to handle traffic lights and speed signs by using image-level labels only, as well as smooth car-following, resulting in a significant reduction of traffic accidents in simulation.
1 Introduction
Autonomous driving approaches trade off explicit environmental modeling against direct sensor-to-control learning. The paper extends direct perception to complex urban navigation through conditional affordances driven by video and high-level commands.
- Existing approaches: Modular pipelines decompose perception, planning, and control but rely on manually chosen representations and costly annotations.Their intermediate representations may be difficult to estimate accurately and may not be optimal for sensorimotor control.
- Existing approaches: Imitation learning maps raw inputs directly to controls, simplifying data collection but requiring high-capacity models and extensive training data.The single-step mapping must cover the variety of real-world situations encountered at test time.
- Direct perception: Direct perception predicts low-dimensional environmental representations for a conventional controller, combining advantages of modular and end-to-end approaches.It avoids both end-to-end sensorimotor learning and pixel-level or box-level annotations.
- Direct perception: Existing direct perception work addressed highway driving, whereas urban navigation also requires traffic-rule compliance, obstacle handling, and multi-direction junctions.Urban scenarios add speed limits, red lights, pedestrians, and intersections with more than one possible direction.
- Proposed approach: Conditional Affordance Learning predicts urban-navigation affordances from video and high-level directional commands, then uses a controller to calculate control commands.The paper targets low-dimensional representations, conditional navigation before intersections, smooth driving, and traffic-rule compliance.
2 Related Work
Prior autonomous-driving research is organized around modular perception stacks, end-to-end imitation learning, and direct perception, often evaluated in simulation. The paper situates its urban-driving contribution against these approaches and their simulator settings.
- Modular pipelines: Modular-pipeline research separately trains traffic-scene detection tasks such as object detection, image segmentation, and motion estimation.These components support construction of an environmental model for driving decisions.
- Imitation learning: Imitation-learning systems have progressed from ALVINN to modern convolutional networks for tasks including off-road navigation and highway following.The cited work spans varied terrains, weather, lighting conditions, and obstacle avoidance.
- Direct perception: Direct-perception research predicts affordance indicators describing driving scenes, with later work comparing convolutional architectures for image-to-indicator mappings.The cited studies demonstrated strong performance in racing simulation and analyzed GoogLeNet and VGG architectures.
- Simulation benchmarks: Driving simulators are widely used because real-environment training and testing are difficult, but TORCS lacks several complex urban-scene elements.The passage identifies missing intersections, pedestrians, and oncoming traffic, while describing GTA V as photorealistic but closed source.
3 Conditional Affordance Learning
The CAL agent uses camera observations and planner commands to predict urban-driving affordances, then combines them with controllers for traffic-rule compliance and smooth navigation.
- System setup: The agent receives a front-facing camera image and a high-level maneuver command before producing throttle, brake, and steering outputs.Commands include going straight or turning left or right.
- Perception: The perception stack extracts feature maps from image sequences, stores the last N maps, and combines them with directional commands using task-specific temporal processing.Task blocks use different temporal receptive fields and dilation factors.
- Affordances: Its affordance set represents traffic lights, speed limits, hazards, road-center alignment, and vehicle orientation for urban navigation.Traffic-light and speed-sign detections update stopping and speed behavior, while obstacles trigger hazard stops.
- Perception: A single multi-task network predicts all intermediate affordances in one forward pass, with the network-controller system averaging 50 ms per pass.This latency is below the cited 100 ms real-time requirement.
- Training: The model is trained on CARLA data with augmentation and camera-pose randomization to improve generalization across simulated driving conditions.Training uses Town 1, while Town 2 is reserved for testing.
- Controller: Longitudinal control uses mutually exclusive cruising, following, over-limit, red-light, and hazard-stop states, while lateral control uses Stanley steering with damping.The following state begins when another car is less than 35 m ahead, and turning reduces target speed by 10 km/h.
4 Results
The evaluation compares Conditional Affordance Learning with established navigation approaches on CARLA goal-directed driving, infraction avoidance, and attention behavior. CAL generally performs strongly, including improved generalization to a new town, while analysis identifies a weather-related perception failure and learned visual cues for hazards.
- Goal-Directed Navigation: The benchmark evaluates four driving tasks of increasing difficulty, from driving straight to town navigation with dynamic obstacles.Episodes require reaching predefined goals within task-specific time limits.
- Goal-Directed Navigation: CAL outperforms the baselines in most goal-directed navigation tasks and conditions, with particularly strong generalization to the new town.For the new town under training weather conditions, CAL is up to 68 % better than the best baseline.
- Goal-Directed Navigation: CAL outperforms MP on almost all tasks, while MP performs slightly better in the training town under new weather conditions; CAL is outperformed by CIL on one task.The reported exception is driving straight under training weather conditions in the test town, where tree shadows affect distance-to-centerline prediction.
- Infraction Analysis: In dynamic-obstacle navigation, CAL exceeds the baselines in almost every measure and performs particularly well at avoiding vehicle and pedestrian collisions.In the new environment, CAL performs more than 10 times better than the best baseline, RL.
- Infraction Analysis: CAL occasionally drives on the sidewalk, primarily during sharp right turns, where larger inter-frame pixel motion challenges sequential perception.This can also cause collisions with static objects such as traffic lights.
- Attention Analysis: Grad-CAM analysis shows attention to roadside regions for traffic lights, pedestrian shadows before hazard stops, and pedestrians’ lower bodies during hazard stops.The hazard-stop behavior was learned from binary image-level labels without pixel-wise supervision.
5 Conclusion
The paper concludes that Conditional Affordance Learning combines modular and end-to-end advantages for urban navigation with high-level directional commands. Extensive CARLA simulation shows significant gains, while future work targets broader environments, improved sensing and control, and transfer to the real world.
- Conclusion: Conditional Affordance Learning combines modular pipelines and end-to-end approaches for complex urban navigation using high-level directional commands.The method is presented as a conditional affordance-learning approach for urban driving.
- Conclusion: Extensive simulation experiments demonstrate significant performance gains over the state of the art on CARLA.The conclusion summarizes the evaluation at the benchmark level without specifying an additional metric.
- Conclusion: Future directions include training in more diverse environments, extending the sensor setup, improving control algorithms, and transferring results from simulation to the real world.The authors specifically mention stereo cameras, model predictive control, and joint controller-perception training.
1 Implementation Details
The implementation combines neural-network conditioning with a state-based longitudinal controller and tuned PID control. Hyperparameters and observation-area geometry are optimized for responsive, smooth vehicle behavior.
- Hyperparameter Search: The implementation searches task-block layer type, node count, dropout ratio p, sequence length seq, and dilation value dil, then records optimized parameters.The search ranges are summarized in Table 3 and the best task-block parameters in Table 4.
- Longitudinal Control: The longitudinal controller uses mutually exclusive states ordered by descending importance, including over-limit, red-light, and hazard-stop states.These states are illustrated in Figure 5, while the listed states define throttle and brake behavior for critical situations.
- Longitudinal Control: An over-limit state activates above 15 km/h over the speed limit, sets throttle to zero, and applies speed-dependent braking for rapid deceleration.Entering a 30 km/h zone at 90 km/h produces brake = 0.9.
- Longitudinal Control: The red-light state triggers when P(red light) exceeds Prl, sets throttle to zero, and uses Prl = 0.9 to reduce false positives while stopping reliably.Its braking multiplier is smaller than in the over-limit state because red lights typically occur in 30 km/h zones.
- PID Controller Tuning: PID controllers govern cruising and following, with coefficients initialized through Ziegler-Nichols tuning and empirically refined for fast but smooth disturbance responses.The tuning procedure uses proportional, integral, and derivative coefficients together with ultimate gain and oscillation period measurements.
- Conditioning in the Network vs. Conditioning in the Controller: Conditioning the neural network avoids predicting irrelevant affordances, which can be visually unavailable or ambiguous and would add training noise and runtime.At intersections, a straight maneuver may leave turning-lane affordances outside the camera view or make their distances undefined.
2 Ground Truth Acquisition
Ground-truth affordances are obtained from CARLA’s simulator measurements by transforming scene objects into an ego-centered coordinate system and detecting them within rectangular observation areas. The setup also uses a high-level planner and compares temporal with non-temporal task blocks.
- Ground Truth Acquisition: CARLA provides agent and scene measurements, including traffic-light status and speed-sign type, but these measurements do not directly express the target affordances.Available agent measurements include speed, acceleration, location, and orientation; scene measurements include cars, pedestrians, traffic lights, and speed-limit signs.
- Observation Area: Object positions are transformed from world coordinates into a local coordinate system centered at the car’s front axle before affordance detection.The local frame uses the car’s lateral axis as x and the up vector as z.
- Observation Area: Rectangular observation areas in the local x-y plane determine whether objects count as detected, with their geometry selected according to the respective affordance.Red-light and speed-sign areas are placed on the agent’s right, while the hazard-stop area is short and directly ahead.
- Task-Block Evaluation: Table 6 compares temporal and non-temporal task blocks using IoU and MAE, where higher IoU and lower MAE indicate better performance.The final column reports relative performance change.
- Navigation Commands: CARLA’s A* topological planner converts the agent’s position and destination into commands to turn left, turn right, or continue straight at intersections.The high-level command represents the maneuver supplied to the driving agent.
3 Additional Experiments
The additional experiments show that temporal task blocks improve every classification and regression task, while qualitative evaluation finds CAL best across all four driving-behavior metrics. CAL’s advantage is especially apparent in smooth turning and longitudinal control, although directional transitions still produce occasional short jerks.
- Comparison of Temporal and Non-temporal Task Blocks: Temporal task blocks improve all classification and regression results, showing that every task benefits from video’s additional temporal information.The comparison evaluates temporal versus non-temporal task blocks.
- Comparison of Temporal and Non-temporal Task Blocks: The relative-angle task block gains the largest relative improvement, with temporal error almost four times lower than non-temporal error.The speed-sign task improves least, so the final model uses a non-temporal block for that task to reduce computation time.
- Driving Behaviour: The qualitative evaluation measures centerline distance and RMS jerk, including separate longitudinal and lateral jerk analyses.Lower values indicate better driving behavior, and the centerline-distance result is reported as the episode median.
- Driving Behaviour: CAL achieves the best performance on all four driving-behavior metrics, with similar results across weather conditions and environments.The evaluation concerns navigation without dynamic objects to assess general driving behavior without car- or pedestrian-stopping challenges.
- Longitudinal Jerk: CAL’s adjustable control parameters support smooth acceleration, deceleration, and constant-speed driving, whereas RL’s binary throttle causes sudden jerk.The ranking on the reported longitudinal-jerk comparison is CAL first, followed by CIL and RL.
- Lateral Jerk while Turning: During turns, CAL slows before corners, steers smoothly, and produces lower jerk peaks than competing approaches, but directional switches can cause brief jerks.CIL generally turns smoothly, while RL makes strong and abrupt steering movements with higher jerk.