Source-linked AI summary
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chenyi Chen, Ari Seff, Alain Kornhauser, Jianxiong Xiao
TL;DR
Autonomous driving needs an abstraction that avoids both the complexity of full scene parsing and the limitations of direct image-to-steering regression. The paper proposes ConvNet-based direct perception of compact affordance indicators, showing effective driving in virtual environments and good real-world perception. The approach relies on a front-facing view and assumes the host car is faster than traffic when judging post-overtaking lane changes.
Problem
Existing systems either construct high-dimensional scene representations or directly regress steering actions, motivating a compact representation tied more directly to driving decisions.
Method
A deep ConvNet maps images to driving affordance indicators, which a simple controller converts into high-level driving actions.
Results
The system drives well across diverse TORCS tracks, traffic conditions, and lane configurations, while smartphone-video and KITTI tests demonstrate good real-world perception.
Takeaways & Limitations
Direct perception provides a compact, task-specific description of driving scenes that supports autonomous control in virtual and real environments.
Takeaways & Limitations
During TORCS testing, the system cannot see behind the car and therefore assumes the host car is faster than traffic to decide when post-overtaking lane changes are safe.
Abstract
from arXiv · showhide
Today, there are two major paradigms for vision-based autonomous driving systems: mediated perception approaches that parse an entire scene to make a driving decision, and behavior reflex approaches that directly map an input image to a driving action by a regressor. In this paper, we propose a third paradigm: a direct perception approach to estimate the affordance for driving. We propose to map an input image to a small number of key perception indicators that directly relate to the affordance of a road/traffic state for driving. Our representation provides a set of compact yet complete descriptions of the scene to enable a simple controller to drive autonomously. Falling in between the two extremes of mediated perception and behavior reflex, we argue that our direct perception representation provides the right level of abstraction. To demonstrate this, we train a deep Convolutional Neural Network using recording from 12 hours of human driving in a video game and show that our model can work well to drive a car in a very diverse set of virtual environments. We also train a model for car distance estimation on the KITTI dataset. Results show that our direct perception approach can generalize well to real driving images. Source code and data are available on our project website.
1. Introduction
The paper introduces direct perception as a third autonomous-driving paradigm between full scene parsing and direct image-to-steering regression. It predicts compact driving affordance indicators that support simpler, higher-level control.
- Motivation: Mediated perception builds a detailed world representation from detected objects, but may add complexity through redundant information and unresolved vision sub-tasks.The paper contrasts this with directly predicting driving-relevant quantities such as car distance.
- Motivation: Behavior reflex models map images directly to steering, but traffic creates ambiguous decisions and steering-angle supervision is too weak to capture higher-level maneuvers.Similar images can support following, passing left, or passing right, while the model must also discover which image regions matter.
- Proposed paradigm: Direct perception maps images to affordance indicators such as road-relative angle, lane-marking distances, and distances to cars in current and adjacent lanes.These indicators are intended to provide a compact, meaningful description of the road situation for driving decisions.
- Approach and evidence: A deep ConvNet trained on 12 hours of TORCS human-driving recordings, paired with a simple controller, drives across varied tracks, traffic conditions, and lane configurations.The system is reported to have a simpler structure than typical mediated-perception systems.
- Approach and evidence: Tests on car-mounted smartphone videos and KITTI demonstrate good real-world perception, extending evaluation beyond the virtual driving environment.The paper positions direct perception as a compact, task-specific scene representation for autonomous driving.
2. Learning affordance for driving perception
The system maps driving images to 13 affordance indicators covering lane-following and collision-avoidance situations, then uses them in a controller for autonomous driving.
- Affordance learning: TORCS provides screenshots and ground-truth driving indicators, including host-car speed, road position, and distances to preceding cars.The data are collected by manually driving a label-collecting car.
- Affordance learning: The ConvNet represents highway scenes across one-, two-, and three-lane configurations while monitoring the current and adjacent lanes.The host car considers traffic in its current lane and the two adjacent lanes.
- Affordance representation: The representation contains 13 indicators based on heading angle, distances to nearby lane markings, and distances to preceding cars.These indicators support lane perception and car perception and feed the driving controller.
- Affordance representation: The in-lane and on-marking systems activate under different conditions, with an overlap area enabling smooth transitions.Indicators can also be inactive when their corresponding lane system or adjacent lane does not exist.
- Controller: The controller computes steering from car position and pose, switches the target center line during lane changes, and adjusts speed for turns and preceding cars.Its baseline desired speed is 72 km/h, and close preceding cars trigger distance-based slowing.
- Controller: The optimal velocity car-following model uses preceding-car distance to produce stable, smooth following across speeds and can stop the host car when necessary.The model uses dist(t), vmax, and calibrated coefficients c and d.
3. Implementation
The implementation uses an AlexNet-based ConvNet trained on varied TORCS tracks, cars, and road textures, with testing based only on front-facing images and car speed.
- Model: The direct-perception model uses five convolutional and four fully connected layers, ending in 13 outputs trained with Euclidean loss.The 13 indicators are normalized to [0.1, 0.9] because they have different ranges.
- Training data: Training data come from seven TORCS tracks and 22 traffic cars, supplemented by over 30 asphalt textures, lane configurations, and programmed traffic behaviors.Screenshots are down-sampled to 280×210 and paired with ground-truth labels.
- Training data: 484,815 images are used for training, with 64-image mini-batches, an initial learning rate of 0.01, and training stopped after 140,000 iterations.The model is trained from scratch without crops or mirrored images.
- Testing assumptions: During TORCS testing, the system accesses only the front-facing image and the host car’s speed.This restricted view creates a limitation immediately after overtaking cars in adjacent lanes.
- Testing assumptions: The overtaking limitation is addressed by assuming the host car is faster than traffic and using a timer before changing into the overtaken lane.The TORCS control frequency is 10 Hz and is described as sufficient below 80 km/h.
4. TORCS evaluation
The TORCS evaluation shows that the direct-perception ConvNet can support collision-free driving, tolerate moderate estimation errors, and outperform hand-crafted or weaker perception baselines.
- Qualitative assessment: The system drives well in TORCS without collisions, although lane changes may briefly overshoot before recovering.Lane perception is accurate, while car perception is reliable up to 30 meters and noisier from 30 to 60 meters.
- System architecture: The ConvNet estimates 13 driving indicators from each TORCS image, after which a controller combines them with the current speed to produce driving commands.The commands are sent back to TORCS to control the host car.
- Qualitative assessment: Moderate indicator-estimation errors do not prevent smooth, collision-free driving because the controller continuously corrects the car’s position.The car is modeled as a continuous system, so scattered erroneous estimates can be tolerated.
- Comparison with baselines: Even on the same track and without traffic, the Caltech lane-detector-based system performs worse than the proposed model.The comparison uses separate training and testing sets collected from the same track.
- Comparison with baselines: The ConvNet-based direct-perception system works considerably better than the GIST-based system, and additional training data improves ConvNet accuracy.The comparisons use the “ConvNet full” model and a reduced-data “ConvNet sub” model.
5. Testing on real-world data
Tests on real driving data evaluate transfer from TORCS to smartphone video and quantitative car-distance estimation on KITTI. The approach shows strong lane perception and performance similar to a mediated-perception baseline.
- Smartphone video: The TORCS-trained model performs reasonably well on smartphone driving videos despite the domain difference between training and testing.Lane perception correctly identifies lane configuration, localizes the car, and recognizes lane-changing transitions.
- Smartphone video: The car-perception module is noisier on real videos, likely because TORCS cars differ visually from real cars.Only estimates are shown because the real videos lack ground-truth measurements.
- KITTI distance estimation: The KITTI experiment trains a separate ConvNet to estimate distances to cars ahead using images from over 40,000 stereo pairs and 3D labels for about 12,000 pairs.The KITTI images were captured in European urban areas and paired with Velodyne LiDAR point clouds.
- KITTI distance estimation: The KITTI model estimates the (x, y) coordinates of nearby cars in three front-of-car regions defined relative to the host car.The coordinate system uses the host-car center as origin, heading as the y axis, and rightward direction as the x axis.
- KITTI distance estimation: A two-ConvNet structure combines close-range estimates from the full image with far-range estimates from a central crop.The close-range and far-range models cover 2–25 meters and 15–55 meters, respectively.
- Comparison with DPM-based baseline: The direct-perception ConvNet has similar performance to the state-of-the-art DPM mediated-perception baseline on KITTI.It produces more false positives on some samples, but its true-car distance estimates are more accurate when false positives are not penalized.
6. Visualization
Visualization indicates that the ConvNets learn driving-specific responses: activations correlate with heading, lane markings, and traffic cars, while response maps emphasize nearby cars and lanes.
- Activation patterns: Activation patterns correlate strongly with the host car’s heading, lane-marking locations, and traffic cars.The patterns are derived by averaging the images that most strongly activate selected neurons.
- Interpretation: The authors interpret these visualizations as evidence that the ConvNet developed task-specific features for driving.This interpretation follows the observed responses to cars and lane markings.
- Response maps: Response maps preserve image location and reveal which regions the ConvNet uses when estimating affordance indicators.The maps display the highest filter-response value at each pixel.
- Response maps: KITTI-based response maps emphasize nearby cars, while TORCS-based maps strongly emphasize lane markings.These responses indicate attention to task-relevant visual regions for distance and lane-related estimates.
7. Conclusions
The paper presents direct perception as a third autonomous-driving paradigm that estimates driving affordances rather than parsing entire scenes or directly regressing commands. Experiments report good performance in virtual and real environments.
- Conclusions: The proposed direct-perception paradigm uses a deep ConvNet to estimate affordances for driving actions instead of full-scene representations or direct driving commands.It is presented as distinct from mediated perception and behavior reflex approaches.
- Conclusions: Experiments show that the approach performs well in both virtual and real environments.The conclusion summarizes the reported experimental scope without specifying a single benchmark metric.