Source-linked AI summary
An Empirical Evaluation of Deep Learning on Highway Driving
Brody Huval, Tao Wang, Sameep Tandon, Jeff Kiske, Will Song, Joel Pazhayampallil, Mykhaylo Andriluka, Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue, Fernando Mujica, Adam Coates, Andrew Y. Ng
TL;DR
Highway autonomous-driving perception needs robust, affordable vision methods supported by data spanning driving conditions. The paper collects labeled highway data and evaluates CNNs for lane and vehicle detection, including an efficient single-frame detector. The evaluated systems achieve real-time rates and good highway detection performance, while remaining subject to distance, occlusion, and inference-bottleneck constraints.
Problem
Autonomous-driving vision requires large labeled data sets representing varied highway driving conditions to assess whether deep-learning perception is robust and practical.
Method
The authors collect sensor-calibrated highway data and train CNNs for single-frame lane and vehicle detection, including depth prediction and an Overfeat-based real-time detector.
Results
The CNN detects lanes and vehicles at 44Hz on a GTX 780 Ti, with 100% ego-lane F1 score up to 50 meters and recall declining beyond 65 meters.
Takeaways & Limitations
Existing CNN algorithms can achieve good performance for highway lane and vehicle detection while meeting real-time requirements on suitable hardware.
Takeaways & Limitations
Detection performance is constrained by image resolution at long distances, and box merging can become an O(n^2) real-time bottleneck.
Abstract
from arXiv · showhide
Numerous groups have applied a variety of deep learning techniques to computer vision problems in highway perception scenarios. In this paper, we presented a number of empirical evaluations of recent deep learning advances. Computer vision, combined with deep learning, has the potential to bring about a relatively inexpensive, robust solution to autonomous driving. To prepare deep learning for industry uptake and practical applications, neural networks will require large data sets that represent all possible driving environments and scenarios. We collect a large data set of highway data and apply deep learning and computer vision algorithms to problems such as car and lane detection. We show how existing convolutional neural networks (CNNs) can be used to perform lane and vehicle detection while running at frame rates required for a real-time system. Our results lend credence to the hypothesis that deep learning holds promise for autonomous driving.
I. INTRODUCTION
The paper motivates highway perception as a relatively structured autonomous-driving setting where computer vision and deep learning can complement existing sensors. It collects labeled highway data to evaluate CNN-based lane and vehicle detection across driving situations.
- Highways are comparatively predictable and orderly, while urban and residential roads have greater unpredictability, generic objects, inconsistent lane markings, and complex traffic flows.
- Cameras provide richer features at a fraction of the cost of radar and sonar, while computer vision supports lane detection and redundant vehicle detection.
- Deep learning offers an alternative to classic computer vision by reducing hand-engineering while requiring substantial data and computation.
- The authors calibrate cameras with LIDAR, GPS, and other sensors to create labeled video data containing lane markings, vehicle locations, and relative speeds.
- The paper empirically evaluates its collected data set and explains a neural network for detecting lanes and cars.
II. RELATED WORK
The paper positions its detector against sensor fusion and road-model approaches, instead relying on a neural network trained from annotations. It applies an efficient Overfeat-based design for empirical evaluation.
- Prior autonomous-driving detectors commonly combine computer vision with multiple sensors, motion models, or road models to improve detection reliability.
- The authors use neither road nor motion-based models, relying instead on neural-network robustness for predictions.
- Their detector operates independently on single monocular-camera frames and predicts object depth from labels extracted from radar returns.
- The network may learn a road model from annotations, but that model is not explicitly engineered.
- For evaluation, the authors use a straightforward Overfeat application because of its efficiency and combine it with related labeling ideas.
III. REAL TIME VEHICLE DETECTION
The vehicle detector adapts Overfeat’s sliding-window CNN into an efficient mask-and-regression system for high-resolution highway images. Design changes address ambiguity and occlusion, while inference can become a computational bottleneck.
- III. REAL TIME VEHICLE DETECTION: The system targets cars beyond 100m and real-time operation above 10Hz using 640 × 480 images.
- III. REAL TIME VEHICLE DETECTION: Overfeat converts fully connected layers into convolutional layers, producing a grid of context-specific feature vectors for sliding-window detection.
- III. REAL TIME VEHICLE DETECTION: The original design uses multiple context views and input scales so objects are classified across the image.
- III. REAL TIME VEHICLE DETECTION: Ambiguous predictions can create false vehicles, while the O(n^2) box-merging step may bottleneck real-time inference.
- III. REAL TIME VEHICLE DETECTION: The combined detector uses Overfeat’s efficient sliding window to produce object masks and regress bounding boxes from a single 640 × 480 resolution.
- III. REAL TIME VEHICLE DETECTION: Shrinking bounding boxes by 75% reduces activation ambiguity in occluded cases by requiring the detector’s central 4×4-pixel region to lie within the object center.
A. Lane Detection
The vehicle CNN extends to lane-boundary detection by adding a lane class and regressing line endpoints with their depths. DBSCAN then clusters predicted segments into semantic lanes and supports 3D lane shapes.
- A. Lane Detection: The lane detector extends the vehicle CNN with an additional class and predicts six regression values for each lane boundary segment.
- A. Lane Detection: Four lane-regression values represent segment endpoints, while two represent endpoint depths relative to the camera.
- A. Lane Detection: DBSCAN clusters line segments into lanes, and the predicted depths allow reconstruction of lane shapes in 3D.
A. Data Collection
The research vehicle combines cameras, lidar, GPS, and radar-related systems to collect highway perception data and generate vehicle and lane annotations.
- The research vehicle is a 2014 Infiniti Q50 equipped with six cameras, a Velodyne lidar, and a Novatel SPAN-SE receiver.
- The collected sensor configuration supports mapping and annotation of highway lanes and vehicles in three-dimensional space.
- Vehicle annotations combine Mechanical Turk bounding boxes with radar returns to obtain vehicle distance and relative speed.
- Lane borders are represented as curves, making frame-level annotation tedious, inefficient, and prone to human errors.
1) Ego-lane boundary generation:
Ego-lane boundaries are generated from mapped reflective lane markings and filtered using the vehicle’s GPS trajectory and geometric cues.
- Ego-lane boundary generation: High-reflectivity points in 3D lidar maps are filtered to identify highway lane markings while excluding asphalt, cars, and traffic signs.
2) Semi-automatic generation of multiple lane boundaries:
Multiple lane boundaries are initialized by shifting ego-lane boundaries according to an assumed lane width, then corrected manually for exceptional road configurations.
- Semi-automatic generation of multiple lane boundaries: Lane boundaries are initialized by laterally shifting ego-lane boundaries by multiples of a predefined lane width.
- Semi-automatic generation of multiple lane boundaries: Human annotators correct the initial estimates for exceptions such as merges and splits.
B. Data Set
The dataset contains highway driving data from 14 days in the San Francisco Bay Area, with separate vehicle and lane annotations and perspective-distortion augmentation.
- The annotated dataset covers 14 days of driving in the San Francisco Bay Area during April–June.
- Vehicle data contains nearly 17 thousand frames and 140 thousand bounding boxes, while lane data contains over 616 thousand frames.
- Training applies translation and seven different perspective distortions to the raw datasets.
- The same perspective distortion is applied to ground-truth labels so they remain aligned with the transformed images.
C. Results
The evaluation measures lane and vehicle detection across distance and driving data, reporting strong ego-lane performance, real-time operation, and identifiable failure cases.
- Runtime and qualitative evaluation: The network runs at 44Hz on a GTX 780 Ti, 2.5Hz on a Tegra K1, and is expected to reach 5Hz on Nvidia PX1.The evaluation also measures predicted depth using radar returns as ground truth and reviews raw frame-independent detections in uploaded videos.
- Lane detection: The lane test set contains 22 clips from 11 data-collection runs, evaluating four lane boundaries at distances from 15 to 80 meters.The evaluation includes up to 56 lane-boundary positions.
- Lane detection: 100% F1 score is obtained for ego-lane boundaries up to 50 meters.Recall drops beyond 65 meters as image resolution fails to capture lane-marking width, while adjacent-lane recall is low at the nearest point because it lies outside the camera field of view.
- Vehicle detection: The vehicle test set contains 13 clips from one day, totaling 1 hour and 30 minutes of driving.Vehicle bounding boxes are evaluated using IOU≥0.5 against AMT ground truth boxes.
- Failure cases: Nearby false positives are especially problematic for ADAS because they can cause needless braking, with overpasses and shading effects identified as the largest problems.Examples of these conditions are shown in Fig. 10.
- Vehicle detection: The car detector is compared with Continental mid-range radar, whose returns were assigned 100% precision after calibration-related matching errors were accounted for.The comparison uses radar returns matched to ground-truth boxes even when IOU<0.5.
V. CONCLUSION
The paper builds a multimodal highway dataset and trains a CNN to detect lanes and cars in one forward pass. The system runs at 44Hz on a GTX 780 Ti, supporting good highway detection performance and motivating future temporal modeling.
- V. CONCLUSION: A highway dataset contains 17 thousand image frames with vehicle bounding boxes and over 616 thousand image frames with lane annotations.Camera, Lidar, Radar, and GPS are used to build the dataset.
- V. CONCLUSION: A CNN detects all lanes and cars in a single forward pass and runs at 44Hz using one GTX 780 Ti.The paper describes this rate as more than adequate for real-time use.
- V. CONCLUSION: The results show that existing CNN algorithms achieve good performance in highway lane and vehicle detection.Future work focuses on frame-level annotations for neural networks that use temporal information across frames.