Source-linked AI summary
DDD17: End-To-End DAVIS Driving Dataset
Jonathan Binas, Daniel Neil, Shih-Chii Liu, Tobi Delbruck
TL;DR
The paper addresses limitations in automotive sensing for challenging conditions by investigating combined APS and DVS data for end-to-end driving. It introduces DDD17, an open annotated DAVIS driving dataset with synchronized vehicle data, and conducts a preliminary CNN steering-prediction study. The dataset and tools were verified as usable, but quantitative accuracy results were inconclusive.
Problem
Automotive sensing must handle conflicting cost, dynamic-range, sensitivity, and lighting requirements, while current ADAS systems face problems in challenging weather and lighting.
Method
The paper introduces DDD17, combining synchronized APS-DVS recordings with vehicle-interface data and testing CNN prediction of instantaneous steering angle from APS and/or DVS inputs.
Results
The authors verified the usability of the DDD17 dataset and associated tools in a preliminary CNN steering study, but quantitative accuracy results were too inconclusive to report.
Takeaways & Limitations
DDD17 provides an open dataset for studying APS-DVS fusion and end-to-end prediction of measured driving actions.
Takeaways & Limitations
The dataset omits LIDAR and route information, is unbalanced, and the experiments used only a small subset of recordings.
Abstract
from arXiv · showhide
Event cameras, such as dynamic vision sensors (DVS), and dynamic and active-pixel vision sensors (DAVIS) can supplement other autonomous driving sensors by providing a concurrent stream of standard active pixel sensor (APS) images and DVS temporal contrast events. The APS stream is a sequence of standard grayscale global-shutter image sensor frames. The DVS events represent brightness changes occurring at a particular moment, with a jitter of about a millisecond under most lighting conditions. They have a dynamic range of >120 dB and effective frame rates >1 kHz at data rates comparable to 30 fps (frames/second) image sensors. To overcome some of the limitations of current image acquisition technology, we investigate in this work the use of the combined DVS and APS streams in end-to-end driving applications. The dataset DDD17 accompanying this paper is the first open dataset of annotated DAVIS driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor recording highway and city driving in daytime, evening, night, dry and wet weather conditions, along with vehicle speed, GPS position, driver steering, throttle, and brake captured from the car's on-board diagnostics interface. As an example application, we performed a preliminary end-to-end learning study of using a convolutional neural network that is trained to predict the instantaneous steering angle from DVS and APS visual data.
1 Introduction
Self-driving systems need sensing hardware that handles conflicting cost, dynamic-range, sensitivity, and lighting requirements. The paper investigates event cameras as a complement to conventional imaging and introduces DDD17 for end-to-end driving research.
- Automotive sensors must balance low cost, high dynamic range, high sensitivity, and resistance to flickering-light artifacts.
- Challenging weather and lighting remain difficult for current ADAS sensors, motivating event-camera augmentation of conventional imagers.
- DVS asynchronously outputs brightness changes, reducing data rate and latency while increasing temporal resolution relative to frame-based systems.
- DDD17 extends prior DVS-CNN work to real-world driving as the first published end-to-end dataset of DVS or DAVIS driving data.
2 Davis Driving Dataset 2017 (DDD17)
DDD17 records synchronized DAVIS visual data and vehicle signals for studying APS-DVS fusion in ADAS. The dataset combines a 346×260 APS+DVS camera, vehicle-interface measurements, and software for synchronization, viewing, and export.
- DDD17 collects DAVIS and car data from Swiss and German road driving under varied conditions, excluding sensors such as LIDAR to focus on APS-DVS fusion.
- The DAVIS346B records event-based and frame-based data simultaneously through the same optics at 346×260-pixel resolution.
- APS recordings use global-shutter frames at 10–50 fps, with a fixed 56° horizontal field of view and low-light improvements over the DAVIS240C.
- An OpenXC interface reads control and diagnostic data from the Ford vehicle’s CAN bus through the OBDII port.
- Recorded vehicle quantities include steering, pedals, engine and vehicle speed, GPS coordinates, lighting, wipers, odometry, torque, gear, fuel, and ignition status.
- The recording framework combines and synchronizes camera and vehicle streams using system timestamps, while companion tools visualize and export the recordings.
3 Recorded data
DDD17 spans more than 12 hours and 1000 km of Swiss and German driving across varied road, weather, and lighting conditions. Its recorded signals show mostly straight driving, broad speeds, and substantial low-light coverage.
- Over 12 h of recordings cover more than 1000 km across varied weather, road, driving, and lighting conditions over six consecutive days.
- Steering angles are dominated by straight driving and small deviations of ±10°, while speed is uniformly distributed from 0–160 km/h.
- Automatically controlled headlights are on about half the time, indicating substantial coverage of low-light conditions.
- The recording viewer presents DAVIS frames, events, progress, headlamp status, steering angle, and speed together.
4 Experiments: Steering prediction network
The paper presents a preliminary CNN study predicting instantaneous steering angle from APS and DVS inputs. The dataset and tools were usable, but the reported quantitative accuracy was inconclusive and the experiments used only a small subset.
- The dataset is intended to support end-to-end control learning without tedious hand-labeling of data or features.
- CNN steering networks take APS and/or DVS data as input and predict the instantaneous steering-wheel angle.
- The study compares steering-prediction accuracy between networks using pure APS data and pure DVS data.
- The example CNN uses four convolutional layers, 8 feature maps per layer, 3×3 kernels, and 2×2 max pooling.
- Quantitative accuracy results were too inconclusive to report, although the authors verified the usability of the dataset and tools.
5 Conclusion
The paper introduces DDD17, an open dataset of DAVIS driving data with end-to-end labels and supporting software tools, and reports a preliminary CNN steering-prediction study demonstrating data usability.
- DDD17 is the first open dataset of DAVIS driving data with end-to-end labeling and necessary software tools.