PaperlayerView exact revision on arXiv

Source-linked AI summary

DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction

Yuhuang Hu, Jonathan Binas, Daniel Neil, Shih-Chii Liu, Tobi Delbruck

arXiv:2005.08605v1cs.CV

TL;DR

To support end-to-end driving studies with event cameras, the paper introduces DDD20 and evaluates steering prediction using APS, DVS, and fused inputs. Across three datasets, fused inputs achieve the best average steering prediction (EVA 0.88), significantly better than either modality alone.

  • Problem

    The paper addresses the need for larger open end-to-end event-camera driving data and evidence on fusing APS and DVS for steering prediction.

    Source: B. DDD20 · p. 2, §IV · p. 5

  • Method

    The authors expand DDD17 into DDD20 and use it for the first study of APS-DVS fusion for steering prediction.

    Source: B. DDD20 · p. 2

  • Results

    EVA 0.88: DVS+APS achieved the best average steering prediction over three datasets, significantly outperforming either modality alone.

    Source: B. Prediction of steering wheel angle · p. 5

  • Takeaways & Limitations

    The findings support combining event and intensity inputs for steering prediction, while DVS without temporal context is blind to non-moving features.

    Source: §IV · p. 6

    Without temporal context, DVS is blind to non-moving features common in driving.

    Source: §IV · p. 6

Abstract

from arXiv · show

Neuromorphic event cameras are useful for dynamic vision problems under difficult lighting conditions. To enable studies of using event cameras in automobile driving applications, this paper reports a new end-to-end driving dataset called DDD20. The dataset was captured with a DAVIS camera that concurrently streams both dynamic vision sensor (DVS) brightness change events and active pixel sensor (APS) intensity frames. DDD20 is the longest event camera end-to-end driving dataset to date with 51h of DAVIS event+frame camera and vehicle human control data collected from 4000km of highway and urban driving under a variety of lighting conditions. Using DDD20, we report the first study of fusing brightness change events and intensity frame data using a deep learning approach to predict the instantaneous human steering wheel angle. Over all day and night conditions, the explained variance for human steering prediction from a Resnet-32 is significantly better from the fused DVS+APS frames (0.88) than using either DVS (0.67) or APS (0.77) data alone.

I. INTRODUCTION

Deep-learning driving systems rely on extensive real and simulated datasets, while driving vision sensors must handle broad lighting conditions. DVS events offer high dynamic range and temporal detail, and DAVIS combines them with APS intensity frames that may supply complementary information.

I. INTRODUCTION

  • Deep-learning autonomous-driving controllers are typically trained on extensive datasets of real-world recordings and simulated environments.These datasets and methods have supported vision tasks including semantic segmentation, object detection, tracking, and motion estimation.Source: §I · p. 1
  • DVS events respond to local brightness changes asynchronously, with a broader dynamic range than conventional cameras (120 dB vs. 60 dB).Events include pixel coordinates, brightness-change sign, and microsecond timestamps.Source: §I · p. 1
  • DAVIS concurrently outputs DVS events and APS intensity frames from circuits sharing the same photodiode.When APS images are over- or under-exposed, blurred, or aliased, DVS events can provide missing information.Source: §I · p. 1

A. Related work

Existing event-camera driving datasets lacked end-to-end human-driving labels, and earlier DDD17 work did not test fused APS and DVS inputs. DDD20 expands that dataset and supports the first study of fused-input steering prediction.

A. Related work

  • Earlier event-camera driving datasets had useful label types, but none included end-to-end human-driving labels.DDD17 itself was limited in road types, weather, and daylight conditions.Source: A. Related work · p. 2

B. DDD20

  • DDD20 adds 39 h to DDD17’s 12 h, with driving across varied locations, road types, and lighting conditions.Recordings include urban and rural roads, difficult sunlight glare, and repeated mountain-highway sections by day, evening, and night.Source: B. DDD20 · p. 2
  • DDD20 includes vehicle control and diagnostic data but not the LIDAR, radar, and other sensors needed for a complete ADAS solution.Source: B. DDD20 · p. 2
  • Fusing APS and DVS improves steering predictions significantly over either modality alone, while APS alone outperforms DVS alone in this study.This differs from the earlier DDD17 comparison, which found better prediction from DVS than APS frames.Source: B. DDD20 · p. 2, A. Related work · p. 2

II. METHODS

The paper announces the DDD20 dataset and its collection methods and software; the supplied passage introduces the dataset section.

II. METHODS

  • The dataset section describes the DAVIS camera and the collection process for DDD20.The dataset is stated to be released through the authors’ website.Source: §II · p. 2

A. DAVIS camera setup

DDD20 recordings use a DAVIS346B camera to capture APS frames and DVS events concurrently through shared optics. The camera’s field of view and variable APS frame rate reflect the recording setup.

A. DAVIS camera setup

  • The DAVIS346B concurrently captures DVS events and APS frames through the same optics, using the same photodiode for both; its 346×260-pixel sensor provides APS output at up to 50 Hz.For DDD20 recordings, the field of view was 71°.Source: A. DAVIS camera setup · p. 2
  • APS frame rate varies from 8 to 50 fps with exposure duration, which is set by auto-exposure in later recordings.The global-shutter mode was used to minimize motion artifacts.Source: A. DAVIS camera setup · p. 2
  • The camera was mounted behind the windshield, and the Ford Focus setup was repositioned daily, producing slightly different viewpoints across recordings.Source: A. DAVIS camera setup · p. 2
  • The example DDD20 pairs show APS images on the left and DVS images on the right, with 25 ms event histograms and APS exposures varying from 50 μs to about 200 ms.Source: Figure 2 · p. 3
  • The viewer example shows over- and underexposure in APS while the DVS frame still exposes those details.Its top panels show APS and DVS; the bottom panel controls the recording and displays status and vehicle data fields.Source: Figure 3 · p. 3
  • The recording framework streamed camera data to the recording system using cAER and local UDP.Source: A. DAVIS camera setup · p. 3

B. Vehicle control and diagnostic data collection

Vehicle control and diagnostic data were collected through an OBDII-connected interface, while recording software synchronized vehicle and camera streams and managed recordings.

B. Vehicle control and diagnostic data collection

  • Vehicle control and diagnostic data were read from the CAN bus through an OpenXC interface connected to the car’s OBDII port, at about 10 Hz.The recordings used a 2015 Ford Mondeo for Swiss/German trips and a 2016 Ford Focus for USA trips.Source: B. Vehicle control and diagnostic data… · p. 3

C. Recording and viewing software

  • The ddd20-utils Python framework records, views, and exports recordings.The passage also identifies Fig. 3 as showing the framework.Source: C. Recording and viewing software · p. 3
  • The computer’s millisecond system time was added to camera and vehicle streams to synchronize them, and the data were stored in HDF5 containers.The vehicle interface sampled at about 10 Hz, so the computer clock was used to synchronize it with the camera streams.Source: C. Recording and viewing software · p. 3

D. Recorded DDD20 data

DDD20 provides recordings across varied driving conditions, and the experiments use the dataset to ask whether combined APS and DVS input improves steering prediction over either modality alone.

D. Recorded DDD20 data

  • 51 h of usable DDD20 data were recorded across varied weather, road, driving, and lighting conditions over about 60 days of intermittent recording.The recordings lasted from a few minutes to an hour, with a median duration of about 700 s.Source: D. Recorded DDD20 data · p. 3
  • DDD20 includes 215 recordings: 40 from DDD17 and 175 additional recordings, with car speeds spanning 0–130 km/h.Steering angles on straight roads were dominated by small deviations of ±10°.Source: D. Recorded DDD20 data · p. 3

III. EXPERIMENTS

  • The experiments test whether fusing APS and DVS provides better steering prediction than either single modality.Source: §III · p. 4

A. Experiment configurations

The experiments use day, night, and combined datasets from selected DDD20 recordings, with sensor-specific preprocessing and filtering for training and testing.

1) Data selection:

  • The study selected 30 recordings—15 day and 15 night—and split each chronologically, using the first 70% for training and the last 30% for testing.The recordings covered a range of road types and lighting conditions, and their ends were pruned where the car was pulling onto or off the road.Source: 1) Data selection · p. 4
  • Three datasets—Night, Day, and All—were prepared to compare DVS+APS, DVS-only, and APS-only inputs under different lighting conditions.Source: 1) Data selection · p. 4

2) Preprocessing inputs for training:

  • DVS input used signed ON/OFF event-count histograms accumulated over 50 ms, clipped at three standard deviations, while APS frames were used at their native rate and duplicated when below 20 Hz.Both sensor inputs were rescaled to [0, 1].Source: 2) Preprocessing inputs for training · p. 4
  • Frames below 15 km/h were excluded because they generally corresponded to parking-space exits or intersection turns, where instantaneous sensor input cannot reveal driver intention.Source: 2) Preprocessing inputs for training · p. 4
  • Training data were balanced by randomly pruning 70% of frames with steering angles between ±5°, while extreme steering outliers were filtered from training and test data.This left about 50% of the original training frames; test data were not pruned for straight-driving angles.Source: 2) Preprocessing inputs for training · p. 4
  • APS and DVS frames were downsampled from 346 × 260 to 172 × 128 pixels, and camera and vehicle inputs were aligned using system-clock timestamps.Source: 2) Preprocessing inputs for training · p. 4

3) Baseline network:

The steering-prediction baseline is a ResNet-32 with a single linear output for steering angle.

3) Baseline network:

4) Training details:

Training used Gaussian weight initialization and Adam-based optimization with MSE; the dataset was large enough to avoid pretraining on another dataset.

4) Training details:

B. Prediction of steering wheel angle

Across the three datasets, fused DVS+APS inputs produced the best steering predictions. Results also varied by lighting condition, and the paper discusses how EVA and RMSE capture different aspects of prediction quality.

B. Prediction of steering wheel angle

IV. CONCLUSIONS

DDD20 extends event-camera end-to-end driving data to more than 50 hours across 4,000 km. The paper reports that fused DVS and APS information best explains steering variance across driving conditions, while identifying temporal context as a direction for future work.

IV. CONCLUSIONS

  • DDD20 provides more than 50 hours of DAVIS recordings across 4,000 km, comparable in duration and distance to major driving datasets, though BDD100k is larger and not E2E.DDD20 is about four times the size of DDD17.Source: §IV · p. 5
  • Fused DVS and APS information best explains steering variance across driving conditions; DVS also contributes information about moving features while lacking non-moving features without temporal context.Source: §IV · p. 6
  • Future work could use event timing and temporal context for optical-flow preprocessing, interframe inference, and prediction of throttle and braking decisions.Source: §IV · p. 6

Related papers

Loading 2005.08605v1…