Source-linked AI summary
DSEC: A Stereo Event Camera Dataset for Driving Scenarios
Mathias Gehrig, Willem Aarents, Daniel Gehrig, Davide Scaramuzza
TL;DR
Challenging illumination and the scarcity of large-scale stereo event-camera driving data limit evaluation of autonomous-driving vision systems. DSEC addresses this gap with synchronized, wide-baseline stereo event and frame cameras plus lidar and RTK GPS across 53 day-and-night sequences. Its baseline evaluation shows that MVSEC-trained stereo methods do not generalize directly, while the dataset provides ground-truth disparity for retraining and evaluation.
Problem
Robust vision in challenging illumination remains an open problem, while large-scale event-camera driving datasets and stereo datasets are rare.
Method
DSEC records synchronized wide-baseline stereo data from high-resolution event and RGB cameras, lidar, and RTK GPS across selected driving sequences.
Results
MVSEC-trained stereo methods do not generalize to DSEC and must be retrained to reach acceptable performance; DSEC provides ground-truth disparity for evaluation.
Takeaways & Limitations
DSEC provides a foundation for studying event-camera utility and algorithm performance across varied illumination and challenging driving situations.
Takeaways & Limitations
The ground-truth filtering approach introduces disparity bias, including sparser labels on moving objects and in challenging conditions such as night driving.
Abstract
from arXiv · showhide
Once an academic venture, autonomous driving has received unparalleled corporate funding in the last decade. Still, the operating conditions of current autonomous cars are mostly restricted to ideal scenarios. This means that driving in challenging illumination conditions such as night, sunrise, and sunset remains an open problem. In these cases, standard cameras are being pushed to their limits in terms of low light and high dynamic range performance. To address these challenges, we propose, DSEC, a new dataset that contains such demanding illumination conditions and provides a rich set of sensory data. DSEC offers data from a wide-baseline stereo setup of two color frame cameras and two high-resolution monochrome event cameras. In addition, we collect lidar data and RTK GPS measurements, both hardware synchronized with all camera data. One of the distinctive features of this dataset is the inclusion of high-resolution event cameras. Event cameras have received increasing attention for their high temporal resolution and high dynamic range performance. However, due to their novelty, event camera datasets in driving scenarios are rare. This work presents the first high-resolution, large-scale stereo dataset with event cameras. The dataset contains 53 sequences collected by driving in a variety of illumination conditions and provides ground truth disparity for the development and evaluation of event-based stereo algorithms.
I. INTRODUCTION
DSEC addresses the lack of large-scale stereo event-camera datasets for autonomous driving, especially under challenging illumination. It combines high-resolution event and standard cameras with diverse conditions to support sensor-fusion and event-based stereo research.
- Robust vision performance in challenging conditions remains an open problem for autonomous driving.
- Event cameras complement standard cameras with microsecond temporal resolution, over 120 dB dynamic range, less motion blur, and submillisecond latency.
- Large-scale driving datasets with stereo event cameras remain scarce, limiting evaluation in environments that test algorithm performance and generalizability.
- Dense depth estimation from stereo events alone is ill-posed, motivating the inclusion of two high-resolution standard cameras for static scenes without motion.
- DSEC separates event and standard cameras spatially within each stereo pair, reducing parallax and creating a sensor-fusion challenge without assuming shared pixels.
- The dataset includes favorable and very challenging illumination conditions, including sequences intended to assess algorithms during adverse driving situations.
II. RELATED WORK
Driving datasets cover specialized tasks, but event-camera datasets remain limited, especially for stereo matching. DSEC addresses this gap with a higher-resolution, wider-baseline stereo setup than MVSEC.
- Existing driving datasets target tasks including scene flow, segmentation, detection, tracking, localization, mapping, and depth estimation, but lack event-camera data.
- The dataset comparison identifies DSEC and MVSEC as the only listed datasets with stereo camera setups, while DSEC offers a baseline and resolution suited to stereo matching.
- Most automotive event-camera datasets provide single-camera data or non-stereo tasks, so they cannot support development of stereo matching algorithms.
- MVSEC is the closest prior dataset, using two DAVIS346B event cameras with roughly 0.1 MP resolution and a 10 cm baseline.
- DSEC provides two Prophesee Gen 3.1 event cameras with three-times-higher resolution and a 60 cm baseline, plus two RGB cameras with a 51 cm baseline.
A. Sensors
DSEC combines stereo event and color frame cameras with lidar and RTK GPS in a multi-camera sensor suite. The cameras use wide stereo baselines while keeping event-frame pairs close for sensor fusion.
- The sensor suite includes stereo event cameras, color frame cameras, a Velodyne lidar, and an RTK GPS system.
- The monochrome event cameras have 640×480 resolution and a 60-centimeter baseline, while the color cameras have 1440 × 1080 resolution and a 51-centimeter baseline.
- The event and frame cameras are mounted close together on each side, producing an approximately 4.5-centimeter cross-type baseline for sensor fusion.
- Frame cameras capture 12-bit raw images at 20 Hz and prioritize exposure-time adjustments over analog gain to reduce noise.
- Event-camera sensitivity is increased in bright daylight and reduced at dawn, dusk, and night to balance scene detail against noisy events.
B. Sequences
DSEC contains close to an hour of driving data collected across urban, suburban, and rural environments in Switzerland. The final dataset comprises 53 sequences spanning day and night conditions.
- The dataset comprises close to an hour of driving data collected under diverse illumination conditions.
- The final dataset contains 53 sequences recorded in three different areas of Switzerland during daytime and nighttime.
- The recordings cover urban, suburban, and rural areas and include diverse illumination conditions.
- More than 4 TB of raw data were recorded before sequences with good recording and calibration quality were selected.
- The sensor data are hardware-synchronized, and the dataset construction includes time synchronization and camera and lidar calibration procedures.
A. Time Synchronization
DSEC synchronizes its sensors through GPS-linked hardware triggers and expresses camera and event timestamps in a shared time reference. Calibration additionally reconstructs event images for checkerboard-based estimation and refines sensor transformations with ICP.
- A. Time Synchronization: A microcontroller connects GPS trigger and time signals with camera triggers to achieve hardware time synchronization.
- A. Time Synchronization: The GPS receiver supplies PPS signals to the microcontroller and lidar, while event cameras record precisely timestamped special events and frame cameras are triggered at 20 Hz.
- A. Time Synchronization: GPS-trigger timestamps are used to compute pulse times and express frame and event timestamps in a common reference.
- A. Time Synchronization: Event-stream image reconstruction enables Kalibr to detect checkerboard corners for camera intrinsic and extrinsic calibration.
- A. Time Synchronization: Sensor transformations are refined by aligning lidar and stereo point clouds with rotation-only point-to-plane ICP, then averaging rotation quaternions across poses.
V. GROUND TRUTH DEPTH GENERATION
DSEC generates denser disparity ground truth by combining localized lidar scans with stereo-image disparity, filtering projected points before producing maps for event and frame cameras.
- Pointcloud construction: LIO-Mapping estimates lidar poses so preceding and proceeding scans can be accumulated into a local pointcloud for each view.This increases ground-truth density beyond a single sparse lidar scan.
- Calibration support: Events are reconstructed into images at frame mid-exposure times, allowing the reconstructions to be used with standard calibration frameworks.This supports calibration involving event cameras and standard cameras.
- Pointcloud filtering: SGM computes disparity from rectified image pairs, providing a reference for filtering projected lidar points.The disparity estimate is not affected by moving objects in the stated procedure.
- Disparity generation: The filtered lidar pointcloud is projected into the left event and frame cameras to generate groundtruth disparity maps at 10 Hz.The resulting maps are described as avoiding artefacts from moving objects or occluded points.
VI. EVALUATION METRICS
The evaluation uses established stereo-matching metrics to measure both coarse and fine-grained disparity errors, alongside average absolute and root-mean-square errors.
- Disparity metrics: The N pixel disparity error metric DN reports the percentage of disparity predictions whose error exceeds N pixels.N is set to 2 and 1 to assess coarse and fine-grained predictions.
- Disparity metrics: The evaluation also reports mean absolute disparity error and root-mean-square error.
VII. EXPERIMENTS
Experiments assess a prior event-based stereo method and a retrained baseline on DSEC, finding poor direct transfer from MVSEC and substantially greater night-time error.
- Experiments: The prior model was retrained on DSEC for 200k iterations, using a 50-millisecond voxel-grid representation for the baseline implementation.The voxel grid was empirically comparable to event queues while being significantly faster to train.
- Results: Night sequences have approximately 50% higher D1 error than day sequences in the baseline evaluation.The authors attribute this to reduced signal-to-noise ratio under low illumination.
- Evaluation setup: The evaluation covers 12 test sequences across three Swiss areas and separates results by location and day or night.The associated disparity ground truth is visualized for event and image data.
VIII. DISCUSSION
The discussion highlights illumination-related sensor difficulties and biases in generated disparity ground truth, while reporting the evaluation setup used to characterize dataset performance.
- Illumination effects: Direct sunlight can significantly reduce camera contrast through lens flare, despite tuning the event-camera biases and settings.The authors suggest higher-quality lenses and improved lens coating could reduce this effect.
- Illumination effects: At night, standard cameras incur motion blur from longer exposure, while event cameras produce noisier events because of reduced signal-to-noise ratio.Flashing lights can also increase event rates and occasionally saturate the event readout, slightly delaying events.
- Evaluation context: The baseline evaluation reports average scores by city and day or night across 12 sequences in three Swiss areas, using DN, MAE, and RMSE.
- Groundtruth limitations: The groundtruth depth-generation approach introduces disparity biases, including sparser labels on moving objects and in challenging image conditions.The stated causes are filtering and reliance on SGM computed from images.
X. CONCLUSION
DSEC is presented as a large-scale, high-resolution event-camera driving dataset with wide-baseline stereo data and synchronized RGB frames. It supports stereo research across varied GPS-mapped routes and challenging nighttime illumination.
- DSEC is the first large-scale driving dataset with wide-baseline, high-resolution event cameras.
- GPS trajectories span urban, suburban, and rural areas in Switzerland, including regions around Lake Zurich, Zurich, and Lake Thun.
- Night sequences show event rates varying with illumination, with flashing street lamps triggering many events and unlit scenes producing much lower rates.
- The dataset records RGB frames time-synchronized with event cameras to support sensor-fusion research.