Source-linked AI summary
The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, Davide Scaramuzza
TL;DR
Event-based cameras offer low-latency, high-temporal-resolution sensing, but their asynchronous output requires new algorithms and datasets. This paper releases DAVIS datasets and an open-source simulator, providing multimodal measurements and ground-truth poses for developing and evaluating such methods.
Problem
Event-based sensors produce unconventional asynchronous outputs, creating a need for datasets and tools to develop algorithms for high-speed robotics and computer vision.
Method
The paper releases DAVIS datasets spanning artificial and natural scenes and an open-source simulator that generates events, intensity frames, and depth from virtual trajectories.
Results
The datasets provide asynchronous events, intensity images, inertial measurements, camera poses, timestamps, and camera calibration information.
Takeaways & Limitations
The released resources support prototyping event-based algorithms and quantitative ego-motion evaluation using synchronized measurements and ground-truth poses.
Takeaways & Limitations
The simulation datasets omit event noise because realistic noise is difficult to model, limiting their fidelity to noisy sensors.
Abstract
from arXiv · showhide
New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional global-shutter camera and an event-based sensor in the same pixel array. These sensors have great potential for high-speed robotics and computer vision because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and very high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. For this purpose, we present and release a collection of datasets captured with a DAVIS in a variety of synthetic and real environments, which we hope will motivate research on new algorithms for high-speed and high-dynamic-range robotics and computer-vision applications. In addition to global-shutter intensity images and asynchronous events, we provide inertial measurements and ground-truth camera poses from a motion-capture system. The latter allows comparing the pose accuracy of ego-motion estimation algorithms quantitatively. All the data are released both as standard text files and binary files (i.e., rosbag). This paper provides an overview of the available data and describes a simulator that we release open-source to create synthetic event-camera data.
Introduction · Related Datasets
Event-based cameras offer low-latency, high-temporal-resolution sensing suited to high-speed and high-dynamic-range robotics, but their asynchronous output requires a different algorithmic paradigm. Compared with prior DAVIS datasets, this work provides broader 6-DOF motion, varied scenes, and precise motion-capture camera poses.
- Introduction: The paper positions event-based cameras as an emerging technology that could transform the robot sensing landscape.Computer-vision research historically focused on rolling- or global-shutter frame cameras.
- Introduction: Event-based cameras provide low-bandwidth, low-storage, and low-power sensing for high-speed and high-dynamic-range robotics.Standard cameras can suffer from motion blur, image saturation, and high latency.
- Introduction: Their fundamentally different information representation requires a paradigm shift from traditional computer-vision sensing and algorithms.The sensors convey asynchronous brightness changes rather than conventional frame-based information.
- Related Datasets: Two recent DAVIS datasets provide related benchmarks for event-based optical flow and visual navigation.The cited works are Rueckauer and Delbruck (2016) and Barranco et al. (2016).
- Related Datasets: Rueckauer and Delbruck’s dataset targets event-based optical-flow comparison using synthetic and real scenes with pure rotational 3-DOF motion.Its scenes are simple and strongly contrasted, with ground truth acquired using an IMU.
- Related Datasets: In contrast, the proposed datasets include arbitrary hand-held 6-DOF motion across diverse artificial and natural scenes.They provide precise ground-truth camera poses from a motion-capture system.
- Related Datasets: Barranco et al.’s dataset supports comparisons of event-based and frame-based methods for 2D and 3D visual navigation.It uses a ground robot equipped with a DAVIS and Microsoft Kinect RGB-D sensor, with the DAVIS mounted on a 5-DOF pan-tilt unit.
The DAVIS Sensor
The DAVIS combines asynchronous, microsecond-timestamped brightness-change events with synchronous frames from the same 240 × 180 pixel array. Its data-driven output offers low-latency, high-temporal-resolution sensing, no redundant transmission, and 130 dB dynamic range.
- Event and frame output: DAVIS transmits asynchronous events alongside conventional frames, with each event encoding pixel coordinates, timestamp, and brightness-change polarity.Events are detected continuously and timestamped with micro-second resolution.
- Event and frame output: Both events and frames originate from the same physical pixels, avoiding spatial offset between the two modalities.The sensor has 240 × 180 pixel spatial resolution.
- Sensor advantages: Event-based cameras provide low latency and high temporal resolution in the micro-seconds range, supporting high-speed mobile robot applications.Their data-driven operation transmits only brightness changes, so bandwidth depends on motion speed and scene type.
- Sensor advantages: The DAVIS offers 130 dB dynamic range, substantially exceeding the 60 dB cited for expensive computer-vision cameras.Its data-driven output avoids transmitting redundant data.
- Dataset motivation: The released datasets enable comparisons of pose tracking, visual odometry, and SLAM algorithms using well-calibrated data without requiring expensive DAVIS equipment.DAVIS sensors cost approximately ∼5, 000 USD, and prior methods used different datasets that prevented direct comparison.
DAVIS IMU · DAVIS Simulator
The DAVIS combines visual sensing with a nearly co-located IMU, while its open-source simulator generates synthetic events, frames, and depth maps from virtual scenes and trajectories. The simulator uses interpolated rendered intensities to model event timing and supports reconstruction checks, but its released datasets omit event noise.
- DAVIS IMU: The DAVIS IMU enables visual-inertial event-based algorithms and is mounted about 3 mm behind and centered beneath the image sensor.Its position nearly coincides with the event sensor’s photoreceptor, rather than the lens-dependent optical center.
- DAVIS Simulator: Piecewise linear interpolation of sampled log-intensity values determines event times and can generate multiple events between rendered-image samples when intensity jumps exceed the contrast threshold.
- DAVIS IMU: The IMU integrates three-axis gyroscope and accelerometer sensors, with ranges of ±2,000 ◦/s and ±16g, digitized by six 16-bit ADCs at 1 kHz.
- DAVIS Simulator: The simulator takes a virtual 3D scene and a moving DAVIS trajectory, generating corresponding event streams, intensity frames, and depth maps from rendered images.Blender was used to generate thousands of rendered images along the specified trajectory.
- DAVIS Simulator: The interpolation scheme provides event timestamps with higher resolution than the rendered images.
- DAVIS Simulator: Under ideal sampling and quantization, images can be reconstructed from accumulated events, with per-pixel intensity error confined to (−C, C).
- DAVIS Simulator: The released simulation datasets omit event noise because realistic noise is difficult to model, while the open-source simulator supports prototyping event-based algorithms.
Datasets
The datasets combine asynchronous events, intensity images, inertial measurements, camera poses, and camera intrinsics with precise timestamps. Ground-truth availability and IMU recording vary by capture setting, with motion-capture precision and frequency specified for indoor datasets.
- Dataset contents: The datasets provide asynchronous events, intensity images at about 24 Hz, inertial measurements at 1 kHz, ground-truth camera poses, and the intrinsic camera matrix.Indoor ground-truth poses come from motion capture with sub-millimeter precision at 200 Hz.
- Ground truth and recording conditions: All information has precise timestamps, but datasets captured outside motion capture, such as office or outdoor sequences, lack ground truth.Some datasets instead use a motorized slider’s position as ground truth.
- Ground truth and recording conditions: Very noisy IMU data was not recorded for datasets collected with a motorized linear slider because of vibrations induced by its motor.The slider’s position supplied ground truth for those datasets.
Data Format
The datasets are released in standard text files and binary rosbag files, with the text-file format specified in Table 1. Supporting conventions define pose representation, SI units, timestamp normalization, and image storage with timestamps.
- Data Format: Datasets are available in standard text form and as binary rosbag files, with the text-file format described in Table 1.The binary-file details are provided on the dataset website.
- Pose Representation: Ground-truth pose uses an arbitrary motion-capture origin with gravity-aligned z-axis and unit-quaternion orientation q = (qx, qy, qz, qw)⊤.The quaternion separates scalar component qw from vector component qv = (qx, qy, qz)⊤.
- Units and Timestamps: All values use SI units, and timestamps are offset by the lowest recorded timestamp so every dataset starts at zero.This normalization helps avoid numerical difficulties with microsecond-resolution event timestamps.
- Images: Images are provided as PNG files, with image filenames and timestamps listed separately; the typical framerate is 24 Hz but varies with exposure time.The image list and timestamp file accompanies the PNG images.
List of Datasets
The dataset collection spans synthetic and real environments with increasing motion difficulty, scene variation, and degrees of freedom. It also includes HDR, outdoor, dynamic, calibration, slider, and simulated-scene data with modality- and setup-specific additions.
- Dataset progression: The datasets vary motion speed, scene, and degrees of freedom, progressing from isolated to combined and faster excitations that increase difficulty and event rate.This progression applies to the shapes, poster, and boxes datasets.
- HDR sequences: HDR sequences use a spotlight to create large intrascene contrasts, with hdr poster spanning 80 lx in dark areas and 2,400 lx in bright areas.The HDR sequences are hdr poster, hdr boxes, and slider hdr.
- Outdoor datasets: Outdoor datasets were recorded in an urban environment during walking and running, without ground truth, but with a return to the same location after a large loop.The datasets are outdoors walking and outdoors running.
- Dynamic datasets: Dynamic datasets depict a moving person in a mock-up office observed by the motion-capture system, transitioning from sitting at a desk to moving around.
- Calibration and slider datasets: A calibration dataset uses a 6 × 7 checkerboard with 70 mm tiles, while intrinsic calibration is provided but calibration datasets are unavailable for the table’s lower-half settings.The calibration data support different camera models and hand-eye calibration methods.
- Simulated datasets: Simulated datasets provide Blender 3D world models, intensity images, events, and per-frame depth maps at 40 Hz encoded as 32-bit floating-point OpenEXR values.
Calibration
The DAVIS was intrinsically calibrated with a checkerboard, followed by hand-eye calibration so provided ground-truth poses refer to the event camera. A calibration dataset was also included to support alternative camera models or improved hand-eye calibration methods.
- Intrinsic calibration: The DAVIS was intrinsically calibrated using a checkerboard pattern.
- Hand-eye calibration: Hand-eye calibration was applied to subsequent recordings so ground-truth poses represent the event camera rather than the attached motion-capture trackable.
- Calibration resources: A calibration dataset was included for alternative camera models or improved hand-eye calibration methods.
Intrinsic Camera Calibration
The dataset uses a standard pinhole camera model with radial-tangential distortion implemented through ROS and OpenCV, and provides calibration coefficients plus postcalibration data for alternative methods.
- Intrinsic Camera Calibration: Calibration uses a standard pinhole model with three radial coefficients and two tangential coefficients, following the ROS and OpenCV implementation.The radial coefficients are k1, k2, and k3 = 0; tangential coefficients are p1 and p2.
- Intrinsic Camera Calibration: Distortion coefficients are provided in calib.txt in the same order as OpenCV.
- Intrinsic Camera Calibration: A postcalibration dataset is provided for use with alternative calibration methods.
Hand-Eye Calibration
Because motion-capture and DAVIS optical frames differ, the datasets use hand-eye calibration to align them. Calibration initializes the eye-to-hand transformation linearly, refines it with joint reprojection-error optimization, and incorporates it into reported ground-truth poses.
- Calibration motivation: The authors calibrate the hand-eye transformation because the motion-capture coordinate frame differs from the DAVIS optical coordinate frame.Indoor datasets provide accurate, high-frequency pose data at 200 Hz from motion capture.
- Calibration procedure: A linear Tsai–Lenz algorithm first initializes the unknown eye-to-hand transformation X from relative hand-hand and eye-eye pose correspondences.The correspondences are expressed as {A_iX = XB_i}^N.
- Calibration refinement: The initial calibration is refined by jointly estimating the hand-eye X and robot-world Z transformations while minimizing image-plane reprojection error.The nonlinear least-squares problem is solved iteratively with Gauss–Newton optimization.
- Calibration resources: A dedicated postcalibration dataset is included for users who prefer applying another calibration method.
- Ground-truth poses: Reported ground-truth poses directly represent the event camera in the world frame by applying the calibrated transformation to motion-capture poses.The transformation is applied as T_W E_j = T_W H_j T_H_j E_j for every pose j.
Camera-IMU Calibration
The calibration dataset enables estimation of the Euclidean camera-to-IMU transformation, which is approximately aligned in orientation with translation mainly along the optical axis and a small measurement lag.
- Camera-IMU Calibration: Running publicly available Kalibr on the calibration dataset computes the Euclidean transformation from the camera reference frame to the IMU reference frame.The camera is denoted as the “eye” E.
- Camera-IMU Calibration: The camera and IMU axes have approximately the same orientation, with translation dominantly along the optical axis at ≈3.42 cm.The IMU is a couple of centimeters behind the camera’s optical center for the used lens.
- Camera-IMU Calibration: IMU measurements lag images and events by ≈2.4 ms because of the IMU’s built-in low-pass filter.Kalibr also reports this temporal shift.
Known Issues · Clock Drift and Offset
The DAVIS and motion-capture clocks are not hardware-synchronized, producing drift and a small dataset-dependent timestamp offset. Clock resets before each short recording make the drift negligible, but the offset remains.
- Clock Drift and Offset: The motion-capture and DAVIS clocks are not hardware-synchronized.
- Clock Drift and Offset: Clock drift is about 2 ms/min.
- Clock Drift and Offset: The clocks are reset before each dataset recording to counteract drift.
- Clock Drift and Offset: Because recordings last about 1 min, drift has a negligible effect on the datasets.
- Clock Drift and Offset: A small dataset-dependent timestamp offset remains between the DAVIS and motion-capture systems.
- Clock Drift and Offset: The timestamp offset arises because the clocks were reset in software.