PaperlayerView exact revision on arXiv

Source-linked AI summary

CED: Color Event Camera Dataset

Cedric Scheerlinck, Henri Rebecq, Timo Stoffregen, Nick Barnes, Robert Mahony, Davide Scaramuzza

arXiv:1904.10772v1cs.CV

TL;DR

Existing event datasets are limited to monochrome events, although color event cameras have emerged and there are no publicly available color-event datasets. The paper releases CED, a color-event extension to ESIM, and comparisons of three reconstruction methods; it concludes that these resources can support future color-event research.

  • Problem

    Existing event datasets are limited to monochrome events, and color-event cameras have emerged without publicly available color-event datasets.

    Source: §1 · p. 2

  • Method

    The paper constructs CED with color events and frames, extends ESIM to simulate color events, and evaluates three event-based image reconstruction methods.

    Source: §1 · p. 2

  • Results

    CED provides color events and frames across diverse scenes, and the paper compares three state-of-the-art methods for color video reconstruction.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors hope CED and the simulator will inspire future research with color events.

    Source: §5 · p. 8

    CED does not provide ground-truth labels for specific tasks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Event cameras are novel, bio-inspired visual sensors, whose pixels output asynchronous and independent timestamped spikes at local intensity changes, called 'events'. Event cameras offer advantages over conventional frame-based cameras in terms of latency, high dynamic range (HDR) and temporal resolution. Until recently, event cameras have been limited to outputting events in the intensity channel, however, recent advances have resulted in the development of color event cameras, such as the Color-DAVIS346. In this work, we present and release the first Color Event Camera Dataset (CED), containing 50 minutes of footage with both color frames and events. CED features a wide variety of indoor and outdoor scenes, which we hope will help drive forward event-based vision research. We also present an extension of the event camera simulator ESIM that enables simulation of color events. Finally, we present an evaluation of three state-of-the-art image reconstruction methods that can be used to convert the Color-DAVIS346 into a continuous-time, HDR, color video camera to visualise the event stream, and for use in downstream vision applications.

1. Introduction

Existing event datasets contain only monochrome events, despite the arrival of color event cameras and potential applications for color information. The paper introduces a diverse color-event dataset, a color-capable ESIM simulator, and comparisons of three reconstruction methods.

  • No publicly available color-event dataset existed, while existing datasets were limited to monochrome events and access to color cameras was limited.Source: §1 · p. 2
  • CED provides color events and synchronized color frames across varied scenes, lighting, dynamics, and camera motions, without task-specific ground-truth labels.The registered sensor images can be used to generate proxy labels for tasks of interest.Source: §1 · p. 2
  • Three state-of-the-art reconstruction methods are compared to visualize color events and support image-based processing of event data.Source: §1 · p. 2
  • The paper extends ESIM to simulate color events, addressing its lack of color-event support.Source: §1 · p. 2

2. Related Works

Existing event-camera datasets are monochrome and often target specific applications, while CED covers a broad range of scenarios. The work also extends ESIM to simulate color events and addresses the effort required to collect color-event data.

2. Related Works

  • Existing event-camera datasets are limited to monochrome events and often target specific applications, whereas CED aims to cover a wide range of scenarios and motions.The authors position CED for broad research topics rather than a particular use case.Source: §2 · p. 2
  • The ESIM extension simulates color events from rendered color frames using an RGBG Bayer pattern and supports ESIM’s bundled rendering engines.The simulator can also provide ground-truth modalities such as color frames, depth, optical flow, and camera poses.Source: §2 · p. 2
  • Earlier color-event segmentation used a custom beam-splitting rig, and its authors had to reconstruct color frames from three event streams.CED aims to save future researchers this setup effort.Source: §2 · p. 2
  • The Color-DAVIS346 is described as the latest color event camera, outputting RGBG events and frames at 346 × 260 resolution.Earlier devices included the SDAVIS192, which output RGBW events and frames at 188 × 192 pixels.Source: §2 · p. 3

3. CED: Color Event Camera Dataset

CED combines synchronized color-event and frame recordings from the Color-DAVIS346 across varied scenes and conditions. The paper also extends ESIM to simulate color events and associated ground-truth modalities.

3. CED: Color Event Camera Dataset

  • The dataset provides synchronized, timestamped events, raw frames, and demosaiced color frames from the 346 × 260 Color-DAVIS346.It uses fixed exposure for indoor sequences and auto-exposure outdoors; Table 1 lists the camera bias settings.Source: §3 · p. 3, Table 1 · p. 3
  • CED contains 50 minutes of footage, 100k color frames, and over one billion color events across five categories.The categories are Simple, Indoors, People, Driving, and Calibration, with varied lighting, scenes, and camera motions.Source: §3 · p. 3
  • The ESIM extension simulates color events by applying an RGBG Bayer pattern to rendered color frames before event simulation.It remains compatible with ESIM’s bundled rendering engines and can provide modalities including color frames, depth, optical flow, and camera poses.Source: §3 · p. 3

4. Color Video Reconstruction

The paper evaluates three event-based reconstruction methods on color events, adapting them to produce color video. HF and E2VID qualitatively match DAVIS frames, while edge cases expose distinct strengths and weaknesses; reconstructed color also qualitatively improves object detection.

4.2. Results

  • HF and E2VID qualitatively match Color-DAVIS346 frames when reconstructing from events alone.The comparison includes MR, HF, E2VID, and DAVIS frames; the caption identifies (c) and (d) as qualitatively matching frame (a).Source: Figure 6 · p. 6, §4.2 · p. 6
  • HF can accumulate noise during fast motion, whereas MR and E2VID reject fast-motion noise; HF also reconstructs slow-moving objects with longer memory at the cost of noise and bleeding.HF’s temporal receptive field is set by its cutoff frequency; MR and E2VID have implicit memory tied to their integration windows or event tensors.Source: §4.2 · p. 8
  • HF preserves full-resolution fine detail, while MR and E2VID spatial smoothing and color-channel upsampling can reduce sharpness.HF preserves the Bayer pattern for demosaicing; MR and E2VID reconstruct color channels independently at quarter resolution before upsampling.Source: §4.1 · p. 6, §4.2 · p. 8
  • All three reconstructions show dark and bright regions clearly in HDR scenes where DAVIS frames have limited dynamic range.The passage attributes the frame limitation to uniform exposure duration across pixels.Source: §4.2 · p. 8
  • In low light, MR, HF, and E2VID reconstructions avoid the motion blur seen in DAVIS frames.This comparison is reported for challenging low-light conditions.Source: §4.2 · p. 8

4.3. Application of Reconstructions

  • Color E2VID reconstructions qualitatively improve YOLO object detection compared with grayscale E2VID images.The paper also notes that reconstructions may be used to generate proxy labels for transfer to events.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

The conclusion presents CED as a diverse dataset of color frames and events, an ESIM-based color event simulator, and an evaluation of color reconstruction methods. It anticipates that these resources will inspire future work with color events.

5. Conclusion

  • CED provides color frames and events across diverse scenes, motions, and lighting conditions.The conclusion identifies it as the first Color Event Camera Dataset.Source: §5 · p. 8
  • The paper releases an open-source color event camera simulator based on ESIM.Source: §5 · p. 8
  • The paper adapts three state-of-the-art event-based image reconstruction methods for color video and compares their strengths and weaknesses.Source: §5 · p. 8
Loading 1904.10772v1…