Source-linked AI summary
Video to Events: Recycling Video Datasets for Event Cameras
Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carrió, Davide Scaramuzza
TL;DR
Event-based learning needs substantial training data, but event datasets are scarce because the sensors are relatively new. This paper converts existing conventional video datasets into synthetic events, showing transfer to real-event tasks and benefits from real-data fine-tuning, while acknowledging limitations from blurry source frames, unmodeled noise, and frame interpolation.
Problem
Event-based learning requires large training datasets, but such datasets are scarce and existing simulators may lack realism.
Method
The paper converts existing real-world video datasets recorded with conventional cameras into large-scale synthetic event-camera datasets.
Results
Models trained on synthetic events generalize well to real events, and fine-tuning them with real events improves object recognition and semantic segmentation results.
Takeaways & Limitations
Existing video datasets can support event-camera research, including challenging HDR and fast-motion scenarios and applications evaluated through recognition and segmentation.
Takeaways & Limitations
Blurry source frames can persist through interpolation, and the generative model does not account for noise in real event cameras.
Abstract
from arXiv · showhide
Event cameras are novel sensors that output brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high dynamic range (HDR), high temporal resolution, and no motion blur. Recently, novel learning approaches operating on event data have achieved impressive results. Yet, these methods require a large amount of event data for training, which is hardly available due the novelty of event sensors in computer vision research. In this paper, we present a method that addresses these needs by converting any existing video dataset recorded with conventional cameras to synthetic event data. This unlocks the use of a virtually unlimited number of existing video datasets for training networks designed for real event data. We evaluate our method on two relevant vision tasks, i.e., object recognition and semantic segmentation, and show that models trained on synthetic events have several benefits: (i) they generalize well to real event data, even in scenarios where standard-camera images are blurry or overexposed, by inheriting the outstanding properties of event cameras; (ii) they can be used for fine-tuning on real data to improve over state-of-the-art for both classification and semantic segmentation.
1. Introduction
Event cameras offer high dynamic range, low latency, and blur-free sensing, but event-based learning is constrained by scarce training data. The paper converts existing video datasets into synthetic event datasets and evaluates their transfer to real-event recognition and segmentation.
- Motivation: Event cameras asynchronously encode brightness changes with high dynamic range, microsecond-scale latency, and no motion blur.These properties make them useful in high-speed and high-dynamic-range conditions where standard cameras are challenged.
- Motivation: Large-scale event datasets are scarce, and the realism and real-camera generalization of networks trained on synthetic events remain open concerns.The shortage is linked to the novelty of event sensors, while existing simulators may lack realism.
- Approach: The proposed framework converts existing real-world video datasets into large-scale synthetic event-camera datasets.It aims to leverage the virtually unlimited supply of conventional video data for event-camera research.
- Results: Models trained on synthetic events generalize well to real events, including scenarios where standard-camera frames are blurry or overexposed.The reported advantage is attributed to inheriting event cameras’ high dynamic range and lack of motion blur.
- Evaluation: The method is evaluated on object recognition and semantic segmentation, including fine-tuning synthetic-event models with real data.The paper reports improvements over the state of the art for both classification and semantic segmentation.
2. Related Work
Related work spans limited event-camera benchmarks, deep-learning applications to event data, and methods for generating events from conventional video. Existing datasets cover recognition, action, pose, and driving-related tasks, while prior generation methods have realism or timestamp limitations.
- Event Camera Datasets: Available event-camera machine-learning datasets remain limited across classification, action recognition, human pose, and driving-related tasks.Examples include N-MNIST, N-Caltech101, N-CARS, ASL-DVS, DVS-Gesture, DHP19, and DDD17.
- Learning from Events: Deep learning for event data has been applied to classification, steering-angle regression, and semantic segmentation using specialized event representations and architectures.The cited progression includes recurrent networks, event-frame representations, and CNN-based approaches.
- Event Generation: Earlier event-generation methods used image-difference thresholds, per-pixel luminance estimation, or frame-based timestamps to synthesize events.The related methods include thresholding image differences and Pix2NVS, whose timestamps are clustered near frame timestamps.
3. Methodology
The method converts low-frame-rate conventional video into synthetic event streams by adaptive frame upsampling and event simulation, then represents those events for CNN-based learning.
- Event Generation: ESIM generates asynchronous, sparse events from the upsampled video by thresholding per-pixel changes in interpolated intensity.An event is produced when the magnitude of an intensity change exceeds the contrast threshold C.
- Frame Upsampling: The pipeline first adaptively upsamples low-frame-rate video to recover intensity signals at the temporal resolution required for event generation.The interpolation method supports arbitrary temporal resolution, while insufficient intermediate frames can cause brightness-signal aliasing.
- Representation and Learning: Synthetic events and original timestamped labels train a network using event windows preceding each label.The approach applies to datasets with precisely timestamped images and labels.
- Representation and Learning: The asynchronous event stream is converted into the Event Spike Tensor representation, enabling conventional CNN architectures to process it.The representation uses C = 15 temporal bins in this work.
4. Experiments
Experiments on object recognition and semantic segmentation show that synthetic events can generalize to real event data and support effective fine-tuning. Frame interpolation improves event quality, while synthetic-event models retain advantages in challenging imaging conditions.
- Overall evaluation: Synthetic-event models generalize to real data across object recognition and semantic segmentation, including blurry or overexposed standard-camera scenarios.The evaluation covers both tasks and reports generalization, fine-tuning gains, and robustness to degraded conventional frames.
- Object classification: Virtual camera motion over 2D image planes produces sim-N-Caltech101 event streams that can be compared directly with real N-Caltech101 events.The method follows saccadic motion and renders frames at arbitrary temporal resolution before event generation.
- Object classification: 75.1% versus 86.3% accuracy shows an 11.2% synthetic-to-real gap before contrast-threshold randomization.The comparison uses a ResNet-34 classifier trained on synthetic events versus a network trained on real events.
- Object classification: Fine-tuning on real events surpasses the real-data baseline, existing event-based methods, and standard-image state of the art at 94.7% classification accuracy.The pretrained synthetic-event network is fine-tuned with a reduced learning rate until convergence.
- Object classification: 68.7% test performance indicates that frame interpolation improves event quality when low-frame-rate video causes unrealistic event generation.Downsampling creates event-stream distortion, while interpolation reconstructs the original video before event generation.
- Semantic segmentation: 45.5% MIoU with EST and 48.2% with Alonso et al.’s representation approach 54.8% from real-event training, while fine-tuning improves performance by an average of 1.2%.Synthetic-only training remains within 4% of 89.8% overall accuracy at a 50 ms event window; fine-tuning reaches state-of-the-art performance after two epochs.
5. Known Limitations
The method is limited by blurry source frames, unmodeled real-camera noise, and artifacts inherited from frame interpolation, especially under large optical flow.
- Blurry video frames persist through interpolation and produce suboptimal results with the generative event model.
- High optical flow can cause repetitive structures to be copied or tires to collapse to linear interpolation, producing missing or incorrect events.
- The generative model does not account for noise present in real event cameras, leaving noise modeling as future work.
- Future improvements in frame interpolation can directly benefit the proposed method.
6. Conclusion
The paper converts existing video datasets into synthetic event datasets using frame interpolation and event generation. Models trained on these synthetic events generalize to real events, and fine-tuning with real events improves object recognition and semantic segmentation.
- The method converts existing video datasets into event datasets, addressing the scarcity of event-camera datasets.
- The approach combines neural-network frame interpolation with a generative model for events.
- Models trained on synthetic events generalize to real events, while fine-tuning with real events consistently improves object recognition and semantic segmentation.
8. Appendix
The appendix reports additional qualitative segmentation results from a model trained on synthetic events and fine-tuned on real data.
- Additional test-set examples use an EST-input network trained on synthetic events and fine-tuned on real data.Fine-tuning was performed for two epochs.