PaperlayerView exact revision on arXiv

Source-linked AI summary

Events-to-Video: Bringing Modern Computer Vision to Event Cameras

Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza

arXiv:1904.08298v1cs.CV

TL;DR

Event cameras provide valuable sensing properties but emit asynchronous event streams that conventional computer vision techniques cannot directly use. The paper reconstructs natural videos with a recurrent network trained on simulated events, then applies off-the-shelf vision algorithms. Reconstruction quality improves by more than 20%, and the same strategy outperforms event-specific methods for object classification and visual-inertial odometry.

  • Problem

    Asynchronous event streams differ fundamentally from natural images, preventing direct use of existing computer vision techniques and pretrained networks.

  • Method

    A recurrent network learns to reconstruct natural videos from event streams using simulated event data and a perceptual loss encouraging natural-image statistics.

  • Results

    The method improves image quality by > 20% over state-of-the-art reconstruction methods and outperforms event-specific algorithms on object classification and visual-inertial odometry.

  • Takeaways & Limitations

    Reconstructed event videos provide a bridge for applying mature conventional computer vision algorithms to event-camera data.

  • Takeaways & Limitations

    Extreme electronic noise not modeled in simulation can cause reconstruction errors that propagate to subsequent reconstructions.

Abstract

from arXiv · show

Event cameras are novel sensors that report brightness changes in the form of asynchronous "events" instead of intensity frames. They have significant advantages over conventional cameras: high temporal resolution, high dynamic range, and no motion blur. Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events. In this work, we take a different view and propose to apply existing, mature computer vision techniques to videos reconstructed from event data. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. Our experiments show that our approach surpasses state-of-the-art reconstruction methods by a large margin (> 20%) in terms of image quality. We further apply off-the-shelf computer vision algorithms to videos reconstructed from event data on tasks such as object classification and visual-inertial odometry, and show that this strategy consistently outperforms algorithms that were specifically designed for event data. We believe that our approach opens the door to bringing the outstanding properties of event cameras to an entirely new range of tasks. A video of the experiments is available at https://youtu.be/IdYrC4cUO0I

1. Introduction

Event cameras offer high-speed, high-dynamic-range sensing but produce asynchronous event streams that conventional vision methods cannot directly process. The paper reconstructs natural videos from events so off-the-shelf computer vision can be applied, achieving strong reconstruction and downstream-task results.

  • Motivation: Event cameras asynchronously encode brightness-change time, location, and polarity, providing 140 dB dynamic range, low motion blur, and latency as low as one microsecond.These properties make them useful in conditions challenging for conventional cameras.
  • Problem: Existing computer vision techniques cannot directly process asynchronous event streams, motivating specialized event-data algorithms.Image-like event representations still differ from natural images, limiting transfer of existing deep networks.
  • Approach: The paper learns a mapping from event streams to natural videos, establishing a bridge between event-camera vision and conventional computer vision.The reconstructed videos are intended for direct use with off-the-shelf computer vision techniques.
  • Approach: The recurrent reconstruction network learns directly from large amounts of simulated event data rather than relying on handcrafted smoothness priors.A perceptual loss encourages reconstructed images to share natural-image statistics.
  • Results: The network generalizes from simulated to real events and reconstructs videos with substantially better image quality than prior methods.The contribution is described as outperforming the state of the art by a large margin.
  • Results: Applying off-the-shelf algorithms to reconstructed videos outperforms event-specific state-of-the-art methods for object classification and visual-inertial odometry.These applications demonstrate the proposed bridge beyond reconstruction quality alone.
  • Implications: The approach is presented as a way to bring mature computer vision algorithms, pretrained weights, and architectures to event cameras.The stated scope includes leveraging event cameras’ temporal resolution, dynamic range, and lack of motion blur across more applications.

2. Related Work

Prior events-to-video work includes model-based reconstruction, direct event integration, and learning-based image reconstruction. This paper instead reconstructs temporally consistent videos from long event streams and evaluates them in downstream vision applications.

  • Earlier reconstruction: Earlier approaches reconstructed intensity images, optic flow, or motion from events, with video reconstruction later extended to dynamic scenes.The related work includes bio-inspired networks, Kalman filtering, and variational energy minimization.
  • Direct integration: Direct event-integration methods avoid scene-structure or motion-dynamics assumptions and can reconstruct videos at arbitrarily high framerates.One cited approach combines event integration with total-variation regularization for real-time GPU performance.
  • Learning-based reconstruction: A learning-based method used K-SVD on simulated data to map integrated-event patches to image gradients before Poisson integration.This approach reconstructs intensity images from small event windows rather than directly estimating pixel intensities.
  • This work: Unlike patch-based image reconstruction, the proposed method uses a recurrent network to reconstruct a temporally consistent video from several seconds of events.It learns pixel-wise intensity estimation directly.
  • Downstream applications: Prior work had not demonstrated downstream vision applications based on events-to-video reconstructions, according to the paper.The paper addresses this gap through object classification and visual-inertial odometry.

3. Video Reconstruction Approach

The approach reconstructs images from event windows by combining a fixed-size event tensor with recent reconstructions in a recurrent convolutional network trained on simulated data.

  • Event Representation: Each event triggers at a pixel when brightness changes reach a threshold, producing asynchronous events with location, time, and polarity.
  • Overview: The event stream is partitioned into non-overlapping windows containing N events, and each window is fused with the previous K reconstructed images to produce a new image.
  • Event Representation: Each event window is converted into a fixed-size 3D voxel tensor by discretizing its duration into B temporal bins and distributing event polarity to nearby voxels.
  • Training Data: Training uses supervised synthetic event sequences with corresponding ground-truth images because no large-scale real dataset provides both event data and suitable images.
  • Network Architecture: The recurrent convolutional network uses a UNet with skip connections and propagates intensity information by updating previous reconstructions with new events.
  • Training Objective: The network is trained with LPIPS perceptual loss so reconstructed images acquire features close to those of natural images.

4. Evaluation

The evaluation compares the reconstruction method with prior approaches on real event-camera data using quantitative and qualitative evidence. It reports higher image quality, finer details, fewer artifacts, and recovery of details in challenging lighting, while also documenting latency and failure cases.

  • Evaluation setup: The evaluation uses real event sequences and compares the method with SOFIE, HF, and MR using MSE, SSIM, and LPIPS.SOFIE is evaluated qualitatively, while HF and MR use author-provided code with tuned parameters.
  • Qualitative results: Qualitative comparisons show finer details, less noise, and fewer ghosting or bleeding-edge artifacts than competing reconstruction methods.Figure 4 highlights textures and bleeding edges, while Figure 5 reports cleaner, more detailed results.
  • Quantitative results: 21% higher SSIM and 23% lower LPIPS than the best prior methods are reported across all datasets.The comparisons are against MR for SSIM and HF for LPIPS.
  • Challenging conditions: The method reconstructs image details in low-light scenes where conventional frames are severely degraded by using events that capture the scene’s full dynamic range.The example is a high-dynamic-range reconstruction from an event camera mounted on a car driving at night.
  • Failure cases: Noisy events generated by the sun can cause a local reconstruction error that becomes amplified in subsequent reconstructions.This failure case is shown on a sequence from the MVSEC dataset.
  • Limitations: Processing events in windows introduces latency ranging from 1 ms to 200 ms, depending on event rate and window size.The method can also fail under extreme electronic noise absent from simulation, with errors propagating to later reconstructions.

5. Downstream Applications

The paper applies reconstructed event videos to object classification and visual-inertial odometry using conventional computer vision algorithms. This strategy outperforms event-specific methods across both downstream tasks, with real-time classification demonstrated on N-Caltech101.

  • Object Classification and Visual-Inertial Odometry: The approach reconstructs images from event sequences, then trains or applies conventional vision models for downstream tasks.Classification uses CNNs on reconstructed images, including ImageNet-pretrained ResNet-18 models for N-Caltech101 and N-CARS; VIO applies VINS-Mono to reconstructed videos.
  • Object Classification: 98.3% versus 99.1%: the method performs slightly worse than HATS on synthetic N-MNIST.The paper attributes this to the synthetic nature of N-MNIST and the limited advantage of reconstruction over handcrafted features there.
  • Object Classification: 91% versus 90.4% for HATS: the method performs slightly better on the N-CARS binary classification task.N-CARS is described as almost saturated in accuracy, limiting the size of the improvement.
  • Object Classification: 86.6% versus 70.0%: reconstructed-event classification outperforms HATS on the challenging N-Caltech101 dataset.The task classifies natural event data into 101 object classes.
  • Runtime: End-to-end N-Caltech101 classification takes less than 10 ms on an NVIDIA RTX 2080 Ti GPU.Sequence reconstruction takes ≤8 ms and object classification takes ≤2 ms.
  • Visual-Inertial Odometry: The method achieves a 0.15 m median translation error, versus 0.27 m for UltimateSLAM (E+I).It performs better on all sequences except ‘shapes 6dof’, where VINS-Mono fails to initialize because the sequence has very few features.
  • Visual-Inertial Odometry: VINS-Mono on the proposed reconstructions has lower tracking error than when run on competing MR and HF reconstructions.The median error is reported as three times as large for MR; the approach is acknowledged to be slower than UltimateSLAM.

6. Conclusion

The paper presents events-to-video reconstruction with a recurrent convolutional network trained on simulated event data. Applying off-the-shelf computer vision algorithms to the reconstructions outperforms event-specific methods for object classification and visual-inertial odometry.

  • The framework reconstructs videos from events using a recurrent convolutional network trained on simulated event data.
  • The method improves real-event reconstruction image quality by more than 20% over state-of-the-art reconstruction methods.
  • Off-the-shelf computer vision algorithms applied to reconstructed event videos outperform state-of-the-art event-specific algorithms on object classification and visual-inertial odometry.
  • The approach provides access to mature algorithms, modern deep architectures, and pretrained weights from large image datasets for event-camera applications.

A. Architecture Details

The reconstruction network uses a recurrent convolutional architecture organized as a UNet with encoder, residual, decoder, and skip-connection components.

  • The network is essentially a UNet composed of four encoder layers, two residual blocks, four decoder layers, and a final image prediction layer.
  • Symmetric skip connections connect the encoder and decoder portions of the architecture.
  • The encoder layers use strided convolutions with stride 2 and kernel size 5.

B. Initialization Phase

During initialization, the proposed method reconstructs scene appearance from few events, unlike direct event-integration methods that recover only intensity changes relative to an unknown initial image.

  • HF and MR recover intensity only up to the unknown initial image I0, producing edge images during initialization.
  • The proposed method reconstructs the scene reasonably well despite the low number of available events.
  • The method uses deep priors learned from simulated event data and embedded in the perceptual loss to support early reconstruction.

C. Why Use Synthetic Training Data?

Synthetic event data offers broad, inexpensive scene and motion variety while avoiding motion blur and HDR ground-truth problems. Randomized contrast thresholds also support generalization across sensor configurations.

  • Simulation captures a large variety of scenes and motions at little cost.
  • Synthetic data avoids motion blur and HDR limitations that make conventional-camera ground truth poor in event-camera conditions.
  • Randomizing event-sensor contrast thresholds improves generalization to different sensor configurations.
  • The reconstruction network uses a UNet with four encoder layers, two residual blocks, four decoder layers, and symmetric skip connections.
  • After 0.5 seconds, event-integration methods produce incorrect intensity differences, edge images, or ghosting, unlike the authors’ network.

D.2. Performance Analysis

The performance analysis compares reconstruction methods using event-processing rate and frame-synthesis time, while accounting for differences in processing modality. The authors report that their method is real-time-capable despite synthesizing fewer images per second.

  • The comparison is difficult to make directly because HF, MR, and the authors’ method process event data differently.
  • Mean event processing rate divides total dataset processing time by the dataset’s number of events.
  • For the authors’ method, frame-synthesis time is the time required to process N events.
  • For MR, frame-synthesis time is the time required to process NMR events.
  • HF’s reported synthesis time consists of bilateral-filtering time because single-event processing is negligible by comparison.
  • The authors’ method synthesizes fewer images per second than MR and HF but can run in real time while providing state-of-the-art reconstruction quality.

E.2. Results on Synthetic Event Data

On synthetic event sequences, the authors report a substantial reconstruction-quality advantage over state-of-the-art methods, while noting a fundamental quantization limit on perfect recovery.

  • 24% improvement in SSIM and 56% decrease in LPIPS preserve or increase the performance gap over state-of-the-art methods on synthetic data.
  • All methods perform better on synthetic than real data because simulated events are free of noise.
  • Perfect reconstruction remains impossible even for noise-free events because the sensor’s contrast threshold imposes a quantization limit.

E.3. Additional Qualitative Results on Real Data

Additional results show qualitative reconstruction behavior across real, HDR, and event-derived image datasets, alongside downstream visual-inertial odometry evaluation. The reconstructions preserve fine detail and support comparisons with conventional and event-specific approaches.

  • Real-data comparisons: Figures 15 and 16 compare the method with MR and HF on real event sequences, emphasizing preserved textures and reduced ghosting.
  • HDR reconstruction: On automotive HDR sequences, reconstructed events retain scene dynamic range when standard frames are underexposed or overexposed.
  • HDR reconstruction: At night, event-based reconstructions recover details that are not visible in severely degraded standard frames.
  • Event-derived image data: N-Caltech101 reconstructions preserve many details and statistics of the original images under planar motion.
  • Visual-inertial odometry: Figures 20–22 show mean translation and rotation errors over travelled distance for the evaluated visual-inertial odometry sequences.
Loading 1904.08298v1…