PaperlayerView exact revision on arXiv

Source-linked AI summary

High Speed and High Dynamic Range Video with an Event Camera

Henri Rebecq, René Ranftl, Vladlen Koltun, Davide Scaramuzza

arXiv:1906.07165v1cs.CV

TL;DR

Reconstructing intensity video from noisy event streams is ill-posed, while handcrafted priors can produce unrealistic results. This paper learns reconstruction with a recurrent network and reports over 20% better image quality than prior methods, while enabling downstream computer vision applications.

  • Problem

    Event-stream reconstruction is ill-posed because real-camera noise undermines naive integration, while handcrafted priors can produce unrealistic reconstructions and artifacts.

  • Method

    A recurrent convolutional network learns events-to-video reconstruction from simulated event sequences using supervised training and a perceptual loss.

  • Results

    >20% improvement in image quality over state-of-the-art reconstruction methods, with downstream classification and visual-inertial odometry outperforming event-specific algorithms.

  • Takeaways & Limitations

    The reconstructions provide a practical intermediate representation that bridges event cameras with mainstream computer vision and supports high-framerate, high-dynamic-range, and color video.

  • Takeaways & Limitations

    Training simulations contain only globally homographic motion, although the network generalizes to scenes with arbitrary motions.

Abstract

from arXiv · show

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. We also demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data.

1 INTRODUCTION

The paper introduces a recurrent network that learns high-quality video reconstruction directly from event streams, avoiding handcrafted priors. It demonstrates high-framerate, high-dynamic-range, and color video reconstruction, plus improved downstream vision applications.

  • Motivation: Event cameras asynchronously report brightness-change events with time, location, and polarity, offering highly compressed visual data and high temporal resolution.Their measurements differ fundamentally from conventional fixed-rate intensity images.
  • Motivation: Real-camera noise and model mismatch make reconstruction ill-posed, while naive integration accumulates noise and handcrafted priors can create unrealistic artifacts.These limitations had prevented convincing high-quality video reconstruction from event data.
  • Method: The proposed recurrent network learns video reconstruction from large-scale simulated event data and uses a perceptual loss to encourage natural image statistics without handcrafted smoothness priors.The authors also report that simulation-trained networks generalize remarkably well to real events.
  • Capabilities: > 5,000 frames per second videos of high-speed phenomena demonstrate the network’s use of event cameras for capturing details beyond conventional cameras.The introduction also identifies high-dynamic-range and color-video reconstruction as demonstrated settings.
  • Applications: The reconstructions support object classification and visual-inertial odometry, outperforming state-of-the-art algorithms designed specifically for event data.They serve as an intermediate representation that allows mainstream computer-vision techniques to process event data.

2 RELATED WORK

Prior events-to-video methods established intensity recovery through event-based geometric reasoning, direct integration, or learned patch representations. This work instead reconstructs temporally consistent videos with a recurrent network and demonstrates downstream vision applications using those reconstructions.

  • 2 RELATED WORK: Early studies showed that intensity information could be recovered from events by reconstructing images for rotation estimation in static scenes.These methods exploited equations provided by events collected during camera motion.
  • 2 RELATED WORK: Direct event-integration methods avoid assumptions about scene structure or motion dynamics and naturally support arbitrarily high framerates.Munda et al. combined manifold-based energy minimization, direct integration, and total variation regularization for real-time GPU reconstruction.
  • 2 RELATED WORK: Barua et al. learned a dictionary from simulated data to map integrated-event patches to image gradients, followed by Poisson integration.Their approach reconstructed intensity images from small event patches rather than synthesizing long temporal streams.
  • 2 RELATED WORK: In contrast, this work uses a recurrent network to synthesize temporally consistent video from several seconds of events and learns pixel-wise intensity directly.It does not reconstruct individual images from small event windows or map patches to image-gradient dictionaries.
  • 2 RELATED WORK: Downstream vision applications based on event-to-video reconstructions had not been demonstrated before this work.The passage identifies this as a prior gap in the literature.

3 VIDEO RECONSTRUCTION

The method reconstructs image sequences from fixed-size event windows using a recurrent convolutional network, with voxel-grid event encoding and training on synthetic data with perceptual and temporal-consistency losses.

  • Video reconstruction: Event streams are partitioned into non-overlapping windows of N events, converted into event tensors, and processed recurrently to update an internal state and produce each reconstructed image.The network receives each window ε_k together with the previous state s_{k−1}.
  • Event representation: Each window’s events are represented in a spatio-temporal voxel grid with B = 5 temporal bins, using polarity interpolation between neighboring voxels.The event duration is discretized into temporal bins, and normalized timestamps determine the interpolation.
  • Training data: Training uses synthetic event sequences because large-scale paired event-image data are unavailable and conventional-camera ground truth is inadequate for high-speed or high-dynamic-range scenes.ESIM generates events and corresponding ground-truth images from MS-COCO scenes under simulated camera motion.
  • Training data: The synthetic training set contains 1,000 sequences of 2 seconds each, but includes only globally homographic motion.The authors report that the network nevertheless generalizes to arbitrary motions in later sections.
  • Network architecture: The recurrent fully convolutional architecture combines UNet-style encoders and decoders, skip connections, residual blocks, and stacked ConvLSTM states.The implementation uses NE = 3 encoder layers, NR = 2 residual blocks, and Nb = 32 head-layer output channels.

4 EVALUATION

The evaluation uses real event sequences with ground-truth frames to compare reconstruction quality and temporal consistency against several state-of-the-art methods. The proposed method achieves substantially better quantitative and qualitative results, including higher SSIM, lower LPIPS, cleaner details, fewer artifacts, and stronger temporal consistency.

  • Dataset and Protocol: The evaluation uses seven Event Camera Dataset sequences containing 1,670 ground-truth frames recorded with a DAVIS240C sensor.Ground-truth grayscale frames are available at 20 Hz.
  • Dataset and Protocol: The method is compared with SOFIE, HF, and MR for image reconstruction quality and temporal consistency.HF and MR use author-provided code with manually tuned parameters on the evaluated sequences.
  • Results and Discussion: The reconstructions preserve small details remarkably well and avoid ghosting and bleeding-edges artifacts visible in competing methods.These qualitative differences are illustrated in the comparison figures.
  • Results and Discussion: The method outperforms competing approaches in temporal consistency, while ground-truth sequences retain a nonzero temporal error from optical-flow errors and occlusions.Lower temporal error indicates higher temporal consistency, and the ground-truth error remains significantly lower than that of all reconstruction methods.

5 APPLICATIONS

The method reconstructs high-framerate, high-dynamic-range, and color videos from events, then uses the reconstructions as natural-image representations for conventional vision algorithms. These applications reveal detailed fast motion, preserve event-camera dynamic range, and improve downstream classification and visual-inertial odometry.

  • 5.1 High Framerate Video Synthesis from Events: The method decompresses sparse event streams into videos with thousands of frames per second, revealing details invisible to the naked eye or consumer cameras despite noisy, artifact-ridden events.Experiments included rifle-shot objects and popped balloons recorded with a Samsung DVS Gen3 sensor; the rifle’s approximate muzzle velocity was 376 m/s.
  • 5.2 High Dynamic Range Video: Event-based reconstructions preserve the sensor’s high dynamic range, enabling visual detail in challenging scenes where conventional-camera frames are severely overexposed.Event cameras provide 140 dB dynamic range versus 60 dB for conventional cameras, and the study compares event reconstructions with smartphone-camera frames.
  • 5.3 Color Video: The method produces cleaner color reconstructions than the compared HF and MR methods, with qualitative behavior consistent with grayscale results.HF preserves the Bayer pattern and uses demosaicing, while MR receives the same color-reconstruction treatment as the proposed method.
  • 5.4 Applications to Conventional Vision Algorithms: Reconstructed images transfer event data into the natural-image domain, allowing conventional computer-vision algorithms and pretrained CNNs to perform downstream inference.The approach leverages mature image-domain architectures and labeled image datasets unavailable to event-data methods in comparable breadth.
  • 5.4 Applications to Conventional Vision Algorithms: 86.6% versus 70.0%: on N-Caltech101, the method outperforms HATS by a large margin in classifying natural event data across 101 object classes.On N-MNIST, it performs slightly worse than HATS, reaching 98.3% versus 99.1%, without hyperparameter tuning.
  • 5.4 Applications to Conventional Vision Algorithms: 0.15 m versus 0.27 m: the method’s median visual-inertial-odometry error is almost half that of UltimateSLAM (E+I), while competing reconstructions produce larger tracking errors.On N-Caltech101, end-to-end classification takes less than 10 ms, including sequence reconstruction ≤8 ms and object classification ≤2 ms on an NVIDIA RTX 2080 Ti GPU.

6 ANALYSIS

The analysis shows that the network provides real-time reconstruction with a quality–efficiency trade-off, while recurrent connections and temporal loss substantially improve reconstruction behavior. It also generalizes better than a vanilla RNN to short event windows and learns to preserve reconstructions when events cease.

  • Computational efficiency: Our method runs in real time, although HF and MR process event data roughly 5 times faster.HF requires filtering for reasonable raw reconstructions, which increases its per-frame computation time.
  • Architecture search: A search over 72 architecture combinations identifies an elbow-shaped accuracy–inference-time distribution, indicating a good trade-off between reconstruction quality and efficiency.The search varies encoders, residual blocks, feature channels, and skip connections.
  • Ablation studies: The temporal loss improves both temporal consistency and image reconstruction quality.The ablation compares the full network with a network trained using λT C = 0.
  • Ablation studies: 65% decrease in temporal error results from the recurrent connection, which also removes high-frequency blinking in reconstructions.Removing the recurrent connection substantially worsens video reconstruction quality and temporal consistency.
  • ConvLSTM versus vanilla RNN: With τ = 5 ms event windows, our network’s reconstruction quality decreases by ≤6% in SSIM, whereas the vanilla RNN produces poor-quality reconstructions.At τ = 50 ms, close to training conditions, the two networks perform similarly; shorter windows require greater reliance on internal memory.
  • Network memory: When events stop, the recurrent network learns during its first iterations to copy pixel values forward and leave the current reconstruction unchanged.This behavior was discovered from training rather than hardcoded into the architecture.

7 CONCLUSION

The paper presents a recurrent-convolutional events-to-video framework trained on simulated event data. It outperforms state-of-the-art reconstruction methods and supports high-framerate, high-dynamic-range, color reconstruction and downstream use as an intermediate representation.

  • 7 CONCLUSION: The proposed events-to-video framework uses a recurrent convolutional network trained on simulated event data.It reconstructs video directly from event streams.
  • 7 CONCLUSION: > 20% improvement over state-of-the-art reconstruction methods is achieved on real event data.The conclusion describes this as a large-margin improvement.
  • 7 CONCLUSION: The method synthesizes high-framerate, high-dynamic-range, and color video reconstructions from event data only.These capabilities are demonstrated within the proposed reconstruction framework.
  • 7 CONCLUSION: The reconstructions are effective as an intermediate representation for event data.The paper demonstrates their applicability beyond reconstruction itself.

APPENDIX A SUPPLEMENTARY VIDEO & CODE · APPENDIX B FORMAL NETWORK DESCRIPTION

The appendices direct readers to supplementary videos covering the paper’s reconstructions, experiments, ablations, and downstream applications, while formally describing the recurrent network’s stateful computation and initialization. The reconstruction code and a pretrained model are also released online.

  • APPENDIX A SUPPLEMENTARY VIDEO & CODE: The supplementary video presents reconstructions from the method on various event datasets alongside visual comparisons with several state-of-the-art methods.
  • APPENDIX A SUPPLEMENTARY VIDEO & CODE: It includes high-framerate videos of high-speed experiments, high-dynamic-range reconstructions, and color video reconstructions.
  • APPENDIX A SUPPLEMENTARY VIDEO & CODE: The supplementary materials show video illustrations of the ablation studies and VINS-Mono running on video reconstructed from events.
  • APPENDIX A SUPPLEMENTARY VIDEO & CODE: Additional qualitative results cover object detection with YOLOv3 and monocular depth prediction with MegaDepth, neither previously shown with event data.
  • APPENDIX A SUPPLEMENTARY VIDEO & CODE: The authors release the reconstruction code and a pretrained model to spur further research.
  • APPENDIX B FORMAL NETWORK DESCRIPTION: The formal network description takes an event tensor E_k at time step k and the previous network state as inputs, then applies a specified sequence of operations.
  • APPENDIX B FORMAL NETWORK DESCRIPTION: The formalism defines ⊕ as an element-wise-sum skip connection and σ as the sigmoid function, with indexed operations specified over encoder and recurrent layers.
  • APPENDIX B FORMAL NETWORK DESCRIPTION: At the first iteration, k = 0, the hidden states for every encoder layer are initialized to zero.

APPENDIX C WHY USE SYNTHETIC TRAINING DATA?

The authors use synthetic event data because simulation cheaply provides diverse scenes and motions, avoids inadequate real-camera ground truth in high-speed and HDR conditions, and enables sensor-threshold randomization for improved generalization.

  • Motivations for synthetic training data: Synthetic data captures a large variety of scenes and motions at very little cost.Simulation provides a scalable source of varied training examples.
  • Motivations for synthetic training data: Synthetic data avoids motion blur and other ground-truth problems that conventional cameras face in high-speed and HDR conditions.These are precisely conditions in which event sensors excel.
  • Motivations for synthetic training data: Randomizing event-sensor contrast thresholds during simulation improves generalization across different sensor configurations.The authors illustrate this motivation by comparing networks trained only on real DAVIS240C data and only on simulated data.

APPENDIX D ADDITIONAL RESULTS · D.1 Results on Synthetic Event Data

On synthetic event sequences, the proposed method is quantitatively and qualitatively compared with MR and HF. Simulated events improve all methods over real data, while the method’s state-of-the-art advantage is preserved and slightly increased.

  • D.1 Results on Synthetic Event Data: The study quantitatively compares the proposed reconstruction method with MR and HF on synthetic event sequences.Results are reported in Table 9.
  • D.1 Results on Synthetic Event Data: Qualitative reconstruction results on the synthetic dataset are presented in Fig. 18.
  • D.1 Results on Synthetic Event Data: All methods perform better on synthetic data than on real data.
  • D.1 Results on Synthetic Event Data: The improvement on synthetic data is attributed to simulated events being free of noise.
  • D.1 Results on Synthetic Event Data: The performance gap between the proposed method and the state of the art is preserved on synthetic data.
  • D.1 Results on Synthetic Event Data: 24% improvement in SSIM is reported for the proposed method over the state of the art on synthetic data.
  • D.1 Results on Synthetic Event Data: 56% decrease in LPI is reported for the proposed method over the state of the art on synthetic data.

D.2 Additional Qualitative Results on Real Data · APPENDIX E OBJECT CLASSIFICATION

Real-data results show that the reconstruction method preserves fine detail, avoids ghosting, and recovers high-dynamic-range content under difficult lighting. The appendix also specifies reconstruction and classifier configurations for N-MNIST, N-CARS, and N-Caltech101 evaluation.

  • D.2 Additional Qualitative Results on Real Data: Qualitative results cover Event Camera Dataset sequences, Bardow et al. sequences, and HDR reconstructions across multiple public datasets.The supplementary video presents additional results more effectively than still images.
  • D.2 Additional Qualitative Results on Real Data: The method reconstructs fine textures on DAVIS240C sequences while avoiding ghosting effects in shape sequences.The comparison uses MR [4] and HF [5] on Event Camera Dataset sequences with ground-truth frames.
  • D.2 Additional Qualitative Results on Real Data: Under direct sunlight, events capture the scene’s full dynamic range while conventional auto-exposed frames become under- or over-exposed.The example comes from the MVSEC automotive dataset.
  • D.2 Additional Qualitative Results on Real Data: At night and across indoor, outdoor, and night-driving sequences, the method recovers details obscured by conventional-camera under- or overexposure.These examples use MVSEC and other publicly available datasets.
  • APPENDIX E OBJECT CLASSIFICATION: N = 1,000 events define each N-MNIST reconstruction window, and the final reconstructed image is binarized at threshold 0.5 for classification.Reconstructed values lie in [0, 1], and train and test images are normalized to match MNIST statistics.
  • APPENDIX E OBJECT CLASSIFICATION: The N-MNIST classifier uses two convolutional layers, max pooling with dropout, and fully connected layers of 128 and 10 neurons.The convolutional layers use stride 5 and output channels 32 and 64, respectively.
  • APPENDIX E OBJECT CLASSIFICATION: 15 epochs of N-MNIST training use cross-entropy loss, ADAM, and learning rate 0.001.The classifier is trained on the final image reconstructed from each event sequence.
  • APPENDIX E OBJECT CLASSIFICATION: N-CARS uses 20 ms event windows and final reconstructed images with an ImageNet-pretrained ResNet18 classifier, while N-Caltech101 uses 10,000-event windows and a two-thirds/one-third random split.N-CARS fine-tuning uses an added two-neuron output layer; N-Caltech101 contains 5,863 training and 2,396 testing sequences.

APPENDIX F VISUAL-INERTIAL ODOMETRY

The appendix provides additional visual-inertial odometry results, tracking mean translation and rotation errors versus travelled distance. It compares the proposed approach with UltimateSLAM variants across aggregate and sequence-specific evaluations.

  • Aggregate evaluation: Overall mean translation error in meters and mean rotation error in degrees are averaged across all evaluation datasets.These aggregate error evolutions are shown in Fig. 25.
  • Translation sequences: Mean translation and rotation errors are reported versus travelled distance for four translation sequences: shapes, poster, boxes, and dynamic.Fig. 26 lists these sequences from top to bottom.
  • 6dof and HDR sequences: Mean translation and rotation errors are reported versus travelled distance for four 6dof sequences and the hdr boxes sequence.The evaluated sequences are shapes 6dof, poster 6dof, boxes 6dof, dynamic 6dof, and hdr boxes.
Loading 1906.07165v1…