Source-linked AI summary
Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks
S. Mohammad Mostafavi I., Lin Wang, Yo-Sung Ho, Kuk-Jin Yoon
TL;DR
Event cameras offer high temporal resolution and dynamic range but produce asynchronous events rather than intensity images, limiting direct use of conventional algorithms. The paper applies conditional GANs with event stacking to reconstruct HDR images and high frame rate videos, demonstrating reconstruction under extreme illumination and fast motion, with videos theoretically reaching up to 1 million FPS.
Problem
Event cameras output asynchronous events instead of intensity images, so existing image-processing algorithms cannot be directly applied to event data.
Method
The paper uses conditional GANs with SBT and SBE event stacking methods to translate adjustable event-stream portions into images and videos.
Results
Up to 1 million FPS is demonstrated as possible, alongside HDR images and less motion-blurred high frame rate videos under extreme illumination and fast motion.
Takeaways & Limitations
Event cameras can generate useful HDR intensity images and high frame rate videos from pure events for broader vision applications.
Takeaways & Limitations
SBE produces poor HDR reconstruction when an interval contains too few events and has another flaw when too many events occur in one time frame.
Abstract
from arXiv · showhide
Event cameras have a lot of advantages over traditional cameras, such as low latency, high temporal resolution, and high dynamic range. However, since the outputs of event cameras are the sequences of asynchronous events overtime rather than actual intensity images, existing algorithms could not be directly applied. Therefore, it is demanding to generate intensity images from events for other tasks. In this paper, we unlock the potential of event camera-based conditional generative adversarial networks to create images/videos from an adjustable portion of the event data stream. The stacks of space-time coordinates of events are used as inputs and the network is trained to reproduce images based on the spatio-temporal intensity changes. The usefulness of event cameras to generate high dynamic range(HDR) images even in extreme illumination conditions and also non blurred images under rapid motion is also shown.In addition, the possibility of generating very high frame rate videos is demonstrated, theoretically up to 1 million frames per second (FPS) since the temporal resolution of event cameras are about 1μs. Proposed methods are evaluated by comparing the results with the intensity images captured on the same pixel grid-line of events using online available real datasets and synthetic datasets produced by the event camera simulator.
1. Introduction
Event cameras provide microsecond-scale temporal resolution and high dynamic range, but output asynchronous intensity changes rather than images. The paper uses conditional GANs and event stacking to reconstruct HDR images and very high frame rate videos, evaluated on real and synthetic data.
- Event cameras transmit asynchronous space-time coordinates of intensity changes instead of fixed-rate intensity frames.
- Microsecond-scale latency, around 1 µs temporal resolution, and high dynamic range distinguish event cameras from traditional cameras.
- The paper targets intensity-image generation from events for applications including object detection, tracking, 3D reconstruction, and SLAM.
- The proposed framework uses pure events to generate HDR images and high frame rate videos without motion blur under fast motion and extreme illumination.
- SBT and SBE stack events by shifting over time or by event count to support HDR and high frame rate reconstruction.
- Experiments use DAVIS real data, an open dataset with more than 17K images, and a synthetic dataset containing 17K simulator-generated images.
2. Related work
Prior work reconstructed intensity images from events using recurrent, probabilistic, sparse-dictionary, variational, and hybrid approaches. The paper identifies limited photorealistic HDR reconstruction and a lack of qualitative cGAN research on event data, while positioning its work as the first application of GANs to event data.
- Earlier methods reconstructed intensity and motion from events using recurrent maps, probabilistic filtering, sparse dictionaries, variational regularization, and denoising.
- Prior pure-event reconstructions were mainly non-photorealistic, while a hybrid autoencoder method handled normally illuminated scenes but failed on HDR scenes under extreme illumination.
- Deep learning studies used event data for steering, relocalization, and object detection, often incorporating APS images or APS-derived pseudo labels.
- The authors claim to be the first to apply generative adversarial networks to event data.
- The paper states that no qualitative research had shown cGAN effectiveness on event data.
3. Proposed method
The method uses conditional GANs for event-to-image translation, adapting image-translation models to event data. Its design addresses the distinct structure of event inputs while learning outputs conditioned on those events.
- Conditional GANs learn a mapping from observed input x and noise vector z to output image y using an adversarially trained discriminator.
- The paper uses cGANs to reconstruct HDR and high temporal resolution images and videos from event data.
- Because event data differ substantially from conventional vision inputs, the authors propose new methods to provide suitable cGAN inputs.
3.1. Event stacking
Event streams are converted into multi-frame stacks that preserve spatial polarity changes while organizing events by time or event count. These stacks support adjustable video timing, although time-based stacking can fail when event density is insufficient or excessive.
- Each event is represented by pixel coordinates, a timestamp, and polarity indicating the sign of brightness change.
- Stacking Based on Time (SBT): Time-based stacking divides an event interval into n equal portions, accumulates polarity values into grayscale frames, and stacks them as network input.The resulting stack retains some temporal information, with larger n preserving more of it.
- Stacking Based on Time (SBT): SBT loses event timing within each interval and struggles when the interval contains too few events for image reconstruction.This limitation arises from the event camera’s lack of events during little or no scene or camera movement.
- Stacking Based on Time (SBT): SBT also faces difficulties when a single time frame contains too many events.
- Stacking Based on the number of Events (SBE): Event-count-based stacking merges successive groups of Ne events into n frames, producing an n-frame stack containing nNe events.This method is described as more asynchronous and as addressing SBT’s limitations.
- Video reconstruction: For video reconstruction, the output frame rate is 1/ts and can theoretically reach 1 million FPS because event-camera temporal resolution is about 1 µs.Overlapping adjacent stacks with duration ∆t′ much greater than ts naturally enforces temporal consistency for nearby frames.
3.2. Network architectures
The framework uses a U-net generator with skip connections to preserve event details and a PatchGAN-like discriminator to assess event-to-intensity domain transfer. Training combines adversarial and L1 objectives, with noise enabling event-conditioned image generation.
- 3.2.1 Generator architecture: The generator uses a U-net with skip connections to preserve high-frequency event details during sparse-to-dense HDR image translation.The architecture addresses information loss and output noise associated with repeated downsampling and upsampling.
- 3.2.2 Discriminator architecture: The adversarial objective trains the generator to produce images the discriminator classifies as real while the discriminator learns to distinguish generated images from real ones.The discriminator’s role is to classify event-generated images as real or fake, creating the adversarial training interaction.
- 3.2.2 Discriminator architecture: The discriminator takes the original APS image and event-generated image, then evaluates whether the generated image respects event-to-intensity domain transfer.Its design is similar to PatchGAN and uses generator feature maps as a condition.
- 3.2.2 Discriminator architecture: An L1 regularizer reduces blurring and directs the discriminator toward high-frequency structure in event-generated images.The L1 term is part of the event-to-image translation loss.
- 3.2.2 Discriminator architecture: Gaussian noise lets the network learn a mapping from event e and noise ϵ to generated image g, matching event-conditioned distributions and producing more deterministic outputs.The parameter λ adjusts the learning rate or regularization weighting in the objective.
3.3. Dataset preparation
The study uses real DAVIS data, a newly collected public dataset, and ESIM-simulated sequences to train and evaluate event-to-image/video translation across varied scenes.
- 3.3. Dataset preparation: Three dataset groups combine prior real-world scenes, newly collected DAVIS recordings, and sequences generated by the ESIM event-camera simulator.The datasets support varied training, testing, and public-release purposes.
- 3.3. Dataset preparation: DAVIS recordings provide event and APS data on the same pixel grid, reducing rectification and warping requirements when pairing modalities.The real scenes include indoor and outdoor environments with camera rotations and translations.
- 3.3. Dataset preparation: The training data pair stacked events with APS or ESIM ground-truth frames from real-world and simulated scenes.The preparation is intended to prevent the network from learning improper properties of the APS data.
- 3.3. Dataset preparation: Blurred APS images are excluded using BRISQUE scores and manual inspection, while events from black-and-white training regions are removed.These filtering steps are used to train HDR image generation from relevant event information.
4. Experiments and evaluation
The experiments evaluate event-to-image reconstruction on real and simulated datasets, compare stacking strategies, and benchmark against prior methods. Results indicate that SBE is generally more robust than SBT, while larger event stacks improve simulated-data reconstruction and outperform prior methods on BRISQUE scores.
- SBT versus SBE: 17K event stack–APS image pairs trained the SBE and SBT comparison, with Δt = 0.03s for SBT, 60K events per SBE stack, and n = 3 for both.These settings were chosen to compare stacking methods while holding the number of frames per stack constant.
- SBT versus SBE: SBE generally produced better reconstructions than SBT, and both methods generated images close to APS references across varied real-world sequences.The reconstructions recovered shapes and appearances of people and buildings, although APS references can suffer motion blur and limited dynamic range.
- Simulated datasets: n = 3 produced better simulated-data results than n = 1, consistent with preserving more temporal information in the event stack.With n = 1, accumulated events distorted a face and the top of a building.
- Comparison to relevant works: Our method achieved better BRISQUE scores than MR and IE on the face, jumping, and ball sequences.The visual and quantitative comparisons reported more details in the reconstructed outputs, with BRISQUE used because no ground-truth images were available.
5. Discussion
The discussion presents event stacks as sufficient for reconstructing HDR images and high-frame-rate video under difficult imaging conditions. The method recovers details under extreme illumination and reduces motion blur during rapid motion, with motion generation demonstrated up to 1 million fps.
- Applications: The reconstructed intensity images are positioned for downstream tasks including object recognition, tracking, 3D reconstruction, self-driving, and SLAM.The discussion connects this downstream use to the event camera’s high temporal resolution and high dynamic range.
- Events to HDR image: Event stacks contain information for HDR reconstruction in regions that APS images cannot reveal under low or high illumination.The proposed method is shown generating HDR images against direct sunlight and other extreme illumination conditions.
- Events to HDR image: Using three frames per stack is more robust than one-frame stacks, whose accumulated events can distort reconstructed images.The simulated-data comparison attributes the improvement to preservation of more temporal information.
- Comparison to relevant works: Compared with prior results, the method produces more details and more natural grayscale variation on dynamic face, jumping, and ball sequences.The comparison includes details such as faces, beards, and jumping poses.
- Events to high frame rate video: A star-shaped object on a fan rotating at 13000 RPM was reconstructed with clear motion and much less blur in the high-frame-rate video.The paper reports theoretical motion generation up to 1 million fps, with qualitative results deferred to supplementary material.
6. Conclusion
The cGAN-based framework reconstructs HDR, non-blurred intensity images and high-frame-rate videos from pure events. Experiments show robustness under extreme illumination and rapid motion, with performance exceeding other reconstruction methods on public datasets.
- The cGAN-based approach reconstructs HDR, non-blurred intensity images and high-frame-rate videos from pure events.
- Two event-stacking methods, SBT and SBE, support both image and video reconstruction from events.
- The framework outperforms other reconstruction methods on publicly available datasets.
- HDR images remain possible under extreme illumination, while reconstructed images remain non-blurred under rapid motion.