Source-linked AI summary
Event Enhanced High-Quality Image Recovery
Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, Wen Yang
TL;DR
Event cameras provide very high temporal resolution but require intensity reconstruction that jointly addresses denoising, deblurring, and super-resolution. The proposed eSL-Net uses an event-enhanced degeneration model and sparse learning to recover high-quality images, outperforming other methods and generating continuous frames at higher frame rates.
Problem
Event-camera intensity reconstruction is an ill-posed problem requiring denoising, deblurring, and super-resolution from asynchronous events and observed images.
Method
eSL-Net combines an event-enhanced degeneration model with sparse learning to reconstruct high-resolution sharp images from observed images and event sequences.
Results
The method outperforms other approaches in quantitative and visual reconstruction quality and generates continuous frames at 21 times the original APS frame rate.
Takeaways & Limitations
Without retraining, eSL-Net can be extended to high-frame-rate video recovery through a simple transformation of the event stream.
Takeaways & Limitations
At excessively high frame rates, few events occur between frames, making consecutive reconstructions subtly different and difficult to distinguish.
Abstract
from arXiv · showhide
With extremely high temporal resolution, event cameras have a large potential for robotics and computer vision. However, their asynchronous imaging mechanism often aggravates the measurement sensitivity to noises and brings a physical burden to increase the image spatial resolution. To recover high-quality intensity images, one should address both denoising and super-resolution problems for event cameras. Since events depict brightness changes, with the enhanced degeneration model by the events, the clear and sharp high-resolution latent images can be recovered from the noisy, blurry and low-resolution intensity observations. Exploiting the framework of sparse learning, the events and the low-resolution intensity observations can be jointly considered. Based on this, we propose an explainable network, an event-enhanced sparse learning network (eSL-Net), to recover the high-quality images from event cameras. After training with a synthetic dataset, the proposed eSL-Net can largely improve the performance of the state-of-the-art by 7-12 dB. Furthermore, without additional training process, the proposed eSL-Net can be easily extended to generate continuous frames with frame-rate as high as the events.
1 Introduction
Event cameras offer extremely high temporal resolution but produce noisy, low-spatial-resolution measurements and blurry APS frames, making unified denoising, deblurring, and super-resolution challenging. The paper proposes eSL-Net, an explainable sparse-learning network using events and APS frames to recover high-quality images and extend them to high-frame-rate video.
- Motivation: Event cameras capture brightness changes asynchronously with very low latency, but events lack absolute intensity measurements and are sensitive to noise.Their asynchronous mechanism also creates a physical burden for increasing spatial resolution.
- Problem: The reconstruction problem combines low-frame-rate blurry APS images, mixed event and frame noise, and low spatial resolution.APS frames have at least 5 ms latency, motion blur affects dynamic scenes, and spatial resolution trades off against latency.
- Contribution: The proposed eSL-Net uses an event-enhanced degeneration model and sparse learning to jointly recover high-quality images from event-camera observations.Events enhance the model’s representation of motion blur, while sparse learning provides a framework for recovery and noise resistance.
- Problem: Few prior studies simultaneously address denoising, deblurring, and super-resolution, motivating a unified framework.Existing super-resolution methods can be applied after reconstructing low-resolution intensity frames, but a comprehensive approach is preferred.
- Contribution: The eSL-Net can be extended without retraining to generate high-frame-rate, high-quality video, and the authors build a synthetic dataset linking events, LR blurry images, and HR sharp images.The dataset connects event sequences with low-resolution blurry observations and high-resolution sharp targets.
2 Related Works
Related work covers event-based intensity reconstruction from pure events or fused APS frames and events, alongside the comparatively limited development of event-based super-resolution. The paper positions its approach as a unified alternative to separately applying super-resolution after intensity reconstruction.
- Event-based Intensity Reconstruction: Early event-based intensity reconstruction estimated motion, optical flow, and intensity changes, while direct event integration avoided assumptions about scene structure or motion dynamics.The literature includes variational optimization and bio-inspired networks for jointly reconstructing frames and motion-related quantities.
- Event-based Intensity Reconstruction: APS frames and events provide complementary information, enabling fusion methods and event-based deblurring for more continuous or sharper intensity reconstruction.Prior work approximated events as intensity-frame time differentials and related blurry APS frames to events.
- Event-based Super-resolution: Event-based super-resolution remains underdeveloped because event cameras have high temporal frequency but relatively low spatial resolution.The paper identifies SRNet as the only recent event-based super-resolution work known to the authors at manuscript preparation.
3 Problem Statement
Event cameras provide asynchronous brightness-change measurements, while intensity frames provide exposure-averaged observations; reconstructing a sharp high-resolution image therefore requires a joint model for event information and degraded intensity data.
- Events encode intensity changes through pixel location, timestamp, and polarity, and can be represented as a continuous-time signal.The polarity indicates whether intensity increases or decreases, while the Dirac-function representation converts discrete events into a continuous signal.
- The observed intensity frame is modeled as an exposure-time average of sharp latent intensity images, with events relating the latent image at different times.This event-conditioned relationship supplies motion information for modeling the observation process.
- Sensor non-idealities, relative motion, measurement noise, and limited spatial resolution make the observation noisy, blurry, and low-resolution.The resulting model treats measurement noise explicitly and includes downsampling of the high-resolution latent image.
- The reconstruction target is a high-quality intensity image from an observed frame, triggered events, and a specified time within the exposure interval.The task combines information from the frame and events to recover the latent image at time tr.
- Recovering the sharp high-resolution image is an ill-posed multitask problem requiring denoising, deblurring, and super-resolution simultaneously.The paper addresses single-image reconstruction first and later extends the method to high-frame-rate video generation.
4 Event Enhanced High-Quality Image Recovery
eSL-Net integrates event information and sparse learning into an explainable network that jointly addresses denoising, deblurring, and super-resolution for event-camera images.
- 4.1 Event-Enhanced Sparse Learning: The method assumes LR and HR images share sparse codes on paired dictionaries, then estimates the sparse code through an LASSO formulation.This shared-code assumption links reconstruction of the observed LR image to recovery of the HR sharp image.
- 4.1 Event-Enhanced Sparse Learning: eSL-Net integrates events into a sparse-learning framework to solve noise, motion blur, and low spatial resolution in one unified model.The framework is motivated by the difficulty of directly applying conventional image-recovery methods to asynchronous event data.
- 4.2 Network: The network unfolds iterative soft thresholding into fixed phases, making each phase correspond to one sparse-code update and preserving interpretability.ReLU implements nonnegative soft thresholding, while convolution layers implement matrix multiplications and event integration.
- 4.2 Network: The architecture approximates the event double integral with convolutional weighted summation and maps the optimized sparse encoding through an HR dictionary implemented with convolution and shuffle layers.The final reconstruction uses convolution layers followed by a shuffle layer to represent the HR dictionary.
- 4.2 Network: Event sequences are converted into 2k polarity-separated grayscale frames and concatenated with the LR observation before producing an s-times-upscaled HR output.The input tensor contains the observed image plus positive and negative event frames for each temporal partition.
- 4.3 Network Training: Training uses ℓ1 loss to bring outputs closer to desired images while making optimization more stable.The loss is applied to the reconstructed image outputs.
5 High Frame-Rate Video Generation
After training for a latent frame at the observed-frame time, eSL-Net can reconstruct latent frames at other times by reordering event inputs rather than retraining.
- 5 High Frame-Rate Video Generation: The trained network reconstructs the latent image at the observation time from an LR frame and its corresponding event sequence.The LR image and events are paired over the interval preceding the target training time.
- 5 High Frame-Rate Video Generation: For a target time different from training time, events earlier than that target are reversed in polarity and temporal order before network input.This reorganization preserves the input pattern expected by the network trained at the original time.
- 5 High Frame-Rate Video Generation: The preprocessing enables reconstruction at any time in the interval without retraining and theoretically supports frame rates as high as the DVS event rate.The reordered events are merged into input frames before being passed to eSL-Net.
6 Dataset Preparation
The authors synthesize paired HR, LR, blurry, noisy, and event data because no large-scale dataset provides all required modalities, then use GoPro videos for training and testing.
- 6 Dataset Preparation: The dataset is synthesized to provide LR blurry noisy images, corresponding HR sharp clear images, and event sequences for eSL-Net training.The authors state that no large-scale dataset containing these paired modalities exists.
- 6 Dataset Preparation: HR clear images use 1280 × 720 GoPro frames, while 320 × 180 bicubic samples provide LR clear ground truth for experiments without super-resolution.The LR clear images serve as ground truth for the non-SR setting.
- 6 Dataset Preparation: LR blurry images are regenerated by first increasing LR sharp-video frame rate to 960 fps and then simulating exposure averaging to avoid unnatural blur trajectories.The procedure addresses limitations of directly averaging the original 120 fps GoPro frames.
- 6 Dataset Preparation: Events are simulated with ESIM from the 960 fps LR sharp frames, with 30% uniformly distributed noisy events added to the sequences.The added noise approximates the ratio of noise events to effective events in simple real scenes.
- 6 Dataset Preparation: The synthetic data split uses 240 videos for training and 30 videos for testing, with 25650 HR sharp frames drawn from 270 videos.LR blurry images and event sequences correspond to the generated LR data.
- 6 Dataset Preparation: Without super-resolution, Table 1 compares eSL-Net with EDI, CF, and MR on the synthetic testing dataset.The supplied table caption identifies the benchmark scope, while the accompanying results passage reports that eSL-Net outperforms the alternatives.
7 Experiments
Experiments compare eSL-Net with event-based reconstruction methods without and with super-resolution on synthetic and real data. eSL-Net achieves the strongest reported denoising, deblurring, edge-recovery, and noise-removal performance, while also supporting high-frame-rate reconstruction.
- 7.1 Intensity Reconstruction Experiments: eSL-Net outperforms EDI, CF, and MR by a large margin without super-resolution and consistently achieves the best visual denoising and deblurring results.Quantitative results are reported in Table 1, with qualitative comparisons on synthetic and real datasets.
- 7.1 Intensity Reconstruction Experiments: With super-resolution, all methods perform worse, but eSL-Net maintains good performance while jointly addressing noise, motion blur, and low resolution.The paper reports quantitative results in Table 2 and visual comparisons in Figures 6 and 7.
- 7.1 Intensity Reconstruction Experiments: Compared with methods combined with RCAN, eSL-Net performs much better in edge recovery and noise removal after super-resolution.The comparison is supported by qualitative results on synthetic and real datasets.
- 7.2 High Frame-Rate Video Experiments: The experiments generate frames at 21 times the original APS frame rate to balance temporal resolution with visible differences between consecutive frames.Although the theoretical frame rate can reach the DVS event rate, very high rates may produce only subtle differences between frames.
- 7.2 High Frame-Rate Video Experiments: Without retraining, eSL-Net generates continuous high-frame-rate frames, preserving more realistic and richer details in real-data reconstruction.Figure 8 compares eSL-Net with EDI+RCAN for high-frame-rate video reconstruction.
8 Conclusion
The paper presents eSL-Net for high-quality event-camera image reconstruction. It uses event-enhanced sparse learning to address motion blur, noise, and low spatial resolution, and extends to high-frame-rate video without retraining.
- 8 Conclusion: eSL-Net is an explainable sparse-learning network for high-quality image reconstruction from event cameras.The network is obtained by unfolding the iterative soft thresholding algorithm.
- 8 Conclusion: Event enhancement enables the degeneration model to address motion blur, noise, and low spatial resolution in event-camera reconstruction.
- 8 Conclusion: A simple transformation of the event stream extends eSL-Net to high-frame-rate video generation without additional training.
- 8 Conclusion: Experiments on synthetic and real-world data demonstrate the effectiveness and superiority of eSL-Net.