Source-linked AI summary
Event-Based Fusion for Motion Deblurring with Cross-modal Attention
Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, Luc Van Gool
TL;DR
Motion deblurring remains difficult because image-only learning methods often generalize poorly to extreme or real-world blur. The paper introduces EFNet, a two-stage network with multi-level event-image cross-modal attention, a deblurring-specific event representation, and the REBlur dataset. EFNet surpasses prior image-based and event-based methods on GoPro and REBlur, including severe blur cases.
Problem
Image-based deblurring methods often fail on extreme or real-world blur, with performance heavily dependent on training-data quality and scale.
Method
EFNet is an end-to-end two-stage restoration network using SCER and multi-level EICA fusion to combine event and image information, evaluated with the REBlur dataset.
Results
EFNet surpasses the prior state of the art on GoPro and REBlur, with ablations showing EICA improves PSNR by 0.6 dB or more over simple fusion strategies.
Takeaways & Limitations
Event information can support motion deblurring when represented specifically for deblurring and fused with image features through cross-modal attention.
Takeaways & Limitations
Event-based restoration is constrained by time- and pixel-varying thresholds, sub-threshold intensity changes, and spurious events.
Abstract
from arXiv · showhide
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the eventbased image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset will be made publicly available.
1 Introduction
The paper targets the poor generalization of image-only deblurring under extreme or real-world blur by combining frame and event information in EFNet. It contributes cross-modal fusion, a deblurring-specific event representation, the REBlur dataset, and state-of-the-art results on GoPro and REBlur.
- Motivation: Learning-based deblurring methods often fail on extreme or real-world blur because performance depends heavily on training-data quality and scale.This motivates more general and reliable deblurring methods.
- Method: EFNet is an end-to-end two-stage restoration network that combines event and frame-camera information for image deblurring.Its first stage uses image and event branches, while the network progressively restores sharp images.
- Method: The EICA module applies channel-wise cross-modal attention at multiple network levels to adaptively fuse event and image features.It attends to event features relevant for deblurring rather than using simple fusion strategies.
- Method: SCER is a symmetric cumulative event voxel representation designed specifically for image deblurring and inspired by the physical model of event-based image formation.The representation is intended to connect blurry-image formation with event generation.
- Dataset: REBlur provides real-world tuples of blurry images, sharp images, and event streams in a challenging evaluation setting for deblurring methods.The dataset was captured with an event camera and includes severe motion-blur cases.
- Results: EFNet sets the state of the art for image deblurring on both the GoPro and REBlur datasets.The paper reports superiority over prior image-based and event-based methods, including challenging blurry conditions.
2 Related Work
Prior work spans image-only and event-based deblurring, but existing event representations and fusion strategies leave important limitations. The paper positions EFNet as a multi-level event-image fusion approach using a representation designed specifically for motion deblurring.
- Image deblurring: CNN-based image deblurring methods improve accuracy and robustness but rely only on images, which do not explicitly capture motion.This limits performance on real-world blurry images, especially under extreme conditions.
- Event-based deblurring: Event-based methods exploit the connection between events and motion, including EDI models, event-image pairs, and voxel-grid representations.These approaches differ in how they preserve or discard the temporal structure of event streams.
- EFNet: EFNet’s architecture uses two UNet-like backbones and an event extraction branch, with event and image features fused after residual convolution blocks.The second backbone further refines the deblurred image.
- Event representations: Counting events or accumulating polarities into 2D frames is compatible with conventional vision tasks but loses temporal information.Voxel grids preserve temporal information better by accumulating event polarity across discretized time bins.
- Event representations: No existing event representation was specifically designed for motion deblurring.This gap motivates the paper’s symmetric cumulative event representation.
3 Method
EFNet formulates event-based deblurring as a two-stage restoration process that combines blurry images with event representations. Its design addresses event noise and uncertainty through SCER, cross-modal attention, gated feature transfer, and a final denoising stage.
- General Architecture of EFNet: The Event Mask Gated Connection transfers encoder features from sharp regions and decoder features from blurry regions into the second stage.It exploits event-derived spatial information because regions containing events are typically more severely degraded.
- General Architecture of EFNet: EFNet maps a blurry image and exposure-window events to an initial sharp image, then applies a second mapping to produce a denoised final image.The first stage jointly processes image and event representations; the second stage corrects errors in event-affected regions.
- General Architecture of EFNet: Real-world event uncertainty arises from variable thresholds, sub-threshold intensity changes, and spurious events.These factors make direct restoration from the ideal event-based equation impractical.
- Two-stage backbone: Both EFNet stages use UNet-based encoder-decoder structures with two down-sampling and two up-sampling layers.Skip connections and a stage-connecting feature-transfer mechanism support progressive restoration.
- Symmetric Cumulative Event Representation: SCER partitions exposure time T into 2N intervals and concatenates 2N signed cumulative event tensors around the central latent sharp frame.With N = 3 in the implementation, SCER retains temporal information while encoding blur-region and sharp-region information.
- Event-Image Cross-modal Attention Fusion: The Event-Image Cross-modal Attention block uses image features as queries and event features as keys and values for multi-level fusion.The attention map has size c × c, and the design aims to model relations that simple multiplication or concatenation cannot fully capture.
4 REBlur Dataset
REBlur addresses the domain gap between synthetic and real event data with a challenging real-world dataset of aligned blurry-sharp images and event streams.
- Motivation: Real event data are needed because synthetic events retain a domain gap caused by background activity noise, dark current noise, and refractory-period false negatives.Most event-based methods train on blurred-image datasets with events generated by ESIM.
- Dataset construction: REBlur was captured in a high-precision optical laboratory with stable illumination using time-aligned event streams and grayscale images.The setup used an Insightness Seem 1 event camera and a DAVIS, producing 260×360 images.
- Motion coverage: The dataset covers 12 linear and nonlinear motion types across three moving patterns and camera motion.This design targets generalization across different objects and moving processes.
- Dataset composition: REBlur contains 36 sequences and 1469 blurry-sharp pairs with associated events, split into 486 training and 983 testing pairs.An additional four-sequence set contains extreme blur without ground truth.
5 Experiments
Experiments evaluate EFNet against image-only and event-based methods on GoPro and REBlur, then isolate the contributions of fusion, representation, and staged restoration.
- State-of-the-art comparisons: EFNet sets the state of the art in image deblurring on GoPro and REBlur, while restoring fine texture and structural patterns better in GoPro visual comparisons.Comparisons include image-only methods, BHA, and event-enhanced SRN+ and HINet+ using SCER.
- REBlur evaluation: EFNet shows superior performance under severe object- and camera-motion blur on REBlur, including the additional set without ground truth.The first two visual-comparison columns come from the test set; the remaining columns come from the additional set.
- Ablation study: 0.6 dB PSNR or more and 0.4% SSIM improvement show that EICA outperforms multiplication or addition for event-image fusion.Multi-level fusion also improves over early fusion by approximately 1 dB PSNR and 0.7% SSIM.
- Ablation study: 6.4 dB PSNR and 3.6% SSIM separate the full model from the image-only one-stage baseline on GoPro.All reported contributions are combined in this comparison.
- Ablation study: 1.53 dB PSNR and 0.6% SSIM result from adding SCER events through early fusion compared with the image-only counterpart.Adding EMGC between stages contributes a further 0.15 dB improvement.
- Event representation: 35.46 dB versus 35.12 dB shows that SCER uses event information more effectively than SBT for GoPro deblurring.Accumulating all events into a single Stack channel performs worse than not using events, making representation choice non-trivial.
6 Conclusion
The paper presents EFNet, SCER, and REBlur as an event-based fusion framework for single-image motion deblurring and reports superior performance on GoPro and REBlur.
- Conclusion: EFNet uses multi-level attention-based event-image fusion, while SCER is designed to utilize events for deblurring.REBlur supplies a challenging real-world evaluation setting with severe motion blur.