Source-linked AI summary
Learning Event-Based Motion Deblurring
Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, Yebin Liu
TL;DR
Severe motion blur makes sharp video recovery ill-posed because temporal information is lost, while event cameras provide dense but noisy temporal cues. The paper unfolds event-based deblurring into an end-to-end recurrent network with directional event filtering, and reports superior performance on synthetic and real-world datasets.
Problem
Severe motion blur is difficult to deblur because blurred images lose temporal order and visual information, while event signals are noisy and lossy.
Method
The paper unfolds a sequential event-based deblurring formulation into an end-to-end recurrent architecture that fuses global and local appearance and temporal cues with directional event filtering.
Results
The approach achieves superior performance over existing image- and event-based methods on synthetic and real-world datasets.
Takeaways & Limitations
The method provides sharper motion-deblurred video from a severely blurred image by combining image guidance with event-derived motion and boundary information.
Takeaways & Limitations
Inconsistent event thresholds and noise make the event approximation insufficient in practice, causing loss of contrast and details.
Abstract
from arXiv · showhide
Recovering sharp video sequence from a motion-blurred image is highly ill-posed due to the significant loss of motion information in the blurring process. For event-based cameras, however, fast motion can be captured as events at high time rate, raising new opportunities to exploring effective solutions. In this paper, we start from a sequential formulation of event-based motion deblurring, then show how its optimization can be unfolded with a novel end-to-end deep architecture. The proposed architecture is a convolutional recurrent neural network that integrates visual and temporal knowledge of both global and local scales in principled manner. To further improve the reconstruction, we propose a differentiable directional event filtering module to effectively extract rich boundary prior from the stream of events. We conduct extensive experiments on the synthetic GoPro dataset and a large newly introduced dataset captured by a DAVIS240C camera. The proposed approach achieves state-of-the-art reconstruction quality, and generalizes better to handling real-world motion blur.
1. Introduction
Severe real-world motion blur remains difficult because blurred images lose temporal information, while event cameras provide dense temporal cues but introduce noisy, lossy signals. The paper addresses this with an end-to-end recurrent architecture and directional event filtering, achieving strong results across synthetic and real-world data.
- 1. Introduction: Severe motion blur is difficult to reconstruct because conventional approaches lose temporal order and visual information.
- 1. Introduction: Event cameras record intensity changes with microsecond accuracy, providing temporally dense cues that reveal scene motion alongside blurred images.
- 1. Introduction: Event signals are lossy and noisy because threshold-based, discrete sampling varies with scene conditions, making texture and contrast restoration difficult.
- 1. Introduction: The paper unfolds a sequential event-deblurring formulation into a trainable recurrent architecture that combines global and local appearance and temporal cues.Each time step uses previous reconstructions and local events for coarse recovery, then network predictions supply fine details.
- 1. Introduction: Directional Event Filtering aggregates motion boundaries from events to produce a sharp deblurring prior.
- 1. Introduction: Experiments on the synthetic GoPro and newly compiled real-world event dataset show state-of-the-art results and better generalization to real-world motion blur.
2. Related Work
Prior work addresses motion deblurring with handcrafted or learned image priors, increasingly using deep architectures for complex motion. Event-camera research provides high-temporal-resolution signals and event-to-video methods, while this paper combines long-term and local cues with event boundary priors for motion deblurring.
- 2. Related Work: Blind motion deblurring has used handcrafted indicators, learned kernels, restoration functions, image priors, and scene geometry to address unknown blur.
- 2. Related Work: Deep deblurring methods address complex motion through larger receptive fields, multi-scale fusion, feature disentangling, and recurrent refinement.
- 2. Related Work: Event cameras detect scene-intensity changes at the microsecond level with low power consumption and support tracking, stereo, optical flow, and event-to-video tasks.
- 2. Related Work: Existing event-to-video methods translate events into plausible intensity images using local motion cues, while this work also models long-term appearance and motion cues and event boundary priors.
3. Learning Event-Based Motion Deblurring
The paper formulates event-assisted motion deblurring as sequential reconstruction of a sharp video from a blurred image and temporally sampled events. It unfolds this optimization into a recurrent learned-denoising process that combines local and long-term image-event cues.
- Given a motion-blurred image, the objective is to recover a sharp video sequence with T frames.
- Events record pixel-intensity changes at microsecond-level temporal resolution, with polarity indicating the direction of local intensity change.An event is associated with coordinates and a time point that may be fractional because of the camera’s high temporal resolution.
- The event-based image relation uses an indicator for event existence and becomes less approximate as Δt and τ approach zero.In practice, inconsistent thresholds and noise make the approximation insufficient, causing loss of contrast and details.
- The sequential formulation first estimates the final latent image, then reconstructs preceding frames backward using temporal relations between adjacent latent images.The backward reconstruction uses a Markov-chain simplification and Bayesian maximum-a-posteriori steps.
- A learned denoiser implicitly supplies the latent-image prior and is reused across deblurring steps, producing a recurrent architecture.The likelihood is centered on an initial estimate from previous reconstruction, while the initial final-frame estimate combines the blurred image and events.
- The resulting process conditions each latent image on both local and long-term cues from images and events.Algorithm 1 initializes the final frame, iterates backward through earlier frames, and returns the deblurred sequence.
4. Network Architecture
The architecture combines global event-motion encoding, appearance initialization, and recurrent local deblurring, with motion compensation and directional event filtering supplying temporal and boundary guidance.
- Global motion and initialization: The read network encodes global event motion, while the initialize network combines blurred-image appearance with motion to produce an initial latent image.Events are binned into time intervals and represented as stacked event frames before recurrent feature accumulation.
- Architecture overview: Figure 2 illustrates recovery of four sharp frames and omits the motion-compensation module because of space limitations.Detailed layer and parameter configurations are deferred to the supplementary material.
- Recurrent deblurring: The process network sequentially deblurs latent images using previous reconstructions, motion-compensated historical images, and DEF boundary guidance.Each step consumes image and event observations to refine the current latent image.
- Directional event filtering: Directional event filtering samples events at learned space-time positions to extract sharp boundary priors aligned with local scene motion.The module predicts temporal centers and filtering coefficients, while sampling follows local velocity rather than crossing the event density surface.
- Directional event filtering: Local velocities are obtained from motion compensation under a constant-velocity assumption and resampled with a Gaussian-kernel Nadaraya-Watson estimator.The differentiable estimator uses a local L × L window, with L = 20 found sufficient in practice.
- Training objective: The training objective combines image reconstruction, adversarial, photometric flow, and total-variation flow-smoothing losses.The adversarial term uses PatchGAN, while the flow-loss weights λa and λt are set to 0.01 and 0.05.
5. Experiments
Experiments evaluate the approach on GoPro and the newly captured Blur-DVS dataset, covering single-image deblurring, video reconstruction, component contributions, and real-world motion blur. The approach achieves the best reported results across the evaluated benchmarks and produces sharper reconstructions, including under fast motion and low-light conditions.
- Experimental Settings: The evaluation uses the GoPro dataset and the newly captured Blur-DVS dataset, with PSNR and SSIM as quantitative metrics.Blur-DVS is captured with a DAVIS240C camera and includes slow and fast subsets.
- GoPro Results: On GoPro, the approach ranks first for both single-image deblurring and video reconstruction.Visual comparisons show cleaner and sharper results than MPN and BHA in fast-moving scenes.
- Blur-DVS Results: On Blur-DVS, the approach achieves the best results for both single-image deblurring and video reconstruction, including against enhanced image-and-event variants.The comparisons attribute competing weaknesses to insufficient image cues, event noise, lossy sampling, or limited image guidance.
- Generalization: The approach achieves the best visual quality on the fast subset containing real-world motion blur.The authors report that this improvement is consistent on real data and associate it with explicit motion modeling and learned deblurring priors.
- Performance Analysis: Component analysis finds that appearance, events, motion compensation, and directional event filtering each contribute to reconstruction quality.Appearance alone deblurs poorly, events alone recover details but not contrast well, motion compensation improves smoothness, and DEF adds sharper details through learned boundary guidance.
- Performance Analysis: Learned boundary guidance improves estimation from 0.786 to 0.827 in SSIM, whereas simply enlarging the network produces no meaningful improvement.The guidance selects motion boundaries from different time centers and improves sharpness and missing-detail recovery.
6. Conclusion
The work reconstructs video from severe motion-blurred images using events, fusing appearance and motion cues across global and local scales. Experiments report superior performance over existing image- and event-based methods on synthetic and real-world datasets.
- The architecture fuses appearance and motion cues at both global and local granularity.
- A directional event filtering module extracts sharp event boundary guidance to improve reconstructed details.
- Extensive evaluations show superior performance over various existing image- and event-based methods on synthetic and real-world datasets.