Source-linked AI summary
Motion Deblurring with Real Events
Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, Jianzhuang Liu
TL;DR
Synthetic-event mismatch and the difficulty of collecting labeled real-event data limit event-based motion deblurring on real scenes. The paper trains a deblurring network with real events and blurred images using optical-flow-based blur and photometric consistency, with piece-wise linear motion to model nonlinearities. The authors report state-of-the-art results on a real blurry event dataset and conclude that the approach bridges simulated and real-world blur.
Problem
Synthetic-to-real event inconsistency limits real-camera inference, while labeled real-event data are difficult to collect and align.
Source: §1 · pp. 1–2Method
Results
The method achieves state-of-the-art results on the real blurry event dataset.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that the framework bridges simulated and real-world motion blur and produces sharp-image sequences and motion flows from a blurry image and events.
Source: §6 · p. 8The linear-motion formulation assumes constant per-pixel flow throughout the exposure, an assumption the proposed piece-wise model addresses.
Source: §1 · p. 2, §4.1 · p. 4
Abstract
from arXiv · showhide
In this paper, we propose an end-to-end learning framework for event-based motion deblurring in a self-supervised manner, where real-world events are exploited to alleviate the performance degradation caused by data inconsistency. To achieve this end, optical flows are predicted from events, with which the blurry consistency and photometric consistency are exploited to enable self-supervision on the deblurring network with real-world data. Furthermore, a piece-wise linear motion model is proposed to take into account motion non-linearities and thus leads to an accurate model for the physical formation of motion blurs in the real-world scenario. Extensive evaluation on both synthetic and real motion blur datasets demonstrates that the proposed algorithm bridges the gap between simulated and real-world motion blurs and shows remarkable performance for event-based motion deblurring in real-world scenarios.
1. Introduction
The introduction motivates learning with real events and real blurred images because synthetic-to-real inconsistency limits event-based deblurring, while assembling labeled real-event data is difficult. The proposed framework combines self-supervision and a piece-wise linear motion model; experiments report state-of-the-art results on a real blurry event dataset.
1. Introduction
- Synthetic-to-real inconsistency degrades inference because simulated events differ from real events in noise and timing variation.Limited read-out bandwidth contributes to timing variation, including differences between fast-motion events and events collected at a slow-motion speed.Source: §1 · pp. 1–2
- Building labeled real-event datasets is difficult because it requires slow motion and good lighting, followed by blurry-image synthesis and temporal alignment.Even then, limited read-out bandwidth leaves inconsistency between events associated with synthesized and real-world motion blur.Source: §1 · p. 2
- The framework trains an event-based deblurring network with real events and real blurred images, using an event-based optical-flow network and self-supervised constraints.The piece-wise linear motion model accounts for motion nonlinearities and supports a more precise blur model.Source: §1 · p. 2
- The method achieves state-of-the-art results on the real blurry event dataset.In Figure 1’s qualitative comparison, RED-Net produces fewer halo artifacts than eSL-Net and LEDVDI.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
Prior motion-deblurring work ranges from kernel and flow-based models to learned networks, including event-based approaches. The review highlights limits from motion assumptions, noisy real events, and training-data mismatch, motivating learning from real-world events and blur images.
2. Related Work
- Kernel-based methods model spatially uniform blur, while flow-based approaches use pixel- or patch-wise motion for complex behavior.Assumptions such as linear motion can produce artifacts when they are not satisfied.Source: §2 · p. 2
- Learned deblurring methods recover sharp-frame sequences from blurred images, but some approaches rely on multiple blurred frames or linear motion assumptions.The review also notes temporal-order ambiguity in some sequence-recovery approaches.Source: §2 · p. 2
- Event-based methods add motion information to deblurring, but real-event noise and simulated-event mismatch limit generalization to real cameras.Blur-DVS uses real events collected at slow speed, yet dataset construction requires blur-free ground truth, suitable lighting, and still leaves fast-versus-slow event inconsistency.Source: §2 · p. 3
- The authors motivate learning from real-world events and motion-blurred images as more adaptive to real-world scenarios and capturable without sophisticated procedures.This approach is presented as an alternative to the burdensome construction of labeled real-event data.Source: §2 · p. 3
3. Problem Statement
The problem is to recover a sequence of sharp images from one motion-blurred image and its corresponding events. The formulation distinguishes synthetic and real data, whose event statistics and motion conditions create challenges for learning in real-world scenes.
3. Problem Statement
- Event-based motion deblurring reconstructs sharp latent images from a single blurred image and events captured during its exposure.Each event records a pixel location, timestamp, and polarity.Source: §3 · p. 3
- Synthetic events differ from real events through spatial and temporal noise that current simulators do not fully model.Real-event noise includes camera imperfections, background activity, false negatives, and timing variance from limited read-out bandwidth.Source: §3 · p. 3
- The real-world dataset contains blurred images and events without sharp ground truth, while slow-speed labeled data cannot represent temporal noise accompanying real motion blur.Real-world motion may also be nonlinear, whereas common deblurring models assume constant flow during exposure.Source: §3 · p. 3
- The stated obstacles are training on real events with real motion blur and accounting for motion nonlinearity in complex dynamic scenes.Source: §3 · p. 3
4. Method
The framework uses a deblurring network and an event-based optical-flow network to produce sharp frames and motion estimates. It trains with blur and photometric consistency, using piece-wise linear motion to model nonlinear blur and combining self-supervision with synthetic ground-truth supervision.
4. Method
- The framework jointly trains Deblur-Net and OF-Net, using real-world blur and events for semi-supervision and synthetic data for supervised learning.Deblur-Net outputs sharp frames, while OF-Net estimates motion from events; the framework adds blur- and photometric-consistency constraints.Source: §4 · pp. 3–4, Figure 2 · p. 4
4.1. Blur-consistency Constraint
- Piece-wise linear motion divides the exposure-time event stream into equal intervals and assumes linear motion within each interval.OF-Net predicts interval-specific motion flows, which define warped latent frames used to re-render the blurry input; matching that re-render to the input supplies blur-consistency self-supervision.Source: §4.1 · p. 4
4.2. Photometric-consistency Constraint
- Photometric consistency compares a sharp frame with an adjacent frame propagated using the optical flow predicted from events.This adds an inter-frame constraint alongside the sequence-level blur-consistency constraint.Source: §4.2 · p. 5
4.3. Optimization
- The optimization applies blur- and photometric-consistency losses to both synthetic and real datasets, and adds reconstruction loss for synthetic data with sharp ground truth.The loss terms are combined with balancing parameters.Source: §4.3 · p. 5
5. Experiments
The experiments compare RED-Net variants across synthetic and real-event datasets, evaluate optical-flow and deblurring performance, and ablate events and motion models. Results favor real-world training and piece-wise linear motion modeling, including on real blurry scenes.
5.1. Experimental Settings
- The study trains RED-Net on GoPro, GoPro+HQF, and GoPro+RBE, with the latter two using self-supervision and real-world events.HQF pairs real events with synthesized blurry images for training; RBE contains real events and real motion blurs. The datasets use DAVIS240C and DAVIS346 cameras, respectively.Source: §5.1 · pp. 5–6
5.2. Results of Optical Flow
- On the fast-moving magic-cube example, RED-RBE produces the most consistent optical flow and the sharpest qualitative deblurred result.RED-GoPro-PLM gives clearer edges and a more stable background than RED-GoPro-LM; with linear motion assumptions violated, RED-GoPro-LM performs worse than EV-FlowNet.Source: §5.2 · p. 6, Figure 3 · p. 6
5.3. Comparisons with State-of-the-art Methods
- 3.5 dB on single-frame prediction and 2.5 dB on sequence prediction are the average gains of the proposed RED-Nets over state-of-the-art methods on HQF with real-world events.Table 1 compares RED-Nets trained on different datasets with prior methods; LEDVDI outputs six frames for sequence prediction while the others output seven.Source: §5.3 · p. 6, Table 1 · p. 7
- RED-RBE gives the best qualitative deblurring on real blurry scenes, while competing methods show oversmoothing or halo artifacts.The comparison covers labfloor and window scenes; RED-RBE is trained with real-world motion blurs, which can alleviate temporal event noise such as timing chaos from limited read-out bandwidth.Source: §5.3 · p. 7, Figure 4 · p. 8
5.4. Ablation Study
- The ablations compare removing synthesized events, using linear versus piece-wise linear blur models, and training with real-world events.The study reports that without events deblurring performance has a large gap, and that inaccurate motion modeling violates blurry consistency and degrades performance.Source: §5.4 · p. 7, Table 2 · p. 7
- Real-world events yield PSNR gains of 1.6 dB with PLM and 1.3 dB with LM on the real-world dataset compared with synthetic-event training.Across the ablations, PLM brings gains up to 0.5 dB with real-world events, versus 0.16 dB without them.Source: Table 2 · p. 7, §5.4 · p. 8
6. Conclusion
The conclusion presents a self-supervised framework using real events and motion blurs, with motion non-linearities modeled to improve physical blur consistency. Experiments validate the consistency losses on real-world data and report state-of-the-art performance with real-world events.
6. Conclusion
- Experiments report state-of-the-art performance with real-world events across the evaluated datasets.The framework is also evaluated on different datasets, with the PLM-based blur-consistency and photometric-consistency validated on the real-world dataset.Source: §6 · p. 8
- The piece-wise linear motion model accounts for motion non-linearities to improve the accuracy of physical blur consistency.Source: §6 · p. 8
- The self-supervised framework uses real-world events and motion blurs to address performance degradation from data inconsistency.Given a blurry image and corresponding events, RED-Net produces sharp intensity images and motion flows between them.Source: §6 · p. 8