Source-linked AI summary
Generalized Video Deblurring for Dynamic Scenes
Tae Hyun Kim, Kyoung Mu Lee
TL;DR
The paper addresses video deblurring in dynamic scenes, where static-scene assumptions do not capture general blur from motion and depth variation. It estimates pixel-wise blur kernels from bidirectional optical flows and jointly optimizes flows and latent frames in one energy model. Experiments report significant improvements in deblurring challenging real videos and more accurate optical flows than a state-of-the-art blurry-image method.
Problem
Static-scene assumptions and global or segment-wise motion parameterizations do not adequately handle general, spatially varying blur in dynamic scenes.
Method
The method approximates pixel-wise blur kernels with bidirectional optical flows and jointly estimates those flows and latent frames using a single energy model.
Results
The method achieves significant improvements on challenging real videos and estimates more accurate optical flows than a state-of-the-art method for blurry images.
Takeaways & Limitations
Joint pixel-wise kernel estimation enables the method to address general blur in dynamic scenes without relying on global or segment-wise parameterization.
Takeaways & Limitations
The piece-wise linear kernel can introduce approximation error and ringing artifacts, while the data and temporal coherence models are invalid at occluded regions.
Abstract
from arXiv · showhide
Several state-of-the-art video deblurring methods are based on a strong assumption that the captured scenes are static. These methods fail to deblur blurry videos in dynamic scenes. We propose a video deblurring method to deal with general blurs inherent in dynamic scenes, contrary to other methods. To handle locally varying and general blurs caused by various sources, such as camera shake, moving objects, and depth variation in a scene, we approximate pixel-wise kernel with bidirectional optical flows. Therefore, we propose a single energy model that simultaneously estimates optical flows and latent frames to solve our deblurring problem. We also provide a framework and efficient solvers to optimize the energy model. By minimizing the proposed energy function, we achieve significant improvements in removing blurs and estimating accurate optical flows in blurry frames. Extensive experimental results demonstrate the superiority of the proposed method in real and challenging videos that state-of-the-art methods fail in either deblurring or optical flow estimation.
1. Introduction
Existing video deblurring methods often assume static scenes, limiting their ability to handle general blur from moving objects, camera motion, and depth variation. The proposed method jointly estimates bidirectional optical flow and latent frames using pixel-wise blur kernels, improving deblurring and flow estimation in challenging dynamic videos.
- Motivation: Static-scene assumptions limit prior methods when blur arises from moving objects and depth variation in dynamic scenes.Spatially varying blur kernels are difficult to parameterize in dynamic scenes with simple homographies.
- Limitations of prior work: Segmentation-based methods struggle with complex object or rotational motions, while exemplar-based methods can fail on moving objects, large blur, and mid-frequency textures.Exemplar-based interpolation can produce excessively smooth results and ringing artifacts when accurate segmentation and deconvolution are absent.
- Proposed approach: The method estimates bidirectional optical flows to construct pixel-wise varying kernels without global motion parameterization or segmentation.This formulation targets coexisting blur from camera shake, complex object motion, and depth variation.
- Proposed approach: Optical flows and latent frames are jointly estimated in a single energy model because each variable is needed to estimate the other accurately from blurry video.The framework includes efficient techniques for optimizing the joint model.
- Results: The system successfully restores a moving car and reports significant improvements on challenging real videos where other methods fail.The authors also report more accurate optical flows than a state-of-the-art method designed for blurry images.
2. Generalized Video Deblurring
The method models general dynamic-scene blur with pixel-wise kernels derived from bidirectional optical flows and jointly estimates latent frames and flows. Its energy model combines data fidelity, temporal coherence, and spatial regularization to handle locally varying motion.
- 2. Generalized Video Deblurring: Pixel-wise kernels derived from bidirectional optical flows model general blurs that global or segment-wise motion parameterizations cannot represent.The approach targets coexisting blur from differently moving objects and camera motion.
- 2.1. Data Model based on Approximated Blur: Assuming constant velocity between adjacent frames, the blur kernel is approximated as piece-wise linear from forward and backward optical flows.The model uses a short-exposure approximation and defines flows at each frame.
- 2.1. Data Model based on Approximated Blur: The data term uses the pixel-wise blur kernel to relate latent frames and blurry observations, with kernel weights constrained to be non-negative and sum to one.Optical flows determine the kernel, while L, u, and B denote latent frames, flows, and blurry frames.
- 2.2. Temporal Coherence with Optical Flow Constraint: The temporal model finds correspondences between latent sharp images, avoiding shift-invariance restrictions and enabling abrupt motion changes.Unlike conventional blurry-frame flow constraints, the proposed joint problem simultaneously estimates latent-frame brightness and optical flows.
- 2.3. Spatial Coherence: Spatial regularization promotes coherence in latent frames and flow fields while allowing discontinuities, using TV for images and edge-coupled TV for flows.The edge-map coupling preserves flow discontinuities at image edges, while the latent-image term uses computationally inexpensive TV regularization.
3. Optimization Framework
The method minimizes a single non-convex objective that jointly estimates latent frames and bidirectional optical flows. Alternating convex subproblems and practical iterative solvers provide approximated solutions.
- The final objective combines data, temporal, and spatial terms after fixing the known camera duty cycle.
- Unlike sequential multi-phase methods, the model jointly optimizes latent frames and optical flows through a single objective function.
- With optical flows fixed, the latent-frame subproblem becomes convex and is solved using a primal-dual scheme with conjugate-gradient updates.
- With latent frames fixed, the temporal term is convex but the data term remains non-convex, so flow estimation uses first-order Taylor linearization.
- The approximated convex flow problem is optimized using a primal-dual update process with flow-specific dual variables and weighting matrices.
4. Implementation Details
Implementation uses a coarse-to-fine alternating framework that restores latent frames, estimates flows, detects occlusions, and refines results. Known or estimated exposure duty cycles initialize the blur model, while post-processing reduces artifacts.
- The coarse-to-fine implementation uses empirically chosen parameters and a 17-level pyramid for 1280x720 video.The pyramid scale factor is 0.9, and bicubic interpolation propagates flows and latent frames.
- The algorithm estimates only forward and backward adjacent-frame flows, composing them to approximate longer-range flows.For example, ui→i+2 = ui→i+1 + ui+1→i+2.
- Algorithm 1 alternates sharp-video restoration with fixed flows and flow estimation with fixed latent frames at each pyramid level.Occlusion detection and post-processing occur before variables propagate to the next level.
- The camera duty cycle is assumed known or estimated from conventional datasets lacking exposure information.For Kinect RGB videos, the duty cycle can be obtained from a public SDK.
- Occlusion-aware spatio-temporal filtering is used after optimization to reduce ringing artifacts caused by kernel errors and occlusions.The filtering includes n = 0 for spatial filtering and uses a fixed similarity parameter σw = 25/255.
5. Experimental Results
Experiments show improved deblurring and optical-flow estimation across dynamic scenes, including moving objects, layered boundaries, motion boundaries, and highly non-uniform facial blur. Temporal coherence also reduces ringing artifacts.
- Compared with the exemplar-based method, the proposed method better restores moving objects and backgrounds under large and complex blurs.The competing method fails with distinct object motion, inaccurate homographies, and excessively smooth mid-frequency textures.
- Compared with the segmentation-based approach, the method restores bi-layer boundaries better because it does not depend on accurate segmentation.Inaccurate segmentation near boundaries causes serious artifacts in the layered model.
- On synthetic blurry images, the proposed model estimates optical flow more accurately than at motion boundaries of moving objects.Its model permits abrupt motion changes, whereas 's assumption does not hold at motion boundaries.
- The temporal coherence term significantly reduces ringing artifacts near edges in the restored frames.
- Real-video experiments include successful restoration of a face with highly non-uniform blur caused by rotational movement.
6. Conclusions
The method removes general blurs in dynamic scenes using pixel-wise kernel estimation and achieves significant improvements. The figures compare blurry inputs, competing results, and the proposed deblurring results.
- The method removes general blurs in dynamic scenes through pixel-wise kernel estimation.
- The proposed framework and efficient solvers achieve significant improvements in removing general blurs in dynamic scenes.
- Figure 6 compares blurry dynamic-scene frames, deblurring results from [7], and the proposed results.
- Figure 7 compares a blurry frame, the result from [7], and the proposed result.
- Figure 8 compares a blurry frame, the segmentation-based result from [28], and the proposed result.
- Figure 9 compares optical-flow estimation results using ground truth flow, [24], and the proposed method.
- Figure 10 compares results without and with the temporal term E_temporal on a real blurry video frame.
- The experiments include real blurry frames and deblurring results, including Kinect-captured RGB datasets.