Source-linked AI summary
Human-Aware Motion Deblurring
Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, Ling Shao
TL;DR
Motion deblurring must handle different degradations affecting foreground humans and backgrounds, while existing approaches face engineering and pipeline limitations. The paper proposes a human-aware, multi-branch network with supervised attention and introduces HIDE; experiments show favorable performance against state-of-the-art methods.
Problem
Dynamic scenes can impose different motion blurs on foreground humans and backgrounds, motivating human-aware deblurring for restoring semantic details.
Method
A fully differentiable network uses supervised soft human-mask attention, separate foreground/background decoder branches, and a primary decoder that fuses their multi-scale information.
Results
The proposed approach performs favorably against state-of-the-art deblurring algorithms on public benchmarks and the HIDE dataset.
Takeaways & Limitations
HIDE provides 8,422 blurry–sharp image pairs with densely annotated foreground human bounding boxes across varied scenes, motions, object sizes, and backgrounds.
Takeaways & Limitations
Related approaches are limited by significant feature engineering, high computational cost, complicated pipelines, and dependence on segmentation preprocessing.
Abstract
from arXiv · showhide
This paper proposes a human-aware deblurring model that disentangles the motion blur between foreground (FG) humans and background (BG). The proposed model is based on a triple-branch encoder-decoder architecture. The first two branches are learned for sharpening FG humans and BG details, respectively; while the third one produces global, harmonious results by comprehensively fusing multi-scale deblurring information from the two domains. The proposed model is further endowed with a supervised, human-aware attention mechanism in an end-to-end fashion. It learns a soft mask that encodes FG human information and explicitly drives the FG/BG decoder-branches to focus on their specific domains. To further benefit the research towards Human-aware Image Deblurring, we introduce a large-scale dataset, named HIDE, which consists of 8,422 blurry and sharp image pairs with 65,784 densely annotated FG human bounding boxes. HIDE is specifically built to span a broad range of scenes, human object sizes, motion patterns, and background complexities. Extensive experiments on public benchmarks and our dataset demonstrate that our model performs favorably against the state-of-the-art motion deblurring methods, especially in capturing semantic details.
1. Introduction
The paper targets dynamic-scene deblurring where foreground humans and backgrounds undergo heterogeneous motion blur. It proposes an end-to-end human-aware network and introduces HIDE to support this task.
- Relative camera-object motion causes foreground humans and backgrounds to undergo different image degradations.
- Existing non-uniform models deblur foreground and background simultaneously, which can cause inferior performance and artifacts by neglecting multiple motion patterns.
- Prior heuristic methods estimate object motion blur kernels but rely on pre-computed foreground masks rather than emphasizing human-focused deblurring.
- The proposed network learns supervised soft human masks and uses three decoder branches to model foreground humans, backgrounds, and fused global results end-to-end.The attention mechanism is differentiable and explicitly guides domain-specific processing.
- HIDE contains 8,422 blurry-sharp image pairs with densely annotated foreground human bounding boxes across varied scenes, motions, human sizes, and backgrounds.
- The contributions include explicit foreground/background blur disentanglement, supervised differentiable attention, multi-head decoding, and the HIDE dataset.
2. Related Work
Prior deblurring datasets and models generally do not target human-aware restoration with foreground annotations. The paper addresses this gap with HIDE and supervised attention integrated into a trainable multi-head architecture.
- Earlier datasets synthesized blur with predefined kernels or motion maps, while later approaches modeled more realistic non-uniform blur.
- Existing datasets have advanced image deblurring but commonly lack ground-truth foreground annotations for human-aware deblurring.
- CNN-based deblurring models generally overlook distinct foreground human motion patterns and backgrounds, leaving human-aware deblurring insufficiently addressed.
- Heuristic foreground-aware methods are limited by feature engineering, computational cost, complicated pipelines, and dependence on segmentation preprocessing.
- The proposed approach integrates soft attention-based foreground and background masks with a fully convolutional encoder-decoder and multi-head decoder.
- Human-aware attention learns a supervised soft human mask, guides foreground/background decoders toward their domains, and preserves end-to-end differentiability.
3. Proposed HIDE Dataset
HIDE is designed for realistic human-aware motion deblurring, covering both wide and close scenes with varied human motions and scene configurations. Its images are generated from high-frame-rate videos and densely annotated.
- Existing dynamic-blur datasets often emphasize camera disturbance or wide-range scenes, while HIDE targets foreground human motion, including close-up shots.
- HIDE examples include human bounding boxes, masks, and attributes.
- Videos were captured at 240fps with a GoPro Hero camera, then frames were integrated to produce plausible motion blurs, with close-up human scenes prioritized for annotation.
- After removing streak-artifact and structurally unsuitable candidates, the dataset contained 8,422 sharp and blurry image pairs.The images were selected from 31 high-fps videos and include 65,784 human bounding boxes.
- The dataset statistics distinguish scattered scenes with 4,202 images from crowded scenes containing 4,220 images.
- HIDE includes long-shot images in HIDE I and regular-pedestrian close-ups in HIDE II, which emphasize different aspects of multi-motion blur.
4. Proposed Algorithm
The proposed model extends an encoder-decoder deblurring network with supervised human-aware attention, domain-specific FG/BG branches, and a primary branch that fuses their information across scales.
- Base Architecture: The vanilla architecture encodes a blurry image into feature representation H and decodes it into a predicted sharp image.The encoder uses convolutional layers, while the decoder uses transposed convolutional layers with nonlinearities.
- Human-Aware Attention: The attention network maps the blurry image to a spatial importance map, then uses sigmoid activation and human annotations to learn a supervised soft FG mask.The attention map is trained with a pixel-wise ℓ2 loss against human annotations.
- Human-Aware Attention: Attention-enhanced features encode FG information with A and BG information with (1−A), while the original feature H preserves overall image information.These feature representations provide separate inputs for human-aware, background-aware, and global deblurring.
- Multi-Head Decoder: The multi-head decoder contains primary, FG, and BG branches that deblur corresponding regions while preserving domain-specific features.The FG and BG branches use separate learnable architectures, and masked losses prevent errors from the opposite region from propagating into each branch.
- Multi-Head Decoder: The primary decoder fuses intermediate FG and BG decoder features rather than merely combining their output images, supporting global deblurring with domain-specific information.The model recursively injects intermediate features from the FG and BG branches into the primary branch across decoder blocks.
- Multi-Scale Structure: The single-scale model is aggregated over three scales using a coarse-to-fine structure with shared weights between scales.The implementation uses HIDE and GoPro training images, with GoPro used to train the BG decoder because it contains few pedestrians.
5. Experiments
The experiments evaluate the model through ablations and comparisons on GoPro and HIDE, showing benefits from attention, multi-head decoding, and multi-scale processing. Qualitative results indicate improved reconstruction of human details, full-frame content, and images with multiple blurs.
- 5.1. Ablation Study: The ablation study evaluates the human-aware attention module, multi-head decoder, and multi-scale framework using PSNR, SSIM, and visual comparisons.The experiments compare a retrained model without attention, separate FG/BG decoder outputs, blending results, and a single-scale baseline.
- 5.1. Ablation Study: Without attention, the baseline performs worse and reconstructs less accurate profiles, facial features, and shapes than the full model.Figure 5 compares deblurred results without and with attention, while Table 3 reports the quantitative ablation.
- 5.1. Ablation Study: The FG and BG decoder branches handle blur in their respective regions, while blending their complementary features restores content across the full image.Figure 6 presents the FG branch, BG branch, and final blending result.
- 5.1. Ablation Study: The multi-scale model converges better and extracts faithfully reconstructed features that guide improved restoration through a feed-forward mechanism.A single-scale baseline is constructed for comparison in Table 3.
- 5.2. Performance on the GoPro Dataset: On GoPro, the method is evaluated on 1,111 blurred test images using PSNR and SSIM, with visual comparisons shown in Figure 7.The reported examples include moving humans with independent motion and scaled scenes.
- 5.3. Performance on the Proposed HIDE Dataset: On HIDE, the multi-branch model better restores images containing multiple blurs, while HIDE II provides the clearest setting for moving-human deblurring.The paper states that the algorithm clearly outperforms previous state-of-the-art methods on HIDE II.
6. Conclusion
The paper studies human-aware motion deblurring by combining a dedicated dataset with a multi-branch network and supervised attention. The approach selectively reinforces foreground humans and background information, blends domain-specific features, and performs favorably against state-of-the-art algorithms.
- 6. Conclusion: The paper introduces HIDE and a human-aware convolutional network for deblurring multi-motion blur caused by camera motion and human movement.The dataset supports the study and is expected to facilitate future research on related topics.
- 6. Conclusion: The model combines multi-branch deblurring with supervised attention to selectively reinforce foreground humans and background regions.Different-domain information is blended to restore blurred images with more semantic details.
- 6. Conclusion: Experimental results show that the approach performs favorably compared with state-of-the-art deblurring algorithms.