Source-linked AI summary
Deep Semantic Face Deblurring
Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, Ming-Hsuan Yang
TL;DR
Face deblurring must recover sharp facial structure despite limited generic priors and diverse unknown blur kernels. The paper uses semantic cues in a multi-scale CNN, local structural losses, and incremental training, achieving favorable restoration, recognition, and speed results while remaining limited on misaligned faces and extreme motion blur.
Problem
Face images have fewer textures and edges for blur-kernel estimation, similar exemplars may be unavailable, and MAP-based optimization is computationally expensive.
Method
A multi-scale CNN uses semantic labels as global priors, local structural losses for facial components, perceptual and adversarial losses, and incremental training for unknown blur kernels.
Results
The method performs favorably against state-of-the-art approaches in restoration quality, face recognition, and execution speed.
Takeaways & Limitations
Combining global and local face semantics helps restore facial shape and detail while the end-to-end design avoids blur-kernel estimation and post-processing.
Takeaways & Limitations
The method may fail on poorly aligned inputs, including side faces and extremely large motion blur.
Abstract
from arXiv · showhide
In this paper, we present an effective and efficient face deblurring algorithm by exploiting semantic cues via deep convolutional neural networks (CNNs). As face images are highly structured and share several key semantic components (e.g., eyes and mouths), the semantic information of a face provides a strong prior for restoration. As such, we propose to incorporate global semantic priors as input and impose local structure losses to regularize the output within a multi-scale deep CNN. We train the network with perceptual and adversarial losses to generate photo-realistic results and develop an incremental training strategy to handle random blur kernels in the wild. Quantitative and qualitative evaluations demonstrate that the proposed face deblurring algorithm restores sharp images with more facial details and performs favorably against state-of-the-art methods in terms of restoration quality, face recognition and execution speed.
1. Introduction
Face deblurring is difficult because generic priors and exemplar-based methods poorly match faces, while MAP optimization is computationally expensive. The proposed end-to-end CNN uses semantic priors, local facial-structure constraints, incremental training, and perceptual/adversarial losses.
- Motivation: Face images have fewer textures and edges for estimating blur kernels, so natural-image priors may not adequately constrain face deblurring.Existing exemplar-based methods also depend on finding a sufficiently similar reference face.
- Motivation: MAP-based methods require iterative optimization of latent images and blur kernels, producing long execution times that limit resource-sensitive applications.
- Approach: The proposed method feeds blurred images and semantic labels into a coarse-to-fine deep CNN and applies local structure losses to eyes, noses, and mouths.Semantic labels provide global priors, while local constraints encourage fine facial details.
- Approach: Incremental training starts with small blur kernels and incorporates larger kernels sequentially to facilitate convergence and improve performance across kernel sizes.Perceptual and adversarial losses are additionally used to generate photo-realistic results.
- Contributions: The method is reported to perform favorably against state-of-the-art approaches in restoration quality, face recognition, and execution speed.
2. Related Work
Related work spans generic and domain-specific priors, exemplar-based face deblurring, and CNN restoration methods. This paper addresses complex motion blur in faces with global and local semantic cues and perceptual/adversarial losses.
- Generic methods: Blind deblurring methods estimate clear images without an available blur kernel, commonly using natural-image priors within a MAP framework.
- Generic methods: Generic edge-restoration approaches are less effective for faces because face images have different statistical properties and fewer textures and edges.
- Domain-specific methods: Domain-specific methods tailor priors or strategies to categories such as text and low-light images, while exemplar-based face deblurring depends on annotated contours and expensive optimization.
- CNN-based methods: CNNs have been applied to restoration, deconvolution, blur-kernel estimation, and dynamic-scene deblurring, offering computational efficiency.
- CNN-based methods: The proposed work targets complex motion blur in face images using global and local semantic cues together with perceptual and adversarial losses.
3. Semantic Face Deblurring
The method combines semantic face parsing, local structural constraints, and multi-scale CNN restoration with perceptual and adversarial objectives. Its architecture and training losses are designed to preserve facial details and produce photo-realistic outputs.
- Global semantic priors: Semantic label probability maps from a face parsing network are concatenated with the blurred face image as global-prior inputs.At the first scale, the input has 64 × 64 resolution and 14 channels: 3 RGB channels plus 11 semantic-probability channels.
- Local structural constraints: Local structural losses target eyebrows, eyes, noses, lips, and teeth because whole-face content loss may miss small facial components.The losses use structural masks and encourage restoration of details in these key components.
- Photo-realistic objectives: Perceptual loss and adversarial loss are added because pixel-wise L1 or L2 objectives typically produce overly smooth results.The perceptual loss is computed on Pool2 and Pool5 of a pre-trained VGG-Face network, while the discriminator distinguishes real images from generator outputs.
- Training objective: The overall objective combines content, local structural, perceptual, and adversarial losses, with content and structural losses applied at all scales.Perceptual and adversarial losses are used only at the finest, second scale; the empirical weights are λs = 50, λp = 1e−5, and λadv = 5e−5.
- Implementation details: The semantic parsing network is trained first until convergence, followed by deblurring-network training using the specified implementation and optimization setup.The parsing network converges within 60,000 iterations, while deblurring training runs for 17 million iterations on an NVIDIA Titan X GPU.
4. Experimental Results
Experiments evaluate semantic parsing, component losses, incremental training, and comparisons with state-of-the-art methods. The proposed approach improves facial detail restoration, robustness across blur kernels, recognition, efficiency, and real-world visual quality, while requiring aligned faces.
- Semantic face parsing: Fine-tuning the semantic parsing network on blurred images improves robustness, especially for facial components degraded by motion blur.The pre-trained model performs poorly on small and thin components such as eyebrows, lips, and teeth, whereas the fine-tuned model parses components well.
- Effect of semantic parsing: Adding global semantic priors better reconstructs face outlines, while local structural losses restore finer details and textures in components such as teeth and eyes.The content-loss-only model produces overly smooth results and poorly recovers face and lip shapes.
- Incremental training: Incremental training outperforms direct training across blur-kernel sizes and restores images well when training must accommodate diverse random kernels.The strategy incorporates more blur kernels sequentially rather than training on all blurred images simultaneously.
- Comparisons with state-of-the-arts: The proposed method performs favorably against seven state-of-the-art algorithms on restoration quality, producing more facial detail and fewer visual artifacts across datasets and blur-kernel sizes.MAP-based methods produce ringing artifacts, while a CNN baseline without semantic information produces overly smooth results.
- Comparisons with state-of-the-arts: The method is more efficient than state-of-the-art deblurring algorithms, preserves face identity well, achieves the highest detection and recognition performance, and produces visually pleasing real-world results.The execution-time comparison uses ten 128 × 128 images; real-world comparisons use the blurred dataset of Lai et al.
- Limitations: The method may fail on poorly aligned inputs, including side faces or extremely large motion blur, motivating future work on larger and non-uniform blur kernels.The authors also identify reducing the face-alignment requirement as future work.
5. Conclusions
The paper presents a CNN-based face deblurring method that uses semantic information as global priors and local structural constraints, with incremental training for unknown blur kernels. Experiments show favorable performance across restoration quality, execution time, and face recognition.
- The method uses face semantic information as global priors and local structural constraints to restore facial shape and detail.
- Perceptual and adversarial losses are used to produce photo-realistic deblurred results.
- An incremental training strategy addresses random and unknown blur kernels in real-world images.
- Experiments demonstrate favorable performance against state-of-the-art algorithms in image deblurring, execution time, and face recognition.