Source-linked AI summary
VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, Ming-Ming Cheng
TL;DR
Blind face restoration still struggles to produce fine-grained details faithful to degraded inputs. VQFR combines an all-region VQ facial-detail dictionary with compression-patch selection and a parallel decoder, and experiments show it surpasses previous works on synthetic and real-world datasets while preserving fidelity.
Problem
Blind face restoration must recover realistic fine-grained details while preserving fidelity under unknown, complicated degradations.
Method
VQFR uses an end-to-end VQ codebook of high-quality facial details and a parallel decoder that fuses input and texture features through texture warping.
Results
VQFR produces high-quality facial details while preserving fidelity and surpasses previous works on synthetic and real-world datasets.
Takeaways & Limitations
A facial-detail dictionary combined with parallel decoding can improve restoration quality while retaining fidelity to degraded inputs.
Takeaways & Limitations
Extreme poses and extremely low-information faces remain difficult for VQFR, with limitations linked to frontal-face-biased training data and ambiguous code quantization.
Abstract
from arXiv · showhide
Although generative facial prior and geometric prior have recently demonstrated high-quality results for blind face restoration, producing fine-grained facial details faithful to inputs remains a challenging problem. Motivated by the classical dictionary-based methods and the recent vector quantization (VQ) technique, we propose a VQ-based face restoration method - VQFR. VQFR takes advantage of high-quality low-level feature banks extracted from high-quality faces and can thus help recover realistic facial details. However, the simple application of the VQ codebook cannot achieve good results with faithful details and identity preservation. Therefore, we further introduce two special network designs. 1). We first investigate the compression patch size in the VQ codebook and find that the VQ codebook designed with a proper compression patch size is crucial to balance the quality and fidelity. 2). To further fuse low-level features from inputs while not "contaminating" the realistic details generated from the VQ codebook, we proposed a parallel decoder consisting of a texture decoder and a main decoder. Those two decoders then interact with a texture warping module with deformable convolution. Equipped with the VQ codebook as a facial detail dictionary and the parallel decoder design, the proposed VQFR can largely enhance the restored quality of facial details while keeping the fidelity to previous methods.
1 Introduction
Blind face restoration must recover realistic fine details while preserving fidelity under unknown and severe degradations. VQFR addresses this tension with an all-region VQ facial dictionary, compression-patch analysis, and a parallel decoder.
- Geometric priors guide accurate face shapes but provide insufficient realistic details and become inaccurate under severe degradations.
- Generative-prior methods improve overall restoration but still miss fine-grained details such as hair and delicate facial components.
- VQFR uses an end-to-end VQ codebook covering all facial areas, providing a comprehensive low-level feature bank without geometric or GAN priors.
- Compression patch size f balances visual quality and fidelity; the authors suggest f = 32 for 512×512 inputs, while noting expression and identity can still change.
- A parallel decoder gradually fuses input features with VQ texture features to preserve fidelity without sacrificing high-quality facial details.
- Experiments with quantitative and qualitative comparisons show VQFR surpasses previous works in restoration quality while maintaining high fidelity.
2 Related Work
Face restoration methods use geometric, generative, and reference priors, each addressing different aspects of degraded inputs. VQFR is positioned as an end-to-end facial dictionary approach that broadens detail coverage beyond selected components.
- Geometric priors include landmarks, parsing maps, and component heatmaps, but severe degradations make them inaccurate and they lack sufficient detail information.
- Generative-prior methods use GAN latent spaces and pretrained generators to provide facial details and achieve decent restoration performance.
- Reference-prior methods commonly rely on same-identity reference images, while DFDNet instead uses an offline facial-component dictionary.
- DFDNet is limited by a recognition-model dictionary and coverage of only eyes, nose, and mouth, leaving regions such as skin and hair untouched.
- VQFR explores an end-to-end trained VQ codebook covering all facial areas to provide realistic details beyond selected facial components.
- Figure 2 compares reconstruction and restoration across compression patch sizes f = {8, 16, 32, 64}, with rows for HQ, LQ, large-degradation, and small-degradation faces.
3 Methodology
VQFR combines a pretrained HQ vector-quantized dictionary with a parallel decoder to restore realistic facial details while preserving degraded-input fidelity. Its design addresses the compression-size trade-off and separates texture generation from fidelity preservation.
- VQFR Framework: VQFR encodes degraded faces, replaces their latent codes using a pretrained HQ codebook, and decodes them through a parallel decoder using HQ codes and input features.The framework contains an encoder, parallel decoder, vector-quantization model, and pretrained HQ codebook.
- Vector Quantization: VQGAN maps each image to spatial latent codes, quantizes each code to its nearest codebook entry, and decodes the quantized representation back into image space.The encoder produces codes of size H/f × W/f, so each code represents an f × f image patch.
- Codebook Training: VQGAN codebooks are trained end-to-end with reconstruction, vector-quantization, perceptual, and adversarial objectives.The vector-quantization objective includes reconstruction, codebook, and commitment terms; perceptual and adversarial losses encourage perceptual quality and realistic textures.
- Compression Patch Analysis: Experiments show that compression patch size controls the trade-off between degradation removal, detailed textures, and fidelity.Small f values may retain degradations and lack detail, whereas large values improve textures but can substantially alter identity and expression.
- Parallel Decoder: The parallel decoder separates realistic detail generation from fidelity preservation by combining texture and main decoder branches.The main decoder uses degraded-input features to warp texture features, while the texture branch supplies high-quality facial details.
4 Experiments
Experiments evaluate VQFR on synthetic and real-world benchmarks, compare its perceptual quality and fidelity with prior methods, and ablate its key architectural components.
- Comparisons with State-of-the-art Methods: VQFR restores high-quality details in facial components including eyes and mouth on CelebA-Test.The qualitative comparisons examine restored facial details across multiple facial regions.
- Comparisons with State-of-the-art Methods: VQFR achieves the lowest LPIPS and best FID and NIQE on CelebA-Test, indicating perceptually close and realistic restorations.It also achieves comparable landmark distance and identity degree to GFP-GAN, preserving facial expressions and detail positions.
- Comparisons with State-of-the-art Methods: On LFW, CelebChild, and WebPhoto, VQFR largely improves realness and perceptual quality across all three real-world datasets.PULSE attains higher perceptual quality on CelebChild but suffers severely reduced fidelity; qualitative results favor VQFR across facial regions.
- Ablation Study: Using input features improves fidelity, whereas SimVQFR without them produces high perceptual quality but a large landmark distance.The visual ablation links the fidelity loss to facial-line deviations and expression changes.
- Ablation Study: The parallel decoder produces clearer, more realistic high-frequency details than the variant without it, which loses detail in the eyes and hair.The comparison uses NIQE and qualitative visualizations to assess the architectural effect.
- Ablation Study: The texture warping module preserves high-quality textures while improving fidelity by warping facial details to match degraded inputs.Without TWM, concatenation fusion cannot adjust fine details and expressions adequately.
5 Conclusion
The paper proposes VQFR, combining a VQ dictionary of high-quality facial details with a parallel decoder to preserve fidelity during restoration. Experiments report that VQFR surpasses previous methods on synthetic and real-world datasets.
- Conclusion: VQFR uses a vector-quantized dictionary of high-quality facial details for face restoration.The dictionary supplies facial detail features for restoration.
- Conclusion: A parallel decoder gradually fuses degraded-input features with VQ-codebook texture features while preserving high-quality facial details.This design targets fidelity without losing restored detail quality.
- Conclusion: VQFR produces high-quality facial details while preserving fidelity.The conclusion attributes this outcome to the VQ dictionary and parallel decoder.
- Conclusion: Extensive experiments show that VQFR surpasses previous works on both synthetic and real-world datasets.The reported evaluation spans both dataset types.
6 Appendix
The appendix details VQFR’s architecture, evaluation, limitations, ablations, and qualitative comparisons. Results show that input features, parallel decoding, and dual discriminators improve fidelity and facial-detail realism, while performance remains constrained in extreme poses and highly uninformative inputs.
- Architecture: VQFR uses six resolution levels, quantization at feature level f32, residual blocks, and a texture warping module in its parallel decoder.The architecture includes texture and main decoder branches, with deformable convolution used in the texture warping module.
- Evaluation: VQFR evaluates perceptual quality with FID, NIQE, and LPIPS, while reporting PSNR and SSIM for reference because distortion and perceptual quality can conflict.Identity fidelity is measured with the ArcFace embedding-angle metric, although the paper notes that it cannot fully reflect detailed spatial fidelity.
- Limitations: Extreme poses cause poor restoration because the codebook is built mainly from frontal training faces, while extremely uninformative inputs produce less-than-satisfactory results.The paper suggests increasing dataset diversity and codebook size, or adding generation ability to model ambiguous code selection.
- Ablation Study: Input features preserve faithful expressions, facial lines, and components, while the parallel decoder preserves VQ-generated details and improves components, hair, skin, and eyes.The ablations compare SimVQFR without input features against VQFR and a single-branch decoder against the proposed parallel decoder.
- Ablation Study: Adding a local patch discriminator to the global discriminator removes regular patterns in generated hair and skin textures.The dual-discriminator design combines a style-based wavelet-driven global discriminator with PatchGAN as the local discriminator.
- Qualitative Comparison: Across WebPhoto, Celeb-Child, and LFW-Test, VQFR produces high-quality facial components and more realistic hair and skin than previous methods.The comparisons include DFDNet, PSFRGAN, PULSE, and GFPGAN.
GFP-GAN
On real-world WebPhoto, Celeb-Child, and LFW-Test images, VQFR is qualitatively compared with prior face-restoration methods. The reported comparisons emphasize improved realism in facial components, skin, and hair, especially eyes and ears.
- WebPhoto: On WebPhoto, VQFR restores more realistic eyes and ears than previous methods.Additional WebPhoto comparisons report more realistic skin and hair.
- Celeb-Child: On Celeb-Child, VQFR restores more realistic eyes and hair than previous methods.The same qualitative outcome is reported across both cited Celeb-Child comparisons.
- LFW-Test: On LFW-Test, VQFR restores high-quality eyes and more realistic skin than previous methods.One comparison also reports high-quality hair alongside the facial components and skin.