Source-linked AI summary
RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs
Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, Ping Luo
TL;DR
Blind face restoration seeks high-quality faces from unknown degradations, while existing local fusion methods underuse facial context. RestoreFormer combines cross-attention between degraded queries and high-quality priors from a reconstruction-oriented dictionary, and extensive evaluations report superior performance and visual quality across synthetic and real-world datasets.
Problem
Blind face restoration must recover high-quality faces from complex, unknown degradations, while prior methods may underuse abundant facial context during information fusion.
Method
RestoreFormer uses multi-head cross-attention to fuse degraded-face queries with high-quality key-value pairs sampled from a reconstruction-oriented HQ Dictionary.
Results
RestoreFormer outperforms advanced state-of-the-art methods on synthetic and real-world datasets and produces faces with better visual quality.
Takeaways & Limitations
Reconstruction-oriented priors provide high-quality facial details while cross-attention models spatial interactions between degraded faces and those priors.
Abstract
from arXiv · showhide
Blind face restoration is to recover a high-quality face image from unknown degradations. As face image contains abundant contextual information, we propose a method, RestoreFormer, which explores fully-spatial attentions to model contextual information and surpasses existing works that use local operators. RestoreFormer has several benefits compared to prior arts. First, unlike the conventional multi-head self-attention in previous Vision Transformers (ViTs), RestoreFormer incorporates a multi-head cross-attention layer to learn fully-spatial interactions between corrupted queries and high-quality key-value pairs. Second, the key-value pairs in ResotreFormer are sampled from a reconstruction-oriented high-quality dictionary, whose elements are rich in high-quality facial features specifically aimed for face reconstruction, leading to superior restoration results. Third, RestoreFormer outperforms advanced state-of-the-art methods on one synthetic dataset and three real-world datasets, as well as produces images with better visual quality.
1. Introduction
Blind face restoration must recover high-quality faces despite unknown, diverse degradations. RestoreFormer addresses limitations of local fusion by combining fully spatial attention with reconstruction-oriented high-quality priors.
- Unknown degradations such as down-sampling, blur, noise, and compression artifacts make blind face restoration challenging.
- Existing prior-based methods fuse degraded-face identity information with high-quality details, but pixel-wise SFT neglects abundant facial context.
- Its HQ Dictionary is reconstruction-oriented, learned from undegraded faces, and rich in details specifically aimed at face reconstruction.
- RestoreFormer learns fully spatial interactions between corrupted queries and high-quality key-value pairs to restore faces with realness and fidelity.
- Extensive experiments report that RestoreFormer outperforms advanced state-of-the-art methods on synthetic and real-world datasets while producing better visual quality.
2. Related Works
Related work uses geometric, reference, and generative priors for blind face restoration, but each has limitations. Geometric priors depend on corrupted inputs, references may require identity matching, and local combinations can ignore facial context.
- Blind face restoration research commonly uses geometric priors, references, or generative priors to supplement degraded faces.
- Geometric priors such as landmark and facial-component heatmaps are limited because they are mainly generated from low-quality faces.
- Reference-based methods require references matching the degraded face’s identity, which is not always accessible.
- Locally combining information ignores the rich facial context in face images.
- Vision Transformers extend contextual modeling to vision tasks, including low-level image processing and face structure modeling.
3. Methodology
RestoreFormer restores degraded faces by fully spatially fusing degraded features with reconstruction-oriented high-quality priors. Its multi-head cross-attention, HQ Dictionary, and multi-level losses jointly target facial realness, fidelity, and detail.
- RestoreFormer pipeline: RestoreFormer extracts degraded features, retrieves nearest priors from an HQ Dictionary, fuses them with two MHCAs, and decodes a restored face.The pipeline uses an encoder for degraded representations, cross-attention for fusion, and a decoder for reconstruction.
- Multi-head cross-attention: MHCA uses degraded features as queries and high-quality priors as keys and values to fully spatially fuse identity information with facial details.Unlike MHSA, which derives queries, keys, and values from degraded features, MHCA combines two information sources.
- Training objectives: RestoreFormer combines pixel-level L1 and perceptual losses with component-level eye and mouth losses and image-level adversarial and identity losses.The training objective addresses image fidelity, perceptual quality, important facial components, realness, and identity preservation.
- HQ Dictionary: The HQ Dictionary is learned from undegraded faces with a vector-quantized generation network, producing reconstruction-oriented priors rich in facial details.Its priors are intended for reconstruction rather than recognition and are designed to provide richer facial information.
4. Experiments and Analysis
RestoreFormer is evaluated on one synthetic and three real-world datasets using fidelity, realness, identity, qualitative, user-study, and ablation analyses. It achieves stronger realness and identity preservation while producing visually detailed results and benefiting from spatial attention and information fusion.
- Datasets and settings: The evaluation covers CelebA-Test plus LFW-Test, CelebChild-Test, and WebPhoto-Test, with synthetic and real-world degradation settings.CelebA-Test contains 3000 images, while LFW-Test contains 1711 images; the other datasets are real-world collections.
- Datasets and settings: RestoreFormer is assessed with FID, PSNR, SSIM, LPIPS, identity distance, and a human user study measuring visual quality.FID evaluates realness, while pixelwise, perceptual, and ArcFace-based identity measures evaluate fidelity and identity preservation.
- Comparison with state-of-the-art methods: On CelebA-Test, RestoreFormer performs better on FID and IDD while remaining comparable on PSNR, SSIM, and LPIPS.The reported comparison emphasizes realness and identity preservation rather than superiority on every metric.
- Comparison with state-of-the-art methods: On real-world examples, RestoreFormer produces more identity-consistent and visually pleasing faces, with improved reconstruction of details such as eyebrows, eyeglasses, and eyes.The qualitative comparisons attribute these differences to using contextual information when fusing degraded inputs and priors.
- Comparison with state-of-the-art methods: A user study with 100 volunteers and 200 samples found RestoreFormer preferred over DFDNet, PSFRGAN, and GFP-GAN at a higher percentage.The samples came from LFW-Test and WebPhoto-Test, with 100 samples drawn from each dataset.
- Ablation study: Ablations indicate that spatial attention and fusion of degraded information with HQ Dictionary priors improve restoration, while the reconstruction-oriented dictionary is important to the method.MHSA improves contextual processing over direct degraded-image features, MHCA outperforms local SFT fusion qualitatively, and fusing both information sources better preserves identity and details.
5. Conclusion
RestoreFormer addresses blind face restoration by modeling contextual information through fully spatial attention and reconstruction-oriented high-quality features. The paper reports superior capability in comparisons with state-of-the-art methods.
- RestoreFormer uses multi-head cross-attention to learn spatial interactions between corrupted queries and high-quality key-value pairs.
- Its key-value pairs come from a reconstruction-oriented dictionary containing high-quality facial features aimed specifically at face reconstruction.
- Extensive comparisons on several datasets demonstrate RestoreFormer’s superior capability for blind face restoration.