Source-linked AI summary
Towards Robust Blind Face Restoration with Codebook Lookup Transformer
Shangchen Zhou, Kelvin C. K. Chan, Chongyi Li, Chen Change Loy
TL;DR
Blind face restoration is highly ill-posed because degradation removes information and leaves many plausible high-quality outputs. CodeFormer uses a learned discrete codebook with Transformer code prediction and controllable feature transformation, and experiments report superior quality, fidelity, and robustness across degradation settings. The method remains limited by codebook expressiveness for some accessories and by weaker performance on side faces.
Problem
Blind face restoration lacks a reliable LQ-HQ mapping when degradation is unknown, so auxiliary guidance is needed to reduce uncertainty and restore lost details.
Method
CodeFormer predicts discrete code sequences in a learned finite codebook space using global Transformer modeling, while controllable feature transformation adjusts LQ information flow.
Results
CodeFormer outperforms state-of-the-art methods in quality and fidelity and shows strong robustness on synthetic and real-world datasets and face inpainting.
Takeaways & Limitations
Discrete codebook priors and global composition modeling provide an effective basis for robust blind face restoration under heavy degradation.
Takeaways & Limitations
CodeFormer remains inconsistent on rare accessories and offers limited superiority or poor results for side faces because the current codebook cannot fully represent those images.
Abstract
from arXiv · showhide
Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting blind face restoration as a code prediction task, while providing rich visual atoms for generating high-quality faces. Under this paradigm, we propose a Transformer-based prediction network, named CodeFormer, to model the global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded. To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a flexible trade-off between fidelity and quality. Thanks to the expressive codebook prior and global modeling, CodeFormer outperforms the state of the arts in both quality and fidelity, showing superior robustness to degradation. Extensive experimental results on synthetic and real-world datasets verify the effectiveness of our method.
1 Introduction
Blind face restoration is highly ill-posed because unknown degradation creates many plausible outputs, while existing priors provide insufficient guidance. CodeFormer addresses this by predicting discrete code sequences with global Transformer modeling and controllable information flow, achieving robust restoration across degradation levels.
- Motivation: Unknown degradation and information loss make blind face restoration an underconstrained LQ-HQ mapping with infinitely many plausible outputs.Earlier approaches struggle to learn this mapping directly in the full image space without auxiliary guidance.
- Motivation: Existing geometric, reference, and generative priors improve details but remain sensitive to degradation or limited in expressiveness.Their insufficient guidance leaves restoration uncertainty and dependence on corrupted LQ inputs.
- Method: A learned discrete codebook converts restoration into code prediction in a finite proxy space, reducing mapping uncertainty while storing rich HQ visual details.The codebook and decoder are learned through HQ self-reconstruction, with an example proxy space containing 1024 codes.
- Method: CodeFormer uses a Transformer to model global face composition and long-range dependencies, improving code prediction when local LQ information is lost.This global modeling helps recover details such as glasses while improving restoration quality and fidelity over nearest-neighbor code matching.
- Method: The controllable feature transformation module adjusts LQ-feature information flow, enabling a quality-fidelity trade-off across different degradation strengths.Smaller information flow can favor quality under heavy degradation, while larger flow can preserve fidelity under mild degradation.
- Experiments: Experiments on existing datasets and the 970-image WIDER-Test dataset show strong performance and robustness, including on face inpainting.The paper reports effectiveness on synthetic and real-world data and on tasks requiring long-range clues.
2 Related Work
Prior blind face restoration methods use geometric, reference, generative, or dictionary-based guidance, but each has important availability, fidelity, or efficiency constraints. CodeFormer is presented as a discrete-prior framework for addressing these limitations.
- Blind Face Restoration: Geometric priors use landmarks, parsing maps, component heatmaps, or 3D shapes, but degraded inputs make these signals difficult to acquire accurately.They also cannot provide rich details for high-quality restoration.
- Blind Face Restoration: Reference-based methods improve details using same-identity guidance, but suitable references are not always available.Dictionary-based variants pre-construct high-quality facial component features as an alternative.
- Blind Face Restoration: Generative priors such as StyleGAN2 support inversion or latent encoding, yet continuous latent projection makes high-fidelity restoration challenging.Encoder-decoder designs embed generative priors to mitigate this issue.
- Dictionary Learning: Learned dictionaries have aided restoration, but traditional dictionary learning and sparse coding often require iterative optimization with high computational cost.Their HQ-dictionary insight has influenced reference-based restoration networks.
- CodeFormer: CodeFormer instead uses a discrete codebook prior and Transformer-based global composition modeling to reduce uncertainty and compensate for lost local details.Its framework is introduced as a response to the limitations of prior guidance strategies.
3 Methodology
The method learns a discrete codebook prior and predicts code sequences from low-quality faces with a Transformer, then uses controllable feature transformation to balance restoration quality and fidelity.
- 3.1 Codebook Learning: The codebook prior reduces restoration uncertainty by replacing direct low-quality-to-high-quality image mapping with prediction in a finite discrete proxy space.Codebook combinations provide a finite representation space for restoration.
- 3.1 Codebook Learning: A quantized autoencoder learns a discrete codebook and decoder from high-quality face self-reconstruction, providing a context-rich prior for restoration.The codebook represents faces in a compressed proxy space, while the decoder stores high-quality visual details.
- 3.2 Codebook Lookup Transformer Learning (Stage II): Nearest-neighbor lookup is unreliable under diverse degradation, so CodeFormer uses a Transformer to model global interrelations and predict accurate code compositions.The network uses nine self-attention blocks, positional embeddings, and a final projection to code-token probabilities.
- 3.2 Codebook Lookup Transformer Learning (Stage II): The predicted code sequence retrieves codebook items whose quantized feature is decoded by the fixed decoder into a high-quality face image.The codebook and decoder remain fixed while the Transformer and low-quality encoder are trained with code-level objectives.
- 3.3 Controllable Feature Transformation (Stage III): The controllable feature transformation module adjusts low-quality information flow with coefficient w, enabling high fidelity for mild degradation and high quality for heavy degradation.Smaller w reduces reliance on heavily degraded inputs, whereas larger w introduces more low-quality information; inference supports w in [0, 1].
4 Experiments
CodeFormer is evaluated on synthetic and real-world face datasets using quantitative, qualitative, ablation, efficiency, enhancement, and inpainting studies. Results show strong restoration quality, identity preservation, degradation robustness, and controllable quality–fidelity behavior, while side-face restoration remains limited.
- Experimental Settings: Models are trained on FFHQ images with synthesized blur, downsampling, noise, and JPEG degradation, then evaluated on CelebA-Test and three real-world datasets spanning mild to heavy degradation.CelebA-Test contains 3,000 images; LFW-Test, WebPhoto-Test, and WIDER-Test represent increasing real-world degradation.
- Comparisons with State-of-the-Art Methods: On CelebA-Test, CodeFormer achieves the best LPIPS, FID, and MUSIQ scores while also obtaining the highest IDS and PSNR scores among compared methods.The qualitative results further show high-quality faces with faithful details and identity preservation under severe degradation.
- Comparisons with State-of-the-Art Methods: On real-world datasets, CodeFormer achieves the best FID under heavy degradation and remains competitive under mild and medium degradation while preserving identity better than PULSE.Figure 4 reports robustness to heavily degraded real-world faces.
- Ablation Studies: Removing the codebook worsens LPIPS and IDS, while Transformer-based code prediction improves fidelity and code prediction accuracy over nearest-neighbor matching and a pure CNN-based predictor.The Transformer has higher code prediction accuracy across all degradation degrees.
- Ablation Studies: Fine-tuning the decoder deteriorates performance, supporting the use of a fixed decoder with the compact codebook to preserve the learned prior.The adopted codebook contains N=1024 entries of dimension d=256.
- Additional Applications: CodeFormer produces natural colors in old-photo enhancement and remains effective for face inpainting, including cases with large mask ratios.These studies extend evaluation beyond standard face restoration.
- Limitations: CodeFormer remains limited on side faces, where sparse representation in the FFHQ training data leaves the codebook without sufficient codes for effective reconstruction and restoration.Identity inconsistency can also persist in rare visual parts such as accessories.
5 Conclusion
CodeFormer addresses blind face restoration with a learned discrete codebook, Transformer-based global modeling, and controllable feature transformation. The method is reported to be expressive, robust to heavy degradation, and adaptable across restoration settings.
- CodeFormer casts face restoration as code token prediction in a small, expressive discrete codebook space.This reduces restoration-mapping uncertainty and eases learning of the restoration network.
- A Transformer models global composition and dependencies in degraded faces for better code prediction.The global modeling targets local information loss in degraded inputs.
- A controllable feature transformation module enables a flexible trade-off between fidelity and quality under different degradations.
- Experiments report strong expressiveness, robustness against heavy degradation, and superiority over previous methods.
- The appendix extends evaluation to face color enhancement, face inpainting, old photo enhancement, and AI-generated image correction.
A.1 Effect of the Number of Codebook
The reconstruction capability of the learned codebook prior constrains CodeFormer's expressiveness. Ablation results indicate that reconstruction improves as more codebook items are activated and learned.
- CodeFormer's expressiveness is affected by the reconstruction capability of its learned codebook prior.
- Reconstruction on FFHQ improves as more codebook items are activated and learned.Table 6 reports reconstruction using LPIPS and PSNR.
A.2 Transformer Structure
The Transformer ablations examine input scale and layer count for code prediction and restoration. Nine layers provide a better computational-complexity/performance trade-off, while larger-scale features can retain degradation.
- The Transformer predicts code tokens by modeling global interrelations among low-quality faces.
- Single-scale and Multi-scale Inputs: Single-scale and multi-scale Transformer variants use encoder features at resolutions 16 or 16, 32, and 64.Multi-scale features are rearranged to 16 × 16, channel-reduced, and summed before Transformer input.
- Number of Transformer Layer: Nine Transformer layers provide a better trade-off among computational complexity and performance.Beyond nine layers, code-prediction gains are slight and LPIPS is not improved.
- Table 7 reports code-prediction Accuracy↑ and restoration LPIPS↓ across Transformer structures on CelebA-Test.
B More Results on Blind Face Restoration
On real-world severely degraded faces, CodeFormer produces high-quality and natural restorations while preserving identity. The Wider-Test qualitative comparison highlights its robustness and richer detail recovery.
- CodeFormer produces high-quality faces and preserves identities on highly degraded real-world inputs.The comparison includes DFDNet, PSFRGAN, GLEAN, GFP-GAN, and GPEN.
- Compared with other methods, CodeFormer recovers richer details and produces more natural faces.
C More Results on Extensions
CodeFormer extends beyond face restoration to color enhancement, inpainting, old photo enhancement, and correction of AI-generated images.
- CodeFormer is evaluated on face color enhancement and face inpainting, with additional examples for old historic photos and AI-generated artwork.
C.1 Face Color Enhancement
For face color enhancement, CodeFormer is finetuned with color augmentations and compared with GFP-GAN on real-world old photos. It produces high-quality faces with natural colors and faithful details.
- CodeFormer is finetuned using random color jitter and grayscale conversion, matching GFP-GAN’s color augmentations.
- CodeFormer is evaluated against GFP-GAN (version 1) on real-world old photos from CelebChild-Test with color loss.
- CodeFormer generates high-quality face images with more natural color and faithful details.
C.2 Face Inpainting
CodeFormer is extended to face inpainting, historical-photo enhancement, and AI-generated face correction. Inpainting results are reported as more natural and free of strokes and artifacts, including under large mask ratios.
- C.2 Face Inpainting: For face inpainting, CodeFormer is retrained on synthetic paired data with randomly drawn irregular polyline masks.
- C.2 Face Inpainting: CodeFormer is compared with CTSDG and GPEN for face inpainting and produces more natural faces without strokes and artifacts.
- C.2 Face Inpainting: Visual comparisons report that CodeFormer maintains superior face-inpainting performance even under very large mask ratios.
- C.3 Old Photo Enhancement: For an old photo of the 5-th Solvay conference from 1927, faces are cropped, aligned, enhanced with CodeFormer, and pasted back into the original photo.
- C.4 AI-Generated Image Correction: CodeFormer is used to fix faces in AI-generated artwork, including examples generated by Stable Diffusion.