Source-linked AI summary
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
Kelvin C. K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, Chen Change Loy
TL;DR
Large-factor super-resolution is severely underspecified because downsampling removes details and textures, while existing GAN-based approaches face fidelity, artifact, or runtime-optimization limitations. GLEAN uses a pre-trained GAN as a generative latent bank in an encoder-bank-decoder architecture, conditioning it with latent vectors and multi-resolution features. The method produces photorealistic high-resolution images in a single forward pass and can generalize across image categories by switching latent banks.
Problem
Large-factor SR at 8×–64× loses most details and textures, creating an underspecified restoration problem that requires informative priors.
Method
GLEAN uses a pre-trained GAN as a generative latent bank, conditioned by latent vectors and multi-resolution convolutional features in an encoder-bank-decoder architecture.
Results
GLEAN handles challenging poses and structures and generalizes across human faces, cats, and buildings by switching pre-trained GAN latent banks.
Takeaways & Limitations
After training, GLEAN reconstructs photorealistic high-resolution images with a single forward pass and may extend to denoising, inpainting, and colorization.
Takeaways & Limitations
GAN-based SR trained from scratch can struggle to approximate the natural image manifold and produce artifacts, while GAN inversion can lose fidelity and require time-consuming iterative optimization.
Abstract
from arXiv · showhide
We show that pre-trained Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR). While most existing SR approaches attempt to generate realistic textures through learning with adversarial loss, our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN. But unlike prevalent GAN inversion methods that require expensive image-specific optimization at runtime, our approach only needs a single forward pass to generate the upscaled image. GLEAN can be easily incorporated in a simple encoder-bank-decoder architecture with multi-resolution skip connections. Switching the bank allows the method to deal with images from diverse categories, e.g., cat, building, human face, and car. Images upscaled by GLEAN show clear improvements in terms of fidelity and texture faithfulness in comparison to existing methods.
1. Introduction
Large-factor super-resolution at 8×–64× is severely underspecified because downsampling removes details and textures. GLEAN addresses this by using a pre-trained GAN as a latent bank within an encoder-bank-decoder architecture.
- At 8× to 64× magnification, downsampling removes most details and textures, making informative priors especially important for restoration.
- Conventional GAN-based SR trains a generator from scratch to learn both natural image characteristics and fidelity, which can limit natural-manifold approximation and produce artifacts.
- GAN inversion improves use of latent spaces but often sacrifices ground-truth fidelity because low-dimensional codes and image-space constraints provide insufficient spatial guidance.
- GLEAN leverages pre-trained GANs such as StyleGAN as rich, diverse priors without image-specific runtime optimization, requiring only a single forward pass after training.
- Its encoder-bank-decoder architecture conditions the latent bank on latent vectors and multi-resolution convolutional features, while passing multi-resolution cues to the decoder.
- Switching pre-trained GAN latent banks allows GLEAN to handle categories including human faces, cats, and buildings, while addressing challenging poses and structures.
2. Related Work
Prior SR methods often trade perceptual realism against fidelity, while GAN inversion uses iterative optimization and may lose spatial information. GLEAN instead conditions a pre-trained GAN with both latent codes and convolutional features in an encoder-bank-decoder design.
- Pixel-wise SR methods can achieve strong PSNR but often produce perceptually unconvincing, over-smoothed outputs.
- GAN-based SR improves photorealism, but generators trained from scratch must learn fidelity and natural image characteristics simultaneously, leaving possible unnatural artifacts.
- Large-factor SR research extends beyond typical 2× or 4× scaling to settings such as 8×, 16×, and extremely coarse inputs.
- GAN inversion optimizes latent codes, and sometimes the generator, to reconstruct degraded images, but low-dimensional codes may fail to preserve important spatial information.
- GLEAN differs by conditioning the pre-trained generator with latent codes and multi-resolution convolutional features, avoiding iterative optimization during inference.
- The GLEAN overview depicts a 32×32 input mapped to a 256×256 output through encoder, latent-bank, and decoder blocks.
3. Methodology
GLEAN exploits a fixed pre-trained GAN as a latent bank within an encoder-bank-decoder architecture. The encoder supplies latent vectors and multi-resolution features, while the decoder progressively fuses bank and encoder features to reconstruct images.
- Generative Latent Bank: GLEAN uses a pre-trained GAN as a latent bank, avoiding expensive image-specific optimization during inference.The latent bank provides generative priors while the model performs upscaling in a single forward pass.
- Encoder: The encoder extracts latent vectors and multi-resolution convolutional features that capture high-level information and local image structure.The latent vectors form a compressed representation, while convolutional features guide structure restoration.
- Generative Latent Bank: The adapted generator outputs multi-resolution features rather than images, enabling the decoder to combine latent-bank and encoder information.This design better fuses the two feature sources before reconstruction.
- Generative Latent Bank: Each StyleGAN block receives a different latent vector, and additional convolutions fuse encoder features into the augmented style blocks.Using separate latent vectors improves expressiveness and produces outputs with fewer artifacts.
- Advantages: GLEAN’s GAN-based dictionary captures image distributions without dependence on selected reference components or images, supporting potentially unlimited size and diversity.This contrasts with imagery dictionaries whose diversity and generalization depend on selected references.
- Decoder: A progressive-fusion decoder combines encoder and latent-bank features through 3×3 convolutions, pixelshuffle layers, and encoder-decoder skip connections.The skip connections reinforce encoder information so the latent bank can focus on texture and detail generation.
4. Experiments
GLEAN is evaluated on large-factor super-resolution across faces and diverse categories, scale factors, poses, and quantitative metrics. It generally preserves fidelity while producing more realistic textures than inversion- and adversarial-loss-based baselines.
- Qualitative comparison: GLEAN preserves identity and high fidelity while synthesizing realistic textures on 16× CelebA-HQ super-resolution.GAN inversion methods fail to preserve identity, while adversarial-loss methods struggle with fine details.
- Categories and robustness: GLEAN applies across categories by switching between StyleGANs trained on different image domains.Reported categories include faces, cats, cars, bedrooms, and towers.
- Scale factors: GLEAN generates perceptually convincing images resembling the ground truth for up to 64× upscaling.The method extends beyond standard large-factor settings while maintaining realistic reconstructions.
- Categories and robustness: Despite training on aligned human faces, GLEAN reconstructs realistic images with non-aligned poses and non-human faces.Its outputs remain faithful where PULSE becomes significantly different from the ground truth at high resolution.
- Quantitative comparison: GLEAN achieves higher cosine similarity to ground truth than baseline methods in ArcFace embedding space.The evaluation uses 100 CelebA-HQ images.
- Quantitative comparison: GLEAN outperforms other methods in most categories on 16× SR using PSNR and LPIPS.The comparison includes ESRGAN+, a larger ESRGAN version with similar FLOPs to GLEAN.
5. Ablation Studies
Ablations show that GLEAN’s encoder features, latent bank, and decoder each contribute distinct restoration benefits. Multi-resolution conditioning restores local structure, latent priors improve textures, and coarse-to-fine decoding enhances natural details.
- Importance of multi-resolution encoder features: Multi-resolution convolutional features progressively improve fine details, local structures, quality, and fidelity beyond latent vectors alone.Latent vectors mainly recover global attributes such as hair color and pose.
- Effects of latent bank features: Latent bank features improve both structure restoration and texture generation as finer features are passed to the decoder.The fixed generator prior reduces the network’s burden of generating realistic details while preserving fidelity.
- Importance of decoder: The decoder aggregates information coarse-to-fine, while multi-scale skip connections reinforce spatial information from encoder features.Without the decoder, perceptually convincing outputs still contain unpleasant artifacts when zoomed in.
- Comparisons with reference-based methods: Unlike DFDNet’s component dictionary, GLEAN can restore coherent images when components such as skin and hair are absent from the dictionary.DFDNet shows strong face-restoration performance but cannot faithfully produce dictionary-absent parts.
- Image retouching: GLEAN eliminates blurry and incoherent artifacts introduced by amateur retouching while preserving natural textures.The result is shown for interactive image retouching.
- Comparisons with reference-based methods: GLEAN avoids explicit texture matching, enabling large-factor SR where global matching is extremely difficult or computationally prohibitive.SRNTT tends to provide blurry textures on 32×32 inputs because global matching becomes prohibitive.
6. Application – Image Retouching
GLEAN is applied to interactive image retouching to correct artifacts introduced by imprecise editing. It removes blurry regions and produces coherent outputs with natural textures in one forward pass.
- Motivation: Amateur interactive retouching commonly introduces unpleasant artifacts that require tedious and precise editing to avoid.The application targets imperfect manually edited images.
- Results: GLEAN eliminates blurry regions from interactive edits and generates coherent outputs with natural textures.The method’s high fidelity and quality support this retouching application.
- Results: A single forward pass makes GLEAN easy to incorporate into existing interactive editing software.This preserves the method’s runtime practicality in the retouching setting.
7. Conclusion
GLEAN uses pre-trained GANs as a generative latent bank for large-scale super-resolution up to 64×. Its encoder-bank-decoder design reconstructs photorealistic high-resolution images in a single forward pass and may extend to other imaging tasks.
- GLEAN exploits pre-trained GANs as a generative latent bank for large-scale super-resolution up to 64×.
- Its encoder-bank-decoder architecture retrieves rich priors effectively, enabling photorealistic high-resolution reconstruction with one forward pass.
- The GAN-based dictionary concept may extend GLEAN beyond super-resolution to denoising, inpainting, and colorization.
A. Training Details of GLEAN
GLEAN trains with fixed pre-trained StyleGAN latent banks on five image categories using fidelity, perceptual, and adversarial objectives. Training uses Adam with cosine annealing for 300K iterations.
- Pre-trained StyleGAN5 or StyleGAN2 serves as the fixed generative latent bank during GLEAN training.Keeping the bank fixed preserves the generative prior and avoids bias toward the training distribution.
- GLEAN is trained on human faces, cats, cars, towers, and bedrooms, with resized datasets because StyleGAN generates fixed-size images.
- The training objective combines MSE fidelity loss, perceptual loss, and adversarial loss.MSE guides fidelity, while perceptual and adversarial losses improve perceptual quality.
- Training uses cosine annealing and Adam for 300K iterations, starting at a learning rate of 10^-4.The batch size is 8 for human faces and 16 for other categories, using two Nvidia V100 GPUs.
B.1. Super-Resolution
Across scale factors from 8× to 64× and categories including faces, cats, cars, bedrooms, and towers, GLEAN outperforms DGP and ESRGAN+ in fidelity and quality.
- GLEAN extends across 8× to 64× super-resolution and multiple categories, including faces, cats, cars, bedrooms, and towers.
- GLEAN outperforms DGP and ESRGAN+ in both fidelity and quality across the reported scale factors and categories.DGP and ESRGAN+ are less promising on categories other than human faces.
B.2. Image Retouching
GLEAN refines imperfectly retouched images by downsampling away artifacts and super-resolving them back to the original resolution. It produces realistic, coherent refinements in a single forward pass, supporting interactive editing.
- Amateur retouching commonly introduces artifacts because precise, high-quality editing is tedious.
- GLEAN refines retouched images by downsampling them until artifacts vanish, then upsampling them back to the original resolution.
- GLEAN suppresses unnatural retouching artifacts while remaining similar to the edited image and coherent with unaltered regions.
- Compared with GLEAN, PULSE differs structurally from ground truths, while DGP and ESRGAN+ show identity differences or fine-detail artifacts.
- On 16×, 32×, and 64× super-resolution examples, GLEAN synthesizes images resembling the ground truths up to 64×.
- For cats, cars, bedrooms, and towers, GLEAN produces realistic outputs highly similar to ground truths at 8× super-resolution.