PaperlayerView exact revision on arXiv

Source-linked AI summary

GCFSR: a Generative and Controllable Face Super Resolution Method Without Facial and GAN Priors

Jingwen He, Wu Shi, Kai Chen, Lean Fu, Chao Dong

arXiv:2203.07319v1cs.CV

TL;DR

Face super-resolution needs to preserve identity while generating fine detail, but prior-based methods can be restricted by GAN training data and fixed-strength, fixed-factor outputs. GCFSR uses an encoder-generator with style and feature modulation, trained end-to-end without additional priors. It reports superior performance to GAN-based methods across small and large factors, with different loss requirements by scale.

  • Problem

    Face SR must restore structure and identity while generating fine details at large upscaling factors, but GAN-prior methods can be dataset-restricted and fixed-factor methods cannot handle diverse sizes or adjustable detail.

    Source: §1 · p. 2

  • Method

    GCFSR uses an end-to-end encoder-generator architecture with style modulation for realistic details and factor-conditioned feature modulation to fuse encoded and generated features without additional priors.

    Source: §1 · p. 2

  • Results

    GCFSR reports superior performance to GAN-based methods across small and large factors; adversarial loss alone can yield visually pleasing results at factors 4 and 8, while factors 16, 32, and 64 use added L1 and perceptual losses.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    GCFSR can adjust generative strength through feature modulation by changing the conditional upscaling factor, including continuously during testing.

    Source: §5 · p. 8

    The study investigates only single-dimensional modulation over upscaling factor; modulation across multiple degradations and general-SR generalization remain to be studied.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Face image super resolution (face hallucination) usually relies on facial priors to restore realistic details and preserve identity information. Recent advances can achieve impressive results with the help of GAN prior. They either design complicated modules to modify the fixed GAN prior or adopt complex training strategies to finetune the generator. In this work, we propose a generative and controllable face SR framework, called GCFSR, which can reconstruct images with faithful identity information without any additional priors. Generally, GCFSR has an encoder-generator architecture. Two modules called style modulation and feature modulation are designed for the multi-factor SR task. The style modulation aims to generate realistic face details and the feature modulation dynamically fuses the multi-level encoded features and the generated ones conditioned on the upscaling factor. The simple and elegant architecture can be trained from scratch in an end-to-end manner. For small upscaling factors (<=8), GCFSR can produce surprisingly good results with only adversarial loss. After adding L1 and perceptual losses, GCFSR can outperform state-of-the-art methods for large upscaling factors (16, 32, 64). During the test phase, we can modulate the generative strength via feature modulation by changing the conditional upscaling factor continuously to achieve various generative effects.

1. Introduction

Face super-resolution must restore structure and identity while generating fine details across large upscaling factors. GCFSR addresses limits in prior-dependent methods and fixed-strength restoration with a controllable, prior-free framework.

1. Introduction

  • Face SR combines restoration—reconstructing correct structures and preserving identity—with the need to generate finer details at upscaling factors of 8–64.This makes face SR both a restoration and generation problem, and existing pipelines often use additional facial priors such as parsing or attribution maps.Source: §1 · p. 1
  • GAN-prior-dependent methods are limited by the training data of the GAN and require extra operations to adapt it for face SR.For example, a GAN trained on FFHQ may restrict a method to similar face images, such as frontal faces at a fixed size.Source: §1 · p. 2
  • Existing face SR methods offer fixed-style outputs, while users may want to adjust generated detail and conventional fixed-factor models cannot handle diverse image sizes.The passage gives old-photo restoration and surveillance-video enhancement as examples with different desired levels of hallucination.Source: §1 · p. 2
  • GCFSR uses an encoder-generator architecture with style and feature modulation to generate realistic details, fuse encoded and generated features, and adjust generative strength without additional priors.The framework is end-to-end trainable from scratch; feature modulation is conditioned on the upscaling factor.Source: §1 · p. 2
  • GCFSR achieves superior performance to GAN-based methods across small and large upscaling factors, with adversarial loss alone possible for factors ≤8.The framework is end-to-end trainable and converges fast; for large factors, the paper reports state-of-the-art performance after adding L1 and perceptual losses.Source: §1 · p. 2

2. Related Work

Prior face SR work uses facial cues or pretrained GANs to support reconstruction and detail generation. GCFSR is presented as an end-to-end alternative that uses neither facial nor GAN priors and allows generative-strength adjustment.

2.1. Face Super Resolution.

  • Facial-prior methods use landmarks, parsing maps, or facial attributes; examples predict maps for refinement, jointly learn landmarks and SR, or derive component heatmaps [6] [43] [28].Other face SR work directly applies deep neural networks without additional facial priors [3] [29] [39].Source: §2.1 · p. 2

2.2. GAN Prior

  • Recent GAN-prior methods extract feature maps or latent vectors and add modules to adapt the pretrained face GAN for restoration [4] [30] [35].GAN inversion methods optimize latent codes [11] [23], while DGP also finetunes a pretrained prior [25]; GLEAN, GPEN, and GFPGAN combine extracted multi-resolution features with GAN features [4] [30] [35].Source: §2.1 · p. 2, §2.2 · pp. 2–3
  • GCFSR instead trains end-to-end without facial or GAN priors and provides user control over generative strength.The related-work passage characterizes it as achieving state-of-the-art face SR performance.Source: §2.2 · p. 3

3. Method

GCFSR replaces adaptation or finetuning of pretrained GAN priors with an end-to-end encoder-generator trained from scratch. Its feature modulation supports multiple upscaling factors and continuous adjustment of generative strength.

3.1. Rethinking on GAN-prior-based Methods.

  • GCFSR converges fast and outperforms state-of-the-art GAN-prior-based methods, while those methods modify fixed priors or finetune them for adaptation.Figure 2 shows convergence curves for 64× SR; prior-based approaches use additional modules or finetuning.Source: Figure 2 · p. 3, §3.1 · p. 3

3.2. Overview of GCFSR

  • The encoder extracts multilevel structure and latent codes, while the generator uses both to produce realistic details and feature modulation to control their contributions by upscaling factor.The target factor also determines the strength of generated details and can be adjusted continuously at test time.Source: §3.2 · pp. 3–4

3.3. Encoder Network

  • The encoder resizes the LR input by bicubic interpolation, extracts multilevel structural features with strided convolutions, and estimates W+ latent codes from its topmost feature.The features carry structural information; the latent codes are used by the style-based generator.Source: §3.3 · p. 4

3.4. Generator Network

  • Style modulation generates details from the topmost encoded feature, while feature modulation fuses encoded and generated features at each level according to the upscaling factor.This lets the generator use both feature sources rather than generating everything from scratch.Source: §3.4 · pp. 4–5

3.5. Training Details

  • For 4× and 8× SR, adversarial-only GCFSR outperforms other blind face restoration methods; adding L1 and perceptual losses targets stronger performance at larger factors.The paper reports state-of-the-art performance at 16×, 32×, and 64× with the added losses.Source: §3.2 · p. 4, §3.5 · p. 5

3.6. Testing

  • At test time, users can vary the conditional factor continuously from 4 to 64 to adjust generative strength and select among smoothly transitioning SR outputs.For an image downscaled by 48×, the paper suggests searching between s = 32 and s = 64.Source: §3.6 · p. 5

4. Experiments

GCFSR is evaluated across large-factor face super-resolution, blind face restoration, and controllable modulation. Results favor GCFSR on the reported reconstruction metrics, while visualizations examine detail generation and feature modulation.

Comparison with state-of-the-art methods

  • GCFSR achieves the best PSNR and LPIPS for 16×, 32×, and 64× SR on CelebA-HQ.The comparison uses one GCFSR model across the three tasks, while GLEAN uses three models.Source: §4 · p. 6, Table 2 · p. 6
  • Qualitatively, GCFSR combines fidelity and naturalness; GAN inversion methods lose fidelity, while some GAN-prior methods distort or alter eye color.Figure 4 shows comparisons at 16×, 32×, and 64×, with the ground truth at 1024² resolution.Source: §4 · p. 6, Figure 4 · p. 6

Evaluation of modulation on generative strength

  • Continuous adjustment of conditional factor s produces satisfactory results between tested endpoints, including for inputs with factors 24 and 48.The modulation transitions smoothly without noticeable artifacts; s = 24 can yield a blurrier effect and s = 48 a stronger generative effect around a target of 32.Source: §4 · p. 7, Figure 5 · p. 7

The Effectiveness on Blind Face Restoration

  • On CelebA-Test blind face restoration, GCFSR leads in PSNR, SSIM, LPIPS, and ArcFace cosine similarity, with comparable FID.The evaluation uses 3,000 CelebA-HQ test images and a degradation model adopted from GFPGAN.Source: §4 · p. 7, Table 3 · p. 7

Analysis and visualization on feature modulation

  • As s increases, encoder scaling vectors approach 0 while generator scaling vectors approach 1, shifting feature contributions toward generated features.The histograms visualize σ64 enc and σ64 gen across conditional upscaling factors.Source: §4 · p. 7, Figure 6 · p. 8

Effects of learning objectives

  • GCFSRadv outperforms the compared blind face restoration methods while using only adversarial loss for 4× and 8× SR.The comparison includes GFPGAN, GPEN, DFDNet, PSFRGAN, and HiFaceGAN; the models are evaluated at 512 × 512 resolution.Source: §4 · p. 8, Table 4 · p. 8

Effects of style modulation

  • Style modulation produces more realistic eyelash and hair details and fewer artifacts, and improves overall performance on most metrics.The visual comparison is on CelebA-HQ at 16×, 32×, and 64× SR.Source: §4 · p. 8, Figure 7 · p. 8

5. Conclusion

GCFSR is an end-to-end trainable, prior-free face super-resolution framework that handles upscaling factors up to 64×. Its modulation modules support identity-faithful reconstruction and user adjustment, while the work identifies limits in modulation scope and generalization.

5. Conclusion

  • GCFSR handles face SR up to 64× without additional priors using an end-to-end trainable encoder-generator architecture.Style modulation generates realistic details, and feature modulation fuses encoded and generated features according to the conditional upscaling factor.Source: §5 · p. 8

Limitations

  • The study investigates only single-dimensional modulation over upscaling factor; modulation across multiple degradations remains a real-world consideration.Source: §5 · pp. 8–9
  • GCFSR's generalization to general image SR remains unstudied.Source: §5 · p. 9

Related papers

Loading 2203.07319v1…