PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors

Chaofeng Chen, Xinyu Shi, Yipeng Qin, Xiaoming Li, Xiaoguang Han, Tao Yang, Shihui Guo

arXiv:2202.13142v3cs.CV

TL;DR

Real-world blind SR must recover details from LR images with complex unknown degradations, while prior approaches can suffer from artifacts or require unavailable HR references. FeMaSR learns HR priors with a VQGAN, matches LR features to a pretrained codebook, and decodes them with residual shortcuts. The authors report more realistic HR images than previous methods, while the method assumes a fixed pretrained prior and nearest-neighbor matching.

  • Problem

    Real-world blind SR must recover missing details from LR images with complex unknown degradations, while prior methods may suffer GAN artifacts or require unavailable HR references.

  • Method

    FeMaSR pretrains a VQGAN codebook and decoder as HR priors, then matches LR encoder features to the codebook and decodes them with residual shortcut connections.

  • Results

    FeMaSR produces more realistic HR images than previous methods in experiments on synthetic and real-world benchmarks.

  • Takeaways & Limitations

    Feature matching with fixed distortion-free HR priors provides a compact route to realistic real-world blind SR with fewer artifacts.

  • Takeaways & Limitations

    The feature-matching setup uses a fixed pretrained HR prior and resolves rare nearest-neighbor ties by random mapping.

Abstract

from arXiv · show

A key challenge of real-world image super-resolution (SR) is to recover the missing details in low-resolution (LR) images with complex unknown degradations (e.g., downsampling, noise and compression). Most previous works restore such missing details in the image space. To cope with the high diversity of natural images, they either rely on the unstable GANs that are difficult to train and prone to artifacts, or resort to explicit references from high-resolution (HR) images that are usually unavailable. In this work, we propose Feature Matching SR (FeMaSR), which restores realistic HR images in a much more compact feature space. Unlike image-space methods, our FeMaSR restores HR images by matching distorted LR image features to their distortion-free HR counterparts in our pretrained HR priors, and decoding the matched features to obtain realistic HR images. Specifically, our HR priors contain a discrete feature codebook and its associated decoder, which are pretrained on HR images with a Vector Quantized Generative Adversarial Network (VQGAN). Notably, we incorporate a novel semantic regularization in VQGAN to improve the quality of reconstructed images. For the feature matching, we first extract LR features with an LR encoder consisting of several Swin Transformer blocks and then follow a simple nearest neighbour strategy to match them with the pretrained codebook. In particular, we equip the LR encoder with residual shortcut connections to the decoder, which is critical to the optimization of feature matching loss and also helps to complement the possible feature matching errors. Experimental results show that our approach produces more realistic HR images than previous methods. Codes are released at https://github.com/chaofengc/FeMaSR.

1 INTRODUCTION

Real-world blind SR must recover details despite unknown degradations, while prior solutions either simplify degradation assumptions, risk GAN artifacts, or depend on unavailable references. FeMaSR addresses this by matching LR features to pretrained distortion-free HR priors, with semantic regularization and residual shortcuts supporting realistic reconstruction.

  • Blind SR is ill-posed because unknown distortions disrupt image details, making simplified degradation assumptions insufficient for real-world cases.Some methods explicitly estimate blur and noise, but generally handle only limited cases of classical degradation models.
  • GAN-based methods synthesize missing textures but are prone to artifacts because GAN training is unstable.
  • FeMaSR reformulates blind SR as matching LR features with distortion-free features in pretrained implicit HR priors.Its HR priors combine a discrete feature codebook with a corresponding pretrained decoder, separating prior learning from LR-to-HR feature mapping.
  • Semantic regularization based on VGG19 perceptual features strengthens the correlation between semantics and codebook features, improving the HR prior.
  • Residual shortcut connections from LR feature space to decoder features facilitate feature-matching optimization and complement matching errors.

2 RELATED WORK

Related SR methods span conventional architectures, degradation modeling, image translation, and pretrained GAN latent spaces. FeMaSR is framed around a two-stage HR-prior and feature-matching framework, illustrated as a pipeline from VQGAN pretraining to LR feature matching.

  • Single Image Super-Resolution (SISR): SISR research has progressed through deeper networks, residual and residual-dense blocks, and attention mechanisms.
  • Figure 2 depicts FeMaSR as two stages: pretraining a VQGAN-based HR prior, followed by super-resolution through feature matching.The pretrained prior consists of a discrete codebook and decoder, while the LR encoder matches input features to the codebook.
  • Explicit degradation methods such as BSRGAN and Real-ESRGAN improve visual quality through manually designed degradation pipelines.
  • Pretrained StyleGAN latent-space approaches show promise for face SR but have limited applicability to natural images because training GANs for natural images remains difficult.

3 METHODOLOGY

FeMaSR treats blind real-world super-resolution as matching degraded LR features to distortion-free HR priors, then decoding the matched features. Its two-stage design combines semantic-guided VQGAN priors, Swin-based LR encoding, nearest-neighbor matching, and residual shortcuts.

  • Framework Overview: FeMaSR pretrains HR priors with a VQGAN encoder, discrete codebook, and decoder, then uses them for blind super-resolution.The codebook and decoder approximately encode complete HR-image information and reconstruct images from corresponding feature codes.
  • Super-Resolution via Feature Matching: Blind SR is reformulated as matching LR feature codes to distortion-free HR codebook features before decoding realistic HR images.The matched HR codes are decoded instead of directly decoding raw LR features.
  • Pretraining of High-Resolution Priors: Semantic regularization aligns codebook features with pretrained VGG19 perceptual features, conditioning texture restoration on image semantics.The regularization adds a term weighted by 𝛾=0.1, with a convolution matching feature dimensions.
  • Super-Resolution via Feature Matching: The LR encoder follows SwinIR with shallow and deep feature extraction, while its scaling is set so the decoder’s ×8 upscaling produces the target factor.The relation is S_up = S_down × 8.
  • Residual Shortcut Module: Multi-scale residual shortcuts bypass VQGAN quantization for direct gradient flow and complement feature-matching errors.Upsampled LR features are added as residuals to decoder features.
  • Training Objectives: Training combines feature-matching, reconstruction, and adversarial losses, with L1 and perceptual reconstruction terms and Gram-matrix texture supervision.The stated weights are 𝛼=𝜆_L1=𝜆_per=1, 𝛽=0.25, and 𝜆_adv=0.1.

4 IMPLEMENTATION DETAILS

The implementation trains on diverse HR image sources and their synthetically degraded LR counterparts, using fixed-size patches and specified optimization settings.

  • Training Dataset: The training set combines DIV2K, Flickr2K, DIV8K, and 10,000 FFHQ face images.The resulting dataset contains 136,205 HR patches of size 512 × 512, with LR images generated using the BSRGAN degradation model.
  • Training Dataset: Training crops non-overlapping 512 × 512 patches, filters patches with few textures, and randomly resizes aligned FFHQ faces before cropping.Random resizing uses scale factors between [0.5, 1.0] to avoid content bias.
  • Optimization Settings: Both HRP and SR networks use Adam with batch size 16 and 256 × 256 HR images for ×2 and ×4 upscaling.The generator and discriminator learning rates are 0.0001 and 0.0004, respectively; the codebook and decoder remain fixed during feature matching.

5 EXPERIMENTS

Experiments examine the pretrained HRP, benchmark performance, visual quality, and ablations of FeMaSR. The results show realistic texture restoration and identify residual shortcuts and semantic guidance as important design components.

  • Comparison with Existing Methods: Table 1 evaluates synthetic benchmarks using PSNR, SSIM, and LPIPS, with LPIPS emphasized as a perceptual texture-quality measure.The table uses mixed BSRGAN and Real-ESRGAN degradations; lower LPIPS is better.
  • Visualization of HRP: The pretrained codebook encodes basic texture elements individually, while combinations of codes produce semantic textures.The visualization decodes code vectors into 8 × 8 RGB patches to inspect these priors.
  • Comparison with Existing Methods: On real-world examples, FeMaSR removes degradations while generating feasible details, whereas competing GAN methods tend toward over-texturing or over-smoothing.Figure 5 reports comparisons at upscale factor 4.
  • Comparison with Existing Methods: FeMaSR outperforms competing methods in LPIPS on 5 of 6 synthetic benchmarks.The authors attribute qualitative improvements to semantic-aware HRP, while competing methods may remove textures, fail to generate feasible textures, or produce artifacts.
  • Ablation Study: Residual shortcuts accelerate feature-matching-loss optimization and mainly complement color and edge information when matching errors remain.Disabling the shortcut causes a performance drop and slower decrease of the feature matching loss.
  • Ablation Study: Semantic guidance makes VQGAN converge faster and perform better, producing a better HRP that improves restoration performance.The effect is evaluated with reconstruction loss and LPIPS in Stage I, alongside ablations in later restoration results.

6 CONCLUSION

The conclusion presents FeMaSR as a feature-matching approach for real-world blind SR using implicit HR priors. Experiments on synthetic and real-world benchmarks support its reported superiority and reduced artifacts relative to prior GAN-based approaches.

  • 6 CONCLUSION: FeMaSR matches LR features to a distortion-free HR feature codebook and decodes them with a pretrained VQGAN decoder.The HRP combines the codebook and associated decoder.
  • 6 CONCLUSION: Semantic information from pretrained VGG19 features improves the HRP, while multi-scale residual shortcuts facilitate optimization of feature-matching loss.The shortcuts connect LR feature space to the pretrained decoder.
  • 6 CONCLUSION: Quantitative and qualitative experiments on synthetic and real-world benchmarks demonstrate FeMaSR's reported superiority for real-world LR images.The conclusion states that fixed, distortion-free HRP enables more realistic results with fewer artifacts than previous GAN-based approaches.

A.1 Network Architectures of VQGAN

The VQGAN architecture uses a 1024 × 512 codebook with 256 × 256 inputs mapped to 32 × 32 feature maps. Experiments select 1024 codes as a balance between reconstruction performance and computation cost.

  • Network Architectures of VQGAN: The VQGAN uses codebook f_t^p with size 1024 × 512 for feature quantization.Convolution layers with k3n#s1 match feature channels to the codebook dimensions before and after quantization.
  • Network Architectures of VQGAN: A 256 × 256 input image is downsampled into 32 × 32 feature maps.The architecture details describe the encoder and decoder configuration around this representation.
  • Network Architectures of VQGAN: Increasing the number of codes generally improves reconstruction, but gains become marginal once the codebook is sufficiently large.The authors empirically select 1024 codes as a performance–computation compromise.

A.2 Details of Synthetic Dataset

Synthetic training data are built from natural-image and face datasets through texture filtering, content-bias mitigation, and online BSRGAN degradation. Testing uses fixed-seed degraded pairs.

  • Details of Synthetic Dataset: HR training patches are generated from DIV2K, Flickr2K, DIV8K, and 10,000 FFHQ face images at 512 × 512 resolution.Natural-image patches are cropped and filtered for texture content before training.
  • Details of Synthetic Dataset: Texture-poor natural-image patches are filtered using Sobel edge-map statistics, including the mean and variation of edge responses.Because edge maps are sparse, greater edge variation indicates more edges.
  • Details of Synthetic Dataset: For FFHQ, random resizing between 0.5 and 1.0 precedes single-patch cropping to reduce content bias from aligned faces.The source faces are 1024 × 1024 and well aligned.
  • Details of Synthetic Dataset: Corresponding LR images are generated online by applying the BSRGAN degradation model with scale factors 2 and 4.Testing pairs additionally use degradation_bsrgan_plus5 with a fixed random seed of 123.

B MORE RESULTS

The learned HR priors are visualized through codebook reconstructions and texture-conditioned code sampling, while the authors report a limitation on artificial textures.

  • Visualization of HRP: The HRP contains 1024 codes visualized individually, with semantic-related codes examined using the OST texture dataset.The analysis obtains pretrained VQGAN codes, computes texture-category distributions, and samples them for visualization.
  • Visualization of HRP: Codes sampled from different texture distributions generate textures resembling their corresponding semantics, supporting the effectiveness of the learned HRP.The sampled codes are randomly arranged into 8 × 8 latent features and decoded into 64 × 64 RGB texture patches.
  • Failure Cases: FeMaSR favors natural textures over artificial textures such as the straight lines dominating building images, often generating curved lines instead.The authors identify this as an empirical limitation and leave its solution to future work.

B.3 Qualitative Results

The qualitative-results material collects visual examples spanning training patches, HRP codebook visualizations, reconstructions, synthetic benchmarks, and real-world test images.

  • Qualitative Results: The supplementary figures include examples of cropped training HR patches and a visualization overview of all 1024 HRP codes.These figures document the training data examples and the learned codebook representation.
  • Qualitative Results: The qualitative material shows reconstruction examples from the OST dataset and code-index distributions across seven texture categories.Additional visualizations depict semantic-related textures encoded in HRP.
  • Qualitative Results: Additional figures report results on synthetic benchmarks and real-world test images.The cited material identifies these as further visual results rather than quantitative evaluations.
Loading 2202.13142v3…