PaperlayerView exact revision on arXiv

Source-linked AI summary

Fourier Space Losses for Efficient Perceptual Image Super-Resolution

Dario Fuoli, Luc Van Gool, Radu Timofte

arXiv:2106.00783v1eess.IVcs.CV

TL;DR

Efficient perceptual super-resolution remains difficult because high-performing models often have impractical complexity and pixel-wise losses can yield blurry results. The paper introduces Fourier-domain supervision and adversarial losses, combines them with spatial losses, and trains a lightweight generator whose 41ms runtime is over 13× faster than ESRGAN and 48× faster than SRFlow on GPU while achieving comparable performance to larger methods.

  • Problem

    High-performing perceptual super-resolution models can be impractical because of large complexity, while pixel-wise supervision tends to produce blurry images.

  • Method

    The paper applies Fourier-domain supervision and GAN losses to a lightweight generator, then combines frequency-domain guidance with spatial supervision, spatial GAN, and perceptual losses.

  • Results

    The generator backbone runs in 41ms on GPU, over 13× faster than ESRGAN and 48× faster than SRFlow, while achieving comparable performance to much larger networks.

  • Takeaways & Limitations

    Combining spatial and frequency-domain losses provides complementary training information, enabling a lower-complexity generator to achieve comparable performance to larger networks.

Abstract

from arXiv · show

Many super-resolution (SR) models are optimized for high performance only and therefore lack efficiency due to large model complexity. As large models are often not practical in real-world applications, we investigate and propose novel loss functions, to enable SR with high perceptual quality from much more efficient models. The representative power for a given low-complexity generator network can only be fully leveraged by strong guidance towards the optimal set of parameters. We show that it is possible to improve the performance of a recently introduced efficient generator architecture solely with the application of our proposed loss functions. In particular, we use a Fourier space supervision loss for improved restoration of missing high-frequency (HF) content from the ground truth image and design a discriminator architecture working directly in the Fourier domain to better match the target HF distribution. We show that our losses' direct emphasis on the frequencies in Fourier-space significantly boosts the perceptual image quality, while at the same time retaining high restoration quality in comparison to previously proposed loss functions for this task. The performance is further improved by utilizing a combination of spatial and frequency domain losses, as both representations provide complementary information during training. On top of that, the trained generator achieves comparable results with and is 2.4x and 48x faster than state-of-the-art perceptual SR methods RankSRGAN and SRFlow respectively.

1. Introduction

Super-resolution must reconstruct high-frequency information lost during downsampling, but pixel-wise supervision often produces blurry images. This paper addresses the efficiency challenge by using Fourier-space losses to improve a lightweight generator’s perceptual quality and compete with larger networks.

  • Super-resolution challenge: Downsampling removes high-frequency information from high-resolution images, making its reconstruction the central task of super-resolution.The low-resolution image has a lower Nyquist frequency than the high-resolution target.
  • Super-resolution challenge: Pixel-wise supervision encourages conservative frequency estimates, which typically produces blurry images that appear lower quality than the high-resolution targets.
  • Efficiency challenge: Low-complexity networks are increasingly important for smartphone deployment because they enable faster inference and less expensive deployment.
  • Paper focus: The paper targets loss design rather than architecture or data, applying proposed losses to maximize the performance of an existing efficient generator.The authors identify architecture, loss function, and data as the three main ingredients of network performance and efficiency.
  • Proposed losses: Fourier-space supervision directly emphasizes frequency content for accurate reconstruction, while a Fourier-domain discriminator learns the high-frequency distribution adversarially.The Fourier transform also provides global guidance, complementing pixel-wise spatial evaluation.
  • Results: Combining Fourier and spatial supervision and GAN losses with a perceptual loss outperforms the other configurations in the ablation study and enables competition with much larger networks.

2. Related Work

Prior super-resolution work spans restoration-oriented methods that reconstruct missing details and perceptual methods that trade restoration fidelity for increased high-frequency generation. Related approaches also use wavelet-domain representations and sub-band weighting, whereas this paper fully decomposes images with the Fourier transform.

  • Restoration learning: Restoration-oriented super-resolution methods aim to reconstruct missing low-resolution image details as close to the ground truth as possible.The related work includes SRCNN, VDSR, and later convolutional approaches.
  • Perceptual super-resolution: Perceptual super-resolution methods improve visual quality by sacrificing restoration quality for increased high-frequency content generation.SRGAN uses adversarial learning to better model the high-frequency distribution and a VGG-based perceptual loss.
  • Frequency-domain approaches: Wavelet-based methods generate missing high-frequency coefficients, sometimes weighting sub-bands heuristically or using only selected sub-bands.
  • Frequency-domain approaches: Compared with these wavelet approaches, the paper applies the Fourier transform without fully limiting the representation to a selected set of sub-bands.

3. Proposed Method

The proposed method strengthens efficient perceptual SR by supervising reconstruction in Fourier space alongside spatial and adversarial domains. It targets missing high frequencies directly while using complementary spatial and frequency information to guide a low-complexity generator.

  • Fourier Transform and SR: SR reconstructs high-frequency content lost when downsampling an HR image to the LR domain.The LR sampling rate imposes a lower Nyquist frequency, limiting representation of higher frequencies.
  • Proposed Method: Combining spatial, Fourier supervision, Fourier GAN, and perceptual losses provides complementary training information and outperforms other tested configurations.The losses are designed to strengthen guidance for an efficient generator while retaining high perceptual quality.
  • Fourier Transform and SR: Fourier-space losses directly emphasize frequency components, including missing high-frequency content, rather than relying only on spatial-domain supervision.The Fourier transform separates frequencies and provides global guidance, whereas pixel-based losses provide local guidance and can produce blurry images.
  • Supervision Losses: The method applies FFT-based supervision to amplitude and phase differences between Hann-windowed generated and ground-truth images.The loss is calculated directly in Fourier space without heuristic frequency weighting; redundant components can be omitted because of Hermitian symmetry.
  • Supervision Losses: Low-frequency components are also included because resampling and imperfect anti-aliasing create mismatches beyond the missing high-frequency region.Applying the loss to these components improves low-frequency restoration in the generated image.
  • Supervision Losses: A fully connected Fourier-domain discriminator processes amplitude and phase components to provide adversarial guidance for frequency content.Convolutional architectures are considered suboptimal after Fourier transformation because spatial invariance no longer holds.

4. Experiments and Results

Experiments evaluate Fourier-space losses on DIV2K and Urban100 using restoration, perceptual, and distributional metrics. Ablations and comparisons show that Fourier losses improve lightweight generators, especially when combined with spatial losses, while substantially reducing inference cost versus larger perceptual SR methods.

  • Experimental setup: Experiments train 4× SR models on DF2K and evaluate them on the DIV2K validation set and Urban100.Training uses paired 64 × 64 LR and 256 × 256 HR crops.
  • Experimental setup: The evaluation reports PSNR and SSIM for restoration, LPIPS for perceptual quality, and FID for distributional similarity.The ablation metrics are calculated on the DIV2K validation dataset.
  • Ablation study: LPIPS improves by 0.081 and FID by 5.93 when Fourier-domain supervision is introduced, with a trade-off in restoration quality.This comparison is between ablation configurations 1 and 2.
  • Ablation study: Fourier losses replace ESRGAN’s spatial losses while preserving comparable perceptual quality and increasing PSNR by 1.12dB and SSIM by 0.045.The comparison is between configurations 4 and 5, with LF and L_F^GAN used in the Fourier-domain formulation.
  • Ablation study: Combining spatial and frequency-domain losses achieves the best LPIPS and FID results across configurations and improves all metrics over ESRGAN losses.Applied to an ESRGAN generator, the full combination yields PSNR +0.44dB and FID -0.57 versus the original ESRGAN formulation.
  • Comparison with state of the art: The efficient generator competes with larger methods while running 2.4× faster than RankSRGAN and 48× faster than SRFlow.It surpasses all three RankSRGAN models on PSNR/SSIM and achieves the highest FID; SRFlow remains 48 times slower on GPU.

5. Conclusion

The paper presents Fourier-domain supervision and GAN losses that strengthen training for perceptual SR, enabling a lower-complexity generator to achieve comparable image quality with substantially lower runtime.

  • 5. Conclusion: Two Fourier-domain losses strengthen the training signal for perceptual image SR and provide complementary information alongside spatial-domain losses.The losses directly emphasize missing high frequencies while spatial losses retain complementary local information.
  • 5. Conclusion: 41ms runtime makes the generator over 13× faster than ESRGAN and 48× faster than SRFlow on GPU.The reported runtime concerns the generator backbone.
  • 5. Conclusion: The improved guidance allows a significantly lower-complexity, faster network to achieve comparable performance to much larger networks.The conclusion frames reduced complexity and runtime as important for practical applications.
  • 5. Conclusion: The proposed losses help the SR network generate plausible high-frequency content while retaining restoration quality.The method separates retained low-frequency and missing high-frequency content in Fourier space.

Supplementary Material

The supplementary material evaluates generalization, perceptual-quality metrics, and the proposed Fourier-space GAN architecture through additional experiments and visual results.

  • Supplementary Material: Additional evaluations cover DIV2K, Urban100, and BSD100 to assess the approach’s generalization capabilities.The supplementary material also includes visual results and an ablation of the Fourier-space GAN architecture.
  • Supplementary Material: The evaluation reports PSNR, SSIM, LPIPS, FID, Ma, NIQE, and PI, while discussing limitations of no-reference metrics for perceptual-quality assessment.The no-reference metrics are calculated on DIV2K and used alongside standard restoration and perceptual metrics.
  • Supplementary Material: A supplementary ablation study examines the proposed Fourier-space GAN architecture.

A. Quantitative Evaluations

Across three datasets, the proposed losses improve both perceptual and restoration quality relative to previous approaches, with strong results on Urban100.

  • A. Quantitative Evaluations: Across 3 datasets, the proposed losses improve both perceptual quality and restoration quality compared with previous approaches.The passage describes these gains as clear benefits and evidence of generalization.
  • A. Quantitative Evaluations: −1.43 FID is the improvement achieved by ESRGAN (Our losses) over the version without the proposed losses on Urban100.
  • A. Quantitative Evaluations: On Urban100, the IMDN configuration with the proposed losses achieves results comparable to SRFlow despite a runtime difference of 41ms versus an unspecified SRFlow runtime in the supplied passage.

A.2. BSD100

On BSD100, the proposed losses achieve high restoration quality and strong perceptual performance, outperforming RankSRGAN models across the reported metrics except for one FID comparison.

  • A.2. BSD100: On BSD100, ESRGAN (Our losses) performs best in perceptual quality while also achieving high restoration quality.
  • A.2. BSD100: Table 5 reports the BSD100 evaluation, with red marking the best results and blue marking the second-best results.
  • A.2. BSD100: Ours (Full) outperforms all RankSRGAN models across all reported metrics on BSD100.
  • A.2. BSD100: Only RankSRGAN (NIQE) achieves a lower FID score than Ours (Full) on BSD100.

A.3. DIV2K - No-reference Metrics

The paper evaluates no-reference image-quality metrics on DIV2K while noting that these metrics lack ground-truth references and may favor methods explicitly trained for them. Among methods not trained for these metrics, the proposed losses perform best overall.

  • Metric choice: No-reference metrics quantify image quality without ground-truth comparison, limiting their objectivity for super-resolution assessment.The paper therefore evaluates distributional similarity with FID alongside LPIPS.
  • Metric choice: RankSRGAN models perform best on the no-reference metrics, but direct comparison is considered unfair because they are explicitly trained for those metrics.The paper also reports unexpected inconsistencies among RankSRGAN variants.
  • Results: Among methods not explicitly trained for these metrics, the proposed losses applied to ESRGAN and IMDN achieve the best overall results.Ours (Full) achieves the highest Ma and PI scores, while ESRGAN (Our losses) achieves the best NIQE score.

B. Visual Results

Visual examples across three datasets are assessed with PSNR and LPIPS, and the individual examples broadly align with the quantitative evaluation while showing image-level variation. The proposed losses improve restoration and perceptual quality across generator settings, including the efficient IMDN configuration.

  • Evaluation: Visual examples span three datasets and use PSNR for restoration quality and LPIPS for perceptual quality.The examples are intended to complement quantitative evaluation with visual inspection.
  • Evaluation: Individual image scores do not always match the overall quantitative evaluation because methods have different strengths and weaknesses across images.The paper deliberately includes such cases in the visual comparison.
  • Results: Applying the proposed losses generally improves restoration and perceptual quality for ESRGAN, ESRGAN (Our losses), and Ours (Full).The comparison is based on the visual examples and accompanying evaluation.
  • Results: IMDN with the proposed losses achieves comparable performance to ESRGAN and SRFlow while providing highly improved runtimes.Ours (Full) also generally improves perceptual and restoration quality compared with RankSRGAN.

C. Fourier GAN Architecture - Ablation

The Fourier GAN ablation compares full and reduced discriminator architectures and finds that discriminator complexity affects restoration and distributional metrics. The full architecture performs better on PSNR, SSIM, and FID, while the reduced model slightly improves LPIPS.

  • Architecture comparison: The ablation compares the full Fourier-space GAN discriminator with a reduced architecture having three layers.Both architectures are tested in configurations 5 and 8 from the ablation study.
  • Results: The higher-complexity discriminator consistently achieves better PSNR, SSIM, and FID scores in configurations 5 and 8.This comparison is reported for the two tested ablation configurations.
  • Results: LPIPS is slightly improved with the reduced three-layer discriminator.The paper notes this improvement alongside the higher-complexity model’s stronger PSNR, SSIM, and FID results.
  • Interpretation: The authors suspect the LPIPS–FID trade-off may arise from increased VGG-loss weight when the discriminator is weaker.This is presented as a hypothesis rather than a demonstrated mechanism.
  • Visual comparison: The visual examples cover DIV2K, Urban100, and BSD100 images for qualitative comparison.The supplied figure captions identify the datasets and example image numbers.
Loading 2106.00783v1…