Focal Frequency Loss for Image Reconstruction and Synthesis

Liming Jiang; Bo Dai; Wayne Wu; Chen Change Loy

Source-linked AI summary

Focal Frequency Loss for Image Reconstruction and Synthesis

Liming Jiang, Bo Dai, Wayne Wu, Chen Change Loy

arXiv:2012.12821v3Dec 23, 2020cs.CVcs.LGeess.IV

TL;DR

Image reconstruction and synthesis can retain gaps between real and generated images in the frequency domain, with limited evidence that spatial losses address hard frequencies. The paper introduces focal frequency loss, which adaptively emphasizes hard frequency components using amplitude- and phase-aware frequency distances. Across reconstruction and synthesis baselines, it improves perceptual and quantitative quality, complements spatial losses, and shows potential for StyleGAN2.

Problem
Real and generated images can differ in the frequency domain, while spatial losses hardly help models synthesize hard frequencies.
Method
Focal frequency loss maps spectrum values to two-dimensional amplitude- and phase-aware vectors and dynamically down-weights easy frequencies.
Results
Focal frequency loss improves representative reconstruction and synthesis baselines across perceptual and quantitative evaluations and shows potential for StyleGAN2.
Takeaways & Limitations
The loss is complementary to existing spatial losses and can improve image reconstruction and synthesis quality across varied models, structures, and tasks.
Takeaways & Limitations
The frequency-representation formulas are presented for gray-scale images, with color images handled by processing each channel separately.

Abstract

from arXiv · show

Image reconstruction and synthesis have witnessed remarkable progress thanks to the development of generative models. Nonetheless, gaps could still exist between the real and generated images, especially in the frequency domain. In this study, we show that narrowing gaps in the frequency domain can ameliorate image reconstruction and synthesis quality further. We propose a novel focal frequency loss, which allows a model to adaptively focus on frequency components that are hard to synthesize by down-weighting the easy ones. This objective function is complementary to existing spatial losses, offering great impedance against the loss of important frequency information due to the inherent bias of neural networks. We demonstrate the versatility and effectiveness of focal frequency loss to improve popular models, such as VAE, pix2pix, and SPADE, in both perceptual quality and quantitative performance. We further show its potential on StyleGAN2.

1. Introduction

Generative models can leave real–generated image gaps in the frequency domain, reflecting artifacts, missing details, and spectral shifts. The paper proposes focal frequency loss to narrow these gaps by emphasizing hard-to-synthesize frequencies alongside spatial losses.

Motivation: Frequency-domain gaps commonly occur between real and generated images, appearing as spatial artifacts or detectable spectral differences.Examples include checkerboard artifacts, periodic spectral patterns, and frequency-region shifts.
Motivation: Spectral bias favors low-frequency functions, causing models to neglect hard frequencies and lose important frequency information during synthesis.The network’s frequency-fitting priority typically progresses from low to high frequencies.
Approach: Spatial pixel losses provide limited help for hard frequencies because every pixel has the same significance for a given frequency.The paper therefore transforms real and generated images into frequency representations using the discrete Fourier transform.
Approach: Focal frequency loss adaptively down-weights easy frequencies and emphasizes hard ones through a dynamic spectrum weight matrix.It represents each spectrum coordinate with a two-dimensional vector containing amplitude and phase information, then uses scaled Euclidean distances.
Contribution: Experiments show focal frequency loss improves representative reconstruction and synthesis baselines and complements existing spatial losses, including perceptual loss.The study also reports potential improvements for StyleGAN2.

2. Related Work

Prior image reconstruction and synthesis methods largely optimize spatial-domain losses, while frequency-domain optimization has received less attention. The paper distinguishes focal frequency loss by dynamically emphasizing hard frequencies directly in the frequency domain.

Image reconstruction and synthesis: Autoencoders and GANs are widely used for image reconstruction, generation, image-to-image translation, style transfer, and semantic image synthesis.These model families support applications including face manipulation and face generation.
Image reconstruction and synthesis: Existing approaches usually improve image quality with spatial losses such as perceptual loss, while seldom optimizing directly in the frequency domain.Spectral regularization is described as a preliminary frequency-domain attempt.
Image reconstruction and synthesis: Focal frequency loss differs by dynamically down-weighting easy frequencies and focusing models on hard frequencies through direct frequency-domain optimization.The paper presents it as distinct from perceptual loss and spectral regularization.
Frequency domain analysis of neural networks: Frequency-domain analysis studies neural-network frequency behavior, including spectral bias and changing frequency-fitting priorities during training.These studies motivate examining frequency gaps in generated images.

3. Focal Frequency Loss

The method represents images in the frequency domain, measures differences using both amplitude and phase, and dynamically emphasizes hard-to-synthesize frequencies. Frequency-domain analysis links missing spectral regions to distinct spatial artifacts, motivating focal frequency loss.

Frequency Representation of Images: 2D DFT converts an image into frequency components, enabling spatial frequencies to be profiled and hard frequencies to be located.Each frequency value depends on all image pixels, while spectrum regions correspond to different sinusoidal components.
Frequency Representation of Images: Missing low, high, or selected frequency regions produce distinct artifacts, including blur, ringing, and checkerboard patterns.Bandlimiting operations simulate the effects of missing frequencies in the spatial domain.
Frequency Distance: A faithful reconstruction requires both amplitude and phase information because amplitude-only reconstruction yields irregular colors, whereas phase-only reconstruction resembles noise.The frequency distance therefore must capture both components.
Frequency Distance: The frequency distance maps each real and fake complex frequency value to a two-dimensional Euclidean vector and measures their distance.The vector representation preserves magnitude and angle information from the complex frequency values.
Dynamic Spectrum Weighting: The dynamic spectrum weighting scheme down-weights easy frequencies and emphasizes frequencies with larger current losses during training.Weights are normalized to [0, 1], with weight 1 assigned to the currently most lost frequency; gradients through the weights are locked.
Dynamic Spectrum Weighting: Focal frequency loss is a dynamically updated weighted average of frequency distances that progressively refines generated images by focusing on hard frequencies.The weighting is applied through a Hadamard product with the frequency distance matrix.

4. Experiments

Experiments across reconstruction, unconditional synthesis, image-to-image translation, and semantic synthesis show that focal frequency loss improves visual quality and quantitative performance across diverse generative models and datasets.

4.1. Settings: FFL is evaluated with vanilla AE, VAE, pix2pix, SPADE, and StyleGAN2 across seven datasets and multiple image-generation tasks.Metrics include PSNR, SSIM, LPIPS, FID, LFD, IS, mIoU, and pixel accuracy, depending on the task.
4.2. Results and Analysis: Vanilla AE reconstructions become clearer and more textured with FFL, while CelebA results show reduced face blur, identity shift, and expression loss.Quantitative gains occur in most cases on DTD and CelebA, with a larger boost on CelebA.
4.2. Results and Analysis: VAE with FFL retains more reconstruction detail and improves unconditional synthesis quality on CelebA and CelebA-HQ.On VAE reconstruction, FFL improves all reported metrics; unconditional synthesis improves in both FID and IS.
4.2. Results and Analysis: FFL improves pix2pix translation on CMP Facades and edges → shoes by reducing unnatural colors, checkerboard artifacts, and other visual defects.The method also improves semantic alignment with the input mask on CMP Facades.
4.2. Results and Analysis: FFL improves SPADE outputs by restoring details in Cityscapes street scenes and ADE20K outdoor and indoor scenes, while outperforming benchmark methods without changing SPADE’s structure.The comparison uses semantic-image-synthesis metrics reported in Table 5 and includes task-specific baselines.
4.2. Results and Analysis: StyleGAN2 with FFL reduces small artifacts and visibly mitigates frequency-domain gaps between real and generated mini-batch average spectra.The experiment uses CelebA-HQ images at 256 × 256 resolution without truncation.
4.2. Results and Analysis: FFL outperforms PL and SpReg across different reconstruction and synthesis tasks, while remaining complementary to PL.Replacing DFT with DCT still outperforms previous methods, although it is slightly worse than the DFT version.
4.2. Results and Analysis: Ablations show that full FFL performs best, and removing frequency representation, amplitude, phase, or dynamic spectrum weighting degrades visual or quantitative results.Amplitude-only reconstructions retain contours but shift colors, whereas phase-only reconstructions lose facial identity information.

5. Conclusion

The proposed focal frequency loss improves image reconstruction and synthesis by adaptively targeting difficult frequency components alongside spatial losses. It outperforms relevant approaches across diverse settings and shows potential for StyleGAN2.

Focal frequency loss adaptively emphasizes frequency components that are difficult to handle, directly optimizing reconstruction and synthesis in the frequency domain.It is designed to ameliorate image quality by focusing learning on harder frequency components.
The loss complements existing spatial losses across baselines differing in categories, network structures, and tasks, outperforming relevant approaches.This supports its use as an addition to established spatial objectives rather than a replacement for them.
Focal frequency loss improves synthesis results for StyleGAN2, demonstrating potential beyond the reconstruction and synthesis methods evaluated earlier.The appendix also reports additional illustrations, implementation details, dataset descriptions, variant studies, and analyses.

A. Additional Illustrations of Methodology

The illustrations represent images as combinations of spatial frequencies and embed the frequency dimension alongside pixel dimensions. They explain why frequency-domain optimization complements spatial losses for addressing hard frequencies.

Frequency representation: A 2D Fourier transform decomposes an image into orthogonal sine and cosine functions indexed by frequency-spectrum coordinates.The coordinates determine angular frequency and the direction of each spatial-frequency component.
Intuitive geometry: An image can be visualized as a cube whose length and width are pixel dimensions, while height represents the frequency domain.A pixel is an orange prism, and a specific frequency is a green plane in this illustration.
Hard frequencies: Each frequency depends on all image pixels, so optimizing individual pixels in the spatial domain does not readily locate hard frequencies.Neural-network bias can cause models to avoid frequencies that are difficult to synthesize.
Frequency-domain optimization: Directly and adaptively optimizing frequencies can improve reconstruction of frequency components and indirectly improve pixel alignment and quality.The passage links these improvements to pixel-based metrics such as PSNR and SSIM.
Complementarity: Spatial and frequency losses consider different image dimensions and are therefore complementary rather than replaceable.FFL is intended to complement existing spatial losses.
Scope of illustration: The cube visualization is a simplified illustration because real image-frequency components may require a higher-dimensional representation.

B. Implementation Details

The experiments use varied autoencoder and GAN baselines, datasets, optimizers, losses, and training configurations. Adding FFL incurs negligible computational overhead in the reported pix2pix test.

Baselines and configurations: The experiments cover five representative autoencoder- and GAN-based methods, including different network structures and FFL comparisons.The implementation studies MLP and CNN structures and compares FFL with perceptual loss and spectral regularization.
Experimental setup: The implementation details cover baselines, network structures, optimization, initialization, spatial losses, and training schedules.
VAE settings: VAE models use Adam with β1 = 0.9, β2 = 0.999, learning rate 0.001, batch size 128, and task-specific training epochs.Training lasts 20 epochs on CelebA and 400 epochs on CelebA-HQ.
StyleGAN2 settings: StyleGAN2 models use Leaky ReLU, standard StyleGAN techniques, Adam with β1 = 0 and β2 = 0.99, and eight V100 GPUs.The spatial loss is non-saturating logistic loss with R1 regularization.
Computational cost: 0.067 seconds per iteration is the average pix2pix training time with FFL, compared with 0.064 seconds without it.Memory consumption increases from 3513 to 3515 MB on one NVIDIA Tesla V100 GPU.

C. Dataset Details

The study evaluates reconstruction and synthesis across seven datasets spanning textures, faces, facades, shoes, and semantic street or in-the-wild scenes, with varied resolutions and dataset sizes.

Texture reconstruction: DTD supports vanilla autoencoder reconstruction of textural images with special frequency patterns at 64 × 64 resolution.The dataset uses 4,512 training images and 1,128 test images.
Face reconstruction and synthesis: CelebA supports vanilla AE and VAE reconstruction plus unconditional VAE synthesis using cropped, aligned 64 × 64 faces.It contains 199,599 training images and 3,000 test images.
High-quality face synthesis: CelebA-HQ supports VAE reconstruction and VAE or StyleGAN2 unconditional synthesis at 256 × 256, with additional StyleGAN2 experiments at 1024 × 1024.The dataset contains 27,000 training images and 3,000 evaluation images.
Image-to-image translation: CMP Facades provides architectural labels and photos for pix2pix mask →image translation at 256 × 256 resolution.The training and test sets contain 400 and 106 images, respectively.
Image-to-image translation: Edges →shoes supports pix2pix translation from edge maps to centered shoe images at 256 × 256 resolution.It contains 49,825 training images and 200 test images.
Semantic image synthesis: Cityscapes and ADE20K support SPADE semantic image synthesis with dense annotations across 30 and 150 semantic classes.Cityscapes images are scaled to 512 × 256, while ADE20K images are resized to 256 × 256.

D. Variant Studies

The variant studies examine how spectrum focusing and patch granularity affect FFL, finding α = 1 a practical trade-off and very small patches detrimental. FFL also improves spatial reconstruction loss.

Variant comparison: The variant studies report similar visual results across alternative FFL forms.The study evaluates variants on vanilla AE reconstruction using quantitative metrics.
Spectrum-weight variants: α controls spectrum-weight focusing: larger values emphasize hard frequencies more strongly by increasing weight differences between easy and hard frequencies.The main version uses α = 1.
Spectrum-weight variants: The α = 1 main version outperforms the no-FFL baseline on all five metrics, whereas α = 2 degrades results, especially FID.The α = 2 model remains better than baseline on most metrics.
Patch-based variants: Patch factors p = 1, 2, and 4 achieve similar performance and substantially outperform the baseline, while p = 8 degrades quantitative results, especially FID.The authors therefore use p = 1 as the main version, while noting possible gains from patch-based variants in some cases.
Training behavior: With identical weight and random seed, adding FFL makes the vanilla AE spatial MSE converge to a lower point.This supports complementarity between FFL and spatial losses in the reported reconstruction experiment.

E.1. Training Loss

FFL narrows frequency-domain gaps during reconstruction and improves VAE image quality, producing clearer outputs and spectra closer to real images.

Training-loss interpretation: The analysis is framed around FFL complementing spatial losses while narrowing frequency-domain gaps between real and generated images.The training-loss comparison uses vanilla AE reconstruction on CelebA.
Spatial reconstruction: Without FFL, reconstructed VAE faces are blurry; with FFL, they become clearer and closer to ground-truth images.The passage reports improved perceptual quality after applying FFL.
Frequency-domain analysis: FFL narrows the frequency-domain gap in VAE reconstruction on CelebA.The effect is illustrated in Figure 15.
Frequency-domain analysis: The VAE baseline concentrates on a limited spectrum region and loses high-frequency information, whereas FFL makes the spectrum distribution closer to ground truth.FFL also generates some essential special spectrum patterns.
Additional ablations: The section extends component ablations to higher-resolution pix2pix image-to-image translation with GANs.The stated goal is to study FFL components while reducing the influence of adversarial loss in the main AE ablation.

E.4. Results on Non-Photorealistic Images

On non-photorealistic Danbooru2019 anime portraits, FFL improves all reported metrics, consistent with its adaptive treatment of difficult frequency patterns.

Anime portraits: FFL boosts all metrics for vanilla AE reconstruction on Danbooru2019 Portraits.The comparison is reported in Table 11 for anime images.
Interpretation: The authors attribute this potential to special frequency patterns in non-photorealistic images that may be hard for networks to learn.FFL is described as adaptive for dealing with these frequencies.

E.5. Higher-Resolution Results on StyleGAN2

On 1024 × 1024 CelebA-HQ synthesis, StyleGAN2 with FFL produces high-fidelity images and improves FID over the original model, including without truncation.

Visual quality: FFL reduces tiny face and eye artifacts and restores teeth details missing in some original StyleGAN2 outputs.The FFL results are described as very photorealistic.
Sampling settings: The StyleGAN2 experiments evaluate both untruncated samples and samples using truncation ψ = 0.5.Additional random samples are provided for both settings.
Quantitative comparison: 3.374 FID with FFL outperforms 3.733 without FFL for StyleGAN2 on CelebA-HQ.The comparison uses 1024 × 1024 images without truncation.