PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Frequency Bias of Generative Models

Katja Schwarz, Yiyi Liao, Andreas Geiger

arXiv:2111.02447v1cs.CV

TL;DR

Generated images can be distinguished from real images by their high-frequency statistics, but it remains unclear whether generator or discriminator architectures are responsible for the mismatch. The paper independently tests both architectures and analyzes proposed measures in full GAN training. It finds distinct architectural biases, yet existing measures do not fully resolve spectral artifacts.

  • Problem

    State-of-the-art GANs can be distinguished from real images by high-frequency content, while the roles of generator and discriminator architectures in this mismatch remain disputed.

    Source: §1 · pp. 1–2

  • Method

    The authors build isolated testbeds for generator and discriminator architectures, then examine whether the findings and proposed measures extend to full GAN training.

    Source: §1 · pp. 2–3

  • Results

    Existing measures do not faithfully recover spectral statistics; spectral discriminators move closer to matching them, but generated images remain distinguishable by their spectra.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    The findings suggest discriminator design deserves more attention alongside generator architecture when seeking to match training-data distributions.

    Source: §1 · p. 3, §7 · p. 10

    The authors note that work on generative models carries risks of manipulation and misleading-content creation.

    Source: §7 · p. 10

Abstract

from arXiv · show

The key objective of Generative Adversarial Networks (GANs) is to generate new data with the same statistics as the provided training data. However, multiple recent works show that state-of-the-art architectures yet struggle to achieve this goal. In particular, they report an elevated amount of high frequencies in the spectral statistics which makes it straightforward to distinguish real and generated images. Explanations for this phenomenon are controversial: While most works attribute the artifacts to the generator, other works point to the discriminator. We take a sober look at those explanations and provide insights on what makes proposed measures against high-frequency artifacts effective. To achieve this, we first independently assess the architectures of both the generator and discriminator and investigate if they exhibit a frequency bias that makes learning the distribution of high-frequency content particularly problematic. Based on these experiments, we make the following four observations: 1) Different upsampling operations bias the generator towards different spectral properties. 2) Checkerboard artifacts introduced by upsampling cannot explain the spectral discrepancies alone as the generator is able to compensate for these artifacts. 3) The discriminator does not struggle with detecting high frequencies per se but rather struggles with frequencies of low magnitude. 4) The downsampling operations in the discriminator can impair the quality of the training signal it provides. In light of these findings, we analyze proposed measures against high-frequency artifacts in state-of-the-art GAN training but find that none of the existing approaches can fully resolve spectral artifacts yet. Our results suggest that there is great potential in improving the discriminator and that this could be key to match the distribution of the training data more closely.

1 Introduction

Generated images can be distinguished from real images by their high-frequency content, raising questions about whether generator or discriminator architectures impair learning those statistics. The paper tests both in isolation and finds distinct frequency biases, while existing measures still do not fully resolve spectral artifacts in full GAN training.

1 Introduction

  • Recent work reports that generated images are distinguishable by high-frequency content, exposing difficulty in learning the correct data distribution.The introduction frames this as a question about the expressivity of both generator and discriminator architectures.Source: §1 · p. 1
  • The authors develop isolated generator and discriminator testbeds before extending their findings to state-of-the-art GANs on large-scale datasets.The testbeds are intended to clarify which architectural factors make existing measures effective.Source: §1 · p. 2
  • Bilinear and nearest-neighbor upsampling bias generators toward little high-frequency content, while zero insertion can create checkerboard artifacts.When the loss is sensitive to those artifacts, learnable filters can compensate, so upsampling alone does not explain spectral discrepancies.Source: §1 · pp. 2–3
  • The discriminator can supervise high-frequency statistics when their magnitude is large enough, but downsampling artifacts can impair its training signal.The paper also finds that discriminator input in frequency-based domains can help, though the signal may remain problematic.Source: §1 · pp. 2–3
  • In full GAN training, spectral discriminators move closer to matching spectral statistics, but generated images remain straightforward to classify from their spectra.The authors argue discriminator design deserves as much attention as generator design.Source: §1 · p. 3

2 Preliminaries

This section introduces GAN training and the frequency-domain representation used to analyze images and detect generated samples.

2 Preliminaries

  • A GAN jointly trains a generator that maps latent variables to images and a discriminator that distinguishes synthesized images from real data.The models update in alternating steps using a non-saturating objective, often with R1 regularization.Source: §2 · p. 3
  • The power spectral density is estimated from squared Fourier-component magnitudes, and the reduced spectrum averages it azimuthally in normalized polar coordinates.The discrete 2D Fourier transform maps a grayscale image into the frequency domain.Source: §2 · p. 3
  • A spectral classifier fits a power law to the reduced-spectrum tail above rc = 0.75 and classifies images using the fit parameters.The tail threshold is specified as a fraction of the Nyquist frequency.Source: §2 · p. 3

3 Are high frequencies more difficult to generate?

The section isolates generator frequency bias using pixel-level reconstruction tests and finds that upsampling choices shape which spectral content is learned. A spectrum loss can let learnable filters compensate for checkerboard artifacts, so upsampling alone does not explain spectral discrepancies.

Experimental setting

  • To isolate the generator from the discriminator, the conditional reconstruction task paired 10 dataset images with 10 latent codes drawn from a normal distribution and optimized the generator using pixel-wise L2 loss.They used a channel-reduced PGAN generator and tested bilinear and nearest-neighbor interpolation, zero insertion, and reshaping in the channel dimension.Source: §3 · p. 4

Datasets and evaluation

  • The Toyset uses equal-magnitude Gaussian spectral peaks spanning low and high frequencies, while CelebA supplies natural images with exponentially decaying spectra.The reduced-spectrum error plots show normalized frequency radius against training iterations, with color encoding relative error clipped above 100%.Source: §3 · p. 4, Figure 2 · p. 5

Do generators exhibit a frequency bias?

  • Bilinear and nearest-neighbor upsampling learn low frequencies earlier, while zero insertion and reshaping learn both Toyset peaks at similar speeds but produce high-frequency checkerboard artifacts.Nearest-neighbor eventually fits the high-frequency peak; bilinear continues to struggle with it.Source: §3 · p. 4, Figure 2 · p. 5

Can the generator learn to compensate for the artifacts?

  • A logarithmic spectrum loss penalizes low-magnitude errors more strongly and can reduce the checkerboard peak from zero insertion or reshaping.Learnable filters can compensate for those artifacts, but bilinear upsampling benefits less; spatial PSNR changes little.Source: §3 · pp. 4–5, Figure 3 · p. 5

Discriminator comparison

  • Discriminator downsampling does not significantly favor a frequency range; the discriminator instead struggles with frequencies of low magnitude.On the Toyset, low- and high-frequency peaks are learned at approximately equal pace, while low-magnitude frequencies remain difficult to correct.Source: Figure 4 · p. 6
  • A spectral discriminator greatly improves spectral statistics on both datasets, whereas frequency-domain hard-example mining and wavelets alter them only slightly.The spectral discriminator directly guides the generator on dataset spectral statistics.Source: Figure 5 · p. 6

4 Can the discriminator provide a good training signal?

The discriminator can learn high-frequency statistics when their magnitude is large, but struggles with low-magnitude frequencies; downsampling can also degrade its training signal. Adding a spectral discriminator improves spectral alignment, though that can come at the cost of image fidelity.

Experimental setup

  • The conditional reconstruction test isolates discriminator supervision by optimizing learnable image tensors directly, rather than relying on a generator architecture.The experiment uses one image per class and evaluates reconstructions from discriminator gradients.Source: §4 · p. 6

Training-signal quality

  • All tested approaches struggle to correct low-magnitude frequencies, while natural-image spectral decay can make this look like a difficulty with high frequencies.On the Toyset, low- and high-frequency peaks are learned at approximately equal rates; downsampling operations also introduce artifacts in the discriminator's training signal.Source: §4 · pp. 6–7, Figure 6 · p. 7

Additional frequency-domain supervision

  • A spectral discriminator greatly reduces spectral error, but its reconstructions can look worse because reduced-spectrum matching discards image information.The spectral discriminator decreases image fidelity; F-mining and wavelets only slightly improve reconstruction.Source: §4 · p. 7, Figure 6 · p. 7

Implications

  • Downsampling can impair the discriminator's training signal, and none of the tested convolutional architectures with downsampling provides artifact-free supervision.The MLP reconstruction is reasonably good, whereas downsampling operations introduce artifacts.Source: §4 · p. 7

5 Improving GANs

In full GAN training, spectral discriminators improve spectral statistics more than wavelet training or frequency mining, but do not fully resolve discrepancies. Results on StyleGAN2 are mixed, with spectral improvement sometimes trading off against image quality.

Upsampling strategies

  • The spectral discriminator reduces the highest-frequency peak for zero insertion and reshaping, but elevated high-frequency magnitude remains and bilinear upsampling still mismatches dataset statistics.Figure 7 reports that the peak is reduced but spectral discrepancies are not fully corrected.Source: Figure 7 · p. 8, §5 · p. 8

PGAN with different discriminator inputs

  • The spectral discriminator is the most effective tested PGAN method for learning spectral statistics, while wavelet training and frequency mining improve them only slightly.Generated images remain classifiable by their spectral statistics, and spatial image quality remains largely unaffected by the spectral discriminator.Source: §5 · p. 8, Table 3 · p. 8

Upsampling strategies

  • Nearest-neighbor upsampling yields the best performance when spatial statistics and image fidelity are considered together.Zero insertion and reshaping remain more detectable than bilinear and nearest-neighbor outputs.Source: Table 4 · p. 8, §5 · p. 9

StyleGAN2

  • On StyleGAN2, spectral improvement trades off with image quality on AFHQ Dog, while FFHQ shows only slight spectral improvement and no corrected highest-frequency peak.The paper reports mixed results across LSUN Cats, AFHQ Dog, and FFHQ.Source: Table 5 · p. 8, §5 · p. 9

6 Other Related Work

Prior work links generated-image detectability to high-frequency content and examines spectral bias in neural networks. This paper calls its image-spectrum analysis frequency bias to distinguish it from the cited one-dimensional regression setup.

Deepfake Detection

  • Generated images can remain distinguishable from real images through high-frequency content despite high photorealism.The passage frames this as a systematic concern across state-of-the-art GAN architectures.Source: §6 · p. 9

Spectral Bias

  • Prior work studies spectral bias in fully connected ReLU networks, where low-frequency modes are fitted faster than high-frequency modes.The passage notes that convolutional-network spectral bias is less theoretically understood.Source: §6 · p. 9
  • The paper uses frequency bias for spectrum measured across all pixels of one image, distinguishing its analysis from one-dimensional regression over output samples.The comparison is to Rahaman et al.'s setup.Source: §6 · p. 9

7 Conclusion

The experiments find that upsampling bias alone does not explain spectral discrepancies, while advancing the discriminator moves GANs closer to the correct data distribution. Yet existing measures do not faithfully recover spectral statistics, and the authors call for more attention to discriminator design.

7 Conclusion

  • Upsampling introduces generator frequency bias, but this alone does not explain spectral discrepancies; discriminator advances help, yet existing measures do not faithfully recover spectral statistics.Generated images remain straightforward to classify using their spectra.Source: §7 · p. 9
  • The highest-frequency peak is significant only under the latter of the two spectral computations considered.The authors note that this difference was confirmed with Jung et al..Source: §7 · p. 9
  • The authors recommend exploring alternative discriminator downsampling and pixel-level supervision, while noting that generative models carry manipulation risks.They also identify discriminator augmentation for high-frequency artifacts as a possible direction.Source: §7 · p. 10

A Implementation

The implementation uses PGAN-derived testbeds for the generator and discriminator, then trains PGAN and finetunes StyleGAN2 for full-GAN experiments.

A.1 Generator

  • The generator follows a reduced-channel PGAN architecture and is trained with Adam at learning rate 0.0001, batch size 10, and equally weighted image- and reduced-spectrum L2 losses.The generator testbed findings persist across training-image counts: interpolation yields too little high-frequency content, while zero insertion and reshaping show checkerboard artifacts.Source: §A.1 · p. 12, Figure 8 · p. 13

A.2 Discriminator

  • The discriminator is a reduced-channel PGAN model; reconstruction experiments use 10 class-conditional samples and stabilize training with R1 regularization and tensor averaging.Across training-image counts, the discriminator has no significant frequency-range bias and generally struggles with low-magnitude frequencies.Source: §A.2 · pp. 12–13, Figure 9 · p. 13

A.3 GAN

  • Full-GAN experiments train PGAN on two V100 GPUs with batch size 256 and learning rate 0.002, and finetune StyleGAN2 using four V100 or GTX 1080 Ti GPUs.PGAN training omits adaptive discriminator augmentation and follows the protocol in.Source: §A.3 · p. 13

B Additional Analysis of the Generator Testbed

The generator testbed's findings persist across different numbers of training images. The analysis also explains why L2 loss gives relatively little weight to low-magnitude frequency errors.

B.1 Number of Images

  • The generator testbed's findings remain consistent as the number of training images varies.Figure 8 describes the compared upsampling behaviors across training-image counts.Source: §B.1 · p. 13

B.2 Low Magnitude Errors

  • L2 loss penalizes low-magnitude frequency errors less than high-magnitude errors because a low-magnitude error contributes only slightly to the frequency-domain sum.The analysis extends the reasoning from a 1D signal to grayscale images.Source: §B.2 · p. 14
  • The low-magnitude-error argument starts with a discrete 1D signal and applies Parseval's theorem to its discrete Fourier transform.The passages identify the frequency magnitude as |F[x]k|2 for the signal and Fourier representation.Source: §B.2 · p. 14

C Additional Analysis of the Discriminator Testbed

The discriminator testbed's findings persist across training-image counts. Additional spectral supervision improves PSNR with a full-Fourier MLP but not its convolutional counterpart; PGAN spectral comparisons show distinct effects across input domains.

C.1 Number of Images

  • The discriminator testbed findings remain consistent when the number of training images varies.Figure 9 reports that the discriminator has no significant frequency-range bias and generally struggles with low-magnitude frequencies.Source: §C.1 · p. 14

C.2 Penalizing the Spectrum

  • Adding the full-Fourier MLP discriminator significantly improves PSNR for every downsampling technique, whereas the convolutional full-spectrum discriminator does not improve PSNR over SD.The table reports PSNR for spectral discriminators on CelebA at resolution 642 pixels.Source: §C.2 · p. 14, Table 8 · p. 15

D.1 PGAN

  • For PGAN, F-mining changes the highest frequencies only slightly; wavelets reduce the highest-frequency peak but underpredict below 0.88fnyq, while SD matches that range more closely.These spectral observations hold across both datasets and support the classification-accuracy results in Table 3.Source: Figure 11 · p. 15, §D.1 · p. 15
  • The smaller PGAN reproduces characteristic ground-truth features despite not being fully photo-realistic; these experiments focus on spectral properties rather than image fidelity.Source: §D.1 · p. 15

D.2 SNGAN

For StyleGAN2, spectral-discriminator finetuning improves spectra substantially on AFHQ Dog but does not fully resolve the high-frequency peak on other datasets. In SNGAN, the spectral discriminator is the most effective tested approach for reducing spectral discrepancies, though some classifier results can be overly optimistic.

D.2 SNGAN

  • Among SNGAN discriminator variants, the spectral discriminator most effectively reduces spectral discrepancies, while its classification accuracies are similar to wavelets.The comparison also includes frequency-domain hard example mining and wavelet-space training.Source: §D.2 · pp. 15–16, Figure 13 · p. 16
  • The fitted-decay spectral classifier can yield overly optimistic near-chance (50%) accuracy, although reported accuracies align with qualitative reduced-spectrum results.FID is mixed across approaches and lower than the corresponding PGAN values, which the authors attribute to their reduced-channel PGAN having fewer parameters than the original SNGAN.Source: §D.2 · p. 16

D.3 StyleGAN2

  • On AFHQ Dog, spectral-discriminator finetuning largely improves spectra but worsens image fidelity; on FFHQ, FID stays similar while spectral improvement is slight.The remaining datasets retain an elevated amount of high frequencies.Source: Figure 14 · p. 16, §D.3 · p. 17, Figure 15 · p. 17

E Datasets

The appendix notes plans to release Toyset-generation scripts and identifies the datasets and license conditions for the study.

E Datasets

  • The authors plan to include scripts for generating Toyset in their code release upon acceptance.Source: E Datasets · p. 18
  • CelebA, FFHQ, LSUN Cats, and AFHQ are available for noncommercial research purposes under the stated licenses.The passage specifies Creative Commons BY-NC-SA 4.0 for FFHQ and Creative Commons BY-NC 4.0 for AFHQ.Source: E Datasets · p. 18
  • The datasets used are CelebA, FFHQ, LSUN Cats, and AFHQ.Source: E Datasets · p. 18

Related papers

Loading 2111.02447v1…