Source-linked AI summary
Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions
Ricard Durall, Margret Keuper, Janis Keuper
TL;DR
Common up-sampling in generative CNNs distorts spectral distributions, limiting faithful reproduction of training data. The paper analyzes this effect and adds spectral regularization, showing that spectral features detect generated content and that regularization compensates spectral errors.
Problem
Current generative neural networks do not correctly approximate training-data spectral distributions, despite producing visually sound outputs.
Method
The paper analyzes generator up-sampling effects and adds a differentiable spectral regularization term to the generator loss.
Results
The spectral distortion is independent of generator architecture, while spectral features enable easy generated-content detection and regularization compensates spectral errors.
Takeaways & Limitations
Spectral analysis provides a practical indicator of generated content, and spectral regularization supports spectrally consistent generative networks.
Takeaways & Limitations
Comparisons with other detection methods may be biased because their real data matched but their fake data were generated independently with different GANs.
Abstract
from arXiv · showhide
Generative convolutional deep neural networks, e.g. popular GAN architectures, are relying on convolution based up-sampling methods to produce non-scalar outputs like images or video sequences. In this paper, we show that common up-sampling methods, i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly. This effect is independent of the underlying architecture and we show that it can be used to easily detect generated data like deepfakes with up to 100% accuracy on public benchmarks. To overcome this drawback of current generative models, we propose to add a novel spectral regularization term to the training optimization objective. We show that this approach not only allows to train spectral consistent GANs that are avoiding high frequency errors. Also, we show that a correct approximation of the frequency spectrum has positive effects on the training stability and output quality of generative networks.
1. Introduction
The paper shows that common up-sampling units cause generative CNNs to reproduce training-data spectral distributions incorrectly, independently of generator architecture. It proposes spectral regularization to compensate these distortions, enabling accurate detection and more stable, higher-quality GAN outputs.
- Current generative neural network architectures cannot correctly approximate the spectral distributions of training data.
- Up to 100% accuracy is achieved on public benchmarks by exploiting spectral distortions to detect generated images and videos.
- Common up-sampling units, known as up-convolutions, cause the observed spectral distortions.
- A novel spectral regularization term compensates for spectral distortions during generative-network training.
- Spectral regularization produces more stable GAN models and improves visual output quality.
2. The Spectral Effects of Up-Convolutions
This section defines a one-dimensional Fourier power-spectrum representation for images and analyzes how common up-convolution methods alter frequency content. Transposed convolution creates high-frequency replicas, whereas interpolation-based up-convolution suppresses high frequencies.
- Spectral representation: The azimuthal integral converts an image’s two-dimensional Fourier power spectrum into a one-dimensional spectrum by radially integrating each frequency component.The processing assumes square images.
- Up-scaling methods: GANs and related generative architectures require up-scaling mechanisms to transform low-dimensional latent representations into high-dimensional outputs such as images.The section identifies interpolation followed by convolution and transposed convolution as the two most common techniques.
- Theoretical analysis: Increasing spatial resolution by a factor of 2 with bed-of-nails upsampling scales the frequency axes by a factor of 1/2.Bed-of-nails interpolation corresponds to transposed convolution in the analyzed formulation.
- Theoretical analysis: Bilinear interpolation averages adjacent values and suppresses artificial high frequencies, but its resulting spectrum is expected to be overly low in the high-frequency domain.This explains the missing high frequencies observed for interpolation-based up-convolution.
- Theoretical analysis: Bed-of-nails upsampling creates high-frequency replicas, making all observed spatial frequencies beyond N/2 potential upsampling artifacts.Removing these replicas requires appropriate smoothing, which commonly used 3 × 3 filters cannot achieve for medium- to high-resolution images.
3. Learning to Generate Correct Spectral Distributions
The paper proposes adding a differentiable spectral-loss term to generator training to correct up-convolution-induced spectral distortions. Experiments show that correction depends on decoder architecture, requiring more and larger learned filters rather than a single small convolution.
- Spectral regularization: The authors propose adding an additional spectral term to the generator loss because image-space losses cannot directly capture or correct spectral distortions.The spectral term is weighted by the hyper-parameter λ.
- Spectral regularization: They define the spectral loss as binary cross entropy between generated azimuthal integration AIout and the mean AIreal from real samples.Azimuthal integration is differentiable, enabling this loss formulation.
- Spectral regularization: Azimuthal-integral values are normalized by the 0th coefficient AI0 to scale them to [0, 1].M denotes the image size.
- Architecture dependence: A single learned 3 × 3 filter cannot correct the observed spectral effects, even for large λ, motivating reconsideration of the architecture parameters.This result follows the paper’s theoretical analysis of up-convolution-induced distortions.
- Architecture dependence: Spectral distortions remain after one or three 3 × 3 convolutions but can be corrected when more, larger filters are learned.The experiment varies decoder filters from 3 × 3 to 11 × 11 and uses either 1 or 3 convolutional layers.
4. Experimental Evaluation
Experiments on public face datasets show that up-convolution artifacts create detectable spectral distortions in generated images. Spectral regularization compensates these distortions and improves GAN training stability and convergence.
- Fake detection: The detection pipeline extracts DFT-based spectral features with azimuthal integration and classifies them using supervised SVM or unsupervised K-Means.Input images are converted to grayscale before DFT, and normalized power spectra are scaled to a fixed feature size.
- Datasets: Faces-HQ 2 provides an annotated high-resolution dataset of 40k publicly available 1024 × 1024 × 3 facial images for evaluation.The collection combines images from CelebA-HQ, Flickr-Faces-HQ, the 100K Faces project, and thispersondoesnotexist.com.
- Fake detection: Generated faces exhibit spectral distortions from up-sampling units, enabling easy fake detection that can outperform complex DNN-based detectors.Real and fake faces form well-delineated clusters in the high-frequency spectral feature space.
- Spectral regularization: Adding spectral loss and three post-upsampling 5 × 5 convolutional filter layers allows GANs to produce images that mimic the real data’s frequency-domain behavior.The spectral loss uses 1000 unannotated real samples to estimate AIreal.
- Training stability: Spectral regularization yields stable, low FID during training, whereas unregularized GANs tend to collapse and exhibit unstable convergence.FID is evaluated using features extracted by an ImageNet-pretrained Inception-v3 network.
5. Discussion and Conclusion
The paper concludes that state-of-the-art convolutional generative networks fail to approximate real data’s spectral distributions. This failure enables generated-sample identification and raises fundamental concerns about training-data generation and transfer-learning approaches.
- State-of-the-art convolutional generative networks, including popular GAN image generators, fail to approximate the spectral distributions of real data.
- This spectral mismatch can identify generated samples and implies that current training-data generation and transfer-learning approaches are fundamentally flawed.
6. Using Spectral Distortions to Detect Deepfakes
This section provides more detailed results from the experiments presented in Section 4.1.
- The section reports additional detail on the experiments presented in Section 4.1.
6.1. More Details on the used Datasets
The section describes Faces-HQ5, a 40K-image high-quality face dataset assembled from established sources because no public high-resolution dataset provided annotated real and fake faces. It also details CelebA-based medium-resolution experiments and the DeepFakeDetection subset of FaceForensics++.
- Faces-HQ5: Faces-HQ5 contains 40K high-quality images collected and labeled from CelebA-HQ, Flickr-Faces-HQ, 100K Faces, and thispersondoesnotexist.com.The dataset was created because no public dataset provided high-resolution images with annotated fake and real faces.
- CelebA: CelebA provides 202,599 celebrity face images with 40 facial-attribute variations at 178x218x3 resolution.The images are treated as medium-resolution in this context.
- Faces-HQ5: Faces-HQ5 has a size of 19GB.
- CelebA: 162,770 CelebA images were used for training and 39,829 for testing after cropping and resizing from 178x218x3 to 128x128x3.DCGAN, DRAGAN, LSGAN, and WGAN-GP were trained on the real dataset to generate fake examples.
- FaceForensics++: The DeepFakeDetection subset contains 363 original sequences from 28 paid actors across 16 scenes and over 3000 manipulated videos with corresponding binary masks.The videos contain trackable, mostly frontal, non-occluded faces that enable automated tampering methods.
6.2. Experimental Results
The experiments show that spectral features consistently separate real and fake samples across Faces-HQ, FaceForensics++, and CelebA, with clustering supporting this separability. The approach also achieves 100% supervised classification accuracy using as few as 16 annotated training samples.
- Spectral separability: Spectral distributions across Faces-HQ, FaceForensics++, and CelebA indicate that classifiers should separate real and fake samples.Faces-HQ and FaceForensics++ fakes are consistent with up+conv up-convolutions or successive high-frequency blurring, whereas CelebA fakes used transconv.
- Spectral separability: AI-feature clustering shows clear separation between the real and fake classes.The clustering visualization uses 1D power-spectrum features from a random Faces-HQ subset, with perplexity 4 and 4000 iterations.
- Deepfake detection: 100% classification accuracy is achieved in a supervised setting with as little as 16 annotated training samples.The approach also works well in an unsupervised setting, while Tables 3–5 examine performance under different training-sample settings across three datasets.
7. Spectral Regularization on Auto-Encoder
On auto-encoders, spectral regularization improves convergence and generated-image quality while bringing frequency spectra closer to the real 1D Power Spectrum distribution. A topology search concludes that three 5x5 convolutional layers after the last up-convolution are sufficient for using the regularization.
- 7. Spectral Regularization on Auto-Encoder: Spectral regularization improves auto-encoder convergence and generated-image quality measured by MSE.Figure 20 evaluates MSE reconstruction loss and BCE spectral loss with and without regularization.
- 7. Spectral Regularization on Auto-Encoder: Both transconv and up+conv show distinct frequency-spectrum behavior, especially in high-frequency components, whereas regularization brings results closer to the real 1D Power Spectrum distribution.The corrected model generates images closer to the real distribution.
- 7. Spectral Regularization on Auto-Encoder: The topology comparison evaluates DCGAN variants that place one, two, or three 32-filter 5x5 convolutional layers at different positions after up-convolutions.The variants include DCGAN v1 through v4, while Real denotes original CelebA face images.
- 7. Spectral Regularization on Auto-Encoder: Three 5x5 convolutional layers after the last up-convolution are sufficient to use spectral regularization.This conclusion follows theoretical analysis and a rough topology search for verification.