PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency Domain Image Translation: More Photo-realistic, Better Identity-preserving

Mu Cai, Hong Zhang, Huijuan Huang, Qichuan Geng, Yixuan Li, Gao Huang

arXiv:2011.13611v3cs.CV

TL;DR

Image translation methods can over-adapt to the reference and lose source identity and structure. FDIT decomposes images into frequency components and preserves frequency information in pixel and Fourier spaces; it reports state-of-the-art translation performance and lower FID than SwapAE.

  • Problem

    Existing image translation models can lose source-image structural characteristics, while methods lack an explicit mechanism for preserving source identity.

    Source: §1 · p. 1, §5 · p. 8

  • Method

    FDIT uses high-frequency components as cues for object structure akin to identity and preserves frequency information in both pixel and Fourier spectral spaces.

    Source: §6 · p. 8

  • Results

    FDIT establishes state-of-the-art image-translation performance, with FID 5.6% lower than vanilla SwapAE, and preserves source identity while producing photo-realistic hybrids across five large-scale datasets and multiple tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support frequency-based information as a way to preserve source identity while producing photo-realistic image hybrids.

    Source: §6 · p. 9

Abstract

from arXiv · show

Image-to-image translation has been revolutionized with GAN-based methods. However, existing methods lack the ability to preserve the identity of the source domain. As a result, synthesized images can often over-adapt to the reference domain, losing important structural characteristics and suffering from suboptimal visual quality. To solve these challenges, we propose a novel frequency domain image translation (FDIT) framework, exploiting frequency information for enhancing the image generation process. Our key idea is to decompose the image into low-frequency and high-frequency components, where the high-frequency feature captures object structure akin to the identity. Our training objective facilitates the preservation of frequency information in both pixel space and Fourier spectral space. We broadly evaluate FDIT across five large-scale datasets and multiple tasks including image translation and GAN inversion. Extensive experiments and ablations show that FDIT effectively preserves the identity of the source image, and produces photo-realistic images. FDIT establishes state-of-the-art performance, reducing the average FID score by 5.6% compared to the previous best method.

1. Introduction

Existing translation methods lack explicit identity preservation and may over-adapt to references, losing source structure and fine detail. FDIT decomposes images into frequency bands and regulates frequency consistency during reconstruction and translation.

1. Introduction

  • Existing methods lack an explicit identity-preservation mechanism, so translations can over-adapt to the reference, lose source identity, and omit fine-grained details.The paper notes that losing fine detail can hinder photo-realistic high-resolution generation.Source: §1 · pp. 1–2
  • FDIT decomposes images into low- and high-frequency components, treating high-frequency edges and object details as cues related to identity.The low-frequency component captures color and illumination; Figure 2 visualizes grayscale components produced with Gaussian kernels of different sizes.Source: §1 · p. 2, Figure 2 · p. 2
  • FDIT regulates frequency consistency in both pixel space and Fourier spectral space during training.The pixel-space loss matches high-frequency components between source and generated images, while FFT-based supervision matches their high-frequency spectra.Source: §1 · p. 2
  • 5.6% lower FID than vanilla Swapping Autoencoder (SwapAE), with FDIT outperforming competitive baselines across all considered datasets.The paper also reports state-of-the-art performance on image translation tasks.Source: §1 · p. 2
  • Evaluations span five large-scale datasets and image translation and GAN-inversion tasks; user studies report 75.40% and 64.39% average preference for FDIT over SwapAE on identity preservation and image quality, respectively.The authors also report ablations of loss terms and frequency-supervision modules.Source: §1 · p. 2

2. Background: Image-to-image Translation

Image-to-image translation generates an image from a source and a reference image. Prior methods encode content and style separately, then combine source content with reference style, but source identity can be lost.

2. Background: Image-to-image Translation

  • Image-to-image translation generates a synthesized image from a source image and an accompanying reference image.The described setup uses an encoder to map images to latent representations and a generator to produce images.Source: §2 · p. 2
  • Prior approaches divide latent representations into content and style codes, combining source content with reference style to generate a translation.The background describes reconstruction as minimizing the L1 distance between the input and its generated reconstruction.Source: §2 · pp. 2–3
  • The source content code may fail to capture identity, allowing identity-related characteristics to be lost during translation.Figure 3 presents FDIT as regulating frequency consistency during both reconstruction and translation.Source: §2 · p. 3, Figure 3 · p. 3

3. Frequency Domain Image Translation

FDIT preserves frequency information during image translation by imposing constraints in both pixel space and Fourier spectral space. Its design pairs local Gaussian filtering with global FFT analysis.

3.1. Pixel Space Loss

  • Pixel-space losses decompose images with Gaussian filtering and match source high-frequency components in translated outputs, preserving identity-linked edges.The reconstruction loss also matches low- and high-frequency components between the input and generator output.Source: §3.1 · pp. 3–4

3.2. Fourier Frequency Space Loss

  • The method uses FFT to map images from pixel space into Fourier spectral space, regulating reconstruction in frequency space and constraining generated images’ high-frequency components for better identity preservation.For 256×256 images, the high-frequency mask complements a low-frequency circle of radius 21; that low-frequency region contains 97.8% of total spectral energy.Source: §3.2 · p. 4

3.3. Overall Loss

  • The overall objective adds frequency-based losses to the original image-translation loss, using λ1 = λ2 = λ3 = λ4 = 1.Source: §3.3 · p. 4
  • Gaussian filtering captures frequency information locally, while FFT uses all pixels to characterize the global frequency distribution; experiments report both effective for identity preservation.For 256×256 images, the selected Gaussian kernel size is k = 21, and FDIT is not sensitive to k within a mild range.Source: §3.3 · pp. 4–5

4. Experiments

Across image translation and GAN inversion, FDIT improves identity preservation and image quality. The experiments assess reference-guided synthesis, attribute editing, loss ablations, inversion, StarGAN v2, and user preferences.

4.1.1 Reference-guided Image Synthesis

  • FDIT preserves source identity and structure while transferring reference color, illumination, and weather; SwapAE can over-adapt and distort composition and details.The experiments incorporate FDIT objectives into the vanilla Swapping Autoencoder.Source: §4.1.1 · p. 5, §4.1 · p. 5
  • 5.6% lower average FID than the current state-of-the-art method; FDIT also achieves the lowest FID across four evaluated datasets.Table 1 covers LSUN Church, Waterfalls, FFHQ, and CelebA-HQ.Source: §4.1.1 · p. 5, Table 1 · p. 5

4.1.2 Image Attributes Editing

  • FDIT edits toward a target domain by interpolating style codes while keeping source content fixed, improving attribute disentanglement versus vanilla SwapAE and StarGAN v2.The paper also evaluates disentangled semantic latent vectors using PCA.Source: §4.1.2 · pp. 5–6

4.2. Ablation Study

  • On LSUN Church, combined pixel- and Fourier-space losses achieve FID 48.21, versus 49.47 with pixel loss alone and 52.34 with neither loss.The ablation identifies the combined frequency-based objectives as most effective among these settings.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. GAN Inversion

  • On 1024 × 1024 images, FDIT improves reconstruction quality over Image2StyleGAN across MSE, MAE, PSNR, and SSIM.The reported qualitative results also describe improved structure, fine details, and color distribution.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Figure 6 · p. 7

4.4. StarGAN v2

  • On CelebA-HQ-Smile, FDIT preserves gender and hairstyle while changing the smiling attribute; FID improves from 17.32 to 16.86 versus vanilla StarGAN v2.The dataset treats smile status as style and identity as content.Source: §4.4 · p. 7

4.5. User Study

  • Across 2,058 user preferences from five datasets, FDIT receives 75.40% for identity preservation and 64.39% for photo-realistic image quality over baseline SOTA models.The study uses a two-alternative forced-choice setting.Source: §4.5 · pp. 7–8, Table 4 · p. 8

5. Related work

Prior translation work either optimizes latent codes or uses conditional GANs, while frequency-based methods had not addressed image-to-image translation. Neural style transfer preserves identity but is less flexible for cross-domain changes.

Generative adversarial networks (GAN)

  • GANs improved image generation, with later methods injecting noise hierarchically and using AdaIN for stylization.Earlier approaches directly fed Gaussian noise to the generator and had unsatisfactory photo-realism.Source: §5 · p. 8

Image-to-image translation

  • Conditional GAN and autoencoder translation is faster than iterative GAN inversion, but StarGAN v2 and Swapping Autoencoder can lose source structure.GAN inversion can also yield suboptimal reconstructions, making mobile deployment difficult.Source: §5 · p. 8

Frequency domain in deep learning

  • Frequency analysis had been applied to GAN artifacts and VAE reconstruction, but prior work had not explored it for image-to-image translation.The cited VAE approach does not use pixel-space frequency information and relies on a separate dataset for its free parameters.Source: §5 · p. 8

Neural style transfer

  • Neural style transfer preserves source content but struggles with cross-domain transformations and can produce images resembling the source domain.FDIT is presented as preserving identity while retaining domain-transfer capability.Source: §5 · p. 8

6. Conclusion

FDIT preserves frequency information in pixel and Fourier spectral spaces and uses high-frequency components to represent object structure akin to identity. Across five large-scale datasets and multiple tasks, it preserves source identity while producing photo-realistic image hybrids.

6. Conclusion

  • FDIT preserves frequency information in pixel and Fourier spectral spaces and directly uses high-frequency components to capture object structure akin to identity.The conclusion presents this as the framework's distinction from existing image translation models.Source: §6 · p. 8
  • Experiments on five large-scale datasets and multiple tasks show FDIT preserving source identity while producing photo-realistic image hybrids.The paper also reports extensive user studies and ablations validating effectiveness qualitatively and quantitatively.Source: §6 · p. 9
  • User studies and ablations further validate FDIT's effectiveness both qualitatively and quantitatively.Source: §6 · p. 9

A. Image Attributes Editing Results

FDIT's image-attribute editing results cover continuous domain interpolation and unsupervised semantic-vector discovery, with source content and identity retained as attributes change.

A. Image Attributes Editing Results

A.1. Continuous interpolation between Different Domains

  • Continuous style-code interpolation changes images smoothly toward target domains while keeping the content code unchanged and adhering to source identity.Figure 8 shows summer source images in the central column and interpolations toward autumn and winter in the remaining columns.Source: §A.1 · p. 12, Figure 8 · p. 12

A.2. Unsupervised Semantic Vector Discovery for Image Editing

  • PCA discovers orthonormal latent-space directions for editing style while seeking to leave image content untouched.Figure 9 shows source images centrally and interpolation results along two orthonormal directions; identity is maintained as illumination and hair color change.Source: §A.2 · p. 12, Figure 9 · p. 13
  • Editing in the full latent space displays more variation than the illustrated semantic-direction results.Source: §A.2 · p. 12

B. Frequency Domain Image Translation Results

Autoencoder-based FDIT results show improved shape preservation across church, bedroom, and waterfall scenes. The supplementary Flickr Mountains and Waterfalls datasets each contain 100,000 training images.

B. Frequency Domain Image Translation Results

  • FDIT better preserves shape across LSUN Church, CelebA-HQ, Flickr Waterfalls, and LSUN Bedroom generation results.The reported visual cues are church outlines, bedroom layouts, and waterfall scenes.Source: B. Frequency Domain Image Translation… · p. 13

C. Constructing the Flicker Dataset

D. Training Details

The appendix reports shared evaluation conditions and baseline-specific implementation settings, alongside figures documenting image translation, latent-space editing, and ablation outputs.

D. Training Details

  • Figures 10–12 show full-latent PCA edits, translation comparisons across four named datasets, and a mountains ablation; Table 1 details Image2StyleGAN and StyleGAN2 results.Source: Figure 10 · p. 13, Figure 11 · p. 14, Figure 12 · p. 15
  • All training and evaluation settings match the baselines, and experiments run on Tesla V100 GPUs.Source: D. Training Details · pp. 14–15
  • Swapping Autoencoder uses a StyleGAN2 backbone, resolution-dependent batch sizes, Adam, L1 reconstruction loss, adversarial loss, and a patch discriminator.Batch sizes are 16 at 256×256 and 4 at 1024×1024 resolution.Source: D. Training Details · p. 15
  • StarGAN v2 uses its official ResBlock-based implementation, batch size 8, Adam, a 10^-4 learning rate, and encoder-generator exponential moving averages at evaluation.Source: D. Training Details · p. 15
  • Image2StyleGAN projects images by iterative optimization in W +-space with StyleGAN-v1, while StyleGAN2 uses an LPIPS-based projector with StyleGAN-v2.GAN-inversion images use 5000 gradient-descent steps; the Image2StyleGAN v2 optimizer uses learning rate 0.01.Source: E. Details of Image2StyleGAN and… · p. 15, D. Training Details · p. 15

F. The qualitative results for Section 4.2

The qualitative ablation indicates that combining pixel- and Fourier-space losses preserves identity better than either loss alone, while Fourier-only loss compromises pixel-space style consistency.

F. The qualitative results for Section 4.2

Loading 2011.13611v3…