PaperlayerView exact revision on arXiv

Source-linked AI summary

UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation

Dmitrii Torbunov, Yi Huang, Haiwang Yu, Jin Huang, Shinjae Yoo, Meifeng Lin, Brett Viren, Yihui Ren

arXiv:2203.02557v3cs.CVeess.IV

TL;DR

Scientific image translation often lacks paired pixel-level data, while some applications require one-to-one mappings between simulation and experimental images. UVCGAN equips CycleGAN with a Vision Transformer and training techniques; it reports outperforming competing methods across diverse benchmarks while retaining cycle consistency.

  • Problem

    Scientific simulation and experimental data often lack exact pixel-to-pixel correspondences, yet scientific translation can require a one-to-one mapping between domains.

    Source: §1 · p. 1

  • Method

    UVCGAN adds a Vision Transformer generator to cycle-consistent CycleGAN and uses gradient-penalty regularization and self-supervised inpainting pre-training.

    Source: §1 · p. 1, §3.3 · p. 4, §3.4 · p. 4

  • Results

    UVCGAN with self-supervised pre-training and gradient-penalty regularization outperforms competing methods on a diverse set of image-translation benchmarks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Cycle consistency supports scientific applications requiring a one-to-one mapping between simulation results and experimental counterparts.

    Source: §1 · p. 1

    The authors speculate that randomly initialized discriminator signals early in fine-tuning may undermine pre-trained generator weights, and that gradient penalty is needed for the best performance with pre-trained networks.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Unpaired image-to-image translation has broad applications in art, design, and scientific simulations. One early breakthrough was CycleGAN that emphasizes one-to-one mappings between two unpaired image domains via generative-adversarial networks (GAN) coupled with the cycle-consistency constraint, while more recent works promote one-to-many mapping to boost diversity of the translated images. Motivated by scientific simulation and one-to-one needs, this work revisits the classic CycleGAN framework and boosts its performance to outperform more contemporary models without relaxing the cycle-consistency constraint. To achieve this, we equip the generator with a Vision Transformer (ViT) and employ necessary training and regularization techniques. Compared to previous best-performing models, our model performs better and retains a strong correlation between the original and translated image. An accompanying ablation study shows that both the gradient penalty and self-supervised pre-training are crucial to the improvement. To promote reproducibility and open science, the source code, hyperparameter configurations, and pre-trained model are available at https://github.com/LS4GAN/uvcgan.

1. Introduction

UVCGAN revisits CycleGAN for unpaired translation, seeking stronger performance while preserving the one-to-one mappings needed in scientific applications. It combines a Vision Transformer generator with training techniques and highlights the need for fair, reusable evaluation.

1. Introduction

  • Simulation and experimental data can form unpaired translation domains because exact pixel-to-pixel correspondences are often difficult or impossible to obtain.The proposed work focuses on developing models for this translation task.Source: §1 · p. 1
  • CycleGAN uses two GANs and cycle-consistency to translate between unpaired domains while requiring each image to map back to itself after a translation cycle.This one-to-one mapping is important for scientific applications where translation should not alter the science.Source: §1 · p. 1
  • UVCGAN combines a Vision Transformer with advanced training techniques and can outperform competing models on several benchmark datasets.The work also reports that architecture changes alone are insufficient; pre-training and gradient penalty are needed.Source: §1 · pp. 1–2
  • The authors identify unmatched prior evaluation results and standardize evaluation to support fair comparisons and reusable benchmarks.Source: §1 · p. 2

2. Related work

Related work spans generative-model training, computer-vision transformers, self-supervised pre-training, and unpaired translation frameworks. The paper motivates combining these ideas while retaining explicit cycle consistency for settings that require one-to-one mappings.

2. Related work

  • GANs learn through a generator-discriminator minimax game, while gradient penalty has been studied as a means to stabilize GAN training.The cited work notes that gradient penalty, rather than the Wasserstein loss itself, was responsible for stabilizing training.Source: §2 · p. 2
  • Transformers can capture non-local patterns more efficiently than CNNs, and self-supervised pre-training initializes networks using unsupervised tasks derived from the data.Source: §2 · p. 2
  • The authors hypothesize that pre-training may help GANs avoid mode collapse and choose inpainting because scientific data may differ substantially from natural images.Inpainting also suits translation models because output and input shapes match.Source: §2 · p. 2
  • Unpaired translation frameworks vary in consistency enforcement: some relax or omit explicit cycle constraints, while U-GAT-IT retains a CycleGAN-like approach with more sophisticated networks.Source: §2 · p. 3
  • Relaxed or implicit consistency methods can increase diversity and evaluation scores, but introduce randomness that makes them unsuitable when one-to-one mapping is required.The cited frameworks also use more parameters than the original CycleGAN, requiring more computation and training time.Source: §2 · p. 3

3. Method

UVCGAN preserves CycleGAN’s bidirectional, cycle-consistent framework while changing the generator and training procedure. It combines a UNet-ViT generator, gradient-penalty regularization, and self-supervised inpainting pre-training.

3.1. CycleGAN-like Models

  • CycleGAN-like models use two generator–discriminator pairs for translation between domains, with generator losses combining adversarial, cycle-consistency, and identity-consistency terms.The cycle-consistency and identity terms are weighted by combination coefficients.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2. UNet-ViT Generator

  • The UNet-ViT generator passes encoder features to the decoder through skip connections and sends bottleneck features through a pixel-wise Vision Transformer.The authors hypothesize that skip connections pass high-frequency features while the ViT learns pairwise relationships among low-frequency features; their ViT uses 12 Transformer encoder blocks.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Discriminator Loss with Gradient Penalty (GP)

  • UVCGAN uses LSGAN with γ-centered gradient-penalty regularization; the authors report more stable, less hyperparameter-sensitive training, while WGAN + GP performed worse.Zero-centered GP was sensitive to hyperparameter values and did not improve training stability in their experiments.Source: §3.3 · p. 4

3.4. Self-Supervised Pre-training by Inpainting

  • Generators are pre-trained on an image-inpainting task that masks 40% of non-overlapping 32 × 32 patches and trains the model to reconstruct the original image using pixel-wise L1 loss.The study considers pre-training on the translation dataset or ImageNet, alongside no pre-training in its ablation.Source: §3.4 · p. 4

4. Experiments

The experiments prioritize Selfie2Anime and two CelebA-derived tasks, GenderSwap and Eyeglasses. UVCGAN training uses inpainting pre-training and a million translation-training iterations, while other models are reproduced using available models or documented settings.

4.1. Benchmarking Datasets

  • The benchmarks are Selfie2Anime, with 3.4K images per domain, and CelebA-derived GenderSwap and Eyeglasses tasks.GenderSwap has about 68K male and 95K female training images; Eyeglasses has 11K images with glasses and 152K without.Source: §4.1 · pp. 4–5

4.2. UVCGAN Training Procedures

  • UVCGAN is trained for one million iterations at batch size one; its generators use self-supervised inpainting pre-training, and the selected configuration is LSGAN + GP with λGP = 0.1 and γ = 100.The reported best-performing configuration across all three tasks also pre-trains generators on the image-translation dataset itself.Source: §4.2 · p. 5

4.3. Other Model Training Details

  • For comparisons, the authors use existing pre-trained models when available and otherwise train models using provided configurations, original-paper descriptions, or similar-task settings.Table 1 reports training time for both translation directions; UVCGAN and some comparators train two generators jointly, while ACL-GAN and Council-GAN train directions separately.Source: §4.3 · p. 5, Table 1 · p. 6

5. Results

UVCGAN scores better on most translation tasks while preserving input-image correlations; the ablations and attention inspection examine its training components and focus.

5.2. Model Evaluation and Reproducibility

  • Evaluations standardize test-set size, CelebA image cropping, and FID/KID computation; benchmark implementations also require dataset-specific resizing and cropping adjustments.FID and KID were delegated to torch-fidelity, with KID subset sizes of 50 for Selfie2Anime and 1000 for the two CelebA datasets.Source: §5.2 · p. 6

5.1. Evaluation Metrics

  • UVCGAN performs better on most image-translation tasks, while its outputs remain strongly correlated with inputs; U-GAT-IT scores lower on anime-to-selfie but can omit input features.The reported correlations include hair color and face orientation; FID and KID are lower-is-better metrics.Source: Table 2 · p. 6, §5.1 · p. 6

5.3. Ablation Studies

  • Gradient penalty with identity loss improves performance, while identity loss without gradient penalty gives mixed results; same-dataset pretraining helps especially in combination with them.The authors speculate that a randomly initialized discriminator can provide a meaningless signal early in fine-tuning and undermine pretraining.Source: §5.3 · pp. 6–7

5.4. Interpretation of Attention

  • Attention maps average head and target-token weights within each of 12 Transformer blocks, with eye and mouth regions receiving the most attention.The 16×16 token attention vector is reshaped, enlarged to image dimensions, and Gaussian-filtered for the heatmap.Source: §5.4 · pp. 7–8, Figure 4 · p. 8

6. Conclusion

The conclusion presents UVCGAN as a cycle-consistent, content-preserving translator combining a ViT generator with training and regularization techniques. It reports benchmark gains while identifying further scientific-data evaluation and potential misuse as considerations.

6. Conclusion

  • UVCGAN combines cycle-consistent, content-preserving translation with a generator intended to handle long-range spatial dependencies in scientific domains.The paper reports that the model, together with self-supervised pretraining and gradient-penalty regularization, outperforms competing methods across diverse image-translation benchmarks.Source: §6 · p. 8
  • The ablation study suggests that gradient penalty and cycle-consistent loss work well with UVCGAN, and attention inspection indicates focus on relevant source-image regions.Source: §6 · p. 8
  • More open scientific datasets are needed to further demonstrate UVCGAN’s handling of long-distance patterns beyond benchmark datasets.Source: §6 · p. 8
  • Training has greater environmental impact than original CycleGAN but considerably less than other advanced models; the authors also acknowledge potential fake-content misuse.The authors note that code and pretrained models are available as a contribution to mitigation efforts.Source: §6 · p. 8

Appendix A. Extended UVCGAN Ablation Studies

The extended ablation examines the hybrid UNet-ViT generator, self-supervised pretraining, and gradient penalty, finding that pretraining’s benefit depends on gradient penalty.

Appendix A. Extended UVCGAN Ablation Studies

  • The hybrid UNet-ViT generator alone typically improves on CycleGAN, even without self-supervised pretraining or gradient penalty.Source: Appendix A · p. 12
  • Self-supervised pretraining without gradient penalty does not seem to improve translation and sometimes makes performance worse.Source: Appendix A · p. 12
  • Self-supervised pretraining helps only when used together with gradient penalty.Source: Appendix A · p. 12

Appendix B. Hyperparameter Tuning for Other Algorithms

The appendix describes hyperparameter-tuning coverage for ACL-GAN, CycleGAN, and U-GAT-IT, including cases where tuning was omitted or algorithms failed on the evaluated tasks.

Appendix B. Hyperparameter Tuning for Other Algorithms

  • Council-GAN tuning was omitted because each translation run takes 300 hours; tuning coverage otherwise skips tasks with supplied pretrained models or author-provided hyperparameters.Tuning results are supplied only when an algorithm did not work on a dataset or task.Source: Appendix B · p. 12
  • ACL-GAN worked on the three studied datasets only for selfie-to-anime, male-to-female, and glasses removal; reverse-direction tuning varied focus-loss parameters.Source: Appendix B · p. 12, Table A2 · p. 12
  • CycleGAN did not work on any of the three datasets; its tuning grid varied generator type, cycle-consistency weight, and GAN mode, but WGAN-GP failed because its GP implementation was improper.Source: Appendix B · p. 12, Table A3 · p. 13
  • U-GAT-IT’s original implementation could not handle non-square images, so the revised evaluation corrected aspect-ratio preprocessing and tested cycle-consistency weights on the CelebA datasets.The revised preprocessing resized a 178×218 CelebA image to 256×313, then used a central 256×256 test crop.Source: Appendix B · pp. 12–13, Table A4 · p. 13

Appendix C. More detail about the UNet-ViT Generator

The generator places a pixel-wise Vision Transformer at the UNet bottleneck, while skip connections carry encoder features to corresponding decoder layers. The ViT processes spatial features as tokens and projects its output back into the decoder.

Appendix C. More detail about the UNet-ViT Generator

  • A UNet encoder sends bottom-most features to a pixel-wise ViT bottleneck and passes features from each layer to the corresponding decoder layer through skip connections.The decoder combines upsampled features with the corresponding encoder features.Source: Appendix C · p. 13
  • Each downsampling stage halves width and height, and the last three stages double feature dimension, yielding bottleneck dimensions (w0/16, h0/16, 8f0).The decoder upsamples and concatenates features from the matching encoder skip connection.Source: Appendix C · p. 13
  • A final 1 × 1 convolution with sigmoid activation converts the decoder output into an image.Source: Appendix C · p. 13
  • The ViT flattens bottleneck features into w × h tokens of length f, concatenates 2D Fourier positional embeddings, and linearly maps them to dimension fv.After 12 Transformer encoder blocks, it projects the output back to dimension f and unflattens it to width w and height h.Source: Appendix C · p. 13
  • For 256 × 256 inputs with f0 = 48, the bottleneck has w = h = 16 and f = 384; the ViT uses 12 encoder blocks and rezero scaling.The feed-forward network uses fh = 4fv, with fp and fv set to f.Source: Appendix C · p. 13

Appendix D. Additional Sample Translations

The appendix points to additional sample translations for three named tasks.

Appendix D. Additional Sample Translations

Related papers

Loading 2203.02557v3…