Source-linked AI summary
Toward Multimodal Image-to-Image Translation
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, Eli Shechtman
TL;DR
Image-to-image translation is ambiguous because one input can have multiple plausible outputs, while training data often provides only one paired example. The paper uses latent codes and jointly enforces mappings between outputs and latent representations through several model variants. BicycleGAN combines both directions and is reported to produce outputs that are more realistic and diverse than alternatives, with systematic evaluation of these properties.
Problem
Image-to-image translation must represent multiple plausible outputs for an input, but paired training data usually contains only one pair and existing methods can suffer mode collapse.
Method
The paper explores cVAE-GAN, cLR-GAN, and BicycleGAN, with BicycleGAN jointly enforcing output-to-latent and latent-to-output constraints.
Results
BicycleGAN produces results that are both diverse and realistic, and the paper systematically evaluates variants using human photorealism judgments and a perceptual diversity metric.
Takeaways & Limitations
Combining objectives for a bijective mapping between latent and output spaces yields results that are more realistic and diverse in conditional image generation.
Abstract
from arXiv · showhide
Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a \emph{distribution} of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.
1 Introduction
Image-to-image translation can be multimodal, so the paper models distributions of outputs while seeking realism, diversity, and faithfulness to the input. It proposes latent–output bijective consistency and evaluates multiple variants systematically.
- Motivation: Many conditional image-generation methods produce a single result, although one input may correspond to multiple plausible outputs.Night-to-day translation can vary with cloud patterns and lighting conditions.
- Motivation: A low-dimensional latent code represents output aspects absent from the input, but existing methods can suffer mode collapse.At inference, sampled latent codes are combined with the input by a deterministic generator.
- Approach: The proposed approach jointly learns mappings from input and latent code to output and from output back to latent code, discouraging different codes from producing the same output.The encoder can represent output-mode information such as sky color, ground lighting, and cloud patterns.
- Variants: The paper explores cVAE-GAN, cLR-GAN, and BicycleGAN objectives for enforcing latent–output connections in different ways.cVAE-GAN encodes ground-truth outputs and regularizes the latent distribution, whereas cLR-GAN samples latent codes and reconstructs them from generated outputs.
- Contribution: BicycleGAN combines constraints in both directions and is reported to produce diverse, visually appealing results across image-to-image translation problems.The study also examines encoder architectures and latent-code injection methods.
- Evaluation: The variants are systematically evaluated using human judgments of photorealism and a perceptual distance metric for output diversity.Code and data are reported as publicly available.
2 Related Work
Related work spans generative models, conditional image generation, and explicit multimodality representations. Conditional GANs improve image quality but commonly lose multimodality when the generator ignores random noise.
- Generative modeling: Generative modeling has used restricted Boltzmann machines, autoencoders, variational autoencoders, autoregressive models, and GANs to model image distributions.These approaches differ in how they represent stochasticity and generate samples.
- Generative modeling: GANs map random values from an easy-to-sample distribution to output images in a single feedforward pass and use a discriminator during training.The discriminator distinguishes target-distribution images from generator samples.
- Conditional image generation: Conditional image-generation methods can be built from conditional VAEs, autoregressive models, or conditional GANs.Image-to-image conditional GANs have produced substantial quality improvements.
- Conditional image generation: Conditional GAN quality has often come at the expense of multimodality because generators largely ignore random noise when given relevant context.This limits the effect of noise as a source of varied outputs.
- Explicitly-encoded multimodality: Explicit multimodality methods provide additional conditioning such as color or shape scribbles, while mixtures of models can produce multiple discrete answers.The passage notes that these approaches have limitations for representing multimodal outputs.
3 Multimodal Image-to-Image Translation
The paper models multimodal image-to-image translation by sampling a low-dimensional latent code and explicitly enforcing consistency between latent codes and generated outputs. It combines reconstruction-based and latent-regression objectives to improve diversity while maintaining realism.
- Problem: A single input image may correspond to multiple plausible outputs, so the goal is to sample diverse and realistic outputs from p(B|A).The model represents ambiguous output aspects with a low-dimensional latent space z.
- Motivation: Conditional GANs often ignore random noise, producing deterministic outputs and motivating explicit enforcement that latent codes capture relevant information.The pix2pix formulation combines adversarial and ℓ1 losses, while the noise vector receives little incentive for use.
- Variants: cVAE-GAN encodes the ground-truth output into z, reconstructs it with the input, and regularizes the latent distribution toward a Gaussian for test-time sampling.Its reconstruction objective uses z sampled from E(B), while KL regularization supports sampling when B is unavailable.
- Variants: cLR-GAN samples z first, generates an output, and trains an encoder to recover z while using adversarial loss rather than requiring ground-truth similarity.The generated output need not match the ground truth because the latent code is randomly drawn, but it should remain realistic.
- BicycleGAN: BicycleGAN combines cVAE-GAN and cLR-GAN so both cycles, B → z → bB and z → bB → bz, constrain the latent-output relationship.The hybrid addresses cVAE-GAN's prior-sampling concerns and cLR-GAN's lack of ground-truth input-output supervision.
- Variants: The study also varies latent-code injection, comparing spatial replication and concatenation at the input layer with injection into intermediate encoder layers.The full objective includes weighted terms controlled by λ, λlatent, and λKL, and a cVAE-GAN++ variant exposes prior samples to the discriminator.
4 Implementation Details
The implementation uses U-Net generators, multiscale PatchGAN discriminators, alternative encoder architectures, and controlled latent-code injection strategies. Training uses LSGAN objectives with fixed loss weights and an eight-dimensional latent code.
- Network architecture: The generator is a U-Net with symmetric skip connections, while two PatchGAN discriminators classify overlapping 70 × 70 and 140 × 140 patches.The encoder experiments compare a convolutional network with downsampling layers against a residual-block classifier.
- Training details: The latent dimension is fixed at |z| = 8 across datasets, and generator and encoder weights are tied in cVAE-GAN and cLR-GAN.The authors report that simultaneously optimizing G and E for the latent reconstruction loss encouraged hiding latent information rather than learning meaningful modes.
- Latent injection: Latent injection is implemented either by concatenating a spatially replicated z with the input image or by adding z to every intermediate generator layer.These correspond to the add_to_input and add_to_all strategies.
5 Experiments
Experiments evaluate BicycleGAN and related variants across one-to-many image-to-image translation tasks, measuring qualitative output quality, diversity, realism, latent-code length, encoder architecture, and injection strategy. BicycleGAN produces diverse and realistic outputs while avoiding the main trade-offs observed in alternative methods.
- Experimental Setup: The study evaluates six model variants and baselines on one-to-many translation datasets, including edges→photos, Google maps→satellite, labels→images, and night→day.All models are trained on 256 × 256 images.
- Qualitative Evaluation: BicycleGAN produces realistic and diverse results across night→day, edges→shoes, edges→handbags, and maps→satellites examples.Figure 4 presents inputs, ground-truth outputs, and four random samples for each task.
- Realism and Diversity: BicycleGAN achieves higher realism while maintaining diversity, whereas pix2pix+noise has little diversity and cAE-GAN produces large sampling artifacts.Diversity is measured with average LPIPS distance, and realism with a real-versus-fake Amazon Mechanical Turk test on Google maps→satellites.
- Qualitative Evaluation: On labels→facades, pix2pix+noise typically yields one realistic output, whereas cAE-GAN adds variation at a large cost to result quality.The hybrid BicycleGAN combines the strengths of the alternative approaches in qualitative comparisons.
- Architecture and Latent Injection: EResNet better encodes output images than ECNN, while add_to_input and add_to_all achieve similar reconstruction performance; the final model uses EResNet and add_to_all.The injection comparison uses reconstruction loss ||B −G(A, E(B))||1.
- Latent Code Length: The optimal latent-code length depends on the dataset, application, and ambiguity of the output; the study compares dimensions 2, 8, and 256.Very short codes may limit diversity, while very long codes can make sampling difficult.
6 Conclusions
The paper evaluates methods for reducing mode collapse in conditional image generation and finds that combining objectives for bijective latent–output mappings yields more realistic and diverse results.
- Combining multiple objectives to encourage a bijective mapping between latent and output spaces produces more realistic and diverse results.The paper identifies semantically meaningful latent attributes and user-controllable image transformations as future directions.
Changelog
The changelog records revisions across four versions, including implementation details, related work, diversity scoring, a bug fix, and affiliation updates.
- Version 2 added implementation details and related work, and replaced VGG-16 with LPIPS for the diversity score.
- Version 3 fixed a minor LPIPS-score computation bug, while Figure 6 was updated with scores that barely changed.
- Version 4 updated an Adobe Research internship affiliation in the acknowledgments.