PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Xun Huang, Arun Mallya, Ting-Chun Wang, Ming-Yu Liu

arXiv:2112.05130v1cs.CV

TL;DR

Existing image-synthesis models often treat modalities separately, despite their complementary information and the practical need to work with only some inputs. PoE-GAN combines available modalities in a product-of-experts framework, and the authors report it outperforms prior multimodal and unimodal approaches while generating diverse outputs. Its multimodal model performs less well than its unimodal-trained counterpart when both are tested unimodally.

  • Problem

    Prior image-synthesis work treats modalities as separate tasks despite their complementary information, while one model must also handle arbitrary subsets of modalities.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    PoE-GAN combines modality-specific distributions with a product-of-experts generator and uses a hierarchical multiscale latent space for spatial inputs.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    PoE-GAN outperforms prior multimodal work and state-of-the-art unimodal approaches in the single-modality setting, and generates diverse images from the same inputs.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A single PoE-GAN framework supports image generation from any trained input subset, including the empty set.

    Source: §1 · p. 2

    Contradictory modalities yield unsatisfactory outputs, and a model trained unimodally performs better than the multimodal-trained model when tested unimodally.

    Source: §5 · p. 8

Abstract

from arXiv · show

Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting. The project website is available at https://deepimagination.github.io/PoE-GAN .

1. Introduction

Prior conditional image synthesis treats each modality as a separate task, despite modalities conveying complementary information and users not always providing every input. PoE-GAN combines available modalities and reports strong results in both multimodal and unimodal settings.

1. Introduction

  • Different modalities convey complementary information: segmentation sets coarse semantic layout, sketches specify within-region structure, and text describes or modifies objects and regions.The examples include mountain ridges in sketches and “frozen lake” or “pink clouds” in text.Source: §1 · p. 2
  • Existing approaches study each modality-to-image task in isolation, leaving them unable to use complementary information available across modalities.Source: §1 · pp. 1–2
  • Multimodal synthesis must combine inputs with different dimensions and structures, handle missing modalities, and avoid mode collapse that limits output diversity.Users may provide only a subset of inputs, while mode collapse can produce identical images for the same conditions.Source: §1 · p. 2
  • PoE-GAN combines available inputs with a product-of-experts generator, uses a hierarchical multiscale latent representation for spatial inputs, and trains with a multimodal projection discriminator and modality dropout.Contrastive losses are also used for input-output alignment.Source: §1 · p. 2
  • PoE-GAN outperforms prior work in multimodal and unimodal settings, including approaches designed specifically for a single modality, and generates diverse images from the same inputs.The framework is designed to accept any subset of modalities, including no inputs.Source: §1 · p. 2

2. Related work

Prior multimodal image-synthesis work includes VAE-based joint-distribution modeling and GAN- or VQGAN-based approaches. PoE-GAN instead targets conditional image generation with a GAN framework for high-resolution, large-scale datasets.

2. Related work

  • VAE-based multimodal work learns joint distributions, whereas PoE-GAN focuses on the image distribution conditioned on other modalities and uses GANs for high-resolution, large-scale experiments.Source: §2 · p. 2
  • Some prior VAE methods use product-of-experts inference networks to approximate the posterior, conceptually resembling PoE-GAN’s combination of modality information.Source: §2 · p. 2
  • TediGAN relies on a pretrained unconditional generator that is difficult to train on MS-COCO, while a concurrent VQGAN method combines modalities similarly to the concatenation-and-dropout baseline.Source: §2 · p. 2

3. Product-of-experts GANs

PoE-GAN trains one image generator to handle any subset of text, segmentation, sketch, and style-reference inputs, including no inputs. It combines modality-specific latent experts and uses a multimodal projection discriminator to align images with observed conditions.

3. Product-of-experts GANs

  • A single model targets image distributions conditioned on any subset of four modalities, including the unconditional distribution when none are supplied.The modalities considered are text, semantic segmentation, sketch, and style reference.Source: §3 · pp. 2–3

3.1. Product-of-experts modeling

  • The product-of-experts model combines modality-specific latent distributions with a prior, so additional inputs impose constraints and narrow the latent distribution.With no modalities, the latent distribution is the prior.Source: §3.1 · p. 3

3.2. Multiscale and hierarchical latent space

  • A hierarchical latent space pairs spatial inputs with latent feature maps at matching resolutions, helping preserve high-resolution control signals.The latent code includes a global feature vector and feature maps at increasing resolutions.Source: §3.2 · p. 3

3.3. Generator architecture

  • The generator aggregates modality encodings globally and uses local latent maps with global features to modulate decoder activations; segmentation and sketch also provide skip connections.Text and style reference provide global information, while segmentation and sketch contribute local details through skip connections.Source: §3.3 · pp. 3–4

3.4.Multiscale multimodal projection discriminator

  • The multimodal projection discriminator adds an image-embedding match for each observed modality, while its multiscale variant computes alignment losses at multiple resolutions.For spatial modalities such as segmentation and sketch, losses are averaged across locations and then resolutions.Source: 3.4.Multiscale multimodal projection… · p. 5

3.5. Losses and training procedure

  • Training uses latent KL regularization, text-only conditional contrastive loss because of memory costs, and modality dropout to improve robustness to missing inputs at test time.The KL loss encourages conditional latents to remain close to the prior and reduces conditional mode collapse.Source: §3.5 · pp. 5–6

4. Experiments

Experiments compare PoE-GAN across multimodal and unimodal image-synthesis settings, reporting stronger results than TediGAN and modality-specific systems. Ablations show that the generator, KL loss, modality dropout, MMPD, and contrastive losses play distinct roles.

4. Experiments

  • Experiments use MM-CelebA-HQ, MS-COCO 2017 with COCO-Stuff annotations, and proprietary landscapes, comparing TediGAN, modality-specific baselines, and StyleGAN2 unconditionally.Source: §4 · p. 6, §4.1 · pp. 6–7

4.1. Main results

  • PoE-GAN obtains much lower FID than TediGAN in all MM-CelebA-HQ settings and outperforms modality-specific state-of-the-art methods on both datasets when conditioned on one modality.The datasets are MM-CelebA-HQ and MS-COCO.Source: §4.1 · p. 7
  • Qualitative MS-COCO comparisons report better image quality and realistic complex objects, including human faces and stop signs.Source: §4.1 · p. 7
  • On the 1024×1024 landscape dataset, PoE-GAN produces high-resolution landscapes with photo-realistic details and diverse outputs for the same conditional inputs.Source: §4.1 · p. 7, Figure 10 · p. 7

4.2. Ablation studies

  • The concatenation baseline performs well only with all modalities; missing inputs reduce its FID, and its outputs have low LPIPS diversity.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
  • Without KL loss, diversity and missing-modality robustness suffer; without modality dropout, the model can over-rely on an informative input such as segmentation.Without KL loss, expert variances approach zero and the latent code becomes a deterministic weighted average of expert means.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
  • MMPD is more effective in all settings than a discriminator that concatenates images and conditional inputs.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
  • Contrastive losses are useful but not essential: image contrastive loss slightly improves FID in most settings, while text contrastive loss improves text-to-image FID.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

5. Conclusion

PoE-GAN converts arbitrary subsets of modalities into images and outperforms prior multimodal and state-of-the-art unimodal approaches. Its limitations include unsatisfactory outputs for contradictory inputs and a performance gap between multimodal- and unimodal-trained models in unimodal testing.

  • PoE-GAN converts arbitrary modality subsets into images and outperforms prior multimodal and state-of-the-art unimodal synthesis approaches.The framework is intended to produce images satisfying the provided conditions.Source: §5 · p. 8
  • Contradictory modalities produce unsatisfactory outputs, and the multimodal-trained model performs worse than the unimodal-trained model in unimodal testing.The authors identify multimodal fusion as an area for future improvement.Source: §5 · p. 8
  • Multimodal image synthesis can make it easier to create desired fake images, including for visual misinformation.The authors propose providing implementations and training data to support forensic detection research and watermarking schemes.Source: §5 · p. 8

A. Datasets

The evaluation uses MM-CelebA-HQ, MS-COCO, and a proprietary landscape dataset, with different annotation sources and image resolutions. The landscape data relies on pseudo-annotations, while the other datasets include human or automatically generated annotations.

  • MM-CelebA-HQ contains 30,000 1024×1024 celebrity images, with human segmentation maps and automatically generated text and sketch annotations.The main experiment uses the original resolution; ablations use 256×256 images.Source: A. Datasets · p. 12
  • MS-COCO contains 123,287 complex-scene images with up to five text descriptions, segmentation maps, and generated sketch maps.The main experiment uses 256×256 images; ablations use 64×64 images.Source: A. Datasets · p. 12
  • The proprietary landscape dataset has around 10 million images above 1024×1024, with pseudo segmentation, sketch, and text annotations.The text annotation uses CLIP image embeddings; 50,000 images are reserved for testing.Source: A. Datasets · p. 12

B. Implementation details

The implementation combines GAN-stabilization techniques with hierarchical latent rebalancing, modality-specific encoders, and multiscale discrimination. The appendices also specify decoder and channel configurations and report the hardware and speed measurements.

B. Implementation details

  • Training uses Adam, exponential-moving-average generator weights, GAN stabilization techniques, mixed precision, and 0.5 modality dropout.The implementation also limits Gaussian log variance in product-of-experts layers and uses a learnable contrastive-loss temperature.Source: B. Implementation details · p. 12

B.1. Decoder

  • Global PoE-Net uses four-layer MLPs, while Local PoE-Net uses four-layer CNNs with 1×1 endpoint kernels and 3×3 middle kernels.The reported latent and style dimensions are 512.Source: §B.1 · p. 12

B.2. Encoders

  • Text uses a four-layer MLP over CLIP embeddings; segmentation and sketch use CNN encoders with input and decoder skip connections, while style uses a residual encoder.The style code concatenates each residual block’s output mean and standard deviation.Source: §B.2 · pp. 12–13, Figure 11 · p. 14

B.3. Discriminator

  • The discriminator computes multiscale projection-discriminator loss from image and modality feature maps using a residual image encoder.Its modality encoders do not share parameters with the generator encoders, and single-vector features are spatially replicated.Source: §B.3 · p. 13

B.4. Hyper-parameters

  • Decoder and discriminator channel sizes match and are four times the modality-encoder channel size, with channel counts increasing as resolution decreases.The appendix describes latent-channel sizes and reports dataset-specific hyperparameters in Table 5.Source: §B.4 · pp. 13–14, Table 5 · p. 13

B.5. Hardware and speed

  • The appendix reports computation infrastructure and training and inference speed for each dataset.Models use V100 GPUs except Landscape, which uses A100 GPUs; inference timing uses a single TITAN RTX workstation GPU.Source: Table 6 · p. 13, §B.5 · p. 14

C. Additional results

Additional experiments report a significantly lower FID than prior text-to-image methods on 256×256 MM-CelebA-HQ and show that PoE-GAN is not larger than the compared models. The authors argue that its improvement does not come from a larger model.

C.1. Model size comparison

  • PoE-GAN uses fewer parameters than some single-modal baselines and no significantly more parameters overall, despite its more challenging task.The authors conclude that the improvement does not come from using a larger model.Source: §C.1 · p. 14

C.2. Comparison on MM-CelebA-HQ (256×256)

  • PoE-GAN achieves a significantly lower FID than the TediGAN and M6-UFC text-to-image baselines on 256×256 MM-CelebA-HQ.Source: §C.2 · p. 14

C.3. Additional comparison with TediGAN

PoE-GAN differs from TediGAN through learned product-of-experts fusion and joint conditional and unconditional training, and it generates images much faster.

C.3. Additional comparison with TediGAN

  • PoE-GAN learns conditional and unconditional generation simultaneously, and its FID improves as more conditions are provided.TediGAN relies on a pretrained unconditional generator, which the passage says upper-bounds its image quality.Source: §C.3 · p. 15
  • Unlike TediGAN’s handcrafted latent-layer rule, PoE-GAN uses learned product-of-experts fusion that is more general across generator architectures and modalities.TediGAN combines top-layer latent vectors from one modality with bottom-layer vectors from another.Source: §C.3 · p. 15
  • 0.07 seconds versus TediGAN’s 51.2 seconds to generate a 1024×1024 image demonstrates PoE-GAN’s faster sampling.TediGAN’s instance-level optimization makes its sampling very slow, according to the passage.Source: §C.3 · p. 15

C.4. Training PoE-GAN with a single modality

The section compares joint multimodal training with single-modality training and reports a user study of image realism preferences.

C.4. Training PoE-GAN with a single modality

  • A model trained for one modality slightly outperforms the joint model when conditioned on that modality, while PoE-GAN’s gains over unimodal state of the art come from its architecture and training scheme rather than extra annotations.The authors attribute the trade-off to the increased task complexity of an additional modality outweighing the benefits of its annotations.Source: §C.4 · p. 15

C.5. User study

  • In the MS-COCO user study, the majority of users preferred PoE-GAN over baseline methods for text-to-image and segmentation-to-image synthesis.The study asked users to choose the more realistic of two images generated from the same conditional input.Source: §C.5 · p. 15

C.6. Additional qualitative examples

The additional examples cover multimodal conditional synthesis, unimodal comparisons, and unconditional generation, while also illustrating behavior on contradictory inputs.

D. Additional qualitative examples

  • PoE-GAN produces diverse samples from two modalities across MM-CelebA-HQ, MS-COCO, and Landscape, with further text-to-image, segmentation-to-image, and unconditional examples.The unconditional samples are uncurated; the text and segmentation comparisons are on MS-COCO.Source: §C.6 · p. 15
  • With contradictory segmentation and text conditions, PoE-GAN ignores the text input in the examples shown.The figure caption identifies these as examples conditioned on contradictory segmentation and text inputs.Source: Figure 12 · p. 16

D. Limitation

PoE-GAN has limitations with contradictory multimodal conditions and incurs a higher FID when providing multimodal synthesis capability.

D. Limitation

  • Contradictory multimodal inputs are a limitation: when text conflicts with segmentation, the text input is usually ignored.The passage attributes this behavior to the text expert usually having larger variance than the segmentation expert.Source: D. Limitation · p. 16
  • The product-of-experts weighting gives less influence to an expert with larger variance, which explains the text expert’s weaker influence in the stated conflict.The reported variance comparison is between the text and segmentation experts.Source: D. Limitation · p. 16
  • Multimodal conditional synthesis still carries a higher FID score, leaving room to improve modality fusion.The authors leave this improvement for future work.Source: D. Limitation · p. 16
Loading 2112.05130v1…