PaperlayerView exact revision on arXiv

Source-linked AI summary

CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training

Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua

arXiv:1703.10155v2cs.CV

TL;DR

Fine-grained image generation requires realistic and diverse samples conditioned on specific categories, but naive VAE-GAN combinations suffer from blurriness and unstable gradients. CVAE-GAN combines conditional variational and adversarial generation with asymmetric mean-feature training and encoder-based pairwise matching, achieving reported gains across applications and datasets. The resulting model supports category-specific synthesis and related tasks including inpainting, attribute morphing, super-resolution, and face-recognition augmentation.

  • Problem

    The paper addresses fine-grained image synthesis for specific identities or object categories despite limitations of naive VAE-GAN combinations, including blurry outputs and gradient vanishing.

  • Method

    CVAE-GAN combines conditional variational encoding and adversarial generation with an asymmetric mean-feature objective, an encoder linking images to latent vectors, and pairwise feature matching.

  • Results

    The authors report realistic and diverse fine-grained samples, superior performance on three datasets, and applications including image generation, inpainting, attribute morphing, super-resolution, and face-recognition augmentation.

  • Takeaways & Limitations

    The trained model supports category-specific synthesis and can provide synthesized data for improving face-recognition training.

  • Takeaways & Limitations

    Mean feature matching reduces but does not eliminate mode collapse, and matching feature centers can still leave generated and real distributions different.

Abstract

from arXiv · show

We present variational generative adversarial networks, a general learning framework that combines a variational auto-encoder with a generative adversarial network, for synthesizing images in fine-grained categories, such as faces of a specific person or objects in a category. Our approach models an image as a composition of label and latent attributes in a probabilistic model. By varying the fine-grained category label fed into the resulting generative model, we can generate images in a specific category with randomly drawn values on a latent attribute vector. Our approach has two novel aspects. First, we adopt a cross entropy loss for the discriminative and classifier network, but a mean discrepancy objective for the generative network. This kind of asymmetric loss function makes the GAN training more stable. Second, we adopt an encoder network to learn the relationship between the latent space and the real image space, and use pairwise feature matching to keep the structure of generated images. We experiment with natural images of faces, flowers, and birds, and demonstrate that the proposed models are capable of generating realistic and diverse samples with fine-grained category labels. We further show that our models can be applied to other tasks, such as image inpainting, super-resolution, and data augmentation for training better face recognition models.

1. Introduction

CVAE-GAN targets realistic, diverse image synthesis within fine-grained categories by combining conditional variational encoding with adversarial generation. Its asymmetric generator objective and encoder-based feature matching address unstable training, mode collapse, and structural fidelity.

  • Approach: CVAE-GAN combines a variational auto-encoder with a generative adversarial network under a conditioned process for fine-grained image categories.The framework targets specific identities and species such as faces, flowers, and birds.
  • Motivation: The naive VAE-GAN combination can produce blurry images, discriminator rejection, gradient vanishing, and outputs resembling VAE results.These issues make the straightforward combination insufficient in practice.
  • Asymmetric objective: The generator minimizes the ℓ2 distance between generated and real mean features rather than using the discriminator’s cross entropy loss.For multiclass generation, category-specific generated samples match the average feature of real data from that category.
  • Asymmetric objective: The asymmetric objective addresses gradient vanishing to some extent and can reduce mode collapse, making GAN training more stable.The passage describes this as partial prevention rather than a complete solution.
  • Feature matching: An encoder maps real images to latent vectors, while reconstruction and pairwise feature matching anchor generated samples to image structure and diversity.These anchors explicitly relate latent space to real image space; pixel reconstruction helps preserve lines and facial structure.
  • Framework and applications: The end-to-end pipeline uses encoder, generator, discriminator, and classifier networks, and the trained model supports generation, inpainting, attribute morphing, and face-recognition data augmentation.The authors report realistic, diverse samples and more stable, faster convergence than GAN, with outperformance over CVAE, CGAN, and other state-of-the-art methods.

2. Related work

Generative-model research progressed from simple statistical assumptions to deep models that capture complex image structure. The related methods include VAE, GAN, conditional variants, feature-based generation, and autoregressive models, while CVAE-GAN combines complementary matching strategies for more stable convergence.

  • Classical models: Early models such as PCA, ICA, and GMM assume simple data formation and struggle with complex patterns of irregular distributions.Later HMM, MRF, and RBM approaches discriminatively train generative models.
  • Deep generative models: Deep hierarchical architectures capture complex data structures and generate natural images that are more realistic than those of conventional generative models.The main themes identified are VAE, GAN, and autoregression.
  • VAE: VAE combines an encoder with a decoder or generator, but injected noise and squared-error measures often produce blurry samples.This limitation motivates complementary adversarial approaches.
  • GAN: GAN jointly trains a generator and discriminator, but training can be difficult to converge and generated samples may be far from natural.Subsequent work sought to improve generated-sample quality.
  • Conditional generation: Conditional VAE and conditional GAN introduce conditioning for probabilistic one-to-many mappings and support tasks including face generation, text-to-image synthesis, and conditional image synthesis.The passage reports impressive results across these conditional-generation applications.
  • Feature-guided generation: Feature-based methods generate images from representations learned by classification models, while PPGN uses gradient ascent and a latent-space prior.These approaches emphasize generation guided by learned features or latent optimization.
  • Autoregression: Autoregressive models represent images pixel by pixel, with PixelRNN and PixelCNN producing excellent samples.Autoregression follows a different modeling strategy from VAE and GAN.
  • Proposed model: CVAE-GAN differs structurally from these models by combining statistic and pairwise feature matching for faster and more stable convergence.The comparison is illustrated in Figure 2.

3. Our Formulation of CVAE-GAN

CVAE-GAN combines encoder, generator, discriminator, and classifier networks for conditioned fine-grained image generation. Its asymmetric mean feature matching and encoder-based pairwise matching address unstable gradients, mode collapse, diversity, and structural preservation.

  • 3. Our Formulation of CVAE-GAN: CVAE-GAN contains encoder E, generative G, discriminative D, and classification C networks trained end-to-end.E maps x to z; G generates x′; D distinguishes real from fake; C measures class probabilities.
  • 3. Our Formulation of CVAE-GAN: The encoder and generator follow CVAE distributions, while the generator and discriminator retain the adversarial real-versus-generated data relationship.E learns P(z|x,c), and G samples images from P(x|z,c).
  • 3.1. Mean feature matching based GAN: Mean feature matching makes the generator align synthesized and real feature centers, replacing its cross-entropy objective while retaining discriminator training.The features are taken from an intermediate discriminator layer; minibatch means and moving historical averages estimate the centers.
  • 3.1. Mean feature matching based GAN: The asymmetric objective addresses gradient vanishing, becomes zero when generated images are good enough, and avoids WGAN parameter clipping.The discriminator is updated with Eq. 1 and the generator with Eq. 2.
  • 3.2. Mean Feature Matching for Conditional Image Generation: For conditional generation, classifier features are matched by category because using classifier softmax loss for the generator would reproduce gradient instability.C outputs class probabilities over K fine-grained categories, while G uses mean feature matching on an intermediate classifier layer.
  • 3.3. Pairwise Feature Matching: Mean feature matching reduces but does not eliminate mode collapse, so encoder-based pairwise matching links latent vectors to real images and preserves structure.The method adds pixel-wise ℓ2 reconstruction and pair-wise feature losses between x and x′.
  • 3.4. Objective of CVAE-GAN: The full CVAE-GAN objective combines the component losses into one training objective for the proposed pipeline.The paper presents the overall loss after introducing mean matching, conditional generation, and pairwise feature matching.

4. Analysis of Toy Example

The toy experiment compares traditional GAN, WGAN, and mean feature matching based GAN under shared settings for learning a ring-shaped real distribution. The proposed method learns the real distribution fastest, while the baselines show mode-collapse or early-training difficulties.

  • Experimental setup: The experiment models real data as a ring centered at (100, 100) and compares traditional GAN, WGAN, and mean feature matching based GAN.All three models use the same settings, including MLP architectures, RMSProp, learning rate 0.00005, and 2M training iterations.
  • Observed behavior: Traditional GAN samples occupy only a limited region of the real distribution throughout training, demonstrating mode collapse.
  • Observed behavior: WGAN fails to learn the real distribution at early iterations, with clamp values causing either gradient vanishing or divergence when poorly chosen.
  • Observed behavior: The proposed feature matching based GAN correctly learns the real data distribution fastest among the compared models.

5. Experiments

Experiments on faces, flowers, and birds evaluate CVAE-GAN for fine-grained generation, image quality, attribute morphing, inpainting, and face-recognition augmentation. Across these settings, the model produces realistic, diverse, category-preserving images and improves downstream recognition accuracy.

  • Experimental setup: Experiments use FaceScrub, 102 Category Flower, and CUB-200 datasets for category-conditioned image generation.All methods use the same network structure and training data, with 128 × 128 images.
  • Visualization comparison: CVAE-GAN samples are realistic, diverse, and preserve category identity better than CVAE, traditional CGAN, and FM-CGAN.CVAE outputs are blurry, traditional CGAN shows limited within-category variation, and FM-CGAN can lose object structure.
  • Attributes morphing: Interpolating between encoded latent vectors produces gradual changes in pose, emotion, color, and flower number within a category.The interpolation is z = αz1 +(1−α)z2, with α ∈[0, 1].
  • Image inpainting: For inpainting, CVAE-GAN encodes a masked image, synthesizes a reconstruction, and can iteratively refine the result.Experiments corrupt a 50 × 50 patch of a 128 × 128 image, and later iterations improve the result.
  • Loss ablation: Removing adversarial, pixel-level, or classification feature losses respectively causes blur, detail loss, or category-information loss.The complete combination of LG(img), LG(D), and LG(C) produces the best reconstructed samples.
  • Data augmentation: Adding synthesized faces to Facescrub training data improves LFW face-verification accuracy by about 1.0% over no augmentation.The result is reported for augmentation with new identities.

6. Conclusion

The paper concludes that CVAE-GAN generates varied fine-grained objects across three datasets and supports several image-generation applications. It also identifies generation of samples from unknown categories as future work.

  • Conclusion: CVAE-GAN generates varied fine-grained images across three different datasets.The conclusion names human faces, flowers, and birds through the evaluated datasets and emphasizes broad object-generation ability.
  • Applications: The model supports image generation, attribute morphing, image inpainting, and data augmentation for face-recognition training.These applications are presented as supported uses of the proposed method.
  • Future work: Future work will explore generating samples for unknown categories, such as faces of people absent from the training dataset.This extends beyond the fine-grained categories represented during training.
Loading 1703.10155v2…