Source-linked AI summary
Attribute2Image: Conditional Image Generation from Visual Attributes
Xinchen Yan, Jimei Yang, Kihyuk Sohn, Honglak Lee
TL;DR
The paper addresses generating natural images from visual attributes, an incomplete description of the desired image. It learns layered conditional variational auto-encoders with disentangled foreground and background factors and uses optimization-based posterior inference; experiments on faces and birds show realistic, diverse generation and improved bird-image quality.
Problem
The paper studies the under-explored problem of generating natural images from incomplete visual attributes rather than complete descriptions.
Method
The paper learns conditional variational auto-encoders with a layered representation that disentangles foreground and background latent variables, then performs posterior inference by optimization.
Results
Experiments on faces and birds produce realistic, diverse samples with disentangled representations, while disCVAE significantly improves generation quality on bird images.
Takeaways & Limitations
The learned models support attribute-conditioned image reconstruction and completion on novel images through optimization-based posterior inference.
Takeaways & Limitations
Training the layered model assumes that the foreground layer and gating variable are observable during training.
Abstract
from arXiv · showhide
This paper investigates a novel problem of generating images from visual attributes. We model the image as a composite of foreground and background and develop a layered generative model with disentangled latent variables that can be learned end-to-end using a variational auto-encoder. We experiment with natural images of faces and birds and demonstrate that the proposed models are capable of generating realistic and diverse samples with disentangled latent representations. We use a general energy minimization algorithm for posterior inference of latent variables given novel images. Therefore, the learned generative models show excellent quantitative and visual results in the tasks of attribute-conditioned image reconstruction and completion.
1 Introduction
The paper introduces attribute-conditioned image generation from incomplete visual descriptions and develops layered models that separate foreground and background factors. It also proposes optimization-based posterior inference for evaluating reconstruction and completion on novel images.
- 1 Introduction: The paper formulates conditioned image generation from visual attributes extracted from natural-language descriptions.The attributes specify properties such as hair color, gender, age, and expression while latent factors support diverse samples.
- 1 Introduction: A layered foreground-background model disentangles latent variables for uncertain object properties and background variation.The background is generated first, while foreground variables combine with attributes to determine the object layer, shape, and background visibility.
- 1 Introduction: An optimization-based method performs posterior inference on latent variables for image reconstruction and completion on novel images.The paper presents posterior inference as a way to evaluate learned generative models beyond ordinary sampling.
- 1 Introduction: Experiments use LFW faces and CUB birds to assess attribute-conditioned generation, latent disentanglement, reconstruction, and completion.The evaluation compares generation with nearest-neighbor search and analyzes foreground-background layers and latent spaces.
- 1 Introduction: The approach learns conditional variational auto-encoders, including a disentangling layered model, for natural images.The contributions identify conditional VAEs as the solution framework and introduce a novel problem of generation from visual attributes.
2 Related Work
Prior work studied generative modeling, layered image representations, and multimodal image-text learning, but natural-image generation from incomplete visual attributes remained under-explored. The paper positions its task as more challenging than generation from complete descriptions or synthetic graphics codes.
- Image generation: Earlier graphics-based models generated rendered objects from complete descriptions using synthetic data, limiting direct applicability to natural images.These models relied on graphics engines that provided virtually unlimited training data or controlled variation pairs.
- Image generation: Generation from incomplete descriptions such as class labels or visual attributes remained under-explored and posed a more challenging task.The related work contrasts incomplete descriptions with complete object identity, viewpoint, and color specifications.
- Layered modeling of images: Layered image models had been used for segmentation, occlusion, inpainting, and denoising before being incorporated into generative image modeling.Prior layered generative approaches included gated restricted Boltzmann machines and explicit occlusion modeling.
- Multimodal Learning: Multimodal learning methods modeled joint image-text distributions or used bidirectional conditional prediction for shared representations.The cited approaches focused on combining image and text modalities rather than the paper’s attribute-conditioned natural-image generation task.
3 Attribute-conditioned Generative Modeling of Images
The paper develops conditional variational auto-encoders for generating images from attributes, extending them with a layered foreground-background representation and disentangled latent variables. The resulting disCVAE is trained with observable foreground and gating layers using a joint variational objective.
- 3.1 Base Model: Conditional Variational Auto-Encoder (CVAE): A conditional variational auto-encoder generates images x from attributes y and latent variables z, using a recognition model to approximate the posterior.Generation samples z from a prior and then generates x conditioned on y and z.
- 3.1 Base Model: Conditional Variational Auto-Encoder (CVAE): The CVAE objective combines KL regularization between the recognition and prior distributions with an expected reconstruction loss.The KL term reduces the gap between qφ(z|x,y) and pθ(z), while the likelihood term becomes a reconstruction loss such as ℓ2.
- 3.2 Disentangling CVAE with a Layered Representation: The layered model represents an image as foreground and background components combined through an occlusion or gating layer.The gating function determines background visibility, while 1 − g determines foreground visibility; imperfect masks motivate a more robust approximation.
- 3.2 Disentangling CVAE with a Layered Representation: Under stable lighting and distant backgrounds, disCVAE assigns foreground factors to zF and background factors to zB, with attributes also conditioning foreground generation.The foreground is generated from μθF(y,zF), the background from μθB(zB), and the gating layer from sθg(y,zF).
- 3.2 Disentangling CVAE with a Layered Representation: Training assumes the foreground layer and gating variable are observable, enabling end-to-end optimization of the joint likelihood with convolutional generation and recognition models.The variational posterior factorizes as qφ(zF,zB|xF,g,x,y)=qφ(zB|zF,xF,g,x,y)qφ(zF|xF,g,y).
4 Posterior Inference via Optimization
The paper addresses inference of latent variables for novel images after training an attribute-conditioned generator. It reformulates posterior inference as energy minimization, allowing the latent code to be optimized through the generation network.
- 4 Posterior Inference via Optimization: Posterior inference seeks the latent variable z that explains a novel image x given its attribute y, a problem not directly provided by straightforward generation.The inferred latent variables enable evaluation of the learned generative model on novel images.
- 4 Posterior Inference via Optimization: The method reformulates posterior inference as minimizing an energy combining image reconstruction loss with prior regularization.This formulation remains applicable when the generation likelihood is non-Gaussian or deterministic.
- 4 Posterior Inference via Optimization: Optimization back-propagates the energy through the neural generation function to the latent variable and is solved using ADAM.The approach uses the generation model for recognition rather than a conventional recognition model for generation.
5 Experiments
Experiments on faces and birds evaluate attribute-conditioned generation, progression, latent disentanglement, reconstruction, and completion. The layered model produces realistic, attribute-consistent images and improves handling of foreground-background structure, especially compared with vanilla CVAE.
- Attribute-conditioned image generation: Generated face samples are realistic and diverse, while disCVAE yields clearer hair-background boundaries than vanilla CVAE.The samples also better reflect input attributes than nearest neighbors from the training set.
- Attribute-conditioned image generation: DisCVAE generates clearer bird shapes and better attribute correspondence than vanilla CVAE, whose samples are blurry and sometimes blend with the background.Bird modeling is more challenging because shapes and color patterns are diverse and attributes are sparse and high-dimensional.
- Attribute-conditioned image progression: Attribute progression changes targeted visual properties while preserving unrelated appearance, demonstrating control by the input attributes.Changing gender or age alters identity-related appearance while preserving viewpoint, background color, and expression; changing expression, eyewear, or hair color produces mainly local differences.
- Latent space disentangling: Changing foreground and background latent variables produces distinct effects, supporting disentanglement of foreground and background factors.Varying zB changes backgrounds while keeping foreground regions similar, whereas varying zF changes foreground viewpoints while maintaining uniform backgrounds.
- Image reconstruction and completion: Attribute-conditioned models reconstruct faces and predict occluded regions well, while vanilla CVAE shows significant failures on bird images.Attributes improve expression and eyewear reconstruction, and pixel-level mean squared error evaluations highlight benefits from attribute conditioning and layered modeling.
6 Conclusion
The paper presents attribute-conditioned image generation with CVAEs and a disentangling layered representation. Experiments on faces and birds support realistic, diverse generation, while optimization-based posterior inference enables reconstruction and completion.
- The paper studies attribute-conditioned image generation and proposes CVAE-based models with a layered, disentangled representation.
- Experiments on faces and birds show that the models generate realistic samples with diverse appearance.
- The disentangling CVAE significantly improves generation quality on bird images.
- An optimization-based posterior inference approach is applied to image reconstruction and completion with quantitative evaluation.
A Derivation of disCVAE objective
The disCVAE objective models full, foreground, and mask observations with foreground and background latent variables. Its variational formulation combines a factorized generative model with an inference model that conditions background inference on foreground information.
- The objective uses x, xF, g, y, and z=[zF,zB] to represent the full image, foreground image, mask, attributes, and foreground/background latents.
- The joint conditional likelihood factorizes into full-image generation, foreground-and-mask generation, and independent priors for zF and zB.
- The recognition model factorizes background inference conditioned on foreground latent information from foreground recognition.
- Foreground-and-mask likelihood is decomposed into image likelihood plus λg-weighted mask likelihood, with reconstruction or cross-entropy losses.
B Network Architecture for disCVAE
disCVAE uses separate foreground and background recognition and generation networks to implement its disentangled architecture. The encoders support training, while the decoders generate images after training.
- disCVAE consists of four convolutional neural networks: foreground and background networks for both recognition and generation.
- The foreground encoder uses five convolution layers followed by two fully connected layers, and merges the attribute stream at recognition-network output.
- Background networks use the same encoder-decoder design with fewer channels and introduce attributes and zF into background recognition.
- Encoder networks are used only during training; trained image generation uses decoder networks alone.
C Quantitative Analysis: Attribute Similarity, Labeled Faces in the Wild
Attribute-conditioned generation is evaluated by predicting attributes from generated samples and comparing them with reference attributes and nearest-neighbor baselines. Generated samples are closer to test references, with explicit foreground-background modeling improving attribute accuracy.
- A separately trained convolutional attribute regressor evaluates whether generated images capture their conditioning attributes.
- For each test attribute vector, ten generated samples are evaluated using cosine similarity and mean squared error against predicted attributes.
- Generated samples are quantitatively closer to reference attributes than nearest-neighbor samples from the training set.
- Explicit foreground-background modeling produces more accurate samples in attribute space.