Source-linked AI summary
A Variational U-Net for Conditional Appearance and Shape Generation
Patrick Esser, Ekaterina Sutter, Björn Ommer
TL;DR
Deep generative models struggle with images involving large spatial deformations because direct synthesis does not capture the interplay of shape and appearance. This paper combines a shape-guided conditional U-Net with a variational appearance representation, enabling conditional generation and transfer across several datasets while improving the state of the art. Its scope is constrained by latent disentanglement assumptions and dataset biases such as rare subjects, pose-attribute correlations, and heavy viewpoint changes.
Problem
Existing generative models perform weakly on image distributions with large spatial variation because they directly synthesize objects without modeling the interplay of appearance and shape.
Method
A conditional U-Net maps shape estimates to images while conditioning on a variational autoencoder representation of appearance, trained end-to-end without paired varying-pose examples.
Results
Experiments on COCO, DeepFashion, shoes, Market-1501, and handbags significantly improve state-of-the-art conditional image generation and transfer.
Takeaways & Limitations
The model can retain shape or appearance while altering the other, and can sample appearance while preserving shape.
Takeaways & Limitations
Performance is constrained by dataset representation and correlations, including rare children, incorrect pose estimates, pose-gender correlations, and heavy viewpoint changes.
Abstract
from arXiv · showhide
Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO, DeepFashion, shoes, Market-1501 and handbags, the approach demonstrates significant improvements over the state-of-the-art.
1. Introduction
Image generation must model both an object's spatial shape and inherent appearance because these factors change differently under pose, translation, self-occlusion, color, and texture changes. The paper proposes a conditional U-Net guided by shape and conditioned on a variational appearance representation to generate and transfer images.
- Shape describes an object's geometry relative to viewpoint, whereas appearance includes inherent properties such as color and texture.
- Figure 1 illustrates appearance inferred from query images and synthesized in different poses.
- Spatial variation remains challenging because direct image synthesis does not model the interplay between appearance and shape.The paper contrasts recoloring attributes such as facial hair with physically moving an arm to a new position.
- The proposed conditional U-Net maps shape estimates to target images while conditioning generation on a variational autoencoder's latent appearance representation.Shape information can come from edges or automatically estimated body-joint locations.
- The model can retain either shape or appearance from a query image while altering, transferring, imputing, or sampling the other component.Appearance sampling uses the stochastic latent representation while preserving shape.
2. Related work
Prior methods provide varying degrees of control over appearance or shape, but commonly restrict appearance variety, pose control, spatial consistency, or training supervision. The paper positions its method as jointly controlling shape and appearance without requiring paired examples of the same appearance in different poses.
- Many prior methods control appearance through class labels, attributes, or textual descriptions but do not jointly provide equivalent shape control.
- Image-to-image methods can control shape, but examples range from a single uncontrollable appearance to varied appearances without changing a given appearance's pose.
- Segmentation-based multimodal synthesis can produce coarse appearance combinations, whereas the proposed method avoids limited-mode assumptions and includes appearance inference.
- Text-conditioned approaches specify appearance coarsely and have difficulty producing the desired shape consistently.
- Prior general-pose person rendering uses a fully supervised two-stage model requiring labeled examples of the same appearance in different poses.
3. Approach
The approach separates shape and appearance by conditioning a U-Net image generator on an estimated shape representation and a stochastic appearance latent variable. A conditional VAE learns appearance from still images while preserving spatial information through the U-Net architecture.
- Latent shape and appearance: The model represents an image through geometrical shape y and intrinsic appearance z, whose combination controls image generation.Shape captures location, shape, and pose, while appearance captures intrinsic characteristics such as color and texture.
- Latent shape and appearance: A standard VAE with two latent variables cannot reliably separate spatial shape information from appearance under a joint standard-normal prior.The paper therefore introduces additional shape information to disentangle y and z.
- Conditional VAE: An estimator e produces a shape estimate ˆy from x, using representations such as edges or automatically estimated body-joint locations.The appearance latent z is then inferred from both the image and the estimated shape.
- Conditional VAE: The conditional prior p(z|ˆy) captures potential interrelations between shape and appearance, while the encoder and generator are trained end-to-end with reparameterization.The paper gives the example that a person jumping may be less likely to wear a dinner jacket than a T-shirt.
- Generator: The generator Gθ maps ˆy and z to an image, using a U-Net whose skip-connections preserve spatial information from the shape input.The appearance representation is concatenated with the encoder bottleneck before decoding.
- Generator: The final objective combines latent-distribution regularization with perceptual reconstruction loss measured through VGG19 features.The perceptual term replaces reliance on pixelwise statistics alone, with layer weights λk controlling its contributions.
4. Experiments
Experiments across five datasets evaluate reconstruction, conditional generation, appearance transfer, generalization, and component ablations. The full model preserves shape while enabling varied appearance generation and transfer, whereas removing appearance information or the KL loss degrades the intended disentanglement.
- Experimental setup: The evaluation covers shoes, handbags, Market-1501, DeepFashion, and COCO, using pix2pix and PG2 as comparison baselines.Experiments use edge images or automatically regressed human body joints as shape estimates.
- Image reconstruction: Compared with pix2pix and PG2, the method achieves higher SSIM than both and improves on pix2pix while remaining comparable to PG2 in IS.These metrics are computed on reconstructed test images from DeepFashion and Market-1501.
- Generalization and transfer: The model generalizes to human-drawn shoe and handbag sketches, generating more coherent appearances with fewer artifacts than pix2pix.It also supports appearance transfer across different shapes and synthesis across a full 360° turn.
- Conditional generation: Sampling appearance from the learned prior produces diverse human images from stickman inputs while preserving the spatial layout better than pix2pix.The comparison is performed on Market-1501 and DeepFashion; pix2pix produces marginally diverse outputs.
- Ablation study: Without appearance information, the generator outputs the mean image for a shape and shows no appearance variation.This ablation models p(x|y) as a unimodal Laplace distribution.
- Ablation study: Replacing the VAE with a simple autoencoder removes the KL term, causing the network to copy and paste appearance inputs according to the shape estimate.The full model instead successfully performs appearance transfer.
5. Conclusion
The variational U-Net models the interplay of shape and appearance: its VAE samples appearance while the U-Net preserves object shape, improving conditional generation and transfer across datasets.
- The variational U-Net models the interplay of shape and appearance for conditional image generation.
- A variational autoencoder samples appearance while the U-Net preserves object shape.
- Experiments on several datasets and diverse objects demonstrate significant improvements over the state of the art in conditional image generation and transfer.
A. Network structure
The architecture visualization shows a network for 128 × 128 images with eight residual blocks, while the experiments generally use seven.
- Although the residual-block parameter can vary by dataset, all paper experiments set n to 7.
- The visualized network architecture generates 128 × 128 images with 8 residual blocks.
B. Examples of appearance sampling in different datasets
The model samples diverse appearances while conditioning on shape, using edge images for shoes and handbags and human body joints for COCO examples.
- The experiments condition on edge images of shoes and handbags while sampling appearance from the learned prior.
- For edge-conditioned examples, the first generated image uses original appearance and five subsequent images sample appearance from the learned prior.
- The model is compared with pix2pix, with odd rows showing pix2pix outputs and even rows showing samples from the proposed model.
- For stickman-conditioned examples, the first generated image uses original appearance and five subsequent images sample appearance from the learned prior.
C. Transfer of shape and appearance
The transfer experiments vary shape and appearance independently across videos, datasets, and object categories, without requiring labeled examples of identical appearances in different shapes.
- The approach transfers fixed appearances to different shapes and fixed shapes to different appearances across several datasets.
- The method does not require labeled examples showing the same appearance in different shapes, enabling application across a broad range of datasets.
- Video transfer infers appearance from COCO and estimates target shape from Penn Action sequences, generating each frame independently without temporal smoothing.
- Cross-dataset experiments transfer shoe appearances to handbags and handbag appearances to shoes using desired edge-defined shapes.
- Additional transfer examples cover COCO, DeepFashion, and Market-1501 datasets.
D. Quantitative results for the ablation study
The ablation study reports Inception Scores in Table 5, while the positive effect of KL-regularization is assessed qualitatively because IS cannot quantify it.
- The positive effect of KL-regularization cannot be quantified by the Inception Score.The paper therefore presents qualitative results in Fig. 9.
- Table 5 reports Inception Scores for the ablation study.
E. Limitations
Performance depends on dataset support and the relationship between appearance and pose. Failures arise from rare examples, pose-attribute correlations, heavy viewpoint changes, and occluded annotations.
- Generated-image quality depends highly on the dataset used for training.Appearance details are faithfully preserved when the dataset provides sufficient support for them.
- COCO's variation in visual quality and appearance leaves little overlap of appearance details across poses.The model therefore focuses on appearance aspects reusable across many poses in that dataset.
- Rare data, inaccurate pose estimates, and pose-correlated attributes produce failures across COCO, Market-1501, and DeepFashion.Examples include underrepresented children, incorrect leg estimates, and gender-associated pose patterns.
- Under heavy viewpoint changes, the model can fail when appearance is unrelated between views, such as a covered shirt becoming visible.It also becomes confused when occluded body parts are annotated despite being uncommon in training samples.
- Figure 19 collects dataset-specific failure cases across different datasets.