Source-linked AI summary
Disentangled Person Image Generation
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, Mario Fritz
TL;DR
Generating realistic person images while independently controlling foreground, background, and pose remains difficult. The paper learns disentangled factor representations and adversarial mappings, enabling targeted manipulation and sampling, with experiments showing realistic outputs and potential re-identification benefits.
Problem
Existing person-image generation methods provide limited control over independently changing foreground, background, and pose according to user intent.
Method
A two-stage pipeline disentangles foreground, background, and pose into embeddings, then adversarially maps Gaussian noise into each embedding space for reconstruction and sampling.
Results
Experiments on Market-1501 and DeepFashion show realistic images, successful factor manipulation and interpolation, and beneficial use of generated data for person re-identification.
Takeaways & Limitations
Disentangled factor sampling provides more control over person-image generation and supports constructing virtual data for person re-identification.
Takeaways & Limitations
Using vanilla GAN loss caused model collapse, so the method relies on its chosen adversarial training strategy.
Abstract
from arXiv · showhide
Generating novel, yet realistic, images of persons is a challenging task due to the complex interplay between the different image factors, such as the foreground, background and pose information. In this work, we aim at generating such images based on a novel, two-stage reconstruction pipeline that learns a disentangled representation of the aforementioned image factors and generates novel person images at the same time. First, a multi-branched reconstruction network is proposed to disentangle and encode the three factors into embedding features, which are then combined to re-compose the input image itself. Second, three corresponding mapping functions are learned in an adversarial manner in order to map Gaussian noise to the learned embedding feature space, for each factor respectively. Using the proposed framework, we can manipulate the foreground, background and pose of the input image, and also sample new embedding features to generate such targeted manipulations, that provide more control over the generation process. Experiments on Market-1501 and Deepfashion datasets show that our model does not only generate realistic person images with new foregrounds, backgrounds and poses, but also manipulates the generated factors and interpolates the in-between states. Another set of experiments on Market-1501 shows that our model can also be beneficial for the person re-identification task.
1. Introduction
The paper addresses limited user control in realistic person-image generation by disentangling images into foreground, background, and pose factors. Its two-stage framework learns manipulatable embeddings and maps Gaussian noise to generate and control novel person images.
- Motivation: Existing image-generation methods provide tools for realistic synthesis, but users often cannot directly control attributes such as background, appearance, clothing, or viewpoint.The paper motivates controllable generation for applications including image editing, person re-identification, inpainting, and on-demand art.
- Problem and Contribution: The proposed task generates natural person images by explicitly representing user intentions through foreground, background, and pose manipulations.These factors are treated as weakly correlated and can be manipulated individually or jointly.
- Problem and Contribution: Together, the framework enables controlled generation of novel person images by sampling or manipulating disentangled embeddings for the three factors.The introduction identifies manipulatable embeddings as the central contribution of the two-stage approach.
- Two-Stage Framework: Stage I disentangles an input person image into embedding features for foreground, background, and pose using a multi-branched reconstruction network.The factor embeddings are learned so the input image can be reconstructed from their composition.
- Two-Stage Framework: Stage II learns adversarial mapping functions that transform Gaussian noise into the learned factor-embedding distributions for novel image generation.The framework first generates embedding features from noise and then uses the stage-I representation to generate images from those features.
2. Related work
Prior work has explored realistic image generation from noise, conditional person-image synthesis, and disentangled or unsupervised representation learning. This paper instead targets unpaired, self-supervised disentangled person image generation by jointly sampling foreground, background, and pose for greater control.
- Image generation from noise: GANs, AAEs, VAEs, and autoregressive models have been used to synthesize realistic images, often mapping noise-generated distributions to real-data distributions.Convolutional VAEs and AAEs also transform autoencoders into generators.
- Conditional image generation: Conditional person-image generation has used pose keypoints, text, body and clothing segments, segmentation masks, and image conditions to control synthesis or pose transfer.Examples include MPI Human Pose generation, full-body clothing synthesis, multi-view generation, and flexible human-pose transfer.
- Unsupervised representation learning: Unsupervised representation-learning methods have leveraged spatial context and jigsaw-puzzle structure to learn from image context without direct supervision.These approaches use relative positions of neighboring patches or all puzzle tiles as supervisory signals.
- Research gap: This paper addresses a different problem through unpaired, self-supervised disentangled person image generation that jointly samples foreground, background, and pose to increase generation control.The approach leverages simultaneous sampling of these three factors rather than the supervised or unsupervised approaches discussed above.
- Disentangled image generation: Disentangled generation methods such as InfoGAN learn unsupervised representations separating factors including writing styles, pose, lighting, and background digits.InfoGAN extends GANs using mutual information and demonstrates factor separation across MNIST, rendered 3D images, and SVHN.
3. Method
The method uses a two-stage pipeline to disentangle foreground, background, and pose into embedding features, reconstruct the input image, and learn adversarial mappings from Gaussian noise to each feature distribution. Combining these mappings with stage-I decoders enables independent factor sampling and composition of new person images.
- Stage-I reconstruction: Stage-I uses a multi-branched reconstruction network to encode foreground, background, and pose separately, then decode their embedding features to recover the input image.The coarse pose mask guides information flow for separating foreground and background, while pose heatmaps and the mask guide the multi-branched network.
- Stage-II feature mapping: Stage-II learns adversarial mapping functions Φ from Gaussian noise to the foreground, background, and pose embedding-feature distributions after stage-I reconstruction is trained.The reconstruction network is fixed during stage-II training, and the mapping functions are optimized with Wasserstein GAN loss.
- Factor sampling: The sampling pipeline combines stage-I decoders with stage-II mappings to transform Gaussian noise into new foreground, background, or pose factors and compose person images.Foreground and background use convolutional U-Net decoders, whereas pose uses a fully connected decoder.
- Network architecture: Each stage uses residual-block architectures, with convolutional branches for foreground and background, fully connected pose reconstruction, and fully connected Gaussian-to-embedding mappings.Foreground and background decoders are U-Net-based, pose encoding and decoding use four fully connected residual blocks, and each stage-II mapping uses four such blocks.
- Training objectives: Stage-I optimizes image reconstruction with L1 and adversarial losses, while pose reconstruction uses L2 loss for keypoint coordinates and visibility information.The L1 term controls low-frequency similarity to the input image, and adversarial training is used for sharper, more realistic reconstruction.
4. Experiments
Experiments on Market-1501 and DeepFashion show that the pipeline supports controllable factor manipulation, realistic person-image generation, interpolation, sampling, and person re-identification. The method produces more realistic samples than comparison models and provides discriminative features and virtual training pairs for re-ID.
- Datasets and evaluation: Experiments use Market-1501 and DeepFashion to evaluate factor manipulation, pose-guided generation, sampling, interpolation, and person re-ID.Market-1501 contains 32,668 images of 1,501 persons; DeepFashion provides 15,079 training and 7,996 testing images after filtering.
- Factor manipulation: The framework successfully manipulates foreground, background, and pose by sampling factors from Gaussian signals or conditioning them on input images and poses.Examples include one-factor sampling and two-factor conditioning for each of the three disentangled factors on Market-1501.
- Pose-guided generation: Compared with PG2, the method generates more realistic details with fewer artifacts, including better-shaped limbs and clearer hair details on DeepFashion.These qualitative observations agree with the Inception Score and mask Inception Score results reported in Table 1.
- Image sampling: Compared with VAE and DCGAN, the model generates more realistic person images, while BodyROI7 produces sharper images with more natural body-part colors.VAE produces blurry images, whereas DCGAN produces sharp but unrealistic images; using a real pose further supports sampling quality for re-ID.
- Person re-ID: The BodyROI7 model achieves 0.338 and 0.355 (with PCA) rank-1 performance using embedding features, exceeding WholeBody and matching unsupervised baselines.Generated virtual image pairs also achieve rank-1 performance 0.338 when training the re-ID model.
5. Conclusion
The paper proposes a novel two-stage pipeline for person image generation that disentangles foreground, background, and pose variations before learning to generate their embedding features from noise.
- Conclusion: The proposed pipeline addresses person image generation through two sequential stages.Stage-I reconstructs images from disentangled factor embeddings, while Stage-II learns to generate those embeddings from noise.
- Conclusion: Stage-I disentangles and encodes foreground, background, and pose variations into embedding features, then decodes them back into an image with a multi-branched reconstruction network.
- Conclusion: Stage-II adversarially learns mapping functions from noise distributions to feature-embedding distributions, guided by the decoders learned in Stage-I.
A. Network architecture
The stage-I architecture uses separate encoders for foreground and background, a U-net-based joint decoder, and a fully connected pose auto-encoder.
- Stage-I architecture: Stage-I comprises five convolutional residual blocks each for the foreground and background encoders, a U-net-based foreground-and-background decoder, and a fully connected pose auto-encoder.These four architectures are used in the two-stage framework on Market-1501.
B. Training details
Training uses dataset-specific factor decompositions and stage-wise minibatch and iteration schedules, with Adam optimization and alternating adversarial updates.
- Market-1501: On Market-1501, the method disentangles foreground, background, and pose using separate stage-I and stage-II training schedules.Foreground and background use minibatches of 16 for ∼70k stage-I iterations and 32 for ∼30k stage-II iterations; pose uses 64 for ∼30k and 32 for ∼60k.
- DeepFashion: On DeepFashion, clean backgrounds reduce disentanglement to appearance and pose, trained with dataset-specific stage-wise schedules.Appearance uses minibatches of 6 for ∼100k stage-I iterations and 16 for ∼60k stage-II iterations; pose uses 32 for ∼30k and 32 for ∼60k.
- Optimization: Both datasets use Adam with β1 = 0.5, β2 = 0.999, and an initial learning rate of 2e-5, while adversarial discriminator and generator updates alternate.The discriminator and generator are optimized alternatively during adversarial training.
C. Image manipulation results
The DeepFashion results extend the main paper with appearance and pose sampling experiments. Each factor is sampled from Gaussian noise while the remaining factors are fixed using embeddings extracted from real data.
- Sampling results: The DeepFashion experiments report results for both appearance sampling and pose sampling.These results are presented in Fig. 11 and Fig. 12 as an extension of Fig. 1 in the main paper.
- Sampling procedure: For each manipulated factor, the embedding feature is sampled from Gaussian noise.The sampling procedure follows the method explained in Sec. 4.2 of the main paper.
- Factor control: The other factors are fixed using embedding features extracted from real data.This setup isolates appearance or pose sampling while preserving the remaining factors.
D. Pose-guided person image generation results
The pose-guided generation results extend the main paper’s Fig. 5 by showing one appearance rendered in various real poses randomly selected from DeepFashion.
- Pose-guided person image generation results: Figure 13 extends the pose-guided person image generation results presented in Figure 5 of the main paper.The figure provides additional generated results.
- Pose-guided person image generation results: The results preserve one appearance while varying the person’s pose.This demonstrates pose variation under a fixed appearance.
- Pose-guided person image generation results: The varied poses are real poses selected randomly from the DeepFashion dataset.The pose examples are drawn from DeepFashion rather than being manually specified.
E. Inverse interpolation results
The method recovers Gaussian codes for two input images and generates intermediate states between images of the same person. Inverse interpolation between different people is more challenging.
- E. Inverse interpolation results: The method successfully generates intermediate states between two images of the same person by recovering their corresponding Gaussian codes.For input images x1 and x2, the method finds Gaussian codes z1 and z2 before performing inverse interpolation.
- E. Inverse interpolation results: Inverse interpolation between images of different persons is more challenging than interpolation between images of the same person.The different-person case is illustrated in Fig. 14(c), whereas same-person results appear in Fig. 14(a)(b).
F. Image sampling results
Sampling results show that the proposed model generates more realistic person images than VAE and DCGAN, with body-ROI conditioning improving sharpness and color naturalness.
- Market-1501 sampling: The proposed model generates more realistic person images than VAE and DCGAN on Market-1501 sampling results.VAE produces blurry images, whereas DCGAN produces sharp but unrealistic person images.
- Market-1501 sampling: Using body ROI produces sharper, more realistic images with more natural colors on each body part.This improvement is observed when comparing the whole-body and BodyROI7 variants.
- DeepFashion sampling: DeepFashion results sample varied appearance factors with pose fixed and varied pose factors with appearance fixed.The results also generate one appearance under various poses.
- Market-1501 interpolation: Inverse interpolation results cover images of the same person and different persons on Market-1501.The figure includes interpolation between two images, three images, and two different-person images.