Source-linked AI summary
Unsupervised Person Image Synthesis in Arbitrary Poses
Albert Pumarola, Antonio Agudo, Alberto Sanfeliu, Francesc Moreno-Noguer
TL;DR
The paper addresses arbitrary-pose person-image synthesis from a single image, an ill-posed problem involving occluded and ambiguous body and clothing regions. It proposes an unsupervised bidirectional GAN with content and style losses, and reports DeepFashion results close to fully supervised approaches despite documented failure cases.
Problem
Single-image arbitrary-pose rendering is severely ill-posed because viewpoint and pose changes create body and clothing ambiguities and self-occluded regions.
Method
A pose-conditioned bidirectional GAN renders the person into a desired pose, renders back to the original pose, and uses content-style losses to preserve identity and appearance without paired target images.
Results
DeepFashion evaluation with unlabeled data produces synthesis quality very similar to recent supervised methods and better than earlier VAE and CGAN implementations.
Takeaways & Limitations
The approach can generate substantially different poses, including views requiring hallucination of unseen body regions, while retaining high-frequency clothing details.
Takeaways & Limitations
The fully unsupervised setting produces texture-transfer, face, geometric, and body-part-as-texture errors.
Abstract
from arXiv · showhide
We present a novel approach for synthesizing photo-realistic images of people in arbitrary poses using generative adversarial learning. Given an input image of a person and a desired pose represented by a 2D skeleton, our model renders the image of the same person under the new pose, synthesizing novel views of the parts visible in the input image and hallucinating those that are not seen. This problem has recently been addressed in a supervised manner, i.e., during training the ground truth images under the new poses are given to the network. We go beyond these approaches by proposing a fully unsupervised strategy. We tackle this challenging scenario by splitting the problem into two principal subtasks. First, we consider a pose conditioned bidirectional generator that maps back the initially rendered image to the original pose, hence being directly comparable to the input image without the need to resort to any training image. Second, we devise a novel loss function that incorporates content and style terms, and aims at producing images of high perceptual quality. Extensive experiments conducted on the DeepFashion dataset demonstrate that the images rendered by our model are very close in appearance to those obtained by fully supervised approaches.
1. Introduction
The paper proposes a fully unsupervised GAN framework that generates images of a person in new camera views and body poses from a single photo. It uses bidirectional pose-conditioned rendering and a content-style loss to avoid paired training images while preserving appearance.
- The framework generates novel views and poses while synthesizing visible regions and hallucinating body parts or clothing absent from the input.It is designed to retain body shape and maintain texture consistency even when input and desired poses differ substantially.
- The model combines pose-conditional adversarial learning, Cycle-GAN-inspired bidirectional rendering, and content-style losses for perceptually convincing images.
- A pose-conditioned bidirectional architecture renders an input image to a desired pose and then back to the original pose for direct comparison.This two-stage strategy circumvents the need for paired images of the same person under different poses.
- DeepFashion experiments using unlabeled data produce results comparable to recent fully supervised approaches.
2. Related Work
Arbitrary-pose rendering from one person image is severely ill-posed because pose and viewpoint changes create appearance ambiguities and self-occlusions. Prior work includes generative models and conditional image-translation methods, but Cycle-GAN approaches do not perform the spatial pixel transformations required for multi-view generation.
- Single-image arbitrary-pose rendering must resolve clothing and body-shape ambiguities, missing regions from self-occlusion, and multiple sources of prior knowledge.Relevant priors include body shape, kinematic constraints, hair dynamics, cloth texture, reflectance, and fashion patterns.
- Early solutions reconstructed 3D objects before synthesizing target views, but were limited to rigid objects represented by CAD models or simple geometric primitives.
- Deep generative modeling brought variational autoencoders, autoregressive models, and GANs into image synthesis.
- Conditional GAN research has used labels, text, images, and multimodal inputs to constrain generation across image-to-image translation and related tasks.
- Cycle-GAN methods learn mappings between image distributions without paired examples, but do not model spatial pixel transformations needed for multi-view generation.Consequently, their strategies and architectures differ substantially from those for multi-view person synthesis.
3. Problem Formulation
The task is to learn an unsupervised mapping from a single-view person image and an original pose to the same person under a desired pose. Poses are represented as 2D skeletons, and training omits the ground-truth target image.
- The goal is photo-realistic pose transformation that preserves the person’s identity and clothing appearance.
- The model overview includes a generator, discriminator, 2D pose regressor, and pre-trained feature extractor, with no ground-truth image or labels.
- The desired output is defined by the mapping (Ipo, pf) → Ipf, where the input is an image and desired pose.
- Each pose is represented by a 2D skeleton with N = 18 joints, each specified by its image-plane pixel location.
- Training uses single-person images without the ground-truth output image Ipf.
4. Method
The method combines a pose-conditioned bidirectional generator, pose estimation, adversarial training, and identity-preserving content-style losses to synthesize a person in a desired pose without ground-truth target images.
- Architecture: The model uses a generator twice: first to render the person in a desired pose, then to render that result back to the original pose.This creates a recovered image that can be compared with the input image.
- Pose representation: The pose representation concatenates Gaussian belief maps for all detected skeleton joints.Each joint receives a probability map over image pixels, and the full pose is their concatenation.
- Architecture: The generator receives the input image and desired pose, while the discriminator evaluates overlapping image patches for realism and high-frequency correctness.A PatchGAN discriminator produces a realness probability for each 26×26 output location.
- Training losses: The full objective combines image adversarial, conditional pose, and identity losses.The identity term is a weighted combination of content and style losses, and additional recovered-image terms improve robustness and convergence.
- Training losses: Adversarial training matches generated-image and training-image distributions, while the pose loss encourages generated images to match the desired pose.The discriminator distinguishes real from rendered images, and the pose regressor supplies the conditional pose error.
- Identity preservation: The identity loss preserves semantic content through render-back comparison and preserves visible textures through pose-conditioned patch-style matching.Content compares pretrained feature activations of the original and recovered images; style compares Gram matrices for visible joint patches.
5. Implementation Details
Training uses least-squares adversarial loss and pretrained VGG16 features, with Adam optimization over 300 epochs and a learning-rate schedule.
- Optimization: Least-squares adversarial loss is used to reduce model oscillation and improve photo-realism.The image features come from VGG16 at layer z = 7.
- Optimization: The model is trained with Adam using learning rates 0.0002 for the generator and 0.0001 for discriminators, batch size 12, and 300 epochs.The learning rate decreases linearly after epoch 100.
6. Experimental Evaluation
The model is evaluated on DeepFashion using quantitative metrics and qualitative comparisons, including challenging pose changes, hallucinated body parts, and texture preservation. Results are promising and comparable to supervised approaches, while failure cases reveal texture-mapping, face, geometric, and background limitations.
- Benchmark: The evaluation uses DeepFashion images resized to 256×256, with ground-truth pose images reserved for quantitative testing rather than training.Training images are associated with desired 2D poses, but no ground-truth warped image is used during training.
- Quantitative results: SSIM and IS scores place the unsupervised approach on par with supervised methods, although differing test splits make the comparison indicative.The approach also outperforms the reported VAE and CGAN implementations.
- Qualitative results: Qualitative results show generalization to poses substantially different from the input, hallucination of unseen body parts, and rendering of high-frequency clothing textures.Examples include 180-degree pose changes, synthesized legs or arms, and detailed blouse or shirt patterns.
- Failure cases: The model exhibits four recurring failure types: incorrect texture transfer, incomplete face removal, geometric pose errors, and body parts mapped as texture.These failures are illustrated in Figure 4.
- Background robustness: With background-containing inputs, the person is rendered relatively consistently but the background becomes over-smoothed.Robust background handling would require more complex datasets and specialized loss functions.
- Loss analysis: An L1-trained variant fails to capture person identity, motivating the identity-focused loss used by the proposed model.Figure 5 compares synthesized and cycle images from the L1-loss setting with the identity-preserving results.
7. Conclusion
The paper presents a fully unsupervised GAN approach for generating images of people in arbitrary poses, with evaluation on DeepFashion showing promising results. It also identifies complex backgrounds as an important issue for future work.
- The proposed GAN generates images of a person under arbitrary poses and can be trained fully unsupervised.
- DeepFashion evaluation shows promising quantitative and qualitative results, including for poses that differ substantially from the input and require hallucinating large image regions.
- Robust handling of complex backgrounds remains an open issue requiring more complex datasets and specialized loss functions.
- Testing on images with backgrounds shows fairly consistent person rendering, but the backgrounds become over-smoothed.