Source-linked AI summary
Pose Guided Person Image Generation
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, Luc Van Gool
TL;DR
The paper addresses person-image pose transfer while preserving identity appearance, a task made difficult by the need to generate pose and detailed appearance together. PG2 separates these goals into pose integration and adversarial image refinement, producing photo-realistic, pose-wise correct images across two person datasets, with failures on rare poses and imbalanced data.
Problem
Person pose transfer must preserve important appearance details while changing the person to a desired pose, but jointly generating correct pose and detailed appearance is challenging.
Method
PG2 uses a two-stage pipeline that first integrates a reference image with a target pose for coarse structure, then refines appearance details using adversarial training and masked L1 regularization.
Results
Experiments on Market-1501 and DeepFashion show photo-realistic, pose-wise correct person images, with the two-stage model retaining more appearance details than a one-stage model.
Takeaways & Limitations
Explicit pose conditioning and staged generation support person-image synthesis with controllable target poses and transferred clothing and facial details.
Takeaways & Limitations
Failures include incomplete legs for rare complicated poses, inaccurate gender under imbalanced training data, and clothing mixtures when similar garments overlap.
Abstract
from arXiv · showhide
This paper proposes the novel Pose Guided Person Generation Network (PG$^2$) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG$^2$ utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128$\times$64 re-identification images and 256$\times$256 fashion photos show that our model generates high-quality person images with convincing details.
1 Introduction
The paper frames pose transfer as controlled person-image generation conditioned on a reference image and a target pose. It addresses the difficulty of jointly producing correct pose and detailed appearance by separating these goals into stages.
- Motivation: Pose transfer is challenging because a complete end-to-end model must generate correct poses and detailed appearance simultaneously.
- Approach: PG2 divides generation into pose integration and image refinement, focusing first on global human structure and then on appearance details.
- Motivation: The proposed task conditions image generation on both a reference person image and an intended pose to manipulate the person into an arbitrary pose.The reference supplies appearance information, while the pose provides explicit and flexible control.
2 Related works
Related work spans deep generative models and conditional image generation. PG2 is situated among approaches that condition synthesis on labels, attributes, text, images, or structured body information.
- Generative image modeling: Deep generative image modeling includes variational autoencoders, autoregressive models, and generative adversarial networks.
- Conditional generation: PG2 combines reference-image conditioning with target-pose conditioning for person pose transfer.
- Generative image modeling: GANs have been widely used because adversarial training can generate sharp images.
- Conditional generation: Conditional generation methods use category or attribute labels, text, images, or body-part segments as conditioning information.
3 Method
PG2 uses a two-stage pipeline: Stage-I integrates a reference image with a target pose to produce coarse structure, while Stage-II refines appearance through conditional adversarial training.
- Stage-I: Pose integration: Stage-I integrates the conditioning image IA and target pose PB with a U-Net-like generator to produce a coarse image matching the target body structure.The pose is represented using 18 heatmaps derived from estimated keypoints.
- Stage-I: Pose integration: The pose mask loss weights foreground and background differently to reduce the influence of background changes during image comparison.The pose mask is approximately constructed to cover the target human body.
- Stage-I: Pose integration: Stage-I captures target-pose structure and low-frequency clothing color, but its L1-trained output is blurry and lacks high-frequency appearance details.
- Stage-II: Image refinement: Stage-II conditions a DCGAN-like generator on the first-stage result and reference image to refine missing appearance details.The generator learns an appearance difference map rather than synthesizing the target image entirely from scratch.
- Stage-II: Image refinement: The discriminator evaluates pairs containing the conditioning image and generated or target image, while alternating optimization trains the refinement generator and discriminator.
- Stage-II: Image refinement: Stage-II combines adversarial and masked L1 losses, balancing sharper details against low-frequency similarity and artifact control through λ.Small λ increases artifact risk, whereas large λ favors blurry results.
4 Experiments
Experiments on DeepFashion and Market-1501 evaluate pose embeddings, poseMaskLoss, and two-stage refinement through qualitative, quantitative, and user-study analyses. The full two-stage model produces more detailed and photo-realistic results, while evaluation also exposes metric and data limitations.
- Experimental setup: PG2 is evaluated on DeepFashion and Market-1501, with comparisons covering pose embeddings, losses, and one-stage versus two-stage generation.DeepFashion contains 256×256 fashion images, while Market-1501 contains 128×64 re-identification images with more varied backgrounds and viewpoints.
- Pose embeddings: The proposed heatmap pose embedding guides stage-I toward correctly posed images, whereas coordinate and alternative heatmap embeddings produce blurrier or incorrectly posed results.The proposed embedding also produces reasonable face details in some DeepFashion samples.
- Loss comparison: PoseMaskLoss sharpens Market-1501 outputs and improves body boundaries by reducing the influence of noisy backgrounds, while differences are smaller on simpler-background DeepFashion images.The paper reports that poseMaskLoss consistently improves quantitative measures across both datasets.
- Two-stage refinement: The two-stage adversarial model generates more details and more photo-realistic images than the one-stage adversarial model on both datasets.Examples include sharper clothing boundaries, retained textures and stripes, and clearer objects such as a stool; quality decreases on the more challenging Market-1501 setting.
- Quantitative and user evaluation: Quantitative scores can be similar for one-stage and two-stage adversarial models because blurry images may receive favorable SSIM despite appearing less realistic.The evaluation therefore combines SSIM, mask-SSIM, IS, mask-IS, qualitative comparisons, and an AMT user study.
- Failure analysis: Failure cases include incomplete legs for rare complicated poses, inaccurate gender from imbalanced training data, and clothing mixtures caused by similar-colored layered garments.These cases are reported on DeepFashion.
5 Conclusions
The paper presents PG2 for synthesizing a person in a target pose from a reference image. Its two-stage process combines global structure generation, poseMaskLoss, and adversarial detail refinement, producing photo-realistic and pose-wise correct images on two datasets.
- 5 Conclusions: PG2 synthesizes person images conditioned on a reference image and target pose using a two-stage generation process.Stage-I captures global structure and applies poseMaskLoss, while stage-II adds appearance details through adversarial training.
- 5 Conclusions: Experiments on two person datasets demonstrate images that are both photo-realistic and pose-wise correct.The paper identifies more controllable and diverse pose-and-attribute conditioning as future work.