Source-linked AI summary
Pose Guided Person Image Generation
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, Luc Van Gool
TL;DR
The paper addresses person-image pose transfer while preserving identity appearance, a task made difficult by the need to generate pose and detailed appearance together. PG2 separates these goals into pose integration and adversarial image refinement, producing photo-realistic, pose-wise correct images across two person datasets, with failures on rare poses and imbalanced data.
Problem
Person pose transfer must preserve important appearance details while changing the person to a desired pose, but jointly generating correct pose and detailed appearance is challenging.
Source: §1 · p. 1Method
PG2 uses a two-stage pipeline that first integrates a reference image with a target pose for coarse structure, then refines appearance details using adversarial training and masked L1 regularization.
Source: §1 · p. 2, §3.2 · p. 5Results
Experiments on Market-1501 and DeepFashion show photo-realistic, pose-wise correct person images, with the two-stage model retaining more appearance details than a one-stage model.
Source: §4.2 · pp. 6–7, §5 · p. 9Takeaways & Limitations
Explicit pose conditioning and staged generation support person-image synthesis with controllable target poses and transferred clothing and facial details.
Source: §1 · p. 1, §4.2 · pp. 6–7Failures include incomplete legs for rare complicated poses, inaccurate gender under imbalanced training data, and clothing mixtures when similar garments overlap.
Source: §4.5 · p. 9
Abstract
from arXiv · showhide
This paper proposes the novel Pose Guided Person Generation Network (PG$^2$) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG$^2$ utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128$\times$64 re-identification images and 256$\times$256 fashion photos show that our model generates high-quality person images with convincing details.
1 Introduction
The paper frames pose transfer as controlled person-image generation conditioned on a reference image and a target pose. It addresses the difficulty of jointly producing correct pose and detailed appearance by separating these goals into stages.
Motivation
- Pose transfer is challenging because a complete end-to-end model must generate correct poses and detailed appearance simultaneously.Source: §1 · p. 1
Approach
- PG2 divides generation into pose integration and image refinement, focusing first on global human structure and then on appearance details.Source: §1 · p. 2
Motivation
- The proposed task conditions image generation on both a reference person image and an intended pose to manipulate the person into an arbitrary pose.The reference supplies appearance information, while the pose provides explicit and flexible control.Source: §1 · pp. 1–2
2 Related works
Related work spans deep generative models and conditional image generation. PG2 is situated among approaches that condition synthesis on labels, attributes, text, images, or structured body information.
Generative image modeling
- Deep generative image modeling includes variational autoencoders, autoregressive models, and generative adversarial networks.Source: §2 · p. 2
Conditional generation
- PG2 combines reference-image conditioning with target-pose conditioning for person pose transfer.Source: Figure 2 · p. 3, §2 · p. 3
Generative image modeling
- GANs have been widely used because adversarial training can generate sharp images.Source: §2 · p. 3
Conditional generation
- Conditional generation methods use category or attribute labels, text, images, or body-part segments as conditioning information.Source: §2 · p. 3
3 Method
PG2 uses a two-stage pipeline: Stage-I integrates a reference image with a target pose to produce coarse structure, while Stage-II refines appearance through conditional adversarial training.
Stage-I: Pose integration
- Stage-I integrates the conditioning image IA and target pose PB with a U-Net-like generator to produce a coarse image matching the target body structure.The pose is represented using 18 heatmaps derived from estimated keypoints.Source: §3.1 · pp. 3–4
- The pose mask loss weights foreground and background differently to reduce the influence of background changes during image comparison.The pose mask is approximately constructed to cover the target human body.Source: §3.1 · p. 4
- Stage-I captures target-pose structure and low-frequency clothing color, but its L1-trained output is blurry and lacks high-frequency appearance details.Source: §3.1 · p. 4
Stage-II: Image refinement
- Stage-II conditions a DCGAN-like generator on the first-stage result and reference image to refine missing appearance details.The generator learns an appearance difference map rather than synthesizing the target image entirely from scratch.Source: §3.2 · pp. 4–5
- The discriminator evaluates pairs containing the conditioning image and generated or target image, while alternating optimization trains the refinement generator and discriminator.Source: §3.2 · p. 5
- Stage-II combines adversarial and masked L1 losses, balancing sharper details against low-frequency similarity and artifact control through λ.Small λ increases artifact risk, whereas large λ favors blurry results.Source: §3.2 · p. 5
4 Experiments
Experiments on DeepFashion and Market-1501 evaluate pose embeddings, poseMaskLoss, and two-stage refinement through qualitative, quantitative, and user-study analyses. The full two-stage model produces more detailed and photo-realistic results, while evaluation also exposes metric and data limitations.
Experimental setup
- PG2 is evaluated on DeepFashion and Market-1501, with comparisons covering pose embeddings, losses, and one-stage versus two-stage generation.DeepFashion contains 256×256 fashion images, while Market-1501 contains 128×64 re-identification images with more varied backgrounds and viewpoints.Source: §4 · p. 5, §4.1 · p. 6, §4.2 · p. 6
Pose embeddings
- The proposed heatmap pose embedding guides stage-I toward correctly posed images, whereas coordinate and alternative heatmap embeddings produce blurrier or incorrectly posed results.The proposed embedding also produces reasonable face details in some DeepFashion samples.Source: §4.2 · p. 6
Loss comparison
- PoseMaskLoss sharpens Market-1501 outputs and improves body boundaries by reducing the influence of noisy backgrounds, while differences are smaller on simpler-background DeepFashion images.The paper reports that poseMaskLoss consistently improves quantitative measures across both datasets.Source: §4.2 · p. 6, §4.3 · p. 8
Two-stage refinement
- The two-stage adversarial model generates more details and more photo-realistic images than the one-stage adversarial model on both datasets.Examples include sharper clothing boundaries, retained textures and stripes, and clearer objects such as a stool; quality decreases on the more challenging Market-1501 setting.Source: §4.2 · pp. 6–7
Quantitative and user evaluation
- Quantitative scores can be similar for one-stage and two-stage adversarial models because blurry images may receive favorable SSIM despite appearing less realistic.The evaluation therefore combines SSIM, mask-SSIM, IS, mask-IS, qualitative comparisons, and an AMT user study.Source: §4.3 · pp. 7–8, §4.4 · p. 8
Failure analysis
- Failure cases include incomplete legs for rare complicated poses, inaccurate gender from imbalanced training data, and clothing mixtures caused by similar-colored layered garments.These cases are reported on DeepFashion.Source: §4.5 · p. 9
5 Conclusions
The paper presents PG2 for synthesizing a person in a target pose from a reference image. Its two-stage process combines global structure generation, poseMaskLoss, and adversarial detail refinement, producing photo-realistic and pose-wise correct images on two datasets.
5 Conclusions
- PG2 synthesizes person images conditioned on a reference image and target pose using a two-stage generation process.Stage-I captures global structure and applies poseMaskLoss, while stage-II adds appearance details through adversarial training.Source: §5 · p. 9
- Experiments on two person datasets demonstrate images that are both photo-realistic and pose-wise correct.The paper identifies more controllable and diverse pose-and-attribute conditioning as future work.Source: §5 · p. 9