Source-linked AI summary
Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
Haoye Dong, Xiaodan Liang, Ke Gong, Hanjiang Lai, Jia Zhu, Jian Yin
TL;DR
Pose-guided person image synthesis is challenged by large geometric transformations that conventional local-convolution models do not adequately model. The paper introduces Soft-Gated Warping-GAN, which predicts target part layouts and warps condition-image features to render them. Experiments report state-of-the-art performance and the best human perceptual score on two large datasets.
Problem
Existing person-image synthesis models struggle with large geometric transformations, including spatial displacements and distinct changes across body parts under arbitrary poses.
Method
Soft-Gated Warping-GAN first predicts a pose-conditioned target part segmentation map, then uses a soft-gated warping-block to map condition-image features into that layout.
Results
Soft-Gated Warping-GAN significantly outperforms existing methods qualitatively and quantitatively, achieving state-of-the-art performance on two large datasets and the best human perceptual score.
Takeaways & Limitations
The approach provides pose-dependent transformation control while rendering detailed appearances through higher-level part structure and warped deep features.
Abstract
from arXiv · showhide
Despite remarkable advances in image synthesis research, existing works often fail in manipulating images under the context of large geometric transformations. Synthesizing person images conditioned on arbitrary poses is one of the most representative examples where the generation quality largely relies on the capability of identifying and modeling arbitrary transformations on different body parts. Current generative models are often built on local convolutions and overlook the key challenges (e.g. heavy occlusions, different views or dramatic appearance changes) when distinct geometric changes happen for each part, caused by arbitrary pose manipulations. This paper aims to resolve these challenges induced by geometric variability and spatial displacements via a new Soft-Gated Warping Generative Adversarial Network (Warping-GAN), which is composed of two stages: 1) it first synthesizes a target part segmentation map given a target pose, which depicts the region-level spatial layouts for guiding image synthesis with higher-level structure constraints; 2) the Warping-GAN equipped with a soft-gated warping-block learns feature-level mapping to render textures from the original image into the generated segmentation map. Warping-GAN is capable of controlling different transformation degrees given distinct target poses. Moreover, the proposed warping-block is light-weight and flexible enough to be injected into any networks. Human perceptual studies and quantitative evaluations demonstrate the superiority of our Warping-GAN that significantly outperforms all existing methods on two large datasets.
1 Introduction
Person image synthesis remains difficult under large pose-induced geometric changes because existing convolution-based models overlook spatial misalignment and higher-level part layouts. Soft-Gated Warping-GAN addresses this with pose-guided parsing and feature warping, achieving superior qualitative and quantitative results.
- Large pose changes create substantial appearance diversity and spatial layout transformations across clothes and body parts.
- Conventional convolution-based methods overlook large spatial misalignment and higher-level part-level structure layouts, producing artifacts and missing clothing appearance.
- Soft-Gated Warping-GAN synthesizes a target part segmentation map, then warps condition-image features into that target layout for appearance rendering.
- Soft gating controls transformation degree according to pose changes, while attention layers adaptively select effective feature maps for warping.
- Soft-Gated Warping-GAN significantly outperforms existing state-of-the-art methods qualitatively and quantitatively, especially for large pose variation.
2 Relation Works
Related work applies GANs and convolutional architectures to image-to-image translation and person image synthesis. Prior person-synthesis methods use pose keypoints, disentangled features, or deformable skip connections.
- Image Synthesis: DCGANs combine convolutional neural networks with GANs to generate realistic images, while pix2pix learns conditional image-to-image mappings.
- Image Synthesis: CycleGAN, DiscoGAN, and DualGAN learn unsupervised mappings between image domains.
- Person Image Synthesis: PG2 synthesizes person images from pose keypoints, and BodyROI7 uses disentangle and restructure methods with different sampling features.
- Person Image Synthesis: DSCF uses a U-Net with deformable skip connections to synthesize person images from decomposed and deformable images.
3 Soft-Gated Warping-GAN
Soft-Gated Warping-GAN decomposes pose-guided person image synthesis into pose-guided parsing and Warping-GAN rendering. It predicts target part layouts, estimates geometric mappings, warps deep features with soft gating, and renders region-specific textures under pose constraints.
- Architecture: The two-stage pipeline first predicts target human parsing from the target pose and condition parsing, then renders textures with a Warping-GAN.The synthesized parsing provides spatial constraints, while stage II jointly uses it with the condition image and target pose.
- Stage I: Pose-Guided Parsing: The pose-guided parser represents target person parts as pixel-wise class labels and 20-channel one-hot masks for class-level image guidance.It is trained on LIP, whose images contain 20 semantic labels, because DeepFashion and Market-1501 lack parsing labels.
- Stage II: Warping-GAN Rendering: Region-wise learning renders texture details in specific segmentation regions, using the synthesized parsing to guide appearance of parts such as faces, clothes, and hands.The final generator combines warped feature maps, pose information, and synthesized parsing; the discriminator uses pyramidal hierarchy features.
- Stage II: Warping-GAN Rendering: The geometric matcher estimates transformations between condition and synthesized parsing using feature extractors, matching, regression, and affine/TPS mapping.The condition image is warped with the resulting transformation grid, without requiring the target image at test time.
- Stage II: Warping-GAN Rendering: The soft-gated warping-block transforms deep feature maps rather than raw pixels using a residual feature pathway controlled by a gate e ∈ [0, 1].The block combines affine and Thin-Plate Spline transformations and adjusts transformation degree according to pose manipulation requests.
- Objective Functions: Training combines adversarial, pixel-wise, perceptual, and pyramidal hierarchy losses, with the generator objective formed as a weighted sum of these terms.The pyramidal hierarchy loss uses L1 differences between feature maps from multiple discriminator layers, weighted by α_i.
4 Experiments
Experiments evaluate Soft-Gated Warping-GAN on two benchmarks using quantitative metrics, human preference tests, qualitative comparisons, and ablations. The method consistently outperforms baselines on SSIM, achieves comparable IS results, receives stronger human preferences, and benefits from both pose-guided parsing and soft-gated warping.
- Experimental Setup: Experiments evaluate the method on DeepFashion and Market-1501 using SSIM, Inception Score, and Amazon Mechanical Turk perceptual studies.The study also includes qualitative comparisons and an ablation analysis.
- Quantitative Results: The proposed method consistently outperforms all baselines on SSIM across both datasets and achieves comparable IS results on both datasets.The authors interpret the IS results as confirming generalization ability.
- Human Perceptual Study: 96.3% of workers preferred the method over BodyROI7 on DeepFashion for realistic person-image generation.The pairwise tests presented two generated images and asked workers to select the more natural result.
- Qualitative Results: Qualitative comparisons show more detailed and precise pose-conditioned images, although some Market-1501 outputs remain blurry because low-resolution images may affect human parsing.Compared methods are described as producing blurry, coarse, or artifact-prone results in these comparisons.
- Ablation Study: Ablations show that pose-guided parsing improves generation and removing the soft-gated warping-block causes a dramatic performance drop.The reported results attribute the warping-block benefit to its transformation mechanism rather than merely to additional parameters.
5 Conclusion
Soft-Gated Warping-GAN addresses pose-guided person image synthesis by combining target part segmentation with feature-map warping. It controls transformation degrees for different poses and achieves state-of-the-art performance on two large datasets.
- The method uses a human parser to produce target part segmentation maps containing higher-level structure information for image synthesis.
- A soft-gated warping-block transforms feature maps to render textures into the predicted segmentation parts.
- The warping-block supports different transformation degrees according to the target pose while rendering realistic and natural results.
- Qualitative and quantitative experiments show state-of-the-art performance on two large datasets, with the best human perceptual score.
Supplementary
The supplementary figures compare the proposed method with PG2, DSCF, and BodyROI7 across DeepFashion and Market-1501. They also include an ablation study on DeepFashion.
- Figure 9 compares the method with PG2 on DeepFashion.
- Figures 10 and 11 compare the method with DSCF and BodyROI7, respectively, on DeepFashion.
- Figures 12–14 compare the method with PG2, DSCF, and BodyROI7, respectively, on Market-1501.