PISE: Person Image Synthesis and Editing with Decoupled GAN

1. Introduction

• Existing methods face challenges from coupled clothing shape and style, uncertain invisible regions, and lost spatial context.Sparse keypoint correspondences also make sharp, reasonable synthesis difficult when source and target poses differ substantially.Source: §1 · pp. 1–2
• PISE uses a two-stage model with per-region control to decouple clothing shape and style, with experiments showing flexibility and superior performance on pose transfer, texture transfer, and region editing.It first predicts a human parsing map to guide clothing shape, then generates the target image.Source: §1 · p. 2
• Joint global and local per-region encoding and normalization predicts styles for invisible regions while preserving the original style of clothing in the target image.Local features guide regions visible in the source, while global source features guide regions that are invisible in the source but visible in the target.Source: §1 · p. 2
• Spatial-aware normalization transfers source-image spatial information to generated features by modulating their scale and bias.This is proposed to retain spatial context relationships in the source image.Source: §1 · p. 2

2.1. Image Synthesis

• Image-synthesis methods including SPADE and SEAN use normalization or per-region encoding for controllable synthesis, but human pose transfer remains limited by sparse keypoint correspondence and pose and texture variation.PISE instead proposes a two-stage model that uses semantic guidance for more controllable image synthesis.Source: §2.1 · p. 2

2.2. Human Pose Transfer

• Pose-transfer methods use keypoint-based feature transfer or human parsing maps for semantic guidance, but flow-based approaches struggle with large occlusions and prior methods do not flexibly disentangle clothing shape and style.PISE is presented as generating more reasonable and realistic images with consistent shape and style, while enabling more flexible editing through their disentanglement.Source: §2.2 · pp. 2–3

3. Method

• The parsing generator predicts a target-pose-aligned human parsing map, and the image generator uses it to synthesize the reposed person with controllable clothing shape and style.The model also supports texture transfer and region editing.Source: §3 · p. 3

3.1. Parsing Generator

• Gated convolution deforms source parsing features toward the target pose, and a decoder produces the aligned parsing map.The gating mechanism avoids treating every pixel as valid information, unlike vanilla convolution.Source: §3.1 · pp. 3–4

3.2. Image Generator

• The image generator transfers source-region textures to the generated parsing map and uses spatial-aware normalization to preserve source spatial context.It conditions the image synthesis on the source image and parsing maps, generated parsing map, and target pose.Source: §3.2 · p. 4

3.2.1 Per-region encoding

• Joint global and local per-region average pooling extracts style codes locally for visible categories and globally for categories absent from the source.This produces codes for all label categories, including unseen ones.Source: §3.2.1 · p. 4

3.2.2 Per-region normalization

• Per-region normalization uses each generated parsing region's style code to predict scale and bias, applying them to the generated feature.The resulting feature contains the source clothing styles by semantic region.Source: §3.2.2 · p. 4

3.2.3 Spatial-aware normalization

• Spatial-aware normalization maps source spatial scale and bias to the target through a correspondence matrix, then modulates features to retain source style and spatial relationships.The correspondence is computed using features aligned in the latent domain with VGG-19 features.Source: §3.2.3 · p. 5

3.3. Loss Functions

• The parsing generator combines cross-entropy and ℓ1 losses to produce high-quality maps that resemble target parsing maps.The ℓ1 term compares generated and target parsing maps.Source: §3.3 · p. 5
• The image generator uses correspondence, reconstruction ℓ1, perceptual, style, and adversarial losses to align features and train realistic image synthesis.The correspondence loss aligns generated features with VGG-19 target-image features; reconstruction ℓ1 measures pixel-level similarity.Source: §3.3 · p. 5

4. Experimental Results

• The evaluation uses 101,966 training pairs and 8,750 test pairs from DeepFashion Inshop, with LPIPS, PSNR, and FID as metrics.The test identities differ from the training identities.Source: §4 · pp. 5–6

4.1. Person Image Synthesis in Different Poses

• For different target poses, PISE generates realistic images with fine details from a source image.Figure 4 shows person-image synthesis results in arbitrary poses.Source: §4.1 · pp. 6–7, Figure 4 · p. 6

4.2. Comparison with State-of-the-art Methods

• Qualitative comparisons report sharper, more natural PISE images with retained spatial context, clothing shape, and reasonable invisible-region predictions.The comparison includes PATN [32], XingGAN [24], BiGraph [23], ADGAN [16], GFLA [20], and PINet [29].Source: Figure 5 · p. 6, §4.2 · p. 7
• PISE achieves the best PSNR, FID, and LPIPS among the compared methods on 8,750 test images.Table 1 presents the quantitative comparison with six state-of-the-art methods.Source: §4.2 · p. 7, Table 1 · p. 7

4.3. Ablation Study

• Joint global and local per-region encoding improves style prediction, while the full model trades slightly lower PSNR for better perceptual similarity and realism.The ablation also reports that spatial-aware normalization produces sharper images with spatial context, and that joint encoding preserves visible styles and predicts reasonable styles for invisible regions.Source: §4.3 · pp. 7–8, Table 2 · p. 7
• The section includes visual results for upper-clothes and pants texture transfer, texture interpolation, and region editing.The texture-transfer figure shows reference images and generated results, with source images in the first column.Source: Figure 7 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

Texture Transfer

• Replacing a region’s style code with one extracted from a reference image transfers upper-clothes or pants texture, producing natural images with detailed texture.The paper shows transfer from reference images to generated images.Source: §5 · p. 8

Texture Interpolation

• Linear blending of two reference styles makes the generated upper-clothes texture gradually change from one reference style to the other.The interpolation operates on the upper-clothes region’s style code.Source: §5 · p. 8

Region Editing

• Editing the input parsing map can change clothing or hair categories, and the generated image is reported to remain consistent with the edited map.Examples include T-shirt to waistcoat, pants to dress, and long hair to short hair.Source: §5 · p. 8

• PISE first synthesizes a human parsing map, then generates the target image using joint global and local encoding and normalization with spatial-aware normalization.Source: §6 · p. 8
• Experiments are reported to produce detailed textures and consistent clothing shapes, and the ablation study is said to verify the designed components’ effectiveness.Source: §6 · p. 8
• The paper reports texture transfer and region editing applications, while reference-conditioned video generation remains future work.Source: §6 · p. 8