Deformable GANs for Pose-based Human Image Generation

Aliaksandr Siarohin; Enver Sangineto; Stephane Lathuiliere; Nicu Sebe

Source-linked AI summary

Deformable GANs for Pose-based Human Image Generation

Aliaksandr Siarohin, Enver Sangineto, Stephane Lathuiliere, Nicu Sebe

arXiv:1801.00055v2Dec 29, 2017cs.CV

TL;DR

The paper addresses pose-conditioned person-image generation while preserving appearance despite spatial misalignment. It introduces deformable skip connections and a nearest-neighbour loss in an end-to-end GAN, outperforming previous work in the evaluated benchmarks and supporting extension to other articulated objects when keypoints are available.

Problem
Generating a person in a novel pose requires preserving appearance while handling spatial misalignment between conditioning and target images.
Method
An end-to-end GAN uses pose-driven deformable skip connections and a nearest-neighbour loss to transfer local features and match details despite misalignment.
Results
The proposed method outperforms previous work on this task according to automatic evaluation metrics and human judgments.
Takeaways & Limitations
The approach can extend beyond human generation to other deformable-object tasks when the object’s pose can be extracted with keypoints.
Takeaways & Limitations
Extension depends on defining suitable articulated body regions and on keypoint detections, which may be missing, false, or localized inaccurately.

Abstract

from arXiv · show

In this paper we address the problem of generating person images conditioned on a given pose. Specifically, given an image of a person and a target pose, we synthesize a new image of that person in the novel pose. In order to deal with pixel-to-pixel misalignments caused by the pose differences, we introduce deformable skip connections in the generator of our Generative Adversarial Network. Moreover, a nearest-neighbour loss is proposed instead of the common L1 and L2 losses in order to match the details of the generated image with the target image. We test our approach using photos of persons in different poses and we compare our method with previous work in this area showing state-of-the-art results in two benchmarks. Our method can be applied to the wider field of deformable object generation, provided that the pose of the articulated object can be extracted using a keypoint detector.

1. Introduction

The paper targets person-image generation across poses, where spatial deformations disrupt conventional image-to-image assumptions. It proposes an end-to-end GAN approach and reports higher qualitative and quantitative results across the evaluated datasets.

Problem: The task preserves a person’s appearance while deforming the foreground structure according to a target pose.The approach focuses on articulated human bodies and can extend to faces or animal bodies when keypoints are extractable.
Motivation: Pose-based generation supports applications including videos with non-trivial movements and rare-pose training data for pose estimation or re-identification.
Challenge: Conventional skip connections assume aligned conditioning and output structures, but large pose deformations violate this assumption.Prior work shows that standard skip connections cannot reliably handle misalignments between poses.
Approach: The proposed GAN is end-to-end trained to account explicitly for pose-related spatial misalignment, contrasting with a prior two-stage generation approach.The earlier approach used masked L1 training followed by adversarial appearance refinement.
Results: The method obtains higher qualitative and quantitative results than prior work on all evaluated datasets and benchmarks.

2. Related work

Prior pose-based generation methods commonly rely on architectures or losses that do not adequately address articulated-object misalignment. The paper instead uses deformable skip connections and a single-stage end-to-end design for pose-conditioned person generation.

Background: VAEs and GANs are common deep-network approaches for visual content generation, while conditional variants generate images dependent on conditioning variables.
Background: Conditional image-to-image translation assumes spatial correspondence between low-level conditioning and output information.
Prior work: Prior person-generation work synthesizes arbitrary poses using two stages: pose generation followed by texture refinement.
Prior work: Spatial transformers learn a global parametric transformation, whereas articulated objects require non-parametric deformations not described by one global affine transform.
Proposed approach: Deformable skip connections shuttle local encoder information to the decoder according to pose differences, enabling simultaneous pose generation and texture-level refinement.
Loss design: The nearest-neighbour loss addresses spatial misalignment differently from perceptual and patch-based similarity approaches.The cited alternatives either compare feature-map elements elementwise or use computationally expensive patch correspondence without using it as a loss.

3. The network architectures

The network conditions generation on appearance and source and target poses, processes aligned and target-pose information in separate streams, and uses pose-driven deformable skip connections to transfer local features.

Inputs: At testing time, the generator produces a person with the conditioning image’s appearance and the second image’s pose, represented by 18 detected joints.The same human pose estimator is used during training and testing, so detections may contain localization errors or misses.
Inputs: Pose heat maps encode joint locations, using Gaussian blurring with σ = 6 pixels to provide widespread location information.
Generator: The generator receives noise through dropout together with the conditioning image and its pose heat maps plus the target pose heat maps.
Generator: Because source appearance and target-pose information can be far apart, the encoder processes xa and Ha separately from Hb before decoder fusion.The first stream concatenates xa and Ha; the second processes Hb without shared weights.
Discriminator: The discriminator receives xa, Ha, a real or generated image y, and Hb, then outputs confidence that y is real.
Deformable skip connections: Deformable skip connections transfer encoder feature maps after pose-driven spatial deformation, using local affine transformations for body regions.The transferred features are merged into decoder processing to handle local spatial misalignment.
Body regions: The body is decomposed into 10 rigid sub-parts defined from subsets of the 18 detected joints, with rotated rectangles used for limbs.A whole-image region can shuttle background texture information.
Body regions: Body-region masks select feature content for affine movement, while missing, occluded, or truncated regions leave the corresponding transform uncomputed.

4. Training

The model combines conditional adversarial training with a nearest-neighbour loss to handle local misalignments and match generated details to the target image.

The generator and discriminator are trained with a conditional adversarial loss combined with the proposed nearest-neighbour loss.
Unlike prior conditional GAN approaches using L1 or L2 losses, the proposed loss addresses their tendency to produce blurred images.
The nearest-neighbour loss compares each generated-image patch with patches in a local neighbourhood of the target image.
The loss uses VGG-19 conv1_2 feature representations, preserving the original image resolution while representing 5 × 5 receptive-field patches.
The final objective combines the adversarial and nearest-neighbour losses, using λ = 0.01 in all experiments.

5. Implementation details

The implementation trains the generator and discriminator with Adam and uses instance normalization, dropout, and resolution-specific generator architecture choices.

G and D are trained for 90k iterations using Adam with learning rate 2 ∗10−4, β1 = 0.5, and β2 = 0.999.
The model follows prior work by using instance normalization.
Dropout is added at rate 50% and used only during training.
The encoder uses two streams, each composed of the specified convolution, normalization, and dropout layer sequence.
The discriminator architecture is defined separately from the generator architecture.
The DeepFashion generator adds one convolution block in both encoder and decoder because its images have higher resolution.

6. Experiments

The experiments evaluate the method on Market-1501 and DeepFashion using standard image-quality metrics, a detection score, human judgments, qualitative comparisons, and ablations. Results generally favor the full pipeline, with stronger texture fidelity and benchmark performance than prior work and simpler variants.

Datasets: Market-1501 contains 32,668 images of 1,501 persons, while DeepFashion contains 52,712 clothing images organized into 200,000 pairs.After human-body-detection filtering, the datasets provide 263,631 and 89,262 training pairs, respectively.
Evaluation: The evaluation combines SSIM, Inception Score, masked variants, Detection Score, and a human study because no single metric fully captures generation quality.Mask-based metrics may favor the compared method, and Inception Score is considered unsuitable for a one-class domain, motivating Detection Score.
Comparison with previous work: On Market-1501, the proposed method achieves the highest performance on all metrics except Inception Score, while on DeepFashion it improves Inception Score but has slightly lower SSIM.The comparison is against Ma et al., with no other prior works available for this task.
User study: Human judgments report a significant quality boost over Ma et al.; on Market-1501, generated-as-real confusion is one order of magnitude higher.The study showed 55 real and 55 generated images per dataset to 30 expert users for one second.
Ablation study: Ablations show progressive improvement from Baseline to DSC to Full, with Full usually outperforming PercLoss on Market-1501 and DeepFashion.Detection Score differences are especially evident; DeepFashion Detection Scores are omitted because all are close to ∼0.96.
Qualitative analysis: Qualitative results show that the full pipeline preserves clothing textures and local details better, whereas Baseline outputs are often blurred or texture-inaccurate.Examples include shirt drawings and clothing stripes in the Market-1501 and DeepFashion figures.

7. Conclusions

The paper presents a GAN-based approach using deformable skip connections and nearest-neighbour loss for pose-conditioned person-image generation, outperforming previous work in experiments.

The approach combines deformable skip connections with a nearest-neighbour loss for pose-conditioned person-image generation.The two components address different types of misalignment between generated and target images.
Experiments using automatic metrics and human judgments show that the method outperforms previous work on the task.
The appendix reports implementation details, additional quantitative and qualitative results, human-body symmetry handling, and a person re-identification application.

A. Nearest-neighbour loss implementation

The nearest-neighbour loss is implemented by searching local feature neighborhoods for matching points, aggregating channel-wise differences, and weighting the resulting loss conservatively.

For each generated feature point, the implementation searches for the most similar point within an n × n neighborhood of the target feature map.
The target feature map is shifted across n^2 translation offsets, with borders filled using +∞.
Each shifted tensor stores channel-wise absolute differences between corresponding generated and target feature points.
Channel differences are summed pointwise to produce scalar tensors representing local L1 distances.
The minimum local distance is selected for each point, forming the nearest-neighbour loss.
Because the loss is not normalized by channels or pixels, it is weighted with λ = 0.01 relative to LcGAN.

B. Exploiting the human-body symmetry

The method decomposes the human body into rigid parts and uses detected symmetric limbs to fill missing regions when pose-estimation detections fail.

The human body is decomposed into 10 rigid sub-parts: the head, torso, and eight limbs.
If a joint is undetected, its region and affine transformation are omitted and the region mask is filled with zero.This can occur when the region is invisible or the human-pose estimator produces false detections.
When a missing limb has a detected symmetric counterpart, information from that twin region is copied into the missing region.
The corresponding affine transformation is then computed using the completed region.

C. Improving person Re-ID via dataaugmentation

The generated images are used to augment person re-identification training data by producing identity-preserving views in new poses and viewpoints.

The generated images support training discriminative person re-identification methods across poses and viewpoints.
The augmentation targets images of the same person with identical clothes but different viewpoints or poses.
For each Market-1501 training image, the experiment randomly selects 10 target poses and generates 10 corresponding images.

D. Comparison with previous work

Compared with Ma et al.’s method, the proposed approach generally produces more realistic images and preserves conditioning-image details more faithfully on both Market-1501 and DeepFashion. The authors attribute this to combining deformable skip connections with nearest-neighbour loss.

The comparison uses the same conditioning-image and target-pose pairs selected in Ma et al.’s evaluation, displaying outputs from both full pipelines.The conditioning image supplies appearance reference, while the target pose specifies the desired body configuration.
On Market-1501, the proposed results are usually more realistic, sharper, and closer to the conditioning image’s clothing textures and facial details than Ma et al.’s results.Some compared outputs from Ma et al.’s method are difficult to recognize as humans, whereas the proposed outputs appear human-like.
On DeepFashion, the proposed method usually produces more realistic images and better preserves details from the conditioning image than Ma et al.’s method.Examples include more faithful clothing textures and a correctly generated leg appearance where the comparison method generated pants.
The authors argue that deformable skip connections and nearest-neighbour loss together transfer local appearance details while accounting for global pose deformation.This combination is intended to capture details such as clothing texture and facial characteristics from the conditioning image.

E. Other qualitative results

Additional qualitative results show progressively better detail synthesis across model variants, while failure cases arise from pose-estimation errors, ambiguous 2D poses, rare poses or appearances, and warping limitations.

Additional qualitative results: Across additional Market-1501 and DeepFashion examples, pose-related global structure is generally generated sufficiently well by all evaluated model variants.The examples use skeleton-based visualization for clarity, although the method uses only point-wise joint locations.
Additional qualitative results: Detail quality usually improves progressively from Baseline to DSC to PercLoss to Full.This progression concerns the synthesis of local image details rather than the broad pose-related structure.
Failure cases: The method has failure cases on both Market-1501 and DeepFashion, documented in separate figures.The authors group the failures into several recurring causes.
Failure cases: Human-pose-estimation errors can place joints incorrectly and lead to badly generated images.Examples include incorrect joint detections in Market-1501 and wrong right-arm localization in DeepFashion.
Failure cases: A 2D pose representation cannot resolve whether an occluded joint lies in front of or behind the body.In one example, the model generates an arm in front because that configuration is more frequent in the training data.
Failure cases: Rare poses and unusual object appearances can cause structural or visual mistakes in generated images.Reported examples include synthesizing a neck instead of a shoulder and mishandling an unusually light-green backpack.
Failure cases: Large pose-dependent warps can distort texture when a narrow profile region must fit a much wider frontal target region.The authors identify this as a limitation specific to their deformable skip connections.