PaperlayerView exact revision on arXiv

Source-linked AI summary

XingGAN for Person Image Generation

Hao Tang, Song Bai, Li Zhang, Philip H. S. Torr, Nicu Sebe

arXiv:2007.09278v1cs.CVcs.LGeess.IV

TL;DR

Pose-conditioned person image generation must translate a person into a desired pose while preserving realistic appearance and shape, yet existing methods can produce artifacts. XingGAN addresses this with crossing, cascaded shape–appearance guidance and reports new state-of-the-art results on Market-1501 and DeepFashion.

  • Problem

    Pose-conditioned person image generation seeks photo-realistic images in desired poses, but existing methods exhibit artifacts because they do not adequately capture joint appearance–shape influence and may rely on pose-only attention.

  • Method

    XingGAN uses cascaded SA and AS generation branches with crossing blocks that mutually update appearance and shape representations, plus co-attention fusion and guided discriminators.

  • Results

    XingGAN achieves new state-of-the-art results on Market-1501 and DeepFashion according to human judgments and automatic evaluation metrics.

  • Takeaways & Limitations

    Crossing shape and appearance guidance supports person image generation with improved visual fidelity and alignment with targeted poses.

Abstract

from arXiv · show

We propose a novel Generative Adversarial Network (XingGAN or CrossingGAN) for person image generation tasks, i.e., translating the pose of a given person to a desired one. The proposed Xing generator consists of two generation branches that model the person's appearance and shape information, respectively. Moreover, we propose two novel blocks to effectively transfer and update the person's shape and appearance embeddings in a crossing way to mutually improve each other, which has not been considered by any other existing GAN-based image generation work. Extensive experiments on two challenging datasets, i.e., Market-1501 and DeepFashion, demonstrate that the proposed XingGAN advances the state-of-the-art performance both in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/XingGAN.

1 Introduction

XingGAN addresses artifacts in pose-conditioned person image generation by jointly and progressively modeling shape and appearance through crossing guidance. Its experiments report improved visual fidelity and pose alignment, with new state-of-the-art results on Market-1501 and DeepFashion.

  • Motivation: Existing pose-conditioned methods can produce unsatisfactory images and visual artifacts because local convolutions miss joint appearance–shape influence and pose-only attention misguides generation.The cited limitations concern both the interaction mechanism and the use of a single guidance modality.
  • Proposed method: XingGAN combines a Shape-guided Appearance-based branch, an Appearance-guided Shape-based branch, and shape- and appearance-guided discriminators.The generator also includes a co-attention fusion module for combining the final branch features.
  • Proposed method: SA and AS blocks progressively update appearance and shape representations under each other’s guidance, enabling cascaded synthesis from both person shape and appearance embeddings.The branches are trained end-to-end so they can mutually benefit during generation.
  • Proposed method: The crossing operation jointly produces attention maps from image and pose modalities to capture their joint influence during generation.This design is intended to improve the consistency of generated shape and appearance.
  • Results: Qualitative and quantitative experiments on Market-1501 and DeepFashion report better visual fidelity and alignment with targeted poses than state-of-the-art methods.The paper describes these findings as new state-of-the-art results on both challenging datasets.

2 Related Work

Prior work uses GANs and varied guidance representations for person image generation, but keypoint-based conditioning remains attractive for practical use. XingGAN extends image–guidance interaction through cross-conditioning.

  • GAN-based generation: GANs use a generator and discriminator to produce images that the discriminator cannot distinguish from real images.Conditional GANs extend this framework to controlled image generation.
  • Person image generation: Person image generation methods use poses to guide synthesis because pixel-wise alignment is unsuitable when the person’s shape deforms between input and target images.Related representations include human keypoints, skeletons, DensePose, 3D pose, and segmented pose.
  • Pose representations: Keypoint-based pose representations are cheaper, more flexible, and easier to apply in practical applications than DensePose, 3D pose, or segmented pose representations.The paper therefore favors keypoint-based pose representation.
  • Conditioning schemes: Image-guidance conditioning schemes include concatenating image and guidance inputs or concatenating their feature representations at a later network layer.Examples include C2GAN’s input concatenation and PG2’s bottleneck feature concatenation.
  • Cross-conditioning: The proposed cross-conditioning strategy lets image and guidance modalities guide and update each other beyond the local interaction used in existing two-way schemes.The paper reports that this strategy further improves person image generation performance.

3 Xing Generative Adversarial Networks

XingGAN uses separate appearance- and shape-based generation branches that exchange information through crossing updates, then fuses their intermediate results with the source image. The generator translates a source person from source pose to target pose while combining appearance and shape guidance.

  • Overall Framework: XingGAN comprises SA and AS generation branches plus a Co-Attention Fusion module for combining appearance and shape representations.The SA branch models appearance under shape guidance, while the AS branch models shape under appearance guidance.
  • Overall Framework: The generator takes a source image, source pose, and target pose to synthesize the person in the target pose.The source image supplies appearance information, while the source and target poses supply shape information.
  • Shape-guided Appearance-Based Generation: Each SA block updates appearance features using a correlation matrix computed between appearance and shape feature maps.The crossing operation lets each refined appearance position aggregate shape-code information according to cross-feature correlations.
  • Crossing Blocks: The crossing differs from a conventional non-local operator by computing pairwise similarity between appearance and shape feature maps rather than within one feature map.The SA and AS branches therefore exchange information between distinct representations during generation.
  • Appearance-Guided Shape-Based Generation: Each AS block analogously updates shape features under appearance guidance through a second cross-feature correlation matrix.The refined shape code aggregates appearance-code positions, is concatenated with the appearance code, and is passed through a convolution layer.
  • Co-Attention Fusion: CAF generates intermediate results from both final codes and uses co-attention maps to select channels from those results and the input image.The selected content is combined to synthesize the final person image, while the co-attention maps use both shape and appearance features.

4 Experiments

Experiments on Market-1501 and DeepFashion evaluate XingGAN quantitatively, qualitatively, and through ablations. The results show improved visual fidelity and pose alignment, while isolating contributions from its branches, co-attention fusion, and Xing generator design.

  • Experimental Setup: Experiments use Market-1501 and DeepFashion, with images resized to 128×64 and 256×256, respectively, and OpenPose 18-channel heat maps for pose guidance.The datasets contain filtered person images and paired training/testing examples.
  • Main Results: XingGAN achieves better visual fidelity and alignment with targeted poses than state-of-the-art methods on both datasets.The paper reports qualitative and quantitative improvements across the two evaluation datasets.
  • Qualitative Comparisons: XingGAN generates fewer visual artifacts and more realistic person images than PG2, VUnet, PoseGAN, and Pose-Transfer in qualitative comparisons.The reported improvements especially concern appearance details, person integrity, and background artifacts.
  • Ablation Studies: The AS branch reaches 0.286 SSIM and 0.798 Mask-SSIM, compared with 0.239 SSIM and 0.768 Mask-SSIM for the SA branch.The authors attribute the difference to appearance representations being easier to learn than shape representations under deformation.
  • Ablation Studies: Adding both SA and AS branches improves SSIM and Mask-SSIM, while adding CAF further gains around 0.065 Mask-IS, 0.003 SSIM, and 0.009 Mask-SSIM.The authors interpret CAF as learning more correlations between appearance and shape representations.
  • Generator Analysis: The nine-block Xing generator performs best, while five Xing blocks outperform 13-block ResNet and PATN generators on most metrics.Increasing the number of blocks beyond nine reduces generation performance, so nine blocks are used by default.

5 Conclusions

XingGAN uses cascaded guidance across shape and appearance branches to learn deformable translation mappings and update both feature types through crossing blocks. Human judgments and automatic metrics show new state-of-the-art results on Market-1501 and DeepFashion.

  • 5 Conclusions: XingGAN uses cascaded guidance with separate generation branches for person shape and appearance features.The framework learns a deformable translation mapping from both feature types.
  • 5 Conclusions: Two novel blocks update shape and appearance features in a crossing way.The blocks are designed to mutually update the two feature representations.
  • 5 Conclusions: XingGAN achieves new state-of-the-art results on two challenging datasets using human judgments and automatic evaluation metrics.The reported datasets are Market-1501 and DeepFashion.
  • 5 Conclusions: The proposed blocks and XingGAN framework may extend to other GAN-based generation and multi-modality fusion tasks.This is presented as the authors’ expectation about broader applicability.

6 State-of-the-Art Comparisons

On Market-1501 and DeepFashion, XingGAN is compared qualitatively with Pose-Transfer and consistently produces photo-realistic results with fewer visual artifacts.

  • 6 State-of-the-Art Comparisons: XingGAN consistently produces photo-realistic results with fewer visual artifacts than Pose-Transfer on Market-1501 and DeepFashion.Pose-Transfer results were generated using the authors’ well-trained models for fair comparison.
  • 6 State-of-the-Art Comparisons: Figures 8 and 9 provide generation results for XingGAN and Pose-Transfer on Market-1501 and DeepFashion.The comparisons are qualitative.

7 More Ablation Results

Ablation results on Market-1501 show that the full XingGAN model produces more coherent and natural images, while about nine Xing blocks improve naturalness and realism.

  • 7 More Ablation Results: The full XingGAN model consistently generates more coherent and natural person images than its ablated variants.The ablation comparisons assess the advantage of each proposed component.
  • 7 More Ablation Results: About 9 Xing blocks make generated person images more natural and realistic on Market-1501.The result reveals benefits from the progressive generation strategy.

8 Visualization of Co-Attention Maps

The paper visualizes XingGAN’s intermediate results and co-attention maps, alongside qualitative comparisons and ablation figures. The co-attention maps activate different content between intermediate outputs and the input image for final image generation.

  • 8 Visualization of Co-Attention Maps: Figure 12 shows 10 randomly chosen intermediate results, their corresponding co-attention maps, and the input attention map.The visualization examines the proposed co-attention fusion module.
  • 8 Visualization of Co-Attention Maps: The co-attention maps activate different content between generated intermediate results and the input image for generating final person images.The authors interpret this as evidence of the co-attention fusion module’s effectiveness.
  • 8 Visualization of Co-Attention Maps: Figures 8 and 9 present qualitative comparisons with Pose-Transfer on Market-1501 and DeepFashion.These figures are separate from the co-attention visualization and provide cross-dataset visual comparisons.
  • 8 Visualization of Co-Attention Maps: Figures 10 and 11 show ablation results for XingGAN variants and different numbers of Xing blocks on Market-1501.Figure 11 identifies the proposed Xing blocks with the abbreviation ‘B’.
Loading 2007.09278v1…