Source-linked AI summary
Progressive Pose Attention Transfer for Person Image Generation
Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, Xiang Bai
TL;DR
Pose transfer requires generating a person in a target pose despite partial observations, deformation, and appearance variation. This paper uses cascaded Pose-Attentional Transfer Blocks to perform progressive transfers, reporting superior visual and quantitative performance while improving efficiency and supporting person re-identification data augmentation. Its scope favors keypoint poses over more expensive DensePose representations and evaluates re-identification augmentation through bounded reference comparisons.
Problem
Pose transfer must generate target-pose images from incomplete observations while preserving appearance and shape across substantial pose and viewpoint variation.
Method
A progressive pose-attention network cascades Pose-Attentional Transfer Blocks that jointly update pose and appearance representations during transfer.
Results
The network exhibits superior visual and quantitative performance, improves computational efficiency, reduces model complexity, and substantially augments person re-identification training data.
Takeaways & Limitations
Progressive attention transfer is presented as a framework for realistic person-image generation and training-data augmentation, with potential adaptation to other non-rigid objects.
Takeaways & Limitations
The evaluation favors keypoint-based pose representations because acquiring DensePose is expensive, and the re-identification comparison is bounded by lower and theoretical upper performance bounds.
Abstract
from arXiv · showhide
This paper proposes a new generative adversarial network for pose transfer, i.e., transferring the pose of a given person to a target pose. The generator of the network comprises a sequence of Pose-Attentional Transfer Blocks that each transfers certain regions it attends to, generating the person image progressively. Compared with those in previous works, our generated person images possess better appearance consistency and shape consistency with the input images, thus significantly more realistic-looking. The efficacy and efficiency of the proposed network are validated both qualitatively and quantitatively on Market-1501 and DeepFashion. Furthermore, the proposed architecture can generate training images for person re-identification, alleviating data insufficiency. Codes and models are available at: https://github.com/tengteng95/Pose-Transfer.git.
1. Introduction
Pose transfer is difficult because partial observations and viewpoint changes require inferring unseen body parts while preserving person identity and shape. The proposed network addresses this through progressive, pose-attentional transfers that generate images efficiently.
- Motivation: Pose transfer must infer unobserved body parts and handle drastic appearance changes across poses and views.These challenges make robust generation difficult, particularly with partial observations.
- Motivation: Recent pose-transfer methods often fail to produce robust results despite deep neural networks’ learning capabilities.
- Approach: The method transfers through intermediate pose representations instead of performing a single global transformation.This progressive scheme performs local transfers intended to avoid the complexity of the global pose manifold.
- Contribution: The network reports superior qualitative and quantitative performance on challenging benchmarks and substantially augments person datasets for re-identification.
- Contribution: The proposed network is designed to address pose transfer with a neat architecture and efficient computation.
- Approach: Cascaded Pose-Attentional Transfer Blocks use pose and appearance features to guide the transfer progressively.Each block updates image and pose representations, allowing the blocks to be applied sequentially.
2. Related work
Prior work applies GANs and related models to conditional and person-image generation, but pose transfer remains difficult because deformation breaks pixel alignment. Existing pose-transfer methods trade off computational cost, geometric robustness, supervision, or pose-representation cost.
- Conditional generation: Conditional GANs work well for pixel-wise aligned image translation, but pose deformation makes pixel-wise alignment unsuitable for pose transfer.
- Person image generation: Person-image generation methods address clothing, viewpoint, foreground-background composition, and virtual try-on settings using diverse model designs.
- Pose transfer: Earlier pose-transfer systems involve large computational budgets, complicated training, extensive affine transformations, or geometric errors.The cited limitations include fragility to inaccurate pose joints and errors from fully unsupervised bidirectional generation.
- Pose representation: Keypoint-based pose representations are cheaper and more flexible than DensePose, whose acquisition cost limits applicability.The paper therefore favors keypoint-based representations.
3. Model
The model progressively transfers a person from condition pose to target pose using cascaded Pose-Attentional Transfer Blocks, while discriminators enforce appearance and shape consistency.
- Generator: The generator takes a condition image and condition and target pose heat maps, then produces a person image in the target pose.The condition image enters the image pathway, while the two pose heat maps are stacked and encoded in the pose pathway.
- Pose-Attentional Transfer Network: PATN consists of cascaded PATBs that progressively update image and pose codes through interacting image and pose pathways.The final image code is decoded into the generated image, while the final pose code is discarded.
- Pose-Attentional Transfer Blocks: Each PATB computes pose-guided attention masks that select image-code regions for transfer and updates the image code with a residual connection.The masks indicate element importance; the residual connection preserves the original image code and eases training across many PATBs.
- Pose-Attentional Transfer Blocks: The pose pathway is updated using the transformed previous pose code together with the updated image code, synchronizing where patches are sampled and placed.The two codes are combined by concatenation before the pose representation is updated.
- Discriminators: Separate appearance and shape discriminators judge whether the generated image preserves the person identity and aligns with the target pose.Their scores are multiplied to form the final discriminator score R = R_A R_S.
- Training: Training alternates generator and discriminator updates using adversarial, pixel-wise L1, and perceptual L1 losses.The perceptual loss uses VGG-19 features and is included to reduce pose distortions and produce smoother images.
4. Experiments
Experiments on Market-1501 and DeepFashion evaluate pose-transfer quality, shape consistency, realism, ablations, and computational efficiency. PATN consistently produces more consistent and realistic images while remaining computationally efficient.
- Experimental setup: Experiments use Market-1501 and DeepFashion, whose low- and high-resolution settings provide complementary pose-transfer challenges.Market-1501 varies in pose, viewpoint, background, and illumination, while DeepFashion has high-resolution images with clean backgrounds.
- Evaluation metrics: PCKh complements existing metrics by explicitly approximating shape consistency through alignment of 18 estimated pose joints.PCKh measures the percentage of keypoint pairs whose offsets fall below half the head-segment size.
- Quantitative and qualitative comparison: 0.96 PCKh on DeepFashion improves the previous best by 2%, while matching the highest Market-1501 score and leading on most other metrics.The authors note that the testing protocol gives previous methods an advantage because their training images may overlap the test set.
- Quantitative and qualitative comparison: Qualitatively, PATN preserves person integrity, detailed appearance, sharpness, correct leg layouts, and identity-relevant items across both datasets.Examples include skin color, whiskers, hair, hats, facial details, crossed-leg layouts, and a bag retained when other methods lose it.
- Model and computation complexity comparison: PATN substantially reduces model parameters and computation compared with previous methods because its PATB building blocks are simple and compact.The comparison uses testing on one NVIDIA Titan Xp and measures GPU time for generating DeepFashion testing pairs.
- Ablation study and result analysis: With equal numbers of building blocks, PATN yields more consistent shape and appearance than a Resnet generator, and 5 PATBs outperform 13 residual blocks on almost all measures.Increasing PATBs from 9 to 13 provides only a marginal additional performance boost.
- Ablation study and result analysis: Attention masks progressively shift from blended pose regions to smaller adjustment areas and finally to background refinement across the cascaded PATBs.This visualization makes the pose-transfer process interpretable as a sequence of increasingly localized adjustments.
5. Application to person re-identification
The paper evaluates generated images as augmentation for person re-identification under reduced-data settings, comparing generated and real-image augmentation across training protocols and generators. The proposed method consistently improves re-ID performance and benefits further from larger augmented training sets.
- Evaluation protocol: The experiment reduces Market-1501 training data to 10%–90%, preserves at least one image per identity, and replaces unselected images with generated same-identity images.The augmented set combines the retained real images with generated images transferred to the replaced images’ poses.
- Evaluation protocol: The study compares ResNet-50 and Inception-v2 trained on reduced real-image sets and corresponding generated-image-augmented sets.The protocol uses nine reduced training sets and nine augmented training sets, following the Open-ReID training and testing procedures.
- Evaluation protocol: The reduced-data and full-real-data reference results provide lower and upper bounds for estimating the potential of data augmentation.The gap between these bounds measures the maximum potential improvement available from augmentation in the experiment.
- Results: Whenever a performance gap exists, data augmentation with generated images produces a performance boost in the re-ID evaluation.The experiment is designed to expose gains caused by alleviating data insufficiency.
- Comparison with prior generators: The proposed method achieves consistent improvements over previous person-image generators across different portions of real images and supports random target-pose selection without an obvious performance difference.These results are reported for the re-ID data-augmentation setting.
6. Conclusion
The paper concludes that progressive pose-attention transfer improves person-image generation while reducing computational cost and model complexity. It also identifies broader applicability to non-rigid-object generation and other GAN-based approaches.
- Main conclusions: The proposed network uses cascaded PATBs to optimize appearance and pose simultaneously and guide deformable transfer progressively.The conclusion presents progressive pose attention as the central design for addressing challenging pose transfer.
- Main conclusions: The network delivers superior subjective visual realness and objective quantitative scores while improving computational efficiency and reducing model complexity.The paper also reports substantial use for alleviating insufficient training data in person re-identification.
- Interpretability and validation: The progressive transfer process can be visualized through attention masks, making the network more interpretable, and its design is experimentally verified through ablation studies.The conclusion links interpretability to visualization of the transfer process rather than to a separate evaluation metric.
- Broader applicability: The network is potentially adaptable to other non-rigid objects, and its progressive attention-transfer idea may benefit other GAN-based image-generation approaches.These statements are presented as potential extensions rather than demonstrated results in the supplied passage.
A. Video demonstration
The video demonstration generates frame-by-frame person images from selected appearances and pose sequences extracted from human-action videos, then reconstructs them into videos.
- Video construction: The demonstration estimates body joints from dancing and self-recorded videos, uses them as target poses, and generates frames conditioned on sampled DeepFashion person images.The generated frames are stacked in their original order to form output videos.
- Video construction: The video model combines the Full method with revised deformable skip-connections between encoder and decoder features of matching spatial size.The added connections are architecturally similar to U-Net connections.
B. Person image generation conditioned on arbitrary poses
The model generates natural-looking person images from a fixed condition image across target poses with substantial viewpoint, scale, and direction changes. Comparisons on DeepFashion and Market-1501 show more consistent appearances and poses with the targets than prior methods.
- Arbitrary-pose generation: On DeepFashion, the model generates natural and realistic results from the same condition image across poses with varied viewpoints, scales, and directions.The passage reports this behavior for poses selected from the test set.
- Arbitrary-pose generation: On Market-1501, the model generates multiple person images conditioned on the same person image and different target poses.The examples demonstrate pose-conditioned variation while retaining the condition identity.
- Comparison with prior methods: Compared with PG2, VUnet, and Deform, the proposed method shows more consistent appearances and poses with target conditions on DeepFashion and Market-1501.The comparisons use the nine-PATB version of the method.
D. More ablation study results
Additional ablation comparisons on DeepFashion and Market-1501 reinforce PATN’s advantages over a ResNet generator and support progressive generation with more PATBs.
- PATN produces advantages over the ResNet generator in qualitative ablation comparisons on DeepFashion and Market-1501.
- Adding more PATBs makes generated-person poses more coherent and natural.The passage identifies this as evidence for the benefits of progressive generation.
E. More visualization examples of the attention masks in PATBs
The paper visualizes attention masks across PATBs and presents generated examples under different target poses, alongside qualitative comparisons and failure cases on DeepFashion and Market-1501.
- Attention-mask visualization: Attention-mask visualizations are shown for a PATN generator with nine PATBs tested on Market-1501.The masks are labeled PATB-1 through PATB-9.
- Target-pose examples: Generated DeepFashion examples are conditioned on different target poses selected from the whole test set.
- Target-pose examples: Generated Market-1501 examples are likewise conditioned on different target poses selected from the whole test set.
- Qualitative comparisons: Qualitative comparisons are presented on DeepFashion and Market-1501 using PG2, VUNet, and Deform as comparison methods.
- Ablations and failure cases: Additional figures show qualitative ablation results and failure cases on DeepFashion and Market-1501.