Source-linked AI summary
Dense Intrinsic Appearance Flow for Human Pose Transfer
Yining Li, Chen Huang, Chen Change Loy
TL;DR
Human pose transfer needs dense correspondences and a way to infer pixels hidden by self-occlusion, challenges that sparse keypoints do not adequately address. The paper synthesizes flow supervision from fitted 3D models, trains a feedforward pose-to-flow predictor, and uses flow and visibility maps for feature warping; results on DeepFashion and Market-1501 demonstrate effectiveness over existing methods.
Problem
Human pose transfer is limited by sparse keypoint correspondences and invisible target pixels caused by self-occlusion, while large pose changes and fine details remain difficult to model.
Method
The method fits 3D models to pose pairs to synthesize dense appearance-flow supervision, then trains a feedforward network to predict 3D flow and visibility maps from 2D poses.
Results
The method demonstrates effectiveness over existing approaches, with realistic arbitrary-pose images, preserved details, and reported improvements on DeepFashion and Market-1501.
Takeaways & Limitations
Dense intrinsic 3D appearance flow and visibility-guided feature warping support high-fidelity pose transfer with preserved clothing textures and other appearance attributes.
Abstract
from arXiv · showhide
We present a novel approach for the task of human pose transfer, which aims at synthesizing a new image of a person from an input image of that person and a target pose. We address the issues of limited correspondences identified between keypoints only and invisible pixels due to self-occlusion. Unlike existing methods, we propose to estimate dense and intrinsic 3D appearance flow to better guide the transfer of pixels between poses. In particular, we wish to generate the 3D flow from just the reference and target poses. Training a network for this purpose is non-trivial, especially when the annotations for 3D appearance flow are scarce by nature. We address this problem through a flow synthesis stage. This is achieved by fitting a 3D model to the given pose pair and project them back to the 2D plane to compute the dense appearance flow for training. The synthesized ground-truths are then used to train a feedforward network for efficient mapping from the input and target skeleton poses to the 3D appearance flow. With the appearance flow, we perform feature warping on the input image and generate a photorealistic image of the target pose. Extensive results on DeepFashion and Market-1501 datasets demonstrate the effectiveness of our approach over existing methods. Our code is available at http://mmlab.ie.cuhk.edu.hk/projects/pose-transfer
1. Introduction
Human pose transfer must handle large 3D-induced appearance changes, self-occlusion, and fine visual details beyond sparse keypoint correspondences. The paper addresses these challenges with dense intrinsic 3D appearance flow, visibility prediction, and feature warping.
- Human pose transfer synthesizes a person’s image in arbitrary poses from a single reference view, supporting video editing and person re-identification data augmentation.
- Large non-rigid pose changes create substantial 2D appearance variation, while self-occlusion makes target-side invisible pixels ambiguous.
- Keypoint-based representations capture only rough spatial deformations, often producing distortions and unrealistic details during large pose changes.
- The method implicitly reasons about 3D geometry from 2D pose representations, using synthesized projections of fitted 3D models to obtain training appearance flow.
- A feedforward flow module maps reference and target poses to a 3D flow map and visibility map, while warped features and visibility gating support final image generation.
- The framework reports consistent improvements over existing image generators and state-of-the-art performance on DeepFashion.
2. Related Work
Related work spans conditional generative models, keypoint- and disentanglement-based pose transfer, geometric representations, and appearance-flow methods. The paper distinguishes its approach by handling articulated humans under arbitrary pose transformations with implicit 3D reasoning.
- Deep generative approaches condition image synthesis on inputs such as class labels, language descriptions, or images from another domain.
- Human pose transfer methods use keypoint heatmaps, target views, local affine transformations, or weakly supervised appearance–pose disentanglement.
- Geometry-based methods impose human-body constraints through surface-based UV coordinates or explicit fitting of 3D human models.
- Appearance flow provides dense pixel correspondences, but earlier methods primarily address simple viewpoint changes or rigid objects.
- The proposed appearance-flow module instead addresses articulated human bodies undergoing arbitrary pose transformations.
3. Methodology
The framework predicts dense intrinsic 3D appearance flow and visibility from a reference-target pose pair, then uses them to warp features and pixels for target-pose image synthesis. Flow supervision is synthesized through 3D model fitting and image projection, while visibility-aware warping addresses self-occlusion.
- Problem formulation and notations: Eighteen-keypoint pose heatmaps represent the reference and target poses used to guide arbitrary pose transfer.Each keypoint channel is filled within an 8-pixel radius around the detected joint.
- Problem formulation and notations: The dense intrinsic appearance flow comprises a flow map F and visibility map V that encode pixel-wise 3D correspondences between image pairs.Both maps have the target image's spatial dimensions, and visibility distinguishes visible, invisible, and background regions.
- Feature warping module: The dual-path generator separately encodes appearance and target-pose features, then warps appearance features using F while incorporating V through spatial gating.Invisible features are not simply discarded because they may contain clothing-style or body-shape information; the differentiable module supports end-to-end training.
- Training objectives: PatchGAN adversarial training distinguishes real and synthesized image patches conditioned on the reference image and target pose.The image-generation objectives also include reconstruction and perceptual losses.
- Pixel warping: Pixel warping complements feature warping by copying high-frequency details from the reference and fusing the warped image with the decoder output.A weighting network uses the warped image, feature-warped image, F, and V to produce the fusion map.
4. Experiments
Experiments on DeepFashion evaluate the method against prior generators using image-quality metrics, qualitative comparisons, user judgments, and ablations. Results show stronger detail preservation and perceived realism, while flow-guided warping and pixel warping contribute distinct benefits.
- Dataset and implementation details: The method is evaluated on DeepFashion using 89,262 training pairs and 12,000 test pairs at 256×256 resolution.
- Quantitative results: The method significantly outperforms other methods on FashionIS and AttrRec-k, while also achieving compelling SSIM and IS results.Results from methods using different data splits are not directly comparable.
- Qualitative results: Qualitative comparisons show successful recovery of large spatial displacements and deformations, with realistic images and detailed clothing textures.
- User study: In a 30-user study, generated images were more often confused with real images and judged higher quality than the baseline.The study used real-or-fake and paired comparison tests on randomly selected full-resolution test samples.
- Ablation study: Flow-guided feature warping improves preservation of clothing layouts and complex textures, while pixel warping helps reconstruct special clothing patterns.
5. Conclusion
The paper concludes that implicit 3D reasoning, intrinsic appearance flow, and visibility estimation enable high-fidelity human pose transfer. On DeepFashion, the method generates realistic details and preserved attributes while outperforming existing alternatives.
- The method implicitly reasons about 3D human-body geometry to generate intrinsic appearance-flow and visibility maps.
- A feedforward network predicts both maps, which warp and gate image features for high-fidelity target-pose image generation.
- Qualitative and quantitative DeepFashion results show realistic arbitrary-pose images with preserved appearance attributes.
- The method significantly outperforms existing pose-based, keypoint-based, and other image generators.
Supplementary Material
The supplementary material identifies the paper and its arXiv version. It lists Yining Li, Chen Huang, and Chen Change Loy as authors.
- The authors listed are Yining Li, Chen Huang, and Chen Change Loy.
- The paper is identified as arXiv:1903.11326v1, dated 27 March 2019.
A. Network Architecture
The generator uses seven feature levels and applies flow-guided feature warping at the first five levels. Pose and warped appearance features are concatenated at the bottleneck.
- The image generator has N = 7 feature levels, with channels increasing linearly from 32 to 128.
- Flow-guided feature warping is applied at the first five levels because higher-level features are not location-sensitive.
- At the bottleneck, encoded pose features and warped appearance features are directly concatenated.
Dataset
The method produces accurate appearance-flow and visibility maps across diverse pose and viewpoint changes, while generating realistic images that preserve key appearance attributes and generalize to difficult poses.
- Predicted appearance-flow maps remain close to ground truth across diverse pose and viewpoint changes.The visibility map identifies invisible regions caused by self-occlusion and out-of-field-of-view conditions.
- Generated images are more realistic than previous works and better preserve key appearance attributes.
- The method effectively generalizes reference images to diverse and difficult target poses.
B.4. Failure Case Analysis
Failure cases arise with rare textures, unusual clothing layouts, front-back pose discrepancies, and inconsistent reference regions during pixel warping.
- Complex textures and special clothing layouts remain difficult to synthesize.Examples include rare textures and a coat wrapped around the person’s waist; larger training data are expected to help.
- The method may incorrectly infer a person’s backside from a frontal appearance.The generated image can remain plausible while producing an incompatible shoulder region.
- Pixel warping can blend an inconsistent reference region when front-back pose discrepancy is large.
C. Experiments on Market-1501 dataset
Market-1501 evaluates the method on lower-resolution surveillance images with more diverse poses and complex backgrounds; the method achieves strong results and preserves fine details such as backpacks and clothing patterns.
- Market-1501 contains 32,668 surveillance images of 1,501 persons at 128×64-pixel resolution.
- The dataset provides more diverse poses and complex backgrounds than DeepFashion.
- The image generator uses a reduced U-Net with N = 5 levels for the lower image resolution.
- The method achieves strong results against state-of-the-art baselines on Market-1501.
- Generated images preserve higher-quality details such as backpacks and clothing patterns.