Source-linked AI summary
Dense Pose Transfer
Natalia Neverova, Riza Alp Guler, Iasonas Kokkinos
TL;DR
Pose transfer requires generating a person in a target pose from one source image and a pose donor while preserving appearance despite occlusions and limited surface observations. The paper combines DensePose-based surface correspondence, UV-space inpainting, and a complementary predictive stream, then fuses their outputs. It reports quantitatively better results than recent state-of-the-art methods on DeepFashion and MVC, while finding DensePose superior to sparser conditioning signals.
Problem
The paper addresses pose transfer when the target pose is supplied by another image, requiring controllable synthesis while handling surface occlusions and incomplete observations.
Method
The system uses DensePose correspondences for predictive and explicit UV-space warping streams, inpaints missing appearance, and blends their predictions through neural refinement.
Results
The method reports quantitatively better results than recent state-of-the-art methods on DeepFashion and MVC, with DensePose providing a superior conditioning signal.
Takeaways & Limitations
Surface-based neural synthesis supports pose transfer while preserving a connection between image generation and the human body surface.
Takeaways & Limitations
Inception score is unstable and has limited relevance for within-class generation; DeepFashion ground-truth images have an average IS of 3.9.
Abstract
from arXiv · showhide
In this work we integrate ideas from surface-based modeling with neural synthesis: we propose a combination of surface-based pose estimation and deep generative models that allows us to perform accurate pose transfer, i.e. synthesize a new image of a person based on a single image of that person and the image of a pose donor. We use a dense pose estimation system that maps pixels from both images to a common surface-based coordinate system, allowing the two images to be brought in correspondence with each other. We inpaint and refine the source image intensities in the surface coordinate system, prior to warping them onto the target pose. These predictions are fused with those of a convolutional predictive module through a neural synthesis module allowing for training the whole pipeline jointly end-to-end, optimizing a combination of adversarial and perceptual losses. We show that dense pose estimation is a substantially more powerful conditioning input than landmark-, or mask-based alternatives, and report systematic improvements over state of the art generators on DeepFashion and MVC datasets.
1 Introduction
The paper uses surface-based representations to gain stronger control over human image synthesis, introducing pose transfer guided by a pose-donor image. Its two-stream system addresses occlusion and correspondence errors while improving results on DeepFashion and MVC.
- Surface-based representations are proposed to provide stronger control over image synthesis than categorical attributes, low-dimensional parameters, or layout constraints.
- DensePose and SMPL associate person images with a full surface model, supporting surface-based interpretation of human images.
- A pose-donor image guides synthesis of the same person in a new pose through shared surface coordinates and transferred appearance.
- An inpainting network handles self-occlusions and DensePose inaccuracies, while a feedforward synthesis module complements its predictions.
- Experiments on DeepFashion and MVC report quantitatively better results than recent state-of-the-art methods.
- The approach is also presented as potentially useful for virtual and augmented reality, dataset augmentation, forgery-detector training, and texture transfer without acquiring a surface texture map.
2 Previous works
Prior work established neural synthesis and controllable human generation, but existing approaches often rely on domain-specific knowledge or constrain pose through explicit surface models. Dense pose transfer combines surface correspondence with generative inpainting to address these limitations.
- Neural synthesis models generate faces, humans, and scenes, with loss functions shaping the realism and stability of their outputs.
- Conditional generation improves control using attributes, parameters, layouts, keypoints, segmentations, or domain-specific person knowledge.
- A controllable surface-based human model can generate realistic clothing combinations, but determining pose through that model can limit rendering a source person according to a target video.
- Image inpainting methods fill occluded regions using occlusion-aware training or score optimization, while UV-space methods target photorealistic face inpainting and pose-invariant identification.
3 Dense Pose Transfer
Dense Pose Transfer uses DensePose in parallel predictive and warping streams: one generates plausible images from dense conditioning, while the other explicitly transfers and inpaints texture in a shared UV surface system. A blending module fuses and refines both outputs.
- Dense Pose Transfer: DensePose assigns human pixels to a surface-based body parameterization, providing the common correspondence used by both streams.
- Predictive stream: The predictive stream handles familiar poses and produces plausible images, but does not generalize to new poses or transfer texture details.
- Predictive stream: The predictive stream conditions a generative model on the input image and DensePose results for both input and target images, combining global part labels with point-level coordinates.
- Warping stream: The warping stream explicitly maps source observations into shared UV coordinates and inversely maps the texture to the target image using spatial transformers.
- Warping stream: Because visible source and target surface regions may not overlap, an inpainting network extrapolates appearance from observed UV nodes to the remainder of the surface.
- Warping stream: Masked ℓ1 reconstruction penalizes only observed UV regions, while multiple views of the same person provide supervision covering more of the texture domain.
- Warping stream: Part-specific autoencoders fuse local encodings with global pose context before decoding each surface-part texture.
- Blending and training: A blending module combines predictive and warping outputs with target DensePose and learns a residual refinement of the predictive result.
4 Experiments
The experiments evaluate pose-transfer quality using structural, perceptual, detection, and user-study measures across DeepFashion and MVC settings. Results favor DensePose conditioning and the combined architecture, while different losses trade off structure, texture, and realism.
- Experimental setup: Experiments use DeepFashion and supplementary MVC evaluations, with DeepFashion containing 52,712 images across 13,029 clothing items.Images are 256×256 on a uniform background; MVC contains 161,260 high-resolution images covering front, back, left, and right views.
- Evaluation metrics: Structural quality is measured with SSIM and MS-SSIM, while realism is assessed with Inception Score and person detection score.The evaluation reports separate structural and perceptual metrics because no single criterion adequately captures both structural fidelity and photorealism.
- Comparison with the state-of-the-art: Our pipeline has a significant structural-fidelity advantage over recent keypoint-guided and multi-view synthesis methods across tested network configurations and training setups.Its Inception Scores are reported as higher quality or directly comparable with existing works, although the authors caution against strong conclusions from this metric.
- Comparison with the state-of-the-art: User evaluations find that prior method has higher perceived realism, whereas our generations improve pose fidelity and anatomical plausibility.Thirty users evaluated 55 real and 55 generated images using realism, anatomy, and pose questions.
- Effectiveness of different body representations: DensePose conditioning produces higher structural quality than masks, segmentation maps, or keypoints and largely fixes keypoint-based failures such as missing limbs and depth-order confusion.The reported failures include discontinuities and collapse of the body’s 3D geometry into a single plane.
- Ablation studies: Adding the warping branch improves performance, and UV-space inpainting improves it further by supplying a fusion prior and preventing sharp boundary artifacts.The full loss combination balances complementary behaviors: perceptual loss tracks structure, style loss sharpens textures, and adversarial loss favors visual plausibility but can disregard structure.
5 Conclusion
The paper introduces a two-stream pose-transfer architecture using dense human pose estimation, combining surface-based inpainting with neural synthesis. Dense pose provides a superior conditioning signal and supports pose transfer in a body-surface parameterization.
- The proposed two-stream architecture exploits dense human pose estimation for pose transfer.
- Dense pose estimation is reported as a clearly superior conditioning signal for data-driven human pose estimation.
- Dense pose also facilitates pose transfer through inpainting in the human body’s natural surface-based parameterization.
6 Appendix A. Inpainting Autoencoder design
The Inpainting Autoencoder uses a wide, shallow, part-based architecture to reconstruct UV-coordinate image representations from DensePose-aligned inputs. Multi-view supervision helps it infer information for unobserved body regions.
- Architecture: The autoencoder is wide and shallow because deeper networks appeared prone to overfitting.The authors plan to explore alternative architectures, regularization methods, and larger datasets.
- Architecture: Twenty-four separate encoder and decoder layers correspond to the 24 body parts, while a context layer fuses their encodings into a 256-dimensional vector.Each layer uses a linear operation, ReLU, and Instance Normalization.
- Architecture: Global pose information is injected through E′ to enable information transfer from visible to unvisible parts.This transfer requires supervision indicating that the inferred information is useful.
- Input representation: A Spatial Transformer Network warps each image according to its DensePose deformation field into the surface-based UV coordinate system.The resulting UV-aligned representations provide the autoencoder input.
- Training: The autoencoder is trained with multiple views of the same person reconstructed from a single input view, using an ℓ1 loss on observable UV coordinates.Neighbor images are warped into the input image’s common UV system and supervise reconstruction only where each view is observable.
7 Appendix B. Additional experiments on the DeepFashion dataset
Additional DeepFashion experiments examine how input and target pose estimators affect data-driven pose transfer. DensePose conditioning on the input remains beneficial even when keypoints condition the target.
- Additional conditioning experiments: DensePose on the input combined with keypoints on the target still produces better results than using keypoints on the target alone.The authors attribute the improvement to better capture of the input image’s pose.
- Additional conditioning experiments: Table 7 reports accuracy measures for data-driven networks conditioned on different input and output pose estimators.The DensePose-to-Keypoints configuration extracts DensePose from the input and keypoints from the target.
8 Appendix C. Experiments on the MVC dataset
The MVC experiments evaluate multi-view image synthesis on a large clothing-image dataset with multiple views per item. The method improves structural accuracy and realism relative to state-of-the-art methods, with qualitative results also shown.
- Dataset and setup: MVC contains 161,260 high-resolution images showing front, back, left, and right views of clothing items from online shopping websites.The dataset supports multi-view image synthesis across clothing-item viewpoints.
- Dataset and setup: The experiment uses 694 854 training pairs and 14 924 test pairs after reserving 1000 clothing items for testing.Images are downsampled to 256×256, while DensePose is extracted from their original 512×512 versions.
- Results: The method shows significant improvements in structural accuracy and realism over state-of-the-art methods for MVC multi-view image synthesis.Quantitative comparisons are reported in Table 8.
- Results: Figure 7 presents qualitative MVC results for a model trained with l2 loss only.The figure provides visual examples complementing the quantitative comparison.