Source-linked AI summary
Person Image Synthesis via Denoising Diffusion Model
Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, Fahad Shahbaz Khan
TL;DR
Pose-guided person synthesis must generate photorealistic people in arbitrary poses while preserving source appearance, yet existing approaches struggle with complex deformations, occlusions, and texture fidelity. PIDM addresses this by decomposing transfer into diffusion steps and adding texture correspondence modeling and disentangled guidance. Across two benchmarks and a user study, it demonstrates photorealistic synthesis and downstream utility.
Problem
Pose-guided synthesis requires photorealistic images with specified pose and appearance, while prior methods struggle with deformations, occlusions, texture fidelity, and sample diversity.
Method
PIDM uses forward-backward diffusion steps, a cross-attention texture diffusion module, and disentangled classifier-free guidance to model and align appearance and pose.
Results
PIDM sets new state of the art on DeepFashion and Market-1501, with user evaluation demonstrating photorealistic outputs and downstream re-identification utility.
Takeaways & Limitations
Diffusion-based staged transfer provides faithful textures and undistorted appearance details while supporting challenging pose transformations and downstream use.
Takeaways & Limitations
VAE-based alternatives can produce blurry, lower-quality outputs because they depend on a surrogate optimization loss.
Abstract
from arXiv · showhide
The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for high-fidelity person image synthesis with strong sample diversity and enhanced mode coverage of the learnt data distribution. Our proposed Person Image Diffusion Model (PIDM) disintegrates the complex transfer problem into a series of simpler forward-backward denoising steps. This helps in learning plausible source-to-target transformation trajectories that result in faithful textures and undistorted appearance details. We introduce a 'texture diffusion module' based on cross-attention to accurately model the correspondences between appearance and pose information available in source and target images. Further, we propose 'disentangled classifier-free guidance' to ensure close resemblance between the conditional inputs and the synthesized output in terms of both pose and appearance information. Our extensive results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios. We also show how our generated images can help in downstream tasks. Our code and models will be publicly released.
1. Introduction
Pose-guided person synthesis seeks photorealistic people in specified poses and appearances, but one-shot generative approaches struggle with deformation, occlusion, texture fidelity, and diversity. PIDM addresses this with staged diffusion, texture correspondence modeling, and disentangled guidance, achieving strong benchmark and downstream results.
- Pose-guided synthesis renders a person with a desired pose and appearance for applications including ecommerce, virtual reality, metaverse, entertainment, and re-identification.
- GAN-based one-shot synthesis often produces deformed textures and unrealistic body shapes for occluded parts, while VAE-based methods can yield blurry, lower-quality outputs.
- PIDM decomposes source-to-target transfer into forward-backward diffusion steps to learn plausible trajectories and model pose-appearance interactions.
- The texture diffusion module exploits source-target appearance and pose correspondences to preserve texture and produce artifact-free images.
- Disentangled classifier-free guidance separately aligns synthesized style with source appearance and pose with the target pose.
- PIDM sets new state of the art on DeepFashion and Market-1501, with user evaluation and demonstrated utility for person re-identification.
2. Related Work
Earlier pose-guided methods primarily use GANs to transfer source style directly into a target pose, but feature misalignment and complex spatial transformations hinder texture preservation. PIDM instead models pose transformation through several conditional denoising steps.
- Early GAN-based methods use coarse-to-fine generation, while direct concatenation of source image, source pose, and target pose can cause feature misalignment.
- VAE-based disentanglement and deformable skip connections were introduced to separate appearance from pose and spatially transform textures through local affine transformations.
- PIDM replaces direct style transfer with several conditional denoising diffusion steps, making each pose-transformation step relatively simple to model.
- Disentangled guidance aligns generated style and pose with source appearance and target pose, respectively.
3. Proposed Method
PIDM reformulates pose-guided person synthesis as conditional denoising diffusion, progressively transferring source appearance to a target pose. Its UNet uses texture-conditioned cross-attention and disentangled guidance to align generated images with both texture and pose inputs.
- Existing GAN-based methods attempt one-shot source-to-target transfer, but struggle with complex spatial transformations and intricate cloth textures.The supplied motivation also notes deformed textures and unrealistic body shapes, especially for occluded parts.
- PIDM decomposes person synthesis into successive conditional forward-backward diffusion steps rather than modeling the full transfer in one operation.DDPM gradually adds noise and learns reverse denoising, converting Gaussian noise into the target data distribution over multiple steps.
- The conditional model takes a source image and target pose, aiming for an output that matches the target pose while retaining the source image’s style.The denoising network is conditioned on xs and xp throughout the synthesis process.
- Texture-Conditioned Diffusion Model: The UNet combines a noise prediction module HN with a texture encoder HE, whose multiscale features enter HN through cross-attention Texture Diffusion Blocks.The texture encoder extracts stacked features from different layers, while TDB units use source-derived keys and values with noise-feature queries at multiple resolutions.
- Disentangled Guidance based Sampling: During inference, disentangled classifier-free guidance separately amplifies pose and style conditioning to strengthen correspondence between inputs and generated images.The guidance combines unconditioned, pose-guided, and style-guided noise predictions, with separate pose and style scales.
- PIDM is evaluated through qualitative comparisons with state-of-the-art methods on DeepFashion.The comparison uses target pose and source image inputs and includes ADGAN, PISE, GFLA, DPTN, CASD, and NTED.
4. Experiments
Experiments on DeepFashion and Market-1501 evaluate PIDM quantitatively, visually, through user preference, ablations, appearance control, interpolation, and person re-identification augmentation. Across these settings, the reported results support high-quality synthesis, effective texture-and-pose control, and improved downstream re-identification performance.
- Experimental setup: Experiments use DeepFashion and Market-1501 at multiple resolutions, with SSIM and LPIPS measuring reconstruction accuracy alongside FID.DeepFashion uses 256 × 176 and 512 × 352 resolutions; Market-1501 uses 128 × 64.
- Quantitative and qualitative comparisons: PIDM achieves the best FID and favorable SSIM and LPIPS results against several state-of-the-art methods on both benchmarks.The reported metrics indicate image quality, accurate structure, and source-texture transfer.
- Quantitative and qualitative comparisons: Qualitative comparisons show PIDM retaining source appearance while producing sharper, more natural images, including complex poses and occluded regions.The comparison covers DeepFashion and Market-1501, including challenging backgrounds and invisible source regions.
- User study: In the user study, PIDM performs favorably against all baselines on R2G, G2R, and Jab for DeepFashion.The study uses 100 participants and evaluates realism judgments and preference for texture patterns and pose structure.
- Ablation study: Ablations attribute improvements to the texture encoder, texture diffusion module, and disentangled classifier-free guidance.The texture diffusion module models appearance-pose interplay, while disentangled guidance aligns style and pose with their respective conditions.
- Appearance control: PIDM supports appearance editing and smooth style interpolation while maintaining pose and identity, with clothing texture gradually transitioning between styles.The interpolation uses DDIM sampling and spherical linear interpolation between noises.
- Application to person re-identification: Augmenting Market-1501 training data with PIDM-generated images consistently improves person re-identification over previous works.The experiment varies the real training-set fraction from 20% to 80% and fine-tunes a ResNet50 on the augmented set.
5. Conclusion
PIDM uses diffusion-based denoising to simplify pose-guided person image transfer, preserving faithful textures and appearance details. Evaluations across two datasets and downstream person re-identification experiments demonstrate its effectiveness.
- PIDM frames pose-guided person synthesis as simpler forward-backward denoising steps for learning source-to-target transformation trajectories.The approach targets faithful textures and undistorted appearance details.
- A texture diffusion module and disentangled classifier-free guidance model appearance-pose correspondences and align generated images with source appearance and target pose.
- PIDM is evaluated through qualitative, quantitative, and human-based experiments on two datasets.
- Generated PIDM images are also shown to help downstream tasks such as person re-identification.