PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, Wei Yang

arXiv:2310.06313v4cs.CV

TL;DR

Pose disparities make it challenging to synthesize a photorealistic target-pose image from a source image and pose alone. PCDMs progressively predict global features, establish dense correspondence, and refine texture. They report the best SSIM, LPIPS, and FID among compared methods on Market-1501, and reasonable target alignment in extreme-pose and large-occlusion DeepFashion examples.

  • Problem

    Pose disparity makes it challenging to generate a photorealistic image in a distinct pose using only the source image and target pose.

    Source: §1 · p. 1

  • Method

    PCDMs use prior diffusion to predict target global features, inpainting diffusion to establish dense correspondence and create a coarse image, and refining diffusion to restore texture and detail consistency.

    Source: §1 · p. 2

  • Results

    PCDMs achieve the best SSIM, LPIPS, and FID among compared methods on Market-1501, and are the only method reported to align reasonably with targets in DeepFashion extreme-pose and large-occlusion examples.

    Source: §4.1 · pp. 7–8

  • Takeaways & Limitations

    PCDMs are reported to produce visually more appealing outcomes, particularly when handling intricate poses.

    Source: §1 · p. 2

    The two additional prior and refining models increase computational resource consumption and inference time.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent work has showcased the significant potential of diffusion models in pose-guided person image synthesis. However, owing to the inconsistency in pose between the source and target images, synthesizing an image with a distinct pose, relying exclusively on the source image and target pose information, remains a formidable challenge. This paper presents Progressive Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between person images under the target and source poses through three stages. Specifically, in the first stage, we design a simple prior conditional diffusion model that predicts the global features of the target image by mining the global alignment relationship between pose coordinates and image appearance. Then, the second stage establishes a dense correspondence between the source and target images using the global features from the previous stage, and an inpainting conditional diffusion model is proposed to further align and enhance the contextual features, generating a coarse-grained person image. In the third stage, we propose a refining conditional diffusion model to utilize the coarsely generated image from the previous stage as a condition, achieving texture restoration and enhancing fine-detail consistency. The three-stage PCDMs work progressively to generate the final high-quality and high-fidelity synthesized image. Both qualitative and quantitative results demonstrate the consistency and photorealism of our proposed PCDMs under challenging scenarios.The code and model will be available at https://github.com/tencent-ailab/PCDMs.

1 INTRODUCTION

Pose disparities make it difficult to synthesize a photorealistic image from only a source image and target pose. PCDMs address this with three progressive diffusion stages that predict global features, establish dense correspondence, and refine texture.

1 INTRODUCTION

  • Pose-guided synthesis must preserve a person’s appearance while matching a target pose, but large pose disparities make this difficult using only the source image and target pose.The task has potential applications in e-commerce, content generation, and person re-identification.Source: §1 · p. 1
  • Earlier GAN, VAE, flow-based, and parsing-based approaches can produce distorted or blurry details, pose misalignment, artifacts under deformation or occlusion, or inconsistent style and texture.GANs are also described as having unstable training dynamics that can limit sample diversity.Source: §1 · pp. 1–2
  • Diffusion methods retain input information through multi-step denoising, but pose inconsistency leaves them doing unaligned conditional image-to-image generation, often without dense correspondence and with less realistic results.The cited gap concerns correspondence across image, pose, and appearance.Source: §1 · p. 2
  • PCDMs progressively predict target global features, use them to establish dense correspondence and generate a coarse image, then refine its texture and detail consistency.The prior model uses source appearance and target pose coordinates; the inpainting model aligns contextual features before the refining model restores texture.Source: §1 · p. 2
  • The paper’s contributions include the three conditional diffusion models and experiments on two public datasets, a user study, and a downstream task evaluation.The prior model mines global alignment between source appearance and target pose coordinates; the inpainting model explores dense correspondence, and the refining model uses post hoc image-to-image techniques.Source: §1 · p. 2

2 RELATED WORK

Earlier person-image methods address pose mismatch through conditional generation, appearance-pose disentanglement, deformation, and parsing, while correspondence and realism remain challenges. Diffusion models offer multi-step generation, and PCDMs organize this approach into prior, inpainting, and refining stages.

2 RELATED WORK

  • Earlier methods use conditional GANs, VAE-based appearance-pose disentanglement, local affine transformations, or flow-based deformation to address pose inconsistency and alignment.Examples include VUnet, Def-GAN, and flow-based methods.Source: §2 · p. 2
  • The PCDM pipeline progresses from global-feature prediction to dense correspondence and texture restoration.Figure 2 presents these as the three stages for bridging the gap between person images under source and target poses.Source: Figure 2 · p. 3
  • Parsing maps and dense correspondence extend alignment approaches, but absent global target-appearance features can reduce realism, and prior multi-stage methods can lose realism when combining pose, texture, background, or body.GFLA uses global flow fields and occlusion masks to warp source patches; CoCosNet extracts dense cross-domain correspondence.Source: §2 · p. 3
  • Diffusion’s multi-step denoising can better preserve input information and sharpen textures; related person-synthesis work includes MGD and PIDM, while PCDMs use prior, inpainting, and refining stages.MGD conditions latent diffusion on pose, garment sketch, and text; PIDM uses texture diffusion and disentangled classifier-free guidance.Source: §2 · p. 3

3 METHOD

PCDMs progressively bridge source- and target-pose person images through global-feature prediction, dense-correspondence inpainting, and coarse-image refinement. Each stage uses pose, source appearance, or outputs from an earlier stage as conditions.

3.1 PRELIMINARIES

  • Diffusion models reverse Gaussian noising with MSE noise prediction; latent diffusion runs in pretrained-autoencoder latent space, while classifier-free guidance balances fidelity and diversity.Guidance scale controls the strength of the condition.Source: §3.1 · pp. 3–4

3.2 PRIOR CONDITIONAL DIFFUSION MODEL

  • The prior stage uses source and target pose coordinates and source-image appearance to predict the target image’s unnoised global embedding.It uses a transformer and CLIP image embeddings for the global image features.Source: §3.2 · p. 4

3.3 INPAINTING CONDITIONAL DIFFUSION MODEL

  • The inpainting stage combines source and target images, poses, masks, prior target global features, and local source features to establish dense correspondences and generate a coarse image.A convolutional pose encoder and DINOv2 source-image encoder provide spatial pose features and fine-grained source features.Source: §3.3 · p. 5

3.4 REFINING CONDITIONAL DIFFUSION MODEL

  • The refining stage conditions on the coarse image and uses source-image texture features through cross-attention to repair textures and improve detail consistency.The coarse image is concatenated with the noisy image at the input to the diffusion model.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments compare PCDMs with state-of-the-art methods on DeepFashion and Market-1501, using quantitative, qualitative, and user-study evaluations. Ablations, refinement tests, and person re-identification experiments further assess the stages and downstream applicability.

4 EXPERIMENTS

  • The experiments use DeepFashion and Market-1501, evaluate objective and user-oriented metrics, and train on dataset-specific image sizes and configurations.DeepFashion has 52,712 high-resolution images and Market-1501 has 32,668 low-resolution images; the study uses SSIM, LPIPS, FID, R2G, G2R, and Jab.Source: §4 · p. 6, §4.1 · p. 6

4.1 QUANTITATIVE AND QUALITATIVE RESULTS

  • PCDMs excels on two of three DeepFashion metrics and achieves the best SSIM, LPIPS, and FID on Market-1501.On DeepFashion, PCDMs has a worse FID score than PIDM but surpasses it on the other two metrics.Source: §4.1 · pp. 6–7
  • On DeepFashion, PCDMs handles extreme poses, large occlusions, complex textures, and logos more consistently than the compared methods.The qualitative comparison also reports better image quality than PIDM when retaining a small necklace.Source: Figure 6 · p. 7, §4.1 · p. 8
  • In the DeepFashion user study, 56.2% of PCDMs images were perceived as real, nearly 20.4% above the next-best model, and its Jab score was 44.1%.Figure 7 reports R2G, G2R, and Jab; higher values indicate better performance.Source: §4.1 · p. 8, Figure 7 · p. 8

4.2 ABLATION STUDY

  • The three-stage PCDMs configuration is visually superior to B1, B2, and B3 on complex poses; B1 distorts limbs, while B2 improves structure and B3 improves texture detail.B1 uses only inpainting; B2 adds the prior model but omits refinement; B3 uses inpainting and refinement but omits the prior model.Source: §4.2 · pp. 8–9, Figure 8 · p. 8
  • Refining conditional diffusion improves all tested SOTA methods, filling details in ADGAN and PIDM and refining textures and shape consistency in other methods.The tested methods include ADGAN, PIDM, GFLA, DPTN, CASD, and NTED.Source: §4.2 · p. 9, Figure 9 · p. 9

4.3 APPLICATION

  • On Market-1501 person re-identification, adding PCDMs-generated images boosts baseline performance and outperforms PTN, GFLA, DPTN, and PIDM.The experiment uses BoT and synthetic images matched by identity and pose; Table 2 reports Rank1 results.Source: §4.3 · p. 9, Table 2 · p. 9

5 CONCLUSION

PCDMs progressively generate person images through global-feature prediction, dense correspondence and inpainting, then refinement for texture restoration and fine-detail consistency. The paper reports consistency and photorealism under challenging scenarios, while noting added computational cost and inference time.

5 CONCLUSION

  • PCDMs use three progressive stages to bridge source- and target-pose images, with qualitative and quantitative results showing consistency and photorealism under challenging scenarios.The stages predict global features, establish dense correspondence and generate a coarse image, then refine texture and fine-detail consistency.Source: §5 · p. 9
  • The prior diffusion model predicts target-image global features by mining alignment between pose coordinates and image appearance.The inpainting stage establishes dense correspondence and aligns contextual features; the refining stage uses the coarse image to restore texture and improve fine-detail consistency.Source: §5 · p. 9
  • Adding prior and refining models increases computational resource consumption and inference time.Future work is proposed to reduce these costs while maintaining equivalent or better quality.Source: §5 · p. 9

6 ETHICS STATEMENT

The ethics statement notes that person-image manipulation could be misused to create false content and spread misinformation. It frames the risk as common across person-image synthesis methods and calls for research, prevention, and external audits to balance value against risks.

6 ETHICS STATEMENT

  • Manipulating real photos to create person images carries a risk of false content and misinformation.The statement identifies this as a risk of the technology, rather than a PCDM-specific outcome.Source: §6 · p. 10
  • The statement says this misuse risk is faced by virtually all person-image synthesis methods.Source: §6 · p. 10
  • The authors cite progress in detecting and preventing tampering and propose support for research and external audits to balance value against unrestricted-access risks.Source: §6 · p. 10

SUPPLEMENTARY MATERIAL

The supplementary material expands on the paper’s methods and experiments, covering notation, implementation details, and additional qualitative and experimental material.

SUPPLEMENTARY MATERIAL

  • The supplement adds broader qualitative comparisons, ablations, random-sampling results, and an explanation of the user studies.It also provides further implementation specifics.Source: SUPPLEMENTARY MATERIAL · p. 13

A SOME NOTATIONS AND DEFINITIONS

  • Table 3 collects notations and definitions used in the paper.The supplied passage identifies the table’s subject but does not enumerate its entries.Source: Table 3 · p. 13

B IMPLEMENT DETAILS

  • The models use AdamW with a 1e−4 learning rate; prior, inpainting, and refining models train for 100k, 200k, and 100k iterations, respectively.Training uses 8 NVIDIA V100 GPUs; inference uses DDIM with 20 steps and guidance scale 2.0 across all stages.Source: B IMPLEMENT DETAILS · p. 13, Table 4 · p. 13

C.1 MORE QUALITATIVE RESULTS WITH REFINING CONDITIONAL DIFFUSION MODEL.

The section presents qualitative results for PIDM combined with the refining conditional diffusion model and additional comparisons of PCDMs with state-of-the-art methods.

C.1 MORE QUALITATIVE RESULTS WITH REFINING CONDITIONAL DIFFUSION MODEL.

  • Figure 10 shows qualitative results for PIDM combined with the refining conditional diffusion model.The passage identifies the figure’s subject but does not state the comparison’s outcome.Source: §C.1 · p. 14

C.2 MORE QUALITATIVE COMPARISONS FOR PCDMS

  • Figure 11 provides additional examples comparing PCDMs with state-of-the-art methods.The passage does not specify which method performs best.Source: §C.2 · p. 14

C.3 MORE QUALITATIVE ABLATION RESULTS ON THE DEEPFASHION DATASET

The qualitative ablations compare three configurations that omit different components of PCDMs: inpainting alone, prior plus inpainting, and inpainting plus refining.

C.3 MORE QUALITATIVE ABLATION RESULTS ON THE DEEPFASHION DATASET

  • B2 combines prior and inpainting diffusion models but omits the refining model.This configuration is contrasted with using inpainting alone (B1) and inpainting plus refining without the prior model (B3).Source: §C.3 · p. 14
  • B3 combines inpainting and refining diffusion models while omitting the prior model.The passage identifies this as a qualitative ablation configuration shown in Figure 12.Source: §C.3 · p. 14
  • B1 uses only the inpainting conditional diffusion model.Figure 12 presents qualitative ablation results on DeepFashion.Source: §C.3 · p. 14

C.4 RANDOM SAMPLES

The appendix presents random samples and pretrained-weight comparisons, describes the user-study setup, and adds qualitative comparisons, ablations, and diffusion-process visualizations.

C.4 RANDOM SAMPLES

C.5 IMPACT OF PRETRAINED

  • Without pretrained weights, the generated outcomes are still described as remarkably realistic.Figure 15 compares results without pretrained weights; the authors attribute text-reconstruction improvement primarily to their framework's rationality and superiority.Source: §C.5 · p. 14, Figure 15 · p. 20

C.6 USER STUDY

C.7 VISUALIZE THE DIFFUSION PROCESS

  • Supplementary figures compare PCDMs with state-of-the-art methods, show qualitative ablations and PIDM refining examples, and visualize stages two and three.For Figure 10, “ours” denotes results using only the prior and inpainting conditional diffusion models.Source: §C.7 · p. 14, Figure 10 · p. 15, Figure 11 · p. 16

Related papers

Loading 2310.06313v4…