Source-linked AI summary
DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, Ira Kemelmacher-Shlizerman
TL;DR
DreamPose tackles the scarcity of informative fashion videos and the difficulty of preserving garment and identity details during animation. It transforms Stable Diffusion into an image-and-pose-guided video model with specialized conditioning and fine-tuning, producing photorealistic, temporally consistent fashion videos across diverse appearances. The method has occasional failures on unusual poses and complex patterns, and its training and inference are slow compared with GAN or VAE methods.
Problem
Still photographs do not show garment motion, while existing video diffusion models can lack realism, temporal consistency, motion control, and detailed appearance preservation.
Method
DreamPose adapts pretrained Stable Diffusion with image-and-pose conditioning, a split CLIP-VAE encoder, multi-pose inputs, and a two-stage fine-tuning strategy.
Results
DreamPose generates photorealistic fashion videos from a single image and pose sequence across diverse fabrics, patterns, clothing shapes, viewpoints, and person identities.
Takeaways & Limitations
The method preserves identity and garment details while generating pose-driven fashion motion with temporal consistency.
Takeaways & Limitations
The method can produce disappearing limbs, hallucinated dress features, backward-facing pose misalignment, flickering on complex patterns, and slow training and inference.
Abstract
from arXiv · showhide
We present DreamPose, a diffusion-based method for generating animated fashion videos from still images. Given an image and a sequence of human body poses, our method synthesizes a video containing both human and fabric motion. To achieve this, we transform a pretrained text-to-image model (Stable Diffusion) into a pose-and-image guided video synthesis model, using a novel fine-tuning strategy, a set of architectural changes to support the added conditioning signals, and techniques to encourage temporal consistency. We fine-tune on a collection of fashion videos from the UBC Fashion dataset. We evaluate our method on a variety of clothing styles and poses, and demonstrate that our method produces state-of-the-art results on fashion video animation.Video results are available on our project page.
1. Introduction
DreamPose addresses the scarcity of informative fashion videos by animating still photographs with image-and-pose-conditioned diffusion. It modifies Stable Diffusion and uses tailored conditioning and fine-tuning to preserve appearance while producing temporally consistent motion.
- Fashion photographs omit garment draping and flow, while fashion videos provide these details but are relatively rare.
- DreamPose generates realistic animated fashion videos from one or more human images and a driving pose sequence.
- Existing video diffusion models often struggle with realism, temporal consistency, motion control, and detailed appearance, especially when primarily conditioned on text.
- A two-stage fine-tuning strategy balances fidelity to the input image with generalization to new poses.
- DreamPose produces photorealistic fashion videos across diverse garment patterns, shapes, fabrics, identities, viewpoints, and poses.
- The method adapts Stable Diffusion with image-and-pose conditioning, a split CLIP-VAE encoder, and multi-pose inputs to improve appearance fidelity and temporal smoothness.DreamPose combines image embeddings from CLIP and VAE with pose representations, including five consecutive poses surrounding the target pose.
3. Background
Latent diffusion models generate images by denoising random noise in an autoencoder’s latent space, while conditioning and classifier-free guidance steer the denoising process. Stable Diffusion uses a VAE and denoising UNet to reconstruct the final image.
- Stable Diffusion performs diffusion and denoising in an autoencoder’s latent space, reducing computational requirements with minimal perceptual-quality loss.
- Stable Diffusion consists of a variational autoencoder and a denoising UNet.
- Conditional information c is represented by embeddings such as text, image, or segmentation-mask features.
- Classifier-free guidance uses conditional and unconditional noise predictions, scaled by guidance weight s, to steer sampling toward the conditional distribution.
4. Method
DreamPose adapts Stable Diffusion into an image- and pose-conditioned fashion animation model, combining architectural changes, multi-pose conditioning, staged fine-tuning, and dual guidance for temporally consistent video generation.
- Overview: DreamPose fine-tunes pretrained Stable Diffusion on fashion videos after adapting it to accept image and pose conditioning for temporally consistent output.The inference process generates video frames from a single image and a pose sequence.
- Image conditioning: A custom adapter combines CLIP image and VAE embeddings into the cross-attention conditioning used by the denoising U-Net.The VAE-conditioning weights are initialized to zero to reduce training shock during fine-tuning.
- Pose conditioning: Five consecutive pose frames condition each output, reducing pose-jitter effects and increasing motion smoothness and temporal consistency.The pose representation uses pi-2 through pi+2 around the target pose pi.
- Fine-tuning: Training proceeds in two stages: dataset-wide fine-tuning, followed by subject-specific refinement of the U-Net, adapter, and VAE decoder.The second phase creates a subject-specific model for inference from one or more input images.
- Fine-tuning: Subject-specific fine-tuning preserves person and garment identity, while image-and-pose augmentation prevents artifacts such as texture-sticking.Random cropping is one example of the augmentation used during training.
- Fine-tuning: VAE-decoder fine-tuning recovers sharper, more photorealistic details and reduces high-frequency noise compared with fine-tuning the U-Net alone.Decoder fine-tuning also increases pose guidance without sacrificing person identity or appearance.
- Inference: Dual classifier-free guidance independently controls similarity to the input image and alignment with the target pose during frame-by-frame inference.A large sI increases appearance fidelity, while a large sp increases pose alignment and helps prevent overfitting to one input pose.
5. Results
DreamPose generates photorealistic fashion videos across diverse appearances and poses, preserving identity, garment details, and temporal quality. Quantitative, qualitative, and ablation results support its design choices and performance against competing methods.
- DreamPose handles diverse human and clothing appearances, viewpoints, loose garments, and driving poses from different videos.
- Guidance and Decoder Finetuning: VAE decoder finetuning improves appearance and person identity even under larger relative pose guidance.The guidance study varies image and pose classifier-free guidance weights.
- Quantitative Analysis: DreamPose outperforms MRAA, TPSMM, and PIDM on L1, SSIM, VGG, LPIPS, FID, FVD, and AED.Evaluation uses 100 UBC Fashion test videos at 256px resolution, with 50 frames per video.
- Qualitative Analysis: Compared with MRAA, TPSMM, and PIDM, DreamPose better preserves face identity, fabric folds, and fine clothing patterns during pose changes.Other methods lose these details; MRAA may also produce disjointed or blurry limbs, while PIDM varies in identity and dress appearance across frames.
- Ablation Studies: The full model outperforms ablated variants across L1, SSIM, VGG, and LPIPS.The ablations remove or alter the CLIP-VAE encoder, VAE decoder finetuning, or five-pose input.
- Ablation Studies: Subject-specific UNet finetuning preserves photorealistic face and garment details, while VAE decoder finetuning improves sharpness without overfitting to the input pose.A single input pose causes noticeable flickering around the feet and hair.
6. Limitations & Future Work
DreamPose has failure cases involving fabric–limb confusion, hallucinated dress features, backward-facing pose misalignment, and flickering on complex patterns. The method also has substantial finetuning and rendering costs, motivating future improvements in data, pose estimation, segmentation, and temporal consistency.
- Rare failures include limbs disappearing into fabric, hallucinated dress features, and directional misalignment for backward-facing poses.The authors suggest improved pose estimation, a larger dataset, or segmentation masks as possible remedies.
- Large and complex fabric patterns can produce minor flickering, and improving their temporal consistency without subject-specific finetuning remains future work.
- Subject-specific finetuning takes approximately 10 minutes for the UNet and 20 minutes for the VAE decoder, while rendering takes 18 seconds per frame.The authors note that these times are slow compared with GAN or VAE methods.
7. Conclusion
The paper presents DreamPose as a diffusion-based method for animating still fashion images into photorealistic videos. From a single image and pose sequence, it animates diverse fabrics, patterns, and person identities.
- DreamPose generates photorealistic fashion videos from a single image and pose sequence across diverse fabrics, patterns, and person identities.
1. Implementation Details
The implementation trains DreamPose on two NVIDIA A100 GPUs using staged UNet finetuning and classifier-free conditioning dropout.
- The first training phase finetunes the base UNet for 5 epochs at a 5e-6 learning rate with an effective batch size of 16.Training uses 4 gradient accumulation steps at 512x512 resolution.
- Pose, image, or joint conditioning inputs are replaced by null values with 5% probability during training.
2. User Studies
Two user studies evaluated DreamPose against existing image-animation methods and assessed how closely its videos matched input images. Participants preferred DreamPose over MRAA and TPSMM, while 85% rated its videos at least 3 out of 5 for realistic input-image animation.
- The studies involved 50 Mechanical Turk workers and 50 unique test-set input images from the UBC Fashion dataset.
- Workers compared DreamPose with MRAA and TPSMM using randomized pairwise video presentations.
- Participants preferred DreamPose over both MRAA and TPSMM for quality and similarity to the input image.
- 85% of workers rated DreamPose videos 3 or higher on a 0–5 scale of realistic animation matching the input image.
3. Different Videos for Source Person and Driving Pose Sequence
DreamPose can drive one subject’s animation with a pose sequence taken from a different person and garment. This separates source appearance from motion and supports animation when matching ground-truth motion is unavailable.
- DreamPose animates an input image using motion from a video featuring a different person and garment identity.
- The method is applicable when ground-truth motion for the input subject is unavailable.
- Figure 11 presents qualitative results conditioning on subject and pose from different videos.
4. Multiple Input Frames
DreamPose supports multiple input images and remains effective across pose transfer and unseen fashion images. Additional views improve consistency, while the method preserves fine details and generalizes beyond its training dataset.
- Multiple Input Frames: Additional input images increase animation quality and viewpoint consistency beyond DreamPose’s single-image capability.
- DeepFashion Results: Despite training exclusively on UBC Fashion, DreamPose performs well on unseen DeepFashion retail images with new backgrounds, identities, accessories, and patterns.
- Pose Transfer Results: For pose transfer, DreamPose better preserves shoe appearance, hemlines, and face identity than DynaST or PIDM.
- Multiple Input Frames: Training with 1, 3, 5, and 7 input images improves pose fidelity, facial identity, and color as the number of frames increases.