Source-linked AI summary
DisCo: Disentangled Control for Realistic Human Dance Generation
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, Lijuan Wang
TL;DR
Realistic social-media dance generation requires generalizability to unseen humans, backgrounds, and poses, plus compositionality across separately sourced attributes. DISCO addresses these needs with disentangled control and human attribute pre-training, achieving strong image and video generation results while remaining limited in hand-posture control and multi-person or human-object scenarios.
Problem
Realistic social-media dance generation challenges motion-transfer methods because they struggle with diverse visual content and require generalizability and compositionality across humans, backgrounds, and poses.
Method
DISCO combines a disentangled-control architecture for compositional generation with human attribute pre-training for generalization to unseen humans and non-generic views.
Results
DISCO generates high-quality human dance images and videos, outperforming DreamPose across reported metrics; adding HAP reduces FID to ∼38 and FVD to ∼280.
Takeaways & Limitations
DISCO demonstrates effective realistic dance synthesis with diverse appearances, flexible motions, and surprisingly good temporal consistency without explicit temporal modeling.
Takeaways & Limitations
The paper treats subject-specific fine-tuning as a separate setting discussed in Appendix B rather than the main reported setting.
Abstract
from arXiv · showhide
Generative AI has made significant strides in computer vision, particularly in text-driven image/video synthesis (T2I/T2V). Despite the notable advancements, it remains challenging in human-centric content synthesis such as realistic dance generation. Current methodologies, primarily tailored for human motion transfer, encounter difficulties when confronted with real-world dance scenarios (e.g., social media dance), which require to generalize across a wide spectrum of poses and intricate human details. In this paper, we depart from the traditional paradigm of human motion transfer and emphasize two additional critical attributes for the synthesis of human dance content in social media contexts: (i) Generalizability: the model should be able to generalize beyond generic human viewpoints as well as unseen human subjects, backgrounds, and poses; (ii) Compositionality: it should allow for the seamless composition of seen/unseen subjects, backgrounds, and poses from different sources. To address these challenges, we introduce DISCO, which includes a novel model architecture with disentangled control to improve the compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans. Extensive qualitative and quantitative results demonstrate that DisCc can generate high-quality human dance images and videos with diverse appearances and flexible motions. Code is available at https://disco-dance.github.io/.
1. Introduction
DISCO reframes social-media dance generation around generalizability and compositionality, addressing limitations of motion-transfer and ControlNet-based methods. Its disentangled architecture and human attribute pre-training target diverse subjects, backgrounds, poses, and appearances.
- GAN-based motion-transfer methods struggle with diverse dance content because mode collapse captures only a limited portion of the real data distribution.
- Diffusion and ControlNet improve synthesis and geometric controllability, but existing methods rely on coarse text guidance or substitute reference images for text conditions.
- Social-media dance generation demands generalizability to unseen subjects, backgrounds, poses, and viewpoints, plus compositionality across separately sourced elements.
- DISCO uses disentangled control for compositionality and human attribute pre-training for generalizability to unseen humans and non-generic views.
- 28.31 v.s 53.78 FID and 55.17 v.s 66.36 FID-VID show DISCO outperforming state-of-the-art approaches without temporal consistency modeling.Adding temporal modeling further boosts FID-VID to 29.37.
2. Related Work
Related work progresses from motion-transfer and multi-stage dance-synthesis systems toward diffusion-based controllable generation. These approaches improve efficiency and control but still leave challenges for realistic, diverse social-media dance content.
- Stable Diffusion provides efficient latent-space generation, while ControlNet adds controls such as sketches, human skeletons, and segmentation beyond text.
- The paper positions social-media dance around human attributes and dance poses, including identity, clothing, hairstyle, and movement.
- Human dance synthesis includes video-to-video synthesis, still-image animation, and motion transfer.
- Earlier methods require lengthy target-person videos for fine-tuning or multiple networks and cascaded stages for background, motion, and occlusion prediction.
- Diffusion models simplify training and motivate follow-up methods tailored to human dance generation.
3. DISCO
DISCO formulates dance generation as conditioning on foreground, background, and pose, then separates their controls and pre-trains human attributes. The resulting design supports faithful, generalizable, and composable generation while optionally modeling temporal continuity.
- 3. DISCO: DISCO generates images or videos from human foreground f, background g, and pose keypoints p, targeting faithful, generalizable, and composable outputs.
- 3.2. Model Architecture with Disentangled Control: Disentangled control combines local foreground image features with separate background and pose controls to preserve attributes while altering pose.
- 3.2. Model Architecture with Disentangled Control: The architecture uses local CLIP foreground embeddings as cross-attention key and value features for fine-grained human semantics.
- 3.2. Model Architecture with Disentangled Control: DISCO's U-Net uses foreground conditioning, while separate ControlNet branches represent pose and background conditions.
- 3.2. Model Architecture with Disentangled Control: Optional one-dimensional temporal convolution or attention layers are added after two-dimensional spatial layers to improve video continuity.
- 3.3. Human Attribute Pre-training: Human attribute pre-training reconstructs whole images from foreground and background features, learning diverse attributes from large-scale images before motion editing.
- 3.3. Human Attribute Pre-training: Freezing U-Net ResNet blocks during pre-training improves reconstruction quality for human faces and fine details.
4. Experiments
Experiments evaluate DISCO on real-world TikTok-style dance data, image editing, video generation, user preference, generalization, and architectural ablations. Results show flexible subject–background–pose composition, strong generation quality, and benefits from HAP and temporal modeling.
- Data and Setup: DISCO is trained on 335 TikTok dance videos and evaluated on 10 videos depicting different people, with separate training and testing identities.The training videos contain single-person dances, while the test split is collected from the web.
- DISCO Applications: DISCO supports five image-editing scenarios combining seen or unseen human subjects, poses, backgrounds, and image views.The scenarios include subject/pose retargeting, unseen poses, unseen subjects, jointly unseen poses and subjects, and full-unseen compositions.
- Main Results: DISCO generates temporally consistent dance videos with faithful human attributes and steady backgrounds, while HAP improves fine-grained appearance details.The comparison uses testing-split subjects, backgrounds, and poses unavailable during training.
- Main Results: FID decreases to ∼38 and FVD to ∼280 after adding HAP, while DISCO without HAP already outperforms DreamPose across reported image and video metrics.The metrics include frame-wise FID, SSIM, LISPIS, PSNR, L1, FID-VID, and FVD.
- Main Results: Adding temporal modeling produces substantial gains in video synthesis metrics, and scaling training to ∼600 TikTok-style videos shows further scaling potential.Table 2 reports the effect of temporal modeling with HAP and classifier-free guidance enabled by default.
- Main Results: Over 80% of 50 user-study participants rated DISCO images and videos at least 4 out of 5.Participants rated overall synthesis quality from 0, worst, to 5, best.
5. Conclusion
The conclusion presents DISCO as a framework for realistic social-media dance synthesis centered on generalizability and compositionality. It reports promising results while identifying limitations in hand posture, multi-person scenes, and human–object interactions.
- Conclusion: DISCO reframes human dance synthesis around generalizability and compositionality for realistic social-media scenarios.The framework combines disentangled control with human attribute pre-training.
- Conclusion: The paper concludes that qualitative and quantitative results demonstrate DISCO’s effectiveness and progress toward real-world applications.The conclusion describes the generated dance quality as promising.
- Limitations: DISCO cannot handle hand posture well without fine-grained hand-pose control and is difficult to apply to multi-person scenes and human–object interactions.These are stated as limitations of the final system.
B. Subject-Specific Finetuning
Subject-specific fine-tuning adapts DISCO to out-of-domain human subjects using limited video frames. The resulting videos preserve human attributes and backgrounds across varied poses.
- Subject-Specific Fine-Tuning: DISCO can be integrated with efficient fine-tuning techniques for subject-specific dance synthesis from multiple images of one human subject.The appendix describes this setting as an extension rather than the paper’s main focus.
- Subject-Specific Fine-Tuning: Subject-specific fine-tuning uses limited videos of a particular subject, including examples involving Elon Musk and anime.The framework is adapted from the main model architecture.
- Implementation Details: Fine-tuning runs for 500 iterations on 2 NVIDIA V100 GPUs using 3–10-second videos, randomized crops, 256 × 256 images, and learning rate 1e−3.The source videos contain relatively simple poses.
- Qualitative Results: After additional fine-tuning, DISCO preserves faithful human attributes and consistent backgrounds across an extensive range of poses for out-of-domain subjects.The result is presented as evidence of DISCO’s potential as a pretrained checkpoint.
C.1. Adaptable to Other Conditions
DISCO can be extended beyond its demonstrated pose controls to face landmarks and hand gestures for finer-grained control.
- Adaptable Conditions: DISCO can be extended to face landmarks and hand gestures to provide more fine-grained control.The passage identifies these as adaptable control conditions.
C.2. More Quantitative Results
Additional experiments examine architectural choices, guidance scale, motion pre-training, and user-study materials for DISCO. The results support VQ-VAE encoding, a classifier-free-guidance scale of 1.5, and motion pre-training as useful design choices.
- Architecture Ablations: FID 65.14 v.s 83.59 favors pre-trained VQ-VAE encoding over four randomly initialized convolution layers for reference-image conditioning.The pre-trained VQ-VAE produces a more descriptive dense representation of the reference image.
- Architecture Ablations: Adding pose conditioning during human attribute pre-training produces similar downstream results but requires additional pose-estimation annotations.
- Guidance and Motion Prior: Classifier-free-guidance scale 1.5 gives the best quantitative results for both image-wise and video-wise fidelity.
- Guidance and Motion Prior: Motion pre-training with AnimateDiff produces superior temporal smoothness after fine-tuning on TikTok data.The result motivates combining video-based motion pre-training with image-based human-attribute pre-training.
- User Study Materials: User-study materials include rating instructions, examples, and image/video score distributions for human ID, pose, and background alignment.
C.3. Qualitative Results
Qualitative results show that DISCO handles varied image sizes, viewpoints, and body coverage while preserving human attributes and backgrounds across continuous pose sequences.
- Image Editing: DISCO synthesizes partial and full human bodies across large viewpoint and skeleton-rotation changes at different image sizes.The examples include 256 × 384 images designed to include more of the human body.
- Video Generation: ControlNet-only and Attention-only baselines fail to maintain human-attribute and background consistency across continuous poses, unlike DISCO.The comparison reports less visually appealing generations for both baselines.
C.4. User Study
A user study compares DISCO and DreamPose on overall and fine-grained alignment criteria for synthesized images and videos. DISCO receives higher human-ID and background-alignment scores.
- Study Design: The study uses 50 participants, 80 rating questions, and 4,000 responses across 10 synthesized images and 10 videos.Five images and five videos come from each of DISCO and DreamPose.
- Study Design: Ratings cover overall quality, human ID alignment, pose alignment, and background alignment on a 0-to-5 scale.
- Results: DISCO achieves clearly higher human-ID and background-alignment scores than DreamPose.The passage attributes this to human attribute pre-training and disentangled control.
C.5. Ethical Concern
The paper identifies ethical risks associated with realistic dance generation, including deceptive media, training-data bias, and intellectual-property concerns.
- Ethical Concern: DISCO’s broad applicability requires attention to misuse in deceptive media, potential training-data biases, and respect for intellectual property.