PaperlayerView exact revision on arXiv

Source-linked AI summary

Everybody Dance Now

Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros

arXiv:1808.07371v2cs.GRcs.CV

TL;DR

Transferring a source performer’s motion to a target is difficult without frame-matched training pairs. The paper uses pose-guided video translation and reports compelling, high-quality motion-transfer results, including across different performers and appearances.

  • Problem

    Motion transfer lacks exact frame-to-frame pose correspondences between source and target subjects, making supervised image-to-image translation difficult.

  • Method

    The method detects and normalizes poses, maps them to the target subject, models temporal coherence, and enhances faces with a dedicated generator.

  • Results

    The method generates compelling, high-quality, detailed motion-transfer videos, including complex motions from realistic in-the-wild inputs.

  • Takeaways & Limitations

    The approach enables untrained amateurs to perform transferred motions such as ballet spins, martial-arts kicks, and vibrant dances.

  • Takeaways & Limitations

    The model struggles with visual artifacts from loose clothing or hair, missed body parts, and high-frequency shirt textures.

Abstract

from arXiv · show

This paper presents a simple method for "do as I do" motion transfer: given a source video of a person dancing, we can transfer that performance to a novel (amateur) target after only a few minutes of the target subject performing standard moves. We approach this problem as video-to-video translation using pose as an intermediate representation. To transfer the motion, we extract poses from the source subject and apply the learned pose-to-appearance mapping to generate the target subject. We predict two consecutive frames for temporally coherent video results and introduce a separate pipeline for realistic face synthesis. Although our method is quite simple, it produces surprisingly compelling results (see video). This motivates us to also provide a forensics tool for reliable synthetic content detection, which is able to distinguish videos synthesized by our system from real data. In addition, we release a first-of-its-kind open-source dataset of videos that can be legally used for training and motion transfer.

1. Introduction

The paper proposes a simple pose-mediated video-to-video method that transfers a source performer’s motion onto a different target subject, producing compelling synthesized videos. It also contributes synthetic-video detection and an open-source dataset for motion transfer and fake detection.

  • 1. Introduction: The method transfers motion from a source video to a target subject by learning a video-to-video mapping between their appearances.It enables amateurs to perform motions such as ballet, martial arts, and vibrant pop-star dancing without performing the source sequence themselves.
  • 1. Introduction: The approach addresses the lack of paired frames by avoiding direct image translation between subjects who did not perform matching motions frame by frame.Even performing the same routine would not reliably produce exact pose correspondences for supervised learning.
  • 1. Introduction: Pose stick figures serve as an intermediate representation because they preserve motion over time while abstracting away subject identity.The approach uses off-the-shelf human pose detectors such as OpenPose [6] [34] and learns pose-to-image translation.
  • 1. Introduction: The paper reports compelling, high-quality motion-transfer results from realistic in-the-wild videos and introduces a tool to detect videos synthesized by the method.The detection application is motivated by the quality of the generated results.
  • 1. Introduction: The released dataset combines five long single-dancer videos for training and evaluation with many short YouTube videos for transfer and fake detection.The single-dancer data are designated as high-resolution open-source data for motion-transfer and video-generation research.

2. Related Work

Prior motion-transfer work spans video manipulation, optical-flow retargeting, 3D graphics, multi-camera capture, and deep generative models. This work instead targets full-body transfer between human subjects using 2D pose representations, personalized video data, and image-to-image translation.

  • Related Work: Earlier methods manipulated existing footage, used optical flow for “Do as I do” and “Do as I say” retargeting, or applied inverse-kinematic 3D graphics.These approaches include synthesizing unuttered speech and retargeting motion between animated characters.
  • Related Work: Calibrated multi-camera systems scan target actors, estimate 3D joints, fit personalized kinematic models, and render new motions.Such methods depend on elaborate multi-view capture and 3D modeling.
  • Related Work: Deep reanimation methods synthesize facial motion or rigged 3D characters [19], while MoCoGAN and Dynamics Transfer GAN [2] disentangle motion from appearance for novel-motion video generation.Unlike these works, the present method transfers full-body motion between human subjects using a skeleton-like intermediate representation.
  • Related Work: Pose-conditioned image and video generation has introduced architectures and losses for new human poses [27] [28] [33], but this work learns one known person’s style from personalized video data for detailed dancing videos.The distinction is between generating views of an unseen person from one image and synthesizing a known person from substantial video data.
  • Related Work: Concurrent methods learn video mappings for facial or pose-to-body transfer [1] [4] [24], including a method with similar quality but greater complexity and computational cost.The approach is enabled by advances in robust pose estimation, including OpenPose [6] [34] and DensePose, and image-to-image translation.

3. Method

The method transfers a source performer’s motions to a target subject through pose-conditioned video-to-video translation. It combines pose detection, global pose normalization, adversarial image synthesis, temporal smoothing, and specialized face refinement.

  • Pipeline: The pipeline detects source poses, normalizes them to the target’s body shape and location, and maps the resulting stick figures to target-person images using adversarial training.This separates motion extraction, retargeting, and appearance synthesis into three stages.
  • Global pose normalization: Global pose normalization transforms source keypoints using pose heights and ankle positions so transferred motions match the target’s limb proportions and camera location.This addresses differences in body shape and subject distance across videos.
  • Temporal smoothing: Because single-frame translation produces temporal artifacts and misses fine human-motion details, the method predicts adjacent frames jointly and adds a learned temporal-coherence model.The first frame uses a zero placeholder for the unavailable earlier generated frame, while the next frame is conditioned on the preceding output.
  • Face GAN: A face GAN refines a 128 × 128 nose-centered crop by predicting a residual added to the main generator’s face output, with perceptual reconstruction against the ground-truth face.The full-image GAN and face GAN are trained in separate stages, with the full-image weights frozen during face-GAN optimization.

4. Experiments

Experiments show that the proposed method is preferred over nearest-neighbor and PoseWarp baselines, while temporal smoothing and the Face GAN each improve video quality and realism. Evaluation combines Mechanical Turk perceptual studies with SSIM and LPIPS measurements on held-out target-subject data.

  • Evaluation setup: The evaluation uses 10-second transfer snippets across filmed target subjects, with 100 distinct MTurk participants per task viewing 144×256 video pairs.The study compares the proposed method against baselines and ablated conditions using perceptual judgments, while ablation quality is additionally measured with SSIM and LPIPS.
  • Comparison to baselines: 91.2% and 79.1%: the FBF+TS ablation was preferred over Nearest Neighbors and PoseWarp, respectively, showing the advantage is not solely due to Face GAN synthesis.The full model was preferred slightly more often than FBF+TS.
  • Ablation study: SSIM and LPIPS are similar across model variants on body regions and full images, but the full model improves face-region results over FBF+TS.Full-image similarity is limited because all variants can generate the static background, while the face residual generator distinguishes the complete model.
  • Ablation study: 58.8% and 53.3%: the full model was preferred over frame-by-frame synthesis and FBF+TS, respectively, in perceptual studies.These results indicate that temporal information improves video results and that the Face GAN provides an additional overall benefit.
  • Qualitative results: Temporal modeling produces greater frame-to-frame coherence, while the specialized facial generator and discriminator add facial detail and realism.The temporal smoothing effect is most visible in qualitative video results and perceptual studies, whereas the Face GAN advantage is also apparent in single-frame comparisons.

5. Detecting Fake Videos

The paper trains a two-frame fake-video detector to identify generated sequences using both frame fidelity and temporal consistency. Evaluated on 14 held-out subjects, it distinguishes real and fake videos across source motions, with lowest accuracy for same-person transfer and highest for Misty Copeland motion.

  • Detector design: The fake-detector classifies two consecutive frames as real or fake, exploiting both individual-frame fidelity and temporal consistency.It is trained in parallel with the synthesis process using ground-truth frames as real examples and generated frames as fake examples.
  • Data and training: Training uses 48 subjects from public 1920 × 1080 dancing videos, with separate synthesis models generating fake content for detector training.Fourteen subjects are held out for testing, and the multi-subject training setup is intended to improve generalization.
  • Evaluation: The detector is evaluated on both same-subject and different-subject source motions, including Bruno Mars and Misty Copeland, for each held-out target subject.This tests whether detection generalizes across the source-target relationship used to synthesize the fake videos.
  • Results: Fake-detection accuracy is lowest for same-person motion transfer and highest when motion comes from prima ballerina Misty Copeland.The evaluation covers synthesized videos for 14 held-out test subjects and reports results in Table 5.

6. Potential Applications

The system supports synchronized multi-subject dance videos and has also enabled commercial augmented-reality and in-game entertainment applications.

  • Multi-subject synchronization: The system can drive multiple trained target subjects with one source video, making them perform the same dance moves synchronously (Figure 8).This can be used to create family reunion videos with multiple subjects.
  • Commercial applications: Prototype-based systems have been used commercially for an augmented-reality stage piece with a floating 3D-rendered dancer and an in-game application making NBA players dance.

7. Limitations and Discussion

The model can generate arbitrarily long, good-quality dance videos from source movements, but its performance is limited by pose representation, detection failures, and visual artifacts. Despite mismatches in limb lengths and camera positions, it generalizes fairly well to a wide range of unseen source motions.

  • Overall limitations: The model usually generates arbitrarily long, good-quality videos of a target dancer following a source dancer’s movements, but it still has several limitations.The method’s capabilities are qualified by the limitations discussed below.
  • Visual artifacts: Loose clothing and hair, missed limbs from OpenPose, and texture artifacts in shirt creases can degrade synthesized video quality.The authors suggest using target videos with varied clothing or scene lighting, improving pose detection, and mitigating artifacts.
  • Pose normalization: Pose normalization does not account for different limb lengths or camera positions, widening the gap between training and testing motions.These discrepancies remain a limitation of the pose representation.
  • Motion generalization: Despite pose mismatches, the model generalizes fairly well to new motions and learns one model covering a wide range of source movements without requiring similar target motions.When filming a target sequence, no specific source motion is assumed, so the target need not perform motions resembling any source.

8. Appendix

The appendix reports comparable quality against vid2vid at 512 × 256, implementation details, dataset collection practices, and pose normalization procedures for matching source and target framing.

  • Implementation details: The model modifies pix2pixHD architectures for temporal prediction, uses progressive 512 × 256 then 1024 × 512 synthesis, and adds a Patch-GAN face-residual pipeline.Training uses λP = 5 and λV GG = 10, with the global stage trained for 5 epochs on the collected dataset; perceptual loss compares five VGG-19 layer outputs.
  • Dataset: The released target-video dataset contains self-filmed 8–17-minute footage from legally consenting subjects, while source videos are in-the-wild single-dancer clips with static cameras.The target set includes four 1920 × 1080 videos and one 1280 × 720 video, with subjects recruited from varied sources.
  • Comparison to vid2vid: At 512 × 256, our system and vid2vid generally perform similarly, producing comparable-quality results in the qualitative comparison.The high-resolution comparison was infeasible because vid2vid requires excessive memory and computing power; Figure 10 compares models without a Face GAN.
  • Pose normalization: Pose normalization translates and scales source keypoints to match the target’s camera-relative position and apparent size, keeping ankle y positions aligned with the ground axis.The method estimates close and far positions from ankle coordinates, uses ankle-to-nose Euclidean distance for subject height, and applies separate close/far scale factors.
  • Pose normalization: The far-position heuristic clusters ankle coordinates above the median, with α = 0.7 generally effective but tunable for camera height and subject motion range.Close position is determined by the maximum average ankle y coordinate across frames.
Loading 1808.07371v2…