Deep Video-Based Performance Cloning

1 INTRODUCTION

• Full-body motion cloning remains challenging because clothed, articulated humans exhibit richer variation than faces, for which dense motion fields and controllable models are more available.Source: §1 · p. 1
• The approach transfers motions from a driving video to a target individual captured in a reference video without requiring motion capture or depth information.Source: §1 · pp. 1–2
• The method extracts 2D poses from a target actor’s reference video and self-supervises frame generation, then uses unpaired pose sequences to improve generalization beyond observed motions.Source: §1 · p. 2
• A shared space-time conditional generator is trained through paired reconstruction and adversarial losses plus unpaired temporal-consistency and adversarial losses.Source: §1 · p. 2
• Success depends on the richness of paired reference data, relevance of unpaired pose sequences, and similarity between driving poses and training poses.Source: §1 · p. 2
• The generated performances are temporally coherent and preserve target identity well, including challenging reference-driving combinations, although they exhibit typical GAN-generated artifacts.Source: §1 · p. 2

Image Translation.

• Prior image-to-image translation methods use paired data or, in unsupervised variants, two sets of unpaired samples for cross-domain translation.Source: Image Translation · p. 2
• The paper combines self-supervised paired training with unpaired training and customized losses to translate 2D human pose sequences into photorealistic videos of a particular target actor.Source: Image Translation · p. 2

Human Pose Image Generation.

• Other human-generation approaches either lack articulation control, sample new latent features from Gaussian noise, or do not model temporal coherence.Source: Human Pose Image Generation · p. 2
• Earlier human pose image-generation methods require paired data, often rely on a single target image, and generally do not address temporal coherence or consistently preserve identity.Source: Human Pose Image Generation · p. 2

Deep Video Generation.

• Deep video-generation methods synthesize plausible futures or decompose temporal and image generation, but do not provide the detailed control required here.Source: Deep Video Generation · p. 2
• Unlike future-pose prediction and latent content-motion generation, this work receives poses from a driving video and renders photorealistic frames of a specified target actor.Source: Deep Video Generation · p. 3
• Facial reenactment benefits from dense motion fields or controllable facial models, whereas comparable full-body motion reenactment from plain video remained unavailable.Source: Performance Reenactment · p. 3

Performance Reenactment.

• Performance-driven animation typically targets rigged 3D characters, while this paper targets a real person captured using ordinary video.Source: Performance Reenactment · p. 3

3 POSES-TO-VIDEO TRANSLATION NETWORK

• The method maps coherent 2D human-pose sequences to video frames depicting a target actor, using pose confidence maps extracted from ordinary videos.Each pose has one Gaussian confidence map per joint, with 18 joints represented in a spatial volume; videos are aligned and normalized across actors.Source: §3 · p. 3, §3.1 · p. 3

3.1 Paired and Unpaired Data

• Paired reference-video frames train the generator to approximate the inverse pose extractor, while unpaired pose sequences extend translation to motions absent from the reference video.The paired data contain target-actor frames and corresponding poses; unpaired data contain poses from other videos without target-actor ground-truth frames.Source: §3.2 · p. 3, §3.1 · p. 3

3.2 Network Architecture

• A single shared generator is trained through paired reconstruction and adversarial losses plus unpaired temporal-consistency and adversarial losses.The total objective combines reconstruction, unpaired-branch adversarial, and temporal-coherency terms; training begins with the paired branch before enabling the unpaired branch.Source: §3.2 · pp. 3, 5, Figure 2 · p. 4
• The paired reconstruction loss uses low-level VGG-19 features and a conditional PatchGAN, producing better reconstructions than naïve RGB ℓ1 loss.VGG features encourage small-scale patterns and textures, while the reported comparison shows RGB ℓ1 yields blurrier images; the advantage remains after adding the adversarial component.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
• The unpaired branch enforces temporal coherence by warping generated frames with driving-video optical flow and comparing them with consecutive generated frames.Differences are weighted more strongly near the skeleton, where smooth motion is required; a single-input PatchGAN discriminator encourages target-video-like patches and counters oversmoothing.Source: §3.2 · p. 5
• The generator processes N = 2 consecutive poses as a space-time tensor, while paired and unpaired PatchGAN discriminators are extended to space-time volumes.The input has JN × H × W dimensions and the output has 3N × H × W dimensions; the two discriminators differ in whether they receive conditional pose channels.Source: §3.2 · p. 5
• Performance depends on reference-video richness, the relevance of unpaired pose sequences, and similarity between driving poses and paired training poses.Because joints do not explicitly control details such as hands, shoes, or faces, unseen poses similar to paired examples perform better; about 3000 target-actor frames are typically sufficient for training.Source: §3.2 · p. 5

• Pose similarity should be measured locally and translation-invariantly because PatchGAN can combine familiar local patches into generated frames.The metric therefore focuses on limb-level similarity rather than whole-pose alignment.Source: §4 · p. 5

Skeleton Descriptor

• The skeleton descriptor represents 12 limbs using their 2D lengths and orientations, making comparisons invariant to image-plane translation.Length captures scale differences, while orientation captures rotated limbs absent from training.Source: §4 · pp. 5–6

Pose to Pose Metric

• Pose-to-pose distance averages corresponding-limb differences, and limbs exceeding threshold 8 are marked as unlikely to reconstruct well.Translation alone does not make a limb problematic when its orientation and length remain similar.Source: §4 · p. 6, Figure 4 · p. 6

Pose to Sequence Metric

• Pose-to-sequence distance finds each driving limb’s nearest neighbor across the reference sequence and averages those distances.This measures whether each limb has a training counterpart with similar length and orientation.Source: §4 · p. 6
• Small pose-to-sequence distances accompany successful reconstruction of unseen combined arm poses, whereas horizontal arm motions yield high distances and poor reconstruction.The reference sequence contained vertical single-arm motion but no horizontal arm motion.Source: §4 · p. 6, Figure 5 · p. 6

• Small pose distances do not guarantee reconstruction quality because body shape, clothing, pose-extraction accuracy, and viewpoint differences are not fully represented.The metric also ignores how frequently a limb appears in the training data.Source: §4 · p. 6

5.3 Online performance cloning

• The system processes pose extraction and generation in 80 ms per frame, allowing 25 fps cloning in real time with two or more GPUs, although quality depends on paired-data diversity.Pose extraction takes 35 ms per frame and generation takes 45 ms per frame.Source: §5 · p. 7, §5.3 · p. 9

5.1 Comparison to Other Algorithms

• Our method generates higher-quality frames than CycleGAN and pix2pix in qualitative comparisons, while the baselines show geometric distortion or temporal artifacts.CycleGAN mainly changes texture without properly modifying body geometry, whereas pix2pix struggles with unseen poses and flickering.Source: §5.1 · p. 7, Figure 7 · p. 8
• Our method preserves target identity better than pose-guided person image generation, which does not address smooth video generation.The comparison with Ma et al. uses the same input pose; the cited model fails to preserve actor identity and is unlikely to translate pose sequences into smooth video.Source: §5.1 · p. 7, Figure 8 · p. 8

5.2 Quantitative Evaluation

• In self-reenactment, our approach consistently achieves lower RGB MSE than CycleGAN and pix2pix across three actors, while nearest-neighbor poses produce larger errors.The test trains on two thirds of a dance sequence and drives the remaining third, whose poses are seldom exact repeats of training poses.Source: §5.2 · pp. 7–8, Table 1 · p. 8, Figure 9 · p. 8
• MSE falls from 18.5 with 1200 training frames to 15.4 with 3400 frames, with the full training set producing the best visual results.The evaluated training-set sizes are 1200, 2000, and 3400 frames from one actor’s reference sequence.Source: §5.2 · pp. 8–9, Figure 10 · p. 9
• Average MSE improves from 17.5 with paired-branch training alone to 16.2 with both branches, while the paired-only results exhibit more temporal artifacts.This ablation is repeated for three different reference videos.Source: §5.2 · p. 9

• The method generates target-performer frame sequences that imitate motions from unseen driving videos, addressing the challenge of transferring substantially different performances.Source: §6 · p. 9
• GAN-generated videos still exhibit artifacts in fine details, especially faces and high-frequency or edge regions.Source: §6 · p. 9
• Performance transfer depends on reference–driving pose similarity and similar skeleton sizes and proportions, preventing transfer between grownups and children.Source: §6 · p. 9
• The current implementation does not generate backgrounds, uses interactive extraction, and relies heavily on a pose-extraction network with its own limitations.Source: §6 · p. 9
• Future work proposes universal and local discriminators to improve motion realism and fine details in faces, hands, and feet for amateur-to-professional performance enhancement.Source: §6 · p. 9