Source-linked AI summary
DwNet: Dense warp-based network for pose-guided human video generation
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, Leonid Sigal
TL;DR
The paper tackles realistic high-resolution human motion transfer from a single source image to motions shown in a driving video. DwNet uses dense pose-guided texture warping, refinement, and previous-frame conditioning to generate temporally coherent videos, with superiority over baselines on two datasets.
Problem
The problem is generating high-resolution videos in which a human subject from one image performs motions expressed by an auxiliary driving video while preserving realism and temporal coherence.
Method
DwNet uses dense pose-guided representations, texture warping and correction, and recurrent conditioning on previously generated frames within a GAN.
Results
DwNet shows improved performance over state-of-the-art baselines on two video datasets, with users significantly more often preferring its videos to competitors’ videos.
Takeaways & Limitations
The approach produces high-quality pose-guided human videos from a source image and driving video while addressing temporal inconsistency and warping errors.
Abstract
from arXiv · showhide
Generation of realistic high-resolution videos of human subjects is a challenging and important task in computer vision. In this paper, we focus on human motion transfer - generation of a video depicting a particular subject, observed in a single image, performing a series of motions exemplified by an auxiliary (driving) video. Our GAN-based architecture, DwNet, leverages dense intermediate pose-guided representation and refinement process to warp the required subject appearance, in the form of the texture, from a source image into a desired pose. Temporal consistency is maintained by further conditioning the decoding process within a GAN on the previously generated frame. In this way a video is generated in an iterative and recurrent fashion. We illustrate the efficacy of our approach by showing state-of-the-art quantitative and qualitative performance on two benchmark datasets: TaiChi and Fashion Modeling. The latter is collected by us and will be made publicly available to the community.
1 Introduction
The paper addresses high-resolution human motion transfer from one source image to a driving video, using dense pose-guided warping and recurrent conditioning to preserve appearance and temporal coherence.
- High-resolution human video generation must simultaneously maintain realistic frames, temporal coherence, coherent subject appearance, and realistic human motion.
- The task generates a high-resolution video of a source subject from one image, conditioned on motion expressed in an auxiliary driving video.
- Dense pose-guided texture warping transfers the source subject’s appearance to target poses while addressing dense pose errors.
- Each generated frame is conditioned on the previously generated frame, supporting more temporally coherent output.
- The approach is evaluated against recent state-of-the-art methods and accompanied by a new high-resolution fashion-video dataset.
2 Related work
Prior work spans conditional image and video generation, pose-guided synthesis, and video-to-video translation. DwNet differs by generating motion-transfer videos from a single source image while using richer pose structure and temporal conditioning.
- Image Generation: Image generation has progressed through VAEs and GANs, while conditional generation adds information beyond a noise vector.
- Pose guided image generation: Pose-guided image generation evolved from keypoint-based models toward architectures that spatially align source and target representations.
- Video Generation: Video generation remains less explored than image generation, with prior pose-guided video methods often requiring a separate network for each person.
- Video Generation: Unlike prior single-image approaches, DwNet exploits and refines richer structure for pose-guided video generation from one image.
3 Method
DwNet generates human-motion-transfer videos by combining dense pose-guided warping with recurrent frame generation. Its warp module refines DensePose-based correspondences to preserve source appearance, while the decoder conditions each frame on the previous output.
- Temporal generation: Each generated frame is conditioned on the previously generated frame to reduce temporal inconsistency and flickering across the video.The recurrent formulation also uses the source frame as the previous frame when generating the first target output.
- Warp module: DensePose provides 3D pixel-to-body-part correspondences for estimating coarse warp grids between source, driving, and successive target poses.These grids support texture transfer and motion-flow estimation, but DensePose can produce false detections, missing parts, and no clothing information.
- Overall architecture: The generator uses a pose encoder, warp module, and decoder to produce video frames iteratively from a source image and driving poses.The decoder combines encoded driving-pose features with warped representations of the source image and previously generated frame.
- Warp module: A refinement branch predicts corrections to coarse warp grids using the source image, coarse grid, and target pose.The refined grid is added to the coarse estimate and applied to the source feature representation; corrections are produced at the 64 × 64 representation resolution using relative grid shifts.
- Training: Training samples four frames from one video, treating one as the source and three consecutive frames as driving and ground-truth target frames.The authors report that four frames balance temporal consistency and computational efficiency.
- Training: The objective combines least-square adversarial, feature-matching, and perceptual losses, with the total loss using λ = 10.The perceptual loss compares VGG-19 features, while feature matching uses discriminator features.
4 Experiments
DwNet is evaluated on Fashion and Tai-Chi using perceptual loss, FID, AKD, user preference, qualitative comparisons, and ablations. It outperforms the compared methods across both datasets and benefits from recurrent conditioning and warp refinement.
- Datasets: Experiments evaluate DwNet on Fashion and Tai-Chi, including a newly collected high-resolution Fashion dataset and a 256×256 Tai-Chi version.The Fashion dataset contains 500 training and 100 test videos; Tai-Chi contains 3049 training and 285 test videos.
- Evaluation: Performance is assessed with perceptual loss for texture similarity, FID for frame realism, AKD for motion transfer, and an AMT user study for overall quality and temporal coherence.The user study compares DwNet videos with competing methods in randomized pairwise choices.
- Comparison with the state-of-the-art: DwNet outperforms MonkeyNet and Coordinate Inpainting on both datasets and across all reported metrics.The authors attribute MonkeyNet’s weakness to complex-pose handling and Coordinate Inpainting’s weakness to independently generated frames and uncorrected DensePose artifacts.
- Comparison with the state-of-the-art: User studies found DwNet videos significantly more often preferred over videos from competing models.Table 2 reports the percentage of pairwise preferences for the proposed method.
- Ablation: Removing previous-frame conditioning worsens realism and real-image feature similarity, while removing warp refinement produces distorted faces and removing all warping produces blurry, inconsistent textures.The ablations connect recurrent conditioning with temporal coherence and warp modules with image realism and texture consistency.
5 Conclusion
DwNet generates pose-guided human videos from a source image and a driving video. Its recurrent modeling addresses temporal inconsistency, while warp correction addresses warping errors, and experiments on two datasets show superiority over baselines.
- DwNet generates high-quality videos of a source subject performing motions from a driving video.The method uses a source image for human appearance and a driving video for motion.
- The model uses Markovian conditioning to address temporal inconsistency and a warp module to correct warping errors.The conclusion identifies both components as central design choices.
- Experiments on two video datasets show superiority over the baselines.The conclusion also identifies multiple-source generation and DensePose improvement as future directions.