Source-linked AI summary
MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, Fangyuan Zou
TL;DR
Video generation needs better controllability, detail, temporal smoothness, and duration than existing approaches provide. MimicMotion addresses these challenges with confidence-aware pose guidance, confidence-based hand-region loss amplification, and progressive latent fusion, and experiments report improved adaptation to noisy poses, hand quality, and temporal smoothness.
Problem
Video generation remains challenging in controllability, video length, imagery detail, and temporal smoothness, while inaccurate pose guidance can cause distorted outputs and training instability.
Method
MimicMotion combines confidence-aware pose guidance, confidence-based hand-region loss amplification, and position-aware progressive latent fusion within a pretrained video-generation framework.
Results
MimicMotion shows improved adaptation to noisy pose estimation, hand quality, temporal smoothness, and visual realism in extensive experiments and ablations.
Takeaways & Limitations
The framework supports high-quality, controllable human-motion videos with long or arbitrary-length pose-sequence guidance.
Takeaways & Limitations
Video-generation technology carries potential misuse risks for deceptive content and therefore requires safeguards such as watermarking, detection methods, and usage guidelines.
Abstract
from arXiv · showhide
In recent years, generative artificial intelligence has achieved significant advancements in the field of image generation, spawning a variety of applications. However, video generation still faces considerable challenges in various aspects, such as controllability, video length, and richness of details, which hinder the application and popularization of this technology. In this work, we propose a controllable video generation framework, dubbed MimicMotion, which can generate high-quality videos of arbitrary length mimicking specific motion guidance. Compared with previous methods, our approach has several highlights. Firstly, we introduce confidence-aware pose guidance that ensures high frame quality and temporal smoothness. Secondly, we introduce regional loss amplification based on pose confidence, which significantly reduces image distortion. Lastly, for generating long and smooth videos, we propose a progressive latent fusion strategy. By this means, we can produce videos of arbitrary length with acceptable resource consumption. With extensive experiments and user studies, MimicMotion demonstrates significant improvements over previous approaches in various aspects. Detailed results and comparisons are available on our project page: https://tencent.github.io/MimicMotion .
1. Introduction
MimicMotion targets pose-guided human video generation with rich details, reference-image fidelity, temporal smoothness, and arbitrary length. It combines confidence-aware pose guidance, hand-region loss enhancement, and progressive latent fusion to address distortion, blurred frames, and long-video discontinuities.
- Motivation: Pose-guided video generation must jointly preserve imagery quality, temporal smoothness, controllability, and substantial video length.The task uses a reference image, raw video frames, and pose guidance to generate detailed human-motion videos.
- Challenges: Existing methods often distort human hands during large movements and sacrifice imagery details for smoother, blurrier frames.These limitations remain common despite prior work on pose-guided video generation.
- Contributions: Confidence-aware pose guidance reduces inaccurate-pose effects by improving temporal smoothness and easing imagery distortion.Confidence is incorporated into pose-sequence representation to handle unreliable guidance.
- Contributions: Hand region enhancement strengthens loss weights for high-confidence hand regions, targeting clearer and more accurate hands.The method focuses regional supervision where pose confidence indicates reliable hand guidance.
- Contributions: Position-aware progressive latent fusion generates arbitrary-length videos while improving smoothness and richness across segment boundaries.The approach generates overlapping segments and merges them progressively rather than directly applying temporal latent fusion.
2. Related Work
Prior work established diffusion-based video generation, pose-to-appearance transfer, and latent fusion, but generated durations remained short and temporal fusion could create visible discontinuities. MimicMotion extends a pretrained video model with position-aware progressive latent fusion for smoother long-video transitions.
- Diffusion-based Video Generation: Latent diffusion models reduce the redundancy and computational cost of pixel-domain generation, and Stable Video Diffusion provides a pretrained basis for image-based video generation.MimicMotion extends SVD for pose-guided video generation.
- Pose-to-Appearance Mapping: Pose-to-appearance methods transfer motion between identities using paired keypoints and transformations designed to reduce distortion.Prior approaches use local affine or Thin-Plate Spline transformations to align source appearance with driving poses.
- Long Video Generation: Diffusion-based video algorithms commonly produce only a few seconds, motivating methods that extend duration through successive-frame prediction or latent fusion.Short duration limits practical applications of generated videos.
- Long Video Generation: MultiDiffusion blends separately generated image regions, while temporal adaptations must address more noticeable discontinuities between video segments.Temporal discontinuities can appear as flickering or abrupt content changes.
- MimicMotion: Position-aware progressive latent fusion adaptively assigns weights according to temporal position to smooth transitions at segment boundaries.This strategy builds on latent-fusion principles while targeting temporal continuity.
3. Method
MimicMotion extends latent video diffusion for pose-guided generation from a reference image, using confidence-aware pose conditioning and progressive latent fusion for long videos.
- Model and data: The model integrates a spatiotemporal U-Net with PoseNet, which extracts pose features and adds them to the U-Net representation.The reference image supplies CLIP features and VAE latents, while video generation operates in latent space.
- Model and data: MimicMotion trains a pose-guided video diffusion model from a reference image, video frames, and corresponding pose sequences.Pose sequences are extracted frame by frame with DWPose, while the model leverages a pre-trained video generation prior.
- Confidence-aware pose guidance: Confidence-aware pose guidance reduces the influence of inaccurate keypoints during training and corrects erroneous pose guidance during inference.Keypoint confidence reflects detection reliability under visibility, self-occlusion, and motion blur.
- Confidence-aware pose guidance: Hand region enhancement strengthens the loss weight for human-hand regions with high pose confidence to reduce hand distortion.Reliable regions are identified by thresholding keypoint confidence scores.
- Long video generation: Progressive latent fusion merges features from overlapping video segments during every denoising step to improve smoothness at segment boundaries.Fusion weights depend on each frame’s relative position within its segment, with heavier weights near the segment center or boundary context described by the method.
- Long video generation: Long sequences are split into fixed-length segments with overlapping frames, enabling arbitrary-length pose-guided video generation within acceptable computational cost.The fusion procedure is training-free and integrated into latent-space inference.
4. Experiments
Experiments evaluate MimicMotion against pose-guided video-generation baselines using qualitative comparisons, quantitative metrics, user preferences, and ablations. Results indicate stronger frame quality, pose adherence, temporal smoothness, and component-level performance, including improved long-video transitions.
- Experimental Setup: Experiments compare MimicMotion with MagicAnymate, MagicPose, Moore, and MuseV on TikTok sequences using center-cropped inputs for fairness.Evaluation includes qualitative and quantitative comparisons, a user study, and ablation studies.
- Qualitative Comparison: MimicMotion produces better hand quality and pose adherence than state-of-the-art methods in qualitative TikTok comparisons.The comparison is conducted on a dataset where MimicMotion was not trained.
- Qualitative Comparison: MimicMotion maintains stable frame differences, while MagicPose has abrupt transitions and Moore and MuseV show flickering textures or text.The result indicates smoother motion with fewer visible temporal artifacts.
- Quantitative Evaluation: MimicMotion is best in all reported Table 1 metrics on the TikTok test split, including FID-VID, FVD, PSNR, and SSIM.The evaluation combines video-level and frame-level metrics.
- User Study: Users strongly prefer MimicMotion over baseline videos, with preference reaching 75.5% against MuseV despite MuseV's higher image quality.The study used 36 participants, each evaluating six video pairs per baseline comparison.
- Ablation Study: Ablations show that confidence-aware pose guiding and hand region enhancement improve all metrics, while progressive latent fusion notably improves FVD and boundary coherence.Progressive latent fusion improves temporal coherence even when other metrics do not reflect the improvement.
5. Conclusion
MimicMotion combines confidence-aware pose guidance with progressive latent fusion to generate high-quality, long human-motion videos. Experiments and ablations show improved adaptation to noisy poses, hand quality, and temporal smoothness.
- MimicMotion uses confidence-aware pose guidance and progressive latent fusion for high-quality, long videos guided by human motion.
- The model shows superior adaptation to noisy pose estimation, enhancing hand quality and ensuring temporal smoothness.The reported improvements are attributed to confidence scores, hand-region loss enhancement, and progressive latent fusion.
Impact Statement
MimicMotion may support creative content production, digital humans, and entertainment while enabling controllable human-motion videos. Its deployment also raises misuse risks involving deceptive content and misinformation.
- MimicMotion has potential applications in creative content production, digital human, and entertainment.
- The framework enables high-quality, controllable human-motion video creation but may be misused to create deceptive content for misinformation.Suggested safeguards include watermarking, detection methods, and clear usage guidelines.
A. Training Details
MimicMotion is tuned from Stable Video Diffusion using a dancing-video dataset and a specified diffusion noise distribution. Training uses fixed resolution, epoch, batch-size, and hardware settings.
- MimicMotion is tuned from Stable Video Diffusion, with samples resized to 576 × 1024.
- The training dataset contains 4,436 human dancing videos collected from the internet.
- The model is trained for 20 epochs at batch size 8 using log σ ∼N(Pmean, P 2 std), with Pmean = 0.5 and Pstd = 1.4.
B. PoseNet Model Architecture
The PoseNet consists of convolutional and activation layers. Its detailed architecture is presented in Table 3.
- PoseNet consists of convolutional and activation layers.
- Table 3 provides the detailed architecture of PoseNet.
C. Additional Details on User Study
The user study compared MimicMotion with baseline methods through an online pairwise video-preference interface focused on perceived image quality and temporal smoothness.
- Participants selected which of two automatically playing videos they perceived as higher quality.MimicMotion appeared randomly on either side of the interface.
D. Cross-Domain Results: Cartoon and Animal Videos
MimicMotion demonstrates cross-domain pose-guided generation for cartoon and animal dancing videos, preserving cartoon style and producing reasonable animal motions despite appearance differences.
- The evaluation includes four cartoon and four animal dancing videos as cross-domain scenarios.
- Cartoon videos: MimicMotion maintains cartoon characters’ artistic style while generating natural motions that follow input poses.
- Animal videos: For animal subjects, MimicMotion produces reasonable or plausible dancing motions despite their significantly different appearance from humans.
- Generalization factors: The authors attribute generalization to shared pose-feature subspaces and limb-length mapping from pose templates to reference characters.These choices make pose guidance closer to the reference character.