Source-linked AI summary
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Ying Shan, Xiu Li, Qifeng Chen
TL;DR
Pose-controllable, text-editable character video generation is limited by missing paired video-pose data and video generative priors. The paper uses a two-stage scheme built on image-pose pairs, pose-free videos, and pretrained text-to-image models, producing continuously pose-controllable videos while retaining text editing and concept composition.
Problem
Text-to-video generation has limited applications because high-quality video datasets and video generative prior models are restricted, while pose-driven methods do not effectively support text editing.
Method
A two-stage training scheme learns pose control from image-pose pairs and temporal consistency from pose-free videos while tuning separate modules in a pretrained T2I model.
Results
The method generates continuous pose-controllable character videos with text-editable appearance while retaining the pretrained T2I model’s concept generation and composition capabilities.
Takeaways & Limitations
The approach supports flexible control through pose sequences and textual descriptions and can generate diverse characters, backgrounds, styles, and multiple characters.
Abstract
from arXiv · showhide
Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e.,image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint-image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models will be made publicly available.
1 Introduction
The paper addresses pose-controllable, text-editable character video generation despite limited high-quality video datasets and video generative priors. It separates pose alignment from temporal coherence, using image-pose pairs, pose-free videos, and a pretrained T2I model.
- Limited high-quality video datasets and video generative priors restrict text-to-video applications.
- The method uses a two-stage scheme that learns pose control from image-pose pairs and temporal consistency from pose-free videos.The first stage adds a pose encoder, while the second trains temporal components to reduce flickering and improve coherence.
- Pose control is added to Stable Diffusion through a multi-layer pose encoder while most pretrained denoising parameters remain fixed.The encoder extracts pose features at multiple resolutions and injects them through residual connections.
- The approach generates pose-controllable character videos with text-editable appearance while retaining diverse character, background, style, and composition capabilities.
- Extensive experiments compare the method with baselines across generation quality, text-video alignment, pose-video alignment, and temporal coherence.
2 Related Work
Prior work spans text-to-video synthesis, pose-driven video generation, and conditional image synthesis. Existing pose-driven methods can generate or drive images resembling the data distribution but do not effectively support text-based video editing.
- 2.1 Text-to-Video Generation: Text-to-video research includes autoregressive token-based models, transformer-based systems, and diffusion models extended from image synthesis.Examples include GODIVA, NÜWA, CogVideo, Video Diffusion Models, Imagen Video, and Make-A-Video.
- 2.1 Text-to-Video Generation: Natural video synthesis remains challenging because videos have complex and high-dimensional structural characteristics.
- 2.2 Pose-to-Video Generation: Pose-to-video methods use driving signals such as keypoints or motion representations to synthesize or animate human images.Vid2vid, FOMM, and FRAA represent different conditional video synthesis and articulated-character motion approaches.
- 2.2 Pose-to-Video Generation: Existing pose-driven methods fall short in effectively editing videos through text.
- 2.3 Conditional Image Synthesis: Conditional image synthesis methods train encoders or adaptors to map controls such as depth maps, keypoints, and segmentation maps into conditioning features.
3 Method
The method extends a pretrained latent-diffusion text-to-image model for pose-guided text-to-video generation by separating pose control from temporal consistency. It trains these capabilities with image-pose pairs and pose-free videos, then uses temporal attention and cross-frame attention for coherent video generation.
- Task and base model: The task generates realistic videos from pose sequences and text-described appearance using a pretrained latent-diffusion model.The network is modified for text-guided pose-controllable video generation.
- Task and base model: Latent diffusion models encode images into lower-dimensional latents and learn to generate samples through a gradual backward denoising process.An autoencoder provides the latent representation, while the diffusion model learns its distribution.
- Two-stage training: Qualified video-pose pairs are scarce, so the method decouples pose control learned from images from temporal consistency learned from videos.This decoupling motivates training the model in two different stages.
- Two-stage training: In the first stage, pose features are extracted at multiple resolutions and injected into the pretrained U-Net through residual connections.The pose encoder processes the input pose sequence at 64 × 64, 32 × 32, 16 × 16, and 8 × 8 resolutions.
- Two-stage training: In the second stage, the first-stage model is fine-tuned on continuous pose-free video-text pairs to improve temporal consistency and background stability.The method tunes temporal self-attention and cross-frame self-attention while using the text-to-image model's generative prior.
- Capabilities: The resulting system supports continuous pose control from image-pose pairs and random videos while retaining pretrained concept generation and composition capabilities.The discussion also states that the design can extend to related conditional video-generation tasks.
4 Experiments
Experiments evaluate implementation, applications, baseline comparisons, and ablations of the pose-controllable video generation method. The results cover multi-person control, textual character and background specification, temporal consistency, and pose-conditioning design choices.
- 4.1 Implementation Details: The model is trained for 100k steps on LAION-Pose and 50k steps on HDVILA using eight 512×512 consecutive frames.
- 4.2 Applications: Multiple pose sequences can be handled while textual descriptions specify character identities and background contexts.
- 4.3 Comparison with Baselines: Compared with ControlNet and T2I adapter, the approach preserves consistent character appearance across generated video frames.
- 4.3 Comparison with Baselines: The approach achieves higher CLIP score and human-preferred video quality, comparable pose accuracy to ControlNet, and stronger frame consistency than ControlNet.Frame consistency is measured using CLIP cosine similarity between consecutive frames.
- 4.4 Ablation Study: Feature-residual pose injection preserves more pretrained image-generation ability than direct concatenation.Concatenation requires retraining the first convolutional layer because it changes the number of input channels.
- 4.4 Ablation Study: Injecting pose controls into more layers improves pose-frame alignment compared with controlling a single layer.Single-layer control produces an arm mismatch in the Iron Man example.
5 Conclusion
The paper addresses text-editable, pose-controllable character video generation by adapting a pretrained text-to-image model with two-stage training. It combines image-pose pairs for pose control with pose-free video for temporal coherence while preserving conceptual combination ability.
- The method targets text-editable and pose-controllable character videos.
- A two-stage scheme learns pose control from large-scale image-pose pairs and temporal coherence from diverse pose-free videos.
- The resulting videos preserve the pretrained text-to-image model’s conceptual combination ability while remaining temporally coherent.