Source-linked AI summary
Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, Yuan Liu
TL;DR
Existing controlled video methods often lack versatile, fine-grained control across different tasks. DaS uses 3D tracking videos as conditioning for a unified image-to-video diffusion model, achieving strong control across mesh-to-video generation, camera control, motion transfer, and object manipulation with improved temporal consistency. Its main limitation is dependence on correct, compatible 3D tracking videos.
Problem
Existing controlled video methods typically specialize in limited control types and struggle with precise, fine-grained modifications.
Method
DaS uses 3D tracking videos encoding 3D-point motion as conditioning for a unified image-to-video diffusion architecture.
Results
DaS demonstrates strong control across mesh-to-video generation, camera control, motion transfer, and object manipulation, with improved camera-control and motion-transfer performance over baselines.
Takeaways & Limitations
Persistent point-color correspondences in 3D tracking videos improve temporal coherence, including when regions disappear and later reappear.
Takeaways & Limitations
DaS can fail when the input image is incompatible with the 3D tracking video or when regions lack 3D tracking points.
Abstract
from arXiv · showhide
Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.
1 INTRODUCTION
DaS addresses the limited versatility and precision of existing video controls by using 3D tracking videos as control signals in a unified diffusion architecture. This representation supports diverse tasks while linking frames for improved temporal consistency.
- Existing controlled video methods typically specialize in one control type and struggle with fine-grained edits such as raising an avatar’s left hand.
- DaS uses 3D control signals because videos are 2D renderings of dynamic 3D content, enabling diverse and precise control within one architecture.
- DaS takes a 3D tracking video containing color-coded 3D-point motion trajectories as input to guide image-to-video diffusion.
- 3D tracking videos link identical points across frames through persistent colors, improving temporal coherence and preserving appearances when regions disappear and reappear.
- DaS supports mesh-to-video generation, motion transfer, camera control, and object manipulation by manipulating 3D tracking videos.
- 3 days of fine-tuning on 8 H800 GPUs with less than 10k videos yields strong control capabilities across the four tasks.
2 RELATED WORK
Prior video-generation research established strong generative quality, but controllable methods generally use specialized signals for individual control types. DaS differs by using 3D tracking videos for more versatile control across multiple tasks.
- 2.1 Video diffusion: Diffusion video models have achieved impressive generation quality and temporal consistency, motivating applications across creative and industrial domains.
- 2.2 Controlled video generation: Existing controllable video methods commonly introduce a specific control signal that supports only one type, such as identity preservation, camera control, or motion transfer.
- 2.2 Controlled video generation: Mesh-to-video methods use diffusion models to texture or render 3D objects from multiple 2D views or related representations.
- 2.2 Controlled video generation: Camera-control methods condition generation with camera or motion representations to enable customized viewpoints and camera trajectories.
- 2.2 Controlled video generation: Motion-transfer methods guide novel videos using depth, diffusion features, motion controllers, attention flow, or 2D motion prompts.
- 2.2 Controlled video generation: Object-manipulation methods mainly use 2D trajectories or bounding-box relationships, while DaS uses richer 3D tracking videos to represent object motion relationships.
- 2.2 Controlled video generation: Concurrent work includes both 2D motion-based and 3D motion-based control approaches.
3 METHOD
DaS adds 3D tracking-video conditioning to an image-to-video latent diffusion model. Its tracking representation encodes moving 3D points with persistent coordinate colors, and a trainable condition DiT injects these features into generation.
- 3.1 Overview: DaS is an image-to-video diffusion model conditioned on both an input image and a 3D tracking video.
- 3.1 Overview: The architecture processes the input image and tracking video with transformer-based latent diffusion and a VAE.
- 3.2 Backend video diffusion model: DaS is fine-tuned from CogVideoX, whose DiT denoises latent representations and whose VAE decoder produces the output video.
- 3.1 Overview: Dynamic 3D points are colorized by their first-frame coordinates and rendered across cameras to form a 3D tracking video.
- 3.3 Conditioning: A trainable copy of the denoising DiT processes the tracking-video latent, while zero linear layers inject its condition features into denoising.
- 3.3 Finetuning: Training combines real-world and synthetic rendered videos, with the original denoising DiT frozen while only the copied condition DiT is fine-tuned.
- 3.3 Finetuning: The training takes 3 days on 8 H800 GPUs.
3.4 Video generation control
DaS constructs task-specific 3D tracking videos from meshes, source videos, camera paths, or manipulated object points. These controls guide video generation while preserving the intended motion, appearance, or camera movement.
- Object manipulation: Object manipulation combines depth estimation and segmentation to extract an object’s 3D points, modify them, and construct a tracking video for generation.
- Animating meshes to videos: Animated meshes are converted into 3D tracking videos, while a depth-to-image FLUX model supplies an initial visually appealing frame.
- Camera control: Prior camera-control methods use camera or ray embeddings that lack true 3D awareness and leave scene structure to be inferred.
- Camera control: DaS estimates depth, converts the result into colored 3D points, projects them along a specified camera trajectory, and uses the resulting tracking video for camera control.
- Motion transfer: For motion transfer, DaS repaints the source video’s first frame, generates a 3D tracking video from the source motion, and conditions target-video generation on it.
4 EXPERIMENTS
DaS is evaluated on five video-control tasks, including camera control, motion transfer, mesh-to-video generation, and object manipulation.
- DaS is evaluated on five tasks spanning camera control, motion transfer, mesh-to-video generation, and object manipulation.
4.1 Camera control
The camera-control evaluation compares DaS with MotionCtrl and CameraCtrl using camera-trajectory accuracy metrics and large fixed movements. DaS achieves stable and accurate control of generated camera poses.
- MotionCtrl and CameraCtrl serve as baselines because both accept camera trajectories and use camera or ray embeddings for control.
- Figure 4 shows DaS results for four large camera movements: left, right, up, and down.
- Camera accuracy is measured by rotation and translation errors between estimated poses from generated videos and input ground-truth poses.Relative poses are reconstructed with SIFT, then represented by normalized quaternions and translation vectors.
- The evaluation includes 100 random RealEstate10K trajectories and larger fixed left, right, up, down, and spiral movements.
- DaS outperforms baseline methods, demonstrating stable and accurate control of generated camera poses.The authors attribute this to 3D tracking videos enabling accurate spatial inference, unlike implicit camera or ray embeddings.
4.2 Motion transfer
The motion-transfer evaluation compares DaS with TokenFlow and CCEdit using text-alignment and temporal-consistency measures. DaS achieves strong motion transfer with temporally coherent generated videos.
- DaS is compared with TokenFlow and CCEdit for motion transfer.TokenFlow uses diffusion-feature consistency across timesteps, while CCEdit uses depth-based control.
- The evaluation reports Text-Ali for semantic alignment with prompts and Tem-Con for temporal consistency between neighboring frames.
- DaS surpasses both baselines in text alignment and frame consistency.
- Qualitatively, CCEdit produces low-quality, temporally incoherent frames, while TokenFlow maintains semantic consistency but struggles with coherent videos.
- DaS accurately transfers video motion with strong temporal coherence.
4.3 Animating meshes to videos
The mesh-to-video evaluation compares DaS with CHAMP using shared human images and animated SMPL meshes. DaS shows better consistency in avatar structure and texture across motions and styles.
- CHAMP and DaS are compared qualitatively using the same human input image, with an animated SMPL mesh supplying the motion sequence.
- DaS generates videos in different styles from the same animated 3D meshes.
- Compared with CHAMP, DaS demonstrates better consistency in avatar 3D structure and texture details across motion sequences and styles.
4.4 Object manipulation
DaS supports object manipulation through translation and rotation, producing photorealistic videos with accurate object motion and strong multiview consistency.
- Object manipulation: DaS evaluates object manipulation through both translation and rotation of segmented, depth-estimated object points.SAM and depth estimation models provide the object points used for manipulation.
- Object manipulation: The generated videos demonstrate accurate object manipulation with photorealistic appearance and strong multiview consistency.
- Related qualitative results: DaS also reports qualitative results for motion transfer, mesh-to-video generation, and comparisons against CHAMP.
4.5 Analysis
The analysis compares 3D tracking videos with depth-map controls and studies tracking-point density, while also reporting inference cost and failure cases.
- Depth maps vs. 3D tracking videos: DaS outperforms a same-architecture depth-map baseline on all reported metrics for video reconstruction.The comparison uses PSNR, SSIM, LPIPS, and FVD between generated and ground-truth videos.
- Depth maps vs. 3D tracking videos: 3D tracking videos improve temporal association because identical colored 3D points link corresponding frames, unlike depth maps.
- Tracking-point ablation: 4900 tracking points perform slightly better than other tested counts, while 2500, 4900, and 8100 produce visually similar quality.The tested range spans 900 (30×30) to 8100 (90×90) points.
- Inference: Inference takes about 2.5 minutes to generate 49 frames at 480×720 resolution on one H800 GPU.The setting uses DDIM with 50 steps and classifier-free guidance magnitude 7.0.
- Failure cases: Incorrect tracking videos can cause scene transitions or leave regions without tracking points uncontrolled.
5 LIMITATIONS AND CONCLUSIONS
DaS uses colored dynamic 3D points and diffusion to generate videos following 3D motion, but remains sensitive to tracking-video compatibility and coverage.
- Conclusions: DaS’s core idea is to use 3D tracking videos as control signals representing the underlying motion of colored dynamic 3D points.
- Conclusions: The diffusion model generates video that follows the motion encoded by the 3D tracking video.
- Limitations: DaS can fail when the input image is incompatible with the tracking video or when regions lack 3D tracking points.These cases may produce implausible scenes, uncontrolled content, or unnatural results.
- Future work: Future work could learn to generate high-quality 3D tracking videos rather than relying on provided animated meshes or existing videos.