Source-linked AI summary
Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, Jing Liao
TL;DR
Text-to-video models lack independent control over camera and object motion, limiting customized video creation. Direct-a-Video combines self-supervised temporal cross-attention for quantitative camera control with training-free spatial modulation for object trajectories, and supports separate or joint control. Its supported scope includes camera pan and zoom rather than complex 3D camera movement.
Problem
Text-to-video methods often lack user-defined, disentangled control over camera movement and object motion, limiting flexibility in video motion control.
Method
Direct-a-Video combines self-supervised temporal cross-attention for quantitative camera pan and zoom control with training-free spatial cross-attention modulation for object trajectories.
Results
Direct-a-Video supports separate and joint control of camera movement and object motion for customized video synthesis.
Takeaways & Limitations
The framework provides an efficient and flexible tool for creative video synthesis with customized motion.
Takeaways & Limitations
Joint control can produce conflicts when user inputs are inconsistent, and camera augmentation is limited to 2D panning and zooming rather than complex 3D movements.
Abstract
from arXiv · showhide
Recent text-to-video diffusion models have achieved impressive progress. In practice, users often desire the ability to control object motion and camera movement independently for customized video creation. However, current methods lack the focus on separately controlling object motion and camera movement in a decoupled manner, which limits the controllability and flexibility of text-to-video models. In this paper, we introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera's pan and zoom movements, as if directing a video. We propose a simple yet effective strategy for the decoupled control of object motion and camera movement. Object motion is controlled through spatial cross-attention modulation using the model's inherent priors, requiring no additional optimization. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. We further employ an augmentation-based approach to train these layers in a self-supervised manner on a small-scale dataset, eliminating the need for explicit motion annotation. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios. Extensive experiments demonstrate the superiority and effectiveness of our method. Project page and code are available at https://direct-a-video.github.io/.
1 INTRODUCTION
Direct-a-Video addresses the lack of disentangled user control over camera and object motion in text-to-video generation. It combines independent camera conditioning with training-free object-motion control to support separate or joint direction.
- Motivation: Existing methods often conflate camera and object motion, so specifying only an object trajectory can correspond to multiple underlying scenes.A right-moving object may reflect object motion, camera panning, or both.
- Motivation: Supervised disentangled control is difficult because motion annotations are costly and object motion is often correlated with camera movement.Large-scale annotation requires laborious object tracking and camera-pose estimation, while video-model training is computationally expensive.
- Contributions: Direct-a-Video decouples camera movement and object motion, allowing users to control either aspect independently or both jointly.The framework is designed for user-directed video creation with customized motion patterns.
- Camera movement control: The camera module uses temporal cross-attention and self-supervised camera-motion augmentation to interpret quantitative pan and zoom parameters without explicit motion annotations.The additional layers are trained while preserving the original T2V model weights and can generalize from a small-scale video dataset to open-domain scenarios.
- Object motion control: Object motion is controlled at inference through training-free spatial cross-attention modulation, with users drawing bounding-box trajectories for one or more objects.The interaction specifies boxes at the first and last frames together with intermediate paths, avoiding video grounding datasets and annotations.
2 RELATED WORK
Related work spans image-to-video, video-to-video, and text-to-video motion control. Direct-a-Video is positioned in text-to-video generation, where prior methods lack some combination of quantitative camera control, object control, multi-object support, or convenient disentangled interaction.
- Motion-control categories: Motion-control research is commonly organized into image-to-video, video-to-video, and text-to-video generation.The paper focuses on text-to-video motion control, particularly when no source video is available.
- Image-to-video: Image-to-video methods often use keypoint dragging, but sparse local points restrict large-granularity motion control.The interaction is intuitive and user-friendly, yet its local and sparse nature limits broader motion control.
- Video-to-video: Video-to-video motion-transfer methods rely on motion priors from source videos, which may not always be available.Approaches include fine-tuning on reference videos or borrowing spatial and sparse features from source footage.
- Text-to-video: In text-to-video control, AnimateDiff lacks quantitative camera and object-motion control, while VideoComposer uses inefficient dense pixel-wise guidance without explicitly separating the two motions.These limitations motivate a more direct and disentangled user interaction model.
- Text-to-video: Peekaboo supports bounding-box object trajectories but originally lacks multi-object and camera-motion control.The paper contrasts these missing capabilities with its broader control scope.
3 METHOD
Direct-a-Video decouples camera and object motion control in text-to-video generation. It learns camera movement with lightweight self-supervision while steering object placement during inference through spatial cross-attention modulation.
- Overview: Users specify camera pan and zoom parameters alongside object words and bounding-box trajectories for independently controlled video motion.The formulation supports horizontal and vertical panning, zooming, and per-object start, end, and intermediate boxes.
- Overview: Independent camera and object controls can be applied separately or jointly, because the two mechanisms operate in distinct training and inference stages.Camera movement is learned during training, whereas object motion is implemented during inference.
- Camera Movement Control: Camera motion is simulated by augmenting stationary-camera videos with parameterized shifting and scaling, avoiding explicit camera-motion annotation.The augmented samples condition newly introduced temporal cross-attention layers through encoded camera parameters.
- Camera Movement Control: The camera embedder separately encodes panning and zooming, while camera modules inject these signals through temporal cross-attention layers in each U-Net block.The panning parameters are jointly encoded, whereas zooming receives a separate encoding to distinguish the two movement types.
- Camera Movement Control: Camera control preserves pretrained model knowledge by freezing original weights and training only the camera embedder and camera module with diffusion noise-prediction loss.The training conditions on camera movement and video captions, with augmented samples serving as the clean inputs.
- Object Motion Control: Object motion is controlled without additional training by modulating spatial cross-attention using object bounding boxes and corresponding prompt tokens.Attention amplification increases object-word responses inside target boxes, while frame-wise modulation determines complete spatial-temporal trajectories.
4 EXPERIMENT
Experiments evaluate Direct-a-Video’s camera, object, joint-control, and attention-modulation capabilities against diffusion-based baselines. Results show precise, disentangled camera control, stronger multi-object motion handling, and benefits from both amplification and suppression.
- Experimental Setup: The evaluation compares Direct-a-Video with AnimateDiff, Peekaboo, and VideoComposer across camera and object motion control tasks.Camera-control comparisons use AnimateDiff and VideoComposer; object-control comparisons use VideoComposer and Peekaboo.
- Camera Movement Control: Direct-a-Video combines or switches camera pan and zoom movements without reloading modules and independently controls foreground-object motion.Unlike VideoComposer’s global motion vector map, the method uses three camera parameters and does not impose motion constraints on foreground objects.
- Camera Movement Control: Direct-a-Video achieves the best FVD and FID-vid scores and a lower flow error than the camera-control baselines.AnimateDiff is excluded from flow-error comparison because it lacks quantitative control.
- Object Motion Control: Direct-a-Video handles multiple moving objects with individually specified motion, while VideoComposer exhibits motion-control, semantic-mixing, and object-absence problems.The method uses per-object attention modulation to address these multi-object cases.
- Object Motion Control: For object control, Direct-a-Video slightly surpasses VideoComposer and significantly exceeds Peekaboo on grounding metrics, while its quality is comparable to Peekaboo.The paper attributes improved grounding to attention amplification in addition to attention masking.
- Joint Control of Camera Movement and Object Motion: Joint control produces different foreground-background motion patterns from the same box sequence, including stationary, walking-right, walking-left, and camera-induced apparent motion.This demonstrates separate specification of camera movement and object motion.
- Ablation Study: Removing attention amplification decreases grounding ability, while attention suppression mitigates semantic mixing between objects.Ablations show less box-following without amplification and resolve texture leakage when suppression is enabled.
- Ablation Study: Jointly encoding pan and zoom raises flow error from 0.46 to 1.68 compared with separate camera encodings.The ablation supports separately encoding panning (c_x, c_y) and zooming (c_z).
5 LIMITATIONS
Direct-a-Video has limitations in joint control, camera-motion scope, and object control with colliding boxes.
- Conflicting object and camera instructions can produce unrealistic motion, such as a house moving while constrained to a static box.The authors therefore require careful and reasonable user interaction for joint control.
- Camera augmentation currently supports only 2D panning and zooming, excluding complex 3D movements such as panning around an object.The authors leave more sophisticated augmentation or synthetic rendered data for future work.
- Colliding bounding boxes create an additional issue for object control.
6 CONCLUSION
Direct-a-Video addresses independent, user-directed control of camera and object motion by decoupling their control mechanisms. Experiments demonstrate separate and joint control, positioning the framework as an efficient and flexible tool for customized video synthesis.
- Direct-a-Video combines self-supervised temporal cross-attention training for camera control with training-free spatial cross-attention modulation for object control.
- The framework demonstrates separate and joint control of camera movement and object motion.
- Direct-a-Video is positioned as an efficient and flexible tool for creative video synthesis with customized motion.
A ADDITIONAL IMPLEMENTATION DETAILS
The method avoids computationally intensive camera-parameter extraction by simulating camera movement through algorithmic cropping-window changes in stationary-camera footage.
- Camera augmentation simulates camera movement by algorithmically altering the cropping window across footage captured by a stationary camera.
A.2 Training Details for Camera Control
Camera movement parameters are sampled independently during training, while a coarse-to-fine scheme trains newly added camera layers at progressively larger video resolutions and frame counts.
- Camera parameters c_cam = [c_x,c_y,c_z] are sampled during training, with each component sampled independently.
- Only the camera embedder and camera module are trainable, preserving the pretrained Zeroscope T2V model state.
- Training uses a coarse-to-fine strategy, beginning with 256×256×8 videos for 100k iterations before larger video configurations for 50k iterations.
A.3 Inference Details for Camera Control.
Camera control uses classifier-free guidance with camera parameters in the conditional branch and a static camera in the unconditional branch. Camera conditioning is applied only during early sampling steps, with a cutoff at 0.85T.
- The predicted noise is guided from the unconditional branch toward the camera-conditional branch using a guidance scale s.
- The conditional branch receives camera parameters [c_x,c_y,c_z], while the unconditional branch uses the static camera status [0,0,1].
- Camera conditioning is needed only during initial sampling because general temporal transitions are determined early.
- The camera module is bypassed when t falls below the cutoff threshold, empirically set to 0.85T.
B ADDITIONAL ABLATION STUDIES
Additional ablations examine attention-amplification placement, hyperparameters, and the effect of camera and object controls on video quality. The results identify practical amplification ranges and indicate that adding control has only a minor quality influence.
- Attention amplification placement: Attention amplification in either the U-Net encoder or decoder significantly enhances object responsiveness.
- Attention amplification hyperparameters: Object responses are more sensitive to cutoff timestep τ than strength λ, with over-responsiveness typically occurring for τ<0.9.
- Control and quality evaluation: Qualitative comparisons evaluate the base model, camera-only control, object-only control, and combined camera-plus-object control.
- Attention amplification hyperparameters: τ∈[0.9T, 0.95T] and λ∈[10, 25] are empirically appropriate for most cases.
- Control and quality evaluation: Adding camera or object control may have a minor, statistically insignificant influence on FVD and FID-vid video-quality metrics.
C ADDITIONAL RESULTS
Additional results provide further examples of camera movement control and object motion control. These results supplement the demonstrated control capabilities with more qualitative cases.
- Additional qualitative results are presented for camera movement control and object motion control.
- The additional results are shown in Figure 11, with dynamic videos available on the project page.
- Figure 11 extends the reported qualitative evidence beyond the main examples.