Source-linked AI summary
OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data
Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, Pengfei Wan
TL;DR
Cloning nuanced camera motion from reference videos remains difficult to control precisely across single- and multi-shot generation. OmniDirector represents camera motion as grid videos and combines this with hierarchical prompt expansion for multimodal control, outperforming all baselines across camera-control metrics, including a 39.3% relative gain in translation precision over CamCloneMaster.
Problem
Existing textual and parametric camera controls trade off cinematographic precision and user accessibility, motivating direct camera-motion cloning from reference videos.
Method
OmniDirector renders camera parameters in a 3D empty scene as grid-motion videos and integrates camera, content, and action controls through hierarchical prompt expansion.
Results
39.3% relative improvement in translation precision (T-Pre) over CamCloneMaster, with OmniDirector outperforming all baselines across evaluation metrics.
Takeaways & Limitations
OmniDirector provides an efficient and accessible paradigm for general multi-shot camera cloning while preserving accurate trajectories and semantically coherent shot transitions.
Takeaways & Limitations
Direct token concatenation struggles to maintain long-term memory and temporal consistency when scaling to significantly longer videos.
Abstract
from arXiv · showhide
Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/
Introduction
OmniDirector addresses limitations in camera-motion control by representing cameras as grid motion videos that support complex single- and multi-shot cloning. It combines this representation with a unified MMDiT framework and hierarchical prompt expansion for director-level control.
- Motivation: Textual camera specifications lack precision, while explicit parameter-based methods struggle with complex motions and multi-shot transitions.The introduction identifies an inherent trade-off between textual descriptions and explicit camera parameters, while related methods are limited to basic motions and shot transitions.
- Camera Grid: The camera grid renders extracted camera parameters as a grid motion video showing movement in a 3D empty scene.This visual representation is designed to unify diverse camera motions for single- or multi-shot cloning.
- OmniDirector: OmniDirector provides director-level control for Multi-Modal Diffusion Transformers using the camera grid as its foundation.The framework is trained on a newly curated dataset of million-scale camera grid-video pairs.
- Prompt Expansion: A hierarchical Prompt Expansion Agent integrates camera control with other signals by generating camera-motion prompts at inter-shot and intra-shot levels.The inter-shot hierarchy handles relationships across shot transitions to maintain semantic coherence in multi-shot scenarios.
Related Work
Related work spans diffusion-based video generation across T2V, I2V, and V2V tasks, while highlighting the need for additional signals beyond text for precise spatial-temporal control. Video-referenced camera control offers an intuitive precision–usability tradeoff, but paired-data scarcity limits scalable training and motivates synthetic alternatives.
- Diffusion-Based Video Generation: Diffusion-based video generation is categorized into Text-to-Video, Image-to-Video, and Video-to-Video tasks with distinct semantic, temporal, and structural objectives.T2V translates semantics to motion, I2V extrapolates temporally while preserving spatial identity, and V2V manipulates attributes while maintaining structural priors.
- Controllable Video Generation: Text-only generation lacks the precise spatial-temporal control required for real-world applications, motivating the study of additional conditional signals.The related work identifies controllable generation as a response to the limitations of text prompts alone.
- Video-Referenced Camera Control: Video-referenced camera control uses source videos as motion exemplars, improving precision over text prompts while remaining more intuitive than manually specifying complex trajectories.This paradigm bridges explicit parameter control, which is precise but cumbersome, and text-driven generation, which is user-friendly but coarse.
- Video-Referenced Camera Control: Paired-data camera-control methods are constrained by scarce real-world pairs, while recent work uses game-engine-rendered synthetic datasets to address data scarcity.The paired videos share camera movements but depict different contents; synthetic alternatives include datasets rendered with Unreal Engine.
Method
The method section presents OmniDirector through three components: the camera-grid representation, the architecture and training strategy, and the hierarchical prompt expansion agent with inference strategies.
- Method: OmniDirector’s method comprises a camera-grid representation, an architecture and training strategy, and a hierarchical prompt expansion agent with inference strategies.These components are introduced in that order in the method section.
Camera Grid
The camera grid visually represents camera motion by rendering a simplified 3D spatial framework from camera poses, and it extends naturally to multi-shot sequences and special camera effects. This representation enables automatic construction of reference-video/grid training pairs at internet scale.
- Spatial Modeling: The method abstracts scenes into an empty room with floor and ceiling grids, trajectory-surrounding tunnel walls, and Y-axis lines that visually encode camera motion.Grid points are sampled on two X-Z planes, while vertical lines form a tubular boundary around the projected camera trajectory.
- Spatial Modeling: Each frame renders the grid under the reference camera pose by transforming world coordinates with [R_i | t_i] and projecting them onto the image plane.This rendering process produces visual changes corresponding to movements such as dolly, zoom, pan, and tilt.
- Special Effects: The camera grid extends beyond standard motion to represent fisheye distortion and dolly zoom through modified rendering schemes.Kannala–Brandt projection bends straight lines into smooth curves, while dolly zoom keeps the subject visually fixed as the background tunnel grid stretches.
- Multi-Shot Representation: The representation generalizes to multi-shot videos by detecting shot transitions, segmenting the reference into sub-clips, and rendering transition frames as pure white signals.Each sub-clip is treated as one shot before camera parameters are estimated and its grid is rendered.
- Scalable Data Construction: Camera grids can be extracted from arbitrary videos to automatically create reference-video/grid pairs, enabling efficient curation of massive, internet-scale training datasets.The pipeline uses DPA-V3 for camera extrinsics and f = a × max(H, W), with a = 0.8, to improve stability and resolution-agnostic generalization.
OmniDirector
OmniDirector represents camera motion as grid videos and integrates camera grids, reference images, video latents, and text within a unified multimodal diffusion-transformer architecture. A self-reconstruction objective on 30% of training samples strengthens camera-trajectory understanding while retaining standard camera-conditioned video generation.
- Camera-grid representation: The camera grid disentangles camera control from other visual attributes and can be encoded through a standard video-processing pipeline.Its spatiotemporal-signal structure enables interference-free injection into the transformer architecture.
- Multimodal encoding: Camera-grid, reference-image, and noisy-video latents are concatenated along the frame dimension into zvis ∈ R^(2T+1)×H×W×C before 3D-convolutional tokenization.The latents are produced by a pretrained 3D-VAE: zc = ε(G) and zI = ε(I).
- Multimodal fusion: Joint attention lets visual and text tokens exchange information through separate attention pathways within the MMDiT architecture.Text is processed by a pretrained encoder such as T5, while the visual modalities are fused before text semantics modulate generation.
- Training objective: 30% of training samples use self-reconstruction, conditioning on zc = ε(G) and training the model to reconstruct zv = ε(G) instead of a natural video.The remaining samples follow standard camera-conditioned video generation with natural target content consistent with the camera grid.
Unified Multimodal Control Signals for Inference
OmniDirector unifies user prompts, reference-image cues, camera grids, and generated camera descriptions into textual conditioning, while separating camera motion from entangled scene semantics. Its inference design further coordinates global camera structure and local semantic refinement through tailored guidance and denoising schedules.
- Motivation: Unified textual conditioning consolidates user prompts, reference-image cues, and camera grids to improve reliable camera controllability in text-to-video generation.Textual conditioning is central because the generation model is pretrained on text-to-video objectives.
- Challenge: Direct MLLM descriptions can entangle camera motion with subjects, actions, and backgrounds, causing conflicting conditioning that weakens camera control.The method therefore targets camera-specific descriptions rather than unrestricted video descriptions.
- Camera Prompt Generation: Camera prompts are generated from camera parameters by decomposing motion into inter-shot relationships and intra-shot pose descriptions, then correcting and merging them into a compact description.The pipeline analyzes shot transitions, derives pose labels from trajectory segments, rectifies them with reference-video cues, and removes redundant motion labels.
- Semantic Fusion: Qwen3-VL fuses pose descriptions, reference images, and user prompts into a cohesive representation for transferring camera motion to novel scenes.Reference-video keyframes help refine inaccurate pose-based descriptions before semantic fusion.
- Inference Scheduling: Camera-grid features establish global spatial structure during high-noise denoising, while other controls refine local content during low-noise denoising.Classifier-Free Guidance uses a black visual unconditional input tailored to the camera-grid representation.
Experiments · Experimental Setups
OmniDirector is evaluated on a diverse 1,094-sample validation set using camera-control, visual-quality, shot-transition, leakage, and pairwise-comparison protocols. The model is trained on 1.8M resized internet videos with triplet conditioning, augmentation, and specified optimization settings.
- Experimental Setups: OmniDirector is trained on 1.8M internet videos spanning diverse domains, with each video resized to 480p.The training data is collected from internet videos such as movies and advertisements.
- Experimental Setups: The model is conditioned on {G, I, T′} and trained for 10k optimization steps with a 5 × 10−5 learning rate and batch size 64.Training also applies random camera-grid colors and pose jitter for robustness.
- Evaluation Metrics: Camera control is evaluated by extracting reference and generated pose trajectories with DPA-V3 and computing Relative Rotation Error and Relative Translation Error.RRE measures angular pose differences, while RTE measures directional translation error.
- Evaluation Metrics: Robustness is assessed with R-Pre and T-Pre, which report the percentages of predictions meeting relative rotation and translation error criteria.These metrics complement the scale-invariant relative pose errors.
- Evaluation Metrics: Gemini 3.1 Pro and GSB quantify results amid pose-estimation errors, while Gemini also measures reference-video leakage at frame and shot levels.GSB pairwise comparisons with CamCloneMaster cover camera, quality, and narrative.
- Evaluation Metrics: Shot-transition accuracy is measured with Tem-Pre for temporal alignment and Sem-Pre for semantic consistency.Tem-Pre counts transitions successful when temporal error is less than 3 frames and uses TransNet-V2 for shot-boundary detection.
- Evaluation Set: The validation set contains 1,094 curated web samples spanning advertising, cinematic content, complex visual effects, domains, resolutions, shot counts, and camera trajectories.It includes in-domain and cross-domain reference video-image pairs, single-shot and multi-shot sequences, and simple and complex trajectories.
Comparisons with State-of-the-Art Methods
OmniDirector outperforms CamCloneMaster, Seedance2.0, and LTX-LoRA in camera-motion cloning, particularly for complex multi-shot scenarios. It combines accurate, coherent shot transitions with lower leakage, stronger camera-control accuracy, and more stable generation.
- Qualitative Comparison: OmniDirector faithfully clones camera trajectories across scene-scale, resolution, and content variations while preserving semantic coherence during multi-shot transitions.The method achieves accurate shot transitions and remains robust under substantial variation in scene conditions.
- Qualitative Comparison: Seedance2.0 and CamCloneMaster produce perceptually inconsistent motion amplitudes in multi-shot videos, while LTX-LoRA suffers substantial content leakage.LTX-LoRA can execute shot transitions, but its outputs exhibit severe leakage from the reference content.
- Quantitative Comparison: OmniDirector outperforms all baselines on every camera-control metric and improves translation precision by 39.3% relative to CamCloneMaster.The improvement is attributed to superior relative-scale generalization compared with methods that struggle with scale inconsistencies between reference videos and source views.
- Leakage Analysis: OmniDirector achieves the lowest leakage rate because its camera grid representation and Prompt Expansion agent decouple camera signals from reference-video content.Baseline methods rely more heavily on the reference video's visual content, with LTX-LoRA showing the most severe leakage.
- Human Evaluation: OmniDirector shows clear advantages over CamCloneMaster across all three GSB dimensions, indicating superior overall effectiveness and generation stability.The comparison is reported in Table 3 using Good/Same/Bad judgments.
Ablation study
The ablation study shows that multimodal prompt fusion, inter-shot relationship modeling, and adaptive CFG are all necessary for accurate camera-motion cloning and coherent shot transitions.
- Prompt expansion agent: Fusing multiple signals via the MLLM improves all metrics and integrates camera motion with the reference image.Visual results show highly plausible videos from this multimodal signal fusion.
- Prompt expansion agent: Removing inter-shot relationship modeling degrades all metrics, especially semantic precision for shot transitions.Without post-transition guidance, videos shift to random scenes unrelated to the reference camera motion.
- Adaptive CFG: Replacing adaptive CFG with full-stage camera-signal injection significantly reduces camera accuracy.Without AdaCFG, the camera rotates more slowly because simultaneous signal processing weakens the response to camera guidance.
Emergent Camera Understanding
The camera grid acts as a visual proxy for spatial movement, sharing highly similar spatiotemporal semantics with RGB videos. This representation unlocks an emergent capability for comprehending and executing camera dynamics under frozen inference parameters.
- Representation: The camera grid functions as a visual proxy for spatial movement with spatiotemporal semantics highly similar to RGB videos.Its visual structure parallels the semantics of ordinary video representations.
- Emergent capability: The representation empirically unlocks an emergent capability in the video generation model for comprehending camera dynamics.This capability emerges from conditioning the model on the camera-grid representation.
- Inference: With model parameters frozen during inference, the model robustly deduces and executes camera motions from diverse conditioning signals.The passage specifically identifies raw reference signals among the conditioning inputs, although the supplied text is truncated.
Conclusion
OmniDirector achieves general multi-shot camera cloning by representing camera motion as grid videos and training on million-scale camera grid-video pairs. A Hierarchical Prompt Expansion Agent integrates camera motion with other multimodal control signals during inference.
- OmniDirector achieves general multi-shot camera cloning.
- Camera parameters rendered within a 3D empty room are represented as grid videos.
- Million-scale camera grid-video training pairs empower OmniDirector with camera control.
- A Hierarchical Prompt Expansion Agent harmoniously integrates camera motion with other multimodal control signals during inference.