Source-linked AI summary
CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation
Qinghe Wang, Yawen Luo, Xiaoyu Shi, Xu Jia, Huchuan Lu, Tianfan Xue, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai
TL;DR
Controllable text-to-video research seeks precise 3D placement and coordinated object-camera manipulation, but existing controls and annotated datasets remain limited. CineMaster addresses this with an interactive 3D workflow, a conditioned diffusion model, and automated annotation pipeline, and reports stronger control than existing methods.
Problem
Existing controllable video methods rely on limited control signals, while large-scale videos with 3D bounding-box and camera-trajectory annotations are scarce.
Method
CineMaster combines an interactive 3D workflow with a diffusion model conditioned on depth layouts, class labels, and camera trajectories, supported by automated data labeling.
Results
CineMaster performs best in reported qualitative comparisons and provides stronger control over object and camera motion than existing methods.
Takeaways & Limitations
CineMaster provides a 3D-aware framework for specifying object placement, object motion, camera motion, and rendered-frame layout.
Takeaways & Limitations
Accurate open-set object-pose estimation is unavailable, so precise object-orientation control from rotating 3D boxes remains future work.
Abstract
from arXiv · showhide
In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals--comprising rendered depth maps, camera trajectories and object class labels--serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D object motion and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation. Project page: https://cinemaster-dev.github.io/.
1. Introduction
CineMaster addresses the gap between controllable text-to-video generation and film-director-level 3D control. It combines an interactive 3D workflow with rendered control signals and an automated annotation pipeline for training and evaluation.
- Existing controllable video methods use condition maps but generally require pre-existing videos, making precise controls difficult to create from scratch.
- CineMaster lets users position labeled 3D bounding boxes and cameras across keyframes, preview rendered frames, and iteratively refine video conditions.
- The second stage conditions a text-to-video diffusion model on camera trajectories, class labels, and rendered depth maps encoding each frame’s desired 3D layout.
- An automated data-labeling pipeline addresses the scarcity of videos with 3D bounding-box and camera-trajectory annotations by constructing a large annotated video dataset.
- The paper evaluates CineMaster against existing methods and uses ablations to assess its core modules.
2. Related Work
Prior work controls video generation through planar maps, 2D motion signals, camera poses, or tracked points, but these approaches leave important limitations in 3D planning and controllability. CineMaster builds on this literature with joint 3D-aware object and camera control.
- Controllable Video Generation via Planar Condition Maps: Video-control methods commonly condition generation on depth, pose, semantic, optical-flow, or other planar maps that are difficult to create precisely from scratch.
- Controllable Video Generation via Planar Condition Maps: Several approaches control object motion with 2D boxes or trajectories, while 3DTrajMaster uses 6D pose sequences for 3D object-motion control.
- Camera Motion Control: Camera-control methods use camera poses or Plücker embeddings, but training on RealEstate10K limits their ability to generalize to in-the-wild scenes.
- Joint Motion Control: Joint-control methods extend object and camera motion to 3D through point tracking, but image-to-video designs cannot plan a 3D shooting from scratch.
- Joint Motion Control: SynFMC renders joint object and camera annotations with Unreal Engine, while limited diversity and domain gap restrict its generalizability.
3. Method
CineMaster constructs 3D-aware control signals interactively, then conditions a text-to-video model on depth, semantic labels, and camera trajectories. An automated pipeline supplies the annotated data needed to train this system.
- Stage 1: 3D-Aware Control Signals: Users construct 3D-aware controls by repositioning object bounding boxes and cameras across keyframes, with previews guiding iterative scene design.The workflow exports camera trajectories and per-frame projected depth maps for generation.
- Stage 2: Conditional Video Generation: Projected depth maps encode frame-level 3D layouts, while camera trajectories and object labels provide additional conditioning for the text-to-video model.The model uses these signals to guide generation toward the intended video content.
- Stage 2: Conditional Video Generation: The Semantic Layout ControlNet fuses projected-depth latents with entity-label embeddings and masks to guide the position of multiple entities.A copied subset of DiT blocks provides the ControlNet architecture.
- Stage 2: Conditional Video Generation: The Camera Adapter injects sequences of 3 × 3 rotations and 3 × 1 translations into DiT blocks, supporting joint control of object and camera motion.Camera features enter through token addition and a residual connection.
- 3.3. Dataset Labeling Pipeline: The labeling pipeline extracts class labels, camera trajectories, and projected depth maps from in-the-wild videos using segmentation, depth estimation, 3D reconstruction, and tracking.It computes per-frame 3D boxes from tracked point displacements and renders them into depth maps.
- Qualitative Comparisons: CineMaster compares moving-object/static-camera, static-object/moving-camera, and jointly moving settings against aligned controls for MotionCtrl and Direct-A-Video.These comparisons evaluate separate and joint control of object and camera motion.
4. Experiments
Experiments evaluate CineMaster’s training strategy and controllability against existing methods through qualitative comparisons, quantitative metrics, and ablations. CineMaster achieves the strongest reported performance, while removing or separating key training components degrades specific control capabilities.
- Quantitative Comparison: CineMaster outperforms previous state-of-the-art baselines on all metrics in the quantitative comparison.Table 1 compares CineMaster with MotionCtrl and Direct-A-Video under aligned control inputs.
- Training Paradigm Ablation: Jointly training the Semantic Layout ControlNet and Camera Adapter achieves the best performance among the evaluated training paradigms.The final Joint Train variant is compared with staged, isolated, and ablated alternatives.
- Qualitative Comparison: CineMaster provides the best control of object motion and camera motion across moving-object and moving-camera settings.The qualitative comparison covers static or moving cameras combined with static or moving objects.
- Training Paradigm Ablation: Removing stage-one depth training produces mediocre Depth-D because the model lacks fine-grained perception of the depth control signal.The full strategy trains a DiT-based ControlNet on dense depth maps before later adaptation and joint training.
- Training Paradigm Ablation: Removing semantic labels restricts object placement to text prompts and results in poor mIoU, Traj-D, and CLIP-T.The semantic injector specifies the class labels associated with 3D boxes.
- Training Paradigm Ablation: Separately training the semantic and camera modules creates a training–inference discrepancy because the modules lack cross-module communication.The isolated setting combines separately trained components only during inference.
5. Limitations and Conclusions
CineMaster concludes with controllable, 3D-aware cinematic video generation while identifying accurate open-set object pose estimation as an unresolved limitation. The framework combines a 3D-native workflow with conditional text-to-video diffusion and projected depth-map control.
- Limitations: Accurate open-set object pose estimation is currently unavailable, so precise orientation control from rotating 3D bounding boxes remains future work.The paper gives a human turning-around sequence as the motivating example.
- Conclusions: CineMaster enables users to manipulate objects and cameras through a 3D-native workflow and trains a conditional diffusion model for user-intended videos.Projected depth maps are emphasized as strong visual control signals.
- Conclusions: Extensive experiments demonstrate controllable and 3D-aware cinematic video generation.