PaperlayerView exact revision on arXiv

Source-linked AI summary

CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation

Haoyu Zhao, Zihao Zhang, Jiaxi Gu, Haoran Chen, Qingping Zheng, Pin Tang, Yeyin Jin, Yuang Zhang, Junqi Cheng, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang Jiang

arXiv:2604.09201v1cs.CV

TL;DR

Camera-controllable video generation still struggles to align camera motion with user intent without labor-intensive trajectory specification. CT-1 uses a Vision-Language-Camera model with diffusion-based trajectory modeling and curated training data, producing high-quality controllable videos while improving camera-control accuracy by 25.7% over prior methods.

  • Problem

    Existing approaches provide imprecise text-based camera control or require labor-intensive manual trajectories, limiting automated camera-controllable video generation.

  • Method

    CT-1 jointly reasons over visual observations and language instructions, models plausible camera-trajectory distributions with a Diffusion Transformer and Wavelet-based Regularization Loss, and conditions video generation on those trajectories.

  • Results

    The framework achieves improved, high-quality camera-controllable video synthesis and is evaluated on out-of-distribution scenarios against models including Wan2.2 and Gen3C.

  • Takeaways & Limitations

    CT-1 bridges spatial reasoning and video synthesis by generating camera trajectories that are spatially aware, temporally coherent, and aligned with semantic instructions.

  • Takeaways & Limitations

    Camera trajectories are non-unique: multiple semantically valid trajectories may exist for the same instruction.

Abstract

from arXiv · show

Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.

1 Introduction

Camera-controllable video generation remains difficult because text-based methods are imprecise while explicit trajectory control requires labor-intensive manual parameters. CT-1 addresses this gap by transferring spatial reasoning into video generation through scene-aware camera trajectory prediction.

  • Evaluation: The framework is evaluated on out-of-distribution scenarios against existing state-of-the-art video generation models, including Wan2.2 and Gen3C.The evaluation explicitly includes comparisons with these camera-control baselines or related models.
  • Motivation: Existing methods either use high-level semantic prompts or explicit camera parameters, but neither reliably provides scalable, intention-aligned control.Semantic methods may fail to follow camera instructions, whereas parameterized methods require manually specified trajectories.
  • Motivation: Vision-Language-Camera models jointly reason over visual content and semantic cues to predict future spatiotemporal camera poses.This design targets improved semantic alignment between camera trajectories and generated video content.
  • Contributions: CT-1 bridges high-level user intention and low-level scene pixels by inferring scene-aware, intention-aligned camera trajectories for diffusion-based video generation.The model is introduced as a specialized Vision-Language-Camera framework for camera-controllable video synthesis.
  • Contributions: CT-1 injects camera-context tokens from vision-language modules into a Diffusion Transformer and applies Wavelet-based Regularization Loss to model camera trajectory distributions.The regularization operates in the frequency domain to support temporally coherent and physically stable camera motion.
  • Contributions: CT-200K provides over 47 million frames across multiple scenarios to support CT-1 training, alongside a dedicated data curation pipeline.The dataset and pipeline are presented as infrastructure for training the proposed model.

2 Related Works

Prior camera-controllable video methods commonly inject predefined camera parameters into diffusion models, while semantic approaches do not adequately connect camera motion to visual content. CT-1 instead combines visual-language reasoning with trajectory distribution modeling and conditioned video synthesis.

  • Video Diffusion Models: Recent video generation methods increasingly use Diffusion Transformers because U-Net capacity can constrain gains under large-scale scaling.The related work contrasts U-Net encoder-decoder designs with the increasing adoption of DiT architectures.
  • Controllable Video Generation: Controllable video generation typically adds conditioning through concatenation or cross-attention, including explicit camera parameters in camera-control methods.Examples include MotionCtrl and CameraCtrl, which inject camera extrinsics to induce view changes.
  • Camera Trajectories: Camera trajectories jointly describe camera pose variations through intrinsics and extrinsics during video capture.Existing methods constrain generation using predefined or video-extracted camera extrinsics.
  • Proposed Framework: CT-1’s framework combines a vision-language module, a camera-trajectory Diffusion Transformer, and controllable generation models conditioned on predicted trajectories.The framework is organized around semantic embedding, trajectory-distribution modeling, and video synthesis.
  • Camera Trajectories: Automatically generating plausible camera trajectories from high-level semantic information remains underexplored, and text-only ChatCam lacks awareness of underlying visual content.CT-1 is motivated by modeling trajectories under joint visual-content and language input.

3 Method

CT-1 is a Vision-Language-Camera framework that predicts scene-aware camera trajectories from visual observations and instructions, then conditions video diffusion on them. Its design combines diffusion-based trajectory modeling, frequency-aware regularization, and CT-200K data curation.

  • VLC formulation: CT-1 maps visual observations and textual instructions to camera trajectories whose poses are represented on the SE(3) manifold.The model targets semantic alignment and temporal continuity between instructions, scene content, and camera motion.
  • Vision-language modules: Dual DINOv2 and SigLIP vision encoders provide complementary local and semantic visual cues for multimodal camera reasoning.A language module fuses these visual representations with instructions, producing a camera-context feature for trajectory prediction.
  • Camera trajectory diffusion: The camera transformer uses a diffusion process to model a distribution over plausible camera trajectories rather than a single deterministic numerical solution.Noisy pose sequences are conditioned on a timestep embedding combined with a learned camera-context embedding.
  • Wavelet-based regularization: Low-frequency wavelet components preserve principal camera translation, rotation, and global displacement, motivating WavReg for stable and smooth trajectories.The method applies an orthonormal Haar DWT over time to separate low-frequency structure from high-frequency temporal disturbances.
  • Video generation: The estimated trajectory conditions a controllable video diffusion model that generates a video sequence from a reference image and description.This connects CT-1’s predicted camera motion to the final video synthesis stage.

4 Experiments

Experiments evaluate CT-1's camera controllability, video quality, robustness, scaling, regularization, data efficiency, and computational cost. CT-1 achieves the strongest reported camera-control accuracy while maintaining comparable video quality and modest inference overhead.

  • Evaluation setup: Success Rate evaluates whether trajectories are semantically consistent with instructions and physically plausible, using agreement between two human experts.CameraBench100 contains 100 samples spanning six camera-motion settings and diverse scenes.
  • Camera control: CT-1 achieves the best average camera-control accuracy, improving Success Rate by 25.7% over the best prompt-input model with Prompt Extension.It also improves over the best VLM-based trajectory-input model by 171.1%, the best AR-based model by 245.8%, and the best prompt-input model without Prompt Extension by 31.4%.
  • Video quality: CT-1 produces video quality comparable to existing state-of-the-art video backbones across VBench metrics.The results indicate reasonable camera trajectories in motion and structure while preserving temporal consistency.
  • Qualitative evaluation: Qualitative OOD comparisons reveal that existing backbones can be insensitive to camera-motion text, sometimes producing nearly static motion for compound instructions.The comparison includes CogVideoX, LTX-Video, and Wan2.2.
  • Ablations: Overall performance consistently improves as CT-1 scales from Base to Large and Huge configurations.The evaluated DiT models contain 33M, 130M, and 458M parameters, respectively.
  • Ablations: Increasing the proportion of reasoning-scenario data improves modeling of complex camera motion and provides richer, complementary supervisory signals.The data-efficiency study compares general-scenario and reasoning-scenario subsets of CT-200K.
  • Efficiency: CT-1 accounts for roughly 5.3% of total inference time, adding modest overhead relative to downstream video generation.For a 49-frame video, CT-1 predicts a 13-step trajectory under 4× temporal compression.

5 Conclusion

The paper concludes that CT-1 transfers spatial reasoning into camera-controllable video generation through expressive camera-motion distributions. Supported by CT-200K, the framework produces high-quality videos and bridges spatial reasoning with video synthesis.

  • 5 Conclusion: CT-1 transfers spatial reasoning knowledge into video generation by learning expressive camera-motion distributions.The framework is supported by the curated CT-200K dataset.
  • 5 Conclusion: The framework consistently produces high-quality camera-controllable videos while bridging spatial reasoning and video synthesis.

6.1 More Implementation Details

CT-1 evaluates camera control through expert-judged success rates and generates scene- and text-conditioned trajectories for downstream video diffusion models. Its visualizations and qualitative applications indicate temporally continuous, semantically aligned camera motion across varied scenes and scenarios.

  • Evaluation: Camera-control success rate is computed from trajectories unanimously judged correct by two independent domain experts.Disagreements are conservatively counted as failures.
  • Implementation: CT-1 is trained on CT-200K and evaluated across CameraBench100, RealEstate10K, MultiCamVideo, and DrivingDoJo.The model receives each video's first frame and associated camera-motion description as input.
  • Trajectory Prediction: CT-1 trajectories are conditioned jointly on reference images and textual descriptions to predict temporally continuous camera poses.The model extracts scene structure from images and parses camera-motion intent from text.
  • Trajectory Quality: The predicted trajectories preserve semantic alignment, temporal continuity, and smoothness without abrupt changes or oscillatory artifacts.They provide a stable camera-trajectory prior for subsequent diffusion-based video generation.
  • Conditioning: CT-1 adapts its predicted motion to different textual emphases and scene layouts, supporting diverse trajectory distributions under changing image-text conditions.The experiments separately vary text for one scene and images for one camera description.

6.3 More Explanation of VLC Model

The VLC model predicts future camera trajectories from visual context and language intent, treating valid motions as a distribution rather than a single deterministic curve. CT-1 implements this framework with Diffusion Transformer-based trajectory generation and explicit camera-aware conditioning.

  • VLC Model: VLC jointly conditions camera trajectories on reference images and textual instructions because identical commands can map to different motions across scene layouts and viewpoints.This distinguishes VLC from text-only camera-motion prediction and generic vision-language regression.
  • VLC Model: VLC predicts plausible future trajectories before video generation and uses them as explicit control signals for synthesis.This differs from trajectory estimation methods that recover motion from observed videos or image sequences.
  • VLC Model: The model connects multimodal understanding with controllable video synthesis through a structured intermediate camera-trajectory representation.The representation makes camera modeling explicit within the generation framework.
  • Trajectory Properties: VLC trajectories are described as semantically aligned continuous curves on SE(3) and temporally Lipschitz continuous for smooth, stable motion.The Lipschitz condition bounds inter-frame changes using the geodesic distance on SE(3).
  • Trajectory Properties: Because multiple trajectories may satisfy one instruction, CT-1 models camera motion as a distribution to enable flexible and diverse generation.This design addresses the inherent non-uniqueness of semantically valid trajectories.

6.4 More Ablation Studies

Ablation studies isolate the effects of diffusion-based trajectory modeling, WavReg, and camera-context tokens. CT-1 remains robust across evaluation subsets, while wavelet regularization and dedicated multimodal conditioning improve trajectory quality and control behavior.

  • Trajectory Generation: CT-1 is compared with deterministic Gaussian Regression and autoregressive transformer variants under matched image-text inputs and training protocols.The comparison evaluates both trajectory prediction and downstream camera-controllable video generation.
  • Statistical Stability: 81.6 average success rate on the fixed CameraBench100 subset exceeds Wan2.2 w/ PE by 16.7 points.The reported bootstrap intervals are 81.6 [74.8, 87.9] and 64.9 [56.8, 72.7], respectively.
  • Statistical Stability: CT-1 wins 20/20 random balanced subsets and maintains its advantage on larger balanced subsets, indicating robustness to subset selection.The analysis examines bootstrap resampling, repeated random subsets, larger subsets, and performance gaps.
  • Frequency-Aware Modeling: WavReg better preserves low-frequency motion trends while suppressing local oscillations than velocity, acceleration, jerk, and low-pass regularization baselines.Simple smoothness losses can over-penalize legitimate motion changes and reduce control accuracy.
  • Camera-Context Token: The proposed <CAM> Modeling performs best against text-only, vision-only, and pooled image-text conditioning.Text-only conditioning lacks scene-dependent geometric grounding, while image-only conditioning cannot reliably follow language-specified directions.

6.5 Camera Trajectory Analysis via Wavelet Decomposition

Wavelet decomposition separates camera motion into smooth low-frequency behavior and rapid high-frequency changes. The analysis uses kinematic quantities and frequency-domain energy to characterize how abrupt temporal dynamics appear in trajectories.

  • Frequency Decomposition: Wavelet decomposition analyzes angular velocity magnitude, linear velocity magnitude, acceleration energy, and high-frequency D1 energy.These quantities are compared across two camera trajectories.
  • Frequency Decomposition: Low-frequency components vary smoothly and represent gradual rotations and steady overall camera motion.They capture the broad temporal trend of the trajectory.
  • Frequency Decomposition: High-frequency components show sharp fluctuations that represent rapid, localized changes in angular velocity.Pronounced high-frequency energy spikes correlate with abrupt trajectory changes.
  • Kinematic Analysis: The analysis decomposes linear and angular velocity energy into low- and high-frequency components using a wavelet transform.The total motion energy is characterized through the kinematic quantities v and ω.

6.6 DiT-based Camera Trajectory Modeling

CT-1 models camera trajectories with a conditional Diffusion Transformer that combines noisy trajectories, diffusion time, and camera-context information. Its composite training objective adds wavelet-domain supervision, whose weight controls the magnitude and direction of parameter updates.

  • DiT-based Camera Trajectory Modeling: CT-1 represents clean camera trajectories as K0 ∈ R^(T×C) and learns conditional denoising from noisy trajectories using a Diffusion Transformer.The model predicts noise or an equivalent denoised representation conditioned on the <CAM> token.
  • DiT-based Camera Trajectory Modeling: The noisy trajectory is projected into latent space, while diffusion time and camera context are fused into a single <CAM> token prepended to the trajectory sequence.Transformer blocks apply global self-attention across time steps before projecting outputs back to trajectory space.
  • DiT-based Camera Trajectory Modeling: Global self-attention lets each camera pose interact with camera-context information, supporting long-range temporal dependencies and camera-aware trajectory dynamics.The resulting design is intended for camera-controllable video generation within a diffusion framework.
  • DiT-based Camera Trajectory Modeling: CT-1 trains with a composite objective combining standard diffusion loss and wavelet-domain supervision, weighted by β.The wavelet term provides auxiliary supervision in the frequency domain.
  • Effect of β in WavReg Loss: β directly scales the wavelet-gradient component, changing both the magnitude and direction of the stochastic-gradient update.The gradient ratio r(β) measures the relative scale of the diffusion and wavelet contributions, while the angle analysis bounds directional deviation when the ratio is small.
  • Effect of β in WavReg Loss: When rk(β) ≪ 1, the update direction remains close to that induced by diffusion loss while wavelet supervision acts as a controlled, bounded perturbation.This conclusion follows from the stated lower bound and deviation-angle bound.

6.8 CT-200K Dataset Construction

CT-200K is built through a multi-stage curation pipeline that aligns camera motion, visual content, spatial reasoning, and estimated poses. The resulting dataset contains 200,829 samples and over 47M video frames spanning general and reasoning scenarios.

  • Examples: Figure 17 shows representative CT-200 examples with video frames paired with fine-grained camera-motion prompts produced by the curation pipeline.The examples illustrate the dataset’s synchronized visual and camera-motion annotations.
  • General video scenarios: For general videos, Video-VLMs generate dynamic prompts describing camera motion, while Image-VLMs describe static scene content and time-invariant elements.The dynamic description captures viewpoint and trajectory evolution across the video.
  • General video scenarios: LLMs refine the dynamic and static descriptions by removing motion information unsupported by frame-level content, reducing hallucinated or irrelevant motion cues.The filtering step enforces semantic alignment between dynamic camera descriptions and static visual descriptions.
  • Reasoning video scenarios: Reasoning scenarios are selected for meaningful object interactions or spatial changes and annotated with object manipulations and spatial relationships.Video-VLM features identify candidate segments, while LLMs use visual evidence including object localization cues to construct annotations.
  • Reasoning video scenarios: The reasoning annotations provide structured, interpretable accounts of object-level interactions and spatial events through spatially explicit, action-oriented descriptions.These annotations are designed to capture fine-grained spatial reasoning within selected video segments.
  • Camera parameters estimation: Camera pose estimation predicts time-varying camera parameters from frame-wise geometry and inter-frame relationships, producing explicit trajectory representations.Each pose contains intrinsic parameters plus rotation and translation extrinsics describing camera orientation and position.
  • Statistics: The curation pipeline produces 120,814 general-scenario samples and 80,015 reasoning-scenario samples, totaling 200,829 samples and over 47M video frames.The dataset covers diverse scenes and dynamic motions, with each sample paired with textual description and associated annotations.
Loading 2604.09201v1…