PaperlayerView exact revision on arXiv

Source-linked AI summary

CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation

Haoyu Zhao, Zihao Zhang, Jiaxi Gu, Haoran Chen, Qingping Zheng, Pin Tang, Yeyin Jin, Yuang Zhang, Junqi Cheng, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang Jiang

arXiv:2604.09201v1cs.CV

TL;DR

Camera-controllable video generation still faces labor-intensive manual trajectory specification and imprecise control from semantic prompts. CT-1 predicts camera trajectories by reasoning over visual observations and semantic instructions, and the paper reports high-quality camera-controllable videos.

  • Problem

    Existing methods require labor-intensive manual camera parameters or provide imprecise semantic control, while consistent alignment with user intent and scene context remains challenging.

    Source: §1 · p. 2

  • Method

    CT-1 jointly reasons over visual observations and semantic instructions to infer camera trajectories, modeling trajectory distributions with a Diffusion Transformer and Wavelet-based Regularization Loss.

    Source: §1 · p. 3

  • Results

    The authors report that CT-1 consistently produces high-quality camera-controllable videos.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The VLC model provides a structured intermediate representation connecting multimodal understanding with controllable video synthesis.

    Source: §6.3.1 · p. 24

    Multiple semantically valid camera trajectories may exist for the same instruction.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.

1 Introduction

CT-1 infers scene-aware, intention-aligned camera trajectories for video generation using vision-language reasoning, a Diffusion Transformer, and Wavelet-based regularization. The paper introduces CT-200K, with over 47 million frames, and reports a 25.7% improvement in camera control success rate over prior methods.

1 Introduction

  • Explicit parameterized control requires labor-intensive manual trajectories, while high-level semantic methods can control motion imprecisely or fail to follow instructions, as in Wan2.2 [27].Existing methods therefore struggle to match camera trajectories to both user intent and scene context across diverse scenarios.Source: §1 · pp. 1–2
  • VLC models jointly reason over visual content and semantic cues to predict future spatiotemporal camera poses, but trajectory learning is difficult because images provide indirect motion cues rather than explicit spatial targets.The paper identifies data construction and temporally coherent trajectory prediction as central challenges.Source: §1 · p. 2
  • 25.7% improvement in camera control success rate is reported when CT-1 is coupled with a controllable video generator, outperforming existing state-of-the-art approaches.The introduction also describes out-of-distribution evaluation against Wan2.2 [27] and Gen3C [24].Source: §1 · pp. 2–3
  • CT-1 injects camera-context tokens from vision-language modules into a Diffusion Transformer to model plausible trajectories, with wavelet regularization promoting smooth, physically stable motion.The model jointly uses visual observations and semantic instructions to infer scene-aware, intention-aligned, temporally coherent camera trajectories.Source: §1 · pp. 2–3
  • CT-200K contains over 47 million frames across multiple scenarios, and its curation pipeline combines task-specific and general-purpose models to address scarce reliable camera-motion annotations.The pipeline was designed to support training CT-1.Source: §1 · p. 3

2 Related Works

Video-generation models have advanced toward scalable Diffusion Transformers, but camera control remains limited by explicit camera inputs and weak semantic–spatial alignment. The related work motivates CT-1 as a way to generate trajectories from visual content and language instructions.

2.1 Video Generation

  • DiT architectures increasingly replace U-Net approaches in video generation, with scalable Transformer depth supporting large-scale training and joint spatial-temporal modeling [22] [5] [8] [27] [36] [37].U-Nets offer efficient inference and architectural flexibility, but limited capacity constrains gains from large-scale scaling.Source: §2.1 · p. 3

2.2 Camera-Controllable Video Generation

  • CameraCtrl and MotionCtrl [32] inject explicit camera parameters, but insufficient semantic–spatial alignment can yield spatially inconsistent or visually implausible video.Controllable video generation commonly adds conditions through concatenation or cross-attention.Source: §2.2 · p. 3

2.3 Camera Trajectory Estimation

  • CameraCtrl and MotionCtrl [32] use predefined or video-extracted extrinsics, while ChatCam [19] generates text-driven motion without visual-content awareness; CT-1 targets joint visual-language trajectory modeling.Automatically generating plausible camera trajectories from high-level semantic information remains underexplored.Source: §2.3 · pp. 3–4

3 Method

CT-1 maps visual observations and language instructions to camera trajectories, then uses them to condition video generation. The method combines multimodal trajectory prediction, frequency-aware trajectory regularization, and a dataset curation pipeline for training.

3 Method

  • The method presents CT-1 as a vision-language-camera framework for predicting camera motion and enabling camera-controllable video generation.Its framework comprises vision-language modules, a camera transformer, and a video generation module.Source: §3 · p. 4

3.1 Problem Formulation

  • CT-1 predicts a camera trajectory from a visual observation and instruction, with poses on SE(3) that must be semantically aligned, temporally continuous, and potentially non-unique.Each pose is parameterized by a rotation matrix and translation vector.Source: §3.1 · p. 4

3.2.1 Vision and language module

  • DINOv2 [21] and SigLIP [39] encode complementary visual cues, while LLaMA-2 [26] fuses visual tokens, instruction tokens, and a camera-context token for trajectory conditioning.The hidden representation associated with the camera-context token summarizes visual and linguistic information for the camera transformer.Source: §3.2.1 · p. 5

3.2.2 Camera transformer

  • A DiT diffusion model learns a distribution over plausible camera trajectories, and Haar-wavelet regularization encourages smooth, stable motion by weighting low-frequency components more strongly.The decomposition separates smooth global motion from rapid local perturbations; low-frequency components preserve principal trajectory structure and overall displacement.Source: §3.2.2 · pp. 5–7

3.2.3 Camera-controllable Video Generation

  • CT-1 and the video diffusion model form a modular two-stage pipeline: predicted trajectories guide synthesis through CameraNoise without modifying the generator backbone.The same trajectory interface can also be used with CameraCtrl and MotionCtrl [32].Source: §3.2.3 · p. 7

3.3 CT-200K: VLC Dataset Construction

  • CT-200K combines VGGT camera-pose annotations and human-reviewed descriptions with general-video and spatial-reasoning scenarios.For general videos, optical-flow filtering selects clips with suitable camera motion and language models align camera descriptions with image content. Reasoning videos use egocentric object-manipulation scenes and reconstructed spatial instructions.Source: §3.3 · p. 8

4 Experiments

CT-1 improves camera-control Success Rate over the compared methods while producing video quality comparable to existing state-of-the-art backbones. Ablations report scaling gains, benefits from WavReg and reasoning-scenario data, and modest inference overhead.

4.1 Implementation Details

  • Evaluation used human-agreed Success Rate for instruction-following, plausible camera motion, and VBench for video quality, alongside CameraBench100's 100 samples across six camera-motion settings.Qualitative evaluation also used RealEstate10K, MultiCamVideo, and DrivingDoJo.Source: §4.1 · p. 8

4.2 Comparisons with Other Methods

  • CT-1 led average Success Rate: +171.1% vs the best VLM trajectory-input model, +245.8% vs the best AR trajectory-input model, +25.7% vs the best prompt-input model with PE, and +31.4% without PE.The comparison evaluates prompt-input and trajectory-input methods on CameraBench100.Source: §4.2 · p. 9, Table 1 · p. 9
  • CT-1's generated-video quality was comparable to existing state-of-the-art video backbones across VBench metrics.The authors interpret this as evidence that CT-1 estimates reasonable camera trajectories while preserving temporal consistency.Source: Table 2 · p. 9, §4.2 · p. 10
  • In dynamic and out-of-distribution scenarios, comparisons show current backbones have low sensitivity to camera-motion text; prompts such as “tracks backward while trucking right” often produce nearly static camera motion.Figure 5 compares CT-1 with CogVideoX [36], LTX-Video [8], and Wan2.2 [27] in out-of-distribution scenarios.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Ablation Study

  • CT-1 performance consistently improved across Base, Large, and Huge DiT scales of 33M, 130M, and 458M parameters, respectively.Table 3 evaluates VBench and Success Rate on Regular Motion and Complex Motion.Source: §4.3 · pp. 10–11, Table 3 · p. 11
  • WavReg ablations showed that performance first improved and then stabilized as β increased, with best results at β = 0.1.At β = 0, WavReg is disabled.Source: §4.3 · p. 11, Table 4 · p. 11
  • CT-1 maintained stable trajectory generation under repeated cross-conditioned tests, while increasing reasoning-scenario data improved modeling of complex camera motion.The tests used 10 camera descriptions across 10 scenes and vice versa; reasoning data supplied complementary supervisory signals for global understanding.Source: §4.3 · pp. 11–12, Table 5 · p. 11, Table 6 · p. 12
  • CT-1 estimation accounted for roughly 5.3% of total inference time, a modest overhead relative to downstream video generation.For a 49-frame video, CT-1 predicts a 13-step camera trajectory under 4× temporal compression.Source: §4.3 · p. 12, Table 7 · p. 12

5 Conclusion

CT-1 is a camera-controllable video generation framework that transfers spatial reasoning knowledge into video generation. Supported by CT-200K, it produces high-quality camera-controllable videos.

5 Conclusion

  • CT-1 is a Vision–Language–Camera model for camera-controllable video generation.The framework learns expressive camera motion distributions and transfers spatial reasoning knowledge into video generation for camera control.Source: §5 · p. 12
  • Supported by the curated CT-200K dataset, the approach produces high-quality camera-controllable videos.The conclusion presents this as bridging spatial reasoning and video synthesis.Source: §5 · p. 12

6 Appendix · 6.1 More Implementation Details

The appendix details CT-1’s expert-rated success-rate evaluation, wavelet-regularized loss, training setup, and integration with video diffusion models. It also describes image reconstruction used to address resolution mismatch in CameraBench100.

6.1 More Implementation Details

  • Because camera trajectories are non-unique, CT-1 is evaluated primarily by success rate rather than point-wise error metrics.Success means the generated trajectory is semantically consistent with the image–text conditions.Source: §6.1 · p. 16
  • On 100 image–text pairs, a trajectory counts as successful only when two domain experts unanimously judge it to match the described camera motion.Disagreements are counted as failures, and success rate is the successful-sample fraction.Source: §6.1 · p. 16
  • WavReg uses scale-dependent weights such as [2.0,1.0,0.5,0.25] and can set λ_dl = 0 to supervise only low-frequency trend alignment.The loss encourages global camera-motion trends while avoiding overfitting to high-frequency noise; β, the auxiliary-supervision weight, can be 0.1.Source: §6.1 · p. 16
  • The diffusion loss supports noise modeling and generative diversity, while the wavelet loss adds structural bias for semantically stable motion without changing sampling.Gradients propagate through the noise-prediction network.Source: §6.1 · p. 16
  • CT-1 was trained on CT-200K using 32 NVIDIA GPUs, 224×224 inputs, a 1×10−5 learning rate, and batch size 512.Evaluation used CameraBench100, RealEstate10K, MultiCamVideo, and DrivingDoJo; the first video frame and its motion description were CT-1 inputs.Source: §6.1 · p. 16
  • At inference, CameraNoise, CameraCtrl, and MotionCtrl receive a reference image, text prompt, and estimated trajectory; experiments used a single NVIDIA GPU.For CameraBench100, low-resolution reference images were redrawn from scene semantics, layout, and prompts to mitigate mismatch with high-resolution video backbones.Source: §6.1 · p. 16

6.2 More Qualitative Results

The qualitative results show CT-1 translating image structure and text intent into semantically aligned camera trajectories, then using those trajectories to guide video generation across varied prompts, scenes, and models.

6.2.1 Camera trajectories predicted by CT-1.

  • CT-1 predicts continuous camera-pose sequences that preserve motion semantics and temporal smoothness, including under long prompts, without abrupt changes or oscillatory artifacts.Because image-text trajectories are non-unique, the visualizations are assessed for semantic fit rather than compared with ground truth.Source: §6.2.1 · pp. 17–18, Figure 6 · p. 17, Figure 7 · p. 18

6.2.2 Multi-instance cross-validation of CT-1 with image and text inputs.

  • With one image, changing camera descriptions—including a key semantic word—produces distinguishable motions; with one description, CT-1 adapts trajectories to different scene layouts.These complementary tests examine responsiveness to textual variation and consistent motion semantics across images.Source: Figure 8 · p. 18, §6.2.2 · p. 19, Figure 9 · p. 19

6.2.3 Performances on challenging scenarios.

  • Across five challenging scenes, including cartoon settings, CT-1 predicts trajectories aligned with user intent, and generated videos follow those trajectories.The results cover complex visual content and semantic structures.Source: §6.2.3 · p. 20, Figure 10 · p. 20

6.2.4 Applying CT-1 to other camera-controllable video diffusion models.

  • CT-1 trajectories drive CameraCtrl on RealEstate10K [44] and MotionCtrl [32] on MultiCamVideo [1], with reported gains in generated-result accuracy and consistency; MotionCtrl produces smaller motion magnitude than CameraCtrl.The results demonstrate integration with both camera-controllable generation methods.Source: §6.2.4 · pp. 20–22, Figure 11 · p. 21, Figure 12 · p. 21

6.2.5 Deep-reasoning-based camera trajectory generation.

  • When prompts imply motion through scene clues rather than explicit left-right directions, CT-1 uses visual context to infer the camera movement, such as moving toward a poster-covered wall.The experiments test visual-semantic reasoning in scenes with rich visual elements.Source: §6.2.5 · p. 22, Figure 13 · p. 22

6.2.6 Application extension: driving scenarios.

  • In DrivingDoJo [31] scenes, CT-1 infers physically plausible trajectories for forward-driving prompts and guides video generation that captures forward motion and viewpoint evolution.The examples test camera control in driving scenes with geometric constraints and complex structure.Source: §6.2.6 · pp. 22–23, Figure 14 · p. 23

6.2.7 User study.

  • In pairwise user comparisons with CogVideoX [36], LTX-Video [8], and Wan2.2 [27], CT-1 receives the best preference scores across image quality, camera control accuracy, motion fidelity, and overall video quality.The study evaluates 30 selected videos.Source: §6.2.7 · p. 23, Figure 15 · p. 23

6.3 More Explanation of VLC Model

The VLC model predicts scene-aware camera trajectories from image–text conditions and uses them as explicit intermediate controls for video synthesis. Its trajectories are semantically aligned, temporally continuous, and modeled as a distribution to capture multiple valid motions.

6.3.1 Discussion of the VLC model.

  • VLC jointly encodes the reference image and instruction to predict a temporally coherent, semantically aligned camera trajectory as an explicit control signal for downstream video synthesis.This separates camera-motion modeling from video generation and supports more precise control.Source: §6.3.1 · p. 24

6.3.2 Camera properties under the VLC model.

  • Trajectories follow instruction semantics as continuous curves on SE(3), while Lipschitz continuity ensures smooth and stable camera motion.The Lipschitz condition is expressed using geodesic distance on SE(3), a Lipschitz constant, and the inter-frame time step.Source: §6.3.2 · p. 24
  • Because one instruction can admit multiple semantically valid trajectories, CT-1 models motion as a distribution and uses a Diffusion Transformer for flexible, diverse trajectory generation.This distribution reflects the inherent diversity of camera motion.Source: §6.3.2 · p. 24

6.4 More Ablation Studies

The ablations support CT-1’s diffusion trajectory model, wavelet regularization, and dedicated camera-context token; additional evaluations show its CameraBench100 advantage is stable across resampling and subset sizes.

6.4.1 Effect of diffusion transformer.

  • Compared with Gaussian regression and autoregressive prediction, CT-1’s diffusion transformer better handles one-to-many camera motion, avoiding regression averaging and long-horizon error accumulation.The comparison uses the same vision-language encoder and camera-context token design where applicable, and evaluates trajectory prediction and downstream video generation.Source: §6.4.1 · p. 25, Table 8 · p. 25

6.4.2 Statistical stability of CameraBench100.

  • 81.6 average success rate on fixed CameraBench100, 16.7 points above “Wan2.2 w/ PE”; CT-1 also wins all 20 random subsets and remains strong on larger balanced subsets.Bootstrap resampling and repeated evaluations indicate the advantage is stable and not tied to a particular subset of examples.Source: Table 9 · p. 25, §6.4.2 · pp. 25–26

6.4.3 Effect of frequency-aware modeling.

  • Against velocity, acceleration, jerk, and Fourier low-pass regularizers, WavReg preserves intended low-frequency motion while suppressing local oscillations without over-penalizing legitimate changes.The alternatives replace only the regularization term while keeping the backbone, trajectory prediction loss, and training protocol unchanged.Source: §6.4.3 · p. 26, Table 10 · p. 26

6.4.4 Effect of camera-context token.

  • On CameraBench100, proposed <CAM> Modeling performs best: text-only misses scene-dependent geometric grounding, image-only misses language-specified directions, and pooled conditioning lacks dedicated camera-aware conditioning.The ablation compares text-only, vision-only, pooled image-text, and proposed <CAM> conditioning under the same backbone.Source: §6.4.4 · pp. 26–27, Table 11 · p. 27

6.5 Camera Trajectory Analysis via Wavelet Decomposition

Wavelet decomposition separates camera motion into smooth, large-scale trends and rapid, localized changes. The analysis uses kinematic and frequency-domain descriptions to explain how high-frequency energy reflects abrupt trajectory dynamics.

6.5 Camera Trajectory Analysis via Wavelet Decomposition

  • Wavelet analysis shows low-frequency components capture smooth overall motion, while high-frequency components reveal rapid localized rotations, movements, and acceleration spikes.Figure 16 compares angular velocity magnitude, linear velocity magnitude, acceleration energy, and high-frequency D1 energy.Source: §6.5 · p. 27, Figure 16 · p. 28

6.5.1 Proof

  • The proof models linear and angular velocities as frequency components, defining high-frequency energy above a cutoff whose spikes correlate with abrupt trajectory changes.The wavelet decomposition assigns smooth overall motion to low frequencies and rapid, localized changes to high frequencies.Source: §6.5.1 · pp. 27–28

6.6 DiT-based Camera Trajectory Modeling

CT-1 models camera trajectories with a Diffusion Transformer that learns to denoise noisy trajectories conditioned on a camera-context token. Its transformer processes the conditioned sequence across time to capture long-range dependencies and camera-aware dynamics.

6.6 DiT-based Camera Trajectory Modeling

  • CT-1 uses a Diffusion Transformer to predict noise or an equivalent denoised representation of noisy camera trajectories conditioned on the <CAM> token.The clean trajectory K0 has temporal length T and dimension C; the model learns conditional denoising as part of a standard diffusion formulation.Source: §6.6 · p. 28
  • The model projects noisy trajectories into a latent space, adds timestep and camera-context embeddings, and prepends the fused <CAM> token to the trajectory sequence.The augmented sequence is further enriched with learnable positional embeddings.Source: §6.6 · p. 29
  • Transformer blocks use global self-attention across time, and the model discards the output <CAM> token to produce predicted noise or a denoised camera trajectory.This design lets camera poses interact directly with camera context and captures long-range temporal dependencies and camera-aware trajectory dynamics.Source: §6.6 · p. 29

6.7 Analysis of the Effect of β in WavReg Loss.

The analysis frames β as the weight on wavelet-domain supervision relative to the diffusion loss and interprets its effect through gradient scale and direction. It shows that when the wavelet-to-diffusion gradient ratio is small, the update direction remains close to the diffusion-only direction.

6.7 Analysis of the Effect of β in WavReg Loss.

  • β > 0 controls the relative contributions of diffusion and wavelet losses and directly scales the wavelet-gradient component, changing the update’s magnitude and direction.Source: §6.7 · p. 29
  • The gradient ratio r(β) quantifies the relative scale of the two gradient contributions and provides a principled interpretation of β; minibatch stochasticity motivates using an expectation or moving average.Source: §6.7 · pp. 29–30
  • When r_k(β) < 1, the combined gradient remains nonzero; when r_k(β) ≪ 1, its direction stays close to the diffusion-gradient direction, with wavelet supervision providing a bounded perturbation.Source: §6.7 · p. 30

6.8 CT-200K Dataset Construction

CT-200K is built through a curation pipeline that creates aligned annotations for general and spatial-reasoning videos and estimates camera poses. The resulting dataset contains 200,829 samples and over 47M frames, each paired with a textual description and camera trajectory.

General video scenarios

  • Video-VLMs describe camera motion over time while Image-VLMs describe static scene content; LLM filtering removes unsupported motion cues to align the descriptions.The resulting annotations jointly describe camera motion and visual content in a synchronized manner.Source: §6.8 · p. 30

Reasoning video scenarios

  • For spatial-reasoning videos, the pipeline selects segments with meaningful object interactions or spatial changes and uses visual localization cues to annotate object transformations and movements.The annotations provide structured, interpretable accounts of object-level interactions and spatial reasoning events.Source: §6.8 · p. 31

Camera parameters estimation

  • A dedicated camera-pose estimation model processes frame-wise geometry and inter-frame relationships to estimate time-varying camera parameters.The pose representation includes camera intrinsics and extrinsics: rotation and translation.Source: §6.8 · pp. 31–32

Statistics

  • The dataset contains 200,829 samples—120,814 from General Scenarios and 80,015 from Reasoning Scenarios—across over 47M frames, with each sample paired to a textual description and camera trajectory.These paired annotations and trajectories provide supervision for camera-controllable video generation and trajectory modeling.Source: §6.8 · p. 32

Related papers

Loading 2604.09201v1…