Source-linked AI summary
AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B. Lindell, Sergey Tulyakov
TL;DR
Foundational video models offer implicit 3D knowledge but often lack precise camera control without sacrificing video quality. AC3D analyzes camera-motion frequencies, transformer representations, and training-data balance, then combines the findings into a camera-control model that improves fidelity and steering while reducing training cost. Its scope remains constrained by camera-annotation bias toward static scenes and trajectories far outside the training distribution.
Problem
Existing camera-control methods for foundational video models often provide imprecise control while compromising synthesis quality.
Method
AC3D combines low-frequency camera-conditioning schedules, conditioning limited to camera-informative layers, and 20K dynamic videos with stationary cameras.
Results
AC3D reports 18% higher video fidelity and 25% more precise camera steering than the closest competitor, while videos are favored in 90% of cases.
Takeaways & Limitations
Principled analysis of camera motion enables more precise and efficient 3D camera control while maintaining high visual quality and natural scene dynamics.
Takeaways & Limitations
Camera-annotated training data is biased toward static scenes, and future work targets trajectories far outside the training distribution.
Abstract
from arXiv · showhide
Numerous works have recently integrated 3D camera control into foundational text-to-video models, but the resulting camera control is often imprecise, and video generation quality suffers. In this work, we analyze camera motion from a first principles perspective, uncovering insights that enable precise 3D camera manipulation without compromising synthesis quality. First, we determine that motion induced by camera movements in videos is low-frequency in nature. This motivates us to adjust train and test pose conditioning schedules, accelerating training convergence while improving visual and motion quality. Then, by probing the representations of an unconditional video diffusion transformer, we observe that they implicitly perform camera pose estimation under the hood, and only a sub-portion of their layers contain the camera information. This suggested us to limit the injection of camera conditioning to a subset of the architecture to prevent interference with other video features, leading to a 4x reduction of training parameters, improved training speed, and 10% higher visual quality. Finally, we complement the typical dataset for camera control learning with a curated dataset of 20K diverse, dynamic videos with stationary cameras. This helps the model distinguish between camera and scene motion and improves the dynamics of generated pose-conditioned videos. We compound these findings to design the Advanced 3D Camera Control (AC3D) architecture, the new state-of-the-art model for generative video modeling with camera control.
1. Introduction
AC3D analyzes camera control in video diffusion models from first principles and combines spectral, architectural, and data findings to improve precision without degrading synthesis quality. Its resulting model reports stronger fidelity and camera steering than prior approaches.
- Motivation: Foundational video diffusion models encode physical-world, 2D dynamic, and 3D structural knowledge implicitly but lack fine-grained camera-motion control.Existing camera-control methods often provide imprecise control while compromising synthesis quality.
- Spectral properties of camera motion: Camera motion concentrates in low frequencies and appears early in denoising, motivating restricted conditioning that improves visual fidelity and camera following.The restricted schedule yields approximately 15% higher visual fidelity and 30% better camera following while mitigating scene-motion degradation.
- Camera motion knowledge in VDiTs: Video diffusion transformers implicitly estimate camera poses, with disentangled camera information peaking in middle layers and motivating conditioning in only the first 30% of the architecture.This design reduces training parameters by approximately 4×, accelerates training and inference by 15%, and improves visual quality by 10%.
- Re-balancing the training distribution: A curated set of 20K diverse videos with dynamic scenes and stationary cameras helps distinguish scene motion from camera motion.The adjustment recovers scene dynamism while retaining effective pose-conditioned video generation.
- Contributions: 18% higher video fidelity and 25% more precise camera steering than the closest competitor, with generated videos favored in 90% of cases.These comparisons are reported for quantitative metrics and user preference studies.
2. Related work
The paper builds on text-to-video diffusion, 4D generation, and emerging camera-control methods. It extends transformer-based video generation with explicit 3D camera control while addressing limitations in prior architectures and scene quality.
- Text-to-video generation: Text-to-video methods have progressed from temporal extensions of image models to transformer-based architectures for scalable and realistic dynamic-scene generation.The paper focuses on controlling camera movements in these transformer-based video models.
- 4D generation: Optimization-based 4D generation provides explicit space-time control but remains less photorealistic than recent 2D video diffusion models.The paper instead investigates dynamic 3D scene generation by extending pretrained video diffusion models with camera control.
- Camera control for video models: The VDiT-CC design adds lightweight camera-processing blocks to a frozen VDiT backbone, mixing camera and video tokens through summation and cross-attention.The camera branch uses 128-dimensional DiT-XS blocks, while the backbone uses large 4,096-dimensional DiT-XL blocks.
- Camera control for video models: Recent camera-control methods condition video models on extrinsic matrices, Plücker coordinates, or learned control mechanisms, but prior approaches largely use U-Net backbones.The paper positions its transformer-based camera-control design against this line of work.
- Concurrent works: Concurrent transformer and U-Net camera-control approaches still report limitations in scene or visual quality, while some alternatives use predefined or non-continuous camera trajectories.These limitations motivate AC3D's focus on camera control in foundational video diffusion transformers.
3. Method
AC3D builds camera control by analyzing when and where video diffusion transformers represent camera motion, then aligning conditioning, architecture, and training data with those findings.
- Base architecture: The camera-control baseline uses a frozen 11.5B-parameter Video DiT with ControlNet-style camera tokens, cross-attention feedback, and classifier-free guidance.Camera trajectories are represented with Plücker embeddings, and the conditioning branch is trained at 256² resolution because camera motion is low-frequency.
- Motion analysis: Camera motion is concentrated in low-frequency components and is largely formed early in denoising, motivating earlier pose conditioning and a truncated noise schedule.Low-frequency components reach approximately 84% at t=0.9, while high-frequency components are not well-modeled until t=0.6.
- Camera-pose representation: Linear probing shows that VDiT implicitly estimates camera pose, with the most accurate camera information concentrated in middle layers.Pose knowledge increases around layer #9 and peaks across layers #13–21.
- Camera-pose representation: AC3D injects camera conditioning only into the first eight DiT blocks, leaving the remaining 24 blocks unconditioned to reduce interference with visual features.This design reduces trainable parameters by approximately four times and improves training speed.
- Training-data rebalancing: The training distribution supplements camera-annotated data with 20K dynamic videos captured by stationary cameras to separate scene motion from camera motion.RealEstate10K is biased toward static scenes, so stationary-camera videos keep camera conditioning active while adding dynamic content.
4. Experiments
Experiments evaluate AC3D against camera-control baselines using visual-quality and camera-steerability metrics on in-distribution and out-of-distribution data. Results show that AC3D achieves state-of-the-art performance, while ablations identify conditioning schedules, data, scaling, context, and layer selection as important design choices.
- 4.2. Main results: AC3D establishes state-of-the-art performance against all baselines, with higher visual fidelity, more precise camera steering, and over 90% overall user preference.The evaluation uses camera pose and visual-quality metrics on unseen trajectories, including RealEstate10K and MSR-VTT.
- 4.3. Ablations: Biasing training noise toward high levels improves motion, visual quality, and camera controllability, while truncating the noise schedule avoids decreased visual quality.AC3D samples training noise from a truncated normal distribution concentrated between 0.6 and 1.
- 4.3. Ablations: Adding dynamic-scene videos with stationary cameras preserves in-domain quality but improves out-of-distribution performance, whereas omitting them worsens MSR-VTT FID by ≈17% and FVD by ≈4%.The curated data helps address the static-scene bias in RealEstate10K.
- 4.3. Ablations: Using metric-scaled camera parameters improves both visual quality and camera following compared with the more ambiguous original parameters.Without scaling, RE10K FVD worsens by ≈10% and trajectory reconstruction worsens by ≈12%.
- 4.3. Ablations: Dropping camera context improves both camera following and visual quality, while joint 2D-video training slightly improves MSR-VTT FVD but makes steering up to 3× worse.Providing context to the camera branch causes ≈4% worse camera following and ≈15% lower visual quality.
- 4.3. Ablations: Limiting camera conditioning to the first 8 of 32 VDiT blocks preserves visual quality while conditioning all blocks worsens it by ≈10%.The result is consistent with middle and late layers relying on processed camera information.
5. Conclusions
The paper concludes that analyzing camera motion in video diffusion models enables more precise and efficient camera control. AC3D combines improved conditioning schedules, layer-specific control, and calibrated training data to achieve state-of-the-art synthesis while maintaining visual quality and natural scene dynamics.
- Principled analysis of camera motion leads to improvements in control precision and efficiency for video diffusion models.
- AC3D achieves state-of-the-art 3D camera-controlled video synthesis while maintaining high visual quality and natural scene dynamics.
- Future work targets data limitations and camera trajectories far outside the training distribution.
B. Limitations
The analysis is limited by trajectory, model, dataset, and motion-estimation assumptions. These constraints affect how broadly the camera-control findings and motion-spectrum details should be interpreted.
- OOD trajectories generalization: OOD camera trajectories remain difficult for AC3D and baselines, especially when they depart substantially from RealEstate10K training trajectories.The authors attribute this boundary partly to entanglement with video representations and limited trajectory diversity in pretraining and fine-tuning data.
- Motion analysis limitations: Motion-spectrum details may change with the optical-flow estimator or when analysis moves from latent space to pixel space.The central conclusions about low-frequency camera motion and its early diffusion onset are reported as visually robust, but exact spectral behavior is estimator-sensitive.
- Linear probing limitations: The linear-probing analysis uses one video-DiT architecture and RealEstate10K trajectories, so camera-information localization may differ across models or datasets.The paper notes that camera knowledge begins at the ninth block in the studied model but could be distributed differently elsewhere.
- Motion analysis limitations: Camera motion is concentrated in low frequencies, and low-frequency components emerge early in the denoising trajectory.The supplied figure passages motivate the motion analysis, while the limitation passage qualifies the precision of its spectral measurements.
C. CogVideoX Results
The CogVideoX implementation reproduces the paper’s motion-analysis pattern, supporting the reported generalizability of the findings. The section provides the implementation context but no additional quantitative comparison.
- CogVideoX generalizability: CogVideoX shows the same motion-analysis pattern as the main experiments, confirming the reported generalizability of the findings.The authors conduct the analysis on CogVideoX and report corresponding results in Figures 7 and 8.
- Scope: The section introduces the training and architectural details of the base VDiT, VDiT-CC, and downstream experiments.No further implementation values or benchmark results are supplied in the provided passages.
D.1. VDiT implementation details
The base VDiT is a large transformer video generator trained in latent space with rectified flow and long-run optimization. Its inference uses deterministic rectified flow with 40 steps as a quality–speed trade-off.
- Architecture: The base VDiT is an 11.5B-parameter, 32-block transformer with 4,096 hidden dimensions, trained in CogVideoX latent space.The model uses a 16-channel autoencoder with 4×8×8 compression and 2×2 patchification.
- Training: The model trains for 750,000 iterations with AdamW, learning rate 0.0001, weight decay 0.01, cosine scheduling, and bfloat16 precision.The setup also supports image animation by encoding and noising the first frame.
- Inference: Inference uses standard deterministic rectified flow, with 40 steps selected as a quality–speed trade-off.The supplied passage also mentions time shifting for higher resolutions and longer videos.
D.2. VDiT-CC implementation details
VDiT-CC adds lightweight camera conditioning to the transformer backbone and trains on RealEstate10K combined with dynamic videos from static cameras. Its training and noise schedules are tailored to low-frequency camera motion.
- Camera conditioning: VDiT-CC uses lightweight 128-dimensional, four-head DiT-XS camera blocks without context cross-attention to preserve visual quality and control precision.Camera poses are represented with Plücker encodings and processed alongside video tokens.
- Training data: The complete setup trains for 6K iterations on 65K RealEstate10K videos plus 20K dynamic videos with static cameras.Optimization uses AdamW with learning rate 0.0001, weight decay 0.01, and cosine scheduling.
- Noise schedule: The method uses truncated and biased noise schedules during both training and inference because camera motion is low-frequency.The final schedule removes unnecessary high-frequency probability mass compared with the non-truncated ablation.
- Ablations: Each ablation is trained for 6K iterations on 32 NVIDIA A100 80GB GPUs.
- Camera encoding: Static-dataset camera conditioning samples extrinsics and intrinsics from RealEstate10K before processing full-resolution poses in the latent-space pipeline.The passage introduces a temporal camera encoder to address the autoencoder’s fourfold temporal compression.
E. Motion analysis details
The motion analysis examines spectral volumes across generated videos and denoising stages, alongside camera-pose probing in VDiT representations. It separates scene and camera motion categories and characterizes the data and processing used for both analyses.
- Spectral analysis: The analysis inspects spectral volumes at different denoising timesteps to compare motion characteristics.The videos are generated with VDiT and analyzed through spectral volumes.
- Video annotations: The study annotates 200 generated 121-frame videos for quality, scene motion, camera motion, and camera smoothness.Annotations use 1–5 scores for quality and motion strengths, plus a binary smoothness flag.
- Spectral analysis: The spectral comparison uses separate groups with scene motion, camera motion, or both scene and camera motion.The categories are selected using manually assigned scene- and camera-motion labels.
- Spectral analysis: Spectral volumes are computed from latent-space optical flow because it is more robust at earlier denoising timesteps.The method follows the latent diffusion setup and avoids decoding out-of-distribution inputs.
- Spectral analysis: The optical-flow pipeline uses coarse-to-fine estimation, samples latent-frame relationships, and applies Fourier transforms to spatial flow coordinates.Flow is estimated for each sixth latent frame against 24 subsequent frames before spatial averaging of amplitudes.
- Pose probing: Camera-pose probing uses 1,000 RealEstate10K test videos across multiple noise levels and extracts internal VDiT representations.The representations are projected with PCA before linear regression probing.
G. Dataset construction details
The dataset construction adds videos with scene motion recorded from stationary cameras, using manual construction rather than relying solely on automated motion heuristics. Camera information is then artificially assigned with random RealEstate10K extrinsics.
- Dataset construction: The constructed dataset targets videos with dynamic scenes recorded by stationary cameras.This setting is intended to provide scene motion without camera motion.
- Dataset construction: Manual construction is chosen because automated optical-flow checks would fail across many corner cases.The proposed heuristic checks motion in the frame center and borders.
- Camera annotations: The videos receive artificial camera information by assigning random extrinsics from RealEstate10K.
H. Scaling bias
The scaling-bias procedure addresses the arbitrary scale of classical feature-based camera estimation by aligning reconstructed depths with metric depth estimates. Properly scaled cameras preserve visual quality while making camera transitions more predictable.
- Scale ambiguity: Classical feature-based camera estimation produces camera trajectories with arbitrary scale because visual features lack absolute-size priors.Objects with different real-world sizes can appear identical from suitable distances, preventing absolute scale recovery from features alone.
- Scale correction: The method rescales COLMAP camera translations by fitting rendered COLMAP depth to metric depth estimated by a pretrained model.The procedure obtains 3D points, renders COLMAP depth, estimates metric depth, solves for a rescaling parameter, and applies it to translation.
- Scaling results: Properly scaled cameras do not degrade visual quality and make camera-transition magnitudes more predictable for users.The authors report a slight visual-quality improvement and more controllable transition magnitudes.