DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation

1. Introduction

• Ray-conditioned models encode camera motion through per-pixel ray directions but lack explicit scene understanding, leaving scene geometry to be inferred implicitly.The paper links this limitation to entangled appearance and structure modeling and suboptimal camera-motion consistency.Source: §1 · p. 2
• Few prior works had explored depth in end-to-end camera-conditioned video generation, and naive depth injection did not consistently improve generation quality.Single-frame depth can lack temporal context, while joint RGB-depth modeling in one branch can cause modality interference.Source: §1 · p. 2
• DualCamCtrl generates camera-consistent RGB and depth sequences in parallel and uses SIGMA to harmonize semantic and geometric representations.Its two-stage training separates decoupled learning from fusion; the paper also analyzes how depth, camera poses, and denoising stages affect generation.Source: §1 · p. 2
• Over 40% reduction in rotation errors compared with previous methods is reported for DualCamCtrl.The paper presents this result alongside its dual-branch framework, SIGMA mechanism, and two-stage training.Source: §1 · p. 2

2. Related Work

• Foundational diffusion video models support text-to-video and conditional synthesis, with later adaptations for camera control, motion transfer, object manipulation, and multimodality.Examples include CogVideoX, SVD, Wan, Lumiere, VideoCrafter, ModelScope-T2V, Latte, and Open-Sora.Source: §2 · pp. 2–3
• Camera-control methods progressed from explicit pose conditioning to ray-based Plücker embeddings and pre-DiT conditioning, but preserving intended camera trajectories remains challenging.The cited lines include MotionCtrl [86], CameraCtrl [24], VD3D [2], and later approaches [25] [78].Source: §2 · p. 3
• Novel-view-synthesis methods use learned image or video priors, while some depth- or point-cloud-based approaches use multi-stage projection and unprojection rather than end-to-end camera-conditional generation.ReconFusion [90] and CAT3D [17] require per-scene optimization; ReconX [50], ViewCrafter [98], and GEN3C [64] use depth or point clouds.Source: §2 · p. 3

3.1. Preliminary

• Camera-controlled video diffusion conditions noisy video latents on context and camera pose, predicts added noise during denoising, and decodes the resulting latent into video.Camera geometry is represented with ray-based Plücker conditioning.Source: §3.1 · p. 3

3.2. Dual-Branch Camera-Controlled Video Generation

• The method uses depth estimated from the input frame because a single-frame depth map alone lacks temporal consistency for guiding long-range generation.The dual-branch design is intended to propagate depth information through the video while minimizing interference with RGB synthesis.Source: §3.2 · pp. 3–4
• DualCamCtrl uses parallel RGB and depth branches conditioned on shared camera poses to generate modality-specific video representations.The input image and its estimated depth map are encoded separately, combined with the camera embedding and noise, and processed by the respective branches.Source: §3.2 · p. 4

3.3. Injecting Depth Cues for Mutual Alignment

• SIGMA aligns independently evolving RGB and depth branches through semantic-guided bidirectional interaction, with RGB anchoring early layers and depth refining geometry later.The design addresses local semantic misalignment from one-way alignment and disrupted appearance coherence from geometry-guided alignment.Source: §3.3 · p. 4, Figure 3 · p. 5
• The 3D fusion block uses 3D convolutions to combine semantic and geometric information across spatial and temporal dimensions.A bottleneck design and frame-wise gating are used to balance computational cost and adapt fusion strength to temporal dynamics.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Two-stage Training Pipeline

• Two-stage training first learns appearance and geometry without cross-branch fusion, then enables fusion to exploit the RGB branch’s appearance cues and depth branch’s geometric structure.The decoupled stage addresses convergence problems when both branches are trained jointly from the outset; the fusion block is zero-initialized.Source: §3.4 · pp. 5–6

4.1. Influence Dynamics Across Denoising Stages.

• Camera-pose similarity to RGB latents peaks in the early-to-mid schedule, when global structure is formed, then drops sharply in late denoising.Mid layers also carry the most accurate camera-pose and depth information; early-stage similarity varies substantially with schedule or conditioning strength.Source: §4.1 · p. 6

4.2. Which stage is important for camera control?

• The stage-scheduling tests begin with 15 steps distributed uniformly across three stages, then allocate additional steps to different stages.Timesteps are sampled linearly within each stage; within-stage distribution changes had negligible impact in pilot experiments.Source: §4.2 · p. 6
• Extra early denoising steps yield the largest quality gains and help establish global camera geometry before appearance refinement.Insufficient early denoising destabilizes alignment and degrades FVD.Source: §4.2 · p. 6, Figure 5 · p. 6
• After early denoising stabilizes coarse structure, additional early camera guidance has diminishing returns, while late steps refine boundaries and high-frequency textures.The reported threshold is 50 steps under the authors’ setting; late refinement also contributes to pose-consistent local details.Source: §4.2 · pp. 6–7, Figure 6 · p. 7

5. Experiments

• The evaluation uses REALESTATE10K training data, fine-tunes pretrained video diffusion models, and assesses distribution, consistency, motion, camera errors, and human preference.The I2V baselines include MOTIONCTRL, CAMERACTRL, WAN-V2.1, and SEVA; T2V comparisons include CAMERACTRL, MOTIONCTRL, and AC3D.Source: §5 · p. 7

5.1. Comparisons

• 40% reduction in rotation and translation errors accompanies consistent objective-metric improvements in I2V and T2V, including a substantial FVD decrease, especially in I2V.Baselines are evaluated on the same split and metrics; the authors exclude one method because its settings and application scenarios differ.Source: §5.1 · p. 7, §5 · p. 7
• DualCamCtrl produces higher visual fidelity and temporal coherence and receives the highest overall preference rate in the human study.Figure 7 compares methods given the same camera pose and input image.Source: §5.1 · p. 7, Figure 7 · p. 8

5.2. Ablation Study

• Removing depth guidance degrades FVD and motion strength, which the authors interpret as evidence that depth supplies structural cues for camera-controlled scenes.The ablation is conducted on REALESTATE10K.Source: §5.2 · p. 8, Table 4 · p. 8
• Direct joint training through only the fusion stage yields unstable convergence and suboptimal temporal coherence; the two-stage variant performs better across all metrics.The authors attribute the improvement to decoupling appearance and geometry learning before fusion.Source: §5.2 · p. 8, Table 3 · p. 8, Figure 8 · p. 8

6. Conclusion

• Over 40% reduction in rotation errors compared with previous methods is reported for DualCamCtrl.The paper presents the model as integrating depth for more accurate camera-controlled video generation.Source: §6 · p. 8
• SIGMA and two-stage training synchronize RGB and depth sequences to improve geometry awareness.DualCamCtrl uses a dual-branch video diffusion model that integrates depth.Source: §6 · p. 8
• The experiments offer insights into camera-controlled video generation, with potential relevance to other video generation tasks.The paper describes this potential as a hope, not a demonstrated transfer result.Source: §6 · p. 8

7.1. Detail of 3D Fusion Block

• The fusion block projects embeddings down, processes them with stacked 3D convolutions, then maps them back through a zero-initialized output layer.A frame-level gate modulates geometric information per frame; the bottleneck is intended to balance accuracy and efficiency.Source: §7.1 · p. 14

7.2. Detail of Training Pipeline

• Training first optimizes RGB and depth branches independently, then couples them through a 3D-aware fusion module for cross-modal learning.The two stages aim to establish modality-specific representations before using complementary cues to strengthen cross-modal correspondence.Source: §7.2 · p. 14, Figure 10 · p. 14

7.3. Detail of T2V Setting

• In the T2V setting, the dual branches generate without an input image or initial frame, while RGB provides semantic guidance to depth.The paper reports consistent overall performance improvement from this cross-modal guidance in Tables 1 and 2.Source: §7.3 · p. 14, Figure 11 · p. 14

8.1. Initial Interpretation of Depth Information

• Depth inputs can be processed like low-light, hazy, or monochromatic visual cues, yet generated videos show minimal distortion or artifacts.The authors attribute this adaptability to the VAE-DiT ([78]) backbone.Source: §8.1 · p. 15, Figure 12 · p. 15

8.2. Extented Analysis

• Depth-branch similarity increases steadily and remains high, especially in middle and late denoising, as depth features reinforce geometric cues.The analysis contrasts depth's persistent local geometric correction with RGB's reliance on early global alignment.Source: §8.2 · p. 15, Figure 13 · p. 15
• Insufficient early-stage denoising destabilizes alignment and degrades FVD performance, with high CKA-similarity variance reflecting the instability.The authors suggest that the camera–scene relationship is primarily established early, and insufficient early refinement disrupts later feature consistency.Source: §8.2 · p. 15

Ray Conditioning

• Camera views are represented by dense per-pixel Plücker rays derived from camera intrinsics and extrinsics.A pretrained WANv2.1 camera pose encoder produces compact latent features from the rays.Source: §9 · pp. 15–16

User Study

• The user study table caption reports normalized 0–1 scores, with higher scores better, averaged across participants and groups evaluated independently.Independent evaluation accounts for different numbers of compared methods.Source: Table 5 · p. 16

Hyperparameter

• Training uses batch size 8 and learning rate 3 × 10^-6, below the original setup's 1 × 10^-5; the scheduler remains linear.The authors report better convergence performance with this configuration.Source: §9 · p. 16

Evaluation Datasets

• Evaluation uses RE10K with frame strides of 2–8 and DL3DV with stride 1, filtering videos with low VBench motion-smoothness and consistency scores.DL3DV is described as already exhibiting large camera motion.Source: §9 · p. 16

Prompt Generation

• Captions are generated at the chosen sample stride to ensure at least one captioned frame per video while avoiding redundant computation.The stated motivation is the computational cost and resource inefficiency of captioning every frame.Source: §9 · p. 16

10.1. More Experimental Analysis

• Plücker-only conditioning can conflate camera motion with appearance, producing unintended rotations and inconsistent parallax across scenes.Source: §10.1 · p. 16
• Static first-frame depth propagation produces temporal artifacts, whereas multi-frame depth synthesis yields more plausible outputs.The comparison includes propagating the first frame's depth to all subsequent frames as a conditioning signal.Source: Figure 16 · p. 17, §10.1 · p. 17
• The user study finds DualCamCtrl outperforms all baselines across camera-trajectory consistency, smoothness and coherence, visual quality and detail, and semantic consistency with ground truth.These are the four evaluation aspects specified for participants.Source: §10.1 · p. 17
• Single-branch RGB-depth integration couples the modalities, causing interference and misalignment that adversely affect generation quality.The integration strategy uses a shared encoder to generate RGB and depth within one branch.Source: Figure 17 · p. 17, §10.1 · p. 17

10.2. More ablation study

• Adding spatial, 3D spatiotemporal, and frame-gating modules progressively improves performance in the 3D fusion ablation.Source: Table 7 · p. 18, §10.2 · p. 18

10.1. More Experimental Analysis

• The dual-branch framework balances RGB-depth interaction without entanglement, achieving consistent geometry and vivid appearance.The comparison describes the single-branch alternative as exacerbating misalignment.Source: §10.1 · pp. 17–18

10.3. More Visual Comparison

• Figures 19 and 20 provide additional qualitative examples of DualCamCtrl.Source: §10.3 · p. 18

10.2. More ablation study

• SIGMA achieves the best video-quality metrics and camera-motion consistency among one-way, geometry-guided, and semantic-guided mutual alignment strategies.The comparison also examines the layer ranges used for RGB and depth feature injection.Source: Table 8 · p. 18, §10.2 · p. 18

• Large inter-frame motion remains a difficulty for stable, coherent end-to-end video generation.The paper identifies this as a challenge shared with video generation more broadly.Source: §11 · p. 18
• Duplicating the backbone for RGB and depth is acceptable at 1.3B scale but less favorable for larger models; compact depth-branch distillation is proposed as a future direction.The paper notes that two WAN V2.1 backbone copies at 1.3B still yield clear performance advantages.Source: §11 · p. 18