PaperlayerView exact revision on arXiv

Source-linked AI summary

VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

Sixiao Zheng, Zimian Peng, Yanpeng Zhou, Yi Zhu, Hang Xu, Xiangru Huang, Yanwei Fu

arXiv:2502.07531v5cs.CVcs.AIcs.LGcs.MM

TL;DR

Controllable I2V must coordinate camera motion, object motion, and lighting, which are physically coupled in scene appearance. VidCRAFT3 combines controls in one framework and reports strong controllability and visual fidelity across these factors, while its lighting representation remains limited to direction.

  • Problem

    Precise joint control of camera motion, object motion, and lighting direction remains challenging because viewpoint, geometry, and illumination jointly define scene appearance.

    Source: §I · p. 1

  • Method

    VidCRAFT3 jointly models camera motion, object motion, and lighting direction, using point-cloud priors, trajectory features, lighting attention, and progressive training.

    Source: §I · pp. 1–2, 1) Camera Motion Control via Point… · p. 5, 2) Object Motion Control through… · p. 5

  • Results

    VidCRAFT3 demonstrates strong controllability and visual fidelity across camera motion, object motion, and lighting direction while preserving temporal and semantic consistency.

    Source: B. Main Results · p. 8

  • Takeaways & Limitations

    The framework supports compositional control while maintaining temporal coherence and physical consistency.

    Source: §V · p. 13

    Lighting is represented only by direction, limiting modeling of spatially varying illumination and high-dynamic-range effects.

    Source: LIMITATIONS · p. 17

Abstract

from arXiv · show

Controllable image-to-video (I2V) generation transforms a reference image into a coherent video guided by user-specified control signals. While precise control over camera motion, object motion, and lighting is essential for high-fidelity creation, existing methods often treat these factors independently. This overlooks the physical coupling among viewpoint, geometry, and illumination in dynamic scenes, leading to visual inconsistencies such as mismatched shadows and perspective drift under simultaneous changes. We present VidCRAFT3, a unified and flexible I2V framework that explicitly models cross-factor interactions among geometry, motion, and illumination, enabling both independent and joint control over camera motion, object motion, and lighting direction. Image2Cloud provides explicit 3D geometric priors for accurate camera motion control. ObjMotionNet encodes sparse object trajectories into multi-scale motion features to guide realistic object motion. A Spatial Triple-Attention Transformer integrates lighting direction through lighting cross-attention for consistent relighting. To address the scarcity of jointly annotated data, we construct the VideoLightingDirection (VLD) dataset with accurate per-frame lighting direction annotations, and introduce a three-stage progressive training strategy that enables robust learning without fully joint annotations. Extensive experiments demonstrate that VidCRAFT3 achieves state-of-the-art performance in control precision and visual coherence across diverse scenarios.

I. INTRODUCTION

Controllable I2V must coordinate camera motion, object motion, and lighting because viewpoint, geometry, and illumination jointly shape appearance. VidCRAFT3 combines these controls in one framework and adds data and training strategies for learning them.

I. INTRODUCTION

  • VidCRAFT3 jointly models camera motion, object motion, and lighting direction, addressing visual inconsistencies that arise when physically coupled factors are controlled independently.The introduction identifies unsynchronized shadows as one such inconsistency.Source: §I · p. 1
  • The framework combines point-cloud-based camera control, sparse-trajectory motion features, and lighting cross-attention, with progressive training and VLD lighting annotations supporting learning without fully joint annotations.VLD provides accurate per-frame lighting direction annotations.Source: §I · p. 2
  • The authors report state-of-the-art performance in control precision, visual quality, and generalization across diverse scenarios.Source: §I · p. 2

A. Image-to-video Generation

  • Prior I2V methods preserve reference-image fidelity during video generation but can struggle with fine-grained details because they rely on global conditions.Source: A. Image-to-video Generation · p. 3

B. Motion-controlled Video Generation

Prior work addresses camera, object motion, and lighting control through largely distinct approaches, with open-domain interactive lighting control remaining limited. VidCRAFT3 combines these controls in an I2V framework using geometry-aware renderings, trajectory features, and lighting-conditioned attention.

B. Motion-controlled Video Generation

  • Camera-control approaches use explicit pose conditioning or 3D lifting, while object-motion methods use flow, dense fields, region cues, keypoints, masks, or 3D cues.Directly mapping camera parameters to generative dynamics can restrict precision and generalization, particularly for trajectories outside the training distribution.Source: B. Motion-controlled Video Generation · p. 3
  • Prior joint motion-control methods address camera and object motion, while VidCRAFT3 is presented as the first framework to add simultaneous lighting-direction control.The paper combines point-cloud rendering, trajectory learning, and a Spatial Triple-Attention Transformer.Source: B. Motion-controlled Video Generation · p. 3

C. Lighting-controlled Visual Generation

A. Overview

  • VidCRAFT3 supports independent or combined controls by rendering a reconstructed point cloud along the camera path, encoding object trajectories, and integrating lighting with image and text through parallel cross-attention.The framework builds on DynamiCrafter and uses geometry-aware renderings to anchor camera motion while the video diffusion model refines appearance and temporal coherence.Source: A. Overview · p. 4, Figure 2 · p. 4

B. Preliminary

Video diffusion models generate videos by iteratively denoising noise, often in a lower-dimensional latent space. For camera control, VidCRAFT3 anchors generation with rendered views from a reconstructed point cloud, then uses the video diffusion model to refine them.

B. Preliminary

  • Video diffusion models learn to iteratively denoise noisy video latents, and latent diffusion reduces computation by performing this process in a lower-dimensional space.The final video is reconstructed from the latent representation.Source: B. Preliminary · pp. 4–5

1) Camera Motion Control via Point Cloud Rendering:

  • Image2Cloud reconstructs a 3D point cloud from the reference image and renders it along a user-defined camera trajectory to provide explicit geometric cues.The rendering provides structure, depth ordering, and visibility cues for camera control and interactions with object motion and lighting.Source: 1) Camera Motion Control via Point… · p. 5
  • VidCRAFT3 uses each per-frame point-cloud rendering as camera-conditioning input, while replacing the first rendering with the reference image for an exact first-frame match.This replacement reduces drift from point-cloud noise and camera-pose inaccuracies.Source: 1) Camera Motion Control via Point… · p. 5
  • The video diffusion model refines coarse point-cloud renderings to address missing regions, occlusions, and geometric distortions while producing temporally consistent frames.The renderings are encoded with the VAE and concatenated channel-wise with noise in latent space.Source: 1) Camera Motion Control via Point… · p. 5

2) Object Motion Control through Trajectory Learning:

VidCRAFT3 represents user-drawn object trajectories as smoothed motion features and integrates per-frame lighting directions with image and text conditions. These controls are designed to guide motion and lighting while retaining video quality and consistency.

2) Object Motion Control through Trajectory Learning:

  • ObjMotionNet extracts multi-scale features from Gaussian-smoothed optical-flow maps of sparse object trajectories and injects them into the UNet encoder.The encoder-only injection balances precise motion guidance with video quality, leaving the decoder to refine details.Source: 2) Object Motion Control through… · p. 5

3) Lighting Direction Control with Spatial Triple-Attention Transformer:

  • User-specified lighting directions are transformed from the reference camera coordinates into per-frame directions using camera extrinsics.The resulting sequence is encoded with degree-4 spherical harmonics as 16 coefficients per frame, then projected into the UNet feature space.Source: 3) Lighting Direction Control with… · p. 5
  • The Spatial Triple-Attention Transformer uses parallel image, text, and lighting cross-attention modules, summing their outputs into a fused feature.Lighting cross-attention modulates spatial features based on the input lighting direction.Source: 3) Lighting Direction Control with… · pp. 5–6

D. Dataset Construction

The constructed datasets use 25-frame video clips at a spatial resolution of 320 × 512.

D. Dataset Construction

1) Camera Motion Control Dataset:

The paper builds separate camera-, object-motion-, and lighting-control datasets, using reconstructed geometry, filtered trajectories, and synthetic scenes with per-frame lighting annotations.

1) Camera Motion Control Dataset:

  • RealEstate10K contributes 62,000 clips with smooth camera trajectories, geometry-aware point-cloud renderings, and captions generated from sampled frames.The first frame is the reference, and DUSt3R reconstructs the point cloud used to render along each clip’s ground-truth camera path.Source: 1) Camera Motion Control Dataset · p. 6

2) Object Motion Control Dataset:

  • WebVid-10M object-motion clips are filtered for motion and camera dominance, then tracked on a 16 × 16 grid and reduced to 1–8 sparse trajectories with smoothed optical flow.Clips are rejected as camera-dominated when at least 60% of tracks exceed 3% of the image diagonal.Source: 2) Object Motion Control Dataset · p. 6

3) Lighting Direction Control Dataset:

  • VLD contains 57,600 Blender-rendered videos from 3,600 static scenes, each rendered under 16 lighting directions along the same camera trajectory.Haven and BOP scenes provide complementary lighting setups, and each scene’s frames receive camera-pose and lighting-direction annotations.Source: 3) Lighting Direction Control Dataset · pp. 6–7
  • VLD pairs one reference frame with 16 target lighting directions, including 15 mismatched directions and one direction matching the reference lighting.The samples include HDR-environment Haven scenes and BOP models placed in textured rooms.Source: 3) Lighting Direction Control Dataset · p. 7

E. Training

Training progresses from camera control to mixed object-motion and lighting supervision, then shifts from dense to sparse trajectories; inference accepts controls independently or in combination.

E. Training

  • The diffusion objective conditions on image, text, camera, object, and lighting signals, with classifier-free dropout removing a random branch or all conditioning.When camera conditioning is dropped, the reference-image latent is repeated across frames; other dropped conditions become zero tensors.Source: E. Training · p. 7
  • Stage 1 fine-tunes the pretrained model on camera-control data for 40,000 iterations, optimizing the UNet while freezing the VAE and CLIP encoders.Point-cloud renderings provide 3D scene supervision for camera alignment and temporal consistency.Source: E. Training · pp. 7–8
  • Stages 2 and 3 each fine-tune for 20,000 iterations on mixed object-motion and VLD data, shifting from dense to sparse trajectories in Stage 3.Stage 2 freezes temporal UNet layers to retain camera control; Stage 3 uses sparse trajectories to simulate user inputs and encourage inference from limited motion data.Source: E. Training · p. 8
  • A three-stage progressive strategy trains the controls without a dataset containing all three annotations jointly, while strategically freezing layers to retain prior knowledge.The paper states that this yields fine-grained control across camera motion, object motion, and lighting direction without fully annotated multi-task data.Source: E. Training · pp. 7–8

F. Inference

  • At inference, users may supply camera trajectories, sparse object trajectories, and lighting direction individually or together, with missing controls replaced by null signals.Image2Cloud renders the camera path, ObjMotionNet encodes object tracks, and lighting embeddings are fused through the Spatial Triple-Attention Transformer before DDIM sampling.Source: F. Inference · p. 8

IV. EXPERIMENTS

Experiments evaluate VidCRAFT3 on domain-specific control datasets and a broader test set, reporting camera-control benchmark gains alongside strong controllability and visual consistency across factors.

1) Implementation Details:

  • The implementation uses 320 × 512 resolution and 25 frames, with Adam at 1×10−5 and a global batch size of 96 on eight H100 GPUs.Inference averages about 20 GB of GPU memory and 42 seconds per sample.Source: 1) Implementation Details · p. 8

2) Evaluation Datasets:

  • Evaluation uses 1,000 samples each from RealEstate10K, WebVid-10M, and VLD, plus a 100-video generalized set spanning varied content categories.The VLD test samples are stratified into 500 Haven and 500 BOP scenes.Source: 2) Evaluation Datasets · p. 8

3) Evaluation Metrics:

  • Video quality is measured with FID, FVD, and CLIPSIM; camera and object adherence use CamMC and ObjMC, while lighting comparisons use LPIPS, SSIM, and PSNR.Motion metrics use estimated camera poses and extracted object trajectories; lighting metrics compare generated frames with ground truth.Source: 3) Evaluation Metrics · p. 8

B. Main Results

  • VidCRAFT3 demonstrates controllability and visual fidelity across camera, object-motion, and lighting-direction tasks while preserving temporal and semantic consistency.Because no open-source I2V method jointly supports all three controls, the paper evaluates each task separately and presents qualitative joint-control results.Source: B. Main Results · p. 8

1) Camera Motion Control:

  • CamMC is 4.07 versus the best baseline’s 4.12, while FID 75.62, FVD 49.77, and CLIPSIM 32.32 are reported as best across camera-control metrics.The paper attributes the gains to Image2Cloud’s explicit 3D priors anchoring motion to scene geometry and reducing drift during large viewpoint changes.Source: Table I · p. 9, 1) Camera Motion Control · p. 9

2) Object Motion Control:

VidCRAFT3 controls object trajectories while preserving object structure, and its lighting-direction control produces coherent relighting across static and time-varying conditions.

2) Object Motion Control:

  • On WebVid-10M, VidCRAFT3 records an ObjMC score of 3.51, lower than all baselines, and outperforms them on the remaining metrics.The paper reports closer target-trajectory alignment and more realistic motion while preserving visual quality, temporal coherence, and semantic alignment.Source: 2) Object Motion Control · pp. 9–10

3) Lighting Direction Control:

  • For lighting control, VidCRAFT3 significantly outperforms DiffusionRenderer in PSNR, SSIM, LPIPS, and CLIPSIM, while DiffusionRenderer has lower FID and FVD.The paper attributes DiffusionRenderer’s FID and FVD advantage as likely related to its substantially larger training volume, including 150,000 auto-labeled real-world video segments.Source: 3) Lighting Direction Control · p. 10
  • Although training samples use static lighting directions, VidCRAFT3 responds coherently to time-varying lighting sequences at test time through frame-wise lighting conditioning.Qualitative comparisons show pronounced shading and cast shadows consistent with lighting changes, unlike the faint or static shadows often produced by DiffusionRenderer.Source: 3) Lighting Direction Control · p. 10, Figure 9 · p. 10

2) Object Motion Control:

  • Qualitative examples vary camera motion, object motion, or lighting independently, and also show simultaneous camera- and object-motion controls against ground truth.The single-factor examples report faithful control following while preserving content fidelity.Source: Figure 10 · p. 11, Figure 11 · p. 11

3) Lighting Direction Control:

  • Lighting results are comparable across Haven and BOP scenes, with slightly better results on BOP and a small gap between scene types.The paper interprets this pattern as evidence of scene-agnostic domain generalization.Source: 3) Lighting Direction Control · p. 11

4) Joint Control:

Joint control composes camera, object, and lighting signals while maintaining coherence across viewpoints. A user study and training ablation provide further evidence on control performance and the role of the camera prior.

4) Joint Control:

  • Camera-plus-lighting control maintains consistent shading and shadows across viewpoints, while all three controls preserve trajectory alignment and stable scene layout.The joint examples also show coherent lighting changes across viewpoint transitions and no interference among controls.Source: 4) Joint Control · p. 11, Figure 12 · p. 11, Figure 13 · p. 11

C. User Study

  • 69.3% camera-motion and 60.6% object-motion preference rates were reported in a user study of 200 results assessed by 10 participants.The study sampled camera and object motion results from copyright-free videos and named test splits, comparing VidCRAFT3 with 10 baselines.Source: C. User Study · p. 12, Table V · p. 12

1) Training Strategy:

  • The full three-stage schedule improves object adherence, relighting, and camera adherence from Stage 2 to Stage 3; removing Stage 1 sharply reduces CamMC.The No-S1 variant has similar object and relighting scores at Stage 3, indicating the importance of the Stage 1 camera and geometry prior.Source: 1) Training Strategy · p. 12, Table VI · p. 12

2) Object Trajectory Sampling:

The ablations compare trajectory sampling and lighting representations, while the conclusion frames VidCRAFT3 as a compositional framework for consistent control across camera, object motion, and lighting.

2) Object Trajectory Sampling:

  • Dense→Sparse trajectory sampling combines dense-motion learning with sparse-trajectory fine-tuning, strengthening object adherence and overall video quality on WebVid-10M.Dense trajectories provide richer motion but mismatch sparse inference inputs; sparse-only trajectories match inference but underrepresent fine motions and local deformations.Source: 2) Object Trajectory Sampling · p. 12

3) Lighting Embedding Integration Strategies:

  • Lighting Cross-Attn outperforms text-cross-attention and time-embedding integration across all reported metrics, supporting a dedicated lighting-attention branch.The proposed branch is inside the Spatial Triple-Attention Transformer.Source: 3) Lighting Embedding Integration… · p. 12

4) Representation of Lighting Direction:

  • SH Encoding outperforms Fourier Embedding across all VLD metrics, with gains attributed to its smooth, rotation-aware angular basis.The lighting embeddings are fed to Lighting Cross-Attn.Source: 4) Representation of Lighting Direction · p. 13

V. CONCLUSION

  • VidCRAFT3 supports simultaneous, consistent control of camera motion, object motion, and lighting direction while maintaining temporal coherence and physical consistency.The conclusion also identifies VLD as addressing the lack of lighting-aware supervision in existing video datasets.Source: §V · p. 13

APPENDIX QUALITATIVE RESULTS OF ABLATION STUDY

The qualitative ablations contrast trajectory-sampling and lighting-design choices, while additional examples show control across camera and object motion. The authors identify data and modeling constraints that bound the current method.

APPENDIX QUALITATIVE RESULTS OF ABLATION STUDY

  • Dense→Sparse combines dense motion detail with stronger alignment; Lighting Cross-Attn yields more accurate lighting control; SH Encoding produces more realistic lighting effects than Fourier Embedding.The comparisons report better ground-truth alignment for Dense→Sparse, Lighting Cross-Attn, and SH Encoding in their respective ablations.Source: APPENDIX QUALITATIVE RESULTS OF… · p. 17, Figure 14 · p. 18, Figure 15 · p. 19

ADDITIONAL QUALITATIVE RESULTS

LIMITATIONS

  • The method remains challenged by complex physical interactions, large nonrigid motion, and drastic lighting changes; limited joint annotations and direction-only lighting constrain its modeling scope.The authors identify spatially varying illumination and HDR effects as outside the current direction-only lighting representation, and suggest richer representations, improved annotations, and more physically grounded architectures.Source: LIMITATIONS · p. 17
Loading 2502.07531v5…