CamI2V: Camera-Controlled Image-to-Video Diffusion Model

1 INTRODUCTION

• Camera poses offer a physics-informed, user-friendly condition, but prior side-input methods neglect their physical knowledge, leading to imprecise control, inconsistencies, and poor interpretability.The section identifies MotionCtrl and CameraCtrl as examples of such methods.Source: §1 · p. 2
• Epipolar attention restricts feature aggregation to corresponding lines, while Plücker coordinates provide absolute 3D ray embeddings for implicit spatial learning.The epipolar constraint narrows the search space to more relevant, less noisy information.Source: §1 · p. 2
• Register tokens address cases where frames have no epipolar intersections, including those caused by rapid camera movement, dynamic objects, or occlusions.The paper identifies these cases as a reason to augment epipolar attention.Source: §1 · p. 3
• Inference uses separate classifier-free guidance scales for image, text, and camera conditions, which can be combined into one forward pass if needed.The scales can be absorbed into the model input, similarly to timestep conditioning.Source: §1 · p. 3
• The central challenge is modeling noisy cross-frame interactions to improve geometric consistency and camera controllability.Temporal attention only relates features at the same image location, while broader 3D attention accesses more noisy conditions that may mislead the model.Source: §1 · pp. 2–3
• 32.96% on RotErr, 25.64% on CamMC, and 20.77% on TransErr improvements over CameraCtrl on RealEstate10K, without compromising dynamics, generation quality, or out-of-domain generalization.The authors also establish a more robust, accurate, and reproducible camera-controllability evaluation pipeline.Source: §1 · p. 3

Diffusion-based Video Generation

• Video-generation research adapted text-to-image models with temporal blocks and fine-tuning, while Transformer diffusion backbones and 3D-VAE technology advanced video generation.The section also surveys image-to-video methods including Dynamicrafter, SVD, Seine, I2vgen-XL, and PIA.Source: §2 · p. 3

Camera-controlled Video Generation

• Figure 3 contrasts camera representations and trajectories in world coordinates with their conversion to pixel-conditioned Plücker ray representations.The figure presents the camera-to-ray parameterization.Source: Figure 3 · p. 4

Controllable Generation

• Controllable video generation has developed across pose, trajectory, subject, and audio control.The section situates these developments within broader image-controllable generation research.Source: §2 · p. 4

Camera-controlled Video Generation

• Camera-control methods include coarse movement generation, pixel-level motion guidance, and camera-pose side inputs, while CamCo applies epipolar attention only to a single reference frame.The paper notes that CamCo does not account for frames lacking overlap with that reference.Source: §2 · p. 4

3.1 PRELIMINARIES

• Camera poses are represented as Plücker ray embeddings for global positional conditioning, while diffusion denoises latent video codes conditioned on text and a reference image.Each ray is associated with a pixel coordinate; the latent code represents video data.Source: §3.1 · pp. 4–5

3.2 OVERALL PIPELINE

• The pipeline extends a camera-conditioned image-to-video model with epipolar attention and uses multiple classifier-free guidance to balance visual quality and camera-pose consistency.The epipolar-constrained module extends temporal attention.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 EPIPOLAR ATTENTION FOR NOISED FEATURES TRACKING

• Epipolar attention limits cross-frame matching to features near corresponding epipolar lines, reducing the search from hw candidates to l, where l ≪ hw.A distance threshold δ defines the mask; the paper selects half the feature-grid diagonal empirically.Source: §3.3 · pp. 5–6
• Register tokens address zero epipolar masks when no corresponding lines fall within any frame, including cases involving rapid camera movement, dynamic objects, or occlusions.The tokens are added to the key/value sequence and are learnable.Source: Figure 6 · p. 6, §3.3 · p. 7

3.4 MULTIPLE CLASSIFIER-FREE GUIDANCE

• Two guidance scales let users trade off image-text and camera control; distilling them can avoid the extra inference time of three forward passes.The scales are applied to text-conditioned image animation.Source: §3.4 · p. 7

• SfM-based evaluation can produce unstable or inaccurate camera estimates on generated videos because of keypoint mismatches, low resolution, dynamic scenes, lighting variation, and distortion.Source: §4 · p. 7
• The pipeline adapts GLOMAP and uses shared ground-truth camera intrinsics so global mapping primarily optimizes camera extrinsics.Source: §4 · p. 7
• Estimated camera-to-world matrices are canonicalized relative to the first frame and scene scale, with metrics averaged across successful trials and samples.Five trials are run for each of 1,000 sampled videos.Source: §4 · p. 7
• RotError sums per-frame rotation-angle errors across 16 frames, while TransError sums translation distances and CamMC combines rotation and translation similarity across frames.Source: §4 · pp. 7–8
• 32.96%, 25.64%, and 20.77% improvement over CameraCtrl on RotErr, CamMC, and TransErr, respectively, on RealEstate10K.Source: Table 1 · p. 8
• FVD measures the distance between generated frames and the training distribution to assess video quality.Source: §4 · p. 8

5.1 SETUP

• Training uses approximately 70K RealEstate10K clips, samples 16 frames, and trains a frozen DynamiCrafter-based model at 256×256 resolution with two register tokens.Inference uses a fixed frame stride of 8; training uses random strides from 1 to 10.Source: §5.1 · p. 8

5.2 QUANTITATIVE COMPARISON

• CamI2V significantly improves camera controllability and visual quality over compared methods, including a 0.2306 RotErr reduction versus CameraCtrl, equivalent to 13.21° lower rotational error.Source: §5.2 · pp. 8–9

5.3 ABLATION STUDY

• The ablation compares progressively larger noisy-context access: baseline spatial-temporal attention, reference-frame-only epipolar attention, all-frame epipolar attention, and 3D full attention.Source: §5.3 · p. 9
• CamI2V performs best among the tested setups; 3D full attention, despite accessing the most noisy conditions, is only slightly better than CameraCtrl and trails reference-frame-only attention.The authors attribute the pattern to an optimal amount of noisy conditions rather than simply increasing their quantity.Source: §5.3 · p. 9

5.4 QUALITATIVE COMPARISON

• Figure 7 compares generated videos with ground truth; CamI2V's camera trajectory aligns more closely and some rendered details appear more realistic.Source: §5.4 · p. 10, Figure 7 · p. 10
• Figure 8 shows CamI2V applied to out-of-domain content, including oil paintings, photography, and animation.Source: §5.4 · p. 10, Figure 8 · p. 10

6 CONCLUSION

• The framework combines Plücker coordinates as 3D ray embeddings with epipolar attention that aggregates features along epipolar lines.The conclusion describes this attention as supporting robust tracking under high noise conditions.Source: §6 · p. 10
• Register tokens address frames without intersections, including cases involving rapid camera movements or occlusions.Source: §6 · p. 10
• The method achieves state-of-the-art performance on RealEstate10K and out-of-domain datasets.Source: §6 · p. 10
• The conclusion identifies high-resolution generation, complex camera trajectories, and generation quality in long videos as remaining challenges.Future work is directed toward these aspects.Source: §6 · p. 10

A CORE CODES

• The spatial attention block applies residual self-attention, cross-attention conditioned on c, and a feed-forward network to feature x.Source: A CORE CODES · p. 16
• The temporal camera-control block adds Plücker ray embeddings to x and applies epipolar attention using mask m, followed by two self-attention layers and a feed-forward network.Source: A CORE CODES · p. 16
• The epipolar-mask algorithm takes camera intrinsics and extrinsics, feature size, and threshold δ as inputs.Source: A CORE CODES · p. 16
• The mask procedure computes epipolar distances for feature coordinates and returns a mask that includes a register token.The algorithm first forms essential and fundamental matrices from the camera parameters.Source: A CORE CODES · p. 16

B COLMAP & GLOMAP CONFIGURATION

• All clips use a simple pinhole camera model, with the same intrinsics assumed across the 16 frames of each video.Source: B COLMAP & GLOMAP CONFIGURATION · p. 16
• SIFT extraction enables affine-shape estimation and domain-size pooling, while camera intrinsics are fixed through the ImageReader camera parameters.Source: B COLMAP & GLOMAP CONFIGURATION · p. 16
• Guided matching is enabled with a maximum of 65536 matches to allow more underlying matches.Source: B COLMAP & GLOMAP CONFIGURATION · p. 16
• Global mapping disables intrinsic optimization in bundle adjustment and relaxes the geometric constraint by setting maximum relative-pose epipolar error to 4.Source: B COLMAP & GLOMAP CONFIGURATION · p. 16

C GPU MEMORY AND SPEED

• Table 3 compares GPU memory usage and speed under DeepSpeed ZeRO-1.Source: Table 3 · p. 16
• The reported DynamiCrafter comparison is a reproduction, and the caption specifies full-parameter fine-tuning results.Source: Table 3 · p. 16
• The model can be trained on 24GB consumer-level GPUs despite the additional epipolar attention.Source: Table 3 · p. 16

D EXTRA OUT-OF-DOMAIN VISUALIZATIONS

• The authors recommend viewing the supplementary visualizations for a more comprehensive evaluation.Source: D EXTRA OUT-OF-DOMAIN VISUALIZATIONS · p. 17
• Figure 9 visualizes the 256×256 model.Source: Figure 9 · p. 17
• Figure 10 shows original outputs from the 512×320 model with no padding removed.Source: Figure 10 · p. 17
• The 512×320 model generates outputs compatible with input images of arbitrary aspect ratio.Figure 11 shows generation by the 512×320 model.Source: Figure 11 · p. 17