PaperlayerView exact revision on arXiv

Source-linked AI summary

RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control

Teng Li, Guangcong Zheng, Rui Jiang, Shuigen Zhan, Tao Wu, Yehao Lu, Yining Lin, Chuanyun Deng, Yepan Xiong, Min Chen, Lin Cheng, Xi Li

arXiv:2502.10059v2cs.CV

TL;DR

Trajectory-guided image-to-video methods improve camera control but remain difficult to use on arbitrary real-world images lacking known depth and scene scale. RealCam-I2V reconstructs metric-scale 3D scenes for training and interactive inference, then uses scene-constrained noise shaping to guide generation. The method improves controllability and video quality on RealEstate10K and out-of-domain images, while supporting looping and interpolation applications.

  • Problem

    Trajectory-guided camera control is more precise than text-based control, but real-world use is limited by unknown depth, scene scale, and difficult trajectory specification.

  • Method

    RealCam-I2V uses monocular metric depth reconstruction, metric-scale camera training, interactive 3D trajectory drawing, and scene-constrained noise shaping.

  • Results

    RealCam-I2V improves video quality and camera controllability on RealEstate10K and out-of-domain images, with over 30% gains on metric-scale results.

  • Takeaways & Limitations

    Metric-scale reconstruction and scene-constrained noise shaping support precise camera control and scene consistency while enabling camera-controlled looping and generative frame interpolation.

  • Takeaways & Limitations

    Real-world application remains constrained by limited data and the lack of a suitable dataset combining dynamic scenes with camera movement.

Abstract

from arXiv · show

Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation. Project page: https://zgctroy.github.io/RealCam-I2V.

1. Introduction

RealCam-I2V addresses the scale and usability limitations of trajectory-based camera control for real-world image-to-video generation. It combines metric-scale 3D reconstruction, interactive trajectory design, and scene-constrained noise shaping to improve controllability and scene consistency.

  • Text-based camera control is intuitive but lacks precise control over camera angle, scale, and movement direction.
  • Relative-scale trajectories create inconsistencies in real-world scenes and make precise trajectory drawing difficult without depth information.
  • RealCam-I2V uses monocular metric depth estimation to reconstruct a stable 3D scene for consistent real-world camera control.The reconstructed scene provides a metric scale for both training and inference.
  • Training aligns COLMAP camera parameters with metric depth, enabling metric-scale trajectories and greater scene consistency across real-world images.
  • Users draw camera trajectories interactively in the reconstructed 3D scene and receive preview videos for real-time feedback.
  • Scene-constrained noise shaping controls high-noise generation stages while preserving dynamic content generation during later denoising.Preview videos provide scene-visible regions whose controlled noise guides initial layout and camera dynamics.
  • RealCam-I2V improves video quality and controllability, including on RealEstate10K and out-of-domain images.

2. Related Works

Related work spans diffusion-based video generation, controllable generation, and camera-control methods. RealCam-I2V is positioned within trajectory-based approaches while using metric-scale reconstruction for interactive control.

  • Diffusion-based Video Generation: Diffusion-based video generation developed from text-to-video systems into image-to-video models using images as control signals.
  • RealCam-I2V Pipeline: RealCam-I2V aligns training cameras to metric scale and lets users draw trajectories in a metric-scale point cloud during inference.
  • Controllable Generation: Controllable generation supports user direction through inputs such as text, pose, audio, identity, and trajectories.
  • Text-based Camera Control: Text-based camera-control methods guide motion with natural-language descriptions but remain limited in precise camera control.
  • Trajectory-based Camera Control: Trajectory-based methods use camera poses and geometric constraints to improve camera control and viewpoint consistency.

3. Method

RealCam-I2V constructs metric-scale 3D scenes from monocular depth, aligns training camera trajectories to that scale, and enables interactive trajectory design with scene-constrained noise shaping. The framework also supports task-specific video concatenation modes and reports improvements over baselines in quantitative comparisons.

  • Metric Depth Estimation: Metric depth estimation converts an input RGB image into a depth map for constructing a 3D point cloud in camera coordinates.The depth map is produced by a metric depth predictor and projected into 3D for interaction.
  • Metric Scene-scale Alignment: Relative-scale camera transformations are aligned with metric and SfM point clouds to rescale translations and produce metric-scale camera-to-world transformations.The alignment yields a scale factor applied to the relative translation component.
  • Interactive Camera Control: Users draw camera trajectories directly in the reconstructed 3D scene and preview the resulting camera motion without costly video diffusion generation.The interactive preview separates trajectory adjustment from slow video generation and provides visual feedback.
  • Scene-constrained Noise Shaping: Scene-constrained noise shaping overlays selected preview-video pixels onto predicted clean latent features during high-noise stages to guide layout and camera dynamics.Visible pixels are selected while neighborhood filtering excludes pixels near potentially inaccurate depth edges.
  • Task-specific Generation: Interpolation, looping, and continuation are supported through different input concatenation modes with minor modifications.The training setup selects clean condition frames differently for interpolation and continuation modes.
  • Quantitative Comparison: Over 30% improvement on metric scale results and over 10% improvement on FVD are reported against baseline methods.The quantitative comparison states that the approach excels all baselines on relative and metric results while improving visual quality.

4. Experiments

RealCam-I2V is evaluated for camera control, video quality, generalization, and applications. Metric-scale alignment and scene-constrained noise shaping improve performance, while the method supports diverse scenes and video-generation tasks.

  • Experimental Setup: The model is trained on RealEstate10K, using 70,000 clips with relative-scale camera poses and metric-depth alignment from reconstructed point clouds.Depth Anything V2 Large Indoor provides metric depth for estimating relative-to-metric scaling factors.
  • Metrics: Camera controllability is evaluated with RotErr, TransErr, and CamMC after canonicalizing poses relative to the first frame.Evaluation uses 1,000 test samples and five trials per clip to reduce SfM-related randomness.
  • Applications: Static-scene training preserves Dynamic Degree on Vbench-I2V, although dynamic datasets could further enhance dynamics.The reported application results also cover diverse styles and domains.
  • Comparison with SOTA Methods: RealCam-I2V improves FVD and camera-control metrics, with over 30% gains on metric-scale results.Comparisons include DynamiCrafter, MotionCtrl, CameraCtrl, and CamI2V, using DynamiCrafter as the base model.
  • Ablation Study: Metric Scene-scale Alignment reduces scale ambiguity, while combining it with Scene-constrained Noise Shaping yields substantial gains in video quality and controllability.Noise shaping alone provides nearly 50% reduction on DynamiCrafter but performs worse than the combined approach and can suppress dynamics when extended to mid-noise stages.
  • Applications: The method generalizes from RealEstate10K to out-of-domain images and supports camera-controlled looping, frame interpolation, and smooth scene transitions.Visualizations include complex trajectories, large movements, and dynamic video generation.

5. Limitation Analysis and Future Work

RealCam-I2V generalizes beyond static RealEstate10K scenes, but its real-world applicability remains limited by available data. The authors identify the absence of a suitable camera-controlled video dataset as a central boundary.

  • Limitation Analysis and Future Work: The current model remains limited by data because no suitable dataset combines dynamic scenes with camera movement for camera-controlled video generation.The limitation concerns real-world applications despite generalization to dynamic scenes, styles, and domains.

6. Conclusion

RealCam-I2V addresses scale inconsistencies and usability challenges in trajectory-based camera-controlled image-to-video generation. It combines metric-scale 3D reconstruction, interactive trajectory design, and scene-constrained noise shaping to improve video quality, control precision, and scene consistency.

  • 6. Conclusion: RealCam-I2V introduces monocular 3D reconstruction as a reliable reference for training and inference.The reconstruction enables absolute-scale training and supports interactive camera trajectory design with preview feedback.
  • 6. Conclusion: Scene-constrained noise shaping enhances scene consistency and camera controllability.
  • 6. Conclusion: RealCam-I2V achieves substantial improvements on RealEstate10K in video quality and control precision.The paper describes this as establishing a new state of the art on the dataset.

A. Dataset

RealEstate10K camera poses have arbitrary per-clip scale because global scene scale is unavailable. This creates training–inference scale inconsistencies for general image and trajectory combinations, which metric-depth reconstruction is designed to bridge.

  • A. Dataset: RealEstate10K camera poses are reconstructed without global scene scale, producing an arbitrary scale for each clip.The trajectory is refined using SLAM and structure-from-motion processing.
  • A. Dataset: Scale inconsistencies emerge when out-of-domain images are paired with in-domain or out-of-domain trajectories.The passage states that these inconsistencies make realistic and controllable video generation impossible.
  • A. Dataset: Metric-depth prediction reconstructs an absolute-scale scene that bridges training and inference.This alignment supports diverse combinations of images and trajectories across scenarios.

B. Training

Training uses multiple DynamiCrafter variants and fixed comparison settings, while metric-depth preprocessing supplies the scene scale used by RealCam-I2V. The implementation also includes outlier filtering for noise shaping.

  • B. Training: RealCam-I2V is trained on four publicly accessible DynamiCrafter variants: 256, 512, 512 interp, and 1024.A 256×256 ablation uses ϵ-prediction, effective batch size 64, and 50,000 training steps.
  • B. Training: The 256×256 models are trained for 50,000 steps with effective batch size 64.This training takes about 50 hours and uses ϵ-prediction.
  • B. Training: Outlier filtering prevents undesired pixels from being mistakenly pasted when the filtering kernel size satisfies k ≥3.
  • B. Training: Results for MotionCtrl and CameraCtrl are reproduced on DynamiCrafter for fair comparison.CamI2V is implemented with hard mask epipolar attention and two register tokens; comparison uses fixed text image CFG 7.5 and camera CFG 1.0.
  • B. Training: Depth Anything V2 Large, an indoor metric-depth model with 335.3M parameters, provides the metric depth predictor.Its indoor model supports monocular metric depth estimation up to 20m.

D. Noise Shaping and Parameter Sensitivity

Noise shaping overlays reference-video features during early diffusion steps, trading imposed layout prior against retained dynamics. Its threshold controls when shaping applies, and experiments report consistency and quality improvements.

  • D. Noise Shaping and Parameter Sensitivity: Noise shaping overlays reference video features on model predictions during early diffusion steps.Fewer shaping applications preserve more dynamics, whereas more applications impose a stronger layout prior.
  • D. Noise Shaping and Parameter Sensitivity: Effective control is typically achieved with a noise-shaping threshold of tNS = 900.Users can switch among tNS values of 800, 900, and 1000 in practice.
  • D. Noise Shaping and Parameter Sensitivity: Noise shaping is applied only when the noise level satisfies t > tNS.The threshold governs application during the early denoising process for t ∈[0, 1000].
  • D. Noise Shaping and Parameter Sensitivity: Noise shaping significantly improves consistency and quality in VBench-I2V results with dynamic scenes and large camera movements.The evaluated RealCam-I2V model is trained on RealCam-Vid with camera traces approaching 360◦.

E. Robust Analysis of Depth Estimation, SfM, Base Model and Resolution

RealCam-I2V remains effective across different depth predictors, structure-from-motion tools, diffusion backbones, datasets, and resolutions. Its metric-space alignment also tolerates depth-estimation errors, while better predictors can further improve performance.

  • The framework remains robust across variations in depth predictors, SfM methods, base models, and resolution.This robustness is reported across the evaluated system components and settings.
  • Metric-space alignment propagates depth-prediction errors consistently across training and inference, avoiding a train–inference gap.The alignment establishes a unified metric space even when predicted absolute depths are imperfect.
  • RealCam-I2V adapts to high-resolution dynamic-scene video and multiple model families, including UNet- and DiT-based backbones.Experiments include RealCam-Vid, DynamiCrafter, and CogVideoX 1.5.

F. Real-time User Interface

RealCam-I2V provides an interactive camera-path design workflow centered on dragging within a reconstructed 3D scene. Parallel rendering and trajectory interpolation support immediate previews and smooth, coherent motion.

  • Users design camera paths by dragging within the reconstructed 3D scene rather than entering 2D or numerical inputs.The interface provides real-time preview feedback, with optional streamed visualization.
  • Preview frames are rendered independently in parallel, enabling streamed playback and nearly real-time interaction.The implementation uses modern rendering engines such as Open3D and supports previews with sampled keyframes.
  • Linear interpolation in SE(3) expands sparse user keyframes into denser trajectories for smooth and continuous camera motion.For example, 4 keyframes can be expanded to 16 interpolated frames while adhering to the user’s input.
Loading 2502.10059v2…