Source-linked AI summary
Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation
Min-Jung Kim, Jeongho Kim, Hoiyeong Jin, Junha Hyung, Jaegul Choo
TL;DR
Camera-controlled novel-view video generation aims to make post-production viewpoint changes possible, but existing methods are limited by depth-reprojection errors and trajectory-data biases. InfCam combines infinite-homography rotation conditioning with end-to-end residual-parallax learning and data augmentation. It is reported to outperform baselines in camera-pose accuracy and visual fidelity across synthetic and real-world settings.
Problem
Camera-controlled novel-view generation seeks fidelity to specified camera poses, while existing depth-reprojection methods face inaccurate-depth errors and trajectory-conditioned methods can inherit training-data biases.
Source: §1 · p. 2Method
Results
InfCam consistently outperforms depth-reprojection and trajectory-conditioned approaches across all metrics, with reported gains in camera-pose accuracy and visual fidelity on synthetic and real-world videos.
Source: §1 · p. 2, §6 · p. 8, §5.3 · p. 8Takeaways & Limitations
The reported approach performs across diverse camera motions and focal lengths, with generalization demonstrated from synthetic to real-world videos.
Source: §1 · p. 2, §6 · p. 8The motivation assumes that source and target intrinsics, rotation, and translation are known for conditioning.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Recent progress in video diffusion models has spurred growing interest in camera-controlled novel-view video generation for dynamic scenes, aiming to provide creators with cinematic camera control capabilities in post-production. A key challenge in camera-controlled video generation is ensuring fidelity to the specified camera pose, while maintaining view consistency and reasoning about occluded geometry from limited observations. To address this, existing methods either train trajectory-conditioned video generation model on trajectory-video pair dataset, or estimate depth from the input video to reproject it along a target trajectory and generate the unprojected regions. Nevertheless, existing methods struggle to generate camera-pose-faithful, high-quality videos for two main reasons: (1) reprojection-based approaches are highly susceptible to errors caused by inaccurate depth estimation; and (2) the limited diversity of camera trajectories in existing datasets restricts learned models. To address these limitations, we present InfCam, a depth-free, camera-controlled video-to-video generation framework with high pose fidelity. The framework integrates two key components: (1) infinite homography warping, which encodes 3D camera rotations directly within the 2D latent space of a video diffusion model. Conditioning on this noise-free rotational information, the residual parallax term is predicted through end-to-end training to achieve high camera-pose fidelity; and (2) a data augmentation pipeline that transforms existing synthetic multiview datasets into sequences with diverse trajectories and focal lengths. Experimental results demonstrate that InfCam outperforms baseline methods in camera-pose accuracy and visual fidelity, generalizing well from synthetic to real-world data. Link to our project page:https://emjay73.github.io/InfCam/
1. Introduction
Camera-controlled video generation could enable post-production viewpoint changes, but existing approaches face depth-reprojection errors or dataset-specific trajectory biases. InfCam combines infinite-homography conditioning with data augmentation to improve trajectory fidelity across synthetic and real-world videos.
1. Introduction
- Post-production viewpoint changes can avoid costly reshoots and enable creative visual effects.Source: §1 · p. 2
- Depth-reprojection methods are vulnerable to errors from inaccurate depth, while trajectory-conditioned methods can inherit biases from their training pairs.These limitations can degrade performance and generalization, respectively.Source: §1 · p. 2
- InfCam combines infinite-homography conditioning with trajectory and focal-length augmentation to improve camera-controlled video generation.The framework conditions on noise-free rotation and learns residual parallax end-to-end.Source: §1 · p. 2
- InfCam is reported to achieve state-of-the-art performance on synthetic and real-world videos.The paper attributes improved trajectory fidelity to infinite homography warping and data augmentation.Source: §1 · p. 2
2. Related Work
Camera-controlled novel-view video methods either reproject depth along a target trajectory or learn trajectory-to-video mappings from paired data. InfCam's infinite-homography approach conditions on rotation while learning residual parallax, addressing a distinct weakness of depth-based reprojection.
2. Related Work
- Depth-reprojection methods estimate per-frame depth, reproject scene structure along a target trajectory, then predict or inpaint newly exposed regions.Their performance remains limited by depth accuracy and resulting reprojection errors.Source: §2 · p. 2
- InfCam conditions on images warped by H∞ so the model learns parallax confined between the epipole and the point at infinity.The figure contrasts this spatially constrained learning with depth-reprojection conditions that can remain inaccurate during training.Source: Figure 2 · p. 3
- Trajectory-conditioned methods learn from paired trajectories and videos, but can inherit biases in the training-data distribution.SynCamMaster targets stationary-camera views, while ReCamMaster uses MultiCamVideo.Source: §2 · pp. 2–3
3. Motivation
Depth-based reprojection can supply unreliable conditions when depth estimates are inaccurate, and fixed reprojection errors are difficult to correct during training. Infinite homography separates rotation from residual parallax, constraining the model's search while allowing end-to-end refinement.
3. Motivation
- Inaccurate predicted depth can create unreliable reprojection conditions and image artifacts, while fixed reprojection positions cannot be corrected through gradients to the depth estimator.The model must compensate for noisy conditions whose corrupted regions and error magnitudes are unclear.Source: §3 · p. 3
- H∞ is independent of translation and depth, and corresponds to image-point alignment under pure camera rotation.The translation-and-depth term represents parallax relative to the plane at infinity, spatially constrained between the epipole and x∞.Source: §3 · p. 3
- With known source and target intrinsics, rotation, and translation, conditioning on an H∞-warped image provides a noise-free rotational reference.The model learns residual information, including Kt t and implicitly predicted depth, while latent warping is used for computational efficiency.Source: §3 · pp. 3–4
- End-to-end training lets the network implicitly refine predicted depth as it learns parallax within the constrained region.The spatial boundary narrows the search space and helps improve camera-pose fidelity.Source: §3 · pp. 3–4
4. Method
InfCam generates novel views from a source video, target camera trajectory, and target intrinsics. Its method combines a homography-guided video model with trajectory, focal-length, and video-pair augmentation.
4. Method
- InfCam takes a source video, source and target intrinsics, and a target camera trajectory relative to the source's initial pose to generate a trajectory-faithful novel-view video.Source: §4 · p. 4
4.1. Model Architecture
- The model freezes pretrained Wan2.1 weights and trains newly introduced camera-encoder and homography-guided attention parameters.The warping module applies infinite homography to the initial source latent for rotation, then adds camera embeddings for translation and intrinsics; attention processes source, target, and warped latents per frame before passing only source and target latents onward.Source: §4.1 · pp. 4–5
4.2. Data Preparation
- AugMCV augments MultiCamVideo, whose synchronized views are shown alongside stationary-camera SynCamVideo and dynamic-camera MultiCamVideo.AugMCV varies starting poses and focal lengths.Source: §4.2 · p. 5, Figure 4 · p. 5
4.2.1. Trajectory Augmentation
- Trajectory augmentation reverses one video from a scene and concatenates it with another, producing 161-frame sequences with varied initial frames while preserving temporal alignment.This addresses the shared-start-frame bias in MultiCamVideo training pairs.Source: §4.2.1 · p. 5
4.2.2. Intrinsic Augmentation
- Intrinsic augmentation resizes and center-crops the input video to apply a randomly sampled higher focal length from 18mm, 24mm, 35mm, or 50mm.The strategy addresses training pairs sampled from scenes with fixed focal length.Source: §4.2.2 · pp. 5–6
4.2.3. Video Pair Selection
- Video-pair selection samples two synchronized cameras from one scene and applies focal-length augmentation to each video with probability 0.5.Augmented videos may serve as source or target, covering focal-length increase and decrease scenarios.Source: §4.2.3 · p. 6
5. Experiments
Experiments compare InfCam with camera-controlled baselines on AugMCV and WebVid using qualitative, quantitative, and component-ablation evaluations. The reported results cover matched and different camera intrinsics, real-world videos, and the effects of trajectory/intrinsic augmentation and warping.
5.1. Experiment Settings
- Evaluation uses 1,680 AugMCV test videos with ground truth and 2,000 WebVid generations; WebVid lacks target-view ground truth and uses perceptual and estimated-pose metrics.The experiments compare against ReCamMaster, TrajectoryCrafter, and GEN3C; models use pretrained Wan2.1 and training on AugMCV’s 47,432 scenes.Source: §5.1 · pp. 6–7
5.3. Quantitative Results
- InfCam outperforms the baselines across all metrics with shared and different intrinsics on AugMCV.Table 1 reports the quantitative comparison for both scenarios.Source: Table 1 · p. 7, §5.3 · p. 8
- InfCam outperforms the WebVid baselines in camera pose accuracy and visual fidelity, with a significant pose-accuracy improvement.Table 2 reports the WebVid comparison.Source: Table 2 · p. 7, §5.3 · p. 8
5.2. Qualitative Results
- Qualitatively, InfCam maintains viewpoint alignment and visual coherence, while ReCamMaster preserves the source’s initial frame and reprojection baselines show alignment or appearance problems.The ReCamMaster interpolation strategy permits initial-frame viewpoint changes but still leaves alignment errors.Source: §5.2 · p. 7
5.4. Ablation Study
- The ablation progresses from a baseline that captures neither trajectory nor intrinsics, through augmentation stages, to warping that substantially improves ground-truth alignment.Trajectory augmentation improves motion consistency but does not resolve target-intrinsic differences; adding intrinsic augmentation helps, while the warping module is crucial for accurate alignment.Source: §5.4 · p. 8, Table 3 · p. 8, Figure 6 · p. 8
- Training with AugMCV outperforms joint training on MCV and SCV, supporting the effectiveness of trajectory-intrinsic augmentation.The comparison is reported in Table 4.Source: §5.4 · p. 8, Table 4 · p. 8
6. Conclusion
InfCam uses infinite homography warping to encode camera rotations in latent space while learning residual parallax, avoiding external depth estimation. The conclusion reports superior performance across diverse camera motions and focal lengths, while identifying longer sequences as future work.
6. Conclusion
- InfCam encodes camera rotations in latent space and learns residual parallax, removing dependence on external depth estimation and erroneous reprojection results.The framework combines infinite homography warping with trajectory-intrinsic augmentation.Source: §6 · p. 8
- InfCam consistently outperforms depth reprojection-based and trajectory-conditioned approaches across all metrics.The conclusion reports superior performance across diverse camera motions and focal lengths.Source: §6 · p. 8
- The authors identify extending InfCam to longer video sequences as future work.Source: §6 · p. 8
Supplementary Material
Figure 7 presents the model overview and specifies the dimensions of its latent features.
Supplementary Material
- Figure 7 labels batch size, latent frame count, latent width and height, and feature dimension using the einops convention.Source: Figure 7 · p. 11
7. Model Architecture Details
The architecture uses a homography-guided attention layer to process source, target, and warped latents, then passes only source and target latents through pretrained Wan2.1 layers.
7. Model Architecture Details
- The homography-guided self-attention layer spatially concatenates source, target, and warped latents, treating frames as individual batch items.The resulting tensor has shape R^(bf)×3hw×d.Source: §7 · p. 11
- After homography-guided attention, the model discards the warped latent and feeds source and target latents into Wan2.1’s self-attention layers as a unified batch.Source: §7 · p. 11
8. Data Preparation Details
Data preparation expands MultiCamVideo trajectories and focal lengths to diversify training pairs while retaining the model’s supported sequence length and observed field of view.
8. Data Preparation Details
- MultiCamVideo supplies 10 trajectories per scene, all sharing an initial camera position and frame, across pan, tilt, translation, arc, random, and static motions.The augmentation targets the shared-start constraint in the original data.Source: §8 · p. 11
- Trajectory augmentation reverses one trajectory, concatenates another, and randomly crops an 81-frame segment.The crop matches the supported sequence length of Wan2.1.Source: §8 · p. 12
- Focal-length augmentation resizes and center-crops frames, increasing focal lengths without introducing regions outside the originally observed field of view.Selecting augmented videos as either source or target yields pairs covering focal-length increases and decreases.Source: §8 · p. 12
- Augmentation expands the dataset from 13,600 to 47,600 videos, with 47,432 for training and 168 for testing without scene overlap.Figure 8 depicts trajectory augmentation and focal-length augmentation in separate rows.Source: §8 · p. 12, Figure 8 · p. 12
9. Additional Experimental Details
This section specifies the diffusion training objective, baseline inference setups, and camera-trajectory accuracy evaluation. It compares predicted trajectories with ground truth using rotation and translation errors.
9.1. Loss Function
- The model minimizes mean squared error between its predicted velocity and the ground-truth velocity for an interpolated noisy video latent.The latent interpolates between random noise and the video latent; the model output is conditioned on text and timestep.Source: §9.1 · pp. 12–13
9.2. Baseline Setup
- Baseline outputs use 81 frames, with TrajectoryCrafter extended by a second segment and GEN3C evaluated on an input padded by repeating its final frame.TrajectoryCrafter’s two segments overlap by one frame; only the first 81 GEN3C output frames are evaluated.Source: §9.2 · p. 13
9.3. Camera Accuracy Metric
- ViPE-extracted camera trajectories are compared with ground truth using rotation error in degrees and translation error in meters.The source video’s first frame is prepended to anchor relative-pose extraction; WebVid comparison tables cover FF-Sync and FF-Async settings.Source: §9.3 · p. 13, Table 5 · p. 13, Table 6 · p. 13
10. Additional Results
InfCam shows stronger camera-trajectory accuracy in the asynchronous evaluation and competitive or better accuracy in the synchronized setting. Additional qualitative examples cover two datasets and varied camera trajectories.
10.1. Additional Quantitative Results
- In FF-Async, InfCam achieves lower rotation and translation errors than competitors, and it is competitive or superior in FF-Sync.The results support alignment with target trajectories across the two settings.Source: §10.1 · p. 13
- Table 2 averages the FF-Sync and FF-Async scores, pairing interpolated ReCamMaster’s asynchronous result with its original synchronized result.Interpolation is unnecessary for the synchronized setting.Source: §10.1 · p. 14
10.2. Additional Qualitative Results
- Qualitative examples cover AugMCV, WebVid, and eight camera trajectories; on WebVid, GEN3C and ReCamMaster fail to change the first-frame camera pose.The examples also include a rotational trajectory with a shorter focal length to illustrate wide-field-of-view generation.Source: §10.2 · p. 14, Figure 9 · p. 15, Figure 10 · p. 16