ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning

1. Introduction

• Camera-control methods for generated videos do not directly address user-provided footage, and re-generating complex scene motion from a requested camera path remains challenging.The task is ill-posed because a reference video does not reveal the scene from every angle without full 4D scene knowledge.Source: §1 · p. 1
• The first stage creates a noisy anchor video with the requested camera path using depth-based point-cloud rendering or a multiview diffusion model.Both options can produce blur, artifacts, black areas, or poor temporal consistency.Source: §1 · p. 2
• Masked video fine-tuning trains a spatial LoRA on source-video frames for appearance and background context, and a temporal LoRA on known anchor pixels and reference data for motion.Unknown regions are excluded from the masked loss.Source: §1 · p. 2
• At inference, the video model fills unknown anchor regions with plausible content and improves temporal consistency; SDEdit with only the spatial LoRA can further reduce blur and jitter.The output retains the anchor's learned motion and layout.Source: §1 · p. 2
• On Kubric, ReCapture outperforms Generative Camera Dolly and other 4D reconstruction methods; ablations validate its components on VBench.The method does not require paired video data.Source: §1 · p. 2

2. Related Work

• Earlier video-generation camera control targets prompt-generated videos, not footage captured in the real world or supplied by a user.The paper uses Stable Video Diffusion as its video diffusion model.Source: §2 · pp. 2–3
• Figure 2 depicts anchor-video generation followed by masked fine-tuning with spatial and temporal LoRAs, then video-model inference for the clean output.The figure summarizes the method's two setup stages and its output-generation step.Source: Figure 2 · p. 3
• Camera-control methods generally add camera-parameter modules or rendering priors to video generation, rather than changing the trajectory of user-provided videos.The cited approaches include supervised modules and training-free rendering-prior methods.Source: §2 · p. 3
• Dynamic-scene reconstruction has progressed from synchronized multiview inputs to monocular-video approaches using time-evolving NeRFs, image-based rendering, diffusion, and dynamic Gaussian splatting.These methods often struggle to extrapolate beyond the original field of view or depend on effective multiview cues.Source: §2 · p. 3
• ReCapture reframes camera control as video-to-video translation, using video-generation motion priors to reangle user footage, including beyond its original field of view.The method does not build an explicit 4D representation.Source: §2 · p. 3

3. Method

• The method’s two stages generate an incomplete, noisy anchor video at the new camera trajectory and regenerate it into a clean, temporally consistent video.Anchor frames can contain missing regions, structural deformations, and flicker; the approach is agnostic to the anchor-generation technique.Source: §3 · pp. 3–4

3.1. Preliminaries

• LoRA updates pretrained model weights through low-rank factors, providing a more computationally efficient fine-tuning approach that can preserve more of the model’s prior knowledge.The method applies LoRA to both spatial and temporal components of the video diffusion model.Source: §3.1 · p. 4

3.2. Anchor Video with New Camera Motion

• Point-cloud rendering suits translations and small rotations, while multiview diffusion generates novel views for larger camera rotations.Point clouds are built from source frames and estimated depth; independently generated multiview frames can have temporal inconsistencies and artifacts.Source: §3.2 · p. 5

3.3. Masked Video Fine-tuning

• Masked temporal LoRA training excludes invalid anchor regions from its loss, so the model learns from valid pixels while completing missing regions at inference.A context-aware spatial LoRA is trained on augmented source-video frames to repair structural artifacts and preserve source appearance; a final spatial-LoRA refinement can reduce blur.Source: §3.3 · pp. 5–6
• The method adds dynamic camera motion to existing videos without large-scale training on 4D multiview video data and generalizes across videos and camera trajectories.The authors state that the method generalizes to a wide array of videos and camera trajectories.Source: §3.3 · p. 6

4. Experiments

• The evaluation combines PSNR and SSIM with VBench semantic measures, a user study, and ablations; Kubric-4D includes 100 evaluation scenes downsampled to 14 frames at 384 × 256.VBench assesses seven dimensions across 35 videos, including subject identity consistency, motion smoothness, and temporal flickering.Source: §4 · pp. 6, 8, §4.1 · pp. 8–9

4.1. Quantitative Evaluation

• ReCapture outperforms existing 4D reconstruction methods on Kubric-4D and Generative Camera Dolly despite not requiring 4D training data, and leads on most VBench dimensions.The reported Kubric-4D comparison follows the video resolution protocol in [82].Source: Table 2 · p. 8, §4.1 · pp. 8–9, Table 1 · p. 8

4.2. Qualitative Comparisons

• Against Generative Camera Dolly [82], ReCapture more accurately follows the camera trajectory and has fewer artifacts and less blurriness, including at substantially different camera positions.ReCapture also preserves subject motion and appearance in those comparisons, while Dolly exhibits significant artifacts.Source: §4.2 · p. 9

4.3. Ablation Studies.

• Masked temporal LoRA improves temporal consistency and visual quality and fills unseen anchor regions; spatial LoRA boosts subject consistency, while SDEdit reduces artifacts.Table 3 compares temporal LoRAs, additional context-aware spatial LoRAs, and subsequent SD-Edit.Source: §4.3 · p. 9, Table 3 · p. 9

4.4. Implementation Details.

• The implementation uses CAT3D [24] for multiview generation and SVD [9] for video diffusion, with rank-16 spatial and temporal LoRAs.Fine-tuning takes 400 steps and about five minutes on a single 80GB A100 GPU.Source: §4.4 · p. 9

• ReCapture generates novel-camera videos from user-provided footage and often preserves complex scene and subject motion and scene details.Source: §5 · p. 9