PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCamMaster: Camera-Controlled Generative Rendering from A Single Video

Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, Di Zhang

arXiv:2503.11647v2cs.CV

TL;DR

ReCamMaster targets the under-explored task of changing camera trajectories in existing videos while maintaining appearance and dynamic synchronization. It re-renders input videos with novel trajectories using video-conditioned pre-trained text-to-video models and a large Unreal Engine 5 dataset, substantially outperforming existing state-of-the-art approaches. The method also shows promise for stabilization, super-resolution, and outpainting.

  • Problem

    Changing camera trajectories of existing videos is under-explored and must preserve multi-frame appearance and dynamic synchronization.

  • Method

    ReCamMaster combines frame-dimension video conditioning for pre-trained text-to-video models with a multi-camera synchronized Unreal Engine 5 dataset and robustness-oriented training.

  • Results

    ReCamMaster substantially outperforms existing state-of-the-art approaches and strong baselines across multiple evaluation dimensions.

  • Takeaways & Limitations

    The framework shows promising applications in video stabilization, video super-resolution, and video outpainting.

  • Takeaways & Limitations

    ReCamMaster’s super-resolution application enhances only patches in the central region rather than providing fully equivalent super-resolution.

Abstract

from arXiv · show

Camera control has been actively studied in text or image conditioned video generation tasks. However, altering camera trajectories of a given video remains under-explored, despite its importance in the field of video creation. It is non-trivial due to the extra constraints of maintaining multiple-frame appearance and dynamic synchronization. To address this, we present ReCamMaster, a camera-controlled generative video re-rendering framework that reproduces the dynamic scene of an input video at novel camera trajectories. The core innovation lies in harnessing the generative capabilities of pre-trained text-to-video models through a simple yet powerful video conditioning mechanism--its capability is often overlooked in current research. To overcome the scarcity of qualified training data, we construct a comprehensive multi-camera synchronized video dataset using Unreal Engine 5, which is carefully curated to follow real-world filming characteristics, covering diverse scenes and camera movements. It helps the model generalize to in-the-wild videos. Lastly, we further improve the robustness to diverse inputs through a meticulously designed training strategy. Extensive experiments show that our method substantially outperforms existing state-of-the-art approaches. Our method also finds promising applications in video stabilization, super-resolution, and outpainting. Our code and dataset are publicly available at: https://github.com/KwaiVGI/ReCamMaster.

1. Introduction

ReCamMaster addresses the under-explored problem of changing camera trajectories for existing videos while preserving appearance and dynamic synchronization. It combines video-conditioned pre-trained text-to-video models with a large Unreal Engine 5 dataset and reports strong empirical performance.

  • Camera trajectory editing remains under-explored because re-rendering an existing video must preserve multi-frame appearance and dynamic synchronization.
  • ReCamMaster re-renders in-the-wild videos at novel camera trajectories using a video conditioning mechanism built on pre-trained text-to-video models.
  • Its training dataset contains 136K realistic videos from 13.6K dynamic scenes, 40 3D environments, and 122K camera trajectories curated to resemble real-world filming.
  • A designed training strategy further improves robustness to diverse inputs.
  • ReCamMaster outperforms state-of-the-art methods and strong baselines by a large margin and shows potential for stabilization, super-resolution, and outpainting.

2. Related Works

Prior camera-controlled generation largely targets text-to-video or image-to-video settings, while camera-controlled video-to-video generation has only recently emerged. Existing video-to-video approaches face domain and conditioning limitations when applied to real-world footage.

  • Camera-controlled generation has primarily studied text-to-video and image-to-video generation, incorporating camera parameters to control output viewpoints.
  • GCD pioneered camera-controlled video-to-video generation but relies on Kubric-simulated training data, limiting generalization to real-world videos.
  • Video-to-video generation includes editing, outpainting, super-resolution, and novel-view synthesis from videos with camera parameters.

3. Multi-Cam Video: A High-Quality Multi-Camera Synchronized Video Dataset

The authors construct synchronized multi-camera training data in Unreal Engine 5 because real-world paired data is costly and existing datasets lack sufficient scene and camera diversity. The resulting dataset combines dynamic scenes with extensive, automatically generated trajectories.

  • Training requires simultaneous multi-camera shots of the same scene to learn 4D-consistent generation, but real-world collection is costly and existing datasets are limited in diversity.
  • The Unreal Engine 5 pipeline combines 3D environments, animated characters, and multiple cameras moving along predefined trajectories.
  • The construction process uses internet-collected characters and animations together with designed camera trajectories for synchronized rendering.
  • Automatic camera-movement rules generate natural and diverse trajectories at scale, while characters and actions are randomly recombined across video sets.
  • The dataset contains 136K visually realistic videos from 13.6K dynamic scenes in 40 high-quality 3D environments and 122K camera trajectories.

4. Camera-Controlled Video Re-Generation

ReCamMaster re-renders a source video along specified camera trajectories by conditioning a latent video diffusion model on the source video and target camera poses. Its frame-dimension conditioning enables source–target spatio-temporal interaction, while camera encoding and targeted fine-tuning support synchronization, consistency, and diverse inputs.

  • 4. Camera-Controlled Video Re-Generation: The model synthesizes a target video sharing the source scene’s dynamics while following specified camera trajectories and preserving multi-frame appearance and synchronized motion.It conditions a pre-trained text-to-video diffusion model on the source video and target camera trajectory.
  • 4.2. Conditional Video Injection Mechanism: Frame-dimension conditioning concatenates source and target video tokens along the frame dimension, doubling the diffusion transformer’s input token count.The base model’s 3D self-attention then enables interaction between source and target tokens without additional aggregation layers.
  • 4.2. Conditional Video Injection Mechanism: Compared with channel- and view-dimension alternatives, frame-dimension conditioning enables spatio-temporal interaction between conditional and target tokens through all blocks of the base model.The paper attributes its substantial comparative advantage to this flexibility in using condition information.
  • 4.3. Camera Pose Conditioning: The camera encoder projects each target camera sequence into the video-token channel space and adds it to visual features in every Transformer block.The model conditions only on the target camera because accurate source-camera trajectories are difficult to estimate during inference.
  • 4.4. Training Strategy: ReCamMaster fine-tunes only the camera encoder and 3D-attention layers while freezing other parameters, and adds moderate training noise to conditional video latents.This strategy preserves the base text-to-video model’s native capability and mitigates Unreal Engine’s synthetic characteristics.
  • 4.4. Training Strategy: The framework inherently supports text-to-video, image-to-video, and video-to-video camera-controlled generation.This capability is reported as part of the model’s broader conditional generation design.

5. Experimental Results

ReCamMaster is evaluated on camera accuracy, synchronization, visual quality, and VBench metrics using WebVid videos and diverse camera trajectories. Experiments and ablations show strong performance and support the proposed conditioning and training strategies.

  • Metrics: The evaluation covers visual quality, camera trajectory accuracy, and view synchronization through FID, FVD, CLIP, rotation and translation errors, CLIP-V, FVD-V, and VBench metrics.
  • Evaluation setup: The evaluation uses 1,000 WebVid videos with 10 camera trajectories, including pan, tilt, translation, zoom, and horizontal arc movements.
  • Baselines: ReCamMaster is compared with state-of-the-art camera-controlled video-to-video methods, including GCD, Trajectory-Attention, and DaS.
  • Comparison results: ReCamMaster outperforms baselines across multiple metric dimensions, while qualitative results show consistent content, temporal synchronization, and plausible regions outside the original view.
  • Video conditioning ablation: Channel- and view-concatenation conditioning produces artifacts, content inconsistency, and asynchronous dynamics, whereas frame concatenation preserves identity and synchronization during complex movements.
  • Training-strategy ablation: Fine-tuning spatial-temporal attention, source-frame dropping, and their combination improve visual quality, with the combined strategy performing best.

6. Applications of ReCamMaster

ReCamMaster supports video stabilization, video super-resolution, and video outpainting by modifying camera trajectories. The reported applications preserve source content while generating or enhancing selected visual regions.

  • Video Stabilization: ReCamMaster stabilizes unsteady DeepStab videos by adjusting camera trajectories while preserving the original content.
  • Video super-resolution: The framework is also applied to video super-resolution, although its enhancement is limited to patches in the central region.
  • Video Outpainting: Zoom-out trajectories enable video outpainting, generating previously invisible areas such as feet and the ground.

7. Conclusion and Limitations

ReCamMaster re-renders dynamic scenes from input videos along new camera trajectories using video conditioning, synthetic synchronized data, and improved training strategies. The paper reports broad application potential alongside computational and generative limitations.

  • Conclusion: ReCamMaster reproduces dynamic scenes from input videos at new camera trajectories using an innovative video conditioning technique and a large Unreal Engine 5 dataset.
  • Applications: The method shows promise for video stabilization, super-resolution, and outpainting.
  • Limitations: Concatenating source and target video tokens improves generation quality but increases computational demands.
  • Limitations: ReCamMaster inherits limitations from pretrained text-to-video models, including less effective hand generation.

Supplementary Material

The supplementary material describes the transformer-based latent diffusion foundation used by ReCamMaster. A 3D-VAE maps videos into latent space for the video diffusion model.

  • ReCamMaster uses a transformer-based latent diffusion model as its base text-to-video generation model.
  • A 3D-VAE transforms videos from pixel space into latent space before video diffusion processing.

B. Details of Data Construction

The dataset combines diverse 3D environments, characters, animations, camera trajectories, movement speeds, and camera parameters to train ReCamMaster.

  • 3D Environments: The rendered dataset uses 40 visually realistic 3D environments spanning indoor and outdoor settings, supplemented by stylized or surreal scenes.
  • Characters and Animations: It includes 70 human 3D character models and approximately 100 animations, combined across video sets to create varied dynamic scenes.
  • Camera Trajectories: Camera trajectories are batch-generated with randomized starting positions, pan-and-tilt motions, translations, arcs, random smoothed paths, and static shots.
  • Camera Movement Speed: Half of the training data uses constant-speed trajectories, while the other half uses variable-speed trajectories generated by nonlinear functions.
  • Camera Movement Speed: The variable-speed trajectory parameter a controls motion timing: positive values start fast, negative values start slow, and larger absolute values increase the change.
  • Camera Parameters: Two camera parameter sets are used: focal=35mm with aperture=2.8, and focal=24mm with aperture=10.

C.1. Ablation on Dataset Construction

Dataset diversity improves ReCamMaster’s generalization, and the framework supports multiple camera-controlled generation settings.

  • Ablation on Dataset Construction: 136K videos from 13.6K dynamic scenes, 40 environments, and 122K trajectories substantially improve visual quality, camera accuracy, and synchronization over less diverse toy data.
  • Ablation on Dataset Construction: ReCamMaster supports text-to-video, image-to-video, and video-to-video camera-controlled generation tasks.

C.3. More Comparison with SOTA Methods

Additional qualitative evaluations compare ReCamMaster with existing methods, examine non-overlapping starting frames, and document failure cases inherited from the base model.

  • Comparison with SOTA Methods: Figure 13 provides qualitative comparisons between ReCamMaster and state-of-the-art methods.
  • Non-overlapping First Frames: Figure 10 evaluates results when generated videos begin from first frames that do not overlap with the input video’s first frame.
  • Failure Cases: Figure 11 visualizes failure cases, including inferior character hand movements and occasional failures when generating very small objects.
  • Non-overlapping First Frames: The model generalizes well to non-overlapping first frames, enabling re-filming from a completely new perspective.
  • Additional Results: Additional synthesized results are presented in Figure 12.
Loading 2503.11647v2…