PaperlayerView exact revision on arXiv

Source-linked AI summary

Vista4D: Video Reshooting with 4D Point Clouds

Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca, Yash Kant, Ryan Burgert, Yuancheng Xu, Koichi Namekata, Yiwei Zhao, Bolei Zhou, Micah Goldblum, Paul Debevec, Ning Yu

arXiv:2604.21915v1cs.CV

TL;DR

Video reshooting must preserve seen content, generate plausible unseen content, and maintain precise camera control despite artifacts in real-world dynamic-scene reconstruction. Vista4D grounds source videos and target cameras in a temporally persistent 4D point cloud, combines it with video diffusion, and trains on artifact-containing dynamic multiview data. The paper reports improved 4D consistency, camera control, video fidelity, and extensions to dynamic scene applications.

  • Problem

    Video reshooting methods struggle with real-world point-cloud artifacts, content preservation, and accurate camera control for challenging trajectories and viewpoints.

  • Method

    Vista4D uses segmented temporally persistent 4D point clouds, target-camera rendering, joint source-video conditioning, and training on noisy reconstructed multiview data.

  • Results

    Extensive evaluations report improved 4D consistency, camera-control accuracy, and video fidelity over state-of-the-art baselines across varied videos and cameras.

  • Takeaways & Limitations

    Vista4D generalizes beyond video reshooting to dynamic scene expansion, 4D scene recomposition, and long-video inference with memory.

  • Takeaways & Limitations

    Vista4D lacks user control over the trade-off between following an imperfect point cloud and relying on the video prior to correct geometry.

Abstract

from arXiv · show

We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our method re-synthesizes the scene with the same dynamics from a different camera trajectory and viewpoint. Existing video reshooting methods often struggle with depth estimation artifacts of real-world dynamic videos, while also failing to preserve content appearance and failing to maintain precise camera control for challenging new trajectories. We build a 4D-grounded point cloud representation with static pixel segmentation and 4D reconstruction to explicitly preserve seen content and provide rich camera signals, and we train with reconstructed multiview dynamic data for robustness against point cloud artifacts during real-world inference. Our results demonstrate improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines under a variety of videos and camera paths. Moreover, our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition. See our project page for results, code, and models: https://eyeline-labs.github.io/Vista4D

1. Introduction

Video reshooting seeks to synthesize an input scene from user-defined camera trajectories while preserving seen content, plausibly generating unseen content, and maintaining precise control. Vista4D addresses limitations in prior methods with a temporally persistent 4D point cloud, video diffusion priors, and artifact-aware training.

  • Video reshooting renders a dynamic scene from an input video along novel camera trajectories and viewpoints.
  • The task requires faithful reconstruction of seen content, plausible generation of unseen content, and precise user-definable camera control.
  • Vista4D combines video diffusion models with a 4D point cloud whose segmented static pixels persist temporally, grounding generation and supplying camera-control signals.The representation explicitly preserves seen content and supports target cameras with limited per-frame overlap.
  • Prior methods often encounter geometry artifacts, temporal flickering, inaccurate camera control, and content-preservation problems on challenging real-world trajectories.
  • Training on dynamic 4D-reconstructed multiview pairs improves robustness to real-world point-cloud artifacts while retaining video-model priors for geometric coherence.
  • Quantitative, qualitative, and user-study evaluations report improved content preservation, camera controllability, and visual quality across varied videos and cameras.The framework also extends to dynamic scene expansion, 4D scene recomposition, and long-video inference with memory.

2. Related work

Related work for video reshooting uses explicit point-cloud priors, implicit camera or video priors, and increasingly capable 4D reconstruction methods. Vista4D’s context is shaped by the trade-off between spatial grounding and robustness to monocular dynamic-scene artifacts.

  • Video reshooting with explicit priors: Explicit-prior methods render per-frame camera-space point clouds from video depth estimators as conditioning for video diffusion models.Related static-scene novel-view synthesis and video-motion-control methods also use depth-estimation priors.
  • Vista4D context: Vista4D builds a temporally persistent 4D point cloud, renders it from target cameras, and jointly conditions a finetuned video diffusion model on the source video and render.
  • Vista4D context: Its training uses dynamic multiview data that exposes non-frontal reconstruction artifacts, unlike double reprojection’s frontal, artifact-free training view.
  • Video reshooting with implicit priors: Implicit-prior methods use camera embeddings or video references with synthetic synchronized multiview data to control video diffusion models.
  • 4D reconstruction: Traditional structure-from-motion methods rely on multiview geometry constraints that are not robust to dynamic scenes.
  • 4D reconstruction: Learning-based depth, SLAM, and end-to-end 4D reconstruction methods provide increasingly efficient approaches to coherent dynamic-scene reconstruction.

3. 4D-grounded video reshooting

Vista4D constructs a temporally persistent 4D point cloud from the source video and conditions a finetuned video diffusion model on its target-camera render, source video, and camera signals. Training with noisy reconstructed multiview data and source-video conditioning targets robustness to real-world point-cloud artifacts while preserving content and motion.

  • 4D-grounded representation: The pipeline lifts the source video into a world-space 4D point cloud using reconstructed depth, camera parameters, and static pixel masks.Static pixels persist across frames, forming explicit 4D context rather than independent per-frame point clouds.
  • Model conditioning: Vista4D renders the persistent point cloud from target cameras and uses the render, alpha mask, source video, and target cameras to generate the reshot video.The target cameras are injected as Plücker embeddings into the diffusion transformer.
  • Robust training: Training mixes synthetic multiview dynamic videos with reconstructed real-world monocular data to improve robustness to noisy 4D reconstruction.The training setup addresses the mismatch between precise training geometry and imperfect real-world inference point clouds.
  • Model conditioning: Source-video conditioning transfers geometry and appearance information and is reported to preserve source content more effectively than alternative conditioning choices.The model uses in-context source-video conditioning alongside point-cloud renders.
  • Evaluation: Vista4D reports the best camera-control errors and lowest SuperGlue landmark reprojection error in Table 1, while outperforming explicit-prior baselines across video-fidelity metrics in Table 3.The user study also favors Vista4D for content preservation, camera control, and overall fidelity.

4. Experiments

Experiments evaluate Vista4D on camera control, 3D consistency, novel-view synthesis, video fidelity, and qualitative real-world reshooting. The reported results show stronger camera control, geometric consistency, motion preservation, robustness to artifacts, and extensions to scene expansion and recomposition.

  • Evaluation setup: The evaluation uses 110 diverse video-camera pairs built from 51 DAVIS and Pexels videos with reconstructed geometry, segmentation, and designed camera trajectories.Baselines include explicit-prior and implicit-prior video reshooting methods.
  • Quantitative comparisons: Vista4D consistently achieves more accurate camera control and significantly better 3D consistency than baselines on the evaluation dataset.Metrics measure camera translation, rotation, and intrinsics errors alongside landmark reprojection error.
  • Quantitative comparisons: On the iphone dataset, Vista4D outperforms baselines in PSNR, LPIPS, and optical-flow EPE, although TrajectoryCrafter has higher SSIM.The authors note that SSIM does not capture visible artifacts in TrajectoryCrafter outputs.
  • Quantitative comparisons: Vista4D outperforms explicit-prior baselines on video-fidelity metrics, while some implicit-prior methods score better on metrics associated with less camera movement.Qualitative comparisons and the user study support higher overall fidelity for Vista4D.
  • Qualitative comparisons: Qualitative comparisons attribute baseline failures to point-cloud artifacts and camera-control failures at non-frontal target views, whereas Vista4D preserves stronger 4D consistency.The source video helps correct imperfect point-cloud geometry and streaking artifacts.
  • Robustness: Training with depth artifacts and in-context source video enables robustness to imperfect 4D reconstruction and segmentation failures.The reported ablations isolate data and conditioning choices related to artifact robustness.
  • Applications: Joint 4D reconstruction with casual scene captures or alternate angles enables dynamic scene expansion with more faithful environment reproduction.The added information reduces reliance on hallucinated scene content and provides stronger control beyond the source video.
  • Applications: Direct editing of the 4D point cloud supports scene recomposition, including subject insertion while maintaining dynamics and blending differing lighting conditions.The cited example inserts a sunlit rhino into an overcast funeral procession.

5. Conclusion

Vista4D resynthesizes dynamic scenes from novel camera trajectories using a temporally persistent 4D point cloud and 4D-reconstructed training data. It achieves robust real-world inference while extending to scene expansion, recomposition, and long-video memory, but lacks user control over the point-cloud/prior trade-off.

  • Conclusion: Vista4D grounds novel-view dynamic scene synthesis in a temporally persistent 4D point cloud and 4D-reconstructed dynamic multiview training data.This combination improves robustness to real-world point-cloud artifacts while preserving seen content.
  • Conclusion: Extensive evaluations and a user study validate improved 4D consistency, camera control accuracy, and video fidelity over state-of-the-art baselines.
  • Conclusion: Vista4D generalizes to dynamic scene expansion, 4D scene recomposition, and long-video inference with memory.
  • Limitations: Vista4D lacks user control over how closely generation follows an imperfect point cloud versus the video-model prior.The proposed extension is an interpolation mechanism between explicit point-cloud and implicit source-video/camera priors.
  • Broader impacts: Camera control over video raises ethical concerns about content ownership and transformative work alongside its creative possibilities.

A. More qualitative results on video reshooting

Additional qualitative experiments compare Vista4D with five baselines, demonstrate 720p reshooting, and extend the framework to dynamic scene expansion, scene recomposition, and long-video inference.

  • Comparison to baselines: Vista4D consistently preserves source content better, controls cameras more accurately, and achieves higher video fidelity than five named baselines.The comparisons use ReCamMaster, CamCloneMaster, EX-4D, TrajectoryCrafter, and GEN3C.
  • Video reshooting at 720p: Vista4D produces 720p video reshooting results with a 1280 × 720 finetuned checkpoint.
  • Dynamic scene expansion: Dynamic scene expansion incorporates additional casual scene captures through joint 4D reconstruction with the source video.
  • 4D scene recomposition: For 4D scene recomposition, the method conditions on an edited source video rendered from the edited point cloud to avoid conflicts with the unedited source video.
  • Long video inference: Long-video inference chunks the source into 49-frame clips and integrates generated clips into an existing dynamic point cloud for persistent memory.Existing frames are subsampled and jointly reconstructed with newly generated frames, followed by camera alignment and registration.

B. Model architecture details

Vista4D fine-tunes a text-to-video diffusion model to condition on the source video, target cameras, point-cloud renders, and alpha masks, producing resynthesized output video.

  • Model architecture: Vista4D builds on Wan2.1-T2V-14B and adds conditioning on source video, target cameras, point-cloud renders, and alpha masks.The model outputs a video synthesizing the source scene from the specified target cameras.
  • Condition encoding: The source video and point-cloud render are encoded into latents, while alpha-mask tokens are reshaped to match latent-space dimensions.Patchify layers initialize from the base video model, with a zero-initialized alpha-mask layer combined with point-cloud tokens.
  • Camera conditioning: Target cameras enter each DiT block through Plücker embeddings and a zero-initialized camera projection before self-attention.An identity-initialized affine projection is also applied before cross-attention and the feedforward network.

C.1. Training dataset

Training combines synthetic time-synchronized multiview data with filtered monocular internet videos, using 4D reconstruction, static-pixel segmentation, camera augmentation, and conditioning dropout to improve robustness and control.

  • Training datasets: Vista4D trains on MultiCamVideo multiview data and a random 60K subset of OpenVidHD-0.4M monocular videos at a 1:1 sampling ratio.OpenVidHD videos labeled with dynamic camera movement are selected rather than those labeled static.
  • MultiCamVideo: MultiCamVideo scenes are reconstructed with STream3R using a moving window of 128, with frame-first ordering across views.Gaussian smoothing is applied to predicted camera intrinsics and extrinsics because the dataset renders smooth cameras.
  • Data processing: The training pipeline removes video cuts, uses dataset captions, and automatically segments dynamic pixels to construct static-pixel masks.Semantic classes and language filtering identify likely dynamic subjects before Grounded SAM 2 segmentation.
  • Additional training outcomes: Vista4D provides qualitative reshooting examples at 720p and demonstrations of dynamic scene expansion.The 720p checkpoint is finetuned at 1280 × 720, while expansion jointly reconstructs casual captures with the source video.
  • Training strategy: Conditioning dropout and image-latent noise augmentation strengthen point-cloud influence, camera control, and quality across inference chunks.The image condition is randomly dropped 30% of the time, and noise augmentation uses α = 0.05.

D. Evaluation dataset and user study details

The evaluation uses a 110 video-camera-pair dataset and a user study, alongside a camera-design interface and reported preprocessing and inference procedures. The dataset combines selected DAVIS and Pexels videos with manually designed target trajectories and zooms.

  • Evaluation dataset: The evaluation dataset contains 110 video-camera pairs built from 13 DAVIS videos and 38 Pexels videos.Videos were selected for high quality and dynamic scenes and/or camera motion.
  • Camera design: Target cameras use two to three designed trajectories and zooms per video, specified through the camera-design UI.The interface supports camera intrinsics and extrinsics keyframes plus interpolation tension and smoothness.
  • Baseline protocol: Baseline inference uses a direct-mode procedure when source and target first-frame cameras do not match, except for TrajectoryCrafter.The procedure freezes and moves the first-frame point cloud before continuing inference.
  • Runtime: Vista4D has negligible preprocessing overhead relative to model inference, but its total inference is slower than the baselines.The reported comparison attributes the difference primarily to in-context conditioning and a slower base model; all methods use 50 inference steps on an NVIDIA A100 80GB.
  • User study: The user study randomly samples 30 video-camera pairs and asks 42 participants to compare methods on content preservation, camera accuracy, and overall fidelity.Participants select preferred results from Vista4D and baseline video-reshooting methods.

E. Quantitative evaluation metric details

The evaluation measures camera accuracy, 3D consistency, novel-view synthesis, video fidelity, and ablation behavior using reconstruction- and perceptual-based metrics. Camera accuracy is assessed through jointly reconstructed source and generated poses, while reprojection error evaluates cross-view 3D consistency.

  • Camera control accuracy: Camera accuracy is measured with translation, rotation, and intrinsics errors computed from predicted and target camera parameters.The evaluation uses camera extrinsics and vertical field of view, with source-camera alignment obtained through Umeyama alignment.
  • 3D consistency: Reprojection Error evaluates 3D consistency by matching 2D points across two known camera views, triangulating them, and measuring reprojection.This avoids relying solely on fixed ground-truth images for unseen regions.
  • Novel-view synthesis: Novel-view synthesis is evaluated on five error-free iPhone sequences using PSNR and related standard pixel-wise synthesis protocols.The moving camera supplies the source video, while the first static camera with continuous frames supplies the target.
  • Video fidelity: Video fidelity evaluation compares generated and source videos with FID and FVD and uses VBench for perceptual dimensions including aesthetic and imaging quality.The evaluation covers Vista4D and all baselines on the 110 video-camera-pair dataset.
  • Ablations: Removing artifact exposure or in-context source conditioning produces geometry artifacts and temporal jittering that the model cannot reliably correct.The ablations compare no source video, cross-attention conditioning, and training without depth artifacts.

F.2. Temporal persistence

Temporal persistence is tested by removing it from the point-cloud representation and examining source-content preservation and camera control. Without persistence, the model struggles with seen static content and target-camera frames that have little overlap with the source point cloud.

  • Content preservation: Without static-pixel temporal persistence, the model struggles to preserve seen content from the source video.Examples include snow and rock mountains, metal fences, roads, and the right side of scenes.
  • Camera control: Without temporal persistence, camera control becomes less accurate when target-camera frames have little overlap with the source video point cloud.The paper notes that this effect is difficult to show in still frames.
Loading 2604.21915v1…