Source-linked AI summary
Follow-Your-Creation: Empowering 4D Creation through Video Inpainting
Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, Qifeng Chen
TL;DR
4D video generation lacks flexible video-input and viewpoint control, while conventional inpainting struggles with large-angle reconstruction and temporal consistency. Follow-Your-Creation fine-tunes a video inpainting foundation model using composite masks, self-iterative tuning, and temporal packing. The framework generates and edits monocular-video-based 4D content with consistent multi-view coherence and reported high-fidelity results under novel camera trajectories.
Problem
4D generation methods have limited support for video inputs and camera viewpoints, while vanilla video inpainting struggles with large-angle reconstruction and temporal consistency.
Method
The framework uses depth-based dynamic point clouds, composite occlusion and editing masks, self-iterative tuning, and temporal-packing inference around a video inpainting foundation model.
Results
Evaluations consistently demonstrate high-fidelity videos under novel camera trajectories, with consistent multi-view coherence and successful ablation support for composite masks, iterative tuning, and temporal packing.
Takeaways & Limitations
The approach supports controllable 4D generation and editing from monocular videos while preserving the original video inpainting model’s performance.
Abstract
from arXiv · showhide
We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During training, we randomly sample different types of masks to construct diverse and challenging inpainting scenarios, enhancing the model's generalization and robustness in various 4D editing and generation tasks. To handle temporal consistency under large camera motion, we design a self-iterative tuning strategy that gradually increases the viewing angles during training, where the model is used to generate the next-stage training data after each fine-tuning iteration. Moreover, we introduce a temporal packaging module during inference to enhance generation quality. Our method effectively leverages the prior knowledge of the base model without degrading its original performance, enabling the generation of 4D videos with consistent multi-view coherence. In addition, our approach supports prompt-based content editing, demonstrating strong flexibility and significantly outperforming state-of-the-art methods in both quality and versatility.
1 Introduction
Follow-Your-Creation addresses the lack of video-input and viewpoint control in 4D generation by reformulating creation as specialized video inpainting. It combines composite masks, iterative training, and temporal packing to generate and edit coherent 4D videos from monocular inputs.
- 4D video methods guide dynamic content with camera trajectories but commonly depend on large-scale data, limited input modalities, and restricted viewpoint control.
- Existing depth-based pipelines render monocular videos into dynamic point clouds, but occlusions and incomplete geometry create masked regions requiring video inpainting.
- Follow-Your-Creation reformulates 4D generation as specialized video inpainting to use Wan2.1 as a generative prior with minimal additional training.
- Composite masks combine rendering-induced occlusions with editing regions, and random mask sampling creates diverse inpainting scenarios for training.
- Temporal packing uses previously generated results as priors to improve coherence across frames and viewpoints during inference.
- Evaluations on synchronized multi-view and large-scale monocular-video datasets show high-fidelity generation under novel camera trajectories.
2 Related Work
Prior work established controllable video generation, video editing, and dynamic novel-view synthesis, but Follow-Your-Creation applies video inpainting to 4D creation with camera control and editing.
- Camera-controlled generation adds signals such as pose, depth, sketches, or motion patterns to align videos with user intentions.
- Video inpainting methods use mask sequences for editing, but non-generative approaches mainly handle partial occlusions while generative approaches focus on content inpainting.
- Follow-Your-Creation distinguishes itself by reformulating 4D video creation as video inpainting rather than treating inpainting only as content editing.
- Dynamic-scene novel-view methods use diffusion models, but some target specific domains or support only limited camera trajectories.
3 Method
Follow-Your-Creation reformulates monocular 4D video creation and editing as video inpainting. It combines depth-based point-cloud rendering, composite masks, angle-progressive self-iteration, and temporal packing to complete novel views while improving multi-view consistency.
- 3.1 Dynamic point cloud: The pipeline converts an input video into dynamic point clouds using estimated depth, camera intrinsics, and trajectory-dependent extrinsics for novel-view rendering.The point clouds bridge original frames and generated camera views; perspective projection renders the point clouds onto camera planes.
- 3.1 Dynamic point cloud: Rendered novel views produce binary visibility masks that mark valid projections as 1 and missing or out-of-view regions as 0 for inpainting.These masks identify holes caused by occlusion, incomplete geometry, or camera motion.
- 3.2 Composite mask dataset: Training uses point-cloud masks, editing masks, and their union, randomly sampling among the three mask types for video-inpainting supervision.Double reprojection aligns point-cloud masks with the original frames, while editing masks specify regions for content modification.
- 3.3 Self-iterative tuning: Self-iterative tuning begins with small viewpoint changes and progressively generates training videos at larger angular ranges using LoRA adaptation and spatial-temporal consistency losses.Each iteration uses the tuned model to generate the next-stage masked data.
- 3.4 Temporal-packing inference: Temporal packing selects generated frames with large inpainting areas and concatenates their tokens with the next hole video to improve consistency across camera trajectories.The pretrained model’s global spatio-temporal self-attention performs the token interaction without additional fusion layers.
4 Experiments
Experiments evaluate Follow-Your-Creation through qualitative comparisons, benchmark metrics, and ablations. The results show high-fidelity, consistent camera-controlled generation and editing, while composite masks, self-iterative tuning, and temporal packing each address distinct failure modes.
- Qualitative comparison: Follow-Your-Creation generates novel-trajectory videos with high fidelity, 4D consistency, and precise pose control.
- Qualitative comparison: The method produces better editing effects with smooth and accurate camera movements than baseline methods, whose outputs contain significant artifacts.
- Quantitative comparison: The evaluation uses VBench, Kubric-4D low-level metrics, visual-quality measures, camera-accuracy errors, and view-synchronization scores.
- Quantitative comparison: Follow-Your-Creation outperforms the baseline across PSNR, LPIPS, and SSIM on the Kubric-4D dataset.
- Ablation study: Removing composite masks causes artifacts and prevents successful 4D creation and editing, whereas the full method follows edited frames and camera trajectories.
- Ablation study: Without self-iterative tuning, temporal coherence degrades as camera motion angles increase, while temporal packing preserves consistent overlap content across multi-view videos.
5 Conclusion
Follow-Your-Creation reformulates 4D video creation as video inpainting, using composite masks and iterative tuning to support controllable generation under camera motion. Temporal packing further improves multi-view coherence during inference.
- Temporal-packing inference enhances generation quality and maintains multi-view consistency across camera trajectories.The ablation compares rightward motion with combined upward and rightward motion having partial spatial overlap.
- The framework reformulates 4D video creation as video inpainting using composite masks generated from dynamic point clouds and double reprojection.This supports realistic and controllable results with minimal additional training.
- Self-iterative tuning gradually increases viewing angles during training to address temporal consistency under large camera motion.