Source-linked AI summary
SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generation
Chun-Han Yao, Yiming Xie, Vikram Voleti, Huaizu Jiang, Varun Jampani
TL;DR
Generating dynamic 3D objects from monocular videos is ill-posed because unseen-view appearance and motion are ambiguous, while limited 4D data makes generalization difficult. SV4D 2.0 addresses these challenges with architectural, data, training, and 4D-optimization improvements, achieving better detail and consistency than SV4D in video synthesis and 4D optimization.
Problem
Dynamic 3D generation from a single-view video is ill-posed because object appearance and motion at unseen views are ambiguous, and limited large-scale 4D datasets make generalization challenging.
Method
SV4D 2.0 combines a multi-view video diffusion model with improved 3D attention, curated dynamic-object data, progressive training, and two-stage 4D optimization.
Results
SV4D 2.0 outperforms SV4D, improving detail by -14% LPIPS and 4D consistency by -44% FV4D in video synthesis, plus -12% LPIPS and -24% FV4D in 4D optimization.
Takeaways & Limitations
The method provides higher-quality novel-view videos and 4D assets, with improved robustness to occlusions and large motion and better generalization to real-world videos.
Takeaways & Limitations
SV4D 2.0 cannot handle videos with strong perspective distortion and remains focused on object-centric rather than scene-level videos.
Abstract
from arXiv · showhide
We present Stable Video 4D 2.0 (SV4D 2.0), a multi-view video diffusion model for dynamic 3D asset generation. Compared to its predecessor SV4D, SV4D 2.0 is more robust to occlusions and large motion, generalizes better to real-world videos, and produces higher-quality outputs in terms of detail sharpness and spatio-temporal consistency. We achieve this by introducing key improvements in multiple aspects: 1) network architecture: eliminating the dependency of reference multi-views and designing blending mechanism for 3D and frame attention, 2) data: enhancing quality and quantity of training data, 3) training strategy: adopting progressive 3D-4D training for better generalization, and 4) 4D optimization: handling 3D inconsistency and large motion via 2-stage refinement and progressive frame sampling. Extensive experiments demonstrate significant performance gain by SV4D 2.0 both visually and quantitatively, achieving better detail (-14\% LPIPS) and 4D consistency (-44\% FV4D) in novel-view video synthesis and 4D optimization (-12\% LPIPS and -24\% FV4D) compared to SV4D. Project page: https://sv4d20.github.io.
1 Stability AI 2 Northeastern University ∗Equal contribution † Equal advising
SV4D 2.0 generates multiple high-fidelity novel-view videos from a monocular input, supporting high-quality 4D asset optimization.
- SV4D 2.0 generates multiple novel-view videos from an input monocular video.
- The generated videos provide detail sharpness and consistency across view and time/frame axes for 4D asset optimization.
1. Introduction
The paper addresses the difficulty of generating spatially and temporally consistent dynamic 3D content from monocular video. SV4D 2.0 combines architectural, training, and optimization changes and reports improved synthesis and 4D-generation quality over SV4D.
- Motivation: Dynamic 3D generation seeks to synthesize moving objects or scenes in 3D while capturing their dynamics across time.Such assets support applications including video games, movie production, and AR/VR experiences.
- Challenge: Generating dynamic 3D objects from single-view video is ill-posed because object appearance and motion are ambiguous at unseen camera views.Limited large-scale 4D datasets and the high-dimensional nature of 4D further complicate generalization across object categories and motion.
- Challenge: NVVS must maintain consistency across novel views and dynamic motion, but prior separate-model and joint-model approaches struggle with blur, motion, and spatio-temporal consistency.These limitations restrict handling of larger object motion.
- Approach: SV4D 2.0 modifies the architecture, data curation, training strategy, and 4D optimization to improve sparse-view synthesis, occlusion robustness, generalization, and output quality.The approach removes SV4D’s separate reference-multi-view dependency, adopts progressive 3D-to-4D training, and uses two-stage optimization with progressive frame sampling.
- Results: -14% LPIPS and -44% FV4D in video synthesis, plus -12% LPIPS and -24% FV4D in 4D optimization, are reported improvements over SV4D.Evaluations cover synthetic datasets and real-world videos, including DAVIS.
2. Related Work
Prior 3D and 4D methods use distilled generative priors, direct reconstruction, or unified synthetic-data training. SV4D 2.0 extends this direction with multi-view video generation and architectural mechanisms for spatial-temporal consistency.
- 3D Generation: 3D-generation methods either distill priors from 2D generative models, directly predict static 3D objects, or reconstruct content from dense consistent multi-view images.SV4D 2.0 follows the multi-view reconstruction strategy but generates consistent multi-view videos instead of images.
- Video Generation: Video-generation models provide strong geometry, motion, and generalization priors because large-scale image and video datasets are more available than 3D or 4D data.SV4D 2.0 adapts a pretrained video-generation model for multi-view video generation.
- 4D Generation: SDS-based 4D methods can require hours, while inference-only pipelines struggle with satisfactory spatio-temporal consistency.Unified 4D generative models instead train on synthetic 4D data, but often remain weak on details, large motion, and self-occlusion.
- SV4D 2.0: SV4D 2.0 uses randomly masked reference latents, 3D attention, and α-blending of 3D and frame attention to combine spatial and temporal information.These mechanisms remove dependence on a separate multi-view diffusion model and support joint training on 3D and 4D data.
3. Multi-view Video Synthesis via SV4D 2.0
SV4D 2.0 synthesizes spatially and temporally consistent multi-view videos directly from a monocular video and camera trajectory, then improves sparse-view handling, motion robustness, and generalization through architectural, data, and training changes.
- SV4D 2.0 generates novel multi-view videos from a monocular video and user-specified camera trajectory, without reference multi-view conditioning.
- Network architecture: 3D attention merges view and spatial dimensions to improve correspondence across sparse views, where identical image locations may represent different 3D regions.
- Network architecture: Learnable α3D and αf weights blend 3D and frame attention, providing flexible control over spatial and temporal information.
- Network architecture: Jointly generating spatio-temporal images removes dependence on potentially conflicting reference multi-views and improves robustness to self-occlusions.
- Data curation: ObjaverseDy++ improves training data by calibrating global motion, filtering unsuitable objects, reducing baked-in lighting, and rendering denser views and longer sequences.
4. 4D Optimization from Multi-view Videos
SV4D 2.0 reconstructs dynamic NeRF-based 4D assets from synthesized multi-view videos, using staged refinement, visibility-weighted losses, and structured view and frame sampling to improve consistency and motion detail.
- Photogrammetry-based optimization learns a dynamic NeRF from SV4D 2.0 multi-view videos used as pseudo ground-truths.
- The 4D representation combines a canonical NeRF with a temporal deformation field to model density and color across space and time.
- A two-stage procedure first optimizes a dynamic NeRF, then denoises its rendered views to create more consistent and pose-aligned reconstruction targets.
- Soft visibility weights derived from camera-ray and surface-normal alignment reduce the influence of inconsistent textures in reconstruction losses.
- Progressive frame sampling and near-orthogonal view sampling stabilize optimization, improve temporal-deformation learning, and capture better motion details.
5. Experiments
Experiments evaluate SV4D 2.0 on synthetic datasets and real-world videos using image-quality and spatio-temporal consistency metrics. Across novel-view video synthesis and 4D generation, it outperforms prior methods visually and quantitatively.
- Evaluation Setup: Evaluations cover NVVS and 4D optimization on ObjaverseDy and Consistent4D, plus an in-the-wild user study on DAVIS videos.The study compares against recent methods using official implementations where available.
- Evaluation Setup: Metrics measure visual quality with LPIPS, CLIP-S, PSNR, SSIM, and MSE, alongside frame, view, diagonal, and full-grid video consistency.FV4D scans all images in bidirectional raster order, while FVD variants isolate different consistency dimensions.
- Visual Comparison: SV4D 2.0 produces sharper, more detailed, and more temporally and spatially consistent novel-view videos than the compared baselines.SV3D preserves details but lacks temporal consistency, Diffusion2 is over-smoothed, and SV4D becomes blurry during large motion.
- Quantitative Comparison: SV4D 2.0 shows consistent gains over all NVVS baselines on both synthetic datasets and across the reported image and video metrics.Lower FVD values support stronger consistency across frames, views, diagonal images, and the full image matrix.
- 4D Generation: SV4D 2.0 also achieves higher-quality 4D outputs, while visual comparisons report improved detail, consistency, and fidelity to input videos.The reported evaluation includes ObjaverseDy and Consistent4D, with state-of-the-art visual quality and temporal smoothness on Consistent4D.
- User Study: A user study on 20 DAVIS videos finds that participants preferred SV4D 2.0 for stability, realism, and resemblance to the input among four methods.The study targets real-world videos with relatively steady cameras and minimal truncation or occlusions.
6. Conclusion
SV4D 2.0 generates high-quality novel-view videos and 4D assets from monocular video, improving consistency, sharpness, generalization, and robustness over SV4D. Its architecture, data, training, and optimization changes support stronger performance in both tasks.
- SV4D 2.0 generates high-quality novel-view videos and 4D assets given a monocular video.
- Compared with SV4D, it supports longer videos and sparser camera views while improving spatio-temporal consistency, detail sharpness, and real-world generalization.
- The method combines changes in data curation, network architecture, progressive training, and 4D optimization.
- It avoids separate reference multi-views during inference, making the system more practical and robust to occlusions in the anchor frame.
Supplementary Material
The supplementary document provides implementation details, ablation analyses, additional results, and a teaser video summarizing the SV4D 2.0 framework.
- The supplement includes implementation details in Section 7, ablation analyses in Section 8, and additional results in Section 9.
- A teaser video summarizes the SV4D 2.0 framework and presents additional visual results.
7. Implementation Details
Implementation uses curated dynamic-object data, progressive 3D-to-4D training, and a two-stage optimization procedure. Ablations examine the architecture, data and training choices, refinement, and sampling strategies.
- Data Processing: Training data is rendered from CC-licensed animatable objects in Objaverse and ObjaverseXL after filtering for licensing, scaling, and motion issues.The data pipeline is illustrated in Figure 7, and off-center motion is addressed through global-local motion disentanglement.
- Training: Training samples four views and 12 frames from 24 rendered views and 24 frames, using precomputed VAE latents and CLIP embeddings.
- Ablations: Ablations evaluate reference-view conditioning, 3D attention with camera embeddings, improved data curation, and progressive 3D-4D training.
- Ablations: Ablations show that stage-2 refinement improves image quality and 3D consistency, while orthogonal-view and progressive-frame sampling slightly improve quality and temporal consistency.
- 4D Optimization: Two-stage 4D optimization uses visibility-weighted reconstruction, perceptual, mask, normal, depth, and smoothness losses with 1500 and 500 iterations.Stage 2 refines pseudo-ground-truth videos by noising and denoising renders from the stage-1 dynamic NeRF.
8. Ablative Analyses
Ablations show that SV4D 2.0’s reference conditioning, network design, curated data, progressive training, and optimization strategy each support output quality and consistency.
- Novel-view video synthesis: Removing reference multi-views substantially worsens results because the model loses a condition that maintains temporal coherence between generations.
- Novel-view video synthesis: Replacing 3D attention with view attention and camera embeddings with view indices worsens image quality and multi-view consistency, especially for sparse novel views.
- Training strategy: Improved data curation strengthens SV4D 2.0’s performance on ObjaverseDy.
- Training strategy: Progressive 3D-to-4D training is evaluated as an important design choice against direct training on 4D data.
- 4D optimization: Stage-2 refinement significantly improves all image and video metrics, while orthogonal views and progressive frame sampling improve image quality and temporal consistency over random sampling.
- Novel-view video synthesis: Reference multi-views can conflict with later video frames, producing blurry outputs; SV4D 2.0 instead leverages input-frame information for sharper details.
- Limitations: The camera parameterization limits the model’s ability to handle videos with strong perspective distortion and restricts applicability to object-centric videos.
9. Additional Results
Additional results compare SV4D 2.0 with prior systems across occlusion, real-world novel-view synthesis, and large-motion 4D optimization, with visual gains in fidelity and detail.
- Robustness to reference conflicts: Conflicting reference multi-views can merge inconsistent information from the input video and cause blurry details in SV4D outputs.
- Robustness to occlusion: Multiple input frames make SV4D 2.0 more robust to self-occlusion than SV4D’s first-frame-conditioned reference-view generation.
- Novel-view video synthesis: SV4D 2.0 produces higher-fidelity details and better generalization than SV4D on real-world DAVIS videos.
- 4D optimization: SV4D 2.0 captures higher-quality texture and geometry details during large motion in 4D outputs compared with SV4D.
- Representation comparison: 4D Gaussians suffer from temporal flickering and blurry artifacts, whereas Dy-NeRF interpolates better across sparse views and fast motion.
- Comparison with L4GM: L4GM generalizes poorly to real-world data and struggles with non-zero-elevation videos, while SV4D 2.0 is evaluated on both settings.
- Efficiency: Generating four 12-frame videos at 512 × 512 takes roughly one minute on an H100, while 4D optimization takes 21 minutes and requires up to 30GB memory.