Source-linked AI summary
SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, Varun Jampani
TL;DR
4D generation from a single monocular video requires consistent unseen views and motion, but prior methods separate these capabilities and often rely on costly SDS optimization. SV4D unifies video and multi-view diffusion to generate temporally consistent novel-view videos, then fits a dynamic NeRF without SDS. The method achieves state-of-the-art novel-view video synthesis and 4D generation across multiple datasets.
Problem
4D generation from one video must infer appearance and motion at unseen views, while existing methods separately model video generation and novel-view synthesis.
Method
SV4D uses a unified diffusion model to generate a view-by-frame grid of novel views, then optimizes a dynamic 3D representation from those videos without SDS losses.
Results
SV4D achieves state-of-the-art performance on novel-view video synthesis and 4D generation across multiple datasets.
Takeaways & Limitations
Consistent generated novel-view videos can serve as inputs for efficient 4D asset optimization without SDS-based optimization.
Abstract
from arXiv · showhide
We present Stable Video 4D (SV4D), a latent video diffusion model for multi-frame and multi-view consistent dynamic 3D content generation. Unlike previous methods that rely on separately trained generative models for video generation and novel view synthesis, we design a unified diffusion model to generate novel view videos of dynamic 3D objects. Specifically, given a monocular reference video, SV4D generates novel views for each video frame that are temporally consistent. We then use the generated novel view videos to optimize an implicit 4D representation (dynamic NeRF) efficiently, without the need for cumbersome SDS-based optimization used in most prior works. To train our unified novel view video generation model, we curate a dynamic 3D object dataset from the existing Objaverse dataset. Extensive experimental results on multiple datasets and user studies demonstrate SV4D's state-of-the-art performance on novel-view video synthesis as well as 4D generation compared to prior works.
1 INTRODUCTION
SV4D addresses 4D generation from a single monocular video by jointly modeling unseen views and object motion. It produces consistent novel-view videos and uses them to optimize dynamic 3D assets without cumbersome SDS optimization.
- Motivation: 4D generation from one video must infer both appearance and motion at unseen camera views, while remaining ill-posed and data-hungry.The paper identifies the lack of large-scale 4D training data and the high dimensionality of shape, appearance, and motion as central challenges.
- Approach: Given a monocular video and camera trajectory, SV4D outputs a V×F grid of images spanning specified views and video frames.Reference multi-views for the first frame are obtained with SV3D, after which SV4D generates the remaining grid jointly.
- 4D optimization: SV4D-generated novel-view videos are used directly to optimize dynamic 3D objects, avoiding the cumbersome SDS loss used by prior optimization-based methods.The pipeline sequentially processes large video inputs and then optimizes a 4D representation from the generated videos.
- Approach: The model combines video and multi-view diffusion priors through view and frame attention to improve dynamic and multi-view consistency.View attention aligns views within each frame, while frame attention aligns frames within each view.
- Results: The paper introduces ObjaverseDy and reports state-of-the-art results for novel-view video synthesis and 4D generation across multiple benchmark datasets.Evaluations cover synthetic and real-world data and include a modified FVD metric for frame and view consistency.
- Approach: SV4D jointly reasons across frame and view axes using a single novel-view video synthesis network trained on 4D data.This unified design replaces separate modeling of video motion and novel-view synthesis.
2 RELATED WORK
Prior work spans static 3D generation, video generation, and 4D generation. Existing 4D approaches either distill diffusion priors through costly optimization or use other reconstruction strategies, motivating unified dynamic multi-view modeling.
- 3D Generation: Static 3D generation methods distill 2D diffusion priors, directly predict 3D models, or reconstruct objects from dense multi-view images.These approaches target static content rather than jointly modeling object motion and view variation.
- Video Generation: Video generation models benefit from large-scale image and video data and are used as foundation models because such data are easier to collect than large-scale 3D or 4D data.Their learned priors support realistic motion and geometry consistency.
- 4D Generation: Optimization-based 4D methods distill pretrained diffusion models with SDS loss but can take hours to generate 4D content.This contrasts with approaches that seek more direct or efficient 4D reconstruction and generation.
3 METHOD
SV4D combines video and multi-view diffusion priors in a unified network that generates a view-by-frame image matrix, extends it with mixed sampling, and fits a dynamic NeRF from the generated views.
- Network design: SV4D combines frame consistency from video diffusion with multi-view consistency from multi-view diffusion.The design uses SVD and SV3D priors within an attention-based architecture.
- Network design: SV4D uses spatial, view, and frame attention in a UNet to align semantic content across views and dynamic content across frames.View attention conditions on corresponding input frames, while frame attention conditions on the corresponding first-frame reference view.
- Problem setting: The model generates an image matrix M ∈ R^(V×F×D) conditioned on a monocular video and first-frame reference multi-views along a camera trajectory.The target contains V camera views and F dynamic frames.
- Inference sampling: Mixed sampling first generates sparse anchor frames, then densely samples intermediate frames using alternating forward and backward conditioning.The strategy addresses memory limits and reduces inconsistencies between consecutive submatrix generations.
- Training data: ObjaverseDy is curated from Objaverse by filtering licenses, animation length, motion, and rendering settings for dynamic-object training data.The dataset addresses the lack of large-scale dynamic 3D datasets.
- 4D representation: The 4D representation uses a canonical NeRF, deformation field, multi-resolution hash grid, and MLPs to map spatial points and time to density and color.Optimization begins from first-frame multi-view images before jointly fitting temporal deformation with sampled views and frames.
4 EXPERIMENTS
SV4D is evaluated on synthetic and real-world data using image-quality, video-consistency, multi-view-consistency, 4D-consistency, visual, and user-preference measures. Across these evaluations and ablations, SV4D generally outperforms prior methods and produces more detailed, faithful, and consistent results.
- Quantitative comparison: Evaluations use ObjaverseDy, Consistent4D, and DAVIS, with LPIPS, CLIP-S, FVD, and modified FVD metrics covering visual quality and multiple consistency dimensions.FVD-F measures frame coherence, FVD-V multiview consistency, and FVD-Diag and FV4D 4D consistency.
- Quantitative comparison: 31.5% and 21.4% reductions in FVD-F are achieved versus SV3D and STAG4D, respectively, on Consistent4D.On ObjaverseDy, SV4D also achieves better FVD-V, FVD-Diag, and FV4D while maintaining comparable image quality.
- 4D generation: SV4D consistently outperforms baselines across LPIPS, CLIP-S, FVD-F, FVD-V, FVD-Diag, and FV4D for 4D outputs.These metrics cover visual quality, frame consistency, multiview consistency, and joint motion-multiview consistency.
- Visual comparison: SV4D produces detailed, faithful, temporally consistent multiview videos, while baselines show inconsistent geometry, blur, flicker, or failures with large motion.The visual comparisons report these differences across SV3D, Diffusion2, and STAG4D.
- Visual comparison: SV4D optimizes smoother 4D assets with realistic, faithful appearance using photometric and geometric losses rather than SDS losses.Prior methods using SDS are described as prone to spatial incoherence and oversaturated texture.
- User study: Users preferred SV4D in 73.3% of multiview-video comparisons and 60% of optimized-4D comparisons.The comparisons included real-world DAVIS videos and synthetic videos against three baselines.
- Ablative analyses: SV4D sampling outperforms AMT and alternative independent or interleaved strategies across all reported metrics.Visual results show faithful motion and details for SV4D sampling, versus blur or missing parts for interpolation.
5 CONCLUSION
The paper presents SV4D as a unified latent video diffusion model for novel-view video synthesis and 4D generation. Its generated novel-view videos support efficient 4D optimization without SDS losses, with reported state-of-the-art consistency and generalization to real-world videos.
- Conclusion: SV4D generates dynamically and spatially consistent novel-view videos by combining video and multiview priors in one architecture.The model uses dynamic priors from SVD and multiview priors from SV3D.
- Conclusion: Generated novel-view videos enable efficient 4D-asset optimization without SDS losses from one or multiple diffusion models.The conclusion describes this as an alternative to SDS-based optimization.
- Conclusion: Experiments report state-of-the-art performance on novel-view video synthesis and 4D generation, including generalization to real-world videos.The reported consistency covers multiple frames and views.
A.1 DATA DETAILS
The ObjaverseDy data pipeline curates CC-licensed animatable objects, renders multiview dynamic sequences under sampled camera trajectories and lighting, and stores latent and conditioning representations for training.
- Dataset construction: ObjaverseDy is built from a curated subset of CC-licensed animatable Objaverse objects, with some objects held out for evaluation.The dataset is intended to provide dynamic 3D objects for model training and evaluation.
- Rendering: Objects are normalized to unit maximum world-space bounding-box extent and rendered with Blender Cycles using sampled views, frames, and HDRI environment maps.The rendering setup limits Cycles samples to reduce rendering time.
- Camera trajectories: Static and dynamic camera orbits use regularly sampled azimuths, while dynamic elevations are generated from weighted sinusoidal combinations and smoothed.Small azimuth noise makes dynamic trajectories irregular.
- Data representation: Rendered images are encoded with SD2.1’s VAE and CLIP, then stored with camera elevations, azimuths, and frame indices.These precomputed representations accompany the images in the dataset.
A.2 SV4D TRAINING DETAILS
SV4D training uses an EMD-based objective with L2 fine-tuning and precomputed conditioning tensors, while inference uses deterministic DDIM and interleaved anchor-to-dense sampling.
- Training: Training uses EMD with an L2 loss, following the training objectives used in SVD and SV3D.The L2 objective is applied during fine-tuning.
- Training: Precomputed latent and CLIP embeddings are loaded during training to improve efficiency and reduce GPU VRAM use.The tensors are stored in advance rather than computed in real time.
- Inference: Inference uses 50 deterministic DDIM steps and SV4D sampling to generate the full V×F image matrix.The example uses V = 8 views and F = 21 frames.
- Inference: Interleaved sampling first generates sparse anchor frames at indexes [4, 8, 12, 16, 20], then uses them as references for dense sampling.This procedure extends sparse anchors to the remaining frames.
A.4 SV4D NETWORK ARCHITECTURE
SV4D uses sequential spatial, view, and frame attention layers to model coherence across image locations, camera views, and video frames. Its sampling strategies process submatrices separately while preserving consistency for arbitrary-length videos.
- Attention layers: SV4D reshapes features for spatial, view, and frame attention, modeling coherence across image locations, views, and frames.Spatial attention operates within each image; view attention links corresponding locations across views; frame attention links corresponding locations across frames.
- Sampling strategies: Independent and interleaved sampling separately generate consecutive or interleaved frame submatrices without relying on previous fully sampled images.
- View attention: View attention models corresponding spatial locations across views using features arranged over the view dimension.
- Frame attention: Frame attention models corresponding spatial locations across frames using features arranged over the frame dimension.
- Feature dimensions: The experiments use 8 views and 5 frames in the described feature arrangement.
- Attention integration: Frame attention is skip-connected to view-attention outputs with learnable per-layer blending weights to merge spatial and temporal information.The sequential design is reported to leverage SVD and SV3D priors with minimal computation overhead.
A.5 DYNAMIC NERF OPTIMIZATION DETAILS
Dynamic NeRF optimization combines photometric, perceptual, mask, normal, depth, and bilateral smoothness losses to reconstruct 4D assets from generated views. The visual comparisons emphasize consistency and detail, while sampling addresses memory limits when generating the full view-frame matrix.
- Reconstruction losses: The reconstruction objective combines pixel-level MSE, LPIPS, mask, normal, depth-smoothness, and bilateral smoothness losses.The losses regularize appearance, masks, surface normals, depth, and high-frequency geometry.
- Geometric regularization: Bilateral smoothness encourages rendered normal gradients to be smooth where input-image gradients are smooth, helping preserve high-frequency details and avoid over-smoothed surfaces.
- Optimization: The losses are applied in both static and dynamic stages using Adam with a learning rate of 0.01.
- Sampling: Because the full view-frame matrix may exceed GPU memory, the method compares independent and interleaved submatrix sampling strategies.
- Visual results: The generated 4D outputs are described as more detailed, consistent, and faithful to input videos than prior SDS-based works.
B ADDITIONAL RESULTS
Additional visual comparisons report that SV4D-generated images and 4D results are more consistent, detailed, and faithful to the conditioning videos than prior works.
- Visual comparisons: SV4D-generated images and 4D results are more consistent, detailed, and faithful to conditioning videos than prior works.
B.2 MORE ABLATIVE RESULTS
Additional results examine temporal consistency across views and architectural choices, while comparing SV4D against prior novel-view and 4D-generation methods on benchmark datasets. The ablation reports that reference multi-view VAE latents for frame conditioning perform better in most metrics than the tested alternatives.
- Architectural ablations: The ablation compares removing frame attention, removing multi-view conditioning, and replacing reference multi-view VAE latents with CLIP embeddings.
- Ablation outcome: Using reference multi-view VAE latents for frame-attention conditioning achieves better performance in most metrics than the other tested design choices.
- Temporal consistency: SV4D produces better temporal consistency in almost all views on the ObjaverseDy dataset.
- Novel-view baselines: Novel-view video synthesis comparisons include SV3D, Diffusion2, STAG4D, and 4Diffusion.
- 4D-generation baselines: 4D-generation comparisons include Consistent4D, STAG4D, 4Diffusion, DreamGaussian4D, GaussianFlow, 4DGen, and Efficient4D.