Source-linked AI summary
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, Varun Jampani
TL;DR
Single-image 3D reconstruction is challenging because novel views can be limited or inconsistent. SV3D adapts image-to-video diffusion with explicit camera control and adds optimization techniques for 3D generation. The paper reports state-of-the-art performance on novel-view synthesis and 3D generation compared with prior methods.
Problem
Single-image 3D reconstruction requires inferring unseen 3D regions from 2D pixels, while existing generation-based methods can produce inconsistent novel views.
Method
SV3D adapts a latent video diffusion model with camera-pose conditioning for multi-view synthesis and uses masked SDS, disentangled illumination, and other techniques for 3D optimization.
Results
SV3D achieves state-of-the-art performance on novel multi-view synthesis and 3D generation, with consistent and generalizable views and detailed 3D meshes.
Takeaways & Limitations
SV3D provides a controllable, multi-view-consistent, and generalizable video-diffusion foundation for image-to-3D generation.
Takeaways & Limitations
Naively adding SDS can destabilize training and produce unfaithful textures, including oversaturation or blurry artifacts.
Abstract
from arXiv · showhide
We present Stable Video 3D (SV3D) -- a latent video diffusion model for high-resolution, image-to-multi-view generation of orbital videos around a 3D object. Recent work on 3D generation propose techniques to adapt 2D generative models for novel view synthesis (NVS) and 3D optimization. However, these methods have several disadvantages due to either limited views or inconsistent NVS, thereby affecting the performance of 3D object generation. In this work, we propose SV3D that adapts image-to-video diffusion model for novel multi-view synthesis and 3D generation, thereby leveraging the generalization and multi-view consistency of the video models, while further adding explicit camera control for NVS. We also propose improved 3D optimization techniques to use SV3D and its NVS outputs for image-to-3D generation. Extensive experimental results on multiple datasets with 2D and 3D metrics as well as user study demonstrate SV3D's state-of-the-art performance on NVS as well as 3D reconstruction compared to prior works.
1. Introduction
Single-image 3D reconstruction is difficult because it must infer unseen 3D structure from 2D pixels, while existing generative approaches often produce inconsistent novel views. SV3D adapts a pose-controlled video diffusion model for consistent novel-view synthesis and uses those views for improved 3D generation.
- Single-image 3D reconstruction must lift 2D pixels into 3D while reasoning about unseen object regions.The problem has applications across game design, AR/VR, e-commerce, and robotics.
- Limited multi-view consistency in generative models leads to inconsistent novel views and unsatisfactory geometric and texture details.Some methods jointly reason about 3D representations, but require high computational and data resources.
- Novel Multi-view Synthesis: SV3D adapts Stable Video Diffusion for pose-controlled novel-view synthesis from a single image.The model is designed to provide pose controllability, multi-view consistency, and generalization for 3D object generation.
- 3D Generation: SV3D-guided 3D generation optimizes a NeRF and DMTet mesh coarse-to-fine, adding masked SDS and disentangled illumination to improve unseen regions and reduce baked-in lighting.These techniques use SV3D novel views while addressing missing details and lighting effects during optimization.
- SV3D produces more consistent, generalizable, and pose-controllable novel views, while its 3D meshes capture intricate geometric and texture details.The paper reports considerably better NVS and 3D-generation outputs than respective state-of-the-art methods.
2. Background
Prior novel-view synthesis methods are organized around generalization, controllability, and multi-view consistency, but existing approaches do not fully exploit video diffusion models. SV3D addresses this gap with controllable, high-resolution multi-view synthesis for subsequent 3D generation.
- Novel-view synthesis research emphasizes generalization, controllability, and multi-view consistency.These properties frame the comparison of existing and proposed methods.
- Image-diffusion NVS methods can condition on an input image and pose, but related multiview methods may require post-processing or incorporate 3D representations.The background discusses Zero123, Zero123XL, Stable Zero123, Consistent123, Wonder3D, SyncDreamer, and ConsistNet.
- Exploiting Video Diffusion Models: Video diffusion models are used to improve generalization and multi-view consistency, including approaches combining view-conditioned and video diffusion models.SVD-MV and IM-3D are described as video-diffusion-based approaches.
- SV3D targets the gap that existing methods do not fully leverage video diffusion models' generalization, controllability, and consistency.The proposed model operates at 576×576 resolution and is leveraged for 3D generation.
- Compared with prior diffusion-guided 3D-generation methods, SV3D produces denser, controllable, and more consistent multi-view images for 3D generation.The paper presents this as the distinction between SV3D and related work.
3. SV3D: Novel Multi-view Synthesis
SV3D repurposes video diffusion’s temporal consistency for controllable, high-resolution novel-view synthesis from a single image. It evaluates static and dynamic camera orbits and reports strong reconstruction quality and user preference against prior methods.
- Approach: SV3D fine-tunes SVD to generate orbital videos conditioned on a single image and arbitrary camera trajectories.The target orbit contains 21 multiview images specified by elevation and azimuth angles.
- Architecture: Camera elevation, azimuth, and diffusion timestep embeddings condition the UNet, while image latent and CLIP embeddings condition its latent and attention pathways.The camera embeddings are combined with the noise-step embedding for residual blocks.
- Training and evaluation: SV3D trains on static and dynamic orbits, with dynamic orbits evaluating controllability and all metrics evaluating multiview consistency.Training uses rendered 21-frame object orbits from Objaverse, while evaluation includes unseen GSO and OmniObject3D datasets.
- Model improvements: Triangle CFG scaling addresses oversharpening near the end of looping orbits caused by SVD’s linearly increasing guidance scale.The paper also uses progressive static-to-dynamic finetuning and additional 3D-generation techniques.
- Results: SV3D achieves state-of-the-art novel multiview synthesis, with better per-frame reconstruction quality and more detailed, faithful, and consistent views than prior methods.The pose-conditioned progressive model SV3Dp is reported as the best-performing SV3D variant overall.
- Results: Users preferred SV3D videos over Zero123XL, Stable Zero123, EscherNet, and Free3D in 96%, 99%, 96%, and 98% of comparisons, respectively.The study used 30 users and 22 real-world images.
4. 3D Generation from a Single Image Using SV3D
SV3D outputs serve as multi-view targets or diffusion guidance for coarse-to-fine 3D reconstruction, with masked SDS and disentangled illumination improving unseen regions and texture quality. The resulting meshes achieve better 2D and 3D metrics than prior methods, while reflective surfaces remain a limitation.
- 3D Reconstruction Strategy: SV3D-generated static or dynamic orbital samples can directly supervise reconstruction, or guide NeRF and DMTet optimization with SDS.The pipeline treats generated images as multi-view pseudoground truth and applies photometric reconstruction losses or diffusion guidance.
- 3D Reconstruction Strategy: A coarse-to-fine pipeline trains an Instant-NGP NeRF, extracts a mesh with marching cubes, and fine-tunes it using DMTet and full-resolution SDS guidance.The coarse stage reconstructs SV3D images without SDS; the fine stage applies SDS-based guidance before UV unwrapping and export.
- Disentangled Illumination: A 24-Spherical-Gaussian illumination model separates lighting from diffuse color, reducing baked-in illumination and enabling more convincing relighting.The model fits illumination jointly with reconstruction using rendered-image losses and Lambertian shading.
- Experiments and Results: Dynamic training orbits are crucial for complete generations from diverse views, while static and dynamic orbit evaluations assess novel-view synthesis.The experiments compare generated views and meshes against prior methods using image-based and 3D reconstruction metrics.
- Masked SDS: Masked SDS applies guidance primarily to unseen or occluded surfaces, preserving visible textures while inpainting missing details and improving convergence speed.Visibility is estimated from surface normals and reference-camera directions, producing a soft mask for the SDS loss.
- Limitations: Reflective surfaces can remain view-inconsistent and are not represented by the Lambertian shading model.The authors identify full camera-matrix conditioning and richer shading as directions for future work.
5. Conclusion
The conclusion presents SV3D as a high-resolution latent video diffusion model for controllable, multi-view-consistent novel-view synthesis and 3D generation. It combines camera conditioning with several 3D optimization techniques and reports state-of-the-art performance.
- Conclusion: SV3D leverages SVD’s generalizability and view-consistent prior while adding camera-pose conditioning for controllable orbital videos.The model generates high-resolution orbital videos on arbitrary camera orbits.
- Conclusion: Triangle CFG scaling, disentangled illumination, and masked SDS are proposed to improve 3D generation from SV3D outputs.The conclusion positions these techniques as additions to the core SV3D model.
- Conclusion: Extensive experiments report that SV3D is controllable, multi-view consistent, generalizable to real-world images, and state of the art on novel-view synthesis and 3D generation.The authors describe SV3D as a foundation model for further 3D object-generation research.
A. Broader Impact
The broader-impact discussion addresses risks from generated 3D content, including training-data issues and the model’s ability to infer hidden details from input images. User-provided views constrain generation, but mislabeled or sensitive training content remains possible.
- Motivation: Generative 3D models can produce innovative content but require attention to potential downsides and mitigation plans.The discussion frames risk management as important before widespread use of generative models.
- Input Constraints: A full front view constrains creative freedom to visible or unoccluded regions, while predicted depth and lighting influence fidelity and realism.These input factors affect the model’s generated output.
- Training Data: The training process excludes Sketchfab objects tagged as Restricted Content, including PG-13 material.The authors report taking this additional filtering step beyond the platform’s content policies.
- Training Data: Sketchfab labels may be incomplete, although the platform provides public listings and community reporting for potentially offensive content.The authors identify incorrect uploader labeling as a residual data-quality risk.
- Training Data: Objaverse contains occasional doll-like nudity, drugs, blood, and weaponry, and SV3D can generate corresponding 3D models to some extent when given such imagery.The authors characterize these categories as present in the training dataset at some level.
- Model Behavior: When input regions are unclear or hidden, SV3D guesses their details from training data, which may produce outputs resembling existing modeling trends.Clear, straightforward input images constrain creativity and reduce the chance of biases appearing.
B. Data Details
The training data consists of curated Objaverse objects rendered into static and dynamic orbital videos with varied backgrounds, lighting, and camera trajectories. These renders are encoded with image and text-free visual representations plus camera metadata.
- Objects: The dataset uses 150K curated CC-licensed Objaverse objects, normalized by bounding-box scale and centered at the world origin.Each object is positioned consistently before rendering.
- Rendering: Each object is rendered as an 84-frame RGBA orbit at 576×576 resolution, with 21-frame training sequences sampled by stride four.Any frame can serve as the first frame of a sampled orbit.
- Rendering: Training doubles orbit samples by rendering each object with random RGB and white backgrounds, then storing SVD VAE latents, CLIP embeddings, and elevation-azimuth metadata.The stored representations pair visual encodings with camera information.
- Camera and Lighting: Lighting is randomized across 20 curated HDRI environment maps, while cameras use a 33.8-degree field of view and adaptive distance.Adaptive positioning keeps object content consistently within the image extents without clipping.
- Camera Trajectories: Static orbits sample elevation from [-5, 30] degrees and advance azimuth regularly, whereas dynamic orbits vary elevation through weighted sinusoids and perturb azimuth with noise.Dynamic elevations use randomized periods, amplitudes, and phase shifts before smoothing.
C. Training Details
SV3D training adapts the EDM framework with a simplified diffusion loss and removes video-specific conditions that are irrelevant to novel-view synthesis. Inference generates orbital videos with deterministic DDIM sampling in approximately 40 seconds.
- SV3D uses the EDM framework with a simplified diffusion loss for finetuning.
- The model removes the fps id and motion bucket id conditions because they are irrelevant to SV3D.
- Training assigns lower weights to frames near the front-view conditioning image so challenging back views receive equal focus.
- 50-step deterministic DDIM sampling generates an orbital video in approximately 40 seconds.
E. Additional Details on Illumination Model
The illumination model represents white-light environment illumination with spherical Gaussians and Lambertian shading, then learns illumination parameters from SV3D-generated images. Reconstruction combines these components with photometric, perceptual, mask, normal, depth, bilateral, albedo, and SDS-related losses.
- Illumination Model: Spherical Gaussians represent white-light illumination using an axis, sharpness, and scalar amplitude.
- Illumination Model: Lambertian cosine shading is approximated with another spherical Gaussian whose axis is the surface normal.
- Illumination Model: Lighting is evaluated by integrating each illumination Gaussian with the cosine Gaussian, summing the results, and multiplying by diffuse albedo.
- Illumination Model: The model learns illumination Gaussian parameters by reconstruction loss between rendered images and SV3D-generated images.
- Losses and Optimization: 3D reconstruction uses pixel MSE, LPIPS, mask, normal, depth, bilateral smoothness, albedo smoothness, masked SDS, and illumination losses.
- Losses and Optimization: Both coarse and fine stages use the losses, but masked SDS is applied only during the final 200 fine-stage iterations.
G.2. Static v.s. Dynamic Orbits
Dynamic camera orbits improve 3D reconstruction when elevation varies moderately, while extreme elevation increases multi-view inconsistency. Soft-masked SDS generally provides the strongest results among the evaluated SDS variants.
- Static v.s. Dynamic Orbits: Dynamic sine-30 orbits produce better 3D outputs than static orbits because they include more top and bottom views.
- Model and Loss Ablations: SV3Dp with Photo+SDS losses achieves the best 2D and 3D metrics among the compared SV3D models and training losses.
- Static v.s. Dynamic Orbits: Higher elevation in sine-50 increases multi-view inconsistency and produces worse 3D reconstruction.
- Static v.s. Dynamic Orbits: Elevation within ±30 degree generally leads to desirable 3D outputs.
- Masked SDS Loss: Adding SDS guidance improves 2D metrics while maintaining similar 3D metrics compared with pure photometric losses.
- Masked SDS Loss: Soft-masked SDS generally achieves the best results among naive, hard-masked, and soft-masked SDS variants.
H. Additional Visual Results
Additional results on OmniObject3D and real-world images show SV3D-generated novel views that are more detailed and consistent than prior state-of-the-art methods.
- SV3D produces more detailed and consistent novel multi-view images than prior state-of-the-art methods.The additional results cover OmniObject3D and real-world images.
- Additional NVS results are presented for OmniObject3D and real-world images in Figures 13 and 14.
H.2. 3D Generation
SV3D produces detailed, high-fidelity 3D generations from consistent multi-view images on OmniObject3D and real-world inputs, including diverse in-the-wild images.
- SV3D generates detailed, high-fidelity 3D objects from OmniObject3D and real-world images.The results are shown in Figures 15 and 16.
- SV3D’s consistent novel views support 3D generation across OmniObject3D and real-world images.The reported pipeline uses SV3D-generated multi-view images for reconstruction.
- The reconstructions capture accurate shape and fine geometric and texture details across diverse inputs.The figure descriptions emphasize accurate shape and details, including for real-world images in-the-wild.