Source-linked AI summary
Stable Virtual Camera: Generative View Synthesis with Diffusion Models
Jensen Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, Varun Jampani
TL;DR
Existing NVS methods are limited in handling large viewpoint changes, temporal smoothness, flexible task configurations, and sampling consistency. SEVA uses a diffusion architecture, structured training, and two-pass sampling to address these constraints, and extensive benchmarks report stronger performance across diverse settings, including +1.5 dB PSNR over CAT3D in its own setup.
Problem
Existing NVS methods struggle with large viewpoint changes, temporal smoothness, flexible input-target configurations, and consistent sampling across diverse tasks.
Method
SEVA combines a diffusion-based architecture without explicit 3D representations, viewpoint-diverse training, and two-pass procedural sampling.
Results
+1.5 dB PSNR over CAT3D is reported in CAT3D’s own setup, alongside generalization to user captures with 1 to 32 input views.
Takeaways & Limitations
SEVA provides a generalist NVS system for diverse tasks, including high-quality videos lasting up to half a minute with precise camera control and seamless loop closure.
Takeaways & Limitations
Performance is constrained by training-data scope, with degraded outputs for humans, animals, dynamic textures, ambiguous scenes, and complex or highly divergent trajectories.
Abstract
from arXiv · showhide
We present Stable Virtual Camera (Seva), a generalist diffusion model that creates novel views of a scene, given any number of input views and target cameras. Existing works struggle to generate either large viewpoint changes or temporally smooth samples, while relying on specific task configurations. Our approach overcomes these limitations through simple model design, optimized training recipe, and flexible sampling strategy that generalize across view synthesis tasks at test time. As a result, our samples maintain high consistency without requiring additional 3D representation-based distillation, thus streamlining view synthesis in the wild. Furthermore, we show that our method can generate high-quality videos lasting up to half a minute with seamless loop closure. Extensive benchmarking demonstrates that Seva outperforms existing methods across different datasets and settings. Project page with code and model: https://stable-virtual-camera.github.io/.
1. Introduction
NVS seeks realistic, 3D-consistent images from arbitrary viewpoints, but existing methods struggle with viewpoint changes, temporal smoothness, flexible configurations, and consistency. SEVA addresses these challenges with a diffusion model, tailored training and sampling, and broad benchmarking.
- NVS generates realistic, 3D-consistent scene images from arbitrary camera viewpoints using camera-posed input views.
- Existing methods struggle to combine large viewpoint changes, temporally smooth samples, flexible input-target configurations, and consistent sampling.These limitations often lead to additional NeRF distillation for coherent representations.
- SEVA is a diffusion-based NVS model that supports varying numbers of input and target views without NeRF distillation.It targets both large viewpoint changes and temporal smoothness with a single network.
- SEVA combines a 2D-prior-compatible architecture, viewpoint-diverse training selection, and two-pass procedural sampling for arbitrary camera trajectories.The design avoids explicit 3D representations while supporting flexible input-target configurations.
- +1.5 dB PSNR over CAT3D is reported in CAT3D’s own setup, while SEVA generalizes to user captures with 1 to 32 input views.The unified benchmark covers 10 datasets and varied experimental settings.
- The contributions include joint modeling of viewpoint changes and temporal smoothness, procedural sampling for long trajectories, comprehensive benchmarking, and open-source model weights.
2. Background
NVS evaluation centers on generation capacity, interpolation smoothness, and input flexibility across set and trajectory tasks. Regression and diffusion models exhibit complementary strengths, motivating a single model designed for both.
- NVS models are evaluated by generation capacity, interpolation smoothness, and the ability to handle variable input and target views.
- Types of NVS Tasks: NVS predicts target images from input images and their cameras, with sparse-view inputs defined as up to 8 views and semi-dense inputs bridging sparse and dense captures.
- Types of NVS Tasks: Set NVS uses unordered target views across potentially large spatial ranges, emphasizing generation capacity for unseen regions.
- Types of NVS Tasks: Trajectory NVS produces ordered views along smooth camera paths as video, emphasizing interpolation smoothness and non-flickering results.
- Related Models: Regression-based models deterministically map inputs and cameras to target views, while persistent 3D representations can serve both set and trajectory NVS.
- Diffusion-based Models: Image diffusion models generally suit unordered set NVS, whereas video diffusion models suit trajectory NVS but require ordered inputs followed by targets.This specialization limits the flexibility of existing diffusion-based approaches.
- Remarks and Motivation: SEVA’s design targets high generation capacity, smooth interpolation, and flexible input conditioning in one model for both tasks.
3. Method
SEVA combines a standard multi-view diffusion model with a two-stage training curriculum and flexible procedural sampling. Its sampling strategies support variable input-target configurations while improving consistency across views and trajectories.
- Model and sampling formulation: SEVA uses a standard “M-in N-out” multi-view diffusion model during training and a variable-length “P-in Q-out” renderer during sampling.P and Q need not equal the training values M and N.
- Model and sampling formulation: The model conditions on CLIP embeddings, input-view VAE latents, and corresponding camera poses, with 3D and view-axis attention added to the U-Net.An optional temporal pathway uses 3D convolutions when frames in a forward pass are spatially ordered.
- Training strategy: SEVA trains in two stages, using context lengths T = 8 and T = 21, while randomly varying the number and selection of input frames.The stages run for 100K and 600K iterations, respectively, with batch sizes 1472 and 512.
- Procedural sampling: When P + Q > T, procedural two-pass sampling first generates anchor frames and then produces target chunks conditioned on spatially distributed anchors.For P + Q ≤ T, SEVA fits the task into one padded forward pass; for larger tasks, target views are split across passes.
- Procedural sampling: For set NVS, nearest procedural sampling assigns target chunks to spatially nearest anchors, while appending input views performs better and becomes the default.Without trajectory priors, the method reverts to one-pass sampling.
- Procedural sampling: For trajectory NVS, interpolation sampling generates uniformly spaced anchors and synthesizes remaining targets as segments between them.The method uses a stride Δ = ⌊Q / (T − 2)⌋ to span the target camera path.
- Sampling results: The interpolation strategy drastically outperforms one-pass and gt + nearest procedural sampling in temporal smoothness.
- Scaling sampling: When Q ≫ T, a memory bank retrieves spatially nearest previously generated anchors to improve long-term 3D consistency.This autoregressive anchor generation addresses inconsistency caused by chunking anchors across separate forward passes.
4. Experiments
SEVA is evaluated across diverse set and trajectory NVS settings, showing strong performance for both viewpoint interpolation and large viewpoint changes. The experiments also examine temporal sampling, flexible context lengths, resolution generalization, and diverse unseen-area generation.
- Set NVS: SEVA achieves state-of-the-art results in most small-viewpoint NVS splits, including a +6.0 dB PSNR gain on LLFF with P = 3.In semi-dense settings, it remains competitive with specialized models, trailing [18] by only 1.7 dB on T&T with P = 32.
- Set NVS: SEVA underperforms in RealEstate10K single-view NVS because unit-normalized cameras and diverse training scales create scale ambiguity without a second view.Sweeping camera normalization from 0.1 to 2.0 addresses this evaluation issue for P = 1.
- Set NVS: SEVA’s advantage is more prominent in large-viewpoint NVS, improving over CAT3D by +0.6 dB PSNR on Mip360 with P = 3.It also obtains a clear performance lead on harder DL3DV and T&T configurations.
- Trajectory NVS: The temporal pathway and interpolation sampling improve trajectory quality, while one-pass sampling produces visible flickering and abrupt visual changes.Interpolation produces the smoothest transitions, and the temporal pathway further boosts performance and smoothness.
- Discussions: One-pass zero-shot context extension improves with additional input frames in semi-dense settings but fails in sparse settings by producing blurry samples.The model also generalizes from square 576 × 576 training images to portrait and landscape resolutions during sampling.
5. Conclusion
SEVA is a generalist diffusion model for novel view synthesis that balances large viewpoint changes with smooth interpolation across flexible configurations. Extensive benchmarking reports 3D-consistent rendering and strong generalization to real-world scenes.
- SEVA balances large viewpoint changes and smooth interpolation while supporting flexible input and target configurations.
- A diffusion-based architecture without 3D representation, structured training, and two-pass procedural sampling produce 3D-consistent rendering across diverse NVS tasks.
- Extensive benchmarking reports superiority over existing methods and strong generalization to real-world scenes.
A. Broader Impact and Limitations
SEVA is presented as a generative model for immersive 3D experiences that synthesizes realistic, temporally consistent views from sparse camera inputs. It also supports intricate, controllable camera paths without the complexity of dense view capture.
- SEVA synthesizes realistic and temporally consistent views from sparse camera inputs for immersive 3D experiences.
- The model allows users to create intricate, controllable camera paths without typical dense-view capture complexity.
Limitations.
Prior NVS approaches divide between direct image generation and video-based synthesis, each with limitations in consistency, viewpoint range, or diversity. Diffusion-based alternatives can sample plausible solutions but may require additional distillation for consistent 3D objects.
- Directly generating target views can struggle with the diversity of generative NVS and modeling multiple plausible solutions.
- Image-based diffusion models can synthesize distant viewpoints but often lack multi-view consistency and support only specific views rather than arbitrary viewpoints.
- Some diffusion-based models require NeRF distillation to obtain consistent 3D objects from sampled images.
- Video models produce smooth sequences but are generally limited to smaller camera motions because of natural frame rates in video training.
C. Benchmark
The benchmark spans 10 public datasets and separates NVS tasks by viewpoint disparity, with distinct small- and large-viewpoint set and trajectory evaluations. It documents dataset-specific splits, metrics, and qualitative comparisons, including consistency against 3DGS renderings.
- C. Benchmark: The benchmark covers 10 publicly available datasets spanning object-level and scene-level data and diverse scene distributions and complexities.
- C. Benchmark: Tasks are grouped into small-viewpoint and large-viewpoint NVS according to the disparity between input and target views, measured using averaged CLIP distance.
- C. Benchmark: Trajectory NVS evaluates smoothly transitioning videos using all scene frames as target views and compares methods with PSNR, LPIPS, and SSIM across reported tables.
- C. Benchmark: WildRGB-D includes easy Oe scenes covering approximately 120 degrees and hard Oh scenes covering approximately 360 degrees of rotation.
- C. Benchmark: SEVA renderings closely resemble 3DGS renderings with minimal perceptual differences.
D.1. Qualitative Results
SEVA generalizes single-view conditioning across diverse camera motions and scene types, including text-prompted and real-world settings.
- SEVA adapts robustly across text-prompted object-centric, text-prompted, real-world object-centric, and real-world scenes.
D.2. Quantitative Results
The paper reports additional quantitative evaluations against baselines for set NVS and trajectory NVS using LPIPS and SSIM.
- Additional evaluations compare SEVA with baselines on set NVS and trajectory NVS using LPIPS and SSIM.Results are reported in Tables 8–10 and Appendix C.
Samples versus 3DGS.
SEVA samples are sufficiently 3D-consistent for close agreement with 3DGS renderings, while the method exhibits padding-related artifacts and saturation in difficult long-range settings.
- Samples versus 3DGS.: SEVA samples contain plausible hallucinations under high uncertainty, while 3DGS renderings remain sharp and close to them.The comparison suggests that SEVA samples are 3D consistent enough.
- Padding.: Zero-shot generalization to fewer frames than the training length causes abnormal color overflows without padding.Generalizing to substantially more frames instead produces excessive blurriness in the sparse-view regime.
- Padding.: Dynamically varying T during training could improve generalization across different sampling lengths by broadening attention-score distributions.
- Long trajectories.: Samples become increasingly saturated when target views are far from inputs and share no content overlap.The issue is particularly observed in open-ended exploration and navigation; diverse input noise is proposed as future work.
- Diverse camera motions and effects.: SEVA supports diverse camera motions and effects in text-prompted and real-world scenes.