Source-linked AI summary
Bolt3D: Generating 3D Scenes in Seconds
Stanislaw Szymanowicz, Jason Y. Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T. Barron, Philipp Henzler
TL;DR
Existing methods either generate 2D observations followed by costly 3D optimization or struggle to model ambiguity directly. Bolt3D uses latent diffusion, a Geometry VAE, and a feed-forward Gaussian head to generate explicit 3D scenes from one or more images. It produces detailed scenes in less than 7 seconds and reduces inference cost by up to 300× versus optimization-based multiview methods.
Problem
Few-view 3D generation must handle ambiguous, incompletely observed scenes while avoiding costly reconstruction procedures.
Method
Bolt3D trains a latent diffusion model to generate multiview appearance and geometry, then uses a feed-forward Gaussian head to produce explicit renderable 3D scenes.
Results
Up to 300×: Bolt3D reduces inference cost compared with multiview image-generation methods requiring per-scene optimization.
Takeaways & Limitations
Bolt3D’s direct generative output models ambiguity and enables detailed one-view reconstructions, including unobserved regions.
Takeaways & Limitations
The method struggles with thin structures fewer than 8 pixels wide and scenes containing substantial transparent or highly non-lambertian surfaces.
Abstract
from arXiv · showhide
We present a latent diffusion model for fast feed-forward 3D scene generation. Given one or more images, our model Bolt3D directly samples a 3D scene representation in less than seven seconds on a single GPU. We achieve this by leveraging powerful and scalable existing 2D diffusion network architectures to produce consistent high-fidelity 3D scene representations. To train this model, we create a large-scale multiview-consistent dataset of 3D geometry and appearance by applying state-of-the-art dense 3D reconstruction techniques to existing multiview image datasets. Compared to prior multiview generative models that require per-scene optimization for 3D reconstruction, Bolt3D reduces the inference cost by a factor of up to 300 times.
1. Introduction
Bolt3D addresses the difficulty of generating detailed, complete 3D scenes by directly producing explicit 3D representations with a latent diffusion model and multiview-consistent training data. It avoids the costly per-scene optimization used by prior multiview generation pipelines while modeling unobserved regions.
- Motivation: Generating detailed 3D scenes remains difficult because high-resolution scene representations and abundant ground-truth 3D data are lacking.Existing generative models mainly sample 2D images, while real-world 3D scene datasets are scarce.
- Motivation: Prior multiview methods sample many synthetic observations and optimize a 3D representation afterward, making inference slow and compute-intensive.The pipeline may require hundreds of augmented images followed by per-scene optimization.
- Approach: Bolt3D uses latent diffusion to generate explicit 3D scenes represented as multiple Splatter Images, including content for unobserved regions.Each grid cell stores parameters of a pixel-aligned 3D Gaussian, and the model uses more Splatter Images than input views.
- Approach: The generation process denoises Gaussian colors and positions, then predicts opacity and shape with a Gaussian head.The model jointly predicts target-view appearance and per-pixel 3D coordinates, using a geometry VAE for high-resolution geometry.
- Results: 300×: Bolt3D reduces inference cost compared with multiview image-generation methods that require per-scene optimization.The model also outperforms prior single- and few-view feed-forward 3D regression methods and synthesizes detailed content in ambiguous regions.
2. Related work
Related work follows two main strategies: feed-forward regression directly reconstructs 3D but struggles with unseen regions, while 2D generation handles ambiguity but requires costly reconstruction optimization. Bolt3D combines generative ambiguity modeling with direct 3D output.
- Feed-forward 3D regression: Feed-forward 3D regression methods associate 3D Gaussians with pixels and can accurately reconstruct observed regions.Follow-up methods add improved architectures, depth conditioning, feature matching, or remove camera-pose requirements.
- Feed-forward 3D regression: Regression-based methods tend to produce blurry results in unseen regions, whereas Bolt3D generates unobserved scene content.The distinction is between reconstructing observed evidence and generating plausible content beyond input coverage.
- Reconstruction via 2D image generation: 2D image-generation pipelines address few-view ambiguity by generating multiple views, but recovering a renderable 3D asset requires optimization.This two-stage approach can produce high-quality results but requires minutes or hours for real-world scenes.
- Reconstruction via 2D image generation: Bolt3D also uses diffusion to generate multiple views, but outputs the 3D scene directly and removes the optimization stage.Its direct output decreases inference cost by multiple orders of magnitude relative to the two-stage pipeline.
- Direct 3D generation: Earlier direct 3D generation methods denoised voxel grids or used rendering bottlenecks, with limited success beyond objects and small camera motion.These approaches were motivated partly by the difficulty of obtaining 3D supervision.
3. Preliminaries
The preliminaries define latent diffusion, 3D Gaussian scene representations, and few-view reconstruction. The task is ill-posed because a small set of posed images can be explained by many possible 3D representations.
- Latent diffusion models: Latent diffusion compresses data into a lower-dimensional latent space and learns a diffusion model there after VAE compression.The denoiser predicts a clean sample from a noisy sample using a weighted ℓ2 objective.
- 3D Gaussian representation: A 3D Gaussian scene representation stores each Gaussian’s mean, opacity, covariance matrix, and color.The Gaussians can be rendered efficiently using splatting.
- Few-view 3D reconstruction: Few-view reconstruction recovers a 3D representation from a small collection of posed images, with N ∈{1 −4} views in this setting.Each view consists of an image and its camera pose.
- Few-view 3D reconstruction: Insufficient scene coverage makes pure optimization-based reconstruction ill-posed because many 3D representations can explain the same posed images.This ambiguity motivates a generative formulation rather than selecting only one representation through direct regression.
4. Method
Bolt3D combines multiview latent diffusion with a feed-forward Gaussian head to generate renderable 3D Gaussian scenes from posed images and target poses. It trains geometry and appearance using dense, multiview-consistent reconstruction data.
- Architecture: Bolt3D takes posed input images and target camera poses, then outputs a 3D Gaussian representation through diffusion and a feed-forward Gaussian head.The diffusion model generates additional views and per-pixel 3D pointmaps; the head predicts complete Gaussian parameters.
- 3D representation: The scene uses K Splatter Images containing pixel-aligned colored 3D Gaussians for observed or generated views.Each Gaussian stores color, 3D position, opacity, and covariance matrix, enabling content beyond input-view coverage.
- Factorized sampling: Bolt3D factorizes generation by denoising color and 3D position first, then predicting opacity, covariance, and refined color.This separates quantities obtained from captured images and dense Structure-from-Motion from the remaining Gaussian properties.
- Geometric multi-view latent diffusion model: The multiview diffusion model jointly predicts target images, target pointmaps, and source pointmaps from conditioned images, poses, and target poses.Images and geometry are encoded and decoded with separate VAEs that spatially downsample the inputs by 8×.
- Geometry VAE: The Geometry VAE encodes pointmaps and camera raymaps into a latent and decodes them back to reconstructed pointmaps and raymaps.Its objective combines a standard VAE loss with geometry-specific reconstruction terms.
- Geometry VAE: The geometry reconstruction loss weights pixels by point distance from the local camera-frame scene center, accounting for lower confidence farther from the camera.Additional gradient reconstruction losses improve decoded pointmap boundary sharpness.
- Gaussian head and data: The Gaussian head calibrates generated pointmaps and predicts refined colors, opacities, and covariance matrices for renderable Splatter Images.It is trained with rendering losses using dense multiview-consistent geometry generated from MASt3R over 20-25 images per scene.
- Training: The model is trained in three stages: multiresolution model training, Gaussian-head training with rendering losses, and latent-diffusion training on 8-view then 16-view setups.The protocol uses 3 million iterations at 256×256, 250k at 512×512, 100k Gaussian-head iterations, and 700k plus 70k diffusion iterations.
5. Experiments
The experiments evaluate Bolt3D against feed-forward regression, feed-forward generative, and optimization-based methods, while also analyzing its Geometry VAE. Across these comparisons, Bolt3D combines strong reconstruction quality with substantially lower inference cost, and geometry-specific autoencoding is important for unbounded scenes.
- 5.1. Comparison to 3D Regression.: Bolt3D is evaluated against feed-forward Gaussian-splat regression methods on single- and few-view reconstruction using standard novel-view synthesis metrics.The protocol uses 512 × 512 center crops unless stated otherwise, with comparisons including Flash3D and DepthSplat.
- 5.1. Comparison to 3D Regression.: Bolt3D improves performance across datasets and input-view counts, with the largest gain in the 1-view setting where ambiguity is greatest.The results connect the strongest improvement to the setting with the least observed information.
- 5.1. Comparison to 3D Regression.: Bolt3D generates realistic details in unobserved and occluded regions, including scenes with small view overlap or difficult feature matching.These cases are illustrated for both single-view and few-view reconstruction.
- 5.2. Comparison to Feed-Forward 3D Generation.: Against feed-forward 3D generative methods, Bolt3D achieves the best image-level metrics and produces more realistic details than LatentSplat’s VAE-GAN.The comparison includes LatentSplat and Wonderland; the paper attributes the advantage over Wonderland to explicit geometry modeling.
- 5.3. Optimization-based 3D Reconstruction.: Bolt3D requires 300× less inference compute than optimization-based methods while remaining competitive in quality and sometimes producing sharper details.The method does not always outperform CAT3D, but the paper reports sharper results in some backgrounds and fine details.
- 5.3. Optimization-based 3D Reconstruction.: Bolt3D reconstructs one scene in 6.25 seconds on an H100 or 15 seconds on an A100, whereas CAT3D takes around 5 minutes depending on the dataset.CAT3D additionally generates 640–800 images per scene before reconstruction.
- 5.4. Image VAEs generalize poorly to geometry: Image-pretrained autoencoders struggle with unbounded pointmap geometry, whereas the geometry-trained autoencoder keeps 80% of points within 1% of ground truth.The geometry autoencoder also avoids the catastrophic outdoor failures observed with image autoencoders.
6. Conclusion
Bolt3D is a fast feed-forward method for detailed 3D scene generation that models ambiguity and supports high-quality one-view reconstruction. It generates scenes in under seven seconds and reduces 3D-generation cost by up to 300×.
- Bolt3D generates detailed 3D scenes in less than 7 seconds.
- Modeling ambiguity enables high-quality 1-view reconstructions where regression-based methods fail.
- Bolt3D reduces 3D-generation cost by 300× compared with existing optimization-based methods.
7. More experimental results
Additional experiments examine comparisons with video-based reconstruction, geometry and appearance ablations, and evaluation across resolutions. Bolt3D remains competitive across these settings, while performance depends on scene coverage, geometry training, and appearance design choices.
- MVSplat360 is less accurate and more than 200× slower than Bolt3D on 2- and 4-view DL3DV.
- MVSplat360 uses MVSplat conditioning, which performs poorly when input views have little or no overlap.
- Geometry VAE ablation: Training the Geometry VAE encoder and retaining geometry-specific losses improve system performance.
- Gaussian head ablation: Using fewer views reduces scene coverage, while cross attention and learned opacity and scale improve Gaussian rendering quality.
- Resolution comparisons: At 256×256, Bolt3D outperforms Depthsplat with the same input information; Depthsplat becomes similar only with 1.75× wider-field-of-view input.
8. Implementation details
Bolt3D normalizes reconstructed scenes into a common camera-centered scale, jointly models image appearance and geometry with latent diffusion, and converts predictions into renderable Gaussian parameters.
- XYZ normalization: Scenes are transformed into the first camera’s coordinate frame before training and normalized so its mean depth equals 1.
- XYZ normalization: The VAE reconstruction loss reweights points by distance from the local scene center to reduce the influence of uncertain or high-magnitude points.
- Diffusion model: The diffusion model jointly models latent appearance and geometry using expanded input and output channels.
- Autoencoder: The geometry autoencoder uses a camera-pose-conditioned encoder and a transformer-based decoder trained at 256×256 and fine-tuned at 512×512.
- Gaussian head: The Gaussian head predicts opacities, shapes, and refined colors from images, geometry maps, and camera poses, using rendering supervision.
- Sampling: Inference uses DDIM with 50 steps, and camera paths use 16 views rather than CAT3D’s 800.
9. Limitations, discussion and future work
Bolt3D has several documented scope and quality limitations, including sensitivity to scene properties, camera-path choices, and reduced quality relative to optimization-based methods. Future work considers better augmentation, more views, and dynamic-scene generation.
- Limitations: Thin structures narrower than 8 pixels remain difficult because of the geometry VAE’s spatial downsampling ratio.The method also struggles with transparent or highly non-Lambertian surfaces, where Structure-from-Motion geometry is typically inaccurate.
- Limitations: Bolt3D is sensitive to target-camera distributions, particularly the camera-path up-vector and scene scale.The authors suggest better data augmentation as a possible remedy.
- Discussion and future work: The geometry VAE design space remains open, including whether depth representations could outperform the chosen pointmap representation.The paper chose pointmaps because of their success in multiview reconstruction, while concurrent work reports complementary or advantageous depth inference.
- Discussion and future work: Bolt3D produces lower-quality 3D scenes than optimization-based methods such as CAT3D, possibly because CAT3D generates approximately 800 views versus Bolt3D’s 16.Generating more views could improve scene quality but would substantially increase the number of 3D Gaussians.
- Discussion and future work: Bolt3D generates only static scenes; future work could combine multiview video diffusion with direct geometry generation for dynamic scenes.
10. Experimental details
The experimental-details passages identify the DL3DV evaluation subset and describe scene- and architecture-related materials, but provide limited procedural detail in the supplied text.
- DL3DV scenes: Evaluation used the intersection of Bolt3D’s test set with the public DL3DV test benchmark.
- DL3DV scenes: The 2-view, 4-view, and 6-view setups used center-crops of selected views.
- Model architecture: The Gaussian head architecture combines convolutional and transformer blocks, using patchification to manage cross-attention sequence length.
- DL3DV scenes: The listed DL3DV evaluation material includes multiple scene identifiers, with an additional identifier continued in a separate passage.