latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction

1 Introduction

• Regression methods can blur high-uncertainty regions or fail in unseen areas, while generative methods sample plausible reconstructions that may contain hallucinated details.Because uncertainty varies across 3D locations, the paper motivates modeling it locally.Source: §1 · p. 2
• Variational 3D Gaussians encode distributions of semantic features at predicted 3D locations, with low variance in observed areas and uncertainty in unobserved areas.A sampled instance is rendered by splatting and decoded with a generative network.Source: §1 · pp. 2–3
• latentSplat outperforms recent methods in two-view reconstruction, with state-of-the-art quality especially for wide-spread input views and view extrapolation.The introduction reports both quantitative and qualitative superiority.Source: §1 · p. 3
• The method is fast and efficient in training and rendering, offering a more scalable solution than previous generative methods.Source: §1 · p. 3
• latentSplat applies to object-centric scenes with 360° views and general scenes, supports mesh reconstruction from 3D-consistent novel views, and trains purely on real videos.Source: §1 · p. 3

2 Related Work

• Earlier generalizable methods often focused on object-level 360° reconstruction, while large-scene extrapolation remained rare, often because of scalability limits.Source: §2 · p. 3

2.1 Regression-based Generalizable NVS

• Regression approaches include methods limited to object categories, small-scale scenes, or interpolation without unseen-area generalization.Multi-plane image methods are limited to small viewpoint changes.Source: §2.1 · p. 3
• Given two views, latentSplat reports 360° object-centric reconstruction plus interpolation and extrapolation on large scenes, using semantic features and explicit uncertainty rather than purely regressed Gaussians.The paper says this enables high-quality reconstruction of partially observed objects.Source: §2.1 · p. 4

2.2 Generative Models for NVS

• Generative NVS methods can be costly through 3D architectures or slow rendering and diffusion sampling, whereas latentSplat uses Gaussian representation and a lightweight decoder for speed and high-resolution scaling.The cited work describes these constraints across 3D diffusion and image-diffusion approaches.Source: §2.2 · p. 4

3.2 Variational 3D Gaussians

• Each variational Gaussian stores a distribution over spherical-harmonic feature coefficients, with location, shape, opacity, and explicit color parameters; reparameterized samples form semantic Gaussians for consistent multi-view renderings.The per-location variance represents uncertainty in the semantic features.Source: §3.2 · pp. 5–6
• For rendered regions without Gaussians, latentSplat samples features using rendered opacity, and its GAN-trained decoder can generate plausible details in those areas.Source: §3.2 · p. 6

3.3 Encoding Reference Views

• The two-view encoder uses epipolar cross-attention and per-ray depth distributions to predict Gaussian locations and properties, modeling uncertainty in both location and local appearance.Epipolar communication allows the encoder to resolve scale ambiguity.Source: §3.3 · p. 7

3.4 Decoding

• The decoder combines rendered RGB and feature images, feeding multi-scale features into a lightweight convolutional VAE decoder; direct RGB supervision helps optimize Gaussian structure.The decoder uses four upsample blocks, each with two residual blocks.Source: §3.4 · p. 7

3.5 Training

• Training uses two reference views and four target views from a video, encoding the references, rendering from target cameras, and decoding to reconstruct target images.The model assumes video frames have camera poses; view selection differs by scene type.Source: §3.5 · p. 8
• The autoencoder uses L1 and LPIPS reconstruction losses, plus an auxiliary RGB-rendering loss to improve gradients for structural Gaussian parameters.The auxiliary loss combines squared L2 error and LPIPS.Source: §3.5 · p. 8
• GAN losses train the decoder to generate realistic images and train a discriminator to distinguish real images from decoded predictions.The full objective combines reconstruction, auxiliary, and generative losses for the autoencoder, with a separate discriminator loss.Source: §3.5 · p. 8

4 Experiments

• The experiments assess interpolation, extrapolation, scene identity, 3D consistency, and whether latentSplat retains Gaussian splatting’s real-time rendering and memory efficiency.Additional results are provided in the appendix.Source: §4 · p. 8

4.1 Experimental Setup

• Experiments cover two-view reconstruction on CO3D objects and RealEstate10k scenes, with baselines spanning regression-based methods and the generative model GeNVS.CO3D view gaps increase to 25 frames, while RealEstate10k training samples target views up to 45 frames before or after references.Source: §4 · p. 8, §4.1 · p. 9

4.2 Object-Centric 3D Reconstruction

• On CO3D, latentSplat outperforms baselines in FID and KID and improves perceptual metrics over pixelSplat, while remaining on par in PSNR and SSIM.The authors note that traditional metrics can penalize generated details that differ from the target.Source: Table 1 · p. 9, §4.2 · p. 10

4.3 Scene-Level 3D Reconstruction

• On RealEstate10k, latentSplat outperforms baselines in FID, KID, LPIPS, and DISTS, generalizes better to extrapolation, and achieves state-of-the-art PSNR.The table caption reports parity with previous works in traditional metrics overall, while qualitative results are especially strong for extrapolation.Source: §4.3 · p. 11, Table 2 · p. 11

4.4 Uncertainty Visualization

• Uncertainty is higher in unseen or partly seen background regions and around foreground edges or teddybear fur than on uniform surfaces.The authors connect this lower certainty about foreground detail with the generative decoder’s higher level of detail.Source: §4.4 · p. 11

4.5 Mesh Reconstruction from Novel Views

• Meshes reconstructed from latentSplat’s synthesized views are reported as better suited for reconstruction and close to ground truth in texture and surface normals.The evaluation samples semantic Gaussians once, renders them from the original video’s camera poses, and reconstructs textured meshes.Source: §4.5 · p. 12, Table 3 · p. 13, Figure 5 · p. 13

4.6 Efficiency

• Compared with pixelSplat, latentSplat encodes slightly faster and renders 1 ms slower, while remaining real-time and memory efficient; it is orders of magnitude faster than GeNVS.The extra rendering time is attributed to the convolutional decoder and splatting additional feature channels.Source: §4.6 · pp. 12–13, Table 5 · p. 14

4.7 Ablations

• The deterministic ablation scores better on PSNR and SSIM but much worse on FID and KID; the generative approach with multi-resolution latent and RGB skip connections gives the best metric balance.The authors attribute the balance to the decoder’s use of global and local context when synthesizing images.Source: §4.7 · pp. 13–14, Table 4 · p. 14

5 Conclusion

• latentSplat achieves state-of-the-art novel-view-synthesis quality, the highest perceptual similarity to ground truth, 3D-consistent views, and faster, scalable rendering than prior generative approaches.The 3D-consistent views enable downstream mesh reconstruction; rendering can reach real time at large resolutions.Source: §5 · p. 14
• Out-of-view extrapolation can produce 3D inconsistencies because Gaussians are confined to input-camera frustums and the 2D decoder inpaints unseen areas.Independent sampling of depth along rays and local appearance also leaves a trade-off between capturing the full conditional distribution and sampling efficiently.Source: §5 · p. 14
• GeNVS rendering time was estimated using pixelNeRF and 25 StableDiffusion denoising steps at 256 resolution because GeNVS code was unavailable.This qualifies the rendering-time comparison for that baseline.Source: §5 · p. 14

Supplementary Material

• Sections A and B provide architecture and evaluation details.These sections supply the supplementary material's technical and evaluation information.Source: Supplementary Material · p. 19
• Section C gives a separate comparison with the closest generative baseline, while Section D evaluates 3D consistency using videos.Source: Supplementary Material · p. 19
• Sections E and F show intermediate results and model uncertainty, followed by additional qualitative results.Source: Supplementary Material · p. 19

A Architecture Details

• The encoder uses a DINOv1 ViT-B/8 backbone, epipolar transformer, and Gaussian sampling head to predict variational Gaussians.It downsamples feature maps by a factor of four for the transformer and adds an original-resolution skip connection.Source: A Architecture Details · p. 19
• The decoder adapts a pretrained LDM VAE decoder with four upsample blocks and adds skip connections for auxiliary RGB renderings and feature maps.The added 1x1 convolution weights are initialized to zero to avoid noisy gradients.Source: A Architecture Details · p. 19
• The discriminator is a finetuned pretrained LDM model with three strided convolutional hidden layers and patch-level likelihood predictions.Its limited receptive field means it does not predict a global likelihood.Source: A Architecture Details · p. 20
• Training starts with auxiliary losses for 100k iterations and delays the generative loss until 125k iterations to avoid fitting features to wrongly positioned Gaussians and an overly strong discriminator.Reconstruction uses L1 and LPIPS weights of 1 each; the generative loss is adaptively weighted using gradient norms.Source: A Architecture Details · p. 20

B Evaluation Details

• For CO3D, each scene uses 20 reference-view pairs separated by 16–25 frames, with random target views selected for interpolation or extrapolation.Source: B Evaluation Details · p. 20
• For RealEstate10k, reference pairs are 45–135 frames apart with at least 60% mutual epipolar-line coverage, and extrapolation targets extend up to 45 frames beyond either reference.Source: B Evaluation Details · p. 20
• Inference-time and memory comparisons use CO3D hydrant extrapolation on an RTX 4090 after models are trained to convergence.Convergence training used 200k iterations for latentSplat, pixelSplat, and pixelNeRF, and 300k for Du et al.'s method.Source: B Evaluation Details · p. 20

• The comparison is not fair because GeNVS code is unavailable and the methods differ in input views and dataset versions.Source: C Comparison with GeNVS · p. 20, Figure 1 · p. 21
• Both methods produce similar image quality, while latentSplat is much faster and more efficient than GeNVS [9].Source: Figure 1 · p. 21, C Comparison with GeNVS · p. 21
• The authors matched the comparison as closely as possible by using the same test examples and input view with a nearby second view.Source: C Comparison with GeNVS · pp. 20–21

• Two-reference-view videos show 360° CO3D hydrant and teddybear views without obvious geometric inconsistencies, while RealEstate10k videos appear realistic without nearby-frame flicker.Source: D 3D Consistency · p. 21
• Sampling Gaussian features once independently of target views yields consistent renderings even under uncertainty.Source: D 3D Consistency · p. 21
• Low-opacity regions outside reference frustums receive independently sampled noisy target-space features and are more prone to cross-view inconsistency, usually in the background.Source: D 3D Consistency · p. 21

• Directly rasterized RGB renderings that skip the VAE-GAN decoder are blurry in high-uncertainty regions and can contain floating-Gaussian artifacts.Source: E Additional Visualizations · pp. 21–22
• PCA-visualized feature maps encode object parts with different latent features and separate the background, where low-density areas contain noise.Source: E Additional Visualizations · p. 22
• Rendered uncertainty is higher in unseen or partly seen background regions and around object edges or teddy-bear fur than on uniform surfaces.The background standard deviation is set to one; the authors relate uncertainty in object details to the generative decoder's advantage for detail.Source: E Additional Visualizations · p. 22

• Intermediate CO3D hydrant and teddybear visualizations show rendered uncertainty, with darker regions indicating higher uncertainty and features displayed after PCA reduction to three dimensions.Source: Figure 2 · p. 23, Figure 3 · p. 24
• The additional figures include CO3D hydrant interpolation and extrapolation results and teddybear interpolation and extrapolation results.Source: Figure 4 · p. 25, Figure 5 · p. 26, Figure 6 · p. 27
• The additional RealEstate10k figures show qualitative interpolation and extrapolation results.Source: F Additional Results · p. 22, Figure 8 · p. 29, Figure 9 · p. 30