Source-linked AI summary
pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
David Charatan, Sizhe Li, Andrea Tagliasacchi, Vincent Sitzmann
TL;DR
pixelSplat addresses the challenge of generalizable novel view synthesis from sparse images, where existing rendering approaches are costly and many fast methods lack editable 3D representations. It predicts a Gaussian-based radiance field from an image pair using scale-aware two-view encoding and differentiable probability-based sampling. The method outperforms prior light-field transformers on real-world benchmarks while substantially reducing inference and training costs, with limitations for unseen regions, duplicate Gaussians, many views, and out-of-distribution viewpoints.
Problem
Generalizable novel view synthesis from sparse images remains costly, while light-field methods do not reconstruct editable or exportable 3D representations and monocular geometry has scale ambiguity.
Method
pixelSplat uses a two-view image encoder and pixel-aligned Gaussian prediction module, sampling Gaussian means from dense 3D probability distributions with differentiable reparameterization.
Results
pixelSplat outperforms baselines on all reported metrics, including perceptual distance, and reduces the cost of encoding one scene plus rendering 100 images by about 650 times versus the next-fastest method.
Takeaways & Limitations
A single forward pass can predict an explicit 3D Gaussian scene from an image pair while retaining fast, memory-efficient rendering and editable 3D structure.
Takeaways & Limitations
The method unions Gaussians from both reference views without deduplication, does not generate unseen scene parts, becomes memory-prohibitive with many views, and can appear billboard-like from out-of-distribution views.
Abstract
from arXiv · showhide
We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D radiance fields parameterized by 3D Gaussian primitives from pairs of images. Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time. To overcome local minima inherent to sparse and locally supported representations, we predict a dense probability distribution over 3D and sample Gaussian means from that probability distribution. We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation. We benchmark our method on wide-baseline novel view synthesis on the real-world RealEstate10k and ACID datasets, where we outperform state-of-the-art light field transformers and accelerate rendering by 2.5 orders of magnitude while reconstructing an interpretable and editable 3D radiance field.
1. Introduction
pixelSplat targets generalizable novel view synthesis from sparse image pairs while addressing the cost and limited editability of existing approaches. It predicts an explicit Gaussian-based 3D radiance field whose differentiable sampling avoids local minima and supports efficient reconstruction and rendering.
- Generalizable novel view synthesis from sparse observations remains memory- and time-intensive because differentiable rendering evaluates many points along each camera ray.
- Light-field transformers are faster than volume rendering but remain non-real-time and do not reconstruct editable or exportable 3D scene representations.
- pixelSplat reconstructs an explicit 3D radiance field from image pairs using 3D Gaussian primitives, enabling real-time, memory-efficient rendering and interpretable structure.
- Gaussian means are sampled from dense predicted 3D probability distributions, with differentiable reparameterization preserving gradient flow despite local minima.
- A single forward pass predicts a 3D Gaussian splatting representation from two images, outperforming light-field transformers on ACID and RealEstate10k while reducing training and rendering cost.
2. Related Work
Prior work spans neural fields, image-based methods, pixel-aligned or transformer features, cost volumes, and light-field representations for generalizable view synthesis. These approaches trade among 3D interpretability, scale handling, and rendering efficiency.
- Single-scene novel view synthesis: Neural fields and volume rendering became standard for reconstructing scenes from posed image collections, but their computation remains a central challenge.
- Prior-based 3D Reconstruction and View Synthesis: Generalizable novel view synthesis uses few images per scene, with proxy geometry, multi-plane images, or full 3D representations supporting different baseline regimes.
- Prior-based 3D Reconstruction and View Synthesis: Pixel-aligned features, transformers, and cost volumes improve generalization or cross-view matching, while light-field representations trade 3D interpretability for faster rendering.
- Scale ambiguity in machine learning for multi-view geometry: Prior approaches address real-world scene-scale ambiguity using scale-invariant losses, depth-statistics heuristics, or encoder conditioning on scene scale.
3. Background: 3D Gaussian Splatting
3D Gaussian Splatting represents scenes with renderable Gaussian primitives that are cheaper than dense field sampling. However, primitive fitting is non-convex and locally supported, creating optimization barriers when correct locations are distant or require non-monotonic movement.
- 3D Gaussian Splatting parameterizes a scene with Gaussian means, covariances, opacities, and spherical harmonics coefficients.
- Gaussian primitives represent a 3D radiance field and render through inexpensive rasterization, reducing time and memory relative to sampling-based dense representations.
- Primitive fitting is non-convex and susceptible to local minima, while EM remains affected by local minima and does not apply directly to inverse graphics without ground-truth 3D volumes.
- Local support causes gradients to vanish when primitives are far from correct locations, and useful movement may lack a monotonically decreasing-loss path through empty space.
4. Image-conditioned 3D Gaussian Inference
pixelSplat infers pixel-aligned 3D Gaussians from two images while addressing arbitrary scene scale and local minima in primitive optimization. It encodes scale-aware depths through epipolar attention and predicts Gaussian locations by differentiable sampling from per-pixel depth distributions.
- Resolving Scale Ambiguity: SfM provides camera poses only up to an arbitrary, scene-specific scale, so reconstruction depths must match that scale.A single image cannot recover the scale factor, making two-view scale information necessary for consistent Gaussian placement.
- Resolving Scale Ambiguity: The two-view encoder samples depths along corresponding epipolar lines and uses epipolar attention to record scale-consistent depth features.Depths computed from the camera poses encode the scene scale; self-attention propagates estimates to pixels lacking correspondences.
- Gaussian Parameter Prediction: For each pixel feature, the prediction module outputs Gaussian covariance, spherical-harmonic coefficients, depth probabilities, and offsets.Gaussians are predicted from both reference views, and their final set is the union of the per-image predictions.
- Gaussian Parameter Prediction: Instead of directly predicting Gaussian locations and opacities, the model predicts a discrete probability distribution over depth buckets and samples a location from it.Depth bins span near and far planes in disparity space, while the sampled bucket and offset determine the Gaussian mean.
- Gaussian Parameter Prediction: A reparameterization trick makes depth sampling differentiable by setting each sampled Gaussian’s opacity to the probability of its sampled bucket.This couples gradients on Gaussian opacity to the probability of resampling that 3D location, mitigating local minima from directly optimizing primitive positions.
5. Experiments
Experiments evaluate pixelSplat on wide-baseline novel view synthesis using RealEstate10k and ACID, comparing fidelity, resource demands, structured 3D outputs, and ablations. The method outperforms baselines while substantially reducing inference cost, though its Gaussian scenes retain known out-of-distribution rendering failures.
- Experimental Setup: pixelSplat is evaluated on RealEstate10k and ACID using wide-baseline image-pair novel view synthesis.Both datasets provide SfM camera poses, motivating the model’s scale-aware design.
- Experimental Setup: The comparison uses pixelNeRF, GPNR, and an epipolar-transformer light-field method, retrained under the same curriculum.The curriculum gradually increases the inter-frame distance between reference views.
- Results: pixelSplat outperforms the baselines on PSNR, SSIM, and LPIPS, with especially significant improvements in perceptual distance.Qualitative results also show better fine-detail capture and 3D inference in regions observed by only one reference view.
- Results: 650 times less cost is required to infer one scene and render 100 images than with the next-fastest baseline.The method also uses significantly less memory per ray during training.
- Results: The predicted Gaussian point clouds support high-fidelity novel-view synthesis for in-distribution camera poses but show transparency and billboard-like artifacts outside the training distribution.These failure modes also occur in scenes optimized with the original 3D Gaussian splatting method.
- Ablations and Analysis: Removing epipolar encoding causes ghosting, motion blur, and a significant performance drop, while removing depth encoding lowers PSNR by ≈1dB.The ablations support the role of cross-view correspondence and scene-scale depth in resolving scale ambiguity.
- Ablations and Analysis: Direct Gaussian-depth regression lowers PSNR by ≈1.5dB compared with probabilistic prediction.This ablation tests whether probabilistic primitive prediction alleviates local minima in primitive regression.
6. Conclusion
pixelSplat reconstructs an explicit primitive-based 3D scene from two images, while addressing scale ambiguity and local minima through dedicated modeling choices. Its ablations identify the epipolar encoder and probabilistic sampling scheme as essential for high-quality synthesis.
- pixelSplat reconstructs a primitive-based parameterization of a scene’s 3D radiance field from only two images.
- Both the epipolar encoder and probabilistic sampling scheme are essential for high-quality novel view synthesis.
- The method introduces a dense probability-distribution parameterization and differentiable reparameterization trick to address local minima in primitive-based regression.
A. Training Details
Training gradually increases the distance between reference views over the first 150,000 steps.
- The training data loader linearly increases reference-view distance from 25 to 45 over the first 150,000 training steps.
A.1. Our Method
The method trains pixelSplat with multi-view supervision, depth regularization procedures, camera-to-world Gaussian transformations, and an epipolar encoder that also supports more reference views. A three-view variant produces slightly better RealEstate10k results than the two-view setup.
- Our Method: pixelSplat is trained for 300,000 steps with batch size 7, rendering four target views and sampling three Gaussians along each ray.
- Our Method: Depth regularization generates a depth map for each rendered view and uses spatial depth differences weighted by image differences.
- Our Method: Predicted camera-space Gaussian parameters are transformed into world space using the camera-to-world extrinsics matrix before rendering.
- Using More Reference Views: The epipolar encoder extends to arbitrary numbers of views by unioning epipolar samples and adding a learnable per-image embedding.
- Using More Reference Views: A three-view pixelSplat variant with an intermediate third reference view produces slightly better RealEstate10k results.
C. Limitations
The model has limitations involving Gaussian handling, unseen-scene content, and scaling the epipolar mechanism to many reference views.
- The model outputs the union of Gaussians predicted from both reference views rather than fusing or de-duplicating them.
- The model does not address generative modeling of unseen parts of the scene.
- With many reference views, the epipolar attention mechanism becomes prohibitively expensive in memory.
D. Additional Results
This section presents additional qualitative results on the Real Estate 10k and ACID datasets. It also compares novel views generated from two versus three reference views.
- Figure 9 compares qualitative novel-view results using two and three reference views.
- The method supports an arbitrary, fixed number of reference views as input.
- The two-view model uses the top and bottom reference views, whereas the three-view model uses all three reference views.
- The qualitative comparison is organized around reference views, target views, the method's output, and its three-view output.
- Additional results are presented for the Real Estate 10k and ACID datasets.