Source-linked AI summary
VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction
Weijie Wang, Yeqing Chen, Zeyu Zhang, Hengyu Liu, Haoxiao Wang, Zhiyuan Feng, Wenkang Qin, Feng Chen, Jia-Wang Bian, Zheng Zhu, Donny Y. Chen, Bohan Zhuang
TL;DR
Feed-forward 3DGS commonly ties Gaussians to image pixels, creating alignment errors, view dependence, and density mismatches with scene complexity. VolSplat instead predicts Gaussians from a unified 3D voxel representation, and achieves state-of-the-art benchmark performance with more geometrically consistent reconstructions. Its scope is limited by an assumption of static scenes, with moving elements potentially causing ghosting or blurring.
Problem
Pixel-aligned feed-forward 3DGS is sensitive to multi-view alignment errors and ties Gaussian density to the 2D image grid, limiting representation of varying scene complexity.
Method
VolSplat aggregates multi-view features into a 3D voxel grid, refines the representation volumetrically, and predicts Gaussian primitives directly from occupied voxels.
Results
VolSplat achieves state-of-the-art performance on several large-scale benchmarks, including RealEstate10K and ScanNet, while producing more accurate and robust reconstructions than pixel-aligned baselines.
Takeaways & Limitations
Voxel alignment decouples reconstruction from input-view constraints, enabling adaptive Gaussian density and more geometrically consistent representations.
Takeaways & Limitations
VolSplat assumes static scenes, so moving objects or changing environments can produce ghosting or blurring in rendered novel views.
Abstract
from arXiv · showhide
Feed-forward 3D Gaussian Splatting (3DGS) has emerged as a highly effective solution for novel view synthesis. Existing methods predominantly rely on a \emph{pixel-aligned} Gaussian prediction paradigm, where each 2D pixel is mapped to a 3D Gaussian. We rethink this widely adopted formulation and identify several inherent limitations: it renders the reconstructed 3D models heavily dependent on the number of input views, leads to view-biased density distributions, and introduces alignment errors, particularly when source views contain occlusions or low texture. To address these challenges, we introduce VolSplat, a new multi-view feed-forward paradigm that replaces pixel alignment with voxel-aligned Gaussians. By directly predicting Gaussians from a predicted 3D voxel grid, it overcomes pixel alignment's reliance on error-prone 2D feature matching, ensuring robust multi-view consistency. Furthermore, it enables adaptive control over density based on 3D scene complexity, yielding more faithful Gaussians, improved geometric consistency, and enhanced novel-view rendering quality. Experiments on widely used benchmarks demonstrate that VolSplat achieves state-of-the-art performance, while producing more plausible and view-consistent results. The video results, code and trained models are available on our project page: https://lhmd.top/volsplat.
1 Introduction
VolSplat replaces pixel-aligned Gaussian prediction with voxel alignment to improve multi-view consistency and adapt representation density to scene complexity. The framework reports stronger and more robust reconstruction than comparable pixel-aligned baselines across large-scale benchmarks.
- Feed-forward reconstruction replaces slow per-scene optimization with a single learned inference pass, supporting real-time applications and large-scale reconstruction.
- Pixel-aligned methods suffer from calibration and discretization errors, inconsistent cross-view sampling, and density distributions rigidly tied to image pixels.
- VolSplat aggregates multi-view features in a shared 3D voxel grid, then refines the grid and predicts Gaussian primitives directly from voxels.
- Voxel alignment reduces view-dependent inconsistency and alignment-induced errors by fusing information volumetrically before Gaussian prediction.
- VolSplat achieves state-of-the-art performance on RealEstate10K, ScanNet, and ACID, outperforming comparable pixel-aligned feed-forward models.
2 Related Work
Prior work spans geometry-based, image-based, and light-field novel-view synthesis, voxel representations, and feed-forward 3DGS. VolSplat’s overview combines multi-view feature extraction, depth-guided voxel construction, sparse 3D refinement, voxel-aligned Gaussian prediction, and rendering.
- Novel view synthesis: Novel-view synthesis methods use explicit geometry, view interpolation, or dense ray sampling, each requiring accurate geometry, dense viewpoints, or both.
- 3D voxelization: Voxel methods evolved from memory-intensive dense grids toward sparse structures and now commonly support 3D CNNs or organize advanced rendering representations.
- Feed-forward 3D Gaussian Splatting: Feed-forward 3DGS predicts Gaussians in one pass; pixelSplat, MVSplat, and DepthSplat respectively emphasize epipolar processing, cost-volume fusion, and monocular depth features.
- VolSplat pipeline: VolSplat extracts image features, builds cost volumes, predicts depths, unprojects features into voxels, refines them sparsely in 3D, and renders from voxel-predicted Gaussians.
3 Method
VolSplat constructs voxel-aligned 3D features from multi-view image evidence, refines them volumetrically, and predicts one Gaussian collection per occupied voxel. This decouples representation density from the 2D image grid while supporting multi-view consistency and benchmark comparisons across input-view settings.
- 3.1 Preliminary and Observation: Pixel-aligned prediction fixes the Gaussian count at H × W × N, causing over-densification on textureless surfaces and insufficient representation for complex geometry.Its Gaussian positions are determined from predicted depth and pixel locations, tying the 3D representation to the 2D image grid.
- 3.2 3D Feature Construction: The method extracts cross-view features, builds plane-sweep cost volumes, and regresses dense per-pixel depths for lifting image features into a shared 3D space.Cross-view attention exchanges information between neighboring views, while cost-volume matching supplies the geometric prior for 3D construction.
- 3.2 3D Feature Construction: VolSplat voxelizes lifted points by voxel size and average-pools their associated image features into voxel features for volumetric reasoning.Each point is assigned an integer voxel index, and features from points within a voxel are aggregated along the channel dimension.
- 3.3 Feature Refinement and 3D Gaussians Prediction: A sparse 3D U-Net predicts residual voxel features that are added to the original grid, enabling multi-scale geometric refinement while preserving coarse information.Hierarchical sparse convolutions, encoder-decoder stages, skip connections, and residual updates support efficient local and global context fusion.
- 3.3 Feature Refinement and 3D Gaussians Prediction: The refined voxel grid predicts per-voxel Gaussian centers, opacities, covariances, and colors, which are trained end-to-end with photometric and perceptual losses.Gaussian centers are constrained to localized neighborhoods around voxel centroids, with the refinement scale set to three times the voxel size.
- VolSplat compares favorably across 6, 12, and 24 input views, consistently achieving the best performance across all metrics on RealEstate10K.The comparison includes pixel-aligned and post-processing baselines retrained under the same evaluation setup.
4 Experiments
Across RealEstate10K, ScanNet, and zero-shot ACID evaluation, VolSplat achieves strong benchmark performance and visually cleaner, more coherent reconstructions. Its voxel-aligned representation adapts Gaussian density to scene complexity and supports a favorable voxel-size and decoder design.
- 4.2 Experimental Results and Analysis: VolSplat achieves state-of-the-art results on both RealEstate10K and ScanNet, while pixel-aligned models degrade substantially under sparse multi-view inputs.The comparison covers mainstream pixel-aligned models and variants; VolSplat is also reported to outperform all baselines on ScanNet.
- 4.2 Experimental Results and Analysis: VolSplat reduces floaters and artifacts, producing cleaner object boundaries and more coherent reconstructions on RealEstate10K and ScanNet.The qualitative comparisons attribute the improvement to resolving multi-view alignment issues in the 3D feature representation.
- 4.2 Experimental Results and Analysis: VolSplat maintains significantly higher performance than baselines when transferred from RealEstate10K to ACID without fine-tuning.This evaluates zero-shot transfer from indoor training data to outdoor scenes.
- 4.2 Experimental Results and Analysis: VolSplat allocates more Gaussians to geometrically complex regions and fewer to flat or empty areas, unlike the constant-density behavior of pixel-aligned methods.The density analysis describes this as a more efficient, geometry-aware representation.
- 4.3 Ablation Study: A voxel size of 0.1 provides the best quality-efficiency trade-off, whereas overly small or large voxels lose coherent context or fine detail.The voxel-size study reports degraded performance at 0.05 and with larger voxels.
- 4.3 Ablation Study: Removing or replacing the sparse 3D refinement decoder reduces performance, supporting the proposed residual 3D U-Net design.The ablation compares removing refinement, removing residual prediction, and replacing the 3D U-Net with a sparse 3D CNN.
5 Conclusion
VolSplat addresses limitations of pixel-aligned feed-forward 3DGS by predicting Gaussians in a unified voxel-aligned space. This design decouples reconstruction from input-view constraints, adapts density to scene complexity, and supports more geometrically consistent reconstructions.
- 5 Conclusion: VolSplat shifts feed-forward Gaussian prediction from 2D pixels to a unified 3D voxel-aligned representation.The framework constructs voxel features and predicts Gaussians directly from them.
- 5 Conclusion: Voxel alignment decouples the reconstructed scene from input-view constraints while enabling adaptive Gaussian density and resolving alignment ambiguities.The stated outcome is more geometrically consistent and faithful reconstruction for downstream tasks.
A More Implementation Details
The implementation extracts and refines multi-view image features before reconstruction, using specified training and evaluation protocols. The codebase is intended to be released openly.
- A More Implementation Details: A weight-sharing ResNet extracts multi-scale features, followed by local-window cross-view attention to aggregate neighboring-view information.The architecture then proceeds to multi-view depth processing.
- A More Implementation Details: Training uses AdamW with separate learning-rate groups, six input views, eight target views, 256 × 256 resolution, and RealEstate10K-to-ScanNet fine-tuning.ACID is evaluated zero-shot, and the anchor-gap range expands progressively during early iterations.
- A More Implementation Details: Evaluation selects target novel views from disjoint camera positions and assesses eight target views per scene under fixed frame-gap rules.This protocol is designed for controlled comparison with prior feed-forward 3DGS methods.
- A More Implementation Details: The authors state that the complete VolSplat codebase will be open-sourced.
B More Experimental Analysis
VolSplat achieves the strongest image-quality results in the controlled efficiency comparison while maintaining competitive inference efficiency. The comparison uses 6 input views on RealEstate10K with a single NVIDIA H20 GPU.
- VolSplat tops all image-quality metrics while retaining competitive inference efficiency against competing approaches.The comparison evaluates all methods with 6 input views on RealEstate10K using a single NVIDIA H20 GPU.
- The reported relative rankings reflect a controlled and fair comparison despite hardware-dependent differences in absolute runtimes.
C Limitation and Societal Impacts
VolSplat’s current framework assumes static scenes, limiting reconstruction in dynamic environments. Its high-fidelity outputs also raise misuse risks involving deepfakes and unauthorized digital replicas.
- VolSplat assumes static scenes, so moving objects or changing environments can produce ghosting or blurring in rendered novel views.The limitation arises because multi-view feature aggregation and voxel construction rely on geometric consistency across views.
- High-fidelity reconstructions from limited data could be misused for deepfakes or unauthorized digital replicas of private spaces.The paper recommends watermarking or authentication protocols for sensitive deployments.
D More Visual Comparisons
The supplementary visual comparisons show VolSplat alongside ground truth and several baselines on RealEstate10K. They cover standard 6-view inputs and denser 12- and 24-view settings.
- Additional RealEstate10K comparisons include VolSplat, ground truth, and five baselines under 6, 12, and 24 input views.The displayed baselines are AnySplat, WorldMirror, MVSplat, DepthSplat, and GGN.
- The visual comparisons are organized to examine performance as the number of input views increases from 6 to 24.