Source-linked AI summary
FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes
Yunsong Wang, Tianxin Huang, Hanlin Chen, Gim Hee Lee
TL;DR
Existing generalizable 3D Gaussian Splatting methods are largely restricted to narrow-range view interpolation, limiting their support for accurate localization and wide-range free-view synthesis. FreeSplat combines efficient cross-view aggregation, pixel-wise triplet fusion, and free-view training for long-sequence reconstruction; the authors report improved color rendering and depth accuracy, with a 94.9% δ < 1.25 result for FreeSplat-fv.
Problem
Method
FreeSplat uses low-cost nearby-view cost volumes, pixel-wise fusion of Gaussian triplets, and free-view training to reconstruct global Gaussians from long image sequences.
Source: §1 · p. 2Results
FreeSplat-fv reaches 94.9% δ < 1.25 and FreeSplat consistently improves novel-view rendering quality and depth accuracy across input-view lengths.
Source: §1 · p. 2, §5.2 · p. 9, §6 · p. 10Takeaways & Limitations
FreeSplat supports feed-forward reconstruction of global Gaussians for free-view synthesis without depth priors.
Source: §6 · p. 10Extremely long sequences exceeding 50 images require over 40GB of GPU memory, and reconstruction accuracy trails methods using 3D supervision or RGB-D inputs.
Source: §A.5 · p. 16
Abstract
from arXiv · showhide
Empowering 3D Gaussian Splatting with generalization ability is appealing. However, existing generalizable 3D Gaussian Splatting methods are largely confined to narrow-range interpolation between stereo images due to their heavy backbones, thus lacking the ability to accurately localize 3D Gaussian and support free-view synthesis across wide view range. In this paper, we present a novel framework FreeSplat that is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis.Specifically, we firstly introduce Low-cost Cross-View Aggregation achieved by constructing adaptive cost volumes among nearby views and aggregating features using a multi-scale structure. Subsequently, we present the Pixel-wise Triplet Fusion to eliminate redundancy of 3D Gaussians in overlapping view regions and to aggregate features observed across multiple views. Additionally, we propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views. Our empirical results demonstrate state-of-the-art novel view synthesis peformances in both novel view rendered color maps quality and depth maps accuracy across different numbers of input views. We also show that FreeSplat performs inference more efficiently and can effectively reduce redundant Gaussians, offering the possibility of feed-forward large scene reconstruction without depth priors.
1 Introduction
FreeSplat targets long-sequence indoor free-view synthesis, addressing the narrow view range and Gaussian localization and redundancy issues of existing generalizable methods. It combines efficient cross-view aggregation and triplet fusion, with experiments reporting improvements across rendering quality and depth accuracy.
1 Introduction
- Existing generalizable 3DGS methods are limited to narrow-range interpolation or object-centric synthesis, with dense view matching becoming computationally intractable for long sequences.Narrow-range supervision can poorly localize Gaussians, while concatenating multi-view Gaussians creates redundancy in overlaps.Source: §1 · p. 2
- FreeSplat combines low-cost cross-view aggregation, pixel-wise triplet fusion, and free-view training for indoor long-sequence synthesis.Its aggregation uses nearby-view adaptive cost volumes and multi-scale features; fusion aligns and combines local triplets to reduce overlap redundancy.Source: §1 · p. 2
- Low-cost Cross-View Aggregation predicts initial Gaussian triplets and enables matching across more nearby views and long-sequence reconstruction training.The approach uses efficient feature matching to make this longer-sequence processing possible.Source: §1 · p. 2
- Pixel-wise Triplet Fusion reduces redundant Gaussians in overlapping regions while aggregating latent features from multiple views.The module progressively fuses pixel-aligned local Gaussian triplets.Source: §1 · p. 2
- FreeSplat is reported to improve both image-rendering quality and novel-view depth accuracy on ScanNet and Replica across different input-view lengths.The authors describe it as the first generalizable 3DGS method for long-sequence reconstruction.Source: §1 · p. 2
2 Related Work
Prior novel-view synthesis approaches trade representation and rendering choices against computational cost, scene-specific optimization, or input requirements. FreeSplat is positioned as feed-forward reconstruction of large scenes from arbitrary-length image inputs without depth priors.
2 Related Work
- NeRF learns implicit geometry and radiance through ray-marching volume rendering, while 3D Gaussian Splatting uses explicit Gaussians and rasterization for faster rendering.The related-work discussion describes ray marching as computationally intensive and 3DGS rasterization as supporting real-time rendering.Source: §2 · pp. 2–3
- Generalizable novel-view synthesis predicts scene information feed-forward, but existing generalizable 3DGS methods remain limited to interpolation or object- and human-centric scenes.The cited methods include pixel-aligned Gaussian prediction, cost volumes, sequence concatenation, and latent Gaussian representations.Source: §2 · p. 3
- FreeSplat is presented as feed-forward reconstruction of large scenes from arbitrary-length inputs without depth priors.The paper contrasts this scope with prior methods limited to view interpolation or object- and human-centric scenes.Source: §2 · p. 3
- Indoor reconstruction alternatives include mesh extraction, RGB-D SLAM, and per-scene implicit fields, each with stated limits on photorealistic synthesis, input, or optimization.SurfelNeRF supports photorealistic rendering but relies on external depth estimation or ground-truth depth maps.Source: §2 · p. 3
3 Preliminary
The preliminary section contrasts per-scene Gaussian optimization with generalizable prediction of pixel-aligned Gaussians and depths. FreeSplat’s overview then traces images through cost volumes and triplet fusion toward global Gaussian parameters.
3 Preliminary
- Vanilla 3DGS represents a scene explicitly with Gaussian primitives parameterized by a 3D covariance matrix and mean.The covariance is decomposed using scaling and rotation matrices to maintain positive semi-definiteness.Source: §3 · p. 3
- Rendering projects 3D Gaussians into 2D and computes color through point-based alpha blending.The Gaussian parameters are optimized with a photometric loss that minimizes differences between renderings and image observations.Source: §3 · p. 3
- FreeSplat builds nearby-view cost volumes, predicts depths and features, unprojects Gaussian triplets, and progressively fuses pixel-aligned local and global triplets.The fused global triplets can be decoded into Gaussian parameters.Source: Figure 2 · p. 4
- Generalizable 3DGS predicts pixel-aligned Gaussian parameters and depths for unprojection, but prior methods are limited to narrow-range interpolation and inaccurate localization.FreeSplat is proposed to reconstruct global 3D Gaussians for free-view synthesis.Source: §3 · p. 4
4 Our Methodology
FreeSplat builds global 3D Gaussian scenes from sparse posed image sequences by matching nearby views and progressively fusing aligned Gaussian triplets. Free-view training supervises renderings across broader view interpolations using randomly sampled context views.
4.1 Overview
- FreeSplat builds adaptive cost volumes between nearby views, predicts depth maps, and unprojects 2D feature maps into 3D Gaussian triplets for global scene reconstruction.A multi-scale encoder-decoder fuses image features with cost-volume features and predicts full-resolution depth candidates.Source: §4.1 · p. 4, §4.2 · pp. 4–5
4.2 Low-cost Cross-View Aggregation
- Nearby views are selected by pose proximity, and their features are warped across uniformly spaced virtual depth planes to form cost volumes.The method uses cosine similarity, feature concatenation, and a 1 × 1 CNN to construct the cost volume.Source: §4.2 · pp. 4–5
4.3 Pixel-wise Triplet Fusion
- Pixel-wise Triplet Fusion aligns local Gaussian triplets with global ones, removes redundant overlap Gaussians, and aggregates multi-view latent features using a lightweight GRU.Aligned triplet pairs are merged using weighted center coordinates, keeping the resulting centers between the pair.Source: §4.3 · pp. 5–6
4.4 Training
- Free-View Training randomly samples 2–8 context views and supervises renderings across broader view interpolations using photometric losses.Training combines MSE and LPIPS losses; the authors report that free-view training contributes to novel-view depth estimation.Source: §4.4 · p. 6
5 Experiments
FreeSplat is evaluated on ScanNet for short- and long-sequence novel-view synthesis, including extrapolation, and on Replica for zero-shot transfer. The reported results cover rendering quality, depth accuracy, efficiency, and module ablations.
5.2 Results on ScanNet
- For 10-view ScanNet reconstruction, FreeSplat-3views significantly outperforms pixelSplat and MVSplat in interpolation and extrapolation, while FreeSplat-fv improves further.Figure 4 places interpolation in the first two rows and extrapolation in the last two.Source: Table 2 · p. 7, §5.2 · p. 8, Figure 4 · p. 8
5.1 Experimental Settings
- ScanNet uses 100 training and 8 test scenes; zero-shot Replica evaluation uses 8 test scenes.Experiments compare 2-, 3-, and 10-view settings; the 10-view setting also evaluates target views beyond the reference sequence.Source: §5.1 · p. 7
5.2 Results on ScanNet
- 300× faster inference than NeuRay, with improved PSNR and LPIPS but slightly lower SSIM; FreeSplat-spec improves rendering quality and efficiency over pixelSplat and MVSplat at 2- and 3-view settings.FreeSplat-fv also offers competitive results with arbitrary input-view counts and becomes more similar to FreeSplat-spec as the number of views increases.Source: §5.2 · p. 7, Table 1 · p. 7
- 94.9% δ < 1.25 for FreeSplat-fv, which consistently outperforms pixelSplat and MVSplat on novel-view depth localization.The depth comparison reports Absolute Difference, Relative Difference, and δ < 1.25.Source: Table 3 · p. 7, §5.2 · p. 9
5.3 Zero-Shot Transfer Results on Replica
- Zero-shot Replica interpolation and novel-view depth estimation outperform existing methods, but long-sequence results degrade with depth inaccuracy and domain gap.The results are reported in Table 4.Source: Table 4 · p. 9, §5.3 · p. 9
5.4 Ablation Study
- Ablations identify cost volume as essential for Gaussian localization; PTF improves rendering and depth estimation across varying view counts, while FVT excels in long-sequence reconstruction and depth rendering.FVT’s depth-rendering supervision imposes stricter constraints on Gaussian localization and can be combined with PTF for varying input lengths.Source: Table 5 · p. 9, §5.4 · pp. 9–10
6 Conclusion
FreeSplat combines efficient cross-view processing with pixel-wise Gaussian fusion to support free-view synthesis from arbitrary input-view counts.
6 Conclusion
- FreeSplat supports free-view synthesis from arbitrary input-view counts using global 3D Gaussians, efficient long-sequence feature aggregation, and fusion that reduces redundant pixel-aligned Gaussians in overlapping regions.The conclusion reports improved novel-view color fidelity and depth-map accuracy, with feed-forward global reconstruction without depth priors.Source: §6 · p. 10
A.1 Experimental Environment
The experiments use a single NVIDIA RTX A6000 GPU with PyTorch 2.1.2 and CUDA 12.2.
A.1 Experimental Environment
- All experiments run on one NVIDIA RTX A6000 GPU using PyTorch 2.1.2 and CUDA 12.2.Source: §A.1 · p. 14
A.2 Additional Implementation Details
The appendix specifies cost-volume and alignment settings, nearby-view counts, and adjustments used to compare methods under GPU constraints.
A.2 Additional Implementation Details
- Cost volumes use 128 virtual depth planes, 64 feature channels, and depth bounds 0.5–15.0; pixel-wise alignment uses δ = 0.05.Source: §A.2 · p. 14
- FreeSplat-fv uses 4 nearby views in training; long-sequence reconstruction forms cost volumes across nearby 8 views.Source: §A.2 · p. 14
- For fair GPU-constrained comparisons, pixelSplat’s 3-view patch size is enlarged to 16 × 16, and pixelSplat and MVSplat use nearby-view matching at 10-view inference.Source: §A.2 · p. 14
A.3 Additional Experiments
Additional experiments compare FreeSplat with prior methods across view counts, datasets, and whole-scene reconstruction. They report improved rendering efficiency, stronger performance with five-view inputs, and practical boundaries for sparse inputs and very long sequences.
Computational Cost
- Rendering speed rises from 39 to 72 FPS with Pixel-wise Triplet Fusion in the 30-view whole-scene comparison.The table compares GPU memory, encoding time, rendering FPS, and novel-view PSNR; pixelSplat inference could not run with 30 input views because of GPU requirements.Source: Table 6 · p. 14, §A.3 · p. 14
Experiments on RE10K and ACID
- FreeSplat performs better than pixelSplat on both RE10K and ACID with 2 views, matches MVSplat, and outperforms both baselines with 5 views.For the 5-view setting, input views are sampled at random intervals of 25–45 because of limited sequence lengths.Source: §A.3 · pp. 14–15, Table 7 · p. 14, Table 8 · p. 15
- With two-view inputs, large image intervals reduce view overlap and weaken multi-view aggregation; five-view inputs with smaller intervals support stronger cost-volume aggregation and Gaussian fusion.The reported average stereo-image intervals are 66 for RE10K and 74 for ACID, compared with 20 for ScanNet and 10 for Replica.Source: §A.3 · p. 15
Comparison with SurfelNeRF
- FreeSplat significantly outperforms SurfelNeRF in rendering quality and efficiency on the same test sequences.FreeSplat jointly learns depths and 3D Gaussians with an MVS-based backbone, whereas SurfelNeRF relies on depths and does not aggregate multi-view features for surfel feature prediction.Source: §A.3 · p. 15, Table 9 · p. 16
A.4 Additional Qualitative Results
Additional qualitative results show FreeSplat localizing Gaussians and rendering depth across broader view ranges and indoor datasets. Whole-scene examples demonstrate fast feed-forward inference while exposing difficulty in textureless and specular regions.
2 and 3-View Interpolation Results
- FreeSplat more precisely localizes 3D Gaussians and captures finer details in 2- and 3-view interpolation, with precise depth maps supporting broader-view rendering.The text directs readers to FreeSplat-spec results in Figure 4 for comparison.Source: §A.4 · p. 16
Results on Replica
- On Replica, qualitative results show superiority over MVSplat and pixelSplat for view interpolation across indoor datasets.Source: §A.4 · p. 16
Results of Whole Scene Reconstruction
- For whole-scene reconstruction, FreeSplat processes about 40 images in about 1 second on one NVIDIA RTX A6000, but depth remains difficult in textureless and specular regions.The authors frame this as an initial step toward geometry reconstruction without ground-truth depth priors.Source: §A.4 · p. 16
A.5 Limitations
The limitations and checklist material notes compute and depth-estimation boundaries, alongside disclosures about code access, statistical significance, and experimental settings.
A.5 Limitations
- GPU use exceeds 40 GB for input sequences longer than 50 images, and reconstruction accuracy still trails methods using 3D supervision or RGB-D inputs.The authors identify textureless and specular regions as examples of the remaining accuracy gap and focus on photorealistic reconstruction with 2D supervision.Source: §A.5 · p. 16
NeurIPS Paper Checklist
- The checklist reports that code is not openly accessible at submission, with release planned upon acceptance, and that statistical significance information is not reported.The authors say they follow related works in the setting for error bars.Source: §2 · p. 21
2. Limitations
- The checklist marks training and test details as specified, referring readers to Section 5.1 for experimental settings.Source: §2 · p. 21