Source-linked AI summary
F4Splat: Feed-Forward Predictive Densification for Feed-Forward 3D Gaussian Splatting
Injae Kim, Chaehyeon Kim, Minseong Bae, Minseok Joo, Hyunwoo J. Kim
TL;DR
Feed-forward 3DGS lacks efficient, controllable Gaussian allocation because uniform parameterizations create redundancy and do not adapt density to scene complexity. F4Splat predicts densification scores in one pass to allocate Gaussians according to spatial complexity and multi-view overlap, with explicit budget control without retraining. It achieves competitive or superior novel-view synthesis quality while using significantly fewer Gaussians than prior uncalibrated feed-forward methods.
Problem
Existing feed-forward 3DGS methods uniformly allocate Gaussians, causing redundancy and limiting efficient control of the total Gaussian count while maintaining fidelity.
Method
F4Splat predicts per-region densification scores in a feed-forward pipeline to adapt Gaussian allocation to spatial complexity and multi-view overlap.
Results
F4Splat achieves on-par or superior novel-view synthesis quality while using significantly fewer Gaussians than prior uncalibrated feed-forward methods.
Takeaways & Limitations
F4Splat produces compact Gaussian representations with explicit control over the final Gaussian budget without retraining.
Takeaways & Limitations
The threshold–budget lookup-table analysis assumes all densification score values are unique.
Abstract
from arXiv · showhide
Feed-forward 3D Gaussian Splatting methods enable single-pass reconstruction and real-time rendering. However, they typically adopt rigid pixel-to-Gaussian or voxel-to-Gaussian pipelines that uniformly allocate Gaussians, leading to redundant Gaussians across views. Moreover, they lack an effective mechanism to control the total number of Gaussians while maintaining reconstruction fidelity. To address these limitations, we present F4Splat, which performs Feed-Forward predictive densification for Feed-Forward 3D Gaussian Splatting, introducing a densification-score-guided allocation strategy that adaptively distributes Gaussians according to spatial complexity and multi-view overlap. Our model predicts per-region densification scores to estimate the required Gaussian density and allows explicit control over the final Gaussian budget without retraining. This spatially adaptive allocation reduces redundancy in simple regions and minimizes duplicate Gaussians across overlapping views, producing compact yet high-quality 3D representations. Extensive experiments demonstrate that our model achieves superior novel-view synthesis performance compared to prior uncalibrated feed-forward methods, while using significantly fewer Gaussians.
1 Introduction
Feed-forward 3DGS enables single-pass reconstruction but inherits inefficient Gaussian allocation without iterative adaptive density control. F4Splat predicts densification scores to allocate Gaussians spatially and control the final budget while maintaining reconstruction fidelity.
- Motivation: Feed-forward 3DGS reconstructs scenes from sparse images in one pass, but removing iterative optimization also removes conventional adaptive density control.This makes efficient Gaussian allocation a central limitation of existing methods.
- F4Splat: F4Splat predicts per-region densification decisions in a single forward pass within a unified feed-forward pipeline.The method treats Gaussian densification as a learnable prediction problem.
- F4Splat: Densification scores estimate spatial complexity and multi-view overlap to avoid over-allocation in simple regions and duplicate Gaussians across overlapping views.This produces spatially adaptive allocation and compact Gaussian representations.
- F4Splat: F4Splat enables explicit control over the final Gaussian count without retraining, including under a limited Gaussian budget.The allocation strategy assigns additional Gaussians where they are needed while retaining compactness.
- Results: F4Splat achieves on-par or superior novel-view synthesis quality while using significantly fewer Gaussians than prior uncalibrated feed-forward methods.The reported comparison concerns methods relying solely on image inputs.
2 Related Work
Feed-forward 3DGS avoids per-scene optimization and supports sparse-view reconstruction, but uniform Gaussian allocation ties representation size to input resolution. This creates redundancy in simple regions and insufficient capacity in complex ones.
- 3D Gaussian Splatting: 3DGS uses explicit Gaussian primitives for efficient rendering, but standard reconstruction typically requires many views and iterative per-scene optimization.Sparse-view and feed-forward approaches address these practical requirements in different ways.
- Feed-Forward 3D Gaussian Splatting: Feed-forward 3DGS methods learn priors from large-scale datasets and predict Gaussian representations in a single pass from sparse views.They retain real-time rendering and generalization to unseen scenes.
- Allocation Limitation: Uniform output parameterizations allocate a fixed number of Gaussians per pixel or spatial unit, coupling total count to input resolution.They do not adapt allocation to scene complexity.
- Allocation Limitation: Uniform allocation produces redundant primitives in simple regions while failing to sufficiently model geometrically complex regions under limited budgets.This yields suboptimal, non-compact representations.
3 Method
F4Splat formulates Gaussian allocation as a feed-forward prediction problem, using densification scores to distribute a user-controlled Gaussian budget across spatially complex and overlapping regions. Its pipeline predicts cameras, multi-scale Gaussian parameters, and scores, then selects non-overlapping allocations by thresholding.
- Problem formulation: F4Splat generates Gaussian primitives from image collections while allowing users to adjust the final Gaussian count through spatially adaptive allocation.The network takes context images and a user-specified target Gaussian budget as inputs.
- Training strategy: Training jointly optimizes camera and Gaussian parameters with camera, rendering, scene-scale, and densification-score losses.The final representation constructed with a sampled threshold is also supervised by rendering loss.
- Overall framework: The Geometry Backbone encodes multi-view images and predicts camera parameters, while Gaussian heads predict multi-scale primitive parameters and densification score maps.The backbone uses image tokens, camera tokens, register tokens, and alternating frame-wise and global self-attention layers.
- Spatially adaptive Gaussian allocation: Densification scores indicate how densely Gaussians should be placed, assigning more capacity to geometrically or photometrically complex regions and reducing redundant allocations across overlapping views.The allocation process is designed to use a limited Gaussian budget efficiently.
- Spatially adaptive Gaussian allocation: For each region, thresholding the predicted score selects a representation level, with higher levels adding Gaussians when scores exceed the threshold and masks preventing overlap across levels.The final representation is constructed from exclusively selected locations across multi-scale maps.
- Training strategy: The densification target is derived from view-space rendering gradients, whose magnitude signals underrepresented regions, and the network learns to predict this signal for inference.Ground-truth target images are used during training because rendering gradients are unavailable during feed-forward inference.
4 Experiments
F4Splat is evaluated across datasets, input-view counts, baselines, and Gaussian budgets, with ablations testing allocation and training components. It maintains strong reconstruction quality using substantially fewer Gaussians, while adaptive allocation and regularization support compact representations.
- Experimental Setup: F4Splat is evaluated on RE10K and unseen ACID using 8, 16, and 24 input views, alongside uncalibrated, pose-free, and pose-required baselines.
- Quantitative Results: 10-28% of the Gaussian primitives still produces competitive results, demonstrating quality retention under substantially reduced Gaussian budgets.
- Qualitative Results: 24-29% of the Gaussian primitives preserves high rendering quality while producing sharper structures, more faithful details, and fewer blurring artifacts on RE10K.
- Generalization: F4Splat achieves superior performance to the uncalibrated baseline on ACID under the challenging two-view setting while remaining competitive with pose-free and pose-required approaches.
- Ablation Studies: Learned densification-score allocation is compared with random and frequency-based strategies, while level-wise supervision and scene-scale regularization are evaluated through ablations.
- Ablation Studies: Removing level-wise supervision degrades performance, whereas removing scene-scale regularization destabilizes training and prevents meaningful reconstruction.
5 Conclusion
F4Splat reconstructs compact 3D Gaussian representations from sparse, uncalibrated inputs through predictive densification. Its spatially adaptive allocation maintains high reconstruction fidelity with significantly fewer Gaussians and explicit budget control.
- F4Splat reconstructs compact representations from sparse, uncalibrated inputs using feed-forward predictive densification.
- Densification-score-guided allocation distributes Gaussians according to spatial complexity and multi-view overlap, avoiding redundancy in simple or overlapping regions.
- Explicit Gaussian-budget control is available without retraining while maintaining high reconstruction fidelity under limited budgets.
- Experiments show competitive or superior novel-view synthesis performance versus prior feed-forward methods while requiring significantly fewer Gaussians.
S1 Budget Matching Algorithm
F4Splat precomputes a threshold–budget lookup table to select a densification threshold matching a target Gaussian budget. The lookup supports efficient budget selection with a bounded discretization gap.
- Lookup-table construction: The lookup table pairs sorted densification thresholds with cumulative Gaussian counts for efficient target-budget matching.Thresholds are sorted in descending order, while Gaussian counts are stored in ascending order through cumulative sums.
- Bound on budget increments: 4^{L-1}-1 bounds the difference between adjacent Gaussian-count entries when score values are unique.Lowering the threshold activates one score value at one spatial position; the largest increase occurs at the coarsest level and includes unresolved finer-level descendants.
- Allocation efficiency: Adaptive allocation reduces the total Gaussian count by 20% compared with uniform pixel-to-Gaussian allocation.This comparison is reported as computational overhead for spatially adaptive Gaussian allocation.
- Budget matching: For any target budget ¯NG, an index k⋆ identifies a threshold whose Gaussian count is within the bounded increment.The selected threshold is τ¯NG = ˜τk⋆ and satisfies the target-budget condition described by Eq. (4).
S2 Additional Results
Additional videos visualize how F4Splat changes Gaussian allocation across budgets and compare it with AnySplat. F4Splat concentrates Gaussians in detailed regions and maintains fidelity with fewer Gaussians.
- Budget-controlled allocation: As the Gaussian budget increases, F4Splat adaptively allocates Gaussians to fine-detail regions while minimizing redundant allocations in overlapping context regions.The videos visualize spatial distributions under varying Gaussian budgets.
- Comparison with AnySplat: Compared with AnySplat under matched Gaussian budgets, F4Splat allocates Gaussians spatially adaptively rather than uniformly over the scene.The comparison uses two input views from RE10K and ACID, with RGB, depth, and allocated-location visualizations.
- Comparison with AnySplat: F4Splat maintains high fidelity even with fewer Gaussians than AnySplat.The matched-budget video comparison reports rendered RGB quality, depth maps, and Gaussian locations.
S3 Additional Experiments
The additional experiments evaluate predicted camera poses on RE10K and ACID and provide further qualitative novel-view comparisons on RE10K.
- Relative pose estimation: F4Splat’s predicted camera poses are evaluated against DUSt3R, MASt3R, VGGT, and AnySplat on RE10K and ACID.The evaluation follows the NoPoSplat protocol and uses multi-view-supervised models.
- Qualitative comparisons: Figure S1 provides additional qualitative novel-view synthesis comparisons on the RE10K dataset.The passage identifies the figure as an additional multi-view experiment.
S4 Additional Details
The experiments freeze only the image tokenizer while fine-tuning the remaining model components, using differential learning rates for inherited pretrained components.
- Training configuration: Only the image tokenizer, DINOv2, is frozen; attention layers, camera and register tokens, and prediction heads are fine-tuned.The fine-tuned components include frame-wise and global attention layers, camera and register tokens, and point and camera prediction heads.
- Training configuration: The base learning rate is 2×10^-4, while inherited pretrained components use a learning rate scaled by 1/10.This configuration is applied across the experiments.
- Evaluation configuration: RE10K and ACID evaluation follows the NoPoSplat protocol and applies test-time camera pose optimization.The passage specifies this evaluation procedure without reporting its numerical outcomes.