PaperlayerView exact revision on arXiv

Source-linked AI summary

SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis

Yecong Wan, Fan Li, Mingwen Shao, Wangmeng Zuo

arXiv:2605.07287v2cs.CV

TL;DR

Existing methods often allocate Gaussians uniformly, despite varying scene complexity; SplatWeaver routes cardinality-specific experts to allocate primitives adaptively, consistently improving rendering quality while using fewer primitives, including a 0.52 dB PSNR gain over Long-LRM.

  • Problem

    Existing Gaussian prediction schemes commonly allocate primitives uniformly, causing redundancy in smooth regions and insufficient capacity in complex areas.

  • Method

    SplatWeaver uses cardinality Gaussian experts, pixel-level routing, and high-frequency guidance to adaptively allocate Gaussian primitives according to scene complexity.

  • Results

    SplatWeaver consistently outperforms alternative methods across benchmarks, achieving a 0.52 dB PSNR improvement over Long-LRM in sparse-view RealEstate10K.

  • Takeaways & Limitations

    Adaptive allocation provides a more compact scene representation while maintaining high-fidelity novel-view rendering across varied viewing scenarios.

Abstract

from arXiv · show

Generalizable novel view synthesis aims to render unseen views from uncalibrated input images without requiring per-scene optimization. Recent feed-forward approaches based on 3D Gaussian Splatting have achieved promising efficiency and rendering quality. However, most of them assign a fixed number of Gaussians to each pixel or voxel, ignoring the spatially varying complexity of real-world scenes. Such uniform allocation often wastes Gaussian primitives in smooth regions while providing insufficient capacity for fine structures, complex geometry, and high-frequency details. This motivates us to predict region-dependent primitive cardinalities rather than impose a fixed primitive budget everywhere, enabling a more expressive 3D scene representation. Therefore, we propose SplatWeaver, a generalizable novel view synthesis framework that is able to dynamically allocate Gaussian primitives over different regions in a feed-forward manner. Specifically, SplatWeaver introduces cardinality Gaussian experts and a pixel-level routing scheme, wherein each expert specializes in producing a specific number of primitives from 0 to M, and the routing scheme coordinates these experts to adaptively determine how many Gaussian primitives should be allocated to each spatial location. Moreover, SplatWeaver incorporates a high-frequency prior with attendant guidance module and routing regularization to stabilize expert selection and promote complexity-aware allocation. By leveraging high-frequency cues, the routing process is encouraged to assign more Gaussian primitives to fine structures and textured regions, while suppressing redundancy in smooth areas. Extensive experiments across diverse scenarios show that SplatWeaver consistently outperforms state-of-the-art methods, delivering more faithful novel-view renderings with fewer Gaussian primitives. Project Page: https://yecongwan.github.io/SplatWeaver/

I. INTRODUCTION

Existing Gaussian prediction methods use uniform or predefined allocations that waste primitives in smooth regions and underfit complex details. SplatWeaver addresses this by dynamically allocating Gaussian primitives according to scene complexity through cardinality experts, pixel-level routing, and frequency-based guidance, achieving high-fidelity generalizable novel view synthesis.

  • Motivation: Known camera poses limit the practical utility of earlier novel-view synthesis methods in unconstrained or in-the-wild scenarios.These approaches reconstruct geometry and appearance from pre-calibrated viewpoints, ranging from sparse dual-view settings to dense sequences.
  • Motivation: Uniform pixel- or voxel-aligned Gaussian prediction creates redundancy in smooth areas while under-fitting intricate regions.Such schemes lack adaptive pruning and densification, preventing Gaussian distributions from adapting to regional scene complexity.
  • SplatWeaver: SplatWeaver adaptively allocates Gaussian primitives according to scene complexity in a feed-forward framework for efficient, high-fidelity generalizable novel view synthesis.Its emergent allocation capability adjusts the Gaussian budget according to view coverage and scene complexity.
  • SplatWeaver: Cardinality Gaussian experts and pixel-level routing enable each spatial location to receive a flexible number of Gaussian primitives.Each expert predicts a specific primitive cardinality ranging from 0 to M.
  • SplatWeaver: A high-frequency prior guidance module and routing regularization promote complexity-aware and structurally sound Gaussian allocation.The framework concentrates primitives in intricate regions while maintaining sparsity in smooth areas.
  • Results: SplatWeaver significantly outperforms alternative methods across diverse benchmarks while producing high-fidelity reconstructions with a more principled allocation.The paper reports consistent state-of-the-art performance across three benchmarks in pose-free generalizable novel view synthesis.

II. RELATED WORK … III. METHODOLOGY

The paper situates SplatWeaver within radiance-field, generalizable novel-view-synthesis, and dynamic-network research, then introduces cardinality Gaussian experts and pixel-level routing for complexity-aware primitive allocation. Its framework combines camera-pose and feature extraction, frequency-prior guidance, expert prediction, and neighborhood aggregation to form an efficient 3D representation.

  • A. Radiance Fields for Novel View Synthesis.: NeRF introduced implicit volumetric scene representation through coordinate-based neural networks, catalyzing extensive radiance-field research for novel-view synthesis.
  • B. Generalizable Novel View Synthesis.: Generalizable novel-view synthesis seeks to eliminate costly scene-specific optimization, extending reconstruction beyond early methods for small-scale scenes from sparse posed observations.
  • B. Generalizable Novel View Synthesis.: Dependence on a priori camera parameters limits generalizable methods in in-the-wild settings, motivating joint estimation of camera poses and scene appearance.
  • C. Dynamic Neural Networks.: Dynamic neural networks adapt weights or structures to inputs, while Mixture-of-Experts architectures use dynamic routing to scale capacity while preserving efficiency.
  • III. METHODOLOGY: SplatWeaver allocates Gaussian primitives according to scene complexity, replacing uniform per-pixel or per-voxel prediction with experts producing specific cardinalities from 0 to M.
  • III. METHODOLOGY: A frequency-prior injection module guides a pixel-level router in selecting cardinality experts, which predict hidden Gaussian positions and latent features before parameter completion.
  • III. METHODOLOGY: The resulting allocation dynamically controls regional and overall Gaussian budgets according to scene complexity, producing a more efficient and expressive 3D representation.

A. Preliminaries · B. Overview of SplatWeaver

The preliminaries formulate generalizable 3D Gaussian splatting as jointly recovering scene geometry, appearance, and camera poses from uncalibrated images. SplatWeaver then combines multi-view feature extraction, frequency-guided pixel routing, and attention-based Gaussian attribute prediction to allocate primitives adaptively.

  • A. Preliminaries: Given N uncalibrated views, generalizable 3D Gaussian splatting jointly recovers scene geometry, appearance, and camera poses.The input images are represented as {I_n}^N_{n=1}, with I_n ∈ R^{H×W×3}.
  • A. Preliminaries: The scene is represented by G anisotropic 3D Gaussians parameterized by position, scaling, rotation, opacity, and spherical-harmonic color coefficients.Each Gaussian uses μ ∈ R^3, s ∈ R^3, q ∈ R^4, α ∈ R^+, and c ∈ R^{3×(k+1)^2}.
  • A. Preliminaries: The model estimates each view’s intrinsic and extrinsic camera parameters and learns a mapping f_θ from input images to 3D primitives and camera poses.The camera parameters are denoted p_n, which encapsulate both intrinsic and extrinsic information.
  • B. Overview of SplatWeaver: SplatWeaver patchifies each uncalibrated input image into p^2 DINOv2 tokens before extracting interactive multi-view features.The multi-view geometry transformer follows VGGT principles and predicts camera pose parameters p_n.
  • B. Overview of SplatWeaver: A DPT-like decoder produces pixel-level per-image features F_n ∈ R^{H×W×D} for subsequent Gaussian generation.The decoder outputs the feature set {F_n}^N_{n=1}.
  • B. Overview of SplatWeaver: A discrete-wavelet frequency prior guides a pixel-level router toward reliable allocations among cardinality Gaussian experts.Each expert produces a specific number of hidden Gaussians, and the router selects the most appropriate expert for each pixel-wise feature.
  • B. Overview of SplatWeaver: Projected pixel features are concatenated with hidden Gaussians to form combined representations, after which neighboring hidden Gaussians predict scale, rotation, opacity, and color.Attention-based aggregation uses K neighboring hidden Gaussians to infer the remaining attributes of each primitive.

C. Cardinality Gaussian Expert Routing

SplatWeaver dynamically routes cardinality-specific Gaussian experts to spatial regions, allocating primitives according to scene complexity. Frequency-prior guidance and staged routing regularization encourage detail-aware yet compact predictions.

  • Expert Design: Cardinality Gaussian experts predict fixed numbers of Gaussian primitives and are dynamically routed to spatial regions according to scene content and geometry.This enables flexible, complexity-aware Gaussian allocation instead of uniform prediction.
  • Frequency Prior Guided Routing: High-frequency energy from a discrete wavelet transform identifies structurally detailed regions and guides the router toward allocating denser Gaussian representations there.The frequency prior guidance module injects this prior into feature representations and supports a dedicated allocation regularizer.
  • Expert Design: Each expert has cardinality me ∈ {0, 1, . . . , M}, with M empirically set to 3, balancing fine-grained representation, prediction reliability, and routing complexity.Thus, an expert predicts at most three Gaussian primitives.
  • Pixel-Wise Expert Router: A pixel-wise router uses Gumbel-Softmax with a Straight-Through estimator to select the top-1 cardinality expert through a hard one-hot routing decision.The selected expert’s output is multiplied by routing probability 1, preserving the physical meaning of spatial predictions such as µ.
  • Routing Regularization: Routing regularization guides allocation during the first half of training, then is removed so the model can explore routing while retaining a budget constraint.The budget penalty applies when predicted primitives exceed ϵNHW, with ϵ = 0.3 by default, encouraging compact representations.

D. Neighbor-Conditioned Gaussian Parameter Prediction

The module predicts Gaussian attributes by aggregating spatial-neighbor features from hidden Gaussians rather than estimating final parameters independently. Attention with relative-position conditioning, residual refinement, and coarse-to-fine matching provide context-aware, spatially coherent predictions.

  • Feature Aggregation and Parameter Prediction: Neighbor features from hidden Gaussians are aggregated to refine Gaussian parameters, preserving each hidden Gaussian’s original pixel-level detail-rich feature.The design combines local geometric context with the corresponding original pixel feature through a constructed feature representation.
  • Feature Aggregation and Parameter Prediction: To reduce the overhead of direct KNN matching across many primitives, the module uses Faiss CUDA acceleration with coarse-to-fine clustering followed by local matching.This strategy establishes spatial context without applying exhaustive direct neighbor matching to all Gaussian primitives.
  • Feature Aggregation and Parameter Prediction: A point transformer–style mechanism computes self-attention over neighboring hidden Gaussian features conditioned on relative spatial positions.Linear query and key projections, relative positional encoding, and attention projection MLPs define the aggregation operation.
  • Feature Aggregation and Parameter Prediction: The aggregated feature is added residually to the original hidden feature and passed through a prediction head to decode final Gaussian attributes.Residual addition preserves the initial representation while enabling context-based refinement before attribute decoding.
  • Feature Aggregation and Parameter Prediction: Local spatial context helps the predicted primitives remain physically coherent and aligned with the scene structure.The neighbor-conditioned prediction process is designed to leverage local geometric consistency in the scene.

E. Training Objective

The training objective combines rendering, routing, budget, camera-pose, and depth losses. Camera parameters and scene geometry are supervised using distilled VGGT predictions, while rendered images use MSE and perceptual losses.

  • Supervision losses: Camera parameters are distilled from a pre-trained VGGT using a Huber pose loss, while depth preserves scene geometry through mean squared error.These losses are denoted Lpose and Ldepth, respectively.
  • Supervision losses: Rendered images are supervised with a combination of mean squared error and perceptual losses.The losses are applied between the rendered images and their supervision targets.
  • Loss weighting: The loss coefficient λ is set to 0.05.This value is specified alongside the preceding loss formulation.
  • Overall objective: The final objective is L = Lrender + λ1Lroute + λ2Lbudget + λ3Lpose + λ4Ldepth.This weighted combination includes rendering, routing, budget, pose, and depth terms.

IV. EXPERIMENTS AND ANALYSIS · A. Experimental Settings · B. Comparison with State-of-the-Art Models

SplatWeaver is evaluated across diverse multi-view datasets and settings, including held-out and zero-shot benchmarks. Against state-of-the-art baselines, it achieves higher rendering quality while using fewer Gaussian primitives through adaptive, complexity-aware allocation.

  • A. Experimental Settings: Experiments use PyTorch on eight NVIDIA A100 GPUs, with a learning rate decayed from 2e-4 to 1e-6 by cosine annealing.Training samples 2–24 context images per batch and limits the longer image side to 448 pixels.
  • A. Experimental Settings: Training samples views from nine public datasets, while primary evaluation uses 140 held-out DL3DV scenes and zero-shot evaluation uses RealEstate10K and Mip-NeRF 360.The benchmarks span intricate indoor, expansive outdoor, and real-estate video scenes.
  • B. Comparison with State-of-the-Art Models: Qualitative comparisons show finer, detail-rich renderings that preserve intricate textures and sharpness, while other methods exhibit detail distortion or scene-estimation failures.The comparisons cover DL3DV, RealEstate10K, and Mip-NeRF 360 across 4, 8, 16, and 24 view settings.
  • B. Comparison with State-of-the-Art Models: SplatWeaver is compared with pixel-aligned, voxel-aligned, pruning-based, and query-based state-of-the-art methods, including NoPoSplat, FLARE, SPFSplat, AnySplat, YoNoSplat, EcoSplat, and C3G.AnySplat combined with LightGaussian offline post-pruning is also included for reference.
  • B. Comparison with State-of-the-Art Models: SplatWeaver outperforms all competitive methods in PSNR, SSIM, and LPIPS, achieving a 1.02 dB gain over AnySplat while using 70% fewer Gaussian primitives under the 16-view setting.The reported efficiency is attributed to adaptive, on-demand cardinality expert routing.
  • B. Comparison with State-of-the-Art Models: Adaptive allocation reduces redundancy in smooth regions and dedicates more primitives to geometrically complex areas, enabling high-fidelity scene modeling with fewer Gaussians.The allocation is described as geometry-aware and physically grounded.
  • B. Comparison with State-of-the-Art Models: EcoSplat reduces Gaussian counts through pruning but suffers substantial performance drops and can become unstable during scene reconstruction and camera pose estimation.The reported failures are associated with its pruning strategy.
  • B. Comparison with State-of-the-Art Models: C3G’s query-based paradigm uses a fixed Gaussian budget, constraining scalability and causing under-representation when viewpoint count, spatial complexity, or scene coverage varies.SplatWeaver instead maintains superior rendering quality with an economical, geometry-aware allocation.

C. Results on Dense Novel View Synthesis · D. Results on Camera Pose Estimation · E. Results on Pose-Known Novel View Synthesis

Across dense-view synthesis, camera pose estimation, and pose-known synthesis, SplatWeaver is evaluated against optimization-based and generalizable methods, showing broad performance advantages and benefits from adaptive Gaussian allocation. The framework also extends to a posed two-view setting using DepthSplat as its backbone under prior-work training conditions.

  • C. Results on Dense Novel View Synthesis: SplatWeaver outperforms optimization-based methods and generalizable frameworks across all metrics in dense novel view synthesis.The comparison includes 3D-GS, Mip-Splatting, Long-LRM, and AnySplat on Mip-NeRF 360 with 64 views.
  • C. Results on Dense Novel View Synthesis: Optimization-based methods require precise camera poses, careful initialization, and substantial training time, while remaining susceptible to overfitting artifacts.These limitations motivate comparison with feed-forward generalizable approaches.
  • D. Results on Camera Pose Estimation: SplatWeaver outperforms VGGT and AnySplat in camera pose estimation, despite both SplatWeaver and AnySplat using VGGT as supervision.The evaluation uses RealEstate10K and Co3Dv2 with 10 random frames.
  • D. Results on Camera Pose Estimation: Adaptive Gaussian allocation improves pose accuracy by reconstructing a sparser yet more representative Gaussian scene.The resulting geometric priors support more reliable camera registration and reduce localization errors.
  • E. Results on Pose-Known Novel View Synthesis: The pose-known evaluation compares SplatWeaver with feed-forward novel view synthesis methods on RealEstate10K using 2 input views at 256 × 256 resolution.This setting is commonly used in prior works.
  • E. Results on Pose-Known Novel View Synthesis: For pose-known synthesis, SplatWeaver integrates its architecture into DepthSplat and retrains the resulting model under the original settings.The comparison is conducted against methods cited as –,,, [19],, and.
  • E. Results on Pose-Known Novel View Synthesis: Table IV reports quantitative comparisons on the posed two-view RealEstate10K setting, with asterisks marking separate training under prior-literature conditions.The separate training protocol is used for fair comparison.

F. Efficiency Comparisons · G. Empirical Analyses

SplatWeaver combines adaptive Gaussian allocation with compact, efficient rendering, while ablations show that routing, complexity-aware specialization, and selected hyperparameters materially affect reconstruction quality and efficiency. Visual analyses further support its physically plausible allocation across scene complexity, expert scale, and geometry.

  • F. Efficiency Comparisons: SplatWeaver achieves the best PSNR among competing feed-forward methods while using the lowest storage and highest rendering speed through rational Gaussian allocation.The compact representation simultaneously improves rendering quality, storage efficiency, and rendering speed.
  • G. Empirical Analyses: Cardinality Gaussian expert routing improves PSNR by 1.63 dB, demonstrating the contribution of adaptive Gaussian allocation to feed-forward 3D reconstruction.The routing scheme is evaluated as a dedicated model component in the ablation study.
  • G. Empirical Analyses: Routing allocates primitives densely to complex, textured, or geometrically intricate regions and sparsely to smooth or simple areas.This produces a physically reasonable, complexity-dependent Gaussian distribution.
  • G. Empirical Analyses: Four experts provide sufficient capacity for scene-complexity variation, whereas reaching five causes slight performance degradation by increasing optimization difficulty.An insufficient number of experts restricts allocation capacity, although even two experts outperform opacity-based or score-based pruning methods.
  • G. Empirical Analyses: Performance peaks at k = 8, with diminishing returns beyond that value, so k = 8 is selected to balance reconstruction quality and computational efficiency.The method is relatively insensitive to the number of neighboring Gaussians.
  • G. Empirical Analyses: The ρ1, ρ2, and ρ3 proportions are robust within a reasonable range, while excessively high ρ3 and ρ2 disrupt allocation balance in smooth regions.The proportion constraint guides only the initial half of training.
  • G. Empirical Analyses: Low-cardinality experts generate large-scale primitives for smooth regions, whereas high-cardinality experts produce fine-grained, small-scale primitives for intricate structures and high-frequency details.The observed specialization aligns with geometric intuition and supports the physical plausibility of adaptive allocation.
  • G. Empirical Analyses: Despite abandoning conventional per-pixel Gaussian modeling, SplatWeaver produces high-fidelity novel views and detailed, accurate depth maps through non-uniform primitive distributions.The visualization emphasizes the structural fidelity of the adaptive allocation framework.

V. CONCLUDING REMARKS

SplatWeaver enables efficient, adaptive Gaussian primitive allocation in a feed-forward framework, addressing the redundancy and deficiency caused by uniform allocation. Extensive experiments demonstrate the method’s effectiveness, superiority, and efficiency across novel view synthesis benchmarks.

  • Contributions: SplatWeaver enables efficient and adaptive allocation of Gaussian primitives in a feed-forward manner.The framework is designed to improve 3D scene representation through adaptive primitive allocation.
  • Motivation: Uniform per-pixel or per-voxel primitive prediction causes redundancy in simple regions and insufficient capacity in complex ones.SplatWeaver addresses these limitations with a cardinality Gaussian expert routing scheme.
  • Experimental Findings: Extensive experiments across various novel view synthesis benchmarks demonstrate SplatWeaver’s effectiveness, superiority, and efficiency.The authors expect the work to provide insights for future generalizable novel view synthesis research.
Loading 2605.07287v2…