PaperlayerView exact revision on arXiv

Source-linked AI summary

GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens

Roni Itkin, Noam Issachar, Yehonatan Keypur, Xingyu Chen, Anpei Chen, Sagie Benaim

arXiv:2604.15284v2cs.CV

TL;DR

Feed-forward 3DGS struggles to balance compactness, speed, and fidelity because dense view-aligned primitive allocation introduces redundancy as views increase. GlobalSplat aligns multi-view inputs into fixed global scene tokens before decoding Gaussians, achieving competitive quality with a compact representation and fast inference. Its main scope boundary is the fixed 16K-Gaussian budget, which may not suit unbounded or city-scale scenes.

  • Problem

    Dense pixel- or voxel-aligned feed-forward 3DGS pipelines introduce redundant view-anchored primitives as input views grow, limiting scalable compact reconstruction.

  • Method

    GlobalSplat fuses multi-view inputs into fixed globally aligned scene tokens, then decodes explicit Gaussians using disentangled attention and coarse-to-fine capacity growth.

  • Results

    GlobalSplat achieves highly competitive novel-view synthesis on RealEstate10K and ACID while capping scenes at 16K Gaussians, under 4 MB, 1.79 GB peak memory, and under 78 milliseconds encoding.

  • Takeaways & Limitations

    GlobalSplat provides a compact, view-invariant explicit 3DGS representation with faster inference and real-time rendering speeds.

  • Takeaways & Limitations

    The fixed 16K-Gaussian budget may eventually lack capacity for unbounded or city-scale environments.

Abstract

from arXiv · show

The efficient spatial allocation of primitives serves as the foundation of 3D Gaussian Splatting, as it directly dictates the synergy between representation compactness, reconstruction speed, and rendering fidelity. Previous solutions, whether based on iterative optimization or feed-forward inference, suffer from significant trade-offs between these goals, mainly due to the reliance on local, heuristic-driven allocation strategies that lack global scene awareness. Specifically, current feed-forward methods are largely pixel-aligned or voxel-aligned. By unprojecting pixels into dense, view-aligned primitives, they bake redundancy into the 3D asset. As more input views are added, the representation size increases and global consistency becomes fragile. To this end, we introduce GlobalSplat, a framework built on the principle of align first, decode later. Our approach learns a compact, global, latent scene representation that encodes multi-view input and resolves cross-view correspondences before decoding any explicit 3D geometry. Crucially, this formulation enables compact, globally consistent reconstructions without relying on pretrained pixel-prediction backbones or reusing latent features from dense baselines. Utilizing a coarse-to-fine training curriculum that gradually increases decoded capacity, GlobalSplat natively prevents representation bloat. On RealEstate10K and ACID, our model achieves competitive novel-view synthesis performance while utilizing as few as 16K Gaussians, significantly less than required by dense pipelines, obtaining a light 4MB footprint. Further, GlobalSplat enables significantly faster inference than the baselines, operating under 78 milliseconds in a single forward pass. Project page is available at https://r-itk.github.io/globalsplat/

1 Introduction

GlobalSplat addresses the scalability and compactness limits of feed-forward 3DGS by aligning multi-view information globally before decoding explicit Gaussians. It maintains competitive novel-view synthesis with a fixed, compact representation and fast inference.

  • GlobalSplat: GlobalSplat uses fixed-size scene tokens independent of input size, together with dual-branch attention and coarse-to-fine capacity growth.The architecture disentangles geometry and appearance, while progressive decoding capacity helps prevent representation bloat.
  • Motivation: Current feed-forward 3DGS methods form primitives from dense view-aligned intermediates, baking redundancy into representations as input views increase.This makes large-context reconstruction harder to scale robustly.
  • GlobalSplat: GlobalSplat fuses all input views into a globally aligned latent scene representation before decoding explicit 3D Gaussians.This shifts primitive formation from view-centric outputs to scene-centric allocation.
  • Results: 16K Gaussians and 4 MB yield a view-invariant representation across large-context settings.The model maintains this budget regardless of the number of input views.
  • Results: 28.5 PSNR for 24 views on RealEstate10K is achieved with 1.79 GB peak GPU memory and under 78 milliseconds encoding time.The evaluation covers RealEstate10K and ACID against feed-forward 3DGS baselines.

2 Related Work

Related work spans optimization-based rendering, feed-forward reconstruction, and generalizable novel-view synthesis. The field has moved toward explicit, scalable scene representations, but dense intermediates and global attention remain important efficiency challenges.

  • Optimization-Based Novel-View Synthesis: Optimization-based methods represent scenes with implicit radiance fields or explicit 3DGS, with Gaussian splatting supporting efficient rendering.These methods generally perform per-scene optimization.
  • Feed-Forward 3D Reconstruction: Feed-forward reconstruction methods progressed from pixel-aligned pointmaps to multi-view global-attention approaches.Full global attention can cause memory and computation to grow rapidly with input length.
  • Feed-Forward Novel-View Synthesis: Earlier feed-forward NVS methods used plane-sweep volumes, MPIs, layered representations, and feature point clouds for fast synthesis.These representations can struggle with large baseline changes and complex occlusions.
  • Feed-Forward Novel-View Synthesis: Feed-forward 3DGS methods commonly rely on dense pixel- or view-aligned intermediates whose overhead grows with input-view count.Later work explores geometric robustness, Gaussian-level aggregation, voxel-aligned prediction, and joint pose estimation.

3 Method

GlobalSplat aligns multi-view information into fixed latent scene tokens before decoding explicit 3D Gaussians. Its architecture incorporates camera-aware inputs, geometry–appearance disentanglement, capacity scheduling, and jointly supervised reconstruction objectives.

  • Input Context Construction: GlobalSplat normalizes scenes and extracts ray-augmented patch features that combine RGB information with camera geometry.Camera tokens use Plücker-ray embeddings and per-view codes derived from camera centers and intrinsics.
  • Training Objective: Training combines rendering, self-supervised consistency, and regularization losses to refine appearance and geometry while constraining predicted Gaussians.Consistency compares geometry-derived opacity and depth maps across independently processed view subsets; regularization includes feature thresholding and a soft frustum constraint.
  • GlobalSplat Architecture: Dual-branch attention separately processes geometry and appearance streams before a mixer updates the latent tokens.Specialized decoder heads then produce geometric properties and texture properties from the refined streams.
  • Input Context Construction: Explicitly re-injecting absolute camera location and focal information improves performance in large-context settings.The camera code is formed from resolution-normalized intrinsics and Fourier-encoded camera centers, then added to every patch.
  • GlobalSplat Architecture: A fixed set of learnable latent tokens fuses multi-view features, keeping token count independent of the number of input views.The architecture sets M = 2048 tokens with dimension d = 512.
  • Coarse to Fine Training Curriculum: The coarse-to-fine curriculum begins by merging 16 candidates per latent slot into one Gaussian and increases capacity through G ∈ {2, 4, 8}.The final model uses G = 8 after global geometry has converged.

4 Experiments

GlobalSplat is evaluated on RealEstate10K and ACID through quantitative, qualitative, efficiency, and ablation studies. Results emphasize compact, view-invariant representations, competitive quality, cross-dataset robustness, and efficient inference, while identifying fixed-budget scalability as a limitation.

  • Efficiency: 1.79 GB peak memory, 77.88 ms inference time, and 3.8 MB disk size make GlobalSplat the most efficient reported method for 24 input views.The efficiency comparison reports peak GPU allocation, inference time, and on-disk size.
  • Quantitative Evaluation: GlobalSplat achieves competitive RealEstate10K quality with fixed 2K–32K Gaussian representations across 12, 24, and 36 input views.Several baselines substantially increase representation size as input views grow.
  • Cross-dataset Generalization: GlobalSplat maintains competitive zero-shot transfer to ACID across input-view settings using the same compact fixed-budget representation.The evaluation transfers from RealEstate10K to substantially different aerial and unbounded environments.
  • Qualitative Evaluation: GlobalSplat produces sharp, artifact-free indoor renderings, while C3G misses fine details and DepthSplat and GGN introduce structural artifacts.Zpressor achieves comparable visual fidelity but uses 393K Gaussians, versus GlobalSplat’s 16K.
  • Compactness-Quality Tradeoff: Increasing latent capacity improves reconstruction quality more than increasing decoded Gaussians per token under fixed Gaussian budgets.At both 2K and 32K budgets, allocating capacity to more latent scene tokens is more effective than increasing decoder density.
  • Model Ablation: Ablations show that the two-stream architecture, coarse-to-fine capacity growth, camera metadata, and consistency loss each contribute to the full model’s performance.Removing these components degrades quality, with consistency-loss removal also increasing structural artifacts.
  • Limitations: A fixed 16K-Gaussian budget may limit representation capacity for unbounded or city-scale environments.The authors suggest adaptive or hierarchical token allocation as future work.

5 Conclusion

GlobalSplat is an efficient feed-forward 3D Gaussian Splatting framework that aligns multi-view information into compact global scene tokens before decoding geometry. It achieves competitive synthesis quality with a compact representation and efficient inference.

  • GlobalSplat aggregates multi-view observations into fixed-size global scene tokens before decoding explicit 3D geometry.The framework uses an “align first, decode later” design to reduce redundancy in view-centric pipelines.
  • GlobalSplat uses a disentangled dual-branch encoder and coarse-to-fine training curriculum.The curriculum progressively increases decoded capacity while retaining a compact representation.
  • GlobalSplat achieves highly competitive novel-view synthesis quality on large-context scenes while capping the representation at 2K-32K Gaussians.
  • The resulting footprint is under 4 MB, with significantly faster inference and generation times, minimal memory utilization, and real-time rendering speeds.

A Qualitative Results on ACID

On ACID, GlobalSplat is compared qualitatively with four baselines across six scenes. Its reconstructions are visually close to ZPressor and sharper and more coherent than the other reported baselines.

  • The qualitative comparison covers GlobalSplat, Zpressor, DepthSplat, GGN, C3G, and ground truth across six ACID scenes.
  • GlobalSplat’s qualitative ranking is consistent with the RealEstate10K results, appearing visually close to ZPressor and better than the other baselines.
  • GGN often fails because output-space merging introduces unstable structure and visible artifacts.
  • DepthSplat and C3G produce oversimplified results because they lack sufficient expressive capacity for fine details and complex scene structure.
  • GlobalSplat produces sharper and more coherent reconstructions, indicating transfer across datasets.

B Implementation Details

Evaluation uses a deterministic resizing and cropping pipeline that produces exact 256 × 256 images while updating camera intrinsics after resizing.

  • Each image is resized to height 256 while preserving its aspect ratio.
  • The resized width is rounded to a multiple of the patch size, which is 8.
  • A centered square crop and, when needed, a final resize produce an exact 256 × 256 image.
  • Camera intrinsics are updated after resizing, and the same deterministic preprocessing is applied to context and target views.

B.1 Model Architecture and Decoder

The decoder maps latent scene tokens to Gaussian parameters through dual geometry and appearance branches, while stage-dependent merging and interpolation provide coarse-to-fine output control.

  • Gaussian parameterization: Each Gaussian is parameterized by a 3D mean, anisotropic scale, rotation, opacity, and view-dependent spherical-harmonic color.The spherical-harmonic representation uses degree 3.
  • Encoder: The encoder separates geometry and appearance processing, uses 8×8 image patches, and produces 512-dimensional latent scene tokens.It uses 2048 latent tokens and four iterative encoder rounds.
  • Coarse-to-fine decoding: The decoder predicts 16 Gaussian candidates for each latent token throughout training, rather than progressively introducing new Gaussians.
  • Coarse-to-fine decoding: Stage-dependent merging exposes progressively finer structure by reducing the amount of merging across stages.At stage 0, one strongly merged Gaussian is exposed; later stages reveal finer structure.
  • Dual-branch prediction: The geometry branch predicts centers, log-scales, rotations, opacity logits, and importance, while the appearance branch predicts spherical-harmonic coefficients.
  • Smooth stage transition: A smooth stage transition linearly interpolates between the previous coarser representation and the current less-merged representation.The transition coefficient λ ranges from 0 to 1.

B.2 Data Sampling and Training Setup

Training uses multi-view video samples with shared preprocessing and augmentation, while a staged schedule controls the number of decoded Gaussians per token.

  • Crop and resize: Images are cropped around the principal point, resized to 256 × 256, and augmented with a shared multi-view crop transformation during training.
  • View sampling: Training samples come from monocular video sequences using a random temporal window of 40–220 frames.
  • View sampling: 13 input views and 12 target views are sampled from each local segment to expose varied camera baselines while preserving geometric relatedness.
  • Training configuration: Color jitter uses identical brightness, contrast, saturation, and hue parameters across all views in a sample to preserve multi-view consistency.
  • Stage schedule: The training schedule uses a stage index to control exposed Gaussians per token, with transitions smoothed over 2k iterations.

B.3 Training Objective and Losses

The objective combines rendered-image supervision, subset consistency, and regularization of Gaussian geometry and parameters. Consistency is enforced in rendered space with symmetric stop-gradient supervision.

  • Rendering loss: Rendered reconstruction uses MSE and perceptual image losses between target images and rendered predictions.
  • Subset consistency loss: Subset consistency reconstructs shared target views from two overlapping, complementary input subsets of the same temporally ordered sequence.
  • Consistency loss: Rendered accumulation and depth maps receive symmetric stop-gradient consistency losses, optionally restricted to pixels with sufficient support.
  • Regularization loss: The regularization loss combines a soft frustum constraint with decoder-side penalties on Gaussian parameters.
  • Frustum constraint: The frustum penalty is zero for Gaussians inside at least one input-view frustum and smoothly penalizes unsupported predicted centers.
  • Decoder-side regularization: Decoder regularization stabilizes compact Gaussian optimization through opacity, scale, rotation, and spherical-harmonics terms.

C.1 Baselines Evaluation Details

Baseline evaluation uses official implementations, weights, or published results where available, with protocol-specific exclusions and adaptations documented for each method.

  • DepthSplat: DepthSplat is evaluated using its large model, official repository, and official weights under the 256×256 RealEstate10K protocol.
  • ZPressor: ZPressor uses the MVSplat+ZPressor variant trained only on RealEstate10K, with reported results for 6-anchor and 3-anchor settings.
  • GGN: GGN uses official code and checkpoints for valid 12- and 24-view settings, but its public implementation is omitted for unstable 36-view reconstructions.
  • C3G: C3G evaluation uses the released multiview Gaussian decoder checkpoint and includes post-inference pose optimization.
  • Published or adapted results: AnySplat and NoPoSplat table values are taken from the C3G benchmark because their official releases do not match the required experimental setup.
  • Published or adapted results: EcoSplat comparison values come from its original paper, while LVSM uses its official implementation and decoder-only 256 × 256 model.

C.2 Efficiency Benchmark Protocol

Efficiency is benchmarked on a single A100-based system using each model’s shipped utility and full test set. Reconstruction time measures scene-representation prediction after warm-up removal.

  • Hardware and implementation: All efficiency benchmarks use one NVIDIA A100 GPU with 64 GB VRAM, 8 CPU cores, and 128 GB system RAM.
  • Benchmark procedure: Each method is evaluated on the full test set with its respective evaluation pipeline using the benchmark utility shipped with the model.
  • Timing metrics: Reconstruction time excludes the first measured sample and denotes the time to predict the scene representation from input views.
  • Timing metrics: Peak GPU memory is recorded during the same evaluation process as reconstruction timing.
Loading 2604.15284v2…