Source-linked AI summary
ZipSplat: Fewer Gaussians, Better Splats
Alexander Veicht, Sunghwan Hong, Dániel Baráth, Marc Pollefeys
TL;DR
Pixel-aligned feed-forward 3DGS ties representation size to image resolution rather than scene complexity, creating redundant and inflexible Gaussian budgets. ZipSplat replaces that grid with dynamically clustered scene tokens that decode to unconstrained 3D Gaussians and adjustable budgets. It achieves state-of-the-art pose-free reconstruction with substantially fewer Gaussians and generalizes zero-shot across additional datasets.
Problem
Pixel-aligned Gaussian prediction ties representation budgets to camera resolution, causing equal capacity for regions with different geometric needs and redundancy across overlapping views.
Method
ZipSplat clusters dense multi-view visual tokens into scene tokens, refines them with attention, and decodes them into groups of Gaussians with unconstrained 3D positions.
Results
ZipSplat achieves state-of-the-art pose-free novel view synthesis on DL3DV and RealEstate10K with up to 6× fewer Gaussians than pixel-aligned baselines and zero-shot generalization to Mip-NeRF360 and ScanNet++.
Takeaways & Limitations
Inference-time clustering provides continuous control over the quality–efficiency trade-off with a single trained model, while token-based placement adapts capacity to scene content.
Takeaways & Limitations
The compact representation is presented as opening future-work paths for semantic scene understanding and 4D dynamic scenes, rather than as evaluated capabilities.
Abstract
from arXiv · showhide
Feed-forward 3D Gaussian Splatting methods reconstruct a scene from posed or pose-free images in a single forward pass, yet current approaches predict one Gaussian per input pixel, tying the representation budget to camera resolution rather than scene complexity. A flat wall and a richly textured object thus produce equally many Gaussians despite very different geometric needs. We propose ZipSplat, a token-based feed-forward model that decouples Gaussian placement from the pixel grid. A multi-view backbone extracts dense visual tokens, and k-means clustering compresses them into a compact set of scene tokens. Cross- and self-attention refine these tokens, and a lightweight MLP decodes each into a group of Gaussians with unconstrained 3D positions. Because clustering is applied at inference, a single trained model spans the quality-efficiency curve without retraining. ZipSplat operates without ground-truth poses or intrinsics, yet sets a new state of the art on DL3DV and RealEstate10K with ${\sim}6{\times}$ fewer Gaussians than pixel-aligned methods, surpassing the best pose-free baseline by 2.1dB and 1.2dB PSNR, respectively. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++, outperforming all comparable baselines. Our project page is at https://veichta.com/zipsplat.
1 Introduction
ZipSplat addresses the inefficiency of pixel-tied Gaussian prediction by representing scenes with compact, content-adaptive tokens and unconstrained 3D Gaussian placement. This design reduces redundancy while retaining strong pose-free reconstruction quality across benchmarks.
- Pixel-aligned methods tie Gaussian budgets to camera resolution, giving flat and richly textured regions equal capacity despite different geometric needs.
- Overlapping views duplicate Gaussians and ray-anchored predictions limit coverage of occluded or unobserved regions.
- ZipSplat decouples Gaussian placement from pixels by refining visual tokens into scene tokens and decoding them into groups of Gaussians with unconstrained 3D positions.
- 6× fewer Gaussians accompany state-of-the-art pose-free novel view synthesis on DL3DV and RealEstate10K, with stable quality as context views grow.
- Feature-space clustering lets one trained model adjust its Gaussian budget continuously at inference without retraining.
2 Related Work
Prior compacting methods reduce redundancy after generating pixel-aligned Gaussians, while ZipSplat predicts a compact representation directly from dynamically formed scene tokens. Its token budget adapts to scene complexity rather than relying on fixed queries or post-hoc reduction.
- Feed-forward 3DGS methods predict Gaussians from sparse images in a single forward pass, extending earlier geometry backbones to high-resolution multi-view inputs.
- Existing compact-representation methods prune, voxel-merge, fuse, deduplicate, or separately compress Gaussians after the full per-pixel set is predicted.
- The pipeline compresses dense visual tokens into scene tokens, refines them with attention, and decodes each into groups of Gaussians with free 3D positions.
- ZipSplat predicts the compact set directly by decoding Gaussians from learned tokens rather than pixel rays.
- Unlike fixed scene-independent queries, ZipSplat forms tokens dynamically from multi-view representations so the budget scales with input views and scene complexity.
3 ZipSplat: Decoupling Gaussians from the Pixel Grid
ZipSplat replaces pixel-anchored Gaussian prediction with compact scene tokens that decode into unconstrained 3D Gaussian groups. Clustering and adaptive placement reduce redundancy while preserving detail, supported by geometric and rendering supervision.
- Motivation: Most feed-forward methods place each Gaussian along a pixel’s viewing ray, tying representation size to camera resolution rather than scene content.This gives each prediction an observed-surface anchor but assigns equal capacity to regions with different geometric complexity.
- Scene tokens: ZipSplat extracts dense multi-view visual tokens, compresses them with k-means into K = r·T scene tokens, refines them with cross- and self-attention, and decodes each into G Gaussians.The compression ratio r is set at inference, allowing the Gaussian budget to change without retraining.
- Decoding Gaussians: The decoder predicts Gaussian parameters directly in 3D, freeing placement from viewing rays and concentrating Gaussians on detailed geometry while using fewer on flat surfaces.Each token produces a group of Gaussians whose positions and shapes adapt to local geometry.
- Decoding Gaussians: Gaussians decoded from one token self-organize into broad groups on flat surfaces and compact groups around fine details and edges without explicit spatial supervision.The shared token representation therefore corresponds to coherent spatial regions despite producing multiple primitives.
- Training: Free placement is stabilized with geometric supervision, initialization, and progressive scheduling, alongside photometric, perceptual, and depth rendering losses.The one-directional Chamfer loss pulls predicted Gaussian means toward valid scene surfaces without forcing uniform coverage of all ground-truth points.
- Results: ZipSplat maintains sharper reconstructions as input views increase, while YoNoSplat and DA3 blur despite using many more Gaussians; C3G is limited by 2K fixed Gaussians.The qualitative comparison spans 6 to 128 input views on DL3DV.
4 Experiments
ZipSplat is evaluated against pose-free and posed feed-forward baselines on in-domain and unseen benchmarks, with analyses of efficiency, generalization, and design choices. It delivers higher reconstruction quality with substantially fewer Gaussians and supports adjustable compression at inference.
- Multi-View Novel View Synthesis: ZipSplat outperforms every pose-free DL3DV baseline and uses 6× fewer Gaussians than per-pixel methods at 6 views.It predicts 62K versus 393K Gaussians, and also surpasses posed DepthSplat and MVSplat in PSNR and SSIM.
- Multi-View Novel View Synthesis: 1.2 dB: ZipSplat reaches 26.20 PSNR on RealEstate10K without poses, ahead of YoNoSplat at 24.99 PSNR.It reaches 27.19 with camera priors, while the posed DepthSplat result is 24.16 PSNR.
- Cross-Dataset Generalization: 18.1 PSNR: ZipSplat generalizes zero-shot to ScanNet++ without poses and reaches 23.5 PSNR with ground-truth poses.On Mip-NeRF360, quality improves from 21.72 to 22.29 PSNR between 32 and 128 views, while baselines plateau or decline.
- Adjustable Gaussian Budget: 22.52 vs. 22.01 PSNR: at r=0.2 with 24 input views, ZipSplat surpasses YoNoSplat with roughly 24× fewer Gaussians.The corresponding Gaussian counts are 50K versus 1.2M.
- Token Selection: At r=0.1, k-means exceeds stride by 0.45 PSNR and random selection by 0.79 PSNR under aggressive compression.At this ratio, each cluster draws from 4.2 views on average for 24-view input, improving cross-view coverage.
- Ablations: 1.5 to 2.7 dB: ZipSplat improves PSNR over matching-backbone per-pixel methods while predicting 13 to 25× fewer Gaussians.The improvement holds for both compared backbones, suggesting the gain stems from the token formulation.
5 Conclusion
ZipSplat decouples Gaussian placement from the 2D pixel grid by predicting unconstrained Gaussians from compact scene tokens. This yields pose-free novel view synthesis with fewer Gaussians and supports future extensions to semantic understanding and dynamic scenes.
- ZipSplat predicts unconstrained Gaussians from compact scene tokens rather than tying primitive placement to the 2D pixel grid.
- Up to 6× fewer Gaussians than pixel-aligned baselines accompany state-of-the-art pose-free novel view synthesis.
- Capacity is allocated according to 3D geometric complexity rather than camera resolution, producing compact and scalable representations from sparse multi-view input.
- ZipSplat’s compact representation may benefit semantic scene understanding and token clustering may extend to compress redundant observations in 4D dynamic scenes.
- Predicting positions directly in 3D loosens the coupling between geometry and pose or depth errors compared with ray-uplifting.
A Additional Implementation Details
ZipSplat uses a pretrained multi-view backbone and attention-based token aggregation, then decodes scene tokens into groups of parameterized Gaussians. Training combines RealEstate10K and DL3DV with scheduled view counts and compression.
- The architecture uses a pretrained DA3-Giant backbone, multi-scale visual tokens, and three cross-then-self-attention blocks for token aggregation.The backbone has 40 layers, width 1536, and 24 heads; tokens come from layers 19, 29, and 39.
- A two-layer MLP decodes each token into G=32 Gaussians with 23 parameters each.
- Gaussian parameters use constrained activations for positions, scales, rotations, opacities, and degree-1 spherical-harmonic colors.Initialization shares a template across 32 Gaussians, while hinge penalties regularize scales and opacities.
- The input view count increases from 2 to 24 while the compression ratio follows a cosine schedule toward r_min=0.5.Each step renders four target views, and pseudo-ground-truth depth for geometric supervision comes from DA3-Giant using ground-truth poses.
B Evaluation Details
Evaluation follows a pose-alignment protocol for sparse pose-free reconstruction and compares methods at a common 252×252 metric resolution. The protocol also records each method’s native encoding and evaluation pipeline.
- Predicted Gaussians are frozen while each target camera pose is optimized for 200 steps using L1 and LPIPS losses with early stopping.A single checkpoint is used across all benchmarks.
- Table 5 reports DL3DV results for 6, 12, and 24 input views under bilinear-plus-antialiasing target-image filtering.Each row combines a method with an evaluation pipeline, and columns report source, encoder, and evaluation resolutions.
- Table 6 isolates resize-filter effects by changing only the ground-truth filter while keeping model, checkpoint, indices, pose alignment, and other settings identical.
- All methods are scored at a common 252×252 resolution while their encoders run at their native training resolutions.The protocol notes that higher-resolution baselines retain a slight edge under this comparison.
B.2 Resize filter
Ground-truth resize filtering materially affects PSNR, making filter reporting necessary for cross-paper comparison. The accompanying figures describe inference and representation efficiency as view counts change.
- Up to 1 dB PSNR changes when switching from PIL Lanczos to bilinear with antialiasing make resize-filter reporting essential for comparison.ZipSplat uses bilinear with antialiasing to match the DL3DV source ground truth.
- The backbone dominates forward-pass time and peak GPU memory across view counts, while token compression and Gaussian decoding remain negligible.The view-dependent schedule keeps memory within 24 GB even at 192 views.
- At 192 views, scaled compression delivers a 10× rendering speedup with 20× less storage.View-dependent scaling merges redundant cross-view observations and keeps storage nearly flat.
C Inference Time Analysis
ZipSplat’s inference cost is dominated by the backbone, while compression keeps later stages practical and substantially reduces representation cost. At moderate view counts, all tested compression ratios remain fast and memory-efficient.
- Inference cost: 95% of forward time at 192 views comes from the backbone under scaled compression, while post-backbone stages remain negligible.The backbone takes 3.9 s of a 4.1 s forward pass; without compression, post-backbone stages rise to 44% because they process 20× more tokens.
- Inference cost: Under scaled compression, 24-view inference completes in under 0.33 s and uses less than 8.6 GB of memory.
- Compression behavior: At extreme compression r=0.01, too few tokens remain to cover the scene, causing novel-view renders to drop entire regions.The supplied figure compares r=1, r=0.1, and r=0.01 on a 24-view scene.
- Inference cost: 13.2 GB peak memory under scaled compression at 192 views compares with 15.7 GB without compression and 18.9 GB for YoNoSplat.
- Output representation: At 192 views, scaled compression produces 102 K Gaussians, 9.3 MB storage, and 401 FPS, versus 2 M Gaussians, 183 MB, and 40 FPS without compression.This corresponds to 20× less storage and a 10× rendering speedup.
D Behavior at Extreme Compression
ZipSplat remains robust well below its training compression range, but extreme compression eventually leaves too few tokens to cover the scene. The failure threshold depends on scene overlap and complexity, and k-means becomes increasingly valuable at low ratios.
- Failure boundary: ZipSplat degrades gracefully to r=0.1, but at r≈0.01 novel-view renders can lose entire scene regions.The onset depends on scene overlap and complexity; denser, more redundant captures tolerate more aggressive compression.
- Token selection: K-means selection’s advantage over random selection grows from 0.45 dB at r=0.2 to 1.09 dB at r=0.05.Mutual nearest-neighbor matching and HDBSCAN did not outperform k-means.
E Additional Qualitative Results
Additional qualitative comparisons cover DL3DV, RealEstate10K, and failure cases. ZipSplat is reported to produce sharper details and more coherent geometry than the compared baselines, while failures arise under challenging scene content and coverage.
- Evaluation scope: The qualitative-results section reports comparisons on DL3DV and RealEstate10K alongside the failure cases.
- DL3DV: Across DL3DV scenes with 6, 12, and 24 input views, ZipSplat produces sharper details and fewer artifacts than YoNoSplat, DA3, and C3G.
- RealEstate10K: On RealEstate10K with 6 input views, ZipSplat recovers sharper details and more coherent geometry than AnySplat and C3G.AnySplat cannot extend into occluded regions and suffers from bad alignment.
- Failure cases: Failure cases include uneven Gaussian allocation, lost detail in high-frequency regions, and degraded quality with low context overlap or moving objects.