Source-linked AI summary
Matryoshka Gaussian Splatting
Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Jeffrey Hu, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, Josef Bengtson, Chenliang Zhou, Weihao Xia, Cristina Nader Vasconcelos, Cengiz Oztireli
TL;DR
3DGS needs adjustable rendering fidelity, but discrete LoD exposes few operating points and existing continuous methods can reduce full-capacity quality. MGS learns ordered Gaussian prefixes with stochastic budget training, and experiments show backbone-level full-capacity performance together with continuous speed-quality control from one model.
Problem
3DGS LoD methods offer either limited discrete operating points or smoother scaling with often degraded full-capacity quality.
Method
MGS learns an ordered, prefix-closed set of Gaussian primitives and trains it with randomly sampled budgets, optimizing both a prefix and the full set without architectural changes.
Results
Across four benchmarks and six baselines, MGS matches the full-capacity reconstruction quality of its 3DGS backbone while enabling continuous quality-speed control from one model.
Takeaways & Limitations
Truncating MGS prefixes provides a dense spectrum of quality-speed operating points without per-budget retraining or auxiliary data structures.
Takeaways & Limitations
The MGS formulation depends on choosing a statistic s(g) that induces a meaningful importance ordering.
Abstract
from arXiv · showhide
The ability to render scenes at adjustable fidelity from a single model, known as level of detail (LoD), is crucial for practical deployment of 3D Gaussian Splatting (3DGS). Existing discrete LoD methods expose only a limited set of operating points, while concurrent continuous LoD approaches enable smoother scaling but often suffer noticeable quality degradation at full capacity, making LoD a costly design decision. We introduce Matryoshka Gaussian Splatting (MGS), a training framework that enables continuous LoD for standard 3DGS pipelines without sacrificing full-capacity rendering quality. MGS learns a single ordered set of Gaussians such that rendering any prefix, the first k splats, produces a coherent reconstruction whose fidelity improves smoothly with increasing budget. Our key idea is stochastic budget training: each iteration samples a random splat budget and optimises both the corresponding prefix and the full set. This strategy requires only two forward passes and introduces no architectural modifications. Experiments across four benchmarks and six baselines show that MGS matches the full-capacity performance of its backbone while enabling a continuous speed-quality trade-off from a single model. Extensive ablations on ordering strategies, training objectives, and model capacity further validate the designs.
1 Introduction
MGS addresses the limitations of discrete and existing continuous LoD for 3DGS by learning ordered Gaussian prefixes that support smooth budget control without sacrificing full-capacity quality. Its stochastic training procedure uses random budgets and two renders per iteration, and experiments show backbone-level quality with a continuous quality-speed frontier.
- Motivation: Real-time rendering budgets vary across hardware and runtime conditions, motivating LoD methods that adapt scene complexity to available resources.Discrete LoD can only expose fixed quality levels and may produce abrupt visual transitions as budgets change.
- Motivation: Conventional 3DGS has no primitive ordering, so removing splats for speed can rapidly collapse reconstruction quality.Existing discrete, pruning, and continuous approaches respectively offer coarse granularity, independently derived operating points, or degraded full-capacity quality.
- MGS: MGS learns an ordered Gaussian set whose prefixes remain coherent, with reconstruction fidelity improving smoothly as the splat budget increases.Rendering different budgets is achieved by truncating the ordered sequence, producing continuous quality-speed operating points from one model.
- Training: Stochastic budget training samples a random splat budget and optimizes both its prefix and the full set using only two forward passes per iteration.The procedure changes the training objective rather than the architecture and can integrate into existing 3DGS pipelines.
- Evaluation: Across four benchmarks and six baselines, MGS matches backbone full-capacity quality while enabling a continuous quality-speed frontier from one model.The evaluation includes ablations of importance scoring, budget sampling, and training objectives.
2 Related Work
Related work extends 3DGS through capacity control, LoD structures, compression, pruning, and nested representations. MGS transfers nested-prefix principles to Gaussian primitives for continuous rendering-budget control within one model.
- 3D Gaussian Splatting: 3DGS represents scenes with explicit anisotropic Gaussian primitives rendered through differentiable rasterization, with cost scaling directly with primitive count.This makes primitive count a natural axis for computational-budget control.
- Capacity Control: Capacity-control methods include fixed-budget sampling and improved densification, but the original 3DGS heuristics do not directly control final primitive count.3DGS-MCMC addresses capacity through Langevin-dynamics sampling under a fixed budget, while Mini-Splatting improves densification.
- Level of Detail: Discrete LoD and progressive streaming construct hierarchical or layered Gaussian structures but typically expose only a small number of quality levels and require auxiliary indices.Compression and pruning can approximate multi-budget behavior through importance-based pruning or quantization.
- Nested Representations: Nested representations make every prefix independently usable, as established by nested dropout and extended by Matryoshka Representation Learning.Slimmable networks similarly pack multiple capacity levels into a single model through different channel widths.
- MGS: MGS transfers the nested-representation principle to Gaussian scene primitives, enabling continuous rendering-budget control through ordered primitive prefixes.The approach packages a dense spectrum of operating points within one model.
3 Method
MGS combines an ordered Gaussian-prefix representation with stochastic budget training, enabling continuous rendering budgets from one model. Randomly sampled prefixes and full-set renders jointly train the representation with two forward passes per iteration.
- Nested Primitive Representation: MGS ranks Gaussians by an importance score and renders nested prefixes to support variable-budget scene representations.Each k-prefix contains the k highest-ranked Gaussians, and changing k traces a continuous quality-speed curve without per-budget retraining or model switching.
- Nested Primitive Representation: The method is agnostic to the scoring statistic, but uses descending opacity because it provides stable multi-budget behavior.Opacity is treated as a proxy for visibility and radiance contribution; alternative scoring criteria are evaluated in ablations.
- Stochastic Budget Training: At each step, stochastic budget training samples a random prefix size and optimizes both the prefix reconstruction and the full-set reconstruction.The full-set term anchors full-quality performance while the prefix term trains partial subsets; each step uses exactly two renders regardless of N.
- Stochastic Budget Training: Uniformly sampling prefix fractions covers every budget from the minimum training ratio through the full set with equal probability.Setting rmin = 1 recovers standard 3DGS training.
- Stochastic Budget Training: The Gaussian ordering is recomputed after each iteration so prefixes contain the most important primitives under the current parameters.Gradient updates change Gaussian parameters and therefore can change the importance scores during training.
4 Experiments
The experiments evaluate MGS across four standard 3DGS benchmarks against discrete and continuous Gaussian LoD baselines. They use PSNR, SSIM, LPIPS, and composite AUC measures to assess both image quality and quality–speed or quality–splat trade-offs.
- Benchmarks and baselines: MGS is evaluated on MipNeRF 360, Tanks & Temples, Deep Blending, and BungeeNeRF using the standard every-8th-image test split.Images with indices 0, 8, 16, and so on are reserved for testing.
- Benchmarks and baselines: The comparison includes four discrete LoD methods and continuous LoD methods that support arbitrary splat budgets.Discrete baselines include H3DGS, Octree-GS, MaskGaussian, and FlexGaussian.
- Evaluation metrics: Evaluation reports PSNR, SSIM, and LPIPS, alongside quality–speed trade-offs at different operating points.A composite quality score combines normalized PSNR, SSIM, and inverse LPIPS.
- Evaluation metrics: AUCfps measures the normalized area under a monotone quality-versus-FPS envelope clipped to 0–500 FPS.The envelope extends leftward because an operating point can be replicated at any lower speed.
- Evaluation metrics: AUCsplat measures the normalized area under a monotone quality-versus-splat-count envelope clipped to 0–5M splats.The envelope extends rightward from operating points and connects the origin to the lowest-budget point.
- Evaluation metrics: Both AUCfps and AUCsplat are multiplied by 100 for readability.
Implementation.
MGS is implemented in the gsplat codebase using the 3DGS-MCMC training strategy with fixed default ordering, weighting, capacity, and training duration.
- Implementation.: The default implementation orders splats by descending opacity, uses equal prefix/full weights with γ=1, and sets capacity to N=5M.Experiments use 50 k training iterations on identical Ubuntu servers with NVIDIA A100 GPUs.
4.2 Results and Comparisons
MGS delivers strong highest-budget image quality and better quality–budget behavior across the evaluated comparisons. Although Octree-GS can achieve higher PSNR on two benchmarks at its highest-quality level, its coarser levels degrade more sharply.
- Highest-budget quality: 28.20 dB PSNR, 0.841 SSIM, and 0.130 LPIPS are achieved by MGS on MipNeRF 360 at the highest splat-count operating point.MGS exceeds Octree-GS’s 27.62 dB PSNR by +0.58 dB and has lower LPIPS, 0.130 versus 0.221.
- Quality–budget trade-off: MGS’s continuous LoD curve achieves the highest quality at every speed and splat budget on MipNeRF 360 while spanning a wider FPS range than any baseline.The figure averages results across all nine MipNeRF 360 scenes and compares continuous curves with discrete operating points.
- Highest-budget quality: On Tanks & Temples, MGS trails Octree-GS by 0.03 dB PSNR while achieving the best SSIM and LPIPS.The reported PSNR values are 24.56 for MGS and 24.59 for Octree-GS.
- Highest-budget quality: Octree-GS obtains higher PSNR than MGS on Deep Blending and BungeeNeRF at its single highest-quality level.
- Quality–budget trade-off: Octree-GS’s coarser LoD levels incur severe quality degradation, producing far lower AUC scores than MGS on Deep Blending and BungeeNeRF.MGS consistently achieves the lowest LPIPS across the reported comparisons.
Quality–Speed Trade-off.
MGS sustains higher quality across varying budgets and generally outperforms continuous and discrete baselines on quality–speed trade-offs. It preserves coherent reconstructions at aggressive reductions, with DrJohnson as an exception at full budget.
- MGS outperforms all baselines in AUCfps and AUCsplats across four benchmarks.It sustains high fidelity across varying speed and splat-budget constraints.
- MGS produces a smooth, dense frontier from 1% to 100% prefix ratios, whereas discrete methods offer only a handful of operating points.The continuous frontier is generated from one trained model without per-budget retraining or model switching.
- At 5–10% budgets, MGS preserves coherent reconstructions while CLoD-3DGS and CLoD-GS exhibit severe artifacts.MGS achieves 21–28 dB PSNR, compared with 11–17 dB for both baselines.
- On stump, train, and rome, MGS maintains higher fidelity across all budget levels.On DrJohnson, CLoD-3DGS reaches higher full-budget PSNR, 29.1 versus 27.7 dB, but degrades more sharply at reduced budgets.
- Opacity-descending importance scoring dominates alternative importance scores across the full budget range.
Continuous vs. Discrete Operating Points.
MGS offers dense, continuous operating-point control from a single model, unlike methods exposing only a few fixed levels. Its full-capacity quality remains close to, and sometimes exceeds, the stand-alone 3DGS-MCMC backbone.
- MGS supports every integer splat budget, while Octree-GS, H3DGS, and FlexGaussian expose only 3–6, 9, and 2–6 levels respectively.Deployment requires simply truncating the splat array, without auxiliary structures, mode switches, or latency spikes.
- MGS trails 3DGS-MCMC by only 0.20 dB on MipNeRF 360 and Tanks & Temples.The reported pairs are 28.20 versus 28.40 dB and 24.56 versus 24.76 dB, respectively.
- MGS surpasses 3DGS-MCMC on Deep Blending and BungeeNeRF.The reported PSNR pairs are 28.41 versus 27.63 dB and 27.13 versus 27.04 dB, respectively.
4.3 Ablation Studies
The ablations evaluate importance scoring under controlled conditions and identify opacity-descending ordering as the strongest strategy for multi-budget performance.
- Experimental setup: The ablation uses the bicycle scene with N=1M splats and 50 k training steps unless otherwise noted.Design choices are isolated while other settings remain at their defaults.
- Importance scores: Seven importance scores are derived from opacity, volume, SH energy, and colour variance, with ascending and descending orders evaluated.
- Evaluation metrics: Table 2 reports raw quality with PSNR, SSIM, and LPIPS, and summarizes quality–efficiency frontiers with AUCfps and AUCsplats.All metrics are reported at full budget, with higher or lower being preferable according to each metric.
- Importance scores: At 10% budget, opacity-descending ordering achieves 22.2 dB PSNR at 493 FPS, versus 17.6 dB for SH-energy descending.Ascending order performs worse overall than the descending variant.
- Importance scores: At 10% budget, fixed-random ordering reaches 21.5 dB, below opacity-descending ordering at 22.2 dB.The comparison supports a semantically meaningful coarse-to-fine ordering for early-prefix quality.
Budget Training.
Budget-training ablations show that jointly optimizing prefix and full-set reconstructions provides the best overall trade-off. A balanced 5:5 weighting is used despite different endpoint preferences.
- Training strategy: Prefix + full training achieves the best trade-off between low-budget and full-quality performance.Prefix-only training improves early prefixes marginally but degrades quality at higher budgets.
- Loss weighting: At the 10% prefix, a 9:1 prefix-heavy ratio reaches 22.2 dB, while a 1:9 full-heavy ratio reaches 21.1 dB.
- Loss weighting: At full quality, the 1:9 ratio achieves 25.5 dB PSNR and 0.169 LPIPS, surpassing 9:1 by +0.2 dB.
- Loss weighting: A 6:4 Prefix/Full Loss Weight ratio provides the best area under the curve.The experiments use a 5:5 ratio for simplicity and balanced performance.
- Loss weighting: The default 5:5 ratio achieves 25.4 dB at full quality and 22.1 dB at 10% prefix quality.
5 Conclusion
MGS enables continuous budget control for 3DGS by learning an ordered, prefix-closed set of Gaussian primitives without architectural changes. Across four benchmarks, it matches full-capacity 3DGS performance while supporting arbitrary quality–speed operating points from one model.
- MGS learns an ordered, prefix-closed Gaussian representation that supports arbitrary splat budgets through prefix truncation.This produces a dense spectrum of quality–speed operating points from a single model.
- Stochastic budget training samples a random budget and jointly optimizes its prefix and the full set within existing 3DGS pipelines.The strategy requires no architectural changes and uses two forward passes per training step.
- MGS matches full-capacity 3DGS performance while enabling continuous level-of-detail control from one model.The conclusion reports this result across four standard benchmarks.
- Ablations validate the proposed ordering strategy, training objective, and budget-sampling designs.