Budgeted-GS: Real-Time Large-Scale Gaussian Splatting via Factoring LOD

Introduction

• The theory defines a measurable capacity floor with error falling as N^-2/dps and a collapse license for merging or dropping locally over-covered Gaussians with bounded volumetric loss.The floor derives from optimal transport in position–direction phase space; the collapse license uses the weighted Wang–Zahl union-volume estimate following the resolution of the three-dimensional Kakeya set conjecture.Source: §1 · p. 1
• Factoring LOD converts one trained model into a factoring tree whose one-parameter scheduler adjusts quality to fit the target GPU’s memory.The tree uses scale bands and moment-matched equivalent planks, with a pixel-footprint scheduler that sweeps a continuous memory–quality curve.Source: §1 · pp. 1–2
• Across 13 public scenes, published compressors lie on an essentially flat frontier, while a retrained official city model renders its test trajectory at native 1920×1080 with full SH in real time on one consumer GPU.The 13-scene evaluation used a preregistered protocol; the paper describes the city result as, to its knowledge, the first post-hoc, full-SH consumer-GPU result in this regime.Source: §1 · p. 2
• Budget-Centered Training trains the densifier directly at the floor-located budget, leading train-then-prune at every matched budget in a single training run.This avoids first growing a large model and then pruning it.Source: §1 · pp. 1–2

Related Work

• Classic point-cloud, mesh, and voxel systems use coarse-to-fine hierarchies with per-frame budgets, whereas raw 3DGS is a flat set of primitives.Examples include surfel splatting, QSplat, Potree, progressive meshes, Nanite, and sparse voxel octrees.Source: §2 · p. 2
• Existing 3DGS structures couple spatial organization to training, while Budgeted-GS builds post-hoc from any model and selects per leaf and scale-band.Its selection uses measurable redundancy and pixel-footprint criteria rather than fixed training-time levels or blocks; comparisons include Octree-GS and CityGaussianV2 runtime LOD.Source: §2 · p. 2
• Prior size-control methods act during training or export, but Budgeted-GS exposes a continuous, renderer-agnostic runtime memory–quality trade-off without an error bound from those methods.The related methods include MCMC-based densification [KRS∗24], pruning and compression approaches such as LightGaussian [FWW∗24], and 2DGS [HYC∗24]; LightGaussian’s score supports a Vanilla-flat budget axis down to 25% of primitives on city-scale scenes, and rasterizer accelerators remain composable with selection.Source: §2 · p. 2

3. Overview

• The capacity floor gives the minimal achievable rendering error for any scene at primitive count N, with per-scene measurement under a preregistered protocol.The floor is stated as theorems and measured in Section 6.1.Source: §3 · p. 2
• Budget-Centered Training reads the floor’s knee N* from the measured curve and trains the densifier directly at that budget.This avoids first growing a larger model and then reducing it, as described in the overview’s life-cycle framing.Source: §3 · p. 2
• Factoring LOD converts any trained model into a factoring tree, whose one-parameter pixel-footprint scheduler selects the per-view working set that fits available memory.Source: §3 · p. 2

4. An optimal-transport theory for 3D Gaussian primitives

• The theory predicts where reconstruction error flattens into a shallow floor and uses that floor to estimate a scene’s minimum primitive count, redundant primitives, and budget allocation across a city.It connects rendering and primitives through construction and capacity to selection rules.Source: §4 · p. 2

4.1. Rendering as optimal transport in phase space

• The theory formulates rendering as optimal transport in phase space P := X × Ω, where primitives occupy spatial-directional cells subject to an uncertainty bound and may be point, wave, or field primitives.The paper positions this phase-space rendering formulation as a first use of optimal transport for rendering.Source: §4.1 · pp. 2–3

4.2. An optimal-transport view of 3DGS

• The framework treats rendering as transport from a representation’s phase-space distribution to per-view pixel marginals, while forming a representation under a complexity budget is a separate transport-cost minimization problem.For compositing, representation loss decomposes into transport cost minus conditional variance, and duality supplies a certificate lower-bounding achieved error.Source: §4.2 · p. 3

4.3. The capacity floor

• On surface scenes, the capacity floor scales as R(N) ∝ N^-1/2, while rendering error is source-limited at N^-2/dps rather than the target-limited N^-2/dc, with dps ≥ dc.The lower bound assumes view completeness; the floor is certifiable through a duality chain that lower-bounds achievable error per budget and test view. The framework also predicts rates of N^-1 for triangles, N^-1/2 for point-like primitives, and N^-1/4 for isotropic voxels on surfaces, and an anisotropic covariance-ellipsoidal optimal cell.Source: §4.3 · p. 4

4.4. The selection rules

• The selection rules use weighted covering to certify when rendering primitives can be collapsed with at most an ε-fraction loss of volumetric mass, and allow sub-pixel footprints to be dropped or collapsed with bounded loss.For cartoon fields, an N-atom approximation has L2 error at most CK N^-1/2(logN)^c, while some fields require at least c0K(N log^2 N)^-1/2; budget allocation yields N* ≍ M^p/(p+1), L* ≍ M^1/(2(p+1)), and exponentially decaying scale-band budgets.Source: §4.4 · p. 5

5. Budgeted-GS: training and deployment

• Collapse licenses merging over-covered primitives, scale truncation licenses dropping sub-pixel content, and budget allocation favors coarser bands; synthetic-scene error rates match the prediction within measurement noise [WZ25].These operational rules are grounded in union-volume estimates and classical adaptive-approximation rates.Source: §5 · p. 5

5.1. Budget-centered training

• Budget-centered training measures the quality curve’s knee N∗, trains directly at that primitive cap, and prunes by local redundancy, stopping when validation improvement falls below 0.5 dB.The initial primitive count and refine-stop fraction are set as fractions of N∗, scaled by measured redundancy θwaste = Nopt/Nfull; opacity-based pruning targets high-multiplicity regions flagged by the θ map.Source: §5.1 · p. 5

5.2. The factoring tree

• The factoring tree partitions primitives by octree leaf and scale band, then moment-matches Morton-ordered buckets into aggregates; construction takes seconds and the representation is 119× smaller than input.Aggregates preserve weighted volume, centroid, inertia, and opacity-weighted SH color moments; raw primitives remain available for the finest tier.Source: Figure 3 · p. 6, §5.2 · p. 6

5.3. The pixel-footprint scheduler

• The pixel-footprint scheduler chooses raw, fine, or coarse rendering according to whether members, cluster aggregates, or neither remain resolvable at threshold t = c·d/f.The scalar c controls collapse: c = 1 is the exactly resolvable reference tier, larger values collapse more content, and distant aerial rendering uses c ≈ 0.3.Source: §5.3 · pp. 6–7

6.1. The floor, measured across 13 scenes

• Across 13 public scenes, floor intactness holds 13/13, while measured scene slopes fall outside the preregistered band even as steepest windows bracket the predicted −1/2 law.Published compression trajectories are much flatter, with slopes from −0.28 to +0.09 and mean −0.04; all 26 published operating points lie above the universal colour-quantization floor.Source: §6.1 · p. 7

6.2.1. Outdoor extension

• On the three outdoor scenes, BCT and the matched train→prune→fine-tune scan agree within 0.15 dB, while published compressor points sit 0.8–2.9 dB above that curve.On stump, the published points are 2.7–2.9 dB above the measured floor at matched budgets, and measured training-time redundancy accounts for the gap.Source: §6.2.1 · p. 8

6.2.2. Budget-aware density control at the ceiling

• Budget-aware density control reaches 36.85 dB on lego with 151k primitives, matching Vanilla’s quality at 339k primitives; across indoor scenes, score ranking beats opacity pruning by +2.7 to +7.9 dB.At matched 70% budgets, θ-pruning improves bonsai/room PSNR over random pruning by +3.0/+6.1 dB; on room, a BCT-born factoring tree renders at 34.19 dB, within 0.16 dB of its root, versus 22.83 dB for a full-model-born tree at the same budget.Source: §6.2.2 · p. 8, §6.2.3 · p. 8

6.3. The LOD quality window

• On MatrixCity Aerial, c=4 renders 68% of primitives at −0.05 dB, while the full ladder provides a continuous memory–quality curve and a ∼20.8 dB point at 110.6 MB.The latter is 17× lower than GPS’s full 1926 MB; on garden, residual-proxy adaptive selection recovers 0.49 dB over a fixed threshold at a comparable primitive count (1.05×).Source: §6.3 · p. 8, Table 2 · p. 9

6.2. Training-side results

• At the matched 529k room budget, BCT reaches 31.94 ± 0.04 dB versus Compact3DGS at 30.23 dB, and beats published compressor points on five of six scenes.BCT leads the train-then-prune pipeline at every matched budget; its room run takes 78 ± 7 minutes versus approximately 175 minutes for the traditional route, which is 2.2× slower and lower quality.Source: §6.2 · pp. 7–8, Table 1 · p. 9

6.5. Pruning baselines

• On Block_A, the contribution-scored export matches Vanilla at 50% and 25% of primitives and beats LightGaussian, RAP, and the PUP score at 10%; its 10% PSNR margins are +0.45, +0.80, and +0.37 dB, respectively.The related systems differ in how they obtain LOD: Octree-GS [RJL∗24] and CityGaussian [LGL∗24] / V2 [LLM∗25] couple it to training, while this layer is post-hoc and renderer-agnostic.Source: §6.4 · p. 8, §6.5 · p. 9

6.6. Real-time rendering at city scale

• At native 1920 × 1080 with full SH, the 6.89M-primitive model renders at 71 fps versus Octree-GS at 11.5 fps on the official 207-view trajectory.Figure 4 shows the same-view comparison; the post-hoc deployment also selects among memory budgets without retraining or re-export.Source: §6.6 · pp. 10–11, Figure 4 · p. 11

6.7. Ablations

• Ablations show that the rate-based stop rule prevents a 0.56 dB over-pruning loss, while a DC-only tree loses 5 dB; construction peaks at 6.45 GiB on an 8 GB RTX 3060 Ti.The 7.07M-primitive Block_all v2 tree also builds on that consumer card.Source: §6.7 · p. 11, §6.8 · p. 11

• Evaluation uses a single GPU, omits transparent and occluded-region extremes, and leaves memory streaming—GPS’s native regime—for future work.Source: §7 · p. 11
• The proof covers compositing renderers, atomic and field representations, and the consistency functional, but leaves three issues open.These are the sharpness of the visibility-exponent bound, SGD trainability of implicit fields at the proved N−1/2 rate, and non-compositing renderers such as caustics and participating media.Source: §7 · p. 11
• The measured −1/2 bracket holds on all 13 scenes within the registered reading window; indoor scenes read −0.457 ≈−1/2.Source: §7 · p. 11

8. Conclusion

• The capacity floor derives from optimal transport, with the Zador law locating it and collapse, truncation, and allocation rules translating it into executable permissions.The paper frames this as a principled answer to what can be given up when city-scale 3DGS scenes exceed consumer GPU capacity.Source: §8 · pp. 11–12
• Budget-centered training grows a model directly at the floor’s prescribed budget, while Factoring LOD makes one trained model a deployment-time knob across continuous memory budgets.Factoring LOD renders a full city in real time on a consumer GPU.Source: §8 · p. 12
• Future work includes out-of-core memory streaming, hysteresis and perceptual validation for tier transitions, and extending phase space to time and more primitive families.Source: §8 · p. 12

Proof scope

• The appendix gives full proofs for the loss identity, duality certificate, transport theorem, Zador construction, solver descent, and taxonomy, but only proof sketches for the visibility-cell rate and new-view bound.It cites Zador’s law, the Wang–Zahl covering estimate, and DeVore’s width bounds to their classical sources.Source: S1. Proofs for the capacity-floor chain · p. 14

Loss identity and duality certificate

• The conditional expectation of the rendering coupling gives rendered colour, and expanding its transport cost yields L = C − V; dual-feasible pairs lower-bound Cfree, with C ≥ Cfree.The certificate follows by combining that inequality with the loss identity.Source: S1. Proofs for the capacity-floor chain · p. 14

Zador law

• For the rendering instance, squared error satisfies R(N) ≤ C_dps ∥λ∥^{dps/(dps+2)} N^{-2/dps}; under view completeness, its matching sharp-constant lower bound is Zador’s theorem [Zad82, GL00].Source: S1. Proofs for the capacity-floor chain · p. 14

Solver convergence and consistency

• Masked visibility-cell atoms achieve L2 ≤ CK_occ^1/2 N^-1/2(logN)^c, while the new-view bound adds angular-regularity and SH/visibility-capacity terms to training error.The visibility-cell rate and new-view bound remain proof sketches in this appendix.Source: S1. Proofs for the capacity-floor chain · p. 14

Taxonomy of primitive families

• Point, wave, and field primitives embed in canonical form (Q,w,ρ); on surfaces, value-only atoms achieve N^-1/2, first-order elements N^-1, and isotropic voxels N^-1/4.The ellipsoidal cell-density law ρA ∝ λ^1/2 is presented as a prediction because Gersho-type cell shapes are conjectural beyond dimension one.Source: S1. Proofs for the capacity-floor chain · pp. 14–15

Solver convergence and consistency

• Gradient descent is monotone for η < 2/L, and every accumulation point satisfies ∇C = ∇V; sampling and meshing achieve N^-2/d and N^-1 on C2 surfaces, respectively.Lloyd steps are exact block minimizations, so iterates are monotone and stationary quantizers are fixed points.Source: S1. Proofs for the capacity-floor chain · p. 14

S3. Measurement appendix

• The appendix records the preregistered Section 6.1 protocol and its four checks, and archives per-band floor curves, the estimate-bias study, downstream uses, and JSON certificates in supplementary artifacts.Source: S3. Measurement appendix · p. 15

S3.1. Protocol and records

• The floor-measurement study preregistered its criteria and froze verdicts before data collection, a discipline also used for other measurements.The authors state that the floor is useful only if it can be verified on real scenes; they refer to Appendix S5 for the other measurements.Source: S3.1. Protocol and records · p. 15

S3.2. Protocol

• Nine budgets trace R(N) after MCMC training at a cap of 2 × 106 primitives, opacity-pruning to N, and fine-tuning; no scan point may cross the fitted floor by more than log10(0.8).The floor-intactness check is applied on each scene.Source: S3.2. Protocol · p. 15
• Published operating points are re-anchored to each paper’s own baseline at matched capacity ratio, with 5% tolerance corresponding to ≈0.22 dB.Source: S3.2. Protocol · p. 15
• Duality certificates lower-bound achieved error at every budget and view.Source: S3.2. Protocol · p. 15

S4.2. Weighted covering

• Weighted covering treats primitives as opacity-weighted planks and certifies when over-covered regions can be collapsed with at most an ε-loss of volumetric mass.The computable redundancy signal θ supports both factoring and the training-side stop rule; the regional over-packing bound controls scale-band decomposition.Source: S4.1. From primitives to planks · p. 15, S4.2. Weighted covering · p. 16

S4.3. Adaptive covering rates

• For piecewise C2/H2 cartoon fields, the adaptive covering upper bound is O(N^-1/2(log N)^c), while a lower bound is Ω((N log^2 N)^-1/2); synthetic validation measured slopes −0.90 to −1.09.The scale-truncation lemma permits dropping atoms below vmin=(ε/C6K)^2 with loss at most ε/2, licensing pixel-footprint pruning; the upper/lower log exponents remain unpinned.Source: S4.3. Adaptive covering rates · pp. 16–17, Figure S1 · p. 16

S4.4. Optimal budget allocation

• Under budget M ≍N(L + 1)^2, optimal allocation scales as N*≍M^(p/(p+1)) and L*≍M^(1/(2(p+1))) to balance geometric and appearance error.For scale bands, coarser bands receive exponentially more budget, instantiated per view by the pixel-footprint resolvability threshold.Source: S4.4. Optimal budget allocation · p. 17

S4.5. Rendering-side error control

• The rendering certificate bounds image change through transferred field error and a model-operator gap, but is loose in practice; stored geometric-spread and SH-residual statistics serve as empirical switch proxies.The certificate was never violated in measured real scenes, though the passage identifies its looseness as a limitation.Source: S4.5. Rendering-side error control · p. 17

S9. Timing of BCT vs. the traditional route

• On room@529k, BCT took 78±7 minutes versus about 175 minutes for train–prune–fine-tune and reached 31.9 versus about 30.5 dB; tree defaults varied garden sanity PSNR by only 0.23 dB.Capped MCMC had 2 collapses in 12 runs in one batch, but none in 24 formal-pipeline runs; same-seed retries recovered expected quality.Source: S7. Run-to-run stability of capped MCMC · pp. 18–19, S9. Timing of BCT vs. the traditional… · p. 19, S8. Tree hyper-parameter sensitivity · p. 19

S10. Scheduler enhancements and export

• Adaptive selection uses geometric-spread and SH-residual proxies, improving garden fixed-threshold LOD by +0.49 dB at 1.05× primitive count; geometric-only pruning can discard visually load-bearing primitives.The passage flags occlusion as an unresolved component of the error control.Source: S10. Scheduler enhancements and export · p. 19
• Render-aware ranking keeps Block_A at Vanilla quality down to 25% of primitives, with 24.3% scoring zero because they are never visible in training views; exported tiers can be consumed by standard splatting rasterizers.The one-pass LightGaussian-significance scan took 234 s for 4565 views at 7.94M primitives on an RTX 4080 SUPER.Source: S10. Scheduler enhancements and export · p. 19

S11.1. Memory–quality Pareto vs. GPS on MatrixCity

• On MatrixCity, the LOD budget–quality curve spans 1240 MB to 34 MB at 28.4 to 17.5 dB, stays near-lossless through c ≤4, transfers to GPS within 0.5 dB, and beats random subsampling at matched budgets.The passage also states c=4 is lossless under both renderers; it does not provide the final numerical margin in the same-budget comparison.Source: S11.1. Memory–quality Pareto vs. GPS… · p. 19
• At 8.9% sampling, random sampling reaches 12.9 dB while LOD retains 20.9 dB; matching ∼20.8 dB takes 110.6 MB versus GPS full’s 1926 MB, a 17× saving.The comparison holds the trained model, renderer, cameras, and ground truth constant; LOD exports are re-rendered through GPS itself.Source: S11.1. Memory–quality Pareto vs. GPS… · p. 20, Figure S15 · p. 28
• On lego, BCT is within 0.34 dB of Vanilla at 44% of primitives and beats MCMC with 12% fewer; across four other scenes, it loses 0.06–0.33 dB at 36–66% of Vanilla budget.Table S5 also reports that Taming needs 39% fewer primitives but loses 0.74 dB; density-controller baselines do not expose a primitive budget.Source: Table S5 · p. 21, Table S6 · p. 23
• On the garden orbit, adaptive selection stays at 41–53 dB as compression saturates near 1.4×, while fixed c=2 reaches 4.7× but loses fidelity; indoor near-field redundancy saturates sooner.The indoor room example shows visible degradation by c=8, consistent with the faster saturation reported for indoor scenes.Source: Figure S7 · p. 22, Figure S10 · p. 25
• At matched budgets, contribution-scored pruning beats opacity-ordered pruning by 2.7–7.9 dB across every CityGaussian scene and budget; on Block_A, the score export matches Vanilla at 50% and 25% and dominates PSNR/SSIM at 10%.The Block_A table uses DC-only evaluation against photographs, so it cautions that absolute values are low and relative comparisons are meaningful.Source: Table S7 · p. 23, Table S9 · p. 26
• At peak VRAM, the reported MatrixCity comparison is 30.4 GiB for Octree-GS versus 4.4 GiB for Budgeted-GS, 6.9× lower for Budgeted-GS.Octree-GS reports 22.95/11.19/6.37/5.50/5.47 fps across levels 0–4; Budgeted-GS reports 71.0 fps mean on its official 207-view trajectory, with full SH at native 1080p. The methods use their respective official paths, and VRAM was sampled at 1600×900.Source: Table S8 · p. 23