PaperlayerView exact revision on arXiv

Source-linked AI summary

RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting

Injae Kim, Minseok Joo, Jihwan Park, Sehyung Kim, Minseong Bae, Hyunwoo J. Kim

arXiv:2610.27712v1cs.CVcs.AI

TL;DR

Budget-controllable FF3DGS methods adjust output size but still incur nearly constant inference latency and peak memory because they generate dense candidates. RLSplat learns to allocate exactly the requested number of Gaussians and decode them with resource-linear cost. It achieves the best rendering quality at every tested budget, with lower memory and latency than F4Splat at 2K.

  • Problem

    Budget-controllable FF3DGS methods produce dense Gaussian candidates before pruning or selection, leaving inference latency and peak memory nearly constant across budgets.

    Source: §1 · p. 2

  • Method

    RLSplat learns a policy to allocate exactly B Gaussians across local regions and refines them with a resource-linear decoder using local cross-attention.

    Source: §1 · pp. 2–3

  • Results

    Across tested budgets, RLSplat achieves the best rendering quality; at 2K, it improves PSNR by 0.9 dB over F4Splat while using 9.6× less peak memory and running 4.1× faster.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    RLSplat can serve as a 3D scene representation for downstream tasks with their own resource limits, such as 3D vision-language models and robot learning.

    Source: A Limitations and Future Work · p. 13

    Image encoding remains a fixed cost independent of B, bounding speed-up at small budgets; evaluation uses 256 × 256 resolution in a single cross-dataset setting.

    Source: A Limitations and Future Work · p. 13

Abstract

from arXiv · show

Feed-forward 3D Gaussian Splatting (FF3DGS) reconstructs 3D scenes from images in a single forward pass, making it well-suited to applications that must operate within limited resource budgets. To further improve its practicality, recent budget-controllable FF3DGS methods allow the number of Gaussians to be adjusted at inference, so that the output meets a given storage limit or rendering-latency constraint. However, these methods overlook the cost of inference itself: to produce a representation with the target number of Gaussians, they first predict a dense set of Gaussians and then prune or select from it, requiring nearly constant latency and peak memory regardless of the budget. We therefore propose RLSplat, a resource-linear budget-controllable FF3DGS framework whose representation size, inference latency, and peak memory all decrease linearly as the Gaussian budget is reduced. Instead of pruning or selection, RLSplat learns a Gaussian allocation policy that distributes the budget across local scene regions, so that exactly the requested number of Gaussians is generated. These Gaussians are then refined through resource-linear Gaussian decoding, in which each Gaussian attends only to nearby Gaussians found via an efficient hierarchical neighbor search, so that the decoding cost decreases linearly with fewer Gaussians. In zero-shot evaluation, RLSplat outperforms prior budget-controllable methods at every budget, surpassing the strongest baseline even with 4$\times$ fewer Gaussians, while running up to 4.1$\times$ faster with 9.6$\times$ less peak memory at low budgets.

1 Introduction

RLSplat addresses the inference cost overlooked by budget-controllable FF3DGS methods that generate dense candidates before pruning or selection. It allocates exactly the requested Gaussian budget and decodes those Gaussians with resource-linear cost, outperforming prior budget-controllable methods across budgets.

1 Introduction

  • Prior budget-controllable methods first predict dense Gaussian candidates and then prune or select them, leaving inference latency and peak memory nearly constant as the budget changes.This means they control output size but overlook the cost of producing it.Source: §1 · p. 2
  • Each Gaussian is refined through local cross-attention over Kgs nearby tokens found by hierarchical neighbor search, reducing decoder cost from O(B2) to O(BKgs).This makes decoding cost scale linearly with the Gaussian budget.Source: §1 · pp. 2–3
  • RLSplat makes representation size, inference latency, and peak memory decrease linearly as the Gaussian budget is reduced.It replaces pruning or selection with a resource-linear approach to generating the requested representation.Source: §1 · p. 2
  • A learned allocation policy assigns exactly B Gaussians across local scene regions rather than selecting from a larger predicted set.The policy is trained with GSPO using rendering quality as its reward.Source: §1 · pp. 2–3
  • At 2K Gaussians, RLSplat improves PSNR by 0.9 dB over F4Splat while using 9.6× less peak memory and running 4.1× faster.It consistently outperforms prior budget-controllable methods across all budgets.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Related work moves from feed-forward prediction toward compact and budget-controllable Gaussian representations, while policy optimization provides a way to learn discrete allocations from rendering rewards.

Feed-Forward 3D Gaussian Splatting

Compact and Gaussian budget-controllable FF3DGS

Policy optimization

  • Policy-gradient methods optimize discrete decisions from scalar rewards, and RLSplat uses GSPO to learn Gaussian allocation with rendering quality as the reward.GSPO normalizes rewards within a group and defines and clips importance ratios at the sequence level (Zheng et al., 2025).Source: §2 · p. 3

3 Method

RLSplat generates a budget-matched Gaussian representation by allocating Gaussians across local scene regions rather than pruning a dense prediction. Its policy and local refinement decoder are designed so inference cost scales linearly with the requested budget.

3.1 Overview of RLSplat

  • RLSplat predicts camera parameters and anchor tokens from uncalibrated images, then distributes the Gaussian budget across anchors representing local 3D regions.The allocations sum to the requested budget, and each anchor spawns its assigned Gaussian tokens for decoding.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 Gaussian Allocation Policy

  • The allocation policy models discrete anchor counts as a reinforcement-learning problem, sampling allocations from proportions that represent each anchor’s expected budget share.At inference, expected counts are deterministically converted to integer allocations summing exactly to B.Source: §3.2 · pp. 4–5
  • GSPO trains the allocation policy using rendered reconstruction loss as a reward and group-relative advantages to update its discrete allocation decisions.The method evaluates rollout allocations at novel target views before computing the reward and policy update.Source: §3.2 · pp. 4–5

3.3 Resource-Linear Gaussian Decoding

  • The decoder spawns exactly B Gaussian tokens and refines them with local cross-attention, using hierarchical search over nearby anchors and their Gaussian tokens.Neighbor sets are computed once from detached initial positions and reused across refinement layers.Source: §3.3 · p. 5
  • Voronoi and offset regularizers constrain Gaussians to remain near their parent anchor’s local region, supporting the validity of hierarchical neighbor search.The Voronoi loss favors the parent anchor centroid over neighboring centroids, while the offset loss limits residual center correction relative to local anchor spacing.Source: §3.3 · pp. 5–6

3.4 Training

  • Training first fits the backbone and decoder with random allocations, then adds the policy and fine-tunes the full model with policy optimization.During policy training, budgets are sampled with a skew toward the lower end of the supported range.Source: §3.4 · p. 6

4 Experiments

Experiments compare RLSplat with budget-controllable and budget-fixed methods in zero-shot ACID evaluation, then examine allocation and decoder ablations. RLSplat retains stronger reconstruction at matched budgets, while the ablations show benefits from learned allocation and hierarchical neighbor search.

4 Experiments

  • The experiments train on RE10K and evaluate zero-shot on 249 ACID test scenes, comparing matched budgets and including budget-fixed baselines.The reported inference-cost measurements cover Gaussian generation, excluding data loading, pose alignment, and rendering.Source: §4 · pp. 6–7, §4.2 · p. 8

4.1 Novel-view Synthesis Results

  • RLSplat achieves the best LPIPS, SSIM, and PSNR at every tested budget and view count, with its PSNR advantage over F4Splat growing as the budget decreases.With 8 context views, the PSNR advantage is 0.74 dB at 4K, 0.99 dB at 2K, and 1.22 dB at 1K Gaussians per view.Source: §4.1 · p. 7, Table 1 · p. 7
  • With 24 context views, RLSplat reaches 25.58 dB PSNR using 24K Gaussians, versus SPFSplat’s 25.00 dB using 1536K Gaussians—a 64× reduction.RLSplat also outperforms budget-fixed methods on LPIPS, SSIM, and PSNR across all view counts.Source: §4.1 · p. 7, Table 2 · p. 7
  • At the same 8K-Gaussian budget, RLSplat assigns 5,388 Gaussians to stair balusters and achieves 23.51 dB local PSNR, versus 20.00 dB for F4Splat and 17.79 dB for EcoSplat.Its full-view PSNR is 26.51 dB, compared with 23.42 dB for F4Splat and 21.82 dB for EcoSplat.Source: §4.1 · pp. 7–8, Figure 4 · p. 8

4.2 Ablation Studies and Analysis

  • Removing the offset loss lowers PSNR by 0.53 dB, removing both locality losses lowers it by 2.51 dB, and removing local refinement lowers it by 0.96 dB.Without projection loss, training fails because anchors are no longer tied to the image regions they encode.Source: §4.2 · p. 8, Table 3 · p. 8
  • The learned allocation policy performs best on all three reported metrics, shifting Gaussians away from low-texture areas and toward complex structures where it improves detail.At the same budget, it assigns 79 rather than 165 Gaussians to a low-texture region at similar PSNR, and 116 rather than 84 to a high-texture region.Source: §4.2 · p. 8, Table 4 · p. 8, Figure 5 · p. 9
  • At 65,536 Gaussians, hierarchical neighbor search is 27× faster than exact search (8.1 vs. 219.6 ms), reducing end-to-end latency from 595.0 to 388.0 ms.With Voronoi loss, it retrieves the exact Kgs nearest neighbors for 99.0% of Gaussians, versus 60.2% without that loss.Source: §4.2 · pp. 8–9, Table 5 · p. 9

5 Conclusion

RLSplat distributes exactly the requested Gaussian budget across anchors and refines the Gaussians with a resource-linear decoder. On ACID, it achieves the best rendering quality among budget-controllable methods at every tested budget while latency and peak memory decrease linearly with the budget.

5 Conclusion

  • RLSplat achieves the best rendering quality among budget-controllable methods at every tested budget on ACID.Its latency and peak memory also decrease linearly with the Gaussian budget, while matching or exceeding baseline quality at a fraction of their memory and latency.Source: §5 · p. 9

A Limitations and Future Work

Image encoding remains a fixed cost independent of the Gaussian budget, limiting achievable speed-up at small budgets. The allocation policy uses deterministic rounding to distribute exactly the requested number of Gaussians across anchors.

A Limitations and Future Work

  • Image encoding remains a fixed cost independent of B, bounding achievable speed-up at small budgets.Evaluation uses 256 × 256 resolution and a single cross-dataset setting; downstream-task evaluation is identified as a next step.Source: A Limitations and Future Work · p. 13

B Allocation Policy Details

  • The allocation policy predicts a logit for each anchor and converts the logits to allocation probabilities with softmax.Its output layers are zero-initialized, so probabilities begin uniformly; budget conditioning uses the log-ratio of the requested budget to the number of anchors, floored at 1/64.Source: B Allocation Policy Details · p. 13
  • At inference, deterministic rounding assigns integer Gaussian counts to anchors while preserving a total of exactly B.Each count is either floor(Bp_n) or ceiling(Bp_n), and anchors with zero allocation probability receive no Gaussians.Source: B Allocation Policy Details · p. 13

C GSPO Training Details

GSPO training uses five rollouts and temperature 1.0, settings that perform best among the tested hyperparameter variants. Decoder details specify token construction, packing, neighbor search, and residual prediction.

C GSPO Training Details

  • All tested GSPO hyperparameter settings are within 0.11 dB PSNR of one another, with C = 5 and τ = 1.0 performing best.The authors use C = 5 and τ = 1.0 throughout; τ = 1.0 samples from the policy itself and outperforms both tempered variants.Source: C GSPO Training Details · p. 13, Table 6 · p. 14

D Decoder Details

  • Each Gaussian token’s relative position divides its anchor’s unit interval into bins, with integrated positional encoding suppressing frequencies that could alias across different counts.All B tokens are packed into one sequence, avoiding padding computation.Source: D Decoder Details · p. 14
  • The decoder finds each Gaussian’s nearest tokens among tokens belonging to its 16 nearest anchors, reusing the anchor search across both decoder layers.Anchor centroids and token distances are detached for the respective nearest-neighbor searches.Source: D Decoder Details · p. 14
  • The residual head predicts center offsets, opacity, scales, quaternion, and spherical-harmonic color coefficients, with its center output scaled by α_μ.The residual head is zero-initialized and shares its architecture with the initial-Gaussian head.Source: D Decoder Details · p. 14

E Training Details

Training separates base encoder-decoder training with random allocations from subsequent policy fine-tuning. The training setup fixes 24 total context views per step and biases policy-training budgets toward low values.

E Training Details

  • Each training step processes 24 context views, with the number of views drawn from divisors of 24 and batch size set to 24/V.Context views are loaded at 256 × 256 and upsampled to 504 × 504 before the backbone; targets, losses, and evaluation remain at 256 × 256.Source: E Training Details · p. 14
  • Base training runs for 40K steps with uniformly random allocations, exposing the decoder to varied per-anchor counts before policy learning.Budgets are sampled from a Beta distribution over [2048, 16384V].Source: E Training Details · p. 14
  • Policy training fine-tunes all parameters for 5K steps using five rollouts at τ = 1.0, with the reconstruction render produced from deterministic allocation.Budgets are drawn from Beta(1, 4) over the same range, biasing training toward low budgets; the total step budget is capped to bound peak memory independently of the per-sample maximum.Source: E Training Details · p. 14

F Ablation Protocol

The ablations vary individual pretraining components and allocation settings, using separate training schedules and a shared evaluation setup.

F Ablation Protocol

  • Pretraining ablations remove local refinement, projection, offset, or offset-plus-Voronoi losses, with variants trained from scratch for 5K steps.Step-dependent schedules are scaled to one eighth of the full schedule.Source: F Ablation Protocol · p. 15
  • Allocation ablations vary group size, temperature, and allocation rule after 1K fine-tuning steps from a shared pretrained checkpoint.Rules include a uniform split and a learning-free Sobel-energy heuristic.Source: F Ablation Protocol · p. 15
  • All ablations use ACID with 8 context views and 249 scenes; allocation ablations use B/V = 1024.Allocation variants train for their own cosine-schedule horizon rather than being truncated from a longer run.Source: F Ablation Protocol · p. 15

G Allocation Policy Visualizations

The visualizations show allocation maps for twelve RE10K two-view test scenes, comparing raw anchor-level patterns with smoothed maps.

G Allocation Policy Visualizations

  • The figures cover twelve RE10K two-view test scenes, with an average of 1024 Gaussians per view.Each anchor’s expected allocation is shown relative to a uniform split across anchors in both views.Source: G Allocation Policy Visualizations · p. 15
  • The maps show expected allocation before the total budget is rounded to integer counts; 1× represents about 0.8 Gaussians per anchor.Corner labels give the fraction of the total budget assigned to each view.Source: G Allocation Policy Visualizations · p. 15
  • Raw maps are paired with maps smoothed by a Gaussian filter over log2 allocation, which removes anchor-level noise while leaving the spatial pattern unchanged.The filter uses σ=1 anchor.Source: G Allocation Policy Visualizations · p. 15

H Additional Qualitative Comparison

Across reduced Gaussian budgets, RLSplat retains fine structures that F4Splat and EcoSplat progressively blur or lose. The allocation visualizations show how budget shifts with scene structure, lighting, and view similarity.

H Additional Qualitative Comparison

  • In indoor scenes, allocation shifts toward views with more structure and away from near-blank walls and doorways.Maps encode raw and smoothed allocation on a 0.25× to 4× log scale, with red for more and blue for fewer.Source: Figure 8 · p. 16
  • Allocation moves away from dark or under-exposed regions, concentrating on lit scene regions or objects.Examples include an arch frame and a darkened hillside.Source: Figure 9 · p. 17
  • When two views are nearly identical, the budget stays close to evenly split between them, while allocation within each view favors buildings, vegetation, and shorelines over water and sky.Source: Figure 10 · p. 18
  • At budgets from 128K to 32K, RLSplat retains thin structures while F4Splat and EcoSplat progressively blur or lose them.Examples include stair balusters, railings, window frames, and foliage; the text reports less quality degradation for RLSplat as budget decreases.Source: H Additional Qualitative Comparison · p. 15, Figure 11 · p. 19, Figure 12 · p. 20

Related papers

Loading 2610.27712v1…