PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Should the Gaussians Go? Reusable Label-Aware Allocation for Budget-Constrained Gaussian Splatting of Cinematic CT

Bowen Xiang, Dingjie Su, Jon S. Heiselman, Michael I. Miga

arXiv:2610.20397v1cs.CV

TL;DR

Fixed Gaussian budgets make placement an unresolved part of cinematic CT splatting, where per-patient optimisation currently determines where Gaussians go. The paper learns a reusable label-aware allocator from optimised scenes, then reports that its neural prior matches per-patient allocation and its closed form comes close on vessels. The results also show that transfer across rendering presets depends on how the methods' inputs relate to rendered appearance.

  • Problem

    Per-patient optimisation decides Gaussian placement, while display hardware imposes a fixed count; existing voxel-based feed-forward prediction does not control that count.

    Source: §1 · p. 2

  • Method

    A reusable front end predicts a density over cells from CT, labels, and transfer-function inputs, then draws exactly B positions without per-patient placement optimisation.

    Source: §1 · p. 2

  • Results

    At equal budgets, the neural prior matches per-patient optimisation on held-out and external patients, while VACF comes close on vessels and both outperform uniform placement and dense voxel-aligned pruning.

    Source: §1 · p. 3, §6 · p. 15, §7 · p. 16

  • Takeaways & Limitations

    The front end can remove placement search from the per-patient pipeline and provides exact count control, while remaining per-patient attribute fitting is still required.

    Source: §6 · pp. 15–16, §7 · p. 16

    The study covers one organ and one task, and complete scenes still require per-patient path-traced fitting of colour, opacity, and shape.

    Source: §6 · p. 16

Abstract

from arXiv · show

Anatomical labels help 3D Gaussian splatting preserve vessels in cinematic renderings of CT, but the representation is still found by optimising each patient's scene, where densification decides where Gaussians go, while real-time displays cap their number. We address one part of this process: deciding where a fixed budget of B Gaussians should be placed. We propose a reusable, labelaware allocation front end that maps a patient's CT, labels and transfer function to a density over 2mm cells and draws exactly B positions from it without per-patient optimisation; the other Gaussian attributes are still fitted per patient. The front end is learned once from per-patient optimised scenes and implemented as a population-trained neural prior and an interpretable seven-parameter visibility-aware closed form (VACF). With identical attribute fitting on 20 held-out patients at 10K-100K Gaussians, both raise vessel-masked PSNR over uniform allocation by 4.5-7.1dB and over pruning a dense voxel-aligned model by 1.6-3.3dB. The neural prior does not differ significantly from per-patient optimised allocation at any budget, and VACF is within 0.6dB of it on vessels; on 19 external patients the neural prior again matches it on vessels and VACF is within 0.7dB. As an initialisation for position optimisation, VACF allocation beats per-patient densification on vessels at 10K Gaussians and, at 25K, reaches a higher vessel PSNR after 118s of optimisation than densification after 248s. Under altered rendering presets, VACF transfers to new colours and lighting but not to a label-free transfer function, whereas the neural prior shows the opposite pattern.

1. Introduction

The paper separates fixed-budget Gaussian placement from per-patient scene optimisation, proposing a reusable label-aware allocator learned from optimised scenes. Its evaluations report improved vessel quality over uniform placement and pruning, while comparisons and transfer remain bounded by the methods and rendering presets tested.

1. Introduction

  • The paper targets the mismatch between per-patient densification and pruning, which choose Gaussian placement through repeated optimisation, and display hardware that caps Gaussian counts.Feed-forward voxel-based prediction avoids per-patient optimisation but does not control the number of Gaussians.Source: §1 · p. 2
  • The proposed front end maps a patient's CT, labels, and transfer function to a density over 2 mm cells, then draws exactly B positions without per-patient placement optimisation.Colour, opacity, and shape remain fitted to that patient's path-traced views.Source: §1 · p. 2
  • The front end is trained on optimised-scene allocations and has two implementations: a population-trained neural prior and an interpretable visibility-aware closed form.VACF uses six tissue weights and a visibility exponent; both methods share supervision and an evaluation protocol that holds attribute fitting fixed.Source: §1 · pp. 2–3
  • Both allocators improve anatomical rendering quality over uniform placement and dense voxel-aligned pruning at equal budgets; the neural prior matches per-patient optimisation on held-out and external patients.VACF also comes close on held-out patients and is within a fraction of a decibel on vessels; as an initialisation, it improves vessel results over densification.Source: §1 · p. 3, Figure 1 · p. 3
  • The paper identifies visibility as the main systematic difference between optimised allocations and anatomy-only allocation rules.Its comparisons do not evaluate the official LAGS pipeline; the authors describe label-aware training as complementary to their allocation.Source: §1 · p. 3

2. Related work

Prior work addresses label-aware optimisation, feed-forward Gaussian prediction, compactness, and visibility, but leaves fixed-count placement under the paper's conditions unresolved. The proposed allocation is presented as complementary to these approaches.

Gaussian splatting for medical volumes

  • Medical-volume methods including LAGS improve Gaussian distribution through per-scene optimisation, while Render-FM predicts one Gaussian per foreground voxel without controlling the count.The paper does not evaluate the official LAGS pipeline; its allocation is described as complementary to label-aware training.Source: §2 · p. 3

Compact Gaussians and budget control

  • Compact-Gaussian methods use pruning, budgeted growth, relocation, densification, or codebook compression, but operate per scene by optimisation.The paper uses significance pruning of a dense voxel-aligned model as a baseline.Source: §2 · p. 4

Feed-forward Gaussian prediction

  • Feed-forward Gaussian predictors typically tie Gaussians to pixels or voxels; this paper's front end instead decides positions under a budget and could precede such a decoder.In this work, attributes are still fitted per patient.Source: §2 · p. 4

Optimal point density and visibility

  • Quantisation theory relates optimal point density to source density, while visibility-driven rendering work uses visibility to classify or emphasise structures rather than represent a volume with a fixed primitive count.Source: §2 · p. 4

3. Method

The method learns a cell-wise allocation from per-patient optimised scenes, then samples an exact Gaussian budget for a new patient. Its allocation model motivates budget reuse and visibility weighting, while explicitly simplifying inter-cell occlusion.

3.1. Setting

  • The allocator maps CT, labels, and transfer-function inputs over 2 mm liver-region cells to a density, from which exactly B positions are drawn.Allocator parameters are learned once; cell features and allocation are computed per patient without optimisation, while remaining attributes are optimised per patient.Source: §3.1 · p. 4

3.2. Oracle allocations from per-patient optimisation

  • Training-patient oracle densities are normalised per-cell counts from teacher 3DGS scenes optimised on 256 path-traced views; test-patient teachers also provide the per-patient reference.A separate budget-capped teacher supplies the per-case reference at exactly B Gaussians.Source: §3.2 · p. 4

3.3. An allocation model

  • The allocation model treats cell distortion as additive with local power-law complexity, implying a normalised optimum independent of B and uniform allocation only when cell complexities match.It models cell complexity as an intrinsic term multiplied by a power of mean transmittance; it ignores neighbouring-Gaussian occlusion and assumes a common exponent, so it serves as motivation for empirical tests.Source: §3.3 · p. 4

3. Method

  • The method uses per-patient oracle allocations to train two allocators, then applies them to draw positions for a new patient before fitting remaining attributes.The allocators are a neural prior and VACF, and the method diagram shows the training-to-deployment sequence.Source: Figure 2 · p. 5

3.4. Neural allocation prior

The neural allocation prior is a 3D U-Net trained to predict oracle cell densities from ten cell-level inputs. Its inputs include anatomy, transfer-function properties, visibility, and depth.

3.4. Neural allocation prior

  • The neural prior uses a 3D U-Net whose softmax output over labelled cells is the allocation density, trained on 56 patients using the teachers' Gaussian positions.Inputs include CT intensity, tissue and vessel fractions, transfer-function extinction and gradient, visibility, and depth below the liver surface.Source: §3.4 · p. 5
  • Input ablations retrain the network after removing groups of channels; the variant without transfer-function inputs retains visibility computed from extinction.Source: §3.4 · p. 5

3.5. Visibility-aware closed form (VACF)

VACF allocates Gaussian density using label fractions, TF-gradient magnitude and visibility, with seven fitted parameters. Its restricted form makes the allocation interpretable and its transfer limits explicit.

3.5. Visibility-aware closed form (VACF)

  • VACF uses six weights for label fractions and TF-gradient magnitude, plus an exponent on cell visibility, to form its allocation density.The weights are fitted on training patients; the exponent is constrained to be nonnegative.Source: §3.5 · p. 5
  • The six weights lie on the simplex, and VACF fits them with a nonnegative visibility exponent using Nelder–Mead on training patients.The exponent is fixed from a power-law fit.Source: §3.5 · p. 5
  • Because visibility is recomputed from TF extinction, VACF adapts to a new transfer function without refitting, but its tissue terms assume rendered content follows labels.Source: §3.5 · p. 5

3.6. How the two implementations relate

  • VACF is a pointwise monotone function of a weighted sum modulated by visibility, while the neural prior can represent interactions and spatial context.The neural prior has 2.7 million parameters and may learn information encoded by its training transfer function.Source: §3.6 · p. 6
  • VACF’s six weights and visibility exponent can be inspected across cell sizes, and its form makes conditions for failure identifiable.Source: §3.6 · p. 6

3.7. Deployment and evaluation protocol

  • Systematic resampling draws integer cell counts summing to B, then jitters positions within cells; evaluation freezes positions so image-quality differences reflect placement.In deployment, positions may also be optimised.Source: §3.7 · p. 6

3.8. Baselines and references

  • Baselines include uniform foreground voxels, TF-extinction density, a hand-designed anatomy heuristic, segmentation-free vesselness and gradient, and pruned dense models.The per-case 3DGS and oracle density require per-patient optimisation and serve as references, not competitors.Source: §3.8 · p. 6

4. Experimental setup

The experiments render path-traced CT views under specified camera, lighting and sampling conditions, and evaluate internal and external patient cohorts. They compare frozen allocators using region-focused image metrics and paired patient-level statistics.

4.1. Cinematic ground truth

  • Ground-truth images use Mitsuba 3 volumetric path tracing with a heterogeneous medium, environment map and directional key light, across cameras on a framing sphere.The rendering uses a Henyey–Greenstein phase function with g = 0.2, 512 × 512 pixels and a 35° field of view.Source: §4.1 · p. 6
  • Training views use 1024 samples per pixel (spp), while 16 held-out test views use 2048 spp.Source: §4.1 · p. 6

4.2. Data

  • The internal cohort is the MSD hepatic-vessel task: 56 training, 10 validation and 20 test patients selected from scans with slice thickness ≤2.5 mm.The source task contains 303 scans; 86 meet the slice-thickness criterion.Source: §4.2 · p. 6
  • External testing uses 19 IRCADb scans from another centre, with labels mapped to the MSD vessel and tumour categories and allocators frozen before evaluation.The annotation protocol differs from MSD, and IRCADb was used once for final evaluation.Source: §4.2 · p. 6

4.3. Rendering presets

  • Preset A is label-aware; preset B changes tissue extinction, colours, intensity window and lighting, while preset C assigns appearance from CT intensity without labels.Presets B and C were evaluated on the first 10 test patients and were not used for decisions.Source: §4.3 · p. 6

4.4. Model selection and use of the test data

  • Allocators and design choices were frozen before the second MSD test evaluation; external patients and presets B and C were unseen during decisions.After the first evaluation, neural-prior inference was corrected to normalise over labelled cells, matching its training loss; the corrected prior is reported.Source: §4.4 · pp. 6–7

4.5. Metrics and statistics

  • Evaluation reports PSNR and SSIM for projected regions, including vessels and thin vessels, alongside LPIPS and centreline ridge recall.Paired comparisons are across patients using two-sided Wilcoxon signed-rank tests with Holm correction and bootstrap confidence intervals; nonsignificance does not establish equivalence.Source: §4.5 · p. 7

4.6. Implementation

  • On an RTX 4090, median VACF and neural-prior allocation times are 0.6 s and 4.9 s, respectively, while frozen-position attribute fitting takes 2.0 min.Feature computation takes about 7 s per patient; all timings are additional to path tracing training views.Source: §4.6 · p. 7

5. Results

At equal budgets, label-aware allocation improves rendering quality over uniform placement and dense-model pruning on held-out and external patients. Results also show how allocation behaves when positions are optimised, rendering presets change, and cell sizes or budgets vary.

5.1. Anatomical rendering quality at equal budgets

  • The neural prior is not significantly different from the oracle density in any of 12 comparisons; VACF is within −0.60 to −0.05 dB of it on vessels at 10K–50K.Per-case 3DGS exceeds the oracle density by 0.21–0.39 dB on vessels because it can optimise positions within cells; at 10K, thin-vessel CR80 is 0.95 for VACF and 0.96 for both the neural prior and oracle.Source: §5.1 · pp. 7, 9, Table 3 · p. 9
  • At 10K, VACF raises vessel-masked PSNR by +6.46 [6.01, 6.90] dB over uniform voxels and +2.67 [1.94, 3.39] dB over significance-pruned dense models.Both VACF and the neural prior stay well above uniform allocation and dense-model pruning across budgets; their advantage over the anatomy heuristic is largest on thin vessels and at small budgets.Source: §5.1 · p. 7, Figure 3 · p. 8, Table 1 · p. 8

5.2. External patients

  • On 19 external IRCADb patients, the neural prior is within −0.18 to +0.07 dB of the oracle on vessels, while VACF is 0.46–0.66 dB below it.Both methods improve over the anatomy heuristic, with larger margins than on the held-out set.Source: §5.2 · p. 9, Table 4 · p. 10

5.5. What drives allocation

  • The oracle favours cells near the liver surface and thin vessels; depth below the surface is the dominant predictor of its allocation difference from the anatomy heuristic.Visibility explains the preference for visible cells; VACF halves the heuristic’s depth bias, while the neural prior removes it.Source: §5.5 · p. 11, Figure 7 · p. 14, Figure 8 · p. 15
  • At 10K, removing visibility lowers VACF foreground PSNR by 0.76 [0.64, 0.91] dB, while using 4 mm rather than 2 mm cells lowers thin-vessel PSNR by 2.5 dB.Refining to 1 mm improves VACF at 10K by +0.33 [0.21, 0.44] dB on vessels and +0.88 [0.76, 1.00] dB on thin vessels; the fitted closed-form parameters remain stable across cell sizes.Source: §5.5 · pp. 11, 14, 15, Table 7 · p. 14

5.3. Allocation as initialisation for position optimisation

  • At 10K, VACF with optimised positions exceeds per-case 3DGS by +1.12 [0.97, 1.26] dB on vessels; at 25K, it reaches 41.9 dB in 118 s versus 41.3 dB in 248 s.Position optimisation improves every allocation, and VACF with optimised positions remains above uniform allocation on vessels at 10K and 50K.Source: §5.3 · pp. 9, 11, Figure 5 · p. 12, Table 5 · p. 12

5.4. Rendering presets: strengths and limits of each implementation

  • Under unseen colours and lighting, VACF remains at oracle level on vessels, but under a label-free transfer function it falls below the oracle while the neural prior with TF inputs reaches oracle level on vessels and thin vessels.The preset-C gap reflects changed appearance-label correspondence: the oracle allocates 31% to the liver-surface shell, while VACF allocates 9%.Source: §5.4 · p. 11, Figure 6 · p. 13, Table 6 · p. 14

5.6. Supporting analysis: one density across budgets

  • Optimised allocation densities are approximately budget-invariant over 25K–200K at 4–8 mm resolution, but invariance was not tested at 2 mm or below 25K.At 8 mm, divergence between budget densities is generally comparable to same-budget seed variability; the single-density results at smaller budgets are empirical support.Source: §5.6 · p. 15, Figure 9 · p. 15

6. Discussion

The allocation front end performs close to per-patient optimisation on vessels, with different strengths across implementations and rendering presets. VACF can also improve position optimisation, but the study and methods retain important scope limits.

6. Discussion

  • Both allocators raise anatomical rendering quality over uniform placement and pruning, while the neural prior matches per-patient optimisation on internal and external patients.VACF comes within a fraction of a decibel of the oracle on vessels.Source: §6 · pp. 15–16
  • As an initialisation, VACF position optimisation beats densification on vessels at 10K Gaussians and reaches higher vessel PSNR after 118 s than densification after 248 s at 25K.The neural prior was not evaluated for position optimisation.Source: §6 · pp. 15–16
  • VACF transfers to new colours, extinctions and lighting, but fails when appearance no longer follows labels; the neural prior’s transfer depends on its TF inputs.Under the label-free preset, the neural prior matches the oracle on vessels; under a TF that changes extinction contrast, it under-serves thin vessels.Source: §6 · pp. 15–16
  • Complete scenes still require per-patient path-traced training views and fitting of colour, opacity and shape, so the front end does not produce scenes in one forward pass.The study covers one organ and task and measures image quality, not clinical utility or on-device frame rates.Source: §6 · p. 16
  • The oracle, renderer and camera distribution constrain the evaluation, while presets B and C were tested on 10 patients and the neural prior was trained only on 2 mm cells.The authors also did not measure how Gaussian budget relates to frame rate on a device.Source: §6 · p. 16

7. Conclusion

The paper separates fixed-budget Gaussian placement from per-patient optimisation and shows that a reusable label-aware front end can allocate positions for new patients. Attribute fitting remains per patient, while rendering presets delimit when each allocator can be trusted.

7. Conclusion

  • A label-aware allocation front end learned from optimised scenes answers where Gaussians should go for new patients without per-patient optimisation.The population-trained neural prior matches per-patient allocations internally and externally; VACF comes within a fraction of a decibel on vessels and runs in under a second.Source: §7 · p. 16
  • Both implementations improve anatomical rendering quality over uniform placement and pruning at equal budgets, and VACF gives position optimisation a better start than densification.Rendering presets delimit when each implementation can be trusted.Source: §7 · p. 16
  • The front end decides positions, but fitting the remaining Gaussian attributes still requires per-patient work.Deciding where Gaussians go no longer has to be repeated for each patient.Source: §7 · p. 16

CRediT authorship contribution statement

Bowen Xiang is credited across the paper’s research and production work; the other listed authors are credited with review, editing and visualization.

CRediT authorship contribution statement

Funding

No specific grant funding is reported.

Funding

  • The research did not receive any specific grant funding.Source: Funding · p. 16

Appendix A. Development notes

The development protocol was shaped by four pitfalls: underpowered experiments obscured allocation differences, teacher settings affected learned densities, and preprocessing or normalization could distort regional allocation.

Appendix A. Development notes

  • At 256 training views and 1024 spp, a real oracle gap appeared after an earlier 64-view, 256-spp experiment compressed differences through noise-limited ground truth and view-limited overfitting.In the earlier experiment, the oracle was within 0.1 dB of the anatomy heuristic; training-view PSNR rose while test-view PSNR fell as models grew.Source: Appendix A · p. 17
  • Default densification thresholds barely densified per-patient teachers on soft translucent images, so their densities mostly reflected their initialisation until absolute-gradient densification at a lower threshold was used.Source: Appendix A · p. 17
  • Smoothing teacher count maps shifted thin-vessel budget share from 2.4% to 0.5%, while raw-count likelihood was unbiased and the model itself provided smoothing.Allocators fitted to smoothed targets, including the first-round closed form evaluated on the test set, under-served thin vessels.Source: Appendix A · p. 17
  • A mismatch between the neural prior's training cells and inference normalization leaked a fifth of its budget outside the organ, motivating regional audits of allocation density.Source: Appendix A · p. 17

Related papers

Loading 2610.20397v1…