PaperlayerView exact revision on arXiv

Source-linked AI summary

FactorSplat: Appearance-Controllable Gaussian Proxies for Medical Volume Rendering

Zhongpai Gao, Benjamin Planche, Meng Zheng, Anwesa Choudhuri, Terrence Chen, Ziyan Wu

arXiv:2610.02382v1cs.CV

TL;DR

Image-trained Gaussian proxies typically bake in one transfer function, limiting their use when region-specific edits must be applied at inference. FactorSplat combines a local lookup and learned appearance residual in a shared Gaussian proxy, improving reconstruction and edit fidelity over region-aware VEG across seven CT and MR scans.

  • Problem

    Image-trained Gaussian proxies bake in one appearance, while inference-time transfer-function control must handle new curves, local responses, and structure coverage without changing geometry.

    Source: §1 · p. 2

  • Method

    FactorSplat trains a per-scene Gaussian proxy with shared geometry, a local transfer-function lookup, and a learned low-rank residual for appearance response.

    Source: §1 · p. 2

  • Results

    Across seven CT and MR scans, FactorSplat improves mean reconstruction and edit fidelity over region-aware VEG.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    One proxy decouples appearance updates from camera motion, supporting interactive color and opacity control instead of separate models for each preset.

    Source: §6 · p. 10

    FactorSplat is evaluated on rendered CT and MR images, not as a clinically validated diagnostic system; clinical use would require task-specific validation and expert oversight.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Transfer functions (TFs) control color and visibility in medical volume rendering, but image-trained Gaussian proxies typically bake one transfer function into their appearance. We present FactorSplat, a per-scene N-dimensional Gaussian splatting (N-DGS) proxy that accepts region-specific intensity-to-RGBA curves at inference. A local lookup applies the authored color and opacity change, while a shared functional encoder and low-rank per-Gaussian factors learn the residual appearance response. Geometry and directional appearance remain shared across presets, with visibility control and TF-aware pruning preserving the ability to hide and reveal structures. On seven CT and MR scans, FactorSplat improves mean PSNR and changed-region error over region-aware VEG across validation, interpolation, unseen composition, and out-of-distribution (OOD) edits. Across these four splits, seven-scan mean PSNR gains over VEG range from 1.10 to 1.52 dB. One checkpoint per scan supports unseen edits without retraining. At $1600^2$, the cached fast renderer averages 524 FPS with 1.17 ms TF switches. Project page: https://gaozhongpai.github.io/FactorSplat/.

1 INTRODUCTION

Image-trained Gaussian proxies bake in their training transfer function, while inference-time control must handle function-valued, spatially heterogeneous edits without changing geometry. FactorSplat uses a shared Gaussian proxy with region-specific transfer-function conditioning and reports improved results over region-aware VEG on unseen transfer functions.

1 INTRODUCTION

  • Single-appearance proxies bake color and opacity into their Gaussians, so another preset requires rebuilding or replacing the proxy and may reveal omitted anatomy.The problem is especially acute when training uses one opaque preset.Source: §1 · p. 2
  • Inference-time transfer-function control must support new sampled curves, spatially heterogeneous responses, and coverage of structures revealed across presets while keeping geometry invariant.The transfer function changes optical appearance rather than anatomy.Source: §1 · p. 2
  • FactorSplat trains a per-scene proxy across transfer functions, using shared geometry and local region–intensity samples to combine direct lookup with a learned low-rank residual for DC color and opacity.Directional SH coefficients remain transfer-function-independent, and the base transfer function preserves exact identity.Source: §1 · p. 2
  • FactorSplat conditions an N-DGS proxy on region-specific intensity-to-RGBA curves to render unseen transfer functions with shared geometry and no refitting.Its appearance model combines an interpretable local lookup, low-rank residual, visibility control, and transfer-function-aware coverage preservation.Source: §1 · p. 2
  • Across seven scans, one checkpoint per scan improves mean PSNR and edit fidelity over region-aware VEG on unseen transfer functions.The contribution statement reports evaluation on unseen transfer functions.Source: §1 · p. 2

2 RELATED WORK

Prior work includes systems that help author transfer functions, image-space surrogates, and Gaussian proxies with different controls or rendering assumptions. FactorSplat is distinguished by grounding function-valued transfer-function inputs in each Gaussian’s local support.

2 RELATED WORK

  • Transfer-function research includes multidimensional mappings for classifying overlapping structures, optimization of mappings against image objectives, and semantic mapping from structures or text prompts.FactorSplat instead reproduces the rendering of an authored transfer function with a compact proxy.Source: §2 · p. 2
  • Image-space surrogates condition generators on viewpoint and rendering parameters, while ViSNeRF factorizes a radiance field for control interpolation but retains ray-based rendering.FactorSplat’s functional encoder modulates a per-scene Gaussian field.Source: §2 · p. 2
  • Prior Gaussian proxies target fixed appearances; iVR-GS composes editable intensity-range models, with cost and size growing with their number, while ECoNGS reduces redundancy across basic scenes.These systems differ in how they support appearance control and share representation.Source: §2 · p. 2
  • VEG stores scalar values and applies unseen one-dimensional color and opacity mappings, whereas FactorSplat uses region-specific intensity transfer functions and cinematic-rendering supervision.FactorSplat adds a direct lookup to an image-trained N-DGS proxy whose residual learns the transfer-function-dependent appearance response.Source: §2 · p. 3
  • Other conditioned Gaussian fields use low-dimensional controls, while FactorSplat grounds function-valued transfer-function inputs in each Gaussian’s local support and maps local codes to appearance offsets.Its per-primitive factors map local codes rather than generating full weights with a hypernetwork.Source: §2 · p. 3

3 METHOD

FactorSplat conditions one per-scene Gaussian proxy on region-specific transfer functions while keeping geometry and directional appearance shared. A physical lookup and learned residual model TF-dependent appearance, with visibility control and training choices designed to preserve structures revealed by different presets.

3.1 PROBLEM FORMULATION AND N-DGS BACKBONE

  • One proxy renders unseen transfer functions without optimization or per-preset parameters, using an opacity-only N-DGS backbone with shared geometry.Centers and covariances remain unchanged across transfer functions, and the backbone has no view-dependent position shifts.Source: §3.1 · pp. 3–4

3.2 LOCAL TRANSFER-FUNCTION CONDITIONING

  • The local lookup averages authored RGBA changes over each primitive’s material–intensity support, while a shared encoder embeds local changes before pooling to retain distinct local responses.The region index routes samples to curves; the encoder consumes curve RGBA values rather than region identifiers.Source: §3.2 · p. 4

3.3 FACTORIZED APPEARANCE RESPONSE

  • The hybrid appearance model combines lookup and learned residual branches to change DC color and opacity while sharing geometry and directional appearance across transfer functions.Both offsets vanish at the base preset; conditioned appearance is cached at transfer-function switches, while camera motion reuses the unchanged rasterizer.Source: §3.2 · p. 4, §3.3 · p. 5
  • A parameter-free visibility gate handles binary region removal, while the conditional branches handle graded appearance.Removal relative to the base preset yields zero opacity when the gate is applied; mixed support is removed when removed-region mass reaches one half.Source: §3.3 · p. 5

3.4 INITIALIZATION AND TRAINING

  • Preset-neutral initialization covers the union of segmented structures, and coverage-aware pruning retains primitives visible under any training preset.Scale-aware descriptors preserve initial material support as geometry is optimized; clone and split children inherit descriptors and factor rows.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

FactorSplat is evaluated across seven CT and MR scans, with comparisons spanning reconstruction, edit fidelity, ablations, and computational cost. It generally outperforms region-aware VEG on the reported quality measures, while ablations identify benefits from its appearance branches, functional input, and visibility gate.

4.1 PROTOCOL

  • The seven scans cover 41 conditions each, including held-out validation, interpolation, unseen compositions, and OOD edits, with all models fitted per scan.The protocol uses five CT and two MR scans, 24 training presets, and 40 held-out cameras shared across presets.Source: §4.1 · p. 6

4.2 RECONSTRUCTION QUALITY AND EDIT FIDELITY

  • 1.45/1.52/1.10/1.41 dB mean PSNR gains over floating-point VEG on validation, interpolation, composition, and OOD; changed-region error has lower means on every split for both CT and MR.The PSNR gains correspond respectively to validation, interpolation, composition, and OOD; the result also reports higher PSNR in 24 of 28 scene–split comparisons.Source: §4.2 · p. 7
  • FactorSplat exceeds specialists on validation and interpolation for all seven scans, but specialists retain a 5.2 dB mean advantage on OOD.On composition, FactorSplat leads on four scans, ties on one, and trails on two; the text cautions that broader camera coverage may contribute to its observed advantage.Source: §4.2 · p. 7

4.3 ABLATION STUDIES

  • Removing the learned residual reduces PSNR in all eight columns; removing the physical lookup causes larger losses, including 2.57 and 2.65 dB on composition.These ablations support combining a direct response to the authored edit with learned corrections to reproduce reference images.Source: §4.3 · p. 7
  • Replacing functional input with nearest-preset embeddings loses 1.02 and 2.84 dB on heart and vascular OOD, where functional input is most useful for curves far from training presets.The embedding variant remains competitive on nearby vascular edits.Source: §4.3 · pp. 7, 9
  • Removing the analytic visibility gate lowers OOD PSNR by 0.32 and 0.52 dB, while OOD delta error rises on both heart and vascular.Validation, interpolation, and composition remain unchanged; the gate benefits material removal.Source: §4.3 · p. 9

4.4 COMPUTATIONAL COST

  • At 1600^2, FactorSplat’s cached fast renderer averages 524 FPS versus VEG’s 144 FPS, with 1.17 ms average TF switches.The fast renderer evaluates conditioning once per TF switch and reuses cached attributes during camera motion.Source: §4.4 · p. 9
  • Mean training time is 9.8 minutes per scan for FactorSplat versus 43.8 for VEG; one FactorSplat checkpoint supports multiple appearances.The estimated cost for separately trained N-DGS specialists across 17 evaluated TFs averages 88.1 minutes per scan.Source: §4.4 · p. 9

5 DISCUSSION

The discussion frames FactorSplat’s local conditioning as a way to compose region-specific edits within one shared Gaussian field, while noting limits to global illumination response and visibility boundaries. Feed-forward proxy generation is proposed as a way to reduce per-scan construction cost.

Implications for TF authoring

  • Local conditioning ties each primitive’s response to the materials it represents, supporting composable authored edits within a shared Gaussian field.The discussion suggests training banks should cover distinct appearance operations and the materials they affect.Source: §5 · p. 9

Local control and global appearance

  • Edits outside a primitive’s support leave its code unchanged, limiting response to global illumination changes; visibility gating also removes whole Gaussians rather than clipping voxel boundaries.Contextual conditioning could address the global-illumination limitation, and persistent descriptors can misalign as centers move.Source: §5 · p. 9

Feed-forward proxy generation

  • FactorSplat learns one proxy per scan; feed-forward prediction of shared geometry and conditional appearance factors could reduce construction cost while retaining inference-time appearance control.The proposed direction builds on Render-FM.Source: §5 · p. 9

6 CONCLUSION

FactorSplat combines authored TF changes with learned appearance corrections in one shared-geometry proxy, improving reconstruction and edit fidelity over region-aware VEG across seven CT and MR scans. Its scope remains research visualization rather than clinically validated diagnosis, and broader unseen edit operations remain future work.

6 CONCLUSION

  • Across seven CT and MR scans, FactorSplat improves mean reconstruction and edit fidelity over region-aware VEG while reducing training time and increasing rendering throughput.Ablations support combining authored optical changes with learned corrections.Source: §6 · p. 10
  • Local TF lookup and a low-rank appearance residual render unseen region-specific transfer functions while Gaussian geometry remains shared.This supports interactive color and opacity control within one proxy instead of separate models for each preset.Source: §6 · p. 10
  • Generalizing FactorSplat to a broader range of unseen edit operations remains an important next step.Source: §6 · p. 10
  • The appendix documents evaluation procedures, VEG adaptation, per-scan results, ablation edit metrics, and timing protocol; repository materials include scripts, TF manifests, metrics, and figure-source records.Source: REPRODUCIBILITY STATEMENT · p. 11
  • FactorSplat is a research visualization method evaluated on rendered CT and MR images, not a clinically validated diagnostic system.Clinical use would require task-specific validation, expert oversight, and access to the original scan.Source: ETHICS STATEMENT · p. 11

A EFFICIENT TRANSFER-FUNCTION EVALUATION

At each TF switch, the shared encoder evaluates region–intensity entries once, then caches per-Gaussian appearance updates across camera changes. The added representation scales linearly with the number of primitives, independently of TF count.

A EFFICIENT TRANSFER-FUNCTION EVALUATION

  • R = 4; lookup gains are learned, while the output scales remain fixed.Source: A EFFICIENT TRANSFER-FUNCTION EVALUATION · p. 14
  • At each TF switch, the shared encoder evaluates M × B region–intensity entries once, then each Gaussian aggregates embeddings over persistent local support to update DC color and opacity.The resulting attributes are cached across camera changes, avoiding repeated TF encoding during view navigation.Source: A EFFICIENT TRANSFER-FUNCTION EVALUATION · p. 14
  • The conditional representation scales linearly with the number of primitives, independently of the number of transfer functions.It adds 4Nr per-Gaussian factors, H(4 + r) shared encoder weights, and four lookup gains.Source: A EFFICIENT TRANSFER-FUNCTION EVALUATION · p. 14

B REGION-AWARE VEG BASELINE

The region-aware VEG adaptation assigns each Gaussian a fixed region and learnable within-region intensity coordinate, then trains on authored presets. Evaluation also describes native quantization and a numerical-stability correction to the released implementation.

B REGION-AWARE VEG BASELINE

  • The VEG extension assigns each Gaussian a fixed categorical region and learnable scalar, initialized from its nearest source seed’s density-weighted majority region.Authored curves are resampled to VEG’s 100 intensity knots; the within-region coordinate is sigmoid-parameterized and interpolated between adjacent bins.Source: B REGION-AWARE VEG BASELINE · p. 14
  • Training uses 24 authored presets for 30,000 iterations, retaining VEG’s optimizer, image losses, regularizers, and weight-based pruning, with no primitive-count cap.Source: B REGION-AWARE VEG BASELINE · p. 14
  • 0.15/0.12/0.10/0.05 dB: native quantization lowers mean PSNR by these amounts on validation/interpolation/composition/OOD, respectively.Both evaluated rows use the same trained checkpoint; the native-quantized row follows VEG’s released deployment path.Source: B REGION-AWARE VEG BASELINE · p. 14
  • The released VEG specular term has a singular derivative at zero for shininess between 0 and 1; masking exact-zero inputs assigns zero gradients there while preserving forward values and other gradients.With this correction, VEG completes training on all seven scans.Source: B REGION-AWARE VEG BASELINE · p. 14

C PER-SCAN RESULTS

Per-scan results show FactorSplat’s PSNR advantage over VEG across all splits on five scans, while VEG has lower edit error throughout heart and kneejoint. Table 3 reports changed-region error and PSNR, including five-edit OOD averages and an unweighted mean.

C PER-SCAN RESULTS

D ABLATION EDIT FIDELITY

The ablation separates reconstruction quality from changed-region edit fidelity. Removing either appearance branch raises edit error, while the functional code improves OOD edit fidelity on both scans.

D ABLATION EDIT FIDELITY

  • Preset embeddings remain competitive nearby, but the functional code improves OOD edit fidelity on both scans.Table 4 complements Table 2’s reconstruction metrics with changed-region ΔTF for the same variants.Source: D ABLATION EDIT FIDELITY · p. 16
  • Unchanged-region leak spans 0.12–0.43×10−2 across the variants, and the visibility gate affects only presets that remove material.Mean rank averages ranks across eight PSNR columns; worst gap is the maximum deficit from a column’s best score.Source: D ABLATION EDIT FIDELITY · p. 16
  • Removing either appearance branch increases changed-region ΔTF error in all eight columns.Table 4 reports changed-region ΔTF, with lower values better.Source: Table 4 · p. 16, D ABLATION EDIT FIDELITY · p. 16

E RUNTIME MEASUREMENTS

Runtime measurements compare deployment throughput and reconstruction quality under matched-view protocols. Additional held-out CT examples cover vascular, heart, intestine, and lower-body anatomy.

E RUNTIME MEASUREMENTS

  • Table 5 reports B200 deployment at 1600², with FactorSplat FPS from cached fast rendering and PSNR from quality rendering.The protocol uses matched views; fast and quality rendering can produce different images.Source: Table 5 · p. 16, E RUNTIME MEASUREMENTS · p. 16
  • TF-update latency is measured separately and incurred once per transfer-function switch, not during fixed-TF view navigation.Measurements use five warm-ups and 20 synchronized wall-time measurements per view.Source: E RUNTIME MEASUREMENTS · p. 16

F ADDITIONAL QUALITATIVE COMPARISONS

  • Held-out CT examples show vascular, heart, intestine, and lower-body scenes, with full-frame PSNR and each row’s best marked.The examples occupy rows 1–3 for vascular, 4–5 for heart, 6 for intestine, and 7 for lower body.Source: Figure 4 · p. 17

Related papers

Loading 2610.02382v1…