PaperlayerView exact revision on arXiv

Source-linked AI summary

CTBakeGS: training-free baking of photorealistic CT volume renderings into 3D Gaussian splats

Haruki Fukuda, Constantin Lux, Wolf Schweitzer, Tadashi Hosoya, Lars Ebert

arXiv:2610.37406v1cs.CVcs.GR

TL;DR

Photorealistic CT volume renderings can convey anatomy, but viewing them requires CT data, dedicated software and a powerful GPU, while existing medical 3DGS approaches require scene training. CTBakeGS directly converts renderings into standard splat models without training. Bone reached about 33 dB in wide views, while close-up vessel views were weakest at 27 dB against converged-renderer references.

  • Problem

    Photorealistic CT viewing requires CT data, dedicated software and a powerful GPU, while existing medical 3DGS methods require per-scene optimisation or prior network training.

    Source: §1 · p. 1

  • Method

    CTBakeGS directly converts CT renderings into standard 3DGS models, baking the renderer’s appearance into splats without training or per-scene optimisation.

    Source: §4 · p. 7

  • Results

    Bone reached about 33 dB in wide views and 30 dB in close-up views, while close-up contrast-enhanced vessels were weakest at 27 dB against converged-renderer references.

    Source: §4 · p. 7

  • Takeaways & Limitations

    Converted models could be shared and viewed interactively without the CT data or a volume renderer, with potential value for communication and teaching.

    Source: §5 · p. 9

    Evaluation used six cases from one public dataset with 1.5-mm voxels; clinical thin-slice scans have up to ten times more voxels and convert more slowly.

    Source: §4 · p. 9

Abstract

from arXiv · show

Photorealistic volume rendering of computed tomography (CT) may convey anatomy intuitively, but it needs the CT data, dedicated software and a powerful graphics processor. Three-dimensional Gaussian splatting (3DGS) displays photorealistic scenes in real time on less powerful devices, but existing medical approaches train each scene for minutes to hours. We aimed to convert a CT volume rendering directly into a 3DGS model without training. Splats were placed on an adaptive octree of millimetre-sized cells. Their colour and opacity were computed on the graphics processor with the renderer's lighting, cell opacity was converted analytically into splat opacity, and faint splats were retained stochastically above the 1/255 opacity floor of standard 3DGS rasterisers. Six public CT scans (1.5-mm voxels) were converted with three transfer functions and compared with converged references by peak signal-to-noise ratio (PSNR), structural similarity and a learned perceptual metric. Two-million-splat models reached a PSNR of 32.6 +/- 0.9 dB for bone, 30.1 +/- 1.0 dB for contrast-enhanced vessels and 32.3 +/- 1.5 dB for skin and bone. A conversion pass (usually two or three per model) took 4-30 s on a current laptop and 9-60 s on a desktop with a 4-GB graphics card. On the laptop, the models rendered at 68 frames per second at 800 x 800 pixels. Training-free conversion produced real-time 3DGS models of photorealistic CT renderings in about a minute or less per conversion pass on ordinary computers, with fidelity that we consider sufficient for interactive viewing.

1 Background

Photorealistic CT renderings can aid anatomical understanding, but viewing them typically requires the CT data, dedicated software and a powerful GPU. CTBakeGS aims to bake a rendering’s appearance directly into 3D Gaussian splats, without per-scene training.

1 Background

  • Existing medical 3DGS methods optimise each scene for minutes to over an hour or require trained networks, while changing transfer functions can require rebuilding models.Photorealistic CT rendering also needs the CT data, dedicated software and a powerful GPU.Source: §1 · p. 1
  • Figure 1 shows the conversion pipeline: adaptive cells are sampled for colour and lighting, mapped to overlapping splats, and faint splats are retained above the rasteriser’s opacity floor.The output is a standard 3DGS file.Source: Figure 1 · p. 2
  • CTBakeGS uses information already defined by the volume—material location, opacity and lit colour—to bake the rendering into Gaussians rather than train them from rendered images.The authors describe removing prerequisites such as rendered training views, a dedicated GPU and per-scene training time.Source: §1 · p. 2

2 Methods

CTBakeGS converts a rendered CT volume directly into standard 3DGS files by assigning cells splat positions, sizes, colour and opacity. The methods describe its conversion pipeline and browser implementation, then specify the scans, references and metrics used for evaluation.

2.1 Conversion

  • Each volume cell becomes a splat whose appearance comes from the lit volume rendering; the pipeline precomputes lighting, samples cells, maps opacity, retains faint splats and writes a standard 3DGS file.Anti-aliasing compensation is applied when splats are drawn.Source: §2.1 · p. 2
  • Lighting is baked into view-independent splat colours using sky visibility and key-light transmittance, while 27 stratified samples per cell determine its colour and opacity.The renderer’s light volume stores sky visibility in 24 directions and transmittance toward a key light.Source: §2.1 · p. 3
  • An adaptive octree splits cells where transfer-function opacity changes, using millimetre-sized cubes down to sub-voxel size; the finest size is adjusted toward a chosen splat count.Conversion is repeated until the count is within 8% of target, usually after two or three passes.Source: §2.1 · p. 3
  • Overlapping Gaussian splats receive analytically mapped opacity to match cell optical depth, while faint splats are retained stochastically above the rasteriser floor.The faint-splat procedure assigns opacity α0 = 0.01 with probability τsplat/τ0; its hash-based choice keeps conversion deterministic.Source: §2.1 · p. 3

2.2 Implementation

  • CTBakeGS is implemented as a single HTML file using JavaScript and WebGL 2 that runs locally in a web browser.Version 1.0 was used for all measurements.Source: §2.2 · p. 4

2.3 Evaluation

  • Evaluation used six TotalSegmentator CT scans with 1.5-mm isotropic voxels and three transfer functions, comparing 3DGS views with converged references rendered from the same volumes and cameras.The references averaged 128 jittered frames; a 256-frame view differed by 54.7–58.7 dB PSNR.Source: §2.3 · p. 4
  • Image quality was assessed with PSNR, SSIM and LPIPS; the study also recorded splat counts, conversion times and frame rates, and tested splat-count and conversion-step variants.Variants included about 1, 2 and 3 million splats and five ablations; metrics were averaged across views, with cases as the unit of analysis.Source: §2.3 · pp. 4–5

3 Results

Two-million-splat models closely matched converged references across three CT looks, while rendering in real time. Ablations showed that opacity mapping and faint splats materially affect fidelity.

3.1 Fidelity

  • At two million splats, wide-view PSNR was 32.6±0.9 dB for bone, 30.1±1.0 dB for contrast-enhanced vessels and 32.3±1.5 dB for skin and bone.SSIM was 0.960–0.980 and LPIPS 0.028–0.091; close-up views were about 3 dB lower. Errors concentrated at dense-structure edges for bone and vessels and across the translucent layer for skin and bone.Source: §3.1 · p. 5, Figure 2 · p. 6

3.2 Splat count and speed

  • A two-million-splat pass took 14.3 s on average (4.2–29.5 s) and rendered at 68 frames per second at 800 × 800 pixels on the current laptop.The desktop with a 4-GB graphics card took 29.4 s per pass and rendered at 28 frames per second.Source: §3.2 · p. 5, Table 3 · p. 7, §3.3 · p. 7

3.3 Ablations

  • Without opacity mapping, PSNR fell in every case for every look, with median drops of 1.3–5.9 dB depending on look and view.Without anti-aliasing compensation, wide-view PSNR fell by 3.8 dB for bone and 0.9 dB for vessels, with little change for skin and bone or close-ups.Source: §3.3 · p. 5
  • At equal splat count, a uniform voxel grid was as faithful as the octree for vessels and skin and slightly better for bone (+0.4 and +0.6 dB, all cases).Voxel-unit cells gave 22.5 dB for bone wide views with 5-mm slices, versus 34.3 dB with millimetre cells; the 1.5-mm data with millimetre cells reached 33.6 dB.Source: §3.3 · p. 5
  • Without faint splats, wide-view PSNR fell by 0.3 dB for bone, 0.4 dB for vessels and 1.6 dB for skin and bone.For case 2 skin and bone, it fell from 29.8 to 23.8 dB; the Table 4 comparison reports per-case PSNR with and without faint splats.Source: §3.3 · p. 7, Table 4 · p. 7

4 Discussion

CTBakeGS converts lit CT renderings into standard 3DGS models quickly, with fidelity varying by transfer function and view. Discussion highlights faint-splat retention, splat-budget trade-offs, and limits of the evaluation and intended scope.

4 Discussion

  • About a minute or less per conversion pass on a current laptop or a 2018 desktop with a 4-GB graphics card, usually two or three passes per model, without training or per-scene optimisation.The method is deterministic, and a new transfer function requires another conversion.Source: §4 · p. 7
  • 42–47 dB for the real-time volume renderer, versus about 33 dB for bone in wide views, 30 dB for bone in close-ups, 29 dB for skin and bone in close-ups, and 27 dB for vessels in close-ups.The discussion notes that dense-bone and thin-vessel edges were smoothed in close-ups.Source: §4 · p. 7
  • About 21–37 dB for trained models is not directly comparable because those studies used different data, resolutions, and references; those methods remain preferable when highest fidelity matters and their cost is acceptable.CTBakeGS removes their prerequisites rather than competing with them on fidelity.Source: §4 · p. 7
  • Without faint-splat retention, wide-view skin-and-bone PSNR in case 2 fell from 29.8 to 23.8 dB, and the thin outer layer was missing.In the Fig. 3 close-up, PSNR was 22.5 dB without faint splats and 30.1 dB with them; the restored layer showed fine grain.Source: §4 · p. 7, Figure 3 · p. 8
  • Isotropic splats may represent thin translucent sheets poorly, while rasterisers sort whole splats rather than samples along each ray.The skin layer is a simplification of real skin’s complex reflectance and subsurface scattering; realistic, easy-to-use CT skin rendering remains future work.Source: §4 · p. 9
  • At equal splat count, the adaptive octree was not more faithful than a uniform grid, but it allows a chosen splat budget independent of voxel count.The six volumes ranged from 0.14 to 2.86 million voxels; two million splats could be chosen for each.Source: §4 · p. 9
  • Six cases from one public dataset with 1.5-mm voxels bounded the evaluation; clinical thin-slice scans may contain up to ten times more voxels and convert more slowly.Image similarity to the same renderer does not establish anatomical accuracy or suitability for diagnostic reading, teaching, or communication.Source: §4 · p. 9
  • Further testing should include clinical thin-slice scans, faster conversion of thin-slice and very large volumes, and reader studies of teaching and communication value.The authors also identify testing preservation of fine structures and using view-dependent colours or anisotropic splats as future work.Source: §4 · p. 9

5 Conclusions

The conclusion reports direct, training-free conversion of photorealistic CT renderings into standard 3D Gaussian splats, displayed in a web browser. The reported wide-view errors and conversion times support interactive sharing and viewing as a potential use.

5 Conclusions

  • The approach directly converts photorealistic CT volume renderings into standard 3D Gaussian splats without training and displays them in a web browser.The authors report conversion in about a minute or less per pass on a current laptop and a 2018 desktop, and under four minutes per pass on a 2017 laptop with integrated graphics.Source: §5 · p. 9
  • About 2.3–3.1% of full scale was the models’ root-mean-square error in wide views.Source: §5 · p. 9
  • The approach could make complex CT data easier to share and view interactively without the CT data or a volume renderer, with potential value for communication and teaching.Source: §5 · p. 9

Abbreviations

The abbreviations section expands the paper’s imaging, rendering, and image-quality terms.

Abbreviations

  • CT means computed tomography; 3DGS means three-dimensional Gaussian splatting; EWA means elliptical weighted average.Source: Abbreviations · p. 9
  • GPU means graphics processing unit; RMS means root-mean-square.Source: Abbreviations · p. 9
  • LPIPS means learned perceptual image patch similarity; PSNR means peak signal-to-noise ratio; SSIM means structural similarity index.Source: Abbreviations · p. 9

Authors’ contributions

H.F. led conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, original-draft writing, visualization, supervision, project administration, and funding acquisition. W.S., C.L., T.H., and L.E. contributed to specified project and writing roles.

Authors’ contributions

Declarations

The study used publicly available, de-identified CT data, so ethics approval, consent to participate, and consent for publication were not applicable. The authors declared no competing interests.

Declarations

  • Ethics approval and consent to participate were not applicable because the study used only publicly available, de-identified CT data.The data came from the TotalSegmentator dataset, licensed CC BY 4.0.Source: Declarations · p. 10
  • The authors declared no competing interests.Source: Declarations · p. 10

Supplementary material

The supplementary material describes the CT cases and the scope and methods of additional comparisons, including thick-slice testing.

Supplementary material

  • The released cases use 1.5-mm isotropic voxels, and the listed conversion times are for one pass at final cell size with two million splats.Source: Supplementary material · p. 12, Table S1 · p. 12
  • The six-case comparisons use per-case metric differences, with metrics averaged over 48 wide or 16 close-up views and exploratory two-sided exact Wilcoxon signed-rank tests.The tests are not adjusted for multiple comparisons; the smallest attainable p-value is 0.031.Source: Supplementary material · p. 13, Table S2 · p. 13
  • Table S3 identifies a thick-slice comparison for case 1 and lists PSNR (dB), SSIM, and LPIPS as the reported measures.Source: Table S3 · p. 13
  • Each condition is compared with a reference rendered from the same data, using either 1.5-mm or 5-mm data, with two million splats.Source: Supplementary material · p. 13

Related papers

Loading 2610.37406v1…