Source-linked AI summary
SpectralCTGaussians: Projection-Domain Reconstruction and Basis Material Decomposition for Spectral CT using 3D Gaussian Splatting
Reinout Vos, Saptarshi Neil Sinha, Michael Weinmann
TL;DR
Spectral CT resolves X-ray measurements into energy channels that can constrain material composition, but reconstruction must handle noisier channels and their coupled physics. SpectralCTGaussians jointly optimizes a Gaussian representation with per-Gaussian material bases; it outperforms traditional baselines in novel-view synthesis and achieves the best reported PSNR for spectral volume reconstruction.
Problem
Conventional CT records a single energy-integrated intensity, while 3DGS-based CT lacks a spectral representation for one-step basis material decomposition.
Source: §1 · pp. 1–2Method
Results
The method outperforms all traditional baselines in novel-view synthesis, achieves the best PSNR on multi-energy volume reconstruction, and performs one-step decomposition with direct RGB output.
Source: §7 · p. 9Takeaways & Limitations
One shared representation covers reconstruction and material decomposition while using approximately a third of the Gaussians of single-channel approaches.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Spectral computed tomography (CT) extends conventional CT by measuring attenuation across multiple energy channels, allowing improved modeling of physical X-ray interactions and energy-dependent material behavior and leading to richer scene understanding. We present a novel method for spectral CT reconstruction and basis material decomposition using 3D Gaussian Splatting by adding per-Gaussian basis material fractions to the set of learnable parameters, which together with a set of energy-dependent basis functions define the attenuation across the full spectral range. The basis functions represent various physical attenuation models such as photoelectric absorption and Compton scattering, and are jointly optimized across all energy channels through a differentiable polychromatic forward model, with material decomposition performed via mean-shift clustering of the resulting coefficients. We evaluate our method on a baseline real-world dataset as well as a synthetic dataset that we introduce, comparing against traditional reconstruction algorithms and state-of-the-art learning-based CT reconstruction methods. Our approach outperforms all traditional baselines in novel view synthesis and achieves the best PSNR among all compared methods for spectral CT volume reconstruction, while describing all energy channels with a single shared representation that requires a number of Gaussians comparable to single-channel Gaussian splatting-based CT reconstruction approaches. For basis material decomposition, no traditional or learning-based baseline offers one-step decomposition with direct RGB material segmentation, and our method additionally recovers the photoelectric basis with higher PSNR than traditional pipelines.
1 Introduction
Spectral CT's energy-resolved measurements can constrain material composition, but they create a noisier, more coupled inverse problem that existing approaches address only partially. SpectralCTGaussians extends 3D Gaussian Splatting with energy-dependent basis coefficients for joint spectral reconstruction and one-step material decomposition.
1 Introduction
- Energy-resolved measurements constrain material composition, unlike conventional CT's single energy-integrated intensity, and support basis material decomposition.The introduction also names contrast-agent quantification and virtual removal of bone or iodine as applications.Source: §1 · p. 1
- Spectral reconstruction is harder because each energy channel receives fewer photons and the channels are coupled through the polychromatic forward model.Existing approaches address the challenge only partially: analytical methods treat channels independently, iterative methods are expensive, two-step pipelines propagate errors, and learning-based methods often train per channel and generalize poorly.Source: §1 · p. 2
- Existing 3DGS-based CT assigns one density per Gaussian, so it cannot represent energy-dependent attenuation or perform one-step spectral basis decomposition.The authors position their work as the first 3DGS approach to support a spectral CT representation with one-step basis material decomposition.Source: §1 · p. 2
- SpectralCTGaussians adds energy-dependent basis coefficients to each Gaussian and optimizes all energy channels jointly through a differentiable polychromatic projection model.The representation uses a number of Gaussians comparable to a single-channel representation.Source: §1 · p. 2, Figure 1 · p. 2
- The method performs one-step basis decomposition using photoelectric, Compton, and learnable K-edge bases, then applies mean-shift clustering to produce material maps.The paper also introduces a synthetic spectral CT dataset with ground-truth material information for quantitative decomposition evaluation.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
Prior spectral CT approaches trade off physical modeling, computational cost, training demands, and cross-channel representation. Multi-view scene representations offer projection-based reconstruction, while existing Gaussian CT methods do not jointly represent spectral attenuation and direct material decomposition.
2 Related Work
- Channel-wise FBP, FDK, ART, SART, and SIRT are fast and easy to apply, but ignore polychromatic beam physics and inter-channel correlation.The text characterizes these methods as mainly useful for quick initialization because beam hardening and cross-channel effects remain unmodeled.Source: §2 · p. 2
- Model-based iterative reconstruction incorporates a spectral forward model, data fidelity, and regularization, but full-spectral methods are computationally expensive and sensitive to regularizer choice.The surveyed methods include low-rank, sparsity, tensor, and dictionary-learning approaches; full-spectral methods can provide beam-hardening correction and direct decomposition.Source: §2 · pp. 2–3
- Learning-based methods range from physics-informed and unrolled models to data-driven U-Nets and GANs, but often require large training sets and train separately by channel.The passage also notes that these methods can hallucinate structures on out-of-distribution data.Source: §2 · p. 3
- NeRF and 3DGS CT methods optimize scene representations from projections; 3DGS avoids dense ray sampling, but existing Gaussian CT representations lack spectral attenuation and direct material decomposition.Spectral NeRF variants add basis physics or query attenuation by energy, while optical spectral 3DGS models view-dependent appearance rather than X-ray attenuation.Source: §2 · p. 3
3 Methodology
The method represents the scanned object with 3D Gaussians carrying geometry, density, and material coefficients, then fits their attenuation to spectral CT projections with a physics-based forward model.
3.1 Spectral CT Gaussian Representation
- Each Gaussian combines a shared density with nonnegative material fractions, whose spatial coefficients weight shared energy-dependent photoelectric, Compton, and K-edge attenuation bases.This factorization separates geometry from material composition while retaining density-based pruning.Source: §3.1 · p. 3
3.2 Spectral CT Rendering
- Projected basis-coefficient line integrals are computed once per iteration, then combined across 64 energy samples to predict channel measurements with a polychromatic Beer–Lambert model.The energy grid uses uniform midpoint samples over 20–140 keV.Source: §3.2 · p. 4
- Photoelectric and Compton bases model distinct energy-dependent attenuation profiles, with an optional jointly optimized sigmoid basis for K-edge materials.The K-edge sigmoid is centered at a learned edge energy; its amplitude and sharpness are optimized with the Gaussian model.Source: §3.2 · p. 4
3.3 Training
- Initialization derives Gaussian geometry and basis fractions from a lowest-energy-channel FDK volume, then optimization fits rendered log-projections using L1, D-SSIM, and total-variation losses.The K-edge parameters are frozen during warm-up, and an additional 2D total-variation term regularizes projected basis-coefficient images.Source: §3.3 · pp. 4–5
3.4 Basis material decomposition
- After reconstruction, mean-shift clustering groups Gaussian basis coefficients into material maps without requiring a predefined cluster count.Clusters are ordered by coefficient ratio and density, then assigned semantic colors for segmentation images.Source: §3.4 · p. 5
4 Datasets
The evaluation uses a real-world multi-energy cone-beam dataset and a synthetic spectral phantom with known basis-material ground truth. The datasets and preprocessing provide held-out views for evaluation.
4 Datasets
- The evaluation uses one real-world multi-energy cone-beam dataset and a synthetic spectral phantom with known ground-truth basis material decomposition.The synthetic phantom supplies ground-truth decomposition because publicly available datasets do not.Source: §4 · p. 5
- Figure 2 presents bird-chest novel-view results with rendered projections beside their absolute-error maps.The figure caption identifies the dataset as multi-energy.Source: Figure 2 · p. 6
- The 128^3 synthetic phantom contains soft-tissue, bone, and K-edge cylinders, with a sharp K-edge transition at 63 keV.Each voxel stores ground-truth photoelectric, Compton, and K-edge coefficients.Source: §4 · p. 6
- Real-world data are normalized by approximate unattenuated incident intensity and log-transformed when needed; methods share 75 training and 100 held-out test views.Datasets are converted to a common format with per-channel folders and shared scanner, spectral, and view-split metadata.Source: §4 · p. 6
5 Implementation details
The method uses custom CUDA rasterization and voxelization kernels in PyTorch, with spectral integration and adaptive Gaussian growth during training.
5 Implementation details
- Training uses a spectral forward model with 64 midpoint samples over 20–140 keV and runs for 30k Adam iterations on one Nvidia RTX 3090.The implementation uses PyTorch with custom CUDA rasterization and voxelization kernels.Source: §5 · p. 6
- The point cloud starts from 50k Gaussians sampled from the FDK volume and grows through adaptive density control to at most 500k Gaussians.Source: §5 · p. 6
- The loss weights are λDSSIM = 0.25, λ3DTV = 0.005, and λTV,PE = λTV,C = λTV,K = 0.05; K-edge parameters are frozen for 2000 iterations before separate-optimizer training.Source: §5 · p. 6
6 Evaluation
The evaluation compares spectral CT reconstruction, novel-view synthesis, and basis material decomposition with traditional and learning-based baselines. Results show strong performance across these tasks, alongside limits in surface reconstruction and sensitivity to spectral configuration.
6.1 Comparison with baseline methods
- The comparison includes channel-wise FDK, SART, CGLS, and PDHG-TV, plus separately trained single-channel learning models; Ours (3) adds a K-edge basis to Ours (2).X-Gaussian and X-Field are excluded from volume comparisons because they cannot produce volumes.Source: §6.1 · p. 6
6.2 Quantitative results
- The method matches R2-Gaussian on novel-view accuracy while using one shared representation, and exceeds traditional methods in PSNR and SSIM across datasets.For LPIPS, it beats every method except FDK; R2-Gaussian uses a separate model per energy channel, requiring three times as many Gaussians overall.Source: §6.2 · p. 7
- For spectral reconstruction, the method leads all compared approaches in PSNR and beats learning-based methods in SSIM and LPIPS, with some exceptions.The volume comparison uses an FDK-derived pseudo ground truth, so the numbers are indicators rather than exact-reference scores; R2-Gaussian has comparable SSIM and LPIPS using a third of the Gaussians.Source: §6.2 · p. 7
- On synthetic basis decomposition, the method leads photoelectric recovery by about 3 dB PSNR and 0.14 SSIM over the strongest traditional solver, and leads average SSIM.Traditional solvers retain higher average PSNR, mainly through stronger Compton-basis PSNR; Ours (3) beats Ours (2) on all three material-map metrics.Source: §6.2 · pp. 7–8
6.3 Qualitative results
- Projection error maps favor the proposed method and R2-Gaussian, while the proposed method’s residual errors concentrate at bone edges and the object silhouette.The two proposed variants look nearly identical because the scanned object has no K-edge material in the scanned energy range; shared geometry also produces smoother projections than channel-wise reconstructions.Source: §6.3 · p. 8
- Axial slices preserve tissue and bone contrast better than the other learning-based reconstructions, but no method produces a fully convincing outside surface.Background noise creates spurious density around the object; the proposed method retains a recognizable shape and comparatively uniform background.Source: Figure 3 · p. 7, §6.3 · p. 8
- Synthetic material maps place and color the phantom structures correctly, while real-data maps separate bone, soft tissue, and metal plausibly despite bone oversaturation.The real bird-chest dataset has no ground-truth material maps, so its decomposition is assessed qualitatively.Source: §6.3 · pp. 8–9, Figure 4 · p. 8
6.4 Ablation Studies
- Ablations show that removing 80 keV costs relatively little, fewer views reduce quality, and removing decomposition controls can collapse basis coefficients.Holding out 50 or 120 keV degrades more because it requires extrapolation; reducing views from 75 to 25 costs under 1 dB per channel. A monochromatic model scores best by fitting channels independently, indicating a need for precise spectral configuration.Source: Table 4 · p. 8, §6.4 · p. 9
7 Conclusion
The method uses one spectral Gaussian representation to support novel view synthesis, volume reconstruction, and basis material decomposition. It outperforms traditional baselines in novel view synthesis, achieves the best PSNR on multi-energy bird chest data, and offers one-step projection-domain decomposition with direct RGB output.
7 Conclusion
- The method outperforms all traditional baselines in novel view synthesis, achieves best PSNR on multi-energy bird chest volume reconstruction, and is first to offer one-step projection-domain decomposition with direct RGB output.It also recovers the photoelectric basis more accurately than traditional pipelines.Source: §7 · p. 9
- A single representation with per-Gaussian basis material fractions and a differentiable polychromatic forward model supports novel view synthesis, volume reconstruction, and basis material decomposition.It uses approximately one-third as many Gaussians as single-channel approaches while remaining competitive with per-channel learning-based methods in novel view synthesis.Source: §7 · p. 9
- Absolute basis scaling remains the model’s weakest aspect; the ablations indicate that decomposition relies on stabilizing components and requires precise forward-model calibration.Future work could improve calibration, reduce photoelectric–Compton cross-talk, and extend the learnable K-edge component to multiple edges.Source: §7 · p. 9