X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

• A unified exterior-and-interior representation aims to provide consistent rendering from any viewpoint and support applications including cultural heritage, industrial inspection, and physically grounded visualization.The paper motivates this as a more complete account of real objects than either modality alone.Source: §1 · pp. 1–2
• Separate RGB and X-ray reconstruction methods leave interiors or surface appearance unmodeled, while independent modality models lack alignment and increase parameter count.RGB approaches such as NeRF [Mildenhall et al. 2021] and Gaussian splatting [Kerbl et al. 2023] capture external surfaces; X-ray methods such as NAF [Zha et al. 2022], X-Gaussian [Cai et al. 2024a], and R2-Gaussian recover internal density.Source: §1 · p. 2
• X-Splat uses one set of Gaussian geometric parameters for both modalities, with appearance attributes only on surface-proximal primitives, reducing duplicated geometry.Both RGB and X-ray gradients update the shared geometry.Source: §1 · p. 2
• It registers modalities using clean RGB multi-view stereo points and an X-ray-derived mesh rather than optimized Gaussians, addressing surface-to-interior misalignment.InsideOut [Lee et al. 2025] instead aligns separate, optimized Gaussian sets, which can contain floaters and optimization artifacts.Source: §1 · p. 2
• Joint optimization lets both modalities update shared parameters and adds an X-ray-derived surface-depth consistency loss to supervise RGB rendering.This targets the underuse of X-ray geometry in RGB optimization identified in prior joint reconstruction.Source: §1 · p. 2
• On InsideOut and the new Flower & Insect dataset, X-Splat reports more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and greater compactness than two independent models.The Flower & Insect dataset combines existing flower scans with newly captured insect specimens.Source: §1 · p. 2

External Appearance Reconstruction

• RGB reconstruction progresses from NeRF and implicit surfaces to real-time Gaussian splatting, with refinements for aliasing and surface fidelity, but does not recover internal structure.Related work also extends appearance reconstruction to sparse-view, pose-free, and dynamic settings.Source: §2 · p. 2

Internal Structure Reconstruction

• X-ray and CT methods recover internal density using analytical reconstruction, neural attenuation fields, Transformers, and Gaussian splatting, but X-rays do not capture external surface appearance.Gaussian methods include Beer–Lambert radiative rasterization in X-Gaussian [Cai et al. 2024a] and bias correction with a differentiable voxelizer in R2-Gaussian.Source: §2 · pp. 2–3

Cross-modal 3D Reconstruction

• InsideOut [Lee et al. 2025] fits two independent Gaussian sets with post-hoc visual alignment, whereas X-Splat uses geometric registration and shared geometry for joint optimization.Other RGB–X-ray combinations address image-level medical fusion, sensing-system calibration, or thickness prediction rather than a unified 3D representation.Source: §2 · p. 3

Imaging Formation: Reflection vs. Transmission

• RGB records surface-dominated radiance with transmittance-weighted color, whereas X-rays penetrate the object and measure a Beer–Lambert density line integral.Source: §3 · p. 3
• Despite their different forward operators, RGB and X-ray observe the same underlying object and share its 3D geometry.Source: §3 · p. 3

3D Gaussian Representation

• 3D Gaussian splatting represents scenes with anisotropic Gaussians whose centers, rotations, and scales define shared geometry [Kerbl et al. 2023].Source: §3 · p. 3

RGB Gaussian Splatting

• RGB Gaussians carry opacity and view-dependent spherical-harmonic color, and EWA-projected footprints are composited front to back [Kerbl et al. 2023].Source: §3 · p. 3

Radiative Gaussian Splatting

• Radiative Gaussian splatting assigns each primitive a scalar density and integrates it along X-ray rays as a closed-form sum of detector-plane 2D Gaussians [Cai et al. 2024a].Source: §3 · p. 3

4.1 Unified Gaussian Representation

• Shared geometry serves both modalities, while X-ray density and surface-only RGB opacity and color remain modality-specific, reducing duplicated parameters and allowing both modalities to refine geometry.Surface-only appearance reflects the assumption that surface reflection dominates RGB for the opaque objects studied, while interior primitives contribute negligibly to RGB pixels.Source: §4.1 · p. 4

4.2 Geometry-based Alignment

• RGB MVS and a coarse X-ray reconstruction provide surface point clouds that are registered using PCA initialization followed by scaled point-to-point ICP, rather than aligning optimized Gaussian sets [Lee et al. 2025].The X-ray surface is extracted from a voxelized CT volume, while the RGB surface comes from calibrated views; the resulting similarity transform maps RGB camera poses into the X-ray frame.Source: §4.2 · p. 4, Figure 3 · p. 4

4.3 Geometry-Constrained Joint Optimization

• Optimization first fits shared geometry and density to X-ray views from an FDK reconstruction, then jointly optimizes inherited X-ray parameters with initialized surface appearance while retaining X-ray photometric supervision.Retaining the X-ray loss during Stage 2 prevents RGB gradients from shifting shared geometry at the expense of X-ray fidelity.Source: §4.3 · p. 5
• The CT mesh defines the surface subset for RGB attributes, and an edge-aware depth loss uses CT depth edges as a stop-gradient target to align RGB and CT depth boundaries without enforcing absolute depth agreement.Surface appearance is initialized from nearby MVS colors, while opacity decays with distance from the CT mesh; the mesh is periodically re-extracted during Stage 2.Source: §4.3 · p. 5

4.4 Adaptive Density Control

• Adaptive density control tracks position gradients separately by modality, adds primitives when either modality underfits, and prunes only when both RGB opacity and X-ray density are low.RGB densification is restricted to the surface subset, X-ray densification can add primitives throughout the object, and low-opacity surface primitives are demoted rather than removed.Source: §4.4 · p. 5

5.1 Experimental Setup

• Evaluation covers nine scenes across InsideOut and Flower & Insect, using 50 training and 120 testing X-ray projections per scene and held-out RGB views.Metrics include RGB PSNR, SSIM, and LPIPS; X-ray 2D-PSNR/SSIM and 3D-PSNR; alignment Chamfer Distance; and Gaussian count and storage.Source: §5.1 · p. 6

5.2 Per-Modality Reconstruction Quality

• Ours improves RGB PSNR by around 1 dB over either 3DGS variant on InsideOut and over 2 dB on Flower & Insect, while matching R2-Gaussian’s X-ray quality across both datasets.FDK initialization does not outperform MVSinit, indicating that better initial geometry alone does not remove the single-modality ceiling; classical FDK reconstruction also degrades substantially relative to Gaussian-based methods.Source: §5.2 · p. 6, Table 1 · p. 7, Table 2 · p. 8

5.3 Joint RGB-X-ray Reconstruction

• PCA+ICP on clean MVS-surface and CT-mesh proxies improves Chamfer Distance by roughly 8× over InsideOut hierarchical alignment, with PCA and ICP providing complementary global and local alignment.PCA alone is insufficient for fine alignment, while identity-initialized ICP can reach local minima and near-degenerate scales on symmetric objects.Source: §5.3 · p. 6, Table 3 · p. 8
• Ours beats the Independent baseline on RGB, matches it on X-ray, and uses around 26% less storage on InsideOut.Sharing geometry and restricting appearance attributes to the surface set reduce storage, while joint X-ray supervision supplies geometric constraints that RGB photometry alone cannot provide; one unified model can render both modalities at the same viewpoint.Source: §5.3 · pp. 6–8, Table 4 · p. 8

5.4 Ablation Studies

• Removing CT-depth consistency lowers average RGB PSNR by around 0.16 dB, while removing the surface set inflates storage by roughly 2.5×.CT-depth consistency anchors RGB surfaces to X-ray-derived geometry; without a surface set, appearance attributes are allocated even to interior primitives that do not contribute to opaque-object RGB rendering.Source: §5.4 · p. 9
• Replacing geometry-based alignment with InsideOut-style hierarchical alignment lowers RGB PSNR by roughly 0.9 dB while X-ray quality remains stable.The resulting surface misalignment artifacts support the greater robustness of alignment based on clean geometry.Source: §5.4 · p. 9, Figure 6 · p. 9
• Compared with separate single-modality models, the unified representation improves RGB without compromising X-ray reconstruction, and outperforms Independent on every measured metric.Freezing CT geometry as a depth prior for a separate RGB model does not recover the gain; storage savings reach approximately 2.2× on Flower & Insect, and X-ray quality stays stable across ablation rows.Source: §5.4 · p. 10, Figure 8 · p. 10
• Under noisy, sparse, and limited-angle X-ray acquisition, degraded CT geometry, and sparse RGB supervision, the framework remains robust and the shared-geometry RGB gain widens as views become sparser.X-ray fidelity remains on par with the X-ray-only baseline, while alignment and RGB quality are essentially unaffected.Source: §5.4 · p. 10

6 Conclusion

• The unified framework improves RGB rendering and cross-modal alignment, preserves X-ray fidelity, and is more compact than two independent models.It shares Gaussian geometry across modalities, restricts appearance attributes to surface-proximal primitives, registers through clean geometric proxies, and uses an X-ray-derived surface-depth consistency loss.Source: §6 · p. 10

Limitations and future work

• Restricting appearance to surface-proximal primitives assumes surface-dominated reflection, so the model does not cover transparent or strongly scattering or transmitting materials.Supporting these materials would require volumetric appearance modeling, and paired RGB–X-ray datasets containing them are not yet available.Source: §6 · p. 10
• The framework may degrade when X-ray inputs are extremely noisy or blurry because shared geometry depends on a reasonably accurate X-ray reconstruction.Source: §6 · p. 10
• Joint training improves RGB but only matches the single-modality baseline on X-ray; enabling RGB supervision to improve X-ray reconstruction remains future work.Source: §6 · p. 10

A Dataset Details

• The appendix provides per-scene specifications and additional acquisition and rendering details to complement the main paper’s Datasets paragraph.Source: A Dataset Details · p. 13

A.1 Per-scene Specifications

• All scenes use 50 training and 120 testing X-ray projections at 512 × 512 pixels, uniformly sampled over [0°, 360°), alongside per-scene RGB and CT specifications.RGB views use a 1:3 train/test split; raw RGB images are downsampled to 800 pixels wide, and CT volumes are resampled or retained at the specified grids.Source: §A.1 · p. 13, Table 6 · p. 13

A.2 X-ray Synthesis Geometry

• X-rays are synthesized with TIGRE using cone-beam geometry, with object volumes scaled so their longest physical extent equals 2 while preserving aspect ratio.InsideOut volumes fill the unit cube; Flower and Insect occupy smaller regions along shorter axes, and Table 7 lists the geometric parameters.Source: §A.2 · p. 13, Table 7 · p. 13
• Insect scenes use a 3.5 × 3.5 scene-unit detector, versus 3.0 × 3.0 for others; synthesis is noise-free, with noisy, sparse, and limited-angle robustness evaluated in Sec. C.The larger detector accommodates the insects’ elongated bodies.Source: §A.2 · p. 13

A.3 RGB Capture and Rendering

• InsideOut RGB uses released photographs [Lee et al. 2025], downsampled from native 1830 × 1830–2597 × 2597 resolutions to 800 × 800 for training and evaluation.Source: §A.3 · p. 13
• Flower RGB views are rendered in Blender at 800 × 800 with 512 samples per pixel from assets, using 80 views along a five-revolution top-to-bottom spiral.The spiral spans elevations [−30°, 85°].Source: §A.3 · p. 13
• Each prepared insect specimen has 36 RGB photographs captured at 2048 × 1536 and downsampled to 800 × 600, preserving the 4:3 aspect ratio.Source: §A.3 · p. 13
• RGB metrics and rendering loss use foreground masks: Flower uses Blender alpha, while InsideOut and Insect masks are SAM-predicted through X-AnyLabeling [CVHub520 and contributors 2024; Kirillov et al. 2023; Ravi et al. 2024] and manually refined.PSNR, SSIM, and LPIPS are computed only inside the foreground mask.Source: §A.3 · p. 13
• MVSFormer [Cao et al. 2022] reconstructs training-view RGB into a dense initialization cloud, filtered by foreground masks and FPS-downsampled for alignment and Gaussian color initialization.This replaces the COLMAP sparse cloud used by 3DGS / InsideOut.Source: §A.3 · p. 13

A.4 Flower & Insect Dataset Gallery

• The five Flower & Insect scenes comprise two rendered Flower assets and three lab-captured insect specimens, each shown with RGB photographs and X-ray projections from different angles.The Flower scenes are Tulip and Eustoma; the insect scenes are E02, E15, and E32.Source: §A.4 · p. 14, Figure 9 · p. 14

B Implementation Details

• The appendix supplements the main-paper Implementation Details paragraph with the full optimization procedure, complete optimizer settings, densification and pruning thresholds, and remaining hyperparameters not explicitly listed in the main paper.Source: B Implementation Details · p. 14

B.1 Full Optimization Procedure

• Algorithm 1 consolidates X-ray pretraining, geometry-based cross-modal alignment, surface-aware initialization, and joint optimization with periodic CT-mesh refresh into one procedure.The procedure uses N1=3,000 and N2=12,000 iterations, refreshes the mesh every Tmesh=500 iterations, and samples 180 virtual geometry views.Source: §B.1 · p. 14
• Stage 2 retains X-ray photometric supervision, evaluates geometric loss from virtual views independent of training RGB views, refreshes the CT mesh, and restricts RGB-driven densification to the surface set.X-ray gradients can spawn primitives throughout the volume, while RGB gradients can spawn them only within the surface set.Source: §B.1 · p. 14

B.2 Optimizer and Learning Rates

• Adam uses β1=0.9 and β2=0.999, while view-dependent color uses spherical harmonics up to degree 3 and position, density, scaling, and rotation learning rates decay over 30,000 steps.Per-attribute rates follow 3DGS/R2-Gaussian defaults; Table 8 reports the rates.Source: §B.2 · p. 14, Table 8 · p. 14

B.3 Surface-aware Initialization

• Surface primitives receive opacity that decays from 0.5 at the surface to 1×10^-3 at the band edge, while primitives outside the surface set receive fixed opacity 1×10^-5.The decay scale is σ=0.2·τd=0.004.Source: §B.3 · p. 14

B.4 Densification and Pruning

• Densification runs every 100 iterations from iteration 500 through 8,000, using position-gradient thresholds 5×10^-5 for X-ray and 1×10^-6 for RGB; pruning and surface-set demotion use αmin=5×10^-3.The pruning density threshold is ρmin=1×10^-5.Source: §B.4 · p. 14

B.5 Alignment Pipeline

• Alignment extracts an X-ray surface proxy from the coarse Gaussian volume using Marching Cubes, aligns it to an RGB surface proxy with PCA initialization and similarity ICP, then assigns surface attributes by distance to the CT mesh.Marching Cubes uses iso-level 0.3; ICP uses a voxel-down-sampled correspondence radius of 0.01 and at most 2,000 iterations. For the geometric constraint, virtual views use radius 5.0 and RGB/CT depth maps are rendered at half the X-ray detector resolution.Source: §B.5 · pp. 14–15, §B.6 · p. 15

B.7 Hardware and Training Time

• InsideOut training takes 14–21 minutes per scene, averaging 16 minutes, for Stage 1 X-ray pretraining and Stage 2 joint optimization, excluding evaluation.The experiments use a single NVIDIA RTX A6000 GPU with 48 GB memory.Source: §B.7 · p. 15, Table 9 · p. 15
• Matryoshka takes the longest because its layered nesteddoll structure reaches approximately 215k Gaussian primitives, versus approximately 50–150k for the other three scenes.Runtime correlates with the final number of Gaussian primitives.Source: §B.7 · p. 15
• Flower and Insect have larger native voxel grids than InsideOut’s uniform 256^3 volumes, increasing X-ray rasterization cost while keeping total runtimes within the same order of magnitude.Source: §B.7 · p. 15

B.8 InsideOut Reproduction

• Because InsideOut’s official implementation and preprocessing are unavailable, the authors reproduce it from the paper’s descriptions and hyperparameters.Source: §B.8 · p. 15
• Only part of InsideOut’s capture data is public, and views from different physical placements lack metadata linking them, so only single-placement views can be registered consistently.For example, 251 of 555 Matryoshka views are released.Source: §B.8 · p. 15
• The reproduction may underestimate InsideOut’s original performance because its reported supplementary RGB metrics use full, non-public captures, whereas the reproduction uses a substantially smaller registrable subset.The authors say the discrepancy likely stems from the unreleased pipeline and smaller view subset.Source: §B.8 · p. 15

Noisy X-ray projections

• Under noisy X-ray training projections, X-ray 3D-PSNR degrades by only ∼0.5 dB for both Ours and R2-Gaussian, while RGB changes by at most 0.04 dB per scene.The passage reports no noise leakage into RGB and no additional fragility relative to the X-ray-only backbone.Source: C Robustness Analysis · pp. 15–16, Table 10 · p. 16

Sparse X-ray projections

• With 25 rather than 50 training projections, Ours matches R2-Gaussian across X-ray 2D-PSNR, 3D-PSNR, and surface CD.Joint RGB supervision does not compromise X-ray reconstruction under sparser acquisition.Source: C Robustness Analysis · p. 16, Table 11 · p. 16

Limited-angle X-ray acquisition

• With X-ray training confined to a 120° arc, Ours remains on par with R2-Gaussian across metrics in-arc and out-of-arc, with only marginal differences.Full-coverage RGB neither completes the unobserved 240° wedge nor measurably degrades reconstruction; X-ray remains self-sufficient.Source: C Robustness Analysis · p. 16, Table 12 · p. 16

Alignment under degraded X-ray geometry

• Re-deriving PCA+ICP alignment from the limited-angle CT mesh changes RGB by at most 0.07 dB per scene, with X-ray 2D-PSNR and surface CD likewise unaffected.This test indicates that geometric-proxy alignment is not fragile to degraded X-ray geometry.Source: C Robustness Analysis · p. 16, Table 13 · p. 16

Sparse RGB views

• With 15 RGB training views, Ours’ gain over single-modality 3DGS widens to +4.16 dB from +2.72 dB at full views, reaching +5.49 dB on Tulip and +6.65 dB on Eustoma.The results indicate that X-ray modality benefits RGB reconstruction more under sparse RGB supervision, especially for thin structures.Source: C Robustness Analysis · p. 16, Table 14 · p. 16

D More Results

• The appendix also extends RGB novel-view synthesis and cross-modal alignment comparisons to additional scenes.Source: D More Results · p. 16, Figure 11 · p. 18, Figure 13 · p. 19
• Additional CT-slice and X-ray-projection scenes show Ours matching R2-Gaussian’s fidelity on both modalities, while FDK exhibits the expected closed-form noise floor.Source: Figure 10 · p. 17
• On InsideOut, the frozen CT-prior RGB model is only 0.06 dB above single-modality 3DGS and 1.33 dB below Ours.The joint gain stems from jointly refining shared geometry under supervision from both modalities.Source: D More Results · pp. 16–17, Table 15 · p. 17
• Across additional unified RGB-X-ray scenes, Ours renders inside and outside consistently; Ours + IO-align has mild surface-alignment artifacts, and the InsideOut reproduction has lower RGB quality.Source: Figure 12 · p. 19