Source-linked AI summary
H-2DGS: Height-Field Gaussian Splatting for Sharp and Deformable Neurosurgical Surface Reconstruction
Zhang Zichuan, Bart Brouwers, Rembrandt Bakker, Maud Landers, Geert-Jan Rutten, Maxime Chamberland
TL;DR
MRI slices and microscope views make it difficult to relate exposed cortical anatomy to internal structures, while volumetric Gaussian splatting can leave an ambiguous boundary that obscures tractography. H-2DGS converts a volumetric reconstruction into an explicit curved surface and uses it for postoperative alignment. Across five cases, it maintains a localized cortical surface in uncovered views and achieves lower multi-view depth inconsistency than baseline approaches.
Problem
MRI slices differ from the exposed cortical view, and limited microscope view coverage can leave conventional 3DGS with a thick, ambiguous opacity band that obscures the boundary around internal anatomy.
Source: §1 · p. 1Method
H-2DGS concentrates a volumetric reconstruction, fuses multi-view depth into a shared TSDF, and represents the resulting surface with curved height-field Gaussian primitives.
Source: §1 · p. 2Results
Across five cases, H-2DGS consistently maintains a localized cortical surface in uncovered views and obtains lower multi-view depth inconsistency than baseline approaches.
Source: §6 · p. 8Takeaways & Limitations
The explicit surface provides a clear cortical boundary for integrating tractography and whole-brain MRI context, and can be deformed toward postoperative MRI geometry while preserving photorealistic appearance.
Source: §6 · p. 8H-2DGS reproduces observed images less accurately than baseline methods, with a remaining photometric gap from reduced high-frequency appearance expressiveness.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Neurosurgical navigation commonly presents magnetic resonance imaging (MRI) as 2D slices, making it difficult to relate the anatomy in those slices to the cortical surface observed through the surgical microscope. We seek a common 3D visualization that combines a photorealistic reconstruction of the exposed cortex during tumor surgery with whole-brain MRI context and diffusion tractography pathways. This requires the reconstructed cortex to form a clear boundary around the brain anatomy rather than a diffuse volume that obscures internal structures. Conventional 3D Gaussian Splatting (3DGS) can reproduce observed views but, under limited surgical view coverage, may distribute opacity through a thick and geometrically ambiguous band. We introduce height-field 2D Gaussian Splatting (H-2DGS), a three-stage pipeline that concentrates a robust 3DGS reconstruction, fuses its depth into a shared truncated signed distance field (TSDF) surface, and represents that surface with curved, surface-oriented Gaussian primitives. Separate height, color, and opacity bases provide local geometry and appearance, while a neighbor-transition objective coordinates overlapping patches. The explicit surface further supports postoperative alignment: MRI-derived depth deforms the reconstruction, and tracked-color supervision with alternating optimization preserves its appearance. Across five real neurosurgical cases, H-2DGS gives low multi-view depth inconsistency, a compact 7.97MB representation, and interactive rendering at 29--42ms per frame. The resulting sharp, deformable surface integrates clearly with tractography and MRI anatomy, including a postoperative resection cavity.
1 Introduction
MRI slices and microscope views make it difficult to relate exposed cortical landmarks to internal anatomy, while volumetric splatting can leave an ambiguous boundary that obscures tractography. H-2DGS builds a sharp cortical surface for a shared 3D view of the cortex, whole-brain MRI, and fiber pathways.
1 Introduction
- MRI slices do not resemble the exposed cortex, leaving surgeons to mentally relate visible landmarks and resection boundaries to anatomy in the MRI volume.The missing 3D spatial context increases cognitive load and makes the relationship less intuitive.Source: §1 · p. 1
- A shared 3D scene combines microscope-derived cortical landmarks, whole-brain MRI context, and diffusion tractography pathways beneath the visible surface.Continuous viewpoint changes can connect the surface seen by the surgeon to internal structures.Source: §1 · pp. 1–2
- Under narrow microscope camera trajectories, conventional 3DGS can explain the same pixels at multiple depths, producing a thick or fragmented band that obscures tractography.The resulting ambiguity blurs the boundary between the visible cortical surface and subcortical anatomy.Source: §1 · p. 1
- H-2DGS progressively concentrates a volumetric reconstruction, fuses multi-view depth into a shared TSDF, and converts it into curved height-field surface primitives.Separate geometry, color, and opacity bases support local surface shape and appearance, while neighbor-transition optimization coordinates overlapping patches.Source: §1 · p. 2
- For postoperative alignment, MRI-derived depth deforms the explicit surface, while tracked-color supervision and alternating geometry–appearance updates preserve cortical texture.The approach addresses brain motion and tissue removal that can change the relationship between the reconstructed surface and MRI.Source: §1 · p. 2
2 Related Work
Prior Gaussian surface methods initialize designed primitives directly from point clouds, which may be weakly constrained under limited view coverage. H-2DGS instead concentrates a volumetric reconstruction and fuses its depth before initializing curved surface charts, with a height basis that also makes each primitive deformable.
2 Related Work
- H-2DGS concentrates a volumetric reconstruction and fuses multi-view depth into a shared TSDF before initializing curved surface charts, rather than initializing designed primitives directly from potentially weak point-cloud geometry.Its height basis makes each primitive a deformable local surface, alongside per-primitive appearance bases related to textured Gaussian representations [19] [20] [23].Source: §2 · p. 2
- Prior surgical reconstruction methods often prioritize view synthesis or temporal tissue motion, whereas H-2DGS focuses on a sharp cortical boundary for tractography and postoperative geometric alignment.Cross-modal methods connect intraoperative imagery to preoperative anatomy through inverse rendering or coupled meshes [4] [5].Source: §2 · p. 2
3 Sharp Height-Field Surface Reconstruction
H-2DGS separates appearance fitting from surface construction, then turns multi-view depth into a shared, explicit cortical surface. Curved patches retain local appearance while coordinating overlaps for continuous geometry.
3.1 Problem formulation and design rationale
- H-2DGS separates appearance acquisition, geometry localization, and explicit-surface encoding because sparse microscope views make direct surface fitting brittle.Stage I bootstraps appearance with flexible volumetric Gaussians; Stage II contracts ray-wise density; Stage III replaces the volume with curved height-field patches.Source: §3.1 · p. 3
3.2 Stages I–II: from appearance to localized geometry
- Stage I establishes visible tissue texture, while Stage II narrows foreground-ray density using differentiable quantiles and retains photometric fitting.The 1% and 99% quantiles define the density interval; contraction is constrained to preserve observed cortical images rather than collapse density at an arbitrary depth.Source: §3.2 · p. 3, Figure 2 · p. 3
3.3 Stage III: TSDF-guided surface conversion
- Stage III fuses rendered depths into a TSDF whose zero level set initializes a shared surface and supplies patch positions and orientations.View-consistency filtering and local quadric fitting produce overlapping charts covering the fused surface.Source: §3.3 · p. 4, Figure 3 · p. 4
3.4 Height-field Gaussian primitive
- Curved height-field patches bend beyond a planar surfel while carrying spatially varying geometry, color, and opacity on one local domain.Independent bases let appearance vary more rapidly than geometry; explicit surface intersections provide camera-linear depth for later alignment.Source: §3.4 · p. 4, Figure 4 · p. 4
3.5 Initialization, continuity, and refinement
- TSDF samples initialize curved charts, and overlap constraints coordinate neighboring patches to reduce interface disagreement while preserving fitted geometry.Continuity, preservation, and curvature terms address patch interfaces, TSDF fidelity, and oscillations, respectively; photometric refinement then recovers cortical appearance.Source: §3.5 · pp. 4–5
4 Postoperative Surface Alignment
Postoperative alignment deforms a microscope-derived height-field surface toward MRI-rendered depth while preserving cortical appearance. Depth fitting, neighboring-chart transitions, tracked color, and alternating update windows address distinct failure modes of naive fitting.
4.1 Problem setup and optimization strategy
- MRI-rendered depth supervises deformation of the initial microscope-derived height-field model, with the goal of following the target while retaining cortical texture.The setup uses image-space depth comparisons for the same cameras, without sparse cross-modal point correspondences.Source: §4.1 · p. 5
- Naive depth fitting can let neighboring patches tear, alter opacity to change the observed surface, or smear texture as geometry moves.Depth loss supplies target shape, transition regularization couples adjacent charts, and tracked self-supervision transports appearance; alternating geometry and appearance windows separate their updates.Source: §4.1 · p. 5
4.2 Depth-supervised surface deformation
- The deformation pass trains centers, rotations, tangent scales, appearance, opacity, and height coefficients, while fixed basis centers and bandwidths preserve local coordinate systems.Opacity is overridden to 0.999 during depth fitting so a mismatched foreground patch cannot reduce the loss by becoming transparent.Source: §4.2 · p. 5
4.3 Tracked appearance preservation
- Tracked samples bind initial rendered colors to local primitive coordinates, then move with deforming charts and enforce pixel-wise color consistency.Each sample is bound to eight candidate primitives, with targets aggregated in image space.Source: §4.3 · pp. 5–6
- Alternating geometry-active and appearance windows lets MRI supervision reshape the surface before texture fields are restored with geometry frozen.The appearance window uses multi-view augmentation; together, the windows separate MRI-driven shape changes from restoration of the original photorealistic appearance.Source: §4.3 · pp. 6–7
5 Experiments
Across five surgical MRI cases, H-2DGS trades some photometric fidelity for stable geometry, compact representation, and fast rendering. Postoperative depth fitting aligns the surface to MRI morphology while appearance updates recover texture.
5.1 Setup and metrics
- Five real neurosurgical cases use 60–80 sampled training images each; evaluation covers appearance, efficiency, and normalized multi-view depth consistency.The methods compared are H-2DGS, 2DGS, 3D-HGS, and QGS.Source: §5.1 · p. 7
5.2 Sharpness, consistency, and fusion
- In uncovered views, H-2DGS remains a recognizable cortical layer while baseline geometry can streak, fragment, or blur; tractography fibers appear attached to a defined surface.This contrasts with diffuse geometry that can obscure anatomical fusion.Source: §5.2 · p. 7
5.3 Postoperative deformation
- Postoperative depth fitting moves the reconstruction toward MRI morphology, stretching the surface laterally and shallowing the cavity without losing major tissue or vessel patterns.Appearance-window updates restore cortical texture with geometry frozen, while stronger transition regularization improves local geometric and texture coherence.Source: §5.3 · pp. 7–8, Figure 9 · p. 7
5.2 Sharpness, consistency, and fusion
- MVDC is 1×10−5–4×10−5, with 8 MB models rendering in 30–42 ms; QGS uses 79–142 MB and 120–188 ms while attaining the strongest photometric scores.H-2DGS has lower training-view PSNR and SSIM than the baselines, mainly losing high-frequency highlights.Source: §5.2 · p. 7, Table 1 · p. 8
6 Discussion and Conclusion
H-2DGS maintains a localized cortical surface across viewpoints and integrates it with MRI and tractography; postoperative fitting accommodates pronounced geometric discrepancy while preserving photorealistic appearance. Its limitations include a photometric gap and sensitivity to dynamic tissue motion affecting SfM poses.
6 Discussion and Conclusion
- H-2DGS maintains a localized cortical surface in uncovered views and achieves lower multi-view depth inconsistency across five cases than baseline approaches.The sharp boundary supports integration with subcortical tractography and whole-brain MRI context without obscuring internal anatomy.Source: §6 · p. 8
- Postoperative fitting accommodates pronounced geometric discrepancy while preserving the original reconstruction’s photorealistic appearance.The experiment separates depth fitting from appearance preservation.Source: §6 · p. 8
- The 2,400-primitive budget and per-primitive texture yield a smaller footprint, but primitive counts are not matched to the baselines.The paper attributes the footprint advantage to the height-field primitive, noting volumetric primitives without local texture generally need more primitives to fit the same images.Source: §6 · p. 8
- H-2DGS reproduces observed images less accurately than baseline methods by PSNR and SSIM, leaving a photometric gap.The thin explicit surface reduces rendering expressiveness for high-frequency appearance variations despite spatially varying color bases.Source: §6 · p. 8
- SfM’s static-scene assumption can fail under cortical pulsation, confounding tissue and camera motion and degrading reconstruction stages.The authors identify calibrated or externally tracked poses, time-varying primitive geometry, and pulsation-aware frame selection or correction as potential solutions.Source: §6 · p. 8
- The authors identify richer geometry-preserving appearance models and cinematic rendering as future directions for improving fidelity and surgical navigation.Addressing appearance expressiveness and dynamic tissue motion is described as important for extending H-2DGS to more diverse surgical videos.Source: §6 · pp. 8–9