PaperlayerView exact revision on arXiv

Source-linked AI summary

SurgGMF: Fully Causal Gaussian Motion Forecasting for Anticipatory Surgical Scene Rendering

Jingqian Sun, Yichao Tang

arXiv:2609.34733v1cs.CV

TL;DR

Surgical scene models largely reconstruct observed frames, while causal forecasting of surgical Gaussian states remains underexplored. SurgGMF forecasts future Gaussian motion residuals and renders them without target-frame attributes; learned temporal models outperform classical dynamics baselines, with accuracy and latency varying across implementations.

  • Problem

    Dynamic surgical scene models remain focused on observed-frame reconstruction, while causal forecasting of surgical Gaussian states under strict target-frame isolation is underexplored.

    Source: §1 · pp. 1–2

  • Method

    SurgGMF predicts future Gaussian position, scale, and rotation residuals from historical motion fields and renders with a full-causal-last protocol that excludes target-frame Gaussian attributes.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Learned temporal models outperform classical dynamics baselines; TKAN achieves the highest average accuracy, while GRU and LSTM offer more favorable module-level accuracy–latency trade-offs.

    Source: §5 · p. 12

  • Takeaways & Limitations

    SurgGMF establishes a framework for causal Gaussian motion forecasting, extending surgical Gaussian representations toward predictive scene modeling.

    Source: §5 · p. 12

    The study focuses on short-horizon geometry forecasting at the module level, and its FPS measurements do not represent end-to-end robotic-system real-time performance.

    Source: §5 · p. 12, §4.5 · p. 12

Abstract

from arXiv · show

Dynamic surgical scene modeling is essential for robotic perception, simulation, and decision support. Although existing neural rendering methods enable efficient reconstruction and rendering of deformable surgical scenes, they remain primarily focused on observed-frame reconstruction rather than forecasting future scene states. To this end, we present SurgGMF, a fully causal Gaussian motion forecasting framework for anticipatory surgical scene rendering. Rather than predicting future RGB images directly, SurgGMF forecasts future Gaussian motion states represented by position, scale, and rotation residuals (X/S/R) from historical Gaussian motion fields. To prevent target leakage, we introduce a full-causal-last rendering protocol, where future Gaussian states are rendered without accessing target-frame Gaussian attributes while preserving causal appearance propagation. We evaluate SurgGMF on 12 EndoNeRF and StereoMIS video slices using neural temporal learners and classical dynamics baselines under a unified forecasting protocol. Learned Gaussian motion forecasting consistently outperforms classical dynamics baselines in render space, demonstrating gains beyond hand-crafted state extrapolation. Latency analysis further reveals an accuracy--efficiency trade-off: under the current implementations, TKAN achieves the highest accuracy, whereas GRU and LSTM provide more favorable module-level latency profiles. These results establish SurgGMF as a reproducible framework for causal Gaussian motion forecasting and advance surgical Gaussian representations from retrospective reconstruction toward predictive scene modeling.

1 Introduction

Surgical scene models need short-term anticipation, but existing dynamic Gaussian methods mainly reconstruct observed scenes rather than causally forecast future Gaussian states. SurgGMF predicts future X/S/R residuals from historical motion fields and evaluates forecasting with target-frame attributes excluded.

1 Introduction

  • Surgical reconstruction supports intraoperative navigation and robotic assistance, motivating representations that anticipate beyond observed frames.Minimally invasive reconstruction is challenging because of nonrigid deformation, partial visibility, weak texture, specular reflection, and tool occlusion.Source: §1 · p. 1
  • SurgGMF addresses the underexplored problem of causally forecasting surgical Gaussian states under strict target-frame isolation.Existing dynamic Gaussian approaches mainly focus on reconstruction, motion modeling, or general-purpose extrapolation.Source: §1 · p. 2
  • SurgGMF forecasts future Gaussian position, scale, and rotation residuals from historical motion fields instead of predicting future RGB images.The framework renders future frames by updating Gaussian states.Source: §1 · p. 2
  • The full-causal-last protocol excludes target-frame Gaussian attributes while inheriting appearance and uncovered primitives from the last causally available frame.This is intended to prevent target leakage during future rendering.Source: §1 · p. 2
  • SurgGMF evaluates neural temporal learners against classical dynamics baselines and analyzes motion-component contributions and accuracy–efficiency trade-offs.The evaluation uses a unified X/S/R forecast-and-render protocol.Source: §1 · p. 2

2 Related Work

Prior work spans predictive world models, surgical reconstruction, dynamic Gaussian modeling, and Gaussian-state forecasting. SurgGMF focuses on causal, short-horizon surgical Gaussian forecast-and-render evaluation under a fixed historical observation boundary.

Predictive World Models and Structured Scene Forecasting

  • World models support future rollout through learned state representations and transition dynamics, but latent- or image-space states do not directly expose explicit geometry for spatial reasoning and view-consistent rendering.GaussianWorld and GaussianAD use structured Gaussian states for semantic occupancy and road-scene forecasting rather than continuously deforming surgical anatomy.Source: §2 · p. 2

Surgical Reconstruction and Dynamic Gaussian Modeling

  • Surgical neural fields reconstruct deformable environments but do not naturally expose persistent primitive-level trajectories for downstream temporal forecasting.Dynamic and surgical Gaussian methods represent and track observed tissue motion, but generally do not evaluate future-state extrapolation under a fixed historical observation boundary.Source: §2 · p. 3

Predictive Gaussian Dynamics and Temporal Modeling

  • GaussianPrediction, graph-based dynamics, ODE-GS, and ParticleGS demonstrate learned future extrapolation in general dynamic scenes.Their applications include motion extrapolation, future rendering, manipulation, or occupancy forecasting.Source: §2 · p. 3

SurgGMF's Positioning

  • SurgGMF evaluates short-horizon position, scale, and rotation residual forecasts from teacher-derived surgical Gaussian trajectories using one full-causal-last renderer.The renderer excludes target-frame Gaussian attributes, distinguishing this controlled surgical evaluation from general Gaussian extrapolation.Source: §2 · p. 3

Predictive Gaussian Dynamics and Temporal Modeling

  • GRU, LSTM, Transformers, and TKAN are treated as interchangeable forecasting backbones and compared with classical dynamics references in a common Gaussian residual space.Source: §2 · p. 3

3 Method

SurgGMF forecasts future Gaussian geometry from causal motion histories, then renders predictions without accessing target-frame Gaussian attributes. The framework compares temporal learners and dynamics baselines using shared inputs and a common rendering protocol.

3.1 Causal Gaussian Motion Forecasting

  • Gaussian trajectories are represented by canonical-relative position, raw scale, and raw rotation residuals, with complete finite history and target windows used for temporal alignment.The formal experiments use H = 10 and P = 5, predicting all five steps in a single forward pass.Source: §3.1 · pp. 4–5

3.2 SurgGMF Framework

  • Each temporally valid trajectory supplies a 10-step, 16-channel X/S/R history, and all learners and dynamics baselines receive the same causal information.The input combines center displacement, velocity, acceleration, scale residuals, and rotation residuals; predicted residuals are converted back into future Gaussian states.Source: §3.2 · pp. 5–6

3.4 Neural Learners and Dynamics Baselines

  • The forecasting backbone can be recurrent, KAN-based, or attention-based, with GRU, LSTM, TKAN, and Transformer learners compared against five residual-space dynamics baselines.All neural learners predict five future steps in one pass and use the same X/S/R target and full-causal-last renderer.Source: §3.2 · p. 5, §3.4 · p. 7

3.3 Full-Causal-Last Rendering Protocol

  • The full-causal-last protocol uses the causal fill frame as its base, injects predicted geometry, and inherits opacity and appearance without using target-frame Gaussian attributes.The target camera is used to render from the target viewpoint, while the target image and valid mask are reserved for image-space evaluation.Source: §3.3 · p. 6

3.4 Neural Learners and Dynamics Baselines

  • Neural learners regress future X/S/R residuals with an attribute-weighted mean-squared objective, assigning weights 1.0 to center displacement and 0.25 each to scale and rotation.No horizon-specific weighting is used.Source: §3.4 · pp. 6–7

4 Experiments and Analysis

Across 12 surgical video slices, SurgGMF evaluates causal Gaussian-motion forecasting in render space against classical dynamics, across prediction horizons, motion targets, and module-level latency. Learned methods outperform classical baselines, while joint X/S/R forecasting and model choice reveal accuracy–efficiency trade-offs.

4.1 Experimental Setup and Baselines

  • Evaluation covers 12 EndoNeRF and StereoMIS surgical-video slices using H = 10 history steps and P = 5 prediction steps, with 1,165 valid frame–horizon pairs per method after boundary filtering.Render-space quality is evaluated after full-causal-last rendering; baselines include constant velocity, linear fit, constant acceleration, and Kalman-CV.Source: §4.1 · pp. 7–8

4.2 Main Benchmark: Classical vs Neural Forecasting

  • Neural learners beat the best classical scores on all four metrics: PSNR +0.983–1.204 dB, SSIM +0.0242–0.0290, LPIPS −0.0040–0.0054, and MAE −0.0025–0.0031.Among classical methods, constant velocity leads PSNR, SSIM, and MAE, while persistence has the best LPIPS; linear fitting and Kalman-CV underperform persistence across all four metrics.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Qualitative and Horizon Analysis

  • Performance generally declines across the five prediction steps, but learned methods remain stronger than classical references at every evaluated horizon under full-causal-last rendering.Target-frame Gaussian attributes remain unavailable at each horizon, with appearance and uncovered primitives held at causal-last states.Source: §4.3 · p. 9, Figure 3 · p. 10

4.4 Gaussian Motion Target Ablation

  • Adding rotation to position improves PSNR by 0.972 dB for LSTM and 1.048 dB for TKAN; joint X+S+R performs best for both backbones.Adding scale to position improves PSNR by 0.330 dB for LSTM and 0.347 dB for TKAN; scale adds complementary information to X+R.Source: §4.4 · p. 10, Table 3 · p. 11

4.5 Accuracy–Efficiency Trade-off

  • TKAN achieves the highest average accuracy, while GRU (5.94 ms) and LSTM (8.25 ms) offer more favorable module-level accuracy–latency trade-offs.With AMP, forecast–render throughput is 55.1 FPS for GRU, 41.6 FPS for LSTM, and 14.0 FPS for Transformer; these are not end-to-end system measurements, and latency depends on the evaluated software backend.Source: Figure 4 · p. 11, §4.5 · pp. 11–12, Table 4 · p. 12

5 Conclusion

SurgGMF forecasts short-horizon surgical Gaussian motion under a full-causal-last protocol. Learned temporal models outperform classical dynamics baselines, while accuracy and module-level latency vary across learners.

5 Conclusion

  • Learned temporal models outperform classical dynamics baselines under the full-causal-last protocol.Independently trained forecasting-target ablations also support joint X/S/R forecasting.Source: §5 · p. 12
  • TKAN achieves the highest average accuracy under the current implementation, while GRU and LSTM offer more favorable module-level accuracy–latency trade-offs.The study focuses on short-horizon geometry forecasting evaluated at the module level.Source: §5 · p. 12
  • Future work will extend the study toward longer-horizon forecasting, joint geometry–appearance prediction, and end-to-end online deployment.Source: §5 · p. 12

Related papers

Loading 2609.34733v1…