PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT

Danial Arbabi, Korab Hoxha, Angelo Henriques, Mirza Imamovic, M. Ali Nasseri

arXiv:2609.07857v1cs.ROcs.CV

TL;DR

Robotic ophthalmic surgery can remove direct tactile feedback, leaving surgeons reliant on visual cues. This paper builds scene-graph-driven visual-to-haptic guidance and reports reduced needle alignment error without a statistically significant change in task completion time.

  • Problem

    Robotic platforms can eliminate direct tactile feedback, leaving surgeons reliant on visual cues and increasing cognitive load during delicate procedures.

    Source: B. Robotic Assistance and the Sensory… · p. 2

  • Method

    The system constructs a real-time surgical scene graph from simulated iOCT and uses a rule-based engine to map scene relations to context-aware haptic cues.

    Source: §I · p. 1

  • Results

    Mean alignment error fell from 7.9°± 2.6° to 6.8°± 2.3° (p = 0.044), while task completion time showed no statistically significant change.

    Source: B. Safety and Error Reduction · p. 5, A. Task Efficiency · p. 5

  • Takeaways & Limitations

    Participants altered their trajectories to silence misalignment vibrations, which the authors report promoted safer behavior without active motor restriction.

    Source: §VI · p. 6

Abstract

from arXiv · show

Robotic ophthalmic surgery offers high precision but introduces a "sensory gap" by decoupling the surgeon from their instrument, resulting in a loss of tactile feedback. This paper presents a novel haptic feedback system for subretinal injection tasks leveraging Scene Graphs (SG). The system bridges the sensory gap by analyzing a physically simulated intraoperative Optical Coherence Tomography (iOCT) feed to construct a real-time surgical SG. The SG serves as a semantic abstraction layer for the surgical scene, which is then utilized by a deterministic, rule-based engine to generate state-dependent haptic feedback on a robotic input device. The system was evaluated in a user study (N=16) using an anthropomorphic head phantom and a custom-built surgical robot. Results demonstrate that the SG-driven haptic feedback improved surgical precision, reducing needle alignment error by 14% (p = 0.044) and improving System Usability Scale (SUS) scores by 8% (p = 0.015), while maintaining comparable task completion times. A needle trajectory analysis revealed the emergence of a safer "Align-then-Approach" strategy, in which our haptic negative reinforcement prompted users to fine-tune the tool's trajectory before approaching the retinal target. This work suggests that SGs can effectively serve as the direct computational foundation for real-time, safety-enhancing context-aware haptic feedback in robotic microsurgery.

I. INTRODUCTION

Robotic ophthalmic surgery can remove direct tactile feedback, leaving surgeons to rely on visual cues. This work uses scene graphs and rule-based inference to connect visual perception with context-aware haptic feedback.

I. INTRODUCTION

  • The sensory gap in robotic surgery removes direct tactile feedback, forcing reliance on visual estimation and possibly increasing cognitive load.Subretinal injection requires sub-millimeter precision, often beyond the physiological limits of human surgeons because of hand tremor.Source: §I · p. 1
  • Visual-to-haptic feedback offers a software-driven alternative to force-sensing hardware by inferring geometry and forces from existing imaging infrastructure such as iOCT.The approach uses computer vision rather than requiring complex hardware changes.Source: §I · p. 1
  • The proposed framework uses a real-time surgical scene graph and a rule-based engine to map scene relations to context-aware haptic cues, validated in a user study with 16 participants.The perception pipeline builds the graph from simulated iOCT captured from the side view of a physical eye phantom.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

Subretinal injection delivers medication to predefined retinal layers and is used for several retinal conditions and therapeutic cargos. It requires precise needle navigation through the vitreous and retina while avoiding damage to underlying layers.

A. Clinical Motivation: Subretinal Injection

  • Subretinal injection delivers a controlled dose of medication into predefined retinal layers, reaching photoreceptors and the retinal pigment epithelium.It is used to treat submacular hemorrhages, macular folds, and age-related macular degeneration, and to administer gene- and stem-cell cargos.Source: A. Clinical Motivation: Subretinal… · pp. 1–2
  • The procedure requires navigating a needle through the vitreous cavity and penetrating the neurosensory retina without damaging underlying layers.Surgeons are often assisted by intraoperative optical coherence tomography.Source: A. Clinical Motivation: Subretinal… · p. 2
  • Physiological hand tremor and drift can exceed the safety bounds for micrometer-scale retinal tissue manipulation.The procedure therefore demands extreme precision.Source: A. Clinical Motivation: Subretinal… · p. 2

B. Robotic Assistance and the Sensory Gap

Robotic assistance can improve motion precision but separates surgeons from instrument touch, while scene graphs offer a semantic route to context-aware haptic guidance. The proposed framework combines imaging, computer vision, and rule-based inference for a physically simulated iOCT-guided setup.

B. Robotic Assistance and the Sensory Gap

  • Robotic platforms filter tremor and scale motion, but separating surgeons from instruments removes direct tactile feedback and can increase reliance on visual cues and cognitive load.A visual-to-haptic approach offers a software-driven alternative that can use existing imaging infrastructure.Source: B. Robotic Assistance and the Sensory… · p. 2

C. Scene Graphs in Surgical Robotics

  • The scene graph serves as a real-time state machine for closed-loop haptic guidance, addressing the open challenge of using surgical graphs for active robotic control rather than passive monitoring.Unlike traditional virtual fixtures based on geometric constraints and force feedback, this approach uses semantic context for vibrotactile guidance.Source: C. Scene Graphs in Surgical Robotics · p. 2

III. SYSTEM DESIGN

  • The system combines compact imaging, computer vision, and a semantic rule-based engine to simulate iOCT-guided robotic surgery in an anthropomorphic workspace.The optical setup is a low-cost proxy for iOCT geometry, while the scene-graph framework is designed to accommodate future clinical OCT feeds.Source: §III · p. 2

A. Anthropomorphic Phantom and Imaging Setup

An anthropomorphic phantom and concealed mirror-based camera provide a constrained workspace and cross-sectional eye view. The perception pipeline processes the feed to track the needle and create scene-graph relations from detected geometry.

A. Anthropomorphic Phantom and Imaging Setup

  • The anthropomorphic phantom reproduces facial constraints such as the orbital rim and nasal bridge that can limit robotic-arm maneuverability.A concealed D405 camera and 45° mirror capture a 2D cross-sectional view of the eye and instruments at 1280 x 720 pixels and 30 fps.Source: A. Anthropomorphic Phantom and Imaging… · p. 2

B. Perception Pipeline

  • The perception pipeline tracks instruments relative to eye anatomy, using precalibrated contours and adaptive Gaussian thresholding with contour detection to identify the needle shaft.Exponential moving averaging smooths needle movement to balance stability and responsiveness.Source: B. Perception Pipeline · pp. 2–3
  • The pipeline modifies the processed image to resemble an OCT feed, then uses sequential eye-globe, retina, and needle detectors to generate primitives for scene-graph construction.Spatial trigger zones map needle position to discrete relations such as approaching or aligned.Source: B. Perception Pipeline · p. 3, Figure 3 · p. 3, Figure 4 · p. 3

C. Surgical Scene Graph

The Surgical Scene Graph (SG) represents the surgical scene as labeled entities and relations, linking camera-derived geometry to higher-level context for decision-making. Geometric zones instantiate graph edges, while the SG separates perception from downstream feedback logic.

C. Surgical Scene Graph

  • The SG is a labeled, directed graph whose nodes represent categorized entities and whose labeled edges encode relations as subject-predicate-object triplets.Its graph at time t is Gt = (V, E, O, R, ϕ).Source: C. Surgical Scene Graph · p. 3
  • For every frame, the system analyzes the image and dynamically computes a corresponding SG.The graph is the semantic middleware between low-level camera data and surgical context.Source: C. Surgical Scene Graph · p. 3
  • A needle tip within the Retinal Zone at distance ≤ 0.7mm adds a touching-retina-surface edge, while the SG decouples perception from feedback logic.The architecture supports adding entities or relations without changing the downstream inference engine and enables queries combining semantic relations.Source: C. Surgical Scene Graph · p. 3

D. Rule-Based State Inference and Haptic Feedback

A deterministic rule-based engine queries the scene graph to infer discrete surgical states and produce interpretable haptic commands. The state machine distinguishes aligned from misaligned trajectories, and its graph representation links active semantic relations to haptic inference.

D. Rule-Based State Inference and Haptic Feedback

  • In the scene-graph illustration, nodes represent entities and edges represent active semantic relations used for haptic inference.The figure pairs a processed camera view of the instrument interacting with the retina with its corresponding graph.Source: Figure 5 · p. 3
  • The listed graph thresholds were tuned to phantom scale and expert feedback, and clinical use requires dynamic adaptation to tissue deformation.The table defines entity categories, semantic relations, and geometric trigger thresholds.Source: Table I · p. 4
  • At each time step, a deterministic rule-based engine queries graph topology Gt to infer surgical state St, which makes the resulting haptic command Ht explicit and interpretable.The engine is used instead of a probabilistic model to keep the system predictable and explainable.Source: D. Rule-Based State Inference and… · pp. 3–4

1) Surgical State Machine:

  • Misaligned states have trajectory deviation θ > 1.7°, while aligned states have θ ≤ 1.7° and are mapped to a haptic grammar.The finite-state machine has seven distinct states arranged in parallel columns according to semantic alignment.Source: 1) Surgical State Machine · p. 4

2) Haptic Grammar and Feedback Strategy:

The haptic grammar uses silence for aligned navigation and continuous vibration for misaligned trajectories, with distinct signals for retinal contact and target arrival.

2) Haptic Grammar and Feedback Strategy:

  • Aligned-path navigation is silent, minimizing sensory fatigue and overload while implicitly indicating that the trajectory is safe to follow.The haptic command is Off in this state.Source: 2) Haptic Grammar and Feedback Strategy · p. 4
  • Misaligned-path states trigger continuous low-frequency vibration until the instrument angle is corrected toward the retinal target.The vibration warns that the current trajectory may contact the retina away from the target.Source: 2) Haptic Grammar and Feedback Strategy · p. 4
  • Retinal contact at an unwanted position triggers rapid pulsing, whereas reaching the target triggers a short double-burst success signal.These critical-event signals are distinct from the continuous guidance cue.Source: 2) Haptic Grammar and Feedback Strategy · p. 4

3) Hierarchical Logic:

The inference engine prioritizes safety-critical graph conditions over lower-priority task information. A 16-participant within-subject study compared visual feedback alone with visual plus haptic feedback using randomized task order and logged performance measures.

3) Hierarchical Logic:

  • A needle-tip-touching-retina-surface graph relation immediately triggers Retinal Contact, overriding lower-priority states so guidance cues cannot suppress critical warnings.Transition logic uses a strict priority hierarchy implemented as if-then-else rules.Source: 3) Hierarchical Logic · p. 4

IV. STUDY SETUP

  • Sixteen participants performed needle insertion under visual-only baseline and visual-plus-haptic conditions in a within-subject teleoperated robotic setup.Condition order was randomized, target position changed each trial, and participants trained with the robotic controls and haptic cues before tasks.Source: §IV · pp. 4–5
  • The evaluation logged task completion time, average alignment angle, time spent in each surgical state, and the order of traversed states; significance used a Wilcoxon signed-rank test at α = 0.05.Participants completed a System Usability Scale questionnaire after each condition.Source: §IV · p. 5

V. RESULTS

Task completion time did not change significantly with haptic feedback. The supplied figures cover the study setup and the distribution of time spent in surgical states.

A. Task Efficiency

  • Haptic feedback did not produce a statistically significant change in task completion time.Source: A. Task Efficiency · p. 5
  • Fig. 7 presents the study setup.Source: Figure 7 · p. 5
  • Fig. 8 compares participants’ time spent in each surgical state across conditions.The box plot’s background colors correspond to the Intraocular Navigation and Retinal Approach states.Source: Figure 8 · p. 5

B. Safety and Error Reduction

Haptic feedback was associated with lower mean alignment error, while participants evaded the negative Retinal Contact state. The strategy analysis describes early alignment before retinal approach, alongside a shallow-insertion angle-estimation limitation.

B. Safety and Error Reduction

  • 7.9° ± 2.6° to 6.8° ± 2.3° (p = 0.044): mean alignment error was lower in the feedback condition.All participants successfully evaded the negative Retinal Contact state.Source: B. Safety and Error Reduction · p. 5, Table II · p. 5
  • Angle estimation is less robust at shallow insertion depths because less of the needle shaft is visible.Advancing the needle stabilizes detection, and minor angle readjustments may accumulate time in the Misaligned Retinal Approach state.Source: B. Safety and Error Reduction · p. 5

C. Behavioral Strategy Analysis

  • Visual-only participants often skipped initial alignment, approaching the retina with a near-zero median time in Aligned Intraocular Navigation.This led to corrective movements near sensitive tissue and potentially increased iatrogenic-injury risk.Source: C. Behavioral Strategy Analysis · p. 6
  • Visual + Haptic participants followed an “Align-then-Approach” strategy, correcting trajectory early before approaching the retina primarily while aligned.The sequence diagrams contrast this with visual-only participants’ later angle corrections.Source: C. Behavioral Strategy Analysis · p. 6, Figure 9 · p. 6

D. Usability and Cognitive Load

Participants gave the system a mean SUS score of 81.9, rated “Good,” with a significant increase of nearly 8%. The discussion describes modular integration of additional contexts and identifies study limitations and future work.

D. Usability and Cognitive Load

  • The system received a mean SUS score of 81.9, corresponding to a “Good” rating; its score increased significantly by nearly 8%, countering concerns that added haptic feedback might overwhelm users.Source: D. Usability and Cognitive Load · p. 6

VI. DISCUSSION AND CONCLUSION

  • The graph structure enables integration of additional surgical contexts, such as anatomical constraints or workflow phases, without re-engineering the control architecture.The current experiment focuses on a specific injection task.Source: §VI · p. 6
  • The authors identify binary alignment, sample size (N = 16), absent tissue deformation, and imaging noise as limitations or areas for future work.They plan to explore continuous guidance and validate on more realistic models and clinical OCT data.Source: §VI · p. 6
Loading 2609.07857v1…