Source-linked AI summary
EndoNav: Semantic-to-Geometric Grounding for Language-Guided Robotic Endoscopic Examination
Jecia Z. Y. Mao, Hisashi Ishida, Kathryn Jung, Masaru Ishii, Russell H. Taylor, Manish Sahu
TL;DR
EndoNav addresses the gap between high-level anatomical surgeon commands and the precise geometric actions required for endoscopic visualization. It uses an anatomy-grounded viewpoint agent to generate structured objectives and target viewpoints, followed by geometry-constrained motion planning and robot control. In cadaveric comparison, visualization agreement with EndoNav was comparable to inter-surgeon agreement, while substantial portions of surgeon-observed surfaces were recovered.
Problem
Existing systems do not resolve how clinical language specifying an anatomical view becomes a precise endoscope position, orientation, and trajectory within patient-specific anatomy.
Method
EndoNav separates language reasoning from physical execution by converting surgeon commands into structured visualization objectives, target viewpoints, and geometry-constrained endoscope trajectories.
Results
87.04% and 84.37% mean visualization IoUs relative to two surgeon examinations approached the 87.44% inter-surgeon IoU, with 92.91% and 93.20% surgeon-surface recalls.
Takeaways & Limitations
The results demonstrate feasibility of grounding high-level anatomical commands into patient-specific geometric objectives and anatomically constrained robotic visualization behaviors.
Takeaways & Limitations
The study is limited by simulation-based robot execution, few anatomical specimens, incomplete modeling of physical-navigation uncertainties, and planning that requires tens of seconds per examination trajectory.
Abstract
from arXiv · showhide
Minimally invasive procedures performed within confined anatomical spaces depend on continuous endoscopic visualization. Current robotic endoscope systems can stabilize or reposition an endoscope, but they do not possess relevant context to provide effective visualization assistance. We present EndoNav, an anatomy-grounded natural-language framework that translates high-level surgeon commands into autonomous endoscopic visualization behaviors within patient-specific sinonasal anatomy. Spoken surgeon commands are transcribed and interpreted by an endoscopic viewpoint agent conditioned on a patient-specific anatomical scene representation. Rather than generating robot motion directly, the viewpoint agent generates structured visualization objectives that are converted into target viewpoints and inspection trajectories, which are then executed through geometry-constrained endoscope motion planning and joint-space control. We evaluate EndoNav using a structured three-pass sinus examination across three CT-derived anatomical models. For one cadaveric specimen, autonomous visualization is compared with sinus examinations performed by two resident surgeons. EndoNav achieved mean visualization IoUs of 87.04% and 84.37% relative to the two surgeon examinations, compared with an inter-surgeon IoU of 87.44%, while recovering 92.91% and 93.20% of surgeon-observed anatomical surfaces, respectively. These results demonstrate the feasibility of grounding high-level anatomical commands into patient-specific geometric objectives and translating them into anatomically constrained robotic visualization behaviors.
I. INTRODUCTION
EndoNav addresses the unresolved problem of translating anatomical-language commands into patient-specific endoscope viewpoints and executable motion. It separates semantic reasoning from geometry-constrained planning to support autonomous sinus examination.
- Endoscopic manipulation in confined, patient-specific corridors requires coordinating anatomical knowledge, visualization goals, and endoscope control, burdening clinicians.
- Existing robotic systems can generate motion after receiving geometric objectives, but do not translate desired anatomical views into targets, trajectories, or low-level commands.
- Clinical language such as “inspect the Eustachian tube” leaves the required position, orientation, and trajectory unspecified, creating an intermediate spatial-reasoning problem.
- The framework grounds surgeon commands to anatomical targets and target viewpoints, then generates anatomy-aware trajectories and robot-executable motion under nostril constraints.
- EndoNav separates probabilistic language reasoning from geometry-constrained execution by producing structured visualization objectives grounded in patient-specific anatomy.
- EndoNav was evaluated through structured three-pass examinations across three CT-derived anatomical models and compared with two resident surgeons in a cadaver study.
A. System Overview
EndoNav grounds surgeon commands in a patient-specific anatomical scene representation, then converts visualization objectives into feasible viewpoints and anatomically constrained endoscope trajectories.
- A. System Overview: The viewpoint agent maps surgeon language and scene descriptions to anatomy-relative target regions or requested viewing directions.Qualitative directional descriptions are mapped to normalized intervals and scaled by patient-specific cavity geometry to define a bounded 3-D tip search region.
- A. System Overview: Patient-specific sinonasal anatomy is represented in compact cavity-relative geometric forms to support semantic scene description and viewpoint planning.Each cavity uses clinically meaningful entry–nasopharynx, lateral–septal, and roof–floor directions; structures are represented relative to their corresponding cavity frames.
- A. System Overview: Feasible candidate viewpoints are sampled and scored using viewing-direction alignment and target framing.The framing term favors smaller target footprints, providing a wider contextual view of surrounding anatomy.
- A. System Overview: Nostril geometry constrains shaft orientations at each waypoint, while globally selecting orientations preserves feasible passage through the nostril during execution.A backward-facing virtual depth camera generates feasible shaft-orientation candidates from the surrounding nasal boundary; rotational continuity resolves the remaining roll degree of freedom.
- A. System Overview: A signed-distance-field navigation graph enables collision-free tip trajectories that are shortcut-smoothed and resampled into waypoints.The planner retains free-space voxels satisfying the required anatomical clearance before computing a path between the current tip and target viewpoint.
A. Experimental Setup
The ex vivo setup aligned a manually operated rigid endoscope, tracked cadaveric head, and CT-derived anatomy in a common frame for comparison with EndoNav.
- A rigid 0° endoscope was manually manipulated by each resident on an ex vivo cadaveric head.
- Optical tracking markers and BB fiducials registered the endoscope, cadaveric head, and CT-derived anatomical frames.
- Camera calibration and hand–eye calibration recovered the physical endoscope pose and imaging parameters.
- Recovered poses and calibrated parameters instantiated a virtual endoscope in AMBF aligned with the recorded physical examination.
B. Validation Procedure
Validation used a structured three-pass nasal examination and registered observed point clouds to segmented anatomy for direct coverage comparison.
- Three-pass examination: Each nostril examination comprised overall nasal-cavity inspection, middle-turbinate and middle-meatus inspection, and posterior-cavity inspection.
- Three-pass examination: The posterior pass included the nasopharyngeal and Eustachian-tube regions.
- Surface comparison: Simulator-generated observation point clouds were registered to CT anatomy using fiducials and refined with iterative closest point alignment.
- Surface comparison: Registered point clouds were associated with segmented surfaces to produce structure-specific visualization maps.
- Metrics: Metrics were computed independently for each structure and nasal side, with inter-surgeon agreement using the same formulation.
C. Results
EndoNav produced visualization patterns and anatomical coverage comparable to resident examinations across the evaluated specimen and generated trajectories across three CT-derived anatomies.
- Agreement with surgeon coverage: 87.04% and 84.37% mean IoUs were achieved relative to residents R1 and R2, versus 87.44% inter-surgeon IoU.
- Agreement with surgeon coverage: 92.91% and 93.20% macro mean recalls were achieved relative to R1 and R2, respectively.
- Agreement with surgeon coverage: The largest discrepancy was a 71.01% IoU for the left middle turbinate relative to R2.
- Patient-specific anatomies: EndoNav generated examination trajectories for both nasal passages across all three CT-derived anatomical specimens.
- Patient-specific anatomies: Mean coverage ranged from 60.95% for the nasopharynx to 71.78% for the nasal cavity across evaluated structures.
- Examination and planning time: The experimental specimen required 298 seconds and 222 seconds for autonomous left- and right-passage examinations, respectively.
- Examination and planning time: Timing comparison with manual examinations was limited by execution differences and an incomplete third left-side pass for R1.
- Examination and planning time: Mean planning times were 17, 14, and 23 seconds for the three passes, totaling 55 seconds per nasal-side examination.
IV. DISCUSSION
EndoNav achieved visualization agreement comparable to inter-surgeon variability in one experimental specimen, while evaluation across three models showed anatomy-dependent coverage and simulation-based limitations.
- Visualization agreement: 87.04% and 84.37% macro mean IoUs matched EndoNav against the two surgeon examinations, compared with 87.44% inter-surgeon IoU.The comparison does not establish equivalence to surgeon performance; it indicates comparable autonomous–surgeon agreement and inter-surgeon variability.
- Visualization agreement: 92.91% and 93.20% surgeon-surface recalls indicate that EndoNav reproduced substantial portions of anatomy visualized by the surgeons.
- Evaluation measure: Surface-based comparison evaluates whether different navigation behaviors achieve similar visualization objectives despite valid variation in poses, trajectories, and manual examination coverage.
- Cross-specimen evaluation: Across three CT-derived specimens, the same framework generated bilateral examination trajectories without anatomy-specific modification, although coverage remained structure- and anatomy-dependent.
- Cross-specimen evaluation: 67.13% mean Eustachian-tube coverage showed the greatest inter-specimen variability, partly because the structure’s small size makes coverage sensitive to segmentation discrepancies.
- Limitations: Simulation-based execution, few anatomical specimens, unmodeled physical-navigation uncertainties, and trajectory planning requiring tens of seconds limit the current study.
V. CONCLUSION
EndoNav connects anatomy-level surgeon intent to autonomous endoscopic visualization through patient-specific semantic-to-geometric grounding. The results support the feasibility of translating clinically meaningful communication into anatomically grounded endoscope motion.
- V. CONCLUSION: EndoNav translates high-level anatomical commands into structured visualization objectives, target viewpoints, and geometry-constrained endoscope motion plans.
- V. CONCLUSION: Across three anatomical models, EndoNav generated consistent examinations across patient-specific geometries while revealing structure- and anatomy-dependent variation in achievable coverage.
- V. CONCLUSION: The findings demonstrate the feasibility of bridging semantic surgeon intent and embodied endoscope navigation through patient-specific anatomical grounding.