Source-linked AI summary
SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learning
Ji Woong Kim, Juo-Tung Chen, Pascal Hansen, Lucy X. Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M. White, De Ru Tsai, Richard Cha, Jeffrey Jopling, Chelsea Finn, Axel Krieger
TL;DR
Autonomous surgery needs dexterous, long-horizon manipulation that generalizes across variable tissue, beyond the simpler controlled tasks addressed by much prior work. SRT-H uses high-level language-based planning alongside low-level trajectory execution and corrective instructions. It achieved 100% operation success across eight unseen ex vivo gallbladders without human intervention, while remaining bounded by out-of-distribution failures and the need for surgeon oversight.
Problem
Autonomous surgery still lacks demonstrated solutions for dexterous, long-horizon manipulation across variable tissues and complex surgical scenes.
Method
SRT-H hierarchically combines a high-level policy generating task and corrective language instructions with a low-level policy executing robot trajectories.
Results
100% operation success was achieved across eight unseen ex vivo gallbladders, with all 17 required tasks completed autonomously without human intervention.
Takeaways & Limitations
SRT-H demonstrates step-level autonomy for cholecystectomy clipping and cutting while generalizing across unseen ex vivo tissues and self-correcting errors in real time.
Takeaways & Limitations
The system is not failure-proof in out-of-distribution scenarios, so a surgeon should always oversee its operation.
Abstract
from arXiv · showhide
Research on autonomous surgery has largely focused on simple task automation in controlled environments. However, real-world surgical applications demand dexterous manipulation over extended durations and generalization to the inherent variability of human tissue. These challenges remain difficult to address using existing logic-based or conventional end-to-end learning approaches. To address this gap, we propose a hierarchical framework for performing dexterous, long-horizon surgical steps. Our approach utilizes a high-level policy for task planning and a low-level policy for generating robot trajectories. The high-level planner plans in language space, generating task-level or corrective instructions that guide the robot through the long-horizon steps and correct for the low-level policy's errors. We validate our framework through ex vivo experiments on cholecystectomy, a commonly-practiced minimally invasive procedure, and conduct ablation studies to evaluate key components of the system. Our method achieves a 100\% success rate across eight unseen ex vivo gallbladders, operating fully autonomously without human intervention. This work demonstrates step-level autonomy in a surgical procedure, marking a milestone toward clinical deployment of autonomous surgical systems.
1. Introduction
SRT-H addresses dexterous, long-horizon surgical manipulation by combining language-based task planning with low-level trajectory generation. In ex vivo cholecystectomy experiments, it autonomously completed the clipping-and-cutting procedure across unseen gallbladders.
- Motivation: Real-world surgical autonomy must handle variable anatomy, changing scenes, occlusions, partial observability, and precise manipulation safely.These challenges complicate both perception and motion planning in surgical environments.
- Prior Work: Prior autonomous demonstrations primarily addressed simpler navigation steps or application-specific hand-crafted strategies, limiting generalization to complex manipulation.Earlier work included simulation, tabletop manipulation, needle steering, and anastomosis, but did not broadly address long-horizon surgical manipulation.
- Contribution: SRT-H targets contact-rich manipulation, variable ex vivo tissues, and surgical steps lasting several minutes with persistent coordination and decision-making.The framework focuses on grabbing, clipping, and cutting across anatomically and visually diverse porcine gallbladders.
- Method: SRT-H combines a high-level policy that issues task or corrective language instructions with a low-level policy that executes robot trajectories.The hierarchy decomposes complex procedures into shorter tasks and allows high-level correction of low-level mistakes.
- Results: 100% operation success was achieved across eight unseen ex vivo gallbladders, with all 17 required tasks completed autonomously without human intervention.The system was trained on 16,000 trajectories and generalized across anatomies while self-correcting mistakes mid-procedure.
RESULTS
SRT-H completed autonomous cholecystectomy steps across unseen ex-vivo gallbladders, while ablations showed the importance of hierarchical recovery, wrist-camera observations, language corrections, and sufficient training data. Hardware failures were unrelated to the framework, and comparison with a surgeon remained preliminary.
- System deployment: The system used wrist cameras mounted near instrument tips, and its cholecystectomy workflow required operator pauses for clip reloading and tool changes.The hardware configuration provided both a global endoscope view and close-up instrument-tissue views.
- Core experiment results: SRT-H completed all procedures successfully on eight unseen gallbladders without human intervention, averaging 317 seconds and approximately six self-corrections per procedure.The reported duration excludes operator time for clip reloading and tool changes.
- Experimental caveat: A scissors failure and three dVRK reinitializations occurred during experiments, but the authors attribute these non-safety-critical events to developmental hardware rather than SRT-H.The first dVRK Si system was still undergoing development and was not yet perfected.
- High-level policy studies: The high-level policy achieved approximately 97% accuracy and F1 for task-instruction prediction, while observation-history removal caused declines exceeding 10%.The policy achieved approximately 95% recovery-mode accuracy and around 70% corrective-instruction accuracy across 18 motion classes.
- Comparison with expert surgeon: Compared with an expert surgeon, SRT-H produced shorter trajectories and lower mean jerk, but the surgeon completed all evaluated tasks faster.The authors describe this as an initial comparison because too few gallbladders supported a more in-depth analysis.
DISCUSSION
SRT-H demonstrates step-level autonomy for cholecystectomy across unseen ex-vivo tissues, while remaining bounded by its evaluation scope and the need for surgeon oversight in out-of-distribution scenarios.
- 100% operation success rate was demonstrated across eight gallbladders, with autonomous execution and real-time self-correction.The framework automated the clipping-and-cutting procedure without human intervention.
- Existing higher-autonomy systems operate in controlled conditions, leaving a considerable gap toward full autonomy in dynamic, soft-tissue environments.CyberKnife and Veebot reduce automation complexity through rigid, stable tissues or structured procedures.
- SRT-H is categorized as Level of Autonomy IV because it executes reliably and autonomously while self-correcting mistakes, but still requires surgeon oversight.The system is not failure-proof in out-of-distribution scenarios.
- The framework operates at LoEC IV and LoTC IV, involving realistic soft tissue and advanced spatial tasks without topological motion or clinical and anatomical knowledge.Topological motion is identified as the further requirement for LoEC V, while clinical and anatomical knowledge is required for LoTC V.
- Translating SRT-H from ex-vivo experiments to in-vivo clinical environments requires addressing bleeding, tissue motion, surgical access, and wrist-camera constraints.The authors state that further studies are required to confirm adaptation to motion and blood.
- Future work includes broader procedural coverage, uncertainty handling, safety switching, and simulation-based refinement before real-world execution.Suggested tools include conservative Q-learning, conformal prediction, surgeon or teleoperation oversight, and simulation.
MATERIALS AND METHODS
The study collects ex vivo gallbladder demonstrations and trains SRT-H as a hierarchical, language-conditioned imitation-learning system. Its high-level policy selects task or corrective instructions, while the low-level policy converts them into robot trajectories.
- Data collection: Training data combines demonstrations from 34 ex vivo porcine gallbladders collected by two trained non-clinical demonstrators.Dataset D1 covered 31 gallbladders and D2 covered three additional gallbladders.
- Task design: The clipping-and-cutting portion of cholecystectomy is decomposed into 17 tasks covering gallbladder grabbing, six clips, and two cuts with retraction motions.The task count is 1 + 12 + 4 = 17, with clipping and cutting each including a primary motion and retraction.
- Experimental protocol: Multiple trials are enabled by disabling clip latching and omitting final scissor closure, with cutting motion extended during post-processing.This procedure supports repeated clipping and cutting motions without permanently altering the duct or artery.
- Data augmentation: Recovery demonstrations and HL DAgger data add examples of recovering from suboptimal states and learning from expert verbal corrections during policy rollouts.Only language predictions are corrected in the HL DAgger dataset.
- High-level policy: SRT-H uses a high-level transformer policy that predicts a task instruction, correction flag, and corrective motion instruction from endoscope-image history.The task instruction receives weight 0.4, while the correction flag and corrective instruction each receive weight 0.3.
- Hierarchical control: Every 3 s, the correction flag selects either the task instruction or corrective motion for the low-level language-conditioned policy.The low-level policy receives endoscope and wrist-camera images and predicts sequences of robot actions.
Supplementary Materials
The supplementary materials include the PDF’s supplementary methods, tables, and figures.
- Supplementary contents: The supplementary PDF contains Methods Tables S1–S2 and Figures S1–S7.These materials are listed as part of the supplementary content.
Supplementary materials and methods
Supplementary materials describe high-level policy training, corrective language, GPT-4o task planning, and supplementary figures and performance tables. They report that GPT-4o made task-transition errors in the surgical setup.
- High-level policy training configuration: The high-level policy has approximately 45 million parameters, trains for 500 epochs on one RTX 4090, and has approximately 25 ms inference time.Training used a 90–10 training-validation split and selected weights based on validation performance.
- Corrective language instructions: The high-level policy can generate 18 corrective instructions for gripper operations and cardinal arm movements.The listed instruction set includes separate left-arm, right-arm, and bilateral gripper actions.
- General-purpose VLM as high-level policy: GPT-4o was evaluated as a general-purpose high-level surgical task planner using current endoscope images and the available task instructions.The evaluation used the same dVRK setup as the high-level policy.
- General-purpose VLM as high-level policy: GPT-4o initially skipped “Grabbing the Gallbladder” and triggered clipping prematurely before the gallbladder had been grabbed.These errors were attributed to difficulties recognizing task completion and transitions in the surgical context.
- Supplementary figures and tables: Supplementary figures include the GPT-4o prompt, early or incorrect responses, gallbladder appearance and anatomy examples, and high-level policy corrections over eight experiments.The supplementary collection also includes RandAugment transformations and high-level policy accuracy and F1 metrics.
- Supplementary figures and tables: Table S1 documents image transformations used in RandAugment, while Table S2 reports high-level policy accuracy and F1 scores for its prediction targets.The targets are task instructions, correction flag, and corrective instructions.