Source-linked AI summary
From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations
Dongho Yee, Juahn Oh, Jinseok Lee, Jiyul Lee, Yechan Seo, Seong Jeong, Minsung Kim, Seonho Shim, Younghoon Noh, Hyuk Choi, Youngbin Kong, Kyu Eun Lee, Hyoun-Joong Kong
TL;DR
Hand-held laparoscopic instrument motion is typically lost after surgery, limiting the demonstrations available for surgical imitation learning. This paper captures instrument state on the surgeon’s own tools, aligns the sensor streams, and trains and selects a bimanual policy without robot-collected demonstrations. Retrained on in-vivo demonstrations, the policy completed appendectomy in three of four live rabbits with the surgeon selecting the surgical phase.
Problem
Hand-held laparoscopic surgery retains endoscope video but loses instrument motion, while surgical imitation-learning demonstrations had relied on robot-collected data in small, platform-tied corpora.
Source: §I · p. 1Method
Results
The policy retrained on 849 in-vivo demonstrations completed appendectomy in three of four live rabbits under shared autonomy.
Source: §V · p. 8Takeaways & Limitations
Demonstrations from surgeons’ own hand-held instruments were sufficient to train, select, and deploy a bimanual policy that completed appendectomies with electrosurgery in live rabbits.
Source: §VI · p. 8The surgeon selected the phase in vivo, and the study used demonstrations from four rabbits and one deployment in each of four others on a single day.
Source: §V · p. 8
Abstract
from arXiv · showhide
Most minimally invasive surgery is still performed with hand-held laparoscopic instruments, and the surgeon's instrument kinematics are lost when the operation ends; only the endoscope video is kept. This paper presents an end-to-end pipeline that captures this motion in the operating room and uses it to train a surgical robot policy, validated on live animals. We introduce a surgical instrument-state logger that mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker. A data pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before forming observation-action pairs. On these demonstrations we train a diffusion policy with a fine-tuned DINOv3 backbone, selecting its design by closed-loop rollouts in a physics simulator reconstructed from depth maps of an ex-vivo rabbit appendix. The policy is then retrained on 849 in-vivo demonstrations from four live rabbits and deployed on four additional live rabbits with electrosurgery armed. With the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals. The results show that demonstrations recorded from a surgeon's own instruments are sufficient to train, select and deploy a bimanual surgical policy in vivo. The robot serves only as the timing reference for sensor calibration and as the executor, and collects no demonstrations. Both demonstration corpora are released to support future surgical robot learning research.
I. INTRODUCTION
Hand-held laparoscopic motion is lost after surgery, leaving a gap in demonstrations for imitation learning. The paper captures that motion on surgeons’ instruments and develops a pipeline for training and deploying a bimanual surgical policy.
I. INTRODUCTION
- Hand-held surgery loses instrument motion when the operation ends, although imitation learning needs demonstrations of that motion.Only endoscope video is kept.Source: §I · p. 1
- Surgical policies had relied on robot-collected demonstrations in small, platform-tied corpora, unlike the discarded signal available from hand-held laparoscopy.The paper asks whether that signal can be captured in the operating room and used to train a surgical robot policy.Source: §I · p. 1
- The proposed pipeline uses an instrument-mounted logger and per-channel latency matching, with the robot serving as a timing reference and executor rather than a demonstration collector.The logger recovers instrument state using an IMU, time-of-flight sensor, and Hall sensor under the trocar’s remote-center-of-motion constraint.Source: §I · p. 2
II. RELATED WORK
- Unlike prior learned surgical policies trained from teleoperation or simulation, this work uses surgeons’ hand-held instruments for procedural steps through tissue division with energy.The authors also distinguish their task and general-purpose-arm platform from prior work.Source: §II · p. 2
- The logger extends hand-held data collection beyond video-estimated kinematics by capturing synchronized, jaw-inclusive demonstrations; the simulator is used to compare configurations before testing the selected one on a real rig.The paper reports that markerless fits can have millimeter-scale errors and that occlusion further increases error; it notes sim-to-real transfer for deformable tissue remains unreliable.Source: §II · pp. 2–3
A. The surgical instrument-state logger
The shaft-mounted logger captures hand-held instrument state during real operations without an external tracker. It combines inertial, time-of-flight, and Hall sensing with timestamped, synchronized streams for policy training.
A. The surgical instrument-state logger
- The logger measures shaft roll and depression with a gravity-referenced IMU, insertion depth with a ToF sensor, and jaw opening with a Hall sensor.The trocar’s remote-center-of-motion constraint leaves four instrument degrees of freedom; depth is computed from shaft length minus board-to-trocar distance.Source: A. The surgical instrument-state logger · p. 3
- Against FR3 tip pose during free motion, latency-matched channels achieve 0.42° and 0.17° RMS attitude errors, 1.36 mm RMS depth error, and 1.38 mm RMS tip displacement over a 1.6 s action chunk.Depth error was 5.6 mm before fixed gain and offset calibration; tip displacement is 2.89 mm at the 95th percentile.Source: A. The surgical instrument-state logger · p. 3
- The operating-room logger uses a compact enclosure, battery, and BLE streaming at 50–200 Hz, with a measured battery life of roughly 20 hours.It uses a faster ToF sensor and moving average than the ex-vivo logger, and has no cable leaving the instrument.Source: A. The surgical instrument-state logger · p. 3
- Recorders timestamp samples at the earliest host receive point, then merge streams offline by absolute time; the board clock is mapped to host time using lower-envelope regression.The regression accounts for one-sided arrival jitter.Source: A. The surgical instrument-state logger · p. 3
B. Per-channel latency matching
Sensor channels have different delays, so the paper measures and corrects each channel separately. This alignment reduces dynamic depth error and improves lift-stage performance in closed-loop evaluation.
B. Per-channel latency matching
- Each action chunk is anchored to a time point, while observations include the endoscope image and per-arm shaft state after latency matching.The illustrated chunk contains each arm’s 16-step tip path, roll and jaw channels, and an energy timeline.Source: Figure 5 · p. 4
- A channel response is modeled as transport delay plus first-order lag; a time shift can correct near-zero lag, but cannot recover motion lost to large lag.The FR3 tip pose provided ground truth while the instrument was driven through a phantom trocar along swept-sine trajectories.Source: B. Per-channel latency matching · pp. 3–4
- Rewinding each channel by its measured latency reduced dynamic depth error from 2.69 to 1.36 mm (−49.5%) and shaft-roll error by 28%.The depression channel changed by only +2% and served as a control.Source: B. Per-channel latency matching · p. 4
- Ex-vivo insertion depth had a 349 ms low-frequency delay, 93% of it from first-order lag, while attitude delays were 24–26 ms and jaw delay was 79 ms.The 325 ms depth–attitude skew equals 3.5 policy steps at 10 Hz, so no common time shift can remove it.Source: B. Per-channel latency matching · p. 4
C. From logs to training pairs
Instrument channels are converted into relative action chunks using gravity- and trocar-referenced observations, then modeled by a diffusion policy with phase conditioning and latency-aware execution. Configuration selection uses the strongest full-approach rate without a trocar violation in the depth-map simulator.
C. From logs to training pairs
- Observations combine endoscope frames with per-arm roll, depression, and insertion depth; actions are 16-step chunks of tip-frame deltas, jaw aperture, and, in vivo, electrosurgery state.Jaw aperture is excluded from policy input to avoid learning to keep closing from an already-closed jaw.Source: C. From logs to training pairs · p. 4
- Anchor-relative action labels are invariant to heading zero and trocar location, so the executor can unwind them using the robot’s measured tip frame.Heading drift is negligible within a 1.6-second chunk, supporting relative motion despite unreliable absolute heading.Source: C. From logs to training pairs · p. 4
D. Policy
- The policy uses a 1-D U-Net diffusion model with DDIM sampling over 16-step action chunks and two observations spanning 0.2 seconds.The selected configuration also uses shared DINOv3 features with attention pooling and only the final transformer block trainable.Source: D. Policy · p. 4
- FiLM conditions the policy on surgical phase and two timing channels, supplied by a video predictor at runtime or the simulator state in simulation.When the surgeon sets the phase by keypress, the timing channels remain the predictor’s video estimates.Source: D. Policy · p. 4
- The executor advances each channel for observation, execution, and inference latency, commits three actions before replanning, and gates electrosurgery through five safety checks.In vivo, the configuration selected by the highest full-approach rate without a trocar violation is run on the real two-FR3 rig and retrained for in vivo.Source: D. Policy · p. 5, Figure 6 · p. 5
E. Simulator and selection protocol
The simulator reconstructs phantom organs from depth data and evaluates candidate policies through fixed closed-loop episodes. Scoring combines procedural progress with trocar violations.
E. Simulator and selection protocol
- Candidate policies run in an Isaac Sim reconstruction with six FEM organs meshed from ex-vivo depth recordings, nominal soft-tissue stiffness, and two FR3s through 6 mm trocars.The tissue stiffness is not fitted to the specimen.Source: E. Simulator and selection protocol · p. 5
- Each configuration is scored over 16 episodes of 15 seconds using stage counts from grasp through cut and a hard failure threshold of more than 10 mm RCM error.RCM error measures the distance between the instrument shaft and trocar point; exceeding the threshold ends an episode as a violation.Source: E. Simulator and selection protocol · p. 5
- Stage-count uncertainty is reported with Wilson intervals, and configurations are compared using Fisher’s exact test.Source: E. Simulator and selection protocol · p. 5
A. Demonstration corpora
The paper records ex-vivo and in-vivo appendectomy demonstrations with instrument-mounted loggers, then selects a policy configuration using simulator rollouts. The selected design balances full-approach performance against trocar safety, while several ablations show that offline error alone is an unreliable selector.
A. Demonstration corpora
- 533 ex-vivo episodes from two specimens and 849 filtered in-vivo episodes from four live rabbits formed the training corpora, recorded using the instrument-state logger.Ex-vivo demonstrations came from the second session day; in vivo, 1,003 raw episodes were filtered, and four additional rabbits were used for deployment.Source: A. Demonstration corpora · p. 5
B. Configuration selection on the ex-vivo corpus
- 81% full-approach rate for single-block DINOv3 exceeded two-block DINOv3 at 50% and frozen DINOv3 at 44%, with 19% versus 6% trocar violations (p = 0.6).The frozen backbone’s low violation rate came with no jaw closure on tissue (grasp depth 0.0 mm).Source: B. Configuration selection on the… · p. 6
- 88% to 6% trocar violations and 12% to 69% lift resulted from replacing ResNet-18 encoders with frozen DINOv3 (Fisher p = 7 × 10−6; p = 0.003).Source: B. Configuration selection on the… · p. 6
- 19% versus 19% trocar violations but 19% versus 81% successful lifts followed removal of latency matching, despite better validation MAE (0.138 vs. 0.148; p = 0.001).The depth–attitude skew was 325 ms, breaking coordinated insertion needed for lifting.Source: B. Configuration selection on the… · p. 6
- Trocar violations rose from 6% to 69% as zero to four transformer blocks were unfrozen, while only one unfrozen block raised full approach above frozen DINOv3 (81% vs. 44%).The four-block versus frozen safety difference had p = 6 × 10−4; the full-approach comparison had p = 0.07.Source: B. Configuration selection on the… · p. 6
- Validation MAE improved from 0.152 to 0.142 across the unfreezing sweep as violations rose from 6% to 69%, and ResNet-18’s 0.145 MAE beat frozen DINOv3’s 0.152.Capacity, surgical-domain pre-training, and attention pooling also showed the reported inversion between offline error and safety.Source: B. Configuration selection on the… · p. 6
- Video-trained phase prediction reduced full approach from 81% to 25%, while committing one action per chunk instead of three reduced it from 81% to 31%.On the real rig, the policy had zero trocar violations and 0.23 mm peak RCM error; the phase predictor saturated at the final phase within a median of 10.6 seconds, so the surgeon set phase in vivo.Source: B. Configuration selection on the… · p. 6
C. In-vivo training
The in-vivo policy is trained from scratch on filtered live-tissue demonstrations, with checkpoint acceptance based on pre-registered event-level gates. Ex-vivo initialization helped offline error only when in-vivo data were scarce.
C. In-vivo training
- 849 in-vivo demonstrations survived filtering from 1,003 raw episodes, and the policy was trained from scratch with the ex-vivo configuration and optimizer-step budget unchanged.Episodes were removed for sensor-stream quality, non-procedure content, missing phase, or early termination.Source: C. In-vivo training · p. 6
- Checkpoint acceptance used pre-registered event-level gates on the last 5% of episodes, requiring jaw-event detection, bounded energy errors, jaw timing within one anchor, and closed-loop gain above 0.8.The reported gains were 0.86/0.82 for left/right arms; energy false-fire and missed-fire rates were 0.019 and 0.032.Source: C. In-vivo training · p. 6
- Ex-vivo pre-training improved offline MAE only at 25% of the in-vivo corpus (1.221 vs. 1.413), not at 50% or 100%.Both random initialization and ex-vivo initialization passed the same gates in all six runs; the paper concludes live-tissue policies must be trained on live-tissue demonstrations.Source: C. In-vivo training · p. 6
D. Deployment on live rabbits
Under shared autonomy, the policy completed appendectomies in three of four live rabbits, with one rollout per animal. Policy-controlled cautery samples met the reported jaw-closure condition, and peak RCM error remained below the abort threshold.
D. Deployment on live rabbits
- Three of four live-rabbit appendectomies were completed under shared autonomy, with one policy rollout in each animal.In the remaining animal, the joint-limit governor aborted at hand-over before the policy acted; the procedure was completed by hand.Source: D. Deployment on live rabbits · pp. 6–7, Table III · p. 7
- In the fourth animal, the joint-limit governor stopped the arm before any policy command reached tissue, and the procedure was completed by hand.Source: D. Deployment on live rabbits · p. 7
- All four animals survived, with no gross abnormality detected during one week of postoperative observation.Source: D. Deployment on live rabbits · p. 7
- Peak policy-controlled RCM error was 0.84 mm against a 5 mm abort threshold, and the cutting jaw was closed at every cautery sample.All 108 policy-controlled cautery samples across the three rollouts had jaw aperture at most 0.7%.Source: D. Deployment on live rabbits · p. 7
V. DISCUSSION
The pipeline produced a policy from instrument-mounted demonstrations and deployed it successfully in three of four live rabbits. Discussion results also identify latency matching and closed-loop configuration selection as consequential parts of the pipeline, while noting phase-control and study-size limitations.
V. DISCUSSION
- Retraining on 849 in-vivo demonstrations yielded appendectomy completion in three of four live rabbits, with the cutting jaw closed at every cautery sample and peak RCM error of 0.84 mm.Source: §V · p. 8
- Latency matching halved dynamic depth error against FR3 ground truth from 2.69 to 1.36 mm and raised closed-loop lift success from 19% to 81%.The logger recorded 533 ex-vivo episodes and 849 in-vivo episodes without an external camera or tracker.Source: §V · pp. 7–8
- Closed-loop rollouts of 31 candidate configurations selected the policy design; unfreezing DINOv3’s last block gave the highest full-approach rate at 81%.A frozen DINOv3 reduced trocar violations from 88% to 6% relative to ResNet-18, while validation error improved as safety degraded.Source: §V · p. 8
- In vivo, the surgeon supplied the surgical phase because the vision predictor agreed with the surgeon on only 24.6% of policy-control steps; the study used four rabbits for demonstrations and four others for one-day deployments.Source: §V · p. 8
VI. CONCLUSION
Demonstrations recorded from surgeons’ hand-held instruments were sufficient to train, select, and deploy a bimanual policy that completed appendectomies with electrosurgery in three of four live rabbits.
VI. CONCLUSION
- Hand-held instrument demonstrations supported training, policy selection, and in-vivo deployment of a bimanual appendectomy policy.Source: §VI · p. 8
- The policy completed appendectomies with electrosurgery in three of four live rabbits.Source: §VI · p. 8
- The demonstrated scope is bimanual appendectomy execution in live rabbits, rather than a broader surgical task or population claim.Source: §VI · p. 8