PaperlayerView exact revision on arXiv

Source-linked AI summary

Recording Hand-Held Laparoscopic Instrument Motion in the Operating Room: Magnetometer-Free Fusion of Inertial, Range and Visual Sensing

Jiyul Lee, Dongho Yee, Juahn Oh, Jinseok Lee, Yechan Seo, Seong Jeong, Minsung Kim, Seonho Shim, Younghoon Noh, Hyuk Choi, Youngbin Kong, Hyoun-Joong Kong

arXiv:2609.25577v1cs.RO

TL;DR

Hand-held laparoscopic motion is not retained, and video-based pose estimates and magnetometer heading have limitations in this setting. The paper presents a clip-on logger that fuses inertial, range, and markerless visual sensing under the trocar constraint. It reports millimeter-scale tracking errors against a robot and concludes the approach could record trajectories while displaying depth and attitude during surgery.

  • Problem

    Hand-held laparoscopic instruments leave no motion record, while monocular video pose estimates remain millimeters to centimeters off.

    Source: §I · p. 1

  • Method

    A clip-on logger fuses a six-axis IMU and ToF rangefinder with markerless camera sensing under the trocar RCM, using shaft-silhouette vision instead of magnetometer heading.

    Source: §I · p. 1

  • Results

    1.22 mm and 3.04 mm RMS tip errors were reported for programmed and teleoperated continuous motion, respectively, alongside 1.21 mm RMS displacement and 0.341° RMS relative-rotation errors; full fusion beat every sensor subset.

    Source: D. Translation Accuracy · p. 5, TELEOPERATED CONTINUOUS MOTION · p. 6, E. Rotation Accuracy · p. 6

  • Takeaways & Limitations

    The approach could record metric tip trajectories during routine hand-held laparoscopy and display insertion depth and attitude live.

    Source: §VI · p. 8, §V · p. 8

    Validation used one device on a phantom with a fixed ceiling camera, teleoperated free trajectories, and monocular reconstruction near 140 mm insertion.

    Source: §V · p. 8

Abstract

from arXiv · show

Most minimally invasive procedures are still performed with hand-held laparoscopic instruments, yet only the endoscopic video is retained; the instrument motion that expresses surgical skill, and that could support skill assessment and robot learning, is lost. Pose from video alone remains millimeters to centimeters off, and an instrument-mounted inertial measurement unit (IMU) cannot rely on its magnetometer, whose field changed with tool pose and between sessions in our measurements. We present a surgical instrument-state logger that clips onto a conventional instrument without modifying the part that enters the patient and fuses a six-axis IMU and a time-of-flight (ToF) rangefinder with a markerless camera in an error-state Kalman filter under the remote center of motion (RCM) of the trocar. Heading comes from the shaft silhouette, segmented by a U-Net, in place of the magnetometer: the rotation-angle error is 0.200°, against 3.58° from the accelerometer and magnetometer alone. Against a Franka Research 3 manipulator, and without alignment to it, the displacement error over 300 translation trials was 1.21mm RMS and the relative-rotation error over 180 rotation trials 0.34° RMS. On continuous trajectories, tracked and displayed in real time, the absolute tip error was 1.22mm (programmed) and 3.04mm (teleoperated) after post-hoc tuning of three filter parameters, and the full fusion beat every sensor subset. Because the estimator uses no magnetic measurement, its accuracy does not rely on an undisturbed field. The same clip-on device could thus record metric tip trajectories during routine hand-held laparoscopy, while displaying the insertion depth and attitude that are hidden once the instrument is inside the patient.

I. INTRODUCTION

Hand-held laparoscopic instruments produce no metric motion record, while video-based pose estimates remain inaccurate and instrument-mounted magnetometers are unreliable around operating equipment. The paper presents a clip-on, markerless logger that replaces magnetic heading with shaft-silhouette vision and validates it against a robot.

I. INTRODUCTION

  • Hand-held instruments leave motion unrecorded, although metric trajectories could serve surgical data science, skill assessment, and robot-learning demonstrations.Robotic systems log the kinematics of the cases they perform, while hand-held instruments log nothing.Source: §I · p. 1
  • Video-based monocular pose estimates remain millimeters to centimeters off, while magnetic field measurements changed with tool pose and between sessions.The on-chip nine-axis heading drifted by up to 2.66° within 30 s at rest.Source: §I · p. 1
  • 1.22–3.04 mm absolute tip error was achieved during continuous tracking against a moving robot, with neither markers nor instrument modification.The measurements were made under different conditions from the video-only estimates.Source: §I · p. 1
  • The logger fuses a six-axis IMU, raw ToF range, and markerless vision under the trocar RCM constraint, using the shaft silhouette for heading instead of a magnetometer.The module clips to the handle side of a standard instrument; the validation is robot-referenced and alignment-free.Source: §I · p. 1

II. RELATED WORK

Prior work estimates instrument pose from video, instrument-mounted inertial and hybrid systems, or demonstration-capture rigs with kinematic constraints. This paper combines these ideas for laparoscopy and emphasizes robot-referenced continuous validation without alignment.

A. Vision-Based Instrument Pose Estimation

  • Monocular vision methods localize instruments less reliably along the viewing ray; reported examples include 9.7–12.0 mm displacement error and 5.7° shaft-orientation error.The cited manual-instrument method also reported 3.8 mm depth error; this paper instead asks vision only for heading and shaft-junction depth.Source: A. Vision-Based Instrument Pose… · p. 2

B. Inertial and Hybrid Instrument Tracking

  • Instrument-mounted IMUs avoid line-of-sight dependence, but magnetometer-based heading is distorted near operating equipment; prior reports include up to 6.4° orientation error and marker-based position errors.Reported marker-system errors include 2.6–4.4 mm and 1.5–4 mm maximum error.Source: B. Inertial and Hybrid Instrument… · p. 2

C. Demonstration Capture and Kinematic Constraints

  • Hand-held demonstration rigs and trocar RCM constraints are adopted for laparoscopy, with robot-referenced validation on continuous trajectories, absolute error without alignment, and sensor ablations.UMI and DexCap capture manipulation demonstrations without a robot; the trocar RCM is used to condition pose estimation and control.Source: C. Demonstration Capture and Kinematic… · p. 2

III. METHODS

  • The methods combine sensing with complementary weaknesses: gyroscope drift, motion-corrupted gravity, intermittent late vision, and ToF measurements that depend on orientation.The section covers the device, trocar geometry, vision and range models, and the fusion filter.Source: §III · p. 2

A. Hardware and Implementation

A handle-side module carries inertial and range sensors and uses a camera for markerless sensing. The RCM geometry anchors the shaft at the port, while robot control projects commands through that constraint.

A. Hardware and Implementation

  • The handle-side module streams IMU and raw ToF samples, while a camera supplies frames for a U-Net trained on 1,740 labeler-generated frames.The BNO085 magnetometer is recorded but never used; the IMU, range, and camera streams run at 200/240, 100, and 15 Hz, respectively.Source: A. Hardware and Implementation · p. 3

B. Geometry and Robot Control

  • The RCM anchors the shaft line, so tip position depends on insertion depth along the shaft direction.The IMU and ToF sensor sit 304 mm and 284 mm proximal to the tip; FR3 forward kinematics provide ground truth, but the estimator uses sensor data and calibration constants only.Source: B. Geometry and Robot Control · p. 3
  • Robot commands are projected onto the port so off-port targets become pivots about the RCM; the commanded twist moves the tip and aligns the shaft to the port direction.The RCM registered for control also serves as the estimator and ground-truth reference.Source: B. Geometry and Robot Control · p. 3

C. Preliminary Study: Magnetic Disturbance and Accelerometer Drift

The preliminary study found that magnetic heading varied with tool pose and between sessions, motivating a markerless shaft-geometry approach. That approach uses silhouette planes as constraints and derives shaft position and insertion depth from the reconstructed geometry.

C. Preliminary Study: Magnetic Disturbance and Accelerometer Drift

  • 3.58° RMS rotation-angle error from accelerometer and magnetometer alone was 18 times the proposed fusion’s error.Across 180 rotation trials, the Earth-field angle changed by up to 4.8° between trial poses, versus a 0.21° gravity residual.Source: C. Preliminary Study: Magnetic… · p. 3
  • Between static sessions, the mean magnetic field fell 21%, while nine-axis heading drift during 30 s holds rose to 2.66°.The study also found unflagged accelerometer recalibration: acceleration magnitude rose 6.3% and settled 5.5% above gravity; disabling it kept magnitude within 0.06% of gravity.Source: C. Preliminary Study: Magnetic… · p. 3

D. Markerless Shaft Geometry from Vision

  • The vision pipeline trains a U-Net on rule-labeled silhouette edges and accepts frames only when its mask and the labeler agree.Acceptance requires intersection-over-union of at least 0.75 and silhouette-line directions within 0.5°.Source: D. Markerless Shaft Geometry from Vision · p. 4
  • 0.35° residual replaced an ill-conditioned axis-direction estimate from silhouette planes only about 3.6° apart.The planes are used as constraints rather than crossed to estimate the axis direction.Source: D. Markerless Shaft Geometry from Vision · p. 4
  • The point on the shaft axis closest to the camera is obtained from the sum of the silhouette-plane normals and is well conditioned.The shaft radius used is 2.59 mm.Source: D. Markerless Shaft Geometry from Vision · p. 4
  • Insertion depth comes from intersecting the shaft-junction ray with the axis, rejecting frames when the ray and axis are nearly parallel.Frames are rejected when 1 − (d^T u)^2 < 0.45.Source: D. Markerless Shaft Geometry from Vision · p. 4

E. Time-of-Flight Range Model

The range model accounts for the tool-fixed beam, its orientation, and the surrounding plane rather than assuming a simple shaft-aligned range. The section also describes smoothing for display while retaining raw valid measurements for filtering.

E. Time-of-Flight Range Model

  • A single VL53L0X range scatters about 3 mm, so firmware applies a five-sample moving average at a cost of 39 ms delay and correlated outputs.The display uses smoothed depth, while the filter uses raw status-valid ranges.Source: E. Time-of-Flight Range Model · p. 4
  • The model treats the ToF sensor as a tool-fixed beam intersecting a world-fixed plane, unlike affine or cone models that omit directional tilt effects.The beam-plane formulation uses eight parameters and unit scale.Source: E. Time-of-Flight Range Model · p. 4
  • ToF range depends on the tool’s whole orientation in the fitted beam-and-plane model, so opposite tilts change range in opposite directions.The fitted beam is 55.8° off the shaft axis and the plane is 13.4° off the neutral axis; absorbing this dependence into a scale factor increases tip error.Source: E. Time-of-Flight Range Model · p. 4

F. Error-State Kalman Filter

The filter propagates a nominal IMU state and estimates small errors that it injects and resets at measurement updates. It combines gravity, vision geometry and ToF range while accounting for correlated measurement errors.

F. Error-State Kalman Filter

  • The error-state Kalman filter propagates an IMU nominal state and estimates, injects, then resets a small error state at each update.This represents orientation error as a minimal three-vector and allows late vision updates while the nominal state advances at the gyroscope rate.Source: F. Error-State Kalman Filter · p. 4
  • Translation and rotation trials use nominal state (R, bg, L), with gyroscope bias bg included.The error-state specification begins with orientation and bias errors.Source: F. Error-State Kalman Filter · p. 4
  • Updates use gravity direction during slow rotation, silhouette constraints with vision depth, and ToF range; tip position follows from the instrument geometry.Each stream is shifted by its measured latency, and initial heading is selected from branches satisfying the silhouette constraints by image reprojection.Source: F. Error-State Kalman Filter · p. 4
  • Measurement errors are pose-locked and systematic, so residual-based noise levels are inflated by the square root of correlated samples to avoid overconfidence.The ToF depth scattered 0.43 mm across repetitions of one pose but 3.04 mm across poses.Source: F. Error-State Kalman Filter · p. 4

G. Real-Time Tracking and Display

For continuous motion, the tracking filter carries tip position and velocity and combines inertial propagation with vision, ToF and the trocar constraint. The live display is compared with robot motion under a protocol that calibrates the camera-to-robot reference separately from the estimator.

G. Real-Time Tracking and Display

  • The continuous-motion filter carries tip position and velocity and transfers IMU acceleration to the tip using angular velocity, angular acceleration, and the lever arm.Gravity updates are down-weighted rather than gated when acceleration magnitude departs from gravity.Source: G. Real-Time Tracking and Display · p. 4
  • The remote center of motion is a soft two-dimensional pseudomeasurement; ToF observes tip distance plus offset, while recent vision is needed to estimate that offset.Vision contributes shaft direction and tip position, and the trocar gate rejects shaft lines more than 3.5 mm from the registered RCM.Source: G. Real-Time Tracking and Display · p. 4
  • The live display refreshes at the gyroscope rate and shows the estimated and robot tip paths, vision lock, tracking errors, filter state, and visited working envelope.The displayed error is split into three-dimensional, radial, and tangential components.Source: G. Real-Time Tracking and Display · p. 4, Figure 3 · p. 5
  • The filter rewinds to late vision capture time and re-propagates, holding its estimate 30 ms behind real time for a reported 33 ms end-to-end display lag.Once inputs arrive, the live trajectory equals the offline causal filter used for dynamic results, without smoothing.Source: G. Real-Time Tracking and Display · p. 5

IV. EXPERIMENTS

  • Experiments reference a robot holding the instrument through the phantom trocar, but robot data never enters the estimator.Motion trials use minimum-jerk profiles about the RCM, and stationary windows bracket trials except in the dynamic experiment.Source: §IV · p. 5, A. Protocol and Calibration · p. 5

A. Protocol and Calibration

  • Camera-to-robot calibration used an ArUco marker for commanded motions, then closed-form hand-eye calibration refined over 30 poses.On 20 held-out poses, calibration left 2.35 mm and 0.74° mean error; displacement metrics are insensitive to that error.Source: A. Protocol and Calibration · p. 5

B. Static Stability

Static tests quantify the logger’s raw sensor noise, while frequency-based measurements compare observation delays across sensing channels. Raw ToF has substantially less lag than firmware-smoothed depth.

B. Static Stability

  • Accelerometer tilt noise was 0.30–0.55° and raw ToF depth noise was 2.95–3.43 mm per sample, while tilt drifted only 0.017° over five minutes.The robot tip held to 0.019–0.023 mm, so the reported noise was attributed to the logger.Source: B. Static Stability · p. 5, Table III · p. 5

C. Observation Latency

  • 9.0 ms ToF lag contrasts with 58.7 ms for firmware-smoothed depth; the IMU and camera lagged 6.6–6.9 ms and 15.5–17.3 ms, respectively.Camera latency was measured with a temporary fiducial; the passage says it applies to the markerless pipeline because frames are timestamped on arrival.Source: C. Observation Latency · p. 5, Table IV · p. 5
  • Latency tests excited insertion depth at 0.1–1.0 Hz and shaft tilt and roll at 0.15–1.6 Hz, under a 50 mm/s tip-speed limit.The measurements used lock-in detection to fit gain and phase against ground truth.Source: C. Observation Latency · p. 5

D. Translation Accuracy

Translation and rotation were evaluated against a robot without alignment, with sensor ablations testing the estimator’s components. Continuous tracking was assessed on programmed sweeps and operator-driven trajectories.

D. Translation Accuracy

  • 1.21 mm RMS displacement error across 300 translation trials rose from 0.52 mm at the held-out 1 mm step to 2.12 mm at 20 mm.The scale was 0.962; y motion was weakest at 1.83 mm, versus 0.61–0.82 mm for x and z.Source: D. Translation Accuracy · p. 5
  • Translation ablations showed that without vision, error grew tenfold and scaled with distance because heading was unfixed; without ToF, direction survived but scale collapsed to 0.749; without accelerometer updates, an error floor appeared even at the 1 mm step.Source: D. Translation Accuracy · p. 6

E. Rotation Accuracy

  • Rotation trials covered pitch, yaw, and roll at a 140 mm insertion depth, with constants fitted on repetitions 1–3 and evaluated on repetitions 4–10.The relative rotation between stationary windows was compared with robot ground truth without alignment.Source: E. Rotation Accuracy · p. 6
  • Relative-rotation and rotation-angle errors were 0.341° RMS and 0.200° RMS, respectively—5.4× lower than BNO085 on-chip nine-axis orientation and 5.9× lower than gyroscope integration; without vision, relative-rotation error became 39.4°.Source: E. Rotation Accuracy · p. 6

F. Dynamic Trajectory Tracking

  • Dynamic tracking used three programmed chirp sweeps and four 120 s free trajectories driven by an operator using a haptic stylus.A fit-only block of ten static poses calibrated the accelerometer and mounting, vision tip offset, and channel delays; the table is titled for programmed and teleoperated tip trajectory error.Source: F. Dynamic Trajectory Tracking · p. 6, Table VII · p. 6

TELEOPERATED CONTINUOUS MOTION

The full sensor fusion tracked continuous motion more accurately than the tested sensor subsets, with distinct results for programmed and teleoperated trajectories. The report also examines degraded vision and post-hoc filter changes.

TELEOPERATED CONTINUOUS MOTION

  • 1.22 mm RMS tip error on programmed sweeps and 3.04 mm on teleoperated motion; full fusion was best in all seven blocks.Orientation error was 0.6–2.2° RMS.Source: TELEOPERATED CONTINUOUS MOTION · pp. 6–7, Figure 5 · p. 7
  • Keeping one accepted vision frame in three and blanking vision for up to 10 s raised block-mean error only to 1.35 mm programmed and 3.24 mm teleoperated.Hiding the shaft junction raised the errors to 6.2 and 7.7 mm, respectively.Source: TELEOPERATED CONTINUOUS MOTION · p. 7, Figure 5 · p. 7
  • The live-display parameters produced 3.45 mm and 10.28 mm errors, while the reported results used three changes selected after inspecting evaluation blocks.The changes loosened gyroscope bias, subtracted the vision tip offset, and restored tip positions from gated-out frames; the authors say these require confirmation on an independently recorded session.Source: TELEOPERATED CONTINUOUS MOTION · p. 7, Table VIII · p. 7
  • Gyroscope bias accumulated 89° during the 102 s roll sweep, while accelerometer-derived roll stayed within 2.1°.The session’s neutral shaft was 28.6° from vertical, leaving most of the error about gravity, where only vision observes it.Source: TELEOPERATED CONTINUOUS MOTION · p. 7, Figure 5 · p. 7

V. DISCUSSION

The logger combines millimeter tracking accuracy with sub-degree relative rotation, and full fusion outperforms every sensor subset without using magnetic measurements. The authors describe possible uses for routine trajectory recording and intraoperative monitoring, while noting that validation remains limited to a phantom setup.

V. DISCUSSION

  • Magnetic disturbance that limits nine-axis fusion outside the laboratory does not reach this estimate, which uses gravity, range, and shaft silhouette instead.The discussion states that steel furniture and electrosurgical equipment affect neither gravity, range to a surface, nor the shaft silhouette in an image.Source: §V · pp. 7–8
  • The clip-on setup could make routine metric tip-trajectory recording practical without modifying the patient-entering instrument part, adding a marker, or requiring a robot.The resulting trajectories could provide the motion data imitation learning for surgical robots needs.Source: §V · p. 8
  • The causal estimate displays insertion depth and attitude at the gyroscope rate with a 30 ms lag, allowing the device to serve for intraoperative monitoring as well as recording.Those quantities are hidden from the surgeon once the instrument is inside the patient.Source: §V · p. 8
  • Validation used one device on a phantom with a fixed ceiling camera; free trajectories were teleoperated, monocular reconstruction was tested near 140 mm insertion, and the RCM was assumed fixed.The labeler was tuned to one shaft appearance, while a clinical endoscope camera moves and adds its own pose.Source: §V · p. 8

VI. CONCLUSION

  • The logger achieved millimeter tracking accuracy and sub-degree relative rotation, with full fusion ahead of every sensor subset and no magnetic measurement in the estimate.The conclusion states that validation in the operating room remains future work.Source: §VI · p. 8

Related papers

Loading 2609.25577v1…