PaperlayerView exact revision on arXiv

Source-linked AI summary

Surgical Kinematics from Monocular Video with Learned Articulated Motion Constraints

Mehmet Kerem Turkcan, Soham Samal, Zoran Kostic

arXiv:2609.27227v1cs.CVcs.RO

TL;DR

Robotic-surgery motion analysis needs instrument kinematics, but synchronized robot states are not routinely available for monocular recordings. The paper combines visual features, motion-aware prediction, and differentiable trajectory fitting to reconstruct kinematics. On the main Open-H benchmark, it improves path-length error and motion-segmentation temporal mAP over LiveMAE.

  • Problem

    Instrument kinematics support objective surgical-motion assessment, but monocular recordings and limited access to synchronized robot states make video-based reconstruction relevant.

    Source: §I · p. 1

  • Method

    A visual and temporal network estimates instrument states and motion increments, then differentiable weighted least squares fits trajectories with separate displacement-magnitude and direction prediction.

    Source: §I · p. 1, C. Learned articulated motion… · p. 3

  • Results

    Path-length MAE is 0.3392 cm for the method versus 0.4510 cm for LiveMAE, and temporal mean average precision for motion segmentation is 54.44% versus 44.54%.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The reconstructions support motion analysis from video when recorded robot states are unavailable.

    Source: §I · p. 1

Abstract

from arXiv · show

Objective assessment of robotic surgery uses instrument kinematics, which must be reconstructed when only video is available. We introduce a kinematic reconstruction network for estimating instrument position, orientation and jaw angle from monocular video. Our visual representation combines global attention pooling of frozen DINOv3 features with local pooling at instrument landmarks from fine-tuned SAM 3.1 masks. Our shared Transformer encoder and temporal convolutional heads integrate this representation with mask geometry, monocular depth and visual state estimates from arm-specific multilayer regression networks. Our position branch predicts displacement magnitude and direction separately to preserve traveled distance. We fit trajectories to predicted state observations and motion increments by differentiable weighted least squares, expressing quaternion observations relative to cumulative predicted rotations to obtain a quadratic orientation objective. We evaluate reconstruction across 2,802 Open-H episodes. Compared with LiveMAE on the main Open-H benchmark, our method reduces path-length mean absolute error from 0.45 to 0.34\,cm and increases temporal mean average precision for motion segmentation from 44.54\% to 54.44\%.

I. INTRODUCTION

Robotic-surgery motion analysis relies on instrument kinematics, but synchronized robot states are often unavailable when recordings are monocular. The paper introduces a reconstruction network and trajectory fitting that combine state estimates with motion constraints.

I. INTRODUCTION

  • Objective assessment uses instrument kinematics to compute path length, speed, smoothness, and working volume, but synchronized kinematic data are not routinely available from commercial platforms.Reconstructing kinematics from video supports motion analysis when recorded robot states are unavailable.Source: §I · p. 1
  • Appearance changes and occlusion complicate reconstruction, while small position errors can accumulate in path length and temporal averaging can attenuate motion.The introduction therefore calls for both state accuracy and motion consistency.Source: §I · p. 1
  • The network combines a shared Transformer and temporal convolutional heads with appearance, mask geometry, monocular depth, and initial visual state estimates.Its position branch predicts displacement magnitude separately from direction and is supervised on local motion and episode path length.Source: §I · p. 1
  • Differentiable weighted least squares fits trajectories to predicted state observations and motion residuals, with cumulative predicted rotations yielding a quadratic orientation objective.Evaluation uses Open-H, published methods, and component ablations.Source: §I · p. 1

II. RELATED WORK

Related work spans surgical segmentation, instrument-state reconstruction, and learned estimation with differentiable optimization. The paper uses instrument masks and a quadratic trajectory objective to connect these areas.

II. RELATED WORK

  • Surgical segmentation work includes prototype-prompt SurgicalSAM, instrument-region text prompts [7], temporal-memory pruning [8], SAM 3 prompting [9], and low-rank adaptation [10].This work fine-tunes SAM 3.1 for instrument masks and uses mask landmarks for local appearance pooling and trajectory reconstruction.Source: §II · p. 1
  • Prior instrument pose work uses landmarks and geometric models, while SurgRIPE, SurgiPose, and LiveMAE address benchmarking or kinematic reconstruction.The paper's trajectory objective constrains both state observations and motion increments.Source: §II · p. 1
  • Kalman filtering and RTS smoothing couple observations through dynamics, while DROID-SLAM and Theseus integrate learned estimation with differentiable optimization.The paper constructs quadratic trajectory objectives and uses a quaternion change of variables to retain that structure for orientation.Source: §II · p. 1

A. Video-to-state reconstruction

The reconstruction target is each patient-side manipulator’s kinematic state in its recorded base frame. Recorded instrument states provide training targets and evaluation references.

A. Video-to-state reconstruction

  • Given images and timestamps, the method estimates each patient-side manipulator’s position, orientation, and jaw angle in its recorded base frame.The state is represented as position p_t, unit quaternion q_t, and jaw angle a_t.Source: A. Video-to-state reconstruction · p. 1
  • Position is measured in meters and jaw angle in radians, and recorded instrument states provide training targets and evaluation references.Each arm is reconstructed independently using shared parameters.Source: A. Video-to-state reconstruction · p. 1

B. Visual representation

The visual representation combines global and landmark-local appearance with mask geometry and relative depth, then supplies descriptors and initial states to temporal refinement. Instrument masks also provide landmarks for pooling and reconstruction.

B. Visual representation

  • SAM 3 and SAM 3.1 are fine-tuned for surgical segmentation, with the SAM 3.1 training data expanded using CRCD anatomy annotations and uTenn instrument labels.The segmenter is selected by validation mask average precision and frozen before kinematic reconstruction.Source: B. Visual representation · pp. 1–2
  • The reconstruction weights observation and motion residuals, with motion constraints stopping at gaps; its displayed normal equation reconstructs position and jaw while orientation follows the stated separate formulation.The figure caption also identifies depth as divided by its image median.Source: B. Visual representation · p. 2, Figure 1 · p. 2
  • The mask descriptor encodes instrument extent, axis, region geometry, segmentation scores, and mask availability; depth features summarize landmark depths and their relative and spatial relationships.Landmark depth is the median of nearby pixels within the instrument mask when available, using relative depth from a frozen EndoSynth Depth Anything V1 network.Source: B. Visual representation · p. 2
  • Frozen DINOv3 features are pooled globally using arm-specific spatial attention and locally around six SAM-mask landmarks, then combined by a shared fusion network.The landmarks include the instrument centroid, principal-axis endpoints, wrist centroid, and two gripper-region centroids.Source: B. Visual representation · p. 2
  • Arm-specific multilayer regressors predict position, orientation, and jaw angle from the fused descriptor, and their orientation and jaw estimates initialize temporal refinement.The regressors are trained against recorded states after centering position targets within training episodes, then frozen.Source: B. Visual representation · p. 2

C. Learned articulated motion constraints

The temporal model combines shared sequence representations with learned state and motion predictions, then fits trajectories to weighted observation and motion constraints. Position refinement predicts displacement magnitude separately from direction, while quaternion refinement uses a change of variables to obtain a quadratic objective.

Learned state and motion predictions

  • The shared Transformer and prediction heads estimate instrument states and motion increments, using visual features, initial state estimates, and elapsed time between frames.The motion head predicts position and jaw increments, rotation vectors, and motion weights from pairwise features.Source: C. Learned articulated motion… · pp. 2–3

Displacement magnitude and direction

  • The position branch predicts displacement magnitude separately from direction to constrain traveled distance directly.A temporal-convolutional branch predicts direction corrections and positive displacement magnitudes.Source: C. Learned articulated motion… · p. 3

Weighted least-squares reconstruction

  • Weighted reconstruction fits position and jaw trajectories to observation and motion residuals, with positive coefficients weighting each residual type.The position balance coefficient is learned with the position head, while jaw and orientation coefficients are fixed.Source: C. Learned articulated motion… · p. 3
  • For orientation, cumulative predicted rotations are removed from observations before fitting a transformed trajectory, and the inverse transform with normalization recovers unit quaternions.With positive observation coefficients and nonnegative motion coefficients, the orientation objective has a unique minimizer obtained through tridiagonal linear systems.Source: C. Learned articulated motion… · pp. 3–4

Training objectives

  • Training gradually adds motion losses on increments, reconstructed position increments, and multiscale path lengths, while refinement stages freeze previously fitted components.Motion losses and constraints do not cross recording gaps; position and jaw refinement precede fitting the displacement branch.Source: C. Learned articulated motion… · p. 4

IV. DATA AND EVALUATION

The evaluation pairs Open-H videos with recorded states for both arms, using grouped cross-validation across main endoscopy, CAO, and Hamlyn. It assesses state accuracy, motion measures, perception, depth, and motion segmentation.

Evaluation cohorts

  • Open-H evaluation retains reviewed intervals with landmark coverage, visible motion, and agreement with recorded states, while including all released CAO and Hamlyn episodes.Main endoscopy spans training, porcine cholecystectomy, and cautery; Table I pairs each frame with states for both arms.Source: §IV · p. 4, Table I · p. 4

Splits

  • Three-fold cross-validation keeps each episode and both arms together, with folds balanced by setting or task and training durations selected on validation groups.The displacement-branch architecture and training duration are selected on main-cohort validation data and held fixed across cohorts.Source: §IV · p. 4

Additional recordings

  • An additional 70 JHU episodes are evaluated using networks fitted on the first main-benchmark training split, without further fitting or checkpoint selection.These recordings retain their released video-state pairings.Source: §IV · p. 4

Reported comparisons

  • Table II covers perception and kinematic reconstruction, including rendered-test depth and main-benchmark reconstruction; Table III covers temporal estimation and component contributions.Figure 3 also presents paired episode errors across the three Open-H cohorts, with values below the diagonal favoring the proposed method.Source: Table II · p. 5, Figure 3 · p. 5, Table III · p. 6

Evaluation measures

  • State evaluation measures aligned position MAE, orientation error, and jaw MAE; motion evaluation includes path-length error, mean speed, log dimensionless jerk, and jaw closures.Path length is rigid-transform invariant and includes the first second; LDJ uses continuous intervals of at least eight frames.Source: §IV · p. 6

Motion segmentation

  • A fold-specific MS-TCN evaluates five motion classes from reconstructed states and motion features against classes defined from recorded kinematics.The classifier and its preprocessing, normalization, and decoding are shared across reconstructions within each fold.Source: §IV · p. 6

V. EXPERIMENTS

The experiments assess surgical perception, kinematic reconstruction, and downstream motion analysis on Open-H. The proposed method achieves lower reconstruction errors than the compared baselines, with additional evaluations across cohorts and recordings.

Surgical perception

  • Fine-tuning increases mask AP from 2.77 to 37.07 and box AP from 3.29 to 41.44.The comparison uses pretrained and fine-tuned SAM 3.1 on identical test images.Source: §V · p. 6

Comparison with a surgical segmenter

  • Fine-tuned SAM 3.1 achieves 85.57% instrument foreground IoU, compared with 73.85% for SurgicalSAM.Both methods are compared on fully annotated test images at native resolution.Source: §V · p. 6

Depth fine-tuning on rendered Open-H

  • Depth fine-tuning reduces mean scale-and-shift-invariant absolute error from 0.9114 to 0.2454, a 73.08% reduction.Improvement is reported on every one of the 72 test episodes.Source: §V · p. 6

Motion analysis from reconstructed states

  • Temporal mAP is 54.44% versus 44.54% for LiveMAE; an illustrative episode reaches F1 of 88.9% versus 57.1% with the same classifier.Fig. 4 shows the reconstruction recovering closed-jaw motion omitted by LiveMAE.Source: Table V · p. 7, Figure 4 · p. 7, §V · p. 8

Kinematic reconstruction comparison

  • Path MAE is 0.3392 cm for our approach versus 0.4510 cm for LiveMAE, a 24.79% reduction on the main benchmark.The metric compares traveled distance independently of position alignment.Source: §V · p. 7

Reconstruction and local temporal refinement

  • Position refinement reduces path MAE from 0.3854 to 0.3392 cm, an 11.98% reduction; removing reconstruction raises all five errors.The inference removals use the complete checkpoint without refitting.Source: §V · p. 7

Additional Open-H tasks and camera views

  • Our method achieves the lowest error on all five metrics in CAO and Hamlyn; Hamlyn path-length MAE is 1.2627 cm versus 7.2639 cm for RTS-smoothed direct regression.CAO and Hamlyn use separately trained networks.Source: §V · p. 7, Table IV · p. 7

Evaluation on additional recordings

  • On 70 additional JHU episodes, the fixed main-benchmark network has the lowest error on all five metrics, with Path MAE of 0.7751 cm versus 0.9684 cm for LiveMAE.Table IV distinguishes these additional recordings from the separately trained CAO and Hamlyn cohorts.Source: Table IV · p. 7, §V · p. 8

VI. CONCLUSION

The network reconstructs instrument kinematics from monocular video using surgical visual features, separate displacement magnitude and direction prediction, and differentiable trajectory reconstruction. On the main Open-H benchmark, it outperforms all baselines on five reconstruction metrics and improves motion-segmentation temporal mAP over LiveMAE.

Main benchmark

  • On the main Open-H benchmark, path-length MAE is 0.3392 cm, 8.12% below Transformer regression with our visual representation and 24.79% below LiveMAE.The method has the lowest errors on all five reconstruction metrics.Source: §VI · p. 8

Motion smoothness

  • Log dimensionless jerk MAE is 0.688 versus 4.940 for Transformer regression using global DINOv3 features, an 86.07% reduction.Source: §VI · p. 8

Downstream motion analysis

  • Motion segmentation achieves 54.44% temporal mean average precision versus 44.54% for LiveMAE.Source: §VI · p. 8
Loading 2609.27227v1…