PaperlayerView exact revision on arXiv

Source-linked AI summary

Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks

Ji Woong Kim, Tony Z. Zhao, Samuel Schmidgall, Anton Deguet, Marin Kobilarov, Chelsea Finn, Axel Krieger

arXiv:2407.12998v1cs.RO

TL;DR

Surgical imitation learning on the da Vinci is challenged by inaccurate forward kinematics, even as large repositories of surgical demonstrations offer potential for robot learning. The paper trains policies using relative actions with approximate kinematics and reports successful manipulation, robustness to configuration changes, and benefits from wrist cameras.

  • Problem

    The da Vinci’s inaccurate forward kinematics make naive policy training with approximate kinematics prone to task failure.

    Source: §1 · p. 2

  • Method

    The paper models policy actions as relative motion to train imitation-learning policies using the da Vinci’s approximate kinematics.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Relative motion is more consistent than absolute motion, and the resulting policies achieve high surgical-manipulation success, with wrist cameras improving performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach is a step toward using large repositories of approximate surgical data for robot learning without further kinematics corrections.

    Source: §6 · p. 8

    The off-the-shelf large wrist cameras are not clinically relevant, and the model acts only on current observations without adapting behavior to human instructions.

    Source: §6 · p. 8

Abstract

from arXiv · show

We explore whether surgical manipulation tasks can be learned on the da Vinci robot via imitation learning. However, the da Vinci system presents unique challenges which hinder straight-forward implementation of imitation learning. Notably, its forward kinematics is inconsistent due to imprecise joint measurements, and naively training a policy using such approximate kinematics data often leads to task failure. To overcome this limitation, we introduce a relative action formulation which enables successful policy training and deployment using its approximate kinematics data. A promising outcome of this approach is that the large repository of clinical data, which contains approximate kinematics, may be directly utilized for robot learning without further corrections. We demonstrate our findings through successful execution of three fundamental surgical tasks, including tissue manipulation, needle handling, and knot-tying.

1 Introduction

The da Vinci’s inaccurate forward kinematics make straightforward imitation learning difficult, despite the scale of available surgical demonstrations. The paper presents relative actions as a way to learn from approximate kinematics and reports successful surgical manipulation, including robustness and wrist-camera findings.

1 Introduction

  • The image-only policy outputs relative pose trajectories for both arms, using approximate kinematics without further correction.The authors identify relative motion as more consistent than absolute forward kinematics.Source: Figure 2 · p. 2, §1 · p. 2
  • The da Vinci’s inaccurate forward kinematics make naive absolute-pose training fail across the explored surgical tasks, despite a large repository of demonstrations.The paper situates this challenge against the potential of large-scale surgical data for robot learning.Source: §1 · pp. 1–2
  • Wrist cameras improve policy performance and generalization to out-of-distribution scenarios, and removable brackets enable sharing them across instruments.The paper evaluates their practical potential despite wrist cameras not being common in clinical settings.Source: §1 · p. 2
  • Relative-action policies achieve high success in surgical manipulation, generalize to unseen realistic tissue, and benefit from wrist cameras.Wrist cameras particularly help during procedure phases requiring precise depth estimation.Source: §1 · p. 2

2 Related Work

Prior imitation-learning research spans varied architectures and training strategies but has largely focused on household manipulation. Surgical learning raises distinct manipulation, perception, and proprioception challenges; prior autonomous-surgery approaches and calibration practices leave practical design questions for learning from existing data.

2 Related Work

  • Imitation-learning methods span image encoders, sequence models, action tokenization, generative objectives, and strategies such as retrieval and human-video use.Most prior work described here focuses on tabletop household activities.Source: §2 · p. 3
  • Surgical tasks combine deformable-object manipulation, difficult perception, and inaccurate proprioception, leaving key design choices for effective learning unclear.The section notes that hysteresis and inaccurate proprioception are less pronounced in industrial arms.Source: §2 · p. 3
  • Hand-eye calibration requires calibration during both data collection and inference for ground-truth kinematics; this work assumes demonstrations were collected without it.The assumed setting reflects already-collected clinical data without precise ground-truth kinematics during training.Source: §2 · p. 3
  • Prior autonomous-surgery methods target specific tasks [53] [56] [59], while imitation learning for knot-tying remains unexplored [61] [63].The section contrasts task-specific policies with end-to-end imitation learning based on robot demonstrations.Source: §2 · p. 3

3 Technical Approach

The approach learns surgical manipulation from dVRK demonstrations despite inaccurate forward kinematics, comparing camera-centric, tool-centric, and hybrid-relative action representations. The hybrid formulation uses a fixed endoscope-tip frame for translation and the current end-effector frame for rotation.

3 Technical Approach

  • The dVRK has an endoscopic camera manipulator and two patient-side manipulators; potentiometer-based measurements make its forward kinematics inaccurate, with errors up to 5 cm.Passive setup joints use only potentiometers, while active joints also use motor encoders for improved precision.Source: §3 · p. 3, Figure 3 · p. 4
  • Demonstrations form trajectories of observations, proprioception, and actions, with observations including endoscope and wrist-camera images.Proprioception records the current PSM poses relative to the endoscope-tip frame; actions are desired waypoints specified through teleoperation.Source: §3 · pp. 3–4
  • The paper compares camera-centric absolute poses, tool-centric relative motion, and hybrid-relative actions as representations for learning surgical manipulation.The camera-centric approach is the baseline; tool-centric actions improve success rates, and the hybrid formulation further improves translation accuracy.Source: §3 · p. 4, Figure 4 · p. 5
  • The camera-centric baseline predicts action sequences as absolute end-effector poses relative to the endoscope-tip frame.The action prediction horizon is C.Source: §3 · p. 4
  • Tool-centric actions represent relative motion in the current, moving end-effector frame, making action motion more consistent than absolute forward kinematics.The formulation cancels endoscope forward-kinematics terms, reducing the number of joints involved; its moving frame requires the policy to localize end-effector orientation from images.Source: §3 · p. 4
  • Hybrid-relative actions define delta translations in the fixed endoscope-tip frame and delta rotations in the current end-effector frame.Using a fixed frame for translation removes the need to localize end-effector orientation to generate translations along its axes.Source: §3 · p. 5

4 Implementation Details

Policies are trained with ACT and diffusion policy using endoscope and wrist-camera images, without kinematics inputs. Their outputs include position, orientation, and jaw-angle commands for both arms.

4 Implementation Details

  • Training uses action chunking with transformers (ACT) [23] and diffusion policy.Both are trained on the endoscope and wrist-camera images, resized to 224 × 224 × 3.Source: §4 · p. 5
  • The policies take endoscope and wrist-camera images as input and omit kinematics data because dVRK kinematics are generally inconsistent.Original endoscope images are 1024 × 1280 × 3 and wrist images are 480 × 640 × 3.Source: §4 · p. 5
  • Each policy outputs end-effector position, orientation, and jaw angle for both arms.Position and orientation outputs are deltas where the action representation uses relative actions.Source: §4 · p. 5

5 Experiments

The experiments compare action representations, wrist cameras, and generalization across tissue lift, needle pickup and handover, and knot-tying. They test whether relative motion remains consistent under robot configuration changes and whether the policies succeed across these tasks.

Experiment Setup

  • Data collection used 224 tissue-lift, 250 needle-handover, and 500 knot-tying trials, collected by one user over multiple days.Evaluation included three tasks with distinct training and test placements, as illustrated in Figure 6.Source: §5 · p. 6, Figure 6 · p. 7

Evaluating the Consistency of Relative Motion vs. Absolute Forward Kinematics

  • Under shifted setup-joint configurations, camera-centric replay failed while tool-centric and hybrid-relative replay tracked more consistently; all representations reconstructed the reference in the unchanged setup.Relative-action tracking errors did not vary significantly across configurations.Source: Figure 5 · p. 6, Table 1 · p. 6, §5 · p. 7

Policy Performance Using Various Action Representations

  • Across tissue lift, needle pickup and handover, and knot-tying, hybrid-relative actions achieved the highest success rates; diffusion policy with this representation underperformed ACT.Success rates are reported in Table 2.Source: Table 2 · p. 7, §5 · p. 8
  • Tool-centric actions often failed during needle handover after a roughly 90-degree gripper rotation, while hybrid-relative actions ground translation in the fixed endoscope-tip frame.The authors conjecture that localizing a moving end-effector frame made translation challenging.Source: §5 · p. 8

Evaluating the Importance of Wrist Camera

  • Omitting wrist cameras significantly reduced performance on needle pickup and handover and knot-tying, where the wrist view helped with depth-sensitive needle transfer.The authors report that this information was difficult to discern from the endoscopic view.Source: §5 · p. 8

Evaluating Generalization

  • Hybrid-relative policies showed high success on pig-loin needle pickup and handover, but motion quality and needle-grasp accuracy were lower than in core experiments.Qualitative tests also showed successful knot-tying on pork and needle grasps on chicken and an unseen 3D pad.Source: §5 · p. 8

6 Limitations and Conclusion

The authors demonstrate imitation learning on the dVRK using approximate kinematics without post-processing corrections. They identify non-clinically relevant wrist cameras and the lack of instruction-conditioned behavior as limitations.

Limitations

  • The off-the-shelf wrist cameras used are large and not clinically relevant.The authors suggest smaller cameras and optimized mounts as possible future improvements.Source: §6 · p. 8
  • The model acts only on current observations and cannot modulate its behavior based on human instruction.The authors identify this as an issue for future work.Source: §6 · p. 8

Conclusion

  • The approach uses approximate dVRK kinematics without further post-processing corrections and models policy actions as relative motion.The authors frame this as a step toward leveraging approximate surgical data for robot learning at scale.Source: §6 · p. 8

A Implementation Details

The implementation adapts ACT and diffusion policy to the surgical camera and end-effector inputs and outputs. ACT uses a six-dimensional rotation representation with a post-processing step to obtain an orthonormal rotation matrix.

ACT

  • ACT accepts four images—left and right endoscope views and left and right wrist-camera views—and outputs a 20-dimensional pose vector for both arms.Each arm's 10 dimensions comprise position, orientation, and jaw angle.Source: A Implementation Details · p. 14
  • ACT represents orientation with six values for the first two rotation-matrix columns, then uses Gram-Schmidt and a cross product to complete the matrix.The post-processing converts predicted vectors to an orthonormal rotation matrix.Source: A Implementation Details · p. 14

Diffusion Policy

B Generalization to Novel Settings

The model generalizes beyond training settings, with successful zero-shot needle pickup and handover across several substrates and zero-shot knot-tying on pork loin.

B Generalization to Novel Settings

  • Successful zero-shot needle pickup and handover on chicken leg, pork loin, and a 3D pad demonstrate generalization beyond training settings.Source: Figure 7 · p. 15
  • Successful zero-shot knot-tying on pork loin extends the reported generalization examples to another surgical task.Source: B Generalization to Novel Settings · p. 15

Related papers

Loading 2407.12998v1…