PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control

Changyeob Shin, Peter Walker Ferguson, Sahba Aghajani Pedram, Ji Ma, Erik P. Dutson, Jacob Rosen

arXiv:1902.01459v2cs.RO

TL;DR

Soft-tissue manipulation is difficult to automate because its dynamics are complex and tissue points are manipulated indirectly. The paper compares vision-based learning MPC approaches using reinforcement learning and expert demonstrations. Simulations and a Raven IV experiment report successful manipulation, with demonstrations providing a strong initial policy.

  • Problem

    Soft-tissue dynamics are complex, and indirectly manipulating tissue points makes automated tissue manipulation difficult.

    Source: §I · p. 1

  • Method

    The paper compares vision-based learning MPC using reinforcement learning to learn dynamics through exploration and learning from demonstration to initialize them.

    Source: §I · p. 2, 2) Model Predictive Control · p. 2

  • Results

    In simulation both algorithms completed the task, while LfD worked immediately near demonstrated targets; on Raven IV, it successfully positioned tissue points.

    Source: 2) Simulation RL vs. LfD · p. 5, §IV · p. 6, B. Surgical Robot Experiment · p. 6

  • Takeaways & Limitations

    LfD can provide a good initial policy in real environments when demonstrations cover relevant workspaces.

    Source: §IV · p. 6

    The workspace was constrained to prevent visual occlusion, and the LfD initialization assumes a fixed camera-to-environment relationship; extending the method to 3D is also identified as future work.

    Source: §IV · p. 6

Abstract

from arXiv · show

Tissue manipulation is a frequently used fundamental subtask of any surgical procedures, and in some cases it may require the involvement of a surgeon's assistant. The complex dynamics of soft tissue as an unstructured environment is one of the main challenges in any attempt to automate the manipulation of it via a surgical robotic system. Two AI learning based model predictive control algorithms using vision strategies are proposed and studied: (1) reinforcement learning and (2) learning from demonstration. Comparison of the performance of these AI algorithms in a simulation setting indicated that the learning from demonstration algorithm can boost the learning policy by initializing the predicted dynamics with given demonstrations. Furthermore, the learning from demonstration algorithm is implemented on a Raven IV surgical robotic system and successfully demonstrated feasibility of the proposed algorithm using an experimental approach. This study is part of a profound vision in which the role of a surgeon will be redefined as a pure decision maker whereas the vast majority of the manipulation will be conducted autonomously by a surgical robotic system. A supplementary video can be found at: http://bionics.seas.ucla.edu/research/surgeryproject17.html

I. INTRODUCTION

Automating tissue manipulation is challenging because soft tissue has complex dynamics and tracked tissue points are manipulated indirectly through robot grasps. The paper proposes learning-based MPC using reinforcement learning or human demonstrations, with demonstrations initializing the learned dynamics.

I. INTRODUCTION

  • Soft-tissue manipulation is difficult to automate because its complex dynamics lack a model that predicts its behavior.Indirect manipulation of points on the tissue adds difficulty.Source: §I · p. 1
  • The task places specified tissue points at desired image-frame positions while robot arms manipulate the tissue indirectly by grasping elsewhere.Colored markers support vision tracking in this research; the paper notes that tissue features could instead be identified through common patterns.Source: §I · p. 2
  • The proposed framework compares reinforcement learning, which learns tissue dynamics through exploration, with learning from demonstration, which initializes those dynamics from expert demonstrations.The demonstrations can be collected by recording robotic tissue manipulation in teleoperation mode.Source: §I · p. 2
  • The algorithms learn nonlinear tissue dynamics in image space with a neural network, avoiding the need for a separate object model in the reviewed model-based approach.The authors describe this as providing flexibility across objects with different physical properties.Source: §I · p. 2

II. METHODS

The methods use model-based reinforcement learning with an internal MPC controller. They assume continuously available, unoccluded vision feedback and a grasped tissue with no slippage.

A. Algorithms

  • Model-based reinforcement learning updates a dynamics function from trial-and-error samples and uses an internal controller to optimize a reward or cost; this work pairs it with MPC for sample efficiency.The paper cites sample efficiency as desirable because collecting data with physical robotic systems is expensive.Source: A. Algorithms · p. 2

1) Assumptions:

  • The methods assume robot and tissue features remain visible and unoccluded, and that manipulation begins after grasping with no slippage.Source: 1) Assumptions · p. 2

2) Model Predictive Control:

The MPC predicts future states by propagating current states, inputs, and learned dynamics, then selects inputs to minimize cost. Its tissue-manipulation formulation uses image-space positions and velocities and can select the number of inputs in the horizon.

2) Model Predictive Control:

  • MPC forward-propagates current states, inputs, and dynamics to predict future states and select inputs that optimize costs.Source: 2) Model Predictive Control · p. 2
  • The formulation represents tissue-point and robot-wrist positions and velocities in image space, with inputs specifying robot movement and a finite control horizon.The tissue-point vectors contain the positions or velocities of all tissue points; corresponding robot vectors represent all robots.Source: 2) Model Predictive Control · p. 2
  • The cost reduces Euclidean distance between tissue points and desired points, while the optimization also selects how many inputs to use in the horizon.Source: 2) Model Predictive Control · p. 2

3) Adaptive MPC:

A neural network supplies the dynamics used by adaptive MPC, and candidate input sequences are optimized before the first input is sent to visual servoing. The reinforcement-learning loop uses vision feedback, exploration, and model training.

3) Adaptive MPC:

  • The dynamics network takes robot-wrist positions, tissue-point positions, and robot control inputs, and predicts tissue-point velocities.Source: 3) Adaptive MPC · p. 2
  • The controller generates random input candidates to obtain an optimal control sequence, then updates desired wrist positions using its first input and visual servoing.The sequence also determines the optimal number of steps in the time horizon.Source: 3) Adaptive MPC · pp. 2–3
  • The block diagram presents reinforcement learning; its workflow saves experience to replay memory, selects a mini-batch to train the dynamics model, and then performs visual servoing of the robots.Source: Figure 2 · p. 3, 3) Adaptive MPC · p. 3

4) Model-based Reinforcement Learning:

The reinforcement-learning controller learns tissue dynamics from image-based robot and tissue positions, then uses those dynamics to select robot actions through model predictive control. It explores early and trains the dynamics network from stored experience before relying on its learned policy.

4) Model-based Reinforcement Learning:

  • The controller extracts robot-wrist and tissue-point positions from images, then computes their velocities from successive frames for learning dynamics.The positions and velocities are stored as experience sets in replay memory.Source: 4) Model-based Reinforcement Learning · p. 3
  • An ε-greedy policy gradually shifts from random exploration to actions selected by model predictive control as the action count increases.The exploration probability decreases linearly from 1 to 0.1.Source: 4) Model-based Reinforcement Learning · p. 3
  • Robot actions are discrete image-space movements scaled by a step size, and the chosen action updates desired robot positions before visual servoing.The action set includes stopping and moving left, right, up, or down.Source: 4) Model-based Reinforcement Learning · p. 3
  • After collecting a predefined amount of experience, the algorithm trains the dynamics network on fixed-size samples from replay memory and then selects optimal actions using the learned dynamics.The process repeats until the predefined exploration period ends.Source: 4) Model-based Reinforcement Learning · p. 3

5) Learning from Demonstrations:

The paper proposes learning from expert demonstrations to initialize tissue-dynamics predictions for model predictive control. Demonstrations can reduce exploration, but their coverage of desired positions and robot workspaces affects whether further exploration is needed.

5) Learning from Demonstrations:

  • Self-exploration from scratch is described as potentially time-consuming and dangerous, motivating the use of expert demonstrations to initialize tissue dynamics.The proposed demonstrations are video sequences capturing human teleoperation of surgical robots.Source: 5) Learning from Demonstrations · pp. 3–4
  • The algorithm extracts robot-wrist and tissue-point positions and velocities from demonstration images, trains the dynamics network, and uses it to initialize model predictive control.The resulting experience sets are stored in a demonstration replay memory.Source: 5) Learning from Demonstrations · p. 4
  • Demonstrations for one desired-position set support only slightly different targets, while broad coverage of desired positions and robot workspaces can permit task completion without exploration.Significantly different targets require exploration when demonstrations cover only one specific set.Source: 5) Learning from Demonstrations · p. 4

6) Computer Vision Algorithm:

The paper uses vision to track robot wrists and tissue features, and describes its controller settings and simulated tissue environment. The vision workspace restriction limits robot dexterity.

6) Computer Vision Algorithm:

  • Colored features are segmented to locate robot wrists and tissue points, whose positions are calculated from contour-point averages.Tissue points are labeled using prior configuration information, and robot motion is restricted to a two-dimensional square workspace to prevent occlusion.Source: 6) Computer Vision Algorithm · p. 4

7) Learning Algorithm Hyperparameters:

  • The dynamics network has two 12-unit ReLU hidden layers, with four tissue points producing 16-element inputs and 8-element outputs.The control period is 0.5 seconds for both algorithms; RL uses step 5 during learning, and its simulation episodes reset every 1,000 actions up to 5,000 actions.Source: 7) Learning Algorithm Hyperparameters · p. 4

B. Simulation

  • The CHAI3D simulation models soft tissue as spheres arranged in a skeleton whose mass, spring, and damper coefficients determine tissue dynamics.The simulation dynamics update at 1 kHz, while boundary elements are fixed for internal stability.Source: B. Simulation · p. 4, Figure 4 · p. 4

C. Surgical Robot Experiment

The surgical-robot experiments used one pair of Raven IV arms to manipulate an elastic latex object marked for vision tracking. The camera operated in single-camera mode, with reduced image resolution and a 30 Hz frame rate.

1) Raven IV:

  • Experiments used one pair of the Raven IV's cable-driven arms, each with 7 degrees of freedom including grippers.The other pair could independently perform tasks such as suturing.Source: 1) Raven IV · p. 4

2) Experiment Environment:

  • The manipulation target was highly elastic colored latex fixed by four clips, with blue tape marking tissue points for computer-vision tracking.A stereo camera was used in single-camera mode.Source: 2) Experiment Environment · p. 4
  • The camera ran at 30 Hz, and a light source was installed to improve computer-vision robustness.The original camera resolution was reduced before image processing.Source: 2) Experiment Environment · p. 5

III. RESULTS AND DISCUSSION

The simulation uses variable step sizes to balance fast error reduction with stable, smaller steady-state error. The section also compares RL and LfD across learning stages.

A. Simulation

  • The simulation MPC evaluates 5,000 robot-action candidate sets per control period over a maximum five-step horizon.Source: A. Simulation · p. 5

1) Effects of Step Size:

  • Large steps reduce error quickly, while smaller steps stabilize and reduce steady-state error; the controller switches step sizes as error falls.It uses a large step above 150 error, a small step between 150 and 70, and single-pixel actions below 70.Source: 1) Effects of Step Size · p. 5
  • Fig. 7 compares LfD with RL at multiple learning stages, with RL exploiting optimal actions under its learned dynamics at each stage.Source: Figure 7 · p. 5

2) Simulation RL vs. LfD:

LfD is effective immediately in simulation when desired points are near demonstrations, and it was also demonstrated on Raven IV. Across three similar-initialization runs, the robot positioned tissue points by stretching the tissue.

2) Simulation RL vs. LfD:

  • LfD performed immediately after initialization when desired tissue-point positions were near the demonstrations, whereas RL required thorough training.RL performance did not necessarily improve during learning until its dynamics were well understood.Source: 2) Simulation RL vs. LfD · p. 5
  • A single demonstration spanning a wide workspace was found sufficient to move tissue points to varied desired positions.Source: 2) Simulation RL vs. LfD · pp. 5–6

B. Surgical Robot Experiment

IV. CONCLUSIONS

The study presents RL and LfD for soft-tissue manipulation, with LfD reducing the exploration needed to learn dynamics and achieving point placement on Raven IV. The authors identify workspace, dimensionality, and camera-frame constraints for future work.

IV. CONCLUSIONS

  • Simulation results showed both algorithms could accomplish the task, while LfD reduced exploration and successfully placed tissue points on Raven IV.The authors state that relevant workspace coverage in demonstrations supports a good initial policy in real environments.Source: §IV · p. 6
  • The approach was tested in a workspace constrained to prevent visual occlusion, and LfD assumes a fixed camera-to-environment relationship.The authors identify occlusion avoidance, 3D manipulation, and use of demonstrations from different camera frames as future work.Source: §IV · p. 6
  • The authors anticipate that future research along similar lines may eventually support clinical automation of surgical subtasks.Source: §IV · p. 6

Related papers

Loading 1902.01459v2…