Source-linked AI summary
World Models for General Surgical Grasping
Hongbin Lin, Bin Li, Chun Wai Wong, Juan Rojas, Xiangyu Chu, Kwok Wai Samuel Au
TL;DR
Surgical grasping systems must handle unseen objects and disturbances, but existing approaches rely heavily on pose estimation and feature tracking. GAS uses world-model-based deep reinforcement learning with pixel-level visual processing, including depth estimation and a compact depth-and-mask representation. The learned controller averages 69% success on diverse real-robot surgical grasping and demonstrates robustness across six disturbance conditions.
Problem
Pose-estimation and feature-tracking methods have difficulty adapting to unseen objects and are vulnerable to estimation error, while surgical grasping also presents small visual signals and uncertain depth.
Source: §I · p. 2Method
GAS learns a pixel-level visuomotor policy with world-model-based deep reinforcement learning, using depth uncertainty estimation and a compact depth-and-mask representation.
Source: §I · p. 2, §VII · p. 11Results
The real-robot controller averages 69% success across diverse unseen surgical objects and grippers and shows robustness across six disturbance conditions.
Source: §I · p. 2Takeaways & Limitations
A single policy trained in simulation and transferred without finetuning can grasp diverse surgical objects with different grippers in real-world environments.
Source: §I · p. 2The reported evaluation is based on experiments in simulation and on real robots assessing performance, generality, and robustness.
Source: §VII · p. 11
Abstract
from arXiv · showhide
Intelligent vision control systems for surgical robots should adapt to unknown and diverse objects while being robust to system disturbances. Previous methods did not meet these requirements due to mainly relying on pose estimation and feature tracking. We propose a world-model-based deep reinforcement learning framework "Grasp Anything for Surgery" (GAS), that learns a pixel-level visuomotor policy for surgical grasping, enhancing both generality and robustness. In particular, a novel method is proposed to estimate the values and uncertainties of depth pixels for a rigid-link object's inaccurate region based on the empirical prior of the object's size; both depth and mask images of task objects are encoded to a single compact 3-channel image (size: 64x64x3) by dynamically zooming in the mask regions, minimizing the information loss. The learned controller's effectiveness is extensively evaluated in simulation and in a real robot. Our learned visuomotor policy handles: i) unseen objects, including 5 types of target grasping objects and a robot gripper, in unstructured real-world surgery environments, and ii) disturbances in perception and control. Note that we are the first work to achieve a unified surgical control system that grasps diverse surgical objects using different robot grippers on real robots in complex surgery scenes (average success rate: 69%). Our system also demonstrates significant robustness across 6 conditions including background variation, target disturbance, camera pose variation, kinematic control error, image noise, and re-grasping after the gripped target object drops from the gripper. Videos and codes can be found on our project page: https://linhongbin.github.io/gas/.
I. INTRODUCTION
GAS targets general and robust surgical grasping, where pose-estimation and feature-tracking methods struggle with errors and unseen objects, while world-model methods face small visual signals and uncertain depth. It combines pixel-level visuomotor learning with depth estimation and a compact visual representation, and is evaluated across surgical objects, grippers, and disturbances.
A. Surgical Autonomous Grasping
- Prior surgical grasping methods rely on pose estimation or feature tracking, limiting robustness to estimation error and adaptation to unseen objects in unstructured environments.The methods also require full geometric models or assume geometric shapes to infer arbitrary point positions.Source: §I · p. 1
I. INTRODUCTION
- The framework learns one visuomotor controller for diverse unseen surgical objects and grippers, with an average 69% success rate and robustness across six disturbance conditions.It is trained in simulation and transferred directly to a real robot without further finetuning.Source: §I · p. 2
- GAS estimates inaccurate rigid-link depth pixels and their uncertainty from an empirical object-size prior, addressing depth errors from conventional depth cameras.The paper reports that visuomotor learning benefits from this depth estimation.Source: §I · p. 2
- Dynamic Spotlight Adaptation encodes object depth and mask images into a 64x64x3 image by zooming into mask regions, minimizing information loss.The paper reports that this representation improves visuomotor learning performance.Source: §I · p. 2
B. Visual Manipulation With World Models on Real Robots
- Earlier world-model robot manipulation work did not demonstrate handling centimeter-scale surgical objects and depth uncertainty, while GAS applies a single controller across diverse surgical grasping tasks.Earlier surgical controllers often used tracked poses and separate controllers for tasks; GAS uses model-based reinforcement learning without pose estimation.Source: A. Surgical Autonomous Grasping · p. 2, B. Visual Manipulation With World… · p. 3
III. PROBLEM FORMULATION AND BACKGROUND
The task is to control an unknown robot gripper to grasp and lift unknown objects from a phantom plane using image-based discrete actions. The formulation treats this as a partially observable decision process with stated operating assumptions.
1) Problem Setup
- The policy maps temporal RGB-D observations to discrete commands for grasping and lifting an unknown object lying on a phantom plane.Target-object and gripper initial poses are randomized within the workspace.Source: A. Problem Setup and Assumptions · p. 3
- At each timestep, the policy observes a 600x600x4 RGB-D image and predicts a discrete command.The command actuates the gripper in 2 mm Cartesian increments and 10-degree rotations about the z-axis.Source: A. Problem Setup and Assumptions · p. 3
- The action space has nine commands, combining independently activated directions with gripper open or closed modes.The grasping horizon is 300 timesteps.Source: A. Problem Setup and Assumptions · p. 3
- A grasp succeeds when the object is grasped and lifted 10 mm above the plane; the gripper and object remain within a rectangular workspace.Desired positions outside the workspace are clipped to the closest point within it.Source: A. Problem Setup and Assumptions · p. 3
2) Assumptions
- The setup assumes the arm stays within joint limits, the target or gripper may be partially but not fully occluded, and pose or visual-feature coordinates are unavailable.Source: 2) Assumptions · p. 3
B. POMDP Formulation
- The control task is formulated as a discrete-time partially observable Markov decision process, with states, actions, rewards, transitions, observations, discount factor, and observation space.The learning objective is to maximize expected future discounted reward.Source: B. POMDP Formulation · p. 3
C. DreamerV2
GAS builds on DreamerV2 world-model learning and processes observations before policy control. Its video-segmentation stage uses a few prompted masks and online tracking to supply object masks in surgical scenes.
C. DreamerV2
- DreamerV2 learns environment dynamics from on-policy rollouts and optimizes a control policy using simulated rollouts from the learned world model.The world model and policy are learned simultaneously; GAS builds on DreamerV2 for surgical grasping.Source: C. DreamerV2 · p. 3
IV. GRASP ANYTHING FOR SURGERY
- GAS processes visual observations through segmentation, depth estimation, and Dynamic Spotlight Adaptation before a world-model-learned policy predicts actions for the robot gripper.Virtual Clutch processes predicted actions, and domain randomization is applied in simulation.Source: §IV · p. 3, Figure 3 · p. 4
A. Video Object Segmentation
- Video object segmentation assigns task-object classes at pixel level to address the small image area occupied by the gripper and grasping object.Their combined projected area is about 1% of the image in complex surgical scenes.Source: A. Video Object Segmentation · p. 4
- Track Anything Model combines SAM and XMem to segment and track objects online from a first-frame mask annotated with a few point prompts.XMem tracks subsequent masks using temporal and spatial correspondence in historical RGB images; the approach avoids preparing datasets or training segmentation models.Source: A. Video Object Segmentation · p. 4
B. Depth Estimation for Object’s Imprecise Region
The depth estimator replaces inaccurate pixels on rigid-link objects using nearby precise regions of the same category, while representing uncertainty through an empirical size bound.
B. Depth Estimation for Object’s Imprecise Region
- Real-world gripper-tip depth pixels are inaccurate in the original image and are replaced by the estimator’s values.The figure also illustrates depth noise from structured-light sensing and the use of median depth from the ground-truth area.Source: Figure 4 · p. 5
- For each inaccurate mask, the method selects the nearest-centroid precise mask in the same category and uses its nonzero median depth for estimation.Masks are classified using a maximum pixel-error threshold; the grasping tasks set it to 10 mm.Source: B. Depth Estimation for Object’s… · p. 5
- The estimator bounds depth uncertainty using the minimal diameter of a spatial bound containing the paired masks across arbitrary object configurations.The depth estimate is sampled uniformly from the estimated depth plus or minus the empirical size bound.Source: B. Depth Estimation for Object’s… · p. 5, Figure 4 · p. 5
C. Visual Representation
Dynamic Spotlight Adaptation encodes task-object masks and estimated depth in a compact image by retaining global mask cues and zooming in on local regions.
C. Visual Representation
- The representation combines 600x600 mask and depth images into a 64x64x3 image through dynamic zooming, aiming to preserve mask-coordinate information and image quality.The paper reports that ordinary downsampling to 64x64x3 degrades visuomotor performance, while high-dimensional images are difficult to scale with the VAE.Source: C. Visual Representation · pp. 5–6
- Global-layer square masks encode object identities, while local mask and depth layers zoom into segmented regions for the gripper tip and target object.The zoom-in mask tracks the gripper-tip mask, and the three layers are stacked before downsampling.Source: C. Visual Representation · p. 6, Figure 5 · p. 6
- The final image replaces bottom-right global-layer pixels with 6x6 encodings of task-level state and gripper toggling state.The visual pipeline stacks the global, mask, and depth layers and downsamples them to the compact image.Source: Figure 5 · p. 6, C. Visual Representation · p. 7
D. Virtual Clutch
Virtual Clutch delays robot movement until the controller has received sufficient temporal visual observations, addressing inaccurate posterior-state estimates at rollout start.
D. Virtual Clutch
- Virtual Clutch is intended to reduce posterior-state estimation error by waiting for temporal observations before robot-environment interaction.The paper links early error to a poor initial prior and slow convergence when few visual observations are available.Source: D. Virtual Clutch · p. 7
- The controller issues an idle action until the clutch threshold, then applies the learned policy’s predicted action.Hclutch sets the timestep at which the clutch closes; the idle command leaves robot joint positions unchanged.Source: D. Virtual Clutch · p. 7
E. Domain Randomization
Training randomizes camera, target, gripper, and image conditions; an FSM defines sparse rewards and task terminations for grasping.
E. Domain Randomization
- Domain randomization adds camera-pose variation, target motion, gripper action noise, and image noise during simulated rollouts.Image perturbations include salt-and-pepper noise, Gaussian blur, and occlusion-like cutouts.Source: E. Domain Randomization · p. 7
F. FSM-Driven Reward and Termination
- FSM-driven rewards assign sparse outcomes to successful grasping, horizon failure, normal progress, and three abnormal events.The abnormal events are workspace-exceeding gripper poses, horizontal target sliding, and the target leaving the DSA zoom-in box; the task-level state determines rewards and terminations.Source: F. FSM-Driven Reward and Termination · p. 8
V. EXPERIMENT SETUP
The study evaluates simulated grasping and real-robot grasping on two phantoms, comparing GAS with pixel-level RL baselines and module ablations.
A. Environment
- Simulation used SurRoL with needles, boxes, and rods sampled equally and object shapes randomly scaled from 0.75 to 1.5.The study used a simulation benchmark and two real-robot benchmarks.Source: A. Environment · p. 8
- Real-robot tests used a da Vinci PSM with RealSense D435 images; the rectangle phantom supported performance, generality, and robustness studies, while the liver phantom provided more realistic shapes and mechanics.A manually sent joystick flag and a gripper-pose height above 10 mm marked successful termination.Source: A. Environment · p. 8
B. Baselines and Ablations
- PPO and DreamerV2 received RGB-D images, masks, and scalar signals; their raw visual input stacked depth, encoded masks, and grayscale images at 64 × 64 × 3.PPO used an R3M visual encoder and DreamerV2 used CNN and MLP encoders.Source: B. Baselines and Ablations · pp. 8–9
- Ablations replaced DSA with RawVR, removed depth estimation, removed VC, or removed domain randomization during training.These variants were designed to assess the proposed modules and sim-to-real domain randomization.Source: B. Baselines and Ablations · p. 9
C. Training and Evaluation
The evaluation measures grasp success and grasping speed, trains controllers in simulation, and tests whether uncertainty-aware depth estimation improves learning.
C. Training and Evaluation
- Success rate and Grasping Score assess grasping outcomes, with the latter introduced to evaluate how quickly a controller achieves a successful grasp.The score is bounded between 0 and 1.Source: C. Training and Evaluation · p. 9
- Controllers trained for 1.8 million simulation timesteps across three seeds, with 50 evaluation rollouts every 200K timesteps and 10K pre-fill timesteps.The pre-fill included 3K scripted-demonstration timesteps and 7K random-policy timesteps; the highest-scoring seed was selected for real-robot deployment.Source: C. Training and Evaluation · p. 9
1) Depth Estimation:
- After 500K timesteps, the controller using depth estimation began improving, while the baseline without it degraded as training continued.The authors attribute the difference to improved task-object depth accuracy, particularly at the gripper tip, given D435 depth limitations.Source: 1) Depth Estimation · p. 9
2) Visual Representation:
DSA and VC address training difficulties in visual representation and world-model learning, while GAS outperforms PPO and DreamerV2 in the reported simulation and real-robot comparisons.
2) Visual Representation:
- RawVR did not improve during training, whereas DSA reduced encoder learning difficulty by removing background imagery and dynamically zooming into task regions.The authors identify small task regions and degraded image quality after downsampling as possible challenges for RawVR.Source: 2) Visual Representation · pp. 9–10
4) Comparisons With SOTA Methods:
- VC significantly improved training stability relative to GAS-NoClutch, whose learning curve degraded earliest; the authors conjecture initial posterior-state error as a cause.The proposed cause is a conjecture, and the authors leave systematic study of instability for future work.Source: 4) Comparisons With SOTA Methods · p. 10
- 87% success rate and a score of 56 were reported for GAS in simulation after 1.8 million timesteps; PPO and DreamerV2 did not learn successful grasps.The authors note that the baseline methods might improve with more training.Source: 4) Comparisons With SOTA Methods · p. 10
2) Generality:
In real-robot tests, GAS grasped diverse unseen objects and maintained success rates above 75% across five disturbance types, with 55% success when re-grasping dropped targets. The conclusion also identifies constraints in workspace, motion speed, and FSM-state identification.
2) Generality:
- On five unseen target objects and an unseen gripper, GAS achieved over 60% success; sponge and fragments were hardest to grasp.Their small size increases vision and control demands, while their low weight lets gripper force slide them horizontally.Source: 2) Generality · p. 10
3) Robustness:
- GAS re-grasped dropped targets with 55% success and maintained success rates above 75% under the other five disturbance types.The disturbances included background and camera-pose variation, image and action noise, target movement, and re-grasping.Source: 3) Robustness · p. 11
VII. CONCLUSION, LIMITATION, AND DISCUSSION
- GAS combines TAM mask segmentation, object-size-prior depth-value and uncertainty estimation, DSA visual representation, VC, domain randomization, and FSM task-state monitoring.The conclusion describes these components as part of its deep-RL framework for pixel-level visuomotor grasping.Source: §VII · p. 11
- Experiments in simulation and on a real robot evaluated controller performance, generality, and robustness.Source: §VII · p. 11
- Failure cases occur when horizontal pushing moves a target out of the workspace; discrete small action steps limit gripper speed, and FSM states are identified through human observation and control.Source: §VII · p. 11
- Future work targets VOS robustness, continuous actions, reduced target sliding, additional surgical tasks such as thread looping, and automated FSM identification.Source: §VII · p. 11
APPENDIX
The appendix includes a table titled “Hyperparameter for GAS.”
APPENDIX
- The appendix contains a GAS hyperparameter table.Source: Table V · p. 13