PaperlayerView exact revision on arXiv

Source-linked AI summary

Digital Twin-Driven VR Teleoperation with Multi-View Spatial Perception for Surgical Robots

Chang Liu, Chenhao Yu, Honghao Zhao, Hao Ding, Haochen Wei, Adnan Munawar, Mathias Unberath, Peter Kazanzides

arXiv:2609.25527v1cs.RO

TL;DR

Video pass-through HMD teleoperation retains a single endoscopic viewpoint and limited depth cues, while bare-hand tracking can hinder robust control. This paper presents markerless, digital-twin VR teleoperation with multi-view cues and controller-based decoupling. In a 15-participant dVRK study, VR reduced path length and jerk versus an MR baseline and outperformed the MTM console on reported trajectory measures.

  • Problem

    Video pass-through HMDs retain a single endoscopic viewpoint and limited depth cues, while bare-hand tracking is susceptible to tremors, tracking loss, and occlusion.

    Source: §I · p. 1

  • Method

    The platform builds a markerless digital twin for multi-view VR rendering and uses clutch-anchored controller mapping to decouple hand posture from instrument alignment.

    Source: §I · p. 1

  • Results

    VR reduced path length by 86% and jerk by 95% versus the MR baseline, and significantly outperformed the MTM console in path length, motion efficiency, and smoothness.

    Source: B. Task Performance · p. 7

  • Takeaways & Limitations

    The authors conclude that rich spatial cues combined with stable controller input can collectively offset the need for strict physical hand–instrument correspondence.

    Source: §VII · p. 8

    The perception pipeline assumes rigid-body geometry, requires an external RGB-D camera absent from standard clinical setups, and its scene-twin update rate lags the 90 Hz robot-twin rate.

    Source: §VI · pp. 7–8

Abstract

from arXiv · show

Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.

I. INTRODUCTION

Console-based RMIS offers intuitive hand–instrument correspondence but restricts surgeon mobility, while HMD video pass-through retains limited viewpoint and depth cues. The proposed digital-twin VR platform uses multi-view spatial cues and clutch-anchored controllers to decouple hand posture from instrument alignment.

I. INTRODUCTION

  • Clutch-anchored relative mapping with stable 6-DoF controllers lets operators use comfortable poses without strict hand–instrument alignment.The paper presents enhanced spatial understanding as able to offset reliance on physical hand–instrument correspondence.Source: §I · p. 1
  • State-synchronized architecture logs structured kinematic and spatial states for scalable multi-view training data and potential use as an auxiliary 3D clinical view.The potential clinical-workflow integration is described as auxiliary, not as a replacement for existing views.Source: §I · p. 1
  • The digital-twin VR platform replaces video pass-through with state synchronization for unconstrained multi-view rendering and enhanced depth and spatial perception.The scene twin is built from markerless perception and streamed to the HMD.Source: §I · p. 1
  • A 15-participant study reports better VR trajectory quality than a state-of-the-art MR baseline and spatial judgment comparable to the traditional console.The study is described as showing significantly better trajectory quality than MR, with spatial judgment comparable to the console.Source: §I · p. 2

II. RELATED WORK

Prior XR teleoperation systems pursue portability but retain constraints from video-based single views and input limitations. The paper situates its live perception-driven digital twin within work on surgical scene models and synchronized dVRK data, then describes an asynchronous state-based teleoperation pipeline.

A. Extended Reality (XR) and Surgical Teleoperation Interfaces

  • Video-based XR restricts operators to a single viewpoint, while a lack of dedicated handheld controllers hinders reliable clutching.The platform addresses these limitations with flexible digital-twin viewpoints and 6-DoF VR controllers.Source: A. Extended Reality (XR) and Surgical… · p. 2

B. Surgical Scene Perception and Digital Twins

  • Prior digital-twin work includes offline surgical automation, training, and simulation; integrating perception-driven scene representations into live robotic control remains computationally demanding.The paper places its approach among systems that bring vision foundation models into live teleoperation.Source: B. Surgical Scene Perception and… · p. 2

III. SYSTEM OVERVIEW

  • The system sends robot states and camera-derived object poses asynchronously to Unity, while controller inputs return as dVRK motion commands.Unity renders state-consistent scenes at headset refresh rate, independently of robot and perception update rates.Source: §III · pp. 2–3
  • Hand–eye calibration establishes a shared camera-centric frame because inaccurate passive Setup Joint kinematics can cause virtual-to-physical robot drift.The calibration bypasses the Setup Joints to align perception and control.Source: §III · p. 3

A. dVRK Digital Twin

The robot digital twin synchronizes a Unity kinematic model to measured dVRK joint states. A separate object-state twin uses camera perception and operator-facing tracking tools to maintain scene objects.

A. dVRK Digital Twin

  • Unity updates the dVRK kinematic chain from measured joint angles and forward kinematics, including the PSM remote-center-of-motion constraint.This lets operators reason visually about arm configuration, joint-limit proximity, and potential collision risks.Source: A. dVRK Digital Twin · p. 3

B. Object State Twin

  • An external RGB-D camera and perception server estimate manipulated-item poses and stream object states to Unity; the GUI supports tracking, confidence monitoring, and manual re-initialization.The object-state twin complements the robot kinematic twin.Source: B. Object State Twin · p. 3

1) Segmentation and Tracking (SAM 2 & SAM 3):

  • SAM 2 propagates prompted instance masks across frames to stabilize object identity through motion and partial occlusion, while optional SAM 3 labels help bookkeeping for similar objects.The masks also serve as a spatial prior for downstream pose estimation.Source: 1) Segmentation and Tracking (SAM 2 &… · p. 3

2) 6D Pose Estimation and State Streaming (FoundationPose):

FoundationPose estimates rigid object poses from RGB-D frames and instance masks, with state updates rendered directly in the shared camera-centric frame. Hand–eye calibration aligns the physical robot and virtual scene using the same coordinate system.

2) 6D Pose Estimation and State Streaming (FoundationPose):

  • FoundationPose uses an object geometry prior with sampling and refinement to estimate rigid 6D poses without object-specific retraining.The estimated poses are expressed in the RGB-D camera frame and streamed as object state.Source: 2) 6D Pose Estimation and State… · p. 3

3) Multi-Object Management and Robust Rendering:

  • The DT manager assigns persistent IDs, can use SAM 3 labels to reduce similar-instance ID switches, and smooths asynchronous updates while holding the last valid pose during dropouts.The system alerts the operator when manual re-initialization is needed.Source: 3) Multi-Object Management and Robust… · p. 3

C. Hand–Eye Calibration and Registration

1) Hand–Eye Calibration :

Hand–eye calibration estimates the camera-to-robot transform while accounting for the fixed, unknown tool-to-marker offset. The calibrated camera frame then anchors the robot and object digital twins in Unity.

1) Hand–Eye Calibration :

  • Hand–eye calibration estimates the camera-to-base transform from synchronized camera-marker and robot-tip poses, accounting for the fixed, unknown tool-to-marker offset.The marker pose comes from camera-based PnP, while the tool-tip pose comes from robot forward kinematics.Source: 1) Hand–Eye Calibration · p. 4

2) Robot Base Registration:

  • The calibrated transform redefines the dVRK base frame so the controller’s kinematic stream is expressed directly in the camera frame, bypassing inaccurate Setup Joint kinematics.Unity uses the camera as its world origin and imports the calibration to anchor the virtual robot.Source: 2) Robot Base Registration · p. 4

3) Object Digital Twin Registration:

  • FoundationPose estimates object poses directly in the camera frame, so object registration needs no additional spatial calibration beyond the shared camera frame.This places virtual objects relative to the robot in the physical workspace.Source: 3) Object Digital Twin Registration · p. 4

D. VR Teleoperation Interface

Controller motion is mapped relative to a clutch-engagement anchor, and the target robot pose is composed from that anchor and the relative displacement. The interface supports ergonomic re-anchoring and coarse-to-fine repositioning.

1) Controller-to-Robot Motion Mapping:

T HMD

  • Computing target pose from a fixed anchor rather than frame-to-frame integration reduces sensitivity to tracking jitter and yields smoother trajectories.Relative-motion control also removes the need for absolute controller-to-robot workspace alignment, enabling re-anchoring.Source: T HMD · pp. 4–5
  • The interface displays real-time status and multi-view feedback, while controller buttons switch motion scale levels for coarse repositioning or fine alignment.The current motion scale is displayed in the UI.Source: Figure 5 · p. 5, 3) Grasp Feedback and Jaw Lock · p. 5
  • Simultaneously clutching translation and rotation lets users reposition and reorient their hands ergonomically without strict physical hand–instrument correspondence.The VR scene lets operators visually understand the instrument state during clutching.Source: T HMD · p. 5

2) User Interface and Teleoperation Controls:

The middle trigger provides analog jaw control, while grasp confirmation and a jaw-lock mode support grasp feedback and prevent accidental reopening during transport.

3) Grasp Feedback and Jaw Lock:

  • Middle-trigger press depth is mapped linearly from a calibrated open/close range to the desired jaw angle.The stated runtime jaw-angle limits are −10° and 60°.Source: 3) Grasp Feedback and Jaw Lock · p. 5
  • A jaw stall at a non-fully-closed angle triggers a haptic pulse to confirm object contact, and jaw-lock latches the current angle during transport.The lock helps prevent accidental reopening.Source: 3) Grasp Feedback and Jaw Lock · p. 5

4) Multi-View Rendering and State-Driven Replay:

The platform provides complementary native-workspace and endoscopic views, and recorded state streams can be replayed from arbitrary virtual-camera perspectives. The study compares VR with traditional-console and mixed-reality teleoperation during a depth-sensitive ring-placement task.

4) Multi-View Rendering and State-Driven Replay:

  • The native workspace view and a fixed endoscopic view provide complementary perspectives for cross-checking depth and alignment, reducing occlusion-related ambiguity.The endoscopic view emulates the conventional surgical camera perspective.Source: 4) Multi-View Rendering and… · p. 5
  • Saved kinematic and spatial state sequences support offline replay of recorded demonstrations, with multiple virtual cameras rendering tasks from arbitrary perspectives.The architecture logs structured state streams rather than relying on high-bandwidth video.Source: 4) Multi-View Rendering and… · p. 5

IV. EXPERIMENTS

  • The study compares the dVRK traditional console, HoloLens 2 mixed-reality teleoperation, and the proposed VR platform on depth perception and spatial understanding.Participants grasped a vertical ring and placed it on a target peg, a task requiring tool-to-object depth judgment and gripper orientation.Source: §IV · p. 5, A. Experimental Setup and Protocol · p. 5

A. Experimental Setup and Protocol

  • Fifteen volunteers tested all three systems in randomized order under a within-subject design; gripper closures per trial served as a proxy for depth-estimation confidence.Fewer closures indicated higher spatial certainty.Source: A. Experimental Setup and Protocol · p. 6

B. Evaluation Metrics

Evaluation uses task and trajectory measures plus subjective feedback, while latency is assessed separately for the robot and scene twins. The control loop measured 59.7 ms, and scene-perception runtime depends on both module and hardware.

B. Evaluation Metrics

  • Task evaluation covers completion time, gripper closures, path length, motion efficiency, integrated squared jerk, and subjective feedback on usability and spatial awareness.Fewer gripper closures indicate higher depth-estimation confidence.Source: B. Evaluation Metrics · p. 6

A. System Latency Evaluation

  • The robot-twin control loop and perception-driven scene-twin update loop are evaluated independently because they use separate state streams and update constraints.The robot twin uses high-rate kinematic streaming; scene-twin updates are bounded by perception throughput.Source: A. System Latency Evaluation · p. 6

1) Teleoperation Latency:

  • 59.7 ms was the measured end-to-end latency from Quest 3S target-pose commands to robot poses returned via UDP.The estimate used cross-correlation of command and measured-pose time series.Source: 1) Teleoperation Latency · p. 6

2) Scene Perception Latency and Scalability:

  • On an RTX 3080 Ti, SAM 2 segmentation took 157.4 ± 6.5 ms per frame and FoundationPose estimation took 50.9±10.2 ms.Scene-twin update rate is compute-bounded and hardware- and backend-dependent; the passage notes potential acceleration with optimized deployment and scheduling.Source: 2) Scene Perception Latency and… · p. 6

B. Task Performance

VR produced shorter, smoother trajectories than the MR baseline and outperformed the MTM console on several trajectory metrics. Subjective ratings favored MTM overall, while VR was consistently ahead of MR and participants valued multi-view rendering.

B. Task Performance

  • Overall differences used a Friedman test, with pairwise Wilcoxon signed-rank tests corrected by Bonferroni at α = 0.05/3.Table I summarizes four objective metrics.Source: B. Task Performance · p. 6, Table I · p. 7
  • 86% shorter path length and 95% lower jerk versus MR; VR also outperformed MTM in path length, motion efficiency, and smoothness.The VR–MR path-length and jerk differences were both p<.001; comparisons with MTM were significant at p = .037 for path length and p < .001 for motion efficiency and smoothness.Source: Table I · p. 7, B. Task Performance · p. 7
  • The MR condition’s high variance and lower performance were attributed primarily to fragile bare-hand tracking and gesture-based clutching, while 6-DoF VR controllers yielded more stable trajectories.The reported issues include tremors, tracking loss, recognition errors, and accidental clutch misoperations.Source: B. Task Performance · p. 7
  • MTM had higher jerk than VR (p < .001), which the authors linked largely to orientation realignment at clutch re-engagement after users repositioned their hands.Users who did not maintain consistent hand orientation during clutching experienced abrupt positional transients on re-engagement.Source: B. Task Performance · p. 7
  • VR had the fewest mean gripper closures (1.27), followed by MR (1.47) and MTM (1.73), indicating the highest depth-perception confidence among conditions.Fewer closures serve as a proxy for operator depth-perception confidence.Source: B. Task Performance · p. 7

C. Subjective Evaluation

  • MTM scored highest overall and VR ranked second; VR and MTM scores were 2.27 vs. 2.47 for ease of learning and 2.13 vs. 2.40 for precision.The Borda scores rank systems across five dimensions; 87% found multi-view helpful, 80% cited better depth perception, and 80% preferred multi-view to a fixed endoscopic view.Source: Figure 8 · p. 7, C. Subjective Evaluation · p. 7

D. Spatial Perception and Control Decoupling

VR users achieved smoother, more efficient trajectories than MTM users despite lacking physical hand–instrument coupling. The paper argues that rich spatial cues can support this decoupling, while identifying perception and clinical-setup constraints on the current system.

D. Spatial Perception and Control Decoupling

  • VR users produced significantly smoother, more efficient trajectories than MTM users despite having no physical hand–instrument coupling.They also required the fewest gripper closures across conditions.Source: D. Spatial Perception and Control… · p. 7
  • The authors argue that strict hand–instrument correspondence compensates for insufficient spatial perception rather than being fundamental to teleoperation.They state that rich visual spatial cues can let operators maintain awareness through visual feedback alone, avoiding restrictions, occlusions, and fatigue associated with virtual-shaft designs.Source: D. Spatial Perception and Control… · p. 7

VI. LIMITATIONS AND FUTURE WORK

  • FoundationPose assumes rigid-body geometry, making real-time extension to highly deformable environments such as soft tissue a significant computational challenge.The system also relies on an external RGB-D camera absent from standard clinical setups; clinical translation will require robust stereo-endoscopic depth estimation.Source: §VI · pp. 7–8

VII. CONCLUSIONS

  • The state-synchronized architecture could support an auxiliary 3D spatial view and multi-view training data collection through offline replay.The conclusion describes the architecture as a scalable foundation for multi-view data generation and clinical augmentation.Source: §VI · p. 8, §VII · p. 8

Related papers

Loading 2609.25527v1…