PaperlayerView exact revision on arXiv

Source-linked AI summary

MR-GLi: Mixed Reality-Based Gripper-Linked Overlays for Underwater Robot Arm Teleoperation via Bilateral Control

Masashi Sasago, Masato Kobayashi, Yuki Uranishi

arXiv:2609.16041v1cs.RO

TL;DR

Visual torque feedback may support underwater bilateral teleoperation, but the relative value of MR and conventional monitor presentation remained unclear. MR-GLi registers the torque indicator and wrist-camera image to the gripper; in a comparison with a 2D monitor, it reduced perceived gaze-shift burden while maintaining comparable torque-regulation performance.

  • Problem

    How visual feedback should be spatially presented during manipulation, and how MR compares with conventional monitor presentation, remained open questions.

    Source: §I · p. 1

  • Method

    MR-GLi spatially registers a reaction torque indicator and wrist-camera image to the robot gripper and was compared with a 2D monitor using consistent control and visual-feedback content.

    Source: §I · pp. 1–2

  • Results

    MR-GLi reduced perceived gaze-shift burden while maintaining torque-regulation performance comparable to the 2D monitor, with no significant differences in workload or overall usability.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    The results demonstrate the feasibility of gripper-linked MR overlays for underwater bilateral teleoperation without degrading torque-regulation performance.

    Source: §V · p. 6

    Future work will evaluate the approach with direct gaze measurements and under more challenging conditions.

    Source: §V · p. 6

Abstract

from arXiv · show

Visual torque feedback supports underwater bilateral teleoperation, but the benefit of mixed reality (MR) over conventional monitor presentation remains unclear. We present MR-GLi, an MR interface that spatially registers a reaction torque indicator and wrist-camera image to the robot gripper. Twenty participants performed lift and pick-and-place tasks with rigid and compliant objects in a counterbalanced within-subject comparison with a 2D monitor, using identical visual-feedback content and four-channel bilateral control. MR-GLi provided gripper-linked access to visual feedback while maintaining a similar level of torque-regulation performance to the 2D monitor. Subjective evaluation further indicated reduced perceived burden associated with shifting attention between the workspace and visual feedback. These results demonstrate the feasibility of gripper-linked MR overlays for underwater bilateral teleoperation and highlight the importance of considering information access in addition to task performance. Additional material: https://mertcookimg.github.io/mr-gli/

I. INTRODUCTION

MR-GLi spatially links torque feedback and wrist imagery to the gripper to address open questions about visual access and MR versus monitor presentation. In a controlled comparison, it reduced perceived gaze-shift burden while torque-regulation performance remained comparable.

  • Underwater drag and buoyancy can make small grasp-regulation torques difficult to perceive, while separate displays may require attention shifts between feedback and workspace.Source: §I · p. 1
  • MR-UBi showed that adding a reaction torque indicator improved underwater torque regulation, but spatial presentation and comparison with conventional monitor presentation remained open questions.Source: §I · p. 1
  • MR-GLi registers the reaction torque indicator and wrist-camera image to the gripper, so both move with the manipulation site rather than staying fixed in the operator’s view.Source: §I · p. 1
  • A counterbalanced within-subject comparison held bilateral control and visual-feedback content constant between MR-GLi and a conventional 2D monitor.Source: §I · p. 2
  • MR-GLi significantly reduced perceived gaze-shift burden, while no significant differences were detected in torque regulation, workload, or overall usability.Source: §I · p. 2

II. RELATED WORK

Prior work studies visual force feedback and underwater bilateral haptics, but does not resolve how visual cues should be presented in underwater bilateral teleoperation or the relative value of MR and monitor displays.

  • A-RIFT and SensARy use visual force cues in telemanipulation, but do not address their presentation during underwater bilateral teleoperation.Source: A. Visual Force Feedback in… · p. 2
  • MR-UBi improved torque regulation by adding an RTI, but its same-HMD comparison did not resolve MR versus monitor presentation; MR-GLi holds the controller, RTI, and wrist-camera image constant.Source: A. Visual Force Feedback in… · p. 2

B. Spatial Presentation and Visual Information Access

The section motivates testing whether gripper-linked cues reduce visual-access burden while preserving torque regulation, then describes the MR-GLi platform and its bilateral-control basis.

B. Spatial Presentation and Visual Information Access

A. System Overview

  • MR-GLi adds a follower-wrist camera and registers both visual elements to the gripper, extending MR-UBi’s fixed-position RTI presentation.Source: A. System Overview · pp. 2–3

B. Robot Platform and Bilateral Control

  • The robot has three rotational joints and a two-fingered gripper, and bilateral control enforces position tracking and the action–reaction law; joint torque is estimated with a reaction torque observer.Source: B. Robot Platform and Bilateral Control · p. 3

C. Visual Feedback Components

The visual feedback combines a close wrist-camera view with a reaction torque indicator (RTI), which encodes torque through bar length and hue. Its torque ranges and encoding are shared across both display configurations.

C. Visual Feedback Components

  • The RTI uses bar length for torque magnitude and hue to indicate the torque range.The bar length represents the filling ratio, and hue follows the range encoding.Source: C. Visual Feedback Components · p. 3, Figure 4 · p. 3
  • The wrist camera provides an RGB close-up of the gripper and the object it holds.The camera is mounted on the follower arm just behind the gripper.Source: C. Visual Feedback Components · p. 3
  • The shared optimal torque band is 0.20–0.40 N·m, with blue, green, and red indicating the three torque ranges.The band is a common reference target for both objects, not a per-object slippage or overload threshold.Source: C. Visual Feedback Components · p. 3

D. Gripper-Linked Overlays and Calibration

MR-GLi positions the wrist-camera image and RTI relative to the moving gripper, keeping the overlays legible as the arm moves. A manual spatial alignment establishes the robot-to-MR coordinate transformation before each session.

D. Gripper-Linked Overlays and Calibration

  • Joint encoder readings and forward kinematics determine the gripper pose, which anchors the wrist-camera image and RTI at a fixed offset.The resulting panel pose follows the gripper as the arm moves.Source: D. Gripper-Linked Overlays and… · p. 3
  • The two panels share a world-space canvas above the gripper that faces the operator at each frame.This orientation keeps the visual information legible regardless of gripper orientation.Source: D. Gripper-Linked Overlays and… · p. 4
  • Before each session, manual alignment adjusts the virtual arm to the physical arm and stores the robot-to-MR transformation.The play-area boundary anchors the stored position and orientation, allowing the coordinate frame to be restored when the application launches.Source: D. Gripper-Linked Overlays and… · p. 4

IV. EXPERIMENTS

The study compares MR-GLi with a 2D monitor while holding the bilateral control and visual-feedback content constant. It assesses torque regulation and task time alongside perceived gaze-shift burden, workload, and usability.

A. Study Design

  • Perceived gaze-shift burden was the primary subjective outcome, and time within the optimal torque range was the primary objective outcome.The study also evaluated workload, usability, torque regulation, and task time.Source: A. Study Design · p. 4

B. Experimental Setup and Display Conditions

  • Both configurations used the same bilateral control, RTI encoding, and wrist-camera image, differing in how that information reached the operator.The table notes that some factors unavoidably covary with placement.Source: B. Experimental Setup and Display… · p. 4, Table I · p. 4
  • In the 2D Monitor condition, operators viewed the tank directly and read the RTI and wrist-camera close-up side by side on a 24-inch monitor.The side-by-side elements allow gripper state and torque to be read together without looking at the tank, making this a strong baseline; the camera view does not cover the arm or approach path.Source: B. Experimental Setup and Display… · p. 4
  • In MR-GLi, operators viewed the workspace through color video passthrough with both visual elements registered to the gripper, keeping them in view without a gaze shift.Viewing geometry was fixed by the room layout but not instrumented.Source: B. Experimental Setup and Display… · p. 4

C. Participants and Procedure

Twenty participants completed a counterbalanced comparison of two display conditions across lift and pick-and-place tasks with rigid and compliant objects. Each condition used the same fixed sequence of task–object combinations.

C. Participants and Procedure

  • Twenty participants completed 160 trials across two display conditions and four task–object combinations.The group included 9 women and 11 men, with a mean age of 22.0 years (SD 2.7; range 18–29).Source: C. Participants and Procedure · p. 4
  • Each condition included Lift and Pick&Place tasks with a rigid polyurethane block and a compliant cellulose sponge, run in the same fixed sequence for every participant.The objects were chosen to span a wide stiffness range; condition order was counterbalanced, with ten participants meeting each display first.Source: C. Participants and Procedure · p. 4
  • The session comprised preparation, task performance, and a questionnaire under the two display conditions.The protocol figure presents the procedure for both display conditions.Source: C. Participants and Procedure · p. 4, Figure 7 · p. 5

D. Evaluation Metrics

Evaluation combines torque-regulation and task-time measures with subjective usability, workload, and gaze/task-focus ratings; paired condition differences are tested at participant level.

D. Evaluation Metrics

  • Optimal torque-range time is the primary objective outcome, while Low and High measure time below and above the target range.Optimal is reported as a percentage; Low and High are also percentages.Source: D. Evaluation Metrics · p. 5
  • MAE measures deviation from target torque, and SD measures variation in grasping torque over the evaluation interval.Both are reported in N·m.Source: D. Evaluation Metrics · p. 5
  • Pick&Place trial time runs from grasp to release; Lift timing is not evaluated because it requires a fixed five-second hold.Source: D. Evaluation Metrics · p. 5
  • Subjective measures include SUS, NASA-TLX workload and subscales, and retrospective ratings of gaze-shift ease and task focus.Participants also chose which configuration was easier to use.Source: D. Evaluation Metrics · p. 5
  • Paired condition differences are tested with participant-level Wilcoxon signed-rank tests, with significance assessed at .05.Effect size is r = |Z|/√n.Source: D. Evaluation Metrics · p. 5
  • Lift requires holding an object at a marked height for five seconds, while Pick&Place requires carrying it across the tank and releasing it into a basket.Both tasks were performed with both objects.Source: Figure 8 · p. 5

E. Torque Regulation

MR-GLi maintained torque-regulation performance comparable to the 2D monitor, while perceived gaze-shift ease improved and other subjective measures showed no significant deterioration.

E. Torque Regulation

  • p = .546, r = .14 for time within the optimal torque range; no other torque-regulation measure differed significantly (all p ≥ .409, r ≤ .19).Source: E. Torque Regulation · p. 5
  • p = .053, r = .43 for the shorter MR-GLi trial time, a nonsignificant difference that may partly reflect the condition-order effect (p = .021).Torque-regulation accuracy showed no corresponding order effect (p = .68).Source: E. Torque Regulation · p. 5

F. Subjective Measures

  • p = .002, r = .68 for improved perceived gaze-shift ease with MR-GLi.Source: F. Subjective Measures · p. 5
  • p = .506 for SUS and p = .261 for overall NASA-TLX; neither differed significantly, although medians numerically favored MR-GLi (77.5 vs. 76.2; 41.7 vs. 49.5).Source: F. Subjective Measures · p. 5
  • p = .034, r = .48 for lower mental-demand ratings, while task focus did not differ significantly (p = .081); both medians favored MR-GLi.Mental-demand medians were 37.5 vs. 62.5; task-focus medians were 4.0 vs. 3.0.Source: F. Subjective Measures · p. 5

G. Discussion

The discussion reports improved access to visual feedback with comparable torque regulation, and the conclusions describe the approach as feasible while identifying further evaluation needs.

G. Discussion

  • MR-GLi improved access to visual feedback while maintaining torque-regulation performance comparable to the 2D monitor.Perceived gaze-shift ease improved, suggesting reduced burden when shifting attention between workspace and feedback.Source: G. Discussion · pp. 5–6
  • Although SUS and NASA-TLX did not differ significantly, both numerically favored MR-GLi; fourteen of twenty participants found it easier to use.Mental demand and task focus also tended to favor MR-GLi, without significant deterioration in overall usability or workload.Source: G. Discussion · p. 6

V. CONCLUSIONS

  • The counterbalanced comparison held controller and visual-feedback content consistent while showing comparable torque regulation and reduced perceived visual-access burden.Source: §V · p. 6
  • The results demonstrate feasibility for underwater bilateral teleoperation without degrading torque-regulation performance; future work will use direct gaze measurements and more challenging conditions.Source: §V · p. 6
Loading 2609.16041v1…