Source-linked AI summary
GHOST in the Robots: Real-Time Exocentric Dual-Robot VR Teleoperation from Onboard Cameras
Yichen Wei, Faisal Zaghloul, Soujanya C Aryal, Aanya K. Agrawal, Chengfan Li, Jason Xinyu Liu, James Tompkin, Stefanie Tellex
TL;DR
Single-operator control of multiple robots requires coherent sensing, manageable attention demands, and low-latency low-level commands. GHOST addresses this with an onboard-sensing exocentric VR workspace and mode-switching dual-robot control, improving success and completion time over a tablet baseline while enabling tasks infeasible with that baseline.
Problem
Existing multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands, while single-operator VR systems face spatial-awareness and attention challenges.
Method
GHOST uses onboard RGB-D sensing to build an aligned, depth-completed exocentric 3D workspace and provides mode-switching low-level control of two mobile manipulators individually or simultaneously.
Results
Across nine tasks, GHOST improved success and typically completion time; for experts, two tablet-infeasible tasks became completable and the remaining seven were 1.47× faster on average.
Takeaways & Limitations
GHOST demonstrates single-operator low-level dual-robot teleoperation using onboard sensing and enables multi-robot capabilities on existing hardware.
Takeaways & Limitations
The evaluation is preliminary because experts were system developers, novice participation was limited, and novices attempted only two tasks each.
Abstract
from arXiv · showhide
Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands. We present GHOST: an open-source VR teleoperation system that enables single operator control of two mobile manipulators via direct lowlevel commands using only onboard sensing. GHOST creates an exocentric 3D workspace by aligning real-time point clouds from the robots' RGB-D cameras, where scene coverage is improved through learning-based completion to aid operator spatial awareness. For control, the operator uses a mode-switching architecture to command either robot individually or both robots simultaneously. Experiments with 15 novice participants demonstrate 1.6-4x the success rate of an off-the-shelf tablet interface. For experts across nine challenging dual-robot tasks, our system enabled completion of two tasks that were infeasible with the tablet, and was 1.47x faster on average than the tablet. Website and code: https://h2r.github.io/GHOST/.
I. Introduction T
GHOST addresses the perception, attention, and control challenges of single-operator dual-robot teleoperation with an exocentric VR workspace and low-level control. It combines completed, aligned RGB-D point clouds with mode-switching control and reports improved success and speed over a tablet baseline.
- System motivation and contribution: GHOST enables one operator to control two mobile manipulators individually or simultaneously from an immersive virtual scene.
- System motivation and contribution: Real-time depth completion aligns sparse onboard RGB-D data into a coherent 3D scene, expanding the field of view and reducing blind spots.
- System motivation and contribution: Mode switching maps two hand controllers to navigation and manipulation across the robots while supporting simultaneous control with locked relative base poses.
- Evaluation: 1.6–4× higher success rate was achieved for 15 novices on two easier tasks relative to an off-the-shelf tablet interface.
- Evaluation: For three experts, GHOST enabled two tasks infeasible with the tablet, while the remaining seven tasks were completed 1.47× faster on average.
- System motivation and contribution: The paper positions GHOST as combining an exocentric 3D scene, low-latency depth completion, mode-switching control, and dual-robot evaluation tasks.
B. Depth Completion
The paper motivates depth completion as a way to address sparse RGB-D measurements while situating GHOST among multi-robot interfaces that trade autonomy, control granularity, and spatial awareness. GHOST targets real-time 3D awareness with low-level independent and synchronized control.
- Depth completion: Depth completion fills holes in sparse depth data using RGB images as guidance, addressing missing measurements from low-reflectance objects, multipath effects, and distant regions.
- Depth completion: PromptDA was selected as a ViT-based model because it balances prediction quality, generalization to novel environments, and interactive runtime needs.
- Teleoperation interfaces: Velocity-based devices can provide precise remote control, but they are less intuitive for mapping inputs to 6D pose.
- Multi-robot teleoperation: Existing interfaces span high-level autonomous systems, low-level multi-operator control, and coordinated single-operator strategies, with differing assumptions about maps, inputs, and attention.
- Multi-robot teleoperation: GHOST occupies a design-space point combining real-time 3D scene depictions with low-level multi-robot control.
- Multi-robot teleoperation: Unlike Body Extension, GHOST integrates 3D vision and coordinated base control for tasks such as cooperative carrying.
III. System Overview
GHOST integrates two Spot mobile manipulators, a Unity VR client, a ROS 2 server, and a GPU-accelerated RGB-D pipeline. The interface combines robot state, shared point clouds, close-up camera feeds, and predictive gripper targets.
- Hardware and architecture: Each Spot combines a quadruped base, 6-DoF arm, gripper, gripper camera, LiDAR, and body-mounted RGB-D cameras, with front cameras used for reconstruction.
- Hardware and architecture: The Unity VR client renders feedback and provides mode-switching control of either robot individually or both simultaneously.
- Hardware and architecture: ROS 2 manages low-bandwidth state streaming, command execution, localization, and synchronized navigation, while SpotObserver handles RGB-D processing on the VR client.
- Scene elements: Live robot models show each robot’s kinematic state, including arm and gripper configurations.
- Scene elements: Real-time point clouds combine both robots’ front-facing RGB-D views using robot pose estimates in the virtual scene.
- Scene elements: Floating gripper-camera panels support fine-grained grasping when front-facing cameras are occluded, while virtual gripper targets provide immediate intended-pose feedback during network and dynamics delays.
B. Vision Processing Pipeline
The vision pipeline is designed for low latency, shared-frame registration, and dense metric point clouds from sparse depth. It uses GPU processing, localization plus ICP alignment, and PromptDA depth completion.
- Pipeline requirements: The pipeline must achieve low latency, register both robots’ point clouds in a shared frame, and complete sparse noisy depth into dense metric point clouds.
- Latency reduction: Dedicated threads process each robot’s RGB-D stream and depth completion outside Unity’s render loop, while CUDA-DX12 interoperability avoids CPU-GPU copies.
- Latency reduction: 71 fps headset rendering and a 4.3 fps scene update rate were achieved, with latency dominated by sensor-frame delivery over Wi-Fi.
- Depth completion: The shared 3D scene is shown before and after real-time depth completion, with completion improving coverage and point-cloud coherence.
- 3D registration: Robot localization provides an initial inter-robot transform, which ICP refines using both robots’ LiDAR data and recomputes every 4 seconds.
- Depth completion: PromptDA cleans sparse noisy depth after nearest-neighbor prefilling, with prefilling and inference executed on the GPU through CUDA and ONNX Runtime.
A. Design
GHOST addresses the control and attention demands of operating two mobile manipulators by combining mode switching with coordinated dual-robot operation. Its evaluation tasks span bimanual manipulation, perceptual blind spots, and spatial context awareness.
- Control design: Five simultaneous control channels motivate GHOST’s mode-switching architecture for managing two robot bases, two arms, and an exocentric camera.The architecture presents task-relevant control combinations through the Meta Quest 3’s two hand controllers.
- Control design: GHOST separates robot selection from control mode, offering Fly, Drive, Arm, and Arm-Drive modes for single-robot operation.A distinct Dual-Robot mode supports coordinated operation.
- Evaluation design: Nine evaluation tasks cover bimanual manipulation, perceptual blind spots, and spatial context awareness, and all require coordinated control of two mobile manipulators.Tasks 1, 2, 5, and 9 involve bimanual manipulation; tasks 3 and 4 address blind spots; tasks 6, 7, and 8 demand spatial context awareness.
- Control design: Dual-Robot mode applies base commands to a shared formation pivot while keeping arm commands independent, enabling synchronized motion with two hands.This lets the operator switch between individual control and synchronized dual-robot motion.
B. Mappings
GHOST maps VR inputs to robot manipulation, navigation, and camera movement through relative 6-DoF control, virtual scene navigation, and synchronous dual-robot motion. The evaluation covers task categories requiring coordinated manipulation, multi-view perception, and spatial reasoning against a two-tablet baseline.
- Mappings: Relative 6-DoF controller motion maps to the robot end-effector, while a latch mechanism lets operators release and resume control.The system registers controller and end-effector poses when the trigger is held, then applies subsequent motion as relative displacement.
- Mappings: Fly mode translates or rotates a movable virtual base frame, letting operators navigate the exocentric scene without physically walking through the workspace.The operator’s head pose is tracked relative to this frame.
- Mappings: Synchronous navigation records initial robot poses and uses a formation pivot to coordinate base movement for jointly carrying shared objects.The pivot position is the centroid of robot positions, with yaw defined as their circular mean.
- Mappings: Each robot receives a waypoint derived from the updated pivot and its recorded offset, with commands sent through the Boston Dynamics API at 10 Hz.Arm controls remain independent across robots in synchronous mode.
- Evaluation mappings: The nine tasks span bimanual manipulation, perceptual blind spots, and spatial context awareness, while the baseline uses two tablets for simultaneous dual-robot teleoperation.The tablet is an egocentric, velocity-control interface.
B. Novice User Study
A within-subjects study evaluated GHOST against a tablet interface with 15 novice participants performing two coordinated dual-robot tasks. GHOST produced higher task success rates and higher average perceived usability scores, although the usability difference was not statistically significant.
- Study design: Two tablets are typically required for teleoperating two robots with the tablet interface, with operators switching between them.The study compared this baseline against GHOST.
- Study design: 15 novice participants with minimal VR and robotics experience attempted plush toy handoff and box pick-and-place with both interfaces.Each participant received 10–15 minutes of familiarization before testing, with interface order counterbalanced.
- Measures: Success rate was measured alongside completion time and System Usability Scale scores.Participants completed the SUS after using each interface.
- Results: 1.6× the tablet’s success rate was achieved on plush toy handoff, and 4× on box pick-and-place, with comparable or faster completion times among successful trials.These were the two novice evaluation tasks.
- Results: 64.8 ± 19.2 versus 52.8 ± 22.4 out of 100 was the average SUS score for GHOST versus the tablet, but the difference was not statistically significant.The reported Wilcoxon signed-rank test gave p=0.16 and d_z=0.34; the authors note the low n.
C. Expert User Study
An expert study compared full GHOST, RGB-only GHOST, and the official tablet interface across nine dual-robot tasks. Full GHOST achieved the highest feasibility and was faster than the tablet on tasks where both interfaces succeeded, while the reconstructed 3D scene particularly supported spatially demanding tasks.
- Study design: Three expert participants attempted all nine tasks with full GHOST, RGB-only GHOST, and the tablet interface.Each participant performed three primary trials with up to two retries after failures.
- Measures: Task success defined feasibility, while completion time was compared only when an expert did not fail more than three of five attempts.Infeasible tasks were excluded from timing comparisons.
- Results: 26/27 feasible participant-task conditions were achieved by full GHOST, compared with 20/27 for the tablet baseline.Eight of nine tasks were feasible for all three experts with full GHOST; bedsheet folding was feasible for two of three.
- Results: 1.47× average per-task speedup was achieved over the tablet on the seven tasks where the tablet succeeded.Full GHOST was slower on successful weighted-door passthrough trials, although feasibility was 3/3 versus 2/3.
- Ablation: 22/27 feasible participant-task conditions were achieved by RGB-only GHOST, with failures concentrated in tasks requiring coordinated spatial reasoning.The authors suggest direct 6-DoF control provides much of the timing benefit, while reconstructed 3D improves reliability and spatial awareness when accurate.
D. Discussion
GHOST’s advantages arise from coordinated dual-arm control, direct hand-pose mapping, predictive gripper feedback, and a unified 3D scene, with reconstruction quality shaping its benefits.
- Evaluation: Full GHOST is faster and more reliable than the tablet on most expert tasks, while two tasks were infeasible with the tablet.RGB-only GHOST generally falls between the tablet and full GHOST, but can match or exceed it when point-cloud quality is poor.
- Coordinated control: GHOST’s coordinated bimanual mapping helps tightly synchronized tasks, while offering little benefit on sequential tasks.Tablet operators alternated between arms, contributing to failure on bedsheet folding; simultaneous control helped little on weighted-door passthrough or cube collection.
- Intuitive high-precision and rotation control: Direct hand-pose mapping improves precise alignment and rotation-intensive manipulation compared with joystick-based velocity control.Fine spatial adjustments directly command end-effector pose, whereas velocity commands make precise placement and orientation cumbersome.
- Virtual gripper target: The virtual gripper target previews commanded pose, reducing perceived latency and improving operator confidence.During tool use, it previews gripper motion but not the tool tip, creating a perceptual mismatch; VR nevertheless outperforms the tablet for tool use.
- 3D representation tradeoffs: 22/27 to 26/27 completions: the reconstructed 3D scene raised completion reliability over RGB-only GHOST, with marginal completion-time gains.The unified spatial reference supports reasoning about object pose, inter-robot alignment, and spatial constraints, but inaccurate reconstruction can reduce efficiency.
VII. Conclusions
The paper concludes that GHOST supports single-operator, low-level dual-robot manipulation using onboard sensing and improves outcomes over an egocentric tablet baseline. The evaluation remains preliminary because of developer participation and limited novice coverage.
- Conclusion: GHOST enables a single operator to perform low-level dual-robot mobile manipulation using only onboard sensing.The system uses an exocentric VR interface for mobile manipulators.
- Conclusion: GHOST improved success rate and typically completion time relative to the egocentric tablet baseline, including tasks infeasible with that baseline.This conclusion summarizes both novice and expert studies.
- Limitations: The user study is preliminary because expert participants developed the system and few novices each attempted only two tasks.Larger studies with independent experts and more novice participants are proposed to characterize modality trends.
- Future work: Future directions include behavior-cloning models, larger multi-robot teams, autonomous shared control, and more precise, consistent 3D reconstruction.These directions are stated as extensions of the collected data, system scale, autonomy, and scene understanding.
Appendix
The appendix provides task descriptions in two tables, divided into Part 1 and Part 2.
- Task descriptions: Table IV lists task descriptions for each task in Part 1.
- Task descriptions: Table V lists task descriptions for each task in Part 2.