Source-linked AI summary
A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot
Patrick Stotko, Stefan Krumpen, Max Schwarz, Christian Lenz, Sven Behnke, Reinhard Klein, Michael Weinmann
TL;DR
Video-based teleoperation confines operators to the robot camera’s view, limiting immersion, situation awareness, and navigation. The paper presents a live RGB-D-to-3D VR telepresence system for remote robot operation and scene exploration. The authors report improved situation awareness and navigation compared with video-based teleoperation.
Problem
Video-based teleoperation confines the operator to the camera view, limiting immersion and situation awareness and making navigation difficult.
Source: §I · p. 1Method
A robot captures RGB-D data that a SLAM-based system reconstructs into a scene model streamed to remote clients for VR exploration and teleoperation.
Source: §I · p. 2, §III · p. 3Results
The user study found higher situation-awareness ratings for VR-based teleoperation on self-localization, narrow-corner maneuvering, obstacle avoidance, terrain assessment, and view control than for video-based operation.
Source: B. Evaluation of User Experience · p. 6Takeaways & Limitations
The system supports immersive navigation and interactive object measurement in reconstructed scenes, capabilities the authors say video-only systems cannot provide at the same level.
Source: §VII · p. 8
Abstract
from arXiv · showhide
Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multi-client telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for e.g. head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to first-person exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot, complementing traditional input devices for teleoperation. We provide a proof-of-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization. Furthermore, we show its benefits over purely video-based teleoperation in a user study revealing a higher degree of situation awareness and a more precise navigation in challenging environments.
I. INTRODUCTION
Video-based teleoperation ties the operator’s view to the robot camera, limiting immersion, situation awareness, and independent navigation. This work presents a live-captured VR teleoperation system and evaluates its implementation and benefits over video-based operation.
I. INTRODUCTION
- The camera-bound view in video-only teleoperation limits immersion and situation awareness and makes navigation around obstacles and to previously observed areas difficult.The problem is especially challenging around narrow doors or corridors.Source: §I · p. 1
- The system reconstructs RGB-D data with real-time dense volumetric SLAM, manages the scene model on a server, and streams a low-bandwidth representation to remote clients.The robot-side system is self-contained, with optional interfaces for closer integration with existing robotic systems.Source: §I · p. 2
- The paper presents a VR system for robot teleoperation and scene exploration in live-captured environments, including contaminated places that people cannot directly access.The introduction frames this as a way for remote users to explore without being physically present.Source: §I · p. 2
- The paper evaluates the system through proof-of-concept scene-exploration experiments and a psychophysical comparison of immersive VR and video-based teleoperation.The authors report benefits for situation awareness and precise navigation in the user study.Source: §I · p. 2
II. RELATED WORK
Prior telepresence and robot-teleoperation systems faced constraints in real-time scene capture, scale, multi-client exploration, or camera-dependent viewing. This work builds on live 3D reconstruction and telepresence to support immersive exploration of robot-captured scenes.
Telepresence Systems
- Earlier telepresence systems struggled to reconstruct and transmit high-quality 3D models in real time, while commodity depth sensors brought room-scale reconstruction but left sensor noise and temporal inconsistency challenges.Holoportation achieved real-time performance with substantial hardware and camera-surrounded capture areas, limiting interactive exploration of larger scenes.Source: §II · p. 2
- Interactive VR exploration systems progressed from single-client streaming with reported bandwidth up to 175 MBit/s to server-maintained models that support multiple clients and reconnection after network interruptions.The server-based approach streams updates after reconnection, preserving the consistent model during outages.Source: §II · pp. 2–3
Robot-based Remote Telepresence
- Robot teleoperation commonly relies on camera video, so operators cannot explore independently of the camera view; a prior mobile-robot VR system used sparse point clouds, while this work captures surface-based mesh models.The cited point-cloud approach could make objects or walls appear to dissolve when approached, and required adapting common selection and manipulation operations.Source: §II · p. 3
Robot Platform
- Momaro’s immersive display increased situational awareness but used colorless registered point clouds alongside 2D images, while Centauro displayed live RGB-D data without aggregation.These systems provide context for the challenge of presenting accumulated scene data during exploration.Source: §II · p. 3
III. OVERVIEW
The overview describes a live-capture pipeline that reconstructs observed scene parts and streams them to remote exploration clients. Its architecture also supports multiple independent viewers alongside the teleoperator.
III. OVERVIEW
- The system links a robot moving through and capturing a scene to reconstruction, server management, and rendering by connected exploration clients.The five components are the robotic platform, optional robot client, reconstruction client, server, and exploration clients.Source: §III · p. 3
- An optional robot client provides information about the robot’s current posture, while exploration clients render the reconstructed scene and can be used for teleoperation.Rendering may be performed on head-mounted displays.Source: §III · p. 3
- The architecture allows multiple exploration clients to inspect the scene independently while one teleoperator maneuvers the robot.Remote users can communicate with one another, for example in disaster-management scenarios.Source: §III · p. 3
IV. ROBOT-BASED SCENE SCANNING
The section describes Mario-based mobile scene scanning with an RGB-D camera, supported by an omnidirectional drive, movable camera arm, and onboard power and data links.
IV. ROBOT-BASED SCENE SCANNING
- Mario is the mobile scanning platform, equipped with a Kinect v2 RGB-D sensor and notebook for processing and streaming the reconstructed scene.Its large footprint provides stability, while its design for high-speed outdoor use can make tight-space maneuvering difficult.Source: §IV · pp. 3–4, Figure 3 · p. 4
- The four-wheel omnidirectional base supports stable operation and precise positioning on mostly flat terrain; indoor exploration speed is limited to 0.15 m/s.The wheel motors can reach up to 4 m/s.Source: §IV · p. 4
- A UR5 arm moves the camera along Z-shaped trajectories to widen the field of view and improve model completeness; the Kinect v2 captures 512×424 RGB-D data at 30 Hz.The paper notes that lower-resolution, lower-frame-rate smartphone RGB-D sensors can also support sufficient reconstruction.Source: §IV · p. 4
- An eight-cell LiPo battery rated at 16 Ah and 29.6 V powers the robot and arm, with operation times up to 1 h depending on task intensity.Source: §IV · p. 4
- A WiFi router transmits scene data to clients, while a gamepad controls omnidirectional translation and rotation; the system does not integrate the available LiDAR.The teleoperation method must support third-person control, usable while standing beside the robot in reality or VR.Source: §IV · p. 4
V. LIVE TELEOPERATION AND EXPLORATION SYSTEM
The section introduces an efficient teleoperation system built from multiple components.
V. LIVE TELEOPERATION AND EXPLORATION SYSTEM
- The live teleoperation and exploration system is organized as a set of components for robot operation and remote scene exploration.Source: §V · p. 4
A. Reconstruction Client
The reconstruction client builds and streams a live 3D model from robot-captured RGB-D data, while the robot client supplies robot-link poses for immersive visualization.
A. Reconstruction Client
- The reconstruction client fuses RGB-D data into a dense 3D model using spatially hashed TSDF voxel blocks, then queues completed blocks for transmission.Visible active blocks are also queued when the robot stops or the session ends; blocks are progressively fetched, losslessly compressed, and streamed.Source: A. Reconstruction Client · p. 4
B. Robot Client
- The robot client transmits poses for all robot links because an actuated camera's pose alone cannot determine the poses of the robot components.For Mario, the data include the six-degree-of-freedom arm posture and wheel orientations.Source: B. Robot Client · p. 4
- Combining robot-link poses with the estimated camera pose enables immersive robot visualization; streaming additional sensors such as LiDAR is possible but outside this minimally invasive system's scope.Source: B. Robot Client · p. 5
C. Server
The server maintains a global model and streams scene updates to exploration clients; those clients can request regions, render the model, and refresh changed areas.
C. Server
- The server converts received TSDF blocks to a bandwidth-efficient Marching Cubes representation and manages updated-block streams for each connected client.On reconnection, it sends the complete block list if previously streamed parts may have been lost.Source: C. Server · p. 5
D. Exploration Client
- Exploration clients request scene parts by current viewing pose or arbitrary order, then reconstruct meshes with three additional levels of detail for rendering.The robot model uses estimated camera and component poses, with a low-pass filter smoothing the robot base pose.Source: D. Exploration Client · p. 5
- Clients can request updates for selected regions after scene changes, causing visible reconstructed parts to be deleted and the affected block list to be propagated.Source: D. Exploration Client · p. 5
VI. EXPERIMENTAL RESULTS
- The experimental evaluation includes a user study, a brief performance evaluation, and proof-of-concept demonstrations of remote scene interaction.Source: §VI · p. 5
- Fig. 4 depicts the reconstructed teleoperation scene, while Fig. 5 contrasts wide-angle-camera baseline operation with the proposed VR system.Source: Figure 5 · p. 5, Figure 4 · p. 5
A. Implementation
The implementation pairs a live reconstruction and exploration setup with a user study comparing VR and video teleoperation. Participants rated VR more highly on several navigation-related aspects, while video completion was faster and VR avoided more collisions.
A. Implementation
- The implementation used laptop reconstruction and server components plus a desktop exploration client, with HTC Vive visualization and WiFi between the computers.The reconstruction used a 5 mm voxel resolution and a 60 mm truncation region.Source: A. Implementation · p. 5
B. Evaluation of User Experience
- The evaluation compared VR exploration of the reconstructed scene with camera-view-only video, using a course task, 7-point ratings, collision counts, and completion time.Twenty participants each completed the task once in each mode, with mode order varied across participants.Source: B. Evaluation of User Experience · pp. 5–6
- VR ratings favored self-localization, narrow-corner maneuvering, obstacle avoidance, terrain assessment, and view control, with non-overlapping median–interquartile boxes on those aspects.The reported ratings also indicate that VR was well-suited to teleoperation and made moving the robot to target positions easier.Source: Figure 6 · p. 6, B. Evaluation of User Experience · p. 6
- Video users completed the course faster, while more collisions were avoided in VR.The authors attribute the extra time in VR to situation inspection and the collision difference to improved situation awareness.Source: Figure 7 · p. 6, B. Evaluation of User Experience · pp. 6–7
- VR was rated slightly worse than video for perceived latency and resolution; VR resolution was limited by voxel resolution.Scene data took a few seconds to stream, with the time depending on camera movement.Source: B. Evaluation of User Experience · p. 6
C. Performance Evaluation
The performance evaluation measured streaming bandwidth and visualized how the reconstructed model filled in as the robot scanned. The stream used a mean 14 MBit/s and a maximum 25 MBit/s, within typical Internet limits.
C. Performance Evaluation
- 14 MBit/s mean and 25 MBit/s maximum bandwidth were observed for streaming voxel data to the benchmark client, within typical Internet limits.The client requested 512-block packages at 100 Hz and discarded received data to avoid benchmark overhead.Source: C. Performance Evaluation · p. 7
- As the robot scanned additional regions, the streamed scene model progressively became more complete in the exploration client.The representation preserves the impression of solid objects and walls when viewed from varying distances, unlike point-cloud techniques.Source: C. Performance Evaluation · p. 7
D. Interaction of Remote Users with the Scene
The system supports controller-based measurements of 3D distances in the scene, including door dimensions relevant to selecting equipment. Measurements of corridor doors had errors of up to 1 cm, which the authors report as sufficient for rescue management.
D. Interaction of Remote Users with the Scene
- HMD controllers enable interactive 3D-distance measurement of door widths in contaminated-site exploration or evacuation, helping users assess whether a robot or required equipment can fit through a narrow space.Measurement accuracy is determined by voxel resolution, which is chosen according to RGB-D camera noise.Source: D. Interaction of Remote Users with… · p. 7
- Door-height and width measurements in the corridor had errors of up to 1 cm, which the authors considered sufficient for rescue management.The reported door ground-truth dimensions were 95 cm×215 cm and 174 cm×222 cm.Source: D. Interaction of Remote Users with… · p. 7, Figure 9 · p. 7
- Remote users can label map areas as interesting, suspicious, or incomplete, but this functionality was left for future development.The proposed labels could guide scan completion or refinement and interactions with robots, including equipment transport.Source: D. Interaction of Remote Users with… · p. 7
VII. CONCLUSION
The paper presents a robot-based live immersive teleoperation system for exploring places inaccessible to humans. Its demonstrations report scene exploration at acceptable bandwidth, improved immersion and situation awareness for precise navigation, and interactive object measurement.
VII. CONCLUSION
- The system enables interactive scene exploration at acceptable bandwidth and immersive teleoperation, with improved immersion and situation awareness for precise robot navigation and interactive object measurement.The authors state that video-only systems cannot reach this level of immersion and interaction.Source: §VII · pp. 7–8