PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmented Reality Interfaces for Human-Robot Collaboration: Development of a ROS 2-Based Sensor Streaming Framework and Validation via SLAM Algorithms

Alessandro Rubert, Stefano Ghidoni, Matteo Terreran

arXiv:2609.31396v1cs.RO

TL;DR

Communication between commercial AR headsets and ROS 2 systems is identified as a practical bottleneck. This thesis develops a Magic Leap 2 sensor-streaming bridge and validates its data with SLAM, reporting trajectories close to physical displacements with mean translational errors of 4–9 cm on medium-range paths.

  • Problem

    Communication between commercial AR headsets and ROS 2-based robotic systems is a practical bottleneck that this thesis aims to overcome.

    Source: §6.1 · p. 85

  • Method

    The thesis develops a software bridge that acquires and streams data from the headset’s integrated sensors to ROS 2.

    Source: §6.1 · p. 85

  • Results

    Mean translational errors were 4–9 cm across medium-range paths, and estimated trajectories closely mirrored physical displacements.

    Source: §6.1 · pp. 85–86

  • Takeaways & Limitations

    Internal tracking confirmed the quality of the transmitted sensory data, supporting the framework’s use as a basis for real-time interaction.

    Source: §6.1 · p. 85

    The module is ill-suited to applications demanding continuous high-frequency streaming because its actual publishing rate remains constrained.

    Source: Limitations and potential issues · pp. 49, 52

Abstract

from arXiv · show

In recent years, Human-Robot Collaboration (HRC) has taken on a central role in Industry 4.0 and collaborative robotics, demanding communication channels that are increasingly bidirectional, intuitive, and efficient. In this context, Augmented Reality (AR) presents itself as a fundamental enabling technology, capable of both displaying information to the operator and gathering spatial data about the surrounding environment. This thesis presents the development of a sensor streaming framework that connects the Magic Leap 2 AR headset with the ROS 2 (Robot Operating System) ecosystem. Using the Unity development environment and the ROSTCP-Connector package, an on-board application for the headset was developed, capable of acquiring real-time data from the integrated sensors (pose tracking, cameras, and environmental sensors) and publishing it to dedicated ROS 2 topics. In order to test the accuracy, latency, and robustness of the generated data stream, the framework was validated using SLAM (Simultaneous Localization and Mapping) algorithms known in the literature. The experimental results demonstrate that the proposed architecture ensures stable data transmission, laying the groundwork for safe real-time interaction and shared spatial awareness, and opening up new perspectives for the control and supervision of robotic systems in complex HRC scenarios.

4.2 Example acquisition of greyscale frames originating from the three ambient

This section illustrates grayscale imagery from the headset’s ambient cameras and depth sensor, then presents trajectory estimates alongside ground-truth comparisons and related plots.

4.2 Example acquisition of greyscale frames originating from the three ambient

  • The ambient-camera examples distinguish the conventional video stream (Stream 0) from the mixed-reality stream (Stream 1), which overlays interface holograms.The camera passage names World Left and World Center; the depth examples show short- and long-range modes and metric Depth32 encoding.Source: §4.2 · p. 11
  • The depth-sensor examples cover short- and long-range Depth32 frames and short-range raw infrared intensity.Source: §4.2 · p. 11

5.1 3D reconstruction of the estimated trajectory compared against the ground truth,

  • The trajectory figures compare estimated motion with ground truth and include position components, Euler angles, translational APE, and an XY-plane projection.Source: §5.1 · p. 12

Introduction

The work addresses the need for bidirectional, transparent communication in human-robot collaboration by developing a scalable Magic Leap 2–ROS 2 sensor-streaming framework and validating its data with monocular ORB-SLAM3 against the headset’s native pose estimates.

1.1.1 Hard automation. Industry 3.0

  • Early industrial robots repeated programmed operations without perceiving their environment and were kept in enclosed cells to prevent collisions with operators.The passage describes these robots as robust, heavy machines used for arduous, hazardous, and repetitive tasks.Source: §1.1.1 · p. 13

1.1.2 Flexible and collaborative automation. Industry 4.0

  • Dynamic markets make hard automation inefficient and expensive, prompting a shift toward quickly reprogrammable flexible automation and collaborative robots.Unlike earlier hard-automation lines, which were dedicated to particular products and segregated for safety, cobots can be adapted and reconfigured for new processes.Source: §1.1.1 · p. 13, §1.1.2 · p. 14

1.1.3 Human-Robot Collaboration (HRC)

  • HRC combines human flexibility and decision-making with cobot precision, but safe, efficient interaction requires bidirectional communication and mutual transparency about actions and intentions.Robots must anticipate human actions and adapt trajectories, while operators need to anticipate robot movements to avoid collisions, reduce cognitive load, and build trust.Source: §1.1.3 · p. 15

1.2.1 Augmented reality (AR)

  • AR overlays digital elements on the physical environment, enriching the user’s sensory field rather than replacing the real world as VR does.The passage describes AR devices such as headsets and smartphones as displaying elements including 3D images and holograms.Source: §1.2.1 · pp. 15–16

1.2.2 AR as a interface whith robotics

  • AR headsets can act as a bidirectional bridge: their cameras and depth sensors track the operator and environment, while robot status and intended motion appear in the operator’s view.This approach is presented as avoiding cumbersome external infrastructure.Source: §1.2.2 · pp. 16–17

1.2.3 The lack of native communication

  • Magic Leap 2 offers advanced spatial tracking and environmental perception but lacks native integration with ROS 2, motivating an intermediate communication middleware.The proposed bridge extracts headset data, especially video streams, and converts and transmits it to the robotic system for tasks such as localization and trajectory planning.Source: §1.2.3 · p. 17, Operating System) · p. 17

1.3.1 The developed framework

  • The project delivers a ready-to-use, scalable software framework bridging Magic Leap 2 and ROS 2 through modular network cooperation.Its architecture is designed for extension with new functionalities and data-processing modules.Source: §1.3.1 · p. 17, sensors · p. 18
  • ROS 2 configuration selects enabled sensors and stream resolution, and the headset streams only requested data to avoid unnecessary bandwidth and processing load.The computer-side node reads the operational parameters from a YAML configuration file.Source: §1.3.1 · pp. 17–18
  • The project was released in a public GitHub repository to support replication, testing with other robotic systems, and further enhancement.The stated aim is to foster scientific reproducibility and open-source collaboration.Source: sensors · p. 18

1.3.2 Validation

  • ORB-SLAM3 validation confirmed correct data routing, reduced transmission latency, and compatibility with robotic state-estimation algorithms.Magic Leap 2 world-camera video was processed in monocular mode and compared with headset-native pose estimates on short indoor trajectories.Source: §1.3.2 · p. 18

1.4 Thesis outline

  • The thesis moves from the state of the art and enabling technologies to framework implementation, experimental validation, and concluding remarks with future directions.The outline identifies bidirectional interaction and advanced collaborative robotics as future research directions.Source: §1.4 · p. 19

State of the art

The state of the art covers AR for HRC and wearable robotics, alongside headset sensing and streaming. It identifies a gap in modular Magic Leap 2 sensor streaming to ROS 2 and describes a bridge intended to address it.

2.1 Augmented reality in human-robot collaboration (HRC)

  • HRC must balance operator safety with interaction fluidity; passive robot stops protect operators but penalise productivity.Source: §2.1 · p. 21
  • AR-based sharing of safety zones and robot intentions reduced idle time by over 50% and execution time by 20–24% versus non-collaborative workstations.Source: §2.1 · p. 21
  • A HoloLens 2 and haptic interface framework uses aerial gestures for null-space posture changes and teaches parametric trajectories modelled with DMPs.Source: §2.1 · pp. 21–22

and exoskeletons

  • Assistive exoskeletons based on predefined walking patterns can behave rigidly in unstructured environments; CFFTG adapts step height and length in real time from RGB-D point clouds.Source: and exoskeletons · p. 22
  • Pelvis- or foot-mounted sensors have restricted downward views and short obstacle-detection range; an egocentric headset-camera approach merges maps from parallel ORB-SLAM3 streams and DUSt3R matching.Source: and exoskeletons · p. 22

headsets

  • HoloLens 2 Research Mode exposed low-level camera, ToF-depth, and inertial streams through C++ APIs [9].Source: headsets · pp. 22–23

2.4 The technological gap and the role of Magic Leap 2

  • The Magic Leap 2 ecosystem lacked an open-source modular ROS 2 sensor-streaming tool; the thesis proposes a Unity/native-Android bridge with negotiated transmission parameters and visual-localization validation.Source: §2.4 · p. 23

Technologies used

This section describes the technologies used to build and validate a Unity-based framework for streaming Magic Leap 2 headset data to ROS 2. It covers the headset and its sensing and development capabilities, the SDK and TCP communication bridge, ROS 2 infrastructure, and SLAM systems including ORB-SLAM3.

Technologies used

  • The framework streams Magic Leap 2 data to ROS 2 through a Unity headset application using ROS-TCP-Connector, with stream quality and synchronization checked using ORB-SLAM3.Source: Technologies used · p. 25

3.1 Magic Leap 2

  • The Magic Leap 2 is an optical see-through AR headset that lets users view the surrounding environment directly while virtual objects remain spatially anchored.Source: §3.1 · pp. 25–26

3.1.1 Hardware configuration and input devices

  • Processing and battery reside in a cabled compute module rather than the glasses frame; a tracked controller supports interaction, while diffractive waveguides project holograms into the real-world view.Source: §3.1.1 · pp. 26–27

3.1.2 Sensors

  • The headset combines inertial, world-tracking, RGB, depth, eye-tracking, and ambient-light sensors for spatial tracking, scene capture, depth reconstruction, gaze, and lighting assessment.Source: §3.1.2 · pp. 27–28

3.1.3 Operating system

  • Magic Leap OS, based on Android/AOSP, supports native C/C++ and Java development and Unity integration while managing distributed processing, optical calibration, and localization.Source: §3.1.3 · p. 28

3.1.4 Magic Leap SDK

  • The SDK supports Unity/OpenXR, Unreal Engine 5, and native C/C++; its features include spatial mapping, eye and hand tracking, and a 6-DoF controller.Source: §3.1.4 · p. 29
  • The Magic Leap SDK exposes headset camera, depth, IMU, microphone, and ambient-light data for application use.These hardware feeds support advanced analysis or custom applications.Source: §3.1.4 · p. 29

3.2.1 Modular architecture

  • ROS 2 uses DDS for decentralized communication among task-specific nodes, with topics, services, and actions supporting distinct exchange patterns.Topics are asynchronous publish-subscribe channels; services use request-response, while actions support long-running tasks.Source: §3.2.1 · p. 30

3.2.2 Coordinate frames and transforms

  • ROS 2’s TF infrastructure represents relationships between coordinate frames as a tree, enabling composite transforms and coordinate conversion.Each tree edge represents a transform, and the structure provides a unique path between frames.Source: §3.2.2 · p. 31

3.2.3 Tools

  • RViz 2 visualizes live topic data, while rosbag2 records and replays node data streams for later analysis, including without the hardware present.RViz 2 can display TFs, point clouds, and images.Source: §3.2.3 · p. 31

3.3 Unity; 3.3.1 Component-based architecture and execution loop

  • Unity 2022.3.62f3 LTS served as the Magic Leap 2’s runtime, not a simulator, rendering holograms and managing the sensor-acquisition application through modular components.Unity represents virtual elements as GameObjects with attached components, and application logic is implemented in C# scripts.Source: §3.3 · p. 32, §3.3.1 · p. 32

3.4 ROS-TCP-Connector and ROS-TCP-Endpoint; 3.4.1 Client-server architecture

  • The ROS-TCP-Connector and ROS-TCP-Endpoint form a TCP/IP client-server bridge between the Unity headset application and ROS 2 nodes on a workstation.The Unity client maintains a socket connection; the ROS 2 server receives and deserializes streams, then instantiates corresponding publishers and subscribers.Source: §3.4 · p. 33, §3.4.1 · p. 33

3.4.1 Client-server architecture

  • The bridge requires the headset and ROS 2 workstation to be reachable on the same subnet, such as through local Wi-Fi.The passage describes this as enabling flexible deployment without complex hardware constraints.Source: §3.4.1 · p. 33

3.4.2 Serialization and message generation

  • ROS message definitions are converted into Unity-usable C# classes, then serialized, sent over TCP, deserialized by the endpoint, and published on ROS 2 topics.Examples include sensor_msgs/Image for visual frames and geometry_msgs/PoseStamped for spatial coordinates.Source: §3.4.2 · pp. 33–34
  • Unity’s ROS 2 communication operates symmetrically when Unity subscribes to topics published by ROS 2.Source: §3.4.2 · p. 34

3.5.1 SLAM algorithms (Simultaneous Localization and Mapping)

  • SLAM builds an unknown-environment map while estimating real-time 6-DoF pose; visual odometry tracks keypoints, back-end optimization addresses drift, and loop closure realigns the global map.The back-end minimizes reprojection error, while loop closure corrects accumulated trajectory drift.Source: §3.5.1 · p. 34

3.5.2 ORB-SLAM3

  • ORB-SLAM3 supports real-time visual, visual-inertial, and multi-map SLAM on standard CPUs, integrating inertial sensors for rapid motion or temporary visual occlusion.Its Atlas system continues through tracking loss by creating a local sub-map and merging it with mapped areas when revisited.Source: §3.5.2 · pp. 34–35

Framework design and implementation

This section describes the design and implementation of a modular Unity framework that streams Magic Leap 2 sensor data to ROS 2. It covers configurable sensor acquisition and publishing, stream characteristics and constraints, synchronization, and extension of sensor and AR features.

4.1 Functional requirements and constraints

  • The design began with analysis of application goals and the hardware and computational limits of embedded devices and wireless transmission.Source: §4.1 · p. 37

4.1.1 Functional requirements

  • The bridge was required to acquire headset video, pose, and auxiliary sensor data, serialize it as standard ROS 2 messages, synchronize camera calibration messages, and support runtime stream configuration.The configuration can set enabled sensors, resolutions, and frame rates through a YAML file parsed at startup, avoiding repeated Unity/Android recompilation.Source: §4.1.1 · pp. 37–38

4.1.2 Project and system constraints

  • The architecture aligned topic names and optical-frame conventions with common robotic vision sensors, limited payload and TCP serialization overhead, and used independent sensor scripts for extensibility.The component-based design was intended to allow future feeds such as eye or hand tracking without rewriting communication logic or interfering with verified modules.Source: §4.1.2 · p. 38

4.2 Framework architecture and parameter negotiation

  • The framework decoupled ROS 2 control from headset execution through two ROS 2 nodes—a communication endpoint and configuration node—and coordinated Unity scripts.Its initial stage dynamically negotiated and configured transmission parameters.Source: §4.2 · p. 39

4.2.1 System components and negotiation workflow

  • At startup, ROS 2 loads YAML parameters; the headset requests them, receives them as JSON, assigns them to sensor modules, then activates only enabled streams.Modules wait for their enablement, resolution, and frame-rate parameters before beginning continuous publishing.Source: §4.2.1 · p. 39

4.2.2 Structure of the configuration file (config.yaml)

  • The config.yaml file centralizes topic prefix, sensor enablement, supported stream types, image resolution and format, and publishing rates, allowing load and bandwidth choices without rebuilding the application.For example, the default topic prefix is /magic_leap2, and supported settings can be tailored to the receiving algorithm.Source: §4.2.2 · p. 40

4.2.3 Component-based architecture and execution model in Unity

  • In Unity, a central GameObject is associated with dedicated, independent C# scripts for each sensory stream, including cameras.Source: §4.2.3 · p. 43
  • Unity sensor modules isolate acquisition processes and launch asynchronous frame publishing when enabled, while pose updates continue independently to keep TF updates unconstrained by frame rate.Sensors not handled through standard Unity APIs, including IMUs and ambient-light sensors, are queried through a native Kotlin plugin.Source: §4.2.3 · p. 43

4.3 Implementation and characterisation of sensory modules

  • The bridge publishes the headset’s ambient tracking cameras, front RGB camera, depth sensor, four eye-tracking infrared streams, IMUs, and ambient-light readings to dedicated ROS 2 topics.The auxiliary sensor readings include linear acceleration, angular velocity, and external illuminance.Source: §4.3 · pp. 43–44

Description and operation

  • Each of the three ambient tracking cameras captures 1016×1016 greyscale images at 30 Hz, producing approximately 250 Mbit/s of raw data per camera.The cameras are wide-FOV sensors optimized for low latency and spatial analysis.Source: Description and operation · p. 45

ROS 2 topics and messages

  • Each independently enabled ambient camera publishes image_raw, camera_info, and metadata topics under /magic_leap2/<name>/, with corresponding transforms registered only when that module is enabled.The camera messages provide acquired frames, calibration information, and sensor metadata.Source: ROS 2 topics and messages · pp. 45–46

Spatial transformations (TF Tree)

  • Camera poses are published on /tf at 60 Hz relative to magicleap_world, with separate ROS-convention and optical-convention frames for each enabled sensor.The optical frame uses Z forward along the optical axis, X right, and Y down.Source: Spatial transformations (TF Tree) · p. 46

Limitations and potential issues

  • Three ambient cameras maintained a stable 30 Hz stream with negligible jitter, while their concurrent activation generated approximately 750 Mbit/s of network load.The reported load can saturate bandwidth available on conventional Wi-Fi networks.Source: Limitations and potential issues · p. 46
  • To avoid Wi-Fi saturation, the framework advises selectively enabling only the camera of interest, such as World Center, through config.yaml.The recommendation addresses the aggregate load produced by enabling all three uncompressed, high-resolution camera streams.Source: Limitations and potential issues · p. 46

4.3.2 High-resolution RGB front camera (Picture Center)

  • The front RGB camera supports standard optical and mixed-reality streams, offers nominal rates of 30 or 60 Hz, and compresses frames onboard as JPEG to constrain bandwidth.The mixed-reality stream overlays headset-displayed graphics and holograms onto the camera view.Source: Description and operation · pp. 47–48

Spatial transformations (TF Tree)

  • Camera and ToF poses are published on /tf at 60 Hz relative to magicleap_world, using ROS-convention and optical frames.The camera frames are color_frame and color_optical_frame; the depth frames are depth_frame and depth_optical_frame.Source: Spatial transformations (TF Tree) · pp. 48, 49, 52

Limitations and potential issues

  • High-resolution capture and continuous JPEG compression raise headset temperature; thermal throttling degrades frame rate over approximately ten minutes, making the module ill-suited to continuous high-frequency streams.It is better suited to periodic visual inspection, static-frame capture, or short mixed-reality supervision sequences.Source: Limitations and potential issues · p. 49

Description and operation

  • The sensor has fixed 544 × 480-pixel resolution and provides Depth32 32-bit floating-point depth maps and DepthRaw 32-bit infrared-intensity data; Long Range spans 0.5–5.0 m with centimetre-level accuracy, while Short Range offers millimetre-level accuracy.Source: Description and operation · pp. 50–51

ROS 2 topics and messages

  • Depth32 and DepthRaw configuration selects corresponding ROS image, camera-info, and metadata topics; eye-camera topics are registered only for enabled modules.The depth topics include /magic_leap2/depth/image_raw and /magic_leap2/infra/image_raw; eye cameras publish monochrome frames and metadata under /magic_leap2/<name>/.Source: ROS 2 topics and messages · pp. 51, 52, 54

Limitations and potential issues

  • Although the Short Range API accepts a selectable 60 Hz setting, its actual publishing rate is capped at 30 Hz; eye-camera APIs expose neither calibration intrinsics nor geometric poses.The missing eye-camera calibration prevents standard CameraInfo topics, while missing camera-to-headset transformations prevent dedicated /tf frames.Source: Limitations and potential issues · pp. 52, 55

4.3.4 Eye tracking cameras and gaze estimation

  • The headset integrates four monochrome infrared cameras directed towards the user’s eyes for ocular tracking and gaze-direction estimation; camera traffic amounts to approximately 153.6 Mbit/s.An individual camera yields a theoretical throughput of approximately 38.4 Mbit/s.Source: Description and operation · p. 54

Limitations and potential issues

  • Four video streams have low latency and a stable acquisition rate of 30 Hz, making them viable for external image processing and pupil-tracking pipelines.The cited passages also note informational constraints regarding spatial metadata.Source: Limitations and potential issues · p. 55

Description and operation

  • The native plug-in retrieves hardware-timestamped sensor readings after standard Magic Leap SDK APIs proved unable to query all three IMUs concurrently or expose original sample timestamps.It interfaces with SensorManager and registers dedicated listeners for the accelerometer, gyroscope, and light sensor; Unity's C# script queries it at runtime.Source: Description and operation · pp. 55–56

ROS 2 topics and messages

Limitations and potential issues

  • IMU sampling is approximately 40 Hz, constant and without significant jitter, while authentic hardware timestamps synchronize it with visual streams for sensor fusion and VIO.The passages attribute the rate limit presumably to operating-system HAL decimation or aggregation, and contrast it with typical inertial sensor maxima above 100–200 Hz.Source: Limitations and potential issues · p. 56

4.4.1 ROS 2 environment setup

  • The bridge is available as a precompiled APK, so end users can deploy it without reopening Unity or recompiling; the ROS 2 workspace uses config_ml2_stream and ros_tcp_endpoint.Users can configure sensor enablement and parameters such as frame rates, streaming modes, and target resolutions before building the workspace with colcon.Source: §4.4 · p. 57, §4.4.1 · p. 57

4.4.2 Application installation and setup on Magic Leap 2

  • The Magic Leap 2 app is installed from an APK, and requires a host at 192.168.0.203:10000 on the same subnet plus Camera, Spatial Mapping/Perception, Sensors, and Local Network permissions.The passages say missing required permissions can immediately crash or freeze low-level acquisition routines.Source: §4.4.2 · p. 58

4.4.3 Start-up sequence and data streaming

  • Startup requires launching the TCP endpoint and configuration server before the headset app, which requests configuration and then publishes continuously for enabled sensors.The app sends a handshake on /request and receives a JSON configuration payload before acquisition coroutines are unblocked.Source: §4.4.3 · pp. 58–59

4.5 Modularity and upgrades

  • The modular, decoupled framework supports network reconfiguration and new sensor streams or AR capabilities; modifying its source and rebuilding an APK requires Unity.Examples of potential additions include hand tracking and on-device computer vision.Source: §4.5 · p. 59, §4.5.1 · p. 59

4.5.1 Unity development environment setup

  • The Unity setup configures Magic Leap 2 OpenXR, imports the project scripts and Android plug-in, and installs the ROS-TCP-Connector with ROS 2 protocol and host settings.The native plug-in setup includes the Kotlin standard-library dependency.Source: §4.5.1 · pp. 59–60

4.5.2 Guidelines for code extension and upgrades

  • The extension pattern adds YAML parameters to AppConfig and dispatches them to target module properties before setting their activation flags.At startup, JSON parsing populates AppConfig from config.yaml.Source: §4.5.2 · pp. 60–61
  • New sensor modules or AR features can use decoupled C# scripts with activation flags, ROS 2 data sampling and serialization, and optional Android plug-in access.Modules can publish in Unity’s Update loop or via an asynchronous coroutine; the native plug-in supports access to low-level hardware telemetry or metadata.Source: §4.5.2 · p. 61

Experimental Validation and Results

This section evaluates the ROS 2 camera stream through SLAM across tested trajectories, assessing temporal synchronization, spatial accuracy, and orientation tracking against the headset reference. The results describe close temporal alignment and orientation tracking, route-dependent translational error, and retained path topology during extended monocular runs despite progressive scale error.

5.2.1 Rationale for trajectory data segmentation

  • For validation, monocular ORB-SLAM3 used World Camera streams one at a time; analysis retained continuous sustained-tracking segments and excluded runs interrupted by tracking failures.The stated validation objective was to check stream fidelity, including packet loss, jitter, or timestamp skew that could corrupt motion estimation.Source: §5.1 · p. 63, §5.2.1 · p. 64

5.2.2 Error metrics and trajectory alignment

5.3.1 Validation on World Center Camera — Trajectory 1

  • The 367 matched pose pairs had a maximum temporal discrepancy below 0.01 s (Δtmax ≤10 ms), with no systematic middleware latency or timestamp drift.This synchronization result is from the first World Center acquisition.Source: §5.3.1 · p. 67

Analysis of the absolute pose error profile

  • RMSE was 4.40 cm, mean error 3.89 cm, and standard deviation 2.06 cm; error stayed below 5 cm for almost the entire acquisition and reached about 3.6 mm at minimum.These translational APE statistics are reported for the World Center Camera — Trajectory 1.Source: Analysis of the absolute pose error… · p. 68
  • The APE peak was approximately 12.97 cm during initial monocular map initialization and near the abrupt direction change at about 9 s.The passage associates the peak with initial Bundle Adjustment convergence and the change of direction.Source: Analysis of the absolute pose error… · p. 68
  • The estimated horizontal x(t) and y(t) trajectories closely followed reference curvature changes and decelerations, while z(t) fluctuations stayed within ±2 cm.The vertical variation is characterized as consistent with subtle walking oscillations.Source: Trajectory alignment and Cartesian… · pp. 69–70

Orientation estimation (roll, pitch, yaw)

  • Yaw tracking nearly overlapped the reference over a rotation exceeding 200° (+120° to −105°), while roll and pitch mean angular discrepancies were below 1.5°–2°.The roll and pitch differences are attributed to monocular ORB-SLAM3 versus the headset’s visual-inertial fusion.Source: Orientation estimation (roll, pitch,… · p. 70

5.3.2 Validation on World Center Camera — Trajectory 2

  • The second World Center session was a different indoor path lasting about 10.9 s and 6.311 m estimated distance (5.960 m by the headset reference), with 305 temporally synchronized poses.The bridge streamed 1016 × 1016-pixel monochrome video at 30 Hz alongside the headset poses; no second-session error result is supplied in this part.Source: §5.3.2 · p. 71
  • For Trajectory 2 (World Center), translational APE via Sim(3) Umeyama alignment yielded RMSE 7.46 cm, mean µ = 6.81 cm, median 6.21 cm, and dispersion σ = 3.05 cm.During the central run, error remained below 8 cm and reached a minimum of 2.61 cm.Source: Table 5.2 · p. 71, Analysis of the absolute pose error… · p. 72
  • On Trajectory 2, x(t) and y(t) tracked straight and curved path segments, while z(t) oscillated by a few centimetres without divergent drift or loss of metric consistency.Uniquely matched pose pairs had maximum temporal discrepancy below 0.01 s, supporting stable timestamp propagation.Source: §5.3.2 · p. 71, Analysis of the absolute pose error… · pp. 73–74

Orientation estimation (roll, pitch, yaw)

  • Yaw progressively rotated beyond 220° (from −120° to over +105°) and was tracked with excellent accuracy and continuous overlap relative to headset ground truth; roll and pitch followed the Magic Leap-recorded dynamic profile.Source: Orientation estimation (roll, pitch,… · p. 74

5.3.3 Validation on World Left Camera — Trajectory 3

  • The run yielded 612 uniquely matched pose pairs with maximum temporal discrepancy ∆tmax ≤10 ms, and the estimated trajectory closely traced the loop.Source: §5.3.3 · pp. 75, 77
  • For Trajectory 3 (World Left), translational APE under Sim(3) Umeyama alignment reached RMSE 32.59 cm, with mean 27.65 cm, median 23.14 cm, and dispersion 17.24 cm.As the longest and kinematically most challenging trajectory evaluated, its elevated absolute error relative to prior runs reflects expected drift accumulation and subtle scale uncertainties characteristic of monocular SLAM.Source: Table 5.3 · p. 75, §5.3.3 · p. 76
  • On Trajectory 3, translational error stayed below 15 cm from t ≈8–17 s, reached 6.47 cm, and rose to ≈61.09 cm during initial convergence and the return-path closing segment.The closing error was associated with an extended odometric chain and perspective shifts.Source: §5.3.3 · p. 76

5.3.4 Validation on World Right Camera — Trajectory 4

  • The monochrome video feed streamed at full framerate alongside headset poses, yielding 438 uniquely matched pose pairs with maximum temporal discrepancy ∆tmax ≤10 ms and no cumulative middleware latency.Source: §5.3.4 · p. 79
  • Trajectory 4 yielded translational APE RMSE = 9.66 cm, mean µ = 7.46 cm, median 4.75 cm, and dispersion σ = 6.14 cm.After the initial transient, error stayed below 5 cm for nearly the entire session (t > 8 s), with minima around 1.05 cm; the ≈28.19 cm peak occurred during initialization and was mitigated by local bundle adjustment.Source: Table 5.4 · p. 79, §5.3.4 · p. 80
  • On World Right Camera Trajectory 4, z(t) briefly fluctuated around 7.0–7.4 s but recovered steady tracking, with deviations bounded within a few centimetres and no divergence.Source: §5.3.4 · p. 82

Orientation estimation (roll, pitch, yaw)

  • Yaw tracked a continuous ≈200° rotation with exceptional overlap, while roll and pitch reproduced head oscillations with mean angular discrepancies of 1.5°–2.5° attributed to estimator differences.Source: Orientation estimation (roll, pitch,… · p. 82

5.3.5 Final considerations on the experimental results

  • Across trials, the framework streamed continuous, coherent ROS 2 data with precise synchronization and no perceptible latency or packet drops; visual streams enabled trajectory reconstruction with remarkable fidelity to native tracking.Source: §5.3.5 · p. 83
  • Medium-range estimates from the central and lateral right cameras closely mirrored physical displacements, with mean discrepancies of a few centimetres and accurate directional tracking.Source: §5.3.5 · p. 83
  • Head rotations and gaze inclinations were reproduced throughout testing without temporal skew or phase lag during frame ingestion.Source: §5.3.5 · p. 83
  • Closed-loop left-camera acquisition preserved path topology, although error grew toward the end from monocular absolute-scale estimation over distance, without tracking failure or filter divergence.Source: §5.3.5 · p. 83

Conclusions and Future Work

The thesis bridges Magic Leap 2 and ROS 2 with a configurable sensor-streaming framework, then validates its data using monocular ORB-SLAM3 and headset tracking. It concludes with three proposed directions for extending the framework toward HRC applications.

6.1 Conclusions

  • The Magic Leap 2–ROS 2 bridge streams integrated headset sensors, with launch-time configuration of sensor selection, resolution, and framerate.Configuration is handled through a YAML file and dedicated ROS 2 node, without source changes or recompilation.Source: §6.1 · p. 85
  • Mean translational errors were 4–9 cm across medium-range paths, with deviations predominantly attributed to monocular SLAM’s intrinsic scale drift.The headset’s tracking was substantially more accurate, using multiple cameras, IMUs, and an active depth sensor.Source: §6.1 · p. 86
  • On closed-loop trajectories exceeding 12 metres in approximately 23 seconds, the video stream remained unbroken and free from pipeline stalls.Head motion and roll, pitch, and yaw were reproduced with high fidelity without perceptible latency.Source: §6.1 · p. 86

6.2 Future work

  • Future work proposes a ROS 2-to-Unity downlink to display computational decisions, manipulator trajectories, and safety volumes as context-aware holograms.This direction is framed as robotic intent visualization and bidirectional interaction.Source: §6.2 · p. 86
  • The framework could support hand tracking and gesture recognition for issuing manipulator commands, selecting 3D targets, or teleoperating collaborative robots.The proposed interactions are intended to supersede conventional industrial teach pendants.Source: §6.2 · p. 86
  • A proposed egocentric-perception extension would combine head-mounted vision with exoskeleton sensors for 3D reconstruction and terrain-aware gait planning.The envisioned applications include mobile robots and lower-limb exoskeletons.Source: §6.2 · p. 87
Loading 2609.31396v1…