Source-linked AI summary
Hardware-Free Robotics Laboratories in Mixed Reality
Santiago Berrezueta-Guzman, Habiba-Loai Khalil, Andrei Koshelev, Vanesa Metaj, Stefan Wagner
TL;DR
Robotics education combines screen-based simulation with limited access to physical hardware, leaving students with fewer opportunities to inspect motion in a real-scale physical context. MR-Robotics LAB converts MATLAB trajectories for room-scale MR playback. In a formative evaluation, students reported positive perceived usability and multi-viewpoint inspection, but the study did not establish a comparative learning advantage.
Problem
Screen-based simulation presents robot motion apart from learners’ physical surroundings, while hardware cost, safety, and limited resources constrain access to physical robots.
Source: §1 · p. 1Method
MR-Robotics LAB converts MATLAB trajectories into a structured format and replays authored joint configurations at real scale in a physics-enabled MR scene.
Source: §1 · p. 1, §1.2 · p. 2, §3.3 · p. 6Results
M = 4.67 for perceived effort and M = 4.56 for inspecting motion from different viewpoints in the formative evaluation.
Source: §5.1 · p. 11, §5.2 · p. 12Takeaways & Limitations
The work provides a reusable interface between robotics simulation software and immersive visualization, with findings limited to participants’ perceived usability, understanding, and willingness to adopt the platform.
Source: §8 · p. 13The formative study involved 18 participants at one institution in one session, with MATLAB always preceding MR and no control group or objective performance measure.
Source: §7 · p. 13
Abstract
from arXiv · showhide
Teaching robotics relies on screen-based simulation, showing robot motion in an abstract coordinate frame rather than at real scale in the learner's own space, while access to physical hardware is limited by cost, safety, and scheduling constraints. We present MR-Robotics LAB, a mixed-reality (MR) platform that replays MATLAB-generated robot trajectories at real scale within the learner's physical environment. A browser-based service validates a MATLAB workspace file (.mat), normalizes units, and publishes a versioned JSON trajectory; a Unity application on a Meta Quest 3 then reproduces the authored joint configurations under position control and replays them at the declared frame rate within a physics-enabled scene that supports collision detection and end-effector grasping. A formative single-group evaluation with engineering students found that participants reported low setup effort (M = 4.67 on a 5-point scale) and perceived support for workspace understanding from multi-viewpoint inspection (M = 4.56), and 83% of participants affirmed their willingness to use the platform in an introductory robotics course. The evaluation instrument records only perceived outcomes, without counterbalancing or a learning measure, so no comparative advantage over desktop simulation is claimed. The contribution is a reusable simulation-to-MR trajectory pathway and design guidance for hardware-free robot visualization in engineering education.
1. Introduction
Screen-based simulation, hardware cost, and scarce laboratory resources constrain experiential robotics learning. MR-Robotics LAB brings MATLAB-authored trajectories into a room-scale, physics-enabled MR environment, with the introduction framing research questions around usability, spatial understanding, and course adoption.
1. Introduction
- Screen-based tools constrain real-world spatial understanding, while hardware cost, safety, laboratory infrastructure, and limited robot time restrict physical access.The introduction motivates situated learning by noting that spatial reasoning predicts STEM achievement and is malleable through spatially enriched training.Source: §1 · p. 1
- MR-Robotics LAB lets students upload MATLAB trajectory files, place a virtual robot in their surroundings, and inspect or interact with its room-scale motion.Playback preserves authored joint configurations and runs at the declared frame rate; the physics-enabled scene supports collision detection and end-effector grasping.Source: §1 · pp. 1–2
1.1. Research Questions
- The research questions ask about setup effort and usability obstacles, accessible spatial properties compared with screen visualization, and willingness to adopt the platform in an introductory course.They also ask what extensions students would request.Source: §1.1 · p. 2
1.2. Contributions
- A browser-based pipeline converts MATLAB trajectory files into a structured MR format without requiring MATLAB at conversion time, with secure, user-scoped storage.Source: §1.2 · p. 2
- The platform combines real-scale MR playback of MATLAB trajectories with collision detection and physics-based object manipulation.Source: §1.2 · p. 2
- Students can place multiple robot instances, inspect joint telemetry, and group robots to execute in parallel or sequence on a combined timeline.Source: §1.2 · p. 2
- The paper documents control-mode separation for prescribed kinematic playback in a physically reactive scene, including grasp-as-constraint and its consequences for observation.Source: §1.2 · p. 2
2. Related Work
Related work describes immersive media as a way to support visualization and spatial engagement, while noting interaction and scaffolding challenges. It positions MR-Robotics LAB against screen-based simulators and hardware-centered AR programming, and motivates its situated, low-complexity design through embodied cognition.
2.1. AR/VR/MR in Engineering Education
- Reviews report positive engagement and visualization outcomes for immersive learning, while usability and interaction complexity recur as barriers.Prior work links spatially enriched engineering tools with conceptual understanding and proposes robotics laboratories as a vehicle for spatial skill development; without structured scaffolding, immersive environments can add extraneous load and impair outcomes relative to slides.Source: §2.1 · p. 2
2.2. Robot Simulation Tools
- Desktop simulators render robot models on-screen, whereas prior AR robot-programming systems support in-workspace programming but target industrial workflows with physical robots [6].MR-Robotics LAB combines hardware-free trajectory playback, room-scale visualization, and physics-enabled object interaction in an instructional workflow based on student-authored MATLAB trajectories.Source: §2.2 · pp. 2–3
- Game-engine robotics work has focused mainly on reinforcement learning and sim-to-real transfer, while prior immersive kinematics work found value in inspecting motion from multiple perspectives.Source: §2.2 · p. 3
2.3. Embodied cognition and spatial learning
- The design uses real-scale placement in the learner’s environment, joint telemetry to connect trajectory values with motion, and simplified interaction to reduce extraneous load.This framing draws on embodied-cognition accounts of sensorimotor interaction and spatial reasoning [10].Source: §2.3 · p. 3
3. The MR-Robotics LAB platform
MR-Robotics LAB pairs a web conversion service with a Unity headset runtime to replay MATLAB trajectories at real scale in the learner’s physical environment. Its workflow validates and publishes trajectory data before students retrieve, place, and play it in the MR scene.
3. The MR-Robotics LAB platform
- The platform coordinates a web service that converts MATLAB workspaces into a self-describing trajectory contract with an MR runtime that reproduces trajectories at real scale.Source: §3 · p. 3
3.1. System Overview
- Students upload a MATLAB workspace, the web platform validates motion data and associates a robot model and gripper, then publishes standardized JSON to Firebase for headset retrieval.The headset filters trajectories by robot model; students place the robot, select a trajectory, and start playback.Source: §3.1 · pp. 3–4
- The robot appears at real scale in the student’s passthrough environment; supported models are six-DOF serial manipulators defined with URDF.The UR5e was included to check whether the URDF model pipeline generalized beyond one manufacturer.Source: §3.1 · p. 4
3.2. Core Technical Challenges
The design addresses inconsistent MATLAB inputs, unit and metadata mismatches, trajectory fidelity, physical interaction, and low-friction use. A browser pipeline validates and normalizes trajectories into a frame-based representation for MR playback.
3.2. Core Technical Challenges
- Challenges 1–4 concern detecting and validating trajectory data, normalizing units and interchange, enriching data for visualization, and browser-based cloud distribution; challenges 5–8 concern fidelity, physical realism, usability, and minimal setup.The web pipeline addresses challenges 1–4, while the MR application addresses challenges 5–8.Source: §3.2 · p. 5
3.3. Web-Based Trajectory Conversion
- The conversion service validates an unstructured MATLAB workspace, normalizes conventions, attaches metadata, and publishes a trajectory the runtime can treat as authoritative.It locates valid matrices by dimensions and name, prompting the student to choose when multiple candidates fit; robot metadata filters scripts to the placed model.Source: §3.3 · pp. 5–6
3.4. Canonical Joint and Gripper
- The converter represents joint motion as frames, converts arm angles from radians to degrees, and assigns optional binary gripper states and timestamps from the declared frame rate.Gripper state changes occur only at frame boundaries; constant gripper input yields an open state for all frames.Source: §3.4 · p. 6
3.5. Derived Kinematic Channels and Cloud Publication
The published trajectory includes derived kinematic channels for student-facing telemetry, alongside the authoritative motion data and metadata.
Derived kinematic channels
- Joint velocities are numerically differentiated from joint positions, using central differences internally and one-sided differences at endpoints, then lightly smoothed; accelerations are differentiated from smoothed velocities and clipped.These channels are derived for telemetry; playback pose remains driven by uploaded joint positions.Source: §3.5 · p. 6
- The per-joint torque proxy uses a simplified, decoupled model with generic nominal coefficients and is for qualitative visualization, not measured or validated actuator torque.The proxy does not affect the commanded trajectory.Source: §3.5 · p. 7
Cloud publication
- The completed record, frame sequence, derived channels, and metadata are published to user-scoped cloud storage, where the MR application retrieves the JSON as the authoritative motion description.Source: §3.5 · p. 7
3.6. Trajectory Fidelity and Motion Playback
The runtime reproduces the published trajectory while separating position-controlled robot motion from physics-driven interaction with the scene. Playback, collision handling, object manipulation, and controls are designed to keep execution inspectable.
3.6. Trajectory Fidelity and Motion Playback
- Every source frame follows the declared frame rate, with interpolation between frames for smooth rendering; gripper state changes only at programmed frames.Playback can pause and resume from the same point, and a collision halts execution until the robot is repositioned clear.Source: §3.6 · p. 7
3.7. Kinematic–Physics Coupling
- Position control preserves the student-authored joint configurations while the physics engine governs scene objects; grasping attaches an object, and collisions halt playback.The robot is driven by prescribed positions rather than forces, while the physics engine independently determines object motion and response.Source: §3.7 · p. 7
3.8. Collision detection and object manipulation
- The scene detects contact with physical and spawned objects; grasping uses articulated gripper states and attaches an object when finger motion stalls, while opening releases it.After a collision, students can reposition the robot and resume in place to inspect the trajectory segment.Source: §3.8 · pp. 7–8
3.9. Interaction Model and User Experience
- The interaction model supports automatic placement, repositioning by controller or hand tracking, model-filtered scripts, playback controls, and optional joint telemetry.A target-position preview supports checking the programmed grasp location against an instantiated object.Source: §3.9 · p. 8
3.10. Coordinated Multi-Robot Execution
The platform supports synchronized execution of user-grouped robots as well as a short upload-to-play workflow. Its design choices are tied to spatial context and reduced interface complexity.
3.10. Coordinated Multi-Robot Execution
- A user-defined robot system runs members in parallel or selection-order sequence, staging each at its trajectory’s first frame before synchronized execution and scrubbing.Sequential execution is intended to support hand-over scenarios; parallel execution makes temporal overlaps and potential spatial collisions observable.Source: §3.10 · p. 8
3.11. Minimal Setup and Accessibility
- The workflow reduces use to browser upload and headset model selection, with automatic parsing, unit conversion, cloud storage, and floor alignment; conversion requires no MATLAB installation.MATLAB remains necessary to author and export the original trajectory.Source: §3.11 · p. 9
3.12. Design Rationale
- Rendering the robot in the student’s physical environment reflects an embodied-cognition rationale, while minimizing configuration is motivated by concerns about extraneous cognitive load.The rationale cites for scale and context and for interface complexity.Source: §3.12 · p. 9
4. Evaluation method
The study used a formative, single-group session to characterize usability and perceived educational value, not to compare MR with desktop simulation. It collected one post-experience questionnaire after participants used both visualization approaches with the same trajectory.
4. Evaluation method
- The evaluation was formative and single-group, with no paired per-condition ratings to support a comparative test of desktop and MR.Each participant completed the MATLAB desktop phase before the MR phase and answered one questionnaire about the session as a whole.Source: §4 · p. 9
4.1. Participants
- Eighteen self-selected students participated, and uneven prior headset experience may have shaped usability ratings through both first-use costs and novelty.Five participants reported prior VR or MR headset use.Source: §4.1 · p. 9
4.2. Procedure
- Participants converted and replayed the same pre-authored MATLAB trajectory in MR, inspecting it from multiple viewpoints and exploring collisions, objects, and telemetry.The MR application ran on a Meta Quest 3, with the robot placed at real-world scale in the participant’s environment.Source: §4.2 · p. 10
4.3. Measures
- A single post-session questionnaire measured perceived usability and spatial understanding, but included no learning, task-performance, or timing measures.It primarily used seven five-point Likert statements and also collected willingness to use the platform and open-ended feedback.Source: §4.3 · p. 10
4.4. Analysis
- Seven Likert items were tested against the neutral midpoint with exact two-sided Wilcoxon tests and Holm correction; ceiling-heavy responses limit effect-size interpretation.The analysis also reported full response distributions and 95% BCa bootstrap confidence intervals; tests cannot establish that MR outperformed MATLAB.Source: §4.4 · pp. 10–11, Table 3 · p. 11
5. Findings
Participants reported positive workflow and spatial-understanding experiences, and most expressed willingness to use the platform in an introductory course. These are formative perceptions, not comparative or learning outcomes.
5.1. Workflow simplicity and usability (RQ1)
- Perceived effort scored M = 4.67 with 94.4% agreement, the highest rating; all four workflow-usability items were rated positively.Participants’ suggestions focused on interface refinements, including a tutorial, clearer prompts, and easier list scrolling.Source: §5.1 · p. 11
5.2. Spatial understanding (RQ2)
- Multi-viewpoint inspection scored M = 4.56 with 94.4% agreement, while connecting MATLAB values to visible behavior scored M = 4.22 with 72.2% agreement.The MATLAB-values item had the widest response dispersion (SD = 1.00); responses record perceived understanding and preference, not paired comparative ratings.Source: §5.2 · p. 12
5.3. Interest in robotics education (RQ3)
- Fifteen of 18 participants (83.3%) said they would test their own scripts on the platform in a future introductory robotics course.Three were undecided and none declined; whether interest persists beyond one session remains unknown.Source: §5.3 · p. 12
6. Discussion
The discussion positions MR as spatial verification alongside MATLAB, then identifies transferable design choices and practical issues. Several requested extensions were implemented after the formative evaluation but remain unevaluated.
6.1. What MR adds to screen-based simulation
- MR’s reported affordance is that viewers move around the robot while the surrounding room provides a metric reference, but this is not a demonstrated advantage.The discussion frames MR as spatial verification between MATLAB trajectory authoring and hardware execution, while MATLAB remains suited to generation, analysis, and debugging.Source: §6.1 · p. 12
6.2. Design implications
- Three proposed design choices are to preserve the existing authoring toolchain, require scale fidelity, and separate preprocessing from the headset.The discussion links scale fidelity to stable room anchoring and unaltered robot dimensions, and browser conversion to use across shared devices.Source: §6.2 · p. 12
6.3. Challenges and trade-offs
- Participants reported variable visual clarity and a need for controller familiarization; later multi-robot and UR5e additions have not been evaluated.It remains open whether grouped playback aids reasoning about multi-arm workcells more than independent playback.Source: §6.3 · pp. 12–13
7. Limitations
The findings are bounded by a small, single-institution formative study and by assumptions in trajectory conversion and runtime support. Comparative effects and broader system portability remain open.
Study design
- The 18-person, single-session study had no control group, always placed MATLAB before MR, and cannot separate order, novelty, or wording effects from reported experience.The design was intended to surface usability obstacles and design requirements, not estimate the medium’s effect.Source: §7 · p. 13
Trajectory conversion and mixed reality application
- Conversion requires a saved finite matrix with six joint columns and an optional gripper column; mobile bases and dual-arm systems require additional work.The runtime was developed and evaluated on a Meta Quest 3, and portability to other MR platforms remains unconfirmed.Source: §7 · p. 13
8. Conclusion
MR-Robotics LAB connects MATLAB-based robotics simulations to room-scale MR through a web-based conversion workflow and standardized JSON trajectories. Its formative evaluation found positive perceived usability and value, while learning or task-performance advantages over conventional visualization remain unestablished.
8. Conclusion
- The end-to-end workflow converts exported MATLAB workspaces into immersive MR experiences without modifying the existing simulation workflow.A standardized JSON trajectory format and web-based conversion platform bridge MATLAB simulations and MR visualization.Source: §8 · p. 13
- In a formative evaluation with 18 engineering students, participants reported little effort, perceived help understanding robot motion and workspace relationships, and broad interest in future course use.The conclusion characterizes the workflow as well received by this sample and specifies that findings concern perceived usability, understanding, and willingness to adopt.Source: §8 · p. 13
- The standardized JSON representation decouples trajectory generation from visualization, supporting additional robot models, simulation environments, and visualization platforms with relatively minor modifications.Source: §8 · p. 13
- Future work will expand support for arbitrary-joint robots, more flexible conversion, additional robotics software, real-time communication, and larger-scale longitudinal evaluation.Source: §8 · p. 13
- Whether the platform improves learning or task performance relative to conventional visualization remains to be established through controlled evaluation.Source: §8 · p. 13