PaperlayerView exact revision on arXiv

Source-linked AI summary

Location-Independent Robot-Assisted Finishing Using Digital Twins and Extended Reality

Jose Outeiro, Jia Holt, Tero Kaarlela, Khalil Chakal

arXiv:2609.09061v1cs.HC

TL;DR

Small-batch AM finishing remains largely manual and time-consuming, and reviewed systems do not combine location-independent teleoperation with immersive cross-platform access and RAF-specific process feedback. This paper presents a human-in-the-loop CPS using an XR-enabled digital twin for remote finishing supervision and control. In laboratory validation, the system achieved a maximum steady-state joint synchronization error of 0.12° and average round-trip latency of 563 ms.

  • Problem

    AM finishing remains largely manual and time-consuming, and no reviewed system combines location-independent teleoperation, immersive cross-platform access, and process-level feedback specific to the RAF system.

    Source: A. Motivation · p. 1, E. Research gap · p. 2

  • Method

    The paper proposes a human-in-the-loop CPS using an XR-enabled digital twin and MQTT for location-independent programming, supervision, training, and teleoperation of RAF.

    Source: B. Scientific Contribution and… · p. 1, E. Communication · p. 3

  • Results

    0.12° maximum steady-state joint synchronization error and 563 ms average round-trip latency were measured in the validated system.

    Source: §V · p. 6

  • Takeaways & Limitations

    Surface-topography visualization supports operator decisions to reposition a part during finishing or terminate the process.

    Source: §V · p. 6

    The laboratory prototype's 563 ms average latency limits operation to a move-and-wait regime rather than continuous motion streaming or direct manual control.

    Source: D. Performance · p. 5, §V · p. 6

Abstract

from arXiv · show

This paper presents a cyber-physical system (CPS) for location-independent programming, supervision, training, and teleoperation of a Robot-Assisted Finishing (RAF) system used to post-process metal additive-manufactured (AM) components. A digital twin (DT) built in Unity is delivered to the operator as a WebGL application that supports both desktop and immersive modes through WebXR-compatible devices. Moreover, it exchanges robot state and pose commands with a collaborative robot through a Message Queuing Telemetry Transport (MQTT) broker. The DT enforces kinematic and collision constraints before a pose is released to the physical robot, and augments the virtual component with a color map of the surface topography that supports operator decisions on part repositioning or process termination. The architecture was validated on a specially designed physical RAF system. A steady-state joint synchronization error of 0.12 deg and a mean round-trip latency of 563 ms were measured, which is adequate for supervisory programming and intermittent teleoperation.

I. INTRODUCTION

For small-batch and lot-size-one AM, finishing remains largely manual and time-consuming, while operators must make process decisions in noisy, difficult-to-observe conditions. The proposed human-in-the-loop CPS combines remote teleoperation with human decision-making to address this setting.

A. Motivation

  • AM finishing remains largely manual and time-consuming, creating a need for automated and flexible methods to improve overall AM profitability.The stated benefits of AM depend strongly on part geometry, material, and batch size.Source: A. Motivation · p. 1, §I · p. 1
  • About 85 dB(A) was recorded during finishing, and continuous operator presence near the cell is undesirable despite the need for supervision and process decisions.Source: A. Motivation · p. 1
  • Flexible robot control for small-batch AM should be operable by personnel with limited robotics expertise to reduce process time and manual labor.Source: A. Motivation · p. 1
  • A human-in-the-loop CPS combines remote robot teleoperation with human decision-making, since full autonomy may not be economically feasible for low-volume AM finishing.Operators can adapt finishing strategies to component geometry and process behavior.Source: A. Motivation · p. 1

B. Scientific Contribution and Research Questions

The work brings an XR-enabled digital twin and MQTT communication into a human-in-the-loop finishing system for location-independent programming, supervision, teleoperation, and training. It targets remote control performance and the process information needed for remote monitoring.

B. Scientific Contribution and Research Questions

II. STATE-OF-THE-ART

The related work frames the paper around digital-twin data exchange, robot programming constraints, XR interfaces, and abrasive finishing. It identifies the need for an operator-in-the-loop approach that supports remote adaptation and process awareness.

A. Digital twins and cyber-physical systems

  • Digital twins are distinguished from digital models and digital shadows by automated bidirectional data flow between the physical and digital entities.Grieves introduced the digital-twin concept for product lifecycle management.Source: A. Digital twins and cyber-physical… · p. 2

B. Industrial robot programming

  • Teach-pendant programming requires robot-specific expertise and physical presence, while handguiding still interrupts production and offline paths need adaptation for lot-size-one parts.When a surface is sufficiently finished depends on in-process observation rather than a fixed program.Source: B. Industrial robot programming · p. 2

C. Extended reality for robot programming and teleoperation

  • Reviewed XR robot interfaces span teleoperation, virtual path design, DT programming, and online learning,,,,, but address generic tasks rather than manufacturing-process information.Source: C. Extended reality for robot… · p. 2

D. Robot-assisted abrasive finishing

  • Centrifugal disk finishing improves as-built metal AM surface finish, but contact count alone does not validly proxy material removal.Simulated contact quantities have been correlated with roughness reduction, with predicted local differences where media accessibility is limited.Source: D. Robot-assisted abrasive finishing · p. 2

E. Research gap

The reviewed systems do not combine location-independent teleoperation, an immersive cross-platform interface, and process-specific feedback for the RAF system. The described DT models the physical RAF equipment and provides operator controls and alternative viewpoints.

E. Research gap

  • No reviewed system combines location-independent teleoperation, immersive cross-platform access, and process-level feedback specific to the proposed RAF system.Table I compares reviewed systems against those targeted capabilities.Source: E. Research gap · p. 2, Table I · p. 2

A. Physical manufacturing system

  • The physical RAF setup pairs a six-degree-of-freedom UR30 robot with a centrifugal-bowl RF-50 finisher and a three-finger pneumatic gripper.The robot has a 1.3 m reach and 30 kg payload; the bowl has a tilt mechanism for media discharge.Source: A. Physical manufacturing system · p. 2

B. Digital Twin

  • The Unity DT provides robot jog, gripper, waypoint recording, simulation, and playback functions, plus virtual viewpoints and the option to hide bowl walls.Its 3D CAD model uses manufacturer dimensions and robot kinematics based on the ROS industrial package.Source: B. Digital Twin · pp. 2–3

C. Classification of the system

The robot layer is a full digital twin with automatic state updates and validated pose commands flowing back to the robot. The twin converts operator-specified Cartesian motions into joint positions while enforcing kinematic and collision constraints.

C. Classification of the system

  • The robot layer is a full DT, with joint and Cartesian states flowing automatically to the twin and validated pose commands returning to the robot.The bowl is not fully instrumented, so the twin represents system kinematics rather than abrasive-media interaction physics; process topography is transferred only at intervals.Source: C. Classification of the system · p. 3

D. Kinematics and pose validation

  • An iterative damped least-squares solver converts operator-specified Cartesian motions into joint increments.The damping factor trades tracking accuracy against conditioning near singularities; the solver degrades gracefully at singular and out-of-reach targets.Source: D. Kinematics and pose validation · p. 3
  • Unity articulation joints enforce kinematic limits, while collision elements cover the gripper, robot arm, pedestal, bowl, part, and finishing media.Source: D. Kinematics and pose validation · p. 3

E. Communication

The system uses MQTT for robot-state and pose-command exchange, with sequence numbers and timestamps to reject stale or out-of-order commands. Safety combines digital-twin validation, independent robot-controller checks, and a light curtain that inhibits remote motion when a local person may enter the cell.

E. Communication

  • MQTT carries robot commands and state; sequence numbers and timestamps let subscribers reject stale or out-of-order commands.The end-to-end latency is dominated by the robot controller polling interval, not the broker.Source: E. Communication · p. 3

F. Safety architecture

  • The twin blocks invalid poses, while the robot controller independently rejects commands outside joint limits and Cartesian safety planes.The controller checks help prevent malformed, replayed, or spoofed messages from moving the robot beyond the safeguarded volume.Source: F. Safety architecture · p. 3
  • A light curtain inhibits remote command execution because a local person may enter the safeguarded space without knowing remote motion is pending.Source: F. Safety architecture · p. 3

G. Cybersecurity

The DT interface exposes robot control and pose recording for programming, teleoperation, and training. Its cloud architecture hosts the WebGL runtime and MQTT broker, while cybersecurity measures restrict encrypted access and command-topic publishing.

G. Cybersecurity

  • The server uses TLS-encrypted connections, per-user broker authentication, and command-topic publication restrictions; the robot controller connects outbound and accepts no inbound connections.Source: G. Cybersecurity · pp. 3–4

IV. RESULTS AND DISCUSSION

  • The DT supports programming, teleoperation, and training through robot controls, pose recording, and program execution.The interface includes viewpoint controls, TCP and joint control, simulation controls, and a display of physical-twin joint positions.Source: §IV · p. 4, Figure 2 · p. 4

A. Architecture

  • The cloud server hosts the browser-run WebGL twin and MQTT broker for bidirectional updates between the DT and RAF system.The DT publishes commands and subscribes to robot state; the robot controller polls desired positions and tool variables.Source: A. Architecture · p. 4

B. Process-awareness visualization

A height map of the component’s surface topography helps operators identify less-exposed areas and decide whether to reposition the part or terminate finishing. The validation workflow combines human decisions with robotic execution and permits programming only for collision-free end positions.

B. Process-awareness visualization

  • Surface-topography height maps help operators identify less-exposed areas, reposition or rotate the part, and decide whether target roughness has been reached.Topography is measured offline at fixed intervals using 3D Coherent Scanning Interferometry.Source: B. Process-awareness visualization · p. 4, Figure 4 · p. 4

C. Validation

  • The pick-and-place validation task integrates approach, pick, placement, finishing, retraction, and gripper control into the interface.Robot positions can be programmed only when the twin validates the chosen end positions as collision-free.Source: C. Validation · p. 4
  • The system architecture depicts the headset, optional controllers, collaborative robot, centrifugal bowl, gripper, and tilt mechanism.Source: Figure 3 · p. 5

D. Performance

The CPS supports pose-by-pose remote programming and intermittent teleoperation, but its latency precludes continuous motion control. Remote supervision also requires process information such as surface topography, beyond robot kinematics.

D. Performance

  • 0.12° maximum steady-state synchronization error and 563 ms average round-trip latency were measured; these support offline programming and intermittent, but not continuous, teleoperation.The 563 ms latency is dominated by the controller polling loop; the operator cannot react to an unexpected event during motion faster than the round-trip delay.Source: D. Performance · p. 5, E. Response to the research question · p. 5, Table II · p. 5
  • The 563 ms latency precludes reactive corrections while guiding a tool into contact, although poses can be recorded and verified one at a time.Replacing controller polling with an event-driven interface is identified as the most effective latency improvement.Source: D. Performance · p. 5, E. Response to the research question · p. 5

E. Response to the research question

  • Remote supervision of centrifugal disk finishing requires process information such as surface topography, beyond the robot’s kinematic state, to guide actions including process termination.The finishing process may be terminated when the target surface roughness is reached.Source: E. Response to the research question · p. 5

V. CONCLUSIONS AND FUTURE WORK

The CPS combines human decision-making through an XR-enabled digital twin with robotic execution to support finishing of low-volume AM components. A laboratory prototype demonstrated remote finishing, while future work includes process sensing, latency reduction, and usability evaluation.

V. CONCLUSIONS AND FUTURE WORK

  • The CPS combines human decision-making through an XR-enabled digital twin with robotic execution for finishing small-batch and lot-size-one AM components.The system supports location-independent programming, teleoperation, and operator training while reducing exposure to harsh working conditions.Source: §V · pp. 5–6
  • Surface-topography visualization supports informed repositioning during finishing or process termination.The authors identify in-situ metrology as future work to measure topography without removing the part from the gripper and reduce total finishing time.Source: §V · p. 6
  • A laboratory prototype demonstrated remote programming and finishing, with 0.12° maximum steady-state joint synchronization error and 563 ms average round-trip latency.The prototype was piloted with a small variety of VR headsets and in desktop mode; it is not yet available for production use.Source: §V · p. 6
  • Future work includes process sensing, replacing controller polling with an event-driven interface, and a user study with non-expert participants.The user study is needed to substantiate the interface’s usability claims.Source: §V · p. 6
Loading 2609.09061v1…