PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

Yuzhu Sun, Mien Van, Nguyen Minh Nhat, Stephen McIlvanna, Sean McLoone

arXiv:2610.12140v1cs.ROcs.SYeess.SY

TL;DR

Changing workspaces can leave a digital twin out of sync, making policy updates necessary and potentially interrupting tasks. The paper combines a camera-synchronized hardware-in-the-loop digital twin with a dual-actor learner; with partial VR demonstrations that never reach the goal, it reports 83.3% and 100% evaluation success, versus 0% and 16.7% for two imitation-loss methods.

  • Problem

    Workspace changes can make a digital twin no longer reflect the physical scene, requiring updates that may interrupt task execution; direct physical-robot RL training is costly, potentially hazardous, and often impractical.

    Source: §I · p. 1

  • Method

    A camera-synchronized hardware-in-the-loop digital twin resumes RL training after workspace changes, while a separate behaviour-cloning actor guides the RL actor through the critic target without direct imitation loss.

    Source: 2) Dual Actor Framework · p. 6, §VI · p. 10

  • Results

    With partial VR demonstrations that never reach the goal, DA-SACBC reached 100% and 83.3% evaluation success, versus 16.7% for SACBC and 0% for SACBC + Q-Filter.

    Source: D. Retraining with Human VR… · p. 9

  • Takeaways & Limitations

    The framework was demonstrated on an Xarm5 robot for obstacle avoidance and retraining after workspace changes.

    Source: 1) State · p. 5, §VI · p. 10

    The current perception pipeline and one-episode look-ahead assume a quasi-static scene, limiting the framework’s treatment of moving obstacles during task execution.

    Source: §VI · p. 11

Abstract

from arXiv · show

Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.

I. INTRODUCTION

Collaborative robots need to adapt when workspaces change, but training directly on physical robots can be costly or hazardous and a twin may become outdated. The paper couples a synchronized digital twin with resumed RL training and human demonstrations, using separate RL and imitation actors to guide adaptation without an imitation loss on the RL actor.

I. INTRODUCTION

  • Workspace changes can leave a separately trained digital twin and policy out of step with the physical scene, potentially interrupting task execution.Training on physical robots is costly, potentially hazardous, and often impractical.Source: §I · p. 1
  • Camera feeds synchronize the PyBullet twin with the physical Xarm5, and RL resumes in simulation when a workspace change makes the current policy fail obstacle avoidance.The approach shifts updates from pre-deployment training toward adaptation during operation.Source: §I · pp. 1–2
  • VR demonstrations are transferred to the twin, where task rewards label transitions automatically, letting operators contribute task knowledge without manual reprogramming or specialist AI expertise.The same environment supports resumed RL training and preparation of human demonstrations.Source: §I · p. 1
  • The dual actor framework exceeded 80% mean final training success with non-optimal demonstrations, while two imitation-loss baselines rarely completed the task.The RL actor learns from digital-twin interactions, while the imitation actor learns from demonstrations; the critic target favours the path with higher estimated value.Source: §I · p. 1

A. Reinforcement Learning for Robotic Manipulation

  • Unlike prior collaborative-robot RL examples that train before deployment, this work investigates hardware-in-the-loop retraining when the digital twin detects an environmental change.The paper positions operator demonstrations as a way to replace robot reprogramming with guidance during retraining.Source: A. Reinforcement Learning for Robotic… · p. 2

B. Digital Twin

The paper situates its work at the intersection of synchronized digital twins and demonstration-guided RL. It distinguishes online policy retraining from prior approaches and uses a twin to label human demonstrations with task rewards.

B. Digital Twin

  • Unlike twins used mainly for synthetic data or pre-deployment training, the proposed synchronized twin feeds real-time environmental changes into the RL training loop.Prior work includes twins synchronized fully or partly, and an online replanning twin that does not update a learning agent during task execution.Source: B. Digital Twin · p. 2

C. Integrating demonstrations into RL

  • Prior demonstration-integration approaches include replay-buffer retention, IL or offline-RL pretraining, actor imitation losses, and controller selection; their training arrangements differ from retraining after a detected change.The cited discussion notes that the effects of non-optimal demonstrations were not evaluated in one replay-buffer study [40].Source: C. Integrating demonstrations into RL · pp. 2–3
  • The synchronized twin replays raw VR motion and assigns task rewards to transitions, enabling actor-critic learning from demonstrations without manual reward annotation.The same twin also supports RL training resumed during operation.Source: C. Integrating demonstrations into RL · p. 3
  • The proposed method uses a separate behaviour-cloning actor to propose actions, while the critic target selects between it and the RL actor without a direct imitation loss on the RL actor.The paper contrasts this design with methods that add an auxiliary imitation loss directly to the actor and with a Q-Filter that depends on critic accuracy.Source: C. Integrating demonstrations into RL · p. 3

III. DIGITAL TWIN ONLINE TRAINING FRAMEWORK

  • The framework section introduces hardware setup, perception, simulation, bidirectional data transmission, and the integrated digital-twin closed loop.The supplied section text names these components but does not elaborate their implementation here.Source: §III · p. 3

A. Digital Twin Setup

The setup links an Xarm5, depth camera, VR controller, and physical task objects to a PyBullet twin. Camera detections are localized in the simulated workspace through calibrated coordinate mapping.

1) Hardware:

  • The physical setup combines an Xarm5, ZED 2i depth camera, obstacles and goal objects, and an HTC VIVE system for operator-guided demonstrations.Camera and depth data locate objects, while the VR controller guides the virtual robot’s end-effector during demonstration collection.Source: 1) Hardware · p. 3

2) Software:

  • PyBullet implements the digital model as an OpenAI Gym environment, while ROS nodes and topics exchange data between physical and virtual components.The task defines states, actions, and rewards through a modular interface for compatible RL algorithms.Source: 2) Software · p. 3

B. Object Detection and Localization

  • YOLOv8 detections combined with ZED depth data provide object positions and heights, which are mapped from camera coordinates into PyBullet world coordinates.The detection model was trained on 400 workspace images and reported 99.4% mean average precision.Source: B. Object Detection and Localization · p. 3
  • The pixel-to-world mapping assumes a fixed camera, table-plane objects, and negligible residual lens distortion; moving or tilting the camera or stacking objects requires recalibration or a different transformation.The linear mapping uses per-axis scaling and offsets calibrated from the four workspace corners.Source: B. Object Detection and Localization · p. 4

C. Hardware-in-the-loop Training

The robot is pre-trained in the digital twin, then monitored during operation with a virtual preview before physical commands are sent. If the twin indicates failure, training resumes from the pre-trained model; the agent’s objective is to maximize cumulative reward.

C. Hardware-in-the-loop Training

  • The agent is pre-trained before deployment on the same goal-reaching and obstacle-avoidance objective, and the only stated requirement is stable success on the initial task.The pre-trained policy need not be optimal or generalize to unseen environments because later changes are handled by retraining.Source: C. Hardware-in-the-loop Training · p. 4
  • The virtual robot previews an episode about 1–2 seconds ahead, and physical joint commands are sent only after a safe, successful simulation; failure triggers updated camera observations, demonstrations, and resumed training.Retraining resumes from the pre-trained model until reward converges, after which the updated policy is tested in the twin.Source: C. Hardware-in-the-loop Training · p. 4

1) Markov Decision Process:

  • The Markov decision process represents the agent’s state, action, transition probability, reward, and discount factor, with the objective of maximizing cumulative discounted reward.A higher discount factor emphasizes long-term rewards, while a lower one tends to encourage immediate rewards.Source: 1) Markov Decision Process · pp. 4–5

2) Soft Actor Critic:

SAC maximizes both lifetime reward and policy entropy to encourage exploration. The task state records the goal, robot motion, obstacle dimensions, and robot–obstacle proximity.

2) Soft Actor Critic:

  • SAC maximizes lifetime rewards and policy entropy, with the temperature parameter controlling entropy’s importance.Its state-value, soft Q-function, and policy functions are represented by neural networks and optimized with separate objectives.Source: 2) Soft Actor Critic · p. 5
  • The paper identifies the three SAC objective functions as its core update rules and omits their implementation details.Further details are referred to in [51].Source: 2) Soft Actor Critic · p. 5

1) State:

  • The state includes time, goal position, obstacle size, the robot TCP’s position and velocity, and closest points between the robot and obstacles.These observations support reaching the cube while avoiding differently sized box obstacles.Source: 1) State · p. 5

2) Action:

The action space uses Cartesian motion commands, while the reward balances progress toward the goal against proximity to obstacles. The reported experiments disable the action-effort term.

2) Action:

  • Cartesian actions ease transfer to robots with built-in inverse kinematics, avoiding per-step inverse-kinematics calculations for Cartesian goals and obstacles.The alternative is defining actions in the robot’s joint space.Source: 2) Action · p. 5
  • The action vector contains bounded relative x, y, and z motions plus a gripper command, although reaching does not actuate the gripper.All four components are bounded to [−1, 1]; the gripper still enters the effort term when that term is active.Source: 2) Action · p. 5, 3) Reward Function · p. 5

3) Reward Function:

  • The reward combines goal-distance progress, action effort, and penalties for getting too close to obstacles.The clearance threshold is 0.1 m; the same proximity penalty applies independently to the wall and obstacle, and contact triggers further penalties for the rest of the episode. In the reported experiments, the effort weight is zero, so only goal progress and obstacle terms contribute.Source: 3) Reward Function · p. 5

C. Integrating Human Demonstrations

The framework integrates human demonstrations with reinforcement learning so operators can guide adaptation without manually reprogramming the robot. Demonstrations are collected by steering a virtual robot with VR controllers.

C. Integrating Human Demonstrations

  • The framework combines imitation learning with SAC while avoiding a direct imitation loss on the RL actor.The section introduces VR-based collection of human demonstrations as part of this integration.Source: C. Integrating Human Demonstrations · p. 6

1) Demonstration Collection:

  • Operators use a VR controller to guide the virtual robot around obstacles toward the goal instead of controlling the physical robot to collect data.The approach is described as a safe and intuitive way to collect demonstrations.Source: 1) Demonstration Collection · p. 6, Figure 5 · p. 6
  • The digital twin labels each demonstrated action with a reward, forming a transition of state, action, next state, and reward.The transition is represented as (s_t, a_t, s_t+1, r_t).Source: 1) Demonstration Collection · p. 6

2) Dual Actor Framework:

The dual actor framework keeps behavior cloning separate from the RL actor and lets demonstrations influence learning through critic targets. The update favors higher-value actor paths while retaining a weighted contribution from the behavior-cloning path.

2) Dual Actor Framework:

  • Demonstrations influence the RL actor through critic targets rather than a direct imitation loss, allowing the RL actor to explore beyond the demonstrator.A separate behavior-cloning actor proposes an alternative action; the critic evaluates both paths, and the bootstrapped target favors the higher estimated value while retaining weight v for the BC path.Source: 2) Dual Actor Framework · p. 6
  • The framework stores agent interactions and demonstrations in separate replay buffers, and trains the second actor only on demonstrations through behavior cloning.The two critics use the lower Q-value estimate to limit overestimation.Source: 2) Dual Actor Framework · p. 6
  • Both evaluated target forms used v = 0.2 and did not differ detectably.The alternative target discounts only a positive estimated advantage of the RL actor over the BC actor.Source: 2) Dual Actor Framework · p. 6

V. COMPARATIVE EXPERIMENTS AND DISCUSSION

The experiments use obstacle avoidance as a proof of concept for retraining an Xarm5 in a digital twin. They compare demonstration sources and imitation-learning approaches, including a Q-Filtered behaviour-cloning baseline.

A. Task Description

  • The Xarm5 must reach cube goals while avoiding box obstacles, and resumes training in the digital twin after a failed or unsafe attempt.The robot is represented in PyBullet, with inverse kinematics mapping actions to poses for its five degrees of freedom.Source: A. Task Description · p. 6
  • Four demonstration sets include deterministic and stochastic policy rollouts, plus complete and partial human VR demonstrations.The partial human trajectories pass the obstacle but do not reach the goal; the digital twin automatically labels each human transition with a reward.Source: A. Task Description · pp. 6–7, Table II · p. 7
  • The comparative Q-Filter baseline uses a behaviour-cloning loss filtered by whether a demonstration’s estimated value exceeds the actor’s.The filter is applied to demonstrations individually; without it, all demonstrations are included.Source: A. Task Description · p. 7

B. Agent Retraining

After a larger obstacle makes the pre-trained policy collide, retraining resumes from its saved checkpoint. With non-optimal demonstrations, the dual actor approach substantially outperforms methods that add imitation loss to the actor, while optimal demonstrations yield high success across methods.

B. Agent Retraining

  • After the workspace obstacle is enlarged, the pre-trained policy collides, and the ROS node launches retraining when it receives a task-failure flag.PyBullet supplies a safety flag to help prevent unsafe simulated attempts from reaching the physical robot; during training, the agent observes its shortest distance to the obstacle.Source: B. Agent Retraining · p. 7
  • With optimal demonstrations, every method’s mean final training success exceeds 90%.Methods resume from the same pre-trained checkpoint and replay buffer and train for 1.5 × 10^5 additional steps; Fig. 6 plots return and terminal safe-success rate.Source: B. Agent Retraining · p. 7, Figure 6 · p. 7
  • With non-optimal demonstrations, DA-SACBC’s mean success rises above 80%, while both methods adding an imitation term to actor loss remain below 8% throughout retraining; three of six seeds per target form meet the convergence criterion.Source: B. Agent Retraining · pp. 7–8
  • The authors caution that the non-optimal-demonstration comparison does not isolate the imitation-loss mechanism because the algorithms also differ in how demonstrations enter training.The baselines mix demonstrations into critic and actor mini-batches, while DA-SACBC trains its critic and RL actor on interaction data and uses demonstrations to train the BC actor.Source: B. Agent Retraining · p. 8

C. Quantitative Comparison

Across the tested settings, DA-SACBC achieves higher final success than imitation-loss baselines with non-optimal demonstrations, including human VR demonstrations that never reach the goal. Its advantage is not uniform across altered scenes: results depend on the geometry and goal location.

C. Quantitative Comparison

  • 83.3% and 100.0% mean deterministic evaluation success for DA-SACBC, versus 0.0% for SACBC and 16.7% for SACBC + Q-Filter, with non-optimal demonstrations.With optimal demonstrations, every method converges in at least five of six seeds and reaches comparable final success; none is shown to accelerate retraining relative to SAC. Evaluation uses 100 deterministic episodes per final policy.Source: C. Quantitative Comparison · pp. 8–9
  • Contact episodes count sampled finger contacts during training, not collisions of the final policy, and may depend on trajectories visited and time spent near the obstacle.The study reports means and standard deviations across six paired seeds; evaluation success is measured separately over 100 deterministic episodes per final policy.Source: C. Quantitative Comparison · pp. 8–9

D. Retraining with Human VR Demonstrations

  • With partial human VR demonstrations that never reach the goal, DA-SACBC reaches 100% and 83.3% evaluation success, versus 16.7% for SACBC and 0% for SACBC + Q-Filter.With complete human demonstrations, both DA-SACBC forms reach 100% evaluation success; the imitation-loss methods reach 33.3% and 50.0%.Source: D. Retraining with Human VR… · p. 9

E. Generalization to More Complex Scenarios

  • In altered scenes, results depend on geometry: DA-SACBC reaches 83% evaluation success with the pillar, ties SAC at 67% with the moved goal, and reaches 17% with the tall box, where SACBC reaches 100%.With partial demonstrations, DA-SACBC still exceeds SACBC in S3 (67% versus 33%) but remains behind it in S2.Source: E. Generalization to More Complex… · p. 9

F. Human Effort and Demonstration Quality

The section reports human-demonstration collection effort and performance, then summarizes the framework’s retraining results and boundaries. Partial VR demonstrations that never reach the goal still support high evaluation success, while moving obstacles remain an open challenge.

F. Human Effort and Demonstration Quality

  • One operator collected 10 accepted complete demonstrations from 230 attempts in 35 minutes and 10 partial demonstrations from 77 attempts in 10 minutes.Each attempt was limited to 50 steps, and accepted demonstrations met the set criterion without sampled wall or obstacle contact.Source: F. Human Effort and Demonstration… · p. 9
  • Raw NASA-TLX workload was 60.8 for the complete set and 46.7 for the partial set; unsuccessful DT trials posed no physical-robot collision risk.The partial set took fewer attempts and less time, but effort and workload were measured for one operator in one session.Source: F. Human Effort and Demonstration… · p. 10

VI. CONCLUSIONS

  • Mean final training terminal safe-success exceeded 80% with fixed non-optimal demonstrations, versus below 8% for two imitation-loss baselines.The final policies also had higher mean evaluation success and fewer sampled-contact episodes during retraining.Source: §VI · p. 10
  • With partial human VR demonstrations, mean deterministic evaluation success was 83–100%, even though the demonstrations omitted the precise final approach and never reached the goal.Source: F. Human Effort and Demonstration… · pp. 9–10
  • Adaptation may be limited when workspace changes fall outside the RL observations or camera coverage.The paper notes that usability across users requires a dedicated study because operator effort was measured for one person in one session.Source: §VI · p. 11
  • Moving obstacles exceed the current quasi-static, one-episode-look-ahead setup; future adaptation needs faster or predictive synchronization and motion-aware states and policies.A trajectory judged safe in the DT may be unsafe by physical execution, so contact-penalty rewards should be complemented by safe-RL mechanisms.Source: §VI · p. 11
Loading 2610.12140v1…