PaperlayerView exact revision on arXiv

Source-linked AI summary

A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration

Alexandra Coroiu, Andrea Vogt, Viktor Werbilo, Andreas Poppele, Johann Christensen, Sven Hallerbach

arXiv:2610.09891v1cs.HCcs.AIcs.RO

TL;DR

RLHF for HRC faces challenges in safety during AI development, feedback quality, and human-robot adaptation. The paper reviews bidirectional methods and tests feedback timing in VR, finding that user-initiated feedback better captures perceived safety.

  • Problem

    RLHF for HRC faces practical challenges involving safety during AI development, uncertain and limited human data, and bidirectional human-robot adaptation.

    Source: §1.3 · p. 2, §1.2 · p. 2

  • Method

    The paper conducts a scoping review of RLHF in HRC and a VR experiment comparing system- and user-initiated binary feedback.

    Source: §3.1 · p. 9, §4 · p. 13

  • Results

    User-initiated feedback better captures psychological safety and is more sensitive to physical safety violations despite lower feedback volume.

    Source: §3.4 · pp. 12–13

  • Takeaways & Limitations

    Feedback timing is key to AI safety because human-data quality depends on immediate feedback after a perceived safety threat.

    Source: §4 · p. 13

    The review’s search terminology may have omitted relevant reports because RLHF terminology is not yet standard across HRC and HITL research.

    Source: §2.4.2 · p. 8

Abstract

from arXiv · show

Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics.

1. Introduction

HRC pairs automation’s efficiency with human flexibility, but shared workspaces create safety and adaptation challenges. The introduction motivates RLHF and closed-loop design as ways to integrate human input into robot learning.

1.1. Background

  • HRC combines automation efficiency with human flexibility, supporting varied tasks and smaller customised batches.Mobile robots support navigation and transport, while robotic arms support pick-and-place, manipulation, and assembly.Source: §1.1 · p. 1

1.2. Motivation

  • Shared workspaces raise collision and contact risks, while collaborative robots must balance safety requirements with system performance and human factors.Relevant human factors include trust, workload, stress, and perceived safety; existing robots often rely on predefined rules rather than real-time human data.Source: §1.2 · pp. 1–2

1.3. Problem Statement

  • RLHF can refine learned robot tasks from human feedback, but safe HRC development still faces simulation-to-reality, data-quality, and human-robot co-adaptation challenges.The paper addresses these challenges through a review of bidirectional methods and a VR comparison of feedback conditions.Source: §1.3 · pp. 2–3

2.1. Research Questions

The review focuses on RLHF for adaptive collaborative robots and the challenges of developing methods for real-world HRC. Its questions cover robot tasks, feedback, training, and evaluation of both human and robot effects.

2.1. Research Questions

  • The review targets RLHF methods for collaborative robots that address bidirectional human-robot adaptation and real-world deployment challenges.Figure 1 frames RLHF as a potential response to requirements for adaptive collaborative robots compared with classic automation.Source: §2.1 · pp. 3–4, Figure 1 · p. 3
  • The review asks which HRC tasks use RLHF, how feedback is collected, and which AI development methods train the models..Source: §2.1 · p. 4
  • It also asks which metrics evaluate effects in both human-to-robot and robot-to-human directions..Source: §2.1 · p. 4

2.2. Methods

The review searched Scopus for 2020–2025 journal and conference publications, screened 199 records, and recorded study characteristics spanning applications, feedback, training, and evaluation.

2.2. Methods

  • The Scopus search covered journal articles and conference papers from 2020–2025 and returned 199 records for screening.Two reviewers independently screened titles and abstracts against the review’s relevance criteria.Source: §2.2 · p. 4
  • Eligible reports studied robot-specific RLHF with direct human participation, human feedback in training, and real human participants.The criteria excluded general AI tasks, feedback limited to reward-model design, and purely synthetic or simulated human data.Source: §2.2 · p. 4
  • The reviewers recorded application and robot task, feedback measurement and format, experiment environment, AI development methods, and human and robot metrics..Source: §2.2 · p. 4

2.3. Results

RLHF supports varied HRC tasks through multiple feedback modalities and formats, while development commonly proceeds through staged training. Studies evaluate robot, human, and feedback outcomes, using environments with different levels of realism.

2.3.1. RLHF Applications

  • RLHF is used across industry, assistance and care, and emergency applications, including mobile-robot navigation and robotic-arm or gripper tasks.Human feedback can fine-tune policies for navigation in 3D space and gripper control.Source: §2.3.1 · p. 4

2.3.2. Human Feedback Methods

  • Feedback collection spans GUI-based subjective evaluations, language, observed behaviour, physiological signals, and multimodal combinations.Language, behavioural, and physiological inputs require additional processing to derive feedback.Source: §2.3.2 · pp. 4–5, Figure 3 · p. 6
  • Feedback formats include preferences and ratings, negative corrections, and demonstrations, which require more advanced input and processing.Preferences require relevant comparison items; demonstrations require comparing desired actions with robot motion to compute feedback values.Source: §2.3.2 · pp. 5–6
  • Human data must be converted into reward scores, but its uncertainty, feedback design, and dataset size and heterogeneity affect reward construction.Complex preprocessing can make it harder to isolate factors shaping the reward-score distribution.Source: §2.3.2 · p. 6

2.3.3. AI Development Methods

  • RLHF development commonly pre-trains an initial policy, then fine-tunes it through later human-feedback sessions.Pre-training may use simulated trial and error or synthetic feedback; Bayesian methods can model human data and its uncertainty.Source: §2.3.3 · p. 6, Figure 4 · p. 6
  • Feedback experiments range from computer simulations to more realistic mock-ups, while pilot studies commonly involve fewer than 25 participants.The reviewed large-scale studies included more participants but were not representative of intended target groups.Source: §2.3.3 · p. 7

2.3.4. Evaluation Metrics

  • Evaluation covers robot learning properties and task or safety performance, alongside human psychological and behavioural factors and feedback metrics.Psychological measures include satisfaction, cognitive load, trust, and perceived safety; feedback measures include data-point counts, timing, and correctness.Source: §2.3.4 · p. 7, Figure 5 · p. 7

2.4. Discussion

The review maps varied RLHF applications, feedback approaches, development stages, and evaluation metrics in HRC, while identifying limits in the evidence base and priorities for future work.

2.4.1. Main Findings

  • RLHF is used across HRC applications facing uncertainty and close collaboration, including disaster-robot navigation and assistive-robot manipulation.These settings include unknown areas, diverse users, or changing tasks.Source: §2.4.1 · p. 8
  • HRC research uses diverse feedback tools and formats, without a universally preferred method; human-response noise and bias remain common concerns.Strategies often seek more feedback, greater information gain, or lower uncertainty.Source: §2.4.1 · p. 8
  • Human feedback alone is insufficient for desirable RL performance; development often combines initial training, real feedback, and later testing in realistic settings.The review describes iterative progression from low-level simulations to realistic environments with real users.Source: §2.4.1 · p. 8
  • HRC evaluations emphasize robot flexibility and feedback quantity, while psychological measures focus mainly on satisfaction and cognitive load as a safety-related metric.The review characterizes this focus as exploratory and more oriented toward AI development than overall system safety and performance.Source: §2.4.1 · p. 8

2.4.2. Limitations

  • The Scopus-only search and inclusion of bidirectional studies may have omitted relevant work, including unidirectional methods requiring validation for closed-loop HRC.The review notes that terminology is not standardized and its search terms may have missed reports using other vocabulary.Source: §2.4.2 · p. 8

2.4.3. Future Research

  • Future RLHF methods should be tailored to target applications and tested with target users in realistic setups that reflect practical constraints.Relevant constraints include environmental conditions, available interaction modalities, time, cognitive limitations, procedural standards, and resources.Source: §2.4.3 · pp. 8–9
  • Future studies should evaluate trust, perceived safety, stress, and cognitive load under varied conditions, since secondary feedback tasks may affect safety and task outcomes.Safety metrics should be assessed in realistic experiments that reflect the target HRC application.Source: §2.4.3 · p. 9
  • Closed-loop studies should account for how collaboration and prior robot policies shape human feedback, including its noise and bias, when modeling rewards.The review recommends feedback strategies that capture both psychological and physical safety.Source: §2.4.3 · p. 9

3. Experiment

The experiment compares user- and system-initiated binary feedback about a robot’s proxemic behavior during a VR collaboration task. It evaluates feedback information value and human effort using safety-related measures and Bayesian models.

3.1. Research Questions

  • The experiment compares system- and user-initiated binary feedback, examining information value and human effort as complementary aspects of feedback strategy.The study frames the comparison as a trade-off between low-effort binary ratings and the feedback volume they may require.Source: §3.1 · p. 9

3.2.1. Application Domain

  • The application is robot navigation in HRC, where proxemic zones vary with navigation speed and interaction context.The experiment focuses on safe navigation and both physical and psychological safety.Source: §3.2.1 · pp. 9–10

3.2.2. Environment and Task

  • In the VR workshop task, participants sorted crates while a co-working robot transported them and crossed paths with two background robots.The co-working robot’s navigation used Unreal Engine AI, with safety stops within a 1 m radius of other actors.Source: §3.2.2 · p. 10, Figure 6 · p. 10
  • Participants provided thumbs-up or thumbs-down judgments about robot distance, either opening prompts themselves or receiving one every third crate.Feedback was a secondary task alongside crate sorting.Source: §3.2.2 · p. 10

3.2.3. Participants and Procedures

  • The between-subjects study included 44 participants: 23 in the user-initiated condition and 21 in the system-initiated condition.Four of 48 completers were excluded for providing no feedback; participants completed demographic and VR-familiarity surveys, a VR session capped at 15 minutes, and post-surveys.Source: §3.2.3 · pp. 10–11

3.2.4. Metrics and Analysis

  • Human effort was measured by prompt duration and cognitive load, while feedback information value covered psychological safety and physical proxemic threat.Psychological safety used perceived-safety and sense-of-control surveys; physical safety used inverse time-to-collision, calculated from the five seconds before each feedback prompt.Source: §3.2.4 · pp. 11–12

3.3. Results

User-initiated feedback reflected perceived safety better than system-initiated feedback. System-initiated feedback yielded more data without credible differences in measured human effort.

3.3. Results

  • M=9.22 (SD=0.76) versus M=5.24 (SD=2.4) feedback volume occurred in system- versus user-initiated conditions, with no credible differences in cognitive load or prompt duration.The authors conclude that system-initiated feedback provided more data without additional human effort.Source: §3.3 · p. 12
  • A weak difference appeared in how feedback encoded maximum proxemic threat to the human (M=-0.16, 95% CrI [-0.01, 0.35]); none was identified for sense of control or threat to other robots.Source: §3.3 · p. 12
  • M=1.0 (95% CrI [0.13, 1.9]) indicated that user-initiated feedback reflected perceived safety better than system-initiated feedback.The credible interval excluded zero; Figure 8 presents the Bayesian posterior sigmoids for perceived-safety information value.Source: Figure 8 · p. 12, §3.3 · p. 12

3.4. Discussion

User-initiated feedback provides greater information value than system-initiated feedback, despite producing fewer data points. The discussion links this difference to feedback timing and notes design constraints on system-initiated prompts.

3.4. Discussion

  • User-initiated feedback captures more meaningful information about psychological safety despite its lower feedback volume.The study also reports that physical-safety data suggest user-initiated feedback is more sensitive to safety violations.Source: §3.4 · pp. 12–13
  • System-initiated prompts may elicit positive responses that miss broader safety concerns because they overlap task completion and may not strongly prompt negative responses to proxemic violations.The safety stop keeps actor distances technically sufficient even when they violate proxemic expectations.Source: §3.4 · p. 13
  • Feedback timing matters because perceived safety may reflect proxemic violations outside the immediate window of a system prompt.The discussion suggests timely feedback action is needed to capture safety perceptions and violations.Source: §3.4 · p. 13
  • Moving to higher-fidelity VR or physical setups requires feedback interfaces and workflows that manage attention shifts and preserve feedback quality.The authors suggest alternative attention-getting modalities in busy industrial settings, while direct-to-robot feedback could improve the effort–information-value trade-off but add loop complexity.Source: §3.4 · p. 13

4. Conclusion

The paper combines a scoping review of closed-loop RLHF in HRC with a VR experiment on feedback quality. Its findings point to feedback timing and realistic evaluation settings as priorities for future work.

4. Conclusion

  • The scoping review maps methods for integrating human feedback at different stages of AI development, with a focus on bidirectional, closed-loop RLHF design.The paper identifies the review and VR experiment as its two contributions to RLHF in HRC.Source: §4 · p. 13
  • In the VR experiment, feedback timing is identified as important for AI safety because data quality depends on immediate feedback action after a perceived safety threat.The authors report that user-initiated feedback better captures perceived safety than system-initiated feedback.Source: §4 · p. 13
  • Future RLHF studies should use high-fidelity, realistic settings and assess whether feedback methods produce high-quality data without compromising human factors.The conclusion also emphasizes viable feedback methods in real environments and relevant safety and performance goals.Source: §4 · p. 13

A. Bayesian Statistics

The appendix describes Bayesian model specifications for cognitive load, feedback duration, and feedback value for psychological and physical safety.

A. Bayesian Statistics

  • Cognitive load is specified with a Gaussian model, while feedback duration is specified with a gamma model.The duration formula includes condition and a participant-level intercept.Source: A. Bayesian Statistics · p. 16
  • Psychological-safety feedback is specified with a binomial model using perceived safety and sense of control in relation to condition.The appendix labels the corresponding parameter-estimate table as feedback on psychological safety.Source: A. Bayesian Statistics · p. 16, Table A.2 · p. 17
  • Physical-safety feedback is specified with a Bernoulli model using threat measures and their interaction with condition.The appendix labels the corresponding parameter-estimate table as feedback on physical safety.Source: A. Bayesian Statistics · p. 16, Table A.1 · p. 16
Loading 2610.09891v1…