PaperlayerView exact revision on arXiv

Source-linked AI summary

Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review

Nicole Robinson, Brendan Tidd, Dylan Campbell, Dana Kulić, Peter Corke

arXiv:2307.15363v1cs.RO

TL;DR

Robotic vision is used to help robots interpret and respond to people, but the surveyed literature had not been comprehensively synthesized across HRI/C applications and techniques. The authors systematically reviewed 3850 articles and analyzed 310 papers; the review reports varied autonomy levels and limited human experimentation and participant diversity.

  • Problem

    Understanding visual information is fundamental to interaction with environments and people, yet the techniques and tools used in HRI/C require synthesis to characterize the field.

    Source: §8 · p. 26

  • Method

    The authors systematically searched, screened, and evaluated 3850 articles, selecting 310 papers for in-depth analysis of robotic vision in HRI/C.

    Source: §1 · p. 2

  • Results

    The surveyed papers spanned autonomy levels 2, 6, and 10, while most use cases were low criticality (N=271, 88%).

    Source: §7.1 · pp. 25–26

  • Takeaways & Limitations

    The review provides an overview of robotic vision across HRI/C domains and performance metrics, including how computer vision has been adapted for robots interacting and collaborating with people.

    Source: §14 · p. 43

    Human experimentation was limited, participant numbers were small, and samples often lacked diversity, potentially directing design and optimization toward restricted populations and contributing to bias.

    Source: §12.4 · p. 41

Abstract

from arXiv · show

Robotic vision for human-robot interaction and collaboration is a critical process for robots to collect and interpret detailed information related to human actions, goals, and preferences, enabling robots to provide more useful services to people. This survey and systematic review presents a comprehensive analysis on robotic vision in human-robot interaction and collaboration over the last 10 years. From a detailed search of 3850 articles, systematic extraction and evaluation was used to identify and explore 310 papers in depth. These papers described robots with some level of autonomy using robotic vision for locomotion, manipulation and/or visual communication to collaborate or interact with people. This paper provides an in-depth analysis of current trends, common domains, methods and procedures, technical processes, data sets and models, experimental testing, sample populations, performance metrics and future challenges. This manuscript found that robotic vision was often used in action and gesture recognition, robot movement in human spaces, object handover and collaborative actions, social communication and learning from demonstration. Few high-impact and novel techniques from the computer vision field had been translated into human-robot interaction and collaboration. Overall, notable advancements have been made on how to develop and deploy robots to assist people.

1 INTRODUCTION

The paper surveys robotic vision for human-robot interaction and collaboration, reviewing work from 2010–2020 to characterize research themes, applications, and methods. It addresses a gap in comprehensive analysis of how vision techniques are integrated and evaluated in real-world HRI/C.

1 INTRODUCTION

  • The review analyzes robotic vision for HRI/C through a systematic search and in-depth analysis of 310 eligible papers drawn from 3850 articles.It examines navigation in human spaces, social interaction, and object handovers.Source: §1 · p. 2

1.2 Scope

  • The review covers 2010–2020 research, including trends, applications, methods, datasets, experiments, populations, and performance evaluations.The period was selected to examine contemporary approaches alongside the introduction of camera hardware such as Kinect.Source: §1.2 · pp. 2–3

1.3 Related Surveys and Systematic Reviews

  • Earlier reviews addressed particular domains, components, or vision techniques, but did not comprehensively analyze robotic vision for HRI/C across techniques, hardware, and real-world evaluation.The paper identifies limited detailed investigation of how vision is integrated into robots and evaluated in real-world HRI/C settings.Source: §1.3 · p. 3

1.4 Contribution

  • The authors systematically extract and synthesize literature to describe current robotic-vision use and discuss ways to improve robot functionality for interaction and collaboration.The review also offers a critical discussion intended to help advance the field.Source: §1.4 · p. 3

2 BACKGROUND

The background traces computer vision and robotic vision, then distinguishes interaction from collaboration and describes how visual information supports both. It connects vision methods to robot perception, action, and collaborative outcomes.

2.1 A Brief History of the Field of Computer Vision

  • Computer vision techniques include object detection, image classification, pixel-level classification, face detection, pose estimation, and motion tracking.These methods provide information about objects and people in visual scenes.Source: §2.1 · p. 4
  • Computer vision developed from handcrafted features and classical learning methods toward deep learning, while RGB-D sensors enabled joint reasoning about color and 2.5D geometry.Deep learning can improve robustness but requires more hardware and development effort.Source: §2.1 · p. 4

2.2 A Brief History of the Field of Robotic Vision

  • Robotic vision combines vision sensors and algorithms that translate images into robot control signals, allowing robots to perceive and respond to people.Robots can also move or adjust their viewpoint to gather further information.Source: §2.2 · pp. 4–5

2.3 Human-Robot Interaction and Collaboration

  • HRI concerns interaction with people and socially appropriate responses, whereas HRC concerns humans and robots working toward shared goals.The passage associates collaboration with potential improvements in speed, productivity, errors, and safety.Source: §2.3 · p. 5

2.3.1 Robots with Computer Vision to Improve Collaborative Outcomes.

  • Visual information can support communication, action selection, human activity understanding, and object handovers in collaborative tasks.Examples include recognizing gestures as commands, predicting actions, and assessing reachability and collision range during handover.Source: §2.3.1 · p. 5

3 REVIEW PROTOCOL

The review uses a reproducible, PRISMA-based search and screening process to select studies of embodied robots that use vision in timely, human-focused interaction. It extracts common study characteristics and classifies application areas to address questions about trends, methods, evaluation, and challenges.

3 REVIEW PROTOCOL

  • The review follows PRISMA and searches multidisciplinary sources for peer-reviewed work published from 2010 through 2020.The search strategy uses broad, iteratively refined terms and includes IEEE Xplore, ACM Library, and Scopus.Source: §3 · pp. 6–8
  • The research questions cover field trends, application domains, interaction taxonomies, vision tools, datasets and models, participant samples, algorithm performance, and future challenges.The questions span both the state of the field and its evaluation.Source: §3 · p. 6
  • Each eligible study was also assessed for upcoming challenges in robotic vision for HRI/C.This is the final topic in the review's research-question set.Source: §3 · p. 6
  • Eligible systems must be embodied robots that use vision, interact or exchange information with a human, and act on visual input fast enough for interaction.The review defines a robot as a system that can act through actuators in response to sensor perception, with possible externally provided goals.Source: §3 · pp. 6–7
  • The review excludes systems without embodied robot action, vision-driven control, human-relevant interaction, or closed-loop exchange, as well as work outside its publication scope.Examples include camera-only systems, tele-operated robots, open-loop systems, and early-stage work without tests with people.Source: §3 · pp. 7–8

3.1 Review Information and Categorisation

  • The review extracts task, robot, interaction, sensor, vision-method, dataset, and evaluation details, along with user-study and participant information where reported.It also computes a task-evaluation metric by rating task complexity, risk, importance, and robot complexity on three-point scales.Source: §3.1 · p. 8
  • Application categories distinguish gestures from other actions, autonomous movement in human spaces, object handovers and collaborative actions, social communication, and learning from demonstration.For example, gesture categories require communicative movement, while handover categorization does not require the person to perform a set pose.Source: §3.1 · p. 8

4 RESULTS

The screening process reduced 3850 assessed papers to 310 studies meeting the final inclusion criteria. Exclusions largely reflected failures to meet physical-robot, human-interaction, and timely vision-driven-action requirements.

4.1 Selected Articles

  • 310 papers, or 8% of the full articles assessed for eligibility, met the final inclusion criteria.The final set followed an additional review of 540 papers that met the initial C1–C3 criteria.Source: §4.1 · p. 9
  • Of 3850 papers assessed in detail, 3310 failed the initial criteria, most often because they lacked a physical robot, followed by missing qualifying interaction or action based on vision.The reported exclusions were 2020 papers on C1, 1005 on C2, and 285 on C3.Source: §4.1 · p. 9
  • The final screening also excluded duplicate studies and papers without a clear human-robot interaction or collaboration experiment or demonstration.Two independent raters assessed 10% of the eligible papers and reached 100% consensus on inclusion and exclusion criteria.Source: §4.1 · p. 9

5 RQ1. WHAT IS THE GENERAL OVERALL TREND OF ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION IN THE LAST 10 YEARS?

Robotic vision research in HRI/C grew moderately but steadily over the decade, with work distributed across several application areas and robot types. RGB-D cameras were the most frequently used camera type, and regional application patterns varied.

Publication trends

  • Publications showed a modest increase over the period, with a small decline in 2020/2021 that the authors attributed as a prediction to the COVID-19 pandemic.Relevant papers appeared most often in conferences, followed by journals and book series.Source: §5 · p. 9, Figure 2 · p. 11

Application areas

  • Gesture recognition (35%) and robot movement in human spaces (22%) were the largest application groups, followed by handover and collaborative actions (17%), action recognition (13%), social communication (10%), and learning from demonstration (3%).Papers could count in multiple application areas (N=335).Source: §5 · p. 10

Domains and robot types

  • Mobile robots were the most common robot type, followed by fixed manipulators, social robots, mobile manipulators, and aerial robots.Common domains included field, industrial, domestic, and urban settings.Source: §5 · p. 10, Figure 3 · p. 12

Camera types

  • RGB-D cameras, including Kinect, were used most frequently, ahead of monocular, stereo, and omnidirectional cameras, and their uptake increased over time.Their use extended across applications, environments, and robot types.Source: §5 · p. 10

Regional trends

  • Gesture recognition and robot movement in human spaces had the highest work volumes across continents, while Europe had a stronger focus on object handover and collaborative actions.Mobile robots and fixed manipulators were the most common robot types across continents.Source: §5 · p. 10, Figure 4 · p. 12

6 RQ2. WHAT ARE THE MOST COMMON APPLICATION AREAS AND DOMAINS FOR ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

The review covers several ways robotic vision supports human-robot interaction, especially gesture and action recognition. Applications range from direct robot control to navigation, teamwork, and social interaction.

6.1 Gesture Recognition

  • Gesture recognition appeared in 116 papers (37%); most recognized body pose (79; 68%) and static gestures (91; 78%), while 25 (22%) used dynamic gestures.RGB-D cameras appeared in 75 papers, and 66 of those (88%) used Kinect.Source: §6.1.1 · p. 13

6.1.1 Overview.

  • For mobile ground robots, gestures commanded movement, turns, stops, following, or navigation toward a location, using hand, pointing, or body movements.Examples include shoulder angle for discrete direction control and arm motions for forward, backward, and turning commands.Source: §6.1.2 · p. 13

6.1.3 Use Case Examples: Manipulator Robot Control.

  • Manipulator gestures controlled arm and gripper actions, selected objects, and supported collaborative tasks such as placing toppings or handing objects to a person.Mobile manipulators and aerial robots also used pointing or body pose to select, direct, and command actions.Source: §6.1.3 · pp. 13–15, §6.1.4 · p. 15

6.1.5 Use Case Examples: State Changes.

  • Gestures also initiated or changed robot states, including following, navigation, walking, parking, and learning-from-demonstration modes.One navigation robot translated a person's pointing gesture into a goal on its map.Source: §6.1.5 · p. 15

6.1.6 Use Case Examples: Team-based Scenarios.

  • Team-based gesture use included selecting robots, directing groups, and commanding swarms; social uses included responsive waving, sign-language spelling, and gesture-based games.A multi-person example localized a person by audio and then selected whom to track when they waved.Source: §6.1.6 · p. 15, §6.1.7 · p. 15

6.2 Action Recognition

  • Action recognition appeared in 44 papers (14%) and covered activities such as daily living, exercise, and walking, often with humanoid robots.RGB-D cameras appeared in 36 papers, and 33 of those (92%) used Kinect.Source: §6.2.1 · p. 16

6.2.1 Overview.

  • Action recognition supported responses to human states, motion prediction, shared-workspace collision avoidance, exercise feedback, mobility assistance, domestic chores, and playful turn-taking.Examples include inferring motion after camera occlusion and adapting a robotic walking aid to different mobility levels.Source: §6.2.2 · p. 16

6.3 Robot Movement in Human Spaces

Vision supports robots moving around people and manipulating objects with them. The reviewed examples combine person detection and tracking with navigation, social distance, shared-space safety, and task assistance.

6.3 Robot Movement in Human Spaces

  • Robot movement in human spaces appeared in 74 papers (24%); following people was the most common task (55; 74%), while body-pose detection was the leading person-identification method (57; 77%).RGB-D cameras appeared in 37 papers, 29 of which (78%) used Kinect.Source: §6.3.1 · pp. 17–18

6.3.1 Overview.

  • Navigation examples included approaching, following, and avoiding people, with proxemic control, occlusion handling, and human-aware mapping supporting movement in shared environments.Other examples used cameras for person detection alongside laser sensing for SLAM, or aerial robots to detect and follow a person's hand or face.Source: §6.3.2 · pp. 18–19

6.4 Object Handover and Collaborative Actions

  • Object handover and collaborative actions appeared in 56 papers (18%); human-aware shared-workspace safety was the largest use case (26; 46%), ahead of direct robot-arm control (15; 27%).Handover systems used force or tactile sensing to determine when to release objects.Source: §6.4.1 · p. 20

6.4.1 Overview.

  • Collaborative tasks used vision to track hands and pose, often with force, tactile, or wearable-IMU sensing for manipulation, assistance, and safety.Applications included assisting with dressing or bathing, pausing when a person disengaged, and adjusting end-effector height to human pose.Source: §6.4.2 · pp. 20–21

6.5 Social Communication

The review describes robotic vision for social interaction and learning from demonstration, although the latter appeared in relatively few papers. Examples range from conversational behaviors to gesture-guided teaching and task imitation.

6.5 Social Communication

  • Social communication appeared in 33 papers (11%), with conversation the most common task (10; 30%), followed by detecting social engagement (6; 18%).Of 16 papers using RGB-D cameras, 12 (75%) used Kinect.Source: §6.5.1 · p. 22

6.5.1 Overview.

  • Social interaction examples used vision and audio to orient toward speakers, detect conversational turns and engagement, recognize facial expressions, and manage social distance or group interaction.Multi-person applications included greeting people, taking orders, and delivering items in a restaurant setting.Source: §6.5.2 · p. 22

6.6 Learning from Demonstration

  • Learning from demonstration appeared in 12 papers (4%), with tasks including manufacturing assistance, human interaction, scene understanding, and behavior learning.Robots learned by watching a person perform a task or by collecting data from interactions between two people; gestures often initiated demonstration or provided instructions.Source: §6.6.1 · pp. 22–23, §6.6.2 · p. 23

6.6.2 Use Case Examples.

  • Demonstration examples taught robots tasks through pointing and speech, peg-grasping, gaze-guided watering, online motion adjustment, and observation of human actions.One example learned to replicate bottle-related actions by observing mirrored human demonstrations.Source: §6.6.2 · pp. 24–25

7 RQ3. WHAT IS THE HUMAN-ROBOT INTERACTION TAXONOMY FOR ROBOTIC VISION IN HUMAN ROBOT COLLABORATION AND INTERACTION?

Across the reviewed studies, lower-criticality and lower-complexity tasks predominated, and most systems were functional robots used in one-human/one-robot teams. Autonomy levels varied, with levels 2, 6, and 10 especially common.

7.1 Summary

  • 271 papers (88%) were classified as low-criticality tasks, while 249 (80%) received low overall task-complexity scores.Medium and high criticality accounted for 10% and 2%; medium and high complexity accounted for 17% and 3%.Source: §7.1 · p. 25
  • 203 systems (65%) were functional robots, and functional robots were more likely than anthropomorphic or zoomorphic robots to appear in medium- and high-criticality studies.Anthropomorphic robots accounted for 34% and zoomorphic robots 1%.Source: §7.1 · p. 25
  • 281 studies (91%) used a 1:1 human-to-robot ratio, and 307 (99%) reported homogeneous robot teams.The review also found high shared interaction in the A formation in 280 studies (90%).Source: §7.1 · p. 25
  • Autonomy levels 2, 6, and 10 were common; mobile and fixed manipulators were most often used at level 10.No level-1 scores were classified because eligibility excluded manually operated robots; the authors note that entry criteria may have skewed autonomy scores upward.Source: §7.1 · p. 26

8 RQ4. WHAT ARE THE VISION TECHNIQUES AND TOOLS USED IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

The review finds that HRI/C vision research commonly uses established detection, pose, gesture, action, tracking, and sensor-fusion methods. It also identifies a lag in transferring emerging computer-vision techniques into HRI/C, with hardware testing and approval requirements contributing to translation delays.

8 RQ4. WHAT ARE THE VISION TECHNIQUES AND TOOLS USED IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

  • Emerging computer-vision techniques remained underrepresented in HRI/C, and their translation may be delayed by approval requirements, hardware errors, and demands for extensive testing.The review frames these methods as potential opportunities for domains facing technical, speed, or accuracy challenges.Source: §8 · pp. 26–27

8.1 Human Detection and Pose Estimation

  • RGB-D skeleton extraction and tracking software was the most common human-detection and pose-estimation approach, appearing in 107 papers.It typically used Microsoft Kinect SDK or OpenNI software, which the review describes as readily available, real-time, and robust.Source: §8.1.1 · p. 29, §8.1 · p. 29

8.1.1 Commercial Software.; 8.1.2 Face Detection.; 8.1.3 Segmentation.; 8.1.4 Region of Interest.

  • Face detection, color or depth segmentation, and learned detectors provided alternatives for locating people and body regions.Face detection using Viola–Jones appeared in 34 papers; deep-learning approaches included R-CNN and single-shot detectors, while OpenPose and Convolutional Pose Machines were the most-used image-based pose estimators (15 papers).Source: §8.1.2 · p. 29, §8.1.3 · pp. 29–30, §8.1.4 · p. 30

8.2 Gesture Classification

  • Static gestures were classified from hand contours or skeleton joint configurations, while dynamic gestures used temporal methods including HMMs, particle filters, and DTW.Deep neural networks were also used to classify gestures from images, skeletal representations, and other sensory inputs.Source: §8.2 · pp. 30–31

8.3 Non-Gestural Action Classification

  • Non-gestural action classification used human keypoints and learned models, with dense trajectories outperforming CNN features under a training–testing data mismatch in one study.A GAN was also trained on skeleton-keypoint sequences to generate plausible motion predictions.Source: §8.3 · p. 31

8.4 Social Classification

  • Social classification covered facial expression, gender, intent to interact, and engagement, using features such as facial keypoints, gaze, orientation, and speech activity.One engagement estimate used gaze analysis to detect gaze aversion during conversation with a social robot.Source: §8.4 · pp. 31–32

8.5 Human Motion Tracking

  • Human tracking combined detection, motion tracking, and reacquisition after occlusion or leaving the frame, using methods such as particle filters, optical flow, SLAM, and Kalman filters.The review notes that group-detection and tracking methods were not present in its HRI/C corpus but could be relevant to social settings.Source: §8.5 · p. 32

8.6 Multiple Sensors

  • Robotic vision was often paired with microphones, range sensors, inertial or force sensors, and other inputs to support perception, interaction, and operator feedback.Force sensors were used alongside vision for tasks including object handover and collaborative manipulation.Source: §8.6 · p. 32

9 RQ5. WHAT ARE THE DATASETS AND MODELS THAT HAVE BEEN USED FOR ROBOTIC VISION IN HUMAN-ROBOT INTERACTION AND COLLABORATION?

The reviewed studies used a mix of custom and established datasets, but only a small number made datasets available. Dataset choices included large action-recognition collections as well as task-specific resources for people, faces, and gestures.

9 RQ5. WHAT ARE THE DATASETS AND MODELS THAT HAVE BEEN USED FOR ROBOTIC VISION IN HUMAN-ROBOT INTERACTION AND COLLABORATION?

  • Only three papers (1.5%) released datasets, including gesture datasets OpenSign and a 160,000-sample hand-gesture collection.OpenSign contained 10 static gestures recorded by 10 volunteers; the other collection covered open and closed hand gestures across different poses.Source: §9 · p. 33
  • Action-classification studies used large resources including NTU RGB-D, with 60 action classes across 56,880 video samples, and H3.6M, with 3.6 million 3D human poses.The review also describes the task-specific MAD dataset, comprising 625 recordings across five objects and five actions.Source: §9 · p. 33
  • Other dataset choices supported people detection, tracking, face detection, and expression or gender classification, including Market-1501, KTP, HollywoodHeads, CK+, AffectNet, and WIDER FACE.The review reports varied dataset sizes, from 8,475 KTP frames to one million facial images in AffectNet.Source: §9 · p. 33

10 RQ6. WHAT HAS BEEN THE MAIN PARTICIPANT SAMPLE, AND HOW IS ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION EVALUATED?

Across the reviewed studies, human-participant reporting was sparse, and evaluations more often measured robot or task outcomes than robotic-vision performance directly. Reported evaluations span gesture recognition, action recognition, robot movement, collaborative tasks, and social communication.

10.1 Main Participant Sample

  • Only 66 of 310 papers (21%) reported human experiments or testing; among studies reporting participant numbers, the total was 1,228 (M=20, range 1–150, SD=22).Participant age was reported in 22% of papers, gender in 35%, and no studies reported participant country of origin.Source: §10.1 · p. 33

10.2 Evaluation Metrics in Each Application Area

  • Evaluations more often used objective metrics (N=384) than subjective metrics (N=40), and commonly assessed perception, task performance, or preferences rather than vision performance directly.Reported questionnaires included NASA-TLX and GODSPEED; Figure 14 depicts team-metric categories and reported metrics by application area, domain, robot type, and camera type.Source: §10.1 · p. 34

10.2.1 Gesture Recognition.

  • In one evaluation of accuracy during a robot gesture game involving 30 participants, 15% of losses were attributed to out-of-distribution gestures and 10% to classification error.Gesture-style preferences were 62% for elbow-to-finger and 38% for eye-to-finger.Source: §10.2.1 · pp. 35–36

10.2.2 Action Recognition.

  • A mobile robot was successfully controlled in 77.4% of interactions, with 8.9% of unsuccessful interactions attributed to fast rotation or large camera-to-person distance.In a human-following study, participants generally found the robot appropriate but most felt uneasy; 6 of 13 would adapt their living environment, while 10 did not want to change walking speed.Source: §10.2.3 · p. 36

10.2.3 Robot Movement in Human Spaces.

  • In a user-controlled UR10 pick-and-place task, inexperienced participants exceeded 75% success after three trials, averaging 69 seconds per interaction.Higher mental load was reported when a robot operated closer to a person at higher velocity; preference findings included reduced workload for personalized shoe fitting, while another study found no time or error-rate difference for instrument passing.Source: §10.2.4 · p. 36

10.2.5 Social Communication.

  • Social-communication evaluations also reported successful drink service at 100% accuracy in a single-person scenario, with a 658 ms response and 49.4 s interaction time.A PR2 answered user requests successfully in all cases, with 72.7% on the first attempt and 18.2% on the second.Source: §10.2.5 · p. 36

10.2.4 Object Handover and Collaborative Actions.

  • In social communication, a study found that gaze and pause detection for identifying when a person had finished speaking doubled talking time compared with filled-pause detection.Other reported results included 99% accuracy detecting interest in engaging and 92% accuracy detecting disinterest; a receptionist study found no task-performance effect from engagement-aware behavior.Source: §10.2.5 · pp. 36–37

11 RQ7. WHAT IS THE STATE-OF-THE-ART IN VISION ALGORITHM PERFORMANCE FOR ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

The review presents selected vision-algorithm examples because inconsistent metrics and results prevent systematic comparisons across studies. The examples show varied gesture, human-tracking, and action-recognition approaches, with reported performance tied to their particular tasks and settings.

11 RQ7. WHAT IS THE STATE-OF-THE-ART IN VISION ALGORITHM PERFORMANCE FOR ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

  • Because few papers directly evaluated robotic-vision performance with standardized metrics, the review selected examples for novelty, impact, or robustness rather than calculating systematic cross-study differences.The authors identify more rigorous experimentation and systematic evaluation of feasibility, capacity, speed, and accuracy as future pathways.Source: §11 · p. 37

11.1 Gesture Recognition

  • Gesture-recognition examples used fine-tuned Inception V3, a robustness-tested hand-gesture classifier, and Kinect skeleton data with dynamic time warping; their reported accuracies are not directly comparable across settings.The Inception V3 system detected five gestures at 40 Hz and reported 99.1% validation accuracy and 98.9% test accuracy; other examples reported 96.7% average total accuracy and greater than 86% accuracy.Source: §11.1 · p. 37

11.2 Human Detection and Tracking

  • Human-detection and tracking examples reported detection up to 8 m with 4 cm maximum position error and 5° orientation, 59.7% mAP for mobile-robot tracking, and a target-contour method at 94% accuracy and 34 fps.The contour-based method was reported as more accurate and faster than comparison methods; a combined Kalman filter, hidden Markov model, and Fast R-CNN system improved performance sevenfold over a dense sliding-window method.Source: §11.2 · pp. 37–38

11.3 Non-Gestural Action Recognition

  • For non-gestural action recognition, one RGB-camera result was 71% accuracy on NTU RGB-D (75% from Kinect) at 15 fps, while predicted human actions yielded lower rollator tracking error than several control methods.Another study evaluated dense-trajectory, multi-view fusion on 25 children performing 12 actions and compared results with 14 adults.Source: §11.3 · p. 38

12 RQ8. WHAT ARE THE UPCOMING CHALLENGES FOR ROBOTIC VISION IN HUMAN-ROBOT COLLABORATION AND INTERACTION?

The section surveys challenges for robotic vision in human-robot collaboration and interaction, from technical and deployment constraints to human data, behavior models, evaluation, and trust.

12.1 General Challenges of Robotic Vision

  • Robotic vision faces real-world robustness, data and hardware constraints, delayed transfer of new computer-vision methods, and difficulty translating visual signals into useful robot actions.Computationally intensive algorithms may require on-robot GPUs, while camera positioning can help recover occluded information.Source: §12.1 · pp. 38–39

12.2 Fair and Ethical Use of Human Data

  • Human-data use raises consent, transparency, storage, ownership, and privacy concerns, especially when robots capture data in public or sensitive settings.The review points to privacy-preserving techniques such as face anonymization and visual SLAM.Source: §12.2 · pp. 39–40

12.3 Human Models

  • Human-behavior models must be selected and optimized carefully because simplistic or poorly matched models and datasets can misrepresent people and contribute to bias.Categorizing emotion into static labels may prompt inaccurate identification or irrelevant robot responses; repeated errors may also undermine trust.Source: §12.3 · pp. 40–41

12.4 Experimental Design and Evaluation

  • Human evaluations have often involved few, unrepresentative participants and simple metrics that may overlook the human, robot, and team dynamic.The authors warn that restricted samples can bias design and create barriers to wider adoption.Source: §12.4 · p. 41

12.5 Appropriate Trust

  • People may overestimate a robot’s visual perception and capability, so deployments should explain its strengths and limitations within the intended context.Misunderstood capabilities can lead people to rely on the robot beyond what it can perform.Source: §12.5 · pp. 41–42

13 PROMISING AREAS OF FUTURE RESEARCH

The section identifies computer-vision techniques and application areas that are underrepresented in HRI/C, while noting potential uses and transfer constraints.

13 PROMISING AREAS OF FUTURE RESEARCH

  • 3D convolutional networks [67] [134] process video for action classification, while 3D human pose estimation is mostly absent from HRI/C despite its potential to support more functional robot actions.Video models can require substantial training data and GPU resources; 3D human modeling can support physics-aware planning and responses.Source: §13 · p. 42
  • Human–object interaction classifiers localize and describe interacting people and objects, helping robots reason about both the object and action in collaborative tasks.Approaches detect human and object boxes before combining appearance, geometry, or pose information.Source: §13 · p. 42
  • Transformers [437] are emerging as vision architectures across image, object, video, and human–object interaction tasks, with potential for joint processing of multimodal robot inputs.The passage describes this as a post-2020 opportunity for HRI/C.Source: §13 · pp. 42–43
  • Sign language recognition, autonomous driving, and anticipation of human actions are additional opportunities for non-verbal communication, safer coordination, and more reactive robot behavior.Sign language recognition was absent from the reviewed corpus; many human-focused autonomous-driving studies had been tested in simulation and without vision.Source: §13 · pp. 42–43

14 CONCLUSION

The review finds robotic vision for HRI/C to be an emerging area, with limited translation of some computer-vision techniques and relatively simple application settings. It points to multi-input platforms and better adaptation as directions for long-term teamwork.

14 CONCLUSION

  • Robotic vision for HRI/C remains emerging: most reviewed work studied one-on-one interaction or a specific interaction feature, and only some high-impact computer-vision techniques had transferred.The corpus also included relatively few multi-human, multi-robot teams.Source: §14 · p. 43
  • Robotic vision was often tested in a simple or single application field for each use case, indicating limited depth in its current form.Source: §14 · p. 43
  • Future platforms could process multiple visual inputs to support more competent operation in dynamic environments and better adaptation to different people over long-term teamwork.The review identifies multi-human, multi-robot collaboration as one potential task area.Source: §14 · pp. 43–44
Loading 2307.15363v1…