PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Multi-Agent Human-Robot Interaction Systems

Abhinav Dahiya, Alexander M. Aroyo, Kerstin Dautenhahn, Stephen L. Smith

arXiv:2212.05286v1cs.RO

TL;DR

HRI research has focused largely on dyadic systems, while systems with multiple humans or robots introduce more complex interactions and control demands. This survey synthesizes multi-agent HRI research across interaction patterns and robot-control approaches. It finds one-to-one interaction remains prevalent, many-to-many systems are emerging, and much optimization-based control work remains theoretical.

  • Problem

    HRI research has largely concerned dyadic systems, while remote interaction with multiple robots can increase users’ cognitive workload, especially as robot numbers grow.

    Source: §1 · p. 2, §4.2.1 · p. 14

  • Method

    The survey synthesizes multi-agent HRI studies across interaction models and optimization-based robot-control approaches.

    Source: §4.3 · p. 15, §5.1 · p. 15

  • Results

    One-to-one interaction remains most prevalent, one-to-many has enabled more efficient interaction, and many-to-many implementations are emerging, while most computational optimization-control research remains theoretical.

    Source: §4.3 · p. 15, §5.3 · p. 17

  • Takeaways & Limitations

    Understanding multi-agent HRI dynamics and developing modeling and control strategies are crucial steps toward humans and robots interacting seamlessly.

    Source: §8 · p. 19

    Wizard-of-Oz control is prevalent in systems with few robots but may constrain scalability and the use of heterogeneous or larger robot teams; ideal configurations remain unsettled.

    Source: §6 · pp. 17–18

Abstract

from arXiv · show

This article presents a survey of literature in the area of Human-Robot Interaction (HRI), specifically on systems containing more than two agents (i.e., having multiple humans and/or multiple robots). We identify three core aspects of ``Multi-agent" HRI systems that are useful for understanding how these systems differ from dyadic systems and from one another. These are the Team structure, Interaction style among agents, and the system's Computational characteristics. Under these core aspects, we present five attributes of HRI systems, namely Team size, Team composition, Interaction model, Communication modalities, and Robot control. These attributes are used to characterize and distinguish one system from another. We populate resulting categories with examples from recent literature along with a brief discussion of their applications and analyze how these attributes differ from the case of dyadic human-robot systems. We summarize key observations from the current literature, and identify challenges and promising areas for future research in this domain. In order to realize the vision of robots being part of the society and interacting seamlessly with humans, there is a need to expand research on multi-human -- multi-robot systems. Not only do these systems require coordination among several agents, they also involve multi-agent and indirect interactions which are absent from dyadic HRI systems. Adding multiple agents in HRI systems requires advanced interaction schemes, behavior understanding and control methods to allow natural interactions among humans and robots. In addition, research on human behavioral understanding in mixed human-robot teams also requires more attention. This will help formulate and implement effective robot control policies in HRI systems with large numbers of heterogeneous robots and humans; a team composition reflecting many real-world scenarios.

1. Introduction

Multi-agent HRI is growing beyond the predominantly dyadic systems studied in earlier work, bringing new coordination and control challenges. This survey organizes that literature, including a broader range of systems and computational studies, to support comparison and identify future research needs.

1. Introduction

  • The authors frame multi-agent HRI as a growing departure from the field’s predominantly dyadic research focus.They connect this shift to improving robot physical and computational capabilities.Source: §1 · p. 2
  • Multi-agent HRI adds interactions among multiple agents, including human–human, robot–robot, and indirect interactions that complicate coordination and control.The paper notes that dyadic control strategies may not apply or work as intended in these systems.Source: §1 · p. 2
  • The survey extends prior reviews to include multiple robots interacting with one or more humans, as well as simultaneous multi-human–multi-robot systems.It also includes theoretical or computational studies without physically present humans or robots when they capture crucial HRI aspects.Source: §1 · p. 3
  • The survey includes system-design, observational, and algorithmic-design studies of multi-agent HRI.Studies must involve interactions among at least three agents, including at least one human and one robot, and have an intended HRI application.Source: §1 · p. 4
  • The survey organizes systems around five attributes derived from team structure, interaction style, and computational characteristics.It presents literature examples, discusses applications, compares categories, and identifies trends and future challenges.Source: §1 · p. 4

2. Multi-Agent Human-Robot Interaction

The paper defines multi-agent HRI through information flow among human and robot agents, represented as a directed interaction graph. It proposes three aspects for comparing these systems and explains why their interactions and control pose distinct challenges.

2. Multi-Agent Human-Robot Interaction

  • An interaction is information flow between agents through communication, action, or presence that changes a receiving agent’s actions, behavior, or mental state.The directed interaction graph represents agents as nodes and interactions as edges pointing with information flow.Source: §2 · p. 4
  • The paper defines multi-agent HRI systems as interaction graphs with at least three nodes and three connected agents, including a human–robot interaction edge.Source: §2 · p. 4
  • Multi-agent HRI can include multi-agent and indirect interactions, human–human and robot–robot interactions, and increasingly difficult behavior modeling and control as agent counts grow.The resulting uncertainty in human–robot interactions is greater than in robot-only groups, and research seeks to make human responses more predictable.Source: §2 · pp. 4–5
  • The proposed comparison framework uses team structure, interaction style, and computational characteristics to describe system setup, agent interactions, and control.The aspects are not exhaustive, and attributes under one aspect can influence others.Source: §2 · p. 5
  • Team structure covers the number and types of agents and their homogeneity, while interaction style covers communication modalities and interaction models.Computational characteristics concern robot planning, control, and ways of influencing human behavior.Source: §2 · p. 5

3. Team Structure

Team structure distinguishes systems by the number and composition of their human and robot agents. Across team sizes, applications range from dyadic interaction to multi-human–multi-robot collaboration, while agent homogeneity can shape interaction effort and capabilities.

3. Team Structure

  • Team structure in multi-agent HRI is characterized by team size and composition: systems may include multiple humans and/or robots, and agents may differ in capabilities and roles.Team structure also shapes agent interaction, information sharing, and action planning (Fincannon et al., 2013, 2011).Source: §3 · p. 6

3.1. Team Size

  • Team size refers to the number of humans and robots present, while team structure also distinguishes the agents’ homogeneity or heterogeneity.The paper treats team size and team composition as the two factors under team structure.Source: §3.1 · p. 6, §3 · p. 6

3.1.1. Single-human – single-robot

  • Single-human–single-robot systems are conventional dyadic HRI and the most commonly studied configuration.Source: §3.1.1 · p. 6

3.1.2. Single-human – multi-robot

  • Single-human–multi-robot systems include operator-supervised robot teams, human-swarm interaction, and robots that guide, navigate with, or entertain a person.Applications include fault intervention, assembly, search-and-rescue, and storytelling.Source: §3.1.2 · p. 6

3.1.3. Multi-human – single-robot

  • Multi-human–single-robot systems include teams managing a robot in search, exploration, and UAV operation, as well as robots assisting or receiving help from groups of people.Robots also serve as resource distributors or moderators in group interactions, and social robots increasingly appear in classrooms and public spaces.Source: §3.1.3 · pp. 6–7

3.1.4. Multi-human – multi-robot

  • Multi-human–multi-robot systems are among the most challenging to understand and control because uncertainty and possible system states grow exponentially.Applications include military coordination, search-and-rescue, supervisory control, task allocation, scheduling, social interaction, and group learning.Source: §3.1.4 · p. 7

3.2. Team Composition

  • Team composition concerns homogeneity among agents, which can reflect robot hardware, capabilities, or interfaces and human roles, capabilities, or authority.Heterogeneous robot teams can provide varied capabilities, but their control and operation are more demanding than homogeneous teams.Source: §3.2 · pp. 7–8

Key Observations

The survey finds substantial work on one-to-many systems, while systems combining multiple humans and robots are emerging. It also identifies varied human and robot roles, supervision challenges, and limited social or human-assisting implementation of heterogeneous robots.

Key Observations

  • One-to-many systems are substantial in the literature, while systems with multiple humans and multiple robots are emerging, mainly in planning and task scheduling.The survey calls for more research on robot interactions in systems with larger numbers of humans.Source: Key Observations · p. 9
  • Humans take varied roles, including supervising multiple robots, participating as social peers, and seeking information in public settings.Human supervision of many robots presents challenges including limited cognitive capacity, increased workload, and decreased situational awareness.Source: Key Observations · p. 9
  • Despite examples of heterogeneous robot research, implementation in social settings or human-assisting applications remains limited.The survey notes a need for further research on human interactions with larger teams and human behavioral understanding in mixed teams.Source: Key Observations · p. 9

4. Interaction Style

The survey treats interaction style as encompassing communication modes, interaction models, interfaces, and other communicative and spatial features. It distinguishes multi-agent interaction through within-group exchanges, one-to-many interactions, and direct or indirect communication.

4. Interaction Style

  • Interaction style covers communication modes, interaction models, interfaces, verbal and non-verbal methods, affect, and spatial relationships.The survey frames these as different aspects of interaction in human-robot systems.Source: §4 · p. 9
  • Multi-agent systems allow human-human and robot-robot exchanges, one-to-many interactions, and additional interactions among agents beyond dyadic HRI.The survey notes that group interactions can occur alongside human-robot interactions.Source: §4 · p. 9
  • Direct interaction is an agent's intentional, explicit communication to a recipient; indirect interaction includes a third party observing that exchange.The survey uses a robot communicating task information to another robot as direct interaction and a human observing them as indirect interaction.Source: §4 · p. 9
  • Distinguishing direct from indirect interaction helps assess whether agents actively communicate or mainly coexist and observe one another.Modeling indirect interactions is necessary to understand how direct exchanges between two agents can affect others' behavior.Source: §4 · p. 9

4.1. Interaction Models

The survey represents multi-agent interaction models as communication channels connecting agents, and distinguishes one-to-one, one-to-many, and many-to-many arrangements. These models show information-flow direction and can be annotated for direct versus indirect interaction, but basic graphs omit several system characteristics.

4.1. Interaction Models

  • Interaction graphs represent who communicates with whom and the direction of information flow; Table 1 illustrates examples, not an exhaustive catalogue.Solid arrows denote direct interactions, while dashed arrows denote indirect, observational interactions.Source: §4.1 · pp. 9, 11, Table 1 · p. 10
  • Basic interaction graphs do not capture embodiment, agent roles, heterogeneity, or communication modes, though some information can be added through modifications.Table 1 uses solid and dashed arrows to mark direct and indirect interactions.Source: §4.1 · p. 11

4.1.1. One-to-one

  • One-to-one interactions connect one robot and one human at a time, including when a person separately commands or monitors robots in a larger team.A leader-follower arrangement can relay a human's control of leader robots to the rest of a swarm.Source: §4.1.1 · p. 11, Figure 5 · p. 11

4.1.2. One-to-many

  • One-to-many interactions let one robot engage multiple humans or one human engage multiple robots, including through group commands translated for individual agents.Group-command nodes can interpret commands and relay the required information or actions to recipients.Source: §4.1.2 · pp. 11, 13, Figure 6 · p. 12

4.1.3. Many-to-many

  • Many-to-many interactions connect multiple humans and robots, often through proxies that interpret and relay commands; conflicts may otherwise require human resolution.A group node can collect information from multiple robots and relay allocations to multiple operators, or translate multiple users' inputs into robot actions.Source: Figure 6 · p. 12, §4.1.3 · p. 13

4.2. Communication Modalities and Interfaces

Communication modalities include verbal and non-verbal channels, and their feasibility depends partly on agent proximity. The survey distinguishes remote systems, which commonly use screen-based interfaces, from co-located systems with auditory, visual, and physical options.

4.2. Communication Modalities and Interfaces

  • Communication modalities include speech, haptics, and screens, and depend on environment, application, proximity, and agent capabilities.The survey groups modalities by proximity because proximity strongly influences which options are feasible.Source: §4.2 · p. 13

4.2.1. Remote communication

  • Remote interaction can raise cognitive workload, increasingly so as robot numbers grow, and challenges include maintaining supervisors' situational awareness.Screen-based interfaces are common; immersive VR or AR interfaces have been shown to improve situational awareness and decrease workload.Source: §4.2.1 · pp. 13–14, Figure 7 · p. 14

4.2.2. Proximate communication

  • Co-located systems can use auditory, visual, and physical channels, alone or in combination, with non-verbal cues and spatial placement also relevant to group interaction.Examples include speech, gestures, facial expressions, posture, gaze, touch, and force; co-located interactions can also affect other agents' behavior and future interactions.Source: §4.2.2 · pp. 14–15

4.3. Key Observations

One-to-one interaction remains most prevalent in multi-agent HRI, while one-to-many is also common and many-to-many implementations are emerging. Screen-based interfaces are common for humans supervising multiple robots; multimodal communication is increasingly used in co-located and social settings.

4.3. Key Observations

  • One-to-one interaction is most prevalent, while one-to-many is also popular, especially in social and co-located settings.Many-to-many interaction is desirable for natural, smooth interactions, and implementations are beginning to appear.Source: §4.3 · p. 15
  • Screen-based interfaces are most common when humans interact with multiple robots, while the usability of VR and AR across settings remains unclear.The survey identifies immersive interfaces as a possible future shift but calls for more research on their applicability.Source: §4.3 · p. 15
  • Multimodal communication is becoming more popular in co-located and social settings, where it can support natural, safer interaction without enforced physical separation.Source: §4.3 · p. 15

5. Computational Characteristics: Robot Control

The survey distinguishes optimization-based from predefined or rule-based robot control, while noting that human behavior is influenced indirectly through robot actions and communication. It focuses on robot control and how multiple agents affect its implementation.

5.1. Optimization-based control

  • Optimization-based control computes actions to optimize measurable performance parameters, using models such as time series, outcome probabilities, or Dynamic Bayesian Networks.Parameters can include task completion time, resource cost, or reward; data-driven methods are also used.Source: §5.1 · p. 15
  • Optimization-based control is suited to systems with reliable behavioral models and quantifiable metrics, but multi-agent information and interaction demands can grow exponentially with agent count.This motivates computationally efficient decision-making and control, including task allocation, navigation, and operator-support applications.Source: §5.1 · p. 16

5.2. Predefined and rule-based control

  • Predefined and rule-based control uses Wizard of Oz, expert if...then rules, or preset action sequences when models are unavailable or researchers need controlled behavior.These methods can simplify decision-making and support studies of subjective outcomes; they can also be combined with optimization-based control.Source: §5.2 · p. 16
  • Preset robot behavior can manipulate test conditions without environmental feedback, while Wizard of Oz uses a hidden teleoperator and can support mixed-initiative control.Preset behavior is used to study responses such as robot perception, knowledge acquisition, group emotions, and perceived legibility.Source: §5.2 · p. 16

5. Computational Characteristics: Robot Control

  • Robot control can use individual-agent parameters or group-level parameters, with the latter suited to coordinating teams or focusing on whole-group behavior.The survey describes optimization-based and expert-knowledge control as two principal control types.Source: §5.2 · pp. 16–17, §5.3 · p. 17

5.3. Key Observations

  • Wizard of Oz is the most widely used control method in studies of human responses to interactions with one or a limited number of robots.The survey treats its prevalence as an indicator that much multi-agent HRI research investigates human behavior.Source: §5.3 · p. 17
  • Most computational research on optimization-based control remains theoretical, although it has produced efficient algorithms for large systems and real-system implementations also exist.The authors attribute the theoretical emphasis to the resources required to implement and test large systems in the real world.Source: §5.3 · p. 17
  • Group-based control can shape whole-team behavior without specifying each individual, while individual-based control extends dyadic research and can support more natural group interaction.The survey notes growing interest in understanding group-level behavioral parameters.Source: §5.3 · p. 17

6. Interactions between the Core Aspects and Attributes

The survey treats team size and composition, interaction model, communication mode, and robot control as configurable attributes, while noting that choices along one aspect can influence another. It concludes that the literature is not yet mature enough to prescribe an ideal configuration.

6. Interactions between the Core Aspects and Attributes

  • Attributes across team structure, interaction, communication, and control can be selected as separate configurations, but choices in one aspect can influence another.Many-to-many interaction is an exception: it requires both multiple humans and multiple robots.Source: §6 · p. 17
  • Large teams may benefit from one-to-many communication, and large robot groups commonly use a group command node when humans cannot readily communicate with every robot.The survey identifies this interaction and control pattern in Human-Swarm Interaction.Source: §6 · p. 17
  • Wizard of Oz is common with limited robot teams because it is easy to implement, but may constrain scalability and robot diversity while demanding operator time and attention.Source: §6 · p. 17
  • The literature is not mature enough to establish an ideal system configuration or determine whether current configurations are design guidelines or opportunities for exploration.Source: §6 · pp. 17–18

7. Challenges and Direction for Future Work

The section identifies agent-related and task-related challenges for multi-agent HRI. It emphasizes understanding behavior in groups and addressing safety, scalability, coordination, and transparency.

7.1. Agent-related Factors

  • Group behavior remains poorly understood, and social-science concepts for group dynamics have not yet been implemented in fully functioning multi-agent HRI systems.The survey calls for multidisciplinary research and for investigating how adding agents changes human behavior and interaction outcomes.Source: §7.1 · p. 18
  • Research on workload and situational awareness largely concerns one human with multiple robots, leaving mixed teams of multiple humans and robots understudied.The survey calls for examining how other humans’ roles and decision authority affect each user as team size increases.Source: §7.1 · p. 18
  • Trust remains insufficiently understood when robots in a multi-agent system differ in autonomy, performance, and failure rates.The survey notes that varied robot roles, performance, and identities complicate links between robot parameters and human-related outcomes.Source: §7.1 · p. 18
  • More humans can reduce system observability and predictability, while robots may need to adapt to individual interaction styles and select partners in larger teams.The survey identifies partner selection by both humans and robots as a limited area of study.Source: §7.1 · pp. 18–19
  • Multi-agent systems need viable interaction models and conflict resolution for disagreements in agent roles and authority.In highly coordinated tasks, human performance can suffer as the number of robots that do not coordinate autonomously increases.Source: §7.1 · p. 19

7.2. Task-related Factors

  • Shared-workspace safety research should address humans and robots collaborating on tasks, not only robots avoiding people in shared environments.The survey highlights physical HRI such as handling the same objects or sharing workspaces.Source: §7.2 · p. 19
  • Scalability methods for growing human-robot teams are mostly theoretical, with limited testing on real systems and practical conditions that may invalidate assumptions.The survey notes that online decisions and control policies with many parameters make scaling an important consideration.Source: §7.2 · p. 19
  • Transparency remains under-explored in multi-agent HRI, and it is not well known how dyadic findings translate to mixed teams.Existing studies mostly consider a single robot or homogeneous robots; the survey calls for more work in mixed teams, particularly in direct interaction.Source: §7.2 · p. 19

8. Conclusion

The survey organizes diverse multi-agent HRI research to compare existing work and identify trends and challenges. It concludes that safety, scalability, coordination, transparency, group behavior, and viable interaction models remain important challenges.

8. Conclusion

  • Multi-agent HRI enables more realistic and practical interaction scenarios but brings challenges beyond those of dyadic systems.The survey connects this research to a vision of robots operating and collaborating as equal partners in human environments.Source: §8 · p. 19
  • The survey offers a way to compare diverse multi-agent HRI research and identifies trends and challenges in the field.It frames modeling and control strategies for multi-agent interaction as important to the vision of humans and robots interacting seamlessly.Source: §8 · p. 19
  • Progress requires addressing safety, scalability, coordination, and transparency, while modeling behavior and interaction among heterogeneous humans and robots.The conclusion particularly identifies viable interaction models for simultaneous interaction among multiple humans and multiple robots.Source: §8 · p. 20
Loading 2212.05286v1…