PaperlayerView exact revision on arXiv

Source-linked AI summary

Negotiation-based Human-Robot Collaboration via Augmented Reality

Kishan Chandan, Vidisha Kudalkar, Xiang Li, Shiqi Zhang

arXiv:1909.11227v3cs.ROcs.MA

TL;DR

Human-robot collaboration requires communication to manage interdependent actions, yet AR has been used in relatively few HRC approaches. ARN combines AR visualization with human feedback and robot re-planning; experiments report better collaboration efficiency than non-AR approaches.

  • Problem

    Although AR can communicate spatial information and multiagent methods support collaboration, few approaches have leveraged AR for human-robot collaboration.

    Source: §I · p. 1

  • Method

    ARN uses AR to visualize robot states and plans, collect human feedback, and support iterative negotiation and re-planning.

    Source: §III · p. 2, §IV · p. 3

  • Results

    ARN significantly improved collaboration efficiency compared with traditional non-AR baselines in real-world experiments, and simulation found feedback improved efficiency over ARN without feedback.

    Source: §I · p. 1

  • Takeaways & Limitations

    ARN supports collaboration in tasks where robots have limited capabilities and need human help at some phases.

    Source: §VII · p. 6

Abstract

from arXiv · show

Effective human-robot collaboration (HRC) requires extensive communication among the human and robot teammates, because their actions can potentially produce conflicts, synergies, or both. We develop a novel augmented reality (AR) interface to bridge the communication gap between human and robot teammates. Building on our AR interface, we develop an AR-mediated, negotiation-based (ARN) framework for HRC. We have conducted experiments both in simulation and on real robots in an office environment, where multiple mobile robots work on delivery tasks. The robots could not complete the tasks on their own, but sometimes need help from their human teammate, rendering human-robot collaboration necessary. Results suggest that ARN significantly reduced the human-robot team's task completion time compared to a non-AR baseline approach.

I. INTRODUCTION

ARN combines AR-based robot-state and plan visualization with human feedback that robots can use to re-plan. Experiments in delivery tasks found improved collaboration efficiency compared with non-AR baselines.

I. INTRODUCTION

  • Human-robot collaboration remains rare in practice despite advances in multiagent systems.The introduction notes examples where people and robots work in separate zones or interact only at delivery,.Source: §I · p. 1
  • ARN addresses limited use of AR for HRC with a framework for spatially distant teammates to iteratively communicate preferences and constraints.The interface visualizes robots’ current status and planned motion trajectories [p07].Source: §I · p. 1
  • When users cannot help immediately, they can give time-bounded feedback that robots incorporate during re-planning.The interface is especially useful where occlusions make robots’ status difficult to observe directly.Source: §I · p. 1
  • ARN significantly improved collaboration efficiency over traditional non-AR baselines in real-world experiments, while simulation comparisons found feedback improved efficiency over ARN without feedback.The evaluation used a human-robot collaborative delivery task.Source: §I · p. 1

II. RELATED WORK

Prior communication approaches provide natural-language, vision, or spatial cues, but AR has been less used for HRC. ARN combines spatial information with bidirectional communication and multi-robot task re-planning.

II. RELATED WORK

  • ARN uses AR to communicate spatial information, including robot entry routes and planned trajectories, in shared physical environments.The human can visualize and interact with robots’ planned actions.Source: §II · p. 1
  • Projection-based spatial cues let nearby humans respond to robot trajectories, but require proximity and make bidirectional communication difficult,,,.ARN aims to retain spatial information while alleviating the proximity requirement and enabling two-way communication.Source: §II · pp. 1–2
  • Earlier AR systems visualized robot or UAV intentions, but conveyed information only from robot to human,,,.ARN supports communication in both directions.Source: §II · p. 2
  • Unlike a related AR system that supports robot prompts and questions,, ARN addresses human-multi-robot collaboration and task re-planning in response to human feedback.The paper describes the robots’ planning capability as enabling negotiation within human-multi-robot teams.Source: §II · p. 2

III. OUR AR INTERFACE FOR HUMAN-ROBOT COMMUNICATION

The AR interface lets people view robots’ current states and intended plans, then send feedback that robots can use to improve plans. Its Visualizer and Restrictor support these two directions of communication.

III. OUR AR INTERFACE FOR HUMAN-ROBOT COMMUNICATION

  • The interface visualizes robots’ current states and intended plans, and lets people provide feedback for plan improvement.It is designed for human-robot communication.Source: §III · p. 2
  • The mobile interface displays robot locations and planned actions and collects human feedback through two interactive buttons.These are the elements shown in Fig. 1.Source: Figure 1 · p. 2
  • The Visualizer turns robot motion plans and live locations into rendered trajectories and robot avatars, shown as a frame at each time step.The plans and locations are augmented using the V_slz function.Source: §III · p. 2
  • The Restrictor collects domain-specific human feedback, stores it, and converts it into activated constraints that robots can use for re-planning.Feedback may come through speech, gestures, or interactive buttons; absent feedback is represented as an empty set.Source: §III · p. 2
  • The interface’s Visualizer presents robot intentions, while its Restrictor gathers human feedback for the Planner, which computes one action sequence per robot.These are the three components identified for the ARN framework.Source: Figure 2 · p. 3

IV. AR-MEDIATED NEGOTIATION-BASED FRAMEWORK

ARN uses an AR interface to support multi-turn, bidirectional communication and iterative negotiation between human and robot teammates. It repeatedly executes robot plans, presents robot status and trajectories, and replans when human feedback changes the active constraints.

IV. AR-MEDIATED NEGOTIATION-BASED FRAMEWORK

  • ARN generates plans for all robots while treating human feedback as constraints, using a task planner, motion planner, and robot team as inputs.The framework begins with initial robot states and goal states.Source: §IV · p. 3
  • For each robot not yet at its goal, ARN executes its current planned action and updates the action queue when that action is complete.The motion planner converts the current action into a trajectory, while the system records each robot’s current pose.Source: §IV · pp. 3–4
  • ARN renders robot trajectories and live poses on the AR device, then checks for human feedback about the plans.Feedback is converted into activated constraints; plans are regenerated when those constraints differ from the previous set.Source: §IV · p. 3
  • The planning-and-feedback loop continues until every robot reaches its goal.The framework checks the goal condition after each iteration and otherwise proceeds to another iteration.Source: §IV · p. 3
  • The real-world domain map includes three loading locations and a base station, with a door that humans can open to help robots enter.Participants saw colored trajectories in the non-AR baseline approach.Source: Figure 3 · p. 4

V. MULTI-ROBOT TASK PLANNING

The implementation uses Answer Set Programming for single-robot task planning and IIDP for joint plans across robot teams. The authors note that task planning is included for completeness rather than being the research’s main contribution.

Single Robot Task Planning

  • Answer Set Programming is used for robot task planning, with five domain actions: approach, opendoor, gothrough, load, and unload.An example plan approaches a door, opens it, and then goes through it.Source: §V · p. 4

Multi-Robot Task Planning

  • IIDP computes joint plans for robot teams and is described as efficient but suboptimal.The paper identifies the algorithm as iterative inter-dependent planning.Source: §V · p. 4
  • The task-planning section is included for completeness; the paper identifies the AR interface and ARN framework as its main contributions.Source: §V · p. 4

VI. EXPERIMENT SETUP AND RESULTS

The experiments compare ARN with non-AR baselines to evaluate collaboration efficiency and the experience of non-professional users.

VI. EXPERIMENT SETUP AND RESULTS

  • The evaluation tests whether ARN improves overall human-robot collaboration efficiency and produces a better experience for non-professional users than non-AR baselines.Source: §VI · p. 4

A. Real-world Experiments

In the real-world delivery study, ARN significantly improved total team task completion time over a 2D visualization baseline. Participants also rated ARN higher on several collaboration-experience questions, with significant differences on three items.

Experiment Setup

  • The study used three Turtlebot-2 robots on delivery tasks requiring human help with a door; 11 volunteers each tried ARN and the baseline in randomized order.Robots had no arms, so loading and unloading actions were not performed in the real-world trials.Source: A. Real-world Experiments · pp. 4–5

Results on Overall Task Completion Time

Baseline (2D Visualization)

  • The baseline displayed robot locations and planned trajectories on a 2D laptop map, while participants decided when to open the door for the robots.The paper notes that mapping locations on the map to real-world locations could be difficult.Source: A. Real-world Experiments · p. 5

Results on Overall Task Completion Time

  • Fig. 4(a) plots participants’ average task completion time against the sum of the three robots’ task times; ARN trials cluster near the bottom-left.Each point is a trial, and the ellipses show two-dimensional standard deviations.Source: A. Real-world Experiments · p. 5

Questionnaires

  • ARN survey scores were higher on Q2–Q5; improvements in Q2, Q3, and Q5 were significant (p-values < .001), while Q4’s improvement was not significant.The authors suggest that comparing mental-demand ratings may be difficult because the interfaces differ substantially.Source: A. Real-world Experiments · p. 5

B. Simulation Experiments

The simulation compares ARN with and without human feedback and reports lower completion times with feedback, including a significant improvement in overall team time. It combines a replicated delivery environment, simulated AR interaction, and scripted human behavior.

B. Simulation Experiments

  • Gazebo replicated the real-world delivery environment, including three robots waiting outside the base station.The simulation used amcl for localization and the ROS navigation stack’s move base package for path planning.Source: Figure 5 · p. 6, B. Simulation Experiments · p. 6
  • The evaluation compared Restrictor-activated feedback with a configuration that disallowed feedback, using 100 trials for each configuration.The simulation was designed to examine the negotiation-based component’s contribution after human-participant experiments had shown ARN’s effectiveness.Source: B. Simulation Experiments · pp. 5–6
  • The Unity-based virtual AR device displayed robot locations and planned trajectories and converted human feedback into task constraints.The virtual device could be rotated left or right, like the real-world setup.Source: B. Simulation Experiments · p. 6
  • The virtual human could work, tilt the AR device left or right, give two- or four-minute busy feedback, or open the door; without feedback, both feedback actions were removed.Door-opening actions were sampled every 20 seconds independently for each waiting robot at probability 0.6; after two-minute feedback, this fell to 0.2 for two minutes, returned to 0.6, and rose to 0.9 when the human finished the task.Source: B. Simulation Experiments · p. 6
  • Overall T_all improved significantly with feedback versus without feedback (p-value < .001), and average completion times were lower across all trials.Table I reports average completion times for the virtual human, the robot team, and the last robot.Source: Table I · p. 6, B. Simulation Experiments · p. 6

VII. CONCLUSIONS

ARN is an AR-mediated, negotiation-based framework for collaboration on non-transferable tasks when robots with limited capabilities need human help. Simulation and human-participant experiments found greater overall collaboration efficiency than a non-AR approach.

VII. CONCLUSIONS

  • ARN addresses collaboration tasks where humans and robots have non-transferable tasks and robots need human help at certain phases.The framework is intended for robots with limited capabilities.Source: §VII · p. 6
  • ARN supports negotiation by visualizing robots’ current and planned actions and incorporating human feedback into robot re-planning.Source: §VII · p. 6
  • Simulation and human-participant experiments found that ARN increased overall collaboration efficiency compared with a non-AR approach from the literature.Source: §VII · p. 6
Loading 1909.11227v3…