PaperlayerView exact revision on arXiv

Source-linked AI summary

CREW: Facilitating Human-AI Teaming Research

Lingyu Zhang, Zhengran Ji, Boyuan Chen

arXiv:2408.00170v3cs.HCcs.AIcs.LG

TL;DR

Human-AI teaming research lacks comprehensive platforms that support multidisciplinary work and more than simplified, single-task scenarios. CREW brings modular tasks, real-time teaming, and human-data collection into one research platform and benchmarks human-guided reinforcement learning. Across tasks, c-Deep TAMER beat baselines on Bowling, learned faster on Find Treasure, and performed similarly on Hide-and-Seek.

  • Problem

    Existing Human-AI teaming platforms often support oversimplified, single-task scenarios or focus on human-teaming research or multi-agent AI algorithms, while the field lacks unified support for scalable, multidisciplinary research.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    CREW provides a modular Human-AI teaming platform with adaptable tasks, real-time interaction, parallel sessions, and human-data collection, and benchmarks human-guided reinforcement-learning algorithms.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    Across tasks, c-Deep TAMER beat RL baselines on Bowling, learned faster on Find Treasure, and performed similarly on Hide-and-Seek; Bowling's average margin was 10 scores for the top 15 subjects.

    Source: §4.2 · p. 12

  • Takeaways & Limitations

    CREW supported 50 human subject studies within a week and provides infrastructure for multidisciplinary, reproducible, and scalable Human-AI teaming research.

    Source: §1 · p. 2

    The reported benchmark involved 50 human subjects.

    Source: §5 · p. 12

Abstract

from arXiv · show

With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.

1 Introduction

Human-AI teaming research needs a unified, multidisciplinary platform, but existing platforms are limited in task variety, real-time interaction, human-data collection, and benchmarking. CREW addresses these needs with modular infrastructure and benchmarks human-guided reinforcement learning with 50 human studies conducted within a week.

1 Introduction

  • Existing Human-AI teaming platforms are often single-task, lack real-time feedback, and do not provide unified support for human physiological data collection and benchmarks.The field also needs scalable, multidisciplinary hypothesis verification.Source: §1 · p. 2
  • CREW combines an extensible, open platform with real-time communication, hybrid Human-AI teaming, parallel sessions, and comprehensive human and agent data collection.Its modular algorithm components are designed to align with machine-learning community practices.Source: §1 · p. 2
  • CREW benchmarks real-time human-guided reinforcement-learning algorithms alongside reinforcement-learning baselines and supported 50 human subject studies within a week.The platform also includes cognitive tests to investigate how human individual differences relate to effectiveness in training AI agents.Source: §1 · p. 2

2 Related Work

Prior work spans human-AI algorithms, interfaces, workflows, and ethical questions, but the paper identifies a gap in unified infrastructure for multidisciplinary Human-AI teaming research. Existing platforms are constrained in task extensibility, physiological data collection, participant scale, and openness.

Human-AI Teaming Research

Human-AI Teaming Platform

  • Existing platforms include Overcooked for coordination and teamwork and StarCraft II Learning Environment for adversarial interaction and learning from demonstrations.Overcooked Carroll et al. (2019); StarCraft II Learning Environment Vinyals et al. (2017).Source: §2 · p. 2
  • Many platforms limit task variety and interaction modes, omit human physiological data, involve few participants, and are closed-source, restricting extensibility and generalizable evaluation.The paper notes that many studies involve fewer than 10 subjects, mostly fewer than 5.Source: §2 · p. 3

Real-Time Human-Guided Learning

  • Real-time human-guided learning methods integrate feedback into policy training, while CREW combines online training and feedback with physiological-data collection and parallel experiments.TAMER and Deep TAMER model discrete feedback; COACH and DeepCOACH model inconsistent feedback through an advantage function.Source: §2 · p. 3

3 CREW: Design and Components

CREW is designed as an extensible platform for multidisciplinary Human-AI teaming research, combining adaptable tasks, real-time human interaction, parallel experimentation, and multimodal data collection. Its components support human-guided learning studies and cognitive research, including configurable tasks, algorithm modules, and an experiment pipeline.

3.1 Platform Vision and Design Philosophy

  • CREW’s open, extensible design supports multidisciplinary researchers and lets them add features or game logic without changing complex infrastructure components.The design vision includes collaboration across human-AI interaction, machine learning, workflow design, and human analysis and training.Source: §3.1 · pp. 3–4

3.4 Multiplayer and Parallel Sessions

  • CREW’s design supports real-time communication, hybrid human-AI teams, parallel sessions, and collection of active human feedback and passive physiological signals.Parallel sessions can run independently across locations to support larger-scale experiments.Source: §3.1 · p. 4

3.2 Environments

  • CREW includes four example tasks, from simple Bowling and single-player Find Treasure to competitive 1v1 and multiplayer NvN Hide-and-Seek, with visual or structured-state inputs.Researchers can extend existing tasks or increase their complexity; the examples were selected for human-guided reinforcement-learning benchmarks.Source: §3.2 · p. 5

3.3 Human and Agent Role Assignment

  • CREW assigns human and AI roles and lets viewers provide direct feedback, including continuous feedback over time, or take control of agents through teleoperation.Role assignment supports experiments involving humans and AI agents with different functions.Source: §3.3 · p. 7

3.5 Human and Agent Data Collection

  • CREW collects human feedback and synchronized audio, gaze, pupillometry, EEG, and ECG data, alongside agent data such as observations, actions, rewards, and policy weights.The platform uses Lab Streaming Layer for time synchronization across machines.Source: §3.5 · p. 8

3.6 Designing Algorithms

  • CREW’s Python, PyTorch, and TorchRL algorithm components provide state-of-the-art human-guided methods, reinforcement-learning baselines, and a uniform protocol for switching tasks and algorithms.Simulated human feedback providers are intended for debugging and not as replacements for real human evaluations.Source: §3.6 · p. 8

3.7 Modular Pipeline Design for Quantifying Human Characteristics

  • The cognitive-test pipeline combines media, inter-trial intervals, scripts, and Unity builds, with one-click sequential execution and restart from interruption points.The pipeline is intended to quantify human characteristics alongside Human-AI experiments.Source: §3.7 · p. 9

A B C Cognitive Tests

CREW was used to benchmark continuous-action human-guided reinforcement learning across three games with 50 human trainers. Results compare c-Deep TAMER with reinforcement-learning baselines and explore how cognitive-test scores relate to agent performance.

A B C Cognitive Tests

  • The experiment workflow lets researchers organize cognitive tests and Human-AI experiments in sequence through CREW's deployment pipeline.Participants completed six cognitive tests before guiding the agent.Source: A B C Cognitive Tests · p. 9, §4.1 · p. 9

4.1 Experiment Setup

  • The benchmark used Bowling, Find Treasure, and 1v1 Hide-and-Seek; the latter two had 15-second episode limits, and agents learned directly from visual inputs.Find Treasure and Hide-and-Seek checkpoints were evaluated for 100 episodes; Bowling checkpoints were evaluated for one game.Source: §4.1 · pp. 9, 11
  • 50 human trainers completed cognitive tests before guiding an agent through Bowling, Find Treasure, and Hide-and-Seek; parallel sessions enabled all studies within one week.The experiment lasted about 40 minutes per participant, including 10 minutes of tests and 30 minutes of agent guidance.Source: §4.1 · pp. 9–10
  • The benchmark compares continuous-action c-Deep TAMER, which learns from discrete human feedback, with DDPG, SAC, and a DDPG heuristic using dense feedback rewards.c-Deep TAMER trains a feedback model to predict human feedback for state-action pairs and selects actions to maximize predicted feedback.Source: §4.1 · pp. 10–11

4.2 Results

  • Cognitive-test scores correlated positively with guided performance across tasks; Find Treasure was significant (p < 0.01), while total scores correlated significantly with rotation, fitness, and overall scores.The authors characterize these findings as early exploration and do not draw a general conclusion.Source: §4.2 · pp. 11–12
  • 10 scores: among top-15 trainers, c-Deep TAMER beat RL baselines on Bowling; it learned faster on Find Treasure but matched baselines on Hide-and-Seek, struggling with greater task complexity.Similar conclusions held across all 50 subjects.Source: §4.2 · pp. 11–12
  • Find Treasure performance correlated significantly with behavior theory, rotation, and fitness scores; Hide-and-Seek correlated with reflex scores, plausibly reflecting spatial reasoning and rapid task dynamics.The authors suggest timely feedback helps align with relevant state-action pairs in Hide-and-Seek.Source: §4.2 · p. 12

5 Conclusion, Limitation, and Future Work

CREW provides a modular platform for Human-AI teaming research, with built-in tasks and tools for human-AI interaction and data collection. Its benchmark involved 50 human subjects and examined how individual differences relate to agent-guiding performance, while future work includes broader tasks, algorithm coverage, and physiological data analysis.

5 Conclusion, Limitation, and Future Work

  • CREW benchmarked a state-of-the-art human-guided RL algorithm against baselines with 50 human subjects and examined links between individual differences and agent-guiding performance.The conclusion identifies broader task coverage, more algorithm benchmarks, and additional physiological data processing and analysis as future work.Source: §5 · pp. 12–13
  • CREW combines extensible environments, real-time human-AI communication, hybrid teaming, parallel sessions, multimodal feedback, physiological data collection, and built-in tasks and baselines.Its algorithm components are designed to be modular and compatible with machine-learning community practices.Source: §5 · p. 12
  • Future work includes more diverse and challenging tasks, including multiplayer tasks with complex strategies and robotics environments requiring physics understanding.The paper characterizes CREW as still in its early efforts across several critical aspects.Source: §5 · p. 12
  • CREW has benchmarked only several algorithms, and further support for human physiological data processing and analysis remains to be investigated.The authors also identify benchmarking algorithms that were not fully open-sourced in a unified environment as a future direction.Source: §5 · pp. 12–13

7 Appendix

The appendix points to CREW’s open-source code and documentation and lists the technologies used to implement its environments and algorithms.

7.1 Accessing CREW

  • CREW’s open-source code and detailed documentation are available in its GitHub repository.The passage provides the repository address.Source: §7.1 · p. 18

7.2 Platform details

  • CREW’s environments use Unity and ML Agents, with Netcode for GameObjects and Nakama; its algorithms use torchrl.The passage specifies Unity 2021.3.24f1, ML Agents 2.3.0-exp.3, Netcode for GameObjects 1.3, Nakama Unity 3.6.0, and torchrl 0.3.0.Source: §7.2 · p. 18

7.3 Hyperparameters

The section points to a table of experimental hyperparameters, including shared settings and entries that vary across the listed configurations.

7.3 Hyperparameters

  • The listed configurations share γ=0.99, learning rate 1e-4, max_grad_norm 0.1, and target update polyak 0.995.The table summarizes hyperparameter settings for the experiments.Source: §7.3 · p. 18
  • Batch size is 16 in the first listed configuration and 240 in each of the other two.Source: §7.3 · p. 18
  • The first listed configuration uses bowling[0.2, 4] and others[0.2, 1] for the credit assignment window; the other two entries are dashes.Source: §7.3 · p. 18

7.4 Algorithm Statistics

The section reports average algorithm updates across three tasks and notes a lower update count for c-Deep TAMER than RL baselines on Find Treasure.

7.4 Algorithm Statistics

  • On Find Treasure, where c-DeepTAMER showed the most advantage, c-Deep TAMER had relatively fewer average updates than the RL baselines.Source: §7.4 · p. 18
  • The average-update statistics cover Bowling, Find Treasure, and Hide and Seek.The table is titled “Average number of updates.”Source: §7.4 · p. 18

7.6 Full Cognitive Test Analysis

The section presents full cognitive-test and c-Deep TAMER score analyses, including regression plots that summarize correlation analysis.

7.6 Full Cognitive Test Analysis

  • The full analysis covers cognitive-test results and c-Deep TAMER scores.The supplied passages do not state specific findings or values.Source: §7.6 · p. 18
  • Figure 12 presents linear regression plots for correlation analysis, including coefficients and p-values.Source: §7.6 · p. 18
  • Figure 11 compares guided-AI performance for the top and bottom 25 cognitive test scorers.Source: §7.6 · p. 19

7.7 Instruction Video Script

The appendix scripts guide participants through cognitive tests and human-guided AI games, from session orientation to feedback and game-specific objectives.

7.7 Instruction Video Script

  • The session begins with cognitive and gaming proficiency tests, proceeds to games with AI algorithms, and ends with a feedback survey.Source: §7.7 · p. 19
  • Five cognitive games take about seven minutes and assess precision, reflexes, prediction, problem-solving, and spatial awareness, with three-second intertrial intervals.Source: §7.7 · p. 19
  • The eye-alignment task has six five-second trials, while the reaction task requires a rapid click after the screen turns green.Clicking during yellow or failing to respond within two seconds after green results in failure.Source: §7.7 · p. 19
  • The theory-of-behavior task asks participants to predict a ball’s position one second after it pauses; rotation and mental fitting are also named as an instruction topic.Source: §7.7 · pp. 19–20
  • One test uses 12 questions with eight seconds per response; the spatial-mapping script uses six video questions, answerable during playback or within three seconds after pausing.Source: §7.7 · p. 20
  • Participants guide AI agents in three games for about 30 minutes, giving positive or negative feedback based on agent performance.Source: §7.7 · p. 20
  • The guided games ask participants to maximize bowling scores, find treasure quickly, and help an agent chase and catch a hider.Source: §7.7 · pp. 20–21

7.7.1 Compensation

  • The appendix states a $20 participant payment and labels Figure 12 as linear-regression plots of correlation analysis.Source: §7.7.1 · pp. 21–22
Loading 2408.00170v3…