Source-linked AI summary
Predicting Team Performance from Communications in Simulated Search-and-Rescue
Ali Jalal-Kamali, Nikolos Gurney, David Pynadath
TL;DR
Agents that support human teams need to understand naturally occurring communication, which can reveal team characteristics and performance. This paper analyzes search-and-rescue transcripts with topic modeling and clustering, finding that conversation-derived clusters relate to outcomes and can be predicted early in a trial.
Problem
Agents assisting teams need to model individuals and their relationships, while naturally occurring communication can be difficult for AI systems to understand.
Source: §1 · p. 1Method
The study applies LDA topic modeling to team transcripts and clusters trial-level topic probability distributions, relating the clusters to team outcomes.
Source: §3.4 · p. 2, §3.5 · p. 2Results
Cluster assignment was significantly related to trial performance (p=0.0008), and struggling clusters showed patterns including inaction, uneven workload, and poor communication balance.
Source: §3.5 · p. 2Takeaways & Limitations
The proposed pipeline uses transcript portions to predict trial clusters and describes potential performance prediction and intervention as early as 10% to 30% of a trial.
Source: §3.6 · p. 2
Abstract
from arXiv · showhide
Understanding how individual traits influence team performance is valuable, but these traits are not always directly observable. Prior research has inferred traits like trust from behavioral data. We analyze conversational data to identify team traits and their correlation with teaming outcomes. Using transcripts from a Minecraft-based search-and-rescue experiment, we apply topic modeling and clustering to uncover key interaction patterns. Our findings show that variations in teaming outcomes can be explained through these inferences, with different levels of predictive power derived from individual traits and team dynamics.
1 INTRODUCTION
Agents supporting human teams must model both individuals and their interactions, but naturally occurring communication is not oriented toward AI systems. That communication can nevertheless provide information about team characteristics and performance.
1 INTRODUCTION
- Team communication can reveal characteristics and performance, but making use of it depends on an agent’s ability to understand naturally occurring conversation.The paper frames this as a challenge for agents that assess or assist teams during task performance.Source: §1 · p. 1
2 EXPERIMENTAL TESTBED DATA
The study analyzes communication and team measures from a three-person Minecraft search-and-rescue experiment. Its data include transcripts, pre-trial profiles, and measures of team effectiveness collected during trials.
2 EXPERIMENTAL TESTBED DATA
- Teams of three completed an urban search-and-rescue task in Minecraft, with medic, engineer, and transporter roles.The medic treated victims, the engineer cleared obstacles, and the transporter moved victims.Source: §2 · p. 1
- The analysis uses audio transcripts alone, alongside pre-trial BEARD profiles and dynamic TED measures of team effectiveness.The transcripts exclude simulation logs and AI advisor data; TED measures team processes throughout trials.Source: §2 · p. 1
3 METHODOLOGY AND RESULTS
The analysis seeks to identify teams that may need intervention and how early communication patterns can support that decision. It relates individual profiles, trial measures, and transcript-derived clusters to performance, then outlines an early prediction and intervention pipeline.
3.1 BEARD Profiles
- Anger correlated negatively with performance, social perceptiveness positively, and transporting skill unexpectedly negatively, possibly reflecting overconfidence.The authors report these relationships from linear regression of BEARD variables against performance.Source: §3.1 · p. 1
3.2 TED Measures
- TED regressions showed positive coefficients for process-effort-agg and comms-total-words, and a negative coefficient for process-skill-use-agg.The analysis selected aggregate, time, and communication-based measures to avoid interdependencies among TED variables.Source: §3.2 · p. 1
3.3 Pre-Processing of Transcripts
- The team split session transcripts into individual trials, removed administrative and redundant material, and retained 222 unique trial transcripts for preprocessing.Document-term matrices used lowercase conversion, punctuation and number removal, and stopword filtering.Source: §3.3 · p. 2
3.4 Intra-team Communication Analysis
- LDA topic modeling evaluated topic counts from 2 to 20 with 100 runs per count and selected 12 topics using average probabilistic coherence.The most probable words showed shared search-and-rescue vocabulary as well as distinct patterns in topics 2, 3, and 5.Source: §3.4 · p. 2, Table 1 · p. 2
3.5 Categorization Abstraction
- Cluster assignment was significantly related to performance (p=0.0008); Cluster 5 had coefficient -196.30, Cluster 2 -147.753, and Cluster 3 -115.095.These were respectively the lowest-, second-lowest-, and third-lowest-performing clusters.Source: §3.5 · p. 2
- Cluster 5 showed negative correlation with social perceptiveness and positive correlations with spatial ability and game skills.TED patterns included high inaction, low process effort, coverage, and triaging, minimal workload distribution, and poor communication balance in Cluster 5.Source: §3.5 · p. 2
- The reported cluster patterns associate effective teams with balanced communication and workload distribution, and struggling teams with more fragmented interaction.Cluster 2 showed negative correlation with communication equity and uneven task distribution; Cluster 3 showed inconsistent coordination and mixed communication patterns.Source: §3.5 · p. 2
3.6 Early Prediction and Intervention Pipeline
- 47% accuracy using the first 1/10 of a transcript and 76% using 1/3 support early trial-cluster prediction.The pipeline uses these predictions to identify low-performing trials and describes prediction and intervention as possible as early as 10% to 30% of a trial.Source: §3.6 · p. 2