PaperlayerView exact revision on arXiv

Source-linked AI summary

A study on human-agent teaming through spoken interaction: the impact of human individual traits and agent characteristics

Lara Gauder, Martin Bernardo Meza, Javier Krick, Alejandro Masin, Luciana Benotti, Marcos Luis Pietto, Luciana Ferrer

arXiv:2610.04079v1cs.HCcs.AIcs.DB

TL;DR

This paper examines whether simple changes in agent communication can improve perceived teammate-likeness, an important but incompletely understood aspect of human-agent teaming. It tests this in a spoken collaborative game with agents that provide the same task-relevant information but use different social behaviors. The team-building agent was rated as more teammate-like, while human traits and speech patterns were also associated with agent perceptions or type.

  • Problem

    Communication, particularly speech, has received limited attention in human-agent teaming, and prior findings about which agents are perceived as teammates remain inconclusive.

    Source: §1 · p. 2

  • Method

    The study uses a spoken collaborative game and a protocol that varies agent communication while holding game-solving capability constant.

    Source: §1 · p. 3

  • Results

    The team-building agent was rated as more teammate-like across most subscales, including Ability, despite identical task-relevant information; performance did not significantly improve, and trust in automation and neuroticism were associated with more and less favorable perceptions, respectively.

    Source: §4 · p. 21

  • Takeaways & Limitations

    Simple team-building behaviors that leave task information unchanged can improve perceived teammate-likeness and are readily implementable in agents using oral communication.

    Source: §4 · p. 22

    The findings come from a structured referential game, so generalization to more complex, open-ended collaboration remains future work.

    Source: §4.1 · p. 22

Abstract

from arXiv · show

Human-agent teams are collaborative systems where humans and agents work interdependently to achieve shared goals. The success of such teams is associated with the human's perception of the agent as a legitimate teammate. This perception is thought to depend not only on the agent's capabilities and reliability but also on social factors. We explore whether simple changes in an agent's communicative behavior can influence the human's perception of the agent's teammate-likeness. To this end, we developed a protocol based on a collaborative game requiring spoken interaction, in which a human subject and a virtual agent collaborate to identify a target object and place it on a board. Each subject interacted with two agents which provided the same task-relevant information but differed in behavior. While one was a neutral tool-like agent, the other was a team-building agent that used simple social strategies such as empathy, politeness, and positivity. Our analysis shows that the team-building agent was perceived as significantly more teammate-like, even in terms of its ability, despite both agents having identical capabilities for game-solving. Further, subjects with a higher propensity to trust automated systems and lower neuroticism tended to have a more favorable perception of the agent. Finally, we found that subjects tended to change their prosodic patterns when talking to the team-building agent, as a classifier based on prosodic features was able to predict the type of agent with better-than-random performance. A practically relevant conclusion is that minimal changes in spoken communication behavior of the agent, easily implemented in a variety of scenarios, can have a significant positive impact on the human's perception of its teammate-likeness.

1. Introduction

Although human-agent teammate-likeness is shaped by agent, task, and human factors, evidence about communication—especially spoken communication—remains limited. This study tests whether simple social communication changes affect teammate perceptions while holding game-solving capability constant.

1. Introduction

  • Teammate-likeness may reflect agent autonomy and reliability, task interdependence, communication, and human traits such as dispositional trust.Source: §1 · pp. 1–3
  • Communication is comparatively underexplored in human-agent teaming: only 2 of 76 reviewed studies used speech as their sole modality, and designed manipulations may not be perceived as intended.The literature review argues that design and perception should be measured separately.Source: §1 · p. 2
  • The study compares a neutral tool-like agent with a socially communicative agent in a spoken game, manipulating communication while keeping game-solving capability unchanged.The team-building agent uses empathy, politeness, and positivity.Source: §1 · p. 3
  • The team-building agent was perceived as more teammate-like, including in altruism and relationship-building, regardless of presentation order.The agents provided the same task-relevant information.Source: §1 · p. 4
  • Subjects with greater trust in automation and lower neuroticism viewed agents more favorably, and prosodic features predicted agent type better than chance.Source: §1 · p. 4

1.1. Research Goals and Hypotheses

  • The hypotheses predict that the team-building agent improves performance and teammate-likeness, favorable ratings correlate with selected traits, and agent type elicits different prosody.Source: §1.1 · p. 4

2. Methods

The methods describe a spoken collaborative-game study comparing neutral and team-building agents, including its participant sample, measures, recordings, and interaction data. They also outline a remote two-session protocol and a smaller neutral-priming study examining participants’ perceptions and behavior.

2.1.1. Task Description

  • In the What-Where game, subjects identified one of 30 objects through spoken questions, then localized it on a board using further questions about relative position.The agent held the target information, while the subject held the board information, making the task interdependent; localization used no grid or measurement units.Source: §2.1.1 · pp. 5–6

2.1.3. Stimuli and Scenarios; 2.1.5. Session Structure

  • The protocol comprised 12 trials, each with two stages; all subjects followed the same trial sequence, but the first post-training agent varied and the final block used the other agent, with corresponding boards mirrored across agents.Source: §2.1.3 · p. 7

2.1.2. Wizard of Oz; 2.1.4. Agent Behavior

  • The neutral and team-building agents provided the same task-relevant information, but the latter added socially oriented language and behaviors such as greetings, enthusiasm, empathy, and encouragement.A Wizard-of-Oz operator encoded subjects’ speech, while automated responses implemented the condition-specific behavior; the operator could not see the board or performance feedback.Source: §2.1.2 · p. 6, §2.1.4 · pp. 7–8

2.2. Surveys

  • Before the main session, subjects completed demographic, Big Five, interpersonal-orientation, and automation-trust surveys; teammate-likeness surveys followed trials and blocks.The Big Five Inventory used 44 items, the Interpersonal Orientation Scale 25 items, and the Scale of Trust in Automated Systems 12 items.Source: §2.2 · p. 8, §2.2.1 · p. 8, §2.2.2 · p. 8

2.2.5. Autonomous Agent Teammate-Likeness

  • The teammate-likeness measure comprised 53 items across seven subscales, and subjects completed it at each block’s end alongside a four-item questionnaire after each trial.The items were adapted from the AATL model (Wynne and Lyons, 2018, 2019), with additional communication, autonomy, and agent-ability items.Source: §2.2.5 · pp. 9–10

2.3. Implementation Details

  • The web application connected subject and experimenter interfaces, recorded audio, and stored survey, task, and interaction data; agent speech used Argentine Spanish synthesis, with common responses prerecorded.Less frequent responses were generated on the fly, occasionally causing minor delays.Source: §2.3 · p. 10

2.4. Dataset Collection

  • The publicly released Teaming-UBA dataset comprises survey responses, complete interaction logs with both agents, and session speech recordings from 99 participants, whose mean age was 32.1 years (SD = 10.2).Eligibility required subjects to be older than 18 and younger than 65, have a Rioplatense accent, normal color vision, and normal or corrected-to-normal visual acuity.Source: §2.4 · p. 10
  • Before gameplay, participants completed a remote intake, consent, equipment and audio checks, and sociodemographic and personality-related surveys.A research assistant checked audio quality before participants scheduled the synchronous session.Source: §2.4 · p. 11
  • In the synchronous session, a research assistant acting as an undisclosed Wizard of Oz checked audio and background noise before sharing the experimental platform.Participants also recorded a phonetically balanced text and described five scenes for at least 30 seconds each; these agent-free recordings served as per-subject neutral references.Source: §2.4 · p. 11
  • Participants completed two blocks of four trials, one with each agent, after practice; the active-priming version primed the team-building agent before its block.Participants completed intermediate surveys after each trial and a final survey after each agent; the second session lasted about 60 minutes.Source: §2.4 · p. 11
  • A second study gave both agents equivalent priming to isolate priming’s influence; 18 participants completed this neutral-priming version.The group’s mean age was 36.1 years (SD = 8.9).Source: §2.4 · p. 11

3. Database Analysis

The database analysis draws on 74 sessions, with a 15-subject neutral-priming comparison, to examine game performance, teammate-likeness ratings, individual traits, and speech analysis. It also evaluates agent-type prediction from prosodic features aggregated across trials within blocks: learned features predicted significantly above chance, while hand-crafted features did not.

3.1.1. Impact of Difficulty and Trial Number on Game Performance

  • Game performance improved on easier trials and in the second block, but did not differ significantly between the team-building and neutral agents.The easy trial had higher overlap and lower distance than hard and medium trials in both blocks; the second block showed a pattern consistent with training. Neither distance nor subject turns differed significantly by agent at any trial.Source: §3.1.1 · p. 13, Figure 4 · p. 13, §3.1.2 · p. 14

3.2.1. Impact of the Agent Type on the Final Survey Scores

  • The team-building agent scored higher than the neutral agent on all seven AATL subscales and their mean, significantly on five subscales including Ability, despite identical task-solving capabilities.The largest mean differences were for Relationship-Building (M=0.557) and Altruism (M=0.517); Autonomy and Cooperative differences were not significant.Source: §3.2.1 · p. 14, Figure 5 · p. 15

3.2.2. Impact of the Agent Type on the Final Survey Scores by Agent Order

  • Relationship-Building, Altruism, and mean ratings favored the team-building agent under both orders, while other subscale advantages were significant only for N-first subjects.Across-order differences and first-block-only comparisons were not significant for any subscale, so the order-specific pattern warrants caution; the authors suggest a contrast effect may explain some task-functional ratings.Source: Figure 6 · p. 15, §3.2.2 · pp. 15–16

3.2.3. Impact of the Agent Type on the Intermediate Survey Scores

  • Intermediate ratings favored the team-building agent for Relationship-Building across all four trials, while the across-subscale mean differed significantly only in trials 3 and 4.No other subscale differed significantly at an individual trial; the authors note that single-item intermediate subscales may make these scores less robust.Source: Figure 7 · p. 16, §3.2.3 · p. 17

3.2.4. Comparison with Neutral-Priming Sessions

  • Neutral priming did not significantly change the perceived rating difference between agents, supporting the interpretation that the team-building agent’s behavior drove its advantage.No subscale or across-subscale mean differed significantly between the 74 active-priming subjects and 15 neutral-priming subjects after correction.Source: §3.2.4 · p. 17

3.2.5. Impact of the Subjects’ Individual Traits on Final Survey Scores

  • Trust in automation was positively associated and neuroticism negatively associated with mean teammate-likeness across agents; social comparison was associated with a larger Communication rating difference.These associations were weak trends, and the individual-trait results only partially supported the hypothesis; extraversion, openness, and need for affiliation were not significantly associated with the across-agent mean.Source: §3.2.5 · pp. 17–18, Figure 8 · p. 18

3.3. Analysis of the Impact of the Agent Type on the Subjects’ Speech

  • The speech analysis used each subject’s eight Stage 2 recordings, four per agent, after detecting and auditing the subject’s speech intervals.Stage 2 was selected because it elicited richer, longer, and more variable speech than Stage 1; silences and agent speech leaking into recordings were removed.Source: §3.3 · p. 19
  • The study compared handcrafted F0 and loudness descriptors with learned prosodic embeddings, combining interval-level features into trial-level representations.Hand-crafted trial vectors were 10-dimensional; learned embeddings were 16-dimensional per speech segment and fused across trial segments.Source: §3.3 · p. 19
  • A random-forest classifier predicted agent type at the block level by averaging scores across each agent’s four trials.Evaluation used speaker-level 7-fold cross-validation, repeated with 10 seeds; performance was measured by pooled-prediction AUC with bootstrapped confidence intervals.Source: §3.3 · pp. 19–20
  • Learned features predicted agent type significantly above chance for raw and speaker-normalized inputs, while hand-crafted features did not.The hand-crafted configurations’ confidence intervals included chance-level AUC = 0.5; learned features were significantly above chance for those two normalization options.Source: Figure 10 · p. 20, §3.3 · p. 20
  • Trial-position normalization slightly degraded performance for both feature sets, suggesting normalization statistics could not be robustly estimated from the available data.Source: §3.3 · p. 20
  • Learned features improved prediction but did not reveal which specific prosodic aspects varied with agent type.Source: §3.3 · p. 20

4. Discussion and Conclusions

The team-building agent was rated more teammate-like than the neutral agent despite identical task-relevant information, while game performance did not significantly improve. Individual traits were associated with perceptions, and agent type affected subjects’ prosody; the study’s scope is limited by sample size and a structured game setting.

4. Discussion and Conclusions

  • The protocol paired each subject with neutral and team-building agents while holding task-relevant information constant, and the released Teaming-UBA dataset contains 117 subjects.The team-building agent used simple social communication strategies; the data include Argentine Spanish speech, survey responses, and interaction logs.Source: §4 · p. 21
  • The team-building agent did not elicit more subject interactions or significantly improve game performance, contrary to the stated hypothesis.The authors relate this to prior work where teammate framing improved affect and perception without improving task performance absent a structured team-building intervention.Source: §4 · p. 21
  • The team-building agent was rated more teammate-like across most subscales, including Ability, despite identical quality and quantity of game-relevant information.The relationship-building difference had the largest effect, and the comparison showed no evidence that priming significantly explained the more favorable perception.Source: §4 · p. 21
  • Higher trust in automation was associated with more favorable agent perceptions, while neuroticism was associated with less favorable perceptions; other tested traits showed no significant association.The authors report partial support for their individual-traits hypothesis.Source: §4 · p. 21
  • Agent type had a minor but significant effect on subjects’ prosodic patterns, as measured by a model trained on learned prosodic features.The agents supplied the same task-relevant information and the task was identical, so the authors expected differences in how subjects spoke rather than what they said.Source: §4 · p. 21
  • The authors conclude that simple team-building behaviors can improve perceived teammate-likeness without changing the amount or quality of task-related information.They describe these changes as easy to implement across agents that rely on oral communication.Source: §4 · p. 22

4.1. Limitations

  • Splitting the sample by agent order reduced observations per group and statistical power, especially for perception analyses where consistent trends did not always reach significance.The authors state that a larger sample would support more reliable conclusions about order effects.Source: §4.1 · p. 22
  • The findings come from a structured referential game; generalization to more complex, open-ended collaborative settings remains future work.Source: §4.1 · p. 22

7. Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors report using generative AI and AI-assisted technologies during manuscript preparation.

7. Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • The authors used Gemini and Claude for writing support and light language editing, reviewed and edited the content, and used Gemini to generate part of the graphical abstract.Source: §7 · p. 22

8. CRediT authorship contribution statement

The statement assigns contributions across the research team; the appendix also describes protocol design materials.

8. CRediT authorship contribution statement

  • The authors assign contributions across conceptualization, methodology, software, formal analysis, investigation, data curation, writing, supervision, and funding acquisition.Source: §8 · p. 22

A. Protocol Design Details

  • The protocol-design appendix covers board-difficulty criteria, the boards used for each trial, and the priming text shown before each agent.Source: A. Protocol Design Details · p. 24

A.1. Difficulty Level Specifications

Board difficulty varied according to target placement and the proximity and alignment of reference objects.

A.1. Difficulty Level Specifications

  • Easy boards placed one or two objects one cell from the target, which lay on a diagonal between two objects and aligned horizontally and vertically with references.Source: §A.1 · p. 24
  • Medium boards had no object one cell from the target; the target lay on a diagonal between two objects and was partially aligned with references.Source: §A.1 · p. 24
  • Hard boards had no object one cell from the target, and the target was not on a diagonal between object pairs, though it was partially aligned with references.Source: §A.1 · p. 24

A.2. Blocks Description

The experiment used three trial blocks, with two assigned to the agents, and introduced the agents through ratings and reviews. Surveys assessed personality and affiliation motivation, with adequate internal consistency reported for both sets of measures.

A.2. Blocks Description

  • The experiment used one training block and two blocks randomly assigned to the agents, with target objects and board configurations varying across blocks.The Stage 2 board configuration also varied in difficulty.Source: §A.2 · p. 24

A.3. Agent Priming Materials

  • Both agents were rated 4 out of 5 for ability, while the team-building agent received 5-star communication and cooperation ratings and the neutral agent received 3 stars.Introductory reviews were presented as coming from previous subjects, although the research team had designed them.Source: §A.3 · p. 25, Figure A.1 · p. 25

B.1. The Big Five Inventory

  • The Big Five questionnaire used 5-point Likert items based on the Argentine Spanish adaptation by Genise et al. (2020).Items covered Agreeableness, Conscientiousness, Extraversion, Openness to Experience, and Neuroticism.Source: §B.1 · pp. 25–26

Neuroticism

  • Big Five internal consistency ranged from α=0.69 for Agreeableness to α=0.83 for Openness to Experience and Extraversion.The five reported values were 0.83 for Openness, 0.82 for Conscientiousness, 0.83 for Extraversion, 0.69 for Agreeableness, and 0.79 for Neuroticism.Source: Neuroticism · p. 27

B.2. The Interpersonal Orientation Scale

  • The Interpersonal Orientation Scale measured affiliation motivation with 5-point items across Attention, Emotional Support, Positive Stimulation, and Social Comparison.Its four subscales had internal-consistency values from α=0.71 to α=0.84.Source: §B.2 · p. 27, Social Comparison · p. 28

Social Comparison

B.3. The Scale of Trust in Automated Systems

The study measured trust in automated systems and used a final questionnaire to assess agent teammate-likeness across several dimensions. The trust scale showed good internal consistency, while the final questionnaire drew on existing and study-specific items.

B.3. The Scale of Trust in Automated Systems

  • Trust in automated systems was assessed using Jian et al.’s (2000) scale, with 7-point ratings and good internal consistency (α=0.86).Distrust-related items were reverse-coded before computing internal consistency.Source: §B.3 · p. 28

B.4. Autonomous Agent Teammate-Likeness

  • The final 53-item questionnaire combined items adapted from teammate-likeness and Godspeed measures with conversation-level annotations and study-specific items.Subjects completed it after interacting with each agent, rating items on a 5-point scale.Source: §B.4 · p. 28, Ability · p. 28, Autonomy · p. 29

Ability

  • Teammate-likeness items assessed ability, altruism, autonomy, communication, cooperation, relationship-building, and anticipation of needs or actions.Items covered task competence, helping, independent action, information exchange, interdependence, rapport, and anticipation.Source: Ability · p. 28, Altruism · p. 28, Autonomy · p. 29

Synchronized Mental Model

The protocol repeatedly assessed subjects’ perceptions of each agent during the game and evaluated the final questionnaire’s internal consistency. Its short intermediate survey sampled four teammate-likeness dimensions.

Synchronized Mental Model

  • The final questionnaire’s seven subscales had acceptable internal consistency (α≥0.71) for both agents.Reliability was computed separately for each agent across all 117 subjects.Source: Table B.1 · p. 30, Synchronized Mental Model · p. 30
  • After each trial, subjects completed a 5-point intermediate questionnaire with four items on helping, communication, rapport, and ability.The items were selected from the longer final questionnaire.Source: Synchronized Mental Model · p. 30
  • The intermediate survey’s ability item asked whether the agent was competent at solving the tasks.Source: Synchronized Mental Model · p. 30

C. Dataset Collection Details

Data collection separated equipment and individual-trait surveys from the game session, where subjects interacted with both agents and completed perception surveys. Subjects also provided read-aloud and spontaneous speech recordings.

C. Dataset Collection Details

  • The first session checked equipment and collected individual-trait surveys; the second involved the game with both agents and intermediate and final perception surveys.Source: C. Dataset Collection Details · p. 30
  • Subjects read a phonetically balanced Argentine Spanish text and described at least five pictured situations to provide spontaneous speech, speaking for at least 30 seconds.Recording could only be stopped after 30 seconds had elapsed.Source: C. Dataset Collection Details · p. 30

D. Additional Database Analyses

This section adds database analyses of task engagement and compares active- and neutral-priming sessions.

D. Additional Database Analyses

D.1. Engagement in the Task

The study identified low-engagement trials by combining poor placement accuracy with few speech interactions, then excluded four subjects who met the criterion repeatedly.

D.1. Engagement in the Task

  • A trial was flagged when its distance score was in the top quartile and its number of subject speech intervals was in the bottom quartile.A speech interval is continuous speech containing pauses no longer than 0.5 seconds.Source: §D.1 · p. 30
  • The protocol included two consecutive interactions with each agent, with a priming stage before each and an AATL questionnaire after each trial.A break of at least one minute separated the two agent blocks.Source: Figure C.1 · p. 31
  • Four subjects were excluded after repeatedly combining poor placement accuracy with few speech interactions across trials.Three met the outlier criterion in four or five trials; a fourth, who met it in three trials, was excluded to preserve gender and agent-order balance.Source: §D.1 · pp. 30–31

D.2. Effect of Priming on Agent Perception

A control condition removed the favorable priming difference between agents; ratings still favored the team-building agent, with no significant priming-group differences.

D.2. Effect of Priming on Agent Perception

  • In the active-priming condition, introductory ratings and feedback portrayed the team-building agent more favorably in teaming behavior.Source: §D.2 · p. 31
  • The neutral-priming control presented both agents with the same ratings and similar hypothetical-user feedback.This removed the difference in priming between the agents for comparison with the active-priming condition.Source: §D.2 · pp. 31–32
  • No AATL subscale or across-subscale mean differed significantly between priming groups, and the team-building agent was favored in both.The reported nonsignificant comparisons provide no evidence that priming type influenced perceptions of the agents’ teaming capabilities.Source: §D.2 · p. 32

E. Learned Prosodic Representations

The learned prosodic representations use a two-stage teacher–student model, then approximate English-trained intonation units with pause-based speech segments for this study.

E. Learned Prosodic Representations

  • The teacher is a GRU autoencoder with a 16-dimensional bottleneck, trained on pitch and energy signals using masked reconstruction and phone-conditioned decoding.The design aims to retain prosody-relevant information while limiting speaker and text information in its embeddings.Source: E. Learned Prosodic Representations · p. 32
  • The student model outputs one 16-dimensional vector per speech segment and avoids pitch or energy extraction at deployment.It is a two-layer bidirectional GRU trained to predict embeddings from clean audio produced by the teacher model, with training corruption and background noise augmentation.Source: E. Learned Prosodic Representations · p. 32
  • Because the model’s training pipeline used English intonation units, the study approximated them with inter-pausal units for Spanish speech.Voice activity detection identified speech intervals; intervals longer than six seconds were split at internal pauses, and segments of one second or less were discarded.Source: E. Learned Prosodic Representations · pp. 32–33

Related papers

Loading 2610.04079v1…