UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library

1 Introduction

• Polyadic CAs support conversations involving multiple users and bidirectional human-AI and human-human interaction, creating social complexities beyond dyadic interaction.These include multi-party interaction, social roles, group hierarchy, and social tension.Source: §1 · p. 2
• Prior CA reviews do not address how polyadic CAs are designed and evaluated or their effects on human-human interaction.The review frames this as a gap in understanding how polyadic CAs can address social-interaction challenges and influence UX.Source: §1 · p. 2
• The review screened 1,302 ACM papers and identified 36 polyadic and 135 dyadic CA papers with user-evaluation results.It examined interaction challenges, evaluated effects, and issues overlooked in CA design.Source: §1 · p. 2
• The paper synthesizes design challenges and practices, identifies overlooked issues, and compares research interests and evaluation metrics in dyadic and polyadic CA studies.It also proposes boundary-aware CA design and identifies future research opportunities.Source: §1 · p. 2

2.1 Landscape of CAs

• CA terminology and typologies overlap, with classifications spanning modality, social scale, knowledge domain, goals, control, embodiment, design approach, platform, and application domain.The literature uses terms including chatbots, voice interfaces, embodied agents, and social robots.Source: §2.1 · pp. 2–3
• The paper includes conversational-agent terminology broadly, including embodied conversational agents that support verbal and nonverbal conversational interaction.The review includes ECA papers when they study UX in conversational interaction.Source: §2.1 · p. 3

2.2 Gaps in Existing Literature Reviews of CAs

• Existing CA reviews largely examine dyadic human-CA interaction, including engagement, UX, relationships, emotional intelligence, personalization, trust, and human-likeness.They do not address the unique design challenges of polyadic CAs.Source: §2.2 · p. 3
• Polyadic CAs mediate human-human communication and can support group members’ emotional awareness and positive emotions, but their empirical use and effects across scenarios remain little understood.Prior work describes applications including personal assistants, customer service, healthcare, and collaboration support.Source: §2.2 · p. 3
• The review focuses on papers with user evaluations because CA studies report pragmatic failures and difficulty sustaining user engagement over time.The paper frames its review as a resource for identifying practices, gaps, lessons, and future research directions.Source: §2.2 · p. 3

3 Method

• The review used four stages: defining criteria and sources, searching, selecting eligible papers, and analyzing them with mixed methods.Figure 1 depicts the process.Source: §3 · pp. 3–4, Figure 1 · p. 4

3.1.1 Inclusion and Exclusion

• Dyadic studies had to examine a bidirectional, conversation-based CA interaction with one aware user and a user study; task performance alone was insufficient.Polyadic studies shared these requirements but needed more than one user and human-human interaction.Source: §3.1.1 · p. 4

3.1.2 Source

• ACM DL was selected after researchers compared qualification rates across five databases, with ACM DL yielding 23.5%.The other rates were IEEE 12.5%, Web of Science 14.6%, Scopus 17%, and ScienceDirect 4.9%.Source: §3.1.2 · p. 4

3.2 Search

• The search combined 15 conversational-agent terms with 13 terms for identifying polyadic work.The terms were developed through iterations and refinement informed by prior reviews.Source: §3.2 · p. 5

3.2.1 Data Preparation

• The authors searched ACM DL on March 3, 2021, found 1,302 papers after deduplication, and crawled metadata and PDFs by DOI.They assembled a metadata CSV and PDF/text corpus, converting PDFs to text with Fitz in PyMuPDF.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.3 Selection

• Full-text screening yielded 470 unique papers for eligibility review and a final set of 36 polyadic and 135 dyadic papers.Potential UX studies were identified through user-study and UX keywords, while polyadic candidates were also screened using 13 terms.Source: §3.3 · p. 5

3.4.1 Thematic Analysis

• The analysis combined thematic coding and topic modeling to examine research practices, interaction challenges, effects, and overlooked issues.Two co-authors independently open-coded papers, reconciled codes, and then coded the remainder separately; final inter-rater reliability was 91%.Source: §3.4 · p. 5, §3.4.1 · pp. 5–6

3.4.2 Topic Modeling

• Separate NMF topic models based on TF-IDF explored discussion topics in dyadic and polyadic papers.The text was preprocessed, tokenized, and lemmatized; the lemmatized results were judged more interpretable and coherent in sampled papers.Source: §3.4.2 · p. 6

4.1 Addressing Fundamental Challenges of Human-Human Interaction (RQ1)

• Most addressed challenges arose in collaborative learning, work, or group discussion, and only some papers explicitly framed them as design pain points.The review groups these challenges around communication, engagement, relational maintenance, and building connections.Source: §4.1 · p. 6

4.1.1 Inefficient Communication

• Polyadic CAs support workflow and discussion to address unstructured communication, difficult consensus-building, and burdensome coordination in group work.Challenges also include organizing ideas, managing task flow, scheduling, and switching tools.Source: §4.1.1 · p. 6

4.1.2 Lack of Engagement

• Multi-user settings face inactive engagement, including difficulty eliciting productive discussion, balancing participation, and sustaining community activity.These challenges occur in peer learning, collaborative work, online communities, and casual social settings.Source: §4.1.2 · p. 6

4.1.3 Barriers in Relational Maintenance

• Collaborative groups can struggle with emotional awareness and mutual understanding, while CA research explores trust-building and privacy boundaries.The review describes these as barriers to maintaining positive relationships.Source: §4.1.3 · p. 7

4.1.4 Need for Building Connections

• People may need help finding common ground, forming impressions, and generating ideas, particularly during ice-breaking or cross-cultural conversations.Cultural stereotypes can influence first impressions when people from different backgrounds meet.Source: §4.1.4 · p. 7

4.2.1 Research Interest Over Time

• Selected CA UX papers date to the early 2000s, then surged from 2018; both dyadic and polyadic papers peaked in 2020.The 2020 peaks were 48 dyadic papers and nine polyadic papers.Source: §4.2.1 · p. 7

4.2.2 Authors’ Affiliations

• Academic authors made up 83.4% of authors on polyadic papers, compared with 63.6% on dyadic papers.The polyadic set had 145 authors across 36 papers; the dyadic set had 596 authors across 135 papers.Source: §4.2.2 · p. 7

4.2.3 Impactful Works in the Area

• Polyadic UX studies commonly examined group environments such as virtual meetings, games, online communities, and collaboration.The dataset also covered productivity and education in both dyadic and polyadic work.Source: §4.2.3 · p. 7

4.3.1 Shared Aspects between Polyadic and Dyadic CAs

• Polyadic CAs spanned education, online communities, group discussion, work, virtual meetings, guiding services, games, family, and undefined domains.Work and productivity accounted for seven papers; education and collaborative learning for eight.Source: §4.3.1 · p. 8
• Text-only interaction appeared in 22 polyadic papers and video in nine; some studies used multiple modalities, while 22 papers studied non-embodied CAs.The remaining studies examined embodied CAs.Source: §4.3.1 · p. 8
• Most studies evaluated original CA designs, while design methods ranged from framework-based and researcher-proposed features to participatory interviews and ideation workshops.Five specified framework-based methods; two used participatory need-finding interviews and two ran ideation workshops.Source: §4.3.1 · pp. 8–9
• Experimental evaluations were most common (16 papers), followed by surveys and interviews (six each); categories could overlap because studies used multiple methods.Other methods included field studies, Wizard of Oz, interaction-log analysis, observation, and focus groups.Source: §4.3.1 · p. 9

4.3.2 Unique Aspects with the Polyadic CAs

• Polyadic CA studies covered relationships from co-learners and co-workers to community members, co-players, and family; relationship type is a distinct design dimension.Contexts also included collaborators, first-time meeting partners, public visitors, guests, and SMS contacts.Source: §4.3.2 · p. 9
• Eighteen papers addressed multi-user settings, ranging from small groups such as three to five co-learners to larger online communities.The reviewed designs also included medium-sized groups, such as ten discussants.Source: §4.3.2 · p. 9
• Theories and frameworks informing polyadic CA designs were scattered, spanning social roles, group processes, trust, and human-agent interaction.Examples included CASA, balance theory, structural role theory, Mutual Theory of Mind, and Input-Process-Output models.Source: §4.3.2 · p. 9

4.4 Proven Effects of Polyadic CAs on Human-Human Interaction (RQ4)

• Most studies focused on dyadic human-CA effects, while evaluation of group effects was relatively insufficient; reported findings nevertheless suggest opportunities to improve human social experience.Dyadic effects included learning effectiveness, self-efficacy, satisfaction, attitudes, acceptance, and human-CA engagement patterns.Source: §4.4 · p. 9

4.4.1 Communication Efficiency

• Polyadic CAs supported collaboration through consensus-building, clearer communication, task coordination, and offloading tedious work.Examples included moderators aligning group and individual opinions, chat tagging and summarization, scheduling, and search assistance.Source: §4.4.1 · p. 10

4.4.2 Group Engagement

• CAs prompted productive discussion and contributions in learning groups and online communities, and some designs encouraged more even participation.Facilitators invited less-engaged learners and discouraged talkative members from controlling discussions; moderation could also support diverse, deliberative opinions.Source: §4.4.2 · p. 10

4.4.3 Relationship Maintenance

• CAs were designed to regulate group emotions and positivity; an agent expressing vulnerability was associated with teammates’ trust-related behaviors and a more trusting atmosphere.Other designs monitored sentiment, suggested compromises, or discouraged negative words.Source: §4.4.3 · p. 10

4.4.4 Building Connections

• A CA’s agreement, disagreement, or bias toward either person can shift partners’ perceptions; agreement or disagreement with both was linked to greater mutual similarity and attraction, while tutor bias polarized a pair.A helper agent for cross-cultural conversations was evaluated positively for building common ground, although social perceptions of self, partners, and cultural stereotypes had mixed findings.Source: §4.4.4 · pp. 10–11

4.5.1 Log Analysis or Observations

• Log and observation studies assessed dyadic language, prosody, dialogue, task, algorithm, and user-related features, while polyadic studies examined task performance and discussion quality.Dyadic work measured linguistic patterns, speech features, dialogue quality and efficiency, task fulfillment, and user perceptions or social behaviors. Polyadic measures included task duration, efficiency, effectiveness, team performance, input frequency, consensus, opinion diversity, and linguistic features.Source: §4.5.1 · p. 11

4.5.2 Survey Scale

• Survey studies measured dyadic conversation quality, social perceptions, usability, and self-reported experience, while polyadic surveys emphasized communication quality and interpersonal dynamics.Dyadic measures included informativeness, perceived social features, usability, satisfaction, trust, and engagement; polyadic surveys also assessed fairness, group climate, social presence, collaboration, and perceptions of group members.Source: §4.5.2 · p. 13

4.5.3 Interview

• Dyadic interviews covered general impressions, agent characteristics, trust, and user behavior, whereas polyadic interviews focused on issue-specific capability, engagement, persuasion, and reflections on self and others.Polyadic interviews also addressed overall experience, improvement directions, and social interaction features related to team contributions and engagement.Source: §4.5.3 · pp. 13–14

4.6 Overlooked Issues of Polyadic CAs (RQ6)

• Overlooked issues in polyadic CAs concern appropriateness, privacy, and ethics across different group settings.These issues prompt questions about the agents’ social influence and relationships with users.Source: §4.6 · p. 15

4.6.1 Polyadic CAs Need to be "Visible"

• Polyadic CAs should make users aware of their presence, though designers must decide whether they should appear as peers or not.The searchbot study suggested that the agent announce itself and remain visible throughout the interaction.Source: §4.6.1 · p. 15

4.6.2 Polyadic CAs Need to be "Ignorable"

• CAs can be ignored or abused by learners, and frequent, poorly timed reminders can feel invasive or distracting [Tegos et al., 2015, Toxtli et al., 2018].Only one paper suggested making a supporting agent easy to ignore; persistent intervention may backfire.Source: §4.6.2 · p. 15

4.6.3 Polyadic CAs Need to be Accountable

• Voice-activated CAs in interpersonal spaces raise questions about handling conflicts without invading privacy and about whom the CA should be accountable to [Luria et al., 2020a, Seering et al., 2020, Zheng et al., 2021].The reviewed papers ask what should happen when an agent crosses a perceived boundary and how it should respond.Source: §4.6.3 · pp. 15–16

4.6.4 Topics Discussed in Polyadic and Dyadic Works

• Polyadic papers’ Topic 1 concerned group discussion (18.82% topic weight), while dyadic papers’ Topic 1 concerned user-agent interaction (44.16% topic weight).Figure 2 compares topics ordered by weight; its x-axis is the probability distribution of each term in its topic.Source: §4.6.4 · p. 16, Figure 2 · p. 16

5.1 Spotlight an Under-Explored Design Space

• Only a very small fraction of ACM-published CAs were designed for polyadic interaction, and little attention addressed their conversational effects on human-human interaction.Some application areas, including public services and health, lack UX research on polyadic CAs, and evaluations lack empirical findings at large human-human interaction scales.Source: §5.1 · p. 17

5.2 Connect with and Contribute to Theories

• Many reviewed works neither used nor contributed to theory, and existing frameworks may not account for the new social dynamics of polyadic CAs.The review notes that the field has not identified the best theories to explain, build on, and support these studies.Source: §5.2 · p. 17

5.3.1 Polyadic CAs are Unique to Design

• Polyadic CAs raise distinct boundary dilemmas when user needs conflict with human-AI interaction, including when to intervene in disagreements or disclose information about others [Tegos et al., 2015, Zheng et al., 2021, Luria et al., 2020b].The review notes that current studies have yet to resolve situations in which CAs may need to react or take sides.Source: §5.3.1 · p. 18

5.3.2 Design CAs with Boundary-Awareness

• The review proposes boundary-aware CAs because polyadic interaction requires managing boundaries between agents and users as well as among users.Privacy management includes ownership, control, and turbulence; clarifying data ownership and learning relationship-specific boundaries over time are described as essential steps.Source: §5.3.2 · p. 18

5.3.3 Adopt and Evaluate the Existing Guidelines

• Existing AI guidelines offer design principles such as efficient dismissal, matching social norms, and mitigating social biases, but their effectiveness should be evaluated in the wild [Amershi et al., 2019, Microsoft, 2018].The review also notes that a CA helped a team reach logical consensus on a contentious topic, while its authors highlighted protecting marginalized participants’ contributions.Source: §5.3.3 · p. 19

5.4.1 Relational Dynamics

• Future research could examine whether polyadic CAs foster new relational dynamics, including effects of social cues, roles, and identities.The review describes relational dynamics as how people interact with technologies, themselves, and others or groups.Source: §5.4.1 · p. 19

5.4.2 Functional Dimensions

• Polyadic CA design should specify the communication challenge, the agent’s unique function, and its effectiveness relative to other methods.Some reviewed works proposed CAs without identifying a fundamental human-human challenge or comparing their effects with alternatives such as no CA, a human, or previously tested CAs.Source: §5.4.2 · pp. 19–20

6 Limitations and Future Work

• Using ACM Digital Library alone could create selection bias, so the authors present findings as samples illustrating emerging themes rather than as comprehensive coverage.They suggest sampling additional HCI research sources, including IEEE, Web of Science, Scopus, and Science Direct.Source: §6 · p. 20
• Selecting only publications that both designed and UX-evaluated CAs may omit works missed by the search terms and novel techniques without UX evaluations, so the findings are not exhaustive.The authors expect the themes of polyadic challenges and evaluation metrics to expand as more works are identified.Source: §6 · p. 20
• Thematic analysis is subjective, and lemmatization and topic modeling may introduce bias; accordingly, the authors do not claim statistically significant differences.The authors note that lemmatization can lose some word meaning and unfiltered text can add noise that interferes with topic interpretability.Source: §6 · p. 20

7 Conclusion

• The review examines human-human interaction challenges addressed by polyadic CA works, evaluated effects, and overlooked design issues.Source: §7 · p. 20
• Researchers and practitioners should design CAs with boundary awareness to support polyadic human-AI interaction.Source: §7 · p. 20
• The review identifies research opportunities in theoretical grounding, relational dynamics, and functional dimensions of conversational human-AI interaction.Source: §7 · p. 20

A.1 Definitions of Different Metrics Used in Polyadic Papers

• Other interaction measures assess discussion quality, social presence and support, annoyance with intervention, and opinions about self, partners, and cultural stereotypes.Source: §A.1 · p. 21
• Measures also cover consensus, privacy concerns, engagement decisions, engagement level, sociability, and the perceived ability to elicit contributions.Source: §A.1 · p. 21
• Behavioral measures include opinion expression, reflections on self and others, willingness to act, proposed improvements, opinion diversity, and even participation.Source: §A.1 · p. 21
• Linguistic-feature measures are extracted from utterances and include verbosity, readability, sentiment, linguistic diversity, and adaptability.Source: §A.1 · p. 21
• Task-success measures cover individual and team performance, while task-related time and efficiency capture completion duration, system turns, and reaction time.Source: §A.1 · p. 21
• Satisfaction measures assess CA understandability, relevance, and efficiency; other measures capture agent perceptions, unmet expectations, conversation control, and overall UX.Source: §A.1 · p. 21
• Communication measures cover goal support, perceived participation fairness, and consensus speed, alongside group climate, perceptions of members, and collaboration experience.Source: §A.1 · p. 21
• Agent-related measures include perceived social appropriateness, human-like behavior, psychological wellbeing, anthropomorphism, and perceived emotional competence.Source: §A.1 · p. 21

A.2 Definitions of Different Metrics Used in Dyadic Papers

• Linguistic measures captured emotional word use, utterance length, pronouns, repetition, and word concreteness.The listed measures also included variation in language and, for speech-based agents, word count and variation in pitch, rate, and loudness.Source: §A.2 · p. 22
• Conversation measures assessed response quality, dialogue efficiency, expressiveness, readability, information quality, smoothness, sentiment intensity, and instruction quality.Efficiency included the number of system turns and average reaction time; smoothness was measured with the Session Evaluation Questionnaire.Source: §A.2 · pp. 22–23
• Task and system measures included task success, system use, user input time, study dropout, intent identification accuracy, and interaction errors.System use combined multiple system-related measures, such as message types, words, and time to complete a task.Source: §A.2 · p. 22
• Other user measures covered trust-related response quality, perceived intimacy, self-reflection, impolite behavior, motivation, and time spent in affective states.The table also defines error rate as errors during user interaction with the CA.Source: §A.2 · pp. 22–23
• CA perception measures included common ground, ease of interaction, sociability, anthropomorphism, warmth, competence, intimacy, naturalness, and safeness.The section also lists usability, ease of use, mental and physical demand, frustration, desire to continue, acceptance, consistency, usefulness, and willingness to recommend.Source: §A.2 · p. 23
• UX measures spanned overall experience, contrast between expectations and experience, interaction quality, satisfaction, engagement, trust, confidence, control, and emotional change.The listed constructs also included serendipity, pleasant surprise, empathy, self-disclosure, anxiety, comfort, collaboration, and decision-making.Source: §A.2 · p. 24
• Additional perception measures addressed CA capabilities, appearance, personality, naturalness, reciprocity, perceived burden, human-likeness, and effectiveness.Personality was described using openness, conscientiousness, extraversion, agreeableness, and neuroticism.Source: §A.2 · p. 24
• Behavioral measures tracked engagement, users’ efforts to learn the system, awareness of CA functions, self-reflection, and effects on judgment or collaboration.Engagement could be rated on a 1-to-5 survey scale.Source: §A.2 · p. 24

• Participants’ daily chatbot practices were assessed by asking them to briefly describe how they had used the chatbot during the previous three weeks.Source: §A.2 · p. 24