PaperlayerView exact revision on arXiv

Source-linked AI summary

One Agent Too Many: User Perspectives on Approaches to Multi-agent Conversational AI

Christopher Clarke, Karthik Krishnamurthy, Walter Talamonti, Yiping Kang, Lingjia Tang, Jason Mars

arXiv:2401.07123v1cs.HCcs.CL

TL;DR

Commercial conversational agents are often specialized, so complex tasks may require users to navigate multiple agents, and it was unclear which interaction mode best supports users. The paper compares a single-agent interface that orchestrates agents in the background with an interface where users select agents; users preferred the single-agent interface, and One For All’s response selection matched human ground truth 70% of the time across task domains.

  • Problem

    Prior work had not explicitly investigated how single-agent and user-choice interaction modes affect decision-making and cognitive load, or which mode best supports interaction and performance.

    Source: §I · p. 1

  • Method

    The authors implemented One For All and Agent Select and evaluated them in user studies, while comparing response-selection engines and agents against human ground truth.

    Source: §I · p. 2, A. One For All vs. Human judgment · p. 8

  • Results

    Users significantly preferred the single-agent interface for usability and performance; One For All using USE selected the same agent as human ground truth 70% of the time across task domains.

    Source: §I · p. 2, A. One For All vs. Human judgment · p. 8

  • Takeaways & Limitations

    The authors release the prototypes and two datasets containing user-study queries and responses, satisfaction labels, and crowdsourced human judgments.

    Source: §I · p. 2, §IX · p. 10

Abstract

from arXiv · show

Conversational agents have been gaining increasing popularity in recent years. Influenced by the widespread adoption of task-oriented agents such as Apple Siri and Amazon Alexa, these agents are being deployed into various applications to enhance user experience. Although these agents promote "ask me anything" functionality, they are typically built to focus on a single or finite set of expertise. Given that complex tasks often require more than one expertise, this results in the users needing to learn and adopt multiple agents. One approach to alleviate this is to abstract the orchestration of agents in the background. However, this removes the option of choice and flexibility, potentially harming the ability to complete tasks. In this paper, we explore these different interaction experiences (one agent for all) vs (user choice of agents) for conversational AI. We design prototypes for each, systematically evaluating their ability to facilitate task completion. Through a series of conducted user studies, we show that users have a significant preference for abstracting agent orchestration in both system usability and system performance. Additionally, we demonstrate that this mode of interaction is able to provide quality responses that are rated within 1% of human-selected answers.

I. INTRODUCTION

Conversational agents serve specialized domains, so complex tasks may require users to navigate multiple agents. This paper compares single-agent and user-choice interfaces, while related work describes unresolved challenges in orchestration and user experience.

I. INTRODUCTION

  • Because agents specialize in particular domains, users may need to learn and adopt multiple agents for complex tasks.Prior work reports concerns about cognitive burden and decision overload, while other studies suggest multi-agent choice can enhance user experience.Source: §I · p. 1
  • Prior work had not explicitly compared how these interaction modes affect decision-making and cognitive load, leaving their relative suitability and performance unclear.The paper also identifies routing users to the best agent, adapting to changing agents, and covering the ensemble’s domains as design challenges.Source: §I · pp. 1–2
  • The study compares two prototypes with 19 participants completing tasks across several domains: One For All and Agent Select.One For All abstracts agent orchestration; Agent Select lets users choose an agent.Source: §I · p. 2
  • Users significantly preferred the single-agent interface over the multi-agent interface in system usability and system performance.The paper examines the interaction experience of single-agent use against user choice of agents.Source: §I · p. 2
  • The authors release both prototypes and two datasets, including participant queries, agent responses, satisfaction labels, and crowdsourced human judgments.They also present design and system challenges and insights for optimizing the interaction experience.Source: §I · p. 2

II. RELATED WORK

  • Related work covers conversational multi-agent systems and conversational AI in HCI, including prior findings of confusion in multi-chatbot interactions and limits of toolkit-based extensions.It also describes agent orchestration approaches and varied HCI evaluation methods.Source: §II · p. 2

III. SYSTEM DESIGN

The system design presents two prototypes for combining commercial conversational agents. One For All sends queries to an ensemble and returns the response ranked highest for semantic relation to the query.

III. SYSTEM DESIGN

  • The system-design section outlines the prototypes’ architecture, software components, and query-processing workflow.One For All and Agent Select represent the two interaction experiences under study.Source: §III · p. 3

A. Conversational Agents

  • The integrated suite comprises Alexa, Google Assistant, Houndify, and Ford Adasa, selected for domain range, popularity, and open programming interfaces.The paper gives further information on each agent in an appendix.Source: A. Conversational Agents · p. 3

B. One For All

  • One For All sends each agent the query, embeds the query and responses, and returns the response with the strongest semantic relation to the query.The architecture uses parallel software components to provide a unified experience.Source: B. One For All · p. 3

1) Speech to Text Engine:

One For All converts spoken questions to text, gathers responses from agents, and ranks those responses by their semantic relation to the query. The ranking approach does not require training or access to agent internals.

1) Speech to Text Engine:

  • Google Speech API transcribes spoken questions for the next component; the authors observed 0% transcription error during their experimentation.Each conversational agent receives the query as text, and its response is passed to the ranking engine.Source: 1) Speech to Text Engine · p. 3

2) Response Ranking Engine:

  • The ranking engine evaluates BERT, Universal Sentence Encoder, and Smooth Inverse Frequency embeddings to represent queries and candidate responses.BERT uses the second-to-last encoding layer; the system uses the Universal Sentence Encoder transformer model and pretrained 50-dimensional GloVe embeddings for SIF.Source: 2) Response Ranking Engine · pp. 3–4
  • For each response, the algorithm computes Euclidean distance between query and response embeddings, then ranks responses in ascending distance.The approach treats semantic relation as an indicator of how well an agent understood the query.Source: 2) Response Ranking Engine · p. 4
  • The response-selection method requires no training and does not rely on agent internals, so the ensemble can add or remove agents.The authors frame this as a way to adapt to agent changes over time.Source: 2) Response Ranking Engine · p. 4

3) Text-to-Speech Engine:

The prototypes use voice input and output but differ in how queries are routed. One For All routes queries to agents in parallel and ranks their responses, while Agent Select lets users choose an agent beforehand.

3) Text-to-Speech Engine:

  • After selecting the highest-ranked response, One For All converts it to audio using Google Cloud Text-to-Speech.Voice attributes such as gender, pitch, volume, and speech rate are configurable.Source: 3) Text-to-Speech Engine · p. 4

C. One For All: Life of query

  • In One For All, a spoken query is transcribed, sent to each agent in parallel, and ranked before the selected response is played as audio.The ranking uses embeddings of the query and responses to calculate semantic relation.Source: C. One For All: Life of query · p. 4

D. Agent Select: Life of query

  • In Agent Select, users choose an agent before speaking; the system transcribes the query and returns that agent’s response without ranking.The prototype reuses One For All components apart from the ranking engine.Source: D. Agent Select: Life of query · p. 4, Figure 2 · p. 4

IV. USER STUDY

  • The experiment compared both prototypes on a computer system; 19 pilot-study participants were recruited, with varied routine assistant use and familiarity across the integrated agents.Participants interacted with each interface to complete tasks.Source: §IV · pp. 4–5, A. Experiment Prototype · p. 5

B. User Study Protocol

Participants tried both interaction prototypes across ten domains, completing tasks and reporting whether the systems met expectations. The protocol also gathered usability and experience feedback.

B. User Study Protocol

  • Participants received briefings and practice time, then completed each prototype study and a post-phase experience questionnaire and SUS test.Researchers also collected demographic and prior-assistant-use information in a prescreening questionnaire.Source: B. User Study Protocol · p. 5

C. Tasks

  • Each participant used both prototypes for at least 10 tasks across 10 domains and reported whether each task was completed as expected.At least one task per domain was required in both phases.Source: C. Tasks · p. 5

1) Task list:

  • A 50-query example list across 10 domains illustrated agent capabilities, but participants could choose different tasks and interact freely.The list included automobile, weather, directions, flight information, time, stocks, dates, travel, restaurants, and general knowledge.Source: 1) Task list · p. 5

2) Phase 1:

  • In Phase 1, participants used One For All at least once in each of the 10 domains.They reported whether the system answered their query correctly.Source: 2) Phase 1 · p. 5, 3) Phase 2 · p. 5

3) Phase 2:

  • In Phase 2, participants selected an agent before querying; researchers recorded the interaction and response correctness, while questionnaire feedback differences were significant (p < .01).Recorded fields included the selected agent, utterance, response, and processing latency; some tasks involved multiple turns.Source: 3) Phase 2 · pp. 5–6, Figure 3 · p. 6

D. System Usability Scale (SUS)

The study used SUS to measure usability and a separate five-question survey to assess prototype quality and participant satisfaction. Table II reports average SUS and overall accuracy/correctness by prototype.

D. System Usability Scale (SUS)

  • SUS is a 10-item questionnaire with five response options, used to calculate each participant’s individual usability score.Responses range from Strongly agree to Strongly disagree.Source: D. System Usability Scale (SUS) · p. 6

E. Questionnaire

  • The post-phase questionnaire used five questions on system quality, information quality, and user satisfaction, with 10-point Likert responses.Questions were adjusted for One For All or Agent Select and assessed selection preference, response accuracy and usefulness, and satisfaction.Source: E. Questionnaire · p. 6
  • Table II reports average system usability scores and overall accuracy/correctness for each prototype.The supplied table caption identifies the measures but gives no values.Source: Table II · p. 6

V. RESULTS: USER EXPERIENCE

One For All outperformed Agent Select on overall usability and task accuracy, and participants rated its system quality, information quality, and satisfaction more favorably. A separate evaluation assessed One For All’s agent selection and response quality against crowdsourced human judgments.

V. RESULTS: USER EXPERIENCE

  • One For All scored 86 (SD=8.9) on SUS versus Agent Select’s 56 (SD=21.7), and achieved 71% accuracy across participant tasks versus 57%.All reported results in the section were statistically significant (Wilcoxon Signed Rank test, p < .01).Source: §V · p. 6

A. System Quality

  • Participants rated One For All’s system quality higher than Agent Select’s, with mean scores of 7.68 versus 4.5.Feedback favored handing off agent selection; participants said choosing an assistant was difficult because they did not know its capabilities.Source: A. System Quality · p. 6

B. Information Quality

  • One For All averaged µ=7.74 on statement 2 and µ=7.95 on statement 3, versus Agent Select’s µ=5.89 and µ=5.95, respectively.The statements assessed query understanding and whether responses were feasible and useful.Source: B. Information Quality · p. 6

C. User Satisfaction

  • One For All received µ=9.16 for overall satisfaction and µ=8.32 for meeting expectations.These measures corresponded to questionnaire statements 4 and 5.Source: C. User Satisfaction · p. 7

A. Establishing Human Performance

  • The performance evaluation compared One For All with human judgments, measuring agent-selection accuracy and the quality of the selected response.Accuracy was the share of correct-agent selections identified by human judges; response quality was rated by crowdsourced workers.Source: A. Establishing Human Performance · p. 7

1) Dataset Collection:

  • The crowdsourced evaluation collected vetted utterances across nine broad task domains, including subdomains such as automobile fuel economy and adaptive cruise control.Workers supplied utterances through Mechanical Turk using scenario- and paraphrasing-based prompts.Source: 1) Dataset Collection · p. 7

2) Response Selection Accuracy:

Workers selected the most appropriate candidate response to establish human judgments, and separate ratings captured response quality beyond a single best-choice label. The supplied passages describe the evaluation setup and table scope but do not report system-performance outcomes.

2) Response Selection Accuracy:

  • Five workers judged each task’s candidate responses, with majority vote defining the ground truth; options were randomized and agent identities hidden.Workers could select only one option, matching One For All’s decision process.Source: 2) Response Selection Accuracy · p. 7
  • The 320 evaluation tasks comprised 160 automobile tasks and 160 tasks evenly distributed across eight other domains, reflecting a focus on driving scenarios.The design tested whether other agents affected One For All on tasks where a domain-specialized agent might be best suited.Source: 2) Response Selection Accuracy · p. 7

3) Response Quality:

  • Because a single best-choice judgment excludes other feasible responses, workers separately rated each response on a five-point quality scale.Three workers rated each response from Very Poor to Very Good, considering task completion and clarity of understanding.Source: 3) Response Quality · p. 7
  • Table III compares agents with human judgment using the share of responses matching the human-selected answer; feasible responses not voted best may still differ.Thus, a non-match does not necessarily mean the response was infeasible.Source: Table III · p. 8

VII. RESULTS: SYSTEM PERFORMANCE

  • The system-performance section evaluates One For All using the collected dataset and the stated evaluation metrics.The supplied passage does not provide the resulting performance values.Source: §VII · p. 8

A. One For All vs. Human judgment

One For All’s response selection was compared with human judgments and isolated assistants, while the discussion identifies response-quality challenges that can affect selection. The reported findings cover selection agreement, crowd-rated quality, and non-desirable responses.

A. One For All vs. Human judgment

  • One For All using USE matched the human-selected agent 70% of the time overall and outperformed any single agent outside the restaurant domain.The authors report the restaurant domain as the exception.Source: A. One For All vs. Human judgment · p. 8

B. Response Quality

  • 87.19% of One For All selections were acceptable or better, compared with 87.92% for human judgment, placing it on par with humans when selecting an agent.The comparison uses each agent’s end response.Source: B. Response Quality · p. 8, Figure 5 · p. 8

VIII. DISCUSSION

  • Users significantly preferred a single-agent interface over a multi-agent interface in system usability and system performance.The discussion states this preference but gives no additional quantitative result in the supplied passage.Source: §VIII · p. 8

A. Avoiding Non-desirable Agent Responses

  • One For All selected non-desirable responses at a 3% rate, although the authors say most such cases occurred when all agents returned non-desirable responses.The authors attribute the low rate to avoiding responses that indicate an inability to handle a query.Source: A. Avoiding Non-desirable Agent… · p. 9, Table IV · p. 9
  • Unfamiliar abbreviations and non-standard terms can challenge selection, as One For All chose Houndify’s response when Adasa had answered an unfamiliar abbreviation correctly.The authors also report that query-response length can affect ranking.Source: A. Avoiding Non-desirable Agent… · p. 9
  • Long, detailed responses to short questions can influence ranking, prompting recommendations to pre-filter such responses or fine-tune the ranking model.The ranking engine considers the entirety of the question, so extra information can affect the score.Source: A. Avoiding Non-desirable Agent… · p. 9

B. Agent Flexibility and Agnosticism

The authors propose modular, agent-agnostic integration and response selection to accommodate changing agents and overlapping domains. They also report released prototypes and datasets alongside their study’s conclusions.

B. Agent Flexibility and Agnosticism

  • Treating each agent as a modular black box allows agents to be added or removed without agent-specific knowledge or model retraining.The authors say this approach supports plug-and-play changes and adaptation to agent improvements without additional computing cost.Source: B. Agent Flexibility and Agnosticism · p. 9

C. Domain Coverage & Overlap

  • For overlapping domains, response selection avoids complex agent-domain mappings; a user preference list can provide choice when needed.The paper notes that mappings can change over time.Source: C. Domain Coverage & Overlap · p. 9

IX. CONCLUSION AND FUTURE WORK

  • The conclusion reports that users preferred the single-agent interface and that multi-agent responses were rated within 1% of human-selected answers.The paper frames these findings as results of its user studies and systematic evaluation.Source: §IX · p. 10
  • The released resources include prototypes and two datasets: participant questions with returned responses and satisfaction judgments, and queries with agent responses and human judgments.The authors hope these resources will encourage further research on multi-agent usage.Source: §IX · p. 10

X. BIOGRAPHY SECTION

The supplied material combines author biographies with descriptions of the conversational agents and example automotive queries used in the paper. It identifies each agent’s role or integration approach, but does not report comparative results for these examples.

X. BIOGRAPHY SECTION

  • The biographies span conversational AI and NLP, scalable AI systems, in-vehicle human factors, and AI entrepreneurship.They describe six authors’ roles and research interests.Source: §X · p. 12

A. Amazon Alexa

  • Alexa was integrated through its audio-only Voice Service API, with speech-to-text and text-to-speech added to the client package.The implementation used the Alexa Voice Service HTTP/2 endpoints.Source: A. Amazon Alexa · p. 13

B. Google Assistant

  • The Google Assistant client used its SDK and gRPC framework, with speech-to-text added because Google did not always return response transcriptions.Source: B. Google Assistant · p. 13

C. Houndify

  • Houndify was configured with prepackaged domains including weather, local search, and Wikipedia.The implementation used the Houndify Client SDK.Source: C. Houndify · p. 13

D. Adasa

  • Adasa was extended beyond in-vehicle controls to cover navigation, maintenance scheduling, fuel level, and car-indicator information.The assistant supports voice queries about driving features and other vehicle-related tasks.Source: D. Adasa · p. 13

E. Automotive

  • Automotive test questions included adaptive cruise control, lane-drift alerts, dashboard indicators, fuel-door location, and estimated driving range.Source: E. Automotive · p. 13

F. Weather

The supplied appendix lists example questions across weather, time, directions, flights, stocks, dates, travel, restaurants, and general knowledge. These examples show the range of task domains represented, rather than evaluation outcomes.

F. Weather

  • Weather questions ask for forecasts, humidity, and expected high temperatures in particular cities.Source: F. Weather · p. 13

G. Time

  • Time queries cover local time, sunset, and current time in other locations or time zones.Source: G. Time · p. 13

H. Directions

  • Directions examples ask for nearby shopping centers, hospitals, coffee, and restaurants.Source: H. Directions · p. 13

I. Flight Info

  • Flight questions include fares and one-way availability for specified destinations and departure dates.Source: §I · p. 13

J. Stock

  • Stock questions ask about market movement, opening times, and prices for named companies.Source: J. Stock · p. 13

K. Date

  • Date questions ask for the current day or date and dates several days ahead.Source: K. Date · p. 13

L. Travel; M. Restaurant

N. General Knowledge

Loading 2401.07123v1…