Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution

Study setting

• The study analyzes a production CTA used by the U.S. Alexa user base, including task discovery, step-by-step interactions, engagement, and optional satisfaction ratings.The system supports voice, text, images, and video.Source: §1 · pp. 1–2

Research aims

• The work examines how users initiate and progress through CTA interaction stages, what intents they express, and their conversational traits.It also compares real-world interactions with crowdsourced CTA datasets.Source: §1 · p. 2
• The analysis identifies behavioral signals associated with user satisfaction in CTA systems.Source: §1 · p. 2

Implications

• The findings point to opportunities in handling uncooperative behavior, improving task discovery, and supporting sustained task engagement.The authors conclude with concrete design insights for multimodal CTAs.Source: §1 · p. 2

Conversational Task Assistants.

• Wizard of Tasks and Task2Dial provide task-centric dialogues for studying intent recognition and commonsense reasoning, but do not fully capture real-world usage variability.Source: Conversational Task Assistants · p. 2
• The Alexa TaskBot Challenge targets robust CTA operation in real user conditions, while systems including GRILLBot, TWIZ, and ISABEL explore multimodal task assistance.Fine-grained analysis of these real-world interactions had not been performed.Source: Conversational Task Assistants · p. 2

User Behavior with CTAs.

• Earlier user studies highlight natural, proactive strategies, multimodal support for autonomy and task understanding, and challenges in timely, context-aware responses.Source: User Behavior with CTAs · p. 2
• Visual aids can support task comprehension and reduce cognitive load, while voice interfaces and noisy settings raise working-memory and ASR challenges.Source: User Behavior with CTAs · p. 2
• The paper analyzes large-scale real-world interactions to characterize CTA use and behavior under everyday deployment conditions.Source: User Behavior with CTAs · p. 3

3 Designing CTAs for Effective Task Guidance

• The system is designed around modular architecture, multimodal integration, and context-aware responses.Its components manage dialogue flow and task content, with a serverless function coordinating modules, a database handling conversation history, and a search system retrieving tasks.Source: §3 · p. 3, §3.1 · p. 3

Dialogue Management.

• An event-driven, stage-based dialogue manager tracks dialogue and task progress through Task Grounding and Task Execution macro-stages.Task Grounding includes greeting, search, and overview; Task Execution includes task steps and completion. Q&A and multimodal sub-flows let users temporarily diverge while maintaining continuity.Source: Dialogue Management · pp. 3–4

Intent Detection.

• Intent detection combines a BERT-based classifier, phoneme-level similarity, and rule-based heuristics through a voting-style mechanism.The system uses detected intent to determine its behavior, addressing the importance of robustness to ASR errors.Source: Intent Detection · p. 4

Task Suggestions and Search.

• Proactive voice and on-screen suggestions support task discovery alongside free-form search, with retrieval combining text and semantic search.Task content comes from WikiHow for DIY and Whole Foods Market for recipes.Source: Task Suggestions and Search · p. 4

Adaptive Task Chunking.

• Task instructions are divided into shorter steps using length and formatting cues, omitting nonessential information to reduce cognitive load.This is particularly important for voice-only interactions, where users rely on working memory.Source: Adaptive Task Chunking · p. 5

UI and Navigation.

• Users interact by voice alone or by voice and touch, with contextual images and videos synchronized to dialogue stages.The interface emphasizes clarity and ease of use.Source: UI and Navigation · p. 5

Interaction with Multimodal Content.

• Users can ask task-related questions using an LLM prompted with dialogue context, request fun facts, or control video playback through query-to-keyframe similarity.Video control compares queries with cross-modal textual and visual embeddings of keyframes.Source: Interaction with Multimodal Content · p. 5

3.3 Contextual Dialogue and Response Design

• Responses balance clarity and engagement through concise, varied wording, with commands placed at the end to aid recall.Templates and an LLM generate contextual responses; safety heuristics and a RoBERTa-based classifier trigger predefined replies, while stage-aware recovery suggests alternatives.Source: §3.3 · p. 5

Data Collection and Filtering

• The dataset contains 30,000 conversations, with more than 5,000 rated (16.7%), after filtering for at least three turns and removing sessions with over 10% unrelated turns.The authors describe it as, to their knowledge, the largest real-world CTA dataset.Source: Data Collection and Filtering · p. 6

Device and Domain.

• Conversations came from 92% unique users; 91% involved screen-equipped devices and 9% voice-only devices.Among users selecting a task, 54% chose recipes and 46% DIY; task selection occurred among 56% of users.Source: Device and Domain · p. 6

4.1 User Progression Through Dialogue Stages

• Fewer dialogues reach successive stages, with a marked drop from Search/Task Overview to Task Steps, suggesting many users explore or evaluate tasks rather than pursue completion.A smaller proportion progress through every stage, pointing to a challenge in sustaining engagement.Source: §4.1 · p. 6
• Task Steps account for 29.7% of all turns, while Task Overview and Task Completed each consist of a single acknowledgment turn.Users spend 68.5% of interaction in Task Grounding, underscoring the importance of task discovery and guidance before execution.Source: §4.1 · p. 7

4.2 Analyzing User Intents

• 64.7% of turns use touch controls, while voice intents center on task navigation; Fallback appears in 8.6% of turns, alongside out-of-scope and sensitive requests.Figure 5 groups intents into Main Flow, Complex, Non-Cooperative, and Assorted categories.Source: §4.2 · p. 7

4.3 Analysis of Conversational User Traits

• Dialogue-level engagement averages 6.8 turns, cooperativeness averages 80%, task starters complete about half the steps, and users generally accept only one or two fallbacks.A small cooperativeness peak near zero reflects fully uncooperative users; task length also influences completion.Source: §4.3 · p. 8, Figure 6 · p. 8
• Utterances average 2.8 words, emotion is mostly neutral (0.54), fluency is generally high (0.82) but drops for longer utterances, and repetition is low.Longer utterances may involve ASR errors or hesitation.Source: §4.3 · p. 8, Figure 6 · p. 8

Real vs. Crowdsourced Behavior.

• Complex actions make up 8.8% of real interactions versus 47.5% in crowdsourced data, while real utterances are shorter and noisier and uncooperative behavior appears in approximately 17% of interactions.Uncooperative behavior is mostly absent from crowdsourced datasets.Source: Real vs. Crowdsourced Behavior · pp. 8–9

4.4 User Rating Analysis in CTAs

• The 5k rated conversations average 3.44, with ratings clustering around 1 and 5 and most rated interactions positive (≥3).The distribution suggests users often provide feedback after strong positive or negative experiences.Source: §4.4 · p. 9
• Users who start tasks rate the system 3.84 versus 3.05 for those who do not, and task completers rate it 4.17; recipes score 4.01 versus 3.72 for DIY.Ratings do not differ substantially between non-screen devices (3.49) and screen-based devices (3.42).Source: §4.4 · p. 9

Behavioral Signals Associated with User Satisfaction.

• No single feature is strongly correlated with ratings; the strongest positive correlations are Task Progression (0.26), Start and Completion (0.24), and reaching Task Execution (0.23).Greeting (-0.16), Search (-0.08), User Verbosity (- 0.14), and longer answer times (-0.06) have negative correlations; all reported feature correlations are statistically significant at p < 0.01.Source: Behavioral Signals Associated with… · pp. 9–10, Figure 8 · p. 10

Subjectivity of Ratings.

• Annotators agree with one another more than with users: inter-annotator agreement is ρ = 0.70, 80%, κ = 0.59, versus ρ = 0.24, 61%, κ = 0.22 with user ratings.The authors attribute the gap to annotators applying stricter criteria, with average scores of 2.7 versus 3.4.Source: Subjectivity of Ratings · p. 10

5 Challenges, Opportunities and Design Implications

• The analysis identifies multiple opportunities and research challenges associated with lower-performing conversations and puts forward CTA design implications.Source: §5 · p. 10

Ambiguous ASR and Contexts.

• ASR errors frequently disrupt interactions, while users also attempt standard voice-assistant features unavailable during CTA use.The authors point to multitasking between core assistant features and third-party applications as a need.Source: Ambiguous ASR and Contexts · p. 10

Agent Knowledge Boundaries.

• Task search is the most common problem when relevant content is not found; queries requiring external knowledge are often associated with incorrect responses.The passage identifies RAG methods as a potential improvement.Source: Agent Knowledge Boundaries · p. 10

User Behavior.

• Most users follow a linear task flow with limited use of complex features, while uncooperative behavior highlights challenges in feature discoverability and proactive support.Source: User Behavior · p. 11

CTA Design Guidelines.

• The authors synthesize real-world interaction findings into design guidelines for more behavior-driven and robust CTAs.Figure 9 summarizes the guidelines.Source: CTA Design Guidelines · p. 11, Figure 9 · p. 11

6 Conclusions

• The study finds that real-world interactions include touch-based preferences, mostly linear task progression, and frequent out-of-domain behavior, with satisfaction associated with task-execution engagement.The authors present empirically grounded, actionable guidelines for more adaptive and context-aware CTAs.Source: §6 · p. 11

• The conversations come from U.S. Alexa users interacting in English, and patterns may differ across regions, languages, cultural backgrounds, and platforms.Source: Limitations · p. 12
• Findings from the system’s cooking and DIY tasks may not directly generalize to all task domains.Source: Limitations · p. 12
• Associations with ratings are correlational and may reflect unmeasured factors such as user motivation or task relevance.Source: Limitations · p. 12

• Users voluntarily interacted with the system and consented to their interactions being used for research.Source: Ethical Considerations · p. 12
• Conversations were fully anonymized, with no personally identifiable information retained.Source: Ethical Considerations · p. 12
• All results were reported only in aggregate form.Source: Ethical Considerations · p. 12