PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding User Experience in Large Language Model Interactions

Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, Jian-Yun Nie

arXiv:2401.08329v1cs.HC

TL;DR

Research on LLMs has largely emphasized ability benchmarks, with less attention to user needs in real-world interactions. This paper develops and validates a user-intent taxonomy and surveys 411 people about LLM use and satisfaction. The survey analysis reports 11 findings, including three statistically relevant usage clusters, while the authors note limits to generalizing from the participant pool.

  • Problem

    LLM research has largely assessed models as independent intelligence, while user-centric evidence about how they meet needs in human-AI collaboration is limited.

    Source: §1 · p. 2

  • Method

    The authors validate a user-intent taxonomy using open-source ChatGPT logs and human review, then survey 411 people about usage, experiences, and concerns.

    Source: §1 · pp. 2–3, §3.1.1 · p. 4

  • Results

    The 11 survey findings include three statistically relevant usage clusters: subjective GUI use, objective GUI use, and API use, with subjective GUI use potentially overlooked by existing research.

    Source: §8 · p. 15

  • Takeaways & Limitations

    The findings show variation in user preferences across scenarios, underscoring the relevance of user needs when refining LLMs for real-world interaction contexts.

    Source: §5.5 · p. 11

    The voluntary, socially disseminated survey is not representative of a random general user group, limiting broader generalization of its findings.

    Source: §4.2 · p. 6, §7 · p. 15

Abstract

from arXiv · show

In the rapidly evolving landscape of large language models (LLMs), most research has primarily viewed them as independent individuals, focusing on assessing their capabilities through standardized benchmarks and enhancing their general intelligence. This perspective, however, tends to overlook the vital role of LLMs as user-centric services in human-AI collaboration. This gap in research becomes increasingly critical as LLMs become more integrated into people's everyday and professional interactions. This study addresses the important need to understand user satisfaction with LLMs by exploring four key aspects: comprehending user intents, scrutinizing user experiences, addressing major user concerns about current LLM services, and charting future research paths to bolster human-AI collaborations. Our study develops a taxonomy of 7 user intents in LLM interactions, grounded in analysis of real-world user interaction logs and human verification. Subsequently, we conduct a user survey to gauge their satisfaction with LLM services, encompassing usage frequency, experiences across intents, and predominant concerns. This survey, compiling 411 anonymous responses, uncovers 11 first-hand insights into the current state of user engagement with LLMs. Based on this empirical analysis, we pinpoint 6 future research directions prioritizing the user perspective in LLM developments. This user-centered approach is essential for crafting LLMs that are not just technologically advanced but also resonate with the intricate realities of human interactions and real-world applications.

1 INTRODUCTION

The paper argues that ability-focused LLM benchmarks may not reflect users’ varied needs in collaborative settings. It studies user intents and satisfaction, then uses its findings to identify future research directions.

Motivation

  • The study addresses limited research on LLMs as user-centric interfaces and the possible mismatch between overall ability scores and users’ scenario-specific needs.A lower-scoring model may outperform a higher-scoring one in particular situations, depending on user expectations.Source: §1 · p. 2

Contributions

  • The authors develop a taxonomy of user intents grounded in real-world interaction logs and human verification.The taxonomy is intended to support fine-grained study of user satisfaction with LLMs.Source: §1 · pp. 2–3
  • The authors analyze firsthand survey feedback from 411 participants about LLM usage frequency and satisfaction with existing interfaces, reporting 11 key findings from the empirical analysis.The dataset will be publicly available alongside the paper’s publication.Source: §1 · p. 3
  • The study identifies gaps in existing LLM research on real-world applications and proposes six directions for future human-AI collaboration research.The directions draw on the user study’s findings.Source: §1 · p. 3

Conclusion

  • The conclusion advocates developing LLMs that are technically sophisticated and beneficial in human-AI collaboration.It frames this as a user-centered approach attentive to human needs and real-world utility.Source: §1 · p. 3

2 RELATED WORK

Prior work studies intents in information-seeking systems and evaluates LLM capabilities, but general LLM interactions and user experience remain less explored. This paper situates its work across intent analysis, LLM evaluation, and human-AI collaboration.

2.1 User Intent Analysis

  • Earlier intent research spans search and question-answering, while generative-AI studies cover narrower settings; search-oriented conversational categories may not capture general LLM use.The paper builds on prior LLM work using closed-source Bing Chat logs and extends the focus to general LLM environments.Source: §2.1 · p. 3

2.2 Evaluation of Large Language Models

  • LLM evaluations include benchmarks, technical reports, and surveys, but often provide overall scores or predefined tasks without fine-grained coverage of real-world usage.Some technical reports include human preference evaluations, yet comparative detail across models is often limited.Source: §2.2 · p. 4

2.3 Empirical Studies on human-AI Collaborations

  • Human-AI collaboration research spans many domains, but user experience with general LLM interfaces—including intents and satisfaction—remains unexplored.The authors position their study as an initial step toward user-perspective research on generative AI.Source: §2.3 · p. 4

3 REAL-WORLD USER INTENTS FOR ENGAGING WITH LARGE LANGUAGE MODELS (RQ 1)

The paper develops a taxonomy for user interactions with general-purpose LLMs, validating a prior taxonomy against open-source ChatGPT logs and refining it for broader interface use. It then presents intent categories covering information seeking, creative and practical uses, text tasks, API use, and other interactions.

3 REAL-WORLD USER INTENTS FOR ENGAGING WITH LARGE LANGUAGE MODELS (RQ 1)

  • The study builds an intent taxonomy for general-purpose LLMs using related analysis, real-world logs, and user studies.The authors frame intent understanding as a basis for tailoring services to user demands.Source: §3 · p. 4

3.1 Taxonomy Development

  • The authors use a prior taxonomy developed from closed-source Bing Chat logs as the starting point for validation in general LLM interfaces.The prior logs came from a mainly search-oriented product.Source: §3.1.1 · p. 4

3.1.1 Step 1: Generation Based on Related Literature.

  • Three annotators validated the taxonomy on 50 randomly selected English ShareGPT conversations through independent annotation, discussion, joint review, and a final independent round.The final stage ended with consensus on the taxonomy.Source: §3.1.2 · p. 5

3.1.2 Step 2: Validation through Real-World Logs.

  • Validation added Seek Creativity, Ask for Advice, and API Usage, while consolidating Learning into Information Retrieval or Solve Problem depending on the information sought.The revisions extend the taxonomy beyond search-related interfaces.Source: §3.1.2 · p. 5

3.2 Classification result

  • The classification includes practical advice and information retrieval, leisure, creativity, and specialized-area problem solving.Examples include career development and personal schedules; factual information; entertainment; brainstorming; and learning in fields such as engineering and natural sciences.Source: §3.2 · p. 5
  • The taxonomy also covers text tasks such as summarizing, translating, revising, and generating text, as well as API use and other uses.API use means interacting through an application programming interface rather than a graphical user interface.Source: §3.2 · p. 6

4 DESIGN OF USER STUDY

The user study surveys usage patterns, experiences across intents, response expectations, tool use, and concerns about LLM interfaces. Its recruitment and demographic information also indicate a possible bias in the participant pool.

4.1 Questionnaires

  • The survey has 12 questions—10 required and two optional—and is designed to take 5–10 minutes.Its five parts cover usage patterns, experiences across intents, expected response types and tool use, major concerns, and comments.Source: §4.1 · p. 6
  • Questions cover services and usage frequency, prior use and satisfaction by intent, answer-type preferences, and expected tool use.The answer-type pairs include detailed or concise, factual or creative, and professional knowledge or common sense; tools include browsing, personalization, programming, and multimedia creation.Source: §4.1 · p. 6

4.2 Participants

  • The survey was distributed through social-media and university-related channels, and the authors note possible demographic bias.Personal information such as age, gender, and profession was not recorded, limiting demographic analysis to IP distribution.Source: §4.2 · p. 6

5 RESULTS ON USER ENGAGEMENT WITH LLMS (RQ 2 AND 3)

A survey of 411 respondents found frequent LLM use alongside distinct patterns in users’ intents, satisfaction, response preferences, tool expectations, and concerns. Across these results, experiences vary by scenario and language, while users identify both capability and trustworthiness as priorities.

5.1 Feedback Statistic; 5.2 Usage Frequency (Q2)

  • Around 80% of participants used LLM interfaces at least weekly; daily use was reported by half of English and 42.09% of Chinese respondents.The study collected 411 valid responses: 297 Chinese and 114 English.Source: §5.2 · p. 7, §5.1 · p. 7, Figure 3 · p. 7

5.3 Intent Analysis (Q3,4); 5.3.1 Usage Distribution.

  • Text Assistant, Information Retrieval, and problem-solving tasks were the predominant intents; Seek Creativity and Ask for Advice each resonated with about 40% of users, while Leisure use was lower.English respondents showed greater propensity for API use; the authors also note that usage patterns point to personal and recreational opportunities and cultural and linguistic differences.Source: §5.3.1 · pp. 7–8, Figure 4 · p. 8

5.3.2 Intents Analysis.

  • The seven intents clustered into three groups: Objective Usage through GUIs, Subjective Usage through GUIs, and Usage through APIs.The study clustered intents based on statistically significant associations in pairwise chi-square tests.Source: Figure 5 · p. 8, §5.3.2 · p. 9

5.4 User Satisfaction (Q5); 5.4.1 Rating Analysis.

  • Text Assistant had the highest satisfaction across English and Chinese users, with over 80% satisfied or very satisfied; Seeking Creativity drew the highest negative feedback, while Chinese problem-solving satisfaction was around 58%.The authors conclude that creative and problem-solving capabilities need more tailored development to satisfy diverse global users.Source: §5.4.1 · pp. 9–10

5.4.2 Satisfaction across Usage Frequency.

  • For Chinese GUI interactions, usage frequency and satisfaction followed an approximate U-shape, with a similar high-at-both-ends, low-in-the-middle pattern in English feedback.The authors suggest improving the visibility and appeal of less frequently used features while improving medium-frequency collaboration.Source: §5.4.2 · p. 10, Figure 7 · p. 10

5.5 Expected Response Types (Q6,7,8)

  • Response expectations varied by intent: under Leisure, fewer than 40% preferred detailed responses, and fewer than 40% favored factual answers in Advice, Text Assistant, Leisure, and Seek Creativity.Only Solve Problems and API use exceeded 50% preference for professional knowledge in both language groups; the authors identify possible misalignment between user preferences and evaluation criteria.Source: §5.5 · pp. 10–11, Figure 8 · p. 11

5.6 Tool Utilization (Q9)

  • Users expected tools to match intent—such as browsing for Information Retrieval and programming or math operations for problem solving—and valued personalization across subjective scenarios.The reported tool pairings also include document or presentation generation for Text Assistant use.Source: §5.6 · pp. 11–12, Figure 9 · p. 11

5.7 Major Concerns (Q11,12)

  • Users’ major concerns and desired improvements fell into capability and trustworthiness; Hallucination, Long Context Processing, and Multimodal Ability each received over 50% of votes.Trustworthiness concerns include private and safe interactions.Source: §5.7 · pp. 12–13, Figure 10 · p. 12

6 DISCUSSION ABOUT FUTURE DIRECTION (RQ 4)

The discussion argues that LLM research should assess systems against real-world user needs, not rely solely on standardized tests or general evaluators. It outlines directions spanning user intent, personalization, tools, trustworthiness, and cross-linguistic and cultural understanding.

6.1 User-centric Evaluation

  • Evaluate generative AI with real-world scenarios and evaluators calibrated to human preferences, rather than relying solely on standardized test performance or general rankings.The discussion notes that benchmarks and evaluation criteria may not match practical applications and user expectations.Source: §6.1 · p. 13

6.2 User Intent Modeling

  • Develop intent modeling that recognizes the different scenarios and expectations behind seemingly straightforward inputs, enabling more adaptable services.The discussion links intent understanding to adapting to diverse scenarios through experience and data.Source: §6.2 · p. 14

6.3 Personalization

  • Build continuous user modeling that handles streaming inputs and preserves immediate and enduring states, then adapts responses as user patterns evolve.The discussion mentions pluggable modules and efficient tuning mechanisms as ways to support regular updates and responsiveness.Source: §6.3 · p. 14

6.4 Tool Utilization

  • Advance tool use from predefined intent-tool pairings to self-exploration of available tools and, eventually, autonomous toolmaking.The proposed progression includes crafting custom utilities for interaction with the digital and physical world.Source: §6.4 · p. 14

6.5 Trustworthiness

  • Address trustworthiness through privacy protections, safer outputs, and ethical development focused on fairness, inclusivity, and bias mitigation.Suggested privacy strategies include excluding sensitive data from training, local message processing, and selective unlearning.Source: §6.5 · p. 14

6.6 Cross Linguistic and Cultural Development

  • Build both general competence and cross-linguistic and cultural understanding in response to observed differences between English- and Chinese-speaking participants.The cited differences include usage distributions and satisfaction across some scenarios.Source: §6.6 · p. 15

7 LIMITATIONS

The survey sample may be skewed toward highly educated people and those already interested in LLMs. The authors caution that this limits how broadly the findings generalize.

7 LIMITATIONS

  • Survey recruitment through graduate-student and professor social media suggests the sample may skew toward highly educated participants.Participation was voluntary and may also attract people already interested in LLMs or anonymous user studies.Source: §7 · p. 15
  • Voluntary participation may select for respondents already interested in LLMs or inclined to join anonymous user studies.Source: §7 · p. 15
  • The authors describe the sample as a specific segment, not a random general-user group, limiting broader generalization.Source: §7 · p. 15

8 CONCLUSION

The paper broadens user-intent classification to general LLM interfaces and uses a survey to examine user experiences and concerns. Its findings identify usage patterns and inform future research directions centered on human-AI collaboration.

8 CONCLUSION

  • The paper develops a taxonomy of 7 intents for general LLM interfaces from open-source ChatGPT logs, with human verification.Source: §8 · p. 15
  • The survey of 411 people yielded 11 findings, including widespread daily use and three intent clusters across GUI and API use.The authors note that subjective GUI use may be overlooked by existing research.Source: §8 · p. 15
  • The empirical findings inform future LLM research directions focused on human-AI collaboration and users’ diverse, evolving needs.Source: §8 · p. 15

A QUESTIONNAIRE

The questionnaire asks about service use and frequency, LLM-use intents, experience and response preferences across intents, useful tools, and service shortcomings.

A QUESTIONNAIRE

  • Respondents rank services by frequency of use and report their usage frequency from daily to never.Source: A QUESTIONNAIRE · p. 19
  • The questionnaire asks which of seven listed intents describe respondents’ LLM use and invites additional comments on the categories.The listed options include advice, information retrieval, leisure, creativity, specialized problem-solving, text assistance, and API use.Source: A QUESTIONNAIRE · p. 19
  • For each intent, respondents rate system performance and indicate whether answers should be concise or detailed and fact- or creation-based.Source: A QUESTIONNAIRE · p. 19
  • Respondents identify tools that could improve responses, including browsing, input analysis, personalization, programming, and content generation tools.Source: A QUESTIONNAIRE · p. 19
  • The questionnaire asks which system aspects most need optimization, offers an attention-check item, and ends with an open comment prompt.Listed concerns include hallucination, long-context processing, multimodal capabilities, personalization, privacy, and safety.Source: A QUESTIONNAIRE · p. 19

B DEMOGRAPHIC INFORMATION

The demographic information is limited to automatically recorded IP addresses; Figure 11 shows their distribution outside China.

B DEMOGRAPHIC INFORMATION

  • Figure 11 presents the IP distribution excluding China.Source: Figure 11 · p. 20
  • 291 Chinese-version respondents’ IPs originated in China and 6 elsewhere; the English version had 66 Chinese IPs and 48 from other countries.Source: B DEMOGRAPHIC INFORMATION · p. 20
  • The English survey was shared with an international graduate program in China, including about 30 English-speaking students there.Source: B DEMOGRAPHIC INFORMATION · p. 20
Loading 2401.08329v1…