PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies

Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, Nagu Rangan, Tara Safavi, Siddharth Suri, Mengting Wan, Leijie Wang, Longqi Yang

arXiv:2309.13063v3cs.IRcs.AIcs.CL

TL;DR

Log-based intent analysis is difficult because intents evolve and logs may lack context, while existing approaches can be costly or inflexible. The paper develops a human-validated LLM process for generating and applying intent taxonomies, and reports that LLMs can collaborate in taxonomy construction and perform annotation after validation. It concludes that the approach supports analysis of search and chat logs, while cautioning against use with certain vulnerable populations.

  • Problem

    User intents are fluid, logs may lack context, and manual or machine-learned approaches can be costly or inflexible for dynamic datasets.

    Source: §1 · p. 1

  • Method

    The paper uses an LLM with human researchers and assessors to generate, refine, validate, and apply user-intent taxonomies to log data.

    Source: §1 · p. 2

  • Results

    The authors report that LLMs can construct a reasonable-quality taxonomy zero-shot, but human verification and fine-tuning are needed; after validation, an LLM can perform annotation effectively and accurately.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors present LLMs as collaborators in taxonomy construction and validation, then as annotation tools for analyzing large-scale data.

    Source: §6 · p. 10

    The authors caution that the method should not be used with certain vulnerable populations, including children.

    Source: §7 · p. 11

Abstract

from arXiv · show

Log data can reveal valuable information about how users interact with Web search services, what they want, and how satisfied they are. However, analyzing user intents in log data is not easy, especially for emerging forms of Web search such as AI-driven chat. To understand user intents from log data, we need a way to label them with meaningful categories that capture their diversity and dynamics. Existing methods rely on manual or machine-learned labeling, which are either expensive or inflexible for large and dynamic datasets. We propose a novel solution using large language models (LLMs), which can generate rich and relevant concepts, descriptions, and examples for user intents. However, using LLMs to generate a user intent taxonomy and apply it for log analysis can be problematic for two main reasons: (1) such a taxonomy is not externally validated; and (2) there may be an undesirable feedback loop. To address this, we propose a new methodology with human experts and assessors to verify the quality of the LLM-generated taxonomy. We also present an end-to-end pipeline that uses an LLM with human-in-the-loop to produce, refine, and apply labels for user intent analysis in log data. We demonstrate its effectiveness by uncovering new insights into user intents from search and chat logs from the Microsoft Bing commercial search engine. The proposed work's novelty stems from the method for generating purpose-driven user intent taxonomies with strong validation. This method not only helps remove methodological and practical bottlenecks from intent-focused research, but also provides a new framework for generating, validating, and applying other kinds of taxonomies in a scalable and adaptable way with reasonable human effort.

1 INTRODUCTION

User-intent analysis must handle fluid, evolving intents and log data with limited context, while current manual and quantitative methods have cost or nuance limitations. The paper proposes an LLM-and-human methodology for generating and applying validated intent taxonomies to search and chat logs.

1 INTRODUCTION

  • Intent extraction from logs is difficult because intents are fluid and logs may lack sufficient context, while AI-driven chat behaviors evolve rapidly.These changes call for on-demand, task-focused labels and taxonomies.Source: §1 · p. 1
  • Manual coding is time-consuming, while existing quantitative approaches may miss the nuance and diversity of user intents and experiences.The paper asks whether LLMs can create and apply meaningful intent categories at scale.Source: §1 · p. 1
  • The paper investigates an end-to-end approach using GPT-4 as an LLM collaborator in iterative qualitative analysis of AI chat logs.The process leverages LLM-generated summaries, questions, and categories from chat transcripts.Source: §1 · p. 2
  • The novelty is integrating LLM-inferred data structure with human assessor curation, reducing effort while validating machine-generated outputs.The authors report applying the pipeline to heterogeneous search and chat logs, with multiple datasets, assessors, and three LLMs.Source: §1 · p. 2

2 RELATED WORK

Prior work has developed intent taxonomies and applied LLMs to research and log analysis. This paper positions its contribution as human-validated LLM taxonomy generation and application to search and conversational logs.

2.1 Taxonomy Generation, Validation, and Use

  • Taxonomies organize concepts hierarchically, but their generation and validation must balance coverage, coherence, consistency, granularity, usability, and adaptability.Requirements may differ across domains and contexts.Source: §2.1 · p. 2
  • The paper reports being the first to use LLMs to generate taxonomies for conversational search and validate them with human assessors.It presents a method intended for researchers to generate taxonomies for their own needs.Source: §2.1 · p. 2

2.2 Use of LLMs in Research

  • Prior work used LLMs for scientific discovery, qualitative analysis, relevance judgments, synthetic IR datasets, and user modeling.The paper proposes human-collaborative LLM use for intent understanding and taxonomy generation in search research.Source: §2.2 · p. 2

2.3 Log Analysis and Insights

  • Log analysis has traditionally involved extracting data with scripts, manual analysis, and optionally human annotation for pattern discovery and model training.LLMs had been proposed as supportive tools for text analysis and annotation; this paper introduces their collaboration with researchers on retrospective log data.Source: §2.3 · pp. 2–3

2.4 Understanding User Intents in Search

  • Earlier search-intent taxonomies were built through manual log inspection, while query clustering and categorization have also been used to derive and apply taxonomies.The paper extends this area by using human-in-the-loop LLMs to generate intent taxonomies and assign search activities to categories.Source: §2.4 · p. 3

3 METHODOLOGY FOR GENERATING AND VALIDATING A USER INTENT TAXONOMY

The paper develops a bottom-up, human-in-the-loop process for generating, validating, and applying a user-intent taxonomy to Bing Chat logs. It reports substantial agreement in annotation and examines where human and GPT-4 labels diverge.

3.1 Data

  • The study sampled 1,149 English-language conversations from May–June 2023, using 1,000 for taxonomy development and the rest for validation and testing.Some requests included non-English words.Source: §3.1 · p. 3

3.2 Evaluating the Taxonomy

  • The researchers evaluated the taxonomy for comprehensiveness, consistency, clarity, accuracy, and conciseness.These criteria informed prompt design, revision, and evaluation of the taxonomy.Source: §3.2 · p. 4

3.3 Phase 1: Taxonomy Generation

  • GPT-4 generated a single-level taxonomy with 4–6 categories, including labels, descriptions, and examples; researchers consolidated variations across generated versions.The design targeted high-level intent understanding.Source: §3.3 · p. 4

3.4 Phase 2: Taxonomy Validation

  • Initial annotator agreement was low, prompting discussions and revised instructions; negative examples and clarified category definitions brought disagreement down to 20%.Annotators sometimes inferred users’ broader goals from requests; the revision emphasized labeling only what the data supported.Source: §3.4 · p. 4

3.5 Phase 3: Taxonomy Application and Testing

  • GPT-4 and two human annotators coded a separate set of 124 conversations with the modified taxonomy; a third annotator coded them, disagreements used majority vote, and Cohen’s kappa between GPT-4 and majority human labels was 0.7212.When all three chose different labels, the case was labeled Other; agreement was described as substantial.Source: §3.5 · pp. 4–5
  • Cohen’s kappa was 0.7620 between human annotators and 0.7212 between GPT-4 and the human majority; the authors describe both as substantial agreement.The authors interpret these results as support for the taxonomy’s validity and GPT-4’s reliability for annotation.Source: §3.5 · p. 5

3.6 Insights About and From Annotations

  • Information Retrieval was the largest category and Learning the next largest; their overlap made IR-versus-LR distinctions the most frequent source of human disagreement.The paper notes that a Learning task in search always contains Information Retrieval, while the boundary between them may be difficult to judge.Source: §3.6 · p. 5, Table 2 · p. 5
  • GPT-4 labeled 12 of 124 segments as Information Retrieval when humans labeled them Learning; repeated GPT-4 annotations had Fleiss’ kappa of 0.8516.The authors report no ground truth for these cases and argue that GPT-4’s labels were more objectively and consistently assigned in this comparison.Source: §3.6 · p. 5

4 ADDITIONAL VALIDATIONS USING OPEN-SOURCE LLMS

The paper tests whether taxonomy generation and application extend beyond GPT-4, including to multilevel taxonomies. Results show broad category consistency across models, while finer-grained subcategories vary more and model–human agreement ranges from moderate to substantial.

4.1 Single Level Taxonomy Generation

  • Across 30 bootstrap runs (10 per LLM), most taxonomies coalesced around the same five intent categories, which were relatively common across all three LLMs, indicating prompt robustness and category stability.Source: §4.1 · p. 6, Table 4 · p. 6

4.2 Multilevel Taxonomy Generation

  • With two-level taxonomies, level-one categories remained relatively consistent across models, but level-two subcategories varied more.Holding level-one categories constant substantially improved subcategory consistency; the authors say further data and iterations could resolve initial disagreements.Source: §4.2 · p. 6
  • Some subcategories captured specific requests, such as restaurant or movie reviews; that granularity may suit some applications but add noise in others.The authors suggest pruning such subcategories when they are not useful.Source: §4.2 · p. 6
  • For meaningful subcategories, the authors recommend requiring a minimum number of supporting examples; deeper taxonomies may need further prompt optimization and human editing.Other applications may require more levels or granularity than the paper’s flat taxonomy, and the paper notes that each new level needs its own consideration.Source: §4.2 · pp. 6–7

4.3 Taxonomy Application

  • Cohen’s kappa agreement among humans and three LLMs ranged from moderate to substantial, indirectly supporting the taxonomy’s applicability to new data.The authors caution that agreement scores do not measure every quality, such as usefulness or objectivity, which may depend on the application.Source: §4.3 · p. 7

5 APPLICATION OF USER INTENT PIPELINE

The pipeline uses an LLM with human involvement to build, validate, and apply an intent taxonomy to Bing search and chat logs. In the comparison, create, learn, and leisure requests leaned toward chat, while information retrieval and advice or recommendation were nearly evenly distributed, with a slight search skew.

5 APPLICATION OF USER INTENT PIPELINE

  • The pipeline applies a human-in-the-loop method to heterogeneous search and chat logs containing only user requests, with human intervention shaping the process and providing light-touch validation.The authors describe this as applying a process for constructing and applying a reliable taxonomy that has already been validated.Source: §5 · p. 7, Figure 3 · p. 8

5.1 Stepwise Process for the Full Pipeline

  • The study sampled users of both Bing Search and Bing Chat, using 500 queries and 500 chat requests for training and reserving the rest for testing.The 2,456 queries and 15,531 chat requests were collected from May–June 2023; the task was considered more challenging because the logs covered different modalities and lacked system responses.Source: §5.1 · p. 7
  • The zero-shot taxonomy began with five categories: advice or recommendation, create, information retrieval, learn, and leisure.The initial prompt specified a maximum of five categories and criteria including comprehensiveness, consistency, clarity, accuracy, and conciseness.Source: §5.1 · p. 7
  • In a random sample of 100 labels manually checked after the LLM labeled the taxonomy’s training data, 95 assignments followed their label definitions, and none was assigned the “Other” category.The LLM labeled the same 1,000 data points used to generate the taxonomy (500 search queries and 500 chat requests).Source: §5.1 · p. 8
  • On test data—1,956 search queries and 15,031 chat requests processed by GPT-4 with the final taxonomy in the prompt—no sample was marked with the ‘Other’ label, and no category had fewer than 2% of samples.Source: §5.1 · p. 8

5.2 Insights about Intents in Search vs. Chat

  • Create, learn, and leisure intents leaned toward chat, while information retrieval and advice or recommendation were nearly even, with a slight skew toward search.The figure presents this distribution from the perspective of where users go for each intent.Source: §5.2 · p. 8, Figure 4 · p. 8
  • The intent comparison uses a single user request as its unit of analysis, so a learning task spanning multiple search queries may not appear as one request.The authors note that users may issue several queries during a search session to accomplish a learning task.Source: §5.2 · p. 9

5.3 Steps for Generating Intent Taxonomies

  • The authors recommend defining the application and data needs, iteratively building and checking the taxonomy, clarifying labels with positive and negative examples, and assessing accuracy and conciseness on test data.Their guidance includes sampling labels for human review, measuring inter-coder reliability, and considering bias checks such as comparing against human-labeled distributions.Source: §5.3 · p. 9

6 CONCLUSION

The paper presents an LLM–human methodology for generating, validating, and applying user-intent taxonomies, demonstrated on AI chat logs and a search-versus-chat case study. The authors report that human verification and fine-tuning help meet taxonomy quality criteria, while the method reduces effort and still needs testing across more taxonomy designs and applications.

6 CONCLUSION

  • Intent identification is challenging because intents shift and log data may lack context, while labeling methods can be costly or miss nuance.For rapidly changing modalities such as AI chat, applying LLMs to build and use taxonomies also risks an unevaluated feedback loop.Source: §6 · p. 9
  • The methodology generates, validates, and applies user-intent taxonomies, demonstrated through AI-chat-log analysis and a search-versus-chat case study.The authors identify the methodology, rather than the application results, as the paper’s primary contribution.Source: §6 · pp. 9–10
  • The case-study pipeline took less than half the time and effort of developing the method, and optional steps in Section 5.3 would take substantially less.The authors say this efficiency could help researchers adapt to evolving intents quickly and at lower cost and effort.Source: §6 · p. 10
  • Zero-shot LLM construction produced a reasonable-quality taxonomy, but human verification and fine-tuning were needed to meet quality criteria reliably.The criteria include comprehensiveness, consistency, clarity, accuracy, conciseness, adaptability, and scalability.Source: §6 · p. 10
  • Further experiments are needed across different taxonomy designs and applications requiring more levels or granularity to assess the methodology’s flexibility and scalability.The authors note that Amirizaniani et al. [4] [5] have also applied the method to audit LLMs using a multi-probe approach and human-in-the-loop.Source: §6 · p. 10
  • After taxonomy construction and validation, an LLM can annotate data at scale; the authors conclude it can act as a copilot rather than replace researchers.They report that GPT-4 sometimes produced labels truer to the data than human annotators when the two disagreed.Source: §6 · p. 10

7 ETHICAL CONSIDERATIONS

The paper’s method relies heavily on LLMs, which can carry risks related to training bias, toxicity, hallucination, and user overtrust. The authors caution against applying it to vulnerable populations and call for due diligence to protect users when using other datasets.

7 ETHICAL CONSIDERATIONS

  • The method should not be used on vulnerable populations such as children and people with disabilities, because results could mislead or harm them.For other datasets, users must independently assess privacy and protection risks; the authors’ own nonpublic data lacked identifying information and received privacy safeguards.Source: §7 · p. 11

Related papers

Loading 2309.13063v3…