PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestion

Jinheon Baek, Nirupama Chandrasekaran, Silviu Cucerzan, Allen herring, Sujay Kumar Jauhar

arXiv:2311.06318v2cs.IRcs.AIcs.CLcs.LG

TL;DR

Personalizing LLM outputs is challenging when per-user retraining is prohibitively expensive, despite the value of tailoring responses to user knowledge and interests. K-LaMP augments prompts with entity context derived from search histories, and human evaluations find it outperforms several LLM-powered baselines for contextual query suggestion. The authors also note that judges may have difficulty inferring users’ goals from limited evaluation data.

  • Problem

    Personalizing LLM outputs to users’ preferences, needs, and knowledge remains challenging because retraining or fine-tuning for individuals is prohibitively expensive.

    Source: §1 · p. 2

  • Method

    K-LaMP augments LLM prompts with retrieved entities from a lightweight personal knowledge store built by aggregating users’ search queries and browsed pages.

    Source: §1 · p. 2, §3.2.1 · p. 4

  • Results

    On real-world Bing search logs, human evaluation found K-LaMP substantially outperformed several LLM-powered baselines in relatedness and usefulness while maintaining high search result quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Entity-based prompt augmentation offers a lightweight approach that relies on existing logging infrastructure and is more amenable to privacy and scalability considerations than profile-based personalization.

    Source: §1 · p. 2

    Judges may have difficulty assessing users’ varied search goals from limited, albeit rich, evaluation data.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) excel at tackling various natural language tasks. However, due to the significant costs involved in re-training or fine-tuning them, they remain largely static and difficult to personalize. Nevertheless, a variety of applications could benefit from generations that are tailored to users' preferences, goals, and knowledge. Among them is web search, where knowing what a user is trying to accomplish, what they care about, and what they know can lead to improved search experiences. In this work, we propose a novel and general approach that augments an LLM with relevant context from users' interaction histories with a search engine in order to personalize its outputs. Specifically, we construct an entity-centric knowledge store for each user based on their search and browsing activities on the web, which is then leveraged to provide contextually relevant LLM prompt augmentations. This knowledge store is light-weight, since it only produces user-specific aggregate projections of interests and knowledge onto public knowledge graphs, and leverages existing search log infrastructure, thereby mitigating the privacy, compliance, and scalability concerns associated with building deep user profiles for personalization. We validate our approach on the task of contextual query suggestion, which requires understanding not only the user's current search context but also what they historically know and care about. Through a number of experiments based on human evaluation, we show that our approach is significantly better than several other LLM-powered baselines, generating query suggestions that are contextually more relevant, personalized, and useful.

1 INTRODUCTION

The paper addresses the challenge of personalizing LLM outputs without costly per-user retraining, focusing on search suggestions that depend on both users’ interests and knowledge. It introduces K-LaMP, which augments prompts with entity-based context from search histories and evaluates it for contextual query suggestion.

1 INTRODUCTION

  • Per-user retraining or fine-tuning is prohibitively expensive, yet personalization is important for applications including query suggestion, recommendations, snippets, and question answering.Prior retrieval-based approaches may be insufficient when deeper understanding of users’ personal knowledge is needed, while deep profiles raise privacy and scalability concerns.Source: §1 · p. 2
  • The entity-centric knowledge store aggregates entities from users’ queries and browsed pages, with views for familiar, unfamiliar, and recently lapsed entities.Retrieved entities provide succinct prompt additions and the approach relies on existing search-log infrastructure.Source: §1 · p. 2
  • Contextual query suggestion conditions recommendations on the web page being read as well as historical queries, so user expertise can affect which suggestions fit.The task differs from approaches based only on surface-level query-page relationships or page titles.Source: §1 · p. 2
  • K-LaMP significantly outperforms several LLM-powered query suggestion baselines on relatedness, usefulness, and ranking, while tying on validity and maintaining high search result quality.The evaluation uses human judgments on real-world Bing search logs; additional analyses report effective retrieval and gains with longer histories.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior work personalizes LLM outputs by adding interaction histories or retrieved context, while this paper focuses on representing users’ knowledge as well as interests. It studies contextual query suggestion conditioned on the full text of the page a user is viewing.

Large Language Models

  • LLMs can generate coherent responses across diverse tasks, but producing outputs that capture individual users’ preferences, needs, and knowledge remains challenging.Prompt engineering can customize responses, and external factual knowledge can augment model knowledge for question answering.Source: §2 · p. 2

Large Language Models for Personalization

  • Earlier personalization methods add interaction histories to prompts or retrieve relevant memories, but large histories can exceed prompt limits and may contain irrelevant material.Some approaches process memories into higher-level insights; deep profiles instead capture knowledge with privacy and scalability costs.Source: §2 · pp. 2–3
  • K-LaMP represents user knowledge and interests with entities mined from search queries and page visits, enabling lightweight retrieval without explicit profiling and reducing prompt length.This contrasts with prior personalization work focused largely on user interests.Source: §2 · p. 3

Search Query Suggestion

  • Contextual query suggestion conditions recommendations on the full text of the page being viewed, unlike prior work using clicked pages through surface-level associations.The task highlights the need to account for what users know as well as what interests them.Source: §2 · p. 3

3 K-LAMP: KNOWLEDGE-AUGMENTED LLMS FOR PERSONALIZED QUERY SUGGESTION

K-LaMP personalizes LLM query suggestions by combining a user’s query history, current page, and retrieved personal knowledge. Its knowledge stores range from a linear history stream to aggregated entities, with retrieval conditioned on the current search context.

3.1 Problem Statement

  • Contextual query suggestion extends conventional query suggestion by conditioning predicted queries on the current query, historical queries, and the text of the page being read.The paper uses an LLM with aggregated entity-centric knowledge in this context to generate personalized recommendations.Source: §3.1 · p. 3

3.2 Knowledge Store for Output Personalization

  • The LLM context concatenates historical queries, the current web page, and a contextually relevant subset of a personal knowledge store.The paper frames the store as auxiliary user-specific context for enriching the prompt.Source: §3.2 · p. 4

3.2.1 Constructing Personal Knowledge Stores.

  • The stores are built from search-engine interaction histories: K_s retains a timestamped stream of queries and clicked pages, while K_e aggregates entities from those interactions.The linear stream is a partial view of actions already logged by search engines; entities are short, aggregatable units for representing user interests and knowledge.Source: §3.2.1 · pp. 4–5
  • Entity linking projects and aggregates mentions onto public knowledge-base subgraphs, supporting a relatively lightweight, scalable store with small privacy concerns and records removable by deleting associated entities.The paper also notes that entity aggregation can support k-anonymization.Source: §3.2.1 · p. 5

3.2.2 Contextual Retrieval from Personal Knowledge Stores.

  • K_e retrieval matches entities in the current query and page to familiar, unfamiliar, or lapsed entities, sampling five for the first two categories and filtering lapsed entities to those not engaged with in the preceding two weeks.Familiar entities are sampled by frequency, unfamiliar entities by inverse frequency, and lapsed entities by frequency after the time filter.Source: §3.2.2 · p. 5

4 EXPERIMENTAL SETUP

The section describes the Bing search-log benchmark, the query-suggestion models, and the human-evaluation procedure used to assess them.

4.1 Data

  • 1,000 users’ logs form the benchmark; the latest 10 sessions per user are prediction targets, with earlier sessions used to build personal knowledge stores.The data covers three months of Bing logs, filtered for clicked pages in Wikipedia or selected news domains and users with at least 100 page visits.Source: §4.1 · p. 5

4.2 Baselines and Our Model

  • Four GPT-4 setups range from query-history suggestions to contextual suggestions using a page, search-and-browsing retrievals, or K-LaMP’s entity-centric knowledge.Older RNN- and BART-based approaches were excluded from the main experiments because they were not designed for long contexts such as the currently viewed page.Source: §4.2 · pp. 5–6

4.3 Evaluation Setup

  • Because existing query-suggestion metrics do not account for the current document and user-specific context, the study uses human evaluation.Judges score Validity, Relatedness, and Usefulness, and rank outputs by likely click affinity to the user’s interests, knowledge, and search context.Source: §4.3 · p. 6
  • 12 annotators evaluated 1,309 result sets, and approximately 27% were double-annotated; reported inter-annotator agreement was moderate to high.The agreement results cover Validity, Relatedness, Usefulness, and Ranking.Source: §4.3 · p. 6, Table 2 · p. 6
  • All baselines and variants used GPT-4 from June 13, 2023, with temperature 0.7 and top-p 0.95; NEMO linked entities for the knowledge stores.The shared GPT-4 setup was used for a fair comparison.Source: §4.3 · pp. 6–7

5 EXPERIMENTAL RESULTS

K-LaMP outperforms the baselines on the principal personalization measures, while auxiliary analyses examine retrieval strategies, history length, LLM choice, and automatic evaluation. These analyses show where personalization gains arise and where automatic metrics do not track human judgments.

5 EXPERIMENTAL RESULTS

  • K-LaMP significantly outperforms all baselines on Relatedness, Usefulness, and Ranking, while tying Contextual Query Suggestion on Validity.The authors hypothesize that Contextual Query Suggestion with K_s underperforms because its retrieved information may be poorly matched to the current search context.Source: §5 · p. 7
  • Entity-centric retrieval yields higher Retrieval Relevance than linear search-history retrieval, with exact entity matching offering more control than fuzzy embedding-based retrieval.Retrieval Relevance is the average score on a yes/no judgment of whether retrieved context fits the current search context.Source: §5 · p. 7

5.1 Additional Studies and Analyses

  • Across entity-matching strategies, Validity is nearly unchanged, while unfamiliar entities achieve the highest Relatedness and Usefulness scores, even above full K-LaMP.The authors suggest that unfamiliar entities may encourage exploration rather than revisiting familiar topics.Source: §5.1 · p. 7, Figure 3 · p. 7
  • As interaction histories lengthen, K-LaMP alone consistently improves Relatedness and Usefulness; Validity remains unaffected.The authors attribute the improvement plausibly to aggregation in K-LaMP’s entity-centric store, unlike linearly stored histories in other approaches.Source: §5.1 · p. 7, Figure 4 · p. 7
  • Query Suggestion is agnostic to the LLM, whereas Contextual Query Suggestion and K-LaMP are not; with GPT-3.5-Turbo, K-LaMP has comparable Usefulness to GPT-4 Contextual Query Suggestion.The compared GPT-3.5-Turbo evaluation uses 128 sets of query suggestions.Source: §5.1 · pp. 7–8, Table 4 · p. 8

5.2 Automatic Evaluation Setup

  • Automatic Validity compares suggestions with search-result titles and snippets, Relatedness with retrieved personal entities, and Usefulness with users’ subsequent queries.Each similarity is computed as a dot product of Contriever representations.Source: §5.2 · p. 8
  • Automatic metrics show moderate correlation with human judgments for Validity and Relatedness, but no correlation for Usefulness.The authors note that recommendations need not align perfectly with actual behavior because users may value surprising suggestions.Source: §5.2 · p. 8, Table 5 · p. 8
  • Human evaluation is slow, expensive, and difficult to scale, and the authors leave rubric-based LLM evaluation and trained metrics for future work.The authors also note that automatically evaluating contextual recommendations against complex personal preferences and knowledge is difficult.Source: §5.2 · p. 8

6 CONCLUSION

K-LaMP combines a lightweight entity-centric personal knowledge store with LLMs for contextual query suggestion, and human evaluation found its recommendations better related to users’ intent and more useful than competing LLM-powered models.

6 CONCLUSION

  • Human evaluation on an extensive test set found K-LaMP’s recommendations better related to user intent, more useful, and consistently ranked above several LLM-powered query-suggestion models.The authors identify improved automatic metrics and real-world deployment as future evaluation directions.Source: §6 · p. 8
  • K-LaMP personalizes LLM outputs by augmenting them with knowledge from historical user interactions with a search engine.The paper evaluates the framework on contextual query suggestion, which requires modeling users’ contextual interests and knowledge.Source: §6 · p. 8
  • Its lightweight entity-centric personal knowledge store is constructed from users’ search queries and viewed web pages.Source: §6 · p. 8
  • The authors conclude that entities are effective atomic units for personal knowledge, offering a middle ground in performance, flexibility, privacy, and scalability versus deep profiles or linearized interaction histories.Source: §6 · p. 8
  • Future work includes applying knowledge augmentation to other search tasks, using additional data sources or domains, and improving evaluation with automatic metrics and real-world deployment.Source: §6 · p. 8

A PROMPTS

This appendix lists the prompts used to elicit personalized contextual query suggestions from K-LaMP and comparison approaches.

A PROMPTS

  • K-LaMP’s full prompt is provided in Table 8 for eliciting personalized contextual query-suggestion responses from LLMs.Source: A PROMPTS · p. 11
  • The appendix provides separate prompts for the query-suggestion baseline, contextual query-suggestion baseline, and contextual query suggestion with Kₛ in Tables 9, 10, and 11, respectively.Source: A PROMPTS · p. 11
  • These prompts instantiate the approaches compared for personalized contextual query suggestion.Source: A PROMPTS · p. 11

B QUERY SUGGESTION EXAMPLES

The examples compare K-LaMP’s suggestions with non-contextual and commercial query-suggestion methods, showing how current-page context and historical interests shape suggestions.

B QUERY SUGGESTION EXAMPLES

  • K-LaMP suggests a contrastive query about Tim Cook and Steve Jobs’ approaches to Apple product development, unlike a non-contextual suggestion that repeats the page topic.The example uses a manufactured user query about Tim Cook, a page about his leadership, and personal entities such as Apple products.Source: B QUERY SUGGESTION EXAMPLES · p. 11, Table 6 · p. 12
  • K-LaMP’s “Tim Cook’s impact on Apple’s product line” better reflects the user’s technophile interests than commercial suggestions such as “Tim Cook boyfriend.”Source: B QUERY SUGGESTION EXAMPLES · p. 11
  • Table 7 presents query-suggestion results from non-contextual and commercial methods, including Bing and Google, for the example in Table 6.Source: B QUERY SUGGESTION EXAMPLES · p. 11, Table 7 · p. 12, Table 6 · p. 12
Loading 2311.06318v2…