PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, Kaixiang Lin

arXiv:2502.09597v1cs.LGcs.CL

TL;DR

LLMs still have limited ability to personalize responses by inferring, remembering, and following user preferences across long conversations. The paper introduces PREFEVAL, a benchmark with diverse preference-query pairs and generation and classification tasks, and finds substantial long-context degradation while fine-tuning improves performance.

  • Problem

    The paper asks whether LLMs can infer, remember, and follow personalized preferences over long-context conversations, an ability important for user satisfaction and engagement.

  • Method

    PREFEVAL uses 3,000 manually curated preference-query pairs across 20 topics, explicit and implicit preferences, two task types, and evaluations of 10 LLMs up to 100k-token contexts.

  • Results

    Preference following remains challenging: zero-shot precision falls below 10% for 10-turn, 3k-token conversations, deteriorates with longer contexts, and improves with PREFEVAL fine-tuning.

  • Takeaways & Limitations

    PREFEVAL provides a resource for measuring, understanding, and improving LLM preference following in personalized conversational agents.

  • Takeaways & Limitations

    GPT-o1-preview may not be a fair comparison because it may require additional test-time compute during a thinking phase.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PrefEval, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PrefEval comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PrefEval contains user personalization or preference information in both explicit and implicit forms, and evaluates LLM performance using a generation and a classification task. With PrefEval, we evaluated the aforementioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in proactively following users' preferences during conversations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PrefEval significantly improves performance. We believe PrefEval serves as a valuable resource for measuring, understanding, and enhancing LLMs' preference following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https://prefeval.github.io/.

1 INTRODUCTION

PREFEVAL addresses the limited understanding of whether LLMs can infer, remember, and proactively follow user preferences across conversations. It benchmarks this ability across preference forms, tasks, models, and context lengths, revealing substantial degradation in long-context settings.

  • PREFEVAL evaluates whether LLMs can infer, remember, and apply user preferences during conversations, a capability important for personalized interactions.The benchmark targets both explicit and implicit preferences and the challenges of extended interaction histories.
  • PREFEVAL evaluates 10 state-of-the-art LLMs across varying context lengths and methods including prompting, iterative feedback, and retrieval-augmented generation.
  • The benchmark contains 3,000 manually curated question-preference pairs across 20 topics and three preference forms.
  • Below 10% preference-following precision occurs in zero-shot 10-turn conversations of 3k tokens without explicit prompting, while longer contexts further degrade performance.
  • Multiple stated preferences can improve adherence, even when preferences conflict, possibly because they reinforce attention on preferences.
  • Fine-tuning on PREFEVAL enhances preference following and generalizes well to longer contexts.

2 THE PREFEVAL DATASET

PREFEVAL models personalization as adherence to a disclosed preference when answering a later query amid potentially distracting conversational context. Its dataset and paired generation/classification protocols test inference, retrieval, following, and proactive use of preferences.

  • 2.1 PROBLEM FORMULATION: PREFEVAL divides multi-turn conversations into coherent topic-focused sessions to assess personalization in conversational settings.
  • 2.1 PROBLEM FORMULATION: Each preference-query pair links a disclosed preference to a later query whose generic answer would likely violate that preference.Unrelated turns are inserted between preference disclosure and the query to test long-range retention and retrieval.
  • 2.1 PROBLEM FORMULATION: The benchmark targets preference inference, long-context retrieval, preference following, and personalization proactiveness.
  • 2.2 PREFEVAL STATISTICS: PREFEVAL contains 1,000 unique preference-query pairs, each represented in three preference forms, yielding 3,000 pairs across day-to-day topics.The dataset intersperses unrelated contextual turns and extends context lengths up to 100k tokens.
  • 2.5 TASK TYPES AND EVALUATION PROTOCOLS: Generation requires a long-form response, whereas classification asks the model to select one preference-aligned option from four choices.
  • 2.5 TASK TYPES AND EVALUATION PROTOCOLS: Generation responses are judged across four binary error types, with preference-following accuracy defined as the absence of every error type.The errors include preference-unaware, hallucination, inconsistent, and unhelpful responses.
  • 2.5 TASK TYPES AND EVALUATION PROTOCOLS: Classification provides faster automatic evaluation, while its results correlate strongly with generation-task results.

3 EXPERIMENTS

The experiments evaluate preference following across models, preference forms, tasks, methods, context lengths, and preference dynamics. Results show steep long-context degradation, added difficulty for implicit preferences, method-dependent gains, distinct error patterns, and improvement after fine-tuning.

  • Models and methods: The study evaluates state-of-the-art open-source and proprietary LLMs using zero-shot, Reminder, Self-Critic, Chain-of-Thought, and retrieval-augmented methods.The evaluation covers multiple context lengths and preference-following settings.
  • Explicit preference following: 80% to below 30%: zero-shot preference-following accuracy drops as conversations reach 5 turns, then approaches zero between 30 and 300 turns.These results are reported for explicit preferences in the generation task.
  • Methods: Reminder and RAG achieve the highest accuracy across models, while Reminder outperforms more complex Self-Critic and CoT prompting.RAG performs best across most models, whereas Reminder matches or exceeds RAG for Claude 3 Sonnet and Mistral 8x7b.
  • Preference forms: Implicit preferences are more difficult to infer than explicit preferences, including at shorter input lengths.The comparison evaluates three preference forms with the preference stated in the first turn.
  • Classification task: Classification achieves higher overall accuracy than generation, and its results correlate strongly with generation-based preference-following accuracy at 0.73.The classification task is presented as a potentially efficient proxy for complex generation evaluation.
  • Error analysis: Error analysis finds preference-unaware violations in zero-shot settings, while advanced methods introduce inconsistency, hallucinated preferences, and unhelpful refusals.These patterns indicate separate difficulties in retrieving preferences and generating aligned responses.
  • Fine-tuning and preference dynamics: Fine-tuning on PrefEval increases attention to preference-related information, with preference-region attention improvements up to 4.97% across test examples.The experiments also examine multiple and dynamically changing preferences across sessions.

4 RELATED WORK

Prior personalization work spans user-style imitation, profile conditioning, role-playing, and temporal consistency, while long-context benchmarks mainly test information retrieval. This work also distinguishes preference inference and proactive application from discrete instruction execution.

  • After fine-tuning, Mistral-7B achieves superior preference-following accuracy on 20% unseen test topics and improved length generalization with longer context interleavings.
  • Earlier personalization systems focused on mimicking user styles and generating personalized news headlines or reviews.
  • Recent LLM personalization benchmarks examine explicit profile conditioning, character-based role-playing, temporal character consistency, and fine-grained role representation.
  • Long-context benchmarks primarily evaluate locating facts or answers through question answering, retrieval, reasoning, and coding tasks.
  • Unlike discrete instruction-following benchmarks, this work evaluates inferring and adhering to preferences across multiple conversation turns.
  • Personalization Proactiveness measures whether models know when and where to apply user preferences, beyond executing discrete instructions.

5 CONCLUSION

The conclusion presents PREFEVAL as a benchmark for preference following in multi-session conversations and reports persistent difficulty for current LLMs. Models struggle especially with long-context recall, proactive application, and implicit preferences, while reminders help but do not solve the problem.

  • PREFEVAL evaluates preference following in multi-session conversations using explicit and implicit preferences, generation and classification tasks, and contexts up to 100k tokens.
  • Across 10 state-of-the-art LLMs, preference-following accuracy drops below 10% in default settings beyond 10 conversational turns.
  • Models struggle to proactively recall and incorporate preferences stated earlier without explicit prompting.
  • Implicit preferences create additional difficulty because models must infer them from conversations.
  • Reminder prompting shows promise for mitigating performance declines, but substantial room for improvement remains.

6 REPRODUCIBILITY STATEMENT

The reproducibility statement describes a manually curated benchmark for evaluating open-source and proprietary models and outlines planned resource releases. It also identifies contextual turns used to construct inter-turn distractors and the provision of prompts.

  • The benchmark includes a manually curated dataset for evaluating current open-source and proprietary models.
  • The authors plan to release the benchmark to enable reproduction of their results.
  • Contextual turns from the Lmsys-1M dataset will be released for constructing inter-turn distractors.
  • The authors also provide the prompts used in the study.

7 ETHICS STATEMENT

The ethics statement frames PREFEVAL’s development around responsible and ethical research practices. It emphasizes privacy, ethical data quality, bias mitigation, and research integrity.

  • PREFEVAL’s development prioritizes responsible and ethical research practices.
  • The stated ethics priorities include data privacy, ethical data quality, bias mitigation, and research integrity.
  • The dataset consists of manually curated preference-query pairs spanning 20 topics.

A.1 LIMITATION

The benchmark evaluates preference following rather than recommendation factuality, and its synthetic preference-query pairs leave room for future use of real user preferences. Implicit preference inference is designed to be likely but is not guaranteed to be perfectly accurate.

  • PREFEVAL does not evaluate the factual accuracy of recommendations.Fact-checking is treated as a separate dimension outside the benchmark’s scope.
  • Future work should incorporate real user preferences to capture more nuanced user interactions.The current preference-query pairs are synthetic despite extensive human filtering.
  • Implicit preference elicitation makes inference likely but does not guarantee 100% accuracy from multiple-choice selections.The options are designed so only one adheres to or violates the target preference.

A.2 DETAILED RELATED WORKS

Related work spans personalized dialogue and other language-generation tasks, alongside benchmarks for long-context reasoning and instruction following. PREFEVAL differs from retrieval-focused long-context tasks by requiring implicit preference inference and dynamic application across conversations.

  • LLM Personalization and Benchmarks: Earlier personalization work conditioned dialogue systems on user profiles to produce engaging conversations resembling users’ personalities or styles.Related tasks also included personalized news headline and review generation.
  • Long Context LLM and Benchmarks: Recent long-context benchmarks primarily evaluate question answering and summarization as models support increasingly large context windows.These benchmarks address long-context reasoning as context lengths expand.
  • Long Context LLM and Benchmarks: PREFEVAL tests preference following beyond needle-in-a-haystack retrieval by requiring implicit inference and dynamic application across conversation contexts.This frames long-context retrieval in a more conversational preference-following setting.

A.3 MODEL VERSION

The experiments benchmark a variety of large language models using the versions listed in Table 3. The table identifies the evaluated models and their corresponding versions.

  • PREFEVAL evaluates a variety of state-of-the-art large language models.
  • The evaluated models are identified together with their respective versions.
  • Table 3 presents the LLMs benchmarked and the versions used for evaluation.

A.4 METHODS DESCRIPTION

The methods compare direct answering with prompting, reminders, self-critique, few-shot chain-of-thought, and retrieval of relevant prior exchanges. These methods aim to help models use earlier preference information when answering later queries.

  • Zero-shot: Zero-shot evaluation has the LLM answer the user’s query directly without additional prompting.
  • Reminder: The reminder method adds an instruction to make the response consistent with the user’s earlier preference.The reminder is inserted after the question and before the response.
  • Self-Critic: Self-Critic generates an initial response, critiques preference adherence, and then produces a revised response.The critique checks whether relevant earlier preferences were followed.
  • Few-Shot Chain-of-Thought: Few-shot chain-of-thought provides five examples demonstrating preference following before the model answers the query.The prompt asks the model to consider the current user’s preferences rather than the examples’ preferences.
  • Retrieval-Augmented Generation: Retrieval-Augmented Generation retrieves the five most relevant prior exchanges using sentence-embedding similarity and presents them as context.The model is instructed to use relevant retrieved preferences while recognizing that some retrieved contexts may be unhelpful.

A.5 CLASSIFICATION TASK CORRELATION PLOT

Generation-based preference-following accuracy and classification accuracy show strong alignment across models and methods, supporting classification as a complementary evaluation approach.

  • 0.73 correlation coefficient indicates strong alignment between generation-task preference-following accuracy and classification-task accuracy.The analysis averages each scatter-plot point over 20 topics across 6 models, 5 methods, and 12 turns.
  • Preference-following evaluation spans 6 models, 5 methods, and 12 conversation turns across the two task formats.The scatter-plot analysis compares generation and classification performance under varied models, methods, and conversation lengths.
  • RAG performance is examined by comparing retrieval accuracy with preference-following performance across different Top-K values.The accompanying analysis specifically evaluates how retrieval settings relate to RAG-based preference following.
Loading 2502.09597v1…