Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale

Bowen Jiang; Zhuoqun Hao; Young-Min Cho; Bryan Li; Yuan Yuan; Sihao Chen; Lyle Ungar; Camillo J. Taylor; Dan Roth

Source-linked AI summary

Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale

Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, Dan Roth

arXiv:2504.14225v2Apr 19, 2025cs.CL

TL;DR

LLMs must infer changing user profiles from interaction histories and apply them to personalized responses in new scenarios. PERSONAMEM benchmarks this ability with simulated, multi-session histories and in-situ queries, finding that frontier models remain challenged despite reasonable preference-change recall.

Problem
It remains open whether LLMs can internalize user traits, track their evolution, and generate personalized responses in new scenarios from interaction histories.
Method
PERSONAMEM evaluates LLMs with over 180 simulated histories of up to 60 sessions across 15 scenarios using first-person in-situ queries and response selection.
Results
Frontier models achieve only around 50% overall accuracy, while recalling preference changes reaches 60–70% accuracy and incorporating latest situations reaches 30–50% accuracy.
Takeaways & Limitations
PERSONAMEM highlights persistent challenges in user-aware personalization, especially applying user knowledge across new scenarios and long contexts.
Takeaways & Limitations
Generative evaluation is limited to open-weight models on the 10-session, approximately 32k-token version because proprietary models often lack required token probabilities and computation is constrained.

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks -- from offering writing support to delivering tailored recommendations or consultations. Over time, the interaction history between a user and an LLM can provide extensive information about an individual's traits and preferences. However, open questions remain on how well LLMs today can effectively leverage such history to (1) internalize the user's inherent traits and preferences, (2) track how the user profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios. In this work, we introduce the PERSONAMEM benchmark. PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn conversations across 15 real-world tasks that require personalization. Given an in-situ user query, i.e. query issued by the user from the first-person perspective, we evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users' profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users' current situations and preferences, with frontier models such as GPT-4.1, o4-mini, GPT-4.5, o1, or Gemini-2.0 achieving only around 50% overall accuracy, suggesting room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future research in the development of truly user-aware chatbots. Code and data are available at github.com/bowen-upenn/PersonaMem.

1 Introduction

PERSONAMEM targets whether LLMs can use interaction histories to internalize user traits, track evolving preferences, and personalize responses in new scenarios. The benchmark finds that current frontier models remain unreliable, especially when applying users’ latest situations.

LLM personalization requires using interaction histories to internalize traits, track profile evolution, and generate responses for new scenarios.
PERSONAMEM contains over 180 simulated interaction histories with up to 60 multi-turn sessions across 15 personalized task scenarios.Histories are built from personas whose characteristics evolve over time, with task-specific conversations concatenated chronologically.
Around 50% overall accuracy is achieved by frontier models including GPT-4.1, o4-mini, GPT-4.5, o1, and Gemini-2.0-Flash using direct prompts.Llama-4-Maverick reaches 43% in the same evaluation setting.
60–70% accuracy is obtained for recalling preference changes, versus 30–50% accuracy for incorporating users’ latest situations into responses.The benchmark also analyzes history length, preference positioning, and memory components.
The benchmark assesses 15 LLMs on seven types of in-situ queries across 15 task scenarios, including whether knowledge transfers to new scenarios.

PERSONAMEM Benchmark: Overview

The PERSONAMEM benchmark represents users with static and evolving characteristics across interleaved multi-session conversations. It evaluates whether models select responses aligned with the user’s current profile, including transfer to new tasks.

Each benchmark persona combines demographic information with dynamic traits, preferences, and life events that change over time.
Sessions span multiple task scenarios and are interleaved according to their temporal order.Users seek suggestions based on their needs and current situations at different points in profile evolution.
In-situ queries are first-person questions evaluated as four-choice selections, with distractors based on outdated or irrelevant profile information.
The benchmark evaluates seven query types covering profile memorization, preference evolution, and personalized response generation in new scenarios.One query type tests transfer of user knowledge from other task scenarios.
PERSONAMEM includes 20 personas and over 180 histories spanning 10, 20, or 60 sessions and approximately 32k, 128k, or 1M tokens.Conversations cover 15 topics, including therapy, legal advice, and recommendations for books, music, movies, and food.

PERSONAMEM At Scale

PERSONAMEM uses a modular synthetic-data pipeline to build coherent, scalable persona-oriented histories and long-context conversations. Structured timelines support grounded questions, distractors, and quality validation.

A modular GPT-4o-powered pipeline synthesizes persona-oriented, multi-session conversations with long contexts at approximately $2 per persona-topic dataset.The design emphasizes cost-effectiveness and scalability.
Personas are augmented with demographics, personal details, and timelines populated with events aligned to each persona.General histories include education, career development, and life experiences.
Each topic-specific history records initial preferences, timestamped events, preference updates, and reasons for those changes.The construction aims to maintain coherent progression while connecting experiences to the persona.
Ground-truth profiles and preferences are extracted at specific times, while distractors overlook or contradict the user persona.Questions answerable without conversation history are excluded.
Timeline segments are expanded into chronological multi-turn sessions covering the corresponding topic-specific history and additional storytelling context.
Sessions are topologically sorted and concatenated while preserving within-topic causality, enabling multiple valid cross-topic interleavings.Short inserted interactions further extend contexts and simulate natural interactions.
Human judgments on 90 random query–response pairs reached 97.8% appropriateness, 95.6% relevance, 97.8% correctness, and 90.0% best-response agreement.

4 Experiment

PERSONAMEM evaluates long-context personalization through discriminative and generative response selection. Models recall simpler facts more reliably than they apply evolving preferences in novel scenarios, while retrieval modules improve performance.

Evaluation Setup: Models receive a user’s history and in-situ query, then select the response matching the user’s current state among four choices.Evaluations include discriminative choice selection and generative scoring of response options.
Long-Context Results: Around 52% overall accuracy is achieved by the strongest models in the multiple-choice setting, leaving substantial room for improvement.GPT-4.5, GPT-4.1, and Gemini-1.5 perform best overall, while reasoning models show no competitive advantage over non-reasoning models.
Long-Context Results: Models perform reasonably well on recalling static user facts but struggle to incorporate the latest preference or user state into responses.Recalling preference evolution generally performs better than applying the latest preference in an answer.
Long-Context Results: Novel suggestions, preference-aligned recommendations, and generalizing reasons to new scenarios are the lowest-performing query types.These tasks require generating personalized responses beyond directly retrieving known information.
History Position: Performance is generally better when relevant information appears near the beginning or end of the history than in middle sessions.This pattern is consistent with information being lost in the middle of long contexts.
External Memory: External memory modules significantly improve accuracy, with RAG outperforming Mem0 across most question types.RAG especially benefits recalling user-shared facts and generalizing to new scenarios, whereas gains are smaller for revisiting reasons behind preference updates.
Generative Evaluation: Generative evaluation shows the same difficulty pattern: factual recall and preference tracking are easier than new suggestions and scenario generalization.The generative setting evaluates open-weight models on the 10-session, approximately 32k-token benchmark version.

5 Related Work

Prior work covers long-term memory, persona-based adaptation, and preference-following, but PERSONAMEM emphasizes coherent evolving personas and personalization beyond retrieval. Its design uses realistic multi-session interactions and fine-grained in-situ queries.

Personas: Persona-based evaluation captures both demographic-level and individual-level preferences, supporting assessment of adaptation to users and environments.Personas are short biographies that represent these preference levels and can be generated at scale.
Benchmark Comparisons: LOCOMO and LongMemEval focus on general long-term memory tasks, whereas PERSONAMEM evaluates personalization beyond memory retrieval.PERSONAMEM grounds conversations in coherent personas with evolving preferences.
Benchmark Comparisons: PERSONAMEM differs from PrefEval by modeling coherent persona-oriented conversations rather than inserting generated preferences into randomly sampled contexts.This design emphasizes simulated humans in user-model interactions and fine-grained personalization.
Evaluation Framing: PERSONAMEM evaluates question-answering performance with questions and answers grounded in specific temporal events.Its personalization evaluation follows an NLP-task formulation rather than automatic judging of generated texts.
Dialogue Personalization: Earlier dialogue benchmarks study personalization within single turns or sessions, while PERSONAMEM uses scalable multi-session user-chatbot histories.Related work includes LAMP, PERSONALLM, IMPLEXCONV, PERSONABENCH, and PERSOBENCH.

6 Conclusion

PERSONAMEM provides a scalable benchmark for testing whether models memorize, track, and apply dynamically changing user profiles. The authors identify persistent challenges in novel scenarios and long contexts while using synthetic data to reduce privacy risks.

Conclusion: PERSONAMEM combines persona-oriented multi-session histories with fine-grained in-situ queries to evaluate dynamic personalization capabilities.The benchmark targets memorizing, tracking, and incorporating changing user profiles into responses.
Conclusion: Comprehensive assessments highlight challenges in delivering personalized conversations, especially in novel scenarios and long contexts.The benchmark covers 15 state-of-the-art models and retrieval-based methods.
Privacy: Synthetic data curation avoids potential privacy risks associated with real user data while enabling personalization research.The setting uses preferences and activities shared in prior user-chatbot interactions without external sharing.

A.2 More advanced retrieval methods

The paper treats retrieval-augmented methods as an initial proof of concept while prioritizing the PERSONAMEM benchmark. It also adapts evaluation to the ambiguity and cost of open-ended personalized generation.

Scope: Retrieval-augmented methods such as RAG and Mem0 are explored as a proof of concept, not the paper’s primary focus.The stated priority is designing and releasing the personalization benchmark.
Benchmark curation: The benchmark removes examples with obvious answer artifacts, including unusually long correct answers or shared keywords with questions.It also excludes queries answerable correctly without conversation context.
Evaluation design: Open-ended personalization permits multiple valid answers and requires costly LLM-as-a-Judge evaluation.The paper therefore uses candidate log-likelihoods as a more reliable benchmarking basis than fully open-ended evaluation.

B Details on Human Evaluation

Human evaluation validates PERSONAMEM’s generated questions and responses using structured judgments from three annotators across four quality dimensions. Ratings exceeded 90% on every dimension, although agreement was lower for the subjective best-response judgment.

Evaluation scope: The evaluation does not measure human performance on the questions because reading the long contexts was considered intractable.Instead, evaluators assess the generated items using the provided questions, answers, conversations, and metadata.
Evaluation dimensions: Evaluators judge appropriateness, relevance, correctness, and whether the designated answer is better than the incorrect responses.The interface exposes the questions, answers, conversations, and metadata grounding each entry.
Annotation procedure: Three authors annotated 90 entries three times each, with majority labels assigned after iterative refinement of the instructions.The entries covered 18 topics and were drawn from a single persona to reduce annotator mental load.
Results: 97.8% of entries were rated appropriate, 95.6% relevant, 97.8% correct, and 90% as having the best response.Gwet’s AC1 values were 0.928, 0.899, 0.877, and 0.560, respectively.
Results: Agreement was very high for appropriateness, relevance, and correctness, but moderate for best response, likely reflecting that dimension’s subjectivity.The reported proportions exceeded 90% across all four dimensions.

C Supplementary Experiment Results

Supplementary experiments extend the benchmark analysis across long-context, retrieval-augmented, and generative settings. The reported patterns remain consistent: retrieval generally helps, while performance varies with the distance to relevant preferences.

Long-context results: Figure 8 reports model performance across seven in-situ query types with 1M-token contexts, showing patterns similar to Figure 3.The accompanying description notes reasonable recall of user facts but difficulty with novel suggestions and applying preferences in new scenarios.
Retrieval-augmented results: RAG improves performance on most question types for GPT-4o and GPT-4o-mini under 128K-token contexts.Figure 9 compares vanilla models with models enhanced by RAG modules.
Generative evaluation: Figure 10 examines performance as sessions elapse after the latest preferences were mentioned, with similar patterns in discriminative and generative evaluations.The figure concerns the 10-session, 32k-token version of PERSONAMEM.

D.2 Different Conversation Topics

The supplementary topic and distance analyses describe a broad dataset spanning varied tasks, session positions, and token distances. A retrieval system substantially reduces query latency relative to Mem0 in the reported GPT-4o-mini experiment.

Reference distance: The dataset distributes queries across session-distance and token-distance ranges, including both nearby and substantially earlier reference information.The supplied breakdown includes session ranges up to 56–60 sessions and token ranges extending beyond 381k tokens.
Retrieval efficiency: RAG completed 589 GPT-4o-mini queries in 6 minutes, averaging 0.61 seconds per query, excluding embedding time.The experiment used a 32k-token context window, and retrieval time was constant with respect to context length.
Retrieval efficiency: Mem0 required 24 hours total, or 150 seconds per query, in the same reported experiment.Its sequential processing of updates, deletions, and additions within the long context accounts for the reported timing contrast.

F Analysis of error patterns

In a manual analysis of 100 GPT-4o failures, lack of personalization and failure to recognize preference updates were the dominant error patterns. Overall, the results indicate that models often favor broadly reasonable responses over options tailored to the user’s current preferences.

The analysis identified five main error types from 100 randomly selected user queries where GPT-4o failed to select the most personalized response.
14% of errors were format errors, while 12% were hallucinations involving preferences never mentioned by the user.
24% of errors involved failure to recognize preference updates, with the model selecting outdated preferences instead of the most recent ones.
48% of GPT-4o errors reflected lack of personalization, where the model chose a generally reasonable option over a more personalized one.
2% of errors fell into the miscellaneous other category.
These results suggest that models struggle to adapt to evolving preferences and tend to prefer broadly reasonable responses over contextually personalized alternatives.