Source-linked AI summary
OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents
Yulin Hu, Zimo Long, Jiahe Guo, Xingyu Sui, Xing Fu, Weixiang Zhao, Yanyan Zhao, Bing Qin
TL;DR
Memory-augmented dialogue systems can overuse personal information, but existing benchmarks largely assess recall and application rather than appropriate use. The paper formalizes over-personalization, introduces OP-Bench, analyzes memory hijacking, and proposes Self-ReCheck; results show memory mechanisms worsen over-personalization, while the filter mitigates it. OP-Bench is limited by synthetic interactions and single-turn textual evaluation.
Problem
Existing benchmarks emphasize whether agents recall and apply user information, while overlooking whether personalization is used appropriately, including responses that become forced, intrusive, or socially inappropriate.
Method
The paper formalizes Irrelevance, Sycophancy, and Repetition, introduces OP-Bench, analyzes attention and attribution, and proposes Self-ReCheck, a model-agnostic memory filter.
Results
Memory mechanisms trigger memory hijacking: models attend to memories more than queries, causing OP-Bench performance drops of 26.2%–61.1% versus BASE; Self-ReCheck reduces over-personalization by 29% while preserving personalization performance.
Takeaways & Limitations
Appropriate personalization requires selective restraint so agents prioritize the user’s immediate intent rather than merely recalling more information.
Takeaways & Limitations
OP-Bench uses synthetic prompts and LLM-generated interactions and evaluates only single-turn textual queries, excluding multi-turn and multimodal dynamics.
Abstract
from arXiv · showhide
Memory-augmented conversational agents enable personalized interactions using long-term user memory and have gained substantial traction. However, existing benchmarks primarily focus on whether agents can recall and apply user information, while overlooking whether such personalization is used appropriately. In fact, agents may overuse personal information, producing responses that feel forced, intrusive, or socially inappropriate to users. We refer to this issue as \emph{over-personalization}. In this work, we formalize over-personalization into three types: Irrelevance, Repetition, and Sycophancy, and introduce \textbf{OP-Bench} a benchmark of 1,700 verified instances constructed from long-horizon dialogue histories. Using \textbf{OP-Bench}, we evaluate multiple large language models and memory-augmentation methods, and find that over-personalization is widespread when memory is introduced. Further analysis reveals that agents tend to retrieve and over-attend to user memories even when unnecessary. To address this issue, we propose \textbf{Self-ReCheck}, a lightweight, model-agnostic memory filtering mechanism that mitigates over-personalization while preserving personalization performance. Our work takes an initial step toward more controllable and appropriate personalization in memory-augmented dialogue systems.
1 Introduction
The paper identifies over-personalization as a neglected failure mode in memory-augmented dialogue, formalizes three categories, and introduces OP-Bench to measure it. It finds widespread over-personalization from memory use and proposes Self-ReCheck to reduce it while preserving personalization.
- Memory-augmented agents may apply personal details when contextually unnecessary, producing forced, intrusive, or socially inappropriate responses.
- Over-personalization comprises Irrelevance, Sycophancy, and Repetition: off-topic responses, excessive deference, and near-identical answers across distinct queries.
- 1,700 questions across 20 users form OP-Bench, which evaluates three categories and six subcategories across 36 model–memory configurations.
- Memory-augmentation mechanisms often retrieve irrelevant memories that receive disproportionately high attention, overshadowing the query and reasoning.
- Self-ReCheck is a lightweight, plug-and-play memory filter that selects memories by query relevance and reduces over-personalization while largely preserving personalization.
2 Related Works
The related work positions OP-Bench within research on persistent memory agents and personalized-agent benchmarks. Existing benchmarks emphasize remembering preferences, evolving profiles, and long-term consistency, motivating evaluation of over-personalization and generalization.
- Memory-augmented agents maintain structured memories of user preferences and interactions for long-term adaptation, extending beyond transient conversational context.
- Recent work on personalized agents has advanced from retrieval of user-related information toward persistent and adaptive memory systems.
- Despite progress, over-personalization remains a challenge, highlighting the need for benchmarks that balance personalization and generalization.
- Existing personalized-agent benchmarks test preference following, evolving user profiles, and long-term memory consistency across extended interactions.
3 OP-Bench
OP-Bench is a human-verified benchmark for evaluating three forms of over-personalization in long-horizon personalized dialogue: Irrelevance, Sycophancy, and Repetition. It constructs targeted tasks from structured user profiles and measures whether systems avoid unnecessary personalization, resist over-alignment, and produce diverse responses.
- Benchmark construction: OP-Bench uses a three-stage pipeline to extract user profiles and topics, construct tasks for three over-personalization modes, and verify data quality with human review.The benchmark preprocesses multi-session LoCoMo dialogues and adapts them to a user–assistant setting through structured profile extraction.
- Irrelevance: Irrelevance tests whether models inject user memories when queries do not require personalization, including fully irrelevant and deceptively relevant baiting queries.The task separates long-term user attributes from short-term conversational goals and uses automatic filtering plus light human verification during construction.
- Sycophancy: Sycophancy measures excessive deference that prioritizes user alignment over objectivity or factual accuracy across factual, memory, and value-level variants.Its evaluation assigns scores from 0 for fully sycophantic behavior to 1 for complete resistance.
- Repetition: Repetition evaluates whether responses remain diverse across semantically distinct but topically related queries rather than repeatedly reusing the same memories or preferences.The benchmark generates multiple subtopics and open-ended questions, then uses mean pairwise embedding cosine similarity to measure response repetition; higher S indicates greater variety and less repetition.
- Quality control: Each constructed instance is reviewed twice independently and accepted only after reviewer consensus or senior adjudication.This human review pipeline checks both label accuracy and overall sample quality.
- Benchmark construction: 1,700 instances across 20 users cover Repetition, Irrelevance, and Sycophancy, with Repetition comprising 51.9%, Irrelevance 24.6%, and Sycophancy 23.5%.All instances are grounded in structured user profiles derived from long-term dialogues.
4 Experiments
Experiments show that memory augmentation consistently increases over-personalization through aggressive retrieval and disproportionate attention to memories. Self-ReCheck reduces these failures while largely preserving or improving personalization.
- RQ1: Does OP exist?: Memory-augmented methods show 26.2%–61.1% relative performance drops on OP-Bench versus BASE across models.More sophisticated memory systems exhibit more severe over-personalization than simpler mechanisms such as RAG.
- RQ2: Why does OP occur?: Models assign substantially more attention to retrieved memories than user queries, with the average memory-to-query ratio exceeding 2×.This pattern appears across benchmark tasks, indicating that memory can function as a privileged signal rather than a conditional cue.
- RQ2: Why does OP occur?: Memory systems retrieve substantial numbers of low-similarity memories, including fully irrelevant entries, instead of abstaining.Retrieved content can therefore require the dialogue model to handle largely irrelevant user-specific information.
- RQ2: Why does OP occur?: Memory-augmented outputs use more agreement-oriented and deferential language, while memory-conditioned responses become more concentrated across related queries.These findings connect memory use with both sycophantic tone and reduced response diversity.
- RQ3: How can OP be mitigated?: Self-ReCheck filters retrieved memories by query relevance before generation, leaving retrieval and generation modules unchanged.The module evaluates each memory independently and passes only selected items to the chat model.
- RQ3: How can OP be mitigated?: Self-ReCheck improves over-personalization by 29% on average while improving personalization by 3% on average across five memory methods.Attention analysis shows reduced memory-token attention and a more balanced memory-to-query ratio.
5 Conclusion
OP-Bench quantifies over-personalization across irrelevance, sycophancy, and repetition, revealing memory hijacking in memory-augmented agents. Self-ReCheck reduces over-personalization while maintaining strong personalization performance.
- Conclusion: OP-Bench evaluates over-personalization across Irrelevance, Sycophancy, and Repetition in 36 configurations.The benchmark exposes memory-related failures beyond conventional assessments of recall and personalized application.
- Conclusion: Models attend to retrieved memories over 2x more than queries, while OP-Bench scores drop by 26.2%–61.1% under memory augmentation.The conclusion attributes these effects to aggressive and irrelevant retrieval.
- Conclusion: Self-ReCheck reduces over-personalization by 29% while maintaining strong personalization performance.The proposed filtering module supports more selective use of memory in personalized dialogue.
Limitations
OP-Bench has limitations in realism, interaction format, and language-cultural coverage. Its synthetic, single-turn English design may not capture the diversity of real-world conversational behavior.
- Synthetic scenarios may not fully capture the diversity and unpredictability of real-world user behavior.
- OP-Bench evaluates only single-turn textual queries, excluding multi-turn dialogue dynamics and multimodal inputs.
- The benchmark focuses on English interactions and may not generalize to multilingual or cross-cultural settings with different personalization norms.
Ethical Considerations
The benchmark uses comparative human review to identify inappropriate personalization while relying on synthetic, template-driven query generation. Its evaluation process includes anonymization, adjudication, quality controls, and model selection for efficient data construction.
- Human evaluation: Annotators retained an instance only when the memory-based response was judged more inappropriate than the base response.This comparative, user-centric procedure targets queries that trigger undesirable personalized behavior.
- Data curation: 10,000 generated candidates were filtered to 4,289 queries before human review.Filtering removed template redundancy, near-duplicates, and queries below a minimum length.
- Data curation: 1,700 queries remained after independent evaluation and adjudication.The independent stage left 3,075 queries, and disagreement cases were resolved by a senior annotator.
- Annotation process: The annotation interface supports double-blind response comparison, comments, adjudication flags, and memory-relevance highlights.Twelve trained annotators collectively contributed around 800 person-hours, with spot-checks and adjudication for quality control.
A.2.4 Evaluation of Synthetic Data from Different Models
The study compares models for generating synthetic over-personalization queries and evaluates automated judges against human assessments. Reasoning-oriented generators achieve better acceptance, while GPT-4o-mini offers the most favorable judge quality-cost balance.
- Synthetic query generation: Each generator produced 180 queries under a fixed sampling budget, with 30 queries per task category.All generated queries were evaluated using the same human annotation protocol.
- Synthetic query generation: Acceptance rate measures the proportion of queries judged by humans to be valid, high-quality over-personalization instances.Table 4 reports this rate alongside the USD cost of generating 180 queries.
- Synthetic query generation: Reasoning-oriented models achieve substantially higher acceptance rates than chat-oriented models, but require more tokens and longer generation time.DeepSeek-v3.2-thinking was selected as the primary generator after balancing quality and cost.
- Evaluation metrics: Irrelevance and Sycophancy use continuous [0, 1] scores where higher values indicate less over-personalization, while Repetition uses embedding cosine similarity.The metrics combine LLM-based judgments with an embedding-based repetition measure.
- Judge selection: GPT-4o-mini provides the most favorable balance between agreement with human judgments and inference cost.Judge quality is assessed with Spearman’s ρ and Pearson’s r over 200 responses.
B.1 Baseline Methods
The appendix describes baseline memory systems, Self-ReCheck’s relevance-based filtering, and supplementary OP-Bench analyses. Across additional results, memory augmentation generally increases over-personalization, with more sophisticated systems showing stronger effects.
- Baseline systems: The baseline suite includes a model without memory and five approaches using retrieval, memory, or agent-based mechanisms.The listed systems are BASE, RAG, LDAgent, Mem0, MemU, and MemOS.
- Baseline systems: MemU organizes persistent memories through an agent-centric file abstraction for long-term interactive scenarios.It is designed for settings such as AI companions and role-playing agents.
- Self-ReCheck: Self-ReCheck prompts an LLM to assess each retrieved memory’s relevance and make a binary filtering decision.The method is presented as a structured reasoning function illustrated in Figure 9.
- Evaluation setup: OP-Bench prompts prioritize the current query while allowing conditioning on retrieved memory without explicitly encouraging excessive memory reliance.For baseline evaluation, official prompt formats and pipelines are used where available.
- Supplementary findings: Additional OP-Bench results show that memory-augmented agents generally exhibit severe over-personalization, with stronger effects in more sophisticated memory systems.The qualitative analysis also finds more deferential, agreement-oriented, and hedging language when memory is enabled.
C.3 Repetition Analysis
The analysis uses PCA to compare response embeddings across users with and without memory. Memory-enabled responses form more compact clusters, indicating reduced diversity and a response-collapse effect.
- PCA visualization: PCA projects response embeddings into two dimensions to visualize response distributions across users.Each point represents a generated answer, using embedding-3-small.
- Response diversity: Memory-enabled responses form more compact clusters, whereas responses without memory are more dispersed.This pattern holds consistently across the analyzed users.
- Interpretation: Retrieved memories act as dominant anchors that constrain generation, producing a response-collapse effect.
C.4 Attention Allocation Analysis
The attention analysis compares how models allocate attention between user queries and retrieved memories. Memory retrieval shifts attention toward memory tokens, while Self-ReCheck balances attention by reducing memory reliance and increasing focus on the query.
- Analysis setup: The analysis examines attention allocation between user queries and retrieved memories across memory configurations.The experiments use Qwen3-8B.
- Memory-augmented model: When memory retrieval is enabled, attention shifts significantly toward memory tokens and often dominates the attention distribution.This behavior aligns with the over-personalization tendencies observed in evaluation.
- Self-ReCheck: Self-ReCheck produces a more balanced attention allocation by reducing attention to retrieved memory and reallocating it toward the user query.The comparison is shown in the Self-ReCheck and memory-only visualizations.
D.2 Case Study with Self-ReCheck Enabled
The case study compares GPT-4o-mini with MemU across OP-Bench task categories before and after Self-ReCheck. The enabled mechanism is presented as mitigating over-personalization across irrelevance, sycophancy, and repetition cases.
- Case-study scope: The case study evaluates GPT-4o-mini with MemU across Irrelevance, Sycophancy, and Repetition subcategories.Each example includes the task type, user query, retrieved memory, response, and over-personalization score.
- Comparison: Self-ReCheck is compared with the same GPT-4o-mini and MemU configuration without the mechanism.The comparison covers Fully Irrelevant, Baiting, Fact-Level, Memory-Level, Value-Level, and Repetition cases.
- Self-ReCheck examples: With Self-ReCheck, example scores are 1.0 for Fully Irrelevant, 0.8 for Baiting, 0.4 for Fact-Level, 0.7 for Memory-Level, and 0.5 for Value-Level.These scores come from the corresponding enabled-case figures.