PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks

Bowen Jiang, Yuan Yuan, Zhuoqun Hao, Yuchen Liu, Maohao Shen, Sihao Chen, Gregory Wornell, Chris Callison-Burch, Lyle Ungar, Dan Roth, Qi Guo, Xiangjun Fan, Camillo J. Taylor, Hanchao Yu

arXiv:2608.21381v1cs.CYcs.CL

TL;DR

Personal intelligence remains under-measured because users’ preferences and needs unfold across platforms and time, while systems must also avoid inappropriate personalization. PersonaMem-v3 addresses this gap with a real-world-grounded, psychology-informed benchmark spanning cross-platform personalization, recommendation, proactiveness, and agentic tasks. The strongest evaluated agents solve only slightly more than half the benchmark, with judgment-heavy capabilities remaining difficult.

  • Problem

    Personal intelligence requires connecting fragmented, evolving user signals across platforms while deciding when personalization should be used or withheld.

  • Method

    PersonaMem-v3 builds privacy-preserving, temporally evolving user worlds from million-scale engagement histories and evaluates agents across connected digital surfaces and tasks.

  • Results

    Even the strongest evaluated agents solve only slightly more than half the benchmark, while recommendation reranking, proactive timing, and hidden-persona reasoning remain weak.

  • Takeaways & Limitations

    The benchmark treats personal intelligence as holistic cross-platform judgment rather than single-app recommendation or memory retrieval.

  • Takeaways & Limitations

    Privacy-preserving release represents multimodal data and platform activity through textual annotations and structured metadata rather than native media or internal platform records.

Abstract

from arXiv · show

Personal intelligence is becoming a central frontier for user-facing AI agents. To be helpful in everyday life, agents must understand users across the digital contexts where their preferences, intents, habits, social relationships, and needs unfold over time. Today's systems can personalize within individual apps or tasks, but personal intelligence as a whole remains under-measured: how agents build cross-context user understanding, support steerable recommendation systems, act proactively across platforms, and avoid over-personalization. We introduce PersonaMem-v3, a real-world-grounded benchmark and evaluation harness for omni-platform personal intelligence. PersonaMem-v3 is seeded from more than one million anonymized real-world engagement histories, most of which are implicit signals, and uses them to construct time-indexed user digital worlds across social media, chatbot, calendar, and AI-companion with preference evolvement over time. The benchmark brings personalization, LLM-powered recommendation, proactiveness, agentic tool use, and geo-temporal reasoning into one framework, anchored in psychology, social-linguistics, and user-behavior theories. It evaluates whether AI agents can infer holistic user understanding from cross-platform evidence, personalize responses, rerank recommendations on social media, follow user steering through natural language, and hold back when personalization would be inappropriate, repetitive, outdated, or unnecessary. PersonaMem-v3 points toward LLM-powered personal intelligent agents that work with existing scalable recommendation infrastructure while making personalization more interactive, agentic, and aligned with how real users experience their digital lives.

1 Introduction

PersonaMem-v3 frames personal intelligence as cross-platform, temporally evolving user understanding that must balance helpful personalization with social appropriateness and restraint. It introduces a real-world-grounded benchmark and evaluation harness spanning personalization, recommendation, proactiveness, and agentic tasks.

  • Motivation: PersonaMem-v3 addresses users whose layered, platform-dependent personas and changing preferences emerge mainly through implicit engagement signals.These properties make purely synthetic-persona benchmarks insufficient for reproducing realistic user understanding.
  • Benchmark scope: The benchmark constructs six connected digital surfaces from shared user personas and behavioral histories, enabling cross-platform reasoning across social media, chatbots, AI Studio, and Calendar.Signals learned on one surface can personalize interactions on another, such as using chatbot preferences for Instagram recommendations.
  • Interactive recommendation: User-steerable recommendation lets people direct an LLM-driven reranker in natural language to request more like this, stop a topic, or leave the usual bubble.This makes recommendation interactable rather than purely one-way.
  • Evaluation: PersonaMem-v3 evaluates personalization, over-personalization, proactiveness, recommendation, and agentic tasks within temporally evolving user-AI activity histories.The harness supports tool calling and agentic search rather than treating test queries independently.
  • Findings: Current agents solve only a limited portion of omni-platform personal intelligence, while recommendation reranking and proactive actions remain especially challenging.The benchmark also examines whether memory improves efficiency without increasing inappropriate or repetitive personalization.

2 Benchmark Overview

PersonaMem-v3 builds realistic, privacy-preserving multi-platform user worlds by enriching anonymized engagement histories into evidence-grounded, evolving personas. Its pipeline validates preferences across repeated signals and organizes them into profiles, voices, platform-specific presentations, and hidden motivations.

  • Real-world data foundation: The benchmark starts from GIST-Bench’s privacy-preserving real-world substrate and samples 200 users with more than 1,000,000 engagement events over 30 days.The events include explicit and implicit positive or negative signals; nearly 63.8% are implicit negative skips.
  • Persona construction: PersonaMem-v3 converts noisy cross-platform engagement histories into holistic user profiles through candidate preference extraction, cross-validation, pruning, and persona organization.The final persona includes a basic profile, writing voice, platform-specific personas, and hidden personas.
  • Preference validation: Candidate preferences are retained only when repeated evidence supports them, with explicit corroboration weighted more heavily than implicit corroboration.The cross-reference score assigns 1 point to each explicit corroborating event and 0.5 points to each implicit one.
  • Preference evolution: Timestamped histories capture preferences that strengthen, fade, become specific, or reverse, while long-context reasoning distinguishes isolated contradictions from repeated temporal patterns.The benchmark separates enduring preferences from short-term intents.
  • Persona layers: The persona is layered across a stable profile, shared writing voice, app-specific self-presentations, hidden motivations, and sensitive life context.Per-app voices vary with audience and platform while remaining within the shared voice.
  • Hidden personas: The benchmark infers twelve hidden-persona types using named theoretical anchors, with eleven discovered from cross-row hashtag patterns and sensitive life events treated as synthetic.These frameworks structure evidence standards, confidence requirements, and downstream use.

3 Evaluation Harness

PersonaMem-v3 is an interactive, time-aware evaluation harness that tests personalization across user histories, response generation, recommendation, over-personalization, and proactive agentic tasks. It combines shared personalization checks with task-specific evaluations, including LLM reranking against time-aligned behavioral ground truth and user steering.

  • Harness design: PersonaMem-v3 evaluates systems by interacting with structured user environments over time, rather than answering a static question set.The harness provides tool calling and temporal awareness for real-world-style evaluation.
  • Harness design: Each user’s platform-specific engagement histories are cut at the query timestamp, preventing systems from accessing future behavior.The databases store full histories separately for platforms including Instagram, Facebook, Threads, Chatbot, AI Studio, and Calendar.
  • Evaluation modes: Evaluations compare long-context prompting, evolving textual or vector memories, and frontier agents using native search or MCP tools.The long-context mode concatenates cross-platform history by timestamp; other modes use memory systems or agentic tool use.
  • Universal rubric: A shared personalization rubric rewards relevant, appropriate, socially aware preference use while treating privacy leaks, stale memory, disliked-topic leaks, and creepy framing as hard failures.Tasks apply the checks appropriate to their setting alongside task-specific metrics.
  • Personalized responses: Personalized response tasks test whether outputs adapt to current, time-masked context instead of relying on stale, disliked, or inappropriate personalization.One subcategory explicitly tests tracking preference changes after a preference or short-term intent has changed or expired.
  • Feed recommendations: Feed recommendation is framed as LLM reranking over mature candidate-generation infrastructure, using cross-platform understanding to rank what matters to the user now.The core task uses a 16-candidate pool containing a near-future positive target, neutral fillers, and hard negatives from the user’s behavior.
  • Agentic and behavioral tasks: The benchmark also evaluates in-feed user steering, over-personalization, and proactive action, including when the correct behavior is to hold back or remain silent.Proactive tasks require discovering whether a legitimate reason to act exists and producing only a short, evidence-grounded message when appropriate.

4 Results

PersonaMem-v3 evaluates eight model-mode configurations across personalization, recommendation, over-personalization, agentic tasks, and proactiveness. Performance, token use, latency, and task strengths vary substantially by configuration and evidence demands.

  • 53.7% overall accuracy is achieved by Claude Opus 4.8 with Claude Code High, followed by GPT-5.5 Long Context at 53.4%.
  • Long-context modes read roughly 410k tokens per query, whereas Textual Memory and Mem0 typically use around 3k-5k tokens.Compact memories remain competitive on broad preference tasks but lose exact details needed for refinding, ranking, and evidence-grounded action.
  • Gemini-3.5-Flash Textual Memory uses 15.1 seconds per query, compared with 29.0 seconds for its Long Context setting.This configuration combines memory compression with the lowest latency among the evaluated settings.
  • Agentic coding harnesses use 131.6k-246.7k tokens and 48.0-80.3 seconds per query for search, evidence selection, and answer composition.
  • Compact memories preserve stable preferences well, while long context and agentic search are stronger for exact items, message threads, and evidence-grounded answers.Agentic search improves community voice drafting, DM inbox digest, and vague memory refinding, but does not solve proactive feed reranking, which remains around 25-35% NDCG@5.
  • Preference trajectories include reinforced, emerging, diminishing, and bursty patterns, while proactiveness requires deciding when to act and when to remain silent.

4.3 Preference evolution

PersonaMem-v3 finds that user preferences evolve through reinforced, emerging, diminishing, and bursty trajectories. Models must therefore weigh recent and longstanding signals and decide whether older preferences remain valid.

  • 1,554 sustained persona-by-category trajectories comprise 77% reinforced, 8% emerging, 13% diminishing, and 1% bursty preferences.Reinforced signals support stable personalization; emerging signals require early detection, diminishing signals require decay, and bursty signals are usually episodic.
  • Preference evolution requires temporal awareness rather than topic awareness alone, because models must promote, retain, down-weight, or ignore signals at answer time.
  • GPT-5.5 Textual Memory scores 79.9% on preference-change tracking with a 25.7% outdated-stance rate.GPT-5.5 Long Context scores 73.1%, while Gemini Textual Memory scores 71.1%.
  • Mem0 with RAG scores 56.9% and has a 51.4% outdated-stance rate, showing that retrieved relevance can remain sticky after preference expiry.

4.4 Over-personalization tax

The over-personalization tax measures whether models can use personal context without unnecessary, sensitive, repetitive, stale, or socially awkward personalization. Memory changes this behavior differently across foundation models by altering which evidence becomes salient.

  • GPT-5.5 Long Context scores 75.4% on three over-personalization tasks, while GPT-5.5 Textual Memory drops to 71.0%.
  • Gemini-3.5-Flash Long Context scores 57.0%, whereas its Textual Memory setting rises to 69.9%.
  • On sensitive-event restraint, Gemini-3.5-Flash improves from 38.0% with Long Context to 63.3% with Textual Memory.Privacy-leak violations fall from 36.6% to 6.5% across the same settings.
  • The ablation attributes model differences to how memory changes attention salience and how each foundation model interprets the resulting representation.
  • Adding 40 sensitive posts reduces three-task restraint by 2.5 points for GPT-5.5 and 5.5 points for Gemini-3.5-Flash.Adding ordinary posts changes the score by less than one point.

4.5 Lost-in-the-middle effects and memory forgetting curves

PersonaMem-v3 observes U-shaped lost-in-the-middle effects: accuracy drops when supporting evidence is neither very recent nor very old. Larger textual memories preserve more concepts, but memory self-evolution retains a middle-position problem.

  • All model-mode configurations show accuracy drops in middle bins when evidence distance is measured by days or intervening tokens.The farthest bins may contain repeatedly reinforced preferences, so the temporal-distance analysis is not a perfectly controlled position test.
  • Concept-level survival rises from 63.2% to 83.1% to 90.8% as the GPT-5.5 Textual Memory cap increases from 2048 to 4096 to 8192 tokens.The corresponding final memories contain 35, 66, and 102 bullet lines.
  • Larger memory better preserves content over time, but textual memory still faces lost-in-the-middle problems during self-evolution.

4.6 Data diversity

PersonaMem-v3 covers diverse personalization settings and evidence types, while testing how memory capacity affects retention and downstream accuracy.

  • Task and data coverage: The benchmark spans personalization, recommendation, over-personalization, agentic, and proactive agentic tasks across varied user evidence and social contexts.Its histories include explicit and implicit signals, multiple content formats, hidden personas, sensitive life events, demographic variation, and different companion roles.
  • Memory capacity: Larger Textual Memory caps preserve more semantically equivalent content, but exact original wording remains fragile.The comparison uses 2048, 4096, and 8192 token caps.
  • Memory capacity: The memory-capacity comparison directly tests whether larger caps close the middle-position dip in downstream task accuracy.

4.7 Error analysis

Error analysis shows that systems often succeed through relevant selection, concrete taste use, and stylistic matching, but fail mainly on evidence retrieval, ranking, timing, and privacy judgment.

  • Success patterns: Correct answers commonly select relevant items, match user style, or use concrete tastes.The success patterns are relatively similar across configurations.
  • Codex performance: GPT-5.5 improves with Codex on community voice drafting from 16.9% to 54.2%, DM inbox digest from 27.5% to 65.2%, and vague memory refinding from 48.8% to 55.6%.These tasks require retrieving multiple past conversational turns, direct messages, or user-authored posts.
  • Codex failures: 46% of Codex failures involve missed evidence or wrong prioritization among found candidates, followed by timing mistakes at 23% and privacy over-application at 15%.
  • Codex failures: 32.3% of Codex agentic-task rows fall below the correctness cutoff, with proactive daily catch-up and trend alert especially weak.Direct send, repost, and reply actions are much stronger.
  • Claude Code failures: Opus-4.8 and Sonnet-4.6 attribute 49% and 51% of failures, respectively, to missed evidence or wrong candidate ordering.Both models also drift off target in 6-7% of failures.
  • Broader implications: Codex High reaches 35.3% on proactive feed ranking, the highest among evaluated settings.The broader error pattern indicates that calibration, temporal awareness, social intelligence, and ranking judgment remain necessary beyond context or tool calls.

4.8 Memory contents

PersonaMem-v3 compares consolidated textual memory with vector retrieval, highlighting differences in inspectability, compression, targeting, and failure modes.

  • Textual Memory: Textual Memory is a consolidated, inspectable profile organized around identity, interests and preferences, people and places, and active context.It is presented as a compact, user-editable alternative to vector retrieval.
  • Comparison: The two memory modes differ in the kinds of information they store and expose to the model.Textual Memory presents a consolidated profile, whereas RAG presents targeted retrieved facts.
  • Textual Memory: Textual Memory compresses thousands of events into a human-readable document that users can inspect, correct, and edit.
  • Vector retrieval: Mem0 with RAG stores short dated facts, embeds them in a vector space, and retrieves a few nearest facts for each query.Its fact store can be visualized as t-SNE clusters of topics and skipped content.

4.9 Automated data-quality verification

PersonaMem-v3 applies repeated evidence checks, rubric-based verification, regeneration, and human inspection to reduce artifacts in released benchmark rows.

  • Evidence grounding: Persona claims and preferences must be supported by repeated evidence, confidence thresholds, and psychological anchors rather than a single event.
  • Verifier criteria: The verifier checks that queries require user context without leaking the relevant preference, and that example responses personalize naturally and appropriately.
  • Verifier criteria: The verifier also checks that inferior responses remain natural and differ on the specific capability being tested, such as outdated signals, privacy leaks, or wrong ranking.
  • Quality-control loop: Failed examples are regenerated and re-checked, or removed if they continue to fail; human inspection iterates over a hundred times.
  • Audit results: 96.6% of released rows pass the average data-quality audit, while GPT-5.5 and Opus-4.8 agree on pass-fail judgments 89.1% of the time.

5 Related Work

PersonaMem-v3 extends prior work from isolated memory, personalization, recommendation, and agent benchmarks toward a unified omni-platform evaluation. It combines million-scale real-world grounding with cross-platform user understanding, steerable reranking, and agentic tasks.

  • Benchmark progression: PersonaMem-v3 progresses from synthetic chatbot histories to real-world-grounded, omni-platform user histories and agents.Table 9 summarizes this within-series progression.
  • Scope: Unlike most benchmarks that isolate one capability, PersonaMem-v3 jointly evaluates memory, personalization, recommendation, proactiveness, temporal preference tracking, agentic tasks, and privacy-sensitive restraint.
  • Memory and personalization: Its evidence is fragmented across social media, chatbot sessions, calendar events, and geo-temporal context, extending memory evaluation beyond storing, retrieving, updating, or forgetting information.
  • Personalization: PersonaMem-v3 uses million-scale anonymized engagement histories and evaluates personalization jointly with recommendation, over-personalization, proactiveness, and personalized agentic tasks.
  • Recommendation: The benchmark treats an LLM as a personalized reranker and assistant over candidate pools rather than assuming it should own the full recommender pipeline.
  • Personal agents: PersonaMem-v3 contributes an open-source data and model specification for omni-platform personal agents with a comprehensive evaluation harness.

6 Conclusion

PersonaMem-v3 frames omni-platform personal intelligence as an under-measured frontier and evaluates it through realistic, temporally evolving user worlds. Its results show that retrieval access helps evidence-grounded tasks, but judgment remains the central bottleneck.

  • Overall findings: PersonaMem-v3 evaluates holistic user understanding across social media, chatbot, companion-chat, calendar, and geo-locations over time rather than treating personalization as single-app recommendation or memory retrieval.
  • Evaluation modes: Long context, textual memory, vector retrieval, and agentic search offer different tradeoffs in evidence access, efficiency, detail preservation, and privacy constraints.
  • Evaluation modes: Agentic search is especially useful for refinding content, summarizing message threads, drafting in a community voice, and acting across platforms.
  • Remaining gaps: Tool access improves evidence access but leaves agents weak on recommendation reranking, proactive timing, mistake-prevention alerts, and hidden-persona reasoning.
  • Memory behavior: Memory updates can overwrite, compress, or under-emphasize middle-segment information, extending lost-in-the-middle failures beyond raw long-context reasoning.
  • Memory behavior: Memory modules reshape attention, evidence prioritization, and privacy behavior differently across foundation models, including opposite over-personalization effects.

7 Limitations

PersonaMem-v3’s open benchmark is shaped by privacy-preserving representation choices and bounded theoretical and platform coverage. These constraints preserve reproducibility while limiting multimodal fidelity, psychological interpretation, and ecosystem breadth.

  • Data representation: Privacy-preserving release replaces native images, videos, and platform records with textual annotations and structured metadata.The authors state that original multimodal data and raw posts cannot be released.
  • Theoretical scope: The benchmark does not validate clinical labels or prove any psychological theory as a complete account of user behavior.The frameworks are used as generation and evaluation scaffolds rather than diagnostic claims.
  • Platform scope: PersonaMem-v3 focuses on Meta-family platforms and chatbot interactions rather than covering every digital footprint, private platform MCP interface, or production database schema.
Loading 2608.21381v1…