PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaVLM: Long-Term Personalized Multimodal LLMs

Chang Nie, Chaoyou Fu, Yifan Zhang, Haihua Yang, Caifeng Shan

arXiv:2604.13074v1cs.CLcs.CV

TL;DR

Long-term personalization in multimodal language models remains limited because existing approaches struggle to track users’ evolving preferences and personality. PersonaVLM combines proactive memory, multi-turn retrieval-based reasoning, and response alignment, improving baseline performance by 22.4% on Persona-MME and 9.8% on PERSONAMEM under a 128k context.

  • Problem

    Existing multimodal language models and personalization methods struggle to dynamically align responses with users’ evolving preferences, personality, and multimodal information over time.

  • Method

    PersonaVLM uses a personality profile, multi-type memory database, multi-step memory retrieval, and iterative user-trait updates for long-term personalized interaction.

  • Results

    PersonaVLM improves baseline performance by 22.4% on Persona-MME and 9.8% on PERSONAMEM under a 128k context.

  • Takeaways & Limitations

    PersonaVLM provides a framework and benchmark for evaluating and supporting long-term, multi-faceted multimodal personalization.

  • Takeaways & Limitations

    Existing agentic personalization frameworks remain constrained by text-only inputs and reliance on proprietary models, limiting multimodal applicability, open research, and privacy.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) serve as daily assistants for millions. However, their ability to generate responses aligned with individual preferences remains limited. Prior approaches enable only static, single-turn personalization through input augmentation or output alignment, and thus fail to capture users' evolving preferences and personality over time (see Fig.1). In this paper, we introduce PersonaVLM, an innovative personalized multimodal agent framework designed for long-term personalization. It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities: (a) Remembering: It proactively extracts and summarizes chronological multimodal memories from interactions, consolidating them into a personalized database. (b) Reasoning: It conducts multi-turn reasoning by retrieving and integrating relevant memories from the database. (c) Response Alignment: It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics. For evaluation, we establish Persona-MME, a comprehensive benchmark comprising over 2,000 curated interaction cases, designed to assess long-term MLLM personalization across seven key aspects and 14 fine-grained tasks. Extensive experiments validate our method's effectiveness, improving the baseline by 22.4% (Persona-MME) and 9.8% (PERSONAMEM) under a 128k context, while outperforming GPT-4o by 5.2% and 2.0%, respectively. Project page: https://PersonaVLM.github.io.

1. Introduction

PersonaVLM addresses the mismatch between increasingly personalized user expectations and existing static personalization strategies that cannot track evolving preferences, personality, and multimodal information. It introduces a long-term personalization framework centered on proactive memory, multi-step reasoning, and response alignment, supported by a synthetic multimodal dataset and a new benchmark.

  • Motivation: Existing MLLMs struggle to produce responses tailored to users because models use fixed-window, one-size-fits-all optimization while preferences and personality continuously evolve.The introduction identifies model-side limitations and user-side dynamism as the two primary causes of personalization failures.
  • Limitations of Prior Work: Static input augmentation recognizes user-specific concepts but cannot manage or update memories, causing outdated recommendations when preferences shift.The passage illustrates this limitation with a shift from Sprite to Coca-Cola.
  • PersonaVLM: PersonaVLM integrates proactive Remembering, multi-step Reasoning, and Response Alignment to support long-term personalized interaction.Its design includes retrieval-based reasoning and alignment with users’ unique, evolving characteristics.
  • PersonaVLM: PersonaVLM’s memory architecture combines a user personality profile with core, semantic, procedural, and episodic memory types.These memory types represent foundational attributes, facts, habits, and events, respectively.
  • Resources: The authors synthesize over 30k personalized multimodal interactions across 500 unique personas to enable local training without data privacy concerns.The dataset is designed to address the scarcity of suitable training data for personalized multimodal interaction.
  • Resources: The work establishes Persona-MME, a benchmark intended to address the static and text-centric limitations of existing personalization benchmarks.The introduction presents this benchmark as part of the paper’s contributions.

2. Related Work

Related work frames MLLM personalization around adaptation, augmentation, and alignment, while identifying scalability, multimodality, privacy, and preference-evolution limitations. PersonaVLM addresses siloed prior capabilities with a unified framework for dynamic, long-term interaction.

  • Personalization paradigms: Existing MLLM personalization methods fall into three streams: adaptation-based, augmentation-based, and alignment-based approaches.General-purpose MLLMs require responses tailored to individual user knowledge and preferences rather than a one-size-fits-all paradigm.
  • Adaptation-based Personalization: Adaptation-based methods encode user-specific knowledge in trainable parameters, including PEFT, learnable embeddings, and soft prompts for personalized multimodal concepts.These methods can distinguish a user’s pet dog from a generic dog, but require fine-tuning for each new user concept and cannot capture evolving preferences.
  • Augmentation-based Personalization: Augmentation-based methods use external databases to retain and retrieve user-specific memories, extending personalization beyond fixed context windows in lifelong dialogues.Related multimodal approaches incorporate visual processing, while text-only and proprietary-model agentic frameworks remain constrained by limited multimodal applicability, open-research barriers, and privacy concerns.
  • Alignment-based Personalization: Alignment-based personalization seeks to replace universal behavioral standards with responses that accommodate diverse user preferences and communication styles.Per-user training creates scalability challenges, and static alignment probes can become outdated as personality evolves during long-term interactions.
  • PersonaVLM: PersonaVLM departs from siloed approaches such as static memory or fixed alignment by introducing a unified agent framework for dynamic, long-term interaction.Its framework uses personalized memory across response and update stages to retrieve memories, generate personality-aligned responses, extract key memories, and update the evolving personality profile.

3. Methods

PersonaVLM combines a personalized memory architecture with collaborative Response and Update stages for long-term personalization. It retrieves and reasons over multimodal interaction history, evolves a Big Five personality profile, updates memories, and is trained through supervised fine-tuning followed by reinforcement learning.

  • Personalized Memory Architecture: The personalized memory architecture stores a quantitative Big Five personality profile and a timeline-based, multi-type memory database supporting CRUD operations.The memory database includes core memory and other specialized memory types for user-related knowledge.
  • Response Stage: The Response Stage generates personalized responses through multi-step reasoning and timeline-based retrieval over the current multimodal query, dialogue context, and prior memory state.The query includes text, an optional image, and a timestamp; recent conversation history within 60 minutes is treated as short-term memory.
  • Update Stage: The Update Stage automatically evolves personality and proactively updates memories during idle periods after response generation.Semantic memory is updated after each turn, while core and procedural memory are updated at session end through dialogue analysis and CRUD operations.
  • Update Stage: The Personality Evolving Mechanism updates a Big Five vector with an exponential moving average and a cosine-decay smoothing schedule, then converts it into a textual profile.The schedule enables rapid adaptation early and greater stability over time.
  • Training: Training uses two stages: supervised fine-tuning on 78k synthetic samples, followed by GRPO reinforcement learning to improve multi-turn reasoning.SFT teaches memory management and reasoning skills, while RL uses structured outputs and rewards accuracy, consistency, and format adherence.

4. Dataset and Persona-MME Construction

This section presents a multimodal interaction-data synthesis pipeline and Persona-MME, a benchmark designed to evaluate long-term, dynamic personalization comprehensively. Persona-MME spans seven dimensions, 14 fine-grained tasks, over 2,000 cases, and two context-length configurations.

  • Dataset Synthesis Pipeline: The authors address limited high-quality training data by constructing a large-scale multimodal interaction dataset through a dedicated synthesis pipeline.The pipeline samples base personas from PersonaHub, enriches them with randomly assigned personality traits, and forms an initial Core Memory.
  • Persona-MME: Evaluating Long-Term Personalization of MLLMs: Persona-MME fills a benchmark gap by evaluating dynamic multimodal personalization beyond existing benchmarks’ siloed focus on profile tracking, static alignment, or concept understanding.Existing benchmarks include PERSONAMEM, ALIGNX-test, and Yo’LLaVA, but none holistically cover the critical dimensions of dynamic personalization.
  • Persona-MME: Evaluating Long-Term Personalization of MLLMs: Over 2,000 in-situ cases derived from 200 diverse personas organize Persona-MME across seven core dimensions: Memory, Intent, Preference, Behavior, Relationship, Growth, and Alignment.These dimensions encompass 14 fine-grained tasks detailed in the appendix.
  • Persona-MME: Evaluating Long-Term Personalization of MLLMs: Persona-MME provides 32k- and 128k-context configurations for shorter and longer dialogues, respectively, with each configuration containing cases from 100 distinct personas.Each test case includes a multiple-choice question on personalized memory and understanding, plus an optional personality-alignment test.

5. Experiments

PersonaVLM is evaluated through quantitative and qualitative experiments targeting personalized understanding, evolving personality alignment, and personalized open-ended generation. The evaluation uses Persona-MME and PERSONAMEM across long-context settings, alongside qualitative assessment of factual accuracy and personality alignment.

  • Research questions: The experiments address three questions concerning personalized understanding and memory recall, evolving personality alignment, and personalized open-ended generation.These are identified as RQ1, RQ2, and RQ3.
  • Personalized understanding: Persona-MME and PERSONAMEM evaluate personalized understanding across 32k- and 128k-token contexts.PERSONAMEM includes seven task types for tracking dynamic user preferences over the long term; results are reported in Table 1 and Fig. 4.
  • Open-ended generation: Qualitative open-ended-generation comparisons assess both factual accuracy and personality alignment using Gemini-2.5-Pro.The comparison is presented in Figure 5.

Appendix Fig. 10 for more comparisons with leading models. · 6. CONCLUSION

PersonaVLM delivers stronger personalized alignment and open-ended generation than competing models across Persona-MME, P-SOUPS, PERSONAMEM, and GPT-4o comparisons. The paper concludes that its Remembering, Reasoning, and Response Alignment capabilities enable long-term, dynamic personalization for multimodal language models.

  • Appendix Fig. 10 for more comparisons with leading models.: PersonaVLM exceeds InternVL3-8B and LLaVA-OneVision-1.5-8B by 8.62% and 14.39% on Persona-MME in the 128k setting, respectively.It also outperforms the larger InternVL3-38B by 3.87% on Persona-MME (128k).
  • Appendix Fig. 10 for more comparisons with leading models.: PersonaVLM improves over GPT-4o by 17.3% and 2.0% on the 32k and 128k configurations of PERSONAMEM, respectively.Although it lags GPT-4o in memory recall, it outperforms GPT-4o by over 10% in Growth Modeling and Behavioral Awareness.
  • 5.2. Personalized Alignment Evaluation: Persona-MME’s Alignment task contains 812 test cases, while P-SOUPS contains 1,800 cases and evaluates Expertise, Informativeness, and Style.Alignment assesses whether a response matches personality inferred from conversational context; P-SOUPS evaluates alignment with a given user profile.
  • 5.3. Qualitative Evaluation: Qualitative cases demonstrate PersonaVLM’s accurate visual recall, contextual memory integration, and long-term personality alignment compared with baseline models and GPT-4o.Other models show memory hallucinations or tonally misaligned responses that ignore user-specific memories.
  • 5.2. Personalized Alignment Evaluation: PersonaVLM consistently outperforms existing models by 9.16% on Persona-MME and 2.46% on P-SOUPS, representing a >12% gain over the baseline.The alignment evaluation compares open-source models and baseline variants using Self-Critic and few-shot prompting.
  • 5.3. Qualitative Evaluation: In open-ended generation, PersonaVLM secures a 79% win rate versus a 16% loss rate against GPT-4o.The automated evaluation samples 200 Persona-MME questions and uses Gemini-2.5-Pro as judge for Accuracy and Personality Alignment.
  • 6. CONCLUSION: The paper introduces PersonaVLM as an agent framework for long-term, dynamic personalization through Remembering, Reasoning, and Response Alignment.It also proposes Persona-MME as a comprehensive benchmark for personalized multimodal understanding.

A. Details of the PersonaVLM Memory Architecture … B. Implementation Details of PersonaVLM

PersonaVLM organizes long-term personalization through a personality profile and multi-type memory database, with multimodal retrieval and distinct storage policies supporting response generation. Its implementation dynamically updates personality, stores structured memories, and retrieves relevant textual or visual information when needed.

  • A. Details of the PersonaVLM Memory Architecture: PersonaVLM’s memory architecture comprises a User Personality Profile and a Multi-Type Memory Database for storing, updating, and retrieving personalized information.The profile and database are the architecture’s two principal components.
  • A.1. Memory Storage: The personality profile represents Big Five traits as a five-dimensional vector with values between 1 and 5, updated after each interaction using an Exponential Moving Average.The inferred turn-level vector uses integer scores from 1 to 5, while the persistent profile is updated with the EMA rule.
  • A.1. Memory Storage: Core, semantic, episodic, and procedural memories store complementary user information, including foundational attributes, timeless knowledge, topic-based conversation records, goals, and habits.Core memory is included every turn; episodic memory retains summaries, keywords, turn indices, and original dialogue data, while procedural memory retains the latest key-value version.
  • A.2. Memory Retrieval: Memory retrieval begins with a tagged query specifying keywords and a start–end time period when external knowledge is needed for a request.The timestamps use the “YYYY −MM −DDHH : MM” format.
  • A.2. Memory Retrieval: Textual memories are retrieved through parallel dense-vector similarity search, using all-MiniLM-L6-v2 and category-specific top-k selection.The stated k values are 2 for procedural, 4 for semantic, and 2 for episodic memories.
  • A.2. Memory Retrieval: Visual retrieval activates for image inputs, detecting salient objects with Grounding DINO and matching their CLIP embeddings against visual concepts in semantic memory.This creates a unified retrieval mechanism across visual and text-based semantic search.
  • A.2. Memory Retrieval: The operational pipeline retrieves memories through iterative model actions, then infers turn-specific personality, updates semantic memory, and outputs the response with updated state.The pipeline maintains conversational context, memory state, and the personality profile across processing steps.
  • A.3. Memory Management: PersonaVLM retains complete raw interaction history while treating semantic and episodic memories as additive structured records that preserve an immutable historical record.The supplied passage contrasts raw conversational history with managed structured memory policies.

B.1. Implementation Process · B.2. Training Details

PersonaVLM uses session-triggered memory consolidation for its offline operational pipeline and trains through SFT and RL with staged data, GRPO-based optimization, and retrieval-focused robustness strategies. The implementation runs on eight NVIDIA H800 GPUs and completes in approximately eight hours.

  • B.1. Implementation Process: PersonaVLM’s end-to-end offline operational pipeline is specified in Algorithm 1.
  • B.1. Implementation Process: A new user session begins when the elapsed time since the previous interaction exceeds a predefined threshold, such as 60 minutes.Session initiation triggers consolidation of the previous session’s long-term Core and Procedural memories.
  • B.2. Training Details: The SFT dataset contains 78k samples, including 6k user-related concept samples, with 43.6% reasoning QA pairs and 56.4% memory-related samples.The dataset is constructed through the synthesis pipeline in Fig. 3 (a) and augmented using [11].
  • B.2. Training Details: The training pipeline is based on Qwen-VL8 and ms-swift9, with hyperparameters for SFT and RL reported in Table 3.
  • B.2. Training Details: 8 NVIDIA H800 GPUs support the two-stage training process, which takes approximately 8 hours: 2 hours for SFT and 6 hours for RL.
  • B.2. Training Details: GRPO replaces the critic model with relative evaluation, estimating advantages by comparing multiple sampled multi-turn trajectories within each group.The method normalizes trajectory rewards across the sampled group and applies KL regularization controlled by β.
  • B.2. Training Details: A deduplication filter prevents any memory entry from being retrieved more than once per reasoning trajectory, encouraging diverse query keywords and time periods.
  • B.2. Training Details: During training, dynamic top-k sampling randomizes episodic retrieval from [2, 5] and semantic retrieval from [6], while inference uses fixed values of 2 and 4.This acts as data augmentation for sparse and dense retrieval contexts.

C. Data Curation Details.

PersonaVLM’s synthesized dataset uses persona-based training and test splits with progressively longer multimodal dialogues to evaluate long-term memory. Its curation also specifies retrieval-compatible memory formatting and two-stage quality validation.

  • Data Distribution: 700 unique personas are sampled from PersonaHub, with 500 allocated for training and 200 for testing.Training dialogues span 20–100 turns over up to one month.
  • Data Distribution: Test dialogues span 20–100 turns under a 32k context window and 100–500 turns under a 128k context window, over up to three months.The training–testing discrepancy is designed to evaluate long-term memory capabilities rigorously.
  • Data Usage: Full synthesized dialogues form the retrieval database, while training QA targets use regenerated answers based only on the current query and retrieved memories.Original answers access the complete dialogue history, unlike the training condition.
  • Multimodal Memory Formatting: Visual semantic memories use structured object labels and Grounding DINO-cropped image patches paired with textual descriptions, while abstract preferences are stored textually and episodic memories retain full multimodal turns.The formatting distinguishes concrete visual objects from abstract preferences and preserves episodic memory integrity.
  • Data Validation: Data validation combines automated rule- and model-based filtering with human review to remove integrity violations, unsafe or incoherent content, and remaining erroneous or repetitive dialogues.Automated checks use structured metadata such as timestamps and dialogue turn indices.

D. Persona-MME: Details and Statistics · E. More Experimental Details · E.1. Benchmarks

Persona-MME evaluates long-term multimodal personalization through diverse, manually reviewed scenarios spanning seven aspects and 14 fine-grained tasks. The paper also compares related personalization benchmarks and reports strong gains from PersonaVLM, while detailing additional evaluation settings and ablations.

  • D. Persona-MME: Details and Statistics: Persona-MME covers seven personalization aspects through 14 fine-grained tasks and 2,034 in-situ test cases.A single test scenario may assess multiple capabilities.
  • D. Persona-MME: Details and Statistics: The benchmark uses 200 unique fictional personas and dialogues spanning diverse topics and scenarios.The dialogue topical breadth is visualized as a keyword word cloud.
  • D. Persona-MME: Details and Statistics: Each test case averages 142.9 conversational-history turns, with 15.87% multimodal content, 22.7-word questions, and 3.05-word answers.Additionally, 34.02% of questions require visual context information.
  • D. Persona-MME: Details and Statistics: GPT-5 does not dominate every sub-task, while Qwen3-30B-A3B can outperform the larger InternVL3-38B despite limitations on visual tasks.Smaller multimodal models including Qwen2.5-VL-7B, InternVL3-8B, and OneVision-1.5-8B often perform near a random baseline on personality alignment.
  • D. Persona-MME: Details and Statistics: 22.46%: PersonaVLM improves the baseline, with the largest gains concentrated in Growth and Alignment.The evaluation compares more than ten proprietary and open-source models under the 128k configuration.
  • D. Persona-MME: Details and Statistics: Persona-MME uniquely combines long-term interaction scenarios, multimodal inputs, and holistic evaluation of memory, understanding, and alignment.This distinguishes it from existing personalization benchmarks.
  • E.1. Benchmarks: PERSONAMEM uses synthetic, multi-session, timeline-aware conversations to test remembering, preference tracking, generalization, and recommendation alignment.Its settings contain 589 and 1,362 multiple-choice questions, with performance measured by accuracy.
  • E. More Experimental Details: The paper includes an ablation study varying the number of retrieved episodic topics for Persona-MME.The study is presented in Figure 13.

E.2. Ablation Study … E.5. Prompts Used in Our Framework

The ablations show that memory components, episodic-memory retrieval, and multi-turn reasoning are essential to PersonaVLM’s performance, while PEM captures evolving personality and aligns responses during long-term interactions. Comparative examples and documented prompts further describe the framework’s personalization behavior and implementation.

  • E.2. Ablation Study: Removing any single memory type degrades Persona-MME performance at both 32k and 128k context settings, with episodic memory the most critical component.Removing episodic memory causes drops of 12.41% and 5.19% in the two settings, respectively.
  • E.2. Ablation Study: Persona-MME accuracy initially increases with retrieved episodic-memory topics before saturating, leading to a default of two topics for performance-efficiency balance.The two-topic setting is used in all main experiments.
  • E.2. Ablation Study: Disabling multi-turn retrieval during response generation reduces performance by 2.75% at 32k context and 3.73% at 128k context.The full reinforcement-learning-trained model also gains 4–7% over its SFT-only baseline on Persona-MME and PERSONAMEM.
  • E.3. Personality Evolving Mechanism: PEM enables PersonaVLM to capture users’ evolving personality accurately during long-term interactions and generate responses aligned with inferred traits.Figures and tables visualize personality evolution, demonstrate P-SOUPS ablations, and show adaptive responses across Big Five trait settings.
  • E.4. More Interaction Examples: Comparative open-ended interaction cases show PersonaVLM delivering superior comprehensive personalization compared with the baseline model and GPT-4o.The comparisons focus on long-term interactions.
  • E.5. Prompts Used in Our Framework: The framework documents prompts for multi-turn reasoning, retrieval, PEM personality inference, memory-module updates, and open-generation evaluation.These prompts are presented in Figs. 16–23.

F. Further Discussion

PersonaVLM improves efficiency and protects data through a self-contained architecture, while reasoning introduces a latency trade-off. The framework remains limited by missing audiovisual person tracking, baseline-model capabilities, and disconnected episodic memories.

  • Efficiency and Data Security: 93.7% lower average token consumption and 4.8× speedup are achieved by PersonaVLM without reasoning over the baseline model.The evaluation uses 100 randomly selected Persona-MME samples and measures average tokens per request and end-to-end response time.
  • Efficiency and Data Security: 20.4% lower token consumption but 21.1% longer response time distinguish standard PersonaVLM from its non-reasoning variant and baseline, respectively.The results expose a trade-off between advanced reasoning capabilities and response latency.
  • Efficiency and Data Security: PersonaVLM’s memory and retrieval operations run independently of external commercial model APIs, supporting data security and mitigating privacy concerns.Its architecture is described as self-contained.
  • Limitations: PersonaVLM does not support person recognition or tracking from video or audio inputs, depends on the underlying baseline model, and cannot connect related episodic memories across time.Its memory system is primarily timeline-based and does not yet merge related episodic memories.
  • Framework Components: The response phase uses multi-turn reasoning and memory retrieval, while separate prompts infer personality and create or update semantic, procedural, core, and episodic memories.The supplied prompts describe context-aware response generation, personality-state key-value extraction, long-term memory analysis, profile updates, and dialogue-topic summaries.
Loading 2604.13074v1…