PaperlayerView exact revision on arXiv

Source-linked AI summary

MMPB: It's Time for Multi-Modal Personalization

Jaeik Kim, Woojin Kim, Woohyeon Park, Jaeyoung Do

arXiv:2509.22820v2cs.CVcs.AI

TL;DR

Recent VLMs remain poorly adapted to individual identities and preferences, even as they are used in multimodal systems where personal context matters. MMPB benchmarks personalization across concepts, queries, and dialogue, and evaluations find persistent weaknesses in preference reasoning, visual cues, and consistency.

  • Problem

    Recent VLMs largely treat users interchangeably, despite user-specific visual context mattering in smart-home and other multimodal applications.

    Source: §1 · p. 1

  • Method

    MMPB evaluates VLM personalization with 111 concepts, preference-grounded queries, three task types, multi-turn dialogue, and 23 widely used models.

    Source: §1 · p. 2

  • Results

    Across evaluated VLMs, personalization remains difficult, especially preference-grounded reasoning, use of visual cues, and consistency across long interactions.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    MMPB provides a diagnostic foundation for personalized VLM behavior, while future benchmarks should examine dynamically evolving user traits.

    Source: §1 · p. 2, §7 · p. 10

    MMPB evaluates fine-grained but static appearances and preferences, not how models track changing identities or tastes over time.

    Source: §7 · p. 10

Abstract

from arXiv · show

Visual personalization is essential in user-facing AI systems such as smart homes and healthcare, where aligning model behavior with user-centric concepts is critical. However, recent large Vision-Language Models (VLMs), despite their broad applicability, remain underexplored in their ability to adapt to individual users. In this paper, we introduce MMPB, the first extensive benchmark for evaluating VLMs on personalization. MMPB comprises 10k image-query pairs and includes 111 personalizable concepts across four categories: humans, animals, objects, and characters, with the human category enriched with preference-grounded queries. We structure personalization into three main task types, each highlighting a different key property of VLMs. Using 23 widely used VLMs including both open- and closed-source models, we evaluate personalization performance via a three-stage protocol: concept injection, multi-turn dialogue, and personalized querying. Our findings indicate that most VLMs (including some closed-source models) struggle with personalization, particularly in maintaining consistency over dialogue, handling user preferences, and adapting to visual cues. Our analysis reveals that the challenges in VLM personalization (such as refusal behaviors and long-context forgetting) highlight substantial room for improvement. By identifying these limitations and offering a scalable benchmark, MMPB offers valuable insights and a solid foundation for future research toward truly personalized multi-modal AI. Project Page: aidaslab.github.io/MMPB

1 Introduction

Recent VLMs lack user-specific adaptation despite their use as multimodal assistants, leaving a gap in evaluating personalization. MMPB addresses this gap with diverse personalizable concepts, preference-grounded queries, and multi-turn dialogue.

1 Introduction

  • Existing VQA benchmarks emphasize general-purpose knowledge, while personalization work lacks concept diversity, unified tasks and metrics, cold-start evaluation, and preference-grounded VQA.Source: §1 · pp. 1–2
  • Human–model curation filters queries answerable from the concept or image alone, targeting cross-modal reasoning and retention through multi-turn dialogue.Source: §1 · p. 2
  • Across 23 widely used VLMs, models struggle with preference-grounded reasoning, visual personalization, human-centric concepts, and long-context consistency.The reported failure patterns also include personalization bias and safety-induced evasiveness.Source: §1 · p. 2
  • MMPB provides 111 concepts across humans, animals, objects, and characters, plus 10,000+ questions and 15 task types, including human preference queries.Source: §1 · p. 2
  • Five reference images and four description levels provide resources for future techniques such as post hoc training.Source: §1 · p. 2

2 Problem Definition

The section defines four criteria for assessing personalized VLM behavior and formalizes personalization as adding a user-specific concept to image and text inputs. It describes concept injection through either visual or textual modalities.

2.1 Core Properties of Personalized VLMs

  • Four criteria assess whether a personalized concept is recognized, used only when appropriate, kept free of contradictions, and retained across long-context or multi-turn exchanges.Source: §2.1 · p. 3

2.2 Formalizing VLM Personalization

  • Personalization extends VLM inputs to include an embedded concept alongside image and text encodings, which can be concatenated without changing model weights.MMPB injects the concept through either visual or textual modalities.Source: §2.2 · p. 3

3 MMPB: Multi-modal Personalization Benchmark

MMPB is a multiple-choice VQA benchmark for evaluating personalization across recognition and user-preference tasks. It assesses whether models retain and apply concepts introduced through images or text across dialogue and personalized queries.

3.1 Overview

  • MMPB contains 10,017 image–query pairs across 111 concepts and four categories, covering Awareness, Appropriateness, and Coherency.The benchmark focuses on recognition and user preference.Source: §3.1 · p. 3
  • Evaluation proceeds through concept injection, multi-turn conversation, and personalized querying to test concept introduction, retention, and application.Concepts are introduced through reference images or textual descriptions.Source: §3.1 · p. 3

3.2 Data Collection

  • Concepts have five reference images, while appearance descriptions span four levels of detail; the benchmark assumes concept states remain static.The assumption excludes temporal changes such as appearance shifts or evolving preferences.Source: §3.2 · p. 4
  • Human preferences are represented as textual likes and dislikes across five domains with six subdomains each, with annotators resolving conflicting assignments.Each human receives a template-based set of preference keywords.Source: §3.2 · p. 4

3.3 Query Design

  • Queries are drafted by a model and refined by annotators, then screened to rule out concept-only and image-only shortcuts in Coherency questions.Distractors are designed to require joint reasoning over image and concept information.Source: §3.3 · pp. 4–5
  • Fifteen tasks apply Awareness, Appropriateness, and Coherency to recognition and human preference queries, with Persistency assessed through multi-turn dialogue.Awareness tests concept presence, Appropriateness tests when references should be suppressed, and Coherency tests context-appropriate responses.Source: §3.3 · p. 5

4 Benchmarking Results

Across 23 VLMs, text-based concept injection outperforms image-based injection, and personalization performance degrades in longer dialogues. Averaged across settings, open models outperform closed models on MMPB.

4.1 Experimental Setup

  • Experiments simulate cold-start personalization with moderate text descriptions or two reference images, and evaluate both 0-turn and standardized 10-turn settings.Overall accuracy is the main evaluation metric.Source: §4.1 · p. 5
  • The evaluation covers 23 widely used models across open- and closed-source families, selected to represent major VLM families based on availability and usage.The sampled families include InternVL2.5, Ovis2, LLaVA, Qwen-VL, DeepSeek-VL2, Llama-3.2, Claude, Gemini, and GPT-4o.Source: §4.1 · pp. 5–6

4.2 Overall Results

  • Text-based injection scores 63.8% vs. 57.8% at 0 turns and 57.9% vs. 52.6% at 10 turns; closed models average 51.4% vs. open models’ 59.9%.Figure 4 reports results across 23 VLMs; a human evaluator averages 92.1%.Source: Figure 4 · p. 6, §4.2 · p. 6
  • Most models experience substantial performance drops from 0-turn to 10-turn settings, indicating difficulty maintaining personalized responses over time.Both open and closed model types show pronounced multi-turn degradation.Source: §4.2 · p. 6
  • Prompt tuning and personalized LoRA fine-tuning on LLaVA-1.5-13B both improve over baseline, with LoRA yielding the strongest gains.These experiments support MMPB’s sensitivity to personalization techniques beyond general VLM ability.Source: §4.2 · p. 6

5 Challenges in VLM Personalization

The analysis finds recent VLMs remain vulnerable to personalization and probes barriers across preference reasoning, refusal, personalization bias, visual complexity, and long-context retrieval.

5.1 Preference-Grounded Reasoning

  • Models strong on general VQA consistently struggle with preference-grounded queries, while recognition-task performance correlates more closely with general-VQA rankings.Recognition relies mainly on deductive matching; preference-grounded VQA requires abductive reasoning over user intentions, tastes, and scene understanding.Source: §5.1 · p. 7

5.2 Resistance to Personalization

  • Closed-source models except Gemini show evasive responses on recognition and preference tasks; up to 2,237 of 7,501 queries trigger evasive answers, mostly in the human category.InternVL2.5-78B shows evasive responses only on recognition tasks; the authors connect identity-related queries with conservative privacy-sensitive behavior.Source: §5.2 · pp. 7–8

5.3 Personalization Bias and Fine-Grained Discrimination

  • 72 of 92 cases show under-personalization, more pronounced after 10 turns; multi-entity inputs reduce Coherency, while image-based injection handles hard negatives better than text-based injection.Models maintain comparable Awareness on single- and multi-entity images, but struggle to distinguish target concepts from visually similar distractors with text-based injection.Source: §5.3 · p. 8, Figure 7 · p. 8, Figure 8 · p. 9

5.4 Visual Personalization

  • Three-keyword text injection matches the 1-image condition in 10-turn settings, and five reference images do not outperform Moderate text injection.The authors characterize the utility of visual examples in recent VLMs as limited.Source: Table 4 · p. 9, §5.4 · p. 9

5.5 Lost Concepts in a Haystack

  • Models often forget midpoint concepts, lose accuracy as multi-concept descriptions grow more detailed, and deteriorate sharply after five turns in dialogues extending to 100 turns.The midpoint forgetting occurs across prompting methods, and multi-concept inputs are evaluated with up to 50 entities.Source: §5.5 · p. 9, Figure 9 · p. 9

6 Related Works

Prior VLM work spans general multimodal tasks and emerging personalization approaches, while benchmarks and existing methods leave personalization evaluation—especially cold-start adaptation—underdeveloped.

6 Related Works

  • VLMs connect visual and linguistic modalities and support tasks including visual question answering, image captioning, and optical character recognition.The authors note that benchmarks still lack means to evaluate their personalization capabilities.Source: §6 · p. 10
  • MMPB targets user-dependent multimodal reasoning, a dimension that conventional VQA and captioning datasets largely leave unexplored.The benchmark is presented as a foundation for future research in personalized vision-language modeling.Source: §6 · p. 10
  • Existing VLM personalization approaches primarily adapt to personalizable concepts in advance, overlooking cold-start scenarios.The paper contrasts this with MMPB's systematic evaluation through hierarchical concept injection.Source: §6 · p. 10

7 Discussion

The discussion summarizes MMPB's findings on personalization weaknesses and identifies static user traits and property-centered evaluation as boundaries of its current scope.

7 Discussion

  • The analysis finds limited preference-grounded reasoning, a tendency toward refusal, and ineffective use of visual cues across recent VLMs.MMPB evaluates diverse personalizable concepts and task types, including preference-grounded VQA.Source: §7 · p. 10
  • MMPB evaluates fine-grained, static appearances and preferences, not identities whose appearances or inclinations evolve over time.The authors recommend future benchmarks include dynamic concept updates such as style changes and evolving preferences.Source: §7 · p. 10
  • MMPB formalizes Awareness, Appropriateness, Coherency, and Persistency, so its assessments center on these properties rather than downstream applications.The authors point to tailored image captioning and personalized robot actions as application-driven scenarios for future benchmarks.Source: §7 · p. 10

Supplementary Material

The supplementary material explains the four concept categories, reviews personalization approaches, notes human-image safeguards, states licensing conditions, and introduces annotation guidelines.

A.1 Rationale Behind the Four Concept Categories

  • The four categories cover socially salient people, animals, personal belongings, and virtual personas or narrative figures.The authors connect these categories respectively to user relationships, tailored animal interactions, smart-home or robot use, and avatars or digital contexts.Source: §A.1 · p. 19

A.2 Additional Related Works

  • Prior approaches use concept heads, special tokens, multi-concept learning, online image alignment, or retrieval for personalization.The paper notes open challenges in dynamic updates and long-term persistency, and says some methods depend on many negative images, increasing data and training costs.Source: §A.2 · p. 19

A.3 Ethical Consideration

  • For human-category images, the benchmark uses images from MC-LLaVA and replaces real names with <sks> to reduce pretraining-data leakage and avoid identifying private individuals where possible.Source: §A.3 · p. 19

B Licensing

  • The dataset is licensed CC BY-NC-SA 4.0 for research use only, and researchers must respect the rights of original image copyright holders.The authors state that unauthorized use or redistribution is the user's responsibility.Source: B Licensing · p. 19

C Annotation Process

  • The supplementary section introduces guidelines for models and human annotators within the paper's three-stage data-construction process.Source: C Annotation Process · p. 19

C.1 Data Collection

This section describes a data-collection protocol spanning concepts and images, text descriptions, preference keywords, and multi-turn conversations. It provides image-collection recommendations and query-generation guidelines for models and human annotators, including controls for consistency, contradiction resolution, and query images.

C.1.2 Text-based concept injection

  • Four levels of model prompts elicit descriptions from five images, progressing from concise keywords to detailed unified accounts of consistent visual traits.Human annotators corrected missing <sks> placeholders, image-specific descriptions, and inaccurate concept descriptions by restoring the placeholder, generalizing, and checking accuracy.Source: §C.1.2 · pp. 21–22

C.1.3 Keywords for preferences

  • Preference keywords are generated with GPT-4o across five domains and their subdomains, then assigned to templates and checked by annotators for contradictory like/dislike pairs.The domains are Entertainment, Travel, Lifestyle, Shopping, and Fashion; conflicts are resolved by replacing one keyword with a non-conflicting keyword from its subdomain’s master set.Source: §C.1.3 · p. 22

C.1.4 Multi-turn conversation

  • For persistency evaluation, every tested VLM receives the same ten-turn general-topic conversations extracted from LMSYS-1M.The appendix provides a default transcript excerpt spanning topics including internet history, databases, geography, and companies.Source: §C.1.4 · p. 24

C.1.1 Images for Concept Injection

  • MMPB covers 111 human, animal, object, and character concepts; humans support recognition and preference tasks, while other categories support recognition only.The image-collection guideline proposed 100 concepts (50 humans, 25 animals, 15 objects, and 10 characters), emphasizing consistent identity across images and well-known individual animals.Source: §C.1.1 · p. 20, §C.1.5 · p. 25

C.1.5 Query Images

  • Query images target Awareness, Appropriateness, and Coherency; they must differ from injection images, with positive concept examples for Awareness and negative examples for Appropriateness.Images should preserve visual identity and, for preference queries, align with the person’s keywords; animal negatives include both same-species and different-species examples.Source: §C.1.5 · pp. 25–26
  • For query images, collect preference-mismatched negatives and positive Coherency examples; prioritize consistent human appearances from MC-LLaVA and character/object images from MC-LLaVA or MyVLM.Positive Coherency images may be reused from the Awareness set.Source: §C.1.5 · p. 26

C.2.1 Models

  • Recognition query drafts cover Awareness and Appropriateness yes/no questions and Coherency four-option questions with a true concept-image relation and plausible distractors.Yes/no questions should include affirmative and negative formulations.Source: §C.2.1 · p. 27
  • Preference-query drafts use affirmative and negative yes/no questions and four-option Coherency questions, filtering out questions solvable from text or image alone.The Coherency distractors include a concept-aligned option that does not match the image and visually plausible options that are conceptually wrong.Source: §C.2.1 · p. 27

C.2.2 Humans

  • Human annotators verify query labels and type coverage, balance affirmative and negative formulations, check Coherency solvability, and ensure clear grammar and exactly four labeled MCQ options.Coherency checks require concept knowledge and at least one distractor that shares the concept but conflicts with the image.Source: §C.2.2 · p. 28

C.3 Query Evaluation

Query evaluation combines model-assisted quality checks with human review, filtering out questions answerable without both the image and personal concept information. Human annotators then correct flagged issues and ensure queries are open-ended.

C.3. Query Evaluation

  • Three models flag draft query issues using the concept descriptions, image, and query-answer pair; human annotators review a flag when at least two models agree.This model-assisted flagging speeds up human quality checks.Source: §C.3 · p. 29

1. Trivial

  • A Trivial query can be answered from the query itself or commonsense, without the image or extra information.The guidelines illustrate this with a question about what someone is doing in a movie theater.Source: §1 · p. 29

2. Concept-Only

  • A Concept-Only query can be answered from the textual description without inspecting the image.Examples include questions answerable from appearance or preference details.Source: §2 · p. 29

3. Image-Only

  • An Image-Only query can be answered from the image without using the personal description.The guidelines distinguish image-only answers from those requiring personal information.Source: §3 · p. 29

C.3.2 Humans

  • Human annotators review flagged issues, correct answer choices when needed, and mark a question None when no valid issue remains.They also rewrite questions that are not open-ended so they can be answered freely.Source: §C.3.2 · p. 30

D Human Evaluation Platforms

Supplementary evaluations examine fine-tuning, model scale, human baselines, computational resources, and dataset statistics. Fine-tuning improves LLaVA-1.5-13B, while larger models show difficulty retaining image-injected personalization in recognition tasks.

D Human Evaluation Platforms

E.1 Computational Resources

  • Open-source models ran locally on NVIDIA H100 GPUs, while closed-source models were evaluated through batch API processing.Most open-source models used one GPU; four listed large models used three, and Table 5b reports average runtimes.Source: §E.1 · p. 31, Table 5 · p. 31

E.2 Statistics

  • MMPB contains 10,017 image–text query pairs, and human preference keywords span 30 subdomains across five domains.Figure 13 presents word clouds for the five domains, generated by GPT-4o.Source: §E.2 · p. 31, Table 6 · p. 32, Figure 12 · p. 32

F.1 Personalization Fine-tuning

  • Both soft prompt tuning and personalized LoRA improve LLaVA-1.5-13B over baseline, with LoRA yielding the strongest gains.The authors report that these results confirm MMPB captures personalization effectiveness.Source: §F.1 · p. 32

F.2 Model Sizes

  • For recognition tasks, image-based injection shows an almost linear performance decline across turns, while text-based injection has more stable drop rates.Preference-task drops are similar for text- and image-based injection because preferences are injected through text in both modalities.Source: §F.2 · p. 32

F.3 Comparison of Ovis2-34B with Human-Created Descriptions

The supplementary analyses report that Ovis2-34B performs better with independently human-authored descriptions, blind-test outcomes align with query-design expectations, and dialogue topic and prompting strategy affect personalization results in different ways.

F.3 Comparison of Ovis2-34B with Human-Created Descriptions

  • Ovis2-34B attains higher accuracy with human-created descriptions than with its own outputs on a held-out set of 10 concepts.The authors interpret this result as evidence that the benchmark does not unduly advantage model-generated content.Source: §F.3 · pp. 32–33

F.4 Blind Test

  • Blind-test results closely match the expected values for Awareness, Appropriateness, and Coherency, supporting the authors’ query-design assessment.The stated expectations are 0% accuracy for Awareness, 100% for Appropriateness, and a 50% chance of a correct Coherency answer without the image.Source: §F.4 · p. 33, Table 9 · p. 34

F.5 Multi-turn Conversation

  • Recognition-related dialogues produce the smallest drop in recognition accuracy, while general conversations produce the largest multi-turn performance drop.For preference tasks, specialized dialogues show similar drop trends, saturating around 6k tokens; general recognition- or preference-related dialogue gives persistency benefits comparable to targeted reminders.Source: §F.5 · p. 34, Figure 15 · p. 35

F.6 Prompting Methods

  • Prompting methods show no statistically significant overall performance differences, although Reminder leads Awareness and slightly degrades Appropriateness.Reminder also achieves the highest Awareness accuracy, while Zero-shot-CoT and both Few-shot variants underperform plain Zero-shot on Awareness; differences remain small.Source: §F.6 · p. 34, Figure 17 · p. 37

F.7 Extended Results

The extended results compare performance across task types and concept categories, with object-category performance aligning most closely with general VQA leaderboard rank. They also report no evidence of nationality- or ethnicity-based personalization bias.

F.7 Extended Results

  • Object-category performance aligns most closely with general VQA leaderboard rank across the eight-task comparison.The extended results also include performance views across four concept categories and three task types.Source: Figure 19 · p. 38, §F.7 · p. 38, Figure 20 · p. 40
  • The comparison of overall performance with each human concept indicates no evidence of nationality- or ethnicity-based personalization bias.Source: Figure 22 · p. 41
  • The recognition-related conversation excerpt shows general questions and answers about hair-dye techniques, ingredients, and risks.Source: §F.7 · p. 42
  • The preference-related conversation excerpt begins with discussion of K-pop and the user's liking for AKMU.Source: §F.7 · p. 43
  • An explanation excerpt attributes GPT-4o's “Unknown” response to explicit safety constraints rather than a lack of capability.Source: §F.7 · p. 44, Figure 25 · p. 44
  • The appendix also presents an explanation of Claude-3.7-Sonnet's evasive response and generated samples of evasive behavior.Source: Figure 26 · p. 44, Table 11 · p. 46

Related papers

Loading 2509.22820v2…