PaperlayerView exact revision on arXiv

Source-linked AI summary

RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models

Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, Xiangyu Yue

arXiv:2410.13360v3cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

MLLMs lack user-specific knowledge needed for effective personalized assistance. RAP addresses this gap with a database-backed remember–retrieve–generate framework and a personalized training dataset, producing assistants that perform strongly across captioning, question answering, and visual recognition while supporting real-time concept updates. Its main limitations are retrieval precision and the additional generation cost from longer contexts.

  • Problem

    MLLMs lack user-specific knowledge and struggle to remember personal concepts, limiting personalized responses without impractical per-user data collection.

  • Method

    RAP stores personal concepts externally, retrieves relevant multimodal information, incorporates it into MLLM inputs, and trains personalized assistants with a dedicated large-scale dataset.

  • Results

    RAP-MLLMs show strong performance and flexibility across personalized image captioning, question answering, and visual recognition, while supporting real-time generation adjustments.

  • Takeaways & Limitations

    Pretrained RAP-MLLMs can adapt to diverse users and new visual concepts without further training, while external database updates enable real-time concept editing.

  • Takeaways & Limitations

    RAP adds generation cost through longer MLLM contexts and depends on the retriever’s precision to select relevant concepts.

Abstract

from arXiv · show

The development of large language models (LLMs) has significantly enhanced the capabilities of multimodal LLMs (MLLMs) as general assistants. However, lack of user-specific knowledge still restricts their application in human's daily life. In this paper, we introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we turn it into a personalized assistant in three steps. (a) Remember: We design a key-value database to store user-related information, e.g., user's name, avatar and other attributes. (b) Retrieve: When the user initiates a conversation, RAP will retrieve relevant information from the database using a multimodal retriever. (c) Generate: The input query and retrieved concepts' information are fed into MLLMs to generate personalized, knowledge-augmented responses. Unlike previous methods, RAP allows real-time concept editing via updating the external database. To further improve generation quality and alignment with user-specific information, we design a pipeline for data collection and create a specialized dataset for personalized training of MLLMs. Based on the dataset, we train a series of MLLMs as personalized multimodal assistants. By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning. Our models demonstrate outstanding flexibility and generation quality across a variety of tasks, such as personalized image captioning, question answering and visual recognition. The code, data and models are available at https://hoar012.github.io/RAP-Project/.

1. Introduction

RAP addresses the difficulty of personalizing MLLMs with user-specific concepts by combining external concept memory, multimodal retrieval, and personalized generation. A dedicated dataset trains RAP-MLLMs to support personalized multimodal tasks without further training for new concepts.

  • RAP requires only one image with personalized information per concept, unlike approaches that require multiple labeled images and continuous model updates.
  • RAP targets MLLMs’ lack of user-specific knowledge, which limits their effectiveness as personalized assistants in daily life.
  • RAP stores each personal concept in a database, retrieves relevant information multimodally, and incorporates it into the MLLM input for personalized generation.The framework is designed to update supported concepts without additional training.
  • The authors create a large-scale personalized-training dataset and train MLLMs evaluated on personalized captioning, question answering, and visual recognition.
  • RAP-MLLMs demonstrate strong generation quality and flexibility across personalized multimodal tasks while adapting to diverse users and new concepts without further training.

2. Related Work

Prior MLLMs excel at general visual understanding and reasoning but lack user-specific knowledge for personalized content. Existing retrieval-based methods improve multimodal generation, yet few address personalization directly.

  • Multimodal Large Language Models: Existing MLLMs support general visual understanding, reasoning, image captioning, and question answering, but lack user-specific knowledge for personalized content.
  • Retrieval-augmented methods retrieve external knowledge to improve multimodal generation, but personalization remains insufficiently considered.

3. Retrieval Augmented Personalization

RAP personalizes MLLMs through a three-stage memory, retrieval, and generation pipeline backed by a concept database. Its dataset pipeline adds grounding, instruction-following, augmentation, and noise-based training for personalized generation.

  • 3.1. RAP Framework: RAP stores each personal concept as an image, name, and description, using the image’s visual feature as its database key.The database supports concepts such as a user’s named dog or identity-related information.
  • 3.1. RAP Framework: An open-world detector identifies regions of interest, which are used to retrieve concept information before it is integrated into the MLLM input.
  • 3.1. RAP Framework: RAP retrieves the Top-K image-text pairs with the smallest visual-feature distances and can also retrieve concepts by name for text-only queries.The visual distance is Dist(v_i, k_j) = ∥v_i − k_j∥.
  • 3.1. RAP Framework: Retrieved concept images and text are converted into visual and language tokens and incorporated into the MLLM input, while detector and retriever parameters remain frozen during training.
  • 3.2. Personalization Dataset: The personalization dataset combines visual grounding and instruction-following data, including captioning, image description, and question answering.It uses cropped target concepts, names, related information, and Gemini-generated annotations.
  • 3.2. Personalization Dataset: Noise concepts are added while preserving the original answer so models learn to filter irrelevant retrieved information during multi-concept generation.

4. Experiment

RAP-MLLMs are evaluated across personalized captioning, question answering, visual recognition, concept scaling, and controlled generation settings. The results indicate that retrieval-based personalization supports accurate user-specific outputs, robust performance as databases grow, and strong generation when relevant concept information is available.

  • Personalized Image Captioning: RAP-MLLMs generate clearer and more accurate personalized captions than comparison methods, especially when multiple user-specific concepts appear together.The database content supports reliable identification of target concepts, whereas limited training data causes imprecise outputs for LLaVA-LoRA and MyVLM.
  • Personalized Image Captioning: RAP-MLLMs achieve superior recall and precision on personalized image captioning, demonstrating data efficiency.The evaluation uses recall, precision, and F1-score; MyVLM weakens with few positive images, while vanilla LLaVA remains ineffective even with retrieved information.
  • Personalized Question Answering: RAP-LLaVA provides balanced visual and text-only question answering, surpasses baseline methods with one image, and reaches performance comparable to GPT-4V.MyVLM lacks text-only question answering, while Yo’LLaVA is limited by insufficient learned-token information.
  • Influence of Number of Learned Concepts: RAP models remain robust as database size and retrieval errors increase, although retriever recall and precision decrease with larger databases.Error-free retrieval improves performance further, indicating that retrieval quality still affects the upper bound of generation quality.
  • Generation Ability of MLLM: With relevant concept information supplied, RAP-LLaVA achieves 100% precision without irrelevant concepts and obtains a higher recall rate.This controlled setting evaluates the trained MLLM’s generation ability while skipping recognition and retrieval.

5. Conclusion

The conclusion presents RAP as a framework for personalizing MLLMs with user-specific concepts and real-time generation adjustments. It combines a large-scale personalized training dataset with trained assistants that perform across multiple personalized generation tasks.

  • 5. Conclusion: RAP-MLLMs understand user-specific concepts, generate personalized captions and answers, and support real-time adjustments to generation settings.The framework is paired with a curated large-scale dataset and a series of trained personalized assistants.
  • 5. Conclusion: Experimental results show exceptional performance across personalized generation tasks, including image captioning, question answering, and visual recognition.

Supplementary Material

The supplementary material provides additional evaluations, implementation details, dataset information, demonstrations, limitation analysis, ethics analysis, and personalized database examples.

  • Supplementary Material: Supplementary sections add evaluations, experimental details, dataset documentation, demonstrations, and examples of the personalized database.
  • Supplementary Material: The supplementary material includes dedicated analyses of the work’s limitations and potential ethics issues.

B.1. Ablation Study

The ablations examine retrieval quality, concept-count trade-offs, knowledge-intensive performance, captioning, robustness to retrieval errors, and personalization time cost. Results show that retrieval scale affects recall and precision, while training choices improve robustness and RAP preserves general capabilities while enabling personalization.

  • Retriever: Top-5 recall exceeds 90% with 500 personal concepts, although larger databases reduce retrieval performance and larger K trades precision for recall.The retriever’s recall and precision decline as database size increases, while increasing K generally improves recall but introduces irrelevant concepts.
  • Impact of Retrieving Different Numbers of Concepts: Retrieving more concepts from a 300-concept database improves captioning recall but lowers precision because additional concepts introduce noise.The result exposes a trade-off between supporting more personalized concepts and maintaining generation precision.
  • Multimodal Benchmark: RAP-LLaVA retains most general knowledge while adding external-knowledge retrieval for knowledge-intensive multimodal benchmarks.Evaluation covers MMMU and InfoSeek with retrieved textual knowledge from external image-caption databases.
  • Image Captioning: RAP-LLaVA achieves comparable general image-captioning performance while enabling personalization, whereas LLaVA declines when given additional concept information.The standard captioning evaluation averages results across three evaluations and concepts, using five augmented captions as ground truth.
  • Robustness to Retrieval Errors: Negative samples improve robustness to retrieval errors, enabling RAP-MLLMs to distinguish irrelevant concepts and generate accurate responses.Models perform even better when retrieval is error-free, while robustness remains strong despite declining retriever recall and precision as databases grow.
  • Time Cost of Personalization: RAP avoids the repeated forward-backward optimization required by MyVLM and Yo’LLaVA for learning each concept, reducing personalization time cost.The comparison evaluates methods for learning a user’s concept set using two A800 GPUs.

C. More Experimental Details

The experimental setup uses visual retrieval infrastructure and multi-concept data collection to evaluate personalized captioning. Standard captioning metrics average performance across evaluations and concepts using augmented captions as ground truth.

  • Implementation Details: RAP uses YOLO-Worldv2-X for concept detection and FAISS with CLIP ViT-L/14-336 embeddings for multimodal retrieval.Each database key is a 768-dimensional visual feature produced by the CLIP visual encoder.
  • Evaluation Protocol: Standard captioning metrics treat five augmented captions as ground truth and average results across three evaluations and all concepts.These results are reported for personalized captions generated by RAP-LLaVA.
  • Multi-concept Data Collection: The multi-concept collection pipeline samples YouTube frames, detects concept pairs, selects matching images with Gemini, and generates five captions per image.Each concept pair has 8 to 13 testing images.

D. Details of Dataset

The dataset section documents its composition, numerical statistics, sources, and instruction templates for grounding, recognition, captioning, description, and question-answer synthesis.

  • Dataset Composition: Figure 9 summarizes the distribution of the dataset’s different components.The figure provides a visual overview of dataset composition.
  • Dataset Statistics: Table 9 reports detailed numerical statistics for each dataset component.The table complements the visual composition summary with component-level counts.
  • Data Sources: Table 10 identifies the data sources used for each dataset component.The source mapping documents where the dataset components originate.
  • Instruction Templates: Instruction templates cover visual grounding and recognition, image captioning, image description, and question-answer synthesis.These templates define examples for the principal personalized multimodal tasks.

E. Additional Demonstrations

Additional demonstrations cover qualitative task outputs, real-time concept editing and addition, personalized conversations, visual recognition, and dataset composition. The discussion also identifies computational and retrieval-precision limitations alongside privacy, security, and ethical concerns.

  • Real-time Concept Updates: Tables 12 and 13 demonstrate real-time concept editing and addition by updating the external database.These demonstrations illustrate database-based changes to supported concepts without describing further model training.
  • Personalized Conversation: Tables 14 and 15 present qualitative personalized conversations from RAP-LLaVA and RAP-Phi3-V.The demonstrations cover conversational personalization for both model families.
  • Additional Qualitative Results: Tables 16–19 provide additional captions and image descriptions from RAP-LLaVA, RAP-Phi3-V, LLaVA, and Phi3-V.These qualitative examples extend the reported demonstrations across captioning and image description.
  • Visual Recognition: Tables 20 and 21 show RAP-LLaVA visual-recognition results, including precise bounding boxes for specific concepts.The demonstrations focus on recognition and localization of personalized concepts.
  • Limitations: RAP’s main limitations are additional generation-time computation from longer contexts and dependence on retriever precision.The paper calls for methods that reduce computational burden and more robust multimodal retrievers.
  • Potential Ethics Issues: Local deployment gives users flexibility, privacy, and security, but improper personalization can produce bias or misleading and harmful content.The paper recommends bias detection, fairness evaluation, and responsible model-use mechanisms.

H. Examples of the Personalized Database

The personalized database stores user concepts with images, names, and optional descriptions, which RAP-MLLMs use to generate concept-aware responses. Examples show that adding or editing stored concepts changes recognition and personalized captioning or question answering.

  • Database and concept editing: RAP stores each concept with an image, name, and optional personalized description that is integrated into the MLLM input during inference.Users can define concept names and descriptions according to their preferences.
  • Database and concept editing: Adding toy2 to the database changes the caption from recognizing toy1 alone to recognizing both toy1 and toy2.The example contrasts outputs before and after the new concept is stored.
  • Personalized question answering: RAP-LLaVA answers personalized appearance questions by recalling stored attributes, including P’s bald head, red beard, clothing, and seated setting.The response supplies a detailed description tied to the named concept P.
  • Personalized question answering: RAP-Phi3-V combines named concepts with visual context to describe actions and attributes, such as Anya feeding Bond and Anya’s clothing.The examples include both multi-person or person-animal activity descriptions and direct attribute questions.
  • Personalized captioning: Qualitative comparisons show RAP models incorporating personalized identities and attributes into captions, such as my cat, named people, pets, figurines, and other stored concepts.The examples span RAP-LLaVA and RAP-Phi3-V across image captioning and personalized image description.
Loading 2410.13360v3…