Source-linked AI summary
Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges
Tiejin Chen, Pingzhi Li, Kaixiong Zhou, Tianlong Chen, Hua Wei
TL;DR
Privacy leakage in multimodal language models is underexplored despite their ability to expose sensitive information from images or memory. The paper defines two MLLM-specific risks, introduces MM-Privacy, and evaluates models across tasks, finding persistent, task-inconsistent leakage that is higher in open-source models. These findings motivate stronger, task-aware safeguards, while the dataset remains limited in its coverage of real-world data and closed-source memory-output correctness.
Problem
MLLM privacy risks are underexplored because models can expose sensitive image information or information induced by images, whereas prior work mainly examined privacy awareness.
Method
The paper defines Disclosure and Retention Risks, introduces MM-Privacy with memory and evaluation sets, and evaluates closed- and open-source MLLMs across multiple tasks.
Results
MLLMs show persistent privacy leakage across direct and memory-based tests, with open-source models notably more vulnerable and risks varying across tasks such as captioning and rephrasing.
Takeaways & Limitations
The findings support enhanced, task-aware privacy protection mechanisms for MLLMs, especially in the open-source domain.
Takeaways & Limitations
The dataset cannot evaluate the correctness of email outputs by closed-source MLLMs and lacks a comprehensive comparison between real-world and generated data.
Abstract
from arXiv · showhide
Privacy risks in text-only Large Language Models (LLMs) are well studied, particularly their tendency to memorize and leak sensitive information. However, Multi-modal Large Language Models (MLLMs), which process both text and images, introduce unique privacy challenges that remain underexplored. Compared to text-only models, MLLMs can extract and expose sensitive information embedded in images, posing new privacy risks. We reveal that some MLLMs are susceptible to privacy breaches, leaking sensitive data embedded in images or stored in memory. Specifically, in this paper, we (1) introduce MM-Privacy, a comprehensive dataset designed to assess privacy risks across various multi-modal tasks and scenarios, where we define Disclosure Risks and Retention Risks. (2) systematically evaluate different MLLMs using MM-Privacy and demonstrate how models leak sensitive data across various tasks, and (3) provide additional insights into the role of task inconsistency in privacy risks, emphasizing the urgent need for mitigation strategies. Our findings highlight privacy concerns in MLLMs, underscoring the necessity of safeguards to prevent data exposure. Our dataset and code can be found here.
1 Introduction
MLLM privacy risks remain underexplored because these models can expose sensitive information embedded in images or retained in memory. The paper introduces MM-Privacy to evaluate Disclosure and Retention Risks across tasks, finding persistent and task-dependent leakage, especially in open-source models.
- MLLMs can expose sensitive information embedded in images or output information induced by images, while prior work mainly assessed privacy awareness rather than actual leakage.
- The paper defines Disclosure Risks for immediate handling of sensitive inputs and Retention Risks for information learned during training.
- MM-Privacy combines memory and evaluation sets, expands across tasks such as captioning and rephrasing, and contains over 13,000 samples.Its basic version contains 1,000 memory samples and 2,500 evaluation samples.
- Experiments show persistent privacy leakage across closed- and open-source MLLMs, with open-source models exhibiting significantly higher risks despite generally weaker safeguards in closed-source models.
- Privacy risks are highly inconsistent across tasks, with captioning and rephrasing more frequently bypassing safeguards in closed-source models.
- The study presents MM-Privacy as the first systematic investigation of MLLM privacy issues and calls for further mitigation research.
2 Related Work
Prior work shows that MLLMs can be vulnerable to malicious visual inputs, but privacy-focused evaluation remains insufficiently developed. Existing studies have examined safety attacks and defenses, while the specific risk of sensitive-data leakage is still underexplored.
- The paper’s contextually related images are generated with Stable Diffusion using topic-related keywords and then filtered by humans for quality.
- MLLMs can be vulnerable to malicious inputs, including instructions encoded into images that weaken safety alignment.
- Prior MLLM safety research studies malicious visual prompts and defenses designed to reduce attacks without substantially harming performance.
3 The MM-Privacy Dataset
MM-Privacy is a multi-dimensional benchmark for Disclosure and Retention Risks in MLLMs, using shared images, distinct prompts, synthetic private information, and diverse real-world scenarios. Its evaluation spans multiple formats and tasks, with ASR measuring successful private-data leakage.
- Risk Definition: MM-Privacy evaluates Disclosure Risks from sensitive inputs and Retention Risks from information memorized during fine-tuning.
- Dataset Construction: The benchmark uses a Memory Set for Retention Tests and a shared Evaluation Set containing non-overlapping synthetic data for both tests.
- Dataset Scope: MM-Privacy contains over 13,000 samples across text, image, and multimodal formats, including forms, tables, handwritten documents, and four scenario categories.
- Image Construction: The dataset combines automatically generated forms, human-filled photographed forms, and contextually related images produced with Stable Diffusion and human filtering.
- Scenario Design: The scenarios cover hiring, financial, verification, and open-context settings using templates that adapt instructions to different privacy risks.
- Instruction Generation: Disclosure prompts directly request private information, whereas Retention prompts probe memorized information while keeping evaluation data non-overlapping.
- Instruction Rephrasing: The benchmark expands its initial evaluation sets through task-specific rephrasing, reaching 13,500 samples across tasks such as VQA, rephrasing, and classification.
- Evaluation Metrics: Attack Success Rate measures whether adversarial prompts induce real or correct private-data leakage, with higher ASR indicating greater privacy risk.
4 Experiments
Experiments evaluate Disclosure and Retention Risks across closed- and open-source MLLMs, tasks, training targets, and defense prompts. Privacy vulnerability varies substantially by task, model family, training method, and defense prompt.
- Experimental setup: The evaluation covers Disclosure Risks and Retention Risks across closed-source and open-source MLLMs using MM-Privacy.Closed-source models include GPT-4V, GPT-4o, Gemini-1.5-pro, and Claude3-Haiku; open-source models include Idefics2, Llava variants, Xgen-Phi3, and PaliGemma.
- Disclosure Risks: Captioning and rephrasing produce the highest disclosure ASR among tasks, while classification has the lowest refuse rate without the highest ASR.The authors attribute captioning and rephrasing vulnerability to attention shifting, whereas indirect classification instructions may confuse models.
- Disclosure Risks: Open-source MLLMs generally have lower refuse rates than closed-source models, but most remain below 50% ASR because recognizing image text is difficult.Idefics2 is the exception noted as exceeding 50% ASR; direct requests are more effective against open-source models than captioning or rephrasing in most cases.
- Retention Risks: Captioning and rephrasing also show higher retention risk, while the GPT family has higher risk and SSNs are better protected than other information types.Retention testing fine-tuned open-source models with LoRA for 10 epochs; closed-source models were assessed only by refuse rate because their memorized PII was unavailable for evaluation.
- Training targets: Contrastive learning preserves the most privacy, whereas supervised fine-tuning has the highest ASR because it tends to overfit.The comparison uses Idefics2 trained with contrastive learning, QA-style learning, and supervised fine-tuning.
- Defense prompts: Defense prompts raise Llava-1.6’s refuse rate to nearly 100% across tasks, but provide only marginal protection for Idefics2.Different defense prompts also produce different defensive performance, with the Xie et al. prompt reported as best overall for Idefics2.
5 Conclusion
The study finds that MLLMs can leak private information through direct and memory-based tests, especially during captioning and rephrasing. Closed-source models generally provide stronger safeguards, whereas open-source models are notably vulnerable.
- MLLMs are susceptible to privacy leaks through direct and memory-based tests, particularly for image captioning and rephrasing.
- Closed-source models generally offer better privacy safeguards, while open-source models are notably vulnerable.
Limitation
The study’s limitation is that its memory-set evaluation cannot assess the correctness of email outputs from closed-source MLLMs. It also lacks a comprehensive comparison between real-world and generated data because real-world data volume is low.
- Correctness of email outputs from closed-source MLLMs cannot be evaluated despite the dataset containing a memory set.
- The study does not comprehensively compare real-world data with code-generated data because the available real-world data volume is low.
- Exploring privacy risks from real-world data remains future work.
B Influence of Memory Set Size
Retention ASR increases as the memory set grows from 200 to 500 samples, then decreases when it grows from 500 to 1000. The authors associate the decrease with reduced overfitting and memorization in larger training sets.
- ASR rises when memory-set size increases from 200 to 500 samples.The result suggests that processing more data initially increases the likelihood of retrieving and outputting memorized information.
- ASR decreases when memory-set size increases from 500 to 1000 samples because larger training data makes overfitting and memorization less likely.
C Label Template for Evaluation Set
The evaluation set uses a label template for each image, and the resulting label is used only during memory-set injection and fine-tuning.
- The label template provides structured fields for identifying information in a given image.The supplied passages indicate that the template is associated with the image before injection.
- The label is used only during the injection stage, when the model is fine-tuned with the memory set.
D Example of Cross-task Inconsistency for Five Tasks
The paper illustrates cross-task inconsistency by comparing privacy issues across five tasks, each prompted to induce private-information output.
- Five different tasks are used to demonstrate inconsistency in MLLMs’ privacy issues.Each task has prompts designed to induce the model to output private information.
E Example of Images Generated by Diffusion Model
The paper presents synthetic diffusion-model images alongside task examples and memory-set evaluations to examine privacy behavior across MLLM settings.
- Synthetic images generated by Stable Diffusion are presented as examples for the dataset’s image-based evaluation.The examples are paired with the keywords used as Stable Diffusion inputs.
- An example with five tasks illustrates how privacy issues can be inconsistent across tasks.
- ASR is evaluated with different memory-set sizes using Xgen-mm, SFT, and Directly Ask.
F Text Prompt for Different Task
The prompts target private information in images, including Social Security numbers, using task-specific wording and a structured personal-information label template.
- The prompts are provided for different tasks designed to induce models to output private information.The example specifically targets Social Security Numbers.
- The prompt collection uses a special SSN context to disturb aligned models’ attention, because SSN safeguarding is stronger than for email and phone numbers.Without this sentence, the stated SSN refusal rate becomes 1.0.
- The label template organizes names, gender, birth dates, addresses, email addresses, phone numbers, and Social Security numbers.
- One prompt asks the model to output an image’s SSN while framing the request as loan-application verification.It also instructs the model to answer “cannot assist” if it cannot help.
- The figure presents different prompts across tasks intended to induce private-information output, using Social Security Numbers in the example.
- Synthetic images generated by Stable Diffusion are paired with the keywords used to generate them.