PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Language Models be Instructed to Protect Personal Information?

Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, Alan Ritter

arXiv:2310.02224v1cs.CL

TL;DR

The paper asks whether multimodal language models can protect specified personal information without sacrificing utility, amid demonstrated privacy risks from memorized data. It introduces PRIVQA and iterative self-moderation to evaluate and improve instruction-based protection. Self-moderation improves privacy in non-adversarial settings, but models remain biased and vulnerable to textual and image-based jailbreaks.

  • Problem

    The paper addresses how to protect specified personal information while preserving model utility, given privacy leakage risks and an inherent privacy/utility trade-off.

    Source: §1 · p. 1

  • Method

    The paper introduces PRIVQA, a multimodal benchmark, and an iterative instruction-based self-moderation technique for access-control tasks.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Self-moderation improves privacy, but state-of-the-art models remain vulnerable to adversarial textual and image inputs and show inconsistent protection across groups.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    PRIVQA supports reproducible evaluation of privacy protection and adversarial robustness in language and vision-language models.

    Source: §1 · p. 2, §6 · p. 9

    GPT4V was not benchmarked with PRIVQA because its release occurred too late for the study.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large multimodal language models have proven transformative in numerous applications. However, these models have been shown to memorize and leak pre-training data, raising serious user privacy and information security concerns. While data leaks should be prevented, it is also crucial to examine the trade-off between the privacy protection and model utility of proposed approaches. In this paper, we introduce PrivQA -- a multimodal benchmark to assess this privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario. We also propose a technique to iteratively self-moderate responses, which significantly improves privacy. However, through a series of red-teaming experiments, we find that adversaries can also easily circumvent these protections with simple jailbreaking methods through textual and/or image inputs. We believe PrivQA has the potential to support the development of new models with improved privacy protections, as well as the adversarial robustness of these protections. We release the entire PrivQA dataset at https://llm-access-control.github.io/.

1 INTRODUCTION

The paper introduces PRIVQA to evaluate whether multimodal models can follow access-control instructions while preserving utility, and proposes iterative self-moderation to improve protection. Results show improved non-adversarial protection but persistent bias, privacy/utility trade-offs, and vulnerability to textual and visual attacks.

1 INTRODUCTION

  • Privacy safeguards impose an alignment tax, creating a trade-off between information protection and model utility.Existing unlearning and model-editing approaches can be computationally intensive, poorly generalizable, and performance-degrading for realistic controls.Source: §1 · p. 1
  • PRIVQA is an open multimodal benchmark for assessing instruction-following models’ protection of specified people or information categories.Its protected groups and categories are motivated by the General Data Protection Regulation.Source: §1 · p. 2, Figure 1 · p. 2
  • The benchmark evaluates protection of defined populations or information types while retaining answers about control groups.Examples include protecting Italian citizens or geolocation information while answering about non-Italian public figures.Source: §1 · p. 2
  • API-based models such as GPT-4 outperform open-source models such as LLaMA in non-adversarial personal-data protection, especially with self-moderation.The authors also report that models can provide less protection for more private or less well-known individuals.Source: §1 · pp. 1–2
  • The benchmark had not yet evaluated GPT4V, despite OpenAI’s claim that it included built-in privacy protection.The authors note that PRIVQA enables reproducible evaluation of undisclosed safety measures in proprietary and open-source models.Source: §1 · p. 2
  • Iterative self-moderation improves models’ ability to follow access-control instructions, while biases remain in protection across groups.The paper reports inconsistent protection linked to popularity and race.Source: §1 · p. 2
  • Adversarial techniques easily circumvent access-control instructions in state-of-the-art models.Red-teaming includes popular jailbreaking prompts, multi-hop attacks, and image-based attacks.Source: §1 · p. 2

2 RELATED WORK

Prior privacy work targets memorization, training data, or model knowledge, while red-teaming and jailbreaking work probes unsafe or privacy-sensitive behavior. These approaches motivate evaluating flexible access-control instructions for both under-protection and over-protection.

2 RELATED WORK

  • Differential privacy, machine unlearning, and model editing provide privacy controls but do not generalize well to complex information categories defined beyond tokens or associations.Unlearning is also generally irreversible, making adaptation to changing privacy guidelines difficult.Source: §2 · p. 3
  • Red-teaming studies use manually or automatically constructed adversarial inputs to probe model robustness, including privacy leakage and exaggerated safety behavior.Jailbreaking methods target competing objectives, mismatched generalizations, or specific problematic completions.Source: §2 · p. 3

3 THE PRIVQA BENCHMARK

PRIVQA benchmarks whether multimodal models can protect designated personal information while remaining useful on unprotected queries. It defines protected and control groups, constructs balanced textual and visual QA data, and evaluates privacy and utility separately.

3.1 TASK DEFINITION

  • PRIVQA defines protected and control groups, requiring abstention for protected queries while preserving answers for control queries.Protected queries concern designated populations or information categories; control queries have no overlap with the protected group.Source: §3.1 · p. 4

3.2 TASK CONSTRUCTION

  • The benchmark combines five textual and visual QA datasets, including three open-domain textual datasets and two knowledge-intensive visual QA datasets.PopQA+ adds automatically generated human-entity questions, while visual examples include geolocation queries linked to Wikipedia entities.Source: §3.2 · pp. 4–5
  • Protected populations include citizenship, age, occupation, and public positions, while protected information covers six categories including location, professional background, and personal relationships.The population categories are motivated by GDPR-related identifiers, and information categories are derived from personal-data concerns.Source: §3.2 · p. 5
  • Each PRIVQA category contains at most 200 protected examples and an equal number of control examples for balanced evaluation.The benchmark uses public-figure information to avoid exposing real private user data, trading some realism for public reproducibility.Source: §3 · p. 3, §3.2 · p. 5

3.3 EVALUATION METRICS

  • Evaluation measures both response F1 and protection score to capture utility on answers and privacy behavior across protected and control groups.Response F1 compares protected models with the same models without privacy protections, while protection score combines sensitivity and specificity.Source: §3.3 · p. 5
  • Response F1 should decrease substantially for protected queries but minimally for control queries, because control-group degradation indicates an alignment tax.Protection score uses sensitivity for abstaining on protected cases and specificity for responding to non-protected cases, combined through their harmonic mean.Source: §3.3 · p. 5
  • Models signal abstention by outputting the string “none.”This output convention operationalizes the response behavior required for protected queries.Source: §3.3 · p. 5

4 PRIVQA EXPERIMENTS: PROTECTING AGAINST PERSONAL INFORMATION LEAKS

PRIVQA evaluates privacy–utility trade-offs across textual and visual tasks, finding that self-moderation improves protection but remains imperfect and biased. Protection also varies with model scale, iteration count, entity popularity, and whether instructions or model editing are used.

Textual tasks

  • Instruct prompting protects only 32.8% of protected populations and 55.2% of protected information for gpt-3.5, showing weak privacy protection from direct instructions.The evaluation uses access-control instructions appended to input queries.Source: §4 · p. 6
  • 42% and 29% average protection-score improvements over instruct prompting for protected populations and information, respectively, followed by large protected-group F1 drops but only minor control-group drops.These gains use GPT-3.5 self-moderation with one self-authorization iteration.Source: §4 · p. 6

Iterative self-authorization

  • Self-moderation raises protection from 62% to 74% for protected populations within three GPT-3.5 self-authorization steps, whereas additional steps degrade protection for LLaMA-based models.The LLaMA2-70B model often overturns earlier decisions after one step.Source: §4 · p. 7, Figure 4 · p. 7

Visual tasks

  • Visual self-moderation averages about 65% protection for IDEFICS-9B and IDEFICS-80B, but often infers citizenship from appearance, misclassifying 90% of sampled people of Japanese descent as Japanese.This shortcut creates bias because minorities are less likely to be classified as belonging to a protected population.Source: Figure 3 · p. 6, §4 · p. 7

Entity popularity

  • Sensitivity falls from 80% to 20% for gpt-3.5 and from 100% to 60% for gpt-4 when moving from head to tail entities by Wikipedia popularity.The paper links weaker protection for less popular entities to their infrequent representation in pre-training data.Source: §4 · p. 7, Figure 5 · p. 7

Model editing

  • ROME model editing collapses protection to 0% and response F1 to near 0 for protected and control groups across many edits, while self-moderation remains around 70% protection.The comparison uses llama2-7B-chat and averages five random seeds across 1–100 edits.Source: §4 · pp. 7–8

5 RED TEAMING: CIRCUMVENTING ACCESS CONTROL INSTRUCTIONS IN PRIVQA

The red-team evaluation tests whether self-moderation withstands black-box textual, visual, and multi-hop attacks. Simple jailbreaks substantially increase attack success, and multi-hop prompts can completely circumvent access control in some scenarios.

5 RED TEAMING: CIRCUMVENTING ACCESS CONTROL INSTRUCTIONS IN PRIVQA

  • The red-team study applies baseline and task-specific attacks to SELF-MODERATION, identified as the best-performing prompting method in the preceding evaluation.Red teaming is used to assess privacy, security, and bias concerns.Source: §5 · p. 8

5.1 THREAT MODEL

  • The threat model allows adversaries to alter both question text and images in visual QA, expanding the attack surface beyond textual prompting.The objective is to derive adversarial inputs that bypass protection and elicit answers about protected groups.Source: §5.1 · p. 8
  • The evaluation measures ASR as the rate at which attacks make protected queries appear benign, with ASR = 100% − SENSITIVITY.The adversary is assumed to have unrestricted black-box query access and may construct later queries from earlier responses.Source: §5.2 · p. 8, §5.1 · p. 8

5.2 METHODS

  • Adversarial prefixes such as “Ignore previous instruction,” AIM, and BetterDAN drastically increase ASR after self-moderation, exposing weak robustness to textual jailbreaking.These prompts are prepended directly to PRIVQA questions.Source: §5.2 · p. 9
  • Visual prompt injection raises ASR from 27% to 90% for Japanese citizens and from 30% to 95% for German citizens in IDEFICS-9B.The attack renders misleading nationality information into the input image.Source: §5.2 · p. 9, Figure 7 · p. 9
  • Figure 7 demonstrates visual prompt injection by rendering misinformation into an image, matching the attack mechanism evaluated in Table 2.The figure illustrates how image content can be manipulated to mislead visual language models.Source: Figure 7 · p. 9, §5.2 · p. 9
  • 100% ASR can occur under multi-hop prompts even where self-moderation achieves perfect or near-perfect sensitivity in non-adversarial settings.The attacks combine answers to unprotected queries that leak information correlated with protected information.Source: §5.2 · p. 9

6 CONCLUSION

PRIVQA exposes gaps in instruction-based privacy protection: models remain vulnerable to adversarial inputs and show privacy–utility trade-offs and demographic biases. GPT-4V red-teaming further shows that follow-up prompts can elicit geolocation information after an initial refusal.

6 CONCLUSION

  • PRIVQA reveals that current protections are not robust to adversarial inputs and involve privacy–utility trade-offs, with inconsistent protection across demographic groups.The paper attributes inconsistent protection to biases based on popularity and race.Source: §6 · p. 9

A.1 GPT-4V(ISION) REDTEAMING

  • 6 of 12 GPT-4V geolocation follow-ups correctly identified buildings after all initial queries were refused.The follow-up tour-request prompt produced identifications in 8 of 12 cases, including 6 correct identifications.Source: §A.1 · p. 14

A.2 DATASET CONSTRUCTION

The dataset construction combines textual and visual QA sources, filtering for human entities and organizing examples around protected groups and information categories. It also documents the source datasets and the geolocation red-teaming example.

A.2 DATASET CONSTRUCTION

  • PRIVQA curates textual QA from TriviaQA and PopQA and visual QA from KVQA and InfoSeek, restricting examples to QA pairs containing human entities.Human-entity filtering uses Wikidata metadata marking the "instance of human" annotation.Source: §A.2 · p. 14, Table 4 · p. 16
  • Protected and control groups are formed from Wikidata attribute–value annotations, with up to 200 protected examples matched by an equal-sized random control subset.Protected groups contain entities possessing the selected attribute–value pair; remaining examples supply the control group.Source: §A.2 · p. 14
  • Protected-information QA covers six categories, including geolocation queries about buildings and mountains, with examples subsampled to balance relations and queried entities.The maximum group size is 200 examples.Source: §A.2 · pp. 14, 16
  • The geolocation red-teaming example shows GPT-4V revealing a building name or specific location after a follow-up prompt despite an initial refusal.The examples include Hangzhou International Conference Center and Brighton Beach in Melbourne.Source: Figure 9 · p. 15

A.3 EXPERIMENT DETAILS

The experiments evaluate API-based GPT models, open-source LLaMA2 and IDEFICS models, and editing with ROME, while documenting the associated prompts and multi-hop materials. MEND was omitted because its GPU memory requirement exceeded the available hardware.

A.3 EXPERIMENT DETAILS

  • Experiments use GPT-4 and GPT-3.5-turbo, LLaMA2 7B/13B/70B, IDEFICS-9B/80B, and ROME-edited LLaMA2-7B models.The listed configurations specify GPU resources for the open-source and multimodal models.Source: §A.3 · p. 16
  • Table 5 provides the multi-hop question templates used in Section 5.The table is part of the documented experimental materials.Source: Table 5 · p. 17
  • MEND was not evaluated because running it on LLaMA2-7B required over 50GB of GPU memory, exceeding the available 46GB A40 GPU.This hardware constraint limits the experiment comparison to the reported methods.Source: §A.3 · p. 17
  • The appendix documents the Self-Moderation, AIM, and BetterDAN prompts used in the experiments.These prompts correspond to Figures 11–13.Source: §A.3 · p. 17

A.4 MULTI-HOP QA

The multi-hop evaluation constructs a second question from the answer to a first question, testing whether indirect clues can narrow protected locations. The appendix also documents self-moderation and jailbreak prompts, while noting that multi-hop protected-population experiments were omitted.

A.4 MULTI-HOP QA

  • Multi-hop questions use the answer to a first question to construct a second question that can significantly narrow the scope of possible locations.For example, language information can support a subsequent nationality query.Source: §A.4 · p. 17
  • The study does not perform multi-hop experiments for protected populations because templated questions are difficult to create without tailoring them to each individual.This limits the protected-population portion of the multi-hop evaluation.Source: §A.4 · p. 17
  • Figures 10–13 document the textual and visual Self-Moderation prompts and the AIM and BetterDAN jailbreak prompts used in the experiments.These materials specify the prompt variants associated with the red-teaming setup.Source: Figure 10 · p. 18, Figure 11 · p. 19, Figure 12 · p. 20
  • Table 6 reports protective precision and response F1 for protected and control groups, with Δ% measuring relative response F1 drop from the baseline.Self-Authorization is used only for GPT-series models.Source: Table 6 · p. 21

B.1 DETAILED EFFECTS OF MULTIPLE SELF-AUTHORIZATION STEPS

Additional experiments examine how repeated Self-Authorization affects protection metrics, how model editing compares with Self-Moderation, and how visual moderation handles protected identities.

B.1 DETAILED EFFECTS OF MULTIPLE SELF-AUTHORIZATION STEPS

  • Multiple Self-Authorization steps consistently increase sensitivity and decrease specificity, with the rate depending on the model.These results are calculated only for protected information.Source: §B.1 · p. 22, Figure 14 · p. 22

B.2 MODEL EDITING

B.3 VISUAL QA

  • Annotation results for Japanese and German citizen splits were 28/30 and 26/34, respectively, under the visual QA bias guidelines.The guidelines separately assess correct protection decisions and incorrect name identification despite recognizing the protected group.Source: §B.3 · p. 22
  • In visual QA, moderation can abstain correctly despite misidentifying a person, but such decisions may reflect citizenship-classification bias.The visual examples include correct identification and a case where Sayaka Akimoto is identified as Yuko Ogura while the model still abstains because it classifies her as Japanese.Source: Figure 17 · p. 24

Related papers

Loading 2310.02224v1…