SIEVE: Selective attention-value Suppression for Vision-Language Models Unlearning

1 INTRODUCTION

• VLMs can infer biographical information such as a person’s name, occupation, or address from a face, motivating removal of sensitive information without unnecessarily degrading remaining capabilities.Recent multimodal benchmarks pair identities with images and textual attributes to measure whether designated information remains recoverable after model updates.Source: §1 · p. 1
• Sensitive and retained associations may share images, identities, visual features, hidden states, and parameters, so reducing sensitive responses can also alter representations needed for retained predictions.The setting targets image–question–response associations, allowing a person’s date of birth to be forgotten while their hobbies remain answerable.Source: §1 · pp. 1–2
• SIEVE combines attention-value suppression on forget examples with reference-based value preservation and sequence-level forgetting and retention objectives.Forget values are encouraged toward zero; retain values are matched to a frozen pre-unlearning model.Source: §1 · p. 2
• The paper formulates association-level selective unlearning: remove designated sensitive image–question–response associations while preserving permitted associations for the same individual.The forget and retain examples can share the same image and identity while differing only in the queried information.Source: §1 · pp. 1–2
• SIEVE’s experiments report consistent reductions in sensitive-information recovery while preserving competitive retained utility.The approach does not require identifying individual sensitive tokens or assuming particular attention values uniquely encode PII.Source: §1 · p. 2

2 RELATED WORK

• Recent VLM work explores visual-language integration through lightweight projection, spatially aware connectors, multiple encoder paths, and native multimodal pretraining.The passage describes LLaVA, Qwen2-VL, Cambrian, Florence-VL, and Intern-VL3.Source: §2 · p. 2
• One cited study reports that visual grounding can be concentrated in a small subset of attention heads.Source: §2 · p. 3
• Multimodal unlearning methods include modality-aware update constraints, vacuum distillation, selected decoder updates, and training-free weakening of forget-related vectors.The cited approaches include MMUnlearner, VL-Eraser, PAVA, and Knowledge Vector Weakening.Source: §2 · p. 3

3.1 PRELIMINARIES

• The task represents each example as an image–question–response tuple and partitions associations into forget and retain sets, which can include queries about the same identity.The forget set targets sensitive responses; the retain set contains responses to preserve.Source: §3.1 · p. 3

3.2 PROPOSED METHOD

• SIEVE applies separate training objectives to forget and retain examples, combining attention-value regularization with sequence-level supervision.It uses a frozen pre-unlearning model to provide value targets for retained examples.Source: §3.2 · pp. 4–5

3.2.1 ATTENTION-VALUE SUPPRESSION FOR FORGETTING

• Forgetting combines a zero-valued target for recorded forget-example values with negative sequence cross-entropy to discourage the designated response.The authors report that attention distributions change only slightly after unlearning, preserving the original causal attention structure to a high degree.Source: §3.2.1 · p. 5

3.2.2 PRESERVING RETAINED BEHAVIOR AND VALUES

• For retained examples, SIEVE matches updated-model attention values to corresponding values from the frozen pre-unlearning model and minimizes retain-sequence cross-entropy.The preservation loss limits changes to attention values for retain examples.Source: §3.2.2 · p. 5

3.2.3 OPTIMIZATION

• SIEVE alternates forget and retain phases, combining value erasure with negative cross-entropy for forgetting and reference-value preservation with positive cross-entropy for retention.The overall objective pairs representation-level controls with supervised sequence-level objectives.Source: §3.2.3 · p. 6

4.1 EXPERIMENT SETUP

• MLLMU-Bench’s 500 fictitious and 153 celebrity profiles were split by question–response association; LLaVA and Qwen were assessed through visual and textual QA across four sets and three task metrics.The benchmark permits the same identity and image to appear in both forget and retain sets with different questions and target responses.Source: §4.1 · pp. 6–7

4.2 QUANTITATIVE RESULTS

• SIEVE had the lowest forget-set classification and cloze scores in all four model–modality settings, reducing LLaVA forget classification by 16.53 points in Visual-QA and 6.13 points in Textual-QA versus the strongest baseline.The corresponding LLaVA forget cloze scores were 15.55 and 5.66, respectively.Source: §4.2 · p. 7, Table 1 · p. 8
• SIEVE retained generation scores close to the initial model, and versus GA Diff it had lower forget classification and higher scores on all three retain metrics.The evaluation includes classification, generation, and cloze tasks on Forget, Retain, Test, and Real sets.Source: §4.2 · p. 7, §4.1 · p. 7
• On Test, SIEVE achieved LLaVA Textual-QA classification and generation scores of 25.50 and 0.349, Qwen Visual-QA classification and cloze scores of 28.03 and 0.33, and Qwen Textual-QA cloze of 3.25.The relative ordering of methods varies across test metrics.Source: §4.2 · p. 7, Table 1 · p. 8
• At 5% and 10% forget ratios, SIEVE was rightmost in all four Figure 3 panels; it combined the largest forget-score reduction with the highest utility in both ROUGE panels.In accuracy panels, it maintained competitive Retain and Real performance, while some baselines preserved higher celebrity accuracy with smaller forgetting gains.Source: §4.2 · p. 7, Figure 3 · p. 9

4.3 CASE STUDY

• In the same-ID case study, GA Diff and MMU gave incorrect retained-hobby activities, NPO gave an off-target response, while KL Min preserved the hobby but retained the targeted profession.The forget question targeted profession and the retain question asked about a permitted hobby.Source: §4.3 · pp. 7–8, Figure 4 · p. 9

4.4 ABLATION STUDIES

• Removing either L_erase or L_forget compromised forgetting, indicating that value suppression and negative cross-entropy provide distinct signals at representation and sequence levels.The ablation compares objective components using LLaVA-1.5-7B under Visual-QA.Source: §4.4 · p. 8, Table 2 · p. 9
• Removing reference-based preservation lowered forget scores but substantially degraded retained utility; compared with key suppression, value suppression preserved more Retain and Real utility while maintaining forgetting performance.The key-versus-value comparison also reports greater robustness to visual-input augmentation on the Test set.Source: §4.4 · p. 9, Table 2 · p. 9

• SIEVE combines attention-value suppression on sensitive forget examples with reference-based value preservation on retain examples.Source: §5 · p. 10
• The framework targets designated image–question–response associations while preserving non-sensitive knowledge.Source: §5 · p. 10
• Future work includes finer-grained selection within forget examples, stronger evaluation of residual-information recovery, and repeated unlearning requests over time.Source: §5 · p. 10

A.1 DIRECT EFFECT OF VALUE SUPPRESSION ON AN ATTENTION HEAD

• With fixed attention weights, changing value representations changes the head output; the bound assumes normalized softmax weights and must be adjusted if attention dropout changes row sums.The analysis keeps hidden inputs, queries, and keys fixed.Source: §A.1 · p. 14

A.2 PROPAGATION ACROSS TRANSFORMER LAYERS

• Under a local Lipschitz assumption, an earlier-layer intervention can affect later hidden states, but the bound neither requires growth across layers nor bounds answer-recovery probability.The final contribution depends on both the local intervention and subsequent transformer blocks.Source: §A.2 · p. 15

A.3 INTERACTION BETWEEN FORGET AND RETAIN UPDATES

• Forget and retain objectives can interact differently across phase-wise updates, motivating a separate retain phase that restores retained behavior after forgetting updates.A sufficiently small retain-phase learning rate locally decreases the retain training objective; the local-step analysis omits stochastic mini-batch variation.Source: §A.3 · p. 16

A.4 EFFECT OF NEGATIVE CROSS-ENTROPY ON FORGET TARGETS

• Negative cross-entropy lowers the supervised forget-target logit and redistributes probability toward competing outputs across supervised prompt and response tokens.Standard cross-entropy instead increases the target logit relative to competing logits for retained examples.Source: §A.4 · p. 17

B.1 CONFIGURATION SETTINGS

• The unlearning update uses low-rank adaptation with pretrained weights frozen, rank and scaling parameter set to 16, and adapter dropout of 0.05.Only adapter parameters are optimized.Source: §B.1 · p. 17

B.2 EVALUATION TASKS AND METRICS

• Evaluation covers unlearning efficacy, generalizability, and utility across classification, generation, and cloze tasks in multimodal and unimodal settings.The section introduces the tasks and metrics used in these evaluations.Source: §B.2 · p. 18

B.2.1 CLASSIFICATION TASK

• Classification presents an image, question, and target response, then measures accuracy by whether the model predicts correctly.The questions are multiple-choice and concern profile attributes.Source: §B.2.1 · p. 18

B.2.2 GENERATION TASK

• Generation evaluates free-form answers against GPT-4o-generated references using ROUGE-L to measure longest-common-subsequence coverage and word-order consistency.Reference answers are based on key profile attributes.Source: §B.2.2 · p. 18

B.2.3 CLOZE TASK

• Cloze prompts provide an individual’s name and mask profile attributes for sentence completion, scoring predictions against ground-truth labels by accuracy.Masked attributes can include residence, employment, and hobbies.Source: §B.2.3 · p. 18

B.3 DATASET CONSTRUCTION AND RESTRUCTURING

• MLLMU-Bench partitions fictitious and celebrity profiles into Forget, Test, Retain, and Real Celebrity sets to assess forgetting, generalization, and utility.Profiles have more than 14 question–answer pairs, and evaluation covers multimodal and text-only settings.Source: §B.3 · p. 19

B.4.1 GRADIENT DIFFERENCE (GA DIFF)

• GA Diff combines gradient ascent on forget examples with fine-tuning on retain examples using a joint loss.Its objective subtracts forget-set loss and adds retain-set loss.Source: §B.4.1 · p. 21

B.4.2 KL MINIMIZATION (KL MIN)

• KL Min maximizes loss on forget examples while minimizing prediction divergence between pre- and post-unlearning models on retain examples.Source: §B.4.2 · p. 21

B.4.3 NEGATIVE PREFERENCE OPTIMIZATION (NPO)

• NPO treats forget-set responses as undesired and penalizes their probability relative to a reference model trained on retain data.Source: §B.4.3 · p. 21

B.4.4 MMUNLEARNER (MMU)

• MMU adaptively selects parameters relevant to forgetting and restricts updates to reduce interference with retained data.Its objective applies a binary parameter mask to the forget loss while retaining the retain loss.Source: §B.4.4 · p. 21

C.1.1 VISUAL QA

• In the Visual-QA example, SIEVE no longer recovers the designated profession, while preserving the correct country response on a retain example.Several baselines still recover the forget answer or produce an incorrect or malformed retain response.Source: §C.1.1 · p. 21

C.1.2 TEXTUAL QA

• In Textual-QA, SIEVE gives a generic response rather than the designated health condition and preserves the retained height answer of 175 cm.The examples show the same selective behavior under textual access.Source: §C.1.2 · p. 22

C.1.3 RETAIN UTILITY ON CELEBRITY SET (REAL-SET)

• On the Real set, SIEVE retains celebrity answers about fashion campaigns and a Primetime Emmy Award, including a more specific award description.These examples provide qualitative support for Real-set measurements.Source: §C.1.3 · pp. 22–23

C.2.1 KEY VS VALUE SUPPRESSION

• Value suppression provides stronger preservation on most Retain and Real metrics than key suppression while maintaining comparable Forget performance.The comparison is on LLaVA-1.5-7B under Visual-QA.Source: Table 7 · p. 24, §C.2.1 · p. 24
• Value suppression reduces the content carried by a value vector, whereas key or query suppression changes attention routing and leaves that content available.The toy example illustrates this distinction but does not establish robustness to adversarial attention manipulation.Source: §C.2.1 · p. 24

C.2.2 RESULTS OF LARGER MODEL

• On Textual-QA, SIEVE has the lowest Forget classification and generation scores, 35.99 and 0.411, while retaining scores of 54.86, 0.532, and 17.53.The Retain scores are for classification, generation, and cloze, respectively.Source: §C.2.2 · pp. 24–25, Table 8 · p. 25
• On Visual-QA, SIEVE lowers Forget classification from 62.82 for Vanilla to 31.53, with best Forget generation and cloze scores of 0.437 and 5.76.Retain generation and cloze are 0.690 and 34.72, versus Vanilla at 0.693 and 34.72.Source: §C.2.2 · p. 24, Table 8 · p. 25
• SIEVE retains Real-set Visual-QA classification at 49.83 versus Vanilla's 50.78, while reducing Test cloze from 27.67 to 2.33.On Textual-QA, Real generation is 0.563 versus Vanilla's 0.558, while Test generation falls from 0.388 to 0.272.Source: §C.2.2 · p. 25, Table 8 · p. 25

C.2.3 EFFICIENCY ANALYSIS

• 11,091 seconds: SIEVE's total training time is comparable to GA Diff's 10,961 seconds and lower than KL Min, NPO, and MMU.The corresponding totals are 12,440, 17,817, and 39,504 seconds; SIEVE does not require the additional preprocessing attributed to the higher-cost methods.Source: §C.2.3 · p. 25, Figure 10 · p. 26

C.2.4 EFFECT ON ATTENTION-VALUE

• SIEVE substantially suppresses value activations for forget samples while retain-sample activations remain comparatively stable.The reported visualizations cover the last and penultimate layers; the text notes that direct value suppression limits information propagation through attention.Source: Figure 11 · p. 26, §C.2.4 · p. 26

C.2.5 ABLATION STUDIES ON CV

• At V = 0, forget scores are lowest in all six metrics, while retain scores are highest in five and Real scores in all six.The exception is Visual-QA retain classification, which falls to 31.03.Source: §C.2.5 · pp. 26–27, Figure 15 · p. 29

D MORE RELATED WORKS

• Prior unlearning approaches balance suppression of designated information with preservation of non-target behavior; SIEVE applies this objective to selective multimodal associations through attention-value representations.The discussion cites TOFU [Maini et al., 2024], NPO [Zhang et al., 2024], and RMU as examples of related approaches.Source: D MORE RELATED WORKS · p. 28

C.2.6 EFFECT ON ATTENTION-MAP

• Figures 16 and 17 compare vanilla-model and SIEVE Pϕ values at the last and penultimate layers.The captions identify the compared models and layers, but provide no quantitative outcome.Source: Figure 16 · p. 30, Figure 17 · p. 30