PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models

Kangwook Ko, Jaehyuk Jang, Wonjun Lee, Hee-Seon Kim, Changick Kim

arXiv:2608.30649v1cs.CLcs.CV

TL;DR

MLLM identity unlearning must remove personal information after deployment without relying on a retain set or damaging shared visual-language computation. The paper localizes identity knowledge to early-to-mid decoder MLPs and introduces PAVA, which uses forget images to anchor visual behavior. Across MLLMU-Bench and ReMem, PAVA provides the strongest forget–retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.

  • Problem

    Existing MLLM identity-unlearning methods commonly rely on retain sets that are difficult to obtain after deployment and raise privacy concerns comparable to those motivating unlearning.

  • Method

    PAVA localizes updates to early-to-mid decoder MLPs identified by causal tracing, weight transplant, and Fisher overlap, while using a visual-attribute anchor distilled from forget images.

  • Results

    PAVA achieves the strongest forget–retain trade-off among forget-set-only methods across MLLMU-Bench and ReMem on two backbones and remains competitive with retain-based baselines.

  • Takeaways & Limitations

    Localization supplies the update target while visual-attribute anchoring constrains what the update preserves, allowing forgetting without collapse of perception and general ability in the evaluated setting.

  • Takeaways & Limitations

    The benchmarks use curated image–fact profiles, experiments cover only similar 7–8B open MLLMs, and relearning attacks do not certify removal.

Abstract

from arXiv · show

Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.

1 Introduction

MLLM identity unlearning must remove personal information after deployment without requiring a difficult-to-obtain retain set or damaging shared visual-language capability. PAVA addresses this by localizing updates to identity-related decoder MLPs and anchoring visual-attribute behavior using only forget images.

  • MLLMs can encode identifiable personal information, including links between faces and names and biographical facts, in their parameters.
  • Retain-based unlearning preserves utility but depends on a retain set that may be unavailable after deployment and recreate privacy concerns.
  • Forget-only updates can damage the visual and language machinery that should remain intact because identity knowledge is not first localized.
  • Causal tracing, weight transplant, and Fisher overlap identify early-to-mid decoder MLPs as a localized editing target with comparatively low interference to visual processing.
  • PAVA restricts updates to those layers and uses visual-attribute content in forget images to preserve image-grounded behavior without a separate retain set.
  • Across MLLMU-Bench and ReMem on two backbones, PAVA achieves the strongest forget–retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.

2 Related Work

Prior MLLM unlearning methods commonly center on retain sets, while related work also uses auxiliary references or precomputed statistics to reduce that dependence. This paper instead emphasizes localization-first editing using forget examples to identify where identity knowledge can be selectively modified.

  • Existing MLLM unlearning methods such as MANU, MMUnlearner, KVW, and MIP-Editor commonly use retain-set information for pruning, masking, weakening, or editing.
  • Retain-set dependence conflicts with post-deployment unlearning because sufficient retain data is hardest to obtain then and rebuilding it recreates privacy exposure.
  • Recent methods reduce retain-data reliance through auxiliary reference images or precomputed Fisher statistics.
  • The paper adopts a localization-first view, using forget examples to identify where identity knowledge can be selectively edited.
  • Causal tracing and activation patching identify components causally involved in behaviors, with prior language-model studies implicating middle-layer MLPs in factual recall.

3 Localizing Identity Knowledge in MLLMs

Three complementary analyses localize identity knowledge to modality-specific content-token MLPs, especially early-to-mid decoder layers, while indicating these layers offer lower-interference targets for visual preservation.

  • 3.3 Fisher Overlap Analysis: Together, the analyses identify early-to-mid decoder MLPs as localized intervention targets for identity unlearning with comparatively low visual collateral damage.The target combines causal retrieval, storage evidence, and Fisher-based interference analysis.
  • 3.1 Multimodal Causal Tracing: Causal tracing restores identity-conditioned answers most strongly through content-token MLPs: visual tokens in mid VQA layers and name spans in early QA layers.Attention effects peak at the final token in mid-to-late layers, consistent with aggregation for generation rather than primary retrieval.
  • 3.1 Multimodal Causal Tracing: VQA MLP indirect effects peak at L7–L12, whereas QA name processing concentrates in L0–L5, showing modality-specific retrieval bands.The two bands have little overlap because names act through surface form while faces require visual integration across image tokens.
  • 3.2 Weight Transplant: MLP weight transplantation restores about 80% of the both-module effect, compared with 58% for attention alone, indicating identity is stored chiefly in MLP parameters.Per-layer gains peak at L8–L15 and overlap the VQA retrieval peak at L7–L12.
  • 3.2 Weight Transplant: Activation intervention and weight transplantation converge on the same layers, supporting identity encoding rather than mere information transit.The transplant test addresses whether causally important layers store identity or only relay information encoded elsewhere.
  • 3.3 Fisher Overlap Analysis: Decoder MLPs have the lowest Fisher overlap between identity-knowledge and visual-attribute signals among tested module families, suggesting lower expected interference with visual processing.The vision encoder and projector show the highest overlap, while decoder MLPs are lower than attention and the other tested families.

4 Method

PAVA localizes retain-free identity unlearning to early-to-mid decoder MLPs and combines identity forgetting with a visual-attribute anchor derived from the forget images themselves. This design preserves image-grounded behavior while avoiding external retain examples.

  • Localization: Early-to-mid decoder MLPs are identified as an effective target because they store identity knowledge without substantially disturbing visual processing.The localization is based on causal tracing, weight transplant, and Fisher-overlap analyses.
  • Localization: PAVA confines training to the selected causally important MLP layers while keeping the remaining model parameters frozen.Layers are ranked by VQA indirect effect, and LoRA adapters implement the selected-layer updates.
  • Forgetting objective: NPO suppresses identity-conditioned answers by lowering their likelihood relative to the pre-unlearning model.This forget signal is intended to mitigate the instability associated with plain gradient ascent.
  • Visual-attribute anchoring: The visual-attribute anchor uses identity-agnostic questions about non-identity image content and labels them with the pre-unlearning model's answers.The resulting self-generated anchor set is used to preserve visual behavior on the forget images.
  • Evaluation: On MLLMU-Bench, PAVA is evaluated at a 5% forget ratio across LLaVA-1.5-7B and Qwen2.5-VL-7B using forget, retain, and real-celebrity splits.The table reports GPT-judged relevance and correctness, fill-in-the-blank accuracy, and ROUGE-L, with higher or lower being better as indicated.
  • Visual-attribute anchoring: Retention is sourced entirely from forget images, without external retain examples or manual labels.The anchor constrains the unlearned model to match the model's own pre-unlearning responses.

5 Experiments

Experiments show that PAVA achieves selective identity forgetting while preserving utility across benchmarks, backbones, transfer settings, sequential unlearning, and relearning tests. Its gains arise from localized layer updates paired with visual-attribute anchoring.

  • Main Results: On Qwen2.5-VL, PAVA reduces forget Correctness from 84 to 36 while retaining substantially more utility than GA and NPO and avoiding MANU’s retention collapse.Forget Correctness is evaluated alongside retain metrics such as ROUGE-L.
  • Main Results: PAVA lowers forget Correctness by 6 points versus KL on LLaVA-1.5 and by 36 points on Qwen2.5-VL while preserving most retain utility.Compared with GD, PAVA is comparable on LLaVA-1.5 and lower on Qwen2.5-VL for forget Correctness, although GD leads on some retain metrics.
  • Main Results: PAVA preserves answer relevance while forgetting identity, unlike GA and NPO, whose Relevance falls from 88 to 76 on LLaVA-1.5 as capability collapses.On ReMem, PAVA nearly matches GD across forget metrics while retaining comparable ROUGE-L and higher EMr.
  • Localization Informs Unlearning: Restricting NPO updates to IE-top MLPs improves forgetting and retention, while VAA enables deeper forgetting as forget Correctness falls from 50 to 36 with modest retain changes.Top-IE MLPs are the only tested targets combining substantial forgetting with high retention; bottom, random, and some alternative targets collapse retention or under-forget.
  • Anchoring Preserves Visual Grounding: PAVA preserves near-transfer grounding close to vanilla and degrades more gradually on far-transfer questions as forgetting deepens.Without the anchor, the gap reaches 7.0 points on near transfer and 6.5 on far transfer; retain-based GD performs substantially worse at matched forgetting.
  • Localized Updates Enable Sequential Unlearning: Pathway-aware updates avoid collateral damage during sequential QA and VQA unlearning, while VAA further improves VQA retention at the same forgetting level.The localized updates compose without the degradation observed under full-decoder NPO.
  • Resistance to Relearning: PAVA has the lowest overall recovery and smallest recovery at 20% and 30% relearning ratios, making forgotten identities harder to recover without collapsing answerability.GA’s apparent forgetting is followed by large recovery after partial re-exposure, whereas PAVA preserves answerability.

6 Conclusion

The conclusion frames retain-free identity unlearning as a problem of where to intervene. PAVA localizes edits to early-to-mid decoder MLPs and replaces a separate retain set with a visual-attribute anchor from forget images.

  • 6 Conclusion: Causal tracing, weight transplantation, and Fisher overlap converge on early-to-mid decoder MLPs as a localized identity-editing target with comparatively low interference to vision.PAVA confines updates to these layers and uses a visual-attribute anchor drawn from the forget images themselves.

7 Limitations

The evaluation is limited to curated synthetic identity profiles and a narrow set of similarly designed 7–8B open MLLMs. PAVA’s localization must therefore be re-identified for each model, and broader identity distributions remain untested.

  • Data and model scope: Curated image–fact profiles may produce sharper localization patterns than identities reinforced across redundant web data.Validation on in-the-wild identities is identified as an important next step.
  • Data and model scope: Experiments cover only 7–8B open MLLMs with similar designs.The paper does not establish behavior beyond this model family and scale.
  • Localization scope: IE-top layers must be re-identified for each model rather than transferred directly.Amortizing localization across related backbones is left as future work.

A.1 Benchmarks and Splits

MLLMU-Bench uses synthetic identity profiles and evaluates forgetting, generalization, retention, and real-celebrity preservation across visual and textual question-answering. The appendix also specifies the selected layers and tracing procedure used for analysis.

  • Benchmarks and splits: MLLMU-Bench evaluates forget, transformed-forget test, retain, and real-celebrity splits using VQA and QA tasks.Metrics include GPT-judged Relevance and Correctness, fill-in-the-blank accuracy, and ROUGE-L.
  • Layer selection: PAVA edits selected decoder MLP gate, up, and down projections while excluding layers 0 and 1.The default selection uses K = L/4, producing backbone-specific layer sets.
  • Training setup: Gradient-based baselines use LoRA across all LLM attention and MLP layers, whereas PAVA places adapters only on selected MLPs.Training uses five epochs, effective batch size four, and a fixed seed of 42.
  • Tracing procedure: Causal tracing corrupts either the target image or name, then restores clean activations for one component, layer, and token group.VQA uses image swaps; QA uses name swaps in the same prompt position.
  • Tracing procedure: Tracing reads intervention effects by restoring all positions in token groups rather than individual tokens.Groups include content tokens, subject-reference phrases, surrounding text, chat markers, and the generation-prefix token.

B GPT-Judged Evaluation

The evaluation separates answer relevance from factual correctness using a GPT-4o-mini judge. On the forget split, the target is relevant but factually nonrevealing output, and PAVA’s reported comparisons favor retention among retain-free methods.

  • Evaluation criteria: Correctness is positive when any specific factual element overlaps with the ground truth.Examples include shared cities, professions, names, institutions, countries, keywords, or approximate numbers.
  • Evaluation criteria: Relevance measures whether an answer addresses the question type, independently of factual correctness.The rubric distinguishes on-topic answers from incoherent or off-topic output.
  • Evaluation criteria: The forget-split target is high Relevance with low Correctness, indicating fluent answers that no longer reveal the fact.The complete judge prompt includes the rubric, nine examples, and output schema.
  • 10% forget evaluation: At the 10% forget ratio, PAVA keeps retention closest to vanilla among retain-free methods, while GA and NPO lose substantial utility.GA has no stable operating point at this ratio.
  • Cross-backbone evaluation: On Qwen3-VL-8B at matched forget Correctness, PAVA substantially improves all three retain metrics over NPO.GA forgets more deeply only as retention collapses, whereas GD and KL obtain high retention through retain-set supervision.
  • Forget–retain trajectories: PAVA retains more utility than GA and NPO at every forgetting level on both backbones.Its retention stays near vanilla on Qwen2.5-VL while GA and NPO decline; trajectory analysis avoids single-epoch threshold sensitivity.

C.4 Layer-Selection Sensitivity and Stability

Layer-selection analyses support a stable, low-interference decoder-MLP target. Moderate layer counts behave similarly, rankings remain stable across identities and forget ratios, and complementary analyses localize the relevant pathway to early-to-mid MLPs.

  • Selection rule: K = L/4 is used as the default selection after excluding layers 0 and 1.The sensitivity analysis tests whether this rule is brittle to the exact number of selected layers.
  • Sensitivity to K: At K = 3, forget Correctness remains 62.0, while K = 7 and K = 9 reach the same forget Correctness; K = 14 slightly reduces retention.The paper therefore treats K = L/4 as a stable operating point rather than a finely tuned constant.
  • Stability across identities: Layer rankings across four disjoint identity groups have Spearman correlation 0.947±0.014 and top-7 overlap 6.60 ± 0.52.The selected top-7 layers are identical for the default split, G1, G2, and G4; G3 differs by one layer.
  • Stability across forget ratios: The 5% and 10% forget splits select the same top-7 layers, with 7/7 overlap and Spearman correlation 0.970.Full-set tracing also matches with 7/7 overlap and correlation 0.978; removing the shared identity preserves the conclusion.
  • Cross-model localization: Across Qwen2.5-VL-7B and Qwen3-VL-8B, MLP effects concentrate at modality-specific content tokens while attention effects concentrate at the final token in mid-to-late layers.VQA content tokens are visual tokens, whereas QA content tokens are the name span.
  • Cross-method localization: MLP-only weight transplantation recovers most of the both-module effect, with per-layer gains concentrated in early-to-mid layers.Fisher overlap likewise finds decoder MLPs have the lowest identity–vision overlap among tested module families.

E License of Datasets and Models

The paper documents the licenses governing the datasets, pre-trained models, and baseline implementations used in the work. It states that all assets are used according to their respective licenses.

  • The paper summarizes the licenses of all datasets, pre-trained models, and baseline implementations.
  • All assets are used in accordance with their respective licenses.
  • Table 11 lists the licenses for the datasets, base models, and baseline methods used in the work.
Loading 2608.30649v1…