PaperlayerView exact revision on arXiv

Source-linked AI summary

AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning

Wonjun Lee, Jaehyuk Jang, Kangwook Ko, Hee-Seon Kim, Changick Kim

arXiv:2608.28312v1cs.CVcs.CL

TL;DR

MLLMs may memorize identity-specific facts, while deletion-time unlearning often assumes retain data or supervision that may be unavailable. AIM analyzes separable identity and perception representations, then anchors and matches an identity-forgetting target through a Fisher-constrained vision update. Experiments report effective identity forgetting while preserving non-deleted identities, prior knowledge, and visual perception on the same images.

  • Problem

    MLLM identity unlearning must suppress deleted identities while preserving non-deleted identities and visual perception when retain data, original questions, or answers are unavailable at deletion time.

  • Method

    AIM learns a universal visual prompt to anchor an identity-forgetting target, then updates the vision encoder to match it under a cached Fisher-based retain constraint.

  • Results

    AIM achieves effective identity forgetting while competitively preserving retain and celebrity-prior performance, visual perception, and continual-unlearning trade-offs without retain data.

  • Takeaways & Limitations

    Identity knowledge can be targeted through the visual pathway while general visual perception on the same images remains comparatively intact.

  • Takeaways & Limitations

    AIM does not remove text-only identity knowledge, requires a precomputed full Fisher, and treats the retain Fisher as fixed during updating.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can memorize identity-specific facts about people in their fine-tuning data, creating privacy risks when a person requests deletion. Existing MLLM unlearning methods often assume access to retain images or ground-truth answers during deletion, which is unrealistic in many practical scenarios. We study identity unlearning when retain images are unavailable at deletion time. Our analysis shows that identity and visual-perception questions occupy distinct regions in fine-tuned hidden states and are organized differently: identity questions cluster by person, whereas perception questions cluster by question type. This suggests that identity knowledge can be suppressed without erasing general visual perception. Building on this observation, we propose AIM, a two-stage method that anchors an identity-forgetting target with a universal visual prompt and then matches the vision encoder to that target under a Fisher-based constraint. Extensive experiments show that AIM achieves competitive identity forgetting while preserving non-deleted identities, prior knowledge, and visual perception on the same images.

1 Introduction

MLLMs can memorize identity-specific facts, creating deletion-related privacy concerns, while existing unlearning methods often require unavailable retain data or supervision. AIM addresses this setting by analyzing selective representation structure and using a two-stage, Fisher-constrained vision-side intervention.

  • MLLM fine-tuning can memorize names, occupations, and affiliations tied to identifiable faces, creating privacy concerns when individuals request deletion.
  • Many existing methods require retain images, external data, or ground-truth answers, so this work studies identity unlearning without retain data at deletion time.
  • Identity and visual-perception questions occupy distinct hidden-state regions and follow different grouping rules, motivating selective identity intervention.
  • AIM first learns a universal visual prompt that anchors an identity-forgetting target, then matches target images to it through a Fisher-constrained vision-encoder update.
  • AIM achieves effective identity forgetting while competitively preserving retain identities, celebrity prior knowledge, and visual perception, including under continual deletion.

2 Related Work

Prior work spans classical objective-based unlearning, representation-level forgetting, modality-specific editing, and visual-side interventions. However, many approaches depend on extra supervision, retain-side signals, or perturbations that can confound forgetting with visual degradation.

  • Classical machine unlearning uses objectives including gradient ascent, gradient-difference training, KL regularization, and preference optimization.
  • Recent methods reduce supervision or data requirements through label-agnostic representations, sensitivity estimates, contribution suppression, and feature-level language-model unlearning.
  • Recent MLLM methods edit modality-relevant neurons, preserve image understanding, tune identity-encoded layers, or stabilize vision-language alignment.
  • Visual-side approaches include concept editing, token regularization, knowledge distillation, and input perturbations, but often rely on extra supervision, retain signals, or perturbations that may blur forgetting with visual degradation.

3 Analysis

The analysis finds that identity and perception responses are separable and organized differently in hidden-state space. A visual prompt transfers identity refusals to held-out questions while largely preserving perception answers on the same images.

  • The study represents each image-question input with the last-layer LLM hidden state at the final input-token position before answer generation.
  • The analysis compares MLLMU-Bench identity questions with constructed visual-perception questions using PCA and t-SNE visualizations.
  • Identity and visual-perception hidden states occupy distinct representation-space regions, a separation attributed to supervised fine-tuning rather than pretraining.
  • Identity questions cluster by person identity, whereas visual-perception questions cluster by question type across different images.
  • A visual prompt trained on 4 identity questions elicits refusals on 11 held-out identity questions while testing effects on 10 visual-perception questions from the same images.
  • For visual-perception questions, unconstrained VP IDK-Rate is 10.0% / 0% and Semantic Preservation is 63.3% / 85.2% on the two evaluated models.
  • These results motivate vision-side identity unlearning because identity responses can shift while same-image visual perception is largely preserved, with the language model fixed.

4 Method

AIM performs retain-data-free identity unlearning by first anchoring a forgetting target with a universal visual prompt, then matching forget-image features through a Fisher-constrained vision-encoder update. It uses only forget-identity information at deletion time while limiting drift affecting non-deleted identities.

  • Overview: AIM updates only the vision encoder, keeping the language model fixed while matching unprompted forget-image features to learned identity-forgetting targets.The method separates target definition from target matching.
  • 4.1 Problem Formulation: At deletion time, AIM uses forget images and constructed probing questions, but not retain images or original fine-tuning VQA pairs.The method also assumes access to attribute categories associated with each forget identity.
  • 4.2 Stage 1: Universal Visual Prompt Learning: Stage 1 learns a universal visual prompt T applied uniformly to forget images and optimized to anchor a fixed identity-forgetting target.The model parameters remain fixed while only T is optimized.
  • 4.2 Stage 1: Universal Visual Prompt Learning: The prompt-learning stage induces refusal responses for forget-image identity questions and penalizes prompt magnitude to reduce the later parameter drift needed to reproduce its displacement.The forget prompt set is built from constructed questions for forget identities.
  • 4.3 Stage 2: Fisher-Constrained Vision Encoder Update: Stage 2 updates the vision encoder so forget features reach the anchored target without applying T, while a Fisher-based retain-drift budget constrains each update.The Fisher statistic is aggregated at the identity level to address image-count imbalance.
  • 4.3 Stage 2: Fisher-Constrained Vision Encoder Update: AIM caches the full Fisher statistic after fine-tuning and computes only the forget-identity Fisher statistic during unlearning, eliminating the need for retain images, original questions, or ground-truth responses.Aligning forget-image feature displacements prevents gradient cancellation and makes the Stage 2 update well-posed.

5 Experiments

Experiments evaluate AIM across benchmarks, models, forget ratios, ablations, visual perception, robustness, and continual deletion under different data scopes. AIM preserves retain-related capabilities while achieving identity forgetting more reliably than several baselines, including settings where forget-only methods collapse.

  • Experimental setup: AIM is compared with retain-utilizing and forget-only baselines across MLLMU-Bench and ReMem, with metrics covering forgetting, retention, celebrity prior, exposure, and test performance.The experiments use LLaVA-1.5-7B and Qwen3-VL-8B across 5% and 10% forget splits.
  • Main results: AIM matches retain-utilizing baselines without retain data while keeping retain and celebrity-prior metrics close to vanilla on both models.On LLaVA, it matches or exceeds retain-utilizing methods such as MMUnlearner and MANU.
  • Main results: On ReMem, AIM reduces target-identity EM while narrowing the gap to retain-utilizing methods, but its Fisher constraint limits how deeply forgetting propagates into internal token distributions.EMt drops from vanilla’s 92.9/94.0 to 45.2/35.1; the remaining exposure gap is attributed to the retain-preservation constraint.
  • Main results: AIM remains stable across forget ratios and training iterations, whereas GA and NPO collapse and some retain-utilizing methods suffer severe retain degradation at 10%.The Fisher constraint bounds retain drift at every step.
  • Ablation and further analysis: AIM preserves visual perception on forget-identity images, producing responses closest to vanilla and remaining coherent, on-topic, and non-refusal.This evaluates perception responses after unlearning rather than only identity forgetting.
  • Ablation and further analysis: In continual unlearning, AIM remains effective on the 2→3 transition while preserving retain capacity, whereas other methods collapse at the second step.AIM reports EMf = 50.0, EMt = 56.3, ROUGE = 85.2, and EMr = 71.8 on that transition.
  • Ablation and further analysis: Removing any Stage 1 loss term or replacing the aligned target causes unbounded target drift, gradient cancellation, trivial solutions, or ineffective forgetting.The ablations show that CE, alignment, norm, and a coherent target each contribute to stable unlearning.
  • Ablation and further analysis: Under common image perturbations, forget ROUGE-L changes by at most 4.7 points and retain ROUGE-L decreases by at most 1.8 points relative to clean inputs.The stress test applies Gaussian noise, JPEG compression, horizontal flipping, and Gaussian blur.

6 Conclusion

The paper studies identity unlearning in MLLMs under a strict setting without retain images, original questions, or ground-truth answers. It analyzes distinct identity and perception representations and introduces AIM, whose experiments support selective, stable, and continual forgetting while preserving utility.

  • Conclusion: The strict setting uses only forget images, constructed questions, and a precomputed Fisher statistic, without retain images, original questions, or ground-truth answers.This frames deletion-time unlearning when retain data are unavailable.
  • Conclusion: AIM defines an IDK-inducing target and internalizes it through a Fisher-conditioned vision-encoder update, using the cached full Fisher for retain protection.The two-stage design separates target definition from target matching.
  • Conclusion: AIM matches or exceeds retain-utilizing baselines on key metrics while remaining robust across forget ratios, visual perception evaluations, and continual deletion scenarios.Ablations support the necessity of the loss components and two-stage design for stable unlearning.

Limitations

The paper’s strict setting has practical and methodological boundaries, including precomputed-information requirements, fixed-Fisher assumptions, small learning rates, and limited robustness coverage.

  • Scope boundaries: AIM does not remove identity knowledge in the text-only modality because it updates only the vision encoder while keeping the language model fixed.The paper suggests combining AIM with a complementary LLM-unlearning method for broader cross-modal deletion.
  • Information requirements: AIM requires a full Fisher precomputed during fine-tuning, so it cannot directly apply to models without that cached statistic.The Fisher is computed when retain images are implicitly accessible, despite their unavailability at deletion time.
  • Approximation assumptions: Treating the retain Fisher as fixed at its initial value may break down under longer schedules or larger update steps.The cached approximation relies on keeping parameters sufficiently close to their initial values.
  • Optimization constraints: AIM requires small learning rates, which may limit forget-signal propagation and contribute to its gap on ReMem’s Exposure metric.This limitation is associated with how deeply the forget signal propagates through the token distribution.
  • Robustness scope: The robustness evaluation covers tested image-side perturbations but not broader prompt-side, cross-modal, or adaptive attacks.The paper identifies more diverse attack settings as a direction for future work.

A.6 Why Two Stages? Limitations of Single-Stage Unlearning

The two-stage design avoids the instability of directly optimizing all three losses on the vision encoder. Single-stage optimization collapses utility because it lacks a fixed, specific forgetting destination.

  • Empirical failure: The single-stage variant collapses across retain and celebrity-prior metrics, and its apparent forgetting improvement reflects general model degradation.Table 9 compares single-stage joint updating with the two-stage IDK method on MLLMU-Bench at the 10% forget split.
  • Loss instability: Removing any auxiliary loss lets its corresponding loss grow, while omitting alignment and normalization causes Lnorm to explode and oscillate.The three loss terms pull the encoder in conflicting directions during single-stage training.
  • Loss instability: The total loss spikes at initialization before settling, and early retain-feature drift becomes irreversible because retain data provide no corrective signal.The Fisher constraint limits per-step parameter shifts but does not prevent the unstable early loss surface from causing permanent retain damage.
  • Two-stage rationale: Two-stage unlearning first constructs a specific forget target, then matches the vision encoder to that fixed target under the Fisher constraint.A fixed destination gives the constrained update a well-defined direction and convergence behavior.
  • Two-stage rationale: Single-stage optimization targets a broad region of jointly satisfying feature configurations, allowing constrained updates to reroute and conflating target construction with target matching.The paper attributes the optimization failure to this lack of target specificity.
  • Fisher support: Identity-balanced Fisher aggregation remains accurate when retain-side and forget-side image counts diverge, supporting retain protection without direct retain-image access.The decomposition shows high cosine similarity and modest relative ℓ2 error across tested forget ratios.

B.3 Gradient Conflict Without the Fisher Constraint

Forget and retain gradients agree within their own groups but conflict across groups, motivating Fisher-based retain protection. The representation analysis further shows that identity structure is fine-tuning-specific and selectively reduced after unlearning.

  • Gradient conflict: The across-group gradient cosine Gf–Gr is negative, with a layer-wise mean of approximately −0.12, and becomes more negative in later encoder layers.Within-group cosines Gf–Gf and Gr–Gr remain high, indicating coherent directions inside each group.
  • Gradient conflict: Forget-direction updates simultaneously move retain features in the opposite direction, creating irreversible retain drift without retain-side correction.The Fisher constraint bounds per-step retain drift using cached full-Fisher information despite unavailable retain images.
  • Representation structure: SFT increases centroid separation from ℓ2 distance 40.7 to 46.4 and raises | cos(PC1, d)| from 0.647 to 0.985.The Fisher criterion also increases from 1.52 to 19.38, showing that the separation becomes more dominant after SFT.
  • Representation structure: Identity-question clustering becomes sharply pronounced at the final layer, whereas visual-perception clustering remains low without a consistent late-layer increase.The analysis evaluates NMI and ARI at selected intermediate and final LLM layers.
  • Robustness: Across LLaVA full tuning, LLaVA LLM-only LoRA, and Qwen3-VL LLM-only LoRA, identity questions cluster by person while perception questions cluster by template.The pattern is stable across model family and tuning scope.
  • Generalization controls: On unseen ReMem images, visual-perception clustering remains comparable but identity clusters fail to form, indicating identity-by-image structure is fine-tuning-specific.This asymmetry identifies learned identity-bound representations as the target for identity unlearning.

C.5 Model type and tuning-recipe robustness

The clustering signature is robust across model families and tuning scopes, while unlearning selectively changes learned identity structure. Stage 1 transfer saturates with a small number of probing questions.

  • Model and tuning robustness: LLaVA full tuning, LLaVA LLM-only LoRA, and Qwen3-VL LLM-only LoRA all show identity-by-person and perception-by-template clustering.This establishes stability across both model family and tuning scope.
  • Model and tuning robustness: Pretrained models group subsets mainly by question template, whereas vanilla SFT develops strong image-identity clustering specifically for identity questions.Held-out identity and visual-perception questions remain similar to pretrained representations before unlearning.
  • Stage 1 design: VP-to-IDK, VP-to-Blank, and Encoder-to-IDK achieve comparable forget–retain trade-offs, supporting robustness to Stage 1 design choices.The comparison is conducted on LLaVA-1.5-7B with ReMem at the 10% forget split.
  • Post-unlearning selectivity: After unlearning, identity clustering falls and question-template grouping rises, while held-out identity and visual-perception representations remain unchanged.Unseen identity-question templates retain their question-template structure after unlearning.
  • Stage 1 transfer: The IDK elicitation rate saturates at around 8 training questions, which is adopted as the default for learning the visual prompt.The curve reports rates for questions used during prompt learning, held-out questions, and all identity questions.
  • Stage 1 transfer: On MLLMU-Bench close-up face images, the learned visual prompt acquires shapes resembling facial landmarks.This reflects the structure of the forget-set images used to optimize the prompt.

D.1 Related-Method Style Comparison

AIM is compared with related unlearning approaches under a setting where only forget images are available. Across datasets, models, resolutions, and utility tests, it maintains a competitive forgetting–retention balance while preserving visual utility.

  • Method comparison: AIM updates only the vision encoder and combines feature-level intervention with a Fisher-based constraint, unlike related methods that require retain images or forget-side question text.The comparison adapts vision-only and LLM-only unlearning directions to MLLMs while preserving the deletion-time constraint.
  • Training stability: Around epoch 45, AIM’s forget and retain metrics reach a stable plateau, whereas GA and NPO rapidly diverge and their retain and celebrity metrics collapse.AIM is not sensitive to iteration count in the trajectory shown for LLaVA-1.5-7B on ReMem at the 10% forget split.
  • Qualitative behavior: AIM produces fluent identity swaps on forget cases while usually recalling retain identities, whereas GA and NPO produce degenerate outputs.A representative retain failure remains, with AIM misidentifying one retain identity where GA_Diff and MMUnlearner answer correctly.
  • Robustness and utility: On Qwen3-VL-8B, native-resolution evaluation slightly lowers absolute scores but preserves the fixed-resolution forget–retain trade-off.AIM also largely preserves visual perception on forget-identity images and general VQA performance on 10K VQAv2 examples.

F License of Datasets and Models

The supplementary materials document licensing coverage for the datasets, pretrained models, and baseline implementations used in the work. Additional tables report backbone and VQAv2 evaluation results alongside this licensing information.

  • Licensing: The authors summarize the licenses of all datasets, pretrained models, and baseline implementations used in the work.They state that all assets are used in accordance with their respective licenses.
  • Additional results: Table 18 reports additional MLLMU-Bench results for LLaVA-1.5-13B and InternVL3-8B at the 10% forget split.The table caption states that AIM achieves a competitive forget–retain trade-off on both backbones.
  • Additional results: Table 19 reports accuracy after unlearning on 10K randomly sampled VQAv2 validation examples.The evaluation measures general visual utility beyond the identity-image tests.
Loading 2608.28312v1…