Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs

1 Introduction

• Existing protocols often test on the same data used for unlearning, risking overestimated effectiveness and instance-wise suppression instead of concept-level forgetting.This distinction matters when the same visual concept appears across images with different viewpoints, backgrounds, and styles.Source: §1 · p. 1
• Many benchmarks emphasize newly fine-tuned knowledge rather than real-world pre-trained knowledge, which is deeply entangled with useful knowledge.The authors argue that this setting may overestimate real-world unlearning effectiveness.Source: §1 · p. 1
• Using identical hyperparameters across methods can make effectiveness conclusions depend on the shared configuration rather than the methods.Different optimization objectives and training dynamics can make one shared configuration suboptimal for a method.Source: §1 · p. 1
• GEMMU evaluates heldout forget and retain sets, real-world pre-trained knowledge, and method-specific hyperparameter searches.The benchmark also includes modality gap analysis.Source: §1 · p. 2, Figure 1 · p. 2
• Re-evaluating nine representative methods with GEMMU uncovers substantial generalization gaps and ranking changes overlooked by previous benchmarks.The authors also describe GEMMU as a more rigorous benchmark and offer principles for future benchmark design.Source: §1 · p. 2

2 Related Works

• Most multimodal unlearning methods use reverse fine-tuning, suppressing forget-set knowledge with gradient ascent while preserving retained knowledge through gradient descent or regularization.Model editing is an alternative; MANU selectively identifies and prunes components associated with knowledge to forget [2025b].Source: §2 · p. 2

Machine Unlearning Evaluation for MLLMs

• Benchmarks expanded from TOFU’s fictitious knowledge setting to multimodal evaluations covering person profiles, cross-modal knowledge, and personal entities with event scenes.The cited benchmarks include MLLMU-Bench, CLEAR, FIUBench, UMU-Bench, and PEBench.Source: Machine Unlearning Evaluation for MLLMs · p. 2
• Existing benchmarks may rely on generated identities rather than real-world pre-trained knowledge, and some omit train–test splits for retain data.PULSE studies pre-trained knowledge but does not analyze the distinction from fine-tuned knowledge or address overfitting to forget or retain sets.Source: Machine Unlearning Evaluation for MLLMs · p. 3

3 Preliminaries

• The forget set contains knowledge to remove, while the retain set contains knowledge to preserve; both comprise image-question-answer triplets, with empty images for text-only questions.Both datasets are partitioned into disjoint training and test sets to evaluate generalizability.Source: §3 · p. 3
• An unlearning method updates the pre-trained model using the training splits, rather than requiring the entire forget and retain datasets.Previous works often use the entire datasets in place of these training splits.Source: §3 · p. 3
• Successful unlearning reduces recovery of forget-set knowledge and preserves retain-set performance on both training and test splits.The heldout test splits provide a more reliable measure of generalizable forgetting and knowledge retention.Source: §3 · p. 3

4.1 Testing on the Training Set Is Problematic for Concept-wise Unlearning

• Without train-test splits for both forget and retain sets, evaluation scores are unreliable; image-text unlearning also has a larger generalization gap than text-only unlearning.Visual concepts can appear across varied viewpoints, backgrounds, and styles, making heldout evaluation especially relevant to concept-wise forgetting.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2 Reliance on Synthetic, Fine-tuned Knowledge

• Fine-tuned knowledge drops sharply within the first epoch, while pre-trained knowledge decays more slowly on both models, consistent with its greater entanglement with useful knowledge.The comparison tracks forget-set performance over unlearning epochs on LLaVA-1.5-7B and Qwen3-VL-8B-Instruct.Source: §4.2 · p. 4, Figure 4 · p. 5

Proposition 1 (Forgetting–Retention Trade-off)

• Under the stated assumptions, reverting to original parameters can yield zero retention cost for fine-tuned knowledge, whereas successful pre-trained forgetting incurs strictly positive cost.The authors conclude that pre-trained knowledge is more challenging to unlearn because forgetting it necessarily costs other useful knowledge under those assumptions.Source: Proposition 1 (Forgetting–Retention… · p. 5

4.3 Hyperparameter Sensitivity

• At a learning rate of 10−5 RMU wins clearly; at 5 × 10−5 KL Min outperforms RMU by a wide margin under shared settings.The opposite rankings show that conclusions about which method is better depend on the shared configuration.Source: Figure 5 · p. 5, §4.3 · p. 5
• Across nine methods, per-method search moves the strongest methods below the indiscriminate-forgetting line; GADiff achieves the strongest overall balance between forgetting and retention.Many configurations without proper tuning lie on y = x.Source: §4.3 · p. 6, Table 2 · p. 6

5 GEMMU

• GEMMU combines real-world concept removal, heldout forget and retain evaluation, per-method hyperparameter search, and image-text versus text-only analysis.The benchmark is also intended to provide practical design guidelines for future evaluations.Source: §5 · p. 6

5.1 Benchmark Construction

• Real-world concepts span 139 identities, landmarks, car brands, and fruits; examples are split into disjoint training and test sets within each concept.The benchmark also includes two synthetic subsets, CLEAR and fake landmarks, each with 40 concepts; its train-test split is 5:10.Source: §5.1 · pp. 6–7

5.2 Participating MLLMs

• The experiments use Qwen3-VL-8B-Instruct and LLaVA-1.5-7B, with LoRA for both synthetic-knowledge fine-tuning and unlearning.The models are evaluated on image-text and text-only tasks, including classification, fill-in-the-blank, and generation, with paraphrased questions.Source: §5.2 · p. 7, §5.1 · p. 7

5.3 Machine Unlearning Methods

• Nine representative methods cover gradient-ascent, distillation, abstention-target, representation-editing, and pruning strategies.The methods include GA, GADiff, KL Min, LLMU, NPO, IDK, SCRUB, RMU, and MANU.Source: §5.3 · pp. 7–8

5.4 Re-benchmarking Unlearning Algorithms

• RMU mainly removes visual associations, whereas GADiff removes whole concepts at the cost of the largest retain-side drop.On Qwen3-VL-8B, RMU’s VQA classification falls to 2.48 while QA remains 83.91; its FITB scores are 1.66 versus 76.88.Source: §5.4 · p. 8
• GADiff and SCRUB are among GEMMU’s strongest methods, yet no method solves real-world pre-trained multimodal unlearning without under-forgetting, over-destruction, or retention loss.GADiff sacrifices retain classification, while SCRUB leaves textual forget knowledge largely intact.Source: §5.4 · p. 8

5.5 On Guidelines for Future Multimodal Unlearning Benchmarks

• Future benchmarks should use train-test splits, tune each method separately, and report both VQA and QA metrics to distinguish association removal from concept removal.Benchmarks should also avoid synthetic fine-tuned knowledge as a stand-in for real-world pre-trained knowledge.Source: §5.5 · p. 8

• GEMMU emphasizes generalization-aware evaluation and covers a broader, more diverse range of real-world concepts than prior benchmarks.Source: §6 · p. 8
• Experiments reveal substantial generalization gaps and challenges removing pre-trained knowledge that is deeply entangled with retained knowledge.Source: §6 · p. 8
• Proper hyperparameter tuning can alter the relative rankings of unlearning methods.Source: §6 · p. 8

• The study uses LoRA for both fine-tuning and unlearning, following common practice in multimodal unlearning.Source: Limitations · p. 9
• Full-model fine-tuning is outside the study’s experimental scope.Source: Limitations · p. 9
• The authors cite substantially higher computational cost and focus on settings feasible on an NVIDIA H100 GPU.Source: Limitations · p. 9

A Extended Results for Learning Dynamics

• Pre-trained knowledge is forgotten more slowly than fine-tuned knowledge across eight learning-based methods, although failures to forget or model collapse narrow the averaged gap.Source: A Extended Results for Learning Dynamics · p. 10, Figure 7 · p. 11

B.1 Analysis setup

• The analysis defines target and heldout knowledge scores and measures retention cost as the original model’s heldout score minus the updated model’s score.For fine-tuned knowledge, the original model already satisfies the forgetting criterion with zero retention cost; for pre-trained knowledge, the analysis establishes a positive lower bound under assumptions.Source: §B.1 · p. 10

B.2 Assumptions

• The pre-trained-knowledge result assumes the target concept is already present and requires a positive score reduction to meet the forgetting threshold.Source: §B.2 · p. 10

Assumption A1 (Lipschitz target score)

• Under the Lipschitz target-score assumption, reducing the target score by γ requires parameter displacement of at least γ/K.Source: Assumption A1 (Lipschitz target score) · p. 10

Assumption A2 (Limited local improvement)

• The limited-local-improvement assumption bounds possible retained-utility improvement near the original model by ε0 times the update’s displacement.It reflects the assumption that the pre-trained model is near-optimal on heldout concepts.Source: Assumption A2 (Limited local… · p. 10

Assumption A3 (Quadratic penalty)

• The quadratic-penalty assumption makes retained utility fall below its first-order approximation as displacement grows, yielding positive retention cost when forgetting requires sufficient movement.The stated condition is γ/K > 2ε0/µ.Source: Assumption A3 (Quadratic penalty) · p. 10

B.3 Proof

• For fine-tuned knowledge, reversing the update returns θu to θ0, satisfying forgetting with zero retention cost.Thus a zero-cost solution exists without Assumptions A1–A3.Source: §B.3 · p. 11
• For pretrained knowledge, every successful unlearning solution within Θ has a positive retention-cost bound under the stated conditions.The proof derives the bound by lower-bounding the required parameter displacement and then the retention cost.Source: §B.3 · pp. 11–12
• The positive bound is conditional: A1–A3 constrain target-score sensitivity, retained-utility gradients, and the quadratic retention penalty, with A3 acknowledged as strong.The result is not a universal guarantee of retention degradation.Source: §B.3 · p. 12

C.1 Hyperparameter and checkpoint selection

• The score equally weights classification accuracy, fill-in-the-blank accuracy, and ROUGE-L across image-text and text-only inputs, scaling all metrics to 0–100.The metric is described as higher-is-better.Source: §C.1 · p. 12
• Hyperparameters are tuned using training results only, selecting configurations by the average retain-minus-forget score across four real-world subsets and checkpoints by each subset’s best epoch.Synthetic subsets reuse the selected configuration but do not affect its selection.Source: §C.1 · p. 12
• With five training images per concept, the authors do not split training data again for validation; they note validation would be beneficial with larger training sets.This limits their selection procedure to the available training results.Source: §C.1 · p. 12
• Both model experiments use batch size 2, seed 42, eight epochs, and LoRA rank 16, scaling 16, and dropout 0.05.These settings apply to LLaVA-1.5-7B and Qwen3-VL-8B-Instruct.Source: §C.1 · p. 12

D More Examples on Hyperparameter Sensitivity

• Only lr = 2 × 10−4 succeeds for IDK on Qwen3-VL-8B Fruits VQA fill-in-the-blank; three lower rates leave forget-set accuracy essentially unchanged after eight epochs.Figure 8 shows the within-method sensitivity.Source: D More Examples on Hyperparameter… · p. 12, Figure 8 · p. 12

E.1 Representative Examples

• Figures 9–14 show the six subsets in a forget/retain × train/test layout, separating image-text VQA from text-only QA inputs.Only VQA includes the displayed image; QA names the concept in text and receives no image.Source: §E.1 · p. 12, Figure 9 · p. 14, Figure 10 · p. 15
• Examples probe associated knowledge through biographical details, literary works, landmark dates, brand history, and fruit attributes, using CLS, FITB, and GEN task types.The figures show selected combinations; questions and reference answers come from dataset rows.Source: §E.1 · p. 13, Figure 14 · p. 19
• Training and test examples use different images of the same concept and rephrased questions within each forget or retain row, for both VQA and QA.For classification, test distractors and option order may also differ.Source: §E.1 · p. 13
• Tables 4 and 5 present hyperparameter search spaces and selected values for Qwen3-VL-8B-Instruct and LLaVA-1.5-7B, respectively.Learning rates are swept with method-specific settings fixed, and method-specific settings are swept at a learning rate of 10−5.Source: Table 4 · p. 20, Table 5 · p. 20