Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy

1. Introduction

• Per-example U-MIAs show that inexact unlearning can leave high membership leakage, while commonly used population U-MIAs overestimate privacy protection across vision and language models.The evaluation spans ResNet-18 on CIFAR-10 and PaLM 2.Source: §1 · p. 2
• Many inexact unlearning algorithms increase privacy leakage for retain-set examples; two state-of-the-art algorithms do so for over half of the training datapoints not selected for unlearning.This creates tension between individual control through unlearning and population-level privacy.Source: §1 · p. 2
• Without a faithful measure of unlearning protection, progress is difficult to assess; the authors encourage formal adversary definitions and threat models.They also suggest deriving formal bounds on U-MIA success, as has been done for MIAs [37].Source: §1 · p. 2
• A per-example attacker tailors its membership decision to each sample, unlike a population attacker instantiated identically across examples.The paper adapts standard MIAs because unlearning is not equivalent to training on the example, the usual membership-inference comparison.Source: §1 · p. 2
• Unlearning algorithms can reduce vulnerability for some forget-set examples while increasing it for others, and examples are unlearned at different rates.Naively tailoring stopping criteria to examples does not improve privacy protection.Source: §1 · p. 3

2. Background on machine unlearning

• Exact unlearning offers guarantees of influence removal but can be costly or restricted; inexact methods are cheaper, yet generally lack rigorous, well-understood removal guarantees.Shard-based exact methods retrain only the subgroup containing the requested example, but assume enough data for accurate partition models and can incur substantial retraining or inference costs.Source: §2 · pp. 3–4

2.1. Terminology

• The original model is trained on D_train, split into a forget set D_f and retain set D_r; an unlearning algorithm takes model weights and forget-set examples and outputs new weights.The algorithm may also use side information, which varies by implementation.Source: §2.1 · p. 4

2.2. Unlearning goals

• Unlearning aims for efficiency versus retraining, efficacy versus retraining on the retain set, and accuracy comparable to the original model on test data.The cost requirement is informal, and the acceptable cost depends on the setting and privacy stakes.Source: §2.2 · p. 4
• Accuracy is a separate goal: the unlearned model should perform similarly to the original on test data, though unlearning may trade off with task accuracy.The extent of this trade-off depends on the setting.Source: §2.2 · pp. 4–5
• Efficacy lacks a standardized similarity measure, so evaluations often compare model predictions through membership inference, despite costs and metric limitations.Weight-space ℓ2 distance can be uninterpretable and sensitive to permutation or training randomness; KL-based comparisons can require many retrained and unlearned models.Source: §2.2 · pp. 4–5

3. How should we evaluate the privacy of machine unlearning?

• U-MIAs operationalize the comparison between unlearned and retrained model distributions by predicting whether an example was forgotten or unseen.The section treats harder correct membership prediction as indicating greater similarity between those model distributions.Source: §3 · p. 5

Definition 3.1. (Membership inference game on machine unlearning.)

• The unlearning membership-inference game lets an adversary distinguish an unlearned training example from an example not drawn from the training set, with worst-case leakage assessed for sampled points.The authors do not focus on worst-case dataset selection; they target points sampled from datasets with task utility.Source: Definition 3.1. (Membership inference… · pp. 5–6

3.1. MIAs against machine learning models

• Population MIAs train one shared attack classifier across examples, whereas per-example MIAs train a dedicated attacker for each example using shadow models that include or exclude it.The section describes LiRA as a per-example likelihood-ratio attack based on the two inclusion and exclusion worlds.Source: §3.1 · pp. 6–7

3.2. Adapting population MIAs for unlearning evaluation

• Prior population U-MIAs vary in their use of shadow models, attack classifiers, model outputs, and training or evaluation sets, but can give an overly optimistic impression of protection.Some reviewed attacks train classifiers or tune thresholds using forget-set and test-set outputs from the unlearned model.Source: §3.2 · pp. 7–8
• Population U-MIAs can overestimate privacy protection because examples differ in how easily they are learned and unlearned, motivating per-example attacks.The authors argue that an attacker tailoring predictions to each example is stronger than one aggregating information across examples.Source: §3.2 · pp. 7–8

3.3. U-LiRA: Adapting a per-example MIA to machine unlearning

• U-LiRA adapts LiRA by modeling examples included in training and then unlearned, and most analysis compares that world with examples never seen in training.The authors note a third world—examples retained after training—but set it aside for most analysis to align with prior unlearning work.Source: §3.3 · pp. 8–9

4.1. Benchmarked unlearning algorithms

• The benchmark covers layer-freezing and reinitialization methods, gradient-based baselines, ℓ1-sparse MU, and SCRUB.The vision-classifier methods include CF-k and EU-k, NegGrad, GradDesc, NegGrad+, and SCRUB.Source: §4.1 · p. 9

4.2. Training and unlearning details

• The experiments train 256 models on random half-splits of CIFAR-10’s 50,000-example training set (25,000 examples each), then unlearn 40 random forget-set selections per model, producing 10,240 unlearned models.For each unlearning algorithm, hyperparameters are selected by grid search.Source: §4.2 · p. 10

4.3. Unlearning membership inference attack details

• The baseline trains a population classifier on losses, while U-LiRA builds per-example in/out distributions from shadow models and evaluates predictions on separate target models.The baseline uses target-model outputs for its attack training and evaluation, yet U-LiRA generally outperforms it.Source: §4.3 · pp. 10–11

4.4. Results

• For SCRUB and SPARSITY, baseline attack accuracy falls below 55%, but U-LiRA’s advantage over random guessing is 2–3× larger than the baseline’s.U-LiRA also outperforms the baseline across the benchmarked unlearning algorithms.Source: §4.4 · pp. 11–12, Figure 1 · p. 12
• Unlearning generally reduces membership risk for many forget-set examples, but every method leaves a non-negligible subset with increased risk.CF-k’s CDF rises sharply around zero for the selected k, indicating relatively little improvement for most forget-set examples; other k values may behave differently.Source: §4.4 · p. 12
• For SCRUB, NegGrad+, CF-k, and SPARSITY, unlearning increases privacy leakage for a significant fraction of retained examples.With NegGrad+, the attack performs better after unlearning on nearly 70% of the retain set.Source: §4.4 · p. 13, Figure 3 · p. 13

Model and training details; Unlearning details; U-LiRA details

• The study fine-tunes PaLM 2 S on LIMA plus 300 handcrafted examples divided into random, PII, and poison subsets, then unlearns each subset with NegGrad or GradDesc.U-LiRA uses shadow models initialized from the pre-trained model and estimates per-example in/out distributions from losses.Source: §5 · p. 14

U-LiRA details

• Training enough shadow models for U-LiRA on a large language model is computationally expensive, but the relatively small fine-tuning dataset makes this evaluation feasible.Source: §5 · p. 14

Results

• With U-LiRA, neither NegGrad nor GradDesc successfully unlearns any samples, and attack accuracy remains above 80% except for random data.The baseline attack has substantially lower success, which could lead to a false conclusion of successful unlearning.Source: §5 · p. 15
• Unlearning behavior varies by data type: attack accuracy gradually decreases for PII and poison, while random-data unlearning is most successful after two epochs before accuracy rises quickly.Source: §5 · p. 15

• Examples become less vulnerable early in unlearning but more vulnerable again as unlearning progresses.Attack success depends heavily on the termination step; protecting against U-LiRA requires near-identical in- and out-shadow distributions for every forget-set example.Source: §6 · p. 15
• Optimization randomness and differing unlearning rates can expose individual examples even when an algorithm targets average-case success.U-LiRA constructs a separate decision rule for each example, creating tension with algorithms that calibrate across the forget set.Source: §6 · p. 15
• 20 of 100 target models gave the sample a membership probability below 60%, and U-LiRA misclassified it as a non-member in 10.The target models differed only in mini-batch data-order randomness.Source: §6 · pp. 15–16
• The U-LiRA-aware NegGrad variant filtered mini-batches using each example’s log probability relative to its shadow-out average and stopped when over half were excluded.The stopping rule was chosen because the loss difference could make membership inference trivial for most remaining examples.Source: §6 · p. 16
• Over 75% U-LiRA accuracy remained at termination, so the reported unlearning was not successful.The passage attributes continued decreases in many samples’ log probabilities partly to unlearning other examples.Source: §6 · p. 16

7. Discussion

• Inexact unlearning schemes can fail against per-example adversaries, and vulnerability often reflects imprecise schedules that are difficult to calibrate for each sample.The discussion identifies termination timing as a challenge for protecting individual samples.Source: §7 · p. 16
• Figure 5 shows well-separated shadow distributions alongside numerous target-model membership prediction errors for a specific CIFAR-10 example.The figure compares shadow-model logit distributions with predicted membership probabilities across target models.Source: Figure 5 · p. 17
• Examples can become more vulnerable again through over-unlearning, and mini-batch versus sequential updates complicate precise per-sample stopping.GradDesc unlearned random strings by epoch 2, but vulnerability rose from epoch 3 and reached pre-unlearning attack performance by epoch 10.Source: §7 · pp. 16–17
• Figure 6 orders the 5,000 class-5 CIFAR-10 examples by their average predicted membership probability and displays variation across target models.The forget set contains 200 examples; averages and standard deviations are computed over 100 target models.Source: Figure 6 · p. 17
• The discussion hypothesizes that existing schemes may struggle to provide equal guarantees across models and data because termination lacks a common hyperparameter setting.The authors connect this difficulty to different unlearning rates across examples.Source: §7 · p. 17
• Gaussian shadow-distribution assumptions can fit poorly after unlearning, so parametric hypothesis tests are not always an optimal attack choice.The paper illustrates the issue with toy and real examples and recommends considering how unlearning affects attack design.Source: §7 · pp. 17–18
• Gaussian-fit U-LiRA accuracy is approximately 58%, while KDE accuracy is nearly 80% in one example; for the CIFAR-10 example, Gaussian-fit accuracy is 50.1% versus 55.6% with KDE.These examples show that parametric assumptions can affect attack performance.Source: §7 · p. 18

8. Conclusion

• Population MIAs led prior work to overestimate inexact unlearning performance, while per-example U-LiRA showed that membership leakage remains high after unlearning.The conclusion calls for evaluating systems with adversaries appropriate to their sensitive-data setting.Source: §8 · p. 18
• Unlearning schemes should precisely define their threat model and state the guarantees they expect, which determines which adversaries to consider.The conclusion compares this need to adversarial robustness work.Source: §8 · pp. 18–19

A. SCRUB filtering

• Figure 9 reports average accuracy and shaded standard deviation across 10,240 models for the three sets after filtering out failures.The models run SCRUB on 200 samples randomly selected from CIFAR-10 class 5.Source: Figure 9 · p. 22, A. SCRUB filtering · p. 22
• SCRUB filtering retains runs with high retain accuracy, forget accuracy similar to forget-validation accuracy, and forget accuracy below retain accuracy.The forget-validation set is sampled from the test set with a distribution similar to the forget set.Source: A. SCRUB filtering · p. 22
• The observed accuracy dip is attributed to the change in learning objective between training and unlearning.Source: A. SCRUB filtering · p. 22

B. More background on CF-k, EU-k and SCRUB unlearning algorithms

• CF-k freezes the first k layers and fine-tunes the rest on the retain set; EU-k reinitializes unfrozen layers and retrains them on the retain set [10].EU-k remains inexact unless all layers are reinitialized, because frozen layers may retain forget-set information.Source: B. More background on CF-k, EU-k and… · p. 22
• SCRUB maximizes prediction KL divergence on the forget set and minimizes it on the retain set, also minimizing unlearned-model cross-entropy [24].The maximization and minimization steps alternate, with the maximization usually stopped earlier.Source: B. More background on CF-k, EU-k and… · p. 22
• Longer unlearning increases forget-set error, but high error is not always desirable because a retrained model may still perform well on that set.Source: B. More background on CF-k, EU-k and… · pp. 22–23
• SCRUB selects the checkpoint whose forget-set error is closest to validation error from a similarly distributed forget-validation set [24].This rewinding procedure yields SCRUB+R.Source: B. More background on CF-k, EU-k and… · p. 23

C. Unlearning hyperparameters

• Table 1 lists the hyperparameters used for each unlearning algorithm.Source: C. Unlearning hyperparameters · p. 23, Table 1 · p. 23
• The table also gives the search space considered to find each algorithm's optimal configuration.Source: C. Unlearning hyperparameters · p. 23
• For EU-k and CF-k, all layers except the last ResNet block are frozen.Source: Table 1 · p. 23

D.1. Choice of class in class unlearning

• Across class choices, SCRUB generally decreases membership risk, but some examples remain unlearned with predicted membership probability much larger than 1/2.The tested forget sets contain 200 examples from class 1 or class 9, alongside the earlier class-5 setting.Source: §D.1 · p. 23
• For each example in the chosen class, the study averages predicted membership probability over 100 target models that included it in the forget set.Source: §D.1 · p. 23
• Figure 10 plots predicted membership probability for all 5,000 examples of the chosen class when included in a forget set of 200.Source: Figure 10 · p. 24

D.2. Three-way hypothesis test

• For a specific CIFAR-10 example, Figure 11 plots U-LiRA shadow distributions before and after unlearning with SCRUB.Before unlearning, forget and retain distributions are identical and have little overlap with the non-member distribution; after unlearning, forget and out distributions overlap.Source: Figure 11 · p. 24, §D.2 · p. 24
• U-LiRA assigns each sample to forget, retain, or test membership by comparing its likelihood under three shadow distributions.Predicting forget as retain, or retain as forget, counts as a failure even though the example is still classified as a training member.Source: §D.2 · p. 24
• After unlearning, forget and test examples are confused more often, but retain examples remain more separable; adding the retain hypothesis does not make membership inference harder.The section reports no decrease in membership-inference accuracy under the three-way test.Source: §D.2 · pp. 24–25