PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn to Forget: Machine Unlearning via Neuron Masking

Yang Liu, Zhuo Ma, Ximeng Liu, Jian Liu, Zhongyuan Jiang, Jianfeng Ma, Philip Yu, Kui Ren

arXiv:2003.10933v3cs.LGcs.CRstat.ML

TL;DR

Neural networks may memorize sensitive user data that users cannot withdraw from trained models. The paper proposes Forsaken, a neuron-masking unlearning method, and forgetting rate, a membership-inference-based measure. Across eight standard datasets, it reports more than 90% average forgetting rate with less than 5% accuracy loss.

  • Problem

    Neural networks can memorize sensitive user data, while users may have no way to withdraw it from a trained model.

    Source: §I · p. 1

  • Method

    Forsaken stimulates neurons to unlearn selected memorization using generated mask gradients, while forgetting rate measures changes from memorized to unknown using membership inference.

    Source: §I · pp. 1–2

  • Results

    More than 90% average forgetting rate with less than 5% accuracy loss is reported across eight standard datasets.

    Source: §I · p. 2

  • Takeaways & Limitations

    Forsaken is described as more utility-friendly than summation-based unlearning and more efficient in time and storage than retraining-based unlearning.

    Source: §I · p. 1

Abstract

from arXiv · show

Nowadays, machine learning models, especially neural networks, become prevalent in many real-world applications.These models are trained based on a one-way trip from user data: as long as users contribute their data, there is no way to withdraw; and it is well-known that a neural network memorizes its training data. This contradicts the "right to be forgotten" clause of GDPR, potentially leading to law violations. To this end, machine unlearning becomes a popular research topic, which allows users to eliminate memorization of their private data from a trained machine learning model.In this paper, we propose the first uniform metric called for-getting rate to measure the effectiveness of a machine unlearning method. It is based on the concept of membership inference and describes the transformation rate of the eliminated data from "memorized" to "unknown" after conducting unlearning. We also propose a novel unlearning method calledForsaken. It is superior to previous work in either utility or efficiency (when achieving the same forgetting rate). We benchmark Forsaken with eight standard datasets to evaluate its performance. The experimental results show that it can achieve more than 90\% forgetting rate on average and only causeless than 5\% accuracy loss.

I. INTRODUCTION

Neural networks can retain sensitive user data even when training avoids direct access to it, while users may lack a way to withdraw that data. The paper introduces Forsaken and forgetting rate, alongside background on existing unlearning approaches.

A. Machine unlearning

  • Summation-based unlearning subtracts a sample’s contribution from stored summations, but can excessively remove unrelated memorization in adaptive neural networks.It is described as working well for non-adaptive models such as Naïve Bayes and C4.5.Source: §I · p. 1
  • SISA-style retraining records intermediate models and resumes from one before the forgotten sample’s contribution, trading storage for retraining time.The paper also notes that partitioning can leave constituent-model performance uncertain and still require substantial computation and storage.Source: §I · p. 1, A. Machine unlearning · p. 2

I. INTRODUCTION

  • Forsaken uses generated mask gradients to stimulate neurons to unlearn selected memorization, adjusting gradient magnitude to avoid unexpected unlearning.It is presented as more utility-friendly than summation-based methods and more efficient in time and storage than retraining-based methods.Source: §I · pp. 1–2
  • Forsaken can unlearn both out-of-distribution and in-distribution training data, with particular focus on out-of-distribution data that may create unintended memorization.The paper notes that such memorization may increase the possibility of private-information extraction.Source: §I · p. 2
  • The proposed forgetting rate uses membership inference to measure how often eliminated data changes from “memorized” to “unknown” after unlearning.The authors describe it as the first indicator directly usable for machine-unlearning evaluation.Source: §I · p. 2
  • More than 90% average forgetting rate with less than 5% accuracy loss is reported for Forsaken across eight standard datasets.These are the paper’s stated benchmark results.Source: §I · p. 2

A. Machine unlearning

  • The paper defines ideal unlearning as making forgotten samples non-members while leaving the model’s outputs on other training samples unchanged.It distinguishes in-distribution samples from out-of-distribution samples relative to the learning task’s desired distribution.Source: A. Machine unlearning · p. 2

B. Membership oracle

Membership oracles infer whether samples belong to a model’s training set, and this paper uses them to evaluate unlearning. Forgetting rate measures the change in oracle classifications for samples selected for forgetting.

B. Membership oracle

  • A membership oracle infers whether a sample was in the target model’s training set using confidence vectors and a binary classifier trained from shadow-model data.The paper uses such oracles to evaluate unlearning and defaults to classification models because existing membership oracles are limited to classification tasks.Source: B. Membership oracle · p. 3

III. FORGETTING RATE

  • Forgetting rate measures how many samples shift from membership to non-membership after unlearning, relative to samples initially identified as members.The paper says an effective method should have AF > BF when BT > 0; FR = 100% means all memorized samples are unlearned, while FR = 0% means none are.Source: §III · p. 3

IV. FORSAKEN

Forsaken uses mask gradients to reduce a neural network’s memorization of selected training samples, with a generator that adjusts their magnitude to limit unintended forgetting. Its design combines a posterior-matching loss with a dynamic penalty intended to protect model performance.

A. Intuition

  • Forsaken’s generator repeatedly produces mask gradients that act on neurons and adjusts their magnitude from the updated model state to avoid catastrophic forgetting.The approach is inspired by active forgetting; the authors also say it applies across neural-network architectures without changing the model architecture or standard training procedure.Source: A. Intuition · p. 3

IV. FORSAKEN

  • Forsaken is presented as more suitable for adaptive neural networks than summation-based unlearning, and more efficient in storage and time than retraining-based unlearning.The paper also claims smaller utility loss than summation-based methods and no obvious accuracy loss on the original model.Source: A. Intuition · p. 3

B. Forsaken in detail

  • At each iteration, Forsaken updates the model, measures posteriors on samples to forget, then optimizes the generator while holding the model’s original parameters fixed.The generator minimizes the distance between those posteriors and a non-member posterior distribution P.Source: B. Forsaken in detail · p. 3

1) Loss function of G:

  • The generator’s loss uses KL divergence to align forgotten-sample posteriors with non-member posteriors, plus L1 regularization to discourage over-unlearning.The paper says L1 is smoother than Lp regularization for p ≥ 2, which can block convergence; it uses L-BFGS for steadier convergence, though more slowly than first-order optimizers.Source: 1) Loss function of G · p. 4
  • A dynamic penalty weights parameter changes using gradients from D0, a very small set of training data (less than 1%) that should not be forgotten.The authors say larger partial gradients on D0 lead to more penalty, to mitigate performance degradation and catastrophic forgetting.Source: 1) Loss function of G · p. 4

2) Non-member data:

Forsaken targets non-member posterior distributions when unlearning OOD or ID data. It uses sampled real-world non-member outputs for OOD data and testing data for ID data; the paper notes that ID unlearning is slower and reduces accuracy more.

2) Non-member data:

  • Because ideal uniform OOD posteriors are rarely seen in practice, Forsaken uses sampled real-world non-member OOD posteriors as the target distribution P.The paper says this target resembles normal non-member OOD samples, increasing the adversary’s difficulty in identifying forgotten data and potentially mitigating catastrophic forgetting.Source: 2) Non-member data · pp. 4–5
  • For OOD unlearning, P can be formed by averaging model posteriors for publicly available data irrelevant to the task, such as alphabet images for digit recognition.For ID unlearning, the paper replaces those OOD samples with testing samples.Source: 2) Non-member data · p. 5
  • ID data are more task-related than OOD data, and the paper says their unlearning leads to more accuracy reduction and slower convergence.The authors state that non-member ID posteriors are usually closer to corresponding member posteriors.Source: 2) Non-member data · p. 5

C. How to apply Forsaken in applications

In the federated-learning example, a user can replace a normal gradient with a Forsaken mask gradient when withdrawing data, and the server aggregates it like other gradients. The evaluation section assesses Forsaken on both OOD and ID data.

C. How to apply Forsaken in applications

  • In federated learning, a withdrawing user computes a mask gradient locally and sends a scaled version to the server instead of a normal gradient.The server accumulates mask gradients with the other users’ gradients; the paper states they have the same size and a similar value range to normal gradients.Source: C. How to apply Forsaken in applications · p. 5
  • Users can complete private-data unlearning in one iteration, and the paper says it is hard to infer which user performs unlearning.The algorithm’s inputs include the user’s local dataset, current model, learning rate, loss function, and maximum unlearning iterations.Source: C. How to apply Forsaken in applications · p. 5

V. EVALUATION

  • The evaluation examines Forsaken’s effectiveness and efficiency on both OOD and ID data.Source: §V · p. 5

A. Experimental Setup

The experiments compare Forsaken with other unlearning methods across OOD and ID data, evaluating forgetting, model utility, efficiency, and stability. Forsaken generally achieves forgetting rates close to or above full retraining while reducing accuracy and runtime costs relative to several alternatives.

A. Experimental Setup

  • Experiments cover eight datasets across image classification, sentiment analysis, and text categorization, with OOD data simulated using mislabeled, cross-task, or label-split samples.The default OOD unlearning set contains 200 samples; ID samples are randomly selected from the training set.Source: A. Experimental Setup · pp. 5–6

B. Performance Comparison

  • Most OOD samples were memorized despite being irrelevant to the original task, and most were correctly classified without obviously affecting target-model performance.The authors present this as evidence that unintended OOD memorization occurs in real applications.Source: B. Performance Comparison · p. 6
  • For both OOD and ID data, Forsaken generally achieves forgetting rates close to or higher than full retraining, while sometimes outperforming retraining-based methods at the cost of slightly more accuracy loss.Full retraining did not reach 100% forgetting rate in these experiments; ID data were harder to unlearn and caused more accuracy drops than OOD data.Source: B. Performance Comparison · pp. 6–7, Table III · p. 7, Table IV · p. 7
  • Except for full retraining, Forsaken has the lowest performance drop; its accuracy drop is higher for ID than OOD unlearning.SISA and SISA-DP reduce performance substantially, which the authors attribute to partitioning the training data.Source: Figure 2 · p. 7, Figure 3 · p. 7, B. Performance Comparison · p. 7
  • Forsaken’s runtime is hundreds of times lower than retraining-based methods and several times lower than SMU.The stated complexities are O(tn0) for Forsaken, O(KN) for full retraining, O(K(N − n0)) for SISA and SISA-DP, and O(N − n0) for SMU.Source: B. Performance Comparison · pp. 7–8
  • Across unlearning sizes of 100 to 300, Forsaken stays close to full retraining and outperforms other methods, with only modest reductions in forgetting rate and accuracy.The authors interpret this stability as comparable to retraining-based methods.Source: Figure 4 · p. 8, B. Performance Comparison · p. 8
  • With SISA shard counts increasing from 1 to 4, SISA and SISA-DP achieve similar forgetting rates to Forsaken but incur greater accuracy drops.The authors attribute this utility loss to dataset partitioning; Forsaken does not show that problem.Source: B. Performance Comparison · p. 8

C. Effect Factors

The experiments examine how generator iterations, penalty settings, and model overtraining affect Forsaken’s forgetting, utility, and runtime. More iterations and stronger penalties involve trade-offs, while deeper model memorization can give Forsaken more room to fine-tune.

C. Effect Factors

  • After tens of generator iterations, Forsaken commonly reaches its best performance; further iterations barely affect forgetting rate but reduce accuracy through over-unlearning of unrelated memorization.The generator’s running time increases linearly with iterations; 30 iterations took seconds even for a network with 14.09M parameters.Source: Figure 5 · p. 8, C. Effect Factors · p. 8, Table VII · p. 9
  • At λ = 0, Forsaken exceeds 97% FR but loses about 9% accuracy; weighted penalty reduces accuracy drop below 4% while maintaining the same FR.Both forgetting rate and accuracy drop decrease as λ increases; at λ = 100, unlearning is blocked, and the authors recommend high penalty when forgetting rate is not strictly constrained.Source: Table VIII · p. 9, C. Effect Factors · p. 9
  • Forsaken performs a little worse when the target model is less overtrained; deeper memorization leaves more room for unlearning fine-tuning.The experiment measures unlearning at different training stages and trains a new membership oracle for each stage.Source: Figure 6 · p. 9, C. Effect Factors · p. 9

D. Visualization

The visualizations use t-SNE to compare data attribution before and after OOD unlearning. Eliminated samples tend to shift from the training-sample group toward the non-member testing group.

D. Visualization

  • The visualization includes 100 training members, 100 testing non-members, and 100 OOD unlearning samples, projecting their top three posteriors into two dimensions and coloring them by attribution.The figures cover C10.T. and I.C.; t-SNE can change training and testing sample locations between visualizations because of randomization.Source: D. Visualization · pp. 9–10, Figure 7 · p. 10, Figure 8 · p. 10
  • Before unlearning, most eliminated samples lie near the training group; afterward, they tend toward the non-member testing group.The authors interpret this shift as changing from “memorized” to “unknown.”Source: Figure 7 · p. 10, Figure 8 · p. 10, D. Visualization · p. 10

E. White-box Membership Oracle

Forsaken is evaluated with white-box membership oracles, in poisoned-data correction, and for reducing unintended memorization exposure. The results report similar white- and black-box performance, improved recall in the poison setting, and lower exposure as forgetting increases.

E. White-box Membership Oracle

  • Forsaken achieves close performance with white- and black-box membership oracles, and its white-box visualizations move eliminated points from the member to non-member area.The visualization uses t-SNE embeddings; the cited figure captions identify the plotted attribution visualizations.Source: E. White-box Membership Oracle · p. 10, Table IX · p. 10, Figure 9 · p. 10

F. Poison Data

  • Forsaken improves recall but not accuracy in the poison-data setting, while SMU, SISA, and SISA-DP fail to improve performance.Poisoning causes about a 1% accuracy drop and an 8.4% recall-rate drop versus the original model; the poison rate is 2%, 4%, or 10%.Source: F. Poison Data · p. 11, Figure 11 · p. 11

G. Unintended Memorization Exposure

  • On Penn TreeBank with 100 canaries, exposure decreases as forgetting rate increases; exposure below 10 corresponds to negligible extraction probability according to.The experiment uses a two-layer LSTM; Fig. 12 compares forgetting rate and test loss across mask-gradient-generator training iterations.Source: G. Unintended Memorization Exposure · p. 11, Figure 12 · p. 11

H. Expansion to Other Models

Forsaken is tested on logistic regression and support vector machines, with image-dataset experiments omitted for those models. The paper also reviews related approaches and notes that it cannot provably hide unlearned data’s indirect footprint.

H. Expansion to Other Models

  • Forsaken shows satisfying forgetting-rate and accuracy-drop performance on logistic regression and support vector machines; image-dataset experiments are omitted for these models.The authors present this as evidence of robustness and stability across adaptive models.Source: H. Expansion to Other Models · p. 12, Table X · p. 12

VI. RELATED WORK

  • Related work spans membership oracles, summation- and retraining-based forgetting, memorization extraction, and privacy-preserving learning; Forsaken targets eliminating memorization rather than extracting it.The paper also distinguishes influence estimate and Cscore as indicators limited by computational complexity and inability to directly identify forgotten instances.Source: §VI · p. 12

VII. CONCLUSION

  • The conclusion presents Forsaken as neuron-based active forgetting and forgetting rate as a measure of memorized-to-unknown transformation, but offers no provable guarantee for hiding indirect footprints.The authors frame the work as a stepping stone for further machine-unlearning research.Source: §VII · pp. 12–13

APPENDIX

The appendix explains forgetting rate and catastrophic forgetting rate, reports Forsaken’s CFR comparison, and describes the eight benchmark datasets and their roles.

A. Forgetting Rate

  • Forgetting rate subtracts samples already classified as unknown before unlearning; the example yields an 89% successful-unlearning rate.The example has 10 of 100 samples initially classified as false and 90 after unlearning.Source: A. Forgetting Rate · p. 14

B. Catastrophic Forgetting Rate

  • Forsaken achieves the lowest catastrophic forgetting rate in all reported experiments.The paper says a good unlearning method should have high forgetting rate and low catastrophic forgetting rate; CFR measures excessive loss of memorization in normal training samples.Source: B. Catastrophic Forgetting Rate · p. 14, Table XI · p. 14

C. Dataset

  • The eight datasets cover image classification, sentiment analysis, and text categorization; STL-10, TinyImage, and Customer Review provide OOD data.The dataset information is listed in Table XII.Source: C. Dataset · p. 15, Table XII · p. 15

D. Membership Inference

The appendix records membership-oracle metrics and model architectures, then reports results for a larger dataset and ID-data unlearning at different sizes. It also identifies a running-time analysis across unlearning sizes.

D. Membership Inference

  • Membership-oracle accuracy, precision, and recall are reported for each dataset, measured over the whole training and testing datasets.Table XIV contains the oracle metrics.Source: D. Membership Inference · p. 15, Table XIV

E. Neural Networks

F. Big Dataset

  • On TinyImage, Forsaken has similar forgetting-rate performance to SISA and SISA-DP, with runtime almost the same as CIFAR-10 and much faster than the other methods.The experiment uses 90 target categories, remaining categories as OOD data, and an unlearning size of 200.Source: F. Big Dataset · p. 15, Table XV · p. 15

G. ID Data Machine Unlearning with Different Unlearning Sizes

H. Efficiency Analysis

Related papers

Loading 2003.10933v3…