Source-linked AI summary
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Aditya Golatkar, Alessandro Achille, Stefano Soatto
TL;DR
Selective forgetting seeks to remove a chosen training cohort’s information from a deep network’s weights, including against white-box probing. The paper proposes scrubbing based on SGD stability and compares scrubbed weights with models trained without the cohort, reporting matched behavior and a computable information bound.
Problem
Selective forgetting asks how to remove a chosen cohort’s information from deep-network weights while preserving retained knowledge and resisting white-box readout attacks.
Method
The paper scrubs trained weights without original training data or full retraining, using an indistinguishability definition, SGD-stability bounds, and transformations or noise guided by those bounds.
Results
Scrubbed models are compared with retrained models without the forgotten data; the procedure aligns weight distributions and can leave less than 1 NAT of accessible cohort information.
Takeaways & Limitations
Selective forgetting requires removing information from weights rather than merely hiding activations, while the proposed approach aims to preserve knowledge about retained data.
Takeaways & Limitations
The information analysis considers a worst-case attacker with full knowledge of training and arbitrarily complex readout functions, and assumes the forgotten cohort is small and training is stable.
Abstract
from arXiv · showhide
We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network. While the effects of the data to be forgotten can be hidden from the output of the network, insights may still be gleaned by probing deep into its weights. We propose a method for "scrubbing'" the weights clean of information about a particular set of training data. The method does not require retraining from scratch, nor access to the data originally used for training. Instead, the weights are modified so that any probing function of the weights is indistinguishable from the same function applied to the weights of a network trained without the data to be forgotten. This condition is a generalized and weaker form of Differential Privacy. Exploiting ideas related to the stability of stochastic gradient descent, we introduce an upper-bound on the amount of information remaining in the weights, which can be estimated efficiently even for deep neural networks.
1. Introduction
The paper frames selective forgetting as removing a chosen cohort’s information from deep-network weights while preserving retained knowledge and resisting white-box attacks. It positions this goal relative to differential privacy, memorization, membership attacks, and prior unlearning approaches.
- Selective forgetting asks whether a network can behave as if it never saw a chosen cohort without sabotaging information about retained data.
- The framework generalizes differential privacy by targeting a selected cohort rather than requiring protection for every individual sample.
- Prior membership and reconstruction attacks expose information leakage but do not provide a constructive way to forget data.
- The paper distinguishes its setting from catastrophic forgetting and earlier unlearning methods that do not apply to deep networks.
- The paper treats weights as carrying information about a dataset through a formalization compatible with its framework.
- White-box robustness is emphasized because attackers may probe model weights, not only black-box outputs, to extract information about forgotten data.
2. Definition and Testing of Forgetting
Selective forgetting is defined by making scrubbed weights indistinguishable from weights produced without the forgotten cohort, while retaining performance on the remaining data. The paper connects this objective to information-theoretic bounds and SGD stability, yielding tractable scrubbing procedures.
- Definition and Testing of Forgetting: A scrubbing procedure transforms weights trained on D = Df ⊔ Dr so readout functions cannot reconstruct information about Df.
- Definition and Testing of Forgetting: The certificate S0 provides a reference distribution independent of Df, making scrubbed weights indistinguishable from a model that never saw the forgotten data.
- Definition and Testing of Forgetting: If the forgetting quantity is zero, a readout function cannot predict better than chance whether the model was trained with or without Df, so membership attacks fail.
- Definition and Testing of Forgetting: Adding high-variance noise can achieve forgetting but may destroy model utility, motivating transformations that retain accuracy while removing cohort information.
- Definition and Testing of Forgetting: The Forgetting Lagrangian balances loss on retained data Dr against a KL-divergence term that measures residual information about Df.
- Stability and Local Forgetting Bound: Under a small forgotten cohort and stable training, the local forgetting bound replaces a complex global KL term with an efficiently estimable average over fixed random seeds.
- Empirical illustration: In logistic regression, the procedure aligned the trained and reference weight distributions with near-perfect overlap, while Figure 1 reports less than 1 NAT of accessible cohort information after forgetting.
- Stability and Local Forgetting Bound: The bound motivates bringing weights trained with D closer to weights trained with Dr, adding noise along their difference, or combining both strategies.
3. Optimal Quadratic Scrubbing Algorithm
Under a local quadratic and gradient-flow approximation, the paper derives a scrubbing transformation that maps a model trained with forget data to the corresponding model trained without it. A robust noisy variant relaxes these assumptions, trading residual information against retained-data accuracy.
- Figure 2 shows that increasing forgetting through λ increases test error on retained data.
- The quadratic analysis assumes both total and retained-data losses are quadratic and optimization follows gradient flow with random initialization.
- The derived transformation h(w) maps training trajectories on D to trajectories on Dr for every initialization and time, thereby scrubbing information about Df.
- After convergence, the transformation reduces to a simple Newton update.
- 3.1. Robust Scrubbing: Because quadratic losses and continuous gradient flow are unrealistic, the robust procedure adds Gaussian noise to compensate for approximation error.
- 3.1. Robust Scrubbing: In a nonquadratic logistic-regression experiment trained with SGD, scrubbing brought KL divergence close to zero despite violated assumptions.
4. Deep Network Scrubbing
For deep networks, the paper offers Fisher-based and variational forgetting variants. The Fisher approach injects noise using an approximation to the retained-data curvature, while variational forgetting optimizes noise directly against the Forgetting Lagrangian.
- The deep-network procedure has two variants: Fisher forgetting and variational forgetting.
- Fisher forgetting: Fisher forgetting approximates the Hessian with the Fisher Information Matrix, using diagonal or Kronecker-factorized approximations because the full matrix is too large to store.
- Fisher forgetting: The diagonal FIM is insufficient for a full Newton step but remains useful for adding noise.
- Fisher forgetting: The Fisher procedure relies on the original and target weights being close, a condition connected to the stability of SGD.
- Variational forgetting: Variational forgetting optimizes noise in the Forgetting Lagrangian directly and adds maximum noise across directions while minimizing the increase in loss.
5. Experiments
Experiments compare proposed scrubbing methods with retraining and intuitive baselines across image datasets and logistic regression. The results show that matching forgotten-set error is insufficient: effective forgetting must remove information from weights while preserving retain-set behavior.
- Experimental setup: Experiments cover MNIST, CIFAR-10, Lacuna-10, and logistic regression, including forgetting classes or 100-image subsets.Image models use All-CNN architectures with pretraining followed by fine-tuning; the logistic-regression study approximates weight distributions across 100 random seeds.
- Readout functions: Fisher and Variational scrubbing produce confidence distributions closer to retraining, whereas Random Labels can make forgotten samples more identifiable through degenerate entropy.The retrained model has lower confidence on Df than the original model, while the proposed methods move confidence toward that reference.
- Evaluation criteria: Retraining is the gold-standard target, because selective forgetting should reproduce a model trained without Df rather than merely achieve 100% error on Df.Forgetting a subset of a class yields 15.3% and 19.3% error on Df for the retrained reference, not 100%.
- Baseline comparisons: Fine-tuning on Dr performs poorly on errors for both Df and Dr, indicating catastrophic forgetting is not the correct solution to selective forgetting.
- Readout functions: Negative Gradient and Random Labels perform well on error readouts but relearn Df quickly, showing that little information is removed from the weights.Re-learn time measures how quickly training on forgotten data reaches a fixed loss threshold; slower recovery is preferred.
6. Discussion
The discussion frames selective forgetting as information removal from weights, connecting it to Differential Privacy and SGD stability. It also identifies realistic readout functions and the increasing difficulty of forgetting larger cohorts.
- Conceptual framing: The framework generalizes Differential Privacy by bounding information that weights retain about the forgotten cohort.The analysis uses Shannon Information but notes that its attacker model assumes full knowledge of training and arbitrarily complex readout functions.
- Open questions: Characterizing forgetting against viable subsets of realistic readout functions remains a promising research direction.
- Scope and difficulty: For a fixed λ, the upper bound on retained information increases with cohort size, so larger cohorts are harder to forget.
A. Additional experiments
Additional experiments test the approach beyond All-CNN models. ResNet-18 results include error and relearning evaluations and show that improper forgetting can also produce a Streisand effect.
- Loss-landscape analysis: The loss landscape between the original model and a ground-truth retrained model is locally convex along their connecting line, supporting the method’s hypotheses.The two models are close along this line, consistent with the stated stability of SGD.
- ResNet-18: ResNet-18 experiments report errors for forgetting techniques and relearn time for a forgotten class.
- ResNet-18: ResNets also exhibit a Streisand effect when an improper forgetting procedure is applied.
B. Implementation details
The experiments evaluate Fisher and Variational forgetting alongside several baselines, using weight perturbations, information bounds, relearning time, and loss-landscape analyses across architectures.
- Architectures and training: Deep-network experiments use All-CNN and reduced ResNet-18 models on Lacuna-10 and CIFAR-10 after pre-training on Lacuna-100 and CIFAR-100.No data augmentation is used.
- Baselines: The experiments compare original, retrained, fine-tuned, negative-gradient, random-label, hiding, Fisher, and Variational models.Fine-tuning and negative-gradient procedures run for 10 epochs with learning rate 0.01 and weight decay 0.0005.
- Scrubbing procedures: Fisher forgetting estimates noise from a positive semi-definite approximation to the Fisher Information Matrix.Variational forgetting instead solves a variational problem for the optimal scrubbing noise; both use λ = 5 · 10^-7.
- Information bound: The information bound applies the same scrubbing procedure to original and retrained models, then computes their KL divergence using a shared random seed.The reported experimental bound uses a single random seed, although the bound should generally be averaged over multiple seeds.
- Evaluation: Relearn-time is the first epoch in which the scrubbed model’s loss falls below a threshold during 50 epochs of retraining on the full dataset.The training uses a constant learning rate of 0.01.
- Additional analyses: For ResNet, forgetting difficulty is evaluated against cohort size, while loss and error are examined along the interpolation from the original to retrained model.The cohort-size experiment varies from one sample to all 404 samples of class ‘5’.
B.1. Pre-training improves the forgetting bound
The local forgetting bound assumes training stability, which may fail for large deep networks trained over many epochs. Pre-training reduces opportunities for path divergence and improves the bound, but excludes pre-training data from guaranteed forgetting.
- Stability limitation: The local forgetting bound may not hold when large deep networks are trained for many epochs because algorithmic stability is not guaranteed.Pre-training is proposed to give training paths a common starting configuration and reduce training time.
- Scope boundary: The bound cannot guarantee forgetting information contained in the pre-training dataset, so the forgetting set must be disjoint from it.The paper states that the resulting bound is greatly improved after pre-training.
B.2. Datasets
The paper evaluates forgetting on MNIST, CIFAR10, and the introduced Lacuna datasets, using celebrity-face data with separated training and test sets.
- Datasets: The experiments include MNIST, CIFAR10, Lacuna-10, and Lacuna-100.Lacuna-10 and Lacuna-100 are introduced in the paper.
- Lacuna construction: Lacuna-10 contains face images from 10 celebrities, with 100 test images per class and at least 500 sampled images per celebrity.Images are resized to 32x32, and the remaining images form the training set.
- Lacuna construction: Lacuna-100 samples 100 celebrities, has no overlap with Lacuna-10, and is used for pre-training before fine-tuning on Lacuna-10.The paper assumes the pre-training data does not need to be forgotten.
C. Proofs
The proofs formalize information-based selective forgetting, characterize training-path transformations under quadratic losses, and derive an optimal Gaussian scrubbing noise covariance. They also connect these results to error readouts and ResNet experiments.
- Information bounds: Proposition 1 bounds information about an attribute of the forgetting set through the information retained by a scrubbed model.The proof uses data processing and KL-divergence non-negativity.
- Information bounds: Proposition 2 represents stochastic training as a deterministic algorithm driven by Gaussian randomness and bounds the resulting scrubbed-model distributions.The training outcome is written as w = A(D, ϵ), with ϵ ∼ N(0, I).
- Training-path transformation: Under quadratic losses and gradient-flow training, Proposition 3 constructs h so that h(At(D, ϵ)) equals At(Dr, ϵ) for every initialization and time.The construction uses Hessians A and B for the full and retain-set losses.
- Optimal scrubbing: Proposition 4 shows that optimal Gaussian scrubbing adds noise after mapping weights through h, with covariance satisfying ΣBΣ = λΣh.This result assumes Gaussian transformation error and a locally quadratic retain-set loss.
- Optimal scrubbing: The proof derives the covariance condition by minimizing the Forgetting Lagrangian with respect to the added-noise covariance.The same optimality condition is stated directly in the derivation.
- Evaluation and definition: Selective forgetting is evaluated by comparing scrubbed models with gold-standard models trained without the forgotten data, including error readouts for ResNet.The framework defines scrubbing as transforming weights so readout functions cannot reconstruct information about the forgetting set.