Source-linked AI summary
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations
Aditya Golatkar, Alessandro Achille, Stefano Soatto
TL;DR
The paper addresses removing a selected training cohort's information from a trained deep network, including information accessible through black-box input-output observations. It introduces activation-based information bounds and an NTK-inspired deterministic-plus-noise scrubbing procedure, reporting uniformly better forgetting metrics than previous proposals in both white- and black-box settings.
Problem
The problem is to remove information about a forgotten cohort so it cannot be recovered from network weights or input-output observations.
Method
The paper derives black-box information bounds from final activations and uses NTK-inspired deterministic scrubbing with Fisher-shaped Gaussian noise.
Results
The proposed procedure works uniformly better than previous proposals on all introduced forgetting metrics in both white-box and black-box settings.
Takeaways & Limitations
Forgetting can be evaluated through observed activations, with bounded-query black-box bounds providing a sharper target than weight-based bounds in overparameterized networks.
Takeaways & Limitations
The method is not yet scalable to production-level models because naively computing and storing its projection matrix requires O(c^2|Dr|^2) memory and O(|w| · c^2|Dr|^2) time.
Abstract
from arXiv · showhide
We describe a procedure for removing dependency on a cohort of training data from a trained deep network that improves upon and generalizes previous methods to different readout functions and can be extended to ensure forgetting in the activations of the network. We introduce a new bound on how much information can be extracted per query about the forgotten cohort from a black-box network for which only the input-output behavior is observed. The proposed forgetting procedure has a deterministic part derived from the differential equations of a linearized version of the model, and a stochastic part that ensures information destruction by adding noise tailored to the geometry of the loss landscape. We exploit the connections between the activation and weight dynamics of a DNN inspired by Neural Tangent Kernels to compute the information in the activations.
1 Introduction
The paper studies how to remove information about a forgotten training cohort from trained networks, including information exposed through weights and black-box input-output queries. It proposes one-shot scrubbing methods and bounds for quantifying residual information, with NTK-inspired analysis improving forgetting across attack settings.
- Problem: The goal is to remove information about a cohort Df so attackers cannot recover it from model weights or queried inputs and outputs.The paper considers both white-box access to weights and the less-studied black-box setting.
- Black-box perspective: The paper reframes forgetting around final activations because different weights can produce identical input-output behavior.This targets information accessible through network behavior rather than the particular weight solution reached during training.
- Black-box perspective: New bounds quantify the average information an attacker can extract per query from scrubbed final activations.The bounds are intended to characterize black-box attacks directly.
- Results: Black-box bounds are reported as orders of magnitude smaller than corresponding white-box bounds at the same target accuracy.The paper presents this as a better forgetting-versus-error trade-off for bounded observations.
- Contributions: The proposed one-shot algorithms outperform the previous method for both white-box and black-box attacks on DNNs.They also generalize to different readout functions and use an NTK-inspired connection between activation and weight dynamics.
2 Related work
Related work positions forgetting as post-training removal of information about a selected subset, distinct from stronger privacy guarantees. It connects the paper to machine unlearning, membership inference, and NTK analyses of neural-network training dynamics.
- Differential privacy: Differential privacy seeks to prevent recovery of information about any particular training sample, whereas forgetting removes information about a selected subset after training.The stronger privacy requirement can cause significant accuracy loss in deep networks.
- Forgetting: Prior machine-unlearning work includes efficient forgetting for statistical-query learning, instantaneous data summarization with deletion, and engineering principles for data elimination.These works address restricted learning settings or general design frameworks.
- Attacks: Membership inference attacks estimate whether samples were used in training and therefore serve as a metric for forgetting quality.The paper constructs a black-box membership-inference attack using a shadow-model-training approach.
- Neural Tangent Kernel: NTK theory shows that linearized-network training dynamics increasingly approximate actual dynamics as network width grows.The paper uses this framework as an approximation during fine-tuning to derive an optimal scrubbing procedure.
3 Out of the box forgetting
The paper develops information-theoretic bounds for black-box forgetting by comparing scrubbed activations with a retain-only reference model, then uses Gaussian and NTK-inspired scrubbing to control residual information. The analysis highlights advantages in overparameterized models but also exposes sensitivity to adversarial queries and approximation assumptions.
- Information-theoretic formalism: The black-box analysis bounds information extracted when an attacker queries an image and observes the corresponding model output.This bound is then used to design a forgetting procedure.
- Information-theoretic formalism: The framework extends weight-based forgetting analysis to final activations and models an attacker reconstructing a function y of the forgotten data Df.The activation pipeline is represented through a Markov chain and data-processing inequalities.
- Information-theoretic formalism: For bounded queries, the activation-based information term can provide a sharper bound than the corresponding full-weight attack term.This central mutual-information term is difficult to compute directly, motivating a computable upper bound.
- Reference-model comparison: The computable bound compares scrubbed activations with those of a reference model trained on retain data Dr without the forgotten cohort.Matching the reference activations means they cannot contain information about Df under the stated comparison.
- Gaussian scrubbing: For Gaussian scrubbing, the activation distribution is approximated by locally linearizing the network around the scrubbed weights.The approximation assumes the noise covariance is relatively small.
- Bound behavior: The black-box bound scales with the number of query points rather than the number of weights, making it potentially much smaller for overparameterized DNNs under bounded queries.This is presented as avoiding the curse of dimensionality and is observed in experiments.
- Null-space effects: A large weight-space difference can coexist with a small activation-space difference when the difference lies in the network's weight null-space.This explains why the black-box bound can be tighter than the white-box bound in overparameterized models.
- Adversarial queries: Queries with small activation gradients can extract substantial information even when scrubbed and reference activations are close.The paper identifies adversarial queries exploiting differences between reference and scrubbed-model null spaces.
4 An NTK-inspired forgetting procedure
The procedure uses NTK-inspired linearized activation and weight dynamics to compute a one-shot transition from training on D to training on the retained data Dr. It combines a deterministic scrubbing shift with Fisher-shaped noise and handles overparameterized null spaces more effectively than Hessian-based approaches.
- Procedure: The proposed scrubbing procedure minimizes both white-box and black-box information bounds and is computed without approximations.It is evaluated against earlier approaches under commonly used forgetting metrics.
- Linearized dynamics: Linearizing activations around pretrained weights makes the network dynamics approximately linear during fine-tuning when weights move little.This NTK-inspired regime supports closed-form analysis of training and forgetting dynamics.
- Linearized dynamics: The method computes a one-shot forgetting vector that moves weights trained on D toward the weights obtained by training on Dr alone.The construction uses the linearized final activations and NTK dynamics.
- Scrubbing update: A projection removes the components of forgotten-sample gradients lying in the span of retained-sample gradients, while M re-weights directions before summation.This separates directions associated with the forgotten cohort from directions needed for retained data.
- Scrubbing update: The stochastic component adds n ∼ N(0, λF −1), with F computed from the Fisher Information Matrix at hNTK(w), to improve robustness to linearization errors.The noise is tailored to the loss-landscape geometry.
- Relation to Fisher forgetting: When the model is overparameterized, the Hessian is degenerate in the activation-preserving null space, whereas the NTK matrix gives the exact convergence point under the linearized model.This explains why the NTK-based construction extends the earlier Hessian or Fisher approach in wide networks.
5 Experiments
Experiments compare the proposed NTK-based scrubbing procedure with baselines across error, relearning, membership-inference, information, and activation readouts on CIFAR and Lacuna. The proposed method generally approaches the retain-only reference while limiting information about the forgotten cohort and preserving behavior on retained data.
- Experimental setup: The experiments use reduced CIFAR-10 and Lacuna-10 datasets with All-CNN and ResNet-18 models, pretrained on corresponding 100-class datasets before fine-tuning.Each reduced dataset contains 500 training and testing images and 125 validation images.
- Baselines: The comparison includes the original model, fine-tuning on Dr, Fisher forgetting, the proposed method, and a model trained from scratch on Dr as the gold reference.The gold reference has never seen Df.
- Error readouts: The proposed method outperforms Fisher forgetting on error readouts because it first moves w(D) toward w(Dr), reducing the noise needed to erase information about Df.Fisher forgetting incurs higher error on both Dr and Dtest at the same forgetting level.
- Relearn time: The proposed method outperforms all baselines on relearning time, which serves as a proxy for information remaining in the weights about Df.Relearning time is the first fine-tuning epoch at which the loss on Df crosses a specified threshold.
- Membership attacks: Membership-inference attack accuracy for the proposed method lies in the optimal green region, whereas fine-tuning does not forget Df and Fisher forgetting may produce a Streisand effect.The attack uses output-probability entropy to identify whether samples from Df were used for training.
- Activation closeness: Along the scrubbing direction, activation distance to the retain-only model decreases on Df while remaining nearly constant on Dr, reaching a minimum at the scrubbed model.Distance is measured using post-softmax L1 activation differences.
- Error-forgetting trade-off: Increasing Fisher-noise variance reduces information in weights and activations but increases test error; the black-box bound is tighter than the white-box bound at matched accuracy.Queries from Df leak more information than queries from retain or test data.
6 Discussion
The discussion frames forgetting as protection against information recovery while noting trade-offs in performance and computational scalability. It also identifies whether black-box and white-box attacks remain equivalent under unlimited observations as an open problem.
- Forgetting can be achieved trivially with a random generator or retraining, but these alternatives respectively sacrifice performance or increase training-time complexity.
- The proposed NTK-based scrubbing procedure minimizes both white-box and black-box information bounds, with the latter targeting bounded-query attackers.
- Whether infinitely many black-box observations can reveal all weights remains open and is linked to a generalized Kalman Decomposition for deep networks.
- The method is not yet scalable to production because computing and storing matrix P naively requires O(c2|Dr|2) memory and O(|w| · c2|Dr|2) time.
Supplementary Material
The supplementary material provides implementation details, additional experiments across datasets and architectures, and proofs for the paper’s propositions and equations.
- Appendix A provides implementation details for the scrubbing procedure.
- Appendix B reports further experiments on CIFAR-10, Lacuna, TinyImageNet, AllCNN, and ResNet.
- Appendix C provides proofs for all propositions and equations in the paper.
A Experimental Details
The experiments use pretrained networks trained with SGD and modified weight decay, while adapting the NTK formulation to classification and evaluating practical linearization choices. The trapezium trick compensates for discrepancies between linearized dynamics and SGD.
- Experimental setup: Models are trained to convergence with SGD using learning rate η = 0.01, momentum m = 0.9, and weight decay λ = 0.1.
- Pre-training and weight-decay: Weight decay pulls weights toward the pretrained initialization w0 rather than zero, helping preserve the linear approximation neighborhood and simplifying activation-weight dynamics.
- NTK matrix: For classification with cross-entropy and weight decay, the NTK matrix incorporates the loss Hessian and the regularization term λI.
- Practical choices: Replacing the output Hessian H_fw0(x) with the identity matrix I works better in practice, possibly because softmax saturation makes the Hessian estimate inaccurate.
- Practical choices: Linearizing around w(D) provides a better estimate of the scrubbing direction than linearizing around w0.
- Trapezium trick: The trapezium trick renormalizes the linear-dynamics scrubbing vector to match real outputs without matching effective learning rates, while improving robustness to curvature-estimation errors.
B Additional Experiments
Additional experiments examine the geometry supporting linearization and report similar qualitative behavior across architectures and datasets. Figure 6 shows smooth loss landscapes and nearby training paths after pretraining.
- Figure 6: Figure 6 uses PCA to visualize the post-convergence loss landscape and training paths for models trained on D and Dr.
- Figure 6: The loss landscape around the pretraining point is smooth and almost convex, while the two training paths remain close.
- Figure 6: The observed geometry supports using a linearized approximation to compute the shift between the full-data and retain-data training paths.
- Additional experiments: Experiments on ResNet-18 and All-CNN across Lacuna, CIFAR-10, and TinyImageNet show similar qualitative behavior.
C Proofs
The proofs formalize information bounds for black-box observations and activations, then use NTK-based linearization to analyze Gaussian scrubbing across architectures and datasets.
- The Markov-chain formulation treats the retain set D_r as observed and the forgotten cohort D_f as hidden, with training producing weights w from D = D_r ⊔ D_f.
- Lemma 1 provides an upper bound on information about D_f extractable from a scrubbed network output f_S(w)(x).
- Figures 7 and 8 report qualitatively consistent behavior across ResNet-18 and All-CNN models on CIFAR, Lacuna, and TinyImageNet.
- The activation analysis linearizes scrubbed activations around h(w), modeling them as f_h(w)(x) plus Gaussian noise n(x) with covariance Σ.
Proof of Proposition 2.
Proposition 2 derives the optimal scrubbing shift for a linearized network by comparing solutions trained on the complete dataset and the retain set, using NTK matrix identities.
- The NTK linear approximation expresses post-fine-tuning activations as f_w(x) = f_0(x) + ∇_w f_0(x)(w − w_0).
- Under continuous gradient descent, the linearized training dynamics converge to the final solution, enabling analytical comparison of training on D and D_r.
- The NTK matrix Θ is formed from stacked output gradients, while Y and f_0(D) stack labels and initial outputs across the training samples.
- The optimal scrubbing vector is δw = w_lin(D_r) − w_lin(D), so adding it to weights trained on D yields weights equivalent to training on D_r alone in the linearized model.
- The derivation simplifies the scrubbing shift by expanding the inverse NTK matrix with block-matrix relations and the Woodbury Matrix Identity.