Source-linked AI summary
NoT: Federated Unlearning via Weight Negation
Yasser H. Khalil, Leo Brunswic, Soufiane Lamghari, Xu Li, Mahdi Beitollahi, Xi Chen
TL;DR
Federated unlearning must remove a participant’s contributions, but existing methods can require extra storage or access to target data. NoT uses weight negation and fine-tuning, and the paper reports strong unlearning and recovery across evaluated architectures and datasets, surpassing existing federated-unlearning baselines and traditional machine-unlearning methods.
Problem
Existing federated-unlearning methods can require costly retraining, auxiliary storage, or target data that may no longer be available or permissible to use.
Source: §1 · p. 1Method
NoT negates selected model-layer parameters to disrupt inter-layer co-adaptation, then fine-tunes on retained data without auxiliary storage or target-data access.
Source: §1 · p. 2Results
NoT shows strong unlearning and recovery across diverse architectures and datasets, surpassing existing federated-unlearning baselines and traditional machine-unlearning methods.
Source: §8 · p. 9Takeaways & Limitations
NoT supports client-wise, class-wise, and instance-wise forgetting without access to the data targeted for removal.
Source: §1 · p. 2NoT loses some useful knowledge and requires access to retained data for recovery; evaluation has so far covered only smaller vision-classification datasets, not larger datasets such as ImageNet.
Source: §7 · pp. 8–9
Abstract
from arXiv · showhide
Federated unlearning (FU) aims to remove a participant's data contributions from a trained federated learning (FL) model, ensuring privacy and regulatory compliance. Traditional FU methods often depend on auxiliary storage on either the client or server side or require direct access to the data targeted for removal-a dependency that may not be feasible if the data is no longer available. To overcome these limitations, we propose NoT, a novel and efficient FU algorithm based on weight negation (multiplying by -1), which circumvents the need for additional storage and access to the target data. We argue that effective and efficient unlearning can be achieved by perturbing model parameters away from the set of optimal parameters, yet being well-positioned for quick re-optimization. This technique, though seemingly contradictory, is theoretically grounded: we prove that the weight negation perturbation effectively disrupts inter-layer co-adaptation, inducing unlearning while preserving an approximate optimality property, thereby enabling rapid recovery. Experimental results across three datasets and three model architectures demonstrate that NoT significantly outperforms existing baselines in unlearning efficacy as well as in communication and computational efficiency.
1. Introduction
Federated unlearning removes data contributions from trained models for privacy and compliance, but existing approaches can require costly retraining, extra storage, or access to target data that may be unavailable. NoT uses weight negation within a framework that seeks strong unlearning perturbations while preserving conditions for efficient recovery.
1. Introduction
- NoT negates selected global-model layers to disrupt inter-layer co-adaptation, then fine-tunes on retained data to recover essential knowledge without auxiliary storage or target-data access.The Figure 1 example reports performance close to Retrain across accuracy metrics with minimal added communication and computation costs.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
- Prior federated-unlearning approaches store historical updates or modify gradients; these approaches can face privacy, storage, data-access, or computational constraints, while prior weight-perturbation work had not explored negation.The related-work discussion contrasts NoT with both update-storage and gradient-modification approaches, and with other weight-perturbation methods.Source: §2 · pp. 2–3
3. Preliminaries
- The unlearning target is a model trained on retained data that approximates the performance of retraining from scratch, which is computationally and communication-intensive.Each client’s target data is the subset requested for removal; its remaining data is retained data.Source: §3 · p. 3
4. Unlearning Framework
- The framework first perturbs a converged model and then fine-tunes from the perturbed parameters on retained data, seeking immediate unlearning and further parameter change without excessive fine-tuning.The perturbation should be strong enough to move parameters away from optimal configurations yet resilient enough to permit fast re-optimization.Source: §4 · p. 3
4.1. The Need for a Strong Perturbation
- The loss gap, |L_Dr(θ) − L_Du(θ)|, measures separation between retained- and target-data losses; the theorem links unlearning time to the target and training conditions.Natural forgetting from an already-converged model can be slow; the discussion identifies high loss, a large Hessian spectrum, or high stochasticity as factors supporting faster unlearning.Source: §4.1 · p. 3
4.2. What is a Resilient Perturbation?
- A resilient perturbation controls the Jacobian and preserves layer-wise optimality, conditions the framework associates with accelerated fine-tuning while maximizing loss.Layer-wise optimality means freezing a chosen layer, randomizing the others, and still obtaining an acceptable optimum through fine-tuning.Source: §4.2 · p. 4
5. NoT - The Unlearning Algorithm
NoT negates selected weights in a converged global model, then fine-tunes on retained data; its theory links disrupted co-adaptation to forgetting while preserving conditions for recovery. Experiments compare NoT across federated and centralized settings, baselines, and ablations.
5.1. Algorithm Overview
- NoT negates specified layers of the converged global model and fine-tunes on retained data, requiring neither auxiliary storage nor access to forgotten data.Negation is computationally negligible and the client only needs to signal an unlearning request.Source: §5.1 · p. 4
5.2. Negation as a Strong and Resilient Perturbation
- Under stated conditions, weight negation maximizes activation changes in perturbed layers; the theory links recovery to preserved layer-wise properties, with ReLU-like cases only conjectured to be approximately LWOP.The LWOP result requires the negated layers to form an antichain without maximal elements and to use specified activation functions; wide-network gradients may vanish for layers at or before the negated layers.Source: §5.2 · pp. 4–5
5.3. Selecting Layers for Negation
- Negating an early layer is generally preferred: it changes low-level features and prompts deeper parameter updates, while negating multiple layers can slow recovery or cancel out across convolution and normalization layers.The cited analysis attributes slower recovery to layers that must be fine-tuned despite likely small gradients.Source: §5.3 · p. 5
6. Experiments
- The evaluation benchmarks seven federated baselines across three datasets and architectures, covering random forgetting, backdoor mitigation, theory validation, centralized comparisons, and ablations.The experiments also examine layer choice, perturbation type, and the ratio of data to forget.Source: §6 · p. 5
6.1. Experimental Setup
- Experiments use CIFAR-10/100 and Caltech-101 with CNN, ResNet-18, and ViT-B/16 models; converged FedAvg models are unlearned with all clients participating each round.Data are IID unless stated otherwise; Caltech-101 is split into training and test sets before client partitioning.Source: §6.1 · p. 5
- Evaluation tracks retain, forget, and test accuracy, membership-inference success, communication and computation costs, and average gaps from Retrain, with lower gaps indicating better performance.Retain and test accuracy measure preserved knowledge; forget accuracy and MIA assess whether forgotten data remain recognizable.Source: §6.1 · pp. 5–6
6.2. Results
- Across federated forgetting settings and backdoor removal, NoT outperforms baselines; after fine-tuning it aligns with Retrain, and centrally it balances efficiency and accuracy without storage or target-data access.The reported settings include IID and non-IID client-wise, class-wise, instance-wise, and centralized unlearning.Source: §6.2 · pp. 7–8
6.3. Ablation Study
- Early-layer negation performs best among tested layers; weight negation beats other perturbations, and NoT closely matches Retrain across forget ratios while retaining low communication costs.Random initialization and noise perform similarly to unperturbed fine-tuning; FCU computation rises as the forget ratio increases.Source: §6.3 · p. 8
7. Limitations
NoT is reported to lose some useful knowledge, making retained data important for recovery. The reported evaluation scope is vision-centric classification on smaller datasets, and the supplied figure captions identify layer and perturbation comparisons.
7. Limitations
- NoT also loses some useful knowledge, so recovery requires access to retained data.That retained-data access is described as essential for optimal model performance.Source: §7 · pp. 8–9
- The centralized comparison uses randomly selected forget data comprising 10% of the training data.The caption says the best average gap is marked in red but provides no cell values.Source: Table 6 · p. 9
- Negating the ViT convolution projection layer produced the best unlearning performance among the tested layers.Source: Figure 4 · p. 9
- Weight negation was the best tested perturbation for inducing unlearning in the ViT convolution projection layer.Source: Figure 5 · p. 9
- NoT was evaluated only on smaller vision-classification datasets, not larger-scale datasets such as ImageNet [52].The cited examples are CIFAR-10/100 and Caltech-101.Source: §7 · p. 9
8. Conclusion
NoT uses layer-wise weight negation for federated unlearning without additional storage or access to target data. The conclusion reports strong unlearning and recovery across architectures and datasets, while Figure 6 reports performance across forget-data ratios.
8. Conclusion
- NoT requires neither additional storage nor access to target data and unlearns by negating specific layer-wise parameters.The conclusion describes this negation as disrupting co-adaptation across layers.Source: §8 · p. 9
- The proposed framework and empirical results support layer-wise negation as an unlearning operation, with the post-negation optimization state enabling rapid fine-tuning recovery.Source: §8 · p. 9
- NoT showed strong unlearning and recovery across diverse architectures and datasets, surpassing existing federated-unlearning baselines and traditional machine-unlearning methods.Source: §8 · p. 9
- Across different forget-data ratios, NoT attains the best unlearning with the least cost compared with baselines.Source: Figure 6 · p. 9
9. Co-Adaptation
Co-adaptation describes components that are adapted in relation to one another rather than in isolation. The paper applies this idea to neural-network layers, whose joint contribution to loss minimization motivates its layer-wise terminology.
9. Co-Adaptation
- The term co-adaptation originates in evolutionary biology, where entities adapt together within an evolving environment.Source: §9 · p. 13
- In computer science, co-adaptation is used for components considered in a system rather than as independently optimal units.Examples include network layers [53] and artificial neurons [24].Source: §9 · p. 13
- A neural-network layer is not meaningful as an independently optimal unit because it contributes to loss minimization jointly with other layers.Source: §9 · p. 13
- The paper calls layers co-adapted when their parameters are co-adapted, meaning the grafted model is optimal with respect to the loss.Source: §9 · p. 13
- Perturbing layers can preserve layer-wise optimality while creating a non-optimal grafting of multiple optimal models, causing the layers to lose co-adaptation.Source: §9 · p. 13
10. NoT Algorithm
The algorithm begins with a client unlearning request, then the server negates selected global-model layers. The remaining clients locally update the model during subsequent communication rounds.
10. NoT Algorithm
- Clients select target data and send an unlearning request to the server.Source: §10 · p. 13
- After receiving requests, the server negates parameters of selected layers in the global model.Source: §10 · p. 13
- Clients outside the target set run CLIENTUPDATE, performing local gradient-descent iterations on retained data and returning their updated model.Source: §10 · p. 13
11. PyTorch Code
The PyTorch code implements layer-wise parameter negation as NoT’s main operation and returns the resulting unlearned model.
11. PyTorch Code
- The code’s main component applies negation to a model’s layer-wise parameters.The implementation is described as applying layer-wise negation.Source: §11 · p. 13
- The function accepts a model and layer indices, defaulting to layer 0.The provided signature specifies layer_indicies=[0].Source: §11 · p. 13
- The function returns the model as the “unlearned model.”Source: §11 · p. 13
12. Extra Theoretical Results and Mathematical Proofs
The appendix gives theoretical results on unlearning time and activation-distance maximization. It outlines a stochastic-gradient proof and identifies an orthogonal activation replacement as optimal in the stated ReLU case.
12.1. Unlearning Time Lower Bound
- The unlearning-time theorem assumes training on retained data with Stochastic Gradient Langevin Dynamics from an initial parameter set.Its proof first treats deterministic gradient descent and then invokes Itô’s formula for the stochastic case.Source: §12.1 · p. 14
- The deterministic proof uses Cauchy–Schwarz in L2([0, t], Rn).Source: §12.1 · p. 14
12.2. Activation Distance Maximization
- For ReLU activations, the maximum distance is attained when the replacement activation is orthogonal to the original.The appendix describes maximizing distance while keeping activation norms approximately equal.Source: §12.2 · p. 14
2. Binary Step Function (σ : a 7→1a>0 + 1
These theoretical results examine activation perturbations, Jacobian bounds, layer-wise optimality, and affine compensation for negation. Their claims are conditional on activation and network assumptions, and low layer-wise optimality does not itself guarantee rapid convergence.
2. Binary Step Function (σ : a 7→1a>0 + 1
- For sigmoid-like and odd activations, negation maximizes activation distance under the stated conditions; for ReLU, the maximum occurs at an orthogonal replacement.For odd activations, the maximizing replacement is y = -y0.Source: §2 · p. 14
- Orthogonal post-activation transforms are generally infeasible when activation values contain a basis of the output space.Source: §2 · p. 15
- Weight randomization is expected to produce low-correlated layer outputs, but the text characterizes this perturbation approach as less effective.Source: §2 · p. 15
12.3. Jacobian Bound
- The Jacobian-bound results assume a Lipschitz feedforward network and a negated-layer set intersecting each maximal chain exactly once.The framework covers dense, split, and concatenation layers; most common architectures can be rewritten using dense layers with tied weights.Source: §12.3 · pp. 15–16
- Wasserstein bounds are also possible, but their constants depend on Lipschitz activation derivatives; without that smoothness, total variation is necessary.Source: §12.3 · p. 17
12.4. Layer-Wise Optimality
- Layer-wise optimality measures proximity to optimal layer parameters, but a low value does not guarantee rapid convergence.It implies the layer can be frozen while the rest of the model is trained from scratch and is expected to reduce gradient descent’s effective search dimension.Source: §12.4 · p. 18
12.4.1. Affine Compensation of Layer Negation.
- Negation is LWO-Lipschitz when the negated layers form an antichain with no maximal element and use sigmoid-like, odd, or even activations.The theorem’s activation examples include the binary step function, tanh, sin, and x^2.Source: §12.4.1 · p. 18
13. Empirical Support of Theoretical Analysis
The experiments support the theoretical account of NoT through unlearning-time estimates, CKA comparisons, a layer-wise optimality test, and gradient-spectrum analysis. They also identify limits of direct covariance-spectrum estimation for larger models.
13.1. Quantitative Unlearning Time Constraint
- The estimated unlearning-time lower bound is significantly below empirical results, while the predicted FT–NoT cost ratio is close to the experimental ratio.The stochasticity term is orders of magnitude smaller than the available loss decrease and can be neglected.Source: §13.1 · p. 19, Table 7 · p. 19
13.2. Further CKA Analysis
- NoT diverges more from FT in deeper layers before fine-tuning, then becomes CKA-similar to Retrain while remaining different from FT.The high CKA similarity between models with original versus negated first-layer weights supports effective pretraining after negation.Source: §13.2 · p. 20
13.3. Direct Test of Layer-wise Optimality
- NR-Freeze performs on par with Retrain after freezing negated layers, reinitializing the others, and fine-tuning, supporting layer-wise optimality.The test covers first-layer negation in CNN and ResNet-18, and convolutional-projection negation in ViT.Source: §13.3 · p. 20
13.4. Spectral Content of Gradient Covariance
- Direct empirical covariance-spectrum estimation is inconsistent in very high dimensions and inconclusive for ResNet and ViT, so the authors use an L∞ algorithm.The direct estimator is described as barely meaningful for shallow CNNs.Source: §13.4 · p. 20
14. Further Implementation Details
The implementation details specify federated training and unlearning settings across architectures, along with a centralized ResNet-18 setup. NoT negates selected layers, and its spectral content reaches 100% with fewer dimensions in the reported configurations.
14.1. Federated Learning
- All 10 clients participate in every communication round; NoT negates the first layer for all architectures, except EfficientNet, whose convolutional and dense layers with indices ≤150 are negated, while ViT negates its convolution projection layer.Communication rounds are 2000 for CNN on CIFAR-10 and CIFAR-100, 800 for ResNet-18 on CIFAR-10, 1000 for ResNet-18 on CIFAR-100, 300 for ViT on Caltech-101, and 500 for ViT on CIFAR-100.Source: §14.1 · pp. 20–22
- NoT reaches 100% spectral content with 20%–30% fewer dimensions across the reported CNN, ResNet-18, and ViT configurations.The configurations use different batch sizes and parameter subsets across the architectures.Source: Figure 8 · p. 22
14.2. Centralized Training
- In these experiments, ResNet-18 on CIFAR-10 is evaluated with a randomly selected 10% forget set, and NoT negates only the first-layer weights for its best performance.The base model uses the official CIFAR-10 training set, SGD with 0.9 momentum, a 0.01 learning rate, 5e-4 weight decay, and cosine annealing for 200 epochs.Source: §14.2 · p. 23