PaperlayerView exact revision on arXiv

Source-linked AI summary

Boundary Unlearning

Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, Chen Wang

arXiv:2303.11570v1cs.CV

TL;DR

Machine unlearning must remove obsolete, poisoned, or privacy-sensitive training data from DNNs, but parameter scrubbing is expensive. Boundary Unlearning shifts decision boundaries in a trained model to imitate retraining, using Boundary Shrink and Boundary Expanding for class-level forgetting with reported utility and privacy guarantees.

  • Problem

    Efficiently unlearning an entire DNN class is important for privacy and invalid data removal, while existing parameter-space approaches are prohibitively expensive.

  • Method

    Boundary Unlearning shifts a trained DNN’s decision boundary to imitate the decision behavior of a model retrained on remaining data, using Boundary Shrink and Boundary Expanding.

  • Results

    Boundary Unlearning rapidly and efficiently forgets an entire class while achieving utility and privacy guarantees in the reported experiments.

  • Takeaways & Limitations

    The method provides a practical class-level unlearning approach that avoids excessive computational cost and intervention in the original training pipeline.

  • Takeaways & Limitations

    The work primarily focuses on forgetting data consisting of all samples from an entire class.

Abstract

from arXiv · show

The practical needs of the ``right to be forgotten'' and poisoned data removal call for efficient \textit{machine unlearning} techniques, which enable machine learning models to unlearn, or to forget a fraction of training data and its lineage. Recent studies on machine unlearning for deep neural networks (DNNs) attempt to destroy the influence of the forgetting data by scrubbing the model parameters. However, it is prohibitively expensive due to the large dimension of the parameter space. In this paper, we refocus our attention from the parameter space to the decision space of the DNN model, and propose Boundary Unlearning, a rapid yet effective way to unlearn an entire class from a trained DNN model. The key idea is to shift the decision boundary of the original DNN model to imitate the decision behavior of the model retrained from scratch. We develop two novel boundary shift methods, namely Boundary Shrink and Boundary Expanding, both of which can rapidly achieve the utility and privacy guarantees. We extensively evaluate Boundary Unlearning on CIFAR-10 and Vggface2 datasets, and the results show that Boundary Unlearning can effectively forget the forgetting class on image classification and face recognition tasks, with an expected speed-up of $17\times$ and $19\times$, respectively, compared with retraining from the scratch.

1. Introduction

Boundary Unlearning targets efficient class-level machine unlearning by shifting a trained DNN’s decision boundary to imitate retraining on remaining data. It uses Boundary Shrink and Boundary Expanding to preserve remaining-data utility while forgetting the target class.

  • Motivation: Privacy regulations and poisoned, outdated, or erroneous training data motivate machine unlearning that removes data and its lineage from ML models.The paper highlights the right to be forgotten and poisoned-data removal as practical drivers.
  • Motivation: Existing DNN unlearning methods either accelerate retraining by modifying the training pipeline or scrub parameters, which is expensive in high-dimensional parameter space.Retraining from scratch is treated as the optimal unlearned model, but parameter-space approaches are prohibitively expensive.
  • Key observations: Retrained-model visualizations show forgetting samples spread through feature space and move mostly toward the borders of other class clusters.Figure 1 represents remaining samples as solid dots and forgotten samples as hollow circles.
  • Approach: Boundary Unlearning destroys the forgetting-class boundary while maintaining remaining-class boundaries, then pushes forgetting data toward other cluster borders.This links the decision-space observations to utility and privacy guarantees.
  • Approach: Boundary Shrink splits forgetting features among other classes, whereas Boundary Expanding remaps and prunes a shadow class assigned to forgetting data.Both methods shift the original boundary to imitate the retrained model’s decision behavior.
  • Evaluation: Experiments evaluate Boundary Unlearning on image classification and face recognition, reporting rapid and effective forgetting that outperforms four state-of-the-art techniques.The supplied passage does not provide dataset-specific speed values.

2. Related Work

Related DNN unlearning work primarily accelerates retraining or updates model parameters. These approaches face training-pipeline intervention, utility concerns, or high parameter-space cost.

  • Overview: DNN unlearning methods are broadly divided into retrain acceleration and parameter updating.Earlier theoretical approaches for convex models do not generalize to DNNs because of non-convex loss functions.
  • Retrain Acceleration: Retraining from scratch on remaining data is a naive but optimal unlearning method, although its training time and resource cost are high.SISA, gradient subtraction, Deltagrad, and federated-learning approaches attempt to accelerate retraining.
  • Retrain Acceleration: Retrain-acceleration methods intervene in the original training pipeline and may hurt model utility, limiting their fit for MLaaS platforms.The cited related-work passages describe this as a practical drawback.
  • Updating Parameters: Parameter-updating methods scrub DNN weights using information about the forgetting data, including Fisher Information Matrix guidance.The supplied passage introduces Fisher Forgetting but ends before its full description.

3. Preliminaries and Notation

The paper formalizes class-level unlearning for supervised DNN classification by updating an original model toward the model retrained on retained data. It defines forgetting data, remaining data, and class-pair decision boundaries.

  • Notation: The training set D contains labeled inputs and belongs to an input-label product space, while the label space contains K total classes.The notation uses D = {x_i, y_i} and labels 1 through K.
  • Notation: The forgetting set D_f is a subset of training data, the remaining set is D_r = D \ D_f, and this work primarily forgets all samples of one class.The remaining data contains information intended for retention.
  • Unlearning objective: Unlearning updates parameters from w_0 to w′ so the resulting model f_w′ resembles f_w∗, which is retrained on remaining data as the optimal unlearning model.The original model is trained on D, whereas the retrained reference uses D_r.
  • Decision boundary: A class-pair decision boundary B^(i,j) consists of inputs where the two class logits are equal and jointly maximal.The supplied definition is partially linearized but states the boundary’s equality-and-maximum condition.
  • Forgetting prediction: For a forgetting sample x_f from target class t, the retrained model determines its new prediction through the maximum-output class rule.The passage introduces x_f ∈ D_f and identifies t as the forgetting-class label.

4. Proposed Methods

Boundary Unlearning shifts decision boundaries rather than directly locating influential parameters. Boundary Shrink uses nearest incorrect labels, while Boundary Expanding remaps forgetting samples through a temporary shadow class before pruning it.

  • Boundary Shrink: Boundary Shrink identifies nearest but incorrect labels by crossing the nearest decision boundary with gradient-sign noise.The reassigned labels guide precise boundary shifting for forgetting samples.
  • Boundary Shrink: Finetuning with reassigned forgetting samples shrinks the forgetting-class boundary from the sides of other classes.This approach preserves the intended direction of boundary movement rather than assigning random labels.
  • Boundary Shrink: Boundary Shrink deactivates the model’s power on forgetting data while barely hurting generalization on remaining classes.It also pushes forgetting samples close to new boundaries, producing low-certainty predictions.
  • Boundary Expanding: Boundary Expanding assigns forgetting samples to an extra shadow class, remapping them into a newly expanded decision-space region.The region is created by adding a neuron to the original model’s last layer.
  • Boundary Expanding: Pruning the added neuron discards the expanded region, disables the forgetting-class neuron, and returns the model to its original size.Because samples are never assigned to remaining classes, their activations change little and utility is maintained.

5. Performance Evaluation

Boundary Unlearning is evaluated on CIFAR-10 and Vggface2 for utility, privacy, computational cost, attention, and decision-space behavior. It generally preserves remaining-class performance while removing forgetting-class information, closely matching retraining with substantially lower time costs.

  • Utility Guarantee: Boundary Shrink preserves remaining-data accuracy while reducing forgetting-data accuracy to 5.94% on CIFAR-10 and 1.54% on Vggface2.Remaining-data accuracy degrades by 0.73% and 1.37%, respectively.
  • Utility Guarantee: Boundary Expanding is faster but leaves more forgetting-class information than Boundary Shrink, preserving 8.96% accuracy on CIFAR-10.It remains below Random Labels’ 10.40% residual accuracy on CIFAR-10.
  • Privacy Guarantee: Boundary Shrink and Boundary Expanding achieve attack success rates close to the retrained model on both datasets, with Boundary Shrink performing better.Finetune remains high, while Negative Gradient is far from retraining on CIFAR-10.
  • Computational Complexity: 29.7× and 4.8× speed-ups are achieved by Boundary Expanding and Boundary Shrink on CIFAR-10, versus 26.6× and 11.7× on Vggface2.Boundary Shrink takes longer because it generates cross samples.
  • Visualization: Boundary Unlearning shifts attention from faces toward backgrounds and moves forgetting samples toward nearest remaining classes while maintaining remaining-class clusters.Boundary Shrink splits the forgetting-class region; Boundary Expanding pushes its cluster away from the center.
  • Output Entropy: Boundary Unlearning makes outputs on forgetting data more uniform and less confident by destroying the forgetting-class decision boundary.The resulting predictions move toward nearest classes, consistent with the decision-space visualizations.

6. Conclusion

Boundary Unlearning removes an entire class from a trained DNN by shifting its decision boundary, without excessive computational cost or intervention in the original training pipeline.

  • Boundary Unlearning removes an entire class from a trained DNN by shifting the decision boundary.The paper presents it as a machine unlearning methodology addressing both utility and privacy guarantees.
Loading 2303.11570v1…