PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Regression Unlearning

Ayush K Tarun, Vikram S Chundawat, Murari Mandal, Mohan Kankanhalli

arXiv:2210.08196v2cs.LG

TL;DR

Removing information on demand from machine-learning models matters because training data is encoded in model weights, while deep-regression unlearning remains underexplored. The paper proposes Blindspot and Gaussian methods, reports better baseline comparisons across four datasets and resistance to privacy attacks, and notes a forecasting-related limit on calculating AIN.

  • Problem

    Deep-regression unlearning remains underexplored, although model weights encode training information that may need to be removed on request.

    Source: §1 · p. 1

  • Method

    The paper proposes Blindspot, which combines a partially retain-exposed model with the original model, and a Gaussian Amnesiac learning method for regression unlearning.

    Source: §1 · p. 2

  • Results

    Across four datasets, the proposed methods outperform regression-unlearning baselines on representation- and ingrained-level metrics and resist membership-inference and inversion attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments report that the proposed methods are effective and generalize across different types of regression problems.

    Source: §7 · p. 9

    AIN could not be calculated for forecasting models because retrained models failed to reach the required performance even after lengthy training.

    Source: §6.4 · p. 9

Abstract

from arXiv · show

With the introduction of data protection and privacy regulations, it has become crucial to remove the lineage of data on demand from a machine learning (ML) model. In the last few years, there have been notable developments in machine unlearning to remove the information of certain training data efficiently and effectively from ML models. In this work, we explore unlearning for the regression problem, particularly in deep learning models. Unlearning in classification and simple linear regression has been considerably investigated. However, unlearning in deep regression models largely remains an untouched problem till now. In this work, we introduce deep regression unlearning methods that generalize well and are robust to privacy attacks. We propose the Blindspot unlearning method which uses a novel weight optimization process. A randomly initialized model, partially exposed to the retain samples and a copy of the original model are used together to selectively imprint knowledge about the data that we wish to keep and scrub off the information of the data we wish to forget. We also propose a Gaussian fine tuning method for regression unlearning. The existing unlearning metrics for classification are not directly applicable to regression unlearning. Therefore, we adapt these metrics for the regression setting. We conduct regression unlearning experiments for computer vision, natural language processing and forecasting applications. Our methods show excellent performance for all these datasets across all the metrics. Source code: https://github.com/ayu987/deep-regression-unlearning

1. Introduction

The introduction frames deletion from machine-learning models as a privacy challenge because training information is encoded in model weights, and identifies deep-regression unlearning as an unresolved gap. It presents two unlearning methods and reports benchmark performance and robustness to privacy attacks.

  • Because model weights encode training information, deletion requests require changing those weights; retraining from scratch is infeasible for large models because of resource and response-time costs.Source: §1 · pp. 1–2
  • Prior unlearning work primarily addressed classification, while deep regression and forecasting remained unexplored and classification metrics did not transfer directly to regression.Source: §1 · p. 2
  • Blindspot uses a partially retain-exposed, randomly initialized model alongside the original model to selectively retain knowledge and forget queried data through weight optimization.Source: §1 · p. 2
  • The paper also proposes Gaussian fine-tuning, described in its contribution list as Gaussian Amnesiac learning, for regression unlearning.Source: §1 · p. 2
  • Across AgeDB, IMDB-Wiki, STS-B (SemEval-2017), and UCI Electricity load, the proposed methods outperform regression-unlearning baselines on representation- and ingrained-level metrics.Source: §1 · p. 2
  • The proposed unlearning methods are reported to resist membership-inference and inversion attacks.Source: §1 · p. 2

2. Related Work

Related work covers exact and approximate unlearning across classification and regression, with deep-learning methods largely focused on classification. Earlier regression methods address linear or convex models, leaving deep regression as an identified gap.

  • Unlearning methods are categorized as exact removal or approximate approaches that seek outputs statistically indistinguishable from a model trained without the forget data.Source: §2 · p. 2
  • Deep unlearning work includes CNN and vision-transformer methods, while the reviewed methods target classification tasks.Source: §2 · p. 3
  • Prior regression work addressed linear or logistic models; existing methods are hard to apply to non-convex deep networks, motivating deep-regression methods and suitable evaluation metrics.Source: §2 · p. 3

3. Regression Unlearning

The section defines the regression unlearning setting and frames success as approximate agreement with a model retrained without the forget set. It also explains why regression makes unlearning and its evaluation challenging.

3.1. Preliminaries

  • The dataset is split into mutually exclusive forget and retain sets; a gold model is trained from scratch using only the retain set.The forget set contains queried points whose information should be removed.Source: §3.1 · p. 3

3.2. Problem Formulation

  • Approximate unlearning seeks output distributions close to those of the retrained model, rather than matching its parameters.Readout functions are used to assess the validity of this output-distribution criterion and the resulting unlearned parameters.Source: §3.2 · p. 4

3.3. Challenges in Regression Unlearning

  • Regression predicts real-valued expected outputs rather than class probabilities, making output distributions difficult to ascertain; regression inference attacks are also understudied.Source: §3.3 · p. 4

4. Proposed Deep Regression Unlearning

The paper proposes Blindspot Unlearning and Gaussian-Amnesiac Learning for deleting queried information from deep regression models. Blindspot uses a retain-trained helper to guide updates to the original model, while Gaussian-Amnesiac Learning fine-tunes on forget samples with Gaussian-sampled replacement labels.

4. Proposed Deep Regression Unlearning

  • The paper proposes Blindspot Unlearning and Gaussian-Amnesiac Learning to delete queried information from deep regression models.Source: §4 · p. 4

4.1. Blindspot Unlearning

  • Blindspot partially trains a randomly initialized helper on retain samples, then combines retain prediction, output similarity, and layer-activation losses to update the original model.On forget samples, the method makes the original model closer to the helper; the helper is frozen during unlearning, and the forget-set information in the result is bounded by that in the helper.Source: §4.1 · pp. 4–5

4.2. Gaussian-Amnesiac Learning

  • Gaussian-Amnesiac Learning replaces forget-set labels with samples from a Gaussian output distribution, then fine-tunes the original model on the resulting data.The authors report that replacing labels by uniform samples gives inferior results to Gaussian sampling.Source: §4.2 · p. 5

5. Evaluation Measures

The evaluation adapts unlearning measures to regression and assesses performance, relearning, output distributions, and privacy leakage. Together, these measures examine representation-level behavior, ingrained information, and inference-attack exposure.

  • Unlearning is evaluated across representation-level measures, ingrained-level measures, and privacy-leakage attacks.The evaluation includes errors and relearning time, weight or output-distribution analyses, and inference attacks.Source: §5 · p. 5
  • The membership attacker is an RBF-kernel support vector classifier using loss, penultimate-layer gradients, and activations; inversion optimizes a random image vector by gradient descent with mean squared error.Source: §5 · p. 6
  • AIN compares the steps for unlearned and retrained models to reach within α% of the original model's forget-set loss; values near 0 indicate suboptimal unlearning and near 1 adequate unlearning.Source: §5 · p. 6
  • Wasserstein distance compares retrained and unlearned forget-set output distributions, while lower density near zero in relative prediction deviations indicates better unlearning.Source: §5 · p. 6
  • Regression errors on forget and retain sets are compared with the corresponding retrained-model errors because common regression errors are unbounded.Source: §5 · p. 6

6. Experiments

Experiments span computer vision, NLP, and forecasting, comparing unlearning methods with retrained models across several metrics. Results vary by dataset and metric, with Blindspot and Gaussian Amnesiac close to retrained-model performance in reported cases.

6.1. Datasets

  • The four datasets were AgeDB and IMDB-Wiki for vision, STS-B for NLP, and UCI Electricity Load for forecasting.They comprise age-labelled images, sentence-pair similarity labels, and hourly electricity consumption from 370 customers.Source: §6.1 · p. 6

6.4. Results and Analysis

  • AgeDB’s 60–100 band forget-set error was 21.01 for Gaussian Amnesiac and 22.87 for the retrained model, while FineTune’s was 13.5.NegGrad’s error was 34.87; FineTune also had the worst Wasserstein distance in the 0–30 band.Source: Table 1 · p. 7, §6.4 · p. 8

6.2. Baselines and Models

  • The experiments used ResNet18 for vision, an LSTM with GLOVE embeddings for NLP, and a Temporal Fusion Transformer for forecasting.Blindspot used λ = 50 in all experiments, with dataset-specific training schedules and losses.Source: §6.2 · p. 7, §6.3 · p. 7

6.4. Results and Analysis

  • In IMDBWiki 0–30 age-band unlearning, membership attack probabilities were 0.26 for FineTune, 0.14 for Gaussian Amnesiac, and 0.07 for Blindspot.The proposed methods’ attack probabilities were reported as closer to the retrained model’s.Source: Table 2 · p. 7, §6.4 · p. 8
  • In STS-B, Blindspot was closest to the retrained model on 0–2 similarity-band forget-set error: 2.47 versus 2.75 for retraining.Blindspot also had the lowest attack probability across the reported unlearning cases and better AIN scores than other methods.Source: Table 3 · p. 8, §6.4 · p. 9
  • For Electricity Load values <= −0.85, Blindspot had the lowest Wasserstein distance, 0.13 versus 2.25 for Gaussian Amnesiac and 0.54 for FineTune.Results for values >= 0.85 were mixed, with outliers cited as the reason no method led across all metrics.Source: Table 4 · p. 8, §6.4 · p. 9
  • AgeDB density curves show Gaussian Amnesiac had the highest density near zero relative prediction difference from retraining in both age bands.Blindspot followed Gaussian Amnesiac; FineTune and the original model were farther from zero.Source: §6.4 · p. 8, Figure 2 · p. 9

7. Conclusion

The paper introduces Blindspot and Gaussian Amnesiac for deep regression unlearning and evaluates their generalization and privacy robustness. Across four datasets, the authors report strong performance on multiple unlearning metrics and better results than baseline methods.

7. Conclusion

  • Blindspot uses a partially trained model and a copy of the original model, with three losses guiding weight optimization to forget query samples.Gaussian Amnesiac is also proposed for deep regression unlearning.Source: §7 · p. 9
  • Across four datasets, the proposed methods are reported to outperform baselines on metrics for relearning effort, output distribution, and privacy attacks.The authors describe the methods as effective and generalizing across different regression problem types.Source: §7 · p. 9
  • The authors suggest that their insights and approaches may inspire future work on deep regression unlearning in other applications.Source: §7 · p. 9

A. Information Bound for Unlearning using Blindspot Method

The appendix relates residual information about forget data after Blindspot unlearning to the Wasserstein distance between blindspot and retrained-model predictions. Its bound decreases with the blindspot model’s training epochs.

A. Information Bound for Unlearning using Blindspot Method

  • Blindspot’s remaining information about the forget set is bounded in proportion to the Wasserstein distance between blindspot and retrained-model predictions on that set.The appendix states that this distance shrinks as the blindspot model is trained for more epochs.Source: A. Information Bound for Unlearning… · p. 12
  • The bound is kc/n, so the stated information bound decreases as the number of blindspot-model training epochs n increases.The experiments train retrained models for 100 epochs, while as few as 2 blindspot-model epochs are reported to yield very good unlearning.Source: A. Information Bound for Unlearning… · p. 12

B.1. Inversion Attacks

The experiments examine inversion and backdoor attacks, repeated forgetting requests, and Gaussian versus regular Amnesiac unlearning. They report attack resistance, retained performance across sequential requests, and stronger unlearning results for Gaussian Amnesiac than Reg Amnesiac.

B.1. Inversion Attacks

  • In MNIST inversion tests, the original model reconstruction shows class-3 edges, while retrained and Blindspot reconstructions show no recognizable pattern for forgotten classes 3 and 5.The paper reports the class-3 result in Figure 4 and the class-5 result in Figure 5.Source: §B.1 · p. 13, Figure 4 · p. 13, Figure 5 · p. 13

B.2. Backdoor Attacks

  • Blindspot reduces backdoor attack accuracy on patched MNIST samples from 98% in the poisoned model to 0.33%, versus 0% after retraining without those samples.The patched samples are unlearned after being used as a backdoor trigger.Source: §B.2 · p. 13

C. Sequential Unlearning Requests

D. Gaussian Amnesiac Vs Regular Amnesiac for Regression Unlearning

  • Gaussian Amnesiac more closely matches retrained predictions than Reg Amnesiac, with attack probability 0.13 vs 0.17 and Wasserstein distance 3.74 vs 5.20.For AgeDB 0–30 forgetting, Gaussian Amnesiac has the highest density around zero among the methods.Source: D. Gaussian Amnesiac Vs Regular… · p. 14

E. Viability of Blindspot in Classification Unlearning Tasks

The reported class-level results indicate that Blindspot leaves appreciable accuracy on the forgotten classes, suggesting room to improve its effectiveness for classification unlearning.

E. Viability of Blindspot in Classification Unlearning Tasks

  • Blindspot’s forgotten-class accuracy remains 10.5% for 1-class and 12.12% for 2-class unlearning, which the authors say should be closer to zero.The authors identify extending Blindspot for classification unlearning as future work.Source: E. Viability of Blindspot in… · p. 15

F. Efficiency Analysis

The efficiency analysis compares unlearning time with retraining, while additional experiments examine the influence of the λ setting and the amount of retain data on unlearning quality and the Streisand effect.

F. Efficiency Analysis

  • 27 seconds for Gaussian Amnesiac and 72 seconds for Blindspot, versus 1666 seconds for retraining on AgeDB; both proposed methods are over 20× faster on IMDBWiki.On AgeDB, fine tuning takes 84 seconds; Gaussian Amnesiac is over 60× faster than retraining, and Blindspot is over 20× faster.Source: F. Efficiency Analysis · p. 15

G.1. Effect of using different values of λ

  • λ values of 10–50 do not drastically influence performance, while results are perceptually bad at λ values 0, 5, or ≥75.The reported results cover AgeDB 0–30 forgetting.Source: §G.1 · p. 16, Table 6 · p. 16, Table 7 · p. 16

G.2. Effect of using different % of retain data in blindspot model

  • Increasing the amount of retain data lowers Wasserstein distance, while membership-attack probabilities of 0.0 indicate a Streisand effect except with 100% retain data.The experiments vary the retain-data proportion for AgeDB 0–30 forgetting.Source: §G.2 · p. 16

G.3. Training with different epochs

The additional experiments analyze training-epoch choices, results across further model architectures, and prediction-difference density curves. They report epoch-dependent trade-offs and find Blindspot often closest to retrained predictions in the tested model and dataset settings.

G.3. Training with different epochs

  • Training the Blindspot model for more epochs brings its prediction distribution closer to retraining; additional unlearning epochs help only when the Blindspot model is sufficiently trained.With one unlearning epoch, Wasserstein distance decreases through five Blindspot-training epochs and shows no significant further difference beyond five; with only two Blindspot-training epochs, more unlearning epochs increase distance.Source: §G.3 · pp. 16–17, Table 9 · p. 17

G.4. Results on multiple deep models per task

  • Blindspot outperforms Gaussian Amnesiac on AgeDB with AllCNN and MobileNetv3, and on STS-B with GRU and DNN.The authors say the AllCNN and MobileNetv3 results align with the ResNet18 results, and the STS-B comparison aligns with LSTM results.Source: §G.4 · p. 17, Table 10 · p. 18, Table 11 · p. 18

G.5. Additional analysis with density curves

  • Blindspot has the highest density around zero for STS-B and Electricity Load, while on IMDB-Wiki Gaussian Amnesiac leads for 0–30 forgetting and methods have similar curves for 60–100 forgetting.For random-sample forgetting on STS-B, the methods have similar density curves.Source: §G.5 · p. 17, Figure 10 · p. 18, Figure 9 · p. 18

Related papers

Loading 2210.08196v2…