PaperlayerView exact revision on arXiv

Source-linked AI summary

Correlation-Guided Fast Machine Unlearning via Hessian Analysis

Ayushi Thakur, Ruchir Gupta, Amit Kumar Jaiswal, Prayag Tiwari

arXiv:2609.12620v1cs.LGcs.AI

TL;DR

Machine-learning security settings need selective data removal, but approximate unlearning can require costly repeated Hessian computations. This paper uses Pearson correlation and a closed-form update for related samples, reporting improved accuracy over state-of-the-art baselines.

  • Problem

    Security applications face poisoned training examples and privacy-driven removal requests, while repeated Hessian-inverse-vector computations make approximate unlearning costly.

    Source: §I · p. 1

  • Method

    The framework uses Pearson correlation to identify related samples and applies a closed-form update, with Hessian damping to address numerical instability.

    Source: §I · p. 2, §III · p. 4

  • Results

    Pearson correlation reports a 43.2% dominance rate for approximating sequential unlearning effects and 10^-2 improvement in accuracy over state-of-the-art baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported evaluation supports efficient, privacy-preserving unlearning without compromising efficacy.

    Source: §I · p. 2

Abstract

from arXiv · show

The increasing adoption of machine learning in network and distributed security systems has created an urgent need for mechanisms that can selectively and efficiently remove the influence of specific training data to eliminate compromised or adversarial data points from production models. Privacy regulations such as GDPR's \emph{right to be forgotten} also pose similar requirements. However, existing approximate unlearning techniques remain computationally prohibitive for deployment in real-world security systems, as they require repeated expensive Hessian-inverse-vector computations for each data point removal, creating a bottleneck when processing multiple related requests in scenarios such as intrusion detection systems, spam filters, and threat intelligence platforms. Thus, we introduce a computationally efficient unlearning framework that identifies correlated data points in the training set and applies a theoretically derived closed-form parameter update rule, achieving an $82\times$ wall-clock speedup over standard influence function unlearning while preserving model utility with a $10^{-2}$ improvement in accuracy over state-of-the-art baselines. Our method establishes theoretical guarantees and ensures numerical stability through Hessian damping. Our evaluation across seven diverse dataset architecture combinations, including large-scale CIFAR-100 with ResNet-50, demonstrates superior forgetting effectiveness, with membership inference attack success rates of 0.660 and tug-of-war scores of 0.950.

I. INTRODUCTION

The introduction presents correlation-guided unlearning as a way to reduce repeated computation for related removal requests, while the preliminaries define sequential removal and the related work contrasts exact and approximate approaches.

I. INTRODUCTION

  • The framework uses Pearson correlation to identify related samples and a Sherman-Morrison-derived closed-form update to avoid repeated Hessian-inverse-vector products.Source: §I · p. 2
  • High Pearson correlation is linked to gradient proportionality for quadratic loss, and the work reports polynomial error bounds and evaluation across seven datasets with varied architectures.Source: §I · p. 2
  • 43.2% dominance rate for approximating sequential unlearning effects and 10^-2 accuracy improvement over state-of-the-art baselines are reported for Pearson correlation.Source: §I · p. 2

A. Preliminaries

  • Sequential unlearning updates a model iteratively after each removal; the naive process recomputes a Hessian inverse, while Pearson α measures centered linear correlation between feature vectors.Source: A. Preliminaries · p. 2

II. RELATED WORK

  • Prior work spans exact retraining and SISA, approximate influence functions, and gradient reversal; provable guarantees may rely on impractical convexity and Hessian-Lipschitz assumptions.Source: §II · pp. 2–3

III. OUR METHOD

The method reuses a previously computed influence direction to approximately unlearn a similar point, avoiding a second Hessian-inverse-vector computation. Hessian damping and a Sherman–Morrison-based derivation provide a closed-form update intended to remain numerically stable.

III. OUR METHOD

  • For a similar point x after unlearning z, the method scales z’s previously computed influence direction instead of recomputing a Hessian-inverse-vector product.The approximation uses the relationship between the points’ influence directions.Source: §III · pp. 3, 5
  • Hessian damping regularizes H with λI to make it positive definite and well-conditioned for matrix inversion.The derivation uses the damped Hessian to support numerical stability.Source: §III · p. 4
  • The similarity assumption approximates x’s gradient as α times z’s gradient at the original optimum.The paper grounds this approximation in local smoothness properties of deep neural networks.Source: §III · p. 4
  • The derivation approximates the updated Hessian as a rank-1 modification, using a Gauss–Newton approximation of the per-sample Hessian.It approximates the Hessian at the new parameters by evaluating it at the original optimum.Source: §III · p. 4
  • The Sherman–Morrison formula is applied to the damped Hessian modification to avoid recomputing its inverse.The derivation sets both rank-1 vectors to the removed point’s gradient.Source: §III · p. 4
  • The closed-form scaling factor uses the damped self-influence score, and its denominator must remain nonzero for stability.The derivation gives a sufficient condition for avoiding a vanishing denominator by choosing damping appropriately.Source: §III · p. 5

IV. THEORETICAL GUARANTEES

The theoretical guarantees bound the parameter-update error of similarity-based unlearning. The supporting gradient approximation result is derived for quadratic loss under a linear relationship between the data points, while the final bounds invoke standardized-data assumptions.

IV. THEORETICAL GUARANTEES

  • The parameter-update error bound compares similarity-based unlearning with standard approximate unlearning using the damped Hessian.The theorem defines the respective parameter states after unlearning z and then x.Source: §IV · p. 5
  • For quadratic loss, the gradient-approximation bound assumes a linear relationship x = βz + c between the two data points.The bound depends on residuals, the scaling factors, and the norms of z and c.Source: §IV · p. 6
  • The stated residual and coefficient bounds rely on standardized inputs and targets, bounded parameter norm, and the listed bounds on α and feature dimension.The standardization assumption specifies k = 4.Source: §IV · p. 6

V. MAIN RESULT: FINAL ERROR BOUND

The final result gives a dimension-dependent bound on parameter-update error under the paper’s standardized-data assumptions. Its practical stability depends on the damped Hessian and a well-behaved scaling factor.

V. MAIN RESULT: FINAL ERROR BOUND

  • Theorem V.1 states a final parameter-update error bound under Assumption IV.3 with k = 4; for large feature dimension d, the derivation takes |β| ≈ 1 to identify dominant terms.Source: §V · pp. 6–7
  • The final bound incorporates the gradient-approximation error, which depends on residuals, scaling factors, and the offset between the data points.The result is obtained by substituting the preceding gradient-approximation bound and data-dependent bounds.Source: §V · p. 6
  • The bound’s stability depends on the condition of the damped Hessian and the scaling factor C.C is well-behaved when its denominator 1 − s(λ)_z does not vanish.Source: §V · p. 7

VI. ALGORITHM DESIGN

The algorithm first unlearns one point with a damped influence update, then estimates unlearning a similar point using a correlation-based scaling rule. It evaluates the approximation against the standard second update.

VI. ALGORITHM DESIGN

  • The algorithm computes a damped Hessian, selects similar point pairs, and first applies an influence-function update to z.It then computes the similarity and scaling factor to estimate the update for x.Source: §VI · p. 7
  • For x, the algorithm uses the previously computed displacement for z and scales it using Pearson correlation and z’s self-influence.This replaces the second Hessian-inverse-vector product.Source: §VI · p. 7
  • The algorithm evaluates both approaches and records the norm difference between their parameter updates to quantify approximation quality.It also evaluates their performance on the dataset.Source: §VI · p. 7

VII. EXPERIMENTS

Experiments compare similarity measures and unlearning costs across regression and classification settings. Pearson correlation leads the similarity comparison, while reusing updates for correlated points reduces sequential-unlearning time.

VII. EXPERIMENTS

  • The evaluation spans seven datasets across regression and classification, with architectures including MLPs, ResNet-18, and ResNet-50.CIFAR-100 uses ResNet-50; CIFAR-10 and LFW use ResNet-18.Source: §VII · pp. 7–9
  • Pearson correlation measures linear relationships between centered variables, while cosine similarity measures direction and projection-based similarity is magnitude-sensitive.The Diabetes comparison uses L2 regularization and randomly subsampled training-point pairs.Source: §VII · p. 8
  • Pearson correlation achieved the highest win rate, 43.2%, among three similarity measures over 4,950 Diabetes unlearning pairs.A measure wins when it produces the smallest approximation error for a pair.Source: §VII · p. 8, Table II · p. 8
  • ∼82× speedup over standard influence function unlearning is reported for subsequent correlated points in sequential unlearning.The wall-clock comparison is for California Housing.Source: §VII · p. 9, Table III · p. 9
  • For correlated subsequent requests, the method replaces repeated Hessian-inverse-vector products with O(d) vector and scalar operations.The costly product is required once for the first unlearned point.Source: §VII · p. 9

A. Results and Analysis

Results are reported across classification and regression tasks, with classification comparisons spanning five datasets. Batch-sequential experiments report bounded parameter drift and utility and unlearning-fidelity outcomes across six dataset-architecture combinations.

A. Results and Analysis

  • For batch sizes k ≤20, parameter drift remains tightly bounded, model utility (AR) degrades by less than 2%, and unlearning fidelity (AU) stays above 80%.These batch sizes are described as corresponding to practical GDPR compliance scenarios.Source: A. Results and Analysis · p. 9
  • On CIFAR-100 (ResNet-50), the method yields >7% higher model utility (AR) and >11% higher unlearning fidelity (AU) than RUM(B).RUM(B) reports AR=84.52%, AU=76.86%, ToW=0.633, and MIA=0.410.Source: A. Results and Analysis · p. 9
  • Across five classification datasets, the method consistently outperforms baselines across all metrics.The paper also reports evaluation of the approach on regression tasks.Source: A. Results and Analysis · p. 9

B. Numerical Stability and Condition Number Analysis

The condition-number analysis evaluates Hessian damping on CIFAR-10. The undamped Hessian is severely unstable, whereas the default damping value yields a condition number described as numerically stable for inversion.

B. Numerical Stability and Condition Number Analysis

  • The analysis evaluates the condition number of H_w* versus H_λ across damping values on CIFAR-10.The sensitivity of a linear system to numerical noise is measured by κ(A) = σmax(A)/σmin(A).Source: B. Numerical Stability and Condition… · p. 9
  • κ(A) < 10^8 is given as a typical threshold for reliable inversion; double-precision solves typically lose log10(κ(A)) digits of precision.The threshold is presented as a practical stability guideline.Source: B. Numerical Stability and Condition… · p. 9
  • The undamped Hessian on CIFAR-10 has κ ≈10^13, while default λ = 0.01 yields a condition number of approximately 49,735.The passage characterizes the latter as within the numerically stable range for matrix inversion.Source: B. Numerical Stability and Condition… · p. 9

C. Safety Analysis

The safety analysis relates a maximum acceptable parameter error to a minimum Pearson correlation threshold. It also reports that the theoretical error bound is conservative relative to observed errors.

C. Safety Analysis

  • Table IV reports batch-sequential scalability results, and Table X is identified as a sensitivity analysis for damping λ.The supplied passages give the table scopes but not their cell values.Source: Table IV · p. 10, C. Safety Analysis · p. 10
  • If α ≥ α_min, the shortcut's parameter error is at most the maximum acceptable error ε_max.The safety analysis derives α_min by solving the overall error bound for a specified tolerance.Source: C. Safety Analysis · pp. 9–10
  • The theoretical upper bound is conservative by approximately 35× compared with observed errors.The passage suggests safety factors of 0.03 for conservative estimation or 0.1 for aggressive deployment when using the bound.Source: C. Safety Analysis · p. 10, Table IX · p. 11
  • Tables VI, VII, and VIII are identified as classification performance, regression performance, and CIFAR-10 (ResNet-18) comparisons, respectively.Table VI defines AR as retained-set average accuracy and AU as normalized accuracy unlearned.Source: Table VI · p. 11, Table VII · p. 11, Table VIII · p. 11

VIII. DISCUSSION

The discussion presents Pearson correlation as a better proxy for sequential unlearning effects and Hessian damping as a basis for stability guarantees. It reports accuracy improvements alongside competitive forgetting performance.

VIII. DISCUSSION

  • 43.2% win rate for Pearson correlation over cosine similarity and projection-based measures in approximating sequential unlearning effects.The discussion attributes this alignment to centered feature vectors capturing shared directional trends while filtering out mean shifts.Source: §VIII · p. 11
  • 10^-2 accuracy improvement over state-of-the-art baselines accompanies competitive forgetting performance via ToW and MIA scores.The discussion says these results do not come at the cost of compromised privacy or utility.Source: §VIII · p. 11
  • Theorem V.1 provides polynomial-scaling error guarantees controlled by the well-conditioned damped Hessian and a computable scaling factor.The discussion describes the bound as a measurable way to ensure stability.Source: §VIII · p. 11
  • Hessian damping ensures positive definiteness and keeps the update denominator nonzero, supporting numerical stability.The discussion reports this condition across experiments with CNNs, ResNets, and regression models.Source: §VIII · p. 11
  • The safety analysis provides a tunable minimum similarity threshold to keep approximation error below a chosen tolerance.The discussion presents this as a conservative, worst-case operational guideline.Source: §VIII · pp. 11–12

A. Implication

The threat model allows an adversary with access to the trained model to issue data-removal requests. The stated error bound and privacy claim depend on explicit assumptions about similarity computation and the underlying model.

A. Implication

  • An adversary may seek to induce model errors through removal requests or infer training data through membership inference attacks.The threat model assumes adversary access to the trained model.Source: A. Implication · p. 12
  • Requested-point removal error remains at most epsilon_max when alpha is at least alpha_min.The method states this as its theoretical error-bound condition.Source: A. Implication · p. 12
  • The privacy claim says request timing and resource consumption do not leak information about forgotten data.The assumptions include secure training, trustworthy model parameters, and accurate similarity computation unaffected by adversarially crafted features.Source: A. Implication · p. 12

IX. CONCLUSION

The conclusion presents correlation-guided unlearning with damped Hessian approximations and a tunable safety threshold. Across seven datasets, the method achieves an 82× wall-clock speedup while maintaining the reported forgetting effectiveness.

IX. CONCLUSION

  • 82× wall-clock speedup over standard unlearning methods, with MIA success rates of 0.660 and ToW scores of 0.950 on CIFAR-10 with ResNet-18.The evaluation spans seven datasets, including large-scale deployments with ResNet-50.Source: §IX · p. 12
  • Theoretical polynomial error bounds and damping support stability in high-dimensional settings and keep Hessian conditioning numerically stable.The conclusion connects these properties to real-world system integration.Source: §IX · p. 12
  • The built-in safety analysis gives practitioners a tunable alpha_min threshold for guaranteeing approximation error bounds.The conclusion presents this as a deployment aid.Source: §IX · p. 12
  • The method requires no changes to existing training pipelines; planned extensions include adaptive thresholds, adversarial evaluation, and federated-learning integration.The conclusion identifies these extensions as future work.Source: §IX · p. 12
Loading 2609.12620v1…