PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Machine Unlearning in ASR

Diogo Dinis, Francisco Teixeira, Bhiksha Raj, Alberto Abad, Isabel Trancoso

arXiv:2609.34092v1cs.CL

TL;DR

ASR unlearning remains comparatively underexplored despite privacy risks and data-erasure requests involving trained models. This work adapts and evaluates approximate unlearning methods and privacy attacks for ASR, finding that attack design and multi-subject settings affect observed privacy and utility outcomes.

  • Problem

    Machine unlearning for ASR remains underexplored, despite ASR privacy risks and the difficulty of removing training-data influence from model weights.

    Source: §1 · p. 1

  • Method

    The paper adapts approximate unlearning methods to an end-to-end ASR model and evaluates privacy and utility against behavior from a model retrained without the forget set.

    Source: §1 · p. 1, §2.4 · p. 2

  • Results

    Gradient-ascent methods achieve strong privacy–utility trade-offs, while informed attacks identify forgotten samples from some more complex methods, and multi-subject unlearning fares worse than single-subject unlearning.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Simple membership-inference evaluations are insufficient to assess unlearning success reliably, and sequential or simultaneous unlearning needs better-suited constructions.

    Source: §1 · p. 1

    Single-subject hyperparameters do not transfer reliably to multi-subject unlearning, where the reported methods can degrade performance or fail in different directions.

    Source: §4 · p. 4

Abstract

from arXiv · show

Machine unlearning (MU) offers a path to compliance with "right to be forgotten" regulations. While MU has received increasing attention for speech tasks, it remains largely unexplored for Automatic Speech Recognition (ASR). In this work, we investigate whether existing MU algorithms and evaluation tools are suitable for ASR. We apply several MU techniques to an ASR model, evaluating privacy-utility trade-offs for single-subject unlearning, then assess the best algorithm under sequential and simultaneous unlearning. Results show that gradient ascent-based algorithms achieve strong utility-privacy trade-offs, whereas more complex approaches over-unlearn samples, making them easier to identify as unlearned. This suggests standard privacy evaluations based on simple Membership Inference attacks are insufficient to reliably assess unlearning success, motivating improved evaluation methods for MU in ASR. Finally, we show that both sequential and simultaneous unlearning yield worse privacy and utility than single-subject unlearning, underscoring the need for unlearning constructions better suited to these settings.

1. INTRODUCTION

Machine unlearning may help address privacy risks and data-erasure requests because information associated with training data can remain in model weights. This work evaluates unlearning methods and privacy measures for ASR, finding that attack choice and multi-subject settings materially affect the observed outcomes.

1. INTRODUCTION

  • ASR unlearning remains underexplored despite ASR deployment, privacy risks, and possible data-deletion requests; prior work includes synthetic-canary and multi-task studies with differing limitations or objectives.The authors note that synthetic canaries may not represent training data distributions, while seeking high forget-set error differs from their unlearning definition.Source: §1 · p. 1
  • The study evaluates end-to-end ASR unlearning by comparing forgotten-sample behavior with a gold-standard model retrained without the forget set.Under this formulation, forgotten samples should not be distinguishable from test samples drawn from the same distribution.Source: §1 · p. 1
  • Gradient-ascent methods achieve strong privacy–utility trade-offs, but informed attacks identify unlearned samples left by more complex methods, and sequential and simultaneous unlearning perform worse than single-subject unlearning.The results also report high EMD between forget-loss distributions of unlearned and retrained models, indicating that avoiding MI detection does not necessarily make model behavior match the gold standard.Source: §1 · p. 1
  • The authors provide a reproducible setup with five unlearning methods and evaluate privacy using both unlearning-unaware and unlearning-informed membership inference attacks.Several methods do not maintain the same privacy guarantees across the two attack settings.Source: §1 · p. 2
  • The paper assesses sequential and simultaneous multi-subject unlearning and reports worse privacy and utility than for single-subject unlearning.The authors identify the need for constructions better suited to these settings.Source: §1 · p. 2

2. MACHINE UNLEARNING IN ASR

The paper defines ASR unlearning as moving model behavior toward a model retrained without the requested speaker’s data, then reviews approximate and exact approaches and their evaluation. It adapts five approximate methods for ASR.

2.1. Problem statement

  • The unlearning goal is a model close to one retrained from scratch without the speaker’s forget set, with forgotten-sample WER and loss distributions close to unseen data.The training data are speech recordings, transcriptions, and speaker identities; the retain set excludes the forget set.Source: §2.1 · p. 2

2.2. Machine Unlearning paradigms

  • Exact methods formally guarantee absence of forget-set data but require specialized training, while approximate methods adapt pretrained models without exact guarantees.SISA confines each data subset to a replica so only affected replicas need retraining; approximate methods often combine forget-set gradient ascent with retain-set finetuning.Source: §2.2 · p. 2

2.3. Evaluation

  • Utility evaluation checks model performance, while unlearning success is commonly assessed through membership inference, with extraction attacks and indistinguishability tests as alternatives.The paper notes that extraction attacks are not well developed for every application and statistical tests can be computationally prohibitive for large models.Source: §2.3 · p. 2

2.4. Proposed Machine Unlearning methods for ASR

  • The ASR study adapts finetuning, CF-k, NegGrad, NegGrad+, SCRUB, and attention-smoothing unlearning using gradient-based or teacher–student updates.NegGrad ascends the forget-set loss; NegGrad+ adds retain-set finetuning, while SCRUB and ASU use teacher–student objectives.Source: §2.4 · p. 2

3. EXPERIMENTAL SETUP

The experiments use a pretrained E-Branchformer and speaker-level LibriSpeech partitions, then select hyperparameters and evaluate utility and privacy. The privacy evaluation compares simple and informed membership-inference attacks alongside loss-distribution distances.

3.1. Model selection and implementation

  • The experiments use a pretrained E-Branchformer trained on LibriSpeech’s 960-hour training set and retrain a gold-standard model excluding ten forget subjects.The model and partitions were selected to support pipeline transparency and traceability; experiments ran on one node with three RTX A6000 GPUs, and the codebase is openly available.Source: §3.1 · pp. 2–3

3.2. Data

  • Ten speaker-level forget/retain pairs use train-clean-100, with test-clean and test-other for evaluation and a separate speaker reserved for hyperparameter search.The forget set contains the target speaker’s train-clean-100 utterances; the retain set contains the remaining speakers in that subset.Source: §3.2 · p. 3

3.3. Hyperparameter selection

  • Unlearning methods are standardized to 10 epochs and batch size 8, with remaining hyperparameters selected by Bayesian search to minimize loss-distribution EMD.The objective compares forget-set losses with pre-unlearning test-set losses; the search uses Optuna’s default Tree-structured Parzen Estimator sampler.Source: §3.3 · p. 3

3.4. Utility evaluation

  • Utility is measured with mean WER and model loss on forget, retain, and test sets for each speaker.The losses include encoder CTC and decoder cross-entropy; decoding uses the default configuration with beam size 5.Source: §3.4 · p. 3

3.5. Privacy evaluation

  • Privacy evaluation uses simple and informed membership-inference attacks scored by AUC and EER, plus EMD comparisons among forget, retain, test, and retrained-model loss distributions.The informed attacker trains on losses from other unlearned subjects to model forgotten-sample behavior; partitions are matched by utterance duration to reduce a confound.Source: §3.5 · p. 3

4. RESULTS

The results compare ASR utility and privacy across unlearning methods, then test whether single-subject settings transfer to sequential and simultaneous unlearning. Gradient-ascent methods offer strong single-subject trade-offs, but multi-subject settings degrade privacy and utility.

Single-subject unlearning

  • NegGrad+ and AttSmooth achieve the best simple-MIA privacy results, but AttSmooth reaches 87.57% AUC under an informed attack versus 52.09% for the simple attack.NegGrad+ has the strongest privacy-utility trade-off when considering only MIA and WER, closely followed by NegGrad; the table compares WER, loss, AUC, EER, and EMD against desired reference values.Source: §4 · pp. 3–4, Table 2 · p. 4
  • SCRUB and AttSmooth match simple-MIA trade-offs but leave unlearned members identifiable to informed attacks, while high forget-loss EMD can reveal distance from the retrained model.The lowest EMD to the gold-standard model is obtained by finetuning and CF-k, despite those methods' weaker privacy performance.Source: §4 · pp. 3–4

Sequential and simultaneous unlearning

  • Sequential unlearning with NegGrad raises retain WER to 2.93% versus 0.41% for the retrained model; NegGrad+ improves forget-set metrics but still raises Test-Other WER to 6.81%.NegGrad+ has Test-Other WER of 4.25% in the single-subject case and 5.02% for the retrained model; its MIA AUC rises from 50.18% to 55.58% relative to single-subject unlearning.Source: §4 · p. 4
  • In simultaneous unlearning, NegGrad fails to unlearn, whereas NegGrad+ over-unlearns; single-subject hyperparameters therefore do not transfer reliably to multi-subject settings.NegGrad's forget WER and loss remain close to the original model, while NegGrad+ reaches a forget loss of 14.91 and EMD of 10.18 from the gold standard.Source: §4 · p. 4

5. CONCLUSIONS

The paper evaluates machine-unlearning algorithms for ASR and finds that simple membership-inference evaluation can miss identifiable forgotten samples. Its results also show weaker privacy and utility in sequential and simultaneous settings than in single-subject unlearning.

Conclusions

  • Four algorithms show strong privacy-utility trade-offs under simple MI evaluation, but SCRUB and AttSmooth leave forgotten samples identifiable to unlearning-informed attacks.Methods most resistant to MIAs were farthest from the gold standard in forget-set loss distribution, while methods closest to it were most vulnerable.Source: §5 · p. 4
  • Sequential and simultaneous unlearning degrade both privacy and utility relative to single-subject unlearning.The study reports an inversion between EMD and MIA, so resistance to MIAs does not by itself establish absence of information about forgotten data.Source: §5 · p. 4
  • The authors recommend unlearning-informed MI constructions and distributional measures to evaluate privacy and similarity to gold-standard models.They present MU for ASR as an open problem and the benchmark as a stepping stone for future research in ASR and other speech tasks.Source: §5 · p. 4

Related papers

Loading 2609.34092v1…