PaperlayerView exact revision on arXiv

Source-linked AI summary

PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition

Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, James Glass

arXiv:2106.05933v2cs.CLcs.LGcs.SDeess.AS

TL;DR

This paper examines whether sparse subnetworks in pre-trained speech SSL can improve low-resource ASR, given the cost and limited gains of existing pruning approaches. It introduces PARP and reports better results than baseline pruning, including WER reductions versus the full model.

  • Problem

    Whether pruning pre-trained speech SSL can yield subnetworks that outperform dense models remains underexplored, while OMP and IMP require repeated, costly downstream finetuning and show no performance improvement over the dense model.

    Source: §1 · p. 2

  • Method

    PARP prunes an initial subnetwork, finetunes it while allowing pruned weights to update, then re-prunes to the target sparsity.

    Source: §1 · p. 2

  • Results

    Across sparsities, finetuning supervision amounts, model scales, and downstream languages, PARP outperforms OMP and IMP; on Librispeech 10min without LM decoding, it yields an absolute 10.9%/12.6% WER decrease versus the full model.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PARP can discover a shared subnetwork for multiple spoken languages in one pass.

    Source: §6 · p. 10

    PARP is not claimed as a plug-and-play way to boost any SOTA ASR pipeline, and its effects across different setups remain uninvestigated due to resource limitations.

    Source: Limitations and Future Work · p. 10

Abstract

from arXiv · show

Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is suboptimal in the computational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared to the original dense network. We present Prune-Adjust-Re-Prune (PARP), which discovers and finetunes subnetworks for much better performance, while only requiring a single downstream ASR finetuning run. PARP is inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely a slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruning methods. In particular, on the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared to the full model. We further demonstrate the effectiveness of PARP via: cross-lingual pruning without any phone recognition degradation, the discovery of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks.

1 Introduction

PARP targets sparse subnetworks that improve low-resource ASR without the repeated downstream finetuning required by conventional pruning. The paper formulates the low-resource task, describes its model and transfer settings, and compares task-aware and task-agnostic discovery methods.

1 Introduction

  • OMP and IMP require multiple downstream finetuning rounds and show no visible WER improvement over the dense model, motivating efficient discovery of better subnetworks.Finding sparse subnetworks for pre-trained speech SSL had received little study, despite evidence in other domains.Source: §1 · p. 2
  • PARP first prunes a task-agnostic subnetwork, then allows masked weights to update during ASR finetuning before pruning again, using a single downstream finetuning pass.The initial subnetwork can come from pre-trained weights or a different task or language.Source: §1 · pp. 2–3
  • On Librispeech 10min without LM decoding, PARP achieves an absolute 10.9%/12.6% WER decrease versus the full model and outperforms OMP and IMP across the reported settings.The reported settings vary sparsity, finetuning supervision, pre-trained model scale, and downstream spoken language.Source: §1 · p. 3

2.1 Problem Formulation

  • The objective is to find a pruning mask and finetuned weights whose masked network minimizes downstream ASR loss on the target dataset.The formulation represents the target task with a small transcribed set and a downstream loss such as CTC.Source: §2.1 · p. 3

2.2 Pruning Targets and Settings

  • The study considers three low-resource ASR settings using pre-trained wav2vec2 and xlsr as bases: low-resource English, high-to-low-resource multilingual transfer, and cross-lingual transfer for multilingual ASR.Source: §2.2 · p. 3

2.3 Subnetwork Discovery in Pre-trained SSL

  • All pruning baselines use unstructured magnitude pruning, which removes the lowest-magnitude weights regardless of network structure.Random pruning is also included as a comparison.Source: §2.3 · p. 4
  • Task-aware discovery prunes weights after target-task finetuning: OMP does this once, while IMP repeats it over multiple finetunings.IMP prunes 10% of the remaining weights at each iteration.Source: §2.3 · p. 4
  • Task-agnostic methods prune pre-trained weights directly or transfer a mask from another language, but generally yield worse subnetworks than OMP or IMP.The resulting masked pre-trained network is then finetuned on the target task to recover pruning loss.Source: §2.3 · p. 4

3 Method

PARP combines a task-agnostic initial mask with brief updates that permit masked weights to change, then reapplies magnitude pruning. High overlap between task-aware and task-agnostic masks motivates this initialization; PARP-P progressively increases sparsity when starting masks may be unsuitable.

3.1 Algorithm

  • PARP jointly discovers and finetunes a subnetwork in one pass, letting masked weights receive gradient updates before re-pruning to the target sparsity.The algorithm initializes from a task-agnostic mask, trains for intervals of n updates, and reapplies unstructured magnitude pruning.Source: §3.1 · p. 4

3.2 Obtaining and Adjusting the Initial Subnetwork

  • Task-aware and task-agnostic pruning masks have high overlap across sparsities, supervision amounts, model scales, and downstream languages.Figure 2 reports greater than 90% similarity among spoken-language OMP masks and with MPI masks at the shown sparsity.Source: §3.2 · p. 5, Figure 2 · p. 5
  • Because the initial mask is similar to task-aware masks, PARP needs only slight adjustment; zeroed weights can recover through gradients without straight-through estimation or an added sparsity loss.Task-agnostic masks, including MPI or a mask from another language, can be obtained without target-task finetuning.Source: §3.2 · p. 5

3.3 PARP-Progressive (PARP-P)

  • PARP-P starts from a lower sparsity and progressively prunes to the target, particularly helping at higher sparsities while retaining PARP's computational cost.Its distinguishing change is the initial starting sparsity; Figure 2 compares language-pair OMP mask overlaps and MPI masks.Source: §3.3 · p. 5, Figure 2 · p. 5

4 Experiments and Analysis

PARP finds sparse subnetworks that improve low-resource ASR results, with experiments spanning LSR, cross-lingual transfer, multilingual pruning, and pretrained language models. The analyses also examine the importance of the initial mask and possible explanations for PARP’s performance.

4.1 Comparing PARP, OMP, and IMP on LSR, H2L, and CSR

  • 10.9%/12.6% absolute WER reduction over full wav2vec2-large on Librispeech 10min without LM decoding is reported for PARP subnetworks.The comparison is against the full wav2vec2-large model.Source: Table 2 · p. 6, §4.1 · p. 6
  • On H2L, PARP and OMP reach 50% sparsity with minimal degradation; on CSR they reach 70%, with PARP’s advantage appearing only at higher sparsities.PARP is more effective than OMP at all sparsities on H2L, while the advantage is limited to higher-sparsity regions on CSR.Source: §4.1 · p. 6, Figure 4 · p. 7

4.2 How Important is the Initial Subnetwork (Step 1) in PARP?

  • PARP with a random initial subnetwork performs nearly as poorly as random pruning, while PARP-P can yield over 50% absolute WER reduction and PARP’s final mask has >99.99% IOU with its initial MPI mask after 20% sparsity.MPI is a poor starting subnetwork at high sparsity, such as 90% on LSR, because most weights are already pruned.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Are Pruning Masks Transferrable across Spoken Languages?

  • Cross-lingual OMP masks can produce language-dependent PER differences under regular finetuning, but PARP shows no cross-lingual PER degradation on H2L.The transfer experiment uses pretrained wav2vec2-base; the passage reports the same observation for CSR with pretrained XLSR in an appendix.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 Discovering a Single Subnetwork for 10 Spoken Languages

  • Joint finetuning retrieves a shared subnetwork for 10 languages in one run, but PARP is only slightly better than OMP on H2L and CSR in this setting.The authors suggest disjoint language-specific important weights may send mixed signals to PARP’s adjustment step.Source: Figure 4 · p. 7, §4.4 · p. 8

4.5 Does PARP work on Pre-trained BERT/XLNet?

  • On pretrained BERT/XLNet, IMP masks overlap by >98% across nine GLUE tasks, and the study tests PARP for cross-task subnetwork transfer.Table 4 compares cross-task transfer on GLUE using accuracy averaged across source tasks for each target task.Source: §4.5 · pp. 8–9, Table 4 · p. 9

4.6 Implications

  • The authors hypothesize that avoiding the pruning of task-specific neurons helps explain PARP’s high-sparsity and cross-lingual results; iterative pruning may help preserve important weights but costs more computation.They also connect preserved task-specific neurons to random pruning retaining accuracy under certain thresholds and settings.Source: §4.6 · p. 9

5 Related Work

The paper presents PARP as a contribution to pruning large pretrained speech models and reports broad ASR and cross-task results. It also sets boundaries on what the experiments establish and records the computational limits of repeated runs.

5 Related Work

  • The authors position this as the first pruning study of large pretrained self-supervised models for low-resource and multilingual ASR.They also state that prior speech-pruning work had not shown pruned models outperforming their dense counterparts.Source: §5 · p. 10

6 Conclusions

  • PARP discovers better speech SSL subnetworks than baseline pruning methods at a fraction of their computational cost, with over 10% WER reduction over the full model.The conclusion also reports minimal cross-lingual adaptation errors, a shared subnetwork for multiple spoken languages in one pass, and reduced cross-task adaptation errors for BERT/XLNet.Source: §6 · p. 10

Limitations and Future Work

  • The study uses unstructured magnitude pruning but does not establish that its sparse models use less memory or energy than dense models.The authors suggest the methodology could extend to more advanced unstructured or structured pruning methods.Source: Limitations and Future Work · p. 10
  • Cross-lingual mask-transfer results for wav2vec 2.0 are limited to ASR, and the authors do not claim transfer across speech tasks.They provide only a pilot study across phone recognition, speaker recognition, and slot-filling.Source: Limitations and Future Work · p. 10
  • The authors do not claim PARP is plug-and-play for state-of-the-art ASR pipelines, and effects across different setups remain under investigation.They report a preliminary experiment combining PARP with transformer-LM decoding.Source: Limitations and Future Work · p. 10

7 NeurIPS Paper Checklist

  • The authors report that experimental results are presented in Section 4, where Figure 3 shows their claim that PARP outperforms LTH; they direct readers to Section 6 for limitations.Source: §7 · p. 20
  • The authors could not extensively rerun experiments because of computational expense, though their recreated baselines match prior reports.The checklist notes that the experiments were not extensively repeated.Source: §7 · p. 20

8 Model Details

The model details explain which components are pruned, how downstream fine-tuning is configured, and how PARP's pruning frequency is set. The implementation uses a fixed compute budget and omits language-model validation during fine-tuning.

8.1 Model and Pruning Configurations

  • Pruning targets the BERT module in wav2vec 2.0 and XLSR-53, while leaving the CNN feature encoder, positional embeddings, and layer-normalization layers unpruned.The feature encoder is fixed during fine-tuning, and the quantization layer is discarded; wav2vec2-base has 12 transformer blocks and 95M parameters, while wav2vec2-large has 24 blocks and 315M parameters.Source: §8.1 · p. 21
  • The pruning setup also leaves the CNN feature extractor, layer-norm running statistics, and positional embedding/task-specific linear layer unpruned.The authors attribute these choices to the fixed feature extractor and to following BERT-Ticket's setup [19].Source: §8.1 · p. 21

8.2 Finetuning Hyper-Parameters

  • Fine-tuning uses 20k, 15k, and 12k steps for wav2vec2 on the 10h, 1h, and 10min splits, respectively, and 12k steps per language for XLSR.The default 10k-step freezing of wav2vec2 except its final layer was omitted because it could destabilize pruned-model training.Source: §8.2 · p. 21

8.3 PARP Hyper-Parameters

  • PARP's pruning-frequency hyperparameter n is set heuristically from 1 to 50 depending on model scale and training split.The authors state that sensible small values, such as 5–50 over more than 10k steps, should yield similar final performance.Source: §8.3 · p. 21

8.4 Implementation

  • For each experiment, models were fine-tuned on 2 or 4 GPUs in parallel without a language model for validation, using FP32; all pruning methods shared a fair computational budget11.Source: §8.4 · p. 22

9 Experimental Setup for LSR, H2L, and CSR

The experimental setup compares pruning across low-resource, multilingual, and cross-lingual ASR settings, using WER or PER without external language-model decoding. The accompanying analyses examine IMP rewinding and overlap between task-agnostic and task-aware masks.

9 Experimental Setup for LSR, H2L, and CSR

  • Across Librispeech 10h, 1h, and 10min, CommonVoice H2L, and XLSR CSR settings, the study uses WER for LSR and PER for H2L and CSR without external LM decoding.LSR varies supervision and model scale; H2L and CSR compare monolingual wav2vec2-base with multilingual XLSR-53 across the same 10 languages.Source: §9 · p. 22

10 How important is the IMP rewinding starting point?

  • IMP rewinding has minimal effect on speech SSL pruning, and that effect diminishes further when the pretrained model scales from base to large.Rewinding starts each pruning iteration partway through downstream ASR fine-tuning rather than from the pretrained initialization.Source: §10 · p. 22, Figure 9 · p. 22

11 OMP Masks Overlap in H2L and CSR

  • For any sparsity, supervision amount, model scale, or downstream language, task-agnostic OMP masks have high IOU with task-aware masks.The analysis compares pairwise masks after source-language fine-tuning for wav2vec2-base in H2L and XLSR in CSR.Source: §11 · p. 23

11.1 OMP Masks Overlap in H2L; 11.2 OMP Masks Overlap in CSR

  • The H2L and CSR overlap analyses obtain OMP masks for each spoken-language ASR task and compare mask pairs at each sparsity.H2L uses fine-tuned wav2vec2-base, while CSR uses fine-tuned XLSR.Source: §11.1 · p. 23, §11.2 · p. 27

12 xlsr Cross-Lingual Mask Transfer

The paper examines transfer of pruning masks across spoken languages for XLSR and across GLUE tasks for pretrained BERT. It reports minimal phone-error degradation for the cross-lingual transfer and better cross-task transfer results with PARP than with BERT-Ticket.

12 xlsr Cross-Lingual Mask Transfer

  • Cross-lingual XLSR mask transfer has minimal PER degradation, using the same transfer setup as the wav2vec2 study but with XLSR.The procedure transfers a source-language OMP mask to a target-language ASR fine-tuning run with PARP.Source: Figure 28 · p. 31, §12 · p. 31

13 Details of Task Transfer Results on Pre-trained BERT

  • For all GLUE tasks, PARP achieves better cross-task transfer results than BERT-Ticket, including CoLA and STS-B, where BERT-Ticket transfers poorly.The figure compares source tasks by row and target tasks by column, with scores expressed relative to same-task transfer.Source: §13 · p. 32, Figure 30 · p. 33

14 Full H2L and CSR Pruning Results

Supplementary figures compare pruning techniques on H2L, CSR, and jointly fine-tuned wav2vec2-base and XLSR across ten CommonVoice languages.

14 Full H2L and CSR Pruning Results

15 wav2vec2 + PARP with Random Seeds and LM Decoding

Seed tests show a similar PARP gain across runs, while moderate-beam LM decoding reduces but does not eliminate its improvement over the full model.

LM Decoding

  • LM decoding was omitted from the main results to isolate pruning's ASR effect and avoid costly hyperparameter searches across many pruned models.The standard wav2vec-series LMs are trained on Librispeech text and can recover acoustic-model errors.Source: §15 · p. 38

Random Seeds

  • ~10% WER reductions versus full wav2vec2-base were observed across eight additional seeds with PARP at 10% sparsity on Librispeech 10min without LM decoding.The result uses Viterbi decoding.Source: §15 · p. 38, Table 5 · p. 38

LM Decoding

  • PARP at 10% sparsity still improves on full wav2vec2-base with 4-gram or transformer LM decoding, though the gain is smaller.The decoding used moderate beam sizes, with hyperparameters searched on dev-other over 128 Ax trials.Source: §15 · p. 38, Table 6 · p. 38

16 wav2vec2 Cross-Task Mask Transfer on SUPERB

Cross-task mask transfer on SUPERB suggests that source-target task similarity is associated with better results, although the reported setup leaves task-specific hyperparameters unoptimized.

Cross-Task Results

  • More similar source and target tasks perform better in the wav2vec2 cross-task transfer experiment at 50% sparsity.Speaker-related source subnetworks perform better on speaker verification, while ASR or phone-recognition sources perform better on phone recognition.Source: §16 · p. 39, Table 7 · p. 39
  • The reported differences are not large and could potentially be reduced through hyperparameter tuning.The upstream wav2vec2 was jointly finetunable, unlike SUPERB's default setup, and task hyperparameters were not optimized.Source: §16 · p. 39

17 Does Observation 1 generalize across Pre-Training Objectives?

The high overlap between task-agnostic and task-aware ASR masks holds within a shared speech SSL pretraining objective, but does not generalize across objectives.

Experimental Setup

  • The cross-objective comparison tests mask IOU at fixed sparsity by pruning task-finetuned wav2vec2 and HuBERT models on SUPERB tasks.The procedure obtains task-specific masks from each model before comparing them.Source: §17 · p. 40

Cross-Objective Mask Similarity

  • At 50% sparsity, mask IOUs between wav2vec-base and HuBERT-base are close to random, so Observation 1 does not generalize across their pretraining methods.The comparison covers SUPERB tasks; wav2vec2 uses contrastive pretraining and HuBERT uses mask-predict pretraining.Source: §17 · p. 40, Table 8 · p. 40
  • Observation 1 holds separately within wav2vec2 and HuBERT, conditioned on using the same speech SSL pretraining objective.It concerns high IOUs between task-agnostic and task-aware non-zero ASR pruning masks across sparsity, supervision, model scale, and languages.Source: §17 · p. 40

18 Pruned Weights Localization Across Layers

Across spoken languages and tested sparsity levels, wav2vec2-base pruning is concentrated more in lower and higher layers than in intermediate layers. The authors interpret the pattern as indicating greater value in intermediate-layer neurons.

18 Pruned Weights Localization Across Layers

  • Bottom and higher wav2vec2-base layers are pruned more than middle layers, which are pruned less.Table 9 reports this pattern for Spanish in the H2L setting at 50% sparsity with OMP.Source: §18 · p. 41, Table 9 · p. 41
  • The same layer-wise pruning distribution appears across 10 spoken languages and sparsities from 10% to 90%.The accompanying figures show sparsity over layers for individual languages and sparsity settings.Source: §18 · p. 41, Figure 41 · p. 41, Figure 42 · p. 41
  • The authors interpret lower sparsity in intermediate layers as evidence that their neurons are more valuable than neurons in lower and higher layers.This interpretation is stated to hold regardless of spoken language.Source: §18 · p. 41

19 Experimental Limitations

The experimental conclusions have several stated scope boundaries, spanning pretraining objectives, model scale and data, speech style, supervision levels, and deployment conditions. The authors identify these as open questions for follow-up work.

19 Experimental Limitations

  • Experiments use contrastive-pretrained models, so generalization to mask-prediction or autoregressive-prediction objectives remains unclear.The authors name HuBERT and APC as examples of other objectives.Source: §19 · p. 53
  • The studied models are relatively large, and wav2vec2-base and wav2vec2-large were pretrained on Librispeech 960 hours.The authors leave small-model pruning and the effects of pretraining-data selection, including source mismatch, for future investigation.Source: §19 · p. 53
  • Fine-tuning uses read speech from Librispeech and CommonVoice, leaving conversational speech and high-resource settings untested.The authors also leave application to end-to-end ASR without self-supervised pretraining for future study.Source: §19 · p. 53
  • The authors leave testing PARP in more realistic settings, such as noisy data or limited bandwidth, to follow-up work.They characterize the study as scientific in nature and say its central idea emerged from an initial pilot study.Source: §19 · p. 53
Loading 2106.05933v2…