Source-linked AI summary
Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding
Yingting Li, Ambuj Mehrish, Shuai Zhao, Rishabh Bhardwaj, Amir Zadeh, Navonil Majumder, Rada Mihalcea, Soujanya Poria
TL;DR
Full fine-tuning of large pre-trained models can be costly and prone to overfitting or catastrophic forgetting, motivating parameter-efficient transfer learning for speech. The paper introduces SURE and ConvAdapter, and reports that parameter-efficient approaches can match or exceed full fine-tuning across speech tasks, with ConvAdapter using a small share of trainable parameters.
Problem
Fine-tuning all parameters of large pre-trained models for each downstream task is resource-intensive and can overfit or cause catastrophic forgetting.
Source: §1 · p. 1Method
The paper introduces SURE to evaluate parameter-efficient approaches across speech tasks and proposes ConvAdapter, alongside adapter tuning, prefix tuning, and LoRA.
Source: §1 · p. 2, §4.6 · p. 6Results
Across speech tasks, parameter-efficient methods achieve performance on par with or better than full fine-tuning; ConvAdapter uses 0.94% of fine-tuning's trainable parameters but is similar to or slightly worse than other adapters.
Source: §4.3 · p. 4, §5 · p. 7Takeaways & Limitations
The experiments indicate that adapters with few parameters can perform comparably to or better than full fine-tuning across the evaluated speech tasks.
Source: §4.3 · p. 4, §5 · p. 7ASR and SF results for ConvAdapter require further hyperparameter tuning for optimal performance.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters on some of the task in SURE. We further explore the effectiveness of parameter efficient transfer learning for speech synthesis task such as Text-to-Speech (TTS).
1. INTRODUCTION
Full fine-tuning of large pre-trained models demands substantial computing resources and can overfit or forget prior knowledge, while speech-domain evidence for parameter-efficient methods remains limited. The paper introduces SURE to evaluate these methods across speech tasks and proposes a lightweight CNN-based ConvAdapter.
Motivation
- Full fine-tuning requires substantial computing resources and dedicated memory, and can overfit or cause catastrophic forgetting when task-specific data are limited.Parameter-efficient methods instead train a small set of task-specific parameters while keeping the pre-trained model frozen.Source: §1 · p. 1
- Prior speech-domain adapter work was limited, motivating evaluation of parameter-efficient methods across downstream speech-processing tasks.The authors describe prior applications in ASR, cross-lingual or multilingual ASR, and speech translation.Source: §1 · p. 1
Contributions
- SURE evaluates adapters, prefix tuning, and LoRA across Speech Emotion Recognition, Speaker Recognition, ASR, and Keyword Spotting.The benchmark is introduced to examine these methods on varied speech-processing tasks.Source: §1 · p. 2
- ConvAdapter uses CNNs to exploit correlations between adjacent time-frequency components and is intended to scale to transformer-based speech models, including HuBERT.The paper uses Wav2Vec 2.0 in this work.Source: §1 · p. 2
2. RELATED WORK
Related work describes SUPERB as an evaluation of downstream speech tasks using frozen self-supervised models and task-specific prediction layers. Adapter use in speech remains narrower than in NLP and computer vision, with applications spanning ASR, translation, continual learning, and speech prompts.
Speech benchmarks
- SUPERB evaluates speech tasks by freezing a self-supervised model and adding a task-specific prediction layer, but does not explore the upstream model's capability.The passage frames the prediction layer as adding task-relevant description to the downstream input.Source: §2 · p. 2
Adapters
- Adapters have been used in NLP and computer vision, while speech applications remain concentrated in ASR and limited translation settings,,.The cited translation studies address zero-shot translation, a single encoder adapter, and streaming multilingual RNN-T, respectively.Source: §2 · p. 2
3. PARAMETER-EFFICIENT TRANSFER LEARNING
The paper describes adapter tuning, prefix tuning, and LoRA as parameter-efficient changes to transformer models. It also introduces ConvAdapter, a convolution-based module designed to learn task-specific speech representations.
Adapter tuning
- An adapter down-projects its input, applies a nonlinearity, then up-projects it, with a residual connection; the paper inserts it after each transformer module's feed-forward layer.The bottleneck dimension m is smaller than the input dimension d.Source: §3 · pp. 2–3
Prefix tuning
- Prefix tuning prepends learnable vectors to attention keys and values at every layer, leaving queries unchanged; only the prefix vectors are updated during training.The new keys and values are used for multi-head attention.Source: §3 · p. 3
LoRA
- LoRA approximates a pre-trained weight update with trainable low-rank matrices and is added at four locations in the multi-head attention layer.The paper writes the update as W + ΔW = W + WdownWup, with rank r < d.Source: §3 · p. 3
Convolutional Adapter
- ConvAdapter uses three lightweight 1D convolutional layers, layer normalization, and a squeeze-and-excite module, then adds a residual connection.One convolution is depth-wise, reducing parameter count and computational cost; the module is inserted where the bottleneck adapter is placed.Source: §3 · p. 3
4. EXPERIMENTS AND RESULTS
The experiments evaluate parameter-efficient methods across SURE speech tasks and TTS. The methods often match or outperform full fine-tuning, while ConvAdapter uses few trainable parameters and TTS results vary by dataset.
4.1. The SURE Benchmark
- SURE covers four speech-processing tasks and uses task-specific metrics, including accuracy, WER, PER, slot-type F1, and slot-value CER.TTS receives both objective and subjective evaluation.Source: §4.1 · p. 3
4.2. Training
- The experiments use pre-trained Wav2Vec 2.0 with adapters at each transformer layer, task-specific output layers, and early stopping on validation performance.Datasets are partitioned into train, validation, and test sets; training stops after five epochs without validation improvement.Source: §4.2 · pp. 3–4
4.3. Results and Discussions
- ConvAdapter uses 0.94% of full fine-tuning’s trainable parameters and performs similarly to standard adapters and LoRA, while prefix tuning outperforms it with 0.55%.Across SURE tasks, parameter-efficient methods are on par with or better than full fine-tuning; KS and ASR results are comparable, while ESD and MELD results exceed fine-tuning, though MELD performance remains poor.Source: §4.3 · p. 4
4.4. Text-To-Speech (TTS)
- For TTS, a multi-speaker Transformer TTS model is pre-trained on VCTK, then adapted on LibriTTS and L2ARCTIC with the base model frozen.Adapter, Prefix, LoRA, and ConvAdapter modules are trained for 40k additional steps.Source: §4.4 · p. 5
4.4.1. Objective Evaluation
- On LTS, Adapter outperforms the other methods on MCD and WER; on L2ARCTIC, ConvAdapter outperforms LoRA and Prefix tuning on both metrics.The authors attribute the difference to L2ARCTIC’s non-native English accents and say ConvAdapter adapts better to accent changes.Source: §4.4.1 · p. 5, Table 4 · p. 6
4.5. Subjective Evaluation
- Parameter-efficient methods achieve MOS scores similar to full fine-tuning for naturalness and speaker similarity, while fine-tuning is slightly better for speaker similarity.Nine researchers rated randomly selected generated samples on five-point scales; the reported scores include 95% confidence intervals.Source: §4.5 · pp. 5–6, Figure 4 · p. 5, Figure 5 · p. 6
4.6. Discussion
- The authors attribute adapter gains with few trainable parameters to gradients propagating through the pre-trained model and distilling its encoded knowledge.They identify combining ConvAdapter with prefix tuning or LoRA, especially for MELD, as future work.Source: §4.6 · p. 6
- In the speaker-recognition trade-off experiment, ConvAdapter performance is unchanged across compression rates, whereas bottleneck performance degrades.The authors suggest speaker information is encoded in early Wav2Vec 2.0 layers, where fine-tuning top layers showed no performance drop.Source: §4.6 · p. 7, Table 5 · p. 7
5. CONCLUSION AND FUTURE WORK
The paper introduces the SURE benchmark and reports that parameter-efficient adapters can match or exceed full fine-tuning across speech-processing tasks, with ConvAdapter achieving state-of-the-art performance using few trainable parameters. For TTS, parameter-efficient methods produce results similar to fine-tuning, but longer sentences remain a challenge.
5. CONCLUSION AND FUTURE WORK
- Adapters with few parameters achieve comparable or better performance than full fine-tuning across speech-processing tasks, and ConvAdapter reaches state-of-the-art performance with limited trainable parameters.The paper introduces SURE to evaluate parameter-efficient transformer-based adapters across a range of speech-processing tasks.Source: §5 · p. 7
- Parameter-efficient methods produce TTS results similar to fine-tuning, but there is still room for improvement.The authors identify sub-par performance on longer sentences and say speech-sample quality could be improved.Source: §5 · p. 7
7. APPENDIX
The appendix describes Wav2Vec 2.0, the TTS architecture and its training components, and the hyperparameters used for the reported task results.
7.1. Pre-trained Model:Wav2Vec 2.0
- Wav2Vec 2.0 maps raw 16 kHz mono speech through a seven-block convolutional feature encoder, quantization module, and 12-block Transformer context network.Its self-supervised training uses contrastive prediction of masked-input quantized representations and a loss promoting diverse quantized representations.Source: §7.1 · p. 10
7.2. TTS Architecture
- The TTS system uses a Transformer-based framework with adapters in every Transformer layer, speaker embeddings, and a pre-trained HiFiGAN vocoder for audio generation.The speaker embeddings have dimension 512; Tables 6 and 7 cover audio preprocessing and optimizer/model parameters.Source: §7.2 · p. 10, Figure 6 · p. 10, Table 6 · p. 10
7.3. Hyperparameters
- The authors report task-specific learning rates that produced the best results, while Prefix tuning and LoRA use default configurations from.Table 8 lists task hyperparameters, including learning rate, compression ratio, and rank; alternative configurations for Prefix tuning and LoRA are left for future work.Source: §7.3 · p. 10, Table 8 · p. 11