PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition

Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi

arXiv:2109.06870v1cs.CLcs.LGcs.SDeess.AS

TL;DR

The paper asks how wav2vec 2.0 architecture choices affect ASR performance and efficiency. It analyzes these trade-offs and introduces SEW and SEW-D, which improve the performance-efficiency balance across training setups, including a 1.9× inference speedup with a 13.5% relative WER reduction. The conclusions are primarily based on inference efficiency, GPU ablations, and evaluated ASR datasets.

  • Problem

    The paper studies performance-efficiency trade-offs in wav2vec 2.0 for practical ASR applications.

  • Method

    The authors analyze wav2vec 2.0 architecture choices and combine the findings into SEW and SEW-D pre-trained model architectures.

  • Results

    Across evaluated setups, SEW models achieve better performance-efficiency trade-offs; SEW-D-mid provides 1.9× faster inference with a 13.5% relative WER reduction than W2V2-base.

  • Takeaways & Limitations

    SEW models can function as direct replacements for W2V2 models while offering faster inference and better ASR performance.

  • Takeaways & Limitations

    The study focuses on inference time, GPU ablations, and selected datasets, so observations may change on other hardware or more diverse data.

Abstract

from arXiv · show

This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition (ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference time, SEW reduces word error rate by 25-50% across different model sizes.

1 INTRODUCTION

The paper analyzes wav2vec 2.0 design choices through performance-efficiency trade-offs for ASR and combines the findings into SEW and SEW-D architectures.

  • 1 INTRODUCTION: Efficiency matters because inference improvements can reduce deployment costs and energy footprints as W2V2-type models spread to practical applications.
  • 1 INTRODUCTION: The study examines temporal resolution, waveform feature extractors, parameter allocation, and predictor-head expressivity in W2V2.It retains standard pre-training and few-sample fine-tuning for ASR.
  • 1 INTRODUCTION: Using different resolutions for representation computation and ASR decoding significantly reduces inference time while retaining similar performance.
  • 1 INTRODUCTION: SEW-D-mid provides 1.9× faster inference and a 13.5% relative WER reduction than W2V2-base.The comparison uses the 100h-960h LibriSpeech setup.
  • 1 INTRODUCTION: SEW and SEW-D are proposed by combining the architecture observations into efficient pre-trained ASR models.

2 RELATED WORK

Related work places W2V2 within unsupervised speech representation learning and the broader transition toward Transformer-based end-to-end ASR.

  • 2 RELATED WORK: W2V2 belongs to unsupervised audio representation learning methods that support end-to-end ASR through pre-training on unlabeled speech.
  • 2 RELATED WORK: End-to-end ASR systems are commonly categorized as CTC, RNN-T, or sequence-to-sequence models with different decoding settings.CTC is associated with fast batch decoding, RNN-T with real-time systems, and sequence-to-sequence models with offline use.
  • 2 RELATED WORK: Speech processing is increasingly adopting Transformer architectures and their variants following advances in NLP.

3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2)

W2V2 converts waveform audio into continuous features, masks contextual inputs, and trains representations by predicting quantized targets with contrastive learning.

  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): W2V2 maps raw waveform audio to continuous feature vectors, then uses a context network to produce context-dependent representations.
  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): During pre-training, masked context representations must discriminate quantized original features from negative samples using InfoNCE loss.
  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): The framework comprises a waveform feature extractor, context network, optional quantization module, and two projection heads.
  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): The waveform feature extractor downsamples raw audio into a sequence of feature vectors representing short overlapping signal windows.For 16 kHz audio, W2V2 uses 25 ms windows with a 20 ms stride.
  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): The quantization module converts unmasked features into discrete prediction targets using grouped codebooks and Gumbel softmax.
  • 3 TECHNICAL BACKGROUND: WAV2VEC 2.0 (W2V2): Projection heads reduce contextual and quantized representations to a shared dimensionality and are discarded before downstream fine-tuning.

4 EXPLORING MODEL DESIGN TRADE-OFFS

The study evaluates wav2vec 2.0 architecture choices under efficiency constraints, finding that temporal resolution, feature-extractor design, and parameter allocation shape the performance-efficiency trade-off.

  • 4.2 DEPTH VS. WIDTH: Scaling down width achieves a better performance-efficiency trade-off than scaling down depth, favoring deep and narrow context networks.These narrow models are used as baselines for subsequent experiments.
  • 4.3 TEMPORAL RESOLUTION VS. MODEL SIZE: Lowering temporal resolution while increasing model size reduces WER without increasing inference time, but higher output resolution is needed to narrow the WER gap.A transposed 1D convolution produces two predictions per 25Hz frame, restoring 50Hz output resolution.
  • 4.3 TEMPORAL RESOLUTION VS. MODEL SIZE: Squeezing the context network encodes features at low resolution while retaining high-resolution contrastive learning, reducing WER at similar inference time.The design changes the context-network stride and adds up-sampling so each low-resolution representation can support high-resolution predictions.
  • 4.4 WAVE FEATURE EXTRACTORS DESIGN: The compact WFE-C extractor distributes computation more evenly across layers and matches WFE-O performance with substantially lower inference cost.WFE-C doubles channels as sequence length is reduced and uses the original extractor’s kernel sizes and strides.
  • 4.4 WAVE FEATURE EXTRACTORS DESIGN: At controlled inference time, allocating more computation to the context network than the feature extractor improves WER, while increasing extractor depth outperforms increasing its width.The controlled-budget study compares squeezed context networks, compact extractors, parameter reallocations, kernel sizes, and predictor heads.

5 SEW (SQUEEZED AND EFFICIENT WAV2VEC)

The paper combines architectural changes into SEW and SEW-D, targeting better performance-efficiency trade-offs than W2V2 under comparable inference budgets.

  • SEW architecture: SEW differs from W2V2 through a squeezed context network, WFE-C, redistributed computation, and MLP predictor heads with BatchNorm.Table 5 compares W2V2 and SEW configurations across inference budgets.
  • Model scaling: The scaling recipe defines SEW-tiny, SEW-small, and SEW-mid by increasing width and, for SEW-mid, doubling model depth.SEW-tiny matches W2V2 width-256 inference time, while SEW-small matches W2V2-base Transformer size.
  • SEW-D: SEW-D extends SEW with disentangled attention while reducing Transformer width to maintain similar computation costs.The resulting model uses fewer parameters and can outperform a larger SEW counterpart.

6 FURTHER EXPERIMENTS

Experiments compare SEW and SEW-D with W2V2 across supervision levels, training durations, and out-of-domain datasets, generally finding faster inference with competitive or improved WER.

  • LibriSpeech 100h-960h: SEW-tiny reduces W2V2-tiny WER from 22.8% to 10.6% on test-clean and from 41.1% to 23.7% on test-other while being faster.SEW-mid also lowers WER versus W2V2-mid with similar inference times, despite slightly longer training.
  • SEW versus SEW-D: SEW-D generally lowers WER relative to SEW with smaller width and half the parameters, although SEW-D-tiny is slower because of implementation differences.The implementation uses a less efficient tensor format and attention implementation than fairseq.
  • Less supervision: With 1h and 10h of supervision, SEW-D-mid outperforms W2V2-base while being more efficient, whereas its 10-minute result is worse.SEW-D-base+ performs significantly better in most setups but is unstable during fine-tuning and requires adjusted hyperparameters.
  • Published comparisons: 46.4% lower inference time and 19.7% and 13.5% lower WER make SEW-D-mid more efficient and accurate than W2V2-base without an LM.Against W2V2-large, SEW-D-base+ achieves 2.7× inference and 3.2× pre-training speed-ups with comparable WER and half the parameters.
  • Out-of-domain transfer: On three out-of-domain datasets with 10h labels, SEW-D-mid is about 30% faster, with lower, similar, or slightly higher WER depending on the dataset.SEW-D-base+ consistently outperforms W2V2-base by a large margin while being only 10% slower.

7 CONCLUSION

The study analyzes W2V2 architecture choices and proposes SEW as a family of models with better ASR performance-efficiency trade-offs and faster inference.

  • Conclusion: SEW achieves better ASR performance with faster inference by jointly considering compute time and model expressivity.The models are presented as direct replacements for W2V2 models, including in recent work.
  • Conclusion: The paper presents its architecture-analysis approach as a recipe for balancing performance and efficiency when optimizing complex models for deployment.It emphasizes inference-time economics as important for real-world deployment beyond W2V2.

A LIMITATION

The evaluation focuses on model inference time and GPU ablations, leaving language-model computation and hardware- and data-diverse behavior outside the main analysis.

  • Limitations: The study excludes language-model computation from inference-time measurements and performs ablations on GPUs, so observations may change on CPUs or embedded systems.The paper also identifies more diverse languages, domains, and demographic groups as areas needing further ASR study.

B EXPERIMENTAL SETUP DETAILS

The experiments use LibriSpeech and several out-of-domain speech corpora to evaluate pre-training, fine-tuning, decoding, performance, and efficiency under consistent ASR settings.

  • Datasets: LibriSpeech provides 960 hours of 16 kHz read-English training audio, with separate development and test sets for evaluation.The study also transfers models to TED-LIUM 3, VoxPopuli, and Fisher+Switchboard.
  • Pre-training: The models are pre-trained with W2V2’s fairseq configuration and Adam optimization, while SEW variants require higher Layerdrop rates to avoid divergence.The reported Layerdrop rates are 0.1 for SEW and 0.2 for SEW-D.
  • Fine-tuning: Fine-tuning varies by supervision level: smaller-label settings initially freeze the context network, whereas 100h training fine-tunes it from the beginning.The waveform feature extractor remains frozen throughout, and the 100h setup uses gradient accumulation to simulate four GPUs.
  • Evaluation: Evaluation reports pre-training time, inference time, and word error rate using CTC greedy decoding, with language-model decoding additionally evaluated using beam search.Experiments run on an NVIDIA V100 GPU with FP32 operations.
  • Predictor-head evaluation: A 2-layer MLP with BatchNorm performs best among predictor heads while adding 7% to pre-training time, with unchanged inference time and parameter count.The comparison uses a W2V2 E512L12 model fine-tuned on 100h.

C ABLATION STUDY ON MLP PREDICTOR HEADS

The ablations examine prediction-head choices, decoding algorithms, and language-model effects as contributors to ASR performance and inference efficiency.

  • MLP predictor heads: Batch normalization improves W2V2 prediction-head performance, while prediction heads add pre-training overhead but no fine-tuning or inference overhead.The heads operate at every pre-training timestep and are discarded downstream.
  • CTC decoding: Greedy decoding is parallelizable and efficient on GPUs because CTC timestep outputs are independent, whereas Viterbi decoding sums over possible paths.The evaluation reports identical WER for the two decoding methods on baseline W2V2 models.
  • Language-model decoding: Language-model decoding substantially improves WER but dramatically increases inference time, making faster LM decoding an important future direction.The slowdown depends on CPU type because the beam-search implementation is likely sequential and CPU-bound.

E ADDITIONAL EXPERIMENTS ON THE KERNEL SIZE OF DOWNSAMPLING

The kernel-size experiments show that downsampling-kernel overhead matters most for small models, while larger kernels can favor performance in larger models.

  • Design trade-off: Reducing the downsampling-layer kernel size can free inference budget for a larger waveform feature extractor and improve performance.This observation motivates additional model-size experiments.
  • Model-size effects: At small model sizes, the overhead of a kernel size of 127 occupies a large share of inference time and becomes prohibitive.The experiments compare kernel sizes 31 and 127 under 100h-960h semi-supervised LibriSpeech training.
  • Model-size effects: For large models, kernel-size overhead becomes relatively small, making the performance boost from the larger kernel favorable.Results are reported for models pre-trained for both 100K and 400K updates.
  • Out-of-domain transfer: The paper also reports transfer results for LibriSpeech pre-trained models on three out-of-domain datasets using only 10h of labels.These results are provided to support rapid comparison in future work.

G DISENTANGLED ATTENTION

Disentangled attention separates content and positional representations into three interaction types, but its additional projections and matrix multiplications make it slower than conventional self-attention.

  • Attention formulation: Disentangled attention combines content-to-content, content-to-position, and position-to-content interactions while keeping content and positional embeddings separate.Content embeddings C and relative-position embeddings P produce output embeddings O.
  • Attention formulation: The attention computation uses trainable projections for content and relative-position embeddings, with relative positions clamped to a maximum range.The relative-position index δ(i,j) selects rows of P or its products.
  • Efficiency trade-off: Disentangled attention has nine multiplication operations versus four for conventional self-attention, making it much slower to compute.This computational cost is a direct efficiency drawback of the design.
Loading 2109.06870v1…