PaperlayerView exact revision on arXiv

Source-linked AI summary

Improvements to deep convolutional neural networks for LVCSR

Tara N. Sainath, Brian Kingsbury, Abdel-rahman Mohamed, George E. Dahl, George Saon, Hagen Soltau, Tomas Beran, Aleksandr Y. Aravkin, Bhuvana Ramabhadran

arXiv:1309.1501v3cs.LGcs.CLcs.NEmath.OCstat.ML

TL;DR

The paper asks how CNNs for LVCSR can be improved beyond their established advantage over DNNs. It analyzes architectures, pooling, fMLLR feature adaptation, and dropout-compatible Hessian-free training, obtaining 2-5% relative improvement over previous best CNN results across Broadcast News tasks.

  • Problem

    The paper investigates how to improve CNN performance for LVCSR beyond prior CNN systems while preserving speech-feature locality.

  • Method

    The paper evaluates weight sharing, pooling, multi-scale representations, fMLLR integration into log-mel features, and fixed-mask dropout during Hessian-free sequence training.

  • Results

    2-5% relative improvement in WER over previous best CNN results is achieved with the proposed fMLLR and dropout ideas.

  • Takeaways & Limitations

    fMLLR and dropout provide the main reported gains, while the tested pooling and weight-sharing alternatives do not improve LVCSR performance.

  • Takeaways & Limitations

    The HF-plus-dropout procedure requires fixed data across conjugate-gradient iterations to preserve guaranteed conjugate search directions.

Abstract

from arXiv · show

Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a variety of LVCSR tasks. In this paper, we describe different methods to further improve CNN performance. First, we conduct a deep analysis comparing limited weight sharing and full weight sharing with state-of-the-art features. Second, we apply various pooling strategies that have shown improvements in computer vision to an LVCSR speech task. Third, we introduce a method to effectively incorporate speaker adaptation, namely fMLLR, into log-mel features. Fourth, we introduce an effective strategy to use dropout during Hessian-free sequence training. We find that with these improvements, particularly with fMLLR and dropout, we are able to achieve an additional 2-3% relative improvement in WER on a 50-hour Broadcast News task over our previous best CNN baseline. On a larger 400-hour BN task, we find an additional 4-5% relative improvement over our previous best CNN baseline.

1. INTRODUCTION

The paper analyzes CNN architectures and training strategies for LVCSR, targeting improvements beyond prior CNN systems. It evaluates weight sharing, pooling, fMLLR features, and dropout during Hessian-free training.

  • Architecture: The paper compares limited weight sharing and full weight sharing across multiple convolutional layers for speech CNNs.Earlier limited-weight-sharing work focused on a single layer, motivating a broader comparison.
  • Pooling: Pooling methods successful in computer vision, including lp, stochastic, overlapping, temporal, and multi-scale pooling, are evaluated for larger-scale speech recognition.The study tests whether these generalization strategies transfer to LVCSR.
  • Features: The proposed fMLLR method transforms log-mel features into an uncorrelated space, applies fMLLR, and transforms the adapted features back.Direct fMLLR application to correlated VTLN-warped log-mel features had produced no improvement.
  • Optimization: Dropout is made compatible with Hessian-free sequence training by fixing the dropout mask per utterance across conjugate-gradient iterations within each HF iteration.Changing the mask can prevent conjugate directions from being guaranteed during HF optimization.
  • Results: 2-3% relative reduction in WER is achieved on a 50-hour Broadcast News task, while 4-5% relative improvement is achieved on a 400-hour task over the prior CNN system.The combined gains particularly come from fMLLR and dropout.

2. BASIC CNN ARCHITECTURE

The baseline CNN uses a fixed architecture and feature set established for LVCSR experiments on a 50-hour English Broadcast News task.

  • Baseline architecture: The baseline architecture has two convolutional layers followed by four fully connected layers.The convolutional layers use 128 and 256 feature maps, while fully connected layers use 1,024 hidden units.
  • Baseline architecture: The first convolutional layer uses pooling size 3, whereas the second convolutional layer uses no pooling.This configuration was selected as appropriate for the baseline system.
  • Experimental setting: The experiments use 50 hours of 1996 and 1997 English Broadcast News data and report results on the EARS dev04f set.Unless otherwise noted, CNNs are cross-entropy trained in a hybrid setup.

3. ANALYSIS OF VARIOUS STRATEGIES FOR LVCSR

The strategy analysis finds that VTLN-warped mel features with dynamic coefficients are a strong input, while several computer-vision-inspired CNN modifications provide limited or no LVCSR benefit.

  • Optimal feature set: VTLN-warped mel filter-bank features with delta and double-delta coefficients are selected as the optimal CNN input feature set.VTLN warping and delta features help, while energy and direct fMLLR adaptation do not improve results in this feature comparison.
  • Optimal feature set: The feature analysis is summarized by Table 1, which reports WER as a function of input feature.The table supports comparison of alternative CNN input representations.
  • Network depth: CNN performance improves as convolutional layers increase up to two, after which performance deteriorates, with CNNs outperforming DNNs using the same input features.The comparison keeps the total number of network parameters constant.

3.3. Number of Hidden Units

The architecture analysis addresses how speech-frequency differences affect weight sharing and convolutional capacity. It finds similar LWS and FWS performance while favoring a simpler FWS configuration.

  • Number of hidden units: Increasing convolutional hidden units reduces WER up to 220 units, while reducing fully connected capacity below 1,024 units increases WER.A 128/256 hidden-unit configuration across the two convolutional layers gives a slight improvement.
  • Number of hidden units: Table 3 reports WER as a function of the number of hidden units in the convolutional layers.The relevant comparison concerns convolutional capacity under a fixed total parameter budget.
  • Limited vs. full weight sharing: Speech frequency regions differ substantially, motivating limited weight sharing so local filters can specialize to confusable signal regions.Full weight sharing instead permits multiple convolutional layers and can use more hidden units to model frequency variation.
  • Limited vs. full weight sharing: With two convolutional layers and matched parameters, LWS improves WER by only 0.1% over FWS, indicating similar performance.FWS is preferred because it is simpler to implement, and the 5.6M-parameter 256/256 setting is used subsequently.

3.5. Pooling Experiments

The pooling experiments test frequency, overlap, pooling type, and time pooling strategies for LVCSR CNNs. Unlike computer-vision results, most regularization-oriented pooling variants provide little benefit on larger speech tasks.

  • Frequency pooling: Pooling reduces spectral variance, and frequency pooling was evaluated across 8 kHz Switchboard and 16 kHz Broadcast News tasks.The experiments compared pooling sizes across tasks with different sampling rates and speaking characteristics.
  • Frequency pooling: Pooling size 3 was optimal for both 50-hour tasks, while the Broadcast News experiment did not include a no-pooling condition.The no-pooling condition had already failed to help on Switchboard.
  • Pooling types: lp and stochastic pooling were compared with max pooling because max pooling can overfit training data and generalize poorly.lp pooling trades off averaging and max pooling; stochastic pooling samples activations using normalized regional probabilities.
  • Pooling types: Stochastic pooling provided slight improvements over max and lp pooling, but these generalization methods offered little benefit on data-rich speech tasks.The authors contrast this with larger improvements reported for computer-vision tasks.
  • Overlapping pooling: Overlapping pooling produced no WER difference from non-overlapping pooling when parameter counts were matched.The authors attribute the lack of benefit to speech tasks having more data and better model estimates than smaller vision tasks.
  • Pooling in time: Time pooling helped slightly with stochastic and lp pooling, but the gains were not large and were likely diminished after sequence training.For large tasks, the authors conclude that time pooling and related regularization schemes are not helpful.

3.6. Incorporating Speaker-Adaptation into CNNs

The paper develops a way to use speaker-adapted fMLLR features with CNNs while preserving the local correlations CNNs require. It decorrelates features for fMLLR, then maps them back into correlated space for CNN processing.

  • Feature requirements: CNNs require input features with locality and correlation in time and frequency, making standard decorrelated feature spaces unsuitable.VTLN-warped log-mel features were previously found effective for CNNs, whereas fMLLR is commonly applied in decorrelated spaces.
  • Motivation: Applying fMLLR directly to correlated log-mel features with a diagonal-covariance assumption produced little WER improvement.Correlated features are better modeled by full-covariance Gaussians, but those models require many parameters and can yield non-robust estimates.
  • Proposed transformation: STC supports decorrelation while sharing a few full-covariance matrices across distributions, with each distribution retaining its own diagonal covariance matrix.This provides the covariance structure used to apply fMLLR in the decorrelated feature space.
  • Proposed transformation: The proposed method first decorrelates correlated features with an STC matrix, estimates fMLLR in that space, and transforms the adapted features back.This preserves a correlated representation suitable for CNN input while allowing diagonal-Gaussian fMLLR modeling.

3.7. Multi-scale CNN/DNNs

The paper tests multi-scale CNN/DNNs by combining representations from different network layers, reflecting their progression from general to specific information. A small gain is observed when DNN- and CNN-style features are combined.

  • 3.7. Multi-scale CNN/DNNs: Multi-scale networks combine outputs from different layers to exploit potentially complementary representations.Lower speech-network layers emphasize speaker adaptation, while higher layers emphasize discrimination.
  • 3.7. Multi-scale CNN/DNNs: The experiment combines outputs from two fully connected and two convolutional layers, then feeds them into four jointly trained fully connected layers.The same log-mel input features serve both the DNN and CNN streams.
  • 3.7. Multi-scale CNN/DNNs: Combining DNN and CNN features produces a small gain.

3.8. I-vectors

The paper improves CNN input features by applying fMLLR in a decorrelated space rather than directly to correlated VTLN-warped log-mel features. This yields a modest improvement over the baseline system.

  • 3.8. I-vectors: Applying fMLLR in a decorrelated space achieves a 0.5% improvement over the baseline VTLN-warped log-mel system.The gain was not observed when fMLLR was applied directly to correlated log-mel features.
  • 3.8. I-vectors: The improved fMLLR approach is evaluated through WER comparisons in Table 9.

3.9. Rectified Linear Units and Dropout

This section examines ReLU, dropout, and Hessian-free sequence training for CNNs, including how dropout masks affect conjugate-gradient optimization. Fixing masks across CG iterations enables dropout during HF training and improves WER.

  • 3.9. Rectified Linear Units and Dropout: Sequence training provides an additional 10-15% relative improvement over a cross-entropy-trained DNN.The sequence-level objective is more closely linked to speech recognition than cross-entropy.
  • 3.9. Rectified Linear Units and Dropout: Hessian-free optimization uses a damped Gauss-Newton curvature approximation and conjugate gradient to minimize a quadratic loss approximation.The gradient is computed first, then CG runs until progress falls below a tolerance.
  • 3.9. Rectified Linear Units and Dropout: Dropout randomly omits hidden units with probability p to prevent complex co-adaptations during training.Scaling by 1-p during training preserves the correct total input when no units are dropped at test time.
  • 3.9. Rectified Linear Units and Dropout: HF training requires fixing each utterance-and-layer dropout mask across all CG iterations to preserve conjugate search directions.Masks may still vary across layers and utterances.
  • 3.9.4. Results: A fixed dropout mask yields a 0.6% WER improvement, whereas varying the mask increases WER and slows later HF loss convergence.The experiments use dropout probability p = 0.5 in the third and fourth layers and zero elsewhere.
  • 3.9.4. Results: Reducing the number of CE iterations before HF training is also investigated because sequence training is the more task-aligned objective.Table 11 reports WER after HF training for different CE iteration counts.

4. RESULTS

The proposed CNN systems are evaluated on 50-hour and 400-hour English Broadcast News tasks against DNN and earlier CNN systems. Hybrid CNNs show the strongest gains, while CNN-based feature improvements are modest or can deteriorate after extraction.

  • 4. RESULTS: The proposed results combine fMLLR and ReLU+dropout improvements in both 50-hour and 400-hour Broadcast News experiments.
  • 4.1. RESULTS: On 50-hour Broadcast News, the proposed CNN hybrid system improves 6-7% relative over the DNN hybrid and 2-3% over the old CNN hybrid.
  • 4.1. RESULTS: On 50-hour Broadcast News, the proposed CNN-based feature system improves 1% over the old CNN-based feature system.Feature-based gains are smaller than hybrid-system gains.
  • 4.2. RESULTS: On 400-hour Broadcast News, the 5,999-hybrid CNN improves 13-16% relative over the DNN hybrid and 4-5% over the old CNN-based feature systems.
  • 4.2. RESULTS: The 512-hybrid CNN improves WER from 14.8 to 14.1 over the old CNN, but performance slightly deteriorates after CNN-based features are extracted.

5. CONCLUSIONS

The paper improves CNN performance by incorporating fMLLR into CNN features and making dropout effective after Hessian-free sequence training. Pooling and weight-sharing strategies explored from computer vision did not improve LVCSR performance.

  • fMLLR-enhanced features and effective post-sequence-training dropout improved previous best CNN results by 2-5% relative.The paper identifies these two ideas as the sources of the overall improvement.
  • Pooling and weight-sharing techniques popular in computer vision did not improve performance on LVCSR tasks.
Loading 1309.1501v3…