PaperlayerView exact revision on arXiv

Source-linked AI summary

Building DNN Acoustic Models for Large Vocabulary Speech Recognition

Andrew L. Maas, Peng Qi, Ziang Xie, Awni Y. Hannun, Christopher T. Lengerich, Daniel Jurafsky, Andrew Y. Ng

arXiv:1406.7806v2cs.CLcs.LGcs.NEstat.ML

TL;DR

The paper examines how to compare DNN acoustic-model design choices when it is difficult to establish which changes improve downstream speech recognition. It systematically varies architectures and training choices, then finds that model size mattered more than depth in its experiments, with large DNNs gaining on frame classification and WER when trained on a large corpus.

  • Problem

    Acoustic-model improvements are difficult to assess without a well-established baseline, and frame-level classification performance has a complex relationship to final-system WER.

    Source: §III · p. 4, A. Cross Entropy Loss Function · p. 5

  • Method

    The authors systematically vary DNN architecture and training choices, including size, depth, optimization, and locally connected variants.

    Source: §II · p. 3, §III · p. 4

  • Results

    With a large training corpus, DNNs up to 400M parameters and seven hidden layers gained on frame classification and WER, while total size mattered more than depth.

    Source: §X · p. 19

  • Takeaways & Limitations

    In these experiments, network size was the most critical factor, while depth mattered chiefly for having more than one hidden layer.

    Source: §X · p. 19

Abstract

from arXiv · show

Deep neural networks (DNNs) are now a central component of nearly all state-of-the-art speech recognition systems. Building neural network acoustic models requires several design decisions including network architecture, size, and training loss function. This paper offers an empirical investigation on which aspects of DNN acoustic model design are most important for speech recognition system performance. We report DNN classifier performance and final speech recognizer word error rates, and compare DNNs using several metrics to quantify factors influencing differences in task performance. Our first set of experiments use the standard Switchboard benchmark corpus, which contains approximately 300 hours of conversational telephone speech. We compare standard DNNs to convolutional networks, and present the first experiments using locally-connected, untied neural networks for acoustic modeling. We additionally build systems on a corpus of 2,100 hours of training data by combining the Switchboard and Fisher corpora. This larger corpus allows us to more thoroughly examine performance of large DNN models -- with up to ten times more parameters than those typically used in speech recognition systems. Our results suggest that a relatively simple DNN architecture and optimization technique produces strong results. These findings, along with previous work, help establish a set of best practices for building DNN hybrid speech recognition systems with maximum likelihood training. Our experiments in DNN optimization additionally serve as a case study for training DNNs with discriminative loss functions for speech tasks, as well as DNN classifiers more generally.

I. INTRODUCTION

The paper investigates DNN acoustic-model design choices because differences between systems make it hard to isolate what improves performance, and classifier gains do not necessarily translate to lower LVCSR word error rates. It reviews the hybrid HMM-DNN pipeline and tests design and training choices on Switchboard and combined Switchboard–Fisher data.

I. INTRODUCTION

  • Classifier improvements do not necessarily predict lower LVCSR word error rates, complicating comparisons of acoustic-model design choices across systems.The paper identifies the relationship between DNN classification performance and downstream transcription performance as complex.Source: §I · p. 1
  • Experiments span 300 hours of Switchboard data and a combined 2,100-hour Switchboard–Fisher corpus, including DNNs up to ten times larger than typical LVCSR models.The Switchboard studies examine size, overfitting, regularization, and dense versus convolutional architectures; the larger corpus enables tests of larger models, optimization, and depth.Source: §I · pp. 1–2

II. NEURAL NETWORK ACOUSTIC MODELS

  • In hybrid HMM systems, neural networks estimate senone posteriors p(y|x), which are converted into acoustic scores for decoding.The paper describes using an unscaled acoustic-model score with an empirically adjusted scaling term during decoding.Source: §II · p. 2
  • Modern DNN acoustic models use context-dependent states and deeper, larger networks, but the roles of model size and depth remain incompletely understood.The paper also situates DNNs among alternatives such as convolutional and recurrent architectures.Source: §II · pp. 2–3
  • The acoustic-model pipeline includes senone labeling, forced alignment, architecture, loss and regularization, and optimization choices.Training uses a baseline HMM-GMM forced alignment; cross entropy is the default loss, while SGD and alternative optimizers are evaluated as design choices.Source: §II · pp. 3–4

III. QUESTIONS ADDRESSED IN THIS WORK

The paper addresses the lack of a well-established acoustic-model baseline by systematically varying design and training choices. Its experiments compare architectures and optimization, assess generalization, and examine model performance and information encoding.

III. QUESTIONS ADDRESSED IN THIS WORK

  • Without a well-established acoustic-model baseline, it is difficult to assess the relative importance of design and training decisions for LVCSR.The work varies several components systematically to test architectural choices and their relevance to task performance.Source: §III · p. 4
  • The study compares network size, depth, optimization algorithms, and dense, convolutional, and locally untied architectures across two training-data settings.DLUNNs use local connections with distinct weights at each input location, unlike DCNNs, which share weights across locations.Source: §III · pp. 4–5
  • The experiments test dropout, early stopping, and early realignment for generalization, and analyze classification errors and information encoding in larger or deeper DNNs.The paper describes the computations for three neural-network architecture classes and the loss and optimization algorithms used.Source: §III · p. 5, §IV · p. 5

A. Cross Entropy Loss Function

The section explains the paper’s cross-entropy training choice and the standard DNN computation used to estimate class probabilities. It distinguishes frame-level classification from the ultimate goal of reducing word error rate.

A. Cross Entropy Loss Function

  • Cross entropy is used for all experiments, with regularization in some experiments; it is a standard classification loss and often precedes task-specific speech losses.The authors use it to make the experiments relevant to large-scale DNN classification as well as acoustic modeling.Source: A. Cross Entropy Loss Function · p. 5
  • The cross-entropy objective is defined over an individual acoustic input and its senone label, using the model’s probability for the target class.The supplied passage identifies K as the number of output classes and y_k as the probability assigned to class k.Source: A. Cross Entropy Loss Function · p. 5
  • Frame-level classification accuracy and final-system WER have a complex relationship, so the paper reports both frame-level error metrics and system-level WER.HMM and language-model constraints can correct some state-level errors, while acoustic spans differ in their importance to word transcription.Source: A. Cross Entropy Loss Function · p. 5

B. Deep Neural Network Computations

  • A DNN transforms an input vector through fully connected hidden layers into a probability distribution over output classes.The network uses weight matrices and biases at each layer, with rectified linear units in hidden layers and softmax at the output.Source: B. Deep Neural Network Computations · pp. 5–6
  • The softmax output supplies class probabilities for the loss, and a sub-gradient is used with gradient-based optimization despite the rectifier’s nondifferentiability at zero.Fully connected networks are a standard acoustic-model architecture, while their total size and depth have not been thoroughly studied.Source: B. Deep Neural Network Computations · p. 6

C. Deep Convolutional Neural Networks

Deep convolutional networks replace the initial fully connected layers with local, shared filters that exploit time-frequency structure. Pooling downsamples the resulting feature maps and introduces invariance to slight input shifts.

C. Deep Convolutional Neural Networks

  • DCNNs use localized connections and shared weights to exploit local time-frequency structure while limiting the number of model parameters.The same localized hidden units are applied at multiple input locations.Source: C. Deep Convolutional Neural Networks · p. 6
  • A convolutional filter is applied across input locations to produce a feature map that retains time and frequency dimensions.In the described layer, the filter moves one step at a time in both dimensions.Source: C. Deep Convolutional Neural Networks · p. 6, Figure 2 · p. 7
  • Non-overlapping max-pooling selects the largest activation in each local region, downsampling feature maps and encoding invariance to slight shifts.Pooling is applied separately to each hidden unit’s feature map; averaging is a common alternative with often comparable results.Source: C. Deep Convolutional Neural Networks · pp. 6–7, Figure 2 · p. 7
  • A DCNN stacks convolution and pooling before densely connected layers and a softmax classifier, requiring additional architectural hyperparameter choices.Dense and convolutional layers are not interleaved because dense layers do not preserve spatial or time-frequency relationships.Source: C. Deep Convolutional Neural Networks · p. 7

D. Deep Local Untied Neural Networks

The paper describes locally connected untied networks and compares two stochastic-gradient optimization methods for acoustic modeling. Untied filters allow different frequency regions to use different weights, while Nesterov’s accelerated gradient looks ahead along the update direction.

D. Deep Local Untied Neural Networks

  • DLUNNs use locally connected units without sharing weights across input regions, allowing different frequency ranges to learn different filters.This differs from DCNNs, which share filters; the untied layers therefore use more parameters.Source: D. Deep Local Untied Neural Networks · p. 7
  • Each DLUNN location learns a unique 5 × 5 weight set, followed by non-overlapping 3 × 3 max-pooling regions.The pooling layer behaves as it does in a convolutional architecture.Source: D. Deep Local Untied Neural Networks · p. 7, Figure 3 · p. 8

E. Optimization Algorithms

  • Classical momentum accumulates gradient updates in a velocity vector, with the momentum constant controlling accumulation across past updates.For extremely ill-conditioned problems, high classical momentum can cause parameter-update fluctuations and slower convergence.Source: E. Optimization Algorithms · p. 8
  • Nesterov’s accelerated gradient (NAG) evaluates the gradient along the update direction to avoid potential fluctuations during optimization.The paper compares NAG with classical momentum (CM) empirically for acoustic modeling.Source: E. Optimization Algorithms · p. 8

V. SWITCHBOARD 300 HOUR CORPUS

The Switchboard experiments use a 300-hour conversational telephone speech corpus and a hybrid HMM/DNN recognizer. They vary DNN size and input context to test whether larger models improve frame-level and final system performance.

V. SWITCHBOARD 300 HOUR CORPUS

  • The experiments use a 300-hour Switchboard corpus, with DNN state likelihoods in a hybrid HMM/DNN recognizer and evaluation on HUB5 2000 test subsets.The system uses MFCCs with approximately 10 context frames, per-speaker CMVN, and fMLLR adaptation.Source: §V · p. 8

A. Varying DNN Model Size

  • The model-size experiment asks whether increasing parameters alone can improve overall system performance, and also tests a larger input context.The authors note that a larger context should offer a direct route to improving frame classification.Source: A. Varying DNN Model Size · pp. 8–9
  • The five-hidden-layer models contain approximately 36M, 100M, or 200M parameters, with 2,048, 3,953, or 5,984 units per hidden layer, respectively.For the larger models, the experiments also test an input context of approximately 20 frames instead of approximately 10.Source: 1) Experiments · p. 9

1) Experiments:

  • Models use rectified linear hidden units and Nesterov’s accelerated gradient, with 512-example mini-batches and learning-rate halving after each epoch.Training stops when held-out development-set cross-entropy improvement falls below a small tolerance.Source: 1) Experiments · p. 9

2) Results:

Larger DNNs improve frame-level metrics and reduce training-set WER, but the training gains do not translate into large evaluation-set gains. The results also discuss training dynamics and describe the dropout experiment’s setup.

2) Results:

  • The 200M-parameter DNN halves development-set cross-entropy cost versus the 36M model, and each size increase brings approximately a 10% absolute frame-classification accuracy increase.Larger context windows improve frame-level metrics, with the best frame-level model using 200M parameters and approximately 20 context frames.Source: 2) Results · p. 9
  • Models larger than 100M parameters provide no evaluation-set WER benefit for this task, despite reducing training-set WER.The authors report a small evaluation benefit over the 36M baseline, but further training-set reductions do not translate to large evaluation gains.Source: 2) Results · p. 9

3) Discussion:

  • Evaluation-set performance is mostly achieved within the first few epochs, while training WER continues to decline afterward.The authors report that larger models do not show a strong pattern of evaluation performance worsening after initially improving.Source: 3) Discussion · p. 9, Figure 4 · p. 10

B. Dropout Regularization

  • The dropout experiment compares same-architecture models, using a dropout probability of 0.1 selected from several tested values.The setup otherwise follows the earlier training and evaluation protocol and uses the baseline HMM-GMM forced alignments.Source: B. Dropout Regularization · p. 10, 1) Experiments · p. 10

2) Results:

The best-case early-stopped 200M model reaches 20.7% WER on the EV subset, with limited improvement over the smaller models. The section also proposes early realignment during DNN optimization to refresh training labels.

C. Early Stopping

  • Early stopping halts loss optimization before convergence, and the paper evaluates it as a possible way to improve generalization for large acoustic models.The early-stopping result is assessed from training and test WER curves.Source: C. Early Stopping · p. 10, 1) Results · p. 11

1) Results:

  • 20.7% WER on the EV subset is the best-case early-stopped result for the 200M model, just 0.1% better than the 100M baseline and 0.5% better than the 36M model.The authors conclude that early stopping helps but may not deliver the full possible benefits of large DNN acoustic models.Source: 1) Results · p. 11

D. Early Realignment

  • The early-realignment rationale is that forced-alignment labels can be imperfect because the baseline system and speech framework do not fully account for pronunciation variation.The approach assumes sufficiently correct noisy labels make significantly corrupted labels appear as outliers.Source: D. Early Realignment · p. 11
  • Early realignment generates a new forced alignment early in DNN optimization, then continues training on the refreshed labels.The authors hypothesize that an early, less-converged network may remove some label distortions before it fits the corruptions in the original alignment.Source: D. Early Realignment · p. 11

1) Experiments:

Early forced-alignment realignment after two epochs was the strongest regularization approach evaluated on Switchboard, outperforming the tested dropout and early-stopping models. The authors connect its benefit to relabeling early in optimization, when the DNN behaves like a higher-bias model, and report a training-time advantage over later realignment.

1) Experiments:

  • The experiment held other training and evaluation procedures fixed while testing 100M-parameter, five-hidden-layer DNNs realigned after two or five epochs.Training began from HMM-GMM alignments and continued with the same DNN weights after labels were regenerated.Source: 1) Experiments · p. 11

2) Results:

  • Early realignment after two epochs outperformed all evaluated dropout and early-stopping models on Switchboard, while realignment after five epochs beat baseline but trailed the two-epoch model.The two-epoch system was the overall best regularization technique evaluated; five-epoch realignment performed comparably to same-size dropout.Source: 2) Results · pp. 11–12, Figure 5 · p. 12

3) Discussion:

  • Realignment initially raised train and test WER, but the model began outperforming the baseline three epochs later.Figure 5 compares WER by training epoch for models with and without realignment after epoch two.Source: 3) Discussion · p. 12, Figure 5 · p. 12
  • Early realignment changed 16.4% of labels, versus 10% after five epochs; the authors interpret early relabeling as using a higher-bias, smoother approximation.They suggest a large-capacity DNN later fits corrupted training samples so well that it reproduces the original labels more closely.Source: 3) Discussion · p. 12
  • A five-epoch realignment system needed three more training epochs—eight total—to match early realignment, amounting to several days of compute at this model scale.Realignment adds implementation work but can use forced-alignment infrastructure already present in modern LVCSR systems.Source: 3) Discussion · p. 12

VI. COMPARING DNNS, DCNNS, AND DLUNNS ON SWITCHBOARD

  • The paper next turns from regularization to comparing deep convolutional and deep locally untied architectures with standard DNNs.Source: §VI · p. 12

A. Experiments

The Switchboard architecture experiments compare standard DNNs, convolutional networks, and locally untied networks using filter-bank and fMLLR features. Their relative performance depends on feature type, and the authors conclude that neither convolutional nor locally untied models should replace DNNs as the default architecture.

A. Experiments

  • DCNNs and DLUNNs were evaluated on the same Switchboard data as DNNs, with filter-bank features added because they have meaningful time-frequency dimensions for local receptive fields.The compared models had five hidden layers and approximately 36M parameters; the table covers DCNNs with one or two convolutional layers and permuted-feature controls.Source: A. Experiments · p. 12, Table III · p. 13

B. Results

  • With filter-bank features, DCNNs and DLUNNs improved over DNNs, and DCNNs narrowly beat DLUNNs; with fMLLR, DLUNNs were about as good as DNNs and outperformed DCNNs.The authors suggest tied convolutional weights help with filter-bank features, whereas DLUNNs’ flexibility helps with fMLLR features lacking meaningful frequency structure.Source: Table III · p. 13, B. Results · p. 13

A. Experiments

  • The reported convolutional and locally untied results use an approximately 10-frame context; an approximately 20-frame context and other preliminary settings performed worse or gave no improvement.The experiments also tested frequency-only convolution, pooling across time and frequency, and overlapping pooling regions.Source: A. Experiments · p. 13

B. Results

  • Randomly permuting filter-bank features harmed performance, yet DCNNs remained fairly competitive, supporting the authors’ conclusion that local models leverage localized input structure.Source: B. Results · p. 13

C. Discussion

  • The authors conclude DCNNs and DLUNNs are not reliable default replacements for DNNs, while DLUNNs merit further investigation as acoustic models.They note that fMLLR’s substantial speaker-adaptation post-processing appears beneficial across the evaluated architectures.Source: C. Discussion · p. 13

VII. COMBINED LARGE CORPUS

  • The authors introduce the larger-corpus experiments to examine architecture choices when training-set size is less limiting than in the 300-hour Switchboard work.They describe the next corpus as combined Switchboard and Fisher data.Source: §VII · pp. 13–14

A. Baseline HMM system

On the combined Fisher–Switchboard corpus, the authors compare optimization settings and scale DNNs up to 400M parameters. NAG with maximum momentum 0.99 and learning rate 0.01 performed best in the initial optimizer comparison; larger models improved WER with diminishing returns.

A. Baseline HMM system

  • The combined training set uses approximately 2,000 hours from Fisher alongside Switchboard; Fisher transcripts are described as slightly less accurate.Source: A. Baseline HMM system · p. 14
  • The baseline uses speaker-normalized MFCCs transformed with LDA, STC, and speaker-specific fMLLR; systems are evaluated on Eval2000 and RT-03.The full combined-data HMM-GMM baseline contains 8,725 tied triphone states and 3.2M Gaussians.Source: A. Baseline HMM system · p. 14

B. Optimization Algorithm Choice

2) Results:

  • Both classical momentum and NAG performed best with maximum momentum 0.99 and initial learning rate 0.01; their WER was nearly identical across evaluation sets.NAG narrowly outperformed classical momentum in frame-level accuracy.Source: 2) Results · p. 14

C. Scaling Total Number of DNN Parameters

  • The scaling comparison varied total parameters across five-hidden-layer DNNs while holding depth and optimization settings fixed.The tested sizes were 36M, 100M, 200M, and 400M parameters.Source: C. Scaling Total Number of DNN… · p. 14
  • The 400M-parameter DNN performed best in frame classification and WER, but scaling from 36M to 400M yielded diminishing relative WER gains on Eval2000.Relative WER gains were 3.8% from 36M to 100M, 2.5% from 100M to 200M, and 1% from 200M to 400M; the diminishing-gain trend also held on RT03 with smaller gains.Source: C. Scaling Total Number of DNN… · p. 15
  • Even the largest DNN leaves room to improve frame classification on this corpus, which the authors describe as challenging because of acoustic variation and quick-transcription errors.Source: C. Scaling Total Number of DNN… · p. 15

D. Number of Hidden Layers

For the combined-corpus task, deep DNNs substantially outperformed single-hidden-layer models, while gains largely leveled off beyond five layers. Error analysis shows that larger models reduce WER mainly through fewer substitutions, with frame-accuracy gains spread across phonemes.

D. Number of Hidden Layers

  • Five hidden layers showed a clear gain over three, while seven layers added little or no performance; deep models substantially outperformed single-hidden-layer models.The authors conclude that five layers may be deep enough for this task and that depth selection mattered more than total model size in their comparison.Source: D. Number of Hidden Layers · p. 15
  • Seven-layer DNNs had slightly lower frame classification than five-layer models at each size, including on training data, which the authors link to difficulty optimizing very deep networks.They suggest depth can act as a regularizer in practice despite deeper networks’ greater theoretical modeling capacity.Source: D. Number of Hidden Layers · p. 15

VIII. WER AND FRAME CLASSIFICATION ERROR ANALYSIS

  • Error analysis decomposes aggregate WER and frame accuracy to compare systems with similar overall scores, including by phoneme and by substitution, deletion, and insertion.Source: §VIII · pp. 15–16
  • As model size increased, lower substitution rates largely drove WER reductions, while insertions and deletions stayed relatively constant and were generally smaller components.The authors suggest insertion and deletion errors may reflect other system shortcomings, such as OOV words or pronunciation-dictionary coverage.Source: Figure 6 · p. 16, §VIII · p. 16
  • Correct senone classification increased or stayed level with model size for every base phoneme, with overall gains spread across many phonemes rather than concentrated in a few.The figure groups classification percentages by base phoneme and separates correct predictions, within-base errors, and errors across base phonemes; non-speech categories are omitted.Source: §VIII · p. 16, Figure 7 · p. 17

IX. ANALYZING CODING PROPERTIES

The section analyzes how trained DNNs encode inputs through hidden-unit activation patterns. Activations become sparser in deeper layers, while representations remain broadly dispersed and larger models do not develop many permanently inactive units.

A. Sparsity and Dispersion

  • The analysis measures each unit’s lifetime activation probability over 512,000 examples; sorted activation probabilities show sparsity and how evenly units share coding responsibility.Sparsity concerns how few units activate for an input, while dispersion concerns how evenly units share coding.Source: A. Sparsity and Dispersion · p. 17
  • The first hidden layer is more active on average—often by nearly a factor of two—while activation probability changes only slightly across deeper layers.The small changes suggest representations are transformed rather than continually compressed.Source: A. Sparsity and Dispersion · p. 17
  • Representations are generally dispersed, with mostly flat activation-probability curves and only a few units at either tail; dispersion trends slightly higher in deeper layers, especially in larger models.Dispersion is similar among layers within a DNN of a particular size.Source: A. Sparsity and Dispersion · pp. 17–18
  • Larger DNNs do not develop a significant set of permanently inactive units; only a small set of units are active for less than 1% of inputs.The authors note that adding parameters is useful only insofar as they are used to encode and transform inputs.Source: A. Sparsity and Dispersion · p. 18
  • Across model sizes, activation probability generally decreases in subsequent layers, but seven-layer models become less sparse in layers six and seven than in layer five.Larger models also tend to be more sparse and slightly more dispersed than smaller models.Source: A. Sparsity and Dispersion · p. 18

B. Code Length

The code-length analysis finds that larger models use more active units per layer, while seven-layer models show a late increase in code length correlated with no improvement over five layers. The conclusion favors model size over additional depth, while noting limits from isolated acoustic-model training and cross-entropy loss.

B. Code Length

  • Larger DNNs use more hidden units per layer; in the first layer, 100M-parameter models use nearly twice the code length of 36M models.The authors leave open whether longer codes capture more information or contain redundancy.Source: B. Code Length · p. 18
  • Code length generally stays constant or decreases with depth, but in seven-layer models it reaches a minimum at layer five and rises in layers six and seven.This pattern occurs at 36M, 100M, and 200M parameters and is correlated with no improvement over five-layer models; further experiments are needed to test the relationship.Source: B. Code Length · p. 18

X. CONCLUSION

  • The combined 2,100-hour corpus yielded a much larger gain than regularizing large DNNs trained with less data.On Switchboard, regularization improved large DNNs that otherwise overfit.Source: §X · p. 19
  • The standard DNN outperformed DCNN and DLUNN variants in frame classification and final-system WER, while DLUNNs performed slightly better than DCNNs.Standard DNNs also avoid assumptions that input features have meaningful time or frequency structure.Source: §X · p. 19
  • With up to 400M parameters, DNNs trained using NAG gained on frame classification and WER when the training corpus was large; total network size mattered more than depth, and depth beyond a point could harm performance.The authors recommend cross-validating size with at least three and no more than five hidden layers.Source: §X · p. 19
  • The experiments trained acoustic models in isolation with cross-entropy, and these models did not perform as well as DNNs trained with discriminative loss functions in previous work.The authors identify isolation from the larger ASR system as a source of uncertainty and hypothesize that larger DNNs may benefit from loss functions assigning them more of the ASR task.Source: §X · p. 19
  • The authors used approximately 300 lines of Python code to train DNNs, presenting simple architectures and optimization procedures as feasible routes to good system performance.They hope the work serves as a reference point for communication and reproducibility.Source: §X · p. 19

Related papers

Loading 1406.7806v2…