PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Gated Recurrent Units for Speech Recognition

Mirco Ravanelli, Philemon Brakel, Maurizio Omologo, Yoshua Bengio

arXiv:1803.10225v1eess.AScs.NEcs.SDeess.SP

TL;DR

Robust ASR remains difficult in noisy and reverberant environments, motivating improved recurrent acoustic models. The paper revises GRUs into Li-GRU by removing the reset gate, using ReLU activations, and coupling them with batch normalization; across varied ASR settings, Li-GRU improves recognition accuracy and reduces per-epoch training time by more than 30% versus standard GRU.

  • Problem

    Speech recognizers still lack satisfactory robustness and flexibility in challenging acoustic conditions involving substantial non-stationary noise and reverberation.

  • Method

    Li-GRU revises GRUs by removing the reset gate, replacing tanh with ReLU activations, and using batch normalization.

  • Results

    Across different ASR paradigms, tasks, features, and environmental conditions, Li-GRU improves recognition accuracy and reduces training time by more than 30% versus standard GRU.

  • Takeaways & Limitations

    The proposed single-gate architecture provides a compact and computationally efficient GRU variant that is effective across DNN-HMM and end-to-end CTC speech recognition.

Abstract

from arXiv · show

A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end CTC models.

I. INTRODUCTION

ASR has advanced with deep learning and RNN-based acoustic models, but robustness remains difficult in noisy and reverberant conditions. This paper revises GRUs for speech recognition by removing the reset gate and replacing tanh with ReLU activations.

  • ASR systems still lack satisfactory robustness and flexibility in acoustic conditions involving non-stationary noise and reverberation.
  • RNNs are used in speech recognition because they manage temporal context and capture long-term speech modulations.
  • The paper revises standard GRUs for speech recognition rather than deriving a general-purpose recurrent architecture.
  • The proposed design removes the reset gate and replaces hyperbolic tangent with ReLU activations.The authors associate reset-gate removal with redundancy alongside the update gate and discuss ReLU with batch normalization.
  • More than 30% lower per-epoch training wall-clock time accompanies improved recognition accuracy across varied ASR tasks, conditions, and frameworks.The evaluations include different datasets, input features, noisy conditions, DNN-HMM systems, and CTC models.

II. GATED RECURRENT UNITS

GRUs simplify LSTMs with two gates that regulate recurrent state updates. Their update gate interpolates previous and candidate states for long-term dependencies, while the reset gate can discard past memory.

  • RNN training can suffer from vanishing and exploding gradients, motivating gated architectures that control information flow.
  • LSTMs use forget, input, and output gates, but their sophisticated gating mechanism can produce an overly complex model.
  • GRUs simplify LSTMs by using two multiplicative gates, the update gate and reset gate.
  • The update gate sets a linear interpolation between the previous state and candidate state, supporting retention of long-term information.When z_t is close to one, the previous state can remain unchanged; when it is close to zero, the candidate state receives more weight.
  • Fig. 1 plots average update- and reset-gate activations for a TIMIT-trained GRU on a segment of one utterance.
  • The reset gate influences the candidate state and allows the model to possibly delete previously computed memory.

III. A NOVEL GRU FRAMEWORK

The proposed GRU framework removes the reset gate because speech evolves slowly and reset behavior may discard useful history. The resulting single-gate model is more computationally efficient, while gate redundancy motivates the simplification.

  • A. Removing the reset gate: Resetting memory may help at major sequence discontinuities, but speech features are typically computed every 10 ms and past history can remain useful.The authors caution that completely resetting memory at strong speech discontinuities can be harmful because phonotactic history remains informative.
  • A. Removing the reset gate: Update and reset gates can produce similar effects when emphasizing current information or past states, indicating redundancy during speech processing.The paper illustrates this redundancy through temporally correlated average gate activations and proposes quantitative analysis with cross-correlation.
  • A. Removing the reset gate: The first architectural change removes the reset gate from the standard GRU.
  • A. Removing the reset gate: A more compact single-gate model improves computational efficiency.

B. ReLU activations

Li-GRU replaces tanh with ReLU and combines the activation with batch normalization. This design aims to retain ReLU’s benefits while limiting numerical instability and simplifying the model.

  • B. ReLU activations: The candidate state computation replaces the standard hyperbolic tangent activation with ReLU.
  • B. ReLU activations: ReLU activations can be effective in feed-forward networks but may cause numerical instabilities in recurrent models over long time series.
  • C. Batch Normalization: Batch normalization normalizes layer pre-activation means and variances for each training minibatch.
  • C. Batch Normalization: Applied to feed-forward connections, batch normalization can be computed independently at each time step and with reduced computational effort.
  • C. Batch Normalization: Batch normalization rescales neuron pre-activations, inherently bounding ReLU neuron values and helping limit numerical issues.
  • C. Batch Normalization: The Li-GRU formulation combines reset-gate removal, ReLU candidate activation, and batch normalization.The paper presents these changes as the defining components of the proposed model.

D. Related work

The paper builds on prior single-gate GRU work and extends ReLU-based recurrent modeling to GRUs for speech recognition. Its novelty integrates reset-gate removal, ReLU activations, and batch normalization, with broader analyses and evaluation than earlier work.

  • Earlier M-GRU work removed the reset gate and matched standard GRU performance in handwritten digit recognition and sentiment classification.
  • The paper extends reset-gate removal and ReLU activations to a GRU architecture designed specifically for speech recognition.
  • Li-GRU integrates reset-gate removal, ReLU activations, and batch normalization as three key design aspects for speech recognition.
  • Compared with the earlier study, this work adds correlation and gradient analyses, examines batch normalization, evaluates more datasets and acoustic conditions, and includes end-to-end CTC validation.

A. Corpora and tasks

The experiments cover close-talking, distant-talking, noisy, reverberant, and larger-scale speech-recognition tasks. They use multiple corpora and bidirectional recurrent architectures with regularization and length-sorted minibatches.

  • The corpora include TIMIT for close-talking phoneme recognition, DIRHA-English for distant-talking domestic conditions, CHiME 4 for noisy environments, and TED-talk for larger-scale ASR.
  • DIRHA-English evaluates speech with non-stationary noise averaging about 10 dB SNR and reverberation averaging about 0.7 seconds T60.
  • CHiME 4 contains real and simulated recordings from buses, cafes, pedestrian areas, and street junctions.
  • TED-talk provides about 166 hours of training speech from 820 talks, with separate development and test talks.
  • The recurrent setup stacks bidirectional RNN layers before a softmax classifier and uses recurrent dropout; training sorts utterances by length into minibatches of eight.

C. DNN-HMM setup

The DNN-HMM experiments compare recurrent acoustic models using several frame-level acoustic features and an HMM decoder that combines acoustic, lexicon, and language-model information. The broader study also includes a bidirectional CTC configuration.

  • DNN-HMM acoustic models predict context-dependent phone targets from 25 ms frames overlapping by 10 ms.
  • The experiments use 39 MFCC, 40 log-mel filter-bank, and 40 fMLLR acoustic features.
  • Frame posteriors are prior-normalized, converted to likelihoods, and decoded with an HMM search graph integrating acoustic, lexicon, and language-model information.
  • The CTC models use five bidirectional RNN layers with either 250 or 465 units and apply weight noise to all weight matrices.
  • The experimental activity primarily uses hybrid DNN-HMM recognizers and extends validation to an end-to-end CTC model.

A. Correlation analysis

The correlation and gradient analyses examine whether GRU update and reset gates play redundant roles. The reported results show strong, rapidly emerging gate correlation and comparatively smaller reset-gate gradient norms.

  • Correlation analysis: A cross-correlation between average update- and reset-gate activations is compared with update-gate auto-correlation across input frames.
  • Correlation analysis: 66% of the maximum update-gate auto-correlation occurs as the zero-delay cross-correlation peak, indicating substantial redundancy between the gates.
  • Correlation analysis: After 3-4 training epochs, the gate-correlation peak reaches its maximum and remains nearly unchanged through later iterations.
  • Gradient analysis: The gradient analysis reports smaller L2 norms for reset-gate matrices Wr and Ur than for the other main GRU parameters.

C. Role of batch normalization

Batch normalization improves ASR performance, with the largest reported relative gain for Li-GRU, whose ReLU activations otherwise create numerical instabilities. Across TIMIT evaluations, Li-GRU consistently outperforms other recurrent architectures while using fewer parameters.

  • Role of batch normalization: 18% relative improvement is obtained for Li-GRU with batch normalization, compared with about 7% for GRU and M-GRU.Without batch normalization, Li-GRU’s unbounded ReLU activations tend to cause numerical instabilities.
  • Other results on TIMIT: Li-GRU consistently outperforms the other RNN architectures across all considered TIMIT input features.Gated recurrent networks outperform a simple ReLU RNN, while GRUs slightly outperform LSTM in the reported comparison.
  • Other results on TIMIT: 14.9% average PER is achieved by Li-GRU with fMLLR features on the TIMIT test set.The authors identify this as the best published performance on the TIMIT test set to their knowledge.
  • Other results on TIMIT: Li-GRU achieves the best PER results for all five considered phonetic categories.The categories are vowels, liquids, nasals, fricatives, and stops.
  • Other results on TIMIT: M-GRU and Li-GRU use about 30% fewer parameters than the standard GRU.M-GRU removes the reset gate, while Li-GRU additionally replaces tanh activations with ReLU.

F. Recognition performance on DIRHA English WSJ

Li-GRU is evaluated on the challenging DIRHA English WSJ distant-talking task, where training and testing differ acoustically through reverberation and noise. It outperforms GRU consistently across real and simulated data, features, and tested reverberation conditions.

  • Recognition performance on DIRHA English WSJ: Li-GRU outperforms GRU consistently on both real and simulated DIRHA English WSJ data across the considered features.The dataset includes acoustic mismatch, non-stationary noise, and reverberation between training and testing conditions.
  • Recognition performance on DIRHA English WSJ: Removing the reset gate yields a small performance gain over standard GRU in the distant-talking setting.The authors hypothesize that reverberation creates forward memory through multiple delayed signal replicas, making the reset gate ineffective.
  • Recognition performance on DIRHA English WSJ: Li-GRU outperforms GRU across reverberation times T60 ranging from 250 to 1000 ms.Performance degrades as reverberation time increases; these experiments use different impulse responses for training and testing without additive noise.

G. Recognition performance on CHiME

On the CHiME benchmark, Li-GRU consistently outperforms GRU, including across noisy environments, while the comparison is not directly against the competition’s best systems.

  • G. Recognition performance on CHiME: The CHiME comparison reports results for multiple GRU architectures and includes official CHiME 4 results for comparison.The official results appear in the first two table rows.
  • G. Recognition performance on CHiME: The reported CHiME results are not directly comparable with the best CHiME 4 competition systems.Multi-microphone processing, data augmentation, system combination, and lattice rescoring were not used.
  • G. Recognition performance on CHiME: About 14% relative improvement is achieved when passing from GRU to Li-GRU on CHiME.Removing the reset gate provides small benefits, while adopting ReLU units produces the largest performance gap.
  • G. Recognition performance on CHiME: Li-GRU outperforms GRU in all four considered noisy environments on the real evaluation set.The relative improvement is 16% in BUS and 9.5% in STR recordings.

H. Recognition performance on TED-talks

On TED-talks, Li-GRU improves recognition performance over GRU, and the paper also relates its compact design to reduced computational cost across datasets.

  • H. Recognition performance on TED-talks: The TED-talks experiments use standard MFCC features and a four-gram language model during decoding.The comparison is between GRU and Li-GRU on the TED-talks corpus.
  • H. Recognition performance on TED-talks: About 14–17% relative improvement is achieved with Li-GRU on both TED-talks test sets.The improvement is statistically significant with the MPSSW test at p-value 0.01.
  • VI. CONCLUSIONS: Li-GRU improves recognition performance and reduces computational complexity across different ASR paradigms, tasks, features, and environmental conditions.The paper reports more than 30% training-time reduction over a standard GRU.
  • VI. CONCLUSIONS: The paper identifies extending Li-GRU to speech enhancement and speech separation as future work.It also proposes exploring Li-GRU in other possible fields.
Loading 1803.10225v1…