Source-linked AI summary
Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
Ying Zhang, Mohammad Pezeshki, Philemon Brakel, Saizheng Zhang, Cesar Laurent Yoshua Bengio, Aaron Courville
TL;DR
End-to-end speech recognition systems commonly rely on recurrent networks, whose sequential computation and difficult optimization can make training demanding. This paper combines hierarchical CNNs directly with CTC, without recurrent connections, and evaluates the approach on TIMIT phoneme recognition. The model achieves competitive phoneme recognition while training faster than the LSTM baseline in the reported setup.
Problem
End-to-end speech recognition relies heavily on RNNs/LSTMs, but their sequential computation can slow training and their optimization can be difficult.
Method
The paper combines CNNs directly with CTC, without intermediate recurrent layers, using sufficient depth and context to model temporal dependencies.
Results
18.2% phoneme error rate on TIMIT’s core test set was slightly better than the LSTM baseline, with 2.5× faster training in the reported setup.
Takeaways & Limitations
CNNs without recurrent connections can learn the temporal relations required for CTC and provide computational-efficiency gains on TIMIT.
Takeaways & Limitations
The discussion highlights over-fitting risk on small datasets such as TIMIT, while phoneme-level ease relative to character-level training remains a conjecture requiring empirical investigation.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) are effective models for reducing spectral variations and modeling spectral correlations in acoustic features for automatic speech recognition (ASR). Hybrid speech recognition systems incorporating CNNs with Hidden Markov Models/Gaussian Mixture Models (HMMs/GMMs) have achieved the state-of-the-art in various benchmarks. Meanwhile, Connectionist Temporal Classification (CTC) with Recurrent Neural Networks (RNNs), which is proposed for labeling unsegmented sequences, makes it feasible to train an end-to-end speech recognition system instead of hybrid settings. However, RNNs are computationally expensive and sometimes difficult to train. In this paper, inspired by the advantages of both CNNs and the CTC approach, we propose an end-to-end speech framework for sequence labeling, by combining hierarchical CNNs with CTC directly without recurrent connections. By evaluating the approach on the TIMIT phoneme recognition task, we show that the proposed model is not only computationally efficient, but also competitive with the existing baseline systems. Moreover, we argue that CNNs have the capability to model temporal correlations with appropriate context information.
1 CIFAR Senior Fellow, 2 CIFAR Fellow
The passage lists contact email addresses associated with the paper’s authors.
- The passage provides contact email addresses for the paper’s authors.
1. Introduction
The introduction motivates replacing recurrent components in end-to-end speech recognition with CNNs combined directly with CTC. It proposes sufficiently deep convolutional architectures as a way to capture temporal dependencies while retaining efficient parallel computation.
- 1. Introduction: Hybrid systems separately train neural and HMM/GMM modules, potentially requiring laborious hyperparameter tuning for each stage.
- 1. Introduction: RNN-based end-to-end systems face slow sequential training and difficult optimization from vanishing or exploding gradients.
- 1. Introduction: The proposed framework combines CNNs with CTC without intermediate recurrent layers for end-to-end speech recognition.
- 1. Introduction: The paper addresses whether CNNs can learn the temporal relationships required for end-to-end CTC speech recognition.
- 1. Introduction: Sufficiently deep CNNs use stacked convolutional layers to capture temporal dependencies across diverse time and frequency scales.
- 1. Introduction: The architecture uses 2D convolution for temporal modeling, frequency-only pooling in the first convolutional layer, fully connected layers, and CTC.
2. Convolutional Neural Networks
The CNN architecture uses stacked convolutions and frequency-only pooling to model local spectrogram structure while preserving temporal sequence length. It applies nonlinear activations before fully connected layers, with ReLU, PReLU, and Maxout considered as activation functions.
- Architecture: The model uses stacked convolutional and pooling layers followed by multiple fully connected layers.Its architecture differs from earlier speech CNNs by using stacked layers and fully connected layers at the top.
- Input features: Log mel-filter-bank coefficients with energy, deltas, and delta-deltas provide input features that preserve local spectrogram correlations.
- Convolution: Each convolution computes feature maps from the input using learned filters and bias parameters.The convolution is expressed as H_i = W_i ∗ X + b_i.
- Convolution: Zero padding along the frame axis preserves the input sequence length after convolution.The model also uses stride 1 for all convolution operations.
- Activation functions: ReLU, PReLU, and Maxout are used as element-wise nonlinear activation functions.ReLU outputs zero for negative inputs, PReLU uses a learned-slope linear response there, and Maxout selects among linear feature-map candidates.
- Pooling: Max-pooling is performed only along the frequency axis to reduce spectral variations, with pooling applied once after the first convolutional layer.Pooling in time is avoided, and the resulting feature maps retain the same sequence lengths.
3. Connectionist Temporal Classification
CTC models variable-length sequence mappings by summing probabilities over latent sequences that collapse to the target, enabling efficient end-to-end training and decoding.
- CTC represents each target sequence as the sum of probabilities of latent sequences that map to it.The latent sequences are produced framewise by applying a softmax to network outputs.
- The mapping uses a many-to-one function from longer latent sequences O to shorter output sequences Z.In speech recognition, X is the acoustic signal and Z is a symbol sequence.
- A dynamic programming algorithm similar to the HMM forward algorithm computes the latent-sequence sum efficiently.Its intermediate values also provide efficient gradients for the neural network outputs.
- Best-path decoding emits the most probable label at each time step, then applies the collapse function σ to obtain the prediction.This relies on the framewise conditional independence assumption.
- The most probable latent path need not produce the highest-probability output sequence; finding that sequence generally requires approximate search such as beam search.The latent path is formalized as π* = Argmaxπ Pr(π|X).
4. Experiments
The TIMIT experiments evaluate a deep convolutional CTC model using standardized phoneme-recognition data and show competitive accuracy with substantially faster training than the LSTM baseline.
- Architecture: The model has 10 convolutional layers, 3 fully connected hidden layers, frequency-only pooling after the first convolution, and 3×5 filters throughout.It uses 128 feature maps in the first four convolutional layers, 256 in the remaining six, and 1024 units per fully connected layer.
- Optimization: Training uses Adam followed by SGD fine-tuning, batch size 20, dropout probability 0.3, and best-path CTC decoding at test time.L2 regularization with coefficient 1e-5 is applied during fine-tuning.
- Results: 18.2% phoneme error rate was achieved on the TIMIT core test set, slightly better than the LSTM baseline and a transducer with an explicit RNN language model.The evaluation uses the standard core test set containing 192 sentences.
- Results: 2.5× faster training was obtained than the LSTM model without deliberately optimizing the implementation.The comparison keeps roughly the same number of parameters.
- Ablations: The structural analysis indicates benefits from more layers, reasonably large context windows, and Maxout activations.These choices provide larger receptive fields, capture spatial and temporal relations, and offer more functional freedom than ReLU variants.
5. Discussion
The discussion attributes competitive performance to depth, context, Maxout nonlinearities, and regularization, while highlighting training-speed advantages and open questions about scope.
- Deeper convolutional architectures with wider frequency and temporal receptive fields can achieve results comparable to the state of the art with CTC.The proposed methodology also uses Maxout nonlinearities and careful regularization.
- Table 1 reports phoneme error rate on TIMIT and compares CNN configurations with recurrent baselines.Its caption states that deeper architectures and larger filter sizes lead to better performance, with the best development-selected model reaching 18.2% test PER.
- The model’s regularization is especially important for small datasets such as TIMIT, where over-fitting happens easily.This identifies dataset size as a practical boundary for the reported methodology.
- The authors conjecture that convolutional CTC may learn phoneme-level sequences more easily than character-level sequences.They attribute this possibility to robust local phoneme structure and fewer long-term dependencies, but call for empirical research.
- Convolutional training can be nearly independent of input-sequence length because convolutions are parallelizable and supported by optimized libraries.RNN training increases at least linearly with input length because its computations are sequential.
6. Conclusions
The paper presents a CNN-based end-to-end speech-recognition framework without recurrent networks and reports promising TIMIT results with improved computational efficiency.
- The framework combines CNNs and CTC without recurrent neural networks for end-to-end speech recognition.The authors conclude that it can learn the temporal relations required for integration with CTC.
- The TIMIT experiments show promising results and a gain in computational efficiency.Training on large-vocabulary datasets and integration with a language model remain future work.