Source-linked AI summary
Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks
Dimitri Palaz, Ronan Collobert, Mathew Magimai. -Doss
TL;DR
Conventional HMM/ANN ASR separates acoustic feature extraction from modeling, while this paper tests CNNs that learn phoneme-relevant features directly from raw speech. The proposed raw-speech CNN achieves comparable or better recognition than the conventional approach, although MFCC input performs slightly better in the reported CNN comparison.
Problem
The paper addresses whether phoneme probabilities can be estimated without separately extracting hand-crafted acoustic features before ANN modeling.
Method
A CNN takes raw speech windows as input to estimate phoneme class conditional probabilities, with a CRF used for sequence decoding.
Results
The raw-speech CNN slightly outperforms the baseline, while MFCC input gives comparable performance and CRF decoding improves accuracy over HMM decoding.
Takeaways & Limitations
CNNs can learn features relevant to phoneme classification automatically from raw speech and can approach the performance of complex hand-crafted features.
Takeaways & Limitations
MFCC input performs slightly better than raw signal for the CNN, and this difference requires further investigation on larger databases and deeper architectures.
Abstract
from arXiv · showhide
In hybrid hidden Markov model/artificial neural networks (HMM/ANN) automatic speech recognition (ASR) system, the phoneme class conditional probabilities are estimated by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or/and speech production knowledge, and, then modeling the acoustic features with an ANN. Recent advances in machine learning techniques, more specifically in the field of image processing and text processing, have shown that such divide and conquer strategy (i.e., separating feature extraction and modeling steps) may not be necessary. Motivated from these studies, in the framework of convolutional neural networks (CNNs), this paper investigates a novel approach, where the input to the ANN is raw speech signal and the output is phoneme class conditional probability estimates. On TIMIT phoneme recognition task, we study different ANN architectures to show the benefit of CNNs and compare the proposed approach against conventional approach where, spectral-based feature MFCC is extracted and modeled by a multilayer perceptron. Our studies show that the proposed approach can yield comparable or better phoneme recognition performance when compared to the conventional approach. It indicates that CNNs can learn features relevant for phoneme classification automatically from the raw speech signal.
1. Introduction
The paper questions the conventional separation of acoustic feature extraction and modeling in HMM/ANN speech recognition. It investigates CNNs that learn from raw speech and reports comparable or better phoneme recognition performance than the conventional approach.
- Motivation: Conventional HMM-based ASR independently separates feature extraction, acoustic modeling, and decision making.Acoustic features such as MFCC, PLP, and linear prediction cepstral coefficients are extracted using speech perception or production knowledge before modeling.
- Motivation: End-to-end systems can learn task-relevant abstract representations directly from raw input through multilayer architectures.This possibility was demonstrated in related sequence-recognition studies in computer vision and text processing.
- Proposed direction: The paper investigates CNN-based estimation of phoneme class conditional probabilities from raw speech for phoneme sequence recognition.The proposed approach is compared with spectral-feature extraction followed by ANN modeling in a hybrid HMM/ANN system.
- Study and findings: Experiments on TIMIT show that the raw-speech approach can produce phoneme recognition performance similar to or better than the conventional approach.The study also evaluates a CRF-based decoding algorithm alongside the proposed CNN acoustic model.
2. Related Work
Prior work explored raw-speech modeling with statistical sequence models and discriminative classifiers, while short-term spectral representations became common intermediate features for neural speech recognition.
- Raw-speech modeling: Early raw-speech approaches modeled signals with linear prediction HMMs, switching autoregressive HMMs, or switching linear dynamical systems.Experiments reported comparable or better performance than cepstral-based HMM systems in clean isolated-word or digit-recognition settings.
- Neural architectures: CNNs were introduced in speech literature as time-delay neural networks.The terminology connects the paper's CNN framing with earlier neural architectures for temporal speech processing.
- Raw-speech modeling: Raw speech and cepstral features were also combined in support vector machines for noisy phoneme classification.This extended raw-signal modeling beyond the earlier clean-condition sequence-modeling studies.
- Intermediate representations: Short-term spectrum features became widely used as intermediate representations between raw signal and classical cepstral features.These representations were successfully applied to speech recognition with artificial neural networks.
3. Proposed system
The proposed system uses a CNN to transform raw speech windows into phoneme probabilities and a CRF to decode phoneme sequences. Its architecture stacks convolution, temporal pooling, and nonlinear stages before classification, with parameters learned by likelihood optimization.
- System overview: The system combines CNN-based phoneme probability estimation with CRF-based sequence decoding.The CNN processes raw speech, while the CRF assigns transition scores between phonemes and uses network scores for decoding.
- Convolutional Neural Network: A raw-speech window is processed through convolution, temporal max-pooling, and tanh stages before a one-hidden-layer MLP and softmax classifier.The optimal architecture used three filter-extraction stages, and the softmax layer computes conditional probabilities for phoneme classes.
- Convolutional layer: A convolutional layer applies the same linear transformation across successive or shifted windows of kW frames in an input sequence.The transformation uses dout filters represented by rows of a dout × din parameter matrix.
- Pooling: Max-pooling performs local temporal maximum operations while preserving the input/output frame dimension.These layers increase robustness to slight temporal distortions in the input.
- Network training: The CNN parameters are learned by maximizing training-set log-likelihood with stochastic gradient ascent.The softmax converts network scores into conditional probabilities before likelihood optimization over layer parameters.
- Decoder: The CRF assigns node scores from network outputs and transition scores between labels, then trains transitions by maximizing conditional likelihood.Path scores are converted into conditional probabilities using a softmax over possible paths.
4. Experimental Setup
The experiments used TIMIT to compare raw-speech and MFCC inputs, CNN and MLP architectures, and CRF and HMM decoding schemes. Network hyperparameters were selected by early stopping on cross-validation data.
- Dataset: The TIMIT setup included 3,696 training utterances, 400 cross-validation utterances, and 192 core-test utterances from separate speakers.The 61 phonetic symbols were mapped to 39 phonemes plus a garbage class.
- Features: Raw inputs were normalized speech-signal windows, whereas MFCC inputs used 13th-order coefficients with first and second derivatives over a 9-frame context.MFCCs were computed with a 25 ms Hamming window and 10 ms shift.
- Network configuration: The CNN hyperparameters included input context, convolution widths and shifts, filter count, hidden-layer width, and pooling width.These settings were tuned by early stopping on the cross-validation set.
- Network configuration: The selected raw-speech CNN used 270 ms context, 90 filters, 500 hidden units, and three pooling width, while the MFCC CNN used 290 ms context and no pooling layer.The MFCC network used 39 input dimensions and the raw network used 1.
- Baselines and decoding: A single-hidden-layer MLP with 500 nodes provided a traditional comparison, while decoding used either a simple CRF or a constrained-duration HMM.The HMM used three-state phoneme durations and equal phoneme priors.
5. Results
The results evaluated phoneme recognition accuracy on the TIMIT core test set across feature types, architectures, and decoding schemes. Raw-speech CNNs slightly outperformed the baseline, and CRF decoding improved accuracy over HMM decoding.
- Evaluation: Phoneme accuracy on the TIMIT core test set was reported for different input features, network architectures, and decoding schemes.The comparison also included the number of model parameters.
- Architecture and features: Using raw speech, the CNN architecture slightly outperformed the baseline.The baseline was described as consistent with other work, though somewhat lower without supplementary processing.
- Decoding: For raw speech, CRF decoding increased phoneme accuracy compared with HMM decoding.
- Architecture and features: CNNs using MFCC features achieved similar performance to CNNs using raw features.
6. Analysis
The analysis examined max-pooling and learned first-layer filters. Pooling improved performance while reducing parameters, and learned filters responded to different frequency bands.
- Advantage of max-pooling layers: Adding max-pooling layers improved phoneme accuracy while providing an easy way to reduce the number of network parameters.The comparison used raw features with HMM decoding and held the input window size constant across architectures.
- Advantage of max-pooling layers: The pooling-layer study varied the number of max-pooling layers and tuned convolution output dimensions to reduce overfitting from excessive parameters.
- Filters trained in the first layer: Five randomly chosen first-layer filters showed responses to different frequency bands of the raw input signal.The filters were interpreted as possible matching filters.
7. Discussion
The discussion finds that CNNs are better suited than single-layer MLPs for raw temporal speech signals and can approach complex hand-crafted features. MFCCs retained a slight advantage, while CRF decoding performed better than HMM decoding.
- Discussion: Over raw speech, CNNs greatly improved on single-layer MLPs and slightly outperformed the baseline with almost no preprocessing.The results support convolution-based architectures for temporal signals.
- Discussion: The results suggest that deep architectures can learn efficient features and achieve performance similar to complex hand-crafted features.
- Discussion: MFCC input seemed to work slightly better than raw-signal input for CNNs.The paper states that this comparison requires further investigation on larger databases and with deeper architectures.
- Discussion: CRF decoding seemed to work better than generative HMM decoding, plausibly because CRF transitions learn a phoneme bigram language model.The CRF was optimized independently from the CNN, although joint training was identified as possible future work.
8. Conclusions
The proposed CNN estimates phoneme class probabilities directly from raw speech and learns features automatically. It outperforms baseline systems, while MFCC input achieves comparable performance.
- CNNs estimate phoneme class probabilities directly from raw speech data.
- The system learns features automatically from raw speech and outperforms baseline systems.
- Using MFCC features as input yields comparable performance.