Articulatory Features for ASR of Pathological Speech

1. Introduction

• Most computer-assisted speech-training systems cannot automatically detect individual speech-sound problems that affect intelligibility.The paper aims to develop robust pathological-speech ASR and incorporate it into systems for detecting these problems.Source: §1 · p. 1
• Limited database size and structure make it generally infeasible to train robust acoustic models that capture dysarthric speech variation.The supplied passage contrasts dysarthric and normal speech database sizes.Source: §1 · p. 1
• State-of-the-art ASR still performs considerably worse on dysarthric speech than on normal speech despite long-running efforts to build speaker- and text-independent systems.Speech disorders involving articulation can reduce intelligibility and impair communication.Source: §1 · p. 1
• Earlier work explored Dutch DNN-HMM training with multiple Dutch varieties and multi-stage training for pathological speech, while prior studies investigated articulatory features for ASR.The previous Dutch work also created a six-hour dysarthric-speech database.Source: §1 · p. 1
• The study combines vocal-tract constriction variables with filterbank features in fCNN models and evaluates them on two pathological-speech corpora against DNN, CNN, and TFCNN systems.The authors expect articulatory information to increase robustness to spectrotemporal deviations associated with reduced articulatory capabilities.Source: §1 · pp. 1–2

2. Speech corpora selection

• The CGN supplies 441.5 hours of normal speech for training: 255 hours of Dutch and 186.5 hours of Flemish.Training components include read speech, conversations, interviews, and discussions.Source: §2 · p. 2
• EST provides Dutch dysarthric training speech from 16 patients with etiologies including Parkinson’s disease, cerebrovascular accident, traumatic brain injury, and birth defect.Speakers range from mild to moderate dysarthria, and the database includes word, sentence, vowel, and text tasks.Source: §2 · p. 2
• The CHASING01 test set uses selected sentence-reading tasks, with pronunciation-error utterances removed and recordings retained from three male patients.The source database includes five patients tested at six treatment time points; the retained speakers were aged 59, 62, and 67.Source: §2 · p. 2
• COPAS supplies a second test set of annotated sentence-reading tasks and text passages, including dysarthric speech from 75 Flemish patients with diverse etiologies and severity levels.The corpus also contains recordings from Flemish normal speakers and people with other speech disorders; the dysarthric test is described as more challenging than CHASING01.Source: §2 · p. 2

3. Extracting Articulatory Features

• Speech inversion estimates vocal-tract constriction-variable trajectories from speech by learning an acoustic-to-articulatory mapping from paired acoustic and articulatory data.The inverse transform is described as ill-posed because it is nonlinear and non-unique.Source: §3 · p. 2
• The inversion model is trained on synthetic English speech generated with TADA and HLsyn, paired with trajectories and augmented with 14 noise types at 10–80 dB SNR.The generated corpus contains 534,322 audio samples, approximately 450 hours, with 88% for training, 2% for cross-validation, and 10% for testing.Source: §3 · p. 2
• The system uses Z-normalized NMC acoustic features and assumes an English-trained inversion model maps reasonably accurately to Dutch because of linguistic similarity.NMCs track amplitude modulations of filtered subband speech signals; features are contextualized by splicing multiple frames.Source: §3 · p. 2

4. Acoustic Models

• The fCNN convolves filterbank acoustic features across frequency and articulatory TV trajectories across time, then fuses the pooled feature maps before a single neural network.The architecture is illustrated in Figure 1.Source: §4 · pp. 2–3, Figure 1 · p. 3
• TFCNNs convolve across both time and frequency and are included as an alternative model, while their comparative advantages are left for future work.The paper motivates TFCNNs as potentially robust to spectrotemporal deviations due to background noise.Source: §4 · p. 3
• Baseline DNN and CNN models use filterbank features because those architectures were found to perform worse with concatenated features in prior work.The prior comparison is identified as.Source: §4 · p. 3

5.1. Database details

• Normal CGN training data comprised 186.5 hours of Flemish and 255 hours of northern Dutch speech; the selected dysarthric Dutch training set contained 4 hours 47 minutes from a 6-hour-16-minute, 16-speaker database.Segments with pronunciation errors, single words, and pseudowords were excluded from the dysarthric training set.Source: §5.1 · p. 3
• Testing used CHASING01 Dutch dysarthric speech from 3 speakers and Flemish COPAS data with 103 dysarthric and 82 normal speakers.CHASING01 contains 721 utterances; COPAS includes normal and disordered sentence tasks.Source: §5.1 · p. 3

5.2. Implementation Details

• A CNN speech-inversion model mapped contextualized acoustic features to vocal-tract-constriction trajectories, while ASR models used GMM-HMM alignments and acoustic features with a 17-frame context window.The acoustic-model inputs included 40-dimensional log-mel filterbanks with deltas and delta-deltas; fCNN also used a time-convolution layer for articulatory features.Source: §5.2 · pp. 3–4

5.3. ASR experiments

• Dutch tests used either normal or dysarthric Dutch training; Flemish tests used normal Flemish alone or combined with normal Dutch, and performance was measured by WER.Combined normal-and-dysarthric Dutch training was omitted after pilot results were similar to normal-Dutch-only training.Source: §5.3 · p. 4

Dutch test results

• On the Dutch test set, using only 6 hours of in-domain dysarthric Dutch speech, the system with both acoustic and articulatory features achieved the best ASR WER: 19.1%, versus 21.1% for CNN and 20.3% for TFCNN.Source: §6 · p. 4
• 14.1% WER was the best result with normal Dutch training, achieved by TFCNN; articulatory information brought no further improvement over ordinary CNN in this setting.This scenario used a large amount of mismatched data for mild-to-moderate pathological speech.Source: §6 · p. 4

Flemish test results

• 32.2% WER with Flemish-only training and 29.0% WER after adding normal Dutch speech made fCNN the best model in both Flemish scenarios.With Flemish-only training, TFCNN had 33.8% WER and CNN had 33.5%; the Flemish test corpus included 103 dysarthric speakers, and a gap from control-data performance remained.Source: Table 2 · p. 4, §6 · p. 4

• Joint articulatory and acoustic features improved ASR when only a limited amount of in-domain training data was available.Source: §7 · p. 4
• Improvements were consistent in more challenging testing conditions with considerably more speakers and more diverse etiologies.Source: §7 · p. 4
• The study trained various acoustic models and tested them on Dutch and Flemish pathological-speech corpora.Source: §7 · p. 4