Source-linked AI summary
Personalizing ASR for Dysarthric and Accented Speech with Limited Data
Joel Shor, Dotan Emanuel, Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, Avinatan Hassidim, Yossi Matias
TL;DR
ASR trained mostly on typical speech often fails on accented voices and voices affected by medical conditions, while non-standard speech is difficult to cover with enough training speakers and varies between individuals. The paper finetunes base ASR models into personalized systems for ALS and accented speech, reporting relative WER improvements for both groups.
Problem
Typical-speech ASR often fails on accented and medically affected voices when those groups are poorly represented in training data, and non-standard speech is difficult to cover with enough speakers and varies across individuals.
Source: §1 · p. 1Method
The paper starts from a base ASR model trained on standard speech and finetunes personalized models for individual speakers.
Source: §1 · p. 1Results
70% relative WER improvement over the base model for dysarthric speech (for both groups) and 35.1% for accented speech were reported.
Source: §5 · p. 4Takeaways & Limitations
With on the order of 1 hour of data, personalized models were significantly better than cloud-based services, and much of the ALS improvement came from the first 5-10 minutes of training data.
Source: §5 · p. 4The paper does not report the proposed comparison between a single model trained on an entire ALS corpus and a single-speaker model, leaving adaptation to general non-standard speech versus the individual unresolved.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Automatic speech recognition (ASR) systems have dramatically improved over the last few years. ASR systems are most often trained from 'typical' speech, which means that underrepresented groups don't experience the same level of improvement. In this paper, we present and evaluate finetuning techniques to improve ASR for users with non-standard speech. We focus on two types of non-standard speech: speech from people with amyotrophic lateral sclerosis (ALS) and accented speech. We train personalized models that achieve 62% and 35% relative WER improvement on these two groups, bringing the absolute WER for ALS speakers, on a test set of message bank phrases, down to 10% for mild dysarthria and 20% for more serious dysarthria. We show that 71% of the improvement comes from only 5 minutes of training data. Finetuning a particular subset of layers (with many fewer parameters) often gives better results than finetuning the entire model. This is the first step towards building state of the art ASR models for dysarthric speech.
1. Introduction
The paper addresses ASR errors for accented and medically affected speech that is underrepresented in typical training data, using personalized finetuning to adapt existing models. It reports improvements across speech types and architectures, with substantial gains possible from little data and limited layer finetuning.
1. Introduction
- Typical-speech ASR often fails to recognize accented voices and voices affected by medical conditions when those groups are poorly represented in training data.The paper notes that large datasets and data-driven algorithms work well when they include substantial data from the voices the system must recognize.Source: §1 · p. 1
- Personalized models address variation within groups such as ALS speakers and people with a particular accent, while a base model trained on thousands of hours of standard speech addresses data scarcity.The paper focuses on ALS speech and non-native English accents as part of Project Euphonia.Source: §1 · p. 1
- The technique yields word error rates of 10.8% for mild dysarthria (FRS of 3), 20.9% for severe dysarthria (FRS of 1-2), and 8.5% on heavily accented data.The finetuned models make recognition mistakes distributionally more similar to standard ASR mistakes on standard speech.Source: §1 · p. 1
- The paper reports significant finetuning improvements across dysarthric and accented speech and across different architectures.These comparisons are presented as a contribution of the work.Source: §1 · p. 1
- 71% of relative WER improvement on ALS speech is achieved with 5 minutes of data, and 80% with 10 minutes.For some models, training only the layers closest to the input yields 90% of total relative improvement.Source: §1 · p. 1
2. Related Work
The related work situates the study among neural ASR architectures, small-data adaptation, and research on pathological speech. It adds an analysis of ASR phonetic errors on ALS audio and which phonemes improved models recognize better.
2. Related Work
- RNN-T uses encoder and decoder networks and can support streaming ASR with unidirectional models; this paper uses a bidirectional encoder without attention.The cited bidirectional version was shown to achieve comparable results.Source: §2 · p. 1
- LAS is an attention-based sequence-to-sequence model that maps acoustic properties to language, and a cited voice-search system achieved 4.1% WER on a 12,500-hour task.Its encoder represents acoustic frames and its attention decoder produces linguistic output.Source: §2 · p. 1
- Figure 1 presents schematic diagrams of RNN-T on the left and LAS on the right, reproduced from [8].Source: Figure 1 · p. 2
- Prior dysarthric-speech methods include adding articulatory features for 4-8% relative WER improvement [16] and adapting models trained on open-source data.The cited approaches are limited by base-model quality or the amount of finetuning data available.Source: §2 · p. 2
- The study adds an analysis of phonetic mistakes made by a production ASR system on ALS audio and of phonemes better recognized by improved models.Prior studies examined acoustic, articulatory, and phonetic differences in conditions including Parkinsons, age, dyslexia, and ALS.Source: §2 · p. 2
3. Experiments
The experiments create per-speaker personalized models by finetuning pretrained ASR systems rather than retraining each model from scratch. They vary the finetuned layers and data amounts across ALS and accented-speech datasets.
3. Experiments
- Personalized ASR models start from a base model trained on standard, unaccented speech, a more resource-efficient alternative to retraining from scratch for each speaker.Most experiments use both RNN-T and LAS models to check that findings generalize beyond one architecture.Source: §3 · p. 2
- The experiments finetune different layer combinations on different amounts of ALS and accented-speech data, with models finetuned per speaker.Source: §3 · p. 2
3.1. Data
The study uses recorded ALS speech and non-native English speech to evaluate its personalization approach. The datasets differ in size, source, and train-test setup.
3.1.1. ALS
- 36.7 hours of audio were collected from 67 people with ALS, who read assigned sentences and recorded at home using custom software.The sentences came from the Cornell Movie-Dialogs Corpus, text used by text-to-speech voice actors, and modified Boston Children’s Hospital selections.Source: §3.1.1 · p. 2
- ALS evaluation uses 17 speakers with speech FRS scores of 3 or below, totaling 22.1 hours; the corpus has a restricted language domain but is phonetically similar to other corpora such as Librispeech.ALS-TDI measured participant FRS scores.Source: §3.1.1 · p. 2
3.1.2. Accented Speech
- The L2 Arctic dataset contains 20 non-native speakers with approximately 1 hour each, recording 1,150 phonetically balanced utterances per speaker.Each speaker’s data was split 90/10 for training and testing; proper-noun sentences were placed in training to prevent test-set gains from memorization.Source: §3.1.2 · p. 2
3.2. Base Models
The paper uses RNN-T and LAS base models trained with noisy, multicondition speech, then personalizes them through finetuning. The two architectures map acoustic input to linguistic output using different model designs.
3.2. Base Models
- Base networks use 80-bin logmel features from 25 ms windows with 10 ms hops, stacked in groups of three and trained with room-simulated noise averaging 12 dB SNR.Training uses multicondition techniques.Source: §3.2 · p. 2
3.2.1. Bidirectional RNN-Transducer
- The 49.6M-parameter bidirectional RNN-T maps acoustic frames to word pieces using a five-layer encoder, two-layer decoder, and joint layer.Its encoder uses bidirectional convolutional LSTMs; the decoder uses LSTMs.Source: §3.2.1 · p. 2
3.2.2. Listen, Attend, and Spell model
- The 132M-parameter LAS maps acoustic frames to graphemes with an attention-based encoder-decoder and was trained on 960 hours of Librispeech.Its base model achieved 5.5% WER on the clean test split and 15.5% on the non-clean test split.Source: §3.2.2 · p. 2
3.3. Finetuning
Finetuning searches compare layer combinations and data amounts for each speaker. RNN-T performed best when training began with input-proximal encoder layers, while LAS performed best when finetuned throughout.
3.3. Finetuning
- Finetuning used four Tesla V100 GPUs for no more than four hours.Source: §3.3 · p. 3
3.3.1. RNN-T
- RNN-T searches found that training from encoder layer E0 upward, with or without the joint layer, consistently outperformed other fixed layer combinations.The search included progressively larger encoder spans, up to the full encoder, for ALS and accented speakers.Source: §3.3.1 · p. 3
3.3.2. LAS
- LAS results consistently came from finetuning all layers; all reported LAS results use the entire network unless specified otherwise.Source: §3.3.2 · p. 3
4. Results
Finetuning substantially improves recognition in very non-standard speech, with smaller gains for ALS speech closer to healthy speech. The paper also reports useful gains from limited data and selected layers, alongside changes in phoneme-error patterns.
4.1. Performance
- Finetuning improved on base models in most settings, with dramatic gains over Google cloud ASR for heavy accents and very dysarthric ALS, but moderate gains for ALS resembling healthy speech.The comparison is imperfect: LAS and RNN-T use the entire audio, while Google cloud ASR only looks backward in time.Source: §4.1 · p. 3
4.4. Phoneme mistakes
- KL distance between standard and ALS mistake distributions fell from 0.26 to 0.10 after finetuning, as unrecognized-phoneme patterns became more similar to standard speech.The largest deletion differences involved p, U, f, k, and Z, accounting for 20% of deletion likelihood; n and m together accounted for 17% of insertion/substitution mistakes.Source: Figure 3 · p. 3, §4.4 · p. 4
4.2. Limited Data
- 75% of the ALS WER improvement came from five minutes of data and 85% from ten minutes, compared with 40 minutes of training data.On Arctic, 14 and 20 minutes yielded 70% and 81% of the WER improvement, respectively.Source: §4.2 · p. 3
4.3. Layers
- RNN-T finetuning of the joint layer and first encoder layer achieved 91% of relative WER improvement on ALS and 86% on Arctic, compared with finetuning the full encoder or network.The ALS average WERs were 18.1% versus 15.1%; Arctic WERs were 11.0% versus 10.5%. LAS performed best when finetuned throughout.Source: §4.3 · p. 3
5. Discussion
The discussion reports substantial relative WER improvements from finetuning standard-speech models for dysarthric and accented speech. It also identifies efficient adaptation patterns, possible explanations, and unanswered questions about what the models learn.
5. Discussion
- Models showed good average absolute performance for dysarthric ALS speakers, with large improvements among very dysarthric speakers.Source: §5 · p. 4
- Much of the improvement came from the first 5–10 minutes of audio, and the joint and first encoder layers captured much of the relative gain.Source: §5 · p. 4
- 70% relative WER improvement for both dysarthric groups and 35.1% for accented speech came mainly from finetuning; about an hour produced models significantly better than Cloud-based services.The finetuning process used four GPUs for fewer than four hours.Source: §5 · p. 4
- RNN-T models performed better with encoder-only training, unlike LAS; finetuning the joint and first encoder layers also achieved 90% of the full-encoder relative improvement.Source: §5 · p. 4
- The authors suggest early gains may reflect linguistic similarity between training and test sentences, or partial finetuning retaining general acoustic and linguistic information; these explanations remain for future testing.Source: §5 · p. 4
- The experiments do not determine whether models adapt to general non-standard speech or to individuals, because the proposed comparison was not included.Source: §5 · p. 4
- The five most deleted or substituted phonemes accounted for roughly 20% of those errors, while the two most incorrectly inserted phonemes accounted for just over 20%.Source: §5 · p. 4
6. Future Work
Future work identifies strongly dysarthric speech as a major challenge and proposes additional low-data methods and phoneme-informed training. It also leaves pooling data from speakers with similar conditions for later study.
6. Future Work
- Building state-of-the-art recognition for strongly dysarthric speech remains a major challenge, and whether low-data techniques such as Virtual Adversarial Training or data augmentation help is open.Source: §6 · p. 4
- Phoneme mistakes could guide training by weighting examples or selecting recording sentences that contain the most problematic phonemes.Source: §6 · p. 4
- Pooling data from multiple speakers with similar conditions was not studied; the authors place speaker-independent non-standard ALS models outside this work's scope.Source: §6 · p. 4