Source-linked AI summary
Personalized Automatic Speech Recognition for a Dysarthric and Tracheostomic Speaker using Artificial Conversations
David Nadrchal, Monorama Swain, Florian Schmid, Gerhard Widmer, Paul Primus
TL;DR
For a Czech speaker with severe dysarthria and a permanent tracheal stoma, existing personalized ASR approaches do not fit the single-speaker setting. The paper develops a personalized system and data-collection protocol, reporting that the resulting ASR is perceived as helpful. It also identifies limits to further error reduction using spoken utterances alone.
Problem
Existing personalized ASR approaches focus heavily on English and multispeaker data, while this speaker’s combined dysarthria and tracheostomic phonation has no prior Czech training data.
Source: §I · p. 1Method
The authors build a personalized ASR system using a multi-stage pipeline and collect speaker data in streaming interactions with a real-time ASR prototype.
Source: §I · p. 2, B. Dataset Statistics · p. 2Results
The resulting system improved Whisper Base from completely unusable to being perceived as helpful, as supported by experimental results and user feedback.
Source: §VI · p. 6Takeaways & Limitations
The system performs on par with experienced listeners for isolated audio-only utterances, and the speaker considers it useful.
Source: §I · p. 2Further error reduction requires considering context beyond spoken utterances, such as longer conversation history or visual input.
Source: §I · p. 2
Abstract
from arXiv · showhide
This work presents an automatic speech recognition (ASR) system personalized for a Czech speaker with a permanent tracheal stoma and severe dysarthria rendering their speech unintelligible to untrained listeners. We release a public dataset containing 33 annotated hours of the speaker's speech, collected using a novel "artificial conversation" protocol designed for high engagement and dialogue realism. We propose a multi-stage training pipeline based on Whisper Base: fine-tuning on standard Czech speech, acoustically simulated tracheostomic speech, and the speaker's data. We evaluate the system across three near real-time scenarios: scripted conversations, question answering, and spontaneous dialogue, achieving a 50\% relative reduction in Character Error Rate compared to Whisper Base baseline and surpassing the average recognition accuracy of their assistants in acoustic recognition of isolated utterances. We demonstrate that even for severely impeded speech, a helpful ASR is achievable, as evidenced by the quantitative results and the feedback from the speaker.
I. INTRODUCTION
The speaker’s severe dysarthria and tracheostomic phonation make their speech unintelligible to untrained listeners, while existing approaches do not fit this single-speaker Czech case. The paper proposes a personalized ASR system and reports a public dataset, artificial-conversation collection, and near-real-time evaluation.
I. INTRODUCTION
- The speaker’s severe dysarthria and permanent tracheal stoma render their speech unintelligible to untrained listeners, although familiar caregivers understand them using non-verbal cues and feedback.The speech also has reduced energy, spectral flatness, and frequent interruptions.Source: §I · p. 1
- Existing personalized ASR solutions focus heavily on English and large-scale multispeaker data, while no prior Czech training data exist for this speaker’s combined dysarthria and tracheostomic phonation.Low-tech AAC is too slow for engaging real-time interaction.Source: §I · p. 1
- The paper proposes personalized, non-invasive ASR, releasing a 33-hour annotated dataset and evaluating near-real-time recognition in scripted conversations, question answering, and spontaneous dialogue.The authors report performance on par with experienced listeners for isolated audio-only utterances and say the speaker considers the system useful.Source: §I · p. 2
II. DATASET
Artificial Conversations combine dialogue-like interaction with scripted targets to support engaging, realistic data collection. The protocol has the speaker repeat their scripted lines during pauses in a synthesized dialogue.
A. Artificial Conversations
- Artificial Conversations address the limited engagement of standard reading sessions and the mismatch between read speech and real-world conversation.Scripts are tailored to the speaker’s interests and everyday life.Source: A. Artificial Conversations · p. 2
- A synthesized dialogue supplies Character B’s lines and silence intervals for the speaker to repeat Character A’s scripted lines.The protocol aims to retain scripted targets while simulating natural dialogue dynamics.Source: A. Artificial Conversations · p. 2, Figure 2 · p. 2
- The dialogue scripts are generated around the speaker’s interests and everyday life to make recordings engaging and representative of their vocabulary.The dialogue audio is assembled from text-to-speech lines, with Character B’s voice ring-modulated to distinguish it from Character A.Source: A. Artificial Conversations · p. 2
B. Dataset Statistics
The dataset combines artificial conversations with other recording formats and includes streaming recordings used for real-time interaction. Artificial conversations yielded more words per session than reading sessions, and a subset of the data is publicly released.
B. Dataset Statistics
- Across 137 sessions, the researchers collected 16,619 sentence-level utterances totaling 37:45 hours of transcribed speech in multiple recording formats.The sessions included artificial conversations, reading, spontaneous conversations, question answering, and singing.Source: B. Dataset Statistics · p. 2
- A subset of artificial-conversation, spontaneous-conversation, and question-answering sessions was recorded as streaming data while the speaker interacted with the real-time ASR prototype.Artificial conversations were annotated from generation timestamps; other formats were transcribed manually.Source: B. Dataset Statistics · p. 2
- Artificial conversations produced 420.1 words per session versus 229.9 for reading, making them more data-efficient in this collection context.Reading yielded slightly more utterances per session: 129.5 versus 112.7; both formats were collected at approximately three sessions per week.Source: B. Dataset Statistics · p. 2
- The released subset excludes personally identifiable information and includes session identifiers and dates for longitudinal analysis and leakage prevention.The release was made with consent from the speaker and their legal guardian.Source: B. Dataset Statistics · p. 3
- The evaluation used a session-wise split, with held-out streaming sessions reserved for testing to avoid channel or condition leakage.All test data were recorded in streaming settings.Source: B. Dataset Statistics · p. 3
III. METHOD
The method fine-tunes Whisper Base through standard Czech, simulated tracheostomic, and speaker-specific data stages, then regularizes training with a Czech language model. Acoustic simulation uses low-level transformations intended to resemble the speaker’s speech.
A. Training Pipeline
- Whisper Base is fine-tuned on standard Czech speech, simulated tracheostomic speech, offline speaker recordings, and streaming data for device adaptation.The final stage targets deployment-device noise suppression and microphone characteristics.Source: A. Training Pipeline · p. 3
B. Knowledge Distillation from a Language Model
- A custom 60M-parameter bidirectional LSTM language model supplies masked-token probability distributions that are distilled into Whisper using KL divergence.The regularization is intended to counteract Whisper’s poor Czech prior and prevent linguistic degradation during fine-tuning.Source: B. Knowledge Distillation from a… · p. 3, Figure 3 · p. 3
C. Acoustic Simulation
- Because neural voice-conversion methods did not model the extreme phonetic degradation adequately, the authors use deterministic transformations: spectral flattening, fourfold time stretching, and random 250 ms silences.Embedding-cluster entropy increased from 0.5 to 0.7–0.8, but the authors describe this as a coarse proxy and report that simulated and real speech remained notably different.Source: C. Acoustic Simulation · pp. 3–4
- A simpler simulation using only time stretching was also tested against the full acoustic-simulation pipeline.The passage does not report the comparison’s outcome.Source: C. Acoustic Simulation · p. 4
D. Streaming Inference
The system streams audio through a smartphone-accessible web interface, with latency the speaker considered acceptable despite exceeding the commercial sub-second standard. Personalized voice activity detection largely closed the validation-set CER gap between automatic and manual segmentation.
D. Streaming Inference
- Audio streams from a smartphone-accessible web interface to an inference server in 62.5 ms chunks for continuous processing.The chunks contain 1000 samples at 16 kHz.Source: D. Streaming Inference · p. 4
- ≈2.5 s latency exceeds the sub-second commercial standard, but the speaker reported it was acceptable and faster than human listeners typically take to decode their speech.The reported latency is for the deployed system.Source: D. Streaming Inference · p. 4
1) Personalized Voice Activity Detection:
- A personalized logistic-regression VAD uses spectral flatness, zero-crossing rate, spectral centroid, and 13-bin MFCCs, with inference thresholds experimentally selected by validation-set CER minimization.It was trained on 1,057 manually annotated one-second chunks: 533 silence and 524 speech.Source: 1) Personalized Voice Activity Detection · p. 4, D. Streaming Inference · p. 4
- On validation, the VAD measures largely closed the CER gap between automatic and manual segmentation; test-set evaluation and comparison with single-session fine-tuned Pyannote VAD are reported in Section V.Source: 1) Personalized Voice Activity Detection · p. 4
2) Decoding and Constraints:
The evaluation covers three scenarios from scripted to spontaneous speech, each compared with an off-the-shelf Whisper Base baseline to separate impairment-related errors from task difficulty. Decoding uses beam search and Czech vocabulary constraints, with low-confidence transcriptions filtered out.
2) Decoding and Constraints:
- Beam search uses width 25 to maximize transcription accuracy despite acoustic ambiguity.Source: 2) Decoding and Constraints · p. 4
- Decoding is constrained to a standard Czech lexicon augmented with training-session vocabulary, while out-of-vocabulary token probabilities are penalized by 10^-4.The method uses Triebased logit biasing.Source: 2) Decoding and Constraints · p. 4
- Transcriptions with average token probability below the experimentally determined threshold of 0.712 are discarded to filter residual non-speech noise.Source: 2) Decoding and Constraints · p. 4
IV. EXPERIMENTAL SETUP
- Three test scenarios range from scripted to spontaneous speech, each paired with an off-the-shelf Whisper Base baseline to disentangle speech-impairment errors from task difficulty.Source: §IV · p. 4
A. Evaluation Protocol
The protocol compares continuous-streaming pathological speech with manually segmented non-impaired baselines and reports Character Error Rate, chosen for its granularity in Czech and noisy pathological speech. Scripted conversations and question answering use parallel recordings with matching linguistic content.
A. Evaluation Protocol
- The pathological setup uses automatic VAD and segmentation on continuous streaming audio, while the non-impaired baseline uses manual utterance segmentation to avoid oracle-segmentation advantage for the target speaker.Source: A. Evaluation Protocol · p. 4
- CER counts character substitutions, deletions, and insertions relative to reference characters; it can exceed 1.0 when hallucinated insertions exceed reference length.CER is preferred because Czech inflection makes minor suffix errors costly under WER, while character-level scoring gives more granular insight into phonetic preservation in noisy pathological speech.Source: A. Evaluation Protocol · p. 4
1) Scripted Conversations:
- The scripted-conversation test uses 329 utterances from three streaming artificial conversations and matching non-impaired recordings of the exact scripts under similar acoustic conditions.Source: 1) Scripted Conversations · p. 4
2) Question Answering (QA):
- The question-answering test uses 353 short utterances from three sessions, with expert transcriptions and a non-impaired baseline re-recorded to match content.Answers are often single words.Source: 2) Question Answering (QA) · p. 4
3) Spontaneous Conversations:
The spontaneous-speech benchmark uses one streaming conversation and a proxy non-impaired baseline because spontaneity cannot be replicated by re-recording. Human comparisons show the model exceeds the average assistant result, while the strongest listener remains ahead.
3) Spontaneous Conversations:
- The pathological spontaneous test contains one fully transcribed streaming conversation with 188 utterances, isolated from the interlocutor using PyAnnote diarization before VAD.Source: 3) Spontaneous Conversations · p. 5
- Because spontaneity cannot be replicated while preserved, the proxy baseline uses 5,136 utterances from 16 Ortofon-v3 natural informal Czech recordings withheld from training stage one.The spontaneous setting is described as the most challenging in-the-wild benchmark.Source: 3) Spontaneous Conversations · p. 5
A. Human Intelligibility Assessment
- Eight Czech listeners unfamiliar with the speaker averaged CER 1.182 on 25 isolated utterances, indicating effective unintelligibility to naive listeners in audio-only assessment.The CER exceeded 1.0 due to insertion errors.Source: A. Human Intelligibility Assessment · p. 5
- CER 0.565 significantly outperformed assistants’ average CER 0.708 (p = 0.046) and three assistants (p < 0.01), but not the top listener (CER 0.423; p = 0.008).The four assistants’ CER range was 0.42–0.90.Source: A. Human Intelligibility Assessment · p. 5
- The top-performing listener was a dataset-curating author with strong prior knowledge of expected content and experience decoding the speaker’s audio-only speech without visual cues or feedback.Source: A. Human Intelligibility Assessment · p. 5
B. Comparison to Baseline
The personalized system substantially reduces CER versus the zero-shot baseline, but remains behind non-impaired speech recognition. Component tests identify streaming adaptations as critical, while pretraining and acoustic simulation contribute more modestly and their effects vary by scenario.
B. Comparison to Baseline
- The system reduces CER by 40 to 90 absolute percentage points versus the zero-shot baseline, but a gap remains against non-impaired recognition.Omitting pretraining has little effect on scripted conversations but severely degrades less constrained modalities.Source: B. Comparison to Baseline · p. 5
C. Component-wise Evaluation
- Removing Personalized VAD or Vocab Guidance degrades performance by > 27% relative, with Device Adaptation's importance attributed to iPhone noise suppression.The authors attribute Device Adaptation's significance to aggressive noise suppression filters in iPhone streaming settings.Source: C. Component-wise Evaluation · p. 5, Table IV · p. 6
- The lightweight VAD outperforms fine-tuned Pyannote VAD across all three modalities, while the Manual VAD Oracle lowers spontaneous CER from 0.637 to 0.506.The authors attribute much of the remaining spontaneous-speech error to false triggers on the interlocutor; they say these do not severely disrupt conversation flow.Source: C. Component-wise Evaluation · pp. 5–6
- Acoustic simulation and pretraining provide moderate gains of 3–6%, while knowledge distillation has mixed results and full simulation benefits spontaneous speech over stretch-only simulation.The authors hypothesize that spontaneous speech more often transitions to whispered, aperiodic speech, which benefits from removing the periodic source during full simulation.Source: C. Component-wise Evaluation · p. 6
- Scripted-speech CER decreases steadily across pipeline stages, while QA and spontaneous speech achieve reliable accuracy only after training on the target speaker's offline data.Table V reports CER across the multi-stage pipeline.Source: C. Component-wise Evaluation · p. 6, Table V · p. 6
D. Qualitative Error Analysis
The analysis finds that many errors preserve semantic intent despite diacritics and homophone substitutions. Text normalization improves scripted-speech CER, while spontaneous speech remains the most challenging domain.
D. Qualitative Error Analysis
- Aggressive normalization of acoustically similar Czech characters reduces scripted-speech CER by 13.4% relatively, to 0.375.The examples include i/y and d/t; the sample transcript is presented in Table VI.Source: D. Qualitative Error Analysis · p. 6, Table VI · p. 6
- Spontaneous speech remains the most challenging domain, even with text normalization.The paper notes that many errors stem from missing diacritics or homophone substitutions, while semantic intent is often preserved.Source: D. Qualitative Error Analysis · p. 6
- The speaker's feedback describes the system as better than before and as understanding them.The quoted feedback excerpt is partial.Source: D. Qualitative Error Analysis · p. 6
VI. CONCLUSION
The paper presents personalized ASR for one Czech speaker with severe dysarthria and a permanent tracheal stoma, using a 33-hour corpus and a four-stage pipeline. The system improves Whisper Base from unusable to helpful, while high CER and the speaker-specific setting limit the scope of the result.
VI. CONCLUSION
- The paper releases 33 hours of impeded speech collected through Artificial Conversations and presents a four-stage pipeline with acoustic simulation and streaming adaptations.The protocol combines dialogue engagement with scripted tractability.Source: §VI · p. 6
- The system's CER remains approximately 0.50, and its reported advantage over experienced human listeners is limited to low-context scenarios.The authors distinguish this case by Czech data scarcity, the severity and rarity of the impairment, and low-latency constraints.Source: §VI · p. 6
- The authors identify multi-speaker generalization and reducing reliance on annotated data as future-work priorities.They also propose multimodal context, conversational memory, and broader evaluation as directions for addressing remaining gaps.Source: §VI · p. 6