Source-linked AI summary
Differentiable and Severity-invariant Discrete Tokens for Dysarthric Speech Recognition
Huimeng Wang, Xurong Xie, Mengzhe Geng, Haoning Xu, Jiajun Deng, Youjun Chen, Chengxi Deng, Xunying Liu
TL;DR
Dysarthric ASR is challenged by scarce data, mismatch with normal speech, speaker heterogeneity, and severity-dependent variation in discrete representations. The paper proposes DSI tokens that combine iterative refinement, end-to-end ASR optimization, and severity-invariant regularization. On UASpeech and TORGO, DSI-based Conformers outperform comparable HuBERT discrete and continuous features, with system combination reaching WERs of 18.90% and 6.38%.
Problem
Dysarthric ASR remains difficult because data are scarce, speech mismatches normal speech, speakers vary widely, and pathology severity creates additional heterogeneity.
Method
DSI tokens combine iterative pseudo-label refinement, end-to-end optimization with a Conformer ASR backend, and severity-invariant regularization across severity groups.
Results
DSI-based Conformers outperform comparable HuBERT discrete and continuous features on UASpeech and TORGO, while system combination reaches WERs of 18.90% and 6.38%.
Takeaways & Limitations
DSI representations improve dysarthric ASR while reducing severity-dependent variation and generalizing across Conformer and Qwen-based SpeechLLM backends.
Abstract
from arXiv · showhide
This paper proposes novel differentiable and severity-invariant (DSI) discrete token approaches that are not only tightly integrated with downstream dysarthric speech recognition tasks, but also minimise discrete token diversity across speech impairment severity groups. Experiments conducted on the UASpeech and TORGO corpora suggest that Conformer models trained using the DSI tokens outperform the comparable baseline HuBERT discrete/continuous features by statistically significant WER reductions of 2.22\%/0.78\% absolute (9.14\%/3.41\% relative) and 1.78\%/1.06\% absolute (18.43\%/11.86\% relative) on the two tasks, respectively. After system combination, the lowest WERs of 18.90\% and 6.38\% were obtained on UASpeech and TORGO. Phoneme-specific T-SNE visualizations show that severity-invariant regularization reduces severity-dependent variation by producing greater overlap and less distinct boundaries among severity-group distributions.
I. INTRODUCTION
Dysarthric speech recognition remains difficult because data are scarce, speech differs substantially from normal speech, and speakers vary widely. The paper addresses limitations of discrete tokens by integrating differentiable, ASR-oriented learning with severity-invariant regularization.
- I. INTRODUCTION: Discrete tokens retain advantages for dysarthric ASR through sequence compression, lower storage and transmission costs, and support for unified text-speech modeling.These properties also support on-device inference under limited computational resources.
- I. INTRODUCTION: Prior discrete-token approaches face training-objective mismatch and pathology-induced speaker heterogeneity, while severity-related variation has been largely overlooked.These factors contribute to performance degradation in dysarthric speech recognition.
- I. INTRODUCTION: DSI tokens combine iterative pseudo-label refinement, end-to-end integration with a Conformer backend, and severity-invariant regularization.The regularization minimizes differences between token distributions across impairment severity groups.
- I. INTRODUCTION: 2.22%/0.78% and 1.78%/1.06% absolute WER reductions over comparable HuBERT discrete/continuous features were reported on UASpeech and TORGO, respectively.The corresponding relative reductions were 9.14%/3.41% and 18.43%/11.86%.
- I. INTRODUCTION: System combination achieved lowest WERs of 18.90% on UASpeech and 6.38% on TORGO.Qwen-based SpeechLLM experiments also reported consistent WER reductions across both tasks.
II. FOUNDATION MODELS AND DISCRETE TOKENS
The section describes HuBERT as a self-supervised foundation model that produces continuous speech representations and quantized discrete units. A bottlenecked, fine-tuned HuBERT supplies the continuous features used for downstream token extraction.
- II. FOUNDATION MODELS AND DISCRETE TOKENS: HuBERT produces continuous representations, contextual Transformer features, and quantized speech units through CNN encoding, masked prediction, and quantization.Its pre-training alternates between offline clustering for pseudo-label generation and masked prediction optimization.
- II. FOUNDATION MODELS AND DISCRETE TOKENS: Discrete token extraction first obtains continuous speech representations and then quantizes them into discrete tokens.
- II. FOUNDATION MODELS AND DISCRETE TOKENS: A cross-domain fine-tuned HuBERT bottleneck changes the temporal stride to 10 ms, reduces dimensionality from 1024 to 256, then restores the representation to 1024 dimensions.The bottleneck uses transposed and standard CNN layers with intervening fully connected blocks.
III. DIFFERENTIABLE DISCRETE TOKENS
Differentiable K-means learns a codebook and soft assignments so continuous speech representations can be quantized while remaining trainable. A straight-through hard assignment is used for the initial token-learning stage.
- III. DIFFERENTIABLE DISCRETE TOKENS: Differentiable K-means learns a K-centroid codebook by minimizing Euclidean distance between continuous representations and their softassigned centroids.K, D, B, and N denote codebook size, feature dimension, batch size, and sequence length.
- III. DIFFERENTIABLE DISCRETE TOKENS: Soft assignments are computed from negative Euclidean distances to codebook centroids, and Gumbel-Softmax samples relaxed assignment vectors.The temperature controls the relaxed assignment, while σ is set to 1.0.
- III. DIFFERENTIABLE DISCRETE TOKENS: The K-means loss minimizes distance between features and selected centroids, with hard one-hot assignments obtained through a straight-through argmax.This loss is used only during initial DKM training; later stages use pseudo-label and ASR objectives.
B. Iterative Discrete Pseudo-label Update
The iterative pseudo-label procedure progressively refines discrete tokens through alternating quantizer training and corpus re-tokenization. Refined tokens can supervise another iteration or serve directly as downstream ASR features.
- B. Iterative Discrete Pseudo-label Update: Each iteration alternates token-supervised quantizer updating with a discrete-token update that generates refined labels.The current token set supervises HuBERT and DKM optimization before the updated models re-tokenize the corpus.
- B. Iterative Discrete Pseudo-label Update: The initial token set is obtained by clustering continuous HuBERT representations, while the CNN feature encoder remains frozen during pseudo-label training.The remaining HuBERT parameters and DKM codebook are jointly updated.
- B. Iterative Discrete Pseudo-label Update: Updated HuBERT and DKM models extract continuous representations and use framewise argmax assignments to produce the next hard token set.
- B. Iterative Discrete Pseudo-label Update: The refined token set can either supervise the next iteration or be used directly as downstream ASR input features.
C. End-to-End Optimization between Discrete Token Extraction and Conformer ASR
The HuBERT encoder, differentiable K-means quantizer, and Conformer ASR backend are jointly optimized with ASR supervision to learn ASR-oriented discrete tokens.
- Joint optimization trains HuBERT, DKM, and the Conformer backend together toward improved recognition performance.ASR loss is back-propagated through the entire pipeline.
- Speech representations pass from HuBERT through DKM into the Conformer, which predicts the corresponding transcript.
- The pipeline uses the multi-task objective LASR = 0.3L_CTC + 0.7L_Att.L_Att. denotes attention loss.
- HuBERT and DKM start from pseudo-label-refinement parameters, while the Conformer is randomly initialized.
- The CNN feature extractor remains frozen during training, following the standard HuBERT fine-tuning setup.
IV. SEVERITY-INVARIANT DISCRETE TOKENS
Severity-invariant regularization augments end-to-end discrete-token learning by aligning dysarthric and healthy utterance-level token distributions.
- α is an empirically set hyperparameter in the regularization formulation.
- For each dysarthric utterance, SIR compares its averaged token distribution with that of a content-parallel healthy utterance.
- The dysarthric utterance-level distribution is formed by averaging soft assignment vectors across the encoded representation sequence.The sequence length is denoted N_d.
- The corresponding healthy utterance distribution is computed similarly during training.
- The SIR term minimizes divergence between dysarthric and healthy token distributions to reduce severity-dependent variation.The healthy control distribution is stop-gradient treated.
A. Task Description
The experiments use UASpeech and TORGO dysarthric-speech corpora with speaker-partitioned training and test sets, while Table I compares representation efficiency.
- UASpeech contains 103 hours from 16 dysarthric and 13 healthy speakers in an isolated-word recognition task.Its test set uses B2 from dysarthric speakers, while training uses B1 and B3 from all speakers plus B2 from controls.
- TORGO contains 13.5 hours from 8 dysarthric and 7 control speakers, including sentences and single words.Training uses all control data and two-thirds of dysarthric data; the remainder is testing.
- Table I compares continuous HuBERT, HuBERT K-means, and compressed discrete representations on Conformer ASR efficiency.It covers relative sequence length, convergence training time, and CPU inference real-time factor on both test sets.
B. Experiment Setup
The setup uses HuBERT-Large2 features with ESPnet Conformer systems, Kaldi hybrid TDNN systems, and Qwen2.5-based SpeechLLM ASR systems.
- HuBERT-Large2 extracts both continuous SSL features and discrete token features.
- E2E Conformer systems use ESPnet, while hybrid LF-MMI factored TDNN systems use Kaldi's standard chain recipe without i-vectors.
- SpeechLLM-based ASR systems are built with Qwen2.5-0.5B-Instruct.
- The DKM codebook size is evaluated at K = 100 and K = 500, with temperature τ fixed at 0.8.
C. Result Analysis
Across UASpeech and TORGO, DSI tokens improve Conformer dysarthric ASR over standard discrete and continuous HuBERT baselines, while system combination achieves the lowest reported WERs. The gains are strongest for more severe speakers, and visualizations support reduced severity-dependent variation.
- Efficiency trade-off: Uncompressed discrete tokens reduce training time versus continuous features, but deduplication and BPE compression cause additional WER degradation.The experiments therefore retain uncompressed tokens because recognition accuracy is prioritized over efficiency.
- Comparison with standard discrete tokens: 2.22% and 1.78% absolute WER reductions over standard K-means tokens are obtained on UASpeech and TORGO, respectively.The UASpeech reduction is statistically significant and reaches 9.14% relative; TORGO reports a statistically significant reduction.
- Severity-group analysis: The largest subgroup gains occur for severe speakers, including UASpeech VL WER falling from 63.01% to 55.90%.On TORGO, Severe WER falls to 12.11%, while Mild and Moderate speakers show marginal changes.
- System combination: 18.90% and 6.38% are the lowest WERs achieved after system combination on UASpeech and TORGO, respectively.On UASpeech, the combined system closely matches the continuous SSL counterpart; TORGO shows the same pattern.
- Representation analysis: Severity-invariant regularization reduces subgroup separation, with greater overlap and less distinct boundaries in phoneme-specific T-SNE distributions.Figure 2 compares representations without and with regularization across four representative phonemes; ellipses mark confidence regions and stars mark centroids.
VI. CONCLUSION
The paper introduces differentiable and severity-invariant discrete tokens for dysarthric speech recognition, combining pseudo-label refinement, end-to-end optimization, and severity-invariant regularization. Experiments on UASpeech and TORGO show improvements over conventional K-means tokens and comparable continuous HuBERT features.
- Differentiable and severity-invariant discrete tokens improve dysarthric speech recognition on both UASpeech and TORGO.The conclusion reports consistent improvements over conventional K-means tokens and comparable continuous HuBERT SSL features.
- The framework combines iterative pseudo-label refinement, end-to-end optimization, and severity-invariant regularization.These components learn ASR-oriented representations with reduced severity-dependent variation.
- Future work will investigate finer-grained local pathological speech characteristics for representation learning.