PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Learning for Non-Canonical Phoneme Recognition via Articulatory Feature Decomposition

Sophia Riaz, Haoze Zheng, Amos Roche, Miyu Zhang, Anamika Ragu, Salvatore Penachio, Kaustav Mukherjee, Aneesh Jonelagadda

arXiv:2608.22273v1cs.SDcs.AI

TL;DR

Non-canonical phoneme recognition is limited by structured pronunciation deviations, scarce clinical labels, and categorical models that omit articulatory structure. The paper introduces hierarchical articulatory multi-task learning with cross-attention fusion and MPL, reporting consistent improvements over strong baselines on L2-ARCTIC and interpretable feature-linked error patterns. The authors position accented speech as a proxy and call for validation on clinically diagnosed datasets.

  • Problem

    Non-canonical phoneme recognition is difficult because clinical labels are scarce and existing categorical models do not explicitly represent structured articulatory deviations.

  • Method

    The paper decomposes phoneme prediction into articulatory feature tasks, fuses their representations through cross-attention, and combines the framework with MPL.

  • Results

    The approach yields consistent phoneme-error-rate improvements over strong baselines, with the largest accuracy improvement from combining multi-task learning, pseudo-labeling, and strong augmentations.

  • Takeaways & Limitations

    Articulatory supervision produces interpretable structured error patterns and is promising for robust non-canonical phoneme recognition.

  • Takeaways & Limitations

    The experiments use accented L2 speech as a proxy rather than clinically validated pathological datasets, and the reduced ARPAbet inventory omits fine-grained phonetic distinctions.

Abstract

from arXiv · show

Pathological and more broadly non-canonical speech present significant challenges for automatic phoneme recognition due to systematic deviations from canonical pronunciation and limited availability of labeled clinical speech data. Existing phoneme recognition systems are typically trained on canonical speech and treat phonemes as atomic categorical labels, limiting their ability to detect structured articulatory errors common in speech disorders and accents. In this work, we introduce a linguistically structured approach to non-canonical phoneme recognition that decomposes phoneme prediction into articulatory feature dimensions such as manner, place, and voicing. We implement this formulation using a hierarchical multi-task learning architecture in which task-specific articulatory feature heads learn feature-level representations that are subsequently integrated through a cross-attention-based fusion module to produce phoneme predictions. To address the scarcity and noise of pathological speech labels, we combine this framework with semi-supervised learning via Momentum Pseudo-Labeling (MPL) and propose a cascaded training strategy that progressively introduces articulatory feature tasks while employing staged unfreezing of a pretrained speech encoder. Experiments on L2-ARCTIC, used as a proxy for pathological speech variation, show that the proposed approach achieves substantial improvements in phoneme recognition performance compared to strong baseline architectures, while yielding interpretable error patterns aligned with phonological feature structure. These results suggest that articulatory feature supervision is a promising strategy for robust and interpretable phoneme recognition in non-canonical speech, and motivate future validation on clinically diagnosed pathological speech datasets.

1. Introduction

The introduction frames phoneme-level analysis of non-canonical speech as difficult because existing systems rely on canonical data and atomic phoneme labels. It proposes articulatory multi-task supervision with cross-attention fusion to improve low-resource recognition and analyze structured errors.

  • Phoneme-level analysis remains challenging even as prosodic analysis is relatively robust in modern audio-processing pipelines.
  • Reported PERs for common conditions such as MSD and CAS range from 42% to 69% because existing models struggle to generalize to disordered speech.
  • Atomic phoneme labels do not explicitly encode articulatory structure, although non-canonical errors may preserve some articulatory properties while deviating along others.
  • The proposed framework jointly predicts articulatory features and phonemes, then integrates feature representations through cross-attention-based fusion.
  • The evaluation uses L2-ARCTIC to measure PER, mispronunciation detection, ablations, and error patterns along articulatory dimensions.
  • The work investigates whether articulatory supervision can incorporate phonological structure and reduce errors organized along articulatory dimensions.

2. Background

The background reviews motor-speech variability, the limits of word-level and categorical ASR, and the scarcity of clinical labels. It motivates articulatory structure as a low-resource modeling direction while positioning the work against prior MPL and multi-task approaches.

  • Motor speech disorders include apraxia of speech and dysarthria, which disrupt motor planning or execution and produce broad phonetic variability.
  • Word-level ASR optimization can obscure fine-grained articulatory deviations relevant to diagnosing and treating motor-speech disorders.
  • Phoneme-level systems rarely characterize how mispronunciations manifest acoustically, while clinical descriptions of disordered speech are inherently feature-based.
  • Clinical speech datasets are orders of magnitude smaller than standard ASR datasets and can exhibit substantial inter-annotator variability, motivating use of L2-ARCTIC as a proxy.
  • Articulatory dimensions describe speech-apparatus configuration, while phonological features encode language-level sound patterns; together they provide a structured multidimensional representation.
  • Prior work combines self-supervised representations with MPL, articulatory auxiliary objectives, or probability conditioning, but hierarchical articulatory supervision remains limited for low-resource non-canonical speech.

3. Methods

The method derives articulatory labels from ARPAbet and uses hierarchical multi-task learning to estimate feature representations before cross-attention-enhanced phoneme classification. A pretrained WavLM backbone, specialist heads, residual fusion, temporal modeling, and speech augmentations support this structured pipeline.

  • 3.1 Articulatory Feature Derivation: ARPAbet phonemes are mapped to linguistically motivated articulatory attributes derived through IPA representations.The derived labels preserve compatibility with the corpus inventory while exposing phonological structure for auxiliary supervision.
  • 3.2 Articulatory Feature Multi-Task Learning Architecture: The architecture predicts phoneme type, consonant, vowel, place, manner, voicing, height, backness, and roundedness as auxiliary articulatory tasks.Separate specialist branches produce task-specific representations before downstream phoneme prediction.
  • 3.2 Articulatory Feature Multi-Task Learning Architecture: A frozen pretrained WavLM backbone supplies acoustic and phoneme representations formed from separate learnable weighted sums of transformer layers.The representations pass through shared projection blocks before specialist estimation and phoneme classification.
  • 3.2 Articulatory Feature Multi-Task Learning Architecture: Cross-attention lets the phoneme branch dynamically weight consonant, vowel, and type specialist representations at each time step.The attended articulatory context is residually fused with the phoneme representation to support context-dependent disambiguation.
  • 3.2 Articulatory Feature Multi-Task Learning Architecture: The fused representation passes through a bidirectional LSTM before final phoneme classification, modeling dependencies across neighboring phonemes.The residual formulation preserves the original phoneme encoding while augmenting it with articulatory evidence.
  • 3.2 Articulatory Feature Multi-Task Learning Architecture: The phoneme task uses CTC, while auxiliary tasks use frame-level cross-entropy or binary cross-entropy on category-relevant frames.The auxiliary losses cover multiclass type, place, manner, height, and backness targets plus binary voicing and roundedness targets.

4. Experimental Setup

The study uses L2-ARCTIC, normalizes its phoneme and silence annotations, derives frame-level articulatory targets, and evaluates on speaker-independent splits. Because non-native phoneme labels can be ambiguous, the setup accounts for annotation variability and masks feature supervision by phonological relevance.

  • L2-ARCTIC contains read speech from 24 non-native English speakers across six L1 backgrounds, with four speakers per L1 and balanced gender.
  • Expert-aligned TextGrid annotations provide canonical and perceived phoneme transcriptions for each utterance.
  • Non-native phoneme ground truth is challenging because productions may fall between canonical categories and annotator judgments vary.Prior phone-level annotation studies report Cohen’s Kappa values often ranging from 0.7–0.85.
  • Normalization removes stress distinctions, unifies silence tokens, removes artificial silences, and collapses consecutive pauses.Stress markers are stripped because stress is outside the articulatory assessment focus.
  • The normalized inventory contains 39 non-silence phonemes plus silence; CTC training uses 39 target phonemes and an appended blank, totaling 40 output classes.
  • Seven auxiliary articulatory targets are deterministically derived at frame level from ARPAbet mappings and time-aligned TextGrid boundaries.Consonant-specific features are masked on vowel frames and vice versa, restricting supervision to relevant frames.
  • Speakers are split into 18 training and 6 testing speakers with no speaker overlap; the test set includes one speaker from each L1 background.The training speakers are further divided 90/10 by utterance for training and validation.

Ablation Study

The ablation study isolates articulatory-task and augmentation contributions through leave-one-out comparisons under controlled training conditions. The implementation uses a WavLM-based cascaded architecture, while the reported table evaluates phoneme error rate across model variants.

  • MTL Heads: The MTL ablation study removes one articulatory auxiliary task at a time to isolate its contribution to phoneme recognition.
  • MTL Heads: Each ablated model is trained under identical conditions, and performance differences quantify the contribution of the removed task.
  • MTL Heads: The full task set includes phoneme, type, place, manner, voicing, height, backness, and roundedness.
  • MTL Heads: Hard ablation removes the corresponding prediction head from the network rather than merely masking its task loss.
  • Training Conditions: Ablation training disables augmentation and MPL and limits both stages to five epochs, supporting relative architectural comparisons rather than fully converged absolute performance.
  • Data Augmentation: Augmentation ablations remove phase perturbation, VTLP, pitch shifting, prosodic time-stretching, or optional additive noise one at a time while holding other settings fixed.
  • Implementation Details: The final architecture uses microsoft/wavlm-base-plus as a general-purpose pretrained speech encoder, with task-specific layers trained before partially unfreezing the backbone.The model freezes WavLM in Stage 1 and unfreezes its last six transformer layers in Stage 2, with 15 epochs per stage.
  • Implementation Details: Table 1 reports model performance on L2-ARCTIC using Phoneme Error Rate (%), comparing baseline, augmented-baseline, and proposed MTL variants.

5. Results and Discussion

The proposed hierarchical multi-task framework improves non-canonical phoneme recognition by integrating articulatory supervision with cross-attention, MPL, and augmentation. Error analyses show gains across articulatory dimensions and identify unequal feature contributions and cross-feature dependencies.

  • Model Evaluation and Comparisons: Fine-tuned non-canonical models outperform zero-shot systems, while WavLM slightly exceeds Wav2Vec2 and MPL further improves overall PER.Data augmentation also consistently improves performance, particularly for unstable or low-energy phonetic segments.
  • Model Evaluation and Comparisons: Hierarchical MTL with cross-attention improves more substantially than parallel MTL, whose marginal gains suggest auxiliary objectives require structured integration.The proposed model is compared against baseline and augmented architectures using PER.
  • Model Evaluation and Comparisons: The strongest baseline has PER = 14.46%, with substitution errors dominating and reflecting confusion between acoustically similar phonemes.Baseline errors include difficulty resolving fine-grained distinctions such as voicing and vowel height.
  • Model Evaluation and Comparisons: MTL reduces substitution errors across every articulatory feature, with the largest reductions for voicing (−10.1%) and vowel height (−5.65%).Place and manner show smaller gains of −3.24% and −3.86%, respectively.
  • Multi-Task-Learning Ablations: The all-head configuration achieves PER 14.13%, while removing individual heads increases PER by +0.08 to +0.44, led by Roundedness (+0.44) and Voicing (+0.41).Height and Type produce intermediate drops, whereas Backness, Place, and Manner produce smaller ones.
  • Multi-Task-Learning Ablations: Removing Roundedness increases roundedness-related errors (+5.7%), insertions (+2.9%), and deletions (+2.5%), whereas removing Voicing increases voicing, roundedness (+7.6%), backness (+4.1%), and insertions (+3.8%).Removing Backness reduces deletions (−4.8%) but increases insertions (+4.9%), revealing compensatory trade-offs.
  • Augmentation Ablations: The full augmentation suite achieves the lowest PER of 13.49% for HMTL+c.a.+MPL, while removing pitch shifting raises PER to 13.94% and removing prosodic time-stretching raises it to 13.87%.Removing phase perturbation and VTLP also degrades performance by +0.34% and +0.31% PER, respectively.

6. Limitations

The framework has several limitations in representation granularity, articulatory labeling, and speech dimensions modeled.

  • The reduced ARPAbet inventory collapses fine-grained phonetic variation, including aspiration, flapping, and vowel reduction.This limits capture of subtle clinically relevant pronunciation errors.
  • ARPAbet omits articulatory distinctions present in IPA, reducing the granularity of supervision and evaluation.
  • Single-label auxiliary heads cannot fully represent phonemes with multiple articulatory properties, such as /W/ having labial and dorsal articulations.
  • The framework models segmental phoneme recognition but excludes suprasegmental aspects such as stress, rhythm, and intonation.

7. Conclusion

The paper frames pathological phoneme recognition as difficult because of structured deviations, limited data, and noisy labels, then addresses these challenges with articulatory decomposition and cross-attention. Experiments report consistent phoneme-error reductions and structured changes in errors, while accented speech remains a proxy for clinical pathology.

  • Pathological phoneme recognition remains challenging because speech deviations are structured and labeled data are limited or noisy.
  • The approach composes phoneme recognition from articulatory feature prediction tasks, fuses their representations through cross-attention, and projects them into phoneme space after temporal modeling.
  • Consistent reductions in phoneme error rate occur across strong baselines and ablated variants.
  • Errors decrease overall and redistribute from within-feature confusions toward cross-feature errors, with ablations producing linguistically interpretable patterns.
  • The experiments use accented speech as a proxy, motivating future evaluation on clinically validated pathological datasets and richer articulatory representations.

Appendix A: Additional Inadequate IPA-ARPAbet Articulatory Mappings

The appendix documents limitations and ambiguities in mapping among ARPAbet, IPA, and L2-ARCTIC annotations, including collapsed phonetic distinctions and diacritic-bearing symbols.

  • Table A.1 lists language-specific missing or ambiguous mappings among ARPAbet, IPA, and L2-ARCTIC annotations.
  • Table A.2 catalogs diphthongs and IPA symbols with diacritics.
  • Table A.3 identifies phonetic distinctions collapsed by the L2-ARCTIC ARPAbet inventory.

Appendix B: Additional Evaluations and Results

The appendix provides additional evaluations of phoneme feature errors, error types, and changes between the best baseline and best model, alongside further PER combinations.

  • Additional PERs are reported for combinations of architectures, training strategies, and augmentation presence.
  • Table B.1 reports changes in errors from the best baseline to the best model.
  • Table B.2 organizes phoneme error types by model.
  • Table B.3 reports phoneme feature errors by model.
Loading 2608.22273v1…