Augmenting Large Audio Language Models with Low-Level Acoustic Features for Dysarthric Speech Detection

1. INTRODUCTION

• Dysarthria can hinder communication and may indicate an underlying neurological disease, while traditional auditory-perceptual assessments are costly and time-consuming.These clinical demands motivate automatic detection approaches.Source: §1 · p. 1
• Prior LALM work either lagged DL baselines or used a separate classifier, while semantic encoders often neglect acoustic information important to dysarthria.The cited work also notes limits on using GPT-4o with datasets such as PC-GITA.Source: §1 · p. 1
• The proposed framework combines speech recordings with textual low-level acoustic features and speaker demographics during LALM fine-tuning.The schematic contrasts conventional raw-audio and text inputs with the proposed addition of acoustic features and metadata to the text.Source: §1 · p. 1, Figure 1 · p. 2
• Both fine-tuned LALMs outperform state-of-the-art DL-based approaches, with ablations reporting state-of-the-art performance when acoustic features and demographics are included.The paper also evaluates zero-shot capability and the effects of adding these inputs.Source: §1 · p. 1

2.1. DL-based automatic dysarthric speech detection

• Self-supervised foundation-model representations combined with classifiers outperform time-frequency-based approaches and establish the current state of the art.The paper uses these self-supervised approaches as its DL-based baselines.Source: §2.1 · p. 2

2.2. Large Audio Language Models

• LALM research on dysarthric speech detection is limited; prior approaches either lag DL baselines or use a separate classifier, motivating an open-source LALM for direct detection.The strongest results of the cited classifier approach depend on closed-source GPT-4o, whose license does not permit use with datasets such as PC-GITA.Source: §2.2 · p. 2

3. PROBLEM FORMULATION

• A tokenizer maps raw text to discrete vocabulary tokens, while a learned audio encoder maps the waveform to continuous frame representations.The audio representations are projected into the language model's embedding space.Source: §3 · p. 2
• The LALM factorizes output-token probabilities autoregressively, conditioned on tokenized text, projected audio, and previously generated tokens.Each next-token factor is a categorical distribution over the text vocabulary.Source: §3 · p. 2
• The model predicts whether a speech sample is dysarthric or neurotypical from a raw audio waveform and textual input.The label is 1 for dysarthric speech and 0 for neurotypical speech; the text contains a task instruction.Source: §3 · p. 2

• Supervised fine-tuning classifies speech as neurotypical or dysarthric, with cross-entropy loss applied to response tokens and prompt tokens masked.The standard approach uses the audio sample and a textual task prompt.Source: §4 · p. 2
• The feature-augmented prompt adds speaker demographics and acoustic information, addressing LALMs’ limited attention to acoustic cues relevant to dysarthria.The method provides these additions alongside raw audio during fine-tuning.Source: §4 · p. 3, Table 1 · p. 3
• The acoustic descriptors are mean and standard deviation of pitch and loudness, mean harmonic-to-noise ratio, and speech rate, selected as an illustrative starting point.The selection follows clinical associations rather than feature-selection or optimization; the framework can accommodate other descriptors.Source: §4 · p. 3
• The method supplies exact numerical feature values in the textual prompt rather than converting them into categorical labels.Source: §4 · p. 3

• PC-GITA provides Spanish recordings from 50 Parkinson’s patients and 50 neurotypical speakers; experiments use monologue, read-text, and sentence subsets.Recordings are resampled from 44.1 kHz to 16 kHz.Source: §5 · p. 3
• Utterances are segmented into 30-second chunks with 50% overlap, and 10-fold cross-validation is repeated with three random seeds.The final residual segment of each utterance is retained; results are reported as the mean and standard deviation across runs.Source: §5 · p. 3
• The evaluated LALMs are Qwen2-Audio and Phi-4-multimodal; Qwen2-Audio performs best on audio under 30 seconds, while Phi-4 can encode long-form audio.For comparability, Phi-4 also receives 30-second segments.Source: §5 · p. 3
• Deep-learning baselines use wav2vec 2.0 or XLS-R representations, temporal averaging, and a two-layer classifier with a ReLU nonlinearity.The selected representation layers are the 10th for wav2vec 2.0 and the 16th for XLS-R.Source: §5 · p. 3
• LALM fine-tuning uses five epochs and a cosine learning-rate scheduler, with model-specific learning rates and adapter configurations.Qwen2-Audio uses learning rate 1 × 10−4 and LoRA; Phi-4-multimodal uses 5 × 10−5, continues its speech LoRA, and unfreezes the audio encoder and projection layer.Source: §5 · pp. 3–4
• Table 2 reports speaker-level accuracy for the considered approaches on PC-GITA using 10-fold cross-validation.Source: Table 2 · p. 4
• Baseline training uses cross-entropy for 100 epochs with Adam and ReduceLROnPlateau; model selection uses validation loss, and performance is speaker-level accuracy.Speaker diagnoses are obtained by voting across a speaker’s utterances.Source: §5 · p. 4
• Low-level features are computed with openSMILE and supplied as numerical values: pitch and loudness means and standard deviations, harmonic-to-noise ratio, and speech rate.Source: §5 · p. 4

Comparison with state-of-the-art baselines

• Both fine-tuned LALMs achieve state-of-the-art performance against DL-based baselines with the proposed feature-augmented framework.The comparison is reported in Table 2.Source: §6 · p. 4

Ablation study: fine-tuning and feature augmentation

• Both LALMs are chance-level zero-shot; fine-tuning accounts for most improvement, while feature augmentation adds a smaller, consistent gain and reduces performance variation across seeds.Table 3 reports the zero-shot and feature-augmented ablation.Source: §6 · p. 4, Table 3 · p. 4

Ablation study: individual acoustic features

• Incrementally adding features improves performance overall, with all four features combined achieving the best result.The individual-feature ranking was HNR, speech rate, loudness, then pitch; the figure reports mean and standard deviation.Source: §6 · p. 4, Figure 2 · p. 4

• The framework augments LALM textual input with low-level acoustic features and speaker demographic information for dysarthric speech detection.Source: §7 · p. 4
• The experiments show that the framework consistently improves LALM performance.Source: §7 · p. 4
• The findings demonstrate the potential of LALMs for dysarthric speech detection and motivate further research.Source: §7 · p. 4