Source-linked AI summary
Beyond Decodability: Do Acoustic Factors Drive Predictions in Speech-Based Alzheimer's Assessment?
Serli Kopar, Alkis Koudounas, Roshan P. Rane, Sam Gijsen, Paula A. Perez-Toro, Kerstin Ritter
TL;DR
Speech-based AD assessment increasingly uses SSL models whose representations encode acoustic factors, raising the question of whether those factors influence predictions. The paper intervenes on noise and reverberation across audio streams and finds prediction changes across backbones, including strong effects from noise despite no significant group difference.
Problem
Although prior work found diagnostic information in non-speech audio, it remained open whether acoustic factors influence AD predictions when speech content is available.
Source: §1 · p. 1Method
The study tests three SSL backbones using controlled noise and reverberation interventions in input and representation space across participant-only, full-recording, and non-speech streams.
Source: §1 · p. 1Results
Takeaways & Limitations
Clinical speech models should be evaluated with intervention-based robustness alongside predictive accuracy because a nonsignificant acoustic group difference does not establish robustness.
Source: §4 · p. 4The study examines only noise and reverberation using post-hoc interventions; validation on in-the-wild multilingual recordings remains future work.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Speech-based Alzheimer's disease (AD) assessments increasingly rely on pretrained self-supervised learning (SSL) models that learn acoustic representations directly from raw audio, exposing the model to recording factors. We ask whether such factors are merely encoded in SSL representations or can systematically alter predictions. Using ADReSSo and three large SSL backbones, we apply controlled noise and reverberation interventions to participant-speech-only, non-speech, and full-recording audio. We combine layer-wise linear decoding, input- and representation-space interventions, and geometric alignment analysis to distinguish acoustic decodability from influence on AD prediction. Our results show that controlled acoustic interventions alter AD predictions across all three SSL backbones. Noise, despite showing no significant diagnostic-group difference in the original data, produces the strongest intervention effects. Importantly, these effects are systematically structured relative to the classifier's decision direction, replicate on the held-out test set and reverse when the representation-space intervention direction is reversed. Together, these findings show that high predictive performance and the absence of a significant diagnostic-group difference in a measured acoustic factor are not sufficient for robustness. We argue that intervention-based robustness tests should become standard for trustworthy clinical speech models.
1. INTRODUCTION
The study tests whether acoustic factors in SSL representations merely can be decoded or also systematically influence AD predictions. Controlled interventions alter predictions across backbones, motivating robustness tests beyond acoustic decodability.
1. INTRODUCTION
- Prior non-speech studies found AD-predictive information in silence and background acoustics, leaving open whether acoustics influence predictions when speech is available.Source: §1 · p. 1
- The study asks whether acoustic factors are merely decodable from SSL representations or systematically influence downstream AD predictions.Source: §1 · p. 1
- The framework intervenes on noise and reverberation in input and representation space across participant-only, full-recording, and non-speech streams.Source: §1 · p. 1
- Across three SSL backbones, controlled acoustic interventions alter AD predictions, with noise producing the strongest changes despite no significant diagnostic-group difference.Source: §1 · p. 1
2. METHODOLOGY
The study uses ADReSSo recordings, three speech-region streams, and controlled acoustic degradations to test how noise and reverberation relate to AD prediction. Layer-wise probes, frozen classifiers, input- and representation-space interventions, and geometric alignment analyses separate acoustic decodability from prediction changes.
2.1. Dataset and Quality Control
- The speaker-disjoint ADReSSo splits contain 166 training and 71 test participants; SNR differs nonsignificantly by diagnosis, while SRMR differs in training but not test.Streams comprise participant speech (PAT), non-speech (NS), and the full recording (FULL), constructed using voice activity detection and participant timestamps.Source: §2.1 · pp. 1–2
2.2. Counterfactual Data Generation
- A 2 × 2 × 4 design varies noise or reverberation, real or synthetic sources, and four degradation severities.Noise uses environmental recordings or Gaussian noise; reverberation uses measured or simulated room impulse responses, and PAT and NS are reconstructed after degrading FULL.Source: §2.2 · p. 2
2.3. Feature Extraction and Prediction Framework
- Mean-pooled layer representations from Wav2Vec2-Large, HuBERT-Large, and WavLM-Large are evaluated with Ridge regression for continuous targets and logistic regression for AD.Models are tuned through nested cross-validation on the 166-speaker training set; HuBERT is presented in the main results, then evaluated once on the held-out test set after retraining.Source: §2.3 · p. 2
2.4. Experimental Framework
- The framework probes layer-wise decoding, tests input-space degradation with frozen AD classifiers, steers hidden representations, and measures intervention alignment with the classifier decision direction.Input and representation interventions quantify prediction flips; geometric alignment is compared with magnitude-matched random directions.Source: §2.4 · pp. 2–3
- Representation steering averages training-fold degradation shifts into a shared direction, estimates typical strength by median projection, and propagates steered representations to the AD-selected layer.A reversed steering direction serves as a reversibility control.Source: §2.4 · pp. 2–3
- For alignment, intervention displacements are standardized to match classifier features, and cosine similarity is interpreted geometrically because it characterizes the linear classifier's logit change.The null intervals use the central 95% of 100 magnitude-matched random-direction controls.Source: §2.4 · pp. 3–4
3. RESULTS
Controlled acoustic changes altered AD predictions, with stronger effects for noise than reverberation and directionally structured shifts. Representation-space effects also generalized across experiments to held-out data.
E1, Linear Decoding
- Across HuBERT layers, SNR remained highly decodable, reaching R2 ≈0.8 in FULL, while MMSE decodability in PAT and FULL reached R2 ≈0.5 in later layers.AD classification performance peaked at layer 12; SRMR was decodable in early speech-containing layers (R2 ≈0.6) and declined with depth.Source: §3 · p. 4
- NS retained strong SNR information and above-chance AD classification, while SRMR and MMSE were essentially not linearly decodable there.SRMR decodability declined with depth in PAT and FULL, while MMSE decodability increased across layers.Source: §3 · p. 4
E2 & E4, Input-space Intervention & Alignment
- In FULL at ℓ=1, SNR(S) flipped 51% of predictions versus 8% for SRMR(S), and flip rates generally increased with severity except for NS under SNR.Under SRMR, AD→HC flips dominated in PAT and FULL, while HC→AD flips dominated in NS; under SNR, AD→HC flips dominated in NS.Source: §3 · p. 4
- For input-space shifts, 44/48 conditions (91.7%) fell outside the central 95% range of magnitude-matched random directions, with alignment varying by factor, stream, and severity.Positive cosine similarity indicates alignment toward AD; SNR shifts in PAT and FULL became more AD-directed as degradation increased, while NS remained strongly HC-directed.Source: §3 · p. 4
E3 & E4, Representation-space Intervention & Alignment
- At r*=9, FULL D4 SNR steering flipped 42% of predictions; train–test effects agreed at CCC = 0.90−0.99 across experiments.The representation-space logit effect reversed when steering changed from α = +1 to −1 (rPearson = −0.98).Source: §3 · p. 4
4. DISCUSSION AND LIMITATIONS
Acoustic interventions affect AD predictions even when measured group differences are absent, and their direction varies across factors, streams, and severities. The study's conclusions are bounded by its focus on noise and reverberation and post-hoc interventions.
4. DISCUSSION AND LIMITATIONS
- Intervention effects vary across factors, streams, and severities, so the findings do not support a simple fix that removes one global acoustic-confound direction.Input- and representation-space interventions can shift decisions differently.Source: §4 · p. 4
- SNR showed no significant diagnostic-group difference yet produced the largest intervention-induced flips, so group-difference tests alone do not establish robustness.The authors recommend intervention-based robustness evaluation alongside predictive accuracy.Source: §4 · p. 4
- The study tests only noise and reverberation, using post-hoc interventions; validation with in-the-wild multilingual recordings remains future work.The interventions span real and synthetic sources.Source: §4 · p. 4
5. COMPLIANCE WITH ETHICAL STANDARDS
The study used existing ADReSSo data from DementiaBank, collected with informed consent and used according to its access requirements.
5. COMPLIANCE WITH ETHICAL STANDARDS
- Existing ADReSSo data derived from DementiaBank were collected with informed consent and used in accordance with DementiaBank access requirements.Source: §5 · p. 5