PaperlayerView exact revision on arXiv

Source-linked AI summary

ParaCalib: Semantically Calibrated Paralinguistic Modeling for Depression Detection

Yuxin Li, Yifei Li, Yi-Wen Chao, Xiangyu Zhang, Eng Siong Chng, Cuntai Guan

arXiv:2610.01103v1eess.AS

TL;DR

Existing speech-based depression methods typically treat acoustic cues as context-independent markers, although their interpretation can depend on utterance meaning and communicative function. ParaCalib converts contextualized vocal descriptions into structured SC-Para states, and achieves the highest mean Macro-F1 among evaluated representation strategies on both DAIC-WOZ and MODMA.

  • Problem

    Speech-based depression methods typically map acoustic form directly to depression-related evidence without explicitly modeling how semantic context and communicative function shape vocal interpretation.

    Source: §1 · p. 2

  • Method

    ParaCalib maps contextualized vocal descriptions into structured SC-Para states for participant-level depression-label classification.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    ParaCalib achieves the highest mean Macro-F1 among evaluated representation strategies on both DAIC-WOZ and MODMA, and PSE-derived evidence changes with semantic context under fixed caption-level acoustic descriptions.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The findings support structured, context-sensitive, and compositional modeling of vocal behavior for speech-based depression detection.

    Source: §7 · p. 9

    The study evaluates DAIC-WOZ and MODMA independently, so its consistent results do not establish cross-dataset or cross-lingual transfer.

    Source: §8 · p. 10

Abstract

from arXiv · show

Vocal behavior provides important signals for speech-based depression detection, but its interpretation often depends on what is being said and how it functions in context. However, existing methods typically treat acoustic cues as context-independent markers, making it difficult to distinguish vocal form from its context-dependent communicative function. We propose ParaCalib, a framework that semantically calibrates paralinguistic behavior by interpreting vocal patterns relative to utterance-level semantic context and inferred communicative function and representing them in a comparable state space. Concretely, ParaCalib uses an Audio-Language Model (ALM) to generate contextualized vocal descriptions and an LLM-based Paralinguistic State Extractor (PSE) to map these descriptions into a structured Semantically Calibrated Paralinguistic (SC-Para) representation. ParaCalib achieves the highest mean Macro-F1 among the evaluated methods, reaching 71.9\% on DAIC-WOZ and 90.5\% on MODMA. Our controlled analysis provides direct evidence of semantic calibration at the PSE stage: under a fixed caption-level acoustic description, varying the accompanying semantic context changes the inferred depression-related paralinguistic evidence. Our exploratory analysis further identifies recurring configurations of paralinguistic states associated with depression labels rather than a single uniformly dominant state.

1 Introduction

Speech-based depression detection often treats vocal patterns as context-independent markers, although their interpretation can depend on utterance meaning and communicative function. ParaCalib represents vocal behavior as semantically calibrated states and reports the highest mean Macro-F1 among evaluated methods on two datasets.

1 Introduction

  • Most existing speech-based depression methods map acoustic form directly to depression-related evidence without explicitly modeling semantic context or communicative function.These methods follow an acoustic-marker paradigm, treating statistically associated vocal patterns as participant-level indicators.Source: §1 · pp. 1–2
  • Depression can involve slowed or accelerated speech, while the same pause, pitch, intensity, or rate pattern can reflect different communicative functions.Examples include hesitation, emphasis, syntactic planning, turn-taking, topic shifts, or semantic content.Source: §1 · p. 2
  • ParaCalib defines semantic calibration as interpreting vocal behavior relative to utterance-level semantic context and inferred communicative function in a structured, comparable state space.It combines ALM-generated contextualized vocal descriptions with PSE mapping into predefined SC-Para dimensions.Source: §1 · p. 2
  • ParaCalib achieves the highest mean Macro-F1 among evaluated methods, outperforming conventional acoustic and SSL representations while providing inspectable intermediate paralinguistic states.Controlled analysis also finds that PSE-derived evidence changes with semantic context under a fixed caption-level acoustic description.Source: §1 · p. 2

2 Related Work

Prior speech-depression systems use handcrafted or learned acoustic representations, while related linguistic and audio-language approaches do not explicitly structure how vocal meaning varies with context. ParaCalib maps contextualized vocal descriptions into fixed SC-Para dimensions to retain context sensitivity while enabling comparison and aggregation.

2.1 Depression Detection from Speech

  • Speech-depression systems progressed from handcrafted acoustic descriptors to learned and pretrained representations, but generally use them as generic participant-level features rather than interpreting vocal behavior in context.Related linguistic work uses syntax or addresses semantic shortcuts, yet does not explicitly represent how vocal-pattern meaning changes with utterance context and communicative function.Source: §2.1 · pp. 2–3

2.2 Audio-Language Models for Affective Speech Understanding

  • Audio-language models support semantically rich vocal descriptions and affective reasoning, but free-form descriptions are hard to compare and closed-set classifiers compress vocal behavior into coarse categories.Existing approaches primarily optimize prediction or explanation generation rather than a structured representation preserving context-dependent distinctions.Source: §2.2 · p. 3
  • ParaCalib maps contextualized vocal descriptions into fixed SC-Para dimensions, combining contextual sensitivity with comparable states that support participant-level aggregation and representation analysis.Its pipeline uses ALM captioning, PSE extraction, and attention-based MIL aggregation of utterance-level vectors.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Methodology

ParaCalib converts utterance-level audio into contextualized vocal descriptions, maps them to structured SC-Para states, and aggregates valid utterance representations for participant-level classification. The downstream classifier is optimized while the ALM and PSE remain frozen.

3.1 Overview

  • ParaCalib uses ALM captioning, PSE-based SC-Para extraction, and attention-based MIL aggregation to predict participant-level depression labels.The ALM and PSE stay frozen during training; only the downstream MIL classifier is optimized.Source: §3.1 · pp. 3–4

3.2 Utterance-level ALM Captioning

  • The frozen ALM describes spoken content and vocal behavior together, including pitch, rate, rhythm, pauses, energy, and voice quality.The captioner processes utterance-level audio without external text prompts, dataset fine-tuning, or participant-level depression labels.Source: §3.2 · p. 4

3.3 SC-Para Extraction

  • The PSE maps descriptions into five three-state paralinguistic dimensions and two holistic scores: depression-related evidence and confidence.The evidence score is normalized from 0–100 to [0, 1], while confidence is represented on [0, 1]; unknown categorical states can exclude an utterance before aggregation.Source: §3.3 · p. 4

3.4 Attention-based MIL Classification

  • Attention weights utterance-level SC-Para vectors within each participant’s bag, and the pooled representation feeds a two-hidden-layer MLP for binary classification.The readout preserves the seven-dimensional SC-Para representation.Source: §3.4 · pp. 4–5

3.5 Optimization Objective

  • Training keeps the ALM and PSE frozen and optimizes the attention module and downstream MLP using weighted binary cross-entropy.The positive-class weight is Nneg/Npos and the negative-class weight is 1.Source: §3.5 · p. 5

4 Data and Evaluation

The evaluation uses DAIC-WOZ and MODMA with participant-level depression labels and held-out participant evaluation. It combines matched-run or fold-level comparisons with diagnostic analyses of context effects, prompt sensitivity, SC-Para states, speaker identity, and acoustic noise.

4.1 Datasets

  • DAIC-WOZ includes 189 English-speaking participants and uses PHQ-8≥10 labels; MODMA experiments use PHQ-9≥10 labels rather than its original diagnostic grouping.After preprocessing, DAIC-WOZ has 57 depression-label and 132 non-depression-label participants; MODMA has 22 and 30, respectively.Source: §4.1 · p. 5

4.2 Evaluation Protocol

  • DAIC-WOZ uses official partitions, while MODMA uses participant-level five-fold cross-validation repeated with five seeds and held-out participants.Participant-level Macro-F1 is primary, with sensitivity, specificity, and AUC also reported; results are summarized as mean and SD.Source: §4.2 · p. 5

4.3 Statistical and Diagnostic Analyses

  • Representation comparisons use paired tests across five matched evaluation units, report 95% confidence intervals, and apply Holm correction within analysis families.Context controls hold the caption-level acoustic description fixed across acoustic-only, non-depression-related, and depression-related contexts; prompt sensitivity uses meaning-preserving paraphrases on the same 100 captions.Source: §4.3 · pp. 5–6
  • Additional analyses screen utterance-level state-label associations, probe speaker-identity leakage, and test additive-noise robustness on DAIC-WOZ test utterances.Noise is evaluated at 20, 10, and 5 dB SNR and is not used for training or model selection; clustered utterance associations are descriptive, not participant-level confirmatory inference.Source: §4.3 · p. 6

5 Results and Analysis

ParaCalib achieves the highest mean Macro-F1 among evaluated methods on both datasets, while representation comparisons and a controlled context analysis examine how its structured states relate to performance. Component, state-level, robustness, and error analyses further characterize its behavior and scope.

5.1 Depression Detection Performance

  • 71.9% Macro-F1 on DAIC-WOZ and 90.5% on MODMA are the highest mean scores among evaluated methods; gains over HAREN-CTC are 14.0 and 17.5 percentage points.The Holm-adjusted paired-comparison p-values are .0232 and .0734, respectively.Source: §5.1 · p. 6

5.2 Representation Comparison

  • SC-Para reaches 71.9% and 90.5% Macro-F1, exceeding the strongest generic SSL representations at 46.3% and 68.8%, respectively.The table compares complete representation strategies under a common downstream protocol, not parameter-matched models.Source: §5.2 · p. 6, Table 2 · p. 7
  • Transcript-masked Caption-PSE reaches 64.6% Macro-F1 on DAIC-WOZ and 60.6% on MODMA, while content masking indicates a stronger semantic-context contribution on MODMA.The authors accordingly characterize ParaCalib as integrating semantic context and vocal-delivery information rather than isolating a purely acoustic marker.Source: §5.2 · pp. 6–7

5.3 Semantic-Context Control

  • With the caption-level acoustic description fixed, depression-related context raises PSE evidence by 8.6 points over non-depression-related context and 8.7 over acoustic-only input.Non-depression-related context differs from acoustic-only input by 0.1 points (p = .92).Source: §5.3 · p. 7

5.4 Component and State-Level Analysis

  • Removing the evidence score, speaking rate, and vocal energy produces the largest DAIC-WOZ Macro-F1 drops: 5.0, 4.7, and 4.5 percentage points.Energy and evidence score contribute primarily to sensitivity, while speaking rate contributes more strongly to specificity; separate component results suggest categorical and holistic dimensions are complementary.Source: Figure 3 · p. 8, §5.4 · p. 8
  • Categorical dimensions correlate from −0.400 to 0.703 yet are not redundant; state associations are selective, and seven combinations meet descriptive screening criteria.Positive associations include low energy, flat pitch, moderate pauses, and specific speaking-rate states; several combinations pair low energy with slow speech, steady rhythm, few pauses, or flat pitch.Source: §5.4 · p. 8

5.5 Robustness and Error Analysis

  • SC-Para shows lower speaker-identity probe AUC than WavLM and stable extraction and paraphrase results, but additive noise degrades performance.The probe AUCs are 52.9% versus 78.7%; repeated decoding agreement is 94–99%, prompt-paraphrase correlations are r ≥.841, and at 5 dB SNR Macro-F1 falls from 71.9% to 62.6%. Both raters judged 18 of 20 captions and 17 of 20 state sets consistent with perceived vocal evidence.Source: §5.5 · p. 8
  • Consistent false positives cluster nearer the PHQ-8 cutoff, while consistent false negatives have fewer valid utterances than reliably classified true positives.Mean PHQ-8 scores are 6.4 versus 2.6 for false positives and controls; false negatives average 66 utterances versus 119 for true positives.Source: §5.5 · p. 8

6 Discussion

The discussion presents semantic calibration as a useful modeling principle and argues that structured paralinguistic states make vocal behavior more interpretable across utterances. The reported analyses remain descriptive and do not establish clinical symptoms or causal feature importance.

6 Discussion

  • Contextualizing vocal behavior with utterance-level meaning and communicative function shifts PSE-derived evidence, supporting semantic calibration as a modeling principle.The controlled analysis holds the caption-level acoustic description fixed while varying semantic context.Source: §6 · pp. 8–9
  • SC-Para outperforms evaluated handcrafted and SSL representations and direct Caption Embedding on both datasets, while providing fixed-dimensional dimensions comparable across utterances and participants.The component analyses suggest categorical vocal states and holistic PSE scores provide complementary information.Source: §6 · p. 9
  • Utterance-level state associations and classifier analyses describe extracted-representation behavior, not causal feature importance or clinically observed symptoms.These interpretations are bounded by the evaluated datasets and protocols.Source: §6 · p. 9

7 Conclusion

ParaCalib maps contextualized vocal descriptions into a compact, inspectable SC-Para representation for participant-level depression-label classification. It achieves the highest mean Macro-F1 among evaluated representation strategies on both datasets, and controlled analysis shows PSE evidence shifts with semantic context.

7 Conclusion

  • ParaCalib achieves the highest mean Macro-F1 among evaluated representation strategies on both DAIC-WOZ and MODMA.It exceeds conventional acoustic, SSL-based, transcript-based, and direct caption-embedding alternatives.Source: §7 · p. 9
  • ParaCalib maps contextualized vocal descriptions into compact, inspectable SC-Para representations for participant-level depression-label classification.The approach combines ALM-generated descriptions with structured state extraction.Source: §7 · p. 9
  • With the caption-level acoustic description fixed, changing semantic context shifts the PSE-derived evidence score.The conclusion presents this as support for structured, context-sensitive, and compositional modeling of vocal behavior.Source: §7 · p. 9

8 Limitations

The authors identify limitations in recording conditions, label scope, model robustness, state-association inference, and generalizability. They call for further validation and more diverse data before extending the findings to clinical or broader settings.

8 Limitations

  • Severe additive noise reduces performance, and participants with fewer valid utterances provide less evidence for participant-level aggregation.The authors identify noise-robust extraction and uncertainty-aware prediction as future work.Source: §8 · p. 9
  • Because targets are binary PHQ-8 and PHQ-9 cutoff labels, results concern depression-label prediction, not clinical diagnosis.Clinical consideration would require prospective validation using independently adjudicated clinical outcomes.Source: §8 · pp. 9–10
  • Frozen generative models limit the tested setup; robustness across alternative ALMs, PSEs, recording conditions, and prompts remains to be examined.The unknown-filtering procedure also warrants continued evaluation.Source: §8 · p. 10
  • State-association findings are exploratory: utterances share participant labels, and displayed q values do not account for within-participant clustering.Participant-clustered or participant-level reanalysis is needed to confirm the patterns.Source: §8 · p. 10
  • DAIC-WOZ and MODMA were evaluated independently, so results do not establish cross-dataset or cross-lingual transfer; paired comparisons reflect five matched runs or fold means.Larger, more diverse cohorts are needed for stronger analyses of demographic robustness, generalizability, and real-world screening utility.Source: §8 · p. 10

Ethics Statement

The study reused existing speech datasets and protected recordings and participant-level data. It frames ParaCalib as a research framework rather than a diagnostic tool, and reports dataset-specific label and audio preprocessing.

Ethics Statement

  • The study used existing datasets without new recruitment, treated speech and labels as protected data, ran ALM and PSE locally, and conducted internal validation with two graduate researchers.The annotators assessed vocal-evidence consistency, not identity, mental state, or diagnosis.Source: Ethics Statement · p. 10

Intended Use and Clinical Interpretation

  • ParaCalib is a research framework, not a medical device; its PHQ-threshold labels are not clinical diagnoses, and prospective clinical use requires external validation and oversight.The authors also note that the evaluated English- and Mandarin-speaking cohorts do not establish cross-lingual or cross-population generalizability.Source: Intended Use and Clinical Interpretation · p. 10

A.1 Dataset Details and Preprocessing

  • Preprocessing corrected labels for DAIC-WOZ participant 409 and MODMA participant 2010037, and shifted DAIC-WOZ transcript timestamps for four participants.MODMA used PHQ-based binary labels rather than its original clinical grouping; it was distributed with sentence-level segmentation.Source: §A.1 · p. 14

A.2 Training Details, Architectures, and Splits

The supplementary sections specify the shared evaluation setup, model and representation configurations, and additional analyses of SC-Para dimensions and PSE stability. The smaller PSE retains performance above the cited baselines, while prompt paraphrases produce modest evidence-score variation.

A.2 Training Details, Architectures, and Splits

  • All compared classifiers share optimization, batching, device, and early-stopping settings on one NVIDIA RTX 6000 GPU.The MLP, LSTM, Bi-LSTM, Transformer, and attentive MIL classifier use distinct stated architectures and parameter counts.Source: §A.2 · p. 14
  • Paired Macro-F1 tests use five matched runs or fold-level means, characterizing evaluation-unit variation rather than participant-resampling uncertainty.Training, development, and test performance are reported for the evaluated splits.Source: §A.2 · p. 14

A.3 Model and Reproducibility Settings

  • Captioning and state extraction use frozen Qwen3-Omni-30B-A3B-Captioner and DeepSeek-R1-Distill-Qwen-32B models with temperature 0 and top-p 1.0.Representation comparisons include eGeMAPSv02 functionals, SSL hidden states, transcript embeddings, and ALM-caption embeddings.Source: §A.3 · pp. 14–15

B.1 Correlations among Categorical SC-Para Dimensions

  • Five categorical SC-Para dimensions have pairwise Spearman correlations up to .70, indicating related but nonredundant dimensions.Repeated decoding shows high exact agreement across SC-Para dimensions.Source: §B.1 · p. 15, §B.2 · p. 15

B.2 PSE Capacity and Stability

  • Replacing DeepSeek-32B with Qwen3-8B yields 68.2 (6.3)% Macro-F1 on DAIC-WOZ, above HAREN-CTC at 57.9 (2.1)% and Caption Embedding at 66.6 (14.2)%.Meaning-preserving prompt paraphrases produce a mean evidence-score difference of 2.30 points, with correlations from .841 to .952.Source: §B.2 · p. 15

B.3 Methodological Details for Statistical and Controlled Analyses

The controlled analysis varies semantic context while keeping each caption-level acoustic description fixed, and finds that depression-related context raises the PSE evidence score. Additional analyses examine identity information, noise robustness, error patterns, representation dispersion, and caption and state validation.

B.3 Methodological Details for Statistical and Controlled Analyses

  • With caption-level acoustic descriptions fixed, depression-related context raises PSE evidence by 8.6 points over non-depression-related context (participant-clustered 95% CI [6.7, 10.6]).The increase over acoustic-only input is 8.7 points; non-depression-related context differs from acoustic-only input by 0.1 points (p = .92).Source: §B.3 · pp. 15–16
  • Seven of the ten highest-ranked state pairs meet both screening criteria (n ≥ 100 and q < .05), but utterance-level q values do not account for participant clustering.The three lower-support configurations (n = 55, 77, and 92) are shown only for context, and the q values are not participant-level confirmatory significance.Source: §B.3 · p. 16

C.1 Speaker-Identity and Additive Noise

  • On DAIC-WOZ, SC-Para’s speaker-verification ROC-AUC was 52.8–52.9% versus WavLM’s 78.7%; adding MUSAN background noise at 20, 10, and 5 dB SNR reduced Macro-F1 from 71.9% to 68.6%, 63.9%, and 62.6%, respectively.Source: §C.1 · p. 16

C.2 Error Analysis and Computational Cost

  • Consistent false positives average PHQ-8 6.4 versus 2.6 for reliable controls, while false negatives average 66 valid utterances versus 119 for true positives.Unknown categorical states are excluded before participant-level aggregation; discard rates are reported by split and label group.Source: §C.2 · p. 16, Table 13 · p. 17

C.3 Attention-Weighted Dispersion Measure

  • In the DAIC-WOZ test split, depression-label participants have lower attention-weighted dispersion than non-depression-label participants (0.3617 vs. 0.3945).The analysis is exploratory given the small groups (n = 14 and n = 33).Source: §C.3 · p. 17

C.4 Human Validation

  • Two annotators judged generated captions consistent with perceived vocal evidence in 18/20 samples and PSE-derived states consistent in 17/20.The internal validation sampled 20 utterances.Source: §C.4 · p. 17, Table 14 · p. 17

D Prompts and Output Schema

Loading 2610.01103v1…