PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaker-Aware Temporal Aggregation Strategies on Segment Representations for Depression Detection in Dyadic Interaction: A Benchmark Study

Anisha Pattanayak, Huang-Cheng Chou, Shrikanth Narayanan, Sudarsana Reddy Kadiri

arXiv:2607.02904v1eess.AS

TL;DR

Temporal aggregation in clinical speech depression detection is difficult to evaluate independently of the backbone, layer, and corpus. DEPOOL benchmarks six aggregation architectures with six frozen backbones across English and Mandarin corpora using learned layer selection, finding that 24 of 72 single-seed configurations collapse to single-class predictions and that apparent architectural stability can fail across seeds.

  • Problem

    It remains unclear whether temporal aggregation rankings reflect aggregation architectures or the particular backbone, layer, and corpus pairing, limiting controlled comparison.

  • Method

    DEPOOL cross-products six aggregation architectures, six frozen SSL backbones, and two corpora while learning layer weights instead of selecting one hand-picked layer.

  • Results

    24 of 72 configurations (33%) collapse to single-class prediction in single-seed runs, while the architecture with no collapses in that grid loses stability across seeds.

  • Takeaways & Limitations

    Temporal aggregation benchmarks should assess robustness to backbone and random seed rather than relying on average accuracy from a single pipeline.

  • Takeaways & Limitations

    The 72-cell grid uses a single fixed seed, and the replication shows that F1 standard deviation reaches 0.42 while the apparent Bi-GRU stability does not hold across seeds.

Abstract

from arXiv · show

Speech-based depression detection compresses features from short audio segments into one speaker-level decision, a step called temporal aggregation rarely studied on its own. Most benchmarks fix a single self-supervised encoder and a single hand-picked layer, so a reported gain may reflect the pipeline rather than the aggregation method itself. We introduce DEPOOL, a controlled benchmark that compares six aggregation architectures with six frozen speech backbones on an English and a Mandarin depression corpus, where each configuration learns which backbone layers matter rather than fixing one by hand. Across the resulting 72-configuration grid, a third of configurations collapse into predicting a single class for every speaker, a failure tied to the backbone as much as to the method, and the architecture that is most stable in a single-seed run becomes unreliable when training repeats across seeds. Robustness to backbone and seed, rather than average accuracy across a single pipeline, should be a first-class benchmarking criterion for temporal aggregation in clinical speech.

I. INTRODUCTION · II. RELATED WORK

DEPOOL frames depression detection as a controlled study of temporal aggregation, varying six architectures, six SSL backbones, two corpora, and learned combinations of all hidden layers. The benchmark shows that aggregation results can be backbone- and seed-sensitive, with roughly a third of configurations collapsing into single-class predictions.

  • I. INTRODUCTION: Speech depression detection aggregates fixed-length segment features into one speaker-level prediction, but interview cues such as psychomotor retardation build across time.This makes temporal aggregation a central design problem rather than a simple pooling step.
  • I. INTRODUCTION: Most aggregation comparisons use one SSL backbone and one manually chosen transformer layer, confounding architecture performance with pipeline selection.The chosen layer often reflects the authors’ own probing results.
  • I. INTRODUCTION: DEPOOL tests whether aggregation rankings persist when the SSL backbone, transformer layer, and corpus vary, rather than reflecting one convenient pipeline.The benchmark directly targets the validity of single-pipeline comparisons.
  • I. INTRODUCTION: 72 configurations combine six aggregation architectures, six SSL backbones, and two corpora, E-DAIC and MODMA, in a controlled cross-product benchmark.Every configuration learns a softmax-weighted combination of all hidden layers instead of using one fixed layer.
  • I. INTRODUCTION: Evaluation uses strictly speaker-independent, stratified 60/20/20 splits, and the pipeline, splits, and results are released openly.Participant-grouped folds prevent test-speaker segments from appearing in training.
  • I. INTRODUCTION: Roughly a third of the 72 configurations collapse into single-class prediction, a failure invisible in one-backbone studies.The collapse is reported as a benchmark-level result across the configuration grid.
  • I. INTRODUCTION: The single-class collapse depends on both backbone and random seed, making robustness an important criterion alongside average performance.The contribution explicitly reports seed sensitivity as well as backbone dependence.

A. Depression detection from speech … A. Datasets

Speech depression detection has progressed from handcrafted acoustic features to self-supervised representations, while segment aggregation remains comparatively understudied. DEPOOL evaluates aggregation across varied speech backbones and datasets, including English E-DAIC and Mandarin MODMA corpora with distinct protocols and labels.

  • A. Depression detection from speech: Early systems combined handcrafted acoustic features such as MFCCs, pitch, jitter, and shimmer with support vector machines or logistic regression.
  • A. Depression detection from speech: Bidirectional LSTM networks improved on static summaries by modeling temporal dynamics, while SSL models learn richer speech representations.
  • B. Segment aggregation in clinical speech: Segment aggregation has received comparatively little attention and is commonly implemented through majority voting, mean pooling, or a single recurrent sequence model.
  • B. Segment aggregation in clinical speech: Attentive statistics pooling, local attention, NetVLAD, and CLS-token transformer pooling provide candidate aggregation mechanisms, but cross-backbone depression screening evaluations remain limited.
  • A. Datasets: Two depression corpora differ in language, elicitation protocol, and population, enabling findings that hold across both to be less dataset-specific.
  • A. Datasets: 122 unique E-DAIC participants contribute speaker-independent training, validation, and test partitions, with depression labeled by PHQ-8 ≥10.
  • A. Datasets: 52 MODMA participants comprise Mandarin clinically diagnosed MDD patients and matched healthy controls across training, validation, and held-out test subsets.

B. Segmentation and Preprocessing · C. SSL Backbones · D. Semi-Fine-Tuned Layer Aggregation

The pipeline isolates and standardizes participant speech into ordered, overlapping 3-second clips, then extracts full-layer representations from six frozen SSL backbones. Each configuration learns normalized layer weights and projects the resulting clip features to a shared 256-dimensional space, controlling downstream comparisons.

  • B. Segmentation and Preprocessing: Participant-only speech is resampled to 16 kHz mono and segmented with 3-second windows and 1.5-second strides, producing 50% overlap.Interviewer and virtual-agent turns are excluded.
  • B. Segmentation and Preprocessing: Each interview becomes an ordered sequence of 3-second clips that preserves coarse session chronology while providing short sequence-model tokens.
  • C. SSL Backbones: DEPOOL evaluates six frozen SSL backbones, including WavLM variants, HuBERT, Wav2Vec2-Robust, Data2Vec-Audio, and XLS-R.The backbones are large Transformer speech encoders pretrained with self-supervised objectives.
  • C. SSL Backbones: For every clip, the method extracts all hidden states and mean-pools each layer, yielding a [L, D] representation.L denotes the number of hidden-state layers and D the backbone hidden size.
  • D. Semi-Fine-Tuned Layer Aggregation: Rather than selecting one layer manually, each configuration learns a soft weighted average across all L backbone layers.This follows the SUPERB-style featurizer convention and avoids hand-picking bias.
  • D. Semi-Fine-Tuned Layer Aggregation: Layer weights are learnable scalars shared across clips and interviews, with softmax normalization enforcing nonnegative weights summing to one.The normalized weight is softmax(wℓ) = exp(wℓ)/ PL ℓ′=1 exp(wℓ′).
  • D. Semi-Fine-Tuned Layer Aggregation: Only per-backbone layer weights and a shared projection are trained, producing a 256-dimensional clip embedding for every downstream aggregation architecture.This controls comparisons by removing incidental differences in backbone dimensionality.

E. Aggregation Architectures

The benchmark compares aggregation heads that compress masked, ordered clip embeddings into one speaker representation before depressed-versus-healthy classification. The architectures range from fixed summary statistics and attention weighting to sequence modeling and cluster-residual encoding.

  • Input and output: Each head aggregates valid, non-padded clip embeddings from one interview into a single speaker vector for binary depression classification.A validity mask excludes padded clips during aggregation.
  • Fixed pooling: Mean pooling averages valid clip embeddings, while statistical pooling concatenates their element-wise mean, standard deviation, maximum, and minimum.Statistical pooling produces a 1,024-dimensional representation from four 256-dimensional summaries.
  • Attention pooling: Self-attention pooling learns normalized per-clip weights, masking padded clips before forming an attention-weighted 256-dimensional representation.The attention mechanism projects clips into a 128-dimensional space and uses a learnable context vector for scoring.
  • Sequence modeling: The Transformer encoder projects clips to 64 dimensions, processes them with a single-layer four-head encoder and CLS token, and uses the CLS output for pooling.Projection and Transformer stages use ReLU and dropout 0.5, with padding excluded by a key-padding mask.
  • Sequence and residual encoders: The remaining heads include a two-layer bidirectional GRU with attention and a two-cluster residual encoding that independently normalizes cluster residuals before concatenation.The residual representation applies soft assignment weights, intra-normalization, and whole-vector L2 normalization.
  • Classification head: Every architecture feeds its pooled representation to an MLP with dropout, one ReLU hidden layer, and a two-way softmax classifier.The classifier outputs the depressed-versus-healthy decision.

F. Speaker-Independent Splitting

Because speech representations contain speaker-identity information, splitting clips from the same participant across subsets could inflate depression-detection performance. The study therefore uses participant-grouped StratifiedGroupKFold splitting to prevent speaker leakage.

  • F. Speaker-Independent Splitting: StratifiedGroupKFold groups all clips by participant ID, preventing the same speaker from appearing in training, validation, and test subsets.This avoids classifiers learning individual voices instead of depression markers, which could silently inflate reported performance.

G. Training and Evaluation

The 72 configurations use a fixed training protocol and are evaluated with classification, discrimination, clinical utility, and class-specific recall metrics. Collapsed runs are explicitly flagged when predictions reduce to a single label for every test speaker.

  • Training protocol: All 72 configurations are trained independently with AdamW for 40 epochs using learning rate 10−3, weight decay 10−4, class-weighted cross-entropy, and a fixed random seed.The best-F1 epoch is tracked for reporting.
  • Evaluation metrics: Reported metrics include accuracy, F1, ROC-AUC, sensitivity for the depressed/MDD class, specificity, and a clinical utility score.The utility score ranges from 0 to 1 and weights sensitivity twice as heavily as specificity.
  • Collapse criterion: A run is flagged as collapsed when (sensitivity, specificity) equals (1, 0) or (0, 1), indicating the model emits the same label for every test speaker.Sensitivity and specificity are computed from test-set speaker predictions, with depressed/MDD as the positive class.

IV. EXPERIMENTAL RESULTS · A. Aggregated Performance by Architecture and Backbone

Across six frozen backbones and six aggregation architectures, performance depended strongly on corpus and backbone. WavLM-Base-Plus was strongest and most stable on E-DAIC, HuBERT-Large led MODMA, and Wav2Vec2-Robust was weakest and least stable on both.

  • IV. EXPERIMENTAL RESULTS: The evaluation reports aggregated performance by architecture and backbone, the best single configuration per corpus, collapse, and seed sensitivity on a focal subset.
  • A. Aggregated Performance by Architecture and Backbone: On E-DAIC, no architecture wins outright across all metrics.Mean Pooling has the highest accuracy and F1, while Statistical Pooling and NetVLAD achieve higher sensitivity and utility.
  • A. Aggregated Performance by Architecture and Backbone: NetVLAD’s specificity is 0.245 on E-DAIC, indicating substantial over-prediction of the depressed class.
  • A. Aggregated Performance by Architecture and Backbone: On MODMA, Bi-GRU with Attention leads on every metric when averaged across all six backbones.
  • A. Aggregated Performance by Architecture and Backbone: WavLM-Base-Plus is the strongest and most stable E-DAIC backbone, with mean F1 0.601 and 0/6 collapsed configurations.
  • A. Aggregated Performance by Architecture and Backbone: On MODMA, HuBERT-Large edges out WavLM-Base-Plus, with mean F1 0.806 versus 0.759.
  • A. Aggregated Performance by Architecture and Backbone: Wav2Vec2-Robust is the weakest and least stable backbone on both corpora, collapsing on 5 of its 6 architecture pairings per corpus.This is identified as the clearest single signal that backbone choice can dominate architecture choice.
  • A. Aggregated Performance by Architecture and Backbone: Under single-seed training, Bi-GRU with Attention never collapses across the 12 runs, but this stability does not survive seed replication.

B. Best Single Configurations · C. The Collapse Phenomenon · D. Seed Sensitivity: A Focal Subset

The strongest single configurations differ by corpus, while collapse affects one-third of the 72-run grid and depends on both backbone and architecture. Replication across seeds overturns the apparent single-seed stability of Bi-GRU with Attention.

  • B. Best Single Configurations: On E-DAIC, Self-Attention pooling with WavLM-Base-Plus is best by F1, with accuracy 0.870 and F1 0.667.Its sensitivity is 0.500 and specificity 1.000.
  • B. Best Single Configurations: On MODMA, Bi-GRU with Attention on Data2Vec-Audio-Large leads, reaching accuracy 0.900 and F1 0.909.Sensitivity is 1.000 and specificity is 0.800; the architecture is flagged as most consistent in Table I.
  • C. The Collapse Phenomenon: 24 of 72 configurations (33%) collapse to one predicted class for every test speaker in the single-seed grid.Collapse rates vary by architecture: Transformer encoder 9/12 (75%), NetVLAD 5/12 (42%), Mean Pooling and Self-Attention 3/12 each (25%), and Statistical Pooling 4/12 (33%).
  • C. The Collapse Phenomenon: Bi-GRU with Attention is the only architecture without collapse in the single-seed runs, at 0/12.The passage notes that this apparent stability is seed-dependent.
  • C. The Collapse Phenomenon: Backbone identity is at least as strongly tied to collapse as architecture: Wav2Vec2-Robust collapses on 10/12 runs (83%), versus WavLM-Base-Plus on 1/12 (8%).Data2Vec-Audio-Large collapses on 5/12 runs (42%).
  • C. The Collapse Phenomenon: Neither architecture nor backbone alone predicts outcome; their combination determines whether a configuration works.Bi-GRU with Attention remains comparatively high and low-variance across backbones, whereas the Transformer encoder ranges from competitive scores to collapse.
  • D. Seed Sensitivity: A Focal Subset: Across three seeds and both corpora, Bi-GRU stability does not replicate: MODMA with WavLM-Base-Plus averages F1 0.31±0.42 at sensitivity 0.25.Some seeds collapse toward the healthy class, F1 standard deviations reach 0.42, and no configuration sustains a balanced trade-off when seeds vary.

V. DISCUSSION AND ANALYSES · A. Robustness vs. average performance · B. Why do some architectures collapse more?

Mean F1 can conceal that a third of runs collapse into single-class predictions, while replication shows that apparent single-seed reliability is fragile. Collapse is plausibly linked to under-trained routing mechanisms in data-limited settings, but diagnosing it directly remains future work.

  • A. Robustness vs. average performance: A third of underlying runs never learn to discriminate, so mean F1 alone can hide substantial architectural instability.Architectures with similar means may differ between consistently moderate performance and alternating strong runs with total collapse.
  • A. Robustness vs. average performance: Under single-seed training, only Bi-GRU with Attention never fails outright, but replication shows that this reliability is fragile.The benchmark therefore does not support recommending any head solely from single-seed behavior.
  • B. Why do some architectures collapse more?: After merging train and validation, E-DAIC has 99 training participants and MODMA has 42.These few dozen training sequences constrain learning for architectures that must learn routing mechanisms above frozen backbones.
  • B. Why do some architectures collapse more?: The Transformer encoder and NetVLAD must learn CLS-token self-attention or soft cluster assignment essentially from scratch on top of frozen backbones.Their routing mechanisms are learned with only a few dozen training sequences.
  • B. Why do some architectures collapse more?: Under class-weighted cross-entropy, an under-trained routing function can minimize loss fastest by degenerating to a fixed-label rule, matching the observed collapse.The proposed mechanism explains why some architectures collapse more, but does not establish a direct training-time diagnostic.
  • B. Why do some architectures collapse more?: Seed replication evaluated the most stable Bi-GRU/WavLM-Base-Plus and least stable Transformer/Wav2Vec2-Robust cells over 3 seeds, with a single-seed final row.F1, sensitivity, and specificity were defined as in Section III-G; the single-seed MD-W2-T row has no standard deviation.
  • B. Why do some architectures collapse more?: The widest error bars occurred on MODMA with Bi-GRU, showing that single-seed stability does not imply multi-seed stability.Error bars represent one standard deviation over 3 seeds, except for the single-seed MD-W2-T result.

C. Backbone choice is not a side detail … VII. CONCLUSIONS

DEPOOL shows that temporal aggregation cannot be evaluated independently of backbone choice or random seed: backbone effects can dominate architecture differences, while single-seed stability may not replicate. Results also differ substantially across corpora, and the benchmark’s labels, demographic imbalances, and evaluation design limit clinical and fairness claims.

  • C. Backbone choice is not a side detail: 83% of Wav2Vec2-Robust configurations collapse, exceeding the gap between any two aggregation architectures.The passage attributes this pattern to differences in pretraining emphasis, while cautioning that the backbone comparison is not fully controlled.
  • D. MODMA vs. E-DAIC: 0.909 vs. 0.667 is the best achievable F1 on MODMA versus E-DAIC, respectively.The passage suggests MODMA may be easier because of clinician-based diagnoses and a constrained protocol, but its test set has only 10 speakers.
  • C. Backbone choice is not a side detail: Backbone effects appear more robust than architecture rankings in the replication across random seeds.F1 standard deviation reaches 0.42, and the Bi-GRU stability result does not hold across seeds.
  • VI. LIMITATIONS AND ETHICAL CONSIDERATIONS: The 72-cell grid uses a single fixed seed, preventing per-cell metrics from separating architectural effects from random noise.Checkpoint selection also tracks best-F1 on the held-out test partition rather than a validation criterion because of computational costs.
  • VI. LIMITATIONS AND ETHICAL CONSIDERATIONS: The corpora’s labels and demographic imbalances preclude claims of clinical deployment or fairness.The passage calls for subgroup evaluation, prospective clinical validation, informed consent, and human oversight in real-world applications.
  • VII. CONCLUSIONS: 72 controlled configurations combine six temporal aggregation architectures, six frozen SSL speech backbones, and two depression corpora.DEPOOL learns layer selection with a softmax rather than fixing one hand-picked layer.
  • VII. CONCLUSIONS: A third of all 72 single-seed configurations collapse into trivial single-class prediction, concentrated in specific architectures and backbones.The conclusion identifies backbone and random seed as dependencies that prevent aggregation architecture from being evaluated in isolation.

APPENDIX

The appendix documents six diverse frozen SSL speech backbones used in DEPOOL, spanning different pretraining objectives, model sizes, and training-data scales. It also defines the checkpoint, layer, dimensionality, parameter-count, and fine-tuning terminology used to describe them.

  • Backbone diversity: Six SSL speech backbones represent contrastive learning, masked prediction, self-distillation, and varied parameter counts and training-data sizes.The evaluated models range from WavLM-Base-Plus at 94M parameters to XLS-R-1B at 965M parameters.
  • Backbone diversity: The backbone set spans self-supervised-only models and models further fine-tuned for automatic speech recognition.SSL denotes self-supervised only, whereas SSL+FT denotes self-supervised then ASR fine-tuned.
  • Backbone specifications: All released checkpoints are frozen during training, while each backbone exposes Transformer hidden-state outputs—including the feature-extractor output—to the featurizer.The available layer count includes the feature-extractor output, and the checkpoints’ approximate total parameter counts are not updated during training.
Loading 2607.02904v1…