PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Adult Addressee Classification Across Child- and Adult-Directed Speech Datasets

Sofoklis Kakouros, Daniil Kocharov, Okko Räsänen

arXiv:2610.10240v1eess.AS

TL;DR

Adult addressee classification remains challenging across controlled and naturalistic speech, with results tied to particular corpora and data-quality and domain-shift constraints. This study benchmarks SSL representations and pooling methods across in-lab and long-form datasets and probes the acoustic information in the representations. SSL representations outperform the eGeMAPS baseline, while pooling differences are relatively small and classifier decisions use only part of the acoustic information in the embeddings.

  • Problem

    Adult addressee classification is difficult in naturalistic recordings, and published results remain corpus-specific amid substantial recording variability and cross-corpus domain shifts.

    Source: §I · p. 1

  • Method

    The study benchmarks SSL and eGeMAPS features across in-lab and long-form datasets, comparing pooling strategies and probing representations and classifiers for acoustic information.

    Source: §I · pp. 1–2, B. Pooling methods · p. 2, C. Classification · p. 3

  • Results

    SSL representations outperform the eGeMAPS baseline, while differences between pooling strategies are relatively small.

    Source: A. Classification and pooling · p. 4, §VI · p. 6

  • Takeaways & Limitations

    SSL embeddings retain substantially richer acoustic information than the classifier uses, and the most salient acoustic cues differ by corpus.

    Source: §VI · p. 6

Abstract

from arXiv · show

In this work, we present a comprehensive analysis of classification performance for distinguishing child-directed speech (CDS) from adult-directed speech (ADS) using speech data from corpora containing natural in-lab and in-the-wild CDS and ADS. We establish classification benchmarks for these datasets using self-supervised learning (SSL) representations, along with a range of time-pooled representations that go beyond first- and second-order statistics by incorporating cross-channel covariances in high-dimensional embeddings. In addition, we probe these representations to examine how different pooling methods capture prosodic information using linear probes. Overall, SSL-based representations prove particularly effective, achieving the best performance, while different pooling methods offer complementary advantages for the task.

I. INTRODUCTION

Adult addressee classification is difficult in naturalistic recordings, where degraded audio and subtle cues compound variable conditions and cross-corpus differences. The paper benchmarks SSL and eGeMAPS approaches across in-lab and long-form data, including pooling strategies and probes of the information they capture.

I. INTRODUCTION

  • Naturalistic recordings make adult addressee classification difficult because degraded or incomplete speech, noise, overlap, and subtle acoustic, linguistic, and contextual cues complicate the decision.Recording conditions and relevant acoustic differences also vary across datasets, environments, and speakers, creating within-corpus variability and cross-corpus domain shifts.Source: §I · p. 1
  • Published CDS–ADS results remain largely corpus-specific, with a gap between controlled-data and long-form-recording performance and no standardized benchmark for the task.The ComParE Addressee Challenge ensemble reached 70.2 UAR, and later approaches did not surpass that benchmark.Source: §I · p. 1
  • The study establishes SSL and eGeMAPS baselines using both in-lab and long-form recordings.It evaluates state-of-the-art SSL representations alongside conventional acoustic-prosodic features.Source: §I · pp. 1–2
  • The analysis compares pooling strategies, emphasizing covariance pooling, and probes representations and classifiers to examine the acoustic information involved in CDS–ADS classification.The introduction notes prior use of related covariance approaches for prosodic tasks such as emotion recognition.Source: §I · p. 2

II. MATERIALS

The study uses complementary CDS–ADS datasets: controlled in-lab recordings from ManyBabies and naturalistic daylong recordings from ACLEW.

A. ManyBabies dataset

  • ManyBabies provides 1,281 CDS and ADS utterances, totaling 1.25 hours, from 15 mothers recorded in a laboratory while speaking to infants and experimenters.The full dataset is used in the study.Source: A. ManyBabies dataset · p. 2

B. ACLEW dataset

  • ACLEW contributes 9,909 manually segmented and annotated utterances from English daylong recordings of children aged 2–24 months across four US and UK corpora.The study samples random two-minute excerpts and diarizes speakers by type before annotation as CDS or ADS.Source: B. ACLEW dataset · p. 2

III. METHODS

The study contrasts compact handcrafted acoustic-prosodic features with frame-level representations from pretrained SSL encoders, including models selected to test robustness and capacity.

A. Features

  • The feature set combines 88-dimensional utterance-level eGeMAPS descriptors with frame-level representations from pretrained HuBERT-base, WavLM-base-plus, and WavLM-large encoders.The SSL embeddings have dimensions of 768, 768, and 1024, respectively.Source: A. Features · p. 2
  • WavLM-base-plus is included because its broader pretraining corpus and denoising objective may improve robustness to real-world variability and noise in daylong recordings.This is the stated rationale for its inclusion, not a reported outcome.Source: A. Features · p. 2
  • WavLM-large is used to test whether greater model capacity and representational power translate into more informative embeddings and improved generalization.This describes the experimental rationale rather than a finding.Source: A. Features · p. 2

B. Pooling methods

Pooling turns frame-level SSL embeddings into utterance-level representations using marginal statistics or covariance across dimensions.

B. Pooling methods

  • Mean and standard-deviation pooling summarize frame embeddings over time, with mean-only output in Rd and concatenated mean-plus-standard-deviation output in R2d.The standard deviation is computed from embeddings centered by their temporal mean.Source: B. Pooling methods · p. 2
  • Covariance pooling computes second-order dependencies between centered embedding dimensions across an utterance, retaining temporal covariation beyond marginal statistics.The resulting covariance matrix preserves internal structure in the representation sequence that mean and standard-deviation summaries do not retain.Source: B. Pooling methods · pp. 2–3

C. Classification

The classification pipeline applies a lightweight DNN to frozen openSMILE or self-supervised speech features. A multilayer classifier predicts ADS or CDS, with the checkpoint selected using development F1.

C. Classification

  • A lightweight feed-forward DNN classifies pooled, frozen openSMILE or SSL features without fine-tuning the upstream encoder.The classifier consumes a batch of pooled representations, and its output classes are ADS and CDS.Source: C. Classification · p. 3
  • The classifier projects features to 512 dimensions, then uses GELU, dropout, hidden layers of 256 and 128 units, and a final two-class output layer.Training uses cross-entropy and AdamW, with learning rate 10^-3 and weight decay 10^-4.Source: C. Classification · p. 3
  • The model trains for 20 epochs with batch size 16 and gradient accumulation over four steps; the best development-F1 checkpoint is retained with early stopping.Early stopping occurs when development F1 fails to improve for five epochs; final evaluation uses the test split.Source: C. Classification · p. 3

D. Linear probes for model interpretability

The section probes SSL representations against acoustic-prosodic features and describes the experiments used to evaluate classification and interpretability. Grouped analyses find substantial acoustic information in SSL embeddings, with retention differing across datasets.

D. Linear probes for model interpretability

  • Ridge linear probes predict sentence-level eGeMAPS features from pooled SSL embeddings or ADS/CDS classifier logits, including mean F0 as an example target.The logits are saved for samples within each cross-validation fold before separate per-feature ridge regressions are run.Source: D. Linear probes for model… · p. 3
  • Grouped analysis finds that SSL embeddings encode substantial acoustic information, but retention differs across datasets; Fig. 1 compares WavLM-Large embeddings and classifier logits.The figure reports mean Pearson r for grouped eGeMAPS features on ManyBabies and ACLEW.Source: Figure 1 · p. 3
  • Probe performance uses Pearson correlation and R2, measuring linear association and the proportion of target-feature variance explained, respectively.The reported overview includes Tables III and IV and Figure 1.Source: D. Linear probes for model… · p. 3

IV. EXPERIMENTS

  • Classification experiments compare openSMILE descriptors and SSL features on ManyBabies and ACLEW, reporting unweighted accuracy and F1 with fixed splits and random seeds.SSL pooling includes mean, mean+std, and covariance; openSMILE uses its originally extracted descriptors.Source: §IV · p. 3
  • Covariance probes were excluded: at d = 768, covariance has 294,528 coefficients, 383.5× mean pooling and 191.75× mean+std, risking unstable estimates with limited data.The larger representation also makes the first projection layer substantially more complex.Source: §IV · pp. 3–4

A. Data splits

The data splits separate test material by recording location for ACLEW and by caregiver–infant family for ManyBabies. ACLEW’s location-based design also supports cross-corpus generalization evaluation.

A. Data splits

  • ACLEW uses its English subset across four recording locations, with one location held out for testing while the other three provide training and development data in each split.The location-based splits enable cross-corpus generalization evaluation.Source: A. Data splits · p. 4
  • ManyBabies uses five fixed splits, assigning 80% of BabyID-grouped families to training and development and 20% to testing, while balancing ADS and CDS overall.No BabyID appears in both training/development and test sets.Source: A. Data splits · p. 4
  • Table II reports ACLEW four-fold cross-validation results as mean ± standard deviation, with the best mean in each metric highlighted.Source: Table II · p. 4

V. RESULTS AND DISCUSSION

SSL representations outperform the eGeMAPS baseline, but performance is substantially higher on ManyBabies than on naturalistic ACLEW. Larger models perform best, while the tested pooling strategies yield broadly comparable results despite covariance pooling’s greater cost.

A. Classification and pooling

  • All SSL representations outperform eGeMAPS; best scores are 91.7% accuracy and 93.3% F1 on ManyBabies, versus 64.4% accuracy and 66.0% F1 on ACLEW.The paper attributes the gap as likely due to recording quality: ManyBabies recordings are high-quality, while ACLEW daylong recordings are noisier.Source: A. Classification and pooling · p. 4
  • WavLM-Large achieves the best overall performance, and WavLM Base+ improves performance over the base model despite unchanged capacity.Differences between HuBERT and WavLM base models are less pronounced.Source: A. Classification and pooling · p. 4
  • Mean, mean+std, and covariance pooling produce broadly comparable classification results, with covariance adding no corresponding performance improvement.Covariance pooling has O(d^2) dimensionality, compared with linear dimensionality for mean and mean+std pooling.Source: A. Classification and pooling · pp. 4–5

B. Linear Probing

SSL embeddings preserve stronger correlations with eGeMAPS features than the classifier logits do, with patterns and feature salience differing across ManyBabies and ACLEW. Grouped feature analysis likewise shows substantial acoustic information in embeddings, but varying retention across datasets.

B. Linear Probing

  • Across both corpora, correlations with SSL embeddings are consistently much stronger than correlations with classifier logits.The comparison identifies eGeMAPS features retained in embeddings, attenuated at the decision level, and patterns shared across datasets.Source: B. Linear Probing · p. 5
  • ManyBabies logit correlations are approximately 0.29–0.54 and embedding correlations 0.67–0.84; ACLEW ranges are about 0.21–0.46 and 0.55–0.84, respectively.In ManyBabies, mfcc2_mean is the highest-ranked logit feature in all four settings; ACLEW’s embedding–logit gap is more pronounced.Source: B. Linear Probing · p. 5
  • ACLEW’s strongest predictors include voice-quality and spectral-amplitude measures such as hnr_mean, formant amplitudes, shimdb_stdn, and fluxv_stdn.Their stronger presence may reflect both speech differences and naturalistic recording conditions, including noise, talker distance, reverberation, and device characteristics.Source: B. Linear Probing · pp. 5–6
  • Grouping the 88 openSMILE features by acoustically meaningful categories shows substantial acoustic information in SSL embeddings, with retention differing across datasets.The categories use a rule-based mapping derived from feature names.Source: B. Linear Probing · p. 6

VI. CONCLUSIONS

The study compares CDS–ADS classification across controlled in-lab and naturalistic long-form recordings, and examines what SSL representations encode and what classifiers use. SSL representations outperform the eGeMAPS baseline, while probing reveals richer acoustic information than the classifier’s final decisions reflect.

VI. CONCLUSIONS

  • The analysis compares controlled in-lab ManyBabies recordings with naturalistic long-form ACLEW audio and examines SSL representations against OpenSMILE acoustic features.It considers both acoustic information encoded in embeddings and the extent to which classifiers use that information for CDS–ADS classification.Source: §VI · p. 6
  • SSL representations outperform the eGeMAPS baseline, with strongest performance from larger and denoising-enhanced models; pooling-strategy differences are relatively small.The conclusion reports these overall patterns without giving specific metric values.Source: §VI · p. 6
  • Linear probing indicates SSL embeddings retain substantially richer acoustic information than the task-specific classifier reflects, with salient cues differing by corpus.The corpus-specific differences may reflect both speech style and recording conditions.Source: §VI · p. 6

AI-Generated Content Disclosure

AI tools assisted with table formatting but not with experiment design, execution, metric computation, or interpretation.

AI-Generated Content Disclosure

  • OpenAI Codex with GPT-5.5 assisted with table formatting, but AI tools were not used to design or run experiments, compute metrics, or interpret results.Source: AI-Generated Content Disclosure · p. 6
Loading 2610.10240v1…