Source-linked AI summary
ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood
Tiantian Feng, Anfeng Xu, Xuan Shi, Aditya Kommineni, Shakhrul Iman Siam, Megan Micheletti, Zhonghao Shi, Helen Tager-Flusberg, Mi Zhang, Lynn K. Perry, Catherine Lord, Daniel Messinger, Shrikanth Narayanan
TL;DR
ChildVox addresses the limited coverage of ASR-centric child speech research by benchmarking broader acoustic signals across childhood. It integrates diverse datasets and tasks and evaluates multiple foundation-model families, finding strong model performance across child-centered signals while supporting applications such as language-level characterization and speech-production tracking.
Problem
Child speech research has focused mainly on ASR, although children communicate through nonspoken vocalizations and physiological sounds across development.
Method
ChildVox integrates more than 20 sub-tasks across 17 child-centered datasets and evaluates self-supervised, ASR-oriented, and large audio-language models.
Results
ChildVox’s evaluated models consistently outperform frontier proprietary models such as Gemini 3.5 Flash across child-centered acoustic signals.
Takeaways & Limitations
The benchmark supports characterizing children’s language levels and tracking speech production with age.
Takeaways & Limitations
Most datasets and the ASR evaluation are English-focused, while demographic information is often incompletely documented, limiting generalization across languages and populations.
Abstract
from arXiv · showhide
We present ChildVox, a novel benchmark for characterizing the diverse acoustic signals through which children communicate. Specifically, ChildVox follows the full developmental trajectory from birth through school age, covering physiological sounds, non-linguistic vocalizations, canonical syllables, and spoken language. ChildVox integrates more than 20 sub-tasks across 17 child-centered audio and speech datasets, enabling systematic cross-corpus and cross-domain comparison. We evaluate a representative range of audio and speech foundation models, including self-supervised, ASR-oriented, and large audio-language models, on tasks including physiological sound classification, vocalization and canonical syllables modeling, and speech quality assessment and recognition. Benchmark results show that ChildVox provides a suite of high-performance models in recognizing a wide range of acoustic signals from children, supporting downstream applications such as characterizing children's language levels and tracking speech production with age.
1 Introduction
ChildVox addresses the narrow ASR focus of child speech research by benchmarking communication-related acoustic signals across development, from physiological sounds through spoken language. It unifies diverse datasets and tasks to evaluate models for child-centered audio understanding.
- Conventional ASR captures only a subset of children’s communicative skills because children also communicate through nonspoken and physiological sounds.
- ChildVox covers children’s full developmental trajectory from birth through school age, including physiological sounds, vocalizations, canonical syllables, and speech.
- 20+ sub-tasks across 17 child-centered audio and speech datasets are integrated into a consistent benchmark protocol.
- The benchmark evaluates audio, speech, and large audio-language models across child-centered acoustic signals.
- ChildVox evaluates physiological sound, vocalization, canonical syllable, speech quality, pronunciation, articulation, and speaker diarization tasks.
4 Benchmark Models
ChildVox evaluates self-supervised audio and speech encoders, ASR-oriented models, and large audio-language models. The model set spans representation learning for generic audio, nonverbal vocalizations, speech, and broad audio understanding.
- The benchmark includes representative audio, speech, and large audio-language foundation models.
- Self-supervised Models: Three self-supervised models—SSAST, voc2vec, and WavLM—learn representations from generic audio events, non-verbal vocalizations, and speech.
- ASR Models: Whisper-Base, Whisper-Small, and Whisper-Large-v3 encoder embeddings are evaluated to examine model-scale effects, alongside additional Whisper-family models.
- Large Audio-Language Models: Qwen2-Audio-Instruct and Audio Flamingo 3 represent large audio-language models that unify understanding across speech, environmental sounds, and music.
5 Experiments
The experiments standardize audio inputs and evaluation procedures across ChildVox tasks, while using parameter-efficient adaptation for most encoder-based models and LALMs. Metrics are selected by task type.
- Data Preprocessing: All ChildVox audio is resampled to 16 kHz, and most experiments require at least 200 ms of input audio.
- Evaluation: Datasets without predefined partitions use 5-fold cross-validation for experimentation.
- Model Parameters: SSAST is fully fine-tuned, while voc2vec-HuBERT, WavLM-Large, and Whisper-family models use frozen backbones with rank-64 LoRA adaptation.
- Model Parameters: LALM LoRA modules are inserted into language-model attention and projection layers and the audio encoder’s feedforward layers.
- Evaluation: Classification uses utterance-level Macro-F1, diarization uses diarization-error metrics, ASR uses WER, and phoneme recognition uses PER.
6 Benchmark Results
ChildVox benchmark results show substantial variation across physiological sounds, vocalizations, canonical syllables, and speech, with different model families excelling on different tasks. Whisper-Large leads diarization and ASR, while specialized ChildVox models and Qwen2-Audio outperform proprietary zero-shot baselines.
- Physiological Sound Classification: WavLM-Large achieves the highest CirCor performance at 0.643, while SSAST leads ICBHI-crackles, ICBHI-wheezes, and SPRSound.Generic audio representations may capture fine-grained spectral and temporal characteristics in physiological signals better than speech-oriented pretraining.
- Vocalization and Canonical Syllables: SSAST achieves the best performance on AudioSet-Child (0.657), ReCANVo (0.444), and Speech Maturity (0.686).Audio-based pretraining remains beneficial for non-linguistic and early developmental vocal behaviors, while Whisper models are competitive on BabbleCor and donate-a-cry.
- Speech Quality Assessment: Whisper-Large achieves the best speech-quality results on all SpeechOcean762 tasks, while WavLM-Large leads speech emotion recognition.The results indicate benefits from speech-based pretraining for children’s speech classification despite predominantly adult training data.
- Speaker Diarization and Speech Recognition: Whisper-Large records 17.70 DER on NLS and 42.50 DER on ADOS2-Mod3, and achieves 14.80 WER on MyST and 40.20 WER on ADOS.The largest Whisper model leads both diarization and speech-recognition tasks, with Parakeet-TDT ranking second on both ASR benchmarks.
- Overall Findings: Model performance varies substantially across physiological sounds, vocalizations, canonical syllables, and speech, with no single encoder dominating every task.Different encoders capture complementary aspects of child-centered acoustic signals.
- Large Audio Language Models: Qwen2-Audio remains competitive with the strongest encoder models, whereas AudioFlamingo3 performs substantially worse on nearly every ChildVox-Balanced task.AudioFlamingo3 frequently returns free-form descriptions instead of requested labels and may summarize or hallucinate on transcription tasks.
- Comparison with Proprietary Baselines: Specialized ChildVox models outperform Gemini 2.5 Flash and Gemini 3.5 Flash on every compared task, with both Gemini models below Macro-F1 0.35 on CirCor, SPRSound, and ReCANVo.The comparison covers five public datasets to avoid transmitting restricted or license-protected data to third-party APIs.
7 Exemplar Applications with ChildVox
ChildVox models support applications that characterize language development and speech production from children’s audio. Diarization-derived utterance rates increase across expert-assigned language levels, while rhotic-production probabilities correlate positively with age.
- Application Setup: ChildVox models are used in two applications measuring children’s speech production and language development.The applications use benchmark models developed from ChildVox.
- Characterizing the Language Level: Diarization-derived utterance rates increase monotonically from LL-1 pre-verbal communication through LL-2 first words to LL-3 word combinations.LL-1, LL-2, and LL-3 are expert-assigned spoken-language levels in the NLS study.
- Characterizing the Language Level: The LL-3 group shows the most varied utterance-rate distribution, indicating greater variability at the highest language level.The reported measure is utterances per minute derived from speaker diarization.
- Tracking Speech Production with Age: On PERCEPT-R, model-predicted rhotic-production probability has a moderate positive correlation with chronological age (r = 0.576).Younger children show greater variability, while older children’s predicted probabilities center more consistently around 0.75; lower-probability outliers remain.
8 Conclusion and Future Work
ChildVox benchmarks diverse child-produced and child-related acoustic signals across development and supports applications such as language-level characterization and age-related speech-production tracking. The authors also identify limitations involving coverage, annotation reliability, and the evaluated foundation-model set.
- Conclusion: ChildVox characterizes children’s diverse acoustic signals from birth through school age across more than 20 sub-tasks and 17 child-centered datasets.Its scope includes physiological sounds, vocalizations, canonical syllables, and speech.
- Future Work: The benchmark supports characterizing children’s language levels and tracking speech production with age.The authors propose future expansion to phoneme recognition, recording environments, and applications in screening, monitoring, and education.
- Language, Demographics, and Task Coverage: Most speech datasets contain primarily English recordings, while other languages and dialects have limited representation.The ASR evaluation is restricted to English, and demographic factors are often incompletely documented, potentially introducing sampling bias.
- Annotation Variability: Subjective labels and inter-rater disagreement may impose a ceiling on classification scores in several child-centered tasks.The affected tasks include affective vocalization, cry-cause, and canonical-syllable classification.
- Restricted Set of Foundation Models: ChildVox evaluates representative foundation models but omits several recent open-source large audio-language models and broader prompt-engineering strategies.Proprietary comparisons are limited to two Gemini Flash models in a zero-shot setting.
- Ethical Considerations: The released models are intended for research and prohibit clinical, diagnostic, surveillance, privacy-invasive, and commercial uses.The authors plan to release code and checkpoints under the Responsible AI License and emphasize privacy and legal compliance.
A Dataset Details
ChildVox combines diverse child-centered audio and speech datasets spanning physiological sounds, vocalizations, canonical syllables, and speech. The benchmark applies dataset-specific labels, filtering, segmentation, and subject- or speaker-level partitioning procedures.
- Physiological sounds: CirCor contains 5,272 phonocardiogram recordings from 1,568 subjects aged 0–21 years, with expert murmur labels.Labels are present, absent, or unknown.
- Physiological sounds: ICBHI provides 920 respiratory recordings from 126 subjects and 6,898 annotated respiratory cycles for condition, crackle, and wheeze tasks.Recordings and cycles use distinct segmentation rules, with short segments discarded.
- Physiological sounds: SPRSound contains 2,683 records and 9,089 respiratory events from 292 participants, labeled as normal or adventitious sound categories.Record-level labels include normal, CAS, DAS, CAS & DAS, and poor quality.
- Vocalizations: Donate-a-cry includes 465 infant cry samples from birth to age two, with highly imbalanced reasons simplified to hungry versus other.The hungry category contains 382 samples.
- Vocalizations: CryBank retains hunger, loneliness, and discomfort labels, merges nearby cry segments into continuous bouts, and partitions data by subject using 5-fold cross-validation.Segments with pain, other, or unknown labels are discarded.
- Canonical syllables: BabbleCor uses 400 ms clips from 52 children exposed to five languages, while SpeechMaturity provides 242,004 labeled vocalizations across more than 25 languages.Both datasets use five vocalization categories and subject-level cross-validation or equivalent processing.
B.1 Training Parameters
ChildVox fine-tunes encoder-based speech models and large audio-language models under standardized training settings. The pipeline uses parameter-efficient adaptation, dataset-specific prompts, and task-dependent optimization schedules.
- Training schedules: Non-ASR fine-tuning runs for 10 epochs with a maximum audio duration of 10 seconds for most classification tasks.Learning rates vary by model family, and LALMs use batch size 256 with mini-batch size 8.
- Training schedules: ASR fine-tuning uses learning rate 1 × 10−4 and batch size 32, with 5 epochs on MyST and 20 epochs on ADOS.Transcript normalization is applied consistently to references and outputs.
- Encoder-based fine-tuning: Encoder-based models aggregate learned representations across convolutional and transformer layers before temporal averaging and classification.A point-wise convolution and fully connected classifier follow the weighted layer average.
- Parameter-efficient adaptation: LoRA is applied across foundation-model backbones, including ASR fine-tuning, for parameter-efficient adaptation.Large audio-language models use rank-64 LoRA with 4-bit training quantization and 16-bit inference.
- Large audio-language models: Dataset-specific system prompts describe child vocalization classes because generalized LALMs may lack child-specific domain knowledge.Prompts instruct models to classify the vocalization and output only the label.
C.2 BabbleCor
BabbleCor-related prompts operationalize vocalization classification across five categories and pronunciation, prosody, fluency, and accuracy assessments. Each prompt directs the model to return a single label without explanation or punctuation.
- Vocalization classification: BabbleCor vocalization labels distinguish crying, laughing, canonical syllables, non-canonical vocalizations, and junk audio.Canonical syllables have clear consonant-vowel structure, whereas non-canonical sounds lack that structure.
- Pronunciation: Pronunciation assessment labels children’s /r/ production as Rhotic or Derhotic based on correctness, distortion, omission, or substitution.The prompt specifically evaluates the r sound.
- Prosody: Prosody assessment classifies intonation, stress, and rhythm as poor, nearly correct, or correct.The correct category denotes natural, native-like intonation and rhythm throughout.
- Fluency: Fluency assessment distinguishes intermittent delivery, generally fluent speech, and fluent speech using pauses, hesitations, and flow.The fluent category indicates smooth continuous speech without notable disfluencies.
- Pronunciation accuracy: Pronunciation accuracy is labeled Poor or Understandable, Good, or Excellent according to phoneme production and overall accuracy.Excellent denotes highly accurate, native-like pronunciation throughout.
C.7 CirCor
The respiratory and cardiac prompts map child-related physiological audio to categorical diagnostic labels. They separately assess murmurs, respiratory conditions, crackles, wheezes, and broader respiratory sound patterns.
- Cardiac sounds: Cardiac recordings are classified as murmur Absent, Present, or Unknown.Unknown covers audio quality or ambiguity that prevents confident determination.
- Respiratory conditions: Lung recordings are classified by respiratory condition as Healthy, Obstructive, or Infectious from breath sounds.Examples include wheezing for obstruction and crackles or consolidation for infection.
- Respiratory events: Separate binary prompts detect whether crackles are present in lung recordings.Crackles are defined as discontinuous explosive fine or coarse sounds.
- Respiratory events: Separate binary prompts detect whether wheezes are present in lung recordings.Wheezes are defined as continuous, high-pitched whistling sounds.
- Respiratory patterns: Respiratory sound patterns are labeled Normal, CAS, DAS, CAS & DAS, or Poor Quality.CAS denotes continuous adventitious sounds, DAS discontinuous adventitious sounds, and Poor Quality unreliable audio.
C.12 ReCANVo
C.12 ReCANVo evaluates child audio through tasks for communicative intent, infant-cry causes, transcription, and phoneme-level pronunciation. The experiments also compare Gemini APIs on selected public datasets under a fixed prompting setup.
- Communicative intent: Communicative-intent classification assigns child vocalizations labels such as Delighted, Dysregulated, Frustrated, Request, Selftalk, and Social.The output is restricted to the predicted label.
- Infant-cry causes: Infant-cry classification identifies likely causes including Hunger, Loneliness, and Discomfort.Loneliness denotes seeking contact or attention, while Discomfort includes cold, fever, and a full diaper.
- Speech transcription: Speech transcription requires exact spoken content, with unintelligible audio represented as [unintelligible].The transcription output excludes labels, explanations, and formatting.
- Pronunciation transcription: Single-word pronunciation transcription uses standard phoneme symbols in ARPABET or IPA format.The expected output is a phoneme sequence such as /b ae t/.
- Prompt experiments: Gemini 2.5 Flash and Gemini 3.5 Flash are evaluated through APIs on five selected public datasets with temperature 1.0 prompting.Each proprietary-baseline prompt experiment uses a single run because of budget limitations.
- Research process: Generative AI tools were used only for manuscript grammar checking and polishing, while the research work was developed and conducted by the authors.The authors state that the conceptual framing, methodology, and scientific contributions were human-generated.