PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions

Anfeng Xu, Tiantian Feng, Somer Bishop, Catherine Lord, Shrikanth Narayanan

arXiv:2601.17640v2eess.AScs.SD

TL;DR

Child–adult interaction analysis needs scalable transcription with accurate speaker roles and timestamps, but existing cascaded systems can propagate errors. This paper jointly models ASR and role diarization in Whisper using serialized outputs, diarization-guided decoding, and structural constraints, reporting consistent gains over cascaded baselines and practical speaker-attributed transcripts.

  • Problem

    Child–adult interaction analysis requires accurate transcripts, speaker roles, and utterance timing, while speech variability and cascaded pipelines make scalable automation challenging.

  • Method

    The framework extends Whisper with serialized speaker-and-timestamp outputs, a frame-level diarization head, silence suppression, and state-machine forced decoding.

  • Results

    Experiments on two child–adult datasets show consistent improvements in mtWER, WER, AER, and DER over two cascaded baselines, with strong diarization accuracy.

  • Takeaways & Limitations

    The framework provides scalable generation of structured, speaker-attributed transcripts with temporal boundaries for developmental and clinical child–adult interaction analysis.

Abstract

from arXiv · show

Accurate transcription and speaker role diarization of child-adult spoken interactions are crucial for developmental and clinical research. However, manual annotation is time-consuming and challenging to scale. Existing automated systems typically rely on cascaded speaker diarization and automatic speech recognition pipelines, which can lead to error propagation. This paper presents a unified end-to-end framework that extends the Whisper encoder-decoder architecture to jointly model ASR and child-adult speaker role diarization. The proposed approach integrates: (i) a serialized output training scheme that emits speaker tags and start/end timestamps, (ii) a lightweight frame-level diarization head that enhances speaker-discriminative encoder representations, (iii) diarization-guided silence suppression for improved temporal precision, and (iv) a state-machine-based forced decoding procedure that guarantees structurally valid outputs. Comprehensive evaluations on two datasets demonstrate consistent and substantial improvements over two cascaded baselines, achieving lower multi-talker word error rates and demonstrating competitive diarization accuracy across both Whisper-small and Whisper-large models. These findings highlight the effectiveness and practical utility of the proposed joint modeling framework for generating reliable, speaker-attributed transcripts of child-adult interactions at scale. The code and model weights are publicly available

I. INTRODUCTION

Child–adult interactions require reliable transcripts with speaker roles and timing, but acoustic and linguistic variability and cascaded-pipeline errors make automated analysis challenging. The paper proposes unified modeling with diarization-guided representations, structured decoding, and silence-aware timestamping.

  • Child–adult interactions support developmental, clinical, and behavioral analyses requiring speaker-attributed transcripts and conversational timing measures.These measures include words per minute, utterance duration, and conversational latency.
  • Child speech differs acoustically and linguistically from adult speech, while naturalistic recordings add ambient noise and expressive variability.
  • Cascaded diarization–ASR pipelines can propagate segmentation errors, whereas ASR-first systems can degrade forced alignment.
  • The proposed framework jointly predicts lexical content, speaker roles, and utterance timestamps using serialized output training and a frame-level diarization head.
  • Diarization-guided silence suppression reduces timestamp drift and improves boundary precision during decoding.
  • A forced-decoding state machine enforces valid speaker and timestamp token ordering, preventing missing special-token errors.
  • Experiments on two child–adult datasets report consistent improvements in mtWER, WER, AER, and DER over two cascaded baselines.

B. Multi-Speaker ASR and Serialized Output Training

Existing speaker-attributed ASR systems commonly separate diarization and transcription, while serialized-output approaches unify multi-speaker decoding. The reviewed baselines use WhisperX, diarization-first segmentation, or frame-level role classification within modular pipelines.

  • Traditional speaker-attributed ASR systems use separate diarization, separation, and ASR modules, creating modular processing stages.
  • Serialized Output Training represents multi-speaker interactions as one sequence containing lexical tokens and explicit speaker or speaker-change markers.
  • A. Zero-Shot: Using WhisperX: WhisperX combines voice activity detection, Whisper transcription, and wav2vec 2.0 forced alignment to produce word-level timestamps.
  • B. Diarization-First: Speaker Role Diarization →ASR: The diarization-first baseline converts frame-level activities into speaker segments, merges nearby regions, discards very short segments, and decodes each segment independently.
  • B. Diarization-First: Speaker Role Diarization →ASR: Frame-level role diarization assigns silence/noise, child, or adult labels to audio frames using Whisper encoder representations.

C. ASR-First: SOT-ASR →Forced Alignment

The ASR-first baseline fine-tunes Whisper with serialized speaker-role output but omits timestamp tokens, recovering utterance boundaries afterward through forced alignment. The proposed architecture instead shares encoder representations between decoding and diarization.

  • The ASR-first baseline fine-tunes Whisper with Serialized Output Training to produce unified multi-speaker transcripts containing child and adult speaker tokens.
  • This baseline predicts lexical content and speaker roles without timestamp tokens and processes audio chunks up to 30 seconds.
  • Forced alignment removes speaker tokens, estimates word boundaries, and uses the earliest and latest aligned words as utterance timestamps.
  • The proposed architecture uses the encoder’s last hidden layer for both the decoder and diarization head.
  • 1) Serialized Output Training Setup:: Serialized output training orders each segment as start timestamp, speaker role, transcript, and end timestamp.

2) Diarization Head.:

The lightweight diarization head adds frame-level speaker-role and silence supervision to the Whisper encoder, while serialized ASR training learns lexical and structural tokens jointly.

  • The diarization head uses 1D convolutions to predict child, adult, and silence probabilities for each encoder frame.
  • The model combines serialized-token ASR cross-entropy with frame-level diarization cross-entropy in a weighted multi-task objective.The ASR loss covers text, timestamp, and speaker tokens, while the diarization loss supervises frame-level class predictions.
  • Diarization supervision explicitly aligns encoder features with speaker roles and speaking activity, strengthening representations for decoding and diarization.

C. Diarization-guided Silence Suppression

Inference uses diarization-guided silence suppression and finite-state forced decoding to improve timestamp placement and enforce valid serialized speaker-text outputs.

  • Silence spans with predicted silence probability above 0.7 are trimmed by 0.2 seconds at each boundary before timestamp suppression.The suppression discourages redundant timestamp tokens in silent regions and improves utterance segmentation reliability.
  • A finite-state decoding graph masks invalid token transitions during beam search, enforcing the expected order of speaker, timestamp, and text tokens.
  • Forced decoding guarantees that every speaker segment contains both start and end timestamps, producing syntactically valid serialized output.Combined with silence suppression, it is intended to improve temporal and structural coherence without sacrificing speech recognition accuracy.

E. Diarization Head Initialization and Fine-tuning

The diarization head is pretrained before joint optimization and then fine-tuned to improve silence prediction, using child-adult interaction data from Playlogue and ADOS.

  • Diarization-head pretraining compares final-layer representations with a learnable weighted average of all Whisper encoder layers.Joint training subsequently uses only the final encoder layer, aligning initialization with the joint model.
  • After joint training, the speaker head is fine-tuned separately while other parameters remain frozen to improve silence prediction.Silence accuracy is important for the model’s timestamp suppression procedure.
  • Experiments use Playlogue and ADOS, both containing child-adult interactions, with ADOS annotations refined by utterance-level forced alignment.The datasets differ in participant age, clinical conditions, and recording setup.
  • Playlogue contains over 33 hours of naturalistic interactions involving typically developing preschool children aged 3–5 years.
  • Segments are constructed from pause midpoints and capped at Whisper’s 30-second maximum, with average durations of 25.0 seconds for Playlogue and 23.2 seconds for ADOS.The split design enforces speaker disjointness between ADOS training/development data and testing data.

B. Metrics

The paper evaluates joint speaker-attributed ASR using metrics that separate transcription, role-attribution, and diarization errors.

  • mtWER jointly measures insertion, deletion, substitution, and speaker-role-attribution errors for each speaker role.It is designed for multi-speaker conditions where both transcription accuracy and role consistency matter.
  • WER excludes speaker-role-attribution errors, while AER separately measures attribution performance for child and adult roles.WER and AER are averaged across the two speaker roles.
  • DER accounts for missed detection, false alarm, and speaker confusion errors in role diarization.The reported DER uses reference speech duration rather than total audio duration and is computed with Pyannote metrics.
  • Final results are reported as average scores across all files.

1) Proposed Method:

The evaluation compares training configurations and cascaded baselines, showing that the proposed joint approach improves multi-talker transcription while avoiding pipeline error propagation.

  • Proposed Method: Joint training uses 20 epochs, batch size 16, Adam optimization, and learning rates of 5e−6 for Whisper-small and 1e−6 for Whisper-large.All encoder, decoder, and diarization-head parameters are unfrozen, and checkpoints are selected by validation loss.
  • Baseline Results: The cascaded baselines expose complementary weaknesses: diarization-first segmentation errors impair ASR, whereas ASR-first systems generally lower WER but increase AER and DER.ASR inaccuracies also make forced-alignment timestamp predictions less reliable.
  • Baseline Results: Oracle speaker-role segments substantially improve ADOS diarization-first WER, but inferred and oracle results are comparable on Playlogue.The comparison highlights dataset-dependent effects of annotation and segmentation quality.
  • Proposed Method: 9.7 ∼19.6% relative mtWER reductions are achieved by the joint approach across Whisper-small and Whisper-large compared with cascaded baselines.The paper attributes the improvement to joint optimization and timestamp prediction, which produce more temporally aligned encoder representations.

C. Proposed Method – DER

The joint model improves speaker-attributed transcription and performs strongly on diarization, although DER depends on dataset conditions and remains higher than diarization-first on Playlogue.

  • DER: The joint model consistently outperforms ASR-first in DER by avoiding forced alignment, where transcription errors propagate into boundary predictions.
  • DER: On Playlogue, the joint model has higher DER than the diarization-first baseline for both Whisper-small and Whisper-large.The paper relates this result to challenging recording conditions.
  • DER: The shared encoder enables the joint framework to match a specialized diarization system under difficult conditions.
  • mtWER: The joint model improves child and adult mtWER by approximately 4% to 6% absolute over the best baseline across models and datasets.
  • Child–Adult Performance: Children show higher mtWER than adults, and high ADOS CSS is associated with higher child mtWER across both age groups.Within each age group, mtWER remains comparable at the same severity level.
  • Overall Results: Overall, the joint model improves mtWER over both cascaded baselines and substantially outperforms ASR-first in diarization while approaching diarization-first depending on dataset conditions.

A. k-NN Classification Accuracy

Representation analyses examine child–adult separability, boundary alignment, and overlap robustness across timestamp and diarization-head configurations.

  • k-NN Classification Accuracy: k-NN classification probes whether encoder outputs locally separate child and adult speaker roles without introducing an additional learned classifier.
  • k-NN Classification Accuracy: Timestamp supervision slightly reduces k-NN accuracy, while adding the diarization head substantially restores speaker-discriminative information across both datasets.The evaluation uses k = 5 and cosine distance.
  • Boundary Alignment: Diarization-head supervision improves all boundary-probing metrics across both datasets, with especially notable gains for child and adult F1 scores.Timestamp supervision alone provides only small silence-F1 and temporal-alignment improvements.
  • Overlapped Speech: With 1.0s overlap, Whisper-small mtWER rises from 33.8% to 47.2% and DER from 22.2% to 42.6%.Deletion errors increase from 12.1% to 28.2% under the same overlap condition.
  • Overlapped Speech: Overlap mainly causes missed speech segments, and the study identifies limited reliable overlap annotations as a boundary for the current evaluation and training setup.Future work is proposed on overlap-aware dataset construction and training strategies.

VIII. ABLATION

The ablations show that diarization-head pretraining and full forced decoding are important for improving diarization and ensuring structurally valid serialized outputs, while preserving recognition performance.

  • A. Ablation on Diarization Head: Pretraining the diarization head improves diarization, with silence suppression producing the best Playlogue mtWER, WER, DER, and AER.The non-pretrained head provides limited benefit on Playlogue, whereas the pretrained head improves all reported metrics.
  • A. Ablation on Diarization Head: On ADOS, a randomly initialized diarization head worsens DER because limited data causes slower optimization and incomplete convergence.Noisy gradients from the incompletely optimized head can degrade encoder speaker representations.
  • A. Ablation on Diarization Head: Pretraining with weighted-average encoder representations yields small, consistent gains over last-layer pretraining while maintaining comparable overall performance across datasets.Joint training with a pretrained head keeps WER comparable to or better than the ASR-only baseline and substantially reduces AER.
  • B. Ablation on State-Machine-Based Forced Decoding: 46.6% and 17.4% of Whisper-small decoding runs omit required speaker or timestamp tokens on ADOS and Playlogue without forced decoding.These failures are primarily driven by incorrect prediction of the initial timestamp token.
  • B. Ablation on State-Machine-Based Forced Decoding: Partial forced decoding resolves most initial formatting errors, but full state-machine decoding completely eliminates missing speaker or timestamp tokens and guarantees valid SOT outputs.Infinite loops remain the only occasional failure mode under both forced-decoding settings.
  • B. Ablation on State-Machine-Based Forced Decoding: Full forced decoding improves mtWER and timestamp-based alignment by keeping utterances properly formatted, while making downstream segmentation and role attribution more stable.The method enforces all structural constraints rather than only constraining the first timestamp token.

IX. APPLICATIONS

The framework supports automated extraction of conversational speech measures from child–adult interactions. Predicted metrics closely match human-derived values across Playlogue and ADOS, supporting scalable analysis while not replacing clinician-administered assessments.

  • IX. APPLICATIONS: The application derives five child speech measures covering speech quantity, utterance length, and speaking rate from generated transcripts.Measures include words and utterances per minute, mean utterance duration, mean words per utterance, and speaking rate.
  • IX. APPLICATIONS: Predicted speech metrics closely approximate ground truth, with PCC ranging from 0.53 to 0.98 on Playlogue and 0.70 to 0.97 on ADOS.The correlations are computed across children for each reported measure.
  • IX. APPLICATIONS: The framework enables scalable analysis by producing structured, speaker-attributed transcripts with temporal boundaries without manual segmentation.This reduces annotation burden and facilitates large-scale longitudinal analyses of expressive language and conversational behavior.
  • IX. APPLICATIONS: The generated transcripts may support future development of clinically meaningful interactional and linguistic measures linked systematically to clinician-annotated outcomes.The paper presents this as a future direction rather than an established clinical validation.
  • IX. APPLICATIONS: The proposed system is not a substitute for clinician-administered assessments, despite offering a pathway toward automated speech and language analytics in clinical and research workflows.This scope boundary qualifies the framework’s clinical applicability.
Loading 2601.17640v2…