An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

1. Introduction

• Speech activity does not map directly to conversational structure: within-turn pauses, rapid transitions, overlap, and cross-channel leakage can mislead segmentation.Turns and backchannels require interaction-level classification, not acoustic boundaries alone.Source: §1 · p. 1
• Backchannels vary in form and duration and depend on conversational context, so duration thresholds or fixed lexical lists alone cannot identify them.They provide listener feedback without taking the floor and may occur during an ongoing turn.Source: §1 · pp. 1–2
• VAD and ASR provide speech timing and words, but neither alone labels speech as a continuing turn, a new floor-holding contribution, or listener feedback.This gap separates general speech-processing outputs from the interaction-level labels needed for conversation analysis.Source: §1 · pp. 1–2
• The pipeline derives speaker-specific temporal units directly from audio, addressing leakage and pause fragmentation before using transcription to help classify them.Unlike NaturalTurn, which starts from speaker-specific word-level ASR output, this pipeline begins earlier in processing.Source: §1 · p. 2
• The pipeline targets turns and backchannels in freely conversing dyads and is intended to provide a consistent first-pass annotation for human review.Its evaluation examines detection, boundary accuracy, listening conditions, parameter settings, and a four-conversation human-agreement comparison.Source: §1 · p. 2

2.1. Conversation unit definitions

• Turns are continuous floor-holding contributions, while backchannels provide listener feedback without attempting to take the floor; crosstalk is distinguished from genuine overlap.An overlapped turn is an algorithmic label for a candidate fully contained within the interlocutor’s turn, not evidence of floor-taking.Source: §2.1 · p. 2

2.2. Human reference annotation

• One annotator labeled all 99 conversations, and a second independently annotated four conversations using the same definitions and procedure.Annotators inspected both participant channels and marked each unit’s type, onset, and offset.Source: §2.2 · p. 2

2.3. Pipeline architecture

• The seven-stage pipeline transforms separate-channel recordings into speech events labeled as turns or backchannels for subsequent human review.It combines speech activity detection, channel-energy filtering, transcription, and rule-based post-processing.Source: §2.3 · p. 2, Figure 1 · p. 4
• rVAD detects speech separately on each channel, then an energy-based filter removes segments below a channel-relative threshold to reduce crosstalk.A 60-Hz high-pass filter is applied before VAD and channel-energy comparison.Source: §2.3 · pp. 2–3
• Same-speaker VAD segments are merged across short gaps or when speech duration meets or exceeds the gap, reducing fragmentation while retaining intervening short events.The 0.5-s gap is a theory-informed criterion, not a turn-boundary definition; merged segments are capped at 60 s for transcription processing.Source: §2.3 · p. 3
• Speech segments receive stronger preprocessing before Whisper transcription, which supplies an intermediate representation rather than the final annotation.The transcript-stage signal is noise-reduced and loudness-normalized, but event boundaries remain those from earlier stages.Source: §2.3 · p. 3
• Word-token entropy serves as a proxy for lexical diversity: segments below the default threshold are initially labeled backchannels, while higher-entropy segments are initially labeled turns.Empty and single-token transcriptions are assigned entropy zero, and preliminary labels are later re-evaluated.Source: §2.3 · p. 3
• Temporal context refines labels, and same-speaker turns may be merged across limited gaps when intervening segments are backchannels or overlapped turns.Before evaluation, pipeline and reference annotations receive the same repeated context-based merging and relabeling normalization.Source: §2.3 · pp. 3–4

2.4. Dataset and recording protocol

• Each dyad completed three 10-minute face-to-face Danish conversations, with each participant recorded on a separate head-worn directional microphone.The audio was recorded at 48 kHz.Source: §2.4 · p. 4
• One conversation was noise-free, while two used asymmetric noise delivered to only one participant through bone-conduction headphones.Condition order was counterbalanced across dyads, and participants were not told in advance when or to whom noise would be presented.Source: §2.4 · p. 4
• The bone-conduction noise did not affect the recorded audio, so the manipulation varied participants’ listening conditions rather than recording background noise.Noise level was calibrated to yield 70% detection of a spoken digit presented at 45 dBA in an independent sample.Source: §2.4 · p. 4
• The analysis covers 99 ten-minute conversations from 33 dyads, totaling approximately 16.5 hours, with four conversations additionally annotated by a second human annotator.Participants were native Danish speakers aged 18–35, unacquainted, and instructed to converse freely without a prescribed topic or task.Source: §2.4 · p. 4

2.5.1. Detection reliability

• Detection reliability matches reference and estimated segments of the same speaker and conversational-unit type using temporal intersection over union (IoU).Each segment can contribute to at most one match; the highest-IoU candidate is retained when multiple candidates qualify.Source: §2.5.1 · p. 4
• Matches require IoU of at least 0.50 for turns and 0.10 for backchannels, with the lower backchannel threshold accounting for their shorter durations.Matched segments are true positives, unmatched references false negatives, and unmatched estimates false positives; F1 is computed separately by unit type and overall.Source: §2.5.1 · pp. 4–5

2.5.2. Temporal precision

• Temporal precision is the absolute difference between estimated and reference onset or offset times for successfully matched segments.The reference and estimated boundaries are denoted tref and test, respectively.Source: §2.5.2 · p. 5

Example: if a human reference segment spans 2.0–

• The example reports onset and offset errors of 0.1 s each.Source: Example: if a human reference segment… · p. 5

2.5.2. Temporal precision

• Median absolute error and interquartile range summarize typical boundary error and its spread, while mean absolute error and standard deviation characterize the influence of larger errors.For listening-condition comparisons, median absolute error is calculated separately for each conversation.Source: Example: if a human reference segment… · p. 5

3.1. Effect of context-based post-processing

• Post-processing reduced segments in the example from 69 to 55 for manual annotations and from 125 to 58 for pipeline output.Across all conversations, mean segment counts fell from 186.6 to 172.2 for manual annotations and from 158.8 to 154.1 for pipeline output; the example also showed less consistent backchannel agreement.Source: §3.1 · p. 5, Table 1 · p. 5

3.2. Overall pipeline performance

• Overall F1 was 0.621 ± 0.151 across conversations, with 0.624 ± 0.143 for turns and 0.618 ± 0.160 for backchannels; matched-segment median boundary errors ranged from 0.130 to 0.180 s.Turn onset and offset MedAEs were 0.150 s and 0.160 s; backchannel onset and offset MedAEs were 0.130 s and 0.180 s.Source: §3.2 · p. 5, Table 2 · p. 5

3.3. Performance across listening conditions

• No systematic performance difference was found between noise-free and asymmetric-noise listening conditions.Detection F1 differences were nonsignificant for turns (p = .885) and backchannels (p = .113), and temporal-precision comparisons were also nonsignificant.Source: §3.3 · pp. 5–6

3.4. Case study: Benchmarking against inter-annotator agreement

• In the four-conversation case study, pipeline F1 was 0.626 ± 0.225 overall, compared with human inter-annotator F1 of 0.845 ± 0.118.Human agreement was also higher for turns (0.868 ± 0.051 versus 0.668 ± 0.179) and backchannels (0.821 ± 0.168 versus 0.585 ± 0.256).Source: §3.4 · p. 6, Table 3 · p. 6
• Pipeline median boundary errors in the case study were larger than human inter-annotator errors for turn onsets and offsets and for backchannel onsets and offsets.Pipeline versus human MedAEs were 0.190 versus 0.080 s and 0.210 versus 0.110 s for turn onsets and offsets; backchannel values were 0.140 versus 0.050 s and 0.200 versus 0.040 s.Source: §3.4 · p. 6, Table 3 · p. 6
• Pipeline performance varied substantially across parameter combinations and the four conversations in the case study.Distributions were broader than human inter-annotator distributions, particularly for backchannel F1 and boundary errors.Source: §3.4 · p. 6, Table 3 · p. 6
• No inferential comparison was performed because the 320 parameter runs repeated across four conversations were not independent observations.The human benchmark also covered only those four conversations.Source: §3.4 · p. 6

4. Discussion

• Pipeline performance did not differ significantly between noise-free and asymmetric-noise listening conditions.The manipulation changed participants’ listening context rather than adding noise to the recorded microphone signal.Source: Figure 3 · p. 7, §4 · p. 7
• An automated first pass can apply consistent operational criteria across recordings while retaining human review for ambiguous cases.The authors present this workflow as a way to support more standardized and reproducible annotation.Source: §4 · p. 7
• Across 99 Danish conversations, overall detection F1 was approximately 0.62, while matched turn-boundary MedAEs were 0.150 s at onset and 0.160 s at offset.Mean absolute errors were approximately 1 s for both turn boundaries, indicating a smaller number of substantially larger mismatches; backchannel errors showed a smaller median–mean difference.Source: §4 · p. 7
• Human reference annotations also vary in segment classification and boundary placement, so human agreement is a descriptive benchmark rather than a strict performance ceiling.The authors call for a larger, systematic study of inter-annotator agreement.Source: §4 · p. 7
• Performance varied across parameter settings, especially for backchannels, and reliance on automatic transcription may affect labels for short or non-lexical responses.Transcription accuracy was not evaluated separately; the authors recommend evaluating ASR on short responses and parameter selection on independent data.Source: §4 · p. 7
• The evaluation was limited to dyadic, separate-channel Danish conversation, and annotation time was not measured directly.The authors recommend testing annotation efficiency and the approach across languages, recording configurations, and interactional settings, including multi-party conversation.Source: §4 · p. 7
• In four conversations, pipeline–human agreement was numerically lower and more variable than human inter-annotator agreement, particularly for backchannel F1 and boundary errors.The comparison is descriptive: 320 runs reused 80 parameter combinations on the same four conversations and are not independent observations.Source: §4 · p. 7, Figure 4 · p. 8

5. Conclusion

• The pipeline is intended as a consistent first-pass segmentation for semi-automated review of spontaneous dyadic conversation.It applies the same operational criteria across recordings to support more standardized and reproducible annotation.Source: §5 · p. 8
• Across 99 conversations, detection reliability was comparable for turns and backchannels, with relatively small median temporal errors for successfully matched segments.Performance was more variable than human inter-annotator agreement and sensitive to parameter selection.Source: §5 · p. 8
• The pipeline provides a consistent starting point for human review rather than a replacement for it.The conclusion frames the workflow as supporting standardized and reproducible annotation procedures.Source: §5 · p. 8

7. Ethics

• The Capital Region Committees of the Danish Science-Ethics Committee approved the procedures (H-16036391), and all participants provided informed consent.Source: §7 · p. 8

8. Author Contributions

• H.H. and H.V.S.R. contributed to conceptualization and methodology; H.H. led investigation and data curation, while I.K. handled funding acquisition and supervision.All four authors contributed to writing or review and editing; I.O. contributed methodology.Source: §8 · p. 8

• The work is supported by the Carlsberg Foundation’s Semper Ardens: Accelerate grant, no. CF22-1251, and by an Innovation Fund Denmark grant, no. 4354-00015B.Source: §9 · p. 8
• The Innovation Fund Denmark also supported the work through grant no. 4354-00015B.Source: §9 · p. 8

• The authors used generative AI to edit and refine the text and assist with code, then reviewed and edited the content as needed and took full responsibility for it.Source: §11 · p. 8