PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription

Ayush Kumar Bar, Promila Ghosh

arXiv:2610.14232v1cs.SD

TL;DR

Speech recognizers omit non-verbal vocalizations and background sounds, despite their relevance to transcription and data curation. The paper combines heterogeneous public datasets, timing recovery, and per-class calibration in a 54-class detector. On held-out recordings, the sensitive profile finds 67% of events across the evaluated classes, while evaluation caveats remain.

  • Problem

    Speech recognizers prioritize words, while timestamped detection of vocal and background sounds inside speech faces incompatible labels, overlapping events, and competing false-tag and missed-event concerns.

    Source: §1 · p. 1

  • Method

    The system maps seven public sources into timestamped classes, recovers timings for transcript-embedded sounds by forced alignment, and uses per-class thresholds for two profiles.

    Source: §1 · p. 2, §3 · p. 4, §4.2 · p. 5

  • Results

    Across 48 classes with at least 10 held-out events, the sensitive profile finds 67% of all 9,367 events, with macro recall 0.491 and macro precision 0.302.

    Source: §8.5 · p. 15

  • Takeaways & Limitations

    The detector runs far faster than real time on a CPU, while evaluation on a properly separated target-domain test set remains an open problem.

    Source: §10 · p. 20

    Evaluation uses validation windows from the same held-out clips, a single run, no standard benchmark, and an event criterion that ignores boundary accuracy.

    Source: §9 · p. 20

Abstract

from arXiv · show

Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.

1 Introduction

Speech transcripts omit vocal and environmental sounds that can matter for conversation analysis, captioning, expressive-speech corpora, and training-data curation. The paper combines heterogeneous data and per-class calibration in a timestamped detector whose events can be merged into Whisper transcripts.

1 Introduction

  • Speech recognizers prioritize words, leaving out vocalizations and background sounds that carry meaning for several transcription and data-curation uses.The cited uses include conversation analysis, captioning, expressive-corpus curation, and filtering training data.Source: §1 · p. 1
  • Timestamped detection inside speech must address incompatible labels, overlapping events, and competing needs to limit false tags or missed events.A single global threshold serves neither the desire for few false tags nor the early users’ concern about missed events.Source: §1 · pp. 1–2
  • The system maps seven public sources to 54 timestamped classes, recovers some transcript-embedded timings by CTC forced alignment, and calibrates two profiles on held-out audio.Its events are merged with Whisper word timestamps without reordering the recognizer’s words.Source: §1 · p. 2, Figure 1 · p. 3

2 Related work

Prior sound-event detection combines temporal recognition with onset and offset estimation, while common audio models target environmental sounds or clip-level tagging. This paper’s contribution is to combine disparate sources into a timestamped detector and calibrate and evaluate it, rather than to introduce a new architecture.

2 Related work

  • Sound event detection recognizes sound classes with onset and offset times, and this model uses a Transformer encoder in place of recurrent layers.The paper describes segment- and event-based metrics with collars and threshold-free measures as evaluation options.Source: §2 · p. 3
  • Synthetic soundscapes commonly mix isolated events into backgrounds at known times; this work instead uses speech as the background because target events occur inside speech.Source: §2 · p. 3
  • The paper’s contribution is combining sources into one timestamped detector and calibrating and evaluating it, rather than proposing a new network architecture.Prior work includes vocalization datasets with varied sound coverage and annotation forms.Source: §2 · pp. 3–4

3 Task and label taxonomy

The task is to produce frame-level class probabilities and convert them into timestamped events. The taxonomy contains 20 speaker sounds and 34 background sounds, with transcript tags marking their placement.

3 Task and label taxonomy

  • For 16 kHz mono audio, the model outputs a probability for each class every 40 ms, then post-processing forms events with class, start, end, and mean probability.Source: §3 · p. 4
  • The trained taxonomy has 54 classes: 20 speaker sounds and 34 background sounds; click is excluded because only 47 annotated events were available.Source: §3 · p. 4
  • Speaker sounds appear inline at their start, while background sounds use opening tags and may receive closing tags after overlapping words.A background sound is written as a lone opening tag when the specified overlap and ending conditions are not met.Source: §3 · p. 4

4 Data

The data pipeline maps seven sources with incompatible labels into a shared taxonomy, recovers some transcript-embedded sound timings, and builds real and synthetic training banks. The source-specific splits and crop-cleaning steps define how those materials are selected.

4 Data

  • Table 1 catalogs seven training and evaluation sources, with indexed full-release sizes and quoted licenses.The section reports statistics recomputed from the run’s cached indices and audio banks.Source: Table 1 · p. 4, §4 · p. 4

4.1 Label unification

  • Ordered rules map incompatible source labels into the shared taxonomy, including Vaani free-text tags and AudioSet-Strong event names.Rules also map transcript emojis and folder names, while some tags are ignored or dropped and some recordings become an unlabelled noise pool.Source: §4.1 · p. 5

4.2 Timestamps for NonverbalTTS by forced alignment

  • Forced alignment and energy-based event placement located 6,360 of 8,620 NonverbalTTS sounds, or 74%.The resulting timestamps were not manually verified, and clips with pauses shorter than about 0.2 seconds could not be located by this procedure.Source: §4.2 · p. 5

4.3 Splits

  • Validation sources use source-specific holdouts, including district-based Vaani validation and official development/test or evaluation splits for NonverbalTTS and AudioSet-Strong.Other listed splits hold out NVS-38K clips by hash, OpenSLR 99 speakers, DNC sessions, and the last 20% of each labelled DEMAND recording.Source: §4.3 · p. 5

4.4 Audio banks

  • Audio banks cap most sources at about 3,000 training crops per class and source, clean speech beds and noise with a preliminary detector, and draw class crops unevenly across sources.AudioSet-Strong supplies most background classes, while NVS-38K and NonverbalTTS dominate several common vocalization classes.Source: §4.4 · p. 6, Table 2 · p. 6, Figure 2 · p. 7

5 Model

The model converts an 8-second waveform into 54 class probabilities at 40 ms intervals. Five convolutional blocks extract features, and six Transformer encoder layers provide context across the window.

5 Model

  • Five convolutional blocks reduce an 8-second, 64-band spectrogram to 201 time steps of 768 features.Each front-end step depends on 31 mel frames, about 0.33 seconds of audio.Source: §5 · p. 8
  • 54 event classes receive one probability every 40 ms after convolutional and Transformer processing.The network has 5,802,790 trainable parameters.Source: §5 · p. 8
  • Six Transformer encoder layers process projected 256-dimensional features, allowing each step to draw on the whole 8-second window.A linear sigmoid head produces the per-class probabilities.Source: §5 · p. 8
  • 82% of the model’s parameters are in the six Transformer encoder layers.The convolutions account for 15%, the projection 3%, and the output layer under 1%.Source: §5 · p. 8, Figure 4 · p. 9
  • The illustrated held-out clip’s bottom panel overlays class probabilities with annotated events, while the intermediate panels show network activations and features.The 7.9-second clip is zero-padded to the 8-second window.Source: §5 · p. 8, Figure 5 · p. 10

6 Training

Training combines synthetic event mixtures with real annotated windows and augmentation. The objective and checkpoint selection favor event detection on real validation audio, while per-class thresholds define balanced and sensitive profiles.

6.1 Training examples

  • Synthetic mixtures place one to three events at known times, while real annotated windows retain events in their natural context.Real and synthetic examples receive augmentation, including background noise; 40% of the noise draws come from the real noise pool.Source: §6.1 · p. 11
  • Training uses 171,000 cached feature examples, including augmented copies, with time shifting and SpecAugment applied during training.The pools include 30,000 synthetic mixtures and real windows from four sources.Source: §6.1 · p. 11

6.2 Objective and optimization

  • Frame targets are dilated by 40 ms on each side, and the loss combines class-weighted frame-level BCE with a clip-level maximum-over-time term.The positive BCE term is weighted by the negative-to-positive frame ratio for each class.Source: §6.2 · p. 11
  • The model is selected by mean frame-level average precision across five real or noisy validation sets; synthetic data does not influence selection.AP is threshold-free, so selection rewards models that rank real events above other frames.Source: §6.2 · p. 12

6.3 Calibration: two threshold profiles

  • Per-class thresholds are tuned to maximize frame F1 for the balanced profile or frame F2 for the sensitive profile, with two-fold cross-validation estimating performance.Weak classes receive raised balanced thresholds or sensitive thresholds clamped to 0.6–0.8.Source: §6.3 · p. 12

7 Inference and transcript merge

Inference converts overlapping-window probabilities into timestamped events using temporal filtering and class-specific thresholds. Transcript merging inserts event tags around words while preserving the recognizer’s word order.

7.1 From probabilities to events

  • Events are derived from averaged 40 ms frame probabilities using median filtering, hysteresis, class-specific gap joining, and a 0.15-second minimum duration.Speaker and background events are joined across gaps shorter than 0.3 and 1.0 seconds, respectively.Source: §7.1 · p. 12

7.2 Merging events into a transcript

  • Transcript merging removes events already represented by filler or laughter tokens, joins nearby same-class pieces, and inserts opening and closing tags around overlapping words.Tags are inserted between words rather than sorted with them, so recognizer word order is preserved even when timestamps are non-monotonic.Source: §7.2 · pp. 12–13

8 Experiments

Experiments evaluate the detector on real held-out recordings using frame- and event-level measures, with event-level results showing a recall–precision trade-off between profiles. Training dynamics, per-class scores, attention patterns, worked examples, and runtime further characterize its behavior.

8.2 Training dynamics

  • The selection score rose from 0.209 after epoch 0 to 0.287 after epoch 6, then stayed near that level as training loss continued to fall.The epoch 6 checkpoint was released; the paper reports that validation performance on real held-out audio stopped improving after about seven epochs.Source: §8.2 · p. 13, Figure 6 · p. 14

8.3 Per-class frame-level results

  • Macro frame AP is 0.265 across 54 classes and 0.324 across 43 non-weak classes; music has the highest listed AP at 0.884.Eleven classes are classified as weak; knock has no positive validation frames, and nose_blow has 36.Source: §8.3 · p. 13, Figure 7 · p. 14

8.4 Operating profiles

  • On 43 reliable classes, moving from balanced to sensitive thresholds raises mean cross-validated recall from 0.414 to 0.521 and lowers mean precision from 0.352 to 0.267.The authors recommend the sensitive profile for users reviewing tags and the balanced profile for unattended transcription.Source: §8.4 · pp. 13, 15, Table 6 · p. 15, Figure 8 · p. 15

8.5 Event-level results

  • On 9,367 events across 48 classes, the sensitive profile reaches macro recall 0.491, macro precision 0.302, and F1 0.322, while the balanced profile reaches 0.369 recall, 0.361 precision, and F1 0.329.Events were evaluated on whole held-out clips, counting same-class temporal overlap as a match; this is more lenient than collar-based metrics. Validation windows and threshold tuning overlap the event-evaluation clips, making absolute numbers optimistic.Source: §8.1 · p. 13, Table 7 · p. 15, §8.5 · p. 15

8.6 What the Transformer attends to

  • Across six encoder layers, mean attended distance grows from 1.76 s to 2.59 s, while the first layer places 54% of its attention within ±1 s.For the last layer, five shared time positions receive 12.7% of attention versus 2.5% under uniform attention; this is a single-clip analysis.Source: Figure 10 · p. 17, §8.6 · p. 17

8.7 Worked examples

  • In eight mechanically selected held-out clips, the sensitive profile finds 8 of 22 annotated events, including all four events in one Vaani clip.The examples also include missed coughs, a dog bark, breaths, music, and six short beeps; unannotated sounds cannot be judged for sources that lack those labels.Source: §8.7 · p. 17, Table 8 · p. 19

8.8 Qualitative example

  • In the illustrated Vaani clip, the detector finds music but misses a sneeze and breath, and adds a false tv_speaker detection.An informal single-recording check tagged a sigh and cough, but also added throat_clear and a spurious yawn; it was not part of the evaluation.Source: §8.8 · pp. 17, 19, Figure 12 · p. 19

8.9 Speed

  • On eight CPU threads, detecting 178 s of audio takes 4.6 s, or 39× real time.The measurement excludes audio decoding and model loading; Whisper transcription time was not measured.Source: §8.9 · p. 19

9 Limitations, licensing and ethics

The evaluation has important data, validation, and boundary-accuracy caveats, while domain coverage and several design choices remain untested. Training-data licenses constrain use, and the authors specify intended uses and prohibited applications.

Limitations, licensing and ethics

  • Evaluation reuses windows from held-out clips for checkpoint selection and threshold tuning, uses one run and no standard benchmark, and ignores boundary accuracy.NonverbalTTS and unverified Vaani training timestamps were not manually checked; the authors identify a carefully labelled target-domain test set as the most informative next step.Source: §9 · p. 20
  • Accuracy on call-centre, meeting, and far-field audio is unknown because most real training audio comes from Mandarin, Indic, English, and YouTube sources.The source description includes Mandarin radio drama and comedy, Indic phone recordings, and English interviews and acted speech.Source: §9 · p. 20
  • Eleven classes are weak, and steady background noise can trigger fan detections.The fan example is from the second Vaani clip in Figure 11.Source: §9 · p. 20
  • Pretrained encoders, other window lengths and loss weightings, and ablations of data sources, augmentations, and bed cleaning were not tested.The contribution of each component is therefore not isolated.Source: §9 · p. 20
  • Training-source licenses lead the authors to release weights for non-commercial use only; broader-use weights would require retraining without restricted sources.The authors say whether trained weights are derivative works of their training data is unsettled.Source: §9 · p. 20
  • The intended uses are rich transcription, corpus analysis, data filtering, and annotation assistance with human review; health inference, surveillance, and decisions about individuals are prohibited.The training data contains recordings of real people, and no audio is redistributed.Source: §9 · p. 20

10 Conclusion

The conclusion presents a compact 54-class detector built from heterogeneous public datasets and evaluated on held-out recordings. It highlights remaining evaluation, class-performance, and licensing constraints.

10 Conclusion

  • The sensitive profile detects two thirds of annotated events among 48 classes with at least ten held-out events, but thresholds were tuned on overlapping held-out data.The model runs far faster than real time on a CPU.Source: §10 · p. 20
  • The system unifies seven datasets into 54 classes, recovers transcript-label timestamps by forced alignment, and trains with real and synthetic windows plus real-noise augmentation.Per-class thresholds are calibrated into two profiles on real held-out audio.Source: §10 · p. 20
  • The conclusion identifies a properly separated target-domain test set, weak classes, and inherited training-data licensing constraints as open problems.Source: §10 · p. 20

A Per-class results

The appendix presents per-class evaluation measures, event counts by source, and the rules used to map heterogeneous labels into canonical classes.

A Per-class results

  • Per-class results are organized by class type and evaluation source, with frame average precision, thresholded cross-validated F1 or F2, and held-out event recall and precision.The table caption defines the event metrics as overlap-based and identifies the sensitive and balanced profiles.Source: Table 9 · p. 23, A Per-class results · p. 24

B Annotated events per class and source

C Label-mapping rules

  • Ordered label-mapping rules convert source tags or ontology names to canonical classes; unmatched or excluded labels can be ignored, and some Vaani tags drop a clip.The listed rules include mappings for vocalizations and environmental sounds; the first matching pattern wins.Source: C Label-mapping rules · pp. 25–26

D Reproducibility

The report documents released materials and computational requirements, while noting that rerunning the procedure will not reproduce the released weights exactly. Randomness is only partly fixed, and training was run once.

D Reproducibility

  • The repository includes inference code, training notebooks, weights, threshold files, post-processing configuration, per-class results, and training logs.The report folder also contains scripts and data used to produce figures, tables, and reported numbers.Source: D Reproducibility · p. 27
  • The full run took about 5 hours after downloads, including about 3 hours of training on one NVIDIA T4 machine.The machine had 16 vCPUs and 62 GB RAM; peak training use was about 11 GB RAM and 4.6 GB GPU memory.Source: D Reproducibility · p. 27
  • Exact reproduction is not possible from the released procedure because the model was warm-started from an unreleased preliminary checkpoint and used it to clean speech beds and the noise pool.The preliminary checkpoint used the same architecture and split functions, and the training notebooks now use the released model for those roles.Source: D Reproducibility · p. 27
  • Fixed seeds make the generated pools and validation sets repeatable, but batch order, time shift, and SpecAugment are unseeded; training was performed once.Source: D Reproducibility · p. 27

Related papers

Loading 2610.14232v1…