PaperlayerView exact revision on arXiv

Source-linked AI summary

BreathGRU: A Novel Semi-Supervised Bidirectional Gated Recurrent Unit Framework for Speech and Breath Segmentation for Respiratory Audio

Sania Fatima Sayed, John W. Holloway, Reyer Zwiggelaar, Faisal I. Rezwan

arXiv:2609.31165v1cs.SDcs.LGeess.AS

TL;DR

Speech-breath segmentation is a fundamental preprocessing step in respiratory audio analysis, and the paper proposes BreathGRU, a semi-supervised BiGRU framework for this task. BreathGRU achieved the highest breath event recall and Mean Match IoU, the lowest onset localisation error, and competitive overall segmentation performance against existing approaches.

  • Problem

    Speech-breath segmentation is a fundamental preprocessing step in respiratory audio analysis for healthcare applications.

    Source: §1 · p. 1

  • Method

    BreathGRU combines bidirectional recurrent modelling with frame-level classification, semi-supervised training using pseudo-labels, and duration-constrained decoding to segment speech and breath.

    Source: §1 · p. 3, §2.2 · p. 4, §2.2.4 · p. 6

  • Results

    BreathGRU achieved the highest breath event recall (0.83), the lowest onset localisation error, and the highest Mean Match IoU (0.81), with competitive overall segmentation performance.

    Source: §4 · p. 13, §5 · p. 14

  • Takeaways & Limitations

    The findings indicate that respiratory event modelling with bidirectional temporal context and duration-constrained decoding improves breath detection and boundary localisation compared with general-purpose VAD models.

    Source: §5 · p. 14

    The reported recordings were covered by local ethics approvals in the UK and Australia, bounding the described study to those approved settings.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Speech-breath segmentation is a fundamental preprocessing step in respiratory audio analysis, enabling applications such as respiratory acoustic biomarker extraction, lung function prediction and disease monitoring. Existing approaches, including threshold methods, Fourier Transform-based techniques, and unsupervised and pretrained voice activity detection (VAD) models, primarily focus on speech detection and often classify breathing events as non-speech or silence, limiting their applicability for precise breath detection. To address this limitation, we propose BreathGRU, a semi-supervised Bidirectional Gated Recurrent Unit (BiGRU) framework specifically designed for speech-breath segmentation. The proposed framework combines frame-level acoustic feature extraction with bidirectional recurrent modelling, pseudo-label refinement and duration-constrained Segmental Viterbi decoding to produce speech and breath segmentation. BreathGRU was evaluated against the existing approaches, using manually annotated recordings. Performance was assessed using event-based, time-based, overlap-based, duration-based and boundary-based segmentation metrics. Experiment results demonstrated that BreathGRU achieved the highest breath event recall (0.83), the lowest onset-localisation error (0.14s) and the highest Mean Match Intersection over Union (0.81), with competitive overall segmentation performance compared to large pretrained VAD models like Silero. Qualitative evaluation on manually annotated recordings further showed close agreement between BreathGRU and manual annotation, with better breath detection compared to Silero. These findings demonstrate that explicit breath event modelling provides advantages over general-purpose VAD models and establish BreathGRU as an effective speech-breath segmentation framework which can be applied for respiratory audio analysis and pulmonary healthcare applications.

1. Introduction

Accurate speech-breath segmentation is presented as a preprocessing need for healthcare audio analysis, while much prior work focuses on speech or non-speech rather than explicit breath events. BreathGRU is introduced for speech and breath segmentation, alongside a comparison of existing approaches on manually annotated recordings.

1. Introduction

  • Speech, voice, phonetic, and respiratory sounds are used as non-invasive biomarkers for pulmonary disease assessment, making speech-breath segmentation a fundamental preprocessing step.The passage states that segmentation errors directly affect extracted-feature quality, but its final clause is incomplete.Source: §1 · p. 1
  • Prior approaches include handcrafted threshold rules, frequency-domain and spectral-mask methods,, autoencoders, and pretrained VAD systems such as Silero and PyAnnote.Alam et al.'s rule-based method extracts energy-envelope and spectral features; segments not meeting its conditions are classified as speech.Source: §1 · pp. 2–3
  • Respiratory audio studies have addressed disease monitoring, classification, and spirometry prediction, but most approaches focus on speech activity or phonemic and vowel sounds rather than explicitly modelling breath events.Many segmentation methods were not specifically developed for speech-breath segmentation.Source: §1 · p. 2
  • Pretrained PyAnnote and Silero are optimised for speech detection and can treat inhalation and exhalation as background or silence rather than breathing events.Existing methods also rarely address respiratory temporal continuity, expected breath durations, and speech-breath boundary localisation.Source: §1 · p. 3
  • BreathGRU is a semi-supervised BiGRU framework specifically designed to segment speech and breath using extracted features and temporal probability estimates.The study also compares rule-based, signal-processing, unsupervised, pretrained VAD, and BreathGRU approaches using manually annotated recordings and multiple metric families.Source: §1 · p. 3

2.1. Dataset

The dataset comprises speech recordings from patients with clinically diagnosed asthma after bronchial hyperresponsiveness testing. Manually timestamped speech and breath segments provide evaluation labels, with additional clips from Coswara and TED talks used to assess generalisation.

2.1. Dataset

  • The dataset contains 453 speech recordings from 44 patients with clinically diagnosed asthma after bronchial hyperresponsiveness testing using methacholine challenge and mannitol.Twenty-seven patients contributed 354 recordings at Newcastle University in Australia, and 17 contributed 99 recordings at the University of Southampton in the United Kingdom.Source: §2.1 · p. 3
  • Recordings included audible wheeze, heavy inhalation, and exhalation while participants read a passage; speech and breath segments were manually annotated with timestamps as evaluation ground truth.Participants read the Rainbow Text or a paragraph from “A Winter Book” for one minute after bronchoconstrictor dosages.Source: §2.1 · p. 4
  • For generalisation evaluation, the study used five slow- and fast-counting clips from Coswara and 60-second clips from five TED talks.Participants gave written informed consent, and the study had local ethics approval in the UK and Australia.Source: §2.1 · p. 4

2.2. Breath Bidirectional Gated Recurrent Unit (BreathGRU)

BreathGRU combines frame-level acoustic features, bidirectional sequence modelling, semi-supervised training, and duration-constrained decoding to segment speech and breath. Its components address acoustic representation, contextual classification, use of unlabelled recordings, and temporal consistency.

2.2. Breath Bidirectional Gated Recurrent Unit (BreathGRU)

  • BreathGRU combines acoustic feature extraction, bidirectional recurrent modelling, frame-level classification, semi-supervised training, and duration-constrained decoding for speech-breath segmentation.Figure 2 divides the architecture into feature extraction, bidirectional GRU units, classification heads, training, and inference components.Source: §2.2 · p. 4, Figure 2 · p. 4

2.2.1. Acoustic Feature Extraction

  • Each frame is represented by 59 acoustic features, including MFCCs, their first- and second-order deltas, and Mel-spectrogram bands.The recordings are resampled to 16 kHz and framed with a 25 ms window and 10 ms hop; features are Z-score standardised.Source: §2.2.1 · pp. 4–5

2.2.2. Bidirectional GRU Network

  • A two-layer BiGRU uses preceding and succeeding frame context to represent speech and breath events, including those at transitions.The network has 96 hidden units per direction in each layer.Source: §2.2.2 · p. 5

2.2.3. Frame-level Classification Head

  • The classification head converts contextual BiGRU representations into frame-level posterior probabilities for speech and breath.Layer normalisation, a fully connected layer with ReLU, dropout, and a two-node Softmax output form the head.Source: §2.2.3 · p. 6

2.2.4. Semi-Supervised Learning Strategy

  • The semi-supervised strategy trains on annotated segments, generates pseudo-labels for unlabelled recordings, and weights training by prediction confidence.Frame-level entropy estimates uncertainty; lower-entropy frames receive greater importance and uncertain predictions contribute less to parameter updates.Source: §2.2.4 · p. 6

2.2.5. Duration-Constrained Viterbi Decoding

  • Segmental Viterbi decoding combines frame posteriors, state transitions, and duration priors to select a temporally consistent speech-breath sequence.Duration distributions are estimated from annotated training recordings; the decoding strategy reduces short, fragmented predictions.Source: §2.2.5 · pp. 6–7

2.3 Evaluation Metrics

The evaluation compares predicted speech and breath segments with manual labels using complementary measures of event detection, temporal agreement, segment duration, overlap, and boundary accuracy.

2.3 Evaluation Metrics

  • Event-based F1 measures correctly identified complete segments under overlap criteria, whereas time-based F1 measures label agreement across recording duration.Event precision, recall, and F1 use correctly detected events; time-based precision and recall are duration-based and combined by harmonic mean.Source: §2.3 · p. 7
  • Onset and offset errors measure the temporal differences between predicted and reference segment boundaries, with lower errors indicating more accurate localisation.Mean Duration Error averages absolute duration differences for matched segments; lower values indicate closer agreement in segment lengths.Source: §2.3 · pp. 7–8
  • Mean Match IoU quantifies overlap between matched predicted and reference segments, while Boundary Tolerance measures the share of predicted boundaries within a predefined reference boundary range.The methods are evaluated against manually annotated labels using overlap-based and boundary-based metrics.Source: §2.3 · pp. 7–8

3. Results

BreathGRU and Silero show peak performance across the evaluated speech-breath segmentation models. BreathGRU leads on breath event recall and overall time-based recall, while Silero has stronger speech-event and speech-frame F1 scores.

3. Results

  • BreathGRU and Silero showed peak performance among the evaluated methods, with metrics calculated separately for speech and breath.The comparison included threshold, STFT, an unsupervised autoencoder, Pyannote, Silero, and BreathGRU.Source: §3 · p. 8

3.1. Event-based Evaluation:

  • Breath event recall was 0.83 for BreathGRU versus 0.67 for Silero, while their breath event F1 scores were 0.61 and 0.55, respectively.For speech events, Silero had the strongest F1 (0.59) and BreathGRU was second best (0.43).Source: §3.1 · p. 8

3.2. Time-based Evaluation:

  • BreathGRU had overall time-based recall of 0.83, with breath recall of 0.95 and speech recall of 0.71; Silero had speech F1 of 0.94 and breath F1 of 0.68.Figure 3 identifies BreathGRU as the best of six methods for breath time-based segment detection; Figure 4 presents speech, breath, and overall event precision, recall, and F1 scores.Source: §3.2 · p. 8, Figure 3 · p. 9, Figure 4 · p. 9

3.3. Onset and Offset Evaluation:

BreathGRU localised segment boundaries accurately, with especially low breath-boundary errors. It matched Pyannote on Mean Match IoU, while Silero had the lowest mean duration error.

3.3. Onset and Offset Evaluation:

  • BreathGRU’s mean onset error was 0.18s, versus 0.22–0.25s for other methods and 0.62s for Pyannote.For breath, BreathGRU’s onset and offset errors were 0.10s and 0.05s; its speech errors were 0.24s and 0.22s.Source: §3.3 · p. 9, Figure 5 · p. 10

3.4. Time and Mean-Match IoU Evaluation:

  • Mean Match IoU was 0.81 for both BreathGRU and Pyannote, compared with 0.79 for Silero.Silero and BreathGRU had Time IoU scores of 0.72 and 0.63, respectively.Source: Figure 6 · p. 10, §3.4 · p. 10

3.5. Duration-based Evaluation:

  • Silero had the lowest mean duration error at 2.41s, compared with 4.89s for BreathGRU and 7.11s for Pyannote.Silero’s signed duration bias was −1.81s for speech and +1.81s for breath.Source: §3.5 · p. 11

3.6. Boundary Tolerance Evaluation:

Silero led overall boundary-tolerance scores, while BreathGRU closely matched manual annotations in a patient recording. On unaudited public clips, visual evaluation showed BreathGRU distinguishing speech and breath or pauses against Silero.

3.6. Boundary Tolerance Evaluation:

  • Silero had 71% of boundaries within tolerance, while BreathGRU had 61%.For speech boundaries, BreathGRU scored 57% versus Silero’s 50%; for breath boundaries, it scored 64% versus Silero’s 93%.Source: §3.6 · p. 11

3.8. Generalisation Testing on publicly available audio recordings:

  • Visual evaluation of randomly selected Coswara counting clips and TED Talks showed BreathGRU distinguishing speech from breath or pauses, with smoother transitions than Silero.The clips had no ground-truth or manually extracted segmentations; the plotted TED Talk featured running speech and an Indian accent.Source: §3.8 · p. 11, Figure 8 · p. 12

3.7. Qualitative Comparison of Segmentation Performance:

  • In a patient recording, BreathGRU closely followed manual annotations and produced fewer false-positive breath events than Silero.Silero missed several breath events and partially detected others.Source: §3.7 · p. 11, Figure 7 · p. 12

4. Discussion

BreathGRU detects more breath events than pretrained VAD models and localizes breath boundaries strongly, while retaining competitive overall segmentation performance. The discussion relates these results to temporal modeling, duration-constrained decoding, and evaluation beyond the training recordings.

4. Discussion

  • Speech and breath signals differ in temporal and spectral features, and separating them is relevant to lung-function assessment using symptomatic sounds.The discussion also notes that inaccurate segmentation can affect downstream applications and predictive models.Source: §4 · pp. 12–13
  • Breath event recall was 0.83 for BreathGRU, compared with 0.67 for Silero and 0.29 for Pyannote.The discussion attributes this advantage to training on annotated speech and breath recordings rather than only distinguishing speech from non-speech.Source: §4 · p. 13
  • BreathGRU achieved the lowest onset localisation error and highest Mean Match IoU (0.81), with second-highest Time-based F1 score (0.74) and Time IoU (0.63).Silero achieved higher overall and duration-based metrics, while BreathGRU more accurately identified breath onsets.Source: §4 · p. 13
  • Bidirectional layers use preceding and succeeding acoustic frames, while log-normal duration priors and Segmental Viterbi decoding reduce rapid class switching.The paper attributes improved onset localisation and Mean Match IoU to this combination of temporal context and duration-consistent decoding.Source: §4 · p. 13
  • BreathGRU was competitive for speech segmentation and superior for breath detection despite using fewer recordings than VAD models trained on thousands of hours of speech.The recordings used for BreathGRU came from asthmatic patients.Source: §4 · pp. 13–14
  • In a Figure 7 example with twelve annotated breath events, BreathGRU detected all twelve and produced two false positives, while Silero correctly detected five, missed five, and partially detected two.The authors describe this as a qualitative illustration rather than a comprehensive evaluation.Source: §4 · p. 14
  • BreathGRU showed consistent segmentation on unseen Coswara and YouTube recordings, but those recordings lacked ground-truth annotations and larger external datasets are needed to establish robustness and clinical applicability.The authors identify airflow assessment and respiratory acoustic biomarker extraction as relevant applications where breath-event errors may have clinical implications.Source: §4 · p. 14

5. Conclusion

The conclusion presents BreathGRU as a speech-breath segmentation framework evaluated against rule-based, signal-processing, unsupervised, and pretrained VAD approaches. It reports strong breath and boundary results and points to future integration into respiratory-health assessment pipelines.

5. Conclusion

  • BreathGRU was evaluated on manually annotated recordings against rule-based threshold, Fourier-based, unsupervised autoencoder, and pretrained VAD models.Source: §5 · p. 14
  • BreathGRU achieved the highest breath event recall, lowest onset localisation error, and highest Mean Match IoU while maintaining competitive overall segmentation performance.Source: §5 · p. 14
  • The conclusion attributes improved breath detection and boundary localisation versus general-purpose VAD models to bidirectional temporal context and duration-constrained decoding.Future work will focus on integrating BreathGRU into respiratory-health pipelines for digital biomarker extraction, lung-function prediction, and disease monitoring.Source: §5 · p. 14
  • The study reports that its code, test data, and analysis scripts are publicly available in the GitHub repository.The code is written in Python 3.9.6, with dependencies specified in requirements.txt.Source: §5 · p. 15

Supplementary Material

The supplementary material describes metric definitions, model configurations, and additional results tables and plots. These materials cover event, time, boundary, duration, and overlap-based comparisons across the evaluated models.

Supplementary Material

  • The supplementary material specifies the STFT and Wiener-like mask configurations for Fourier-based segmentation and shows the autoencoder's encoder-decoder architecture.Source: Metric Formulas · p. 19
  • The event-based results table reports precision, recall, and F1 for speech, breath, and overall, with Silero best at event precision and BreathGRU better at breath recall.The supplementary material also labels event-based F1 and provides context for event recall.Source: Metric Formulas · pp. 17, 19
  • The time-based table reports mean precision, recall, and F1 for speech, breath, and combined segments, with Silero best among the six methods.The accompanying metric material includes time-based F1 and definitions involving overlap and reference duration.Source: Metric Formulas · pp. 17, 20
  • The onset- and offset-localisation table compares errors in seconds and describes Silero and BreathGRU as competitive.The supplement labels mean onset error among its boundary-error measures.Source: Metric Formulas · pp. 18, 20
  • The IoU table compares speech, breath, and overall Time IoU and Mean Match IoU; Silero leads speech and breath Time IoU at 0.89, while BreathGRU leads Mean Match IoU.The supplementary text defines IoU using segment overlap and combined duration, and Mean Match IoU over matched segments.Source: Metric Formulas · pp. 18, 21
  • The duration-error table reports absolute and signed errors for speech, breath, and average scores, with Silero having the least absolute error in duration boundaries.Source: Metric Formulas · p. 21
  • Silero's boundary-tolerance match was 0.71 overall and 0.93 for breath, while the table also reports speech and breath results for the other models.The supplement describes this measure in terms of predicted boundaries within a tolerance threshold.Source: Metric Formulas · pp. 18, 22
  • Supplementary Figure 2 presents six frequency plots showing speech and breath segmentation by the models on a 30-second recording.Source: Metric Formulas · p. 23
Loading 2609.31165v1…