PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise-Robust Forced Alignment for Spontaneous Speech

Laurin Wagner, Mario Zusag

arXiv:2610.22583v1cs.LG

TL;DR

Forced aligners face gaps in robustness to noisy spontaneous speech and in modeling fillers and vocal events, despite the importance of accurate timing for downstream applications. This paper combines a self-bootstrapped GMM-HMM with WavLM features, explicit event units, and a noise-robust projection; it reports lower boundary MAEs than the strongest baselines on conversational and disfluent speech.

  • Problem

    Existing aligners leave joint gaps in noise robustness, explicit filler and vocal-event modeling, and verbatim training for spontaneous speech, although accurate timing supports downstream speech applications.

    Source: §1 · p. 1

  • Method

    A self-bootstrapped GMM-HMM uses WavLM features, dedicated filler and vocal-event units, and a noise-robust feature projection.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2

  • Results

    20.6 ms on Buckeye and 76.9 ms on FluencyBank, respectively 19% and 17% below the strongest baselines.

    Source: §5.1 · p. 3

  • Takeaways & Limitations

    Accurate, robust timing provides stronger supervision for downstream learning and supports applications including subtitling, speech editing, and dubbing.

    Source: §1 · p. 1

Abstract

from arXiv · show

We show that forced aligners degrade on spontaneous, noisy real-world speech, and that fillers, laughter, and disfluencies corrupt word boundaries, impacting many downstream applications. We revisit the classical GMM-HMM aligner with upsampled WavLM features instead of MFCCs, explicit phone units for fillers and non-lexical vocal events trained on verbatim-annotated spontaneous speech, and a discriminative projection fitted on clean and noise-corrupted feature pairs to improve robustness. The pipeline is self-bootstrapped from 149h of curated audio without manual or external boundary targets. On conversational (Buckeye) and disfluent (FluencyBank) speech it reaches word-boundary MAEs of 20.6 and 76.9ms, 19% and 17% lower than the strongest of ten baselines. Catastrophic misalignments (>200ms) are 2.7 times rarer than Montreal Forced Aligners (MFA) best model even after widening MFA's search beam, which we show removes a large pruning artifact from its conversational error; event units halve the error next to laughter, while being meaningfully more robust than existing aligners.

1. INTRODUCTION

Accurate timing supports clinical, corpus, training, and media applications, but existing aligners leave joint gaps in noise robustness, event modeling, and verbatim spontaneous-speech training. The paper addresses these gaps with a self-bootstrapped GMM-HMM aligner and dedicated noise-robust features and event units.

Motivation

  • Accurate word and phone timing supports clinical measures, corpus segmentation, training objectives, subtitling, editing, and dubbing.Systematic boundary errors act as label noise for downstream training.Source: §1 · p. 1

Research gap

  • Existing aligner families do not jointly provide sufficient noise robustness, explicit filler and vocal-event modeling, and verbatim spontaneous-speech training.Untranscribed hesitations or laughs can become mislabeled audio stretches during training.Source: §1 · p. 1

Contributions

  • The paper presents a self-bootstrapped GMM-HMM aligner that improves over ten competitive baselines on conversational and disfluent speech.Its contributions include WavLM features with a noise-paired projection, dedicated filler and vocal-event units, and verbatim data curation.Source: §1 · p. 1

Capability comparison

  • Table 1 is a capability grid in which “Lbl.-free” means no manual or external boundary labels and “∼” denotes partial robustness.The partial robustness is inherited from augmentation or large-scale pretraining.Source: Table 1 · p. 2

2. TRAINING DATA

The training set combines seven sources totaling 149 hours, with roughly half conversational speech. It includes verbatim transcripts and adds vocal-event examples through both annotations and synthetic insertion.

Training sources

  • The seven training sources total 149 hours, roughly half conversational speech, and all transcripts follow verbatim conventions.The conversational sources include ICSI, AMI, DisfluencySpeech, and part 6 of the Singapore National Speech Corpus.Source: §2 · p. 2

Annotated data

  • ICSI and DisfluencySpeech provide vocal-sound annotations, predominantly laughter, alongside verbatim transcripts that include fillers.The read-speech sources add recording-condition and speaker diversity as well as clean speech.Source: §2 · p. 2

Vocal-event overlay

  • Vocal-event clips from VocalSound and Nonspeech7k are inserted into detected pauses in LibriSpeech training audio, with matching transcript tags.MFA identifies candidate pauses of at least 200 ms; its boundaries are not acoustic-model training targets.Source: §2 · p. 2

3. METHOD

The method keeps a classical GMM-HMM alignment architecture while changing its phone inventory and acoustic features. Training is self-bootstrapped, and a projection fitted on clean and noise-corrupted features targets noise robustness.

3.1. Phonemization

  • The phonemizer maps espeak-ng IPA pronunciations to 39 canonical English phones and extends the inventory with filler and vocal-event units.The added units cover [UH], [UM], and twelve non-lexical vocal events, each represented by a one-phone lexicon entry.Source: §3.1 · p. 2

3.2. Self-bootstrapped WavLM-GMM aligner

  • The aligner retains a Kaldi-style GMM-HMM architecture while replacing MFCCs with interpolated, PCA-reduced WavLM layer-22 features.The 39-dimensional representation is interpolated to a 10 ms grid and provides a bottleneck for stable flat-start GMM training.Source: §3.2 · p. 2
  • Each training stage uses its own alignments, progressing from a flat-start monophone model to a triphone model trained on LDA-projected WavLM features.The recipe uses no speaker adaptation or MLLT because neither improved boundary accuracy in the reported ablations.Source: §3.2 · p. 2

3.3. Noise-robust feature projection

  • The noise-robust variant fits its LDA projection on clean features and MUSAN-corrupted copies carrying identical frame labels.The projection is intended to separate phones while making directions invariant to the noise variation.Source: §3.3 · p. 2

4. EXPERIMENTAL SETUP AND METRICS

Evaluation uses held-out read, conversational, and disfluent speech, with additional noise tests on Buckeye. The protocol scores matched words and reports timestamp error and threshold accuracy, using widened beams for MFA where needed.

Benchmarks

  • TIMIT, Buckeye, and FluencyBank are held out from training, and Buckeye is also tested with MUSAN and Gaussian noise at 20, 10, 5, and 0 dB SNR.The benchmarks cover read, conversational, and stuttering speech, respectively.Source: §4 · p. 3

Protocol

  • ASR-based systems are scored only on exactly matched words, and non-lexical event tokens are excluded from word-level metrics for every system.MFA comparisons include released models and an unmodified MFA recipe trained on the paper’s 149-hour corpus.Source: §4 · p. 3

Decoding beams

  • MFA results use a wide beam because its defaults prune optimal paths on spontaneous speech, while the proposed aligner is essentially beam-independent.The reported comparison uses beam 400 with retry 4000 for MFA; read-speech results are unaffected by the defaults.Source: §4 · p. 3

Metrics

  • MAE is the mean of each matched word’s average absolute onset and offset error, while threshold accuracy measures the share within specified offset-error limits.The thresholds are 25, 50, 100, and 200 ms.Source: §4 · p. 3

5. RESULTS

The final aligner improves over the strongest baseline on conversational and disfluent speech, with fewer severe errors than MFA. Dedicated event units reduce errors near laughter, while WavLM features and a noise-paired projection improve robustness under noise.

5.1. Main comparison and component ladder

  • 19% on Buckeye and 17% on FluencyBank: the final system’s margins over the best baselines, with MAEs of 20.6 ms and 76.9 ms, respectively.On Buckeye, errors beyond 200 ms are 2.7× rarer than for MFA’s best configuration (0.37% vs. 0.98%).Source: §5.1 · p. 3
  • 99% of our Buckeye words fall within 97 ms of reference, versus 196 ms for MFA’s best configuration.The typical-word regime is similar between HMM systems, but our error profile stays below MFA’s as severity increases.Source: §5.1 · p. 3, Figure 1 · p. 3

5.2. Alignment around disfluencies

  • 26–29 ms next to fillers and cutoffs, and 53 ms next to laughter; dedicated vocal-event units reduce laughter-adjacent error a further 2.3×.In the WavLM arm, event units reduce laughter-adjacent error from 107.8 to 52.4 ms.Source: §5.2 · p. 3, Table 4 · p. 4

5.3. Noise robustness

  • At 0 dB SNR under Gaussian noise, ours has 96 ms MAE, versus 298 ms for MFA our data and 417 ms for MFA us arpa.MFA english mfa has 2.0 s MAE in this condition; its rows use a wide beam to restore full coverage.Source: §5.3 · p. 4, Table 5 · p. 4
  • At 0 dB SNR, replacing MFCC with WavLM reduces MAE from 170 to 74 ms under MUSAN and from 273 to 136 ms under Gaussian noise.The noise-paired projection is fitted using MUSAN music; the Gaussian-noise columns therefore test generalization to an unseen noise family.Source: §5.3 · p. 4, Table 5 · p. 4

6. CONCLUSION AND FUTURE WORK

Adjustments to training data, phonemization, and feature extraction yield state-of-the-art spontaneous-speech alignment under noise. The authors note that multilingual verbatim transcription makes extending the approach beyond English feasible.

  • Training-data, phonemization, and feature-extraction decisions yield state-of-the-art alignment for spontaneous speech under noise.Source: §6 · p. 4
  • The conclusion identifies multilingual verbatim transcription as making extension beyond English feasible.Source: §6 · p. 4
  • CrisperWhisper 2.0 is named as the multilingual verbatim-transcription basis for that potential extension.Source: §6 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The study is an algorithmic contribution using public benchmark datasets and required no ethical approval.

7. COMPLIANCE WITH ETHICAL STANDARDS

  • The study uses publicly available benchmark datasets and required no ethical approval.Source: §7 · p. 5

Related papers

Loading 2610.22583v1…