Source-linked AI summary
SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter
Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, Jeffrey P. Bigham
TL;DR
Stuttering-event detection is hindered by small, inconsistently annotated datasets and substantial variation across speakers and contexts. The paper introduces SEP-28k, a large podcast-derived dataset with five event labels, and benchmarks acoustic models on it and FluencyBank. Increasing training data improves relative F1 performance by 24% on FluencyBank and also improves performance on SEP-28k.
Problem
Existing dysfluency datasets are small and have few or inconsistent annotations, while dysfluency expression varies across speakers and contexts.
Method
The paper introduces SEP-28k, a podcast-derived dataset of over 28k clips labeled for five stuttering event types, and benchmarks acoustic models on SEP-28k and FluencyBank.
Results
A 24% relative F1 improvement occurs on FluencyBank when training on all 25k SEP training samples compared with the 3k FluencyBank set, while SEP-28k performance increases with more training samples.
Takeaways & Limitations
The released annotations support further exploration of stuttering-event detection and related tasks beyond those evaluated in the paper.
Takeaways & Limitations
Block-label results are more speculative because blocks are difficult to assess from audio alone and annotator agreement was low.
Abstract
from arXiv · showhide
The ability to automatically detect stuttering events in speech could help speech pathologists track an individual's fluency over time or help improve speech recognition systems for people with atypical speech patterns. Despite increasing interest in this area, existing public datasets are too small to build generalizable dysfluency detection systems and lack sufficient annotations. In this work, we introduce Stuttering Events in Podcasts (SEP-28k), a dataset containing over 28k clips labeled with five event types including blocks, prolongations, sound repetitions, word repetitions, and interjections. Audio comes from public podcasts largely consisting of people who stutter interviewing other people who stutter. We benchmark a set of acoustic models on SEP-28k and the public FluencyBank dataset and highlight how simply increasing the amount of training data improves relative detection performance by 28\% and 24\% F1 on each. Annotations from over 32k clips across both datasets will be publicly released.
1. INTRODUCTION
The paper addresses automatic detection of dysfluencies in speech, motivated by communication difficulties and limited success applying speech-pathology knowledge to speech recognition. It introduces a larger, annotated dataset to support detection of five stuttering event types.
- Frequent dysfluencies can make social interaction and use of speech technologies challenging, motivating automatic detection for clinical assessment and accessible speech recognition.
- Variation across dysfluency expressions, speakers, situations, and environments makes reliable detection difficult.
- SEP-28k provides 28k annotated clips totaling 23 hours, alongside annotations for 4k FluencyBank clips, addressing small datasets and limited or inconsistent annotations.
- The study focuses on blocks, prolongations, sound repetitions, word or phrase repetitions, and interjections using signal-processing, language-model, and acoustic-model approaches as related foundations.
2. DATA
The study constructs SEP-28k from curated podcast speech and adds consistently processed annotations to FluencyBank. Clips are sampled near pauses, labeled for multiple dysfluency types, and evaluated with agreement-aware procedures and standard detection metrics.
- SEP-28k contains 28,177 three-second clips extracted from 385 episodes across eight shows, with sampling focused near pauses where dysfluencies are more likely.
- The dataset adds 4,144 annotated clips from FluencyBank using recordings from 32 adults who stutter and the same annotation process as SEP-28k.
- Annotators used binary labels on three-second clips, allowing multiple stuttering event types and non-dysfluency labels in one clip.
- At least three trained non-clinician annotators labeled the clips, with agreement strongest for word repetitions and weakest for prolongations and blocks.
- F1 and Equal Error Rate evaluate binary labels individually and through a combined Any label, while the datasets use separate training, validation, and test splits.
3. METHODS
The approach extracts multiple frame-level acoustic feature types, models their temporal structure with LSTM-based architectures, and predicts clip-level dysfluency labels. Training compares baseline and improved objectives, architectures, and input features.
- The model extracts acoustic features per frame, applies a temporal model, and outputs clip-level dysfluency labels while comparing features, architectures, and losses.
- Inputs include mel-filterbank energies, pitch and voicing, articulatory vocal-tract constriction variables, and phoneme probabilities.
- Pitch, voicing, and articulatory features are hypothesized to help detect blocks or gasps, while phoneme probabilities may help identify repeated sounds.
- The baseline uses a single-layer LSTM, whereas ConvLSTM adds feature-specific convolutions, learned feature weighting, and two output branches.
- Improved models combine weighted focal loss for fluent/dysfluent prediction with concordance correlation coefficient loss for per-dysfluency outputs.
4. EXPERIMENTS & ANALYSIS
Experiments show that model design and training-data scale both affect stuttering-event detection, while performance varies substantially across event types and datasets.
- 4.1. Model Design: Spectral features with pitch generally performed well, while adding articulatory signals produced the best performance for the improved model.Phoneme-based models performed worst despite their potential relevance to sound repetitions.
- 4.1. Model Design: ConvLSTM architecture and CCC loss moderately improved F1, likely because CCC explicitly represents annotator uncertainty.Table 3 reports F1 by dysfluency type for baseline LSTM and improved ConvLSTM models.
- 4.1. Model Design: Blocks and Word Repetitions had lower performance, whereas Interjections and Prolongations were easier to detect.SEP-28k performance was consistently worse than FluencyBank, likely because it includes more individuals and speaking styles.
- 4.2. Data Quantity & Type: 24% relative F1 improvement was achieved on FluencyBank when training with all 25k SEP training samples instead of 3k FluencyBank clips.Using only 5k SEP clips already improved FluencyBank performance by 16% F1.
- 4.2. Data Quantity & Type: Performance on SEP-28k increased with larger training subsets, whereas training only on FluencyBank produced the worst SEP-28k performance.The results support the hypothesis that existing datasets are too small for effective dysfluency detection models.
5. CONCLUSION
The paper introduces SEP-28 with substantially expanded annotations and adds annotations to FluencyBank. It encourages broader uses of the released data and identifies language models and other speech disorders as future directions.
- 5. CONCLUSION: SEP-28 contains over an order of magnitude more annotations than existing public datasets, alongside new FluencyBank annotations.The authors encourage exploration of the data, labels, and splits beyond the analyses presented.
- 5. CONCLUSION: Future work should explore language-model approaches that may improve detection of difficult dysfluency types.
- 5. CONCLUSION: Future work should also examine dysfluency detection in people with other speech disorders, including dysarthria.The paper notes that such disorders may be characterized differently from stuttering.