PaperlayerView exact revision on arXiv

Source-linked AI summary

From Detection To Anticipation: Continuous Onset Forecasting For Stuttered Speech

Tampu Ravi Kumar, Ayoub Daliri, Vivek Gupta

arXiv:2610.19235v1cs.SDcs.LG

TL;DR

Stuttering systems generally detect dysfluencies after they occur, leaving continuous onset forecasting from preceding speech as a research gap. This paper builds an onset-aligned dataset and models frozen Whisper speech representations with a causal Conformer. The model achieves AUROC 0.6300, and a discrete-time survival formulation provides coherent forecasts across multiple horizons.

  • Problem

    Most computational systems detect speech segments that already contain dysfluencies, while next-segment prediction does not localize onset or continuously estimate its imminence.

    Source: §1 · p. 1

  • Method

    The paper augments FluencyBank Timestamped with reviewed onset annotations and models frozen Whisper speech representations with a causal Conformer for continuous forecasting from preceding speech.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    AUROC 0.6300 is achieved by the causal Conformer, which outperforms a wider static Whisper window, while the survival formulation produces coherent multi-horizon forecasts.

    Source: §4.2 · p. 3, §4.4 · p. 4

  • Takeaways & Limitations

    The findings provide evidence that speech preceding stuttering onset carries measurable predictive structure and establish an empirical baseline for continuous forecasting.

    Source: §6 · p. 4

    The onset-aligned annotation layer covers 2,872 events across 36 speakers, and larger, more diverse cohorts are needed to scale fine-grained anticipatory modeling.

    Source: §7 · p. 4

Abstract

from arXiv · show

Automatic stuttering systems primarily detect dysfluencies after they have occurred, limiting their use for anticipatory intervention. We address this gap by formulating stuttering prediction as continuous causal time-to-event forecasting, where a model estimates the risk of an upcoming stuttering onset every 100 ms using only preceding speech. To support this task, we augment FluencyBank Timestamped for 36 speakers with manually reviewed repetitions, prolongations, and blocks, yielding 2,872 forecasting events. We then model the temporal evolution of frozen Whisper-large-v3 speech representations using a compact causal Conformer with 3.67M trainable parameters. The proposed approach achieves an AUROC of 0.63. Temporal analysis further shows that explicitly modeling preceding speech states outperforms simply increasing the static acoustic window, while a discrete-time survival formulation provides coherent forecasts from 300 ms to 1.5 s using a single model. These results provide initial evidence that stuttering onset is preceded by measurable temporal information in speech and establish a framework for future anticipatory stuttering systems.

1. INTRODUCTION

Most stuttering systems detect dysfluencies after they occur, while segment-level prediction does not localize onset or continuously estimate its imminence. This paper formulates continuous causal forecasting from preceding speech and augments FluencyBank Timestamped with onset-aligned events.

1. INTRODUCTION

  • Most computational stuttering work detects dysfluencies in segments that already contain them, while next-segment prediction neither localizes onset nor continuously estimates its imminence.The paper asks whether onset can be forecast continuously using only speech observed before the event.Source: §1 · p. 1
  • The paper formulates stuttering prediction as continuous causal time-to-event forecasting, estimating future onset risk every 100 ms from preceding speech.The model estimates the probability that a new event will begin within a future horizon.Source: §1 · p. 1
  • The FluencyBank Timestamped augmentation provides 2,872 forecasting events from 36 speakers, including manually reviewed repetitions, prolongations, and blocks.The contribution also evaluates forecasts by event type, speaker, and horizons from 300 ms to 1.5 s, including under an operational duty-cycle constraint.Source: §1 · p. 1

2. TASK FORMULATION AND DATA

The task uses onset-aligned annotations and prediction times on a 100-ms grid, with inputs restricted to speech available at or before each prediction time. A discrete-time survival formulation extends forecasts across multiple horizons while accounting for right-censored frames.

2.1. Data and Annotation Augmentation

  • The augmented FluencyBank Timestamped annotations draw on 36 speakers, with end-to-end human review adding missed repetitions, prolongations, and blocks and an independent reliability check on 185 clips.Fleiss’ κ was 0.54 for repetitions, 0.73 for prolongations, and 0.56 for blocks; matched-event median absolute onset discrepancy was 80 ms.Source: §2.1 · p. 1
  • Events are placed at the acoustic start of each dysfluency; adjacent or closely spaced events without an intervening fluent word are consolidated, yielding 2,872 events.The inventory includes 2,593 repetitions, 129 prolongations, and 150 blocks; the latter two types were annotated entirely by hand.Source: §2.1 · pp. 1–2

2.2. Forecasting Target

  • Each prediction uses preceding speech only, with the encoder summarizing the prior 2 s and the causal model reaching back roughly 22 s through 200 states.The 2-s window determines local audio summarization, not the full temporal context; nothing read by the model extends beyond the prediction time.Source: Figure 1 · p. 2, §2.2 · p. 2
  • A 100-ms prediction grid targets the time until the next event beginning strictly after the prediction time, labeling whether onset falls within a chosen future horizon.The positive input never contains the target event itself.Source: §2.2 · p. 2
  • Of 229,966 grid points, 164,694 are scorable after excluding frames inside an active event or outside transcribed participant speech; 4.93% are positive at 300 ms.Silence within a transcribed segment remains scorable.Source: §2.2 · p. 2
  • Positive and negative frames occur within 2 s of a previous onset at comparable rates, 25.4% and 19.8%, respectively.This comparison addresses whether recent events alone could make the model appear to forecast onset.Source: §2.2 · p. 2

2.3. Multi-Horizon Formulation

  • The multi-horizon model predicts 15 conditional hazards at 100-ms intervals in one forward pass, representing onset in each bin given no earlier onset.Cumulative onset probabilities are derived across bins for horizons from 0.3 to 1.5 s.Source: §2.3 · p. 2
  • Masked binary cross-entropy trains the survival model over at-risk bins, excluding right-censored future bins; its resulting risk trajectory is coherent and non-decreasing.The evaluable set decreases from 156,669 frames at 0.3 s to 128,167 at 1.5 s.Source: §2.3 · p. 2

3. CAUSAL TEMPORAL FORECASTING

The system summarizes only preceding audio with frozen Whisper representations, then applies causal temporal modeling to estimate onset risk. Its training setup uses different losses for fixed-horizon and multi-horizon forecasting.

3. CAUSAL TEMPORAL FORECASTING

  • The causal model attends over up to 200 states, so 2-s audio summaries spaced 100 ms apart yield roughly 22 s of total receptive reach without future audio.Causal attention, left-padded convolutions, and LayerNorm prevent future information from reaching a prediction.Source: §3 · p. 2, §3.2 · p. 3

3.1. Acoustic Representation

  • A frozen Whisper-large-v3 encoder summarizes the 2 s of audio ending at each prediction time, producing one 1,280-dimensional state per 100 ms.The final encoder layer was selected on validation average precision, and representations were cached before temporal-model training.Source: §3.1 · p. 3

3.2. Temporal Model

  • The causal Conformer uses four blocks, four-head causal attention, and causal convolutions to map speech-state sequences to frame-level onset risk.It has 3.67M trainable parameters and attends over up to 20 s of prior states; Whisper remains frozen.Source: §3.2 · p. 3

3.3. Training

  • Fixed-horizon forecasting uses focal loss, while the 15-bin survival head uses plain binary cross-entropy over at-risk bins to preserve calibrated hazards.Training sequences use a 200-step context buffer followed by 200 scored steps; multi-horizon training uses a 50-step buffer.Source: §3.3 · p. 3

4. EXPERIMENTS AND RESULTS

At 300 ms, the causal Conformer outperforms the TCN and static-window predictors, with temporal modeling improving pooled discrimination. Results also show event-type and speaker variation, useful history saturating near 5 s, and a coherent multi-horizon hazard model with operational coverage–duty-cycle trade-offs.

4.1. Setup

  • The six speaker-disjoint folds use shared grids and targets, three seeds per fold, out-of-fold predictions, and paired speaker-level bootstrap intervals.Because only 4.93% of frames are positive, the study reports AP, lift over prevalence, AUROC, and within-speaker AUROC rather than accuracy.Source: §4.1 · p. 3

4.2. Does Temporal Modeling Improve Forecasting?

  • At 300 ms, the Conformer reaches AP 0.0769 (1.56× the prevalence floor) and AUROC 0.6300, improving pooled AUROC over the TCN by 0.0411 (95% CI [0.0196, 0.0615]).Widening the static Whisper window from 2.0 to 3.8 s moves AUROC from 0.5682 to 0.5666, while temporal modeling reaches 0.6300.Source: Table 1 · p. 3, §4.2 · p. 3

4.3. Event Type, Speakers, and Temporal Context

  • At 300 ms, Conformer AUROC is 0.630 for repetitions, 0.655 for prolongations, and 0.612 for blocks, improving pooled discrimination over the TCN for all three types.Type comparisons exclude frames positive for another type.Source: Table 2 · p. 3, §4.3 · p. 3
  • Repetition forecasting has mean wAUROC 0.576 (95% CI [0.564, 0.590]) across 36 speakers, with 97% above chance; prolongation and block mean wAUROCs are 0.540 and 0.541, with intervals overlapping chance.Sufficient positive frames were available for only 17 prolongation and 23 block speakers; pooled-to-within-speaker gaps indicate speaker heterogeneity, particularly for rarer types.Source: §4.3 · p. 3
  • With the 2-s Whisper representation fixed, AUROC rises from 0.592 at 2-s attention history to 0.621 at 5 s, with no measurable gain at 10 or 20 s.Useful temporal information extends beyond the local acoustic window and saturates near 5 s.Source: §4.3 · p. 3

4.4. Multi-Horizon and Operational Forecasting

  • The discrete-time hazard model produces coherent forecasts at all horizons from one predictor, while independent cumulative outputs violate temporal ordering on 68.9% of frames.It is competitive with five separately trained binary models.Source: Table 3 · p. 4, §4.4 · p. 4
  • At a 25% alarm duty cycle, the Conformer anticipates 58.6% of scorable events; at a higher-recall point, it anticipates 91.6% at 66.5% duty.At the validation-selected maximum-F1 threshold, it anticipates 43.9% of events with 196 ms mean lead time and 27.6 false warning bouts per minute.Source: §4.4 · p. 4

5. RELATED WORK

Prior work has examined detection, segment-level prediction, and other signals associated with upcoming dysfluency. This paper’s setting differs by forecasting annotated onsets continuously from preceding audio and evaluating event coverage under an alarm budget.

Pre-onset evidence and prediction

  • Liu and Howell and Warner et al. test whether pre-onset material predicts stuttering, but neither builds an acoustic predictor.The paper situates forecasting in evidence about stuttering distribution, self-reported anticipation, and pre-onset neural and auditory differences.Source: §5 · p. 4

Related audio prediction

  • Kozak predicts whether a three-second SEP-28k clip contains dysfluency, whereas this work forecasts exact annotated onsets every 100 ms at sub-second horizons and evaluates event coverage under an alarm duty-cycle budget.The earlier audio predictor is described as a compact CNN with signal concentrated on blocks and sound repetitions.Source: §5 · p. 4

Related methodological precedents

  • The study evaluates event-level sensitivity against time spent in warning, following seizure-prediction practice, and its multi-horizon head uses a discrete-time survival formulation.Source: §5 · p. 4

6. CONCLUSION

The paper formulates stuttering anticipation as continuous causal time-to-event forecasting and builds an onset-aligned corpus layer and benchmark framework. Its findings identify measurable pre-onset structure, advantages for modeling temporal evolution, and coherent multi-horizon forecasts, while operational trade-offs remain an open challenge.

Conclusion

  • The work formulates continuous causal time-to-event forecasting, constructs an onset-aligned corpus layer, and establishes a benchmark framework for pre-onset risk trajectories.Source: §6 · p. 4
  • The findings indicate statistically measurable predictive structure before stuttering onset and gains from modeling sequential contextualized speech states rather than expanding static acoustic context.Source: §6 · p. 4
  • A discrete-time survival formulation produces coherent multi-horizon forecasts without violating temporal ordering, but operational trade-offs leave pre-onset forecasting an open challenge.Source: §6 · p. 4

7. LIMITATIONS AND FUTURE WORK

The primary constraint is annotation scale: precise frame-level onset timestamps are required, but existing segment-level corpora do not provide them. The 2,872-event layer across 36 speakers is a first step; larger, more diverse onset-aligned cohorts remain necessary.

Limitations and Future Work

  • The primary constraint is annotation scale: precise frame-level onset timestamps are needed, while existing segment-level corpora do not provide them.The augmented layer contains 2,872 events across 36 speakers; scaling annotations to larger, more diverse cohorts is identified as essential for fine-grained anticipatory modeling.Source: §7 · p. 4

8. COMPLIANCE WITH ETHICAL STANDARDS

The work uses the publicly available FluencyBank Timestamped dataset under its existing license and positions its models as research toward supervised assistive tools, not autonomous systems.

8. COMPLIANCE WITH ETHICAL STANDARDS

  • The study uses the publicly available FluencyBank Timestamped dataset under its existing license.Source: §8 · p. 4
  • The augmented annotations contain no additional personally identifiable information beyond what FluencyBank Timestamped already provides.Source: §8 · p. 4
  • The forecasting models are intended as a research step toward clinician- or user-supervised assistive tools, not as standalone diagnostic or autonomous intervention systems.Source: §8 · p. 4

Related papers

Loading 2610.19235v1…