Source-linked AI summary
Low-Latency State Space Voice Activity Detection with Robust Onset Time Evaluation
Elad Cohen, Arnon Netzer, Hai Victor Habi
TL;DR
VAD classification metrics do not capture onset timing, and noisy annotations can destabilize latency estimates. The paper models annotation uncertainty and introduces S4VAD, reporting the lowest onset latency with competitive AUROC.
Problem
Classification metrics such as AUROC do not capture VAD detection timing, while onset-label noise can make latency measurements unstable.
Source: §I · p. 1Method
The paper estimates annotation-noise statistics to correct onset-latency measurements and introduces S4VAD, an SSM-based VAD for efficient streaming inference.
Source: §I · pp. 1–2, §III · pp. 2–3Results
S4VAD achieves the lowest onset latency while maintaining competitive AUROC among the evaluated low-latency VAD models.
Source: §I · p. 2, §VI · p. 6Takeaways & Limitations
The approach provides more stable onset-latency estimates under annotation jitter, while S4VAD combines long-range context with fast local dynamics for streaming speech detection.
Source: §I · p. 1, §VI · p. 6
Abstract
from arXiv · showhide
Voice Activity Detection (VAD) systems are commonly evaluated using metrics such as the area under the receiver operating characteristic curve (AUROC), but these metrics do not account for temporal responsiveness. For low-latency applications, however, accurately measuring speech onset delay is essential. This is particularly challenging because onset latency evaluation is affected by noise and systematic misalignment in annotation timestamps. In this work, we introduce a probabilistic framework for evaluating VAD onset time under noisy temporal labels. We model annotated onset times as noisy observations of latent acoustic onsets and estimate the resulting discrepancy distribution from data. We show that this approach provides a more stable and robust estimate of algorithmic latency. In addition, we introduce S4VAD, the first state-space-model-based (SSM-based) VAD architecture. S4VAD supports efficient streaming inference while directly controlling the decay of past acoustic evidence, enabling fast and responsive speech-onset detection. We evaluate VAD onset latency across several low-latency architectures, including CNN, Transformer, and RNN models. Our proposed VAD achieves the lowest latency while maintaining a competitive AUROC.
I. INTRODUCTION
Conventional VAD classification metrics do not capture onset timing, while noisy and misaligned annotations make latency estimates unstable. The paper proposes uncertainty-aware onset evaluation and S4VAD, reporting the lowest onset latency with competitive AUROC.
I. INTRODUCTION
- Classification metrics such as AUROC are largely insensitive to detection timing, although onset latency affects streaming responsiveness and downstream behavior.Algorithmic latency is typically measured from an annotated speech onset to the first VAD frame that indicates speech.Source: §I · p. 1
- Manual re-annotation reveals onset-label jitters of tens to hundreds of milliseconds, making naive latency measurements unstable and potentially misleading.Tolerance windows used in related areas reduce penalties for small shifts but mask boundary errors rather than explicitly estimate delay.Source: §I · p. 1
- The proposed evaluation models annotated onsets as noisy observations of true acoustic onsets and uses EM-estimated noise statistics to correct naive latency estimates.The resulting estimates are described as more stable under realistic annotation jitter.Source: §I · pp. 1–2
- S4VAD is introduced as the first SSM-based VAD, combining efficient streaming inference with control over how quickly past acoustic evidence decays.This provides an architectural mechanism for trading long-context robustness against fast onset responsiveness.Source: §I · pp. 1–2
- S4VAD achieves the lowest onset latency while maintaining competitive AUROC across the evaluated low-latency VAD models.The comparison includes CNN-, RNN-, Transformer-, and SSM-based models.Source: §I · p. 2
II. BACKGROUND
Prior VAD work reports frame-level classification performance, while latency and temporal-label uncertainty remain distinct evaluation concerns. Existing tolerance regions address boundary uncertainty heuristically, and onset annotations themselves can be imprecise.
A. Related Work
- Prior VAD approaches include classical signal-processing methods and neural CNN, RNN, and Transformer models, but AUROC and EER do not capture temporal responsiveness.Latency-sensitive speech work has focused mainly on endpointing, and tolerance regions address boundary uncertainty without explicitly modeling annotation noise.Source: A. Related Work · p. 2
B. Onset Time Uncertainty
- Onset annotations vary with human labeling, dataset resolution, and alignment procedures, with reported boundary deviations ranging from tens to hundreds of milliseconds.AVA-Speech provides 1 Hz annotations; LibriSpeech timestamps are typically forced-aligned, while manually aligned corpora still show non-negligible deviations.Source: B. Onset Time Uncertainty · p. 2
- The paper motivates estimating onset time by jointly modeling the true onset and annotation-noise distribution rather than treating noisy timestamps as exact.This framework targets temporal uncertainty that tolerance regions handle through heuristic windows.Source: B. Onset Time Uncertainty · p. 2, A. Related Work · p. 2
C. State Space Models (SSM)
State-space models represent sequential data with explicit latent temporal memory. Their dynamics govern how past inputs affect future outputs, with discretization making the memory evolution explicit.
C. State Space Models (SSM)
- A linear state-space model represents sequential data through a latent dynamical system with explicit temporal memory.In continuous time, its state and output are defined from an input signal.Source: C. State Space Models (SSM) · p. 2
- The state matrix A governs state dynamics, while B and C control input coupling and output readout; negative-real-part eigenvalues of A yield stable, exponentially decaying memory.The latent state x(t) and output y(t) are driven by input u(t).Source: C. State Space Models (SSM) · p. 2
- Discretization with step size ∆ makes temporal memory explicit: past inputs affect future outputs through the latent state, with decay determined by the discrete state matrix.The discrete formulation uses ¯A = exp(∆A).Source: C. State Space Models (SSM) · p. 2
III. ONSET TIME EVALUATION
The method estimates VAD onset bias from noisy onset labels by modeling annotation error rather than treating timestamps as exact. It combines the corrected bias with hardware inference latency to define total onset latency.
III. ONSET TIME EVALUATION
- Observed annotation time t is modeled as a noisy observation of latent true onset t*, with annotator error δ.The input also includes audio segment x around the annotated onset.Source: §III · p. 2
- The estimated onset is derived from target VAD scores under a bounded-noise assumption, using threshold τ over interval T.The noise distribution π(δ) is assumed to have bounded support [δL, δR].Source: §III · p. 3
- Algorithmic latency is defined as onset bias: the average delay between a VAD decision and the true speech onset.Source: §III · p. 3
- The corrected onset bias B is combined with hardware inference latency because a streaming decision follows both model delay and computation time.The method estimates dataset-level B by learning one global noise model with EM and a reference VAD, then evaluating the dataset with that model.Source: §III · p. 3
A. Annotator Noise Distribution Estimation
The method estimates a dataset-level annotation-noise distribution with EM, using a reference VAD to help infer onset-label error. The estimate can also reflect reference-model bias, which is not separately identifiable without additional supervision.
A. Annotator Noise Distribution Estimation
- The EM procedure estimates annotation noise from noisy labels and reference VAD scores to obtain a robust onset-bias estimate.The estimated distribution may capture both annotation noise and systematic reference-VAD bias.Source: A. Annotator Noise Distribution… · p. 3
- The E-step estimates posterior annotator error using the observed annotation, audio, and noise model, with an independence assumption and a hazard-style likelihood approximation.The likelihood approximation uses the activity score of a well-trained reference VAD.Source: A. Annotator Noise Distribution… · p. 3
- The M-step maximizes the expected objective subject to π being a valid distribution.Solving the constrained optimization yields the update, after which the E- and M-steps are alternated from a uniform initialization.Source: A. Annotator Noise Distribution… · p. 3
- Annotation noise and systematic reference-VAD bias are not separately identifiable without additional supervision, so residual reference-dependent bias may remain.The authors interpret the estimated shift primarily as annotation noise because they expect a high-quality reference VAD to have relatively small systematic bias.Source: A. Annotator Noise Distribution… · p. 3
- The estimated EM noise distributions closely match the clean-subset shape, and negative δ indicates annotators tend to label onset early.Source: A. Annotator Noise Distribution… · p. 3, Figure 2 · p. 4
B. Annotation Noise Analysis
The analysis estimates dataset-level annotation shifts and compares the inferred noise distributions with manually identified onsets. The results support using a constant dataset-level correction rather than per-sample adjustments.
B. Annotation Noise Analysis
- Mean errors were 5 ms for LibriSpeech and 13 ms for AVA-Speech, and estimated distributions closely matched those from manually identified onsets.The comparison used 100 noisy onset instances from each dataset and five EM iterations.Source: B. Annotation Noise Analysis · p. 4
- The noise is sharply concentrated around its mean, suggesting dataset-level bias dominates sample-specific variability and favoring a constant correction.The authors note that per-sample adjustments may introduce additional variance.Source: B. Annotation Noise Analysis · p. 4
- µδ = −70 ms for LibriSpeech and −136 ms for AVA-Speech, the estimated dataset-level annotation shifts.Source: B. Annotation Noise Analysis · p. 4
- Noisy labels anchor evaluation to the true onset event, while the reference VAD acts as a weak local observer rather than a standalone latency comparator.A direct target-to-reference comparison would measure relative latency and inherit sample-level reference errors.Source: B. Annotation Noise Analysis · p. 4
IV. SSM-BASED VAD
S4VAD uses a diagonal-structured state-space architecture to model acoustic sequences, with explicit control over memory timescales and support for streaming inference.
IV. SSM-BASED VAD
- S4VAD uses the S4D variant, whose diagonal complex-valued state-transition matrix supports modeling short- and long-term dependencies.Each state dimension updates independently, avoiding cross-channel mixing in the state dynamics.Source: §IV · p. 4
- Larger discretization steps make past information decay faster for more local, responsive dynamics, while smaller steps preserve longer context.The step controls the hidden-state memory timescale.Source: §IV · p. 4
- S4VAD supports efficient convolutional training and streaming inference through an equivalent recurrent state update.The implementation uses four S4 blocks with hidden dimension 64 and supports frame-by-frame processing with constant memory and compute.Source: §IV · pp. 4–5
V. EXPERIMENTAL RESULTS
The experimental results section covers implementation details, a comparison of low-latency VAD models, and an ablation study.
A. Implementation Details
The experiments use synthetic noisy LibriSpeech mixtures and AVA-Speech, with specified preprocessing, training, and evaluation procedures.
A. Implementation Details
- The synthetic dataset combines LibriSpeech speech with diverse DNS Challenge noise, producing transitions from background noise to noisy speech.Noise splits use an 8:1:1 ratio, and the original LibriSpeech splits are preserved; the training subset contains 100 hours.Source: A. Implementation Details · p. 5
- Audio is sampled at 16 kHz and converted to 64-dimensional log-Mel features using 32 ms frames and a 10 ms hop.Models train on synthetic mixtures across −5 to 20 dB SNR, with white-noise augmentation, binary cross-entropy, and AdamW for 100 epochs.Source: A. Implementation Details · p. 5
- Evaluation uses synthetic LibriSpeech mixtures at −10 to 10 dB SNR in 5 dB steps and also includes AVA-Speech.Onset bias uses T = 50 ms; thresholds target validation-set FPR, and hardware latency is measured on an Apple M4 Max using XNNPACK.Source: A. Implementation Details · p. 5
B. Low-Latency VADs Comparison
Across low-latency VAD models, S4VAD has the lowest onset latency while retaining competitive AUROC; the section also examines annotation-noise sensitivity and a discretization-step ablation.
B. Low-Latency VADs Comparison
- The proposed onset-bias method remains stable and tracks noise-label shifts, whereas the naive method is strongly affected.Figure 3 compares the naive and proposed onset biases under added Gaussian label noise.Source: B. Low-Latency VADs Comparison · p. 6, Figure 3 · p. 6
- The section includes an ablation study of the discretization step.Source: Table III · p. 6
- S4VAD achieves the lowest onset latency while remaining competitive in AUROC across the compared low-latency VAD models.The comparison includes MarbleNet, TrVAD, ResectNet, and S4VAD, evaluated at a target FPR of 3%.Source: B. Low-Latency VADs Comparison · pp. 5–6
- The model comparisons discuss how architecture relates to latency: fixed-window aggregation can delay MarbleNet, while TrVAD has higher hardware latency.ResectNet has low latency on LibriSpeech but higher latency on AVA-Speech; S4VAD combines a compact recurrent state with fast local dynamics.Source: B. Low-Latency VADs Comparison · p. 6
C. Ablations
The ablations show that S4VAD has the lowest onset bias across FPR operating points, while its discretization timescale controls the AUROC–onset-bias tradeoff; the conclusion reports the fastest onset latency with competitive AUROC.
C. Ablations
- S4VAD exhibits the lowest onset bias across all FPR operating points.The figure compares onset bias with FPR on LibriSpeech.Source: Figure 4 · p. 6, C. Ablations · p. 6
- dtmin = 0.6 provides the lowest onset bias while maintaining competitive AUROC; both overly short and long memory timescales can degrade onset responsiveness.Changing dtmin changes the memory timescale and produces a systematic AUROC–onset-bias tradeoff.Source: C. Ablations · p. 6
- Under Gaussian perturbations of onset labels, naive latency is strongly affected by noise, whereas the proposed method adapts to its distribution for more stable estimates.The comparison evaluates onset bias using perturbed labels.Source: C. Ablations · p. 6
VI. CONCLUSIONS
- S4VAD preserves long-range temporal context while maintaining fast local dynamics for responsive streaming speech detection.The conclusion also reports the fastest onset latency with competitive AUROC.Source: §VI · p. 6