Source-linked AI summary
Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
Sameep Chattopadhyay, Alexander Erdmann, Mari Ostendorf
TL;DR
Conversational streaming ASR is vulnerable to long silences and backchannels because limited memory can lose or corrupt useful context, while turn-level resets impair onset recognition. The paper introduces State Carry-over and State Rollback, and evaluates them across two models and two dialogue benchmarks, with the best method reducing onset WER by 15–21% relative to reset baselines.
Problem
Limited acoustic memory makes conversational streaming ASR vulnerable to long silences and backchannels, while resetting state at turns discards context and harms onset tokens.
Method
The paper compares Fresh Start, Continuous State, State Carry-over, and State Rollback, which selectively preserve or discard cross-utterance model state.
Results
15–21% relative WER reduction at turn onsets is achieved over standard state-resetting baselines across two streaming models and two telephonic dialogue datasets.
Takeaways & Limitations
Selective state management preserves useful prior context while remaining robust to conversational noise, with Rollback the best-performing strategy across evaluated models and datasets.
Takeaways & Limitations
Evaluation is limited mainly to American English telephonic conversations, and downstream voice-agent behavior is not directly measured.
Abstract
from arXiv · showhide
Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
1 Introduction
Streaming ASR in conversational agents must operate with limited acoustic memory, making long silences and backchannels harmful. Resetting state at turn boundaries avoids some contamination but discards context and can cause turn-onset errors.
- Streaming ASR models operate with only a few seconds of acoustic memory to meet real-time latency requirements.Conversational voice-agent pipelines target response delays below 500 ms, with ASR contributing no more than 200 ms.
- Long silences and backchannels can displace crucial prior-utterance information with task-irrelevant acoustic content.
- Resetting state at each turn provides a fresh start but loses useful prior context and can impair tokens at turn onsets.Early ASR errors may also affect downstream language-model processing.
- The study proposes state-management strategies that preserve cross-utterance context and evaluates them on two streaming models and two telephonic dialogue datasets.The reported outcome is a 15–21% relative WER reduction at turn onsets over state-resetting baselines.
2 Streaming ASR
Streaming ASR systems either carry state continuously or reset it at utterance boundaries, creating different failure modes for conversational speech. Prior context has been used through text history or trained audio-state transfer, whereas this paper targets inference-time state management.
- Conversational agents commonly reset model state at each utterance boundary, trading contamination avoidance for lost acoustic and linguistic context.
- Continuous streaming carries context across recordings but can accumulate silences and backchannels that corrupt later acoustic representations.
- Prior approaches inject text dialogue history into the prediction network or use prior audio context during encoder training.
- The paper’s approach differs from earlier cross-utterance methods by requiring neither training nor fine-tuning.
- FastConformer-RNNT enables streaming through fixed left-context attention, rolling key-value caches, right-context lookahead, and localized convolution caches.Its RNN-T decoder maintains linguistic memory in the prediction network’s hidden and cell states.
3 Study Design
The study evaluates state-management strategies for cache-aware streaming ASR on two models and two telephonic dialogue benchmarks. It compares resetting, continuous processing, carry-over, and rollback under real-time latency settings.
- Models: The experiments use pre-trained fastconformer-114m and nemotron-streaming-0.6b models sharing a cache-aware FastConformer-RNNT architecture.Their state includes encoder caches and decoder hypothesis-related components.
- Datasets: Evaluation uses CallHome and Switchboard test sets containing backchannels, extended silences, occasional crosstalk, and utterance-level speaker timestamps.Individual unmixed channels are used, and nearby same-speaker utterances are merged into turns when separated by less than 1 s of silence.
- State Management: Fresh Start resets state per turn, Continuous State streams without manipulation, State Carry-over retains the prior turn’s state, and State Rollback discards state after short turns.Rollback returns to the last substantial turn when the previous turn contains fewer than k tokens.
- State Management: The comparison includes four strategies, while Algorithm 1 formalizes Fresh Start, State Carry-over, and State Rollback as state-update rules.Continuous State is excluded because it streams through inter-turn silence without state manipulation.
- Evaluation: WER is measured with greedy batch-size-one decoding using 5.6 s of left context and 160 ms of right-context latency.Rollback thresholds are selected from k ∈ {1, 5, 10}, with k = 5 for fastconformer-114m and k = 10 for nemotron-streaming-0.6b.
- Resource Cost: State Carry-over adds no memory beyond standard streaming, whereas State Rollback stores one additional state of approximately 2.6 MB or 7.32 MB.These correspond to approximately 0.6% and 0.3% of the respective model resources described in the passage.
4 Results and Analysis
The experiments show that streaming ASR state management determines robustness to turn onsets, long pauses, and short preceding turns. State Rollback performs best overall by preserving useful context while discarding harmful context.
- Continuous State degrades after silences exceed the 5.6 s acoustic-memory length and after short turns that often correspond to backchannels.State Carry-over outperforms Continuous State across the tested conversational conditions.
- State Rollback achieves the lowest overall WER across all evaluated model-dataset pairs.It consistently outperforms Fresh Start and State Carry-over across the reported metrics.
- Fresh Start improves robustness to long pauses and backchannels but incurs a substantial penalty on initial-word WER.This trade-off is particularly pronounced for fastconformer-114m.
- RB remains the best-performing strategy below 250 ms latency and continues to outperform existing approaches with VAD-derived boundaries.At 160 ms latency, fastconformer-114m RB outperforms Fresh Start at 560 ms, while relative WER increases remain below 6% across both models.
- State Carry-over outperforms Continuous State on both models and benchmarks, while its gains over Fresh Start mainly occur for fastconformer-114m.For nemotron-streaming-0.6b, utterances following backchannels can still degrade under State Carry-over, which State Rollback addresses selectively.
5 Conclusions
The study finds that resetting streaming ASR state at turn boundaries discards useful context and worsens onset recognition. State Carry-over and State Rollback preserve prior context while improving recognition, especially at turn onsets.
- State-resetting pipelines discard useful cross-utterance context, causing disproportionate degradation at utterance onsets.Early ASR errors may propagate to downstream language-model components.
- State Carry-over and State Rollback preserve prior context across turns while remaining robust to conversational noise.
- Experiments across two streaming ASR models and two conversational benchmarks show improved overall recognition accuracy, especially at turn onsets.
6 Limitations
The evaluation is bounded primarily by American English telephonic conversations, and the rollback threshold is empirically tuned. The study also covers only selected conversational failure modes and does not directly measure downstream voice-agent behavior.
- The benchmarks consist primarily of American English telephonic conversations, leaving generalization to other languages and settings open.
- State Rollback relies on an empirically tuned threshold that may benefit from dynamic adaptation to conversational context.
- The analysis focuses on state resets, long pauses, and backchannels rather than noisy acoustics, code-switching, or multi-speaker settings.
- The work improves ASR performance but does not directly evaluate intent prediction, dialogue quality, task success, or other downstream behavior.
A.1 Inference Setup
Inference uses fixed software, hardware, decoding, and deployment conditions to emulate real-world streaming evaluation. All results come from deterministic greedy decoding with batch size 1.
- All experiments use Python 3.10.19, NeMo ASR toolkit 2.7.0, Hugging Face checkpoints, and NVIDIA L40 GPUs.
- Evaluations use batch size 1 to emulate real-world streaming deployment conditions.
- Both streaming ASR models use deterministic greedy decoding, so every reported result comes from a single run.
A.2 Architecture Details
The experiments use FastConformer-RNNT streaming models whose state combines encoder caches and decoder variables. The architecture details include component shapes, memory footprints, and WER-reduction significance reporting.
- The FastConformer state includes attention KV and convolution caches plus a valid cache-length counter.
- The RNNT state contains the accumulated decoder hypothesis, last predicted token, and number of emitted tokens.
- Table 2 reports tensor shapes and memory footprints for the model-state components.
- Table 3 reports relative WER reduction from FS to RB and marks significance using MAPSSWE and per-utterance Wilcoxon tests.
A.3 Dataset Statistics
The evaluation uses two telephonic dialogue benchmarks with contrasting conversational styles and different test-set sizes.
- Switchboard contains telephone conversations between strangers discussing assigned topics, producing relatively structured dialogue with clearer turn-taking.
- CallHome contains conversations between friends and family, making its speech more spontaneous, informal, and conversationally dense.
- The CallHome test set comprises 40 channels and 1,335 turns, while Switchboard contains 214 channels and 5,822 turns.Each channel corresponds to approximately 5 minutes of conversational audio.
A.4 Statistical Significance Testing
The study uses statistical tests and multiple experimental settings to assess state-management strategies, while treating turn-onset analyses as the primary evidence.
- Statistical significance: MAPSSWE and utterance-level Wilcoxon signed-rank tests compare FS and RB using differently weighted WER differences.MAPSSWE weights utterances by reference-word count, whereas Wilcoxon gives each turn equal weight.
- Statistical significance: RB improvements are statistically significant under both tests, with p < 10^-3 for all but one result.For nemotron-streaming-0.6b on CallHome, MAPSSWE gives p < 0.05 while the per-utterance Wilcoxon test gives p < 10^-3.
- Statistical significance: The significance tests are secondary evidence because the primary contribution concerns recognition at turn onsets rather than overall utterance-level WER.They support that the strategies improve rather than degrade overall transcription quality.
- State rollback: RB selects a threshold k from {1, 5, 10} to discard short-turn context while preserving context from longer utterances.The selected configuration uses the best overall validation WER.
- Latency analysis: RB consistently outperforms alternative strategies across models, datasets, and voice-agent-compatible latency settings.Reducing latency from 560 ms to 160 ms causes less than a 6% overall-WER increase under RB for both models.
- VAD-derived turn boundaries: With VAD-derived timings, overall WER is consistently higher than with ground-truth timings, while strategy trends remain similar and the methods outperform existing strategies.The degradation is attributed to residual crosstalk leakage and imperfect VAD detection.