The Trade-off Was in the Labels: Causal Supervision for Turn-Aware Streaming ASR

Bojie Li; Noah Shi

Source-linked AI summary

The Trade-off Was in the Labels: Causal Supervision for Turn-Aware Streaming ASR

Bojie Li, Noah Shi

arXiv:2609.04225v1Jul 12, 2026eess.AScs.SD

TL;DR

Voice agents need to distinguish internal pauses from completed turns, but silence thresholds cannot do so reliably. This paper introduces a causal-labeling recipe for a small turn-aware streaming ASR model that jointly transcribes, endpoint-detects, handles dictation, and uses context. The resulting system reaches a deployment-matched operating point unavailable to silence timeouts, while counterfactual supervision also reduces contextual copying.

Problem
Silence thresholds cannot distinguish within-turn pauses from between-turn gaps, while existing open systems lack a complete disclosed training recipe for turn-aware streaming ASR.
Method
A LoRA adapter on Qwen3-ASR-0.6B is trained with labels computable from information available at each streaming decision, plus counterfactual constructions for temporal and contextual supervision.
Results
0.10 to 1.00 fire recall after adding one second of silence exposed clairvoyant-label failure, while the causal recipe produced a checkpoint dominating eight predecessors and reduced contextual intrusion below 1%.
Takeaways & Limitations
Streaming supervision should prevent targets from depending on future or untrustworthy signals, using counterfactual data constructions to make decisions causally learnable.
Takeaways & Limitations
The evidence is limited to single-speaker, single-channel English, with endpointing evaluated on held-out AMI meetings and longer-than-trained dictation remaining unresolved.

Abstract

from arXiv · show

A voice agent must decide, moment to moment, whether the user has finished; silence rarely settles it: a caller reading a phone number pauses mid-digits, a one-word "Stop!" ends a turn, a long question carries pauses longer than real turn-gaps. A voice-activity detector plus a silence timeout (the deployed default) cannot separate these, because within-turn pauses routinely exceed between-turn gaps; what distinguishes them is whether the words so far form a complete thought: what a recognizer computes to produce a transcript. We present the first open training recipe and benchmark for turn-aware streaming ASR: a small LoRA adapter on Qwen3-ASR-0.6B, trained in hours on one GPU, that transcribes, detects end-of-turn from meaning and silence, handles dictation, and grounds transcription in context. On a deployment-matched benchmark it reaches 0.97 boundary recall at 0.39 s median latency with 0.3 false fires per speech-minute, replicated on a fresh test set; no silence timeout reaches this point. The recipe rests on one principle: every streaming-decision label must be computable from input up to the decision point. Offline corpora violate it, encoding the future; such clairvoyant labels manufactured oscillation and a phantom recall-versus-precision trade-off, exposed when one appended second of silence raised a "broken" model's end-of-turn recall from 0.10 to 1.00. The same leak recurred with context: an always-matching biasing prefix became a copied shortcut (40% intrusion), and counterfactuals disagreeing with the audio cut this to 0.8% while keeping most of a +28.9 pp entity-recall benefit.

1 Introduction

The paper introduces an open recipe for turn-aware streaming ASR and argues that causal labels—not architectural complexity—determine reliable streaming decisions. A single small model handles transcription, endpointing, dictation, and context grounding while counterfactual supervision removes misleading temporal and contextual shortcuts.

Recipe and system: A single LoRA-adapted recognizer transcribes incrementally, emits end-of-turn markers, holds through dictation pauses, and grounds transcription in context.The system uses Qwen3-ASR-0.6B and roughly 20k synthesized examples, trained in hours on one GPU.
Causal supervision: Clairvoyant labels use post-decision audio or future speaker information unavailable to streaming models, thereby encoding the future into supervision.The paper identifies this leakage as the central training problem for streaming endpointing.
Causal supervision: 0.10 to 1.00 fire recall after appending one second of silence exposed the apparent recall-versus-precision frontier as a labeling artifact.A causal rule based on semantic completeness and at least 0.3 s of silence, combined with minimal pairs, produced a checkpoint dominating eight predecessors.
Generalization: 40% wrong-entity intrusion under matched-only context training fell below 1% with counterfactual audio-context disagreements while preserving the biasing benefit.The same construction addresses contextual leakage as temporal leakage.
Artifacts: The released artifacts include the recipe, code, probes, and benchmark for a unified turn-aware streaming ASR checkpoint.The paper presents these as reusable research artifacts rather than merely a production system.

2 Problem formulation and system overview

The problem is that silence duration cannot reliably identify turn boundaries because dictated numbers and long questions contain substantial internal pauses. The proposed system therefore makes endpointing part of the recognizer’s chunk-wise decode, using transcript meaning, silence, context, and marker tokens rather than a downstream timeout alone.

Problem formulation: Dictated numbers and long questions require holding through silence, whereas completed utterances require firing almost immediately.These opposing cases make a single silence threshold inadequate.
Problem formulation: Within-turn pauses and between-turn gaps overlap almost completely, so endpointing depends on whether the words so far form a complete thought.That completeness signal is computed during recognition.
System overview: Audio is processed in 0.5 s chunks, with a context prefix optionally kept visible throughout the session.The system outputs transcript segments and end events after an energy gate.
System overview: The recognizer re-reads recent audio and emitted words at each chunk, then may emit an end-of-turn token in the same decode as the transcript.This replaces a separate detector plus downstream silence timer.
Implementation: The model adds end-of-turn markers to Qwen3-ASR-0.6B through a lightweight LoRA adapter.Rank 16 is used for endpointing experiments and rank 32 for the full model.

3 Causal supervision

Streaming supervision must use only information available at the decision point: labels that depend on future audio create contradictory training targets and false trade-offs. Causal labels and counterfactual pairs teach one model to combine semantic completeness, silence, dictation structure, and trustworthy context.

Causality principle: A streaming label is clairvoyant when its value depends on input after decision time t, violating causal supervision.
Causality principle: Clairvoyant labels split identical observable prefixes into conflicting targets, producing oscillating training and an illusory recall–precision frontier.
End-of-turn supervision: A causal rule fires only when the words so far are semantically complete and at least 0.3 s of silence has been observed.
End-of-turn supervision: Minimal pairs vary silence or preceding completeness independently, preventing completeness-only fires and teaching the model to hold through incomplete pauses.
Dictation: Dictation supervision holds during partial phone numbers and fires after complete numbers plus silence, reducing five mid-number interruptions to one fire at +0.42 s.
Context biasing: Matching-only context training copies a profile, whereas counterfactual disagreements make the target follow audio and remove the shortcut.

4 Evaluation protocol

Evaluation replays continuous, deployment-like audio through the live streaming stack rather than forced-alignment clips. It uses causal endpointing labels and dedicated dictation and entity probes, with comparisons gated to avoid uninterpretable silence hallucinations.

Endpointing benchmark: Streaming replay delivers recorded audio chunk by chunk exactly as deployment would, while forced-alignment clip endings are excluded because they misdirect development.
Endpointing benchmark: Endpointing uses continuous 30–60 s single-channel AMI stretches with utterances at true timeline offsets and intervening silence.
Endpointing benchmark: All endpointing comparisons run behind an energy gate because speech-initial models hallucinate markers on silence at roughly two fires per second.
Dedicated probes: The dictation probe scores premature fires, final-boundary recall, and digit accuracy on replayed ten-digit phone numbers with inter-group pauses.
Endpointing benchmark: The main endpointing table reports development results on 25 stretches and confirmation results from a fresh twice-larger set collected after development ended.
Dedicated probes: The spelled-entity probe measures exact match and wrong-profile intrusion with user-profile prefixes, including wrong-user distractors, on enlarged post-development sets.

5 Results

The causal model outperforms timeout-based and prior supervision approaches across endpointing, dictation, and context-biasing probes. Its remaining costs are a breadth–precision trade-off and offline WER regression from narrow synthetic fine-tuning.

End-of-turn detection: The causal model beats every timeout setting on the deployment-matched benchmark, escaping the latency–false-fire curve by using utterance completeness.It requires only 3 fire-candidate suppressions versus 241 for the mixed-pools model.
End-of-turn detection: None of the evaluated external turn-aware systems reaches the causal model’s operating point on the common protocol.The comparison is as-shipped rather than retrained or tuned for the benchmark’s microphone conditions.
Context biasing: A relevant hotword prefix raises Earnings-22 entity recall from 66.7% to 95.6%, while the advantage remains flat after up to 12 s of unrelated speech.The prefix also nudges WER slightly down.
Context biasing and dictation: Counterfactual context training reduces wrong-profile intrusion to 0.8% while preserving dictation and matching-context accuracy.Matched-profile-only training instead produced copying, with intrusion increasing from 11.3% to 40% as training lengthened.
Unified model: The unified rank-32 adapter folds endpointing, dictation, and context grounding into one checkpoint without deployment-time checkpoint swapping.It is trained on roughly 20k examples and is evaluated on a held-out replay set.
Costs: 0.97 false fires per speech-minute versus 0.3 for the pure-endpointing checkpoint reveals a genuine breadth–precision cost.The endpoint fine-tune also adds 1.1/2.7 pp offline WER on LibriSpeech clean/other, while streaming WER is unaffected.

6 Analysis

The analysis shows that clairvoyant offline labels, not architecture or optimization instability, create the apparent endpointing trade-off. A silence-append intervention and controlled relabeling expose contradictory supervision as the mechanism behind oscillation and misranking.

Controlled record: Eight supervision compositions shared architecture, adapter, data volume, and source corpora, isolating labels as the load-bearing difference.The record was designed as a controlled comparison of supervision rather than model capacity.
Supervision record: 89 premature fires per speech-minute made the offline-clips composition unusable live despite perfect offline scoring.Adding successive pools moved models along an apparent recall–precision frontier without resolving it.
Diagnosis by intervention: Fire recall rose from 0.10 to 1.00 after appending 1.0 s of silence to the opposed-pools model’s evaluation clips.The mixed-pools model rose from 0.82 to 0.98, while the offline-labeled control was unchanged.
Diagnosis by intervention: Forced-alignment segmentation makes offline benchmarks demand fires on audio conditions that deployment never presents.Streaming pools instead teach the model to wait until silence is observed.
Label contradiction: At the decision point, disfluent pauses and turn boundaries have nearly indistinguishable silence-gap distributions, while their labels depend on future speaker information.This makes the historical distinction clairvoyant for streaming decisions.
Why oscillation: Opposed-pools training oscillates between fire and no-fire modes because disjoint, identically distributed pools are separately learnable but incompatible in aggregate.With identical audio receiving opposite labels, the model instead settles deterministically on one side.
Evaluation consequences: The legacy offline benchmark inverted live model rankings, making its offline champion the worst live model because of premature firing.The deployment-matched protocol re-ranked the supervision compositions.
Synthetic replication: A synthetic transformer reproduces the same anti-correlated fire–hold behavior as the fraction of clairvoyant labels increases.At zero clairvoyant-label fraction, training is monotone; mode flipping begins only at high fractions.

7 Discussion

The discussion turns the diagnosis into a supervision checklist and identifies scope boundaries for the prototype. It also distinguishes the real breadth–precision and generalization costs from artifacts caused by non-causal labels.

Supervision checklist: Streaming labels should be computable from input up to the decision time, with twin, phantom-condition, and oscillation tests checking causal consistency.Minimal pairs should differ in an observable rather than assigning different labels to causally identical prefixes.
Scope boundaries: The evidence is limited to single-speaker, single-channel English and conversational endpointing on held-out AMI meetings.Cross-corpus generalization and multispeaker mixed-channel audio remain future work.
Scope boundaries: Dictation transfers to shorter enumerations but fails beyond the trained length, firing at a learned ten-digit completeness point on 16-digit card numbers.Transcription remains unaffected, while the completeness judge does not extrapolate reliably to longer patterns.
Scope boundaries: Sub-three-word utterances are marked incomplete, so genuine one-word commands such as “Stop!” are not guaranteed near-instant firing.Distinguishing imperatives from acknowledgments remains the same learned-completeness problem in a sharper form.
Prototype boundary: The checkpoint is a research prototype trained on roughly 20k synthesized examples on one GPU, not a production-ready system.Scaling the recipe to production-grade data is left to future work.

8 Related work

Prior work covers endpointing, turn-taking, streaming ASR, and context biasing, but the paper identifies a missing open training artifact. Its contribution is a unified recipe plus evidence that some reported trade-offs arise from non-causal supervision.

Endpointing and end-of-turn detection: Existing endpointing systems use VAD silence thresholds or learned acoustic and recognizer-integrated signals, but published training details are absent.The paper positions its recipe as the missing open artifact.
Turn-taking: Turn-taking research includes text end-of-turn prediction and future-window voice-activity projection, while this paper diagnoses future-dependent labels used for streaming decisions.The cited projection objectives are described as causally honest because they predict future distributions.
Streaming and unified ASR: The paper inherits unified streaming-ASR substrates and adds an in-transcript turn event together with a training recipe.Related systems span RNN-T, weakly supervised models, decoder-only streaming, and unified speech LLMs.
Context biasing: Context-biasing research ranges from attention-based deep context to retrieval-scale, reinforcement-tuned, and cue-based methods.This work contributes plain prompt-slot measurements for the open model class.

9 Conclusion

The paper argues that streaming decisions must be computed from observable inputs and demonstrates this with a small recognizer that jointly transcribes, detects turns, handles dictation, and grounds context.

Conclusion: A causal labeling rule, two pair constructions, and a deployment-matched benchmark turn a small open model into a unified turn-aware streaming ASR system.The system transcribes, emits end-of-turn events, handles dictation, and grounds transcription in context.
Conclusion: Offline corpora encode future information, producing clairvoyant labels that manufacture oscillation and phantom recall-versus-precision trade-offs.The paper identifies supervision, rather than architecture, as the central obstacle.
Conclusion: Counterfactual construction cures the same leakage pattern for both temporal endpointing and contextual grounding.The temporal fix contrasts observable silence tails, while the contextual fix contrasts matching and disagreeing context.

B Benchmark specification and diagnostics

The benchmark uses continuous replay, observable-silence minimal pairs, and gated diagnostics to expose failures that legacy offline evaluation can hide.

Benchmark protocol: 0.5 s chunks and committed-prefix decoding evaluate fires at chunk boundaries on continuous streaming audio rather than clips ending at speech boundaries.The protocol uses a deployment-oriented replay setup with explicit recall windows and development, confirmation, and held-out sets.
Label diagnostics: Each complete utterance is paired with and without a 0.3–1.2 s observable silence tail, changing the target from transcript only to transcript plus marker.The construction isolates silence as the feature that should distinguish hold from fire.
Diagnostics: r=0.997 links ungated spurious fires to silence content, making ungated recall uninterpretable and motivating the energy gate.Speech-initial training examples hallucinate markers during silence at approximately 1.9 fires per second of silence.
Benchmark validity: The offline champion becomes the worst live model, firing 89 times per speech-minute when audio continues flowing.The opposed-pools model, despite only 10% offline recall, becomes the best prior endpointer on the replay benchmark.
Baselines: Table 6 distinguishes separate turn detectors, cascades, and the deployed VAD-plus-timeout baseline in the comparison landscape.This taxonomy frames the benchmark against both modular and cascade systems.

D Additional results

Additional evaluations show that the causal recipe remains deployable across horizons, datasets, seeds, probes, and ablations, while exposing specific accuracy and robustness boundaries.

Confirm-horizon sweep: h=0 is already deployable for the causal model, whereas h=2 over-suppresses fires and pushes mixed-pools recall to 0.812 beyond the latency budget.A confirm horizon can trade latency and false fires, but the causal checkpoint needs no confirmation horizon to be deployable.
Timeout comparison: The timeout family never reaches the causal model’s recall, latency, and false-fire point; matching recall costs 3× latency and approximately 5× false fires.This comparison holds across development and fresh confirmation sets.
Fresh-set confirmation: The fresh confirmation set reaches 0.924 recall, 0.42 s P50 latency, 0.70 s P95 latency, and 0.21 false fires per speech-minute.The recall difference from development is consistent with sampling variation because the 95% interval includes zero.
Probe results: The enlarged probes report 0.88 final recall, 0.93 email-with-context, 0.008 intrusion, and 0.982 boundary recall on the 100-stretch replay set.These results include confidence intervals for dictation, context, intrusion, and replay endpointing.
Dictation: Re-scoring dictation gives 0.968 on-time final recall, zero late fires, and 0.07 genuine mid-sequence fires per number.The residual shipped-window penalty is eagerness within one chunk of the true speech end rather than lateness.
Unified model: At matched latency, the unified checkpoint reaches 0.97 boundary recall with 0.3 false fires per speech-minute, while a one-chunk horizon yields 0.958 recall and zero false fires.The latter setting favors conservatism over latency.
Seed robustness: Three-seed retraining keeps premature-fire, context, and intrusion behavior stable, supporting recipe-level rather than checkpoint-level robustness.The unified capability remains stable under independent seed retraining.
Ablations: Removing the minimal pair raises gated false fires from 1.08 to 5.28 per speech-minute, while removing silence schemas produces 1458 ungated silence markers versus 8.The ablation reproduces the predicted dependence on the pair constructions and silence competence.

F The synthetic clairvoyant-fraction experiment

A synthetic stream experiment varies the fraction of clairvoyant labels while keeping the holdout causal, showing a graded transition from stable learning to anti-correlated behavior and mode flips.

Experimental setup: At f=0, both classes reach 1.00 accuracy with zero mode flips and worst post-warmup holdout accuracy of 0.99.The causal rule fires based on phrase completeness, while the clairvoyant rule uses future gap length.
Dose response: The fire/hold accuracy anti-correlation grows from −0.3 at f=0.1 to −1.0 at f=1.0.Completeness and gap length are independent, so the two labeling rules demand opposing behavior on identical prefixes.

G Training-recipe notes

The recipe notes isolate training, optimization, and serving choices that preserve streaming behavior while exposing where offline metrics and deployment assumptions can mislead. Key interventions include protected marker rows, replay data, cosine decay, AdamW over Muon, bounded re-feeding, and force-flushed resident state.

Optimization hygiene: Marker-row weight decay silently shrinks the frozen vocabulary and output head, so the recipe assigns those parameters a zero-decay group.At η=2×10−4 and λ=0.01, the tied 151k-row matrix shrank ∼2.4% over 12k steps.
Optimization hygiene: The weight-decay fix did not improve offline WER: the fixed unified checkpoint scored 3.9%/7.3% versus 3.6%/7.0% without it.The text attributes the regression to narrow-schema drift from fine-tuning on ∼12k examples without plain-ASR replay.
Data mixture: Plain-ASR replay recovered offline WER from 3.9%/7.3% to 3.5%/6.9% and reduced spurious offline marker fires from ∼10–20% to 3.2%.The unified model mixes ∼4k plain transcription examples into the ∼20k-example pool.
Optimization: Cosine decay produced the most stable, highest holdout-composite trajectory, reaching a plateau of 0.992 from step 6k.The schedule decayed from 2×10−4 to 10−5 after a 100-step warmup.
Optimization: Muon plateaued at holdout composite 0.846 versus AdamW’s 0.985, with conversational schema collapsing despite faster early raw-loss descent.The comparison used identical data and a 100× larger Muon base learning rate.
Serving: Serving requires bounded re-feeding and force-flushing: appending each chunk as a new placeholder caused 88% WER, while bounded re-feeding achieved 3.8%.A 20 s force-flush fixes WER and compute tails, while P95 remains flat through 8 concurrent sessions on a 0.15-GPU slice.