Source-linked AI summary
TurnFSM for Full-Duplex Dialogue System: Internalizing State-Machine Logic for Streaming Semantic Voice Activity Detection and Utterance-Level Rejection
Zhiwei Lin, Tianjiao Du, Qiaochu Huang, Zihan Zhang, Naijun Zheng, Longshuai Xiao, Yunfei Lu, Jun Chen, Zhiyong Wu
TL;DR
Full-duplex assistants need low-latency turn control while listening during speech, but existing end-to-end and cascaded approaches have capability or overhead limitations. TurnFSM internalizes control as serial finite-state transitions with first-order state dependency, and achieves competitive semantic VAD and rejection performance against task-specific models.
Problem
Full-duplex turn control must handle streaming semantic VAD and rejection, while existing end-to-end models may affect reasoning capabilities and cascaded pipelines add overhead and handcrafted logic.
Method
TurnFSM uses explicit finite-state transitions to serialize submission and rejection, with each state conditioned on the acoustic prefix and immediately preceding state.
Results
TurnFSM achieves performance comparable to corresponding single-task models on semantic VAD and rejection detection.
Takeaways & Limitations
TurnFSM provides a unified streaming controller that separates heterogeneous decisions across transition stages while supporting compact inference without historical state-token accumulation.
Abstract
from arXiv · showhide
Full-duplex voice assistants must continuously listen while speaking, handling user interruptions under low-latency and resource-constrained streaming conditions. Existing end-to-end full-duplex models can compromise reasoning-related capabilities after speech-domain adaptation, whereas cascaded pipelines introduce extra inference overhead and handcrafted control logic. We propose TurnFSM, an LLM-based state prediction framework that internalizes turn control as explicit finite-state transitions, unifying streaming semantic VAD and utterance-level rejection. TurnFSM decomposes submission and rejection into a serial decision process, reducing multi-task interference while maintaining performance comparable to single-task models. We further introduce a first-order state transition mechanism that enforces the dependency on only the previous state during training, enabling compact inference with the standard causal mask and original LLM positional encoding while avoiding historical state-token accumulation and unnecessary step-by-step state generation. Experimental results show that TurnFSM consistently outperforms the binary-head baseline and remains competitive with task-specific models.
I. INTRODUCTION
TurnFSM addresses full-duplex turn-control challenges by replacing parallel semantic VAD and rejection predictions with explicit serial state transitions. A first-order dependency on the previous state enables compact streaming inference while reducing control overhead and cross-task interference.
- Full-duplex assistants must perform reliable turn control under low-latency, resource-constrained streaming conditions, including segmentation, submission timing, and rejection.
- End-to-end full-duplex models may degrade reasoning-related capabilities after speech-domain adaptation, while cascaded pipelines add inference overhead, error propagation, and handcrafted coordination logic.
- Parallel semantic VAD and rejection heads face gradient conflicts because the tasks use different decision criteria and acoustic or semantic cues.
- TurnFSM serializes control through explicit states, determining semantic completeness during listening before evaluating rejection after submission.
- TurnFSM jointly supports streaming semantic VAD and utterance-level rejection within a unified state-transition framework.
- The first-order mechanism conditions each state on the acoustic prefix and immediately preceding state, avoiding historical state-token accumulation and unnecessary step-by-step generation.
II. METHODOLOGY
TurnFSM embeds cascaded turn-control logic in a learnable LLM state-transition model. Its serial decomposition assigns submission and rejection to separate stages, avoiding competition between heterogeneous task representations.
- TurnFSM replaces handcrafted control states with a learnable state-transition model embedded in the LLM.
- The model predicts each control state from streaming audio and the previous state within a unified state space.
- TurnFSM performs submission first and rejection detection second, assigning semantic completeness and rejection to different state stages.
A. Model Architecture
TurnFSM combines streaming audio encoding, an audio adapter, and an LLM backbone for state prediction. The architecture progressively converts audio into LLM-compatible tokens before prediction.
- TurnFSM consists of a streaming audio encoder, an audio adapter, and an LLM backbone.
- The audio encoder uses a two-layer VGG-style front-end with 4× downsampling followed by a 6-layer Conformer, producing acoustic frames at 25 Hz.
- A lightweight MLP audio adapter applies 2× downsampling and projects acoustic representations into the LLM input embedding space.
- The model uses Qwen2.5-7B-Instruct as its LLM backbone for streaming state prediction.
B. Finite-State Turn Control Formulation
TurnFSM models full-duplex turn control as explicit finite-state transitions rather than independent binary decisions. The sequence separates semantic completeness, submission, and utterance validity into distinct control stages.
- Baseline and motivation: The binary-head baseline predicts semantic completeness and input validity as separate submit/wait and accept/reject decisions.Semantic VAD determines whether the current prefix is complete, while rejection determines whether accumulated audio is a valid intent-bearing input.
- Finite-state formulation: TurnFSM maintains a discrete control state at each time step within a finite state machine.The formulation explicitly models the cascaded control process as state transitions.
- State definitions: The state set includes pseudo-initial, silence, listening, submission, acceptance, and rejection states.Silence monitors input without buffering, listening accumulates candidate utterances, and submission finalizes a segment for validation.
- Transition logic: TurnFSM transitions from listening to submission after semantic completeness, then from submission to acceptance or rejection based on utterance validity.This serial process assigns semantic completeness and rejection to different state stages.
C. First-Order State Transition Mechanism
The first-order mechanism trains each state prediction to use only the acoustic prefix and immediately preceding state. Position tying aligns interleaved training with compact inference using the standard causal mask and original positional encoding.
- Motivation: A standard causal mask can make each state prediction attend to the entire history of previous state tokens.This history-dependent implementation repeatedly accumulates earlier states and is unnecessary for the intended FSM dependency.
- Motivation: Historical state-token conditioning creates train–inference inconsistency by requiring intermediate state generation before final decisions.The issue applies even to non-streaming or utterance-level inference.
- Training mechanism: During training, a modified attention mask restricts St to the acoustic prefix A≤t and immediately preceding state St−1.Attention to earlier states S<t−1 is masked, enforcing the first-order dependency.
- Training mechanism: Position tying assigns each state token the positional encoding of the following audio token, preserving the original acoustic temporal order.The following audio token is only a positional reference, not an input for predicting the state.
- Inference mechanism: At inference, TurnFSM predicts St from (A0, . . . , At, St−1) with the standard lower-triangular causal mask and original LLM positional encoding.This avoids historical state-token accumulation and unnecessary step-by-step state generation.
A. Experimental Setup
TurnFSM uses separate cross-modal alignment and state-prediction training stages, supported by large open-source speech corpora and two internal control datasets.
- Training stages: TurnFSM is trained in two stages: cross-modal alignment followed by state prediction.The alignment stage uses ASR training before the control-state datasets are used.
- Cross-modal alignment: Cross-modal alignment uses AISHELL-1, AISHELL-2, and WenetSpeech, totaling 11,200 hours of audio.The bridging layer is trained on an ASR task.
- State prediction data: The semantic VAD dataset contains 70K training samples and 800 test samples totaling 194 training hours.Samples are evenly split between pauses and no pauses, with human-annotated acoustic start and end timestamps.
- State prediction data: The rejection-detection dataset contains 700K training samples and 3K test samples totaling 2,000 training hours.It has an even positive/negative split and covers a wide range of real-world scenarios.
B. Metircs and Baselines
The evaluation measures streaming semantic VAD timing and rejection detection, comparing TurnFSM with a binary-head alternative and task-specific semantic VAD models.
- Metrics: Semantic VAD evaluation reports early-cut rate, timeout rate, success rate, and mean latency.Overall mean latency includes successful and timeout cases, assigning timeout cases a fixed latency of 2000 ms.
- Metrics: Predictions within ±100 ms of the human-labeled endpoint are treated as correct rather than early cuts.The tolerance window accounts for noise in manual endpoint annotation.
- Metrics: Rejection detection is evaluated with false rejection rate and false acceptance rate.These metrics assess rejection and acceptance errors respectively.
- Baselines: TurnFSM is compared with a binary classification head and three open-source semantic VAD models: TEN Turn Detection, Easy Turn, and Smart Turn V3.21.The open-source baselines are offline models whose streaming latency is simulated using Silero VAD speech chunks.
- Baseline handling: Smart Turn receives probability-driven adaptive waiting, merging a new speech chunk when it arrives within the predicted waiting time.Otherwise, the current chunk is treated as semantically complete and submitted.
C. Main Result
TurnFSM performs competitively with single-task models on streaming and utterance-level semantic VAD while integrating rejection detection. It also outperforms the binary-head baseline and supports efficient whole-utterance inference.
- 3.35% FAR and 16.04% FRR keep TurnFSM close to the Rejection-only model on rejection detection.The Rejection-only model records 3.31% FAR and 16.16% FRR.
- TurnFSM outperforms the binary-head baseline on most metrics, including success rate, timeout rate, latency, and FRR.The shared-backbone binary-head design produces stronger multi-task coupling than the state-machine formulation.
- TurnFSM achieves the best overall balance among open-source semantic VAD baselines, with stable submission decisions and lower overall latency.Its 82.41% success rate is highest, while Easy Turn and Smart Turn exhibit trade-offs involving timeout or latency.
- The first-order transition mechanism enables TurnFSM to process a complete utterance in one forward pass without generating historical state tokens step by step.This compact inference form applies to utterance-level semantic completeness prediction.
- 91.00% overall accuracy places TurnFSM second on the EasyTurn test set, behind Easy Turn but ahead of TEN Turn Detection and Smart Turn V3.2.The compared systems score 97.00%, 87.99%, and 74.67%, respectively.
D. Ablation Study
Ablations show that FOSTM preserves task performance while reducing inference cost, whereas cross-modal alignment is important for semantic VAD but less influential for rejection detection.
- FOSTM: Removing FOSTM slightly lowers semantic VAD success and raises early-cut rate, while increasing successful-case latency from 80.9 ms to 111.35 ms.Rejection detection changes only slightly, with minor FAR and FRR differences.
- FOSTM: FOSTM preserves the original FSM’s modeling capability with stable performance across semantic VAD and rejection detection.
- FOSTM: FOSTM reduces redundant state-token accumulation and avoids unnecessary step-by-step state generation during inference.Training enforces pθ(St | A≤t, St−1), enabling compact inference with the standard causal mask.
- Cross-modal alignment: Removing cross-modal alignment increases semantic VAD early-cut rate from 12.06% to 15.83%.Alignment helps audio representations remain consistent with semantic content for completeness detection.
- Cross-modal alignment: Cross-modal alignment has limited rejection-detection impact, with FRR increasing from 16.04% to 17.10% after removal.The FAR slightly decreases when alignment is removed.
IV. CONCLUSION
TurnFSM unifies semantic VAD and utterance-level rejection through serial finite-state transitions and uses first-order state prediction for compact inference. It achieves strong performance, improves over the binary-head baseline, and remains competitive with task-specific models.
- TurnFSM unifies semantic VAD and utterance-level rejection by transforming parallel multi-task prediction into serial finite-state decisions.The model separates submission and rejection into different states to reduce cross-task interference.
- The first-order state transition mechanism aligns interleaved training with compact inference and avoids historical state-token accumulation during deployment.
- TurnFSM is presented as an effective low-latency plug-in module for full-duplex dialogue systems.