PaperlayerView exact revision on arXiv

Source-linked AI summary

TurnFSM for Full-Duplex Dialogue System: Internalizing State-Machine Logic for Streaming Semantic Voice Activity Detection and Utterance-Level Rejection

Zhiwei Lin, Tianjiao Du, Qiaochu Huang, Zihan Zhang, Naijun Zheng, Longshuai Xiao, Yunfei Lu, Jun Chen, Zhiyong Wu

arXiv:2609.04240v1eess.AScs.SD

TL;DR

Full-duplex assistants need low-latency turn control while listening during speech, but existing end-to-end and cascaded approaches have capability or overhead limitations. TurnFSM internalizes control as serial finite-state transitions with first-order state dependency, and achieves competitive semantic VAD and rejection performance against task-specific models.

  • Problem

    Full-duplex turn control must handle streaming semantic VAD and rejection, while existing end-to-end models may affect reasoning capabilities and cascaded pipelines add overhead and handcrafted logic.

    Source: §I · p. 2

  • Method

    TurnFSM uses explicit finite-state transitions to serialize submission and rejection, with each state conditioned on the acoustic prefix and immediately preceding state.

    Source: §I · p. 2

  • Results

    TurnFSM achieves performance comparable to corresponding single-task models on semantic VAD and rejection detection.

    Source: C. Main Result · pp. 5–6

  • Takeaways & Limitations

    TurnFSM provides a unified streaming controller that separates heterogeneous decisions across transition stages while supporting compact inference without historical state-token accumulation.

    Source: §I · p. 2

Abstract

from arXiv · show

Full-duplex voice assistants must continuously listen while speaking, handling user interruptions under low-latency and resource-constrained streaming conditions. Existing end-to-end full-duplex models can compromise reasoning-related capabilities after speech-domain adaptation, whereas cascaded pipelines introduce extra inference overhead and handcrafted control logic. We propose TurnFSM, an LLM-based state prediction framework that internalizes turn control as explicit finite-state transitions, unifying streaming semantic VAD and utterance-level rejection. TurnFSM decomposes submission and rejection into a serial decision process, reducing multi-task interference while maintaining performance comparable to single-task models. We further introduce a first-order state transition mechanism that enforces the dependency on only the previous state during training, enabling compact inference with the standard causal mask and original LLM positional encoding while avoiding historical state-token accumulation and unnecessary step-by-step state generation. Experimental results show that TurnFSM consistently outperforms the binary-head baseline and remains competitive with task-specific models.

I. INTRODUCTION

TurnFSM addresses full-duplex turn-control challenges by replacing parallel semantic VAD and rejection predictions with explicit serial state transitions. A first-order dependency on the previous state enables compact streaming inference while reducing control overhead and cross-task interference.

  • Full-duplex assistants must perform reliable turn control under low-latency, resource-constrained streaming conditions, including segmentation, submission timing, and rejection.Source: §I · p. 1
  • End-to-end full-duplex models may degrade reasoning-related capabilities after speech-domain adaptation, while cascaded pipelines add inference overhead, error propagation, and handcrafted coordination logic.Source: §I · pp. 1–2
  • Parallel semantic VAD and rejection heads face gradient conflicts because the tasks use different decision criteria and acoustic or semantic cues.Source: §I · p. 2
  • TurnFSM serializes control through explicit states, determining semantic completeness during listening before evaluating rejection after submission.Source: §I · p. 2
  • TurnFSM jointly supports streaming semantic VAD and utterance-level rejection within a unified state-transition framework.Source: §I · p. 2
  • The first-order mechanism conditions each state on the acoustic prefix and immediately preceding state, avoiding historical state-token accumulation and unnecessary step-by-step generation.Source: §I · p. 2

II. METHODOLOGY

TurnFSM embeds cascaded turn-control logic in a learnable LLM state-transition model. Its serial decomposition assigns submission and rejection to separate stages, avoiding competition between heterogeneous task representations.

  • TurnFSM replaces handcrafted control states with a learnable state-transition model embedded in the LLM.Source: §II · p. 2
  • The model predicts each control state from streaming audio and the previous state within a unified state space.Source: §II · p. 2
  • TurnFSM performs submission first and rejection detection second, assigning semantic completeness and rejection to different state stages.Source: §II · p. 2

A. Model Architecture

TurnFSM combines streaming audio encoding, an audio adapter, and an LLM backbone for state prediction. The architecture progressively converts audio into LLM-compatible tokens before prediction.

  • TurnFSM consists of a streaming audio encoder, an audio adapter, and an LLM backbone.Source: A. Model Architecture · p. 2
  • The audio encoder uses a two-layer VGG-style front-end with 4× downsampling followed by a 6-layer Conformer, producing acoustic frames at 25 Hz.Source: A. Model Architecture · p. 2
  • A lightweight MLP audio adapter applies 2× downsampling and projects acoustic representations into the LLM input embedding space.Source: A. Model Architecture · p. 2
  • The model uses Qwen2.5-7B-Instruct as its LLM backbone for streaming state prediction.Source: A. Model Architecture · p. 3

B. Finite-State Turn Control Formulation

TurnFSM models full-duplex turn control as explicit finite-state transitions rather than independent binary decisions. The sequence separates semantic completeness, submission, and utterance validity into distinct control stages.

Baseline and motivation

  • The binary-head baseline predicts semantic completeness and input validity as separate submit/wait and accept/reject decisions.Semantic VAD determines whether the current prefix is complete, while rejection determines whether accumulated audio is a valid intent-bearing input.Source: B. Finite-State Turn Control Formulation · p. 3

Finite-state formulation

State definitions

  • The state set includes pseudo-initial, silence, listening, submission, acceptance, and rejection states.Silence monitors input without buffering, listening accumulates candidate utterances, and submission finalizes a segment for validation.Source: B. Finite-State Turn Control Formulation · p. 3

Transition logic

  • TurnFSM transitions from listening to submission after semantic completeness, then from submission to acceptance or rejection based on utterance validity.This serial process assigns semantic completeness and rejection to different state stages.Source: B. Finite-State Turn Control Formulation · p. 3

C. First-Order State Transition Mechanism

The first-order mechanism trains each state prediction to use only the acoustic prefix and immediately preceding state. Position tying aligns interleaved training with compact inference using the standard causal mask and original positional encoding.

Motivation

  • A standard causal mask can make each state prediction attend to the entire history of previous state tokens.This history-dependent implementation repeatedly accumulates earlier states and is unnecessary for the intended FSM dependency.Source: C. First-Order State Transition… · pp. 3–4
  • Historical state-token conditioning creates train–inference inconsistency by requiring intermediate state generation before final decisions.The issue applies even to non-streaming or utterance-level inference.Source: C. First-Order State Transition… · p. 4

Training mechanism

  • During training, a modified attention mask restricts St to the acoustic prefix A≤t and immediately preceding state St−1.Attention to earlier states S<t−1 is masked, enforcing the first-order dependency.Source: C. First-Order State Transition… · p. 4, Figure 3 · p. 4
  • Position tying assigns each state token the positional encoding of the following audio token, preserving the original acoustic temporal order.The following audio token is only a positional reference, not an input for predicting the state.Source: C. First-Order State Transition… · p. 4

Inference mechanism

  • At inference, TurnFSM predicts St from (A0, . . . , At, St−1) with the standard lower-triangular causal mask and original LLM positional encoding.This avoids historical state-token accumulation and unnecessary step-by-step state generation.Source: C. First-Order State Transition… · p. 4

A. Experimental Setup

TurnFSM uses separate cross-modal alignment and state-prediction training stages, supported by large open-source speech corpora and two internal control datasets.

Training stages

  • TurnFSM is trained in two stages: cross-modal alignment followed by state prediction.The alignment stage uses ASR training before the control-state datasets are used.Source: A. Experimental Setup · p. 5

Cross-modal alignment

  • Cross-modal alignment uses AISHELL-1, AISHELL-2, and WenetSpeech, totaling 11,200 hours of audio.The bridging layer is trained on an ASR task.Source: A. Experimental Setup · p. 5

State prediction data

  • The semantic VAD dataset contains 70K training samples and 800 test samples totaling 194 training hours.Samples are evenly split between pauses and no pauses, with human-annotated acoustic start and end timestamps.Source: A. Experimental Setup · p. 5
  • The rejection-detection dataset contains 700K training samples and 3K test samples totaling 2,000 training hours.It has an even positive/negative split and covers a wide range of real-world scenarios.Source: A. Experimental Setup · p. 5

B. Metircs and Baselines

The evaluation measures streaming semantic VAD timing and rejection detection, comparing TurnFSM with a binary-head alternative and task-specific semantic VAD models.

Metrics

  • Semantic VAD evaluation reports early-cut rate, timeout rate, success rate, and mean latency.Overall mean latency includes successful and timeout cases, assigning timeout cases a fixed latency of 2000 ms.Source: B. Metircs and Baselines · p. 5
  • Predictions within ±100 ms of the human-labeled endpoint are treated as correct rather than early cuts.The tolerance window accounts for noise in manual endpoint annotation.Source: B. Metircs and Baselines · p. 5
  • Rejection detection is evaluated with false rejection rate and false acceptance rate.These metrics assess rejection and acceptance errors respectively.Source: B. Metircs and Baselines · p. 5

Baselines

  • TurnFSM is compared with a binary classification head and three open-source semantic VAD models: TEN Turn Detection, Easy Turn, and Smart Turn V3.21.The open-source baselines are offline models whose streaming latency is simulated using Silero VAD speech chunks.Source: B. Metircs and Baselines · p. 5

Baseline handling

  • Smart Turn receives probability-driven adaptive waiting, merging a new speech chunk when it arrives within the predicted waiting time.Otherwise, the current chunk is treated as semantically complete and submitted.Source: B. Metircs and Baselines · p. 5

C. Main Result

TurnFSM performs competitively with single-task models on streaming and utterance-level semantic VAD while integrating rejection detection. It also outperforms the binary-head baseline and supports efficient whole-utterance inference.

  • 3.35% FAR and 16.04% FRR keep TurnFSM close to the Rejection-only model on rejection detection.The Rejection-only model records 3.31% FAR and 16.16% FRR.Source: C. Main Result · p. 6
  • TurnFSM outperforms the binary-head baseline on most metrics, including success rate, timeout rate, latency, and FRR.The shared-backbone binary-head design produces stronger multi-task coupling than the state-machine formulation.Source: C. Main Result · p. 6
  • TurnFSM achieves the best overall balance among open-source semantic VAD baselines, with stable submission decisions and lower overall latency.Its 82.41% success rate is highest, while Easy Turn and Smart Turn exhibit trade-offs involving timeout or latency.Source: C. Main Result · p. 6
  • The first-order transition mechanism enables TurnFSM to process a complete utterance in one forward pass without generating historical state tokens step by step.This compact inference form applies to utterance-level semantic completeness prediction.Source: C. Main Result · p. 6
  • 91.00% overall accuracy places TurnFSM second on the EasyTurn test set, behind Easy Turn but ahead of TEN Turn Detection and Smart Turn V3.2.The compared systems score 97.00%, 87.99%, and 74.67%, respectively.Source: C. Main Result · p. 6

D. Ablation Study

Ablations show that FOSTM preserves task performance while reducing inference cost, whereas cross-modal alignment is important for semantic VAD but less influential for rejection detection.

FOSTM

  • Removing FOSTM slightly lowers semantic VAD success and raises early-cut rate, while increasing successful-case latency from 80.9 ms to 111.35 ms.Rejection detection changes only slightly, with minor FAR and FRR differences.Source: D. Ablation Study · p. 6
  • FOSTM preserves the original FSM’s modeling capability with stable performance across semantic VAD and rejection detection.Source: D. Ablation Study · p. 6
  • FOSTM reduces redundant state-token accumulation and avoids unnecessary step-by-step state generation during inference.Training enforces pθ(St | A≤t, St−1), enabling compact inference with the standard causal mask.Source: D. Ablation Study · p. 6

Cross-modal alignment

  • Removing cross-modal alignment increases semantic VAD early-cut rate from 12.06% to 15.83%.Alignment helps audio representations remain consistent with semantic content for completeness detection.Source: D. Ablation Study · pp. 6–7
  • Cross-modal alignment has limited rejection-detection impact, with FRR increasing from 16.04% to 17.10% after removal.The FAR slightly decreases when alignment is removed.Source: D. Ablation Study · p. 7

IV. CONCLUSION

TurnFSM unifies semantic VAD and utterance-level rejection through serial finite-state transitions and uses first-order state prediction for compact inference. It achieves strong performance, improves over the binary-head baseline, and remains competitive with task-specific models.

  • TurnFSM unifies semantic VAD and utterance-level rejection by transforming parallel multi-task prediction into serial finite-state decisions.The model separates submission and rejection into different states to reduce cross-task interference.Source: §IV · p. 7
  • The first-order state transition mechanism aligns interleaved training with compact inference and avoids historical state-token accumulation during deployment.Source: §IV · p. 7
  • TurnFSM is presented as an effective low-latency plug-in module for full-duplex dialogue systems.Source: §IV · p. 7

Related papers

Loading 2609.04240v1…