PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue

Yihang Li, Chenhui Chu

arXiv:2609.03321v1cs.CL

TL;DR

NFSM’s synthetic text supervision limits natural full-duplex turn-taking because it cannot reproduce fine-grained acoustic timing. The paper decouples supervision across real HH speech and HA text, adds event-guided serialization and SAC Loss, and reports stronger turn-taking with recovery of the foundation LLM’s semantic capability.

  • Problem

    NFSM’s reliance on synthetic text data limits turn-taking naturalness because LLMs cannot faithfully simulate fine-grained acoustic and temporal dynamics in human dialogue.

  • Method

    The paper learns turn-taking from real HH spoken dialogues, shapes semantics with HA text dialogues, serializes events through deterministic rules, and applies SAC Loss.

  • Results

    The approach substantially improves turn-taking proficiency over the synthetic-data baseline while recovering the foundation LLM’s semantic capability to the zero-shot upper bound.

  • Takeaways & Limitations

    Decoupling data sources channels each capability toward the supervision source that best supports it while preserving the shared FSM tape format.

  • Takeaways & Limitations

    The single causal tape compresses continuous-time information into discrete token order and cannot represent paralinguistic information such as prosody, emotion, and voice quality.

Abstract

from arXiv · show

The Neural Finite State Machine (NFSM) framework offers a pragmatic path to full-duplex dialogue by serializing turn-taking control and response generation onto a single causal tape under the standard next-token prediction objective, thereby preserving semantic prowess at a low fine-tuning cost. However, its reliance on synthetic text data fundamentally limits turn-taking naturalness, as Large Language Models (LLMs) cannot faithfully simulate the fine-grained acoustic temporal dynamics of real human dialogues. In this work, we propose a decoupled data approach that learns turn-taking from real Human-Human (HH) spoken dialogues while shaping semantic behavior through configurable Human-Agent (HA) text dialogues. To operationalize this approach, we introduce a rule-based event-guided data transformation method that serializes HH spoken dialogues into FSM tapes by classifying turn-taking events and applying deterministic mapping rules, enabling scalable supervision without LLM-generated annotations. We further propose a Source-Aware Calibrated (SAC) Loss that jointly calibrates the long-tailed distribution of state transition tokens and channels each data source toward the capability it best supervises. Experiments show that our approach substantially improves turn-taking proficiency while recovering the foundation LLM's semantic capability. Our code and model are available at https://github.com/Liyht/def-fsm.

1 Introduction

NFSM separates turn-taking control from response generation on one causal tape, but synthetic text supervision cannot reliably reproduce human acoustic timing. The paper therefore decouples data sources and adds SAC Loss to improve turn-taking while preserving semantic capability.

  • Motivation: NFSM serializes turn-taking control and response generation onto one causal tape, but its synthetic data cannot faithfully simulate fine-grained acoustic and temporal dynamics.Foundation LLMs are mainly pretrained on structured, non-overlapping text, making jointly realistic turn-taking and rich semantics unreliable supervision.
  • Approach: The decoupled approach learns fine-grained turn-taking from real HH spoken dialogues and shapes semantic behavior with configurable HA text dialogues.Each capability is aligned with the data source best suited to supervise it.
  • Approach: A rule-based event-guided transformation serializes HH dialogues into causal FSM tapes by classifying turn-taking events and applying deterministic mapping rules.The pipeline is designed to scale without LLM-generated annotations.
  • Optimization: SAC Loss calibrates the long-tailed distribution of state-transition tokens and specializes optimization across the two data sources.It addresses rare but operationally important state-switch tokens and asymmetric supervisory value.
  • Results: Experiments show substantially better turn-taking proficiency and semantic capability than the conventional synthetic-data baseline, with larger-scale training recovering the foundation LLM’s intrinsic semantics.The reported gains hold under strict token-volume constraints, and upsampling preserves the source mixture ratio.

2 Related Work

Related work organizes turn-taking supervision around event representations and data sources. This paper adopts IPU-based events because their boundaries make rule-based serialization onto an FSM tape straightforward.

  • Turn-Taking Event Classification: Turn-taking event classification assigns labels along the dialogue timeline for downstream event prediction and analysis.Existing schemes differ in the temporal unit used to define events.
  • Turn-Taking Event Classification: Chunk-based methods label equal-length frames, while IPU-based methods define events at silence-delimited boundaries that yield linguistically coherent units.Chunk-based methods provide fine temporal resolution; IPU-based methods align with pauses and linguistic structure.
  • Data Strategies for Full-Duplex Supervision: Full-duplex supervision uses synthetic dialogues, real spoken corpora with implicit audio supervision, or heuristic state labels from fixed acoustic chunks.The paper’s setting differs by serializing real spoken interactions into an FSM tape rather than classifying fixed acoustic chunks.

3 Data Transformation

The transformation pipeline converts HH audio into FSM tapes by asymmetrically preprocessing user and agent channels, classifying IPU-aligned events, and applying deterministic serialization rules. HA text dialogues receive a separate filtering, rewriting, and schema-compliant serialization pipeline for semantic supervision.

  • Pipeline Overview: The method serializes real HH spoken dialogues for turn-taking dynamics and HA text dialogues for semantic capability into one causal FSM tape format.The two corpora are processed through complementary pipelines.
  • Asymmetric Preprocessing: User audio is transcribed with the FSM’s perception module into timestamped chunks and silence tokens, while agent audio uses fine-grained ground-truth transcripts.This asymmetry exposes realistic ASR behavior while teaching clean agent-text generation.
  • Event Classification: The dialogue timeline is partitioned into seven event types: turn change, continuation, pause, gap, backchannel, floor-taking interruption, and butting-in.IPUs are formed using a pause threshold, and the union of channel boundaries defines the discrete timeline.
  • Event-Guided Tape Serialization: The serialization splits [S.LISTEN] into natural and interrupt variants, assigns chunks by timestamp, and applies 14 deterministic rules based on event and initiator.Four heuristics then govern token placement, overlap ordering, silence preservation, and boundary-token collapsing.
  • Human-Agent Transformation: HA text dialogues are cleaned, rewritten into spoken style, filtered again, segmented into clauses, and serialized with a trivial turn-taking structure.This lets HA data focus its supervisory signal on semantic content.

4 Model

The model uses SAC Loss to address class imbalance in turn-taking tokens and specialize learning across heterogeneous HH and HA data sources.

  • SAC Loss calibrates the long-tailed state-transition distribution and decouples learning targets across diverse data sources.It is designed to address both turn-taking class imbalance and joint optimization of distinct capabilities.
  • Selective masking computes loss only on agent response and state-transition tokens, partitioned by token type and HH or HA source.Prompts and user text are excluded from loss computation.
  • Logit Adjustment is applied to state-transition tokens, while response tokens retain standard cross-entropy to counteract rare state-switch marginalization.The adjustment uses priors computed over the restricted state-transition vocabulary.
  • The overall objective is a weighted average over unmasked tokens, with token weights determined by type and data source.The weighting mechanism dynamically adjusts each token's importance according to its origin and role.
  • With α ∈ (0.5, 1], SAC emphasizes HH turn-taking tokens and HA response tokens to specialize the two optimization targets.Higher weight α is assigned to (HH ∩ T) and (HA ∩ R), while the complementary subsets receive 1 − α.

5 Experiments

Experiments show that decoupled HH and HA data, combined with SAC loss, improves turn-taking while preserving or recovering semantic capability across controlled, scaled, and architecture-agnostic evaluations.

  • Settings: The experiments use Switchboard and Fisher for HH speech, ShareGPT-based HA text, VoiceBench for semantics, and matched token volumes against NFSM.The study evaluates two perception configurations and reconstructs synthetic NFSM data as a baseline.
  • Main Results: Our model significantly outperforms NFSM under both perception configurations, while Faster-Whisper delivers higher semantic capability than SimulStreaming at comparable turn-taking performance.SimulStreaming’s hyper-granular outputs fragment user text, creating a trade-off between interruption opportunities and semantic comprehension.
  • Ablation Study: SAC loss improves turn-taking proficiency, preserves semantic capability at a comparable level, and raises accuracy on rare, operationally critical state-switch tokens.Figure 3 compares standard cross-entropy and SAC confusion matrices for state transition and response tokens on Switchboard.
  • Ablation Study: Combined HH and HA training substantially outperforms synthetic-only training, while HH-only and HA-only data each improve one capability without improving both.The same mixture trends remain consistent when Faster-Whisper is replaced by SimulStreaming.
  • Ablation Study: Our model substantially outperforms NFSM in both machine-interrupts-user and user-interrupts-machine evaluations.These interruption tests follow the NFSM protocol and complement turn-taking F1 measurements.
  • Scaling to the Proportionate Upsampling Mixture: Scaling the proportionate mixture improves both capabilities and brings the Faster-Whisper model within 0.09 VoiceBench points of the zero-shot Qwen3-4B upper bound.The scaled FSM model also exceeds the reported end-to-end Moshi reference on VoiceBench.
  • Scaling to the Proportionate Upsampling Mixture: On Full-Duplex-Bench, the model achieves its strongest gains in backchanneling and ranks first or second across smooth turn-taking, user-interruption semantic quality, and all four v1.5 overlap scenarios.The authors attribute these gains to event-guided tapes and SAC loss.
  • Scaling to the Proportionate Upsampling Mixture: Pause handling is a trade-off: the model’s takeover rates trail Freeze-Omni and Gemini Live, although it outperforms every baseline across most of the twelve metrics.Freeze-Omni’s more conservative pause handling is accompanied by weaker smooth turn-taking.

6 Conclusion

The paper presents a decoupled data approach for FSM-based full-duplex dialogue, assigning turn-taking supervision to real HH speech and semantic shaping to configurable HA text. Rule-based tape transformation and SAC Loss improve turn-taking while recovering the foundation LLM’s semantic capability to the zero-shot upper bound.

  • Turn-taking is learned from real HH spoken dialogues, while semantic behavior is shaped through configurable HA text dialogues.
  • A fully rule-based event-guided transformation serializes HH speech into causal FSM tapes without LLM-generated annotations.
  • SAC Loss calibrates long-tailed state-transition tokens and directs each data source toward the capability it best supervises.
  • The approach substantially improves turn-taking over the synthetic-data baseline while recovering the foundation LLM’s semantic capability to the zero-shot upper bound.

Limitations

The study identifies information, expressiveness, data-alignment, and scope limitations. The tape loses continuous-time and paralinguistic information, agent content may not match voice-assistant style, and the corpora cover only two-party question-answering dialogue.

  • Serializing two parallel audio channels into one causal tape compresses continuous-time information into discrete token order, causing information loss.
  • The text-mediated tape cannot represent prosody, emotion, or voice quality, while rendered-speech expressiveness is bounded by the off-the-shelf TTS module.
  • The study focuses on turn-taking naturalness and leaves paralinguistic cues and perceptual evaluation for future work.
  • Agent-channel utterances retain ordinary human participants’ content, which may not align with voice-assistant response style.
  • The corpora cover only two-party dialogue and a question-answering assistant role, leaving multi-party and role-conditioned extensions for future work.

A Data Construction Details

The data construction details compare reconstructed synthetic tapes with real HH tapes, motivate calibration from their token distributions, and specify deterministic serialization and prompt-based processing procedures. The construction merges boundary tokens using FSM states while keeping HA turn-taking structurally trivial so it supervises semantics.

  • The reconstructed NFSM baseline is documented parameter by parameter, including whether each setting is specified in the original paper or adopted for reproduction.
  • State-transition tokens are strongly long-tailed: continuation tokens comprise 72.78% with Faster-Whisper and 80.45% with SimulStreaming of state-transition tokens.
  • Switching to SimulStreaming raises [C.LISTEN] from 8.75% to 13.77%, reflecting greater interleaving within user content and a larger semantic gap.
  • Synthetic tapes allocate 88.38% to response tokens and only 2.40% to critical state-switch tokens, versus 7.26% and 5.88% in real HH tapes under Faster-Whisper and SimulStreaming.
  • Boundary merging reduces adjacent token pairs to four FSM state-pair cases, including continuation, speaking switches, and subtype inheritance for SPEAK-to-LISTEN transitions.

B.1 Training Hyperparameters and Dataset Partitioning

Training uses a fixed multi-GPU optimization setup with early stopping, while preprocessing filters noisy HH dialogues and the event-guided pipeline applies explicit timing thresholds.

  • Fine-tuning uses four NVIDIA RTX A6000 GPUs, batch size 256, AdamW, peak learning rate 1.0 × 10−5, and context length 1024 tokens.
  • Training runs for up to 300 iterations in the token-matched setting and 700 iterations for the proportionate upsampling mixture, with validation-loss early stopping.
  • HH dialogues with Word Error Rate above 0.3 are discarded to remove broken audio and cross-channel leakage.
  • The transformation uses a 32-millisecond IPU pause threshold, 0.64-second silence tokens, and a 1-second maximum duration for heuristic backchannel identification.

C.1 Analysis on Data Mixture, Loss, and Model Scale

The analysis finds a trade-off between turn-taking and semantic capability across data mixtures and SAC-loss weighting, while model scaling benefits semantics but does not monotonically improve turn-taking.

  • Data Mixture Ratios: Increasing HH data consistently enhances turn-taking proficiency, whereas increasing HA data improves semantic capability; the main experiments therefore use a 1:1 HH:HA ratio.The internal Fisher-to-Switchboard ratio showed no significant trend.
  • Impact of α in SAC Loss: As α increases, VoiceBench improves while turn-taking F1 decreases, with state-transition recall rising and precision dropping significantly.The SAC loss therefore aligns better with recall than precision.
  • Impact of Base Model Size: Larger base models consistently improve VoiceBench, but turn-taking F1 peaks at the 4B scale.The authors hypothesize that larger models may require more effort to fit the specialized FSM-tape pattern.
  • Ablation Studies: The ablation study includes comparisons without SAC loss and evaluates loss and training-data composition under SimulStreaming perception.The supplied ablation entry reports 0.6234 and 54.49 without SAC loss, but does not identify the metric labels.
  • Ablation on Training Data: Under SimulStreaming perception, SAC loss preserves turn-taking performance, while neither HH nor HA data alone jointly optimizes semantic and turn-taking capabilities.These trends are reported as consistent with the Faster-Whisper-based model.

C.3 Bidirectional Interruption Evaluation

The evaluation tests bidirectional interruptions using controlled synthetic dialogues and reports higher interruption-related scores than the NFSM baseline for both perception-module variants.

  • Evaluation Protocol: The evaluation replicates the NFSM interruption protocol for Machine-interrupts-User and User-interrupts-Machine settings.The study measures responses at interruption points and evaluates reconstructed dialogues after removing state-transition tokens.
  • Machine-interrupts-User: For Machine-interrupts-User, 600 dialogues contain a deliberate commonsense error in the user’s final statement, with responses extracted when the FSM first emits [S.SPEAK].Qwen3.5-27B serves as evaluator under the original NFSM prompts.
  • User-interrupts-Machine: For User-interrupts-Machine, 600 dialogues are evenly distributed across denial, affirmation, environmental noises, and topic shifting categories.The protocol inserts [S.SPEAK] after the interruption and evaluates Proper Response Rate.
  • Results: Both perception-module variants achieve substantially higher MiU F1 and UiM PRR scores than the NFSM baseline.The reported improvement covers turn-taking dynamics and semantic appropriateness within the interruption sub-scenario.
  • Broader Evaluation: The complete Full-Duplex-Bench results separately assess dialogue-wide turn-taking in v1.0 and overlap handling during model speech in v1.5.The benchmark uses official protocols for the Faster-Whisper-based proportionate upsampling model.
  • Latency: End-to-end FTED measurements across perception, cognitive, and motor modules indicate real-time responsiveness for the Faster-Whisper-based FSM system.The latency test samples 100 questions from four VoiceBench subsets.

E Design of FSM

The FSM design combines specialized decoding, legal-transition constraints, bounded look-ahead, and tape rewriting to manage speech generation and interruptions while preserving responsive interaction.

  • Decoding Strategy: Transition tokens are decoded greedily, while response tokens use standard sampling temperatures to separate control reliability from lexical diversity.Mid-clause transition tokens are discarded and the decision is deferred to the next probe.
  • Constrained Decoding: Illegal state-transition tokens are masked during decoding, providing deterministic protection against structural failures.The model rarely produces illegal transitions, but hard masking enforces the constraint explicitly.
  • Bounded Look-ahead: Bounded look-ahead generates one additional clause while fewer than K clauses remain in flight, balancing interruption latency against wasted computation.Generation idles once the bound is reached.
  • Interruption Handling: During interruptions, the agent continues speaking while the transition decision is computed, then rewrites the tape according to the resulting transition token.This supports natural backchannels but creates temporal inconsistency between tape segments before and after the decision.
  • Transition-Specific Rewriting: The FSM applies distinct tape transformations for [C.SPEAK], [S.LISTEN.I], and [S.LISTEN.N] to determine whether speech continues, stops immediately, or finishes the current clause.The transformations respectively retain or discard look-ahead content according to the interruption type.
  • ASR Robustness: A soft lexical filter discards frequent Whisper hallucinations when their average token log-probability falls below −0.3, while retaining higher-confidence segments.The filter is intended to prevent spurious interruption triggers without removing genuine user utterances.
Loading 2609.03321v1…