PaperlayerView exact revision on arXiv

Source-linked AI summary

The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

Donghang Wu, Tianyu Zhang, Yuxin Li, Hexin Liu, Chen Chen, Eng Siong Chng, Yoshua Bengio

arXiv:2603.17837v5eess.AScs.CL

TL;DR

Full-duplex dialogue models need to reason while listening to streaming speech, rather than leaving the listening phase unused or relying on explicit text thought sequences. FLAIR trains causal latent reasoning with an ELBO-based objective and a Global-aware Expert. The paper reports improved response quality across diverse benchmarks and strong real-time interaction performance without additional inference latency.

  • Problem

    Existing spoken dialogue models often serialize listening and responding, while full-duplex systems need to use the listening phase without violating streaming causality.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    FLAIR trains a causal SDLM to perform continuous latent reasoning during listening using an ELBO objective and a Global-aware Expert.

    Source: §1 · p. 2, §4.2 · pp. 4–5

  • Results

    FLAIR improves response quality across diverse benchmarks, especially reasoning tasks, while maintaining competitive turn-taking and barge-in performance without additional inference latency or computational overhead.

    Source: §7.1 · p. 7, §8 · p. 8

  • Takeaways & Limitations

    The results support latent reasoning during speech perception as a way to improve full-duplex dialogue response quality while retaining real-time interaction efficiency.

    Source: §8 · p. 8

Abstract

from arXiv · show

During conversational interactions, humans subconsciously engage in concurrent thinking while listening to a speaker. Although this internal cognitive processing may not always manifest as explicit linguistic structures, it is instrumental in formulating high-quality responses. Inspired by this cognitive phenomenon, we propose a novel Full-duplex LAtent and Internal Reasoning method named FLAIR that conducts latent thinking simultaneously with speech perception. Unlike conventional "thinking" mechanisms in NLP, which require post-hoc generation, our approach aligns seamlessly with spoken dialogue systems: during the user's speaking phase, it recursively feeds the latent embedding output from the previous step into the next step, enabling continuous reasoning that strictly adheres to causality without introducing additional latency. To enable this latent reasoning, we design an Evidence Lower Bound-based objective that supports efficient supervised finetuning via teacher forcing, circumventing the need for explicit reasoning annotations. Experiments demonstrate the effectiveness of this think-while-listening design, which achieves competitive results on a range of speech benchmarks. Furthermore, FLAIR robustly handles conversational dynamics and attains competitive performance on full-duplex interaction metrics.

1. Introduction

Existing spoken dialogue models often process user speech and agent responses sequentially, unlike human concurrent listening and speaking. FLAIR uses latent reasoning while listening, trained with an ELBO-based framework, to improve responses while preserving causal, low-latency interaction.

1. Introduction

  • Existing SDLMs commonly serialize user speech and agent responses, contrasting with human systems that listen and speak concurrently.Full-duplex interaction continually ingests streaming user speech during response prediction and supports proactive turns, backchannels, and barge-ins.Source: §1 · p. 1
  • The ELBO-based training framework uses a Global-aware Expert and teacher forcing to supervise latent reasoning without explicit reasoning annotations.The paper also describes reinforcement learning as an orthogonal post-training step.Source: §1 · p. 2
  • FLAIR improves response quality across tasks including factual QA, multi-turn QA, open-ended generation, and multiple-choice benchmarks.The introduction reports low response latency and high interaction success rates.Source: §1 · p. 2
  • FLAIR introduces fully causal latent reasoning during listening, without explicit reasoning datasets or additional inference computation.The method is designed for full-duplex SDLMs and can also be adapted to half-duplex speech LLMs.Source: §1 · p. 2
  • The paper presents FLAIR as the first latent-reasoning approach for speech LLMs and says it can be adapted to traditional half-duplex models.Its full-duplex implementation addresses the need to think while listening.Source: §1 · p. 2

2. Related Work

Prior work explores continuous latent computation, full-duplex speech systems, and explicit reasoning for language models. FLAIR addresses the listening-phase gap in full-duplex systems with continuous cognitive activity.

2. Related Work

  • Latent-reasoning approaches include feeding hidden states back as inputs, looping latent updates, and generating soft concept states for implicit exploration.Coconut and CODI explore alternatives in latent space; Ouro and looped transformers internalize multi-step reasoning; Soft Thinking and SoftCoT++ use continuous mixtures of token embeddings.Source: §2 · p. 2
  • Full-duplex SDLM work covers streaming speech components and end-to-end systems, but listening phases often use pause tokens without reasoning.Streaming CoT approaches may require specialized datasets or fail to guarantee causality; FLAIR enables continuous cognitive activity while listening.Source: §2 · p. 3
  • Text reasoning methods such as CoT, self-consistency, and tree-of-thoughts generally require full problem context and non-causal computation.The paper contrasts these stop-and-think methods with the causality and low-latency requirements of streaming conversation.Source: §2 · p. 3

3. Preliminary

Full-duplex SDLMs process streaming speech while generating responses, but conventional systems use silence tokens during listening. The preliminary section also describes ELBO as a variational-inference objective balancing reconstruction and latent regularization.

3.1. Full-duplex SDLMs

  • A typical full-duplex SDLM combines a streaming speech encoder, an LLM backbone, and a speech generation module.The encoder produces continuous speech embeddings, the LLM predicts text tokens, and an autoregressive decoder predicts speech tokens.Source: §3.1 · p. 3
  • Conventional full-duplex systems fill the listening phase with silence tokens, leaving that window unused and potentially degrading response content.The stated objective is to optimize the assistant’s output while the user is speaking.Source: §3.1 · p. 3

3.2. Evidence Lower Bound

  • The ELBO provides a lower-bound objective for maximizing marginal likelihood when direct integration over a high-dimensional latent space is intractable.Its reconstruction term captures data features while its KL term regularizes the approximate posterior toward a prior.Source: §3.2 · pp. 3–4

4. Methodology

FLAIR uses vocabulary-weighted continuous embeddings as next-step inputs during listening, then trains latent reasoning with an ELBO-based teacher-forcing framework.

4. Methodology

  • During listening, FLAIR converts previous-step text logits into a vocabulary-weighted embedding for the next LLM input, rather than reusing the hidden state directly.The embedding is computed by weighting the LLM vocabulary embedding matrix.Source: §4 · p. 4

4.1. ELBO in Full-duplex SDLMs

  • The ELBO optimizes conditional response likelihood via an expert posterior, combining response reconstruction with regularization that aligns the causal prior to that posterior.The formulation shifts the model’s focus from data reconstruction to intent-based response generation.Source: §4.1 · p. 4

4.2. SFT for Latent Reasoning

  • For teacher-forced SFT, a non-causal Global-aware Expert uses the full dialogue to derive latent embedding labels that replace <SIL> padding embeddings during user speech.The expert projects its hidden representation to vocabulary weights, which are applied to the vocabulary embedding matrix.Source: §4.2 · p. 4
  • SFT combines assistant-turn reconstruction loss with listening-phase variational regularization, while a learned timing indicator selects latent reasoning or explicit response tokens at inference.The expert’s vocabulary-weighting distribution supervises the LLM’s distribution during the user’s speaking phase.Source: §4.2 · p. 5
  • At inference, the Expert is discarded; latent embeddings feed the next step during listening, while text tokens drive streaming speech after the assistant begins responding.When the user interrupts, the model stops speaking and returns to latent reasoning.Source: §4.2 · p. 5, Figure 2 · p. 5

5. Data Synthesis and Model Training

The training data combine synthetic speech continuation and instruction-following dialogues with real noisy ASR-QA data. Training proceeds through pre-training, latent-reasoning SFT, and speech-synthesis SFT.

5.1. Dialogue Data Generation

  • The synthetic dataset contains 530K hours of speech continuation, 70k hours of instruction-following QA, and 20k hours of ASR-QA from a real noisy ASR corpus.LLMs and voice-cloning TTS systems generate the user and agent streams.Source: §5.1 · p. 6
  • To mitigate overfitting to synthetic acoustic artifacts, the authors diversify data with augmentation, a broad speaker pool, and multiple TTS models.These measures are described as a strategy for reducing that overfitting risk.Source: §5.1 · p. 6

5.2. Training pipeline

  • Training has three stages: speech-continuation pre-training, two-substage latent-reasoning SFT, and speech-synthesizing SFT with other parameters frozen.The latent-reasoning stage first trains the Expert with Lreco, then jointly trains the LLM and Expert with Lelbo; speech synthesis trains on codec tokens and next-token prediction.Source: §5.2 · p. 6

6. Evaluation and Benchmark

The evaluation covers response accuracy and quality, conversational behavior, and synthesized speech quality. It combines QA benchmarks with full-duplex interaction tests and a speech MOS measure.

6. Evaluation and Benchmark

  • Response evaluation spans factual QA, open-ended instruction following, and multiple-choice speech understanding and reasoning, using accuracy or GPT scores as specified.Open-ended QA uses a 1–5 GPT score, while other listed tests report accuracy; generated speech is transcribed with Whisper-large-v3 for metrics.Source: §6 · p. 6
  • Conversational behavior is assessed with turn-taking latency, barge-in latency, and barge-in success rate, plus Full-Duplex-Bench takeover, score, and latency measures.The impatient dataset has interruptions approximately every two seconds; Full-Duplex-Bench includes noisy CANDOR turn-taking and synthetic barge-in scenarios.Source: §6 · p. 6
  • Synthesized speech quality is evaluated with MOS calculated by UTMOS.Source: §6 · p. 6

7. Result and Analysis

FLAIR improves response quality across QA benchmarks, particularly on reasoning tasks, while retaining full-duplex conversational performance. Its latent embeddings also form a bridge between user-audio and target-text embeddings.

7.1. Reasoning Abilities

  • Latent reasoning improves nearly all QA tasks and outperforms full-duplex systems on most, especially MMSU and OpenbookQA; gains are larger on reasoning-heavy MMSU.Table 1 reports accuracy or GPT-Score across QA benchmarks.Source: Table 1 · p. 7, §7.1 · p. 7
  • For open-ended QA, FLAIR produces shorter responses that the authors describe as more concise and effective for spoken agents; CommonEval results also support effectiveness on real human speech.The authors note that longer responses tend to receive higher GPT scores on AlpacaEval and CommonEval.Source: §7.1 · p. 7

7.2. Conversational Behavior

  • Turn-taking and barge-in performance remains comparable with or better than non-thinking methods, and the authors report robustness in noisy scenarios after noise-augmented training.The added training subset mixes background noise from 0 dB to 60 dB; Table 3 covers Full-Duplex-Bench turn-taking and barge-in performance.Source: §7.2 · pp. 7–8, Table 3 · p. 8

7.3. Discussion and Visualization

  • In a t-SNE visualization of 300 Llama Questions examples, latent embeddings connect input-audio embeddings to target-text embeddings, including trajectories toward targets outside the input cluster.The authors also describe latent embeddings as taking a path through feasible-manifold regions.Source: §7.3 · p. 8, Figure 3 · p. 8

8. Conclusion

FLAIR enables full-duplex spoken dialogue models to reason latently while perceiving speech. Evaluations report improved response quality across benchmarks while retaining real-time conversational performance without additional inference latency or computational overhead.

8. Conclusion

  • An ELBO-based objective and Global-aware Expert enable supervised fine-tuning without explicit reasoning datasets.The framework models concurrent reasoning using variational inference.Source: §8 · p. 8
  • Evaluations report enhanced response quality across factual QA and open-ended conversation, especially on tasks requiring reasoning.The conclusion characterizes the evaluations as extensive.Source: §8 · p. 8
  • FLAIR maintains turn-taking and barge-in performance without additional inference latency or computational overhead.The conclusion frames this efficiency as necessary for real-time interaction.Source: §8 · p. 8

A. Evidence Lower Bound

The Evidence Lower Bound (ELBO) provides an optimization route when a latent-variable model’s marginal likelihood is intractable. In FLAIR, its conditional form shifts the objective to response generation and aligns a causal latent prior with a posterior inferred from full dialogue context.

A. Evidence Lower Bound

  • ELBO optimizes a lower bound on marginal likelihood when direct computation requires intractable integration over latent states.Variational inference uses a parameterized distribution to approximate the unavailable true posterior.Source: A. Evidence Lower Bound · p. 13
  • The ELBO balances expected reconstruction likelihood, which rewards latent states that reconstruct observations, against KL regularization toward the prior.The regularizer is described as structuring the latent space and preventing model collapse.Source: A. Evidence Lower Bound · p. 13

B. ELBO in Full-duplex SDLMs

  • FLAIR’s conditional ELBO replaces the standard data likelihood with response likelihood given the user input, shifting the objective to response generation.The formulation is presented as two components that drive latent reasoning in SDLMs.Source: B. ELBO in Full-duplex SDLMs · p. 13
  • Conditional reconstruction uses a full-context Global-aware Expert posterior, while KL regularization aligns the causal SDLM prior with that posterior.The expert can use the future response; the SDLM prior is generated causally and chronologically.Source: B. ELBO in Full-duplex SDLMs · pp. 13–14

C. Data Details

The training data combines diverse synthetic speech dialogues, QA interactions, interruption scenarios, real acoustic contexts, and augmented noisy speech. These components target conversational variety, turn-taking, acoustic robustness, and instruction following.

Model Diversity and Prompt Speaker Pool

  • More than 100k speech segments from over 20k speakers supply acoustic variety, using diverse LLMs, TTS systems, and LibriTTS, YODAS, and Hifi-TTS voice prompts.The text-generation ensemble includes GPT-OSS-120b 1, Qwen2.5-72B-Instruct 2, and Llama3.1-70b-Instruct 3; synthesis uses Chatterbox 4, Magpie-TTS 5, and Mooncast 6.Source: C. Data Details · p. 14

Speech continuation data

  • Speech continuation data converts text passages into synchronized two-stream dialogues, with stochastic turn lengths and role-swapped speakers.A turn ends after one sentence with 80% probability; turns switch roles if they exceed 200 words.Source: C. Data Details · p. 14

Instruction-Following QA Data

  • Instruction-following QA data covers single-turn topics and multi-turn conversations up to four minutes, with varied textual contexts to reduce redundancy.The generated dialogue is converted into speech through a multi-speaker TTS pipeline.Source: C. Data Details · p. 14

Simulation of user interruption

  • Synthetic interruptions target barge-in handling by inserting probabilistic user interruptions and requiring the agent to stop and return to latent reasoning.For responses over four seconds, interruption probability is 10%; an eight-token delay precedes forced EOS.Source: C. Data Details · p. 14

ASR-QA Data

  • ASR-QA data pairs real speech contexts with questions generated from transcripts, adding natural background noise but limited knowledge density.The passages identify the source corpora as NeMo ASRSET (Noroozi et al., 2024b) and the QA-pair procedure as following Noroozi et al. (2024a).Source: C. Data Details · pp. 14–15

Acoustic Augmentation

  • Acoustic augmentation applies SpecAugment and overlays background-noise clips onto user speech with varied signal-to-noise ratios.Noise is injected with 50% probability, using 10,000 clips and an SNR sampled uniformly from 0 dB to 60 dB.Source: C. Data Details · p. 15

D. Training Details

FLAIR uses a frozen speech encoder and audio codec with a Qwen2.5-7B-Instruct backbone, trained on 64 A800 GPUs. Training details specify loss weighting, turn-boundary scaling, optimization, and precision.

Implementation details

  • The implementation initializes the LLM from Qwen2.5-7B-Instruct and uses a 600M Parakeet-based causal speech encoder, with both encoder and codec frozen during training.The LLM and speech encoder operate at 12.5 Hz, while the codec runs at 25 Hz, yielding two predicted speech tokens per LLM frame.Source: D. Training Details · p. 15

Latent Reasoning SFT

  • Latent Reasoning SFT sets α = 3 and β = 5, with reconstruction loss scaled 20-fold at assistant BOS and 10-fold at assistant EOS.The weights were selected through empirical evaluations across hyperparameter combinations.Source: D. Training Details · p. 15

Training optimization

  • Training uses AdamW, inverse square-root annealing, BFloat16 mixed precision, and gradient clipping at 1.0.Learning rates are 5e−4 with 2500-step warm-up during interleaved pretraining and 5e−5 during SFT.Source: D. Training Details · p. 15

E. Ablation and Discussion

The ablations identify encoder scale as important for turn-taking and find comparable speech quality across two synthesis backends. The discussion also reports emergent QA behavior after speech-continuation pretraining and evaluates expert-model trade-offs.

Speech continuation data

  • Speech-continuation pretraining spontaneously yielded rudimentary instruction following, including spoken-query MCQ responses despite no MCQ training examples.The authors suggest original LLM SFT may supply this capability while speech continuation aligns acoustic queries to text and retains text-domain knowledge.Source: E. Ablation and Discussion · pp. 15–16
  • After pretraining, open-ended QA responses were mostly one sentence; instruction-following QA increased response length while remaining concise relative to LLM responses.The authors attribute the pretraining model’s brevity to its speech-continuation data length distribution.Source: E. Ablation and Discussion · p. 16

Streaming Speech Encoder

  • 70.7% response success with a 120M encoder rose to 100% with a 600M encoder and a linear projection on LLaMA-Q.The context window was held at the ASR default; the authors link the smaller model’s lower success to hidden-embedding dimensional mismatch with the LLM backbone.Source: E. Ablation and Discussion · p. 16

Global-aware Expert Model

  • T5-Large 11 matched the BERT baseline, while T5-3B 12 brought marginal gains judged disproportionate to its computational overhead.The authors retain BERT as the efficiency-effectiveness trade-off.Source: E. Ablation and Discussion · p. 17

Streaming Speech Synthesizer

  • MOS was 4.2 for streaming Flow Matching and 4.3 for multi-layer Audio Codec generation, with negligible perceptual differences and most A/B ratings tied.The authors conclude that FLAIR’s latent reasoning framework is agnostic to the waveform-generation backend.Source: E. Ablation and Discussion · p. 17
Loading 2603.17837v5…