PaperlayerView exact revision on arXiv

Source-linked AI summary

Liberating LLM Capabilities in Full-Duplex Speech Models

Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao

arXiv:2606.07547v1cs.CLcs.AIcs.SD

TL;DR

Speech-based LLMs constrain text-native capabilities by forcing code, structured analysis, and other persistent outputs into speech. LWS addresses this with a token-schema tri-channel model that listens, writes visible text, and speaks in parallel, achieving strong benchmark performance while exposing writing as a first-class realtime channel.

  • Problem

    Speech-based LLMs must project text-native capabilities such as code generation and structured analysis into spoken language, while prior systems generally keep text hidden or auxiliary.

  • Method

    LWS uses one autoregressive LLM with concurrent Listening, Visible Writing, and Speaking channels, implemented through a Token Schema and trained with causally aligned per-second annotations.

  • Results

    LWS performs strongly on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing–speaking consistency, and outperforms internal ablations on URO-Bench.

  • Takeaways & Limitations

    Visible writing can serve as a first-class output channel for realtime speech interaction while preserving a standard autoregressive modeling interface.

  • Takeaways & Limitations

    Realtime operation constrains reasoning depth, and the current formulation assumes speech-only user input.

Abstract

from arXiv · show

Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: https://royalzhang.com/project/lws-page/.

1 Introduction

Speech interfaces constrain LLMs to spoken outputs, suppressing text-native capabilities such as code and structured analysis. LWS makes visible writing a primary channel alongside listening and speaking, using token-based training and reporting strong benchmark results.

  • Motivation: Speech-only interaction forces executable code and structured information into linear spoken language that is harder to use and revise.Text-based LLM capabilities are naturally expressed through code, tables, derivations, and other persistent written artifacts.
  • Motivation: Prior approaches add reasoning before, during, or after speech, but generally retain text as hidden or auxiliary support for spoken responses.The related approaches improve spoken reasoning or full-duplex behavior without making text a first-class user-facing output.
  • LWS: LWS combines continuous listening, visible writing, and realtime speaking within one shared causal context.The paradigm is text-first: visible free-form writing is the primary output while spoken response is produced in parallel.
  • LWS: The Token Schema implements LWS in a standard autoregressive Transformer without architectural changes.The training pipeline also synthesizes per-second cognitive annotations aligned with the progressively revealed input timeline.
  • Results: LWS demonstrates strong full-duplex interaction, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing–speaking consistency, and outperforms internal ablations on URO-Bench.These are the headline results reported for the proposed paradigm.

2 Related Work

Prior speech-LLM research addresses reasoning, full-duplex interaction, or multiple output streams, but typically sacrifices at least one capability relevant to rich realtime interaction. LWS is positioned against these complementary paradigms as a system exposing visible free-form text.

  • Research directions: Related work spans reasoning-augmented speech LLMs, full-duplex spoken interaction, and multi-channel output architectures evaluated across four capability dimensions.These research directions address complementary aspects of speech interaction and are summarized comparatively in Table 1.
  • Reasoning-augmented speech LLMs: Think-Before-Speak and Interleaved Think-Speak models place reasoning before or during reply generation, with different latency and user-turn cognition trade-offs.The cited examples include Step-Audio, TVS, STITCH, Mini-Omni-Reasoner, and MPS.
  • Full-duplex interaction: Full-duplex models support overlap handling, turn-taking, and low-latency response, but typically expose only speech as the user-facing output.Cascaded ASR–LLM–TTS systems preserve text generation quality but generally revert to turn-based interaction.
  • Multi-channel outputs: Parallel text-speech and protocol-based systems emit multiple streams, but text is mainly used for synthesis, control, or hidden analysis rather than always-on visible writing.LWS instead targets free-form writing that remains visible to the user.

3 Method

LWS organizes realtime speech interaction into concurrent Listening, Visible Writing, and Speaking channels within discrete Units and a shared causal context. A token schema and asynchronous execution realize the design without additional architectural alignment modules.

  • 3.1 Architecture overview: LWS runs Listening, Visible Writing, and Speaking as concurrent channels in one full-duplex autoregressive process.Listening processes incoming audio, writing externalizes text-native cognition, and speaking produces the realtime oral response.
  • 3.1 Architecture overview: Each Unit is a configurable temporal interval; experiments use 1-second Units.Units partition the interaction timeline and determine the available channel combination.
  • 3.1 Architecture overview: Listening Units combine audio ingestion with incremental visible writing, while Speaking Units add speech generation to both channels.Writing continues across both phases, whereas speech is emitted only during Speaking Units.
  • 3.1 Architecture overview: All audio, visible-writing, and spoken tokens share one causal attention computation, eliminating separate cross-channel alignment modules.Shared attention serves as the information-fusion mechanism across channels.
  • 3.2 Token schema design: Special tokens delineate channel boundaries and encode the tri-channel behavior within a standard autoregressive Transformer.The token schema replaces architectural modifications with token-level structure.
  • Inference: Inference continuously processes audio Units, starts speech when a spoken segment is available, and keeps visible writing active while speech synthesis runs.Incremental processing also allows incoming user speech during model speaking to be encoded into later Units for interruption handling.

4 Data Construction

LWS training data is built by converting text QA conversations into Unit-based interactions with per-second cognitive annotations. Offline synthesis creates aligned text streams, and online construction combines them with audio timelines to produce 500K examples.

  • Pipeline motivation: The data pipeline starts from standard text QA pairs because public corpora lack per-second cognitive annotations aligned to audio timelines.The target format is the Unit-based interaction sequence used by LWS.
  • Stage 1: Offline cognitive synthesis: Stage 1 uses a teacher LLM to synthesize streaming reasoning, voice response, and structured response streams.These supervise listening cognition, speech, and reply cognition respectively.
  • Stage 1: Offline cognitive synthesis: Stage 1 segments interactions into second-by-second steps and constrains each step to information already revealed by that time.A plausible speech duration and strict causal prompting align cognitive annotations with streaming inference.
  • Stage 2: Online timeline construction: Stage 2 combines synthesized streams with real recordings and CTC-based character alignments to construct the final Unit sequence.It builds global timelines, inserts random inter-turn silences, and assigns Listening or Speaking Units according to the active speaker.
  • Dataset: The final dataset contains 500K Chinese and English training examples in a 1-second Unit-based token format.The Stage 1 prompt template is provided in Appendix D.

5 Experiments

Experiments evaluate LWS across multilingual reasoning, reply quality, channel consistency, full-duplex interaction, and joint training stability. LWS performs strongly against baselines and ablations while maintaining aligned outputs and smooth convergence.

  • LWS outperforms both internal writing ablations on all four displayed Basic/Pro averages across Chinese and English.The averages are Chinese Basic 82.6, Chinese Pro 84.6, English Basic 81.9, and English Pro 78.0.
  • 4.72 on VoiceBench AlpacaEval places LWS above all listed open-source baselines and 0.06 below GPT-4o-Audio.The evaluation scores the Visible Writing reply channel through a speech-to-text protocol.
  • 92.6% channel consistency indicates that LWS’s spoken and visible written responses remain factually aligned in 589 of 636 cases.An LLM-as-judge assigns consistency between the Speaking and Visible Writing outputs.
  • On Full-Duplex-Bench, LWS reaches 0.97 Candor TOR at 0.48 s latency, 4.02 GPT-4o quality at 0.65 s interruption latency, and TOR 0.01 for pause handling.The results cover smooth turn-taking, user interruption, and pause handling settings.
  • The three channels converge stably, with losses of 1.16 for ls cogn, 0.93 for speak, and 0.84 for reply cogn, and coefficient of variation below 8%.The joint overlay shows no obvious optimization instability in this setup.

6 Limitations

The study identifies limits in reasoning depth and input breadth: real-time Unit-level generation constrains longer-horizon tasks, while speech-only input excludes richer contextual modalities.

  • Real-time writing and speech within each Unit constrain longer-horizon reasoning, multi-step planning, and complex tool-mediated workflows.The paper notes that deeper reasoning may require slower generation or deferring speech while written reasoning unfolds.
  • The speech-only input formulation limits applicability when users combine speech with code, screenshots, tables, or other context.Adaptive multimodal input is identified as a direction for future work.

7 Conclusion

LWS unifies full-duplex listening, visible writing, and real-time speaking in one autoregressive LLM through a token schema and two-stage data pipeline. Its results support visible writing as a first-class speech-interaction channel while preserving the standard autoregressive interface.

  • LWS unifies full-duplex listening, visible writing, and real-time speaking within a single autoregressive LLM.
  • The design uses a token schema without architectural changes and is supported by a practical two-stage data pipeline.
  • LWS performs strongly on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing–speaking consistency, and outperforms internal ablations on multilingual URO-Bench.
  • The results support visible writing as a first-class output channel for realtime speech interaction within a standard autoregressive modeling interface.

Ethics Statement

LWS combines continuous listening, visible writing, and speaking in a single full-duplex pipeline, while raising deployment concerns about amplified trust in incorrect or unsafe outputs.

  • Ethics Statement: Because polished writing and fluent speech may increase perceived authority, incorrect, biased, or unsafe content can become more actionable in real time.The paper states that visible writing improves inspectability but does not guarantee correctness, faithfulness, or safety.
  • Ethics Statement: The authors recommend synchronized moderation, hazardous-request refusals, output auditing, and disclosure that visible writing is not verified ground truth.These recommendations address both spoken and written channels.
  • Inference pipeline: LWS keeps audio ingestion active during speaking, allowing incoming user speech to enter later Units for interruption handling.The design combines incremental audio processing, early speech synthesis, and writing that continues during TTS.
  • Inference pipeline: LWS processes audio continuously while generating visible writing and speech through coordinated inference Units.Listening Units produce incremental visible writing; Speaking Units continue listening while producing speech and reply cognition.
  • Inference pipeline: The token trace shows speech and visible writing diverging in granularity: one spoken sentence accompanies a detailed multi-sentence written analysis.The written channel remains the primary unconstrained output modality in this example.

B Data Construction Details

The data construction pipeline synthesizes causally consistent per-second annotations, aligns them with real audio, and uses explicit channel and state tags to organize full-duplex generation.

  • Stage 1: Offline cognitive synthesis: Per-second annotations are constrained to use only information revealed by the current time step, preventing access to future input.The constraint applies separately to listening-phase and reply-phase writing.
  • Stage 2: Alignment and sequencing: Stage 2 combines real audio, CTC character-level alignments, and Stage 1 outputs to construct final Unit token sequences.The global timeline inserts random silence intervals between turns to simulate conversational rhythm.
  • Stage 2: Alignment and sequencing: Units are assigned as Speaking Units when the AI has speech text, while Listening Units cover early, effective, and post-user-speaking phases.Speaking Units include reply cognition; Listening Units receive phase-appropriate writing content.
  • State separation and causality: Explicit state tags distinguish listening cognition, reply cognition, and speaking, reducing the mixture penalty caused by a unified cognition tag.The paper attributes the penalty to substantially different state-conditional distributions and their KL divergences.
  • State separation and causality: The listening-to-reply boundary is marked explicitly so speaking generation cannot use unfinished listening-phase content and violate temporal causality.A unified cognition tag would make the transition ambiguous.

D Data Construction Prompt Template

The prompt template creates per-second streaming reasoning and a concise spoken response from progressively revealed input, enforcing causal access and structured output constraints.

  • Input preparation: The template instructs the model to emulate streaming reasoning over audio that arrives one second at a time.At second N, only text from seconds 0 through N is available.
  • Input preparation: Substantive reasoning begins at t0 = min(2, T), while earlier seconds serve as ASR-style confirmation.T denotes the total number of input seconds.
  • Output structure: The voice response should be concise, friendly, and approximately 30%–40% of the original response without introducing new details.The final voice response must reflect the full written reply.
  • Causal constraints: The prompt requires stable per-second reasoning length and forbids mentioning content that arrives after the current second.These instructions preserve causal consistency throughout the streaming sequence.
  • Output structure: The required JSON contains a per-second streaming reasoning chain and a natural spoken paraphrase.The prompt uses the StreamingColloquialRewrite schema and structured parsing.

E Evaluation Prompts for Reply Quality

Reply-quality evaluation uses transcribed instructions and responses with separate GPT-5 judging prompts for factual accuracy, overall quality, and naturalness.

  • Evaluation setup: The evaluation prompts provide the transcribed user instruction and model response to GPT-5 judges.The placeholders are filled with the respective instruction and response transcriptions.
  • Overall quality: Overall quality is rated from 1 to 5 according to relevance, accuracy, concision, clarity, and completeness.The highest score represents an exceptionally relevant, accurate, concise, and on-point response.
  • Naturalness: Naturalness is rated from 1 to 5 according to how fluent, conversational, and everyday the spoken response sounds.The scale distinguishes robotic speech from responses resembling spontaneous talk.
  • Factual accuracy: Factual accuracy is rated from 1 to 5 based on correctness and absence of hallucinated or invented information.A score of 5 means every detail is accurate and contains no hallucination.
Loading 2606.07547v1…