PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems

1 Introduction

• Existing spoken dialogue systems combine ASR, response generation, and TTS, while multimodal LLMs have enabled end-to-end spoken communication.The introduction situates this work among LLM applications to response generation, ASR, and TTS.Source: §1 · p. 1
• Chain-of-Modality systems generate text questions and answers before speech answers, and speech sequences are much longer than text sequences.These are the two response-latency challenges identified for existing methods.Source: §1 · p. 1
• PSLM uses multiple input-output sequences to handle text and speech tokens, enabling parallel generation and allowing speech generation to begin after the speech question.The authors describe these sequences as “streams.”Source: §1 · p. 1
• The study tests whether PSLM preserves CoM response quality while improving latency, and whether multiple speech streams reduce latency without sacrificing quality.The introduction frames these as research questions RQ1 and RQ2.Source: §1 · p. 2

2.1 Speech Discretization

• PSLM uses HuBERT for speech tokenization and a non-autoregressive HiFi-GAN vocoder to reconstruct waveforms from speech tokens.The paper adopts the vocoder approach to reduce overall latency.Source: §2.1 · p. 2

2.2 Integrating LMs with a Speech Stream

• New speech embedding and output-projection layers surround unchanged Transformer layers, whose hidden features produce text and speech logits.Text and speech embeddings are summed before entering the Transformer; right-padding aligns text and speech sequence lengths.Source: §2.2 · p. 2

2.3 Introducing Multiple Speech Streams

• Multiple speech streams partition speech tokens across sequences, reducing Transformer sequence length to 1/S for an approximate S-fold single-instance speedup.Speech-stream losses are weighted by 1/S to balance their contribution against text-token losses.Source: §2.3 · pp. 2–3

2.4 Streaming Inference with HiFi-GAN

• For streaming inference, HiFi-GAN receives partial speech tokens and generates waveform fragments, with the first fragment available after ⌊R/2⌋+1 tokens.R denotes HiFi-GAN’s receptive field; non-causal convolution is used to maintain speech quality.Source: §2.4 · p. 3

2.5 Overall Latency

• Latency is measured from utterance end to the initial response; PSLM uses external ASR instead of prior TQ and TA generation, while speech tokenization overlaps ASR.The latency formulation includes ASR, speech-question prefill, decoding, and speech detokenization.Source: §2.5 · p. 3

3.1 Dataset

• Training used 1.8M written QA pairs; evaluation used 396 manually reviewed Japanese QA pairs converted to single-speaker synthetic speech.The evaluation set was filtered to exclude text questions or answers exceeding 140 characters.Source: §3.1 · p. 3

3.2 Configuration

• The models use rinna/nekomata-7b and were trained for 50k steps with batch size 16 on eight NVIDIA A100 GPUs.The model uses the GPT-NeoX library and Adam optimizer.Source: §3.2 · p. 3

3.3 Baselines

• Three CoM baselines differ in whether they receive only speech questions, transcribed text questions, or gold text questions.The three-stage training procedure was ineffective in the authors’ preliminary experiments, so they used the Section 3.2 configuration.Source: §3.3 · p. 4

3.4 Evaluation Metrics

• Automatic evaluation uses GPT-3.5 Turbo scores, character error rate, failure rate, and simulated median latency.The GPT-based scores cover text answers and speech-answer transcriptions; latency simulations set the LM decode rate to 50 tokens per second.Source: §3.4 · p. 4
• Human evaluators rated text naturalness, speech naturalness, and perceived response speed on five-point scales.Fifty samples were randomly selected, with twenty in-house workers rating twenty samples each.Source: §3.4 · p. 4

4.1 Automatic Evaluation

• PSLM achieved response-quality scores comparable to CoM while significantly improving latency, with a similar trend for PSLM-ASR versus CoM-ASR.Table 1 reports automatic evaluation results; its caption identifies T-score, S-score, failure rate, and latency values for gold-TQ inputs.Source: §4.1 · p. 4, Table 1 · p. 5
• Two speech streams achieved scores comparable to PSLM, whereas three streams degraded quality, supporting decoding in up to two streams without quality loss.PSLM-3x was trained with a batch size of 4 because of its increased parameter count.Source: §4.1 · p. 4

4.2 Human Evaluation

• In human evaluation, PSLM-ASR matched CoM-ASR in text naturalness, and its response-speed score was significantly higher than those of CoM-ASR and CoM-SQ.All evaluated methods scored higher than Ground-Truth for speech naturalness.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 Detailed Latency Analysis

• CoM latency rises linearly with text-answer length, while PSLM-ASR latency remains constant; the gap is especially marked for long answers.PSLM-2x-ASR further reduces latency; Figure 3 plots latency against TA length for different methods and TPS settings.Source: §4.3 · p. 5, Figure 3 · p. 5

• PSLM uses multiple input-output streams to generate text and speech tokens in parallel, and the study evaluates its effects on response quality and latency.Source: §5 · p. 5
• On spoken question answering, PSLM significantly reduced latency compared with existing methods while maintaining response quality.Source: §5 · p. 5
• Future work includes testing larger datasets and real speech, and extending PSLM to multi-turn dialogues.Source: §5 · p. 5

• PSLM requires an external ASR module for spoken dialogue use because it sacrifices ASR capability for faster response.Source: §6 · p. 6
• The experiments used single-speaker synthetic SQ and SA, which lack diversity in accent, rhythm, emotion, and timbre.Source: §6 · p. 6
• The study did not investigate multi-turn dialogue settings.Source: §6 · p. 6

A Sequence Length Distributions

• The training-set analysis calculates sequence-length distributions for SQ, TQ, TA, and SA and lists the results in Table 3.Source: A Sequence Length Distributions · p. 8
• CoM prompting generates about 70 text tokens—36.5 TQ plus 33.8 TA—before generating SA.The passage says eliminating these tokens can greatly reduce overall latency.Source: A Sequence Length Distributions · p. 8
• Speech tokens are more than 11 times longer than text tokens, highlighting the need for efficient speech-token generation.Source: A Sequence Length Distributions · p. 8

B Implementation Details of HiFi-GAN

• A waveform fragment for token i depends on tokens from i − ⌊R/2⌋ through i + ⌊R/2⌋.The local dependency allows waveform generation to begin before all SA tokens are generated.Source: B Implementation Details of HiFi-GAN · p. 8
• HiFi-GAN generates its first waveform fragment after Noffset = ⌊R/2⌋+ 1 tokens, then shifts the input window by one token for subsequent fragments.Source: B Implementation Details of HiFi-GAN · p. 8
• The model was trained for 300k iterations to generate 24 kHz waveforms from 50 Hz tokens, yielding R = 26.Input speech tokens were embedded into 256-dimensional features; upsampling rates were [8, 6, 5, 2], with other configuration kept as in Kong et al. (2020).Source: B Implementation Details of HiFi-GAN · p. 8

C ChatGPT Evaluation Prompt

• A prompt was used for ChatGPT-based evaluation.Source: C ChatGPT Evaluation Prompt · p. 8
• The original evaluation prompt was written in Japanese.Source: C ChatGPT Evaluation Prompt · p. 8
• The presented prompt is a translated version of the original.Source: C ChatGPT Evaluation Prompt · p. 8

D Speech Evaluation Instruction

• An instruction was used for speech evaluation.Source: D Speech Evaluation Instruction · p. 8
• The original speech-evaluation instruction was written in Japanese.Source: D Speech Evaluation Instruction · p. 8
• The presented instruction is a translated version of the original.Source: D Speech Evaluation Instruction · p. 8

• The ablations included PSLM variants trained without pretraining, without TQ, without SQ, and 2x and 3x variants without weighted loss.Source: E Ablation Study · p. 8
• Removing pretraining degraded all metrics and removing TQ caused large response-quality degradation, while removing SQ left scores comparable to PSLM.Source: E Ablation Study · p. 8, Table 4 · p. 9
• PSLM-2x-no-WL achieved almost comparable scores to PSLM, whereas PSLM-3x-no-WL degraded significantly, indicating weighted loss is especially effective as speech streams increase.Source: E Ablation Study · p. 8, Table 4 · p. 9
• For the current SQA task using synthetic speech, the no-SQ result suggests intonation, rhythm, and emotion are not essential.Source: E Ablation Study · p. 8