Textually Pretrained Speech Language Models

1 Introduction

• Text’s dominance on the web limits the ability to construct large SpeechLMs, while the benefit of transferring textual LMs remains uncertain because speech and text tokens have different granularities.Earlier joint speech-text LMs focused on translation or transcription tasks.Source: §1 · pp. 1–2
• TWIST initializes SpeechLMs from pretrained textual LMs and improves performance consistently across the evaluated automatic and human metrics.The paper also analyzes design choices and scale, presents large SpeechLMs, and introduces two spoken StoryCloze versions.Source: §1 · p. 2
• The paper presents SpeechLMs with up to 13B parameters and approximately 150k speech hours of training data.These are described as the largest SpeechLMs to date to the authors’ knowledge.Source: §1 · p. 2
• The two spoken StoryCloze versions target fine-grained temporal commonsense relations and coarse global sentence coherence, respectively.They are created by synthesizing the original test stories with either the original distractor or a randomly chosen one.Source: §1 · p. 2

2.1 Background

• The GSLM pipeline separates speech tokenization, SpeechLM modeling, and token-to-speech conversion into modules trained separately.The paper uses a token-based vocoder approach for its token-to-speech models.Source: §2.1 · pp. 2–3
• Speech tokenizers encode speech into discrete tokens, and SpeechLMs model those tokens without textual transcriptions.The described setup encodes speech representations and applies k-means to obtain discrete tokens; the paper follows HuBERT based on prior reported overall performance.Source: §2.1 · p. 3

2.2 Textually Warm-Initialized Speech Transformer Language Models

• TWIST replaces a pretrained text LM’s vocabulary and lookup table with speech-token equivalents, keeps the rest of the network unchanged at initialization, and then trains the full SpeechLM on speech data.The authors note that speech tokens cover 20–40 ms windows while text tokens span longer concepts such as subwords.Source: §2.2 · pp. 3–4

3.1 Datasets

• The training collection combines five public English-filtered speech datasets, totaling about 150k hours, or 13.5B tokens at 50Hz and 9B at 25Hz.The authors describe this as the first SpeechLM training effort at this data scale.Source: §3.1 · p. 4

3.2 Model & Hyperparameters

• The model comparisons include OPT-125M, OPT-350M, and OPT-1.3B, with TWIST warm starts compared against randomly initialized COLD-INIT models.BLOOM and Pythia are also tested at sizes equivalent to OPT-350M and OPT-1.3B; SpeechLM parameter counts are about 10–25% below their text models because of smaller speech embeddings.Source: §3.2 · p. 4
• HuBERT representations are quantized with 100, 200, or 500 clusters, and the study also trains a lower-sampling-rate HuBERT tokenizer on mixed speech data.The setup uses HuBERT followed by k-means and a token-based HiFi-GAN vocoder with a duration predictor.Source: §3 · p. 4, §3.2 · p. 5

3.3 Evaluation

• Evaluation covers zero-shot modeling, human ratings of speech continuations, and two spoken StoryCloze variants synthesized from test stories.The variants distinguish original adversarial endings, which test fine-grained causal and temporal commonsense, from randomly sampled endings, which test continuation coherence.Source: §3.3 · pp. 5–6
• Zero-shot evaluation uses speech-token PPL, sWUGGY for lexical modeling, and sBLIMP for syntactic modeling, while human raters score continuation naturalness on a 1–5 scale.PPL is not comparable across tokenizers; human evaluation samples 150 prompts per method and uses 10 raters per sample.Source: §3.3 · p. 5
• Figure 2 reports that TWIST is more sample-efficient and converges faster than COLD-INIT.Panel (a) plots PPL against training-set and model sizes; panel (b) plots validation PPL against training steps.Source: Figure 2 · p. 6
• Transcription-based PPL and auto-BLEU are affected by vocoder quality, ASR errors, and sampling, and preliminary results showed high variance.The authors therefore emphasize a human study and report these metrics for their main model family in the appendix.Source: §3.3 · p. 6

Scaling improves SpeechLMs

• TWIST initialization consistently improves results across metrics, and using 10% of the data yields comparable or better performance than COLD-INIT with 100% across model scales.Increasing model size and dataset size improves performance in almost all cases.Source: §4 · p. 7

TWIST converges faster

• TWIST reaches the same validation PPL in about one quarter of the training updates required by COLD-INIT.Source: §4 · p. 7

Not all warm initializations are equally important

• BLOOM- and Pythia-initialized SpeechLMs show trends similar to OPT, while ImageGPT initialization substantially underperforms COLD-INIT.Source: §4 · p. 7

Human evaluation

• TWIST-7B performs better than smaller models in human speech-generation evaluation, while TWIST outperforms COLD-INIT.Source: Figure 3 · p. 8

Speech Large Language Models

• The authors introduce 7B- and 13B-parameter SpeechLMs initialized from LLaMA-7B/13B [Touvron et al., 2023].Source: §4 · p. 8
• TWIST-13B outperforms all evaluated methods on both sWUGGY setups; LLaMA-7B/13B initialization improves PPL by ~8/10% and sWUGGY by ~1.7/2.5% relative to TWIST-1.3B.TWIST-13B’s sBLIMP results are lower than Borsos et al. [2022], whose method the authors describe as orthogonal.Source: §4 · p. 8

Spoken StoryCloze

• On spoken StoryCloze, models and humans perform better on continuation coherence (TSTORYCLOZE) than on finer-grained relations (SSTORYCLOZE), and model scaling improves both benchmarks.The SpeechLM–human performance gap is about 15% on TSTORYCLOZE and widens substantially on SSTORYCLOZE.Source: §4 · p. 8, Table 4 · p. 8

Human evaluation

• In an example, COLD-INIT-1.3B makes grammatical errors, both 1.3B models stay on topic, and TWIST-7B gives semantically richer continuations.Source: Figure 4 · p. 9, §4 · p. 9

• Textual language models developed into foundational models used as bases for downstream tasks [Shoeybi et al., 2019, Radford et al., 2019, Brown et al., 2020, Lieber et al., 2021, Zhang et al., 2022, Hoffmann et al., 2022, Chowdhery et al., 2022].Source: §5 · p. 9
• Other-modality work adapted language-modeling approaches to image generation using pixel-space masked prediction or discrete image tokens [Chen et al., 2020, Yu et al., 2022, Ramesh et al., 2021, Chang et al., 2023].Source: §5 · p. 9
• Speech and audio research used discrete representations for speech modeling, conversion, dialogue, audio generation, music, and speech-to-speech translation [Lakhotia et al., 2021, Kharitonov et al., 2021, Borsos et al., 2022].Related approaches also modeled music and combined visual and speech tokens for resynthesis tasks [Kreuk et al., 2022, Agostinelli et al., 2023, Donahue et al., 2023, Lee et al., 2021, 2022, Popuri et al., 2022, Hsu et al., 2022].Source: §5 · p. 9
• Joint speech-text training had improved translation and transcription tasks [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022, Ao et al., 2021, Chen et al., 2023], but prior work did not focus on SpeechLM modeling and generation.Source: §5 · pp. 9–10

• Textual initialization improves speech-based language-model performance on automatic and human-generated measures.Source: §6 · p. 10
• The study evaluates speech tokenizers, model and dataset sizes, model architecture, and modality pretraining to examine their effects on performance.Source: §6 · p. 10
• The authors present 7B- and 13B-parameter SpeechLMs and two spoken StoryCloze versions to assess contextual understanding.Source: §6 · p. 10

Limitations

• SpeechLMs’ limited semantic understanding may lead to ungrammatical, off-topic, or inaccurate responses, and TWIST showed no major semantic knowledge transfer.The paper says this limitation is shared broadly by SpeechLMs and should be studied further.Source: §7 · p. 10
• At 50Hz or 25Hz, tokenization yields 750 or 500 tokens for roughly 30-second segments, contributing to long inference time and harder optimization.Source: §7 · p. 10

Broader impact

• SpeechLMs may extend audio access to applications and unwritten or sparsely written languages, while also sharing risks such as manipulation, bias, and toxic or untrustworthy generations.Source: §7 · p. 10

A.1 Model & hyperparameters

• Most models use batch size 64, 400k steps, an inverse-sqrt schedule, 100 warmup steps, and scenario-specific learning rates.Maximum/final learning rates are 4e-4/8e-5 without a pretrained LM and 8e-5/2.5e-5 with one.Source: §A.1 · p. 16
• LLaMA-7B/13B instead use cosine scheduling, 500 warmup steps, batch size 1024 across 32 GPUs, and 75k steps.Their maximum and final learning rates are 1e-4 and 1e-5, respectively.Source: §A.1 · p. 16
• The HuBERT tokenizer’s fourth iteration adds a convolutional layer with strides yielding 25Hz, 16.6Hz, or 12.5Hz features.Source: §A.1 · p. 16
• Early ablations found 25Hz features with 500 tokens best for language modeling, so the authors train models on those tokens and compare other tokenizations.Source: §A.1 · p. 16

Automatic evaluation of speech generation

• Text-PPL is sensitive to sampling parameters, vocoder quality, and ASR errors, and repeated speech can score well despite poor diversity.The authors therefore pair text-PPL with diversity measures.Source: §A.2 · p. 16
• The evaluation thresholds temperature using ASR confidence, then calibrates it so generation diversity matches the original utterance as much as possible.The diversity measure is auto-BLEU.Source: §A.2 · p. 16
• Speech prompts comprise 1,024 test examples from LS-clean, LS-other, and LL, with roughly 3-second prompts and 10-second continuations.Calibration uses 128 randomly selected examples from corresponding validation sets.Source: §A.2 · p. 16
• Table 5 reports text-PPL and auto-BLEU for original continuations and generations from the OPT models using the 25Hz tokenizer.Source: §A.2 · p. 16, Table 5 · p. 16
• TWIST models outperform COLD-INIT models on text-PPL, while calibrated diversity is comparable across models.The 350M models have worse text-PPL than 125M models but better auto-BLEU diversity.Source: §A.2 · p. 16, Table 5 · p. 16
• Figure 5 presents PPL as a function of training-set and model size for models using the 200-token, 50Hz tokenizer.Source: Figure 5 · p. 17

A.3 Speech resynthesis results

• All evaluated tokenizers incur WER loss in resynthesis, while 500 tokens at 25Hz provide the best performance.Resynthesis bypasses the SpeechLM and measures recovery of speech content after tokenization; Table 6 reports WER across token counts and frequencies.Source: §A.3 · p. 17, Table 6 · p. 17

A.4 Model and data scaling results

• The full scaling results report PPL, sWUGGY, and sBLIMP across model sizes, data sizes, and TWIST settings.Figure 5 gives the PPL counterpart for the 200-token, 50Hz tokenizer.Source: §A.4 · p. 17, Table 7 · p. 18

A.5 The effect of LM architecture

• SpeechLMs initialized from text reach better performance across all metrics; BLOOM and Pythia comparisons cover roughly 350M parameters with 25Hz and 50Hz tokenizers and roughly 1.3B with 25Hz.Source: §A.5 · p. 17

A.6 The effect of different modality pretraining

• ImageGPT-pretrained models perform much worse than text-pretrained models, while scratch-trained models achieve performance comparable to previously reported models.Table 10 reports PPL, sWUGGY, and sBLIMP and indicates that image pretraining substantially hurts performance.Source: §A.6 · pp. 17–18, Table 10 · p. 19

A.7 Full human evaluation (MMOS) results

• Table 11 provides the full human-evaluation results as MMOS means with 95% confidence intervals.Source: Table 11 · p. 19, §A.7 · p. 19