PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Joint Language Modeling for Speech Units and Text

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli

arXiv:2310.08715v1cs.CLcs.AIcs.SDeess.AS

TL;DR

Speech and text have mostly been modeled separately, despite the potential value of their complementary information. This paper trains a joint speech-unit and text language model and evaluates its cross-modal ability. Mixed speech-text data improves cross-modal ability and performance on automatic metrics and downstream tasks.

  • Problem

    Speech and text have largely been studied separately in language modeling, despite speech prosody and written text providing complementary information.

    Source: §1 · p. 1

  • Method

    The paper trains a joint autoregressive language model on speech units and text, proposes automatic cross-modal metrics, and evaluates transfer through downstream spoken-language-understanding tasks.

    Source: §1 · p. 2

  • Results

    Mixed speech-text data improves cross-modal ability and performance on both automatic metrics and downstream tasks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The proposed automatic metrics require no fine-tuning and are indicative of cross-modal transfer ability on downstream tasks.

    Source: §1 · p. 2

    The downstream evaluation covers only two SLUE tasks, and the study uses 350M-parameter models, leaving the effect of scaling unclear.

    Source: §7 · p. 9

Abstract

from arXiv · show

Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.

1 Introduction

Speech and text have largely been studied separately, despite complementary information in speech prosody and written text. This work develops a joint autoregressive model and examines how to mix the modalities.

1 Introduction

  • Speech prosody and written text offer complementary information, while written text is more abundant than any available speech dataset.Speech contains prosodic information absent from text, whereas sources such as Wikipedia provide structural knowledge missing from most speech datasets.Source: §1 · p. 1
  • The paper presents a joint autoregressive language model trained on both speech and text.Speech signals are converted to discrete units and combined with text for language-model training.Source: §1 · pp. 1–2
  • Automatic metrics requiring no fine-tuning are proposed to evaluate the model, and are reported as indicative of downstream cross-modal transfer ability.The paper also fine-tunes models on spoken-language-understanding tasks using speech or text and evaluates them in either modality.Source: §1 · p. 2
  • Larger-span SentencePiece speech units outperform local units from existing self-supervised models, while proposed speech-text mixing improves cross-modal ability.The tokenizer comparison is reported in Sec. 5.5.1; the mixing result is reported in Secs. 5.5.3 and 5.5.4.Source: §1 · p. 2

2 Related Work

Related work covers self-supervised speech representations, speech modeling without text, and joint speech-text transformers. This paper uses discrete speech units in a decoder-only joint language model focused on spoken-language content.

2.1 SSL speech models

  • Self-supervised HuBERT representations are quantized into discrete speech units, which are combined with text to train a single joint language model.The paper uses HuBERT (Hsu et al., 2021) and a quantizer to produce speech units.Source: §2.1 · p. 2

2.2 Textless NLP

  • Textless NLP models speech without textual data through a speech-to-unit tokenizer, unit language model, and unit-to-speech detokenizer.The unit language model predicts subsequent units from prior context and can generate unit sequences autoregressively.Source: §2.2 · p. 2

2.3 Joint speech-text transformers

  • Prior joint speech-text transformers mainly use encoder-decoder or encoder-only designs, often requiring multiple losses and explicit paired-data alignments.The paper explores decoder-only joint modeling; unlike VALL-E (Wang et al., 2023), which targets controllable text-to-speech synthesis, it focuses on spoken-language content.Source: §2.3 · p. 2

3 Method

The method trains a next-token language model on speech and text sequences, using tokenization to address differences in their sequence lengths. It constructs paired data through either concatenation or word-aligned alternation.

3 Method

  • The SUTLM predicts each sequence token from its prior context and is trained to maximize the data's log-probability.The data consists of sentence sequences whose tokens may be text or speech units.Source: §3 · p. 3

3.1 Speech-only: unit LM (uLM)

  • HuBERT representations are clustered with k-means, repeated units are removed, and SentencePiece combines the remaining units into larger chunks.The subword tokenizer is used to mitigate the sequence-length mismatch between phone- or sub-phone-level speech units and text chunks.Source: §3.1 · p. 3

3.2 Text-only: text LM (tLM)

  • A separate SentencePiece tokenizer converts the text-only corpus into subword tokens with a vocabulary of around 45k..Source: §3.2 · p. 3

3.3 Concatenated speech-text (CST)

  • Concatenated speech-text (CST) sequences join speech units and their transcriptions in uLM and tLM formats, explicitly indicating their correspondence.The format is intended to encourage the model to learn dependence between speech units and corresponding text transcriptions.Source: §3.3 · p. 3

3.4 Alternating speech-text (AST)

  • Alternating speech-text (AST) uses forced-aligned word boundaries as switching points, alternating speech-unit and text chunks with modality-switch tokens.The sequence uses <U2T> when switching from speech units to text and <T2U> when switching from text to speech units.Source: §3.4 · p. 3

4 Evaluation Metrics

The section introduces automatic metrics for evaluating speech-text joint language models without fine-tuning and describes their use alongside downstream spoken language understanding evaluations. It presents Context Retrieval Accuracy (CRA), external-LM perplexity, and cross-modal SLUE testing.

4 Evaluation Metrics

  • The proposed Context Retrieval Accuracy (CRA) evaluates an SUTLM’s cross-modal ability without fine-tuning, addressing the lack of a reliable cross-modal metric.The authors report that their automatic metrics indicate downstream cross-modal transfer ability.Source: §4 · pp. 3–4

4.1 Context Retrieval Accuracy (CRA)

  • CRA tests whether a language model assigns a continuation higher conditional probability with its matching prompt than with random prompts.For cross-modal CRA, prompt and continuation use different modalities; this measures whether the SUTLM learns shared speech-text representations.Source: §4.1 · p. 4

4.2 Perplexity under External LM (PELM)

  • PELM scores sampled SUTLM continuations by transcribing prompt-continuation sequences and measuring continuation perplexity with an external text language model.The method controls the sampled continuation’s modality; the external model is OPT-6.7B.Source: §4.2 · pp. 4–5

4.3 Evaluation on SLUE tasks

  • SLUE evaluation fine-tunes models on sentiment analysis and named entity recognition using speech or text, then tests them with either input modality.Both speech data and transcriptions are provided for the tasks.Source: §4.3 · p. 5

5 Experiments

The experiments use speech-only, mixed speech-text, and text-only training data, HuBERT-derived speech units, and a shared transformer language model. The training batches balance the three data formats equally.

5.1.1 Speech-only

  • The speech-only dataset is a 5% subsample of a corpus collection, totaling 65k hours of speech.The collection includes Multilingual LibriSpeech, VoxPopuli, Common Voice, and Spotify Podcast & People’s Speech.Source: §5.1.1 · p. 5

5.1.2 Mixed speech-text (CST and AST)

  • The mixed speech-text dataset uses MLS and VoxPopuli without subsampling and contains 45k hours of speech and 2.7B words.Source: §5.1.2 · p. 5

5.1.3 Text-only

  • The text-only dataset combines OPT web data, Wikipedia, and LibriLM, then subsamples 5% to yield 8.5B subwords.Source: §5.1.3 · p. 5

5.2 SSL speech tokenizer

  • Speech units come from a HuBERT Base model pretrained on 221K hours of unlabeled speech in eight languages, with final-layer representations clustered into 2000 k-means units.Source: §5.2 · p. 5

5.3 Model architecture and training

  • The 350M-parameter, 24-layer transformer is trained for 500k updates, with speech-only, mixed, and text-only data resampled equally in each batch.The architecture uses 16 attention heads, 1024-dimensional embeddings, and a 4096-dimensional feed-forward layer.Source: §5.3 · p. 5

5.4 Evaluation setup

This section evaluates automatic continuation metrics and SLUE transfer across tokenizer and mixing choices. AST improves cross-modal alignment and downstream transfer, with task-specific limits.

5.4.1 Automatic Metrics

  • Automatic-metric experiments use moderate-length sentences; CRA covers 100 utterances and PELM 500, with continuations sampled at temperature 0.6 and p = 0.95.Continuations are truncated to 10 words; CRA is more computationally expensive, and longer sequences can reduce metric sensitivity.Source: §5.4.1 · p. 6

5.4.2 Downstream Tasks

  • SLUE-SA uses Macro F1 after self-attention pooling, while SLUE-NER uses Micro F1 for generated transcription and entity-tag outputs; both report dev results.The SLUE-NER test set is not publicly available.Source: §5.4.2 · p. 6

5.5.1 What kind of speech units works the best?

  • SP 10k performs best on PELM, SLUE-SA, and SLUE-NER, but its CRA is slightly worse than with original HuBERT units.The authors hypothesize that nearly 1.0 u2u CRA across unit types reflects low-level acoustic information; SP 10k sequences are shorter and may support more coherent continuations.Source: §5.5.1 · p. 6

5.5.4 Does alternating speech-text (AST) help learn shared representations?

  • AST improves cross-modal CRA over CST and further improves SLUE-SA cross-modal transfer.The comparison covers CRA directions and SLUE-SA fine-tuning/evaluation modalities.Source: Table 3 · p. 7, Table 4 · p. 7, §5.5.4 · p. 8

5.5.2 Do we need paired data to learn shared representations?

  • Without paired data, cross-modal CRAs and SLUE-SA scores are nearly random, and SLUE-NER shows no cross-modal transfer.The unpaired model also performs worse than the speech-unit-only counterpart on SLUE-NER.Source: §5.5.2 · p. 6, Table 3 · p. 7, Table 4 · p. 7

5.5.3 Does concatenated speech-text (CST) help learn shared representations?

  • CST raises cross-modal CRAs over unpaired training, while its SLUE-SA text-to-speech score exceeds direct speech fine-tuning, 0.51 vs. 0.48.For SLUE-NER, CST improves over speech-unit-only training when fine-tuned on speech; on SLUE-SA, extra speech-text data can perform worse than speech-unit-only training.Source: §5.5.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

5.5.4 Does alternating speech-text (AST) help learn shared representations?

  • AST continuation evaluation uses CRA and PELM; lower PELM for u2u and t2u than u2t and t2t may reflect unit-transcriber recognition errors.PELM for t2u is lower than for u2u, while u2t is only marginally worse than t2t.Source: Table 3 · p. 7, §5.5.4 · p. 8
  • On SLUE-NER, AST yields non-trivial speech-to-text transfer after speech fine-tuning, but no text-to-speech transfer after text fine-tuning.The latter task setup is described as nearly trivial because its input and output are nearly identical apart from entity tags.Source: Table 5 · p. 7, §5.5.4 · p. 8

5.6 Limitations of PELM

PELM can misrepresent continuation quality when models repeat prompts, and the SUTLM’s cross-modal continuations remain only locally consistent. The reported results also identify temperature as a possible contributor to lower PELM scores.

5.6 Limitations of PELM

  • The SUTLM continues after a cross-modal prompt, but its continuation is only locally consistent, which the authors attribute to its 350M-parameter architecture.Source: §5.6 · p. 8
  • PELM fails to reflect continuation quality when a model simply repeats the prompt.For row (E), u2t and t2u continuations repeat prompt content.Source: §5.6 · p. 8
  • For row (E), prompt bigram proportions are 0.53 for u2t and 0.42 for t2u, consistent with repeated prompt content.Source: §5.6 · p. 8
  • The u2t and t2t PELMs are below ground-truth PELM, which the authors say is likely due to temperature 0.6 hurting diversity and coherence.Source: §5.6 · p. 8

5.7 Implications for SLU Downstream Tasks

Mixed speech-text training improves cross-modal performance on SLU tasks. The reported pattern differs by task: SLUE-SA supports zero-shot transfer between modalities, while SLUE-NER reports non-trivial transfer from speech to text but not vice versa.

5.7 Implications for SLU Downstream Tasks

  • Mixed speech-text data enables zero-shot transfer between speech and text on SLUE-SA.Source: §5.7 · p. 8
  • The SLUE-SA setup removes the SUTLM output layer and attaches a classification head so outputs are valid classes.Source: §5.7 · p. 8
  • On SLUE-NER, mixed speech-text data directly improves performance, with non-trivial transfer from speech to text but not vice versa.Source: §5.7 · p. 8
  • The authors note that different fine-tuning strategies can produce different results.Source: §5.7 · p. 8

6 Conclusion

The study develops joint modeling of speech units and text, evaluating the models with automatic metrics and downstream fine-tuning. It reports improved cross-modal ability and performance on both metric evaluations and downstream tasks when using mixed speech-text data.

6 Conclusion

  • Using mixed speech-text data improves cross-modal ability and performance on automatic metrics and downstream tasks.Source: §6 · p. 8
  • The study develops evaluation metrics and fine-tunes the joint model on speech and text data.Source: §6 · p. 8
  • The authors hope the work motivates further exploration of speech-text integration for more comprehensive language modeling.Source: §6 · p. 8
  • Future work could investigate the optimal speech-text data balance and multimodal data beyond speech and text.Source: §6 · p. 8

7 Limitations

The approach depends on HuBERT speech tokenization and an off-the-shelf recognizer for word-level alignment, components that may be harder to obtain or less effective for lower-resource languages. The study also has computational, benchmark-coverage, and model-scale limitations.

7 Limitations

  • For lower-resource languages, the HuBERT tokenizer and off-the-shelf recognizer may be harder to obtain or may perform less well.Source: §7 · p. 9
  • CRA complexity grows at O(N^2), potentially making evaluation expensive at larger utterance counts or model sizes; downstream evaluation covers only two SLUE tasks.Source: §7 · p. 9
  • The study tests only 350M-parameter SUTLMs, leaving the effect of scaling unclear.Source: §7 · p. 9

A Appendix

The appendix presents speech/text continuation examples; speech continuations are transcribed for display.

A Appendix

  • The U2U sample includes a continuation about seeking a home and the poor's hospitality and care.Source: A Appendix · p. 12
  • The T2T sample repeats the phrase “a hundred toy guns.”Source: A Appendix · p. 12
  • Table 6 presents speech/text continuation examples, with speech continuations transcribed.Source: Table 6 · p. 12
Loading 2310.08715v1…