PaperlayerView exact revision on arXiv

Source-linked AI summary

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, Furu Wei

arXiv:2305.16107v1cs.CLcs.SDeess.AS

TL;DR

VALL-E X is designed only for speech synthesis, leaving open whether one decoder-only generative model can cover recognition, synthesis, and translation. VIOLA uses discrete codec tokens to unify speech and text tasks in a conditional language model. Evaluations report support for single-modal and cross-modal tasks, with in-context learning retained for speech synthesis.

  • Problem

    VALL-E X is designed only for speech synthesis, leaving open whether one decoder-only generative model can handle speech recognition, synthesis, and translation.

    Source: §1 · p. 1

  • Method

    VIOLA converts speech to discrete codec codes and uses a decoder-only Transformer with task and language IDs to model speech and text tasks as conditional language modeling.

    Source: §1 · p. 1

  • Results

    Evaluations show VIOLA supports single-modal and cross-modal speech and text tasks while maintaining in-context learning for speech synthesis.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments demonstrate the effectiveness of codec-based decoder-only models across speech recognition, translation, and synthesis tasks.

    Source: §5 · p. 9

    VIOLA is optimized with supervised data, demonstrates in-context learning only for speech synthesis, and supports only cascaded inference for speech-to-text and speech-to-speech translation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.

1 Introduction

Against a backdrop of convergence in model architectures, training techniques, and inference methods, the paper asks whether one decoder-only generative model can handle speech recognition, synthesis, and translation. VIOLA applies that approach across speech and text tasks and reports support for single-modal and cross-modal tasks.

  • Recent work shows convergence in model architectures, training techniques, and inference methods, while generative models show potential for few-shot and zero-shot NLP tasks.Source: §1 · p. 1
  • VALL-E X extends codec language modeling to cross-lingual synthesis and speech-to-speech translation, but is designed only for speech synthesis.This limitation motivates asking whether one decoder-only generative model can handle recognition, synthesis, and translation.Source: §1 · p. 1
  • VIOLA unifies speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks as conditional codec language modeling with a decoder-only Transformer.It converts continuous speech into discrete codec codes with an offline EnCodec model and uses task and language IDs to distinguish tasks and languages.Source: §1 · p. 1
  • VIOLA supports evaluated single-modal and cross-modal speech and text tasks while maintaining in-context learning for speech synthesis.The evaluations cover speech-to-text recognition and translation, text-to-text translation, text-to-speech synthesis, and speech-to-speech translation.Source: §1 · p. 2

2 Related Work

Prior work spans task-specific speech architectures, universal speech representations, and encoder-decoder models for spoken-language tasks. VIOLA positions a decoder-only Transformer as an approach for all spoken-language tasks, extending codec-based models beyond synthesis.

2 Related Work

  • Earlier speech systems used varied architectures, including CTC, AED, transducer, and duration-based models, while NLP explored encoder-only, decoder-only, and encoder-decoder models.Source: §2 · p. 2
  • HuBERT and WavLM learn universal speech representations, while SpeechT5 uses a universal encoder-decoder and SpeechNet uses modality-specific encoders and decoders.Whisper uses an encoder-decoder framework for speech recognition and translation.Source: §2 · p. 2
  • VIOLA extends VALL-E and VALL-E X beyond synthesis to more speech tasks with a decoder-only Transformer.The paper describes it as the first quantitative exploration of codec-based autoregressive models across various spoken-language tasks.Source: §2 · p. 3
  • SpeechGPT uses HuBERT hidden units, which contain less acoustic information than codec codes, and reports no quantitative cross-modal instruction-following or speech-dialogue results.Source: §2 · p. 3

3 Multi-Task Codec Language Model

VIOLA extends the autoregressive codec language-model approach to recognition, translation, and synthesis by converting speech and text tasks into token-sequence modeling. Its multi-task model combines task and language IDs with task-specific inputs and outputs, then supports several tasks through its training and inference procedures.

3.1 Background

  • VIOLA builds on VALL-E (X), which models TTS with autoregressive prediction of first-layer acoustic tokens and non-autoregressive generation of the remaining layers.VIOLA extends this approach to additional speech tasks.Source: §3.1 · p. 3

3.2 Problem Formulation

  • VIOLA formulates speech and text tasks as conditional codec language modeling by converting speech to discrete codec tokens and text to phoneme sequences.The modeled tasks include ASR, MT, and TTS, whose inputs and outputs vary by task.Source: §3.2 · p. 3

3.3.1 Multi-Task Auto-Regressive Codec LM

  • The autoregressive codec model combines embedding modules, a Transformer decoder, and a prediction layer, with language and task IDs distinguishing inputs.Tokens in the inputs can attend across input positions; output tokens attend to prior outputs and all inputs.Source: §3.3.1 · p. 4, Figure 2 · p. 4

3.3.2 Pre-Net and Pos-Net

  • Semantic and acoustic tokens are encoded separately; acoustic embeddings are averaged across layers and passed through a unidirectional LSTM.The LSTM produces acoustic features for the autoregressive model; the non-autoregressive synthesis model also uses an LSTM to encode acoustic tokens.Source: §3.3.2 · p. 4

3.4 Training Objectives

  • The model jointly optimizes ASR, text-to-text translation, and TTS objectives, with task IDs identifying the task.ASR predicts phonemes from eight layers of acoustic tokens; MT maps source to target phonemes; TTS predicts first-layer acoustic tokens from phonemes.Source: §3.4 · p. 5

3.5 Inference Methods

  • VIOLA uses a cascaded pipeline for speech-to-text and speech-to-speech translation, leaving end-to-end versions for future work.Inference uses sampling for synthesis and beam search for other tasks; five synthesis candidates are scored by speaker similarity or by speaker similarity and WER.Source: §3.5 · p. 5

4 Experiments

The experiments evaluate VIOLA across speech and text tasks, using multilingual datasets, discrete speech tokens, and decoder-only Transformer models. Results compare task performance with strong baselines and examine acoustic-token depth and LSTM embedding modules.

4.1 Data

  • VIOLA is evaluated on WenetSpeech development for speech-to-phoneme ASR, WMT2020 test for MT, EMIME for ZH⇒EN S2TT, zero-shot Chinese-prompted EN TTS and ZH⇒EN S2ST, and LibriSpeech dev-clean/test-clean for EN TTS.Training uses 10,000 hours of Chinese multi-domain labeled speech from WenetSpeech and 60,000 hours of English unlabeled audiobook speech from LibriLight; MT training uses AI Challenger and WMT2020, with 63M English-Chinese sentence pairs in conversion, drama, and news domains.Source: §4.1 · pp. 5–6

4.2 Data Pretreatment

  • Speech is encoded as EnCodec acoustic tokens, while ASR transcripts and MT text are converted to phoneme sequences for model training.EnCodec uses a 6 kbps bandwidth and produces eight tokens per frame; phoneme conversion uses dataset lexicons and the BigCiDian IPA-based unified set.Source: §4.2 · p. 6

4.3 Model Architecture

  • VIOLA uses a Transformer decoder with layer-specific acoustic-token embeddings and a three-layer unidirectional LSTM; 12-layer and 18-layer variants have 178M and 250M parameters.Comparisons include encoder-decoder baselines and single-task decoder-only language models.Source: §4.3 · p. 6

4.4 Training Details

  • The multi-task model is trained jointly on ASR, MT, English TTS, and Chinese TTS for 800K steps on 32 V100 GPUs.Training uses a maximum 20-second sentence length and accumulates the four task losses for each step.Source: §4.4 · p. 6

4.5 Evaluation Metrics

  • Evaluation uses PER and BLEU for recognition and translation, and WER, speaker similarity, and speech naturalness for generated speech.Generated TTS and S2ST speech is transcribed by a HuBERT-Large ASR model for WER and BLEU evaluation; speaker similarity is measured with an automatic speaker-verification model.Source: §4.5 · pp. 6–7

4.6 Main Results

  • VIOLA performs comparably to strong baselines across speech recognition, machine translation, speech-to-text translation, and speech synthesis, with task-specific advantages and trade-offs.VIOLA (18L) reaches 56.97 BLEU on WMT2020 with 250.6M parameters versus AED's 242.5M; on S2TT it scores 55.85 versus 55.98 with 48.7% fewer parameters.Source: §4.6 · pp. 7–8
  • VIOLA (18L) achieves best reported zero-shot ZH⇒EN S2ST results, with average SS 0.50, BLEU 47.27, and speech naturalness 3.55.The S2ST evaluation uses a cascaded ASR, MT, and TTS pipeline.Source: §4.6 · p. 8

4.7 Analysis

  • With eight acoustic-token layers, speech-recognition PER falls 54.5% relative to one layer; adding LSTM embedding modules reduces ASR PER by 8.7% and improves TTS results.VIOLA with LSTMAT+NAT has 12.5% lower average WER than VALL-E X.Source: §4.7 · pp. 8–9, Table 6 · p. 8, Table 7 · p. 9

5 Conclusion

VIOLA is an autoregressive codec language model for speech and text tasks. By representing speech as discrete acoustic tokens, it formulates spoken processing as one conditional language-modeling task, with experiments demonstrating effectiveness across tasks.

5 Conclusion

  • VIOLA is an autoregressive codec language model for multimodal tasks involving speech and text.The paper reports experiments across speech recognition and translation, machine translation, text-to-speech synthesis, and speech-to-speech translation.Source: §5 · p. 9
  • Discretizing speech into acoustic tokens lets spoken processing tasks share one conditional language-modeling task and optimization objective.Source: §5 · p. 9
  • Experiments across speech-to-text recognition and translation, machine translation, text-to-speech synthesis, and speech-to-speech translation demonstrate VIOLA's effectiveness and superiority.Source: §5 · p. 9

6 Limitations

VIOLA has three stated limitations: it uses supervised data, shows in-context learning only for speech synthesis, and lacks end-to-end inference for speech translation tasks.

6 Limitations

  • VIOLA is optimized with supervised data and does not fully use large-scale text corpora or unlabeled speech.Source: §6 · p. 9
  • VIOLA shows in-context learning ability for speech synthesis, but not for all speech-processing tasks.Source: §6 · p. 9
  • For speech-to-text and speech-to-speech translation, VIOLA supports cascaded inference but lacks end-to-end capability.Source: §6 · p. 9
Loading 2305.16107v1…