Source-linked AI summary
AudioPaLM: A Large Language Model That Can Speak and Listen
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, Christian Frank
TL;DR
Speech systems often require separate models for understanding, translation, and generation, limiting unified multimodal processing. AudioPaLM extends a pretrained text LLM with discrete audio tokens and trains it across speech-text tasks. It achieves state-of-the-art speech translation, competitive speech recognition, zero-shot translation on unseen language pairs, and voice transfer, while depending strongly on audio-tokenizer quality and full-model finetuning.
Problem
Existing speech and text systems are heterogeneous, motivating a single model that can process and generate both modalities across speech-text tasks.
Method
AudioPaLM expands a pretrained text-only decoder’s vocabulary with discrete audio tokens and finetunes the unified model on mixed speech-text tasks.
Results
AudioPaLM exceeds baselines on AST and S2ST, is competitive on ASR, and performs zero-shot AST for unseen language combinations.
Takeaways & Limitations
Text-model pretraining transfers to audio tasks, while AudioPaLM also supports voice transfer across languages from a short spoken prompt.
Takeaways & Limitations
AudioPaLM depends strongly on audio-tokenizer quality and requires finetuning the whole model rather than freezing most pretrained weights.
Abstract
from arXiv · showhide
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples
1 Introduction
AudioPaLM unifies speech and text modeling in one decoder-only multimodal system. It combines a joint vocabulary and text-model initialization to support multiple speech-text tasks, including translation and recognition.
- AudioPaLM combines text and audio vocabularies so one model can train in both directions across speech-text tasks.The motivation is to let a single multimodal model handle tasks traditionally served by separate systems.
- A joint vocabulary and task markup let AudioPaLM process arbitrarily interleaved speech and text with one decoder-only architecture.The model can consume and produce both modalities while using textual tags to specify tasks.
- Initializing AudioPaLM from PaLM or PaLM-2 transfers pretrained linguistic and common-sense knowledge to speech processing.The architecture permits reuse of a large text model’s weights during multimodal training.
- AudioPaLM supports ASR, AST, S2ST, TTS, and related speech-text applications in a unified training setup.The reported contributions include state-of-the-art AST and S2ST results, competitive ASR, voice transfer, and zero-shot AST.
2 Related work
Prior work combines modality-specific encoders, text decoders, or discrete speech representations for multimodal and speech generation tasks. AudioPaLM instead uses a decoder-only model that can generate arbitrary audio and text tokens.
- Multimodal encoders aim to learn richer, aligned representations across modalities for downstream inter-modality tasks.These approaches include masked-language-model and contrastive training strategies.
- Encoder-decoder systems such as Flamingo, PaLI, and Whisper combine audio or vision encoders with text decoders but output only text.Pretrained components can sometimes be frozen during multimodal finetuning.
- AudioPaLM’s decoder-only design models arbitrary audio and text tokens rather than restricting generation to text.Figure 1 shows pretrained text-model expansion with audio tokens and downstream audio reconstruction.
- Discrete speech representations support autoregressive speech generation, trading temporal modeling or reconstruction fidelity depending on the representation.Self-supervised representations model long-term patterns, while neural codecs provide higher-fidelity reconstruction with less temporal compression.
- SPEAR-TTS uses a text encoder and AudioLM-style decoder to synthesize speech with only 15 minutes of labeled speech.Its low-data result suggests close correspondence between text and semantic speech representations.
- Speech-to-speech translation systems range from cascades of ASR, MT, and TTS to direct or discrete-representation approaches.Direct systems can preserve acoustic information and potentially reduce compound errors and computation.
3 Method
AudioPaLM tokenizes speech and text into a shared vocabulary, extends a pretrained text decoder with audio embeddings, and finetunes it across combined tasks. Audio outputs are reconstructed through AudioLM or SoundStorm-style decoding and can preserve a prompted voice.
- Model representation: A decoder-only Transformer models sequences of text and audio tokens from a multimodal vocabulary initialized with a pretrained text-only checkpoint.Text uses SentencePiece tokens, while speech is represented by discrete audio tokens.
- Audio tokenization: Raw speech is converted into discrete audio tokens by extracting speech embeddings and quantizing them into a limited vocabulary.The paper evaluates w2v-BERT, USM-v1, and USM-v2 tokenization approaches.
- Modifying the decoder: The pretrained decoder is modified by expanding its embedding matrix from t × m to (t + a) × m, where a is the number of audio tokens.The corresponding output matrix changes consistently because the decoder architecture is otherwise agnostic to vocabulary size.
- Multitask finetuning: New audio-token embeddings are added to the pretrained text model, and the whole model is trained on mixtures of speech recognition, translation, synthesis, and text tasks.Task tags identify the requested operation and input/output languages; combining tasks from one dataset improved performance.
- Audio decoding: Audio tokens are decoded into SoundStream tokens and then converted into waveforms with a convolutional decoder.AudioLM uses Stage 2 for voice-conditioned low-bitrate generation and Stage 3 to reconstruct residual quantizer levels for higher quality.
- Audio decoding: SoundStorm provides a non-autoregressive alternative that produces audio of the same quality as AudioLM while being two orders of magnitude faster.Its iterative procedure operates in parallel across tokens.
- Voice transfer: A 3-second voice sample conditions acoustic generation, allowing the translated speech to preserve the original speaker’s voice across languages.The conditioning is represented with both audio tokens and SoundStream tokens.
- Multitask finetuning: Including multiple tasks from the same dataset improved performance in the reported experiments.The authors specifically describe combining tasks such as ASR and AST.
4 Data and Metrics
AudioPaLM is trained and evaluated on mixtures of speech, text, and synthetic datasets supporting ASR, AST, and S2ST. Evaluation uses task-specific metrics on corresponding test splits.
- Datasets: The training data includes CoVoST2, CVSS, VoxPopuli, Common Voice, conversational Spanish-English speech, YouTube ASR, and synthetic TTS resources.CVSS provides synthesized target speech, while VoxPopuli supplies transcripts and spoken translations.
- Datasets: Training-hour counts reflect dataset balancing and finite training steps rather than the full underlying dataset sizes.The reported hours correspond to audio seen by the AudioPaLM variants during training.
- Datasets: AudioPaLM training uses datasets supporting ASR, AST, and S2ST tasks.Table 1 documents the training datasets, while the mixtures combine tasks from those datasets.
- Datasets: The method also benefits from text training data through pretrained PaLM checkpoints and from data used to train audio tokenizers.These sources supplement the datasets listed for AudioPaLM training.
- Metrics: AST is evaluated with BLEU, ASR with word error rate, and CVSS S2ST with BLEU computed from ASR-transcribed model audio.Evaluations use the corresponding datasets’ test splits, and S2ST metrics are designed for comparison with prior work.
5 Experiments
The experiments compare AudioPaLM variants against existing baselines across ASR, AST, and S2ST. The unified method exceeds baselines on speech translation while remaining competitive on speech recognition.
- Top-level results: AudioPaLM reports significant improvements over prior results on AST and direct S2ST, with competitive ASR performance.These are the paper’s top-level experimental outcomes.
- Experimental setup: The compared models include an AST variant trained without S2ST and TTS data and an S2ST variant trained with those tasks that can also produce speech.S2ST audio generation uses SoundStorm.
- Initialization: AudioPaLM is initialized from a PaLM-2 8B checkpoint and extended to process audio tokens as inputs and outputs.Additional audio-token embeddings are initialized to zero, while input and output embeddings remain shared.
- Top-level results: The method exceeds baselines on AST and S2ST, is competitive on ASR, and approaches a cascaded AudioPaLM-2 ASR-plus-text-translation system on AST.The cascaded comparison uses a separately fine-tuned AudioPaLM-2 text-to-text translation model.
5.2 Zero-shot behaviour
AudioPaLM is evaluated for zero-shot speech-to-text translation on FLEURS across languages with AST data and languages having only ASR data. AudioPaLM-2 outperforms Whisper in both settings and improves substantially over AudioPaLM.
- Setup: FLEURS evaluates zero-shot AST on 102 languages, using separate groups with AST-observed and ASR-observed training languages.The ASR-observed group has transcription data but no speech-to-text translation data for the evaluated language pairs.
- Results: AudioPaLM-2 significantly outperforms Whisper on AST-observed languages.These languages had corresponding X→English speech-to-text translation data during AudioPaLM training.
- Results: AudioPaLM-2 also outperforms Whisper on ASR-observed languages, where AudioPaLM receives no AST data for the evaluated language pairs.Whisper’s comparison is not zero-shot because it was trained on AST data for these languages.
- Model scaling: 28% increase for AST-observed languages and 107% increase for ASR-observed languages result from using AudioPaLM-2 instead of AudioPaLM.The passage connects these gains to AudioPaLM-2’s stronger text translation capabilities transferring to audio.
5.3 Quality of generated speech
AudioPaLM’s generated speech is evaluated for both perceptual quality and preservation of the source speaker’s voice using objective and subjective measures. It outperforms Translatotron 2 and the ground-truth synthesized recordings across these evaluations.
- The evaluation measures audio quality with a no-reference MOS estimator and voice-transfer quality with speaker-embedding cosine similarity.The subjective studies separately assess generated-speech quality and voice similarity.
- Subjective ratings use a 1-to-5 scale, producing mean opinion scores and similarity mean opinion scores for each system.Raters evaluate quality or voice identity using trained examples and pilot-study calibration.
- AudioPaLM significantly outperforms voice-preserving Translatotron 2 in both audio quality and voice similarity.The comparison uses objective metrics and subjective ratings on the CVSS-T test split.
- AudioPaLM achieves higher quality and better voice similarity than the ground-truth synthesized CVSS-T recordings across most metrics.The ground-truth recordings were synthesized with voice transfer to mimic the source speaker’s voice.
- AudioPaLM’s objective and subjective metrics show no significant variation between high-resource and low-resource language groups.The high-resource group includes French, German, Spanish, and Catalan; the remaining languages form the low-resource group.
5.4 Impact of model and data choices
AudioPaLM’s performance is shaped by choices about task mixtures, initialization, tokenization, model scale, and output modalities. These experiments reveal substantial gains from pretrained checkpoints, suitable tokens, combined tasks, more data, and larger models, alongside trade-offs across tasks.
- Task mixtures: 2.5 BLEU points—from 16.0 to 18.5—resulted when ASR tasks were added to CoVoST2 AST training.The authors hypothesize that ASR helps connect new audio inputs with the model’s existing text knowledge.
- Initialization: Finetuning a pretrained PaLM 8B checkpoint substantially outperforms training from scratch on both CoVoST2 ASR and AST tasks.The audio-token embeddings are newly initialized, but the pretrained model’s parameters are fine-tuned rather than frozen.
- Tokenization: The choice of tokenization substantially affects performance, with USM-v2 outperforming the other tested tokenization schemes.The authors attribute this to differences in the information exposed by each tokenizer and how readily the model can process it.
- Task mixtures: Expressing AST as a combination of simpler tasks improves AST performance but causes a small reduction in ASR performance.The model remains unified rather than becoming a pipeline, because it can attend to all previous audio and text tokens during decoding.
- Output modalities: Adding S2ST training creates S2ST capability but modestly decreases AST BLEU and ASR WER on the CoVoST2 test split.The added training task requires the model to output audio tokens in addition to text tokens.
- Model scale: Larger models improve downstream ASR and AST results, with further gains from scaling 1B to 8B after large improvements from 128M to 1B.The 1B model also outperforms Whisper 1.5B Large on VoxPopuli ASR and translation according to the reported comparisons.
6 Conclusion
AudioPaLM unifies speech and text processing in one pretrained-and-finetuned language model, achieving strong speech translation and recognition results while supporting zero-shot translation and voice transfer. Its capabilities remain constrained by audio-tokenizer quality, whole-model finetuning, and less-developed generative-audio evaluation benchmarks.
- AudioPaLM processes and generates speech and text interchangeably using a pretrained text LLM extended with discrete audio tokens.Textual task tags allow one model to train on mixtures of speech-text tasks.
- AudioPaLM achieves state-of-the-art speech translation, competitive speech recognition, zero-shot translation on unseen language pairs, and voice-transfer S2ST.Voice-transfer quality is evaluated with automatic metrics and human raters.
- Native audio generation depends strongly on the quality of the audio tokenizer.The paper identifies this dependency as an empirically demonstrated limitation.
- AudioPaLM requires finetuning the whole model, unlike approaches that freeze most weights and preserve pretrained component capabilities.
- Generative audio research is limited by a less-developed set of established benchmarks and metrics compared with text.The paper focuses on speech recognition and speech translation, where benchmarks are more mature.
A Author Contributions
The author contributions span AudioPaLM architecture and evaluation, audio tokenization, multimodal data and infrastructure, speech-to-speech translation, voice-quality assessment, and project coordination.
- Paul initiated the project, created the AudioPaLM architecture, demonstrated its viability across pretraining, recognition, and translation, and contributed to the report.
- Chulayuth conducted audio-tokenization experiments and integrated audio with PaLM2.
- The team developed USM-v2 tokens, token-learning methods, large-scale tokenization infrastructure, and semi-supervised ASR datasets.
- Contributors improved task mixtures with ASR data, speech-generation models, TTS tasks, text-to-text data, and synthetic speech-to-speech datasets.
- Researchers advanced speech-to-speech translation, ASR-BLEU evaluation, PaLM2 integration, cascaded baselines, and speech-quality metrics and voice consistency.
- Additional contributors supported AudioLM details, project planning, speech-to-speech baselines, related work, and overall coordination.
B Detailed results of AST models performance
The appendix identifies detailed AST results through BLEU scores on CoVoST2 and WER results on Vox Populi.
- Table 14 reports BLEU scores on CoVoST2.
- Table 15 reports WER (%) on Vox Populi.
C Detailed results of S2ST models performance
The appendix reports S2ST performance on CVSS using ASR-BLEU scores.
- Table 16 reports S2ST performance on CVSS with ASR-BLEU scores.
D Detailed results of AST zero-shot performance
This section reports zero-shot AST performance on FLEURS using per-language BLEU scores and language-specific audio-training hours. It also distinguishes languages by whether AST or ASR was observed during training.
- Per-language FLEURS results pair BLEU scores with the hours of audio used to train AudioPaLM in each language.The reported training hours exclude audio used in self-supervised audio-tokenization training.
- Languages marked § belong to the “AST observed” set, while languages marked † belong to the “ASR observed” set from Section 5.2.
- The final column summarizes the average BLEU score across all 82 languages.