Source-linked AI summary
On decoder-only architecture for speech-to-text and large language model integration
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, Yu Wu
TL;DR
Speech-LLaMA addresses the limited exploration of integrating speech signals with LLMs and of decoder-only architectures for speech-to-text. It combines a CTC compressor and audio encoder to map continuous acoustic features into an LLM’s semantic space, and evaluates the approach on multilingual speech translation. The method outperforms strong baselines, while a decoder-only model trained from scratch achieves comparable performance with fewer parameters.
Problem
Speech-LLM integration and decoder-only architectures for speech-to-text remain insufficiently explored, with speech-text alignment and integration cost posing challenges.
Method
Speech-LLaMA combines a text-based LLM, a CTC compressor, and an audio encoder to map compressed continuous acoustic features into the LLM’s semantic space.
Results
Speech-LLaMA significantly outperforms strong sequence-to-sequence baselines on 13-language-to-English speech translation, while a decoder-only model achieves comparable performance with around 40% fewer parameters.
Takeaways & Limitations
Decoder-only models show potential for speech-to-text conversion and can be competitive with encoder-decoder architectures while using fewer parameters.
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.
1. INTRODUCTION
The paper presents Speech-LLaMA, an efficient end-to-end approach that integrates acoustic information into text-based LLMs while investigating decoder-only architectures for speech-to-text tasks. Experiments indicate strong translation performance and competitiveness with encoder-decoder systems.
- Motivation: The work addresses speech-text alignment, integration cost, and limited exploration of decoder-only backbones for speech-to-text processing.Speech sequences are typically longer than text sequences, creating alignment challenges when integrating modalities.
- Approach: Speech-LLaMA integrates speech with a text-based LLM by mapping compressed continuous acoustic representations into the LLM’s semantic space.The acoustic feature is compressed, transformed by an acoustic encoder, and consumed by the LLM for transcription or translation.
- Approach: The proposed method uses a pre-existing LLM, an acoustic feature compressor, and an acoustic encoder with only a small number of additional free parameters.Unlike approaches using discretized speech tokens, it directly maps continuous speech representations into the LLM’s semantic space.
- Experiments: The authors investigate acoustic compression, attention-mask selection, and fine-tuning strategy as practical factors affecting speech-LLM integration performance.These aspects are examined alongside multilingual speech translation experiments.
- Results: The decoder-only architecture is shown to be as competitive as encoder-decoder architectures for speech-to-text tasks and more parameter efficient.The comparison is conducted on large, diverse, real-world data.
2. RELATED WORK
The related work covers speech-LLM integration, decoder-only language-model backbones, CTC-based speech compression, and LoRA-based parameter-efficient adaptation.
- Speech-LLM integration: Speech-LLM integration methods include cascaded ASR-then-LLM systems, unified decoder-only networks, jointly optimized audio-token conversion, and discrete self-supervised speech tokens.These approaches represent different strategies for combining speech signals with language models.
- CTC compression: CTC compression reduces speech-feature sequence length by removing redundant information through a CTC branch and posterior-based representation compression.Prior work jointly optimizes the CTC branch with the main cross-entropy objective.
- Parameter-efficient adaptation: LoRA adapts large Transformer models by adding low-rank matrices while freezing the original model parameters.Only the newly introduced matrices are updated during fine-tuning, reducing training memory requirements.
3. OUR APPROACH
Speech-LLaMA integrates acoustic embeddings into a text LLM for speech translation. Its architecture compresses speech, maps it into the LLM’s semantic space, and supports both pretrained-LLM and from-scratch decoder-only variants.
- Speech-LLaMA conditions text generation on both a text prompt and acoustic embeddings mapped into the text-LLM’s semantic space.The model converts speech into acoustic embeddings so a pretrained text LLM can generate transcription or translation.
- The proposed model combines a pretrained text LLM, a CTC compressor, and a relatively small randomly initialized audio encoder.The CTC compressor reduces speech length, while the audio encoder transforms the compressed signal into continuous vectors consumed by the LLM.
- The CTC compressor matches audio and text durations by selecting representative frames through blank-removal or frame-averaging.Blank-removal discards blank-predicted frames; frame-averaging averages consecutive frames predicted as the same class.
- The pretrained-LLM setup explores task prompts, attention masks, and LoRA fine-tuning while keeping most LLaMA parameters frozen.LoRA is applied after audio-encoder training, and the from-scratch variant replaces the prompt, compressor, audio encoder, and LLaMA with randomly initialized components.
- The from-scratch decoder-only model uses an ⟨SOS⟩ token at the end of the acoustic sequence to mark the start of text generation.Its generation is conditioned only on the audio signal and previously generated text.
4. EXPERIMENTS
The experiments evaluate the proposed speech-to-text methods on multilingual speech translation from 13 source languages into English.
- Speech translation is the primary benchmark, with systems translating spoken language from 13 source languages to English.
4.1. Data and metric
The study uses multilingual speech data and evaluates translation quality with BLEU on CoVoST 2 test sets.
- Training includes 1K hours per source language across 13 languages plus 1K hours of English data, totaling 14K hours.The languages were selected based on training and testing data availability.
- Performance is evaluated on the 13 languages using CoVoST 2 test sets and BLEU scores.
4.2. Models configuration
The experiments compare CTC and convolutional speech compression, several integration and fine-tuning choices, and a 240M-parameter encoder-decoder baseline against the proposed models.
- The CTC compressor uses two convolutional layers and four Transformer layers for 4-times subsampling, with 15.8M parameters.It is pretrained with a CTC objective on paired speech-text data from 13 languages.
- The convolutional subsampling baseline adds three 1D convolution layers for 32-times feature-length reduction and jointly trains them with the audio encoder.
- The audio encoder has four Transformer layers and projects its final output to 4096 dimensions to match LLaMA’s semantic embeddings.
- Training samples concatenate text-prompt embeddings and audio-encoder representations as a prefix sequence for LLaMA generation.
- The study compares causal attention with non-causal full attention restricted to the text-prompt and audio-representation prefix.
- LoRA adds 2.1M parameters through rank-2 adapters while freezing the CTC compressor and LLaMA during adaptation.
- The seq2seq baseline is a 240M-parameter Whisper model with 12-layer audio and text Transformer encoders.
- Table 1 reports BLEU scores for 13 languages across the baseline and proposed models.The seq2seq decoding and reranking experiments use n = 5 beam-search candidates.
4.3. Training and evaluation
The experiments use standardized log mel-filterbank acoustic features and train models with AdamW, staged learning-rate schedules, and model-dependent batch sizes and training durations.
- Acoustic inputs are 80-dimensional log mel-filterbanks extracted with 25 msec windows and 10 msec hops, followed by global mean and variance normalization.
- All models are trained with AdamW using β1 = 0.9 and β2 = 0.98 on 16 V100 GPUs with warmup and linear decay learning rates.
- CTC compressor pretraining uses 100K steps, while Speech-LLaMA first-stage training uses 500K steps and LoRA fine-tuning adds 100K optimization steps.
- From-scratch decoder-only models use a peak learning rate of 0.001 for at most 300K steps, and decoder-only decoding uses beam search with beam size 4.
- Both seq2seq and decoder-only models tokenize with an English-only byte pair encoding model.
5. RESULTS AND DISCUSSIONS
Speech-LLaMA configurations substantially outperform the baselines, with performance shaped by compression, attention-mask, and fine-tuning choices. The randomly initialized decoder-only model approaches seq2seq performance with fewer parameters.
- 5.1. Baselines: B2 improves over B1 by 0.9 BLEU through LLaMA n-best rescoring, showing that shallow LLM integration benefits speech models.
- Up to 4.6 absolute BLEU improvement, or 21.2% relative, is achieved by Speech-LLaMA configurations over the baselines with limited learnable parameters.
- 5.3. CTC compressor: The CTC compressor consistently outperforms convolutional compression despite being frozen, suggesting source-language transcription pretraining contributes to its effectiveness.
- 5.3. CTC compressor: Frame-averaging exceeds blank-removal by 1.5 average BLEU, indicating greater robustness to information loss during CTC compression.
- 5.4. Effect of non-causal attention mask: Non-causal attention improves average BLEU by 1.5 with blank removal, 0.7 with frame averaging, and 0.8 in LoRA fine-tuning.
- 5.5. Effect of LoRA fine-tuning: LoRA fine-tuning adds 1.5 average BLEU with causal attention and 1.6 with non-causal attention using only 2.1M additional parameters.
- 5.6. Decoder-only vs Encoder-Decoder: The from-scratch decoder-only model is only 0.4 BLEU below the seq2seq baseline while using significantly fewer total parameters.
6. CONCLUSION & FUTURE WORK
The paper presents Speech-LLaMA as a deep integration of acoustic representations with an off-the-shelf LLM and evaluates practical design choices. It reports strong speech-translation gains and comparable decoder-only performance with fewer parameters.
- Speech-LLaMA directly maps acoustic representations into the LLM’s semantic space while exploring compression, attention-mask design, and adapter fine-tuning.
- On 13-language-to-English speech translation, Speech-LLaMA significantly outperforms a strong sequence-to-sequence baseline.
- A decoder-only architecture trained from scratch achieves comparable performance with around 40% fewer parameters than the encoder-decoder baseline.