LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT

• Existing unified decoder-only audio-and-text LLMs quantize audio into discrete tokens, a representation associated with information loss and ASR performance degradation; multi-step codec synthesis also requires many prediction steps.Predicting indices for other codec groups is challenging because codec tokens have a multimodal distribution.Source: §1 · p. 2
• LauraGPT processes audio and text inputs and generates either modality, using continuous audio representations for inputs and discrete codec codes for outputs.The authors state that this representation improves audio-input task performance and facilitates joint autoregressive modeling for audio generation.Source: §1 · p. 2
• The one-step codec vocoder predicts the sum of codec token groups with a transformer-based predictor and reconstruction losses, replacing individual-index prediction with one feed-forward calculation.The design addresses the multimodal distribution challenge of codec tokens.Source: §1 · p. 2
• LauraGPT consistently achieves comparable-to-superior performance against strong baselines across the broadest set of audio recognition, understanding, and generation tasks among existing decoder-only unified audio-and-text LLMs, to the authors’ knowledge.The tasks span content, semantics, paralinguistics, and audio-signal analysis.Source: §1 · p. 2

• Audio-to-text LLMs generate text from audio, while text-to-audio LLMs generate audio from text; unified audio-and-text LLMs process and generate both modalities.The surveyed tasks include recognition and understanding for audio-to-text, TTS and text-to-music for text-to-audio, and tasks such as speech enhancement and speech-to-speech translation for unified systems.Source: §2 · pp. 2–3
• Text-to-audio systems use continuous targets such as embeddings or Mel-frequency spectrograms, or generate discrete codec tokens with diffusion models or autoregressive language models.The passage notes that continuous representations challenge unified text-and-audio modeling.Source: §2 · p. 3
• Some codec-based audio generators use multiple output heads, while VALL-E predicts the first quantizer autoregressively and remaining quantizers non-autoregressively, requiring numerous prediction procedures.The passage also identifies multimodal codec-token distributions as a challenge for predicting remaining quantizer indices.Source: §2 · p. 3
• Unified systems include controller-based approaches that interface specialized models and backbone-based models, while discrete-token decoder-only models lose information through audio quantization.Controller-based approaches are described as complex, resource-intensive, and prone to error accumulation; encoder-decoder systems may require modality-specific pre-nets and post-nets.Source: §2 · p. 3

3.1 Modified Language Model for Unifying Audio-and-Text Modeling

• Continuous audio encodings and text embeddings share a GPT backbone, which predicts text or audio tokens according to a task token.Inputs are downsampled to 60 ms, and only the first codec group is selected for audio outputs to control sequence length.Source: §3.1 · pp. 3–4

3.2 Audio Tokenizer

• The audio tokenizer uses an encoder and the first RVQ quantizer, whose outputs become audio tokens for generation.The codec uses 32 quantizers with vocabulary size 1024; using more RVQ quantizers trades sequence length and efficiency against performance.Source: §3.2 · p. 4

3.3 One-step Codec Vocoder for Audio Generation

• The one-step vocoder predicts summed codec embeddings from first-quantizer tokens and uses a pretrained codec decoder to reconstruct waveforms.Text and audio inputs condition the predictor; for zero-shot TTS they include text and prompt-audio features, while speech enhancement uses noisy-speech features.Source: §3.3 · pp. 4–5

3.4 Multi-task Finetuning

• LauraGPT fine-tunes seven basic tasks in one model, using audio encodings or text embeddings as inputs and audio or text tokens as outputs.Its modular task sequence can combine S2TT and TTS for speech-to-speech translation, with original speech included when maintaining speaker identity is desired.Source: §3.4 · p. 5

Model Architecture

• LauraGPT uses Qwen-1.8B as its GPT backbone and has 2B parameters; the Conformer and GPT components can be replaced by other models.Qwen-1.8B has 24 transformer layers, hidden size 2048, and 16 attention heads.Source: §4 · p. 5

Training Setup

• Experiments initialize the Qwen backbone and audio encoder from pretrained checkpoints, then optimize model parameters through multi-task fine-tuning.The training process uses three stages to address differences in data volume across tasks.Source: §4 · p. 6

5.1 Results on All Tasks

• LauraGPT achieves comparable-to-superior performance against strong baselines across diverse speech recognition, understanding, and generation tasks.Table 1 covers speech recognition, understanding, and generation benchmarks; the authors caution that its SOTA comparisons are not fair across models.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Analysis

• More codec groups can reduce the ASR performance gap, but increase token rates, sequence lengths, and computational demands.Using more quantizers improves performance but reduces efficiency.Source: §5.2 · pp. 6–7
• LauraGPT outperforms Discrete IO on ASR, S2TT, and SE, where discrete inputs degrade ASR, yield BLEU scores of 5.1 and 5.0 on translation test sets, and do not improve noisy-speech enhancement.The authors attribute the SE result probably to noisy speech being too complex for four groups of discrete tokens to represent accurately.Source: §5.2 · p. 7, Table 2 · p. 8

Comparison on Audio Synthesis Schemes

• On the SE task, the one-step scheme outperforms the multi-step scheme on CER, WER, and PESQ while matching STOI.The one-step vocoder formulates synthesis as regression and uses a single feed-forward calculation.Source: Table 3 · p. 8, Comparison on Audio Synthesis Schemes · p. 8
• For limited-data AAC, SLU, and SER tasks, multi-task training generally outperforms single-task training, though SER WA is slightly worse.On SLURP, the reported gains are +2.9 absolute in intent accuracy and +22.5 absolute in SLU-F1; minority-class accuracies also improve.Source: Table 4 · p. 8, Comparison on Audio Synthesis Schemes · p. 8

• LauraGPT handles audio and text inputs and outputs in one model for audio recognition, understanding, and generation.Source: §6 · p. 8
• Its design combines continuous and discrete audio features with a one-step codec vocoder and multi-task learning.Source: §6 · p. 8
• Experiments show comparable-to-superior performance against strong baselines across diverse speech tasks spanning content, semantics, paralinguistics, and audio-signal analysis.Source: §6 · p. 9

Limitations

• LauraGPT trains all parameters except the codec vocoder, incurring substantial computation; the authors plan to investigate parameter-efficient fine-tuning and possible text-capability forgetting.The paper also notes that parameter-efficient fine-tuning may help preserve the pretrained text LLM’s capabilities.Source: Limitations · p. 9
• Limited computation restricted multi-task versus single-task comparisons to low-resource AAC, SLU, and SER, leaving richer-resource tasks such as ASR for future study.The authors report better multi-task results on limited-data tasks and plan broader comparisons, including tasks with potentially conflicting objectives.Source: Limitations · p. 9
• LauraGPT’s speech-generation performance strongly depends on audio-tokenizer quality, which the authors plan to analyze across tasks and improve with new tokenizers.The proposed future work targets tokenizers suited to unified audio-and-text LLMs and generative tasks.Source: Limitations · p. 9
• The envisioned speech foundation model would add few-shot adaptation, instruction following, and general speech, non-speech audio, and music modeling.These are presented as desired capabilities, not as demonstrated abilities of current LauraGPT.Source: Limitations · p. 9
• The authors describe LauraGPT as a milestone toward a speech foundation model, with further progress focused on more task data, pretraining, and fine-tuning rather than architectural changes.They frame the broader question as designing efficient, scalable unified GPT-style models with competitive performance across speech tasks.Source: Limitations · p. 9

A.1 Basic Tasks

• The evaluated tasks cover speech transcription, semantic understanding, translation, emotion recognition, audio captioning, enhancement, and speech synthesis.SE is reformulated as codec-token classification; SLU identifies intent and slots, while AAC describes audio clips in natural language.Source: §A.1 · p. 14

A.2 Training Datasets

• The authors use publicly available datasets with varied licenses and state that they did not check whether the data contained identifying information or offensive content.They say their dataset use follows intended uses and prior research practice.Source: §A.2 · pp. 14–15
• ASR uses Chinese AISHELL-1, AISHELL-2, and WenetSpeech plus English LibriSpeech and GigaSpeech; S2TT uses BSTC and CoVoST 2 with augmented parallel data.The S2TT augmentation adds about 2,000 hours of translated speech data and ParaCrawl v9 parallel text.Source: §A.2 · p. 14
• SER uses seven listed corpora, SLU uses SLURP, AAC uses AudioCaps, WavCaps, and Clotho, and TTS uses LibriTTS and 3D-speaker.SLURP covers 18 scenarios; no other corpora are used for SER.Source: §A.2 · p. 14
• SE training pairs clean speech with noise mixed at 2–15 SNR and simulated reverberation, yielding approximately 6,000 hours of paired data.The clean speech comes from AISHELL-1, AISHELL-2, LibriSpeech, and WSJ; noise includes FSD-50K audio and room impulse responses.Source: §A.2 · p. 14

A.3 Evaluation Datasets and Metrics

• Evaluation uses task-specific datasets and metrics, including CER and WER for recognition or content consistency, BLEU for S2TT, and PESQ and STOI for enhancement.The appendix also describes speaker similarity and synthesized-speech naturalness metrics, plus captioning metrics SPICE, CIDEr, and SPIDEr.Source: §A.3 · p. 15
• SER metrics are WA, UA, and WF1; SLU uses intent accuracy and SLU-F1, which balances Word-F1 and Char-F1.WA is overall accuracy, while UA and WF1 average class-wise accuracy and F1, respectively.Source: §A.3 · p. 15

A.4 Details of Training and Inference

• Training initializes Qwen and the audio encoder from pretrained checkpoints, then fine-tunes all tasks in three stages to address differences in task data volume.The middle stage focuses on smaller-data tasks; the final stage returns to all tasks at a reduced learning rate.Source: §A.4 · p. 15
• The codec-vocoder predictor is trained on TTS and SE data, while its pretrained decoder remains frozen during LauraGPT multi-task fine-tuning.The predictor uses Adam with a peak learning rate of 0.001 and 25K warmup steps.Source: §A.4 · p. 15
• Training combines audio-encoder or text embeddings with a task-ID embedding and target embeddings, applies masked cross-entropy, and masks losses on inputs and task tokens.At inference, Qwen generates autoregressively until the end-of-sequence token; the codec vocoder converts audio-format output tokens to waveforms.Source: §A.4 · pp. 15–16
• Table 5 reports basic-task training-data statistics and defines Corpus×N as copying a corpus’s samples N times; Table 6 lists task evaluation datasets and metrics.Table 6’s arrows indicate whether higher or lower metric values are desirable.Source: Table 5 · p. 16, Table 6 · p. 16

A.5 Details of the SER Evaluation

• For SER, training emotion labels are unified into nine classes, while “like” and “happiness” map to “joy” at test time to match MELD.LauraGPT generates emotion labels autoregressively, and out-of-domain outputs count as errors.Source: §A.5 · p. 16

B Comparison with Related Unified Audio-and-Text Models

• The paper warns that differences in experimental settings, datasets, and open-source availability make fair comparison with related unified models difficult.It nevertheless presents the most relevant available comparisons.Source: B Comparison with Related Unified… · p. 17
• On Chinese AISHELL-1 and AISHELL-2 test sets, LauraGPT outperforms Whisper by -3.9 and -2.3 absolute on CER, respectively, despite using much less training data.On English LibriSpeech test-clean and test-other, LauraGPT performs worse than Whisper Large V2, which used more English training data.Source: B Comparison with Related Unified… · p. 17
• LauraGPT’s LibriSpeech test-other WER of 7.7% is comparable to SpeechT5’s reported 7.3% WER without CTC and LM.The paper notes that LauraGPT was trained directly on labeled multi-task data without SpeechT5’s self-supervised pretraining.Source: B Comparison with Related Unified… · p. 17
• The authors conclude that LauraGPT notably outperforms VioLA on ASR, using comparisons through Paraformer because VioLA reports phoneme error rate rather than word or character error rates.They note substantial training-data differences and lack of VioLA code and models, limiting direct comparison.Source: B Comparison with Related Unified… · p. 17
• AudioPaLM is not directly compared because its training and evaluation datasets are disjoint from LauraGPT’s and its pretrained model is unreleased.The authors say an empirical comparison would require substantial effort.Source: B Comparison with Related Unified… · p. 17

C.1 Effectiveness of Multi-task Finetuning on the SER task

• On MELD, multi-task LauraGPT substantially outperforms the single-task model on UA and WF1, though WA is slightly worse; it also improves several low-accuracy or previously unpredicted emotion classes.WF1 is identified as MELD’s primary metric because of class imbalance; multi-task training improves disgust and fear, which the single-task model could not predict.Source: §C.1 · pp. 17–18

C.2 Batch normalization versus layer normalization in audio encoder

• On speech enhancement, replacing batch normalization with layer normalization greatly reduces the loop ratio and improves PESQ and STOI, although broader normalization methods remain an open research need.Batch normalization is associated with a high inference loop ratio, particularly for long-sequence tasks; the evaluation focuses on speech enhancement, generally the longest-sequence task.Source: §C.2 · pp. 18–19

C.3 Impact of initialization from pre-trained models

• Pre-trained initialization significantly affects ASR and S2TT performance but has relatively limited influence on speech enhancement, suggesting GPT prior knowledge matters more for text generation than audio generation.The authors hypothesize that pre-training the GPT backbone with audio token sequences as well as text could improve generated audio quality.Source: §C.3 · p. 19

D Supporting More Complex Tasks

• LauraGPT’s modular design supports complex tasks by chaining basic tasks and passing raw inputs and subtask outputs between them.The paper describes this as an extensible framework beyond the basic tasks used in training.Source: D Supporting More Complex Tasks · p. 19
• Rich transcription can combine content, speaker, paralinguistic, and high-level semantic outputs using different task IDs, avoiding pipelined error accumulation and individual task runs.Source: D Supporting More Complex Tasks · p. 19
• For noise-robust ASR, the proposed chain feeds noisy-speech and enhanced-speech embeddings through speech enhancement and ASR task IDs; joint training may let LauraGPT exploit both embeddings.Source: D Supporting More Complex Tasks · p. 19