AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs

• LLMs already support diverse language tasks, including summarization, generation, retrieval, and translation, but their training criteria can mismatch user preferences.Source: §1 · p. 1
• Earlier multimodal approaches commonly align a pretrained modality encoder with an LLM using dedicated or curated paired data.Source: §1 · p. 1
• Prior speech-oriented LLM extensions support tasks such as recognition, translation, or synthesis but do not fully exploit LLM flexibility for broader open-ended tasks.Source: §1 · p. 2
• AudioChatLlama extends an instruction-tuned Llama to end-to-end speech response generation while retaining conversational context for reasoning.Source: §1 · p. 2

2 AudioChatLlama: LLM with General-Purpose Speech Abilities

• AudioChatLlama is designed to support audio prompts as a direct replacement for text while preserving the original language model’s capabilities.The system is intended to enable speech conversation and cross-modal recognition and reasoning without curated paired data.Source: §2 · p. 2

2.2 General-Purpose Alignment Using Unpaired Data

• The alignment objective is modal invariance: audio and text containing the same semantic information should induce the same response.Figure 2 states the target of generating an equivalent response from spoken input instead of its text version.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Architecture

• The architecture uses a decoder-only multimodal design in which a CTC-pretrained conformer audio encoder and projection layer produce embeddings consumed autoregressively by the language model.The audio encoder maps speech representations to the LLM’s embedding dimension, while the decoder generates textual responses.Source: §2.1 · pp. 2–3
• Llama-2-chat 7B is kept completely frozen, while only the audio encoder is trained to learn how to prompt the instruction-tuned conversational model.Source: §2.1 · p. 3

2.2 General-Purpose Alignment Using Unpaired Data

• Using generic (audio, transcript) ASR pairs, Llama-2-chat generates responses from transcripts, and those responses form (audio, response) pairs for audio-encoder alignment.This avoids dedicated speech-response datasets, although ASR corpora provide transcripts rather than naturally paired answers.Source: §2.2 · p. 3
• The alignment data has important scope limitations: audiobook utterances may be poor prompts, and short ASR segments can produce nonsensical transcript prompts.The paper nevertheless considers alignment successful when audio induces the same response, even if the source prompt and generated response are nonsensical.Source: §2.2 · pp. 3–4

3.1 Dataset & Generation

• The experiments use the English Multilingual LibriSpeech split, a 50k-hour audiobook-derived ASR corpus, with utterances segmented to at most 20 seconds.Reference transcripts are fed to Llama-2-chat to generate responses for training.Source: §3.1 · p. 4
• Generated responses use greedy decoding with a maximum length four times the input prompt, producing a larger dataset because instruction-tuned LLM replies are often longer.The responses are generated from the reference text rather than from curated speech-response annotations.Source: §3.1 · p. 4

3.2 Model Architecture & Training

• The audio encoder processes 80-dimensional filterbanks, uses convolutional and conformer layers, and is initially pretrained with CTC over a 1.5k-token SentencePiece vocabulary.The CTC output layer is discarded after pretraining.Source: §3.2 · p. 4
• Encoder features are stacked, projected to 4096 dimensions to match Llama-2-chat 7B, inserted into the standard chat prompt structure, and trained to predict response tokens.The audio embeddings replace the user prompt while the system prompt is empty.Source: §3.2 · pp. 4–5
• Evaluation compares response perplexity for AudioChatLlama and cascaded speech-recognition-plus-LLM systems under the transcript-conditioned Llama-2-chat response.Cascaded systems additionally report prompt WER because they first transcribe the audio.Source: Table 2 · p. 5
• Llama-2-chat is kept frozen throughout training, leaving the audio encoder as the only trainable component and reducing mismatch between text-based response generation and audio training.The model is selected for both data generation and training to preserve consistent system behavior across modalities.Source: §3.2 · p. 5

4.2 Perplexity

• AudioChatLlama outperforms cascaded ASR + LLM systems on MLS and TriviaQA-TTS perplexity without two-stage decoding.The evaluation compares direct audio prompting with cascaded transcription followed by LLM response generation.Source: §4.2 · p. 6

4.3 Human Evaluation

• In TriviaQA-TTS human evaluation, AudioChatLlama performs better than the baseline at 37.5% and 14.3% WER, and matches it at 4.3%.Success rate measures the fraction of predicted responses agreeing with reference responses across 50 samples per error level.Source: §4.3 · p. 6, Table 3 · p. 6
• On RealSQA, AudioChatLlama outperforms the cascaded model at 31.5% and 13.3% WER and matches baseline performance at 3.5%.The same trend appears on recorded speech questions, with larger baseline degradation at higher ASR error rates.Source: §4.3 · pp. 6–7, Table 4 · p. 6

4.4 Embedding Space Alignment

• For a human-recorded example with a frozen LLM, pairwise cosine similarity between 40 audio embeddings and 10 transcript embeddings shows nearly monotonic alignment despite different sequence lengths, with “Obama” weakly aligned.Uninformative start and end portions correspond to deliberate silence in the recording.Source: §4.4 · pp. 7–8

4.5 Extended Capabilities

• Users can switch between audio and text inputs during a conversation, while conversational history helps interpret difficult speech such as rare words.The system treats aligned audio embeddings as a replacement for text and uses prior context to guide interpretation.Source: §4.5 · p. 8
• Automatic evaluation is difficult for several extended capabilities because instruction-tuned LLMs may produce valid responses with additional content.The speech-translation example first transcribes audio, then responds and adds information relevant to the prompt.Source: §4.5 · p. 8
• The model extends text capabilities to speech translation and audio summarization without explicit task-specific training.Its instruction-tuned LLM foundation supports these capabilities through the speech interface.Source: §4.5 · p. 8
• Performance on extended tasks depends on the original LLM: if the underlying model cannot translate, the end-to-end speech system cannot translate either.This establishes the capability boundary for inherited text-to-speech extensions.Source: §4.5 · p. 8

Audio understanding

• AudioChatLlama still has limited audio understanding beyond the capabilities presented in the paper.The authors propose extending it to generic audio understanding and reasoning.Source: §5 · p. 8
• Replacing the small conformer with self-supervised encoders such as Wav2Vec2 or HuBERT is proposed to produce richer representations for broader tasks.The current conformer was trained on a limited amount of data, yet it provides a stepping stone toward stronger audio models.Source: §5 · p. 8

Data generation

• Training on a segmented ASR dataset can introduce drawbacks when segmentation breaks sentence structure, potentially reducing response quality.The authors also used greedy decoding for generated replies and suggest beam search to obtain higher-quality training responses.Source: §5 · p. 8

Conclusion

• AudioChatLlama extends an instruction-tuned LLM with end-to-end speech processing and reasoning without carefully curated paired data.The model is intended to maintain the LLM’s wide range of original capabilities.Source: §6 · p. 8
• Audio prompts can replace text in sustained conversations, with support for spoken QA, speech translation, and audio summarization.These capabilities cover cross-modal recognition and reasoning tasks.Source: §6 · p. 8
• Prior conversational context can guide recognition and reasoning when standard speech recognition systems struggle with rare words.The conclusion contrasts this contextual use with alternative approaches based on standard speech recognition.Source: §6 · p. 8

Basic Speech Translation

• The examples include basic speech translation from an audio prompt into an Italian text response.The system translates “How are you today?” as “Come stai oggi?”.Source: A Examples of Extended Capabilities · p. 10

Audio Summarization

• The system summarizes spoken content about Troy by returning a shorter description of its location and mythological significance.The example preserves the city’s location in present-day Turkey and its connection to the Trojan War.Source: A Examples of Extended Capabilities · p. 11

Interchanging Modalities

• The modality-switching example demonstrates that users can alternate between audio and text questions during interaction.The section explicitly presents interchanging modalities as an extended capability.Source: A Examples of Extended Capabilities · p. 11

Contextualization

• A multi-round text exchange preserves conversational context by answering that Barack Obama’s vice president was Joe Biden.The response follows an earlier exchange identifying Obama.Source: A Examples of Extended Capabilities · p. 11
• A subsequent audio question about Obama’s eligibility receives a response that uses the preceding conversational context.The answer states that Obama is no longer eligible to run for president.Source: A Examples of Extended Capabilities · p. 11
• The Iceland example shows contextual handling of a rare name, with Jökulsárlón inferred and understood correctly from the surrounding conversation.The conversation concerns winter travel, Reykjavik, and the Glacier Lagoon before the response addresses the route to Jökulsárlón.Source: A Examples of Extended Capabilities · p. 11

Interchanging Modalities

• Additional audio questions receive text responses about Reykjavik attractions and Blue Lagoon ticket costs, illustrating open-domain conversational answering.The examples also include travel-distance information from Reykjavik to the glacier lagoon.Source: A Examples of Extended Capabilities · p. 11