PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM can Read Spectrogram: Encoder-free Speech-Language Modeling

Ruchao Fan, Yiming Wang, Yuxuan Hu, Bo Ren, Yufei Xia, Xiaofei Wang, Yao Qian, Shujie Liu, Jinyu Li

arXiv:2606.10231v4eess.AScs.SD

TL;DR

Speech-LLMs typically use pretrained speech encoders, and this paper asks whether an LLM can read Mel spectrograms directly instead. Mel-LLM feeds projected Mel inputs to the LLM without dedicated speech-encoder blocks. It reports competitive ASR performance, acoustic-task benefits, and weaker spoken-QA results, with data scale and multimodal initialization affecting ASR performance.

  • Problem

    Speech-LLMs typically rely on pretrained speech encoders, and the paper asks whether an LLM can process Mel spectrograms directly without one.

    Source: §I · p. 1

  • Method

    Mel-LLM removes pretrained speech-encoder blocks and projects chunked Mel spectrograms into the LLM, optionally using lightweight temporal downsampling.

    Source: §I · p. 1, B. Speech Understanding Input Path · p. 3

  • Results

    Mel-LLM achieves competitive ASR results with limited degradation versus encoder-initialized models at scale, while direct Mel access benefits acoustic tasks and spoken-QA results are weaker.

    Source: §I · p. 1, §VI · p. 6

  • Takeaways & Limitations

    The results support encoder-free Mel input as a speech-understanding approach, with Phi-4-MM initialization important in limited-data settings.

    Source: §I · p. 1, A. ASR: Main Results and Scaling · p. 4

    The reported spoken-QA results are weaker, and limited-data ASR depends critically on Phi-4-MM initialization.

    Source: A. ASR: Main Results and Scaling · p. 4, §VI · p. 6

Abstract

from arXiv · show

Recent speech-aware large language models (Speech-LLMs) rely on pre-trained speech encoders to convert audio into semantic/acoustic rich representations consumable by LLM. In this work, instead, we explore: can an LLM learn to read Mel spectrogram directly without a dedicated speech encoder? We propose Mel-LLM, an encoder-free Speech-LLM that feeds lightly pre-processed Mel-spectrogram patches directly into the LLM through a linear projection, allowing the LLM to learn speech-text alignment purely through its own parameters. We focus on speech understanding tasks, including automatic speech recognition (ASR), spoken QA and audio understanding. For ASR, we evaluate on the OpenASR Leaderboard public sets and production-level scaling experiments, demonstrating that the encoder-free solution achieves competitive performance with only limited degradation compared to encoder-initialized counterparts. We find that when data is limited, initialization from a multimodal checkpoint (Phi-4-MM) is crucial for maintaining performance. We also present ablation studies suggesting which LLM layers are most involved in speech adaptation. Beyond ASR, we extend Mel-LLM with general speech/audio understanding tasks, revealing an acoustic-semantic trade-off: directly exposing the LLM to Mel-spectrogram input improves paralinguistic and non-ASR acoustic tasks, while knowledge-intensive spoken QA remains more challenging than encoder-anchored systems. We additionally include a text-to-speech (TTS) proof-of-concept with a next-token VAE decoder, showing that direct Mel generation is possible but still trails stronger latent-diffusion generation.

I. INTRODUCTION

The introduction asks whether an LLM can read Mel spectrograms without a dedicated speech encoder, then presents Mel-LLM as an encoder-free approach for speech understanding and exploratory generation. It reports competitive ASR results and highlights both acoustic-task benefits and limits of the approach.

I. INTRODUCTION

  • Mel-LLM removes pretrained speech-encoder blocks and projects chunked Mel spectrograms directly into the LLM, optionally using lightweight temporal downsampling.The LLM learns to interpret spectral features and align them with text; the architecture also includes an exploratory speech-generation path.Source: §I · pp. 1–2, Figure 1 · p. 2
  • Mel-LLM achieves competitive OpenASR results, with limited degradation versus encoder-initialized models at scale.The introduction also reports benefits on paralinguistic understanding and identifies exploratory TTS as a remaining challenge.Source: §I · p. 1
  • Direct Mel inputs preserve low-level acoustic cues that benefit paralinguistic understanding and other non-ASR acoustic tasks.The introduction frames this as a contrast with speech encoders that provide compressed representations.Source: §I · p. 1
  • Multimodal Phi-4-MM initialization is critical with limited data, while layer-freezing evidence points to lower and middle LLM layers as important for speech adaptation.These findings suggest that the LLM can learn part of the function normally handled by a dedicated encoder.Source: §I · p. 1
  • The paper includes a preliminary TTS proof of concept, but high-fidelity continuous Mel generation remains difficult with an LLM backbone.Speech understanding is the main focus of the work.Source: §I · pp. 1–2

II. RELATED WORK

Related work establishes encoder–projector–LLM systems as the prevailing paradigm for speech understanding and surveys encoder-free multimodal models, continuous-Mel TTS, and unified speech-text models. The paper positions its contribution as direct continuous-Mel input for speech understanding and reports detailed encoder-based versus encoder-free comparisons across data scales.

II. RELATED WORK

  • Prior speech-understanding systems commonly use pretrained encoders such as Whisper or HuBERT before feeding semantic features to an LLM.The broader encoder–projector–LLM paradigm is associated with –[3],, –[35].Source: §II · p. 2
  • Encoder-free vision and video models motivate direct modality input, while speech works also explore continuous Mel generation and unified speech-text modeling.Examples include Fuyu, ELVA, MELLE, and discrete-token systems such as SpeechGPT, VoxtLM, and Spirit LM [19].Source: §II · p. 2
  • Mel-LLM differs from discrete-token speech-text approaches by using continuous Mel features directly as LLM inputs.The paper treats TTS as exploratory; its main contribution is encoder-free speech understanding.Source: §II · p. 2
  • The authors describe this as the first academic publication with detailed encoder-based versus encoder-free Speech-LLM comparisons across data scales.This claim is limited to their knowledge and the stated comparison scope.Source: §II · p. 2

A. Architecture Overview

Mel-LLM builds on Phi-4-MM and simplifies the speech encoder component for speech understanding and generation. For speech understanding, normalized Mel features may be downsampled, projected into the LLM space, and combined with text prompt embeddings for transcription generation.

A. Architecture Overview

  • Mel-LLM builds on Phi-4-MM and simplifies the speech encoder component while supporting speech understanding and generation.The architecture shares most of its architecture knowledge with the LLM, especially Phi-4-MM.Source: A. Architecture Overview · p. 2

B. Speech Understanding Input Path

  • The input path begins with 80-dimensional log-Mel features, with optional convolutional downsampling to sequence length T/r; r is 8 for ASR and 2 for TTS.Convolution layers are disabled when r = 1.Source: B. Speech Understanding Input Path · pp. 2–3
  • A randomly initialized linear projection maps speech features to the LLM hidden dimension, where the LLM learns to interpret the spectral features.The projected speech embeddings are concatenated with text-prompt embeddings before autoregressive transcription generation.Source: B. Speech Understanding Input Path · p. 3
  • The LLM is adapted using LoRA with rank 320.Source: B. Speech Understanding Input Path · p. 3

C. Exploratory Speech Generation Path

The exploratory TTS path uses the LLM to predict speech-position hidden states and a lightweight Mel head to generate continuous Mel spectrograms. It is presented as a proof of concept, not a competitive TTS system.

C. Exploratory Speech Generation Path

  • The TTS path predicts speech-position hidden states from text and decodes them into Mel frames with a lightweight VAE-style Mel head.The head uses a residual MLP, postnet, and stop predictor; training combines reconstruction, KL, stop-token, and temporal-smoothness losses.Source: C. Exploratory Speech Generation Path · p. 3
  • The authors frame direct Mel generation as a proof of concept for future unified modeling rather than a competitive TTS system.Source: C. Exploratory Speech Generation Path · p. 3

IV. EXPERIMENTAL SETTINGS

The experiments use a Phi-4-MM-based Mel-LLM configuration and public ASR data, with an expanded mixture for speech and audio understanding. The setup also compares encoder-free and encoder-initialized models across data scales and records a historical OpenASR benchmark definition.

A. Model Configuration

  • The 12.5Hz Mel-LLM setting removes the main Conformer encoder blocks while retaining NeMoConv downsampling; the TTS mel-input reduction factor is 2.The Phi-4-MM LLM has 3072 hidden dimensions, 32 layers, 24 attention heads, and 8 KV heads; LoRA uses r = 320 and α = 640.Source: A. Model Configuration · p. 3, Table I · p. 3

B. Training Data and Settings

  • OpenASR ASR training uses about 31M utterances, or approximately 64k hours, from publicly available English speech corpora and no proprietary or internal data.For expanded speech understanding experiments, the mixture contains 55.3M weighted examples, including speech QA and audio/speech understanding data, with held-out evaluation examples excluded.Source: B. Training Data and Settings · p. 3
  • Training freezes the base LLM layers and trains only LoRA, iterating over the data three times; the production-scale experiments use about 10× anonymized in-house training data for one iteration.Training uses 16 H100 GPUs and an effective batch size of 512.Source: B. Training Data and Settings · p. 3
  • TTS preliminary training uses 50k hours of Libriheavy English data, swept for five epochs.The setup uses dropout of 0.5 for the input projection and output Mel head.Source: B. Training Data and Settings · pp. 3–4
  • The OpenASR numbers follow the earlier benchmark definition, which included TED-LIUM before its removal from the leaderboard suite.Source: B. Training Data and Settings · p. 4
  • With 10× scaled anonymized in-house data, the encoder-free versus encoder-initialized WER gap narrows significantly, while encoder-free degrades notably with limited public data.The table describes in-house test sets and attributes the narrowing to data scaling.Source: Table III · p. 4

C. Evaluation

The evaluation covers public and in-house ASR, heterogeneous speech and audio understanding tasks, and zero-shot TTS. Metrics and output handling vary by task, including WER for ASR and classification-specific matching procedures.

C. Evaluation

  • ASR is evaluated on diverse OpenASR public benchmarks and production-level in-house call-center, conversation, and dictation sets, with WER reported for both.Source: C. Evaluation · p. 4
  • Speech understanding evaluation spans acoustic and paralinguistic classification, speaker and age tasks, music genre, audio understanding, and spoken QA benchmarks.Classification outputs are normalized against labels or choices, invalid or unmatched outputs count as wrong, and age regression extracts a numeric prediction or range midpoint.Source: C. Evaluation · p. 4
  • Zero-shot TTS is evaluated on LibriSpeech test-clean using Whisper-large-v3 WER for intelligibility and UTMOS for perceptual quality.Source: C. Evaluation · p. 4

V. EXPERIMENTAL RESULTS

Mel-LLM’s ASR results show that encoder-free performance is competitive, with limited-data initialization and data scale affecting the gap to encoder-based systems. Gemma-4-12B-it is unstable on noisy long-form ASR inputs.

A. ASR: Main Results and Scaling

  • Gemma-4-12B-it is unstable on noisy long-form ASR, frequently refusing or reporting that it does not detect speech on AMI-like inputs.Source: A. ASR: Main Results and Scaling · p. 4
  • 7.12% average WER on OpenASR with Phi-4-MM initialization is 0.15% behind the 6.97% random-encoder baseline, while random LLM initialization reaches 7.44%.The encoder-free model uses Phi-4-MM LoRA parameters with a randomly initialized convolution frontend and adapters.Source: A. ASR: Main Results and Scaling · p. 4
  • With 10× scaled anonymized in-house data, the encoder-free gap narrows to +3.8% relative (12.52%→12.99%) across in-house test sets.With limited public data, average WER across those sets is 12.52%→13.93%.Source: A. ASR: Main Results and Scaling · p. 4

B. ASR: Ablation Studies

The ASR ablations examine token-rate efficiency and which initialized LLM layers contribute to speech adaptation. The reported trade-offs favor 12.5Hz for speed and suggest upper layers can remain frozen with little degradation.

B. ASR: Ablation Studies

  • Mel-LLM improves tasks relying on acoustic or paralinguistic cues, while encoder-anchored Phi-4-MM and Gemma-4-12B-it remain stronger on knowledge-intensive MMLU-speech.Source: Table VI · p. 5
  • 12.5Hz achieves a 1.57× training speedup over the encoder-based baseline while maintaining competitive WER.Higher token rates of 100Hz and 50Hz yield lower WER (6.58%, 6.71%) but are slower; 25Hz and 12.5Hz have similar WER (7.21% vs. 7.12%).Source: B. ASR: Ablation Studies · p. 5, Table IV · p. 5
  • Freezing LoRA layers 24–31 yields 7.43%, nearly matching the fully trainable setting, whereas extending the freeze to layers 20–31 or 16–31 increases degradation.Freezing layers 28–31 causes only 0.28% degradation (7.12%→7.40%). The pattern suggests lower layers matter more for adapting raw spectral patterns.Source: B. ASR: Ablation Studies · p. 5, Table V · p. 5

C. Understanding: Acoustic Cues vs. Semantic Anchoring

Mel-LLM improves performance on several acoustic and paralinguistic tasks, while encoder-anchored systems remain stronger on knowledge-intensive spoken QA. The paper also reports preliminary continuous-Mel TTS results that trail a latent-diffusion baseline.

C. Understanding: Acoustic Cues vs. Semantic Anchoring

  • Mel-LLM improves over Phi-4-MM on IEMOCAP emotion classification (40.69→66.16), gender classification (92.57→97.73), WavLLM speaker understanding (76.64→87.98), and MMAU-mini (50.70→54.10).With ASR initialization, emotion reaches 72.36, gender 98.21, WavLLM speaker 92.10, and GTZAN music genre 53.55.Source: C. Understanding: Acoustic Cues vs.… · pp. 5–6
  • Phi-4-MM remains stronger on knowledge-intensive MMLU-speech, while Mel-LLM is close on text-only MMLU (61.0→58.68) but scores 41.30 on MMLU-speech.The authors identify semantic anchoring and audio-to-reasoning alignment as the main bottleneck under current evidence.Source: C. Understanding: Acoustic Cues vs.… · p. 6

D. Preliminary Speech Generation Results

E. Scope and Practical Implications

Mel-LLM can read Mel spectrograms without a dedicated speech encoder, but its benefits and limits vary across tasks and depend on initialization and training conditions. The conclusion presents encoder removal as a task-specific trade-off, not a universal replacement.

E. Scope and Practical Implications

  • Mel-LLM reads Mel spectrograms without a dedicated speech encoder, but it is not a universal replacement because acoustic access and semantic alignment remain in tension.The authors identify its practical value in simplifying the audio stack and preserving acoustic and paralinguistic information.Source: E. Scope and Practical Implications · p. 6, §VI · p. 6

VI. CONCLUSION

  • ASR performance is competitive with encoder-initialized models when training data is sufficient, while Phi-4-MM initialization is critical in low-resource settings.The authors also report only a limited performance gap compared with encoder-initialized models.Source: §VI · p. 6, E. Scope and Practical Implications · p. 6
  • Direct Mel access brings strong gains on emotion, gender, speaker, and music understanding, while spoken QA remains weaker than in encoder-anchored systems.The conclusion attributes the spoken-QA difficulty to losing a semantic anchor and an increased audio-to-reasoning bottleneck during acoustic adaptation.Source: §VI · p. 6, E. Scope and Practical Implications · p. 6
  • Direct continuous Mel generation with an LLM backbone remains substantially harder than speech understanding, and latent-diffusion generation remains stronger for TTS.The authors identify stronger latent generation heads as a direction for speech synthesis.Source: §VI · p. 6, E. Scope and Practical Implications · p. 6
  • Future work should explore native speech-text joint training, text distillation, transcript or teacher-encoder supervision, and curricula that avoid overwhelming text-side performance.These are proposed approaches to anchor speech representations and preserve LLM reasoning.Source: §VI · p. 6

VII. GENERATIVE AI USE DISCLOSURE

  • The authors state that AI tools assisted with writing and formatting, while they produced and verified all technical content, experiments, and conclusions.The disclosed uses include grammar correction, spelling checks, formatting, and drawing LaTeX tables.Source: §VII · p. 7
Loading 2606.10231v4…