PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Audio Token Compression in Large Audio Language Models

Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass

arXiv:2511.20973v2eess.AScs.AIcs.CL

TL;DR

High-rate audio features make LALM inputs long and computationally challenging. The paper compresses audio tokens before the LLM and uses LoRA to realign compressed embeddings. Compressed models reduce tokens by up to three times without significantly worsening ASR and S2TT performance.

  • Problem

    High-rate audio features create long LALM inputs that pose computational challenges for transformer-based LLMs.

    Source: §2 · p. 1

  • Method

    The paper compresses audio tokens using unsupervised segmentation or uniform pooling and downsampling, then uses LoRA finetuning to realign compressed embeddings with the LLM.

    Source: §2 · pp. 1–2

  • Results

    Compressed LALMs reduce audio tokens by up to three times without significantly worsening ASR and S2TT performance.

    Source: §6 · p. 4

  • Takeaways & Limitations

    Token merging by unsupervised segmentation or uniform averaging outperforms uniform sampling on WER and BLEU scores.

    Source: §5.1 · p. 3

    The study evaluates ASR and S2TT; future work would benchmark compressed LALMs on audio classification and understanding.

    Source: §6 · p. 4

Abstract

from arXiv · show

Large Audio Language Models (LALMs) deliver strong performance across speech and audio tasks, but their audio encoders generate high-rate token sequences (e.g., 25 tokens/s), making attention computation costly and limiting scalability. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens before they are consumed by the LLM decoder. To mitigate potential performance degradation, we employ low-rank adapters during finetuning. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal, and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count up to three times before the LLM backbone.

2. Introduction

LALMs perform well on audio tasks, but high-rate audio features create long inputs that challenge transformer-based LLM computation. The paper studies token compression methods and LoRA finetuning for preserving lexical content while reducing those inputs.

2. Introduction

  • High-rate audio features create long LLM inputs, and transformer computation scales quadratically with sequence length.A 10-second clip can yield around 500 features at 50 features per second.Source: §2 · p. 1
  • Existing approaches use uniform pooling or speech transcription, but transcription discards paralinguistic information such as speaker identity, prosody, emotion, and environmental context.Uniform pooling typically reduces token counts by a factor of two, while audio token counts remain higher than text token counts.Source: §2 · p. 1
  • The paper explores unsupervised unit discovery, uniform average pooling, and uniform downsampling to reduce audio tokens while studying recovery of underlying lexical content.Unsupervised unit discovery segments audio into acoustically homogeneous units, whose boundaries can merge frame-level features into segment-level features.Source: §2 · p. 1
  • A shift to compressed representations disrupts alignment with the LLM input space, so the authors use LoRA finetuning to restore alignment and recover most frame-level performance.Evaluation focuses on speech recognition and speech-to-speech translation using CommonVoice and CoVoST; some other tasks might allow more aggressive downsampling.Source: §2 · pp. 1–2

3. Related Work

Related work describes LALMs that combine audio encoders with language-model backbones, as well as systems that use transcripts or specialized encoders. Later systems expand audio representations and training approaches for broader audio tasks.

3. Related Work

  • Pengi, SALMONN, LTU, LTU-AS, and AudioGPT illustrate varied LALM designs, including hierarchical or dual encoders, open-ended question answering, transcript input, and routing to task-specific models.AudioGPT, for example, uses Whisper for speech recognition and a Pyramid Transformer for sound detection.Source: §3 · p. 2
  • GAMA combines information from multiple AST layers through an Audio Q-former and uses an improved instruction-tuning dataset for complex reasoning..Source: §3 · p. 2
  • Qwen2-Audio and Audio Flamingo systems improve or train audio encoders and report performance across speech, audio, and music tasks.Qwen2-Audio fully finetunes its Whisper encoder and LLM backbone; Audio Flamingo variants focus on improved audio encoders and audio reasoning.Source: §3 · p. 2

4. Audio token compression in large Audio Large Language Models

The compression methods operate on audio encoder features before the LLM: segmentation uses adjacent-frame dissimilarity to locate boundaries, while local pooling reduces feature sequences by uniform averaging or sampling. LoRA adapters are added to the LLM to realign compressed audio embeddings.

4. Audio token compression in large Audio Large Language Models

  • The audio encoder uses Whisper-large v3 features, with preprocessing and pooling that produce 50 features per second.Audio is resampled to 16 kHz; the mel-spectrogram uses a 25 ms window and 10 ms stride, and the final features cover approximately 40 ms each.Source: §4 · p. 2
  • Unsupervised segmentation detects candidate boundaries from adjacent-frame dissimilarity, then uses a peak detector to locate peaks in that signal.Dissimilarity is defined as 1 minus the similarity between adjacent audio-encoder outputs; a peak exceeds both neighboring values.Source: §4 · p. 2
  • Local pooling downsamples features by a factor of K, using either average pooling with kernel and stride K or sampling every Kth frame.The passage describes the resulting rate as 25/K features per second.Source: §4 · p. 2
  • LoRA adapts the LLM to realign modified audio embeddings while keeping pretrained LLM weights frozen and leaving the audio encoder without adapters.Adapters are placed in key and query projection layers; the configuration adds approximately 9M learnable parameters.Source: §4 · p. 2

5. Experiments

Experiments test token compression on Qwen2-Audio across ASR and speech translation, finding that pooling or segmentation can reduce tokens while retaining strong performance, with results varying by task, compression method, and training data.

5. Experiments

  • Experiments add uniform average pooling, uniform sampling, or unsupervised segmentation to Qwen2-Audio and evaluate ASR and S2TT using LibriSpeech, CommonVoice, and CoVoST data.Source: §5 · p. 2

5.1. Performance on ASR and S2TT task

  • Without LoRA or finetuning, compression factors of 2 and 3 degrade ASR and S2TT performance, severely at factor 3; LoRA realignment recovers most frame-level performance.The authors report relative FLOPs can be halved at compression factor 3 without major performance loss after realignment.Source: §5.1 · p. 3, Table 1 · p. 3
  • A <1 WER increase accompanies halving audio tokens, while reducing them to one-third raises WER by <1.5; merging methods outperform sampling across ASR and S2TT.Unsupervised segmentation performs slightly better than uniform average pooling, which outperforms uniform sampling.Source: §5.1 · p. 3, Table 1 · p. 3
  • At sixfold compression, uniform averaging doubles WER, while uniform sampling's WER increases significantly more as compression rises.The authors suggest averaging retains information better than sampling and frame the choice as a compute–WER trade-off.Source: §5.1 · p. 3, Figure 2 · p. 3

5.2. Does realignment on one language transfer to other languages

  • English LibriSpeech training performs reasonably across languages; adding English CommonVoice lowers WER across languages, and adding other languages yields the best performance.The authors hypothesize that English CommonVoice training helps the model learn the CommonVoice distribution.Source: §5.2 · p. 3, Table 2 · p. 4

5.3. Does realignment on one task transfer to other tasks

  • ASR training on LibriSpeech transfers reasonably to S2TT, but CommonVoice additions do not improve S2TT; CoVoST data or S2TT training yields the best performance.Source: §5.3 · p. 4

5.4. Learnable compression modules

  • A learnable convolution compression layer performs worse despite increasing trainable parameters, and placing compression before the audio encoder raises WER.The convolution module uses 0.14% and 0.16% trainable parameters at compression factors 2 and 3, respectively; the authors suggest it may need more training.Source: §5.4 · p. 4, Table 3 · p. 4

5.5. Impact of WER on understanding and meaning

  • For uniform averaging at compression factors 2 or 3 and unsupervised segmentation, higher WER does not significantly reduce meaning or readability scores; large WER increases from sampling coincide with lower meaning scores.The evaluation uses ChatGPT to score meaning, readability, and misspelled proper nouns.Source: §5.5 · p. 4, Table 4 · p. 4

6. Conclusions and Future Work

The paper uses LoRA to align compressed audio tokens with the LLM embedding space, reducing token counts by up to three times without significantly worsening ASR and S2TT performance. Future work will explore alternatives to additional LoRA weights and evaluation on other tasks.

  • Up to 3 times fewer audio tokens can be used without significantly worsening ASR and S2TT performance.Source: §6 · p. 4
  • LoRA aligns compressed audio tokens to the LLM embedding space.Source: §6 · p. 4
  • Future work will explore methods without additional LoRA weights and benchmark audio classification and understanding.Source: §6 · p. 4

Related papers

Loading 2511.20973v2…