PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren

arXiv:2610.13600v1cs.SDcs.AIeess.AS

TL;DR

Long audio-token sequences make LALM inference costly, while common compression methods may overlook phonetic transitions. AudioTETRIS combines an encoder-derived transition prior with depth-adaptive decoder attention to select tokens. It reports near-lossless performance with 25% token retention and gains over training-free baselines.

  • Problem

    Audio encoders produce hundreds of tokens for short clips, and existing compression methods based on coarse attention or feature similarity can ignore phonetic structure, while decoder self-attention cost grows quadratically with sequence length.

    Source: §1 · pp. 1–2

  • Method

    AudioTETRIS combines a phoneme-transition prior derived from encoder hidden-state changes with depth-adaptive decoder attention to progressively select audio tokens.

    Source: §1 · p. 2

  • Results

    With as few as 25% of tokens, AudioTETRIS retains near-lossless accuracy, outperforms training-free baselines, and can surpass full-sequence models at moderate compression.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Encoder hidden-state temporal variance serves as a proxy for phonetic transitions without fine-tuning or external alignments.

    Source: §1 · p. 3

    The transition prior is validated for Whisper- and Conformer-based encoders but is unconfirmed for different objectives or aggressive downsampling; unseen backbones require calibration.

    Source: LIMITATIONS · p. 9

Abstract

from arXiv · show

Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.

1 INTRODUCTION

AudioTETRIS addresses the computational cost of long audio-token sequences and the risk that compression overlooks phonetic transitions. It combines an encoder-derived acoustic prior with decoder attention and reports strong performance at high compression.

1 INTRODUCTION

  • Audio encoders produce hundreds of tokens for short clips, raising inference costs because decoder self-attention scales quadratically with sequence length.Existing compression approaches based on isolated attention or local feature similarity ignore encoder-captured phonetic structure.Source: §1 · pp. 1–2
  • Speech information concentrates around rapid phonetic transitions, so compression must preserve these acoustically rich regions rather than treat tokens uniformly.The authors link encoder hidden-state changes to phonetic transitions and report validating the relationship between representation variance and segment transitions.Source: §1 · pp. 1–2
  • AudioTETRIS propagates a temporal-derivative acoustic prior into the decoder and fuses it with depth-adaptive attention to guide token survival by structural and semantic relevance.The method is training-free and uses intra-audio attention in early layers and cross-modal text-to-audio attention in later layers.Source: §1 · pp. 2–3
  • With as few as 25% of tokens, AudioTETRIS retains near-lossless accuracy, outperforms training-free baselines, and can surpass full-sequence models at moderate compression.Evaluations span Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3 across speech recognition, emotion recognition, audio question answering, and translation.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers training-free pruning or merging, training-dependent pooling, and multimodal compression. The approaches differ in their use of acoustic structure, task-specific relevance, and extra training or modalities.

2 RELATED WORK

  • Audio-token compression seeks to reduce long sequences without sacrificing the dense lexical content of speech.The long audio sequences create computational overhead during prefill and generation.Source: §2 · p. 3
  • Training-free methods prune or merge tokens using attention or local similarity, but these approaches may lack comprehensive context or phonetic cues.SpeechPrune uses initial-layer attention and speech-text similarity; LTBM merges nearby tokens using local feature similarity.Source: §2 · p. 3
  • Alternative methods use trainable parameters or cross-modal alignment, but can require fine-tuning or rely on visual-audio streams.The paper positions AudioTETRIS as training-free and acoustically grounded in response to this trade-off.Source: §2 · p. 3

3 METHODOLOGY

AudioTETRIS extracts a transition-aware acoustic prior from encoder states, then combines it with depth-adaptive decoder attention to progressively prune audio tokens.

3 METHODOLOGY

  • AudioTETRIS has two stages: it extracts a phoneme-transition-aware prior from encoder hidden states, then fuses that prior with depth-adaptive decoder attention to prune tokens progressively.Figure 1 illustrates the pipeline.Source: §3 · p. 3

3.3 STAGE 2: DEPTH-ADAPTIVE ATTENTION SCORING

  • Audio-token attention mass shifts from intra-audio attention in early layers toward text-conditioned interaction in middle and late layers.In Qwen2-Audio on LibriSpeech, intra-audio attention is 76.91% in early layers, 46.73% in middle layers, and 40.60% in late layers.Source: Figure 2 · p. 4, §3.3 · p. 4

3.1 PRELIMINARIES

  • The encoder’s audio tokens are projected into the LLM space and concatenated with query embeddings; decoder attention operates on a sequence that shrinks as tokens are pruned.The attention matrix is indexed by layer, head, and sequence positions.Source: §3.1 · p. 4

3.2 STAGE 1: PHONEME TRANSITION SCORE

  • Stage 1 measures squared Euclidean distance between consecutive encoder hidden states and normalizes the distances into acoustic weights with unit mean.The first token receives zero derivative because it has no preceding token.Source: §3.2 · p. 4

3.3 STAGE 2: DEPTH-ADAPTIVE ATTENTION SCORING

  • Stage 2 finds the crossover layer from a calibration set, fuses layer-adaptive attention with the acoustic weight, and retains the top fraction at each pruning layer.The crossover is the first layer where mean cross-modal attention mass meets or exceeds mean intra-audio attention mass.Source: §3.3 · p. 5

4 ANALYSIS: VALIDATION OF ACOUSTIC STATE DYNAMICS

Encoder-state transitions align with phoneme boundaries most strongly in mid-network layers across all three backbones. A label-free kurtosis diagnostic and qualitative trajectory analysis identify the same mid-network sweet spot.

4 ANALYSIS: VALIDATION OF ACOUSTIC STATE DYNAMICS

  • The transition diagnostic smooths adjacent-state distances and compares them with forced-alignment boundaries, while kurtosis checks transition prominence without external alignments.The correlation analysis uses a Gaussian kernel of width σ = 2 and both discrete and smoothed alignment targets.Source: §4 · p. 5
  • Transition alignment peaks at layer 16 for Qwen2-Audio (rsoft = 0.68), layer 14 for Phi-4-Multimodal (0.59), and layer 15 for Audio Flamingo 3 (0.62), alongside a mid-network kurtosis peak.The correlation and kurtosis diagnostics identify the same transition-rich region across backbones.Source: §4 · p. 5, Figure 3 · p. 6
  • Mid-network hidden states show tight phoneme clusters separated by sudden trajectory jumps, whereas early and late layers show less phonetically consolidated structure.The qualitative t-SNE analysis corroborates the mid-network transition pattern.Source: §4 · p. 5

5 EXPERIMENTS & RESULTS

Across four speech-text tasks, AudioTETRIS retains strong performance under compression, with widening advantages over baselines at higher compression and reported compute and latency savings. Results on two additional backbones indicate that the pattern extends beyond Qwen2-Audio.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers ASR, emotion recognition, audio question answering, and English-to-Chinese speech translation across three distinct LALM backbones, with four compression rates from 0.36 to 0.75.The compared baselines are SpeechPrune, SparseVLM, LTBM, and matched-rate random pruning; pruning is scheduled at layer 11 and a backbone-specific later layer, using a mid-network encoder layer selected by kurtosis.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • At rate 0.36, AudioTETRIS reports 4.52 WER, 58.04% IEMOCAP, and 42.36% MMAU-mini versus 4.61, 55.11%, and 42.20% uncompressed; at rate 0.75, WER is 8.50.At rate 0.75, IEMOCAP, MMAU-mini, and CoVoST2 remain within 2 points of uncompressed baselines, while random pruning reaches 31.59 WER and AudioTETRIS beats the best baseline by over 1.2 WER points.Source: §5.2 · p. 7

5.3 EFFICIENCY ANALYSIS

  • At 0.75 compression, pruning within the LLM reduces total TFLOPs from 7.89 to 4.32 and latency from 1.31 s to 0.62 s.Real-world latency savings are smaller than the FLOP reduction, partly because of RoPE reindexing and KV-cache trimming overhead.Source: §5.3 · p. 7

5.4 GENERALIZATION ACROSS BACKBONES

  • At compression ratio 0.51, AudioTETRIS stays within 1 point of uncompressed WER and achieves strongest or near-strongest MMAU-mini accuracy on Phi-4-Multimodal and Audio Flamingo 3.The reported pattern is consistent with transition-prior transfer across differing encoder architectures and subsampling rates.Source: §5.4 · p. 7

6 ABLATION STUDIES

The ablations support L2 as the transition-score distance and show that combining the acoustic prior with attention performs better than either stage alone, particularly under aggressive compression.

6 ABLATION STUDIES

  • The ablations isolate the scoring-function components and compare transition-score distance metrics, with the stage comparison set against the full pipeline at compression ratio 0.51.Extended component results across ratios and benchmarks are reported in Table 6.Source: §6 · p. 7, §6.2 · p. 8

6.1 DISTANCE METRIC FOR THE TRANSITION SCORE

  • L2 is the strongest transition-score distance on all four benchmarks; cosine raises ASR WER by over 50% (9.04% vs. 5.77%).The comparison holds other hyperparameters fixed at compression ratio 0.51, and L2 is used as the default.Source: §6.1 · pp. 7–8, Figure 5 · p. 9

6.2 COMPONENT AND STAGE ABLATION ANALYSIS

  • Across ratios and benchmarks, both stages together outperform either stage alone; at ratio 0.75, Stage 2 alone gives 15.56% WER versus 8.50% combined.The acoustic prior preserves phonetically contiguous spans while attention targets task relevance, with the advantage especially visible on LibriSpeech.Source: §6.2 · pp. 8–9

7 CONCLUSION

AudioTETRIS uses encoder hidden-state dynamics as a phoneme-transition prior, fused with depth-adaptive decoder attention. It retains near-lossless performance at light compression and widens its advantage over training-free baselines as compression becomes aggressive.

7 CONCLUSION

  • AudioTETRIS retains near-lossless performance at light compression, while its advantage over training-free baselines widens as compression becomes aggressive.Source: §7 · p. 9
  • Encoder hidden-state dynamics provide a phoneme-transition prior fused with depth-adaptive decoder attention, without auxiliary training or fine-tuning.The paper argues that modality-specific encoder representations contain structural geometry worth propagating to the LLM interface.Source: §7 · p. 9
  • ASR is highly sensitive to token loss, making speech-aware structural priors important for exact lexical recovery.Source: §7 · p. 9

LIMITATIONS

  • The acoustic prior assumes large-norm hidden-state transitions at phonetic boundaries; this is validated for Whisper- and Conformer-based encoders, not other objectives or aggressive downsampling.Source: LIMITATIONS · p. 9
  • A one-time, backbone-specific calibration pass is needed to locate the later crossover layer, so the method is not fully zero-shot on unseen architectures.Source: LIMITATIONS · p. 9
  • Progressive decoder pruning adds a small TFLOP overhead versus single-shot methods, though the paper says accuracy gains largely offset it.Source: LIMITATIONS · p. 9
  • Evaluation covers four English-source benchmarks; noisier or multilingual audio is untested, and higher compression would likely erode lexical accuracy first.Source: LIMITATIONS · p. 9

REPRODUCIBILITY STATEMENT

AudioTETRIS is training-free, and the paper documents its scoring, calibration, pruning schedule, checkpoints, and dataset splits for reproduction.

REPRODUCIBILITY STATEMENT

  • No training procedure, optimizer, or learning-rate schedule is needed because AudioTETRIS is training-free.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper provides the transition and attention scores, crossover-layer discovery, fused-score pruning pseudocode, pruning schedule, checkpoints, dataset splits, and licenses.Source: REPRODUCIBILITY STATEMENT · p. 10
  • A one-time, per-backbone calibration locates the crossover layer from attention statistics without gradients or parameter updates.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors state that they will release the implementation and evaluation scripts used to produce Tables 1–5.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1.1 BACKBONES AND CHECKPOINTS

Table 2 identifies the pretrained checkpoints used in the work, and the experiments use unmodified weights.

A.1.1 BACKBONES AND CHECKPOINTS

  • All experiments use unmodified pretrained weights; only load-time decoder-forward monkey-patches implement AudioTETRIS pruning.Source: §A.1.1 · p. 14
  • Table 2 lists the pretrained checkpoints used in the work.Source: Table 2 · p. 14
  • A checkpoint is marked for non-commercial research use only, with full terms in its checkpoint card.Source: §A.1.1 · p. 14

A.1.2 DATASETS, SPLITS, AND EVALUATION

The appendix presents extended backbone and component-ablation results alongside analyses of AudioTETRIS’s pruning layers, transition-aware scoring, and implementation choices. It also examines pruning schedules and qualitative examples, including the complementary token information preserved by two-stage pruning.

A.1.3 SPARSEVLM ADAPTATION TO FIXED-RATE SPEECH PRUNING

  • Because SparseVLM’s public releases omit its original dynamic controller, the authors adapt its attention-guided scoring to fixed retention rates at layers 1 and 15 for controlled comparison.The adaptation retains SparseVLM’s text-token attention scoring and evaluates retention fractions 0.36, 0.51, 0.64, and 0.75.Source: §A.1.3 · p. 14

A.1.4 ATTENTION BACKEND FOR SCORE-BASED PRUNING

  • Attention-score pruning requires materialized attention probabilities, so the authors use eager attention only at scoring layers and restore FlashAttention-2 elsewhere, capturing eager-fallback cost in latency but not TFLOPs.This hybrid path is applied identically to AudioTETRIS and attention-based baselines; the fallback occurs at two decoder layers and could be replaced by a custom fused kernel.Source: §A.1.4 · pp. 14–15

A.2.2 PRUNING LAYER SELECTION

  • AudioTETRIS first prunes at layer 1, the earliest stage to maximize cumulative FLOP reduction, then sets crossover layers at 15 for Qwen2-Audio, 13 for Phi-4-Multimodal, and 18 for Audio Flamingo 3.At layer 1, the fused score combines attention-based importance with an acoustic transition prior from encoder hidden states, compensating for weak early attention.Source: §A.2.2 · pp. 15–17

A.2.4 BOUNDARY CORRELATION ON COVOST2 AND IEMOCAP

  • On CoVoST2 and IEMOCAP, boundary correlation again peaks mid-network across all three backbones; peak rsoft is higher on CoVoST2 for Qwen2-Audio (0.67 vs. 0.60) and Audio Flamingo 3 (0.61 vs. 0.58).IEMOCAP uses released forced alignments, whereas the authors generated CoVoST2 English-source phoneme alignments with Montreal Forced Aligner; the analyses use σ = 2.Source: §A.1.2 · p. 14, §A.2.4 · pp. 18–19, Figure 9 · p. 19

A.2.5 SENSITIVITY TO THE SMOOTHING WIDTH σ

  • Across all backbones, rsoft and rhard peak at σ = 2 tokens and decline on either side, as narrower smoothing misses context and wider smoothing blurs phonetic transitions.The transition prior smooths the raw distance between consecutive encoder states, while the forced-alignment transition labels are smoothed with the same σ.Source: §A.2.5 · pp. 19–20, Figure 10 · p. 20

A.2.6 QUALITATIVE ENCODER EMBEDDING TRAJECTORIES

  • In Qwen2-Audio’s t-SNE visualization, middle-layer tokens cluster compactly by phoneme, while early- and late-layer tokens show less phoneme consolidation and less distinct boundaries.The authors treat this single-utterance visualization as illustrative because t-SNE preserves local neighborhoods but not global distances.Source: §A.2.6 · p. 20, Figure 11 · p. 20

A.3.1 EXTENDED RESULTS ON PHI-4-MULTIMODAL AND AUDIO FLAMINGO 3

  • Across four compression ratios, AudioTETRIS maintains near-lossless performance at low compression and widens its advantage over baselines as compression increases on Phi-4-Multimodal and Audio Flamingo 3.The extended results are reported in Table 4 for Phi-4-Multimodal and Table 5 for Audio Flamingo 3.Source: §A.3.1 · p. 21, Table 4 · p. 21, Table 5 · p. 22

A.3.2 EXTENDED COMPONENT-WISE ABLATION

  • Across all four compression ratios and benchmarks, the full two-stage pipeline outperforms either isolated stage, with a wider gap at higher compression.The schedule ablations also vary pruning-stage placement and compression-budget allocation; Table 6 reports the component comparisons.Source: §A.3.2 · p. 21, §A.3.3 · pp. 21–22, Table 6 · p. 22

A.3.3 SINGLE-STAGE AND ALTERNATIVE MULTI-STAGE SCHEDULES

  • At rate 0.51, two-stage {1, 15} reaches 5.77% WER, beating {1, 19} (6.02%) and single-stage layer 15 (6.93%); two-stage splits improve across rates.Single-stage layer 15 also outperforms layer 1 (8.92% WER at rate 0.51), highlighting text-conditioned attention's role in token ranking.Source: §A.3.3 · p. 23, Table 9 · p. 23
  • At the evaluated rates, three-stage {1, 8, 15} yields small WER gains over two-stage {1, 15} at three of four rates, but at rate 0.51 scores 5.93% versus 5.77%.The two-stage schedule remains the default; at rate 0.51, three-stage and dense schedules have 1.04× and 1.21× latency, respectively, without consistent downstream gains.Source: §A.3.3 · pp. 23–24
  • At rate 0.36, front-loading yields 4.99% vs. default 4.52% WER, while at 0.75 back-loading worsens to 14.53% vs. 10.73%; later pruning and 11-stage schedules incur penalties.The authors attribute the later-pruning penalty to cross-modal attention dominating in later layers; the 11-stage dense schedule underperforms at all rates.Source: §A.3.3 · p. 24

A.3.4 QUALITATIVE EXAMPLES OF TOKEN SURVIVAL

  • Token-survival examples show complementary stages: Stage 1 protects dense, distributed phonetic detail, while Stage 2 adds task-relevant tokens Stage 1 would not prioritize.The examples cover three representative LibriSpeech utterances; the figure distinguishes tokens kept uniquely by each stage and by the combined method.Source: §A.3.4 · p. 24, Figure 12 · p. 25
Loading 2610.13600v1…