PaperlayerView exact revision on arXiv

Source-linked AI summary

Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper

Chanhee Cho, Junhyuk Choi, Bugeun Kim

arXiv:2608.30927v1cs.SDcs.AI

TL;DR

Whisper’s fixed audio-token interface may contain more redundancy than downstream ASR and SpeechLM tasks require. The paper introduces train-free stride-k subsampling at the convolutional stem or encoder output, retaining every k-th token. Across Whisper and Whisper-based SpeechLMs, stride-2 substantially reduces tokens and computation while usually preserving performance, with larger costs on harder or weaker settings.

  • Problem

    Prior efficiency methods leave Whisper’s input-side redundancy unaddressed, while train-free methods rely on data-dependent criteria or intervene after encoder processing.

  • Method

    Stride-k subsampling deterministically retains every k-th token after the convolutional stem or encoder transformer without training or auxiliary computation.

  • Results

    Compound stride-2 reduces audio tokens by 75%, total GFLOPs by 52–58%, and SpeechLM latency by 19.6–27.4%, with small WER or accuracy costs on most stronger settings.

  • Takeaways & Limitations

    Whisper’s preprocessing overlap and encoder representations support substantial train-free audio-token reduction, with the largest wall-clock effect when tokens enter downstream LLM prefill.

  • Takeaways & Limitations

    The method assumes sufficient temporal overlap in the frontend and requires access to internal encoder computation, excluding raw-waveform generalization and closed API-only models.

Abstract

from arXiv · show

Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.

1 Introduction

Whisper’s fixed audio-token interface carries preprocessing and encoder redundancy that can be reduced without training. Stride-k subsampling targets this redundancy at the stem or encoder output, with stride-2 retaining performance while reducing computation.

  • Motivation: Whisper exposes ASR decoders and SpeechLMs to a fixed 1500-token audio interface whose length governs downstream cost.ASR decoders cross-attend to the sequence, while SpeechLMs insert the tokens into an LLM context.
  • Motivation: Stride-2 after the stem introduces no temporal coverage gap because overlapping receptive fields still cover the full input span.The retained receptive fields overlap across approximately 62% of the input span before self-attention.
  • Motivation: Prior efficiency methods either retrain or prune representations after the encoder has processed them, leaving input-side redundancy unaddressed.The supplied related-work passage describes distillation, fine-tuning, retraining, and downstream pruning limitations.
  • Method: Stride-k retains every k-th token through indexing alone at either the convolutional stem output or encoder output.The operation adds no parameters or auxiliary computation and can be inserted into Whisper-based models with one indexing step.
  • Mechanism: CKA links input-side stability to acoustic overlap and output-side stability to attention-induced redistribution.Input-side similarity weakens with frame distance, whereas output-side similarity drops sharply before plateauing.
  • Results: Compound stride-2 reduces audio tokens by 75% and total GFLOPs by roughly 52–58%, while mostly retaining ASR WER and SpeechLM accuracy on stronger baselines.It reduces SpeechLM end-to-end latency by 19.6–27.4%, with larger accuracy costs on harder benchmarks or weaker baselines.

2 Related Work

Prior work reduces Whisper or audio-token costs through compression, adaptation, pruning, or token merging. The identified gap is a train-free operation that reduces sequence length before the encoder transformer without data-dependent criteria.

  • Existing approaches: Distillation-based methods compress Whisper, while parameter-efficient fine-tuning adapts it for low-resource settings.These approaches modify or retrain model behavior rather than providing a purely train-free indexing operation.
  • Existing approaches: Other train-free methods prune or merge audio sequences using speech-text similarity, attention scores, cosine similarity, or locality.The cited methods operate on hidden states, downstream audio tokens, or intermediate encoder tokens.
  • Gap: Prior methods leave two gaps: they do not shorten input before encoder processing and they rely on data-dependent similarity or attention analysis.These limitations motivate deterministic subsampling at the encoder input.

3 Stride-k Subsampling

Stride-k subsampling deterministically selects every k-th Whisper representation, either before or after the encoder transformer. Its input-side placement reduces encoder and decoder attention costs, while output-side placement reduces decoder cross-attention costs only.

  • Preliminaries: Whisper’s 25 ms Hann window, 10 ms hop, and stride-2 convolutional stem produce overlapping representations in a 1500-token sequence.Stride-2 after the stem leaves no temporal coverage gap and approximately 62% accumulated overlap, whereas stride-3 leaves 8.3%.
  • Preliminaries: Stride-k is defined as H′=H[::k,:]∈R⌈T/k⌉×d, retaining every k-th token without training, fine-tuning, distillation, or auxiliary computation.The operation reduces a representation of T tokens and hidden size d to approximately T/k tokens.
  • Positions: Input-side subsampling occurs after the convolutional stem and before the transformer, reducing both encoder self-attention and decoder cross-attention FLOPs.The encoder processes 1500/k tokens instead of 1500.
  • Positions: Output-side subsampling occurs after the transformer and before decoder cross-attention, leaving encoder FLOPs unchanged.Only decoder cross-attention is reduced, because the decoder receives fewer target tokens.

4 Diagnosing the stride-k Subsampling

Experiments across three benchmarks and five Whisper scales show that stride-2 is generally stable, while degradation depends on task difficulty, model scale, and subsampling position. Input-side performance degrades more sharply beyond k=2 than output-side performance.

  • Experimental setup: WER is evaluated across three benchmarks, five Whisper scales, stride values k=1,…,6, and both subsampling positions.The benchmarks span clean read speech, expressive speech, and varied crowd-sourced recording conditions.
  • Benchmark difficulty: On LibriTTS, Whisper-Large output-side WER changes from 4.31 at baseline to 4.30 at k=2 and 5.47 at k=4.On Common Voice, the corresponding values are 11.66, 12.70, and 16.26.
  • Model scale: At input-side k=2 on LibriTTS, WER increases relative to baseline by factors of 2.40 for Tiny, 1.82 for Base, 1.60 for Small, 1.35 for Medium, and 1.07 for Large.Larger models therefore tolerate more aggressive subsampling before degradation.
  • Subsampling position: At k=2, output-side WER is lower than input-side WER for Whisper-Small and Whisper-Medium on LibriTTS.The reported pairs are 8.04 versus 5.13 for Small and 6.22 versus 4.70 for Medium.
  • Subsampling position: At k=3, input-side WER rises abruptly, whereas output-side WER remains near baseline through k=4.For LibriTTS Whisper-Medium, input-side WER changes from 4.61 to 19.51 at k=3, while output-side degradation remains gradual.
  • Overall pattern: Stable k ranges are associated with lower baseline WER, indicating that stronger-performing configurations tolerate more token reduction.This pattern holds across benchmarks, model scales, and both subsampling positions.

5 CKA Analysis of Adjacent-token Similarity

CKA reveals distinct similarity-decay patterns at Whisper’s conv-stem and encoder-output positions. These patterns align with the different degradation profiles observed under stride-k subsampling.

  • Measurement: CKA measures step-wise similarity decay between anchor tokens and tokens at frame distances d=1 through 5 across both representation positions.The analysis uses LibriTTS, ten anchor tokens per utterance, and five Whisper scales.
  • Results: At the conv stem, similarity decays continuously as frame distance grows, consistent with diminishing receptive-field overlap.On Whisper-Base, the step-wise change moves from −6.48 at d=1 to −3.25 at d=4; post-stem tokens span about 65 ms and are spaced every 20 ms.
  • Results: At the encoder output, similarity changes sharply over the first two steps and then plateaus across larger distances.On Whisper-Large, the changes move from −14.03 and −11.26 at d=0,1 to −0.49 at d=4, with the same pattern across scales.
  • Results: The two decay shapes align with stride-k WER behavior: input-side performance stays near baseline at k=2 but worsens sharply from k=3, while output-side degradation remains gradual through k=4.The positional asymmetry reflects similarity-decay shape rather than absolute CKA level.
  • Interpretation: The encoder transformer redistributes redundancy across a wider local neighborhood instead of concentrating it only in adjacent tokens.This output-side plateau exceeds the continuous decay predicted from locality alone and supports a wider stable subsampling range.

6 Compound Stride-2

Compound stride-2 applies subsampling at both Whisper encoder positions, reducing tokens and computation while retaining performance most reliably on stronger models, cleaner inputs, and tasks using broadly distributed acoustic evidence.

  • Whisper ASR: WER remains reasonably close to baseline on LibriTTS and ESD, but Common Voice shows increases of +13.40, +9.56, and +9.90 for Small, Medium, and Large-v3.On LibriTTS, the corresponding increases are +4.20, +2.40, and +1.07; on ESD, they are +4.84, +3.50, and +1.01.
  • Whisper-based SpeechLMs: SpeechLM accuracy drops least for Audio Flamingo 3, is intermediate for Qwen2-Audio, and is largest for LLaMA-Omni 2 across the benchmark blocks.The largest individual drop is −13.72 on LLaMA-Omni 2 Reasoning Linguistic Semantics.
  • Efficiency: 19.6–27.4% lower end-to-end latency accompanies 51.9–56.6% lower GFLOPs across the three SpeechLMs.ASR latency improves much less despite approximately 57% lower GFLOPs, reaching −6.0% on Large and near-zero or slightly negative changes on smaller scales.
  • Robustness: Robustness follows baseline strength and input stability: stronger Whisper checkpoints and cleaner or controlled inputs absorb compound stride-2 more reliably.As SNR falls from clean to 0 dB, the Large-v3 WER gap on LibriTTS grows from +0.9 to +28.6, while output-side k = 2 stays within about one point of baseline.
  • Temporal evidence: Compound stride-2 is most fragile when decisions require sparse, fine-grained temporal cues, and more reliable when they use broad acoustic patterns.Hurt cases concentrate around pause boundaries, word-level intonation, and event counting; help cases include coarse prosody, tempo, and stable timbre.
  • Composability: Stride-k can stack with other compression methods, including SpeechPrune and Distil-Whisper, because it reduces encoder-side tokens while decoder compression acts on another axis.On Distil-Whisper large-v3, compound stride-2 cuts total FLOPs by roughly a further 54%, with WER close to baseline on LibriTTS and ESD.

7 Conclusion

The paper introduces train-free stride-k subsampling for Whisper and finds that compound stride-2 substantially reduces token and compute costs with generally modest performance losses. The results indicate that Whisper’s standard audio-token interface is over-provisioned for many downstream uses, with the largest latency gains when tokens enter downstream LLM prefill.

  • Method: Stride-k retains every k-th Whisper token after the convolutional stem or encoder transformer, requiring no training or auxiliary computation.Compound stride-2 reduces audio tokens by 75% and GFLOPs by 52–58%.
  • Results: Across Whisper ASR and SpeechLMs, compound stride-2 keeps performance close to baseline on many settings while larger costs occur on harder benchmarks or weaker baselines.SpeechLM latency decreases by 19.6–27.4%.
  • Implication: Audio-token reduction has its largest wall-clock effect when tokens enter downstream LLM prefill, while ASR benefits more through FLOPs reduction and composability with decoder-side compression.The conclusion distinguishes latency gains in SpeechLMs from the smaller ASR speedups.

Limitations

The method’s scope is bounded by its reliance on overlapping windowed-frame frontends and access to internal encoder computation, and the study does not examine retraining or fine-tuning.

  • Scope and assumptions: Stride-k assumes overlapping windowed-frame processing, covers open-weight models with internal encoder access, and excludes closed API-only models.The study evaluates the operation purely without retraining or fine-tuning.

Usage of Large Language Models

Whisper’s overlapping acoustic preprocessing creates redundant encoder representations with broad temporal coverage. The paper analyzes how deterministic stride-k pruning changes spacing, overlap, and coverage across the encoder sequence.

  • Preprocessing redundancy: Whisper uses 25 ms Hann windows with a 10 ms hop, while convolutional aggregation combines several overlapping frames into each encoder token.This gives tokens an effective receptive field wider than their spacing.
  • Receptive field: 65 ms is the effective receptive field produced by Whisper’s window, hop, and approximately five-frame aggregation configuration.The receptive field is determined from the STFT window, frame hop, and number of aggregated frames.
  • Subsampling operation: Stride-k retains one token every k encoder tokens, increasing retained-token spacing while reducing sequence length through deterministic indexing.The convolutional stem outputs tokens every 20 ms, so stride-k changes the spacing to k times that interval.
  • Coverage analysis: The capped global overlap ratio is normalized over the 30-second input and bounded at 100% to prevent accumulated overlap from exceeding the input duration.Table 4 summarizes this capped overlap and accumulated coverage loss for different stride factors.
  • Coverage analysis: Coverage gaps emerge when retained-token spacing exceeds the effective receptive field, while accumulated overlap and normalized global coverage quantify remaining redundancy.The equations estimate both overlap across retained representations and the stride at which uncovered temporal regions begin.

A.3 Result for Different Values of k

The paper evaluates stride-k at multiple Whisper encoder positions and across ASR and SpeechLM settings. It combines theoretical overlap analysis, CKA measurements, efficiency estimates, and frozen-model experiments to assess performance–efficiency trade-offs.

  • A.3 Result for Different Values of k: 62.4% capped global temporal overlap remains after stride-2 removes half of the encoder tokens across the full 30-second sequence.The original k = 1 representations provide complete capped overlap coverage, while stride-3 falls to 8.3%.
  • A.3 Result for Different Values of k: Stride-2 is identified as the moderate subsampling factor that reduces computational redundancy while preserving dense temporal acoustic coverage.Explicit temporal coverage loss first emerges at k ≥4, when retained-token spacing exceeds the receptive field.
  • Subsampling positions: Input-side subsampling shortens both encoder-transformer and decoder cross-attention sequences, whereas output-side subsampling shortens only the decoder’s encoder-output sequence.For input-side pruning, T and Tenc become approximately ⌈1500/k⌉; output-side leaves T at 1500 while reducing Tenc.
  • Representation analysis: CKA compares adjacent-token representations at the convolutional-stem output and final encoder output across five Whisper scales and token distances.The measurements use frozen checkpoints and independently compute linear CKA for each position, scale, and distance.
  • SpeechLM evaluation: Three Whisper-based SpeechLMs are evaluated on MMSU and MMAU without training, fine-tuning, or calibration after subsampling.Their pipelines differ in how speech encoder outputs become LLM-side audio tokens; LLaMA-Omni2 maps 1,500 baseline tokens to 300 and compound stride-2 maps 375 to 75.

H ASR Noise Robustness

Noise robustness experiments compare baseline, input-side, output-side, and compound stride-2 under matched SNR conditions. They show that output-side pruning remains near baseline, while input-side costs grow as acoustic difficulty increases.

  • Experimental design: The experiment uses the same utterances and noise realizations across model scales and stride conditions, comparing every subsampling condition with its same-SNR baseline.This design separates the cost of subsampling from the cost of added noise.
  • Noise robustness: +57.5 WER at 0 dB is the compound stride-2 cost for Small, compared with +28.6 for Large-v3.On clean speech, the corresponding costs are +3.7 for Small and +0.9 for Large-v3.
  • Noise robustness: Output-side stride-2 stays within about one WER point of baseline at every tested SNR on both Whisper scales.It is slightly better than baseline in two reported cases: −0.11 on clean Small and −0.09 at 10 dB.
  • Noise robustness: Baseline WER rises from 4.72 to 27.73 on Small and from 4.21 to 15.34 on Large-v3 as conditions move from clean speech to 0 dB.The widening subsampling gap is described as reflecting a shrinking redundancy margin rather than a fixed additive noise cost.
  • Error analysis: Added compound-stride errors primarily involve omissions and substitutions of function words, including articles and plural suffixes.These short, low-energy units are more vulnerable when the temporal sampling rate is halved, while downstream language-model context can restore them more effectively in SpeechLMs.

I Multilingual Evaluation

Multilingual and cross-frontend results show that stride-2 remains viable when adjacent encoder tokens retain temporal overlap, but performance depends on overlap margin and task difficulty.

  • Multilingual Evaluation: Output-side k=2 stays within one CER point of baseline at every Chinese scale and improves slightly on Large-v3 by −0.20.
  • Multilingual Evaluation: Input-side k=2 increases Chinese CER by +6.01, +6.73, and +3.78, while compound stride-2 produces the largest increases.
  • Multilingual Evaluation: Broader multilingual evaluation, including Japanese and other Common Voice languages, remains future work.
  • Frontend Generalization: The operative transfer condition is sufficient temporal overlap between adjacent preprocessing tokens, not the Whisper architecture itself.
  • Frontend Generalization: Whisper’s frontend gives each post-stem token a 65 ms receptive field spaced every 20 ms, leaving 25 ms overlap under stride-2.
  • Frontend Generalization: Phi-4-multimodal retains positive overlap but with only 5 ms residual overlap, predicting greater degradation than Whisper under stride-2.
  • Frontend Generalization: Phi-4-multimodal degrades under compound stride-2, with MMAU accuracy falling from 62.20 to 56.50 and MMSU Reasoning from 70.41 to 58.97.

L.2 Results

Failure-case analyses show that stride-2 disproportionately affects temporally fine-grained tasks, while aggregate degradation follows baseline strength and model robustness.

  • Results: 9.4%, 14.8%, and 24.5% are the MMSU overall failure rates for AF3, Qwen2-Audio, and LLaMA-Omni 2, respectively.
  • Results: 6.8%, 12.6%, and 20.6% are the corresponding MMAU overall failure rates, preserving the ordering AF3 < Qwen2-Audio < LLaMA-Omni 2.
  • Category-Level Results: MMSU Intonation and Pause have the highest non-Others failure rates, averaging 20.4% and 20.3% across models.
  • Category-Level Results: MMAU Counting averages 20.7% failure and shows a baseline-flat profile of 21.4% / 22.0% / 18.8%.
  • Category-Level Results: Other categories spread more widely across baselines, so their failure rates are attributed to baseline strength rather than intrinsic category vulnerability.
  • Discussion: The residual cost of compound stride-2 falls disproportionately on temporally fine-grained perception.
  • Efficiency: Compound stride-2 reduces total FLOPs by roughly a further 54% for Distil-Whisper large-v3 while WER remains close to baseline on LibriTTS and ESD.
Loading 2608.30927v1…