Source-linked AI summary
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Cihan Xiao, Yuan Tseng, Matthew Wiesner, Desh Raj, Leibny Paola Garcia Perera, Sanjeev Khudanpur, David Harwath
TL;DR
Acoustic encoders produce long, costly sequences for LLMs, and compression must preserve task-oriented information. ASSET learns variable-duration segmentation and pooling from task feedback; it approaches a character-boundary oracle with fewer LibriSpeech tokens and outperforms AISHELL oracles at a comparable phone token rate.
Problem
Acoustic encoders produce high-frequency sequences that are costly for LLMs, and suitable auxiliary alignments for compression are less obvious for tasks such as emotion recognition.
Source: §1 · p. 1Method
ASSET uses reinforcement learning to learn task-guided boundaries for variable-duration segments and trains a residual pooler and decoder with teacher-forced cross-entropy.
Source: §1 · p. 1Results
ASSET approaches the character oracle with fewer tokens on LibriSpeech and reaches 7.53% CER at 7.7 tokens/s on AISHELL, outperforming both oracles at a phone-comparable rate.
Source: §4.2 · pp. 3–4Takeaways & Limitations
Exploratory non-ASR experiments find that segmentation varies across tasks and, within a policy, with how the same utterance is expressed.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence length, while acoustic encoders tend to produce substantially longer sequences than typical textual inputs. This motivates learning to compress encoder outputs while preserving task-oriented information. To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token. We use group-relative policy optimization (GRPO) to train a segmenter with task performance feedback and a token-rate penalty. In automatic speech recognition (ASR) experiments, ASSET achieves performance comparable to a transcript-assisted character-boundary oracle with approximately 25% fewer audio tokens on English LibriSpeech, while improving on a phone-boundary oracle at a similar token rate on Mandarin AISHELL. Exploratory experiments on non-ASR tasks further illustrate how learned segmentation changes when the framework is adapted to tasks beyond transcription.
1. INTRODUCTION
Acoustic encoders produce long, costly sequences for LLMs, motivating compression that preserves task-relevant information. ASSET learns task-guided segmentation and pooling, addressing the difficulty of training discrete boundary decisions and the lack of suitable auxiliary alignments for some tasks.
1. INTRODUCTION
- Long acoustic prefixes increase context use and decoder self-attention cost, which grows quadratically with input sequence length.The paper targets cases where downstream tasks do not require the encoder’s full temporal resolution.Source: §1 · p. 1
- Fixed-rate stacking or pooling uses evenly distributed boundaries [6], while CTC compression uses auxiliary alignments that are less obvious for tasks such as emotion recognition.This motivates learning compression directly from the target task.Source: §1 · p. 1
- Discrete boundary decisions block ordinary backpropagation, so reinforcement learning scores their effect on downstream task performance without requiring an auxiliary frame-level target.Pooling can be optimized by backpropagation once segment assignments are given.Source: §1 · p. 1
- ASSET combines a learned segmenter with a residual pooler, using task feedback for boundaries while teacher-forcing trains the pooler and decoder.The paper also reports cross-language ASR experiments and exploratory non-ASR studies of task- and speaking-style-dependent policies.Source: §1 · p. 1
2. RELATED WORK
Prior speech-input compression includes CTC collapse, fixed pooling, and adaptive merging, while other adaptive approaches use soft alignments, fixed reduction, or RL objectives outside LLM-based ASR. ASSET distinguishes itself by learning boundaries from task performance feedback while retaining frames in a monotonic partition.
2. RELATED WORK
- Decoder-only speech systems use CTC collapse or fixed pooling [6]; EFIN pretrains the encoder, while AdaTS adaptively merges tokens by pairwise similarity.ASSET instead learns boundary policies from task-performance feedback while retaining every frame in a monotonic partition.Source: §2 · p. 1
- Prior adaptive methods include soft monotonic alignment, extreme fixed reduction, phone-error and frame-rate RL, and predicted-phoneme-perplexity rewards.ASSET optimizes boundaries against LLM decoder transcription loss and uses supervised order-aware residual pooling.Source: §2 · p. 1
- The ASSET overview shows a policy partitioning frames into variable-duration segments, with each segment pooled to an audio token and task losses scoring sampled boundaries.Cross-entropy updates the pooler and decoder; GRPO updates the segmenter.Source: Figure 1 · p. 2
3. METHOD
ASSET compresses encoder frames into variable-duration segments and pools each into an audio token before the LLM decoder. It learns boundary decisions from task feedback while training the pooler and decoder directly.
3. METHOD
- ASSET uses frozen encoder features, a learned boundary policy, and a residual recurrent pooler to create audio tokens that condition decoder text generation.The resulting tokens are prepended to the decoder input.Source: §3 · p. 2
3.1. Adaptive segmentation
- The segmenter makes causal Bernoulli boundary decisions, where each decision can depend on previous boundaries, to partition frames into contiguous variable-duration segments.The frame kept ratio is M/T, and the token rate is that ratio multiplied by the encoder rate.Source: §3.1 · p. 2
3.2. Residual segment pooling
- Because a segment mean is permutation-invariant, a BiGRU adds order-explicit information as a residual to the mean-pooled representation.Zero-initializing the residual projection makes the pooler exactly mean pooling at initialization.Source: §3.2 · p. 2
3.3. Task-guided policy optimization
- For each utterance, ASSET samples boundary sequences and scores their pooled prefixes using the decoder’s normalized teacher-forced continuation loss.For ASR, the continuation is the reference transcript.Source: §3.3 · p. 2
- A soft retained-ratio regularizer discourages both lazy over-segmentation and excessive compression that could cause training collapse.The regularizer is combined with continuation loss to define the reward.Source: §3.3 · pp. 2–3
- GRPO trains the boundary policy from relative advantages, while cross-entropy separately trains the pooler, projection layer, and decoder adapters.Segmentations with above-average returns receive positive advantages.Source: §3.3 · p. 3
4. EXPERIMENTS
Experiments evaluate ASSET on English and Mandarin ASR, then explore emotion recognition, speaker counting, and how learned boundaries vary across tasks and speaking styles. ASSET approaches the character-boundary oracle on LibriSpeech with fewer tokens and outperforms both AISHELL oracles at a rate comparable to phone boundaries.
4.1. Experimental setup
- The 960h systems use frozen WavLM-Large features at approximately 50 Hz with Llama-3.2-1B-Instruct; training includes a character-boundary cold start, pooler/decoder adaptation, then GRPO segmenter updates.The cold-start and adaptation stages precede the final joint stage, which continues decoder fine-tuning while updating the segmenter.Source: §4.1 · p. 3
4.2. ASR results
- The 100h WER sweep varies fixed strides and policy models, including Transformer and BiGRU configurations.The figure’s left panel compares fixed strides and models; its right panel details the policy and pooler.Source: §4.2 · p. 3, Figure 2 · p. 3
- 2.79/5.63% WER at 10.9/10.6 Hz: the 960h Transformer system approaches the character-boundary oracle with fewer tokens and uses 8–9% fewer tokens than the CNN system at comparable WER.It improves on Phone-CTC by 1.25/1.98 WER points at similar rates.Source: §4.2 · p. 3, Table 1 · p. 4
- 7.53% CER at 7.7 tokens/s: on AISHELL-1, ASSET outperforms both oracles at a token rate comparable to phones.Phone alignment produces a higher token rate than character alignment, 7.1 versus 3.2.Source: Table 3 · p. 4, §4.2 · p. 4
- 2.8×/2.7× higher batched throughput: CNN/Transformer ASSET outperform no downsampling at their largest stable tested batches, 32 versus 16.The full-path RTF measurements show decoder savings outweigh segmentation and pooling overhead.Source: §4.2 · p. 4
- 0.14/0.41 WER improvement: GRPO updates improve over a frozen Transformer segmenter while using about 3 fewer tokens/s.Both arms start from the same Transformer policy and train the BiGRU pooler and decoder; the segmenter is frozen in one arm and updated in the other.Source: §4.2 · p. 4
4.3. Exploratory non-ASR extensions
- For emotion recognition, the adapted system roughly halves the frozen policy’s token rate with a similar mean macro-F1.The experiments also adapt the cold-start initialization for speaker counting using a task-specific rate band.Source: §4.3 · p. 4
- Emotion and speaker-count policies retain about 3/4 of ASR boundaries entering vowels from consonants, but retain 36.5% and 2.1%, respectively, when leaving vowels for consonants.For the emotion policy, boundary agreement with the default rendition across happy, sad, and confused speech is 67.7–75.2% F1.Source: §4.3 · p. 4
5. CONCLUSION
The conclusion presents ASSET as a learned speech-compression framework and reports results across English and Mandarin ASR, with exploratory evidence that segmentation varies by task and speaking style.
5. CONCLUSION
- ASSET compresses speech LLM inputs through learned segmentation and residual recurrent pooling.The approach uses GRPO task feedback for the boundary policy and teacher-forced cross-entropy for the pooler and decoder.Source: §5 · p. 4
- On LibriSpeech-960h, ASSET achieves WER comparable to a transcript-assisted character-boundary oracle with fewer audio tokens; Mandarin experiments show similar benefits.The conclusion also reports that exploratory non-ASR segmentation varies across tasks and with how the same utterance is expressed.Source: §5 · p. 4
- Exploratory non-ASR experiments find that segmentation varies across tasks and, within one policy, with how the same utterance is expressed.Source: §5 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The study used existing speech datasets, and the institution determined that no additional ethics approval was required.