PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

Jiaqi Li, Chaoren Wang, Xiaohai Tian, Mingjie Chen, Xinyu Liang, Xu Li, Yufan Lin, Junwen Qiu, Jun Zhang, Lu Lu, Haizhou Li, Zhizheng Wu

arXiv:2606.31247v1cs.SDeess.AS

TL;DR

Existing spoken language models use fixed frame rates, limiting their ability to adapt quality and speed to speech information density. FlexiSLM introduces dynamic, controllable frame rates for speech input and output, achieving leading speech-to-speech performance at 12.5 Hz while remaining strong at 6.25 Hz.

  • Problem

    Existing spoken language models use fixed frame rates, ignoring speech’s time-varying information density and preventing inference-time quality–speed control.

  • Method

    FlexiSLM applies dynamic frame-rate representations to speech input and output, with direct frame-rate conditioning for controllable operation.

  • Results

    At 12.5 Hz, FlexiSLM achieves the best speech-to-speech QA score among 7B SLMs and remains stronger than fixed-rate 7B baselines at 6.25 Hz.

  • Takeaways & Limitations

    FlexiSLM demonstrates a practical paradigm for efficient spoken language models with controllable quality–speed operating points.

  • Takeaways & Limitations

    FlexiSLM is not streaming, and its training data does not cover reasoning-intensive tasks, multi-turn dialogues, or many multiple-choice questions.

Abstract

from arXiv · show

Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .

1 Introduction

SLMs unify speech understanding and generation but typically use fixed-rate speech representations, limiting efficiency and inference-time quality–speed control. FlexiSLM extends dynamic frame-rate coding to spoken language modeling, motivated by the efficiency and controllability benefits demonstrated by prior codecs.

  • Background: SLMs unify speech-to-speech dialogue, ASR, TTS, and audio understanding through joint text–speech modeling with an LLM backbone.Representative applications include speech-to-speech dialogue, automatic speech recognition, text-to-speech, and audio understanding.
  • Prior work: FlexiCodec uses frame merging to achieve strong audio reconstruction quality at an average of 6.25 Hz while allowing inference-time average frame-rate steering.The dynamic frame-rate codec adaptively compresses speech based on information density.
  • Contribution: FlexiCodec was validated only in a 0.3B-parameter TTS pipeline, motivating extension of dynamic-rate coding to more compute-intensive end-to-end SLMs.The paper presents Flexible Spoken Language Model (FlexiSLM) as the first spoken language model with dynamic and controllable frame rates.

2 Related Work

Prior work spans fixed-rate acoustic and semantic speech tokenization, newer efficient and dynamic-rate representations, and end-to-end spoken language models that extend text LLMs to speech. SLMs commonly use a pretrained speech encoder to condition a decoder-only LLM in a multi-stage design.

  • Speech Tokenization: Early neural audio codecs use residual vector quantization to produce acoustic tokens at fixed rates such as 50 Hz or 75 Hz, prioritizing reconstruction fidelity.The passage identifies SoundStream and EnCodec as representative codecs.
  • Speech Tokenization: Semantic tokens from self-supervised models such as HuBERT capture linguistic content and are increasingly used in speech language modeling.The cited examples include HuBERT and several subsequent speech-language-modeling studies.
  • Speech Tokenization: Efficient speech representations include single-codebook approaches at 50–75 Hz and semantic-enhanced codecs at 12.5–50 Hz.These approaches aim to maintain high audio quality while reducing representation cost.
  • Speech Tokenization: Recent tokenization work explores dynamic frame rates by leveraging speech’s temporal sparsity to obtain lower average frame rates.The supplied passage continues with sequences at 5 Hz or 6.25 Hz.
  • Spoken Language Models: Spoken language models are general-purpose speech systems that follow natural-language instructions across diverse speech tasks and directly comprehend and generate speech.The passage characterizes SLMs as extensions of text-based LLMs to speech.
  • Spoken Language Models: A dominant end-to-end SLM design uses a pretrained speech encoder to extract acoustic features that condition a decoder-only LLM, followed by an additional transformer module.The supplied passage describes this as a three-stage paradigm.

3 Method

FlexiSLM combines pretrained speech encoding, adaptive frame merging, a Thinker–Talker architecture, and dynamic-frame-rate speech synthesis. Direct frame-rate conditioning enables predictable inference-time control, while staged training supports speech generation, perception, and controllable rates.

  • Architecture: FlexiSLM uses a pretrained Qwen2.5-Omni audio encoder, frame merging, Thinker–Talker components, and FlexiCodec-based speech decoding.The audio encoder extracts 25 Hz continuous features; FlexiCodec semantic tokens are paired with frame-length attributes for reconstruction.
  • Adaptive Frame Merging: Frame merging greedily averages adjacent frames whose cosine similarity exceeds threshold τ, retaining each group’s original frame length.The merged representations are produced through interleaving and a lightweight Transformer with local attention.
  • Frame-Rate Control: Direct conditioning on the target average frame rate replaces indirect threshold control, which otherwise yields variable rates and unpredictable speedups across utterances and datasets.During training, empirical rates from randomly sampled thresholds are used as conditioning signals; at inference, users specify the desired rate.
  • Training: Training uses three stages: Talker pre-training, multi-task LoRA fine-tuning, and full fine-tuning with the Talker-to-Thinker connection enabled.Stage 1 uses about 100K English training hours, while Stage 2 activates the input-side merger and trains on mixed speech-processing tasks.
  • Training Objective: The weighted loss combines text, speech-code, and speech-frame-length cross-entropies, with λtext = 2 and λspeech = λspeech_len = 1.For non-speech sequences, the speech-code and frame-length losses are set to 0.

4 Experiments

Experiments show that FlexiSLM outperforms fixed-rate 7B baselines at 12.5 Hz and remains competitive at 6.25 Hz, while supporting accurate frame-rate control and substantial efficiency tradeoffs. Ablations indicate that dynamic output and input frame rates improve generation quality and speech understanding, respectively.

  • Main results: At 12.5/12.5 Hz, FlexiSLM-Stage3 scores 72.4/67.2 overall s2t/s2s, surpassing Qwen2.5-Omni-7B by 5.7/3.9 points.It also exceeds Kimi-Audio-7B (69.7/57.2) and Mimo-Audio-7B (70.6/59.0).
  • Main results: At 6.25/6.25 Hz, FlexiSLM reaches 70.2/64.3 overall s2t/s2s, remaining above all 7B baselines on s2s and competitive on s2t.Reducing only output to 6.25 Hz preserves scores at 72.3/66.2 while using half as many output speech tokens.
  • Main results: At 5.0/5.0 Hz, overall s2t/s2s falls from 70.2/64.3 to 69.0/60.4, and at 4.0/4.0 Hz it falls further to 67.2.These operating points demonstrate controllability at very low frame rates, although quality declines more noticeably.
  • Frame-rate controllability: Direct frame-rate control is more accurate than merging-threshold control, whose compression criterion produces high per-utterance variability.FlexiSLM adopts direct control to reduce generation ambiguity and make compute budgets more predictable for deployment.
  • Inference efficiency: Varying input and output rates moves FlexiSLM from 1.17 to 0.57 RTF without retraining, while reducing output from 12.5 to 6.25 Hz lowers compute from 4.57 to 3.41 TFLOPs.Reducing input from 12.5 to 6.25 Hz yields only a modest RTF improvement from 1.17 to 1.09.
  • Ablations: Replacing dynamic input merging with uniform merging reduces s2t Overall AVG from 68.7 to 67.5 and worsens ASR test-other WER from 7.20 to 7.97.The ablations evaluate all variants at 6.25 Hz input and output, and dynamic output frame rates are reported to improve generation quality at 6.25 Hz.

5 Additional Materials

The appendix provides visualizations, data and training details, additional evaluations, implementation information, broader discussion, and documentation of artifact and AI-tool use.

  • Supplementary analyses: The appendices visualize FlexiSLM’s audio outputs at different target frame rates and document data construction, hyperparameters, and prompt templates.Appendix B covers output visualizations; Appendices A, C, and J cover data, training, and prompts.
  • Supplementary analyses: Additional evaluations report speech-generation WER, ablations, audio-understanding results, and qualitative examples.These materials appear in Appendices D–F and Table 13.
  • Implementation and discussion: Implementation details and broader discussion cover the audio decoder, tokenizer, future directions, artifact documentation, and AI-tool use.Appendices G–I and L–M provide these materials.

6 Conclusion

FlexiSLM is presented as the first spoken language model with dynamic and controllable frame rates for both speech input and output. It operates from 12.5 to 4.0 Hz without retraining and achieves strong speech-to-speech performance among 7B models.

  • Contribution: FlexiSLM is, to the authors’ knowledge, the first spoken language model with dynamic and controllable frame rates.It applies dynamic frame rates to both speech input and output.
  • Frame-rate controllability: Direct frame rate conditioning enables one FlexiSLM model to operate from 12.5 down to 4.0 Hz without retraining.This conditioning supports controllable frame rates within a single model.
  • Results: At 12.5 Hz, FlexiSLM achieves the best speech-to-speech QA score among 7B spoken language models.The comparison is among 7B SLMs.
  • Results: At 6.25 Hz, FlexiSLM remains stronger in speech-to-speech performance than fixed-rate 7B baselines.The reported comparison concerns speech-to-speech performance at the 6.25 Hz operating point.

Limitations

FlexiSLM has not yet explored post-training for improved quality and alignment, does not support streaming, and has limited training-task coverage. Key gaps include real-time conversation and reasoning-intensive, multi-turn, and multiple-choice tasks.

  • Post-training: FlexiSLM has not explored RLHF or DPO, leaving potential improvements in response quality and alignment untested.The paper identifies both post-training techniques as possible future avenues.
  • Streaming: The current model is not streaming, so causal audio-decoder adaptation is needed for real-time conversation.Streaming support is described as an important next step.
  • Training coverage: Training data does not cover reasoning-intensive tasks, multi-turn dialogues, or many multiple-choice tasks.This limits the scope of tasks represented during training.

A Construction of FlexiSLM-Data · B Visualizations

FlexiSLM-Data is a large-scale, filtered single-turn speech-to-speech dataset built from collected prompts and synthesized speech. Visualizations show that FlexiSLM produces fewer, longer-duration tokens at lower target frame rates while preserving word-level alignment.

  • A Construction of FlexiSLM-Data: FlexiSLM training uses a task mixture emphasizing speech-to-speech dialogue, mainly FlexiSLM-Data, alongside TTS, ASR, audio understanding, and text-to-text dialogue.The listed sources include Emilia and MLS for TTS, MLS and LibriSpeech for ASR, LLaSO-Instruct for audio understanding, and TriviaQA for text-to-text dialogue.
  • A Construction of FlexiSLM-Data: FlexiSLM-Data is constructed as a large-scale speech-to-speech dialogue dataset and is currently limited to single-turn interactions.The pipeline begins by collecting text prompts from public QA, instruction-following, and dialogue datasets.
  • A Construction of FlexiSLM-Data: Prompt collection uses the first-turn user utterance from multi-turn datasets as the prompt, skipping samples whose first turn is a…The supplied passage truncates the condition after “a…”.
  • A Construction of FlexiSLM-Data: Responses are synthesized with Qwen3-TTS using the fixed speaker “Ryan”, while speech prompts use Fish-Audio TTS3 with randomly sampled English Emilia speaker prompts.The sampled Emilia utterances are longer than 5 seconds.
  • A Construction of FlexiSLM-Data: Filtering removes code, formulas, excessive punctuation, and non-target languages, then applies correctness filtering and Whisper-medium ASR filtering for WER greater than 20%.The supplied passage names the DeepSeek-V4-Flash API for correctness filtering.
  • A Construction of FlexiSLM-Data: 1.4M speech-to-speech dialogue samples comprise 9.9K hours of speech, including 2.7K hours of user prompts and 7.2K hours of assistant responses.Sample prompt-response pairs are shown as text transcriptions in Table 9.
  • B Visualizations: Figure 4 visualizes generated mel spectrograms and aligns text, fixed-frame-rate tokens, and FlexiSLM tokens across progressively lower target rates.The visualization compares FlexiSLM with a fixed 12.5 Hz representation.
  • B Visualizations: At lower requested frame rates, FlexiSLM uses fewer and longer-duration speech tokens than the fixed 12.5 Hz representation while preserving word-level alignment.The top panel shows the generated mel spectrogram for the spoken response.

C Additional Model and Training Details

FlexiSLM’s Talker matches the Thinker LLM’s Transformer implementation, while training uses staged optimization, LoRA adaptation, warmup and cosine decay, distributed training, and specified step budgets. FlexiCodec and its SenseVoice semantic encoder together contain 450M parameters.

  • Model configuration: The Talker uses 20 Transformer layers, 8 attention heads, hidden size 1280, and intermediate size 5,120.It uses the same Transformer implementation as the Thinker LLM.
  • Model configuration: FlexiCodec and the SenseVoice semantic encoder contain 450M parameters in total.SenseVoice is identified as An et al. (2024).
  • Training configuration: AdamW is used as the optimizer for training.The passage cites Loshchilov and Hutter (2017).
  • Training configuration: The Talker, Talker-to-Thinker connection, and Frame Merging Module use learning rate 2 × 10−4 in each stage.The LLM backbone uses learning rate 2 × 10−5 in Stage 2; the supplied passage also states 1 × 10−6 but is truncated thereafter.
  • Training configuration: Each stage uses a 5% linear warmup, cosine decay to 1/10 of the maximum learning rate, and a dynamic batch size of 3K tokens per GPU.Stages 1 and 2 use Distributed Data Parallel, while Stage 3 uses DeepSpeed ZeRO-2.
  • Training configuration: Training spans 300K steps in Stage 1, 240K steps in Stage 2, and 160K steps in Stage 3, with gradient accumulation set to 1.Stage 2 comprises 3 epochs.

D Speech Generation Quality Evaluation

FlexiSLM delivers strong speech intelligibility at 12.5 Hz and remains acceptable down to 6.25 Hz, while 5.0 Hz incurs a clear WER cost. Error analysis attributes competing systems’ high WER to synthesis failures, alignment or response-completeness problems, ASR-window truncation, and verbalization artifacts.

  • Evaluation setup: Table 8 evaluates TTS and dialog speech intelligibility with WER, using E2TTS-Evalkit on LibriSpeech-PC and Whisper large-v3 transcripts on OpenAudioBench.Baselines include CosyVoice and three spoken language models.
  • Quantitative results: 4.52% dialog WER at 12.5 Hz and 4.41% at 8.0 Hz both beat Qwen2.5-Omni (6.33%) and are comparable to Qwen3-Omni (4.32%).The dialog metric compares generated text with Whisper large-v3 transcription of generated speech.
  • Quantitative results: 2.87% TTS WER and 5.83% dialog WER at 6.25 Hz remain acceptable, whereas 5.0 Hz rises to 4.16% and 9.03%, respectively.The higher 5.0 Hz error is consistent with spoken-QA degradation in Table 3.
  • Error analysis: Mimo-Audio’s high WER often reflects alignment or output-format failures, including missing speech, incomplete responses, divergence from text, unintended continuation, and language switching.These findings come from manual inspection.
  • Error analysis: Qwen2.5-Omni errors often involve responses exceeding the 30-second ASR window, incomplete transcription, number-reading errors, and word-fragmentation artifacts.The ASR-window and verbalization issues are reported as distinct sources of high WER.

E Additional Ablation Study … H Audio Tokenizer Details

FlexiSLM’s ablations identify key architectural trade-offs, while its audio understanding remains strong under frame-rate compression. The system uses a dynamic-frame-rate FlexiCodec target and a two-stage flow-matching decoder to synthesize speech.

  • E Additional Ablation Study: Removing the input merging Transformer worsens ASR WER from 2.92 to 6.45 on clean speech and from 7.20 to 12.33 on other speech.Spoken QA also drops from 68.7/63.0 to 65.7/60.1, confirming the Transformer re-aligns dynamically merged features for LLM consumption.
  • F Audio Understanding Performance: At 12.5 Hz, FlexiSLM achieves 65.8% average audio-understanding accuracy, exceeding Gemini-2.5-Pro at 48.3% and LLaSO-3B at 58.3%.The benchmark covers multiple-choice emotion, accent, vocal sound, instrument, music-source, and gender classification tasks.
  • F Audio Understanding Performance: 64.7%, 64.0%, 63.4%, and 64.1% average accuracy are retained at 8.0, 6.25, 5.0, and 4.0 Hz input frame rates, respectively.These sequence-level classification tasks can preserve global acoustic statistics despite aggressive frame merging.
  • G Flow-Matching Audio Decoder Details: The audio decoder is a two-stage pipeline consisting of a VoiceBox-style NAR flow-matching model followed by a vocoder.The flow-matching model consumes FSQ semantic tokens and frame lengths, conditions on a short speaker-reference prompt, and generates mel spectrograms.
  • H Audio Tokenizer Details: FlexiSLM’s Talker Transformer predicts FlexiCodec FSQ semantic tokens and per-token frame lengths, which the flow-matching decoder converts into the final waveform.FlexiCodec provides the dynamic-frame-rate speech-token training target.
  • H Audio Tokenizer Details: FlexiCodec creates variable-length sequences by merging adjacent 12.5 Hz ASR-feature frames whose cosine similarity exceeds threshold τ, with τ from 0.8 to 1.0 supporting average rates from 3 Hz to 12.5 Hz.At 12.5 Hz, no merging occurs; FlexiSLM uses the semantic branch while acoustic reconstruction is handled by the flow-matching decoder.

I Future Work … M Use of AI

The paper identifies future extensions spanning interactive, streaming, post-trained, broader-capability, and general-audio FlexiSLMs. It also documents prompt templates, qualitative comparisons, artifact-use practices, and limited AI assistance in writing.

  • I Future Work: Future work will extend FlexiSLM toward full-duplex interaction, using rendered speech tokens to provide coherent acoustic context across turns.The proposed direction is for systems that listen, think, and speak concurrently.
  • I Future Work: The current model is non-streaming on input and output; causal or chunk-based modules could enable real-time conversation across cloud and edge budgets.This would adapt the Frame Merging Module and audio decoder while retaining frame-rate controllability.
  • I Future Work: Future work has not explored RLHF or DPO for spoken dialogue, where reward models could capture prosody, naturalness, and turn-taking.The stated goal is to improve response quality beyond supervised distillation alone.
  • I Future Work: Expanding data to multi-turn dialogue, chain-of-thought reasoning, tool use, and general audio could broaden capabilities while preserving the dynamic-rate backbone.The proposed general-audio scope includes music, environmental sound, and mixed audio.
  • J SLM Prompt Template: Training and evaluation use a Qwen-Omni system prompt, task prompts from LLaSO-Instruct, question audio clips for S2S, and audio boundary tokens.The system prompt is prepended to every training sample.
  • K Qualitative Analysis of Model Outputs: The qualitative analysis presents representative speech-to-speech cases, performance gains, and corresponding error patterns in Table 13.Table 13 compares FlexiSLM with Qwen2.5-Omni and highlights wrong answers in red.
  • L Artifact Use and Intended Purpose: External artifacts are used consistently with their intended use and distribution terms, including permissively licensed prompt datasets and open-source TTS systems.The artifacts are used for research-context prompt collection, response generation, and speech synthesis.
  • M Use of AI: AI writing assistants were used to polish the paper’s language and presentation, not to generate ideas, design experiments, or write substantive scientific content.This disclosure distinguishes editorial assistance from research contributions.
Loading 2606.31247v1…