VoxServe: Streaming-Centric Serving System for Speech Language Models

Keisuke Kamahori; Wei-Tzu Lee; Atindra Jha; Rohan Kadekodi; Stephanie Wang; Arvind Krishnamurthy; Baris Kasikci

Source-linked AI summary

VoxServe: Streaming-Centric Serving System for Speech Language Models

Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha, Rohan Kadekodi, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

arXiv:2602.00269v1Jan 30, 2026cs.LGcs.AIcs.DCcs.SDeess.AS

TL;DR

Streaming SpeechLM deployment needs low latency, high throughput, and support for diverse architectures, but existing serving systems are often architecture-specific. VoxServe decouples model execution from system optimizations and adds streaming-aware scheduling and asynchronous execution, achieving 10–20× higher throughput at comparable latency while maintaining uninterrupted streaming.

Problem
SpeechLM serving lacks standardized frameworks for diverse, multi-stage architectures, forcing developers to reimplement core optimizations for each model variant.
Method
VoxServe decouples model-specific architecture from system optimizations through a unified execution interface, streaming-aware scheduling, and an asynchronous inference pipeline.
Results
10–20× higher serving throughput at comparable latency was achieved across multiple SpeechLMs while maintaining uninterrupted audio streaming.
Takeaways & Limitations
VoxServe provides a single framework for efficiently serving diverse SpeechLM architectures and supporting streaming and other deployment scenarios.

Abstract

from arXiv · show

Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.

1. Introduction

SpeechLM deployment requires low-latency, cost-efficient serving that coordinates heterogeneous multi-stage pipelines and supports uninterrupted streaming. VoxServe addresses these challenges with a unified, architecture-decoupled system, streaming optimizations, and 10–20× higher request rates than existing implementations at similar latency.

SpeechLM adoption in virtual assistants, content generation, and language access services increases demand for low-latency, cost-efficient serving systems.
SpeechLMs combine LLM backbones with audio-specific modules, creating heterogeneous multi-stage inference pipelines that require coordinated scheduling, caching, and streaming.
Existing serving implementations use fragmented, bespoke inference stacks, limiting holistic pipeline management and making it costly to switch between model families.
VoxServe unifies diverse SpeechLM architectures through a model-execution abstraction that decouples model design from system-level optimizations.The system applies model-agnostic optimizations across multiple SpeechLM families under one abstraction.
10–20× higher request rate: VoxServe serves three models at similar response latency to existing implementations while ensuring streaming viability.VoxServe also introduces optimized scheduling and an asynchronous pipeline for streaming applications.

2. Background and Motivation

SpeechLM serving is difficult because models combine heterogeneous, architecturally diverse audio and language components, while streaming requires specialized latency and continuity objectives. Existing architecture-specific stacks force repeated engineering and lack system-wide coordination, motivating a unified, efficient serving system.

Challenge 2: Optimizing for unique streaming performance metrics: Streaming speech serving must minimize TTFA while ensuring every subsequent audio chunk arrives before playback of the previous chunk ends.TTFA includes prefill, token generation, detokenization, and, when applicable, speech encoder inference; streaming viability is a binary per-chunk condition.
Challenge 1: Supporting diverse and multi-stage speech pipelines: SpeechLMs combine heterogeneous multi-stage pipelines whose audio modules, tokenization rates, and LLM data representations vary substantially.Architectures may use shared text/audio tokens, parallel codebooks, continuous audio features, or depth-wise LLMs.
Challenge 1: Supporting diverse and multi-stage speech pipelines: Architecture-specific inference engines require developers to reimplement batching, chunk-wise detokenization, and CUDA graph optimizations for each model variant.The lack of standardized SpeechLM serving frameworks couples engines to specific architectures.
Challenge 2: Optimizing for unique streaming performance metrics: Optimal scheduling depends on detokenizer invocation intervals and component cache policies, while existing LLM serving metrics do not capture streaming speech requirements.These choices are use-case dependent even for a fixed model.
Deployment gaps: SpeechLM deployment remains fragmented because bespoke libraries support individual architectures and rarely optimize concurrent streaming requests or coordinate shared hardware resources.Combining separate LLM and audio frameworks can overlook system-wide optimization, and new architectures require reimplementing serving optimizations.
Motivation: The target is a serving system that works uniformly across diverse modern SpeechLMs and provides high efficiency for multi-tenant, streaming inference.This goal addresses both architectural fragmentation and inefficient resource management.

3. Design

VoxServe uses a unified, model-agnostic execution interface to support diverse SpeechLM architectures while enabling system-wide optimizations. Its streaming-oriented design combines modular inference stages, asynchronous execution, and phase-aware scheduling.

Unified model execution: VoxServe abstracts diverse SpeechLM architectures behind a common model interface, allowing system-level optimizations and step-wise request scheduling across models.Each model-interface function represents an inference-workflow step that the Scheduler can schedule and batch.
Inference workflow: The model workflow separates preprocessing, LLM forward, sampling, and postprocessing, with interfaces supporting heterogeneous text, audio, token, mask, and feature representations.Postprocessing detokenizes generated audio into waveform chunks and supports batch inference without dynamic tensor shapes.
System architecture: The execution process comprises Scheduler, Worker, and Model components, while the interface process exposes an HTTP endpoint for request submission.The Scheduler orchestrates request lifecycles, and the Model provides a common abstraction across SpeechLMs.
Streaming generation: Chunk-based detokenization delivers generated audio incrementally, with operator-selected chunk sizes and per-request cache state for detokenizers requiring prior-chunk information.Some models also use an optional depth-forward/sampling method for autoregressive multi-codebook audio generation.
Streaming-aware scheduling: The scheduler distinguishes startup and steady-state phases, prioritizing first-chunk generation initially and later prioritizing requests according to streaming-viability risk while limiting startup concurrency.This exploits temporary slack in requests whose streaming quality is not immediately degraded by delayed inference.

4. Evaluation

VoxServe is evaluated against official serving implementations across three SpeechLMs using TTFA and streaming viability under varying request rates. It sustains 10–20× higher request rates at comparable TTFA while extending performance through optimized scheduling, asynchronous pipelining, multi-GPU deployment, and throughput-oriented scheduling.

Evaluation Setup: Evaluation covers CosyVoice 2.0, Orpheus 3B, and Step-Audio 2 against their official serving implementations on a single NVIDIA H100 GPU.TTFA and streaming viability are measured using requests from LibriTTS and VoiceBench, with Poisson-distributed arrivals over a 60-second run.
Streaming Performance: 10–20× higher request rates are sustained by VOXSERVE at comparable TTFA while maintaining high streaming viability across all three models.For CosyVoice, VOXSERVE maintains 500 ms p90 TTFA up to 4.0 req/s with 100% streaming viability, versus approximately 0.4 req/s for the baseline.
Streaming Performance: VOXSERVE avoids baseline queue buildup through system-wide scheduling and efficient detokenizer batching, including cache-state maintenance for batched Step-Audio inference.Baseline Step-Audio detokenizer batching is infeasible because of cache-management constraints, while VOXSERVE maintains cache state under batched inference.
Ablation Study: 3.5 req/s with optimized scheduling achieves comparable TTFA to only 1.5 req/s without optimization, while 2.0 req/s reduces TTFA by approximately 2.5×.Asynchronous pipelining provides additional improvements beyond optimized scheduling, particularly at higher request rates.
Distributed Inference: 16 req/s versus 4 req/s is sustained by data parallelism with four H100 GPUs under a 500ms TTFA constraint, indicating near-linear scaling in serving capacity.The data-parallel setup instantiates one scheduler process per GPU and randomly routes incoming requests among schedulers.
Throughput-Oriented Deployment: 134× realtime throughput is achieved with VOXSERVE’s optimized scheduler, compared with 53× without scheduling optimization and approximately 10× for the baseline.This throughput-oriented experiment uses CosyVoice with 1,000 concurrent LibriTTS requests and measures inverse Real-Time Factor.

5. Related Work

Prior serving systems improve text-generation efficiency and extend to multimodal models, but do not address high-throughput, real-time streaming SpeechLM generation. VoxServe fills this gap by optimizing TTFA and streaming viability across diverse SpeechLM architectures.

Modern LLM serving improves throughput and latency through KV cache management, disaggregation, and operation-level optimizations.
Recent systems extend LLM serving to multimodal models, including EPD disaggregation for dedicated resources and CornServe’s support for any-to-any modalities.
SpeechLM serving remains unaddressed for high-throughput, real-time streaming generation, including diverse architectures with stateful detokenizers, depth-wise models, and varying codebook representations.
VoxServe addresses this gap by optimizing TTFA and streaming viability with abstractions that account for SpeechLM architectural diversity.

6. Conclusion

VOXSERVE is a unified, streaming-centric serving system for modern SpeechLMs that separates model architecture from system optimizations and combines streaming-aware scheduling with asynchronous execution. Across multiple models and deployment scenarios, it achieves 10–20× higher serving throughput than model-specific implementations at comparable latency while maintaining uninterrupted audio streaming.

6. Conclusion: VOXSERVE decouples system-level optimizations from model-specific architectural details through a unified model execution interface.This enables one serving framework to support a wide range of SpeechLM designs.
6. Conclusion: VOXSERVE combines streaming-aware scheduling with an asynchronous execution pipeline to optimize TTFA and sustained streaming viability.The two mechanisms jointly target both initial responsiveness and continued streamability.
6. Conclusion: 10–20× higher serving throughput is achieved at comparable latency than existing model-specific implementations across multiple state-of-the-art SpeechLMs and deployment scenarios.The reported advantage spans multiple models and deployment settings rather than a single evaluation case.
6. Conclusion: VOXSERVE maintains uninterrupted audio streaming while delivering its throughput gains.This indicates that the system preserves streaming viability alongside improved serving efficiency.

A. Evaluation Setup Details

The evaluation follows official model inference configurations while varying chunking, sampling, conditioning, and batching across CosyVoice, Orpheus, and Step-Audio. VOXSERVE consistently achieves significantly lower p90 TTFA than the baseline for CosyVoice across datasets and request rates.

Evaluation Protocol: Experiments adopt chunking, sampling, and reference-audio conditioning schemes from each model provider’s official inference implementation.
Model Configurations: CosyVoice uses chunk size 15, fixed LJSpeech reference audio, and detokenizer inputs containing reference and newly generated audio tokens.Sampling uses temperature = 0.8, top p = 0.95, top k = 50, and repetition penalty = 1.1.
Model Configurations: Orpheus uses chunk size 28 with overlap 21, returns only each chunk’s middle portion, and conditions voice through a preset prompt voice.Sampling uses temperature = 0.6, top p = 0.8, and repetition penalty = 1.3.
Model Configurations: Step-Audio uses chunk size 25, lookahead 3 tokens, fixed repository reference waveform, and previous-iteration KV and activation caches.Sampling uses temperature = 0.7, top p = 0.9, and repetition penalty = 1.05.
Batching: CosyVoice and Orpheus use maximum batch size 128, whereas Step-Audio uses maximum batch size 32 because its KV cache consumes more memory.
Evaluation Results: VOXSERVE consistently achieves significantly lower p90 TTFA than the baseline across CosyVoice datasets and request rates, with stable trends across input statistics.The result indicates robustness to dataset-specific properties.

B. Additional Evaluation Results

Additional evaluations show that VOXSERVE generalizes across a broader range of SpeechLM architectures and remains robust to varying input data distributions, complementing the main experiments in §4.

Generalization: VOXSERVE generalizes across a broader set of SpeechLM architectures.These results complement the main experiments in §4.
Robustness: VOXSERVE remains robust under varying input data distributions.The additional evaluations assess robustness beyond the main experiments.
Evaluation scope: The additional evaluation results complement the main experiments in §4.They extend the evaluation across architectures and input data distributions.

B.1. Other Models

VoxServe is evaluated on four additional modern SpeechLMs with diverse architectures and generation characteristics, maintaining low time to first audio across increasing request rates while preserving high streaming viability.

B.1. Other Models: VoxServe evaluates Chatterbox TTS, CSM, GLM-4-Voice, and Zonos-v0.1, covering diverse SpeechLM architectures and generation characteristics.Figure 10 reports p90 and p99 TTFA under increasing request rates.
B.1. Other Models: Across all four models, VoxServe maintains low TTFA and high streaming viability over a wide operating range.Serving performance is measured by p90 and p99 TTFA as request rates increase.

B.2. Varying Input Statistics

VoxServe’s robustness to input distribution shifts is evaluated by measuring serving performance across three speech datasets.

B.2. Varying Input Statistics: Serving performance is measured across LibriTTS, Hi-Fi Multi-Speaker English TTS, and LJ Speech to evaluate robustness to input distribution shifts.The Hi-Fi Multi-Speaker English TTS dataset is attributed to Bakhturina et al. (2021), and LJ Speech to Ito & Johnson (2017).