Source-linked AI summary
vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu
TL;DR
Any-to-any multimodal models combine heterogeneous stages that existing single-paradigm serving systems cannot natively express or efficiently coordinate. vLLM-Omni models these pipelines as stage graphs and independently serves their components with configurable resources and unified data transfer. It reduces Qwen3-Omni job completion time by up to 91.4% versus the baseline, while its scope includes an explicit limitation on supported generation stages and modality combinations.
Problem
Existing serving frameworks are specialized for single AR-decoding or DiT-denoising paradigms and cannot natively express heterogeneous multi-stage any-to-any pipelines.
Method
vLLM-Omni decomposes any-to-any architectures into stage graphs and independently serves stages with per-stage batching, flexible resources, and unified connectors.
Results
91.4%: vLLM-Omni reduces Qwen3-Omni job completion time by up to 91.4% compared with the baseline method.
Takeaways & Limitations
The system provides efficient serving support for diverse any-to-any models by independently optimizing and executing their stages.
Takeaways & Limitations
The diffusion engine is described as supporting diffusion generation stages, while not all any-to-any models support every input and output modality.
Abstract
from arXiv · showhide
Any-to-any multimodal models that jointly handle text, images, video, and audio represent a significant advance in multimodal AI. However, their complex architectures (typically combining multiple autoregressive LLMs, diffusion transformers, and other specialized components) pose substantial challenges for efficient model serving. Existing serving systems are mainly tailored to a single paradigm, such as autoregressive LLMs for text generation or diffusion transformers for visual generation. They lack support for any-to-any pipelines that involve multiple interconnected model components. As a result, developers must manually handle cross-stage interactions, leading to huge performance degradation. We present vLLM-Omni, a fully disaggregated serving system for any-to-any models. vLLM-Omni features a novel stage abstraction that enables users to decompose complex any-to-any architectures into interconnected stages represented as a graph, and a disaggregated stage execution backend that optimizes resource utilization and throughput across stages. Each stage is independently served by an LLM or diffusion engine with per-stage request batching, flexible GPU allocation, and unified inter-stage connectors for data routing. Experimental results demonstrate that vLLM-Omni reduces job completion time (JCT) by up to 91.4% compared to baseline methods. The code is public available at https://github.com/vllm-project/vllm-omni.
1. Introduction
Any-to-any multimodal models unify understanding and generation across text, images, video, and audio, but their heterogeneous multi-stage architectures challenge existing serving systems. vLLM-Omni addresses this gap with stage-graph modeling and disaggregated execution, reducing Qwen3-Omni job completion time by up to 91.4% versus the baseline.
- Any-to-any models extend multimodal understanding and generation across text, images, video, and audio through unified architectures.
- Modern pipelines may combine multiple autoregressive LLMs and diffusion transformers for audio and visual generation.
- Existing LLM and diffusion serving frameworks lack native support for interconnected pipelines containing heterogeneous model components.
- vLLM-Omni represents complex architectures as stage graphs whose nodes are model stages and whose edges transform and route intermediate data.
- The system combines stagewise execution, unified data transfer, and diffusion support, and experiments report consistent gains across diverse models and tasks.
- 91.4%: vLLM-Omni reduces Qwen3-Omni job completion time by up to this amount compared with the baseline method.
2. Background & Motivation
Any-to-any models use diverse architectures, including multiple autoregressive decoders and modular combinations of AR models with specialized generators. Existing step-centric serving abstractions cannot express or efficiently execute these heterogeneous multi-stage pipelines, forcing manual orchestration and limiting optimization.
- Any-to-Any Multimodal Models: Any-to-any models process and generate multiple modalities, although individual models need not support every input and output modality.
- Multiple AR LLM decoders: Qwen-Omni exemplifies a Thinker–Talker pipeline with separate autoregressive components for text and audio generation.
- AR with specialized generators: Other models combine autoregressive semantic generation with diffusion transformers, lightweight decoders, or modality-specific experts for synthesis.
- Challenges to Existing LLM Serving Frameworks: Step-centric LLM serving frameworks encapsulate iterative decoding and KV-cache management for single-pass text generation.
- Challenges to Existing LLM Serving Frameworks: Their abstraction cannot represent coordinated execution and data flow across heterogeneous AR, DiT, and other stages.
- Challenges to Existing LLM Serving Frameworks: Manual orchestration and co-located execution prevent continuous batching, chunked prefill, fine-grained resource allocation, and dynamic pipeline adjustment.
3. vLLM-Omni Designs
vLLM-Omni decomposes any-to-any models into independently served stages connected through a unified data-transfer layer. Its orchestrator, per-stage engines, configurable resources, streaming, and diffusion support enable disaggregated execution across heterogeneous pipelines.
- Architecture: The backend orchestrates independent stage engines, enabling stage scaling, resource allocation, and intra-stage request batching.
- Stage Abstraction: vLLM-Omni exposes a stage-graph interface in which nodes are model stages and edges are stage-transfer functions.
- Stage Abstraction: Qwen2.5-Omni can be implemented as Thinker, Talker, and DiT Vocoder stages connected by preprocessing and transfer functions.
- Stage Execution: Each stage receives configurable accelerator resources and runtime settings, allowing larger or more compute-intensive components to be provisioned differently.
- Stage Execution: vLLM-Omni supports AR stages through vLLM and integrates a dedicated diffusion engine with denoising, caching, and parallelization optimizations.
- Stage Execution: Streaming stage output lets downstream components begin processing before preceding stages complete their full outputs.
- Data Transfer: A unified connector transfers embeddings, hidden states, and audio or image tensors across stages, including distributed deployments.
4. Experimental Evaluation
The evaluation covers multimodal autoregressive, audio, and diffusion-based models on dedicated testbeds. Across these settings, vLLM-Omni improves end-to-end completion time, throughput, and execution efficiency over baselines.
- Models and settings: Experiments evaluate Thinker–Talker, BAGEL, Mimo-Audio, and DiT-based models across audio, image, video, text-to-speech, and generation tasks.The testbed includes Qwen-Omni models, Vbench, SeedTTS, and Diffusers comparisons.
- End-to-end performance: Qwen2.5-Omni sees 61.4% lower RTF and 61.6% lower JCT than the baseline Transformers implementation.These results use the first 100 queries from each of the evaluated input datasets.
- End-to-end performance: Qwen3-Omni sees 90.7% lower RTF and 91.4% lower JCT than the baseline Transformers implementation.The same evaluation reports substantial end-to-end gains for Qwen3-Omni.
- Throughput analysis: vLLM-Omni achieves 12.97× higher Thinker TPS and 7.98× higher Talker TPS for Qwen3-Omni.For Qwen2.5-Omni, the corresponding improvements are 1.29× and 1.97×.
- Execution analysis: The Talker dominates Qwen3-Omni latency because audio generation produces more tokens and requires more decoding iterations than text generation.For video-input tasks, average output counts are 150.9 text tokens and 545.4 audio tokens.
- Micro experiments: 2.40× and 3.72× speedups are achieved for BAGEL text-to-image and image-to-image generation, respectively.JCT decreases from 23.12s to 9.64s for T2I and from 41.39s to 11.12s for I2I.
- Micro experiments: 11.58× speedup is achieved for Mimo-Audio, while the diffusion engine delivers a 1.26× overall speedup over Diffusers.Mimo-Audio reaches RTF 0.12 with graph compilation; the diffusion result spans image and video generation tasks.
- Connector analysis: The unified connector adds negligible data-transfer overhead while abstracting movement across heterogeneous deployment topologies.This supports disaggregated execution with minimal performance cost.
5. Related Work
Existing serving systems support text-oriented autoregressive or diffusion workloads but do not natively represent complex any-to-any pipelines. vLLM-Omni addresses this gap by combining autoregressive and diffusion serving under a unified system.
- Existing systems: LLM serving systems optimize autoregressive decoding, while diffusion frameworks optimize DiT denoising for image and video generation.These systems are specialized around separate generation paradigms.
- Existing systems: Current multimodal LLM serving systems primarily support multimodal inputs with text-only outputs.Their optimizations include EPD disaggregation and multimodal embedding caches.
- vLLM-Omni: vLLM-Omni unifies autoregressive and diffusion models to support complex any-to-any multimodal pipelines.The paper presents this capability as essential for deploying next-generation any-to-any models at scale.
6. Conclusion
The paper presents vLLM-Omni as a serving system for efficient deployment of any-to-any multimodal models. Its central approach decomposes these architectures into independently optimized stages, with experiments showing substantial improvements over existing approaches.
- Conclusion: vLLM-Omni decomposes complex any-to-any architectures into a stage graph whose stages can be independently optimized and executed.A disaggregated stage-execution backend provides serving support across diverse any-to-any models.