Source-linked AI summary
LongCat-Flash-Omni Technical Report
Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, Chen Chen, Chengxu Yang, Chengzuo Yang, Cong Han, Dandan Peng, Delian Ruan, Detai Xin, Disong Wang, Dongchao Yang, Fanfan Liu, Fengjiao Chen, Fengyu Yang, Gan Dong, Gang Huang, Gang Xu, Guanglu Wan, Guoqiang Tan, Guoqiao Yu, Haibo Qiu, Hao Lu, Hongbo Liu, Hongyu Xiang, Jiaheng Wu, Jian Yang, Jiaxing Liu, Jing Huang, Jingang Wang, Jinrui Ding, Juchao Jiang, Jun Kuang, Jun Wang, Junhui Mei, Ke Ding, Kefeng Zhang, Lei Chen, Liang Shi, Limeng Qiao, Liming Zheng, Lin Ma, Liuyang Guo, Liya Ma, Luying Sun, Man Gao, Mengshen Zhu, Miao Cao, Minliang Lin, Nuo Xu, Peng Shi, Qi Zhang, Qian Fang, Qian Wang, Qian Yang, Quanxiu Wang, Rongxiang Weng, Rongxin Guo, Ruoxuan Liang, Senbin Yang, Shanbo Xu, Shanglin Lei, Shengze Ye, Shimin Chen, Shuaiqi Chen, Shujie Hu, Shuo Li, Siqi Yang, Siyu Xu, Siyu Ren, Song Li, Songxiang Liu, Tianhao Bai, Tianye Dai, Wei Hong, Wei Wang, Weixiao Zhao, Wengang Cao, Wenlong Zhu, Wenlong He, Xi Su, Xi Nan, Xiaohan Zhao, Xiaohao Wang, Xiaoyu Zhao, Xiaoyu Wang, Xiaoyu Li, Xin Pan, Xin Chen, Xiusong Sun, Xu Xiang, Xudong Xing, Xuezhi Cao, Xunliang Cai, Yang Yang, Yanli Tan, Yao Yao, Yerui Sun, Yi Chen, Yifan Lu, Yin Gong, Yining Zhang, Yitian Chen, Yiyang Gan, Yuchen Tang, Yuchen Xie, Yueqian Wang, Yuewen Zheng, Yufei Zhang, Yufeng Zhong, Yulei Qian, Yuqi Peng, Yuqian Li, Yuwei Jiang, Zeyang Hu, Zheng Zhang, Zhengkun Tian, Zhiqing Hong, Zhixiong Zeng, Zhuqi Mi, Ziran Li, Ziwen Wang, Ziyi Zhao, Ziyuan Zhuang, Zizhe Zhao
TL;DR
LongCat-Flash-Omni addresses the challenge of combining strong offline multimodal understanding with real-time audio-visual interaction across heterogeneous modalities. It uses progressive multimodal training, efficient perception and speech modules, a sparse LongCat-Flash backbone, and modality-decoupled training infrastructure. The model reaches state-of-the-art open-source omni-modal benchmark performance while remaining competitive across unimodal tasks and supporting low-latency interaction.
Problem
Training one model for strong offline multimodal understanding and real-time audio-visual interaction is difficult because modalities differ substantially and streaming requires distinct capabilities.
Method
The model combines progressive multimodal training with efficient modality encoders, speech reconstruction, a sparse LongCat-Flash backbone, and modality-decoupled parallelism.
Results
LongCat-Flash-Omni achieves state-of-the-art performance on open-source omni-modal benchmarks and matches or exceeds closed-source systems on key unimodal tasks.
Takeaways & Limitations
The unified open-source model supports multimodal understanding and generation with low-latency audio-visual interaction across text, image, video, and audio.
Takeaways & Limitations
Vision comparisons exclude some models’ unavailable modalities, and video evaluation reports only VideoMME with audio for Gemini models under a 32K-token context limit.
Abstract
from arXiv · showhide
We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.
1 Introduction
LongCat-Flash-Omni addresses the challenge of combining strong offline multimodal understanding with low-latency real-time audio-visual interaction. It uses progressive early-fusion training, interaction-focused data, efficient architecture, and modality-decoupled training to achieve broad multimodal performance.
- Challenges: The model addresses cross-modal heterogeneity, offline-versus-streaming integration, low-latency interaction, and large-scale multimodal training efficiency.These challenges arise from modality differences, distinct streaming requirements, and heterogeneous data and model workloads.
- Training: Progressive early-fusion pretraining incorporates audio and visual data into a text foundation model using balanced multimodal mixtures while preserving unimodal performance.The approach progressively adds modalities and aligns their representations within a unified sequence-modeling framework.
- Data: Human-in-the-loop interaction data and vision-speech question-answering data support long-term memory, multi-turn dialogue, and transfer of offline multimodal understanding to interactive scenarios.The data construction targets natural speech output and interactive use cases.
- Efficiency: The system combines a Shortcut-connected MoE backbone, lightweight modality encoders and decoder, synchronized chunk-wise processing, and modality-decoupled parallelism for efficient real-time interaction and training.The training system sustains more than 90% of the throughput achieved during pure text training.
- Results: LongCat-Flash-Omni achieves state-of-the-art results on omni-modal benchmarks while delivering competitive performance across text, image, video, speech understanding, and speech generation tasks.Subjective evaluations also confirm high-quality, low-latency audio-visual interaction.
2 Architecture
LongCat-Flash-Omni unifies text, image, video, and audio understanding and generation in an end-to-end architecture built for efficient streaming interaction. Lightweight modality encoders, a sparse LongCat-Flash backbone, and specialized audio components support multimodal processing and speech generation.
- Unified multimodal architecture: The end-to-end model accepts text, audio, images, videos, or arbitrary combinations, then generates text and speech tokens through one LLM framework.Vision and audio features are projected into a shared token space, while an audio decoder reconstructs waveforms from generated speech tokens.
- Efficient multimodal components: Lightweight vision, audio, and audio-decoder modules support efficient streaming around the LongCat-Flash LLM backbone.Each modality component has approximately 600M parameters, while the backbone retains LongCat-Flash’s sparse Mixture-of-Experts design with about 27B parameters activated on average.
- Vision encoder: LongCat-ViT combines native-resolution image and video encoding with Transformer refinements for varied visual inputs.Its design includes unified patchification, 2D-RoPE, SwiGLU, RMSNorm, LayerScale, Query-Key normalization, and a vision-language projector with 2× pixel unshuffle.
- Vision encoder: Progressive visual training moves from fixed low-resolution pretraining to native-resolution fine-tuning and introduces video data later to reduce computational overhead.Feature distillation from a frozen vision model supports early convergence, and contrastive pretraining uses 14.6 billion samples.
- Audio components: The audio stack combines four-codebook speech tokenization, streaming audio encoding, and low-latency waveform reconstruction.The tokenizer operates at 16.67 Hz, with one codebook for semantic information and three for acoustic details; the encoder reduces sequence length by eight through frame splicing.
- Streaming audio-visual interaction: Audio and visual streams are independently encoded, time-aligned, chunked, and interleaved before entering the LLM for real-time audio-visual understanding.Dynamic video sampling uses a default rate of 2 FPS, with adjustments based on video duration and a frame-count constraint.
3 Pre-Training
Pre-training combines diverse multimodal data with a curriculum-inspired progression from text to speech, images, video, and long-context modeling. Stage-1 jointly trains text, speech, and ASR objectives to align modalities while preserving text capability.
- 3.1 Data Curation: A 2.5-trillion-token corpus spans audio, image-text, OCR, grounding, STEM, multi-image, video, and long-context multimodal data.
- 3.1 Data Curation: Speech and image-text datasets use filtering, alignment, clustering, and diversity sampling to improve data quality and coverage.
- 3.2 Training Strategy: The curriculum progresses from text pretraining to speech, image-text, video, expanded 128K context, and later audio refinement as sequence complexity increases.
- 3.2.2 Stage-1 Text-Speech Continued Pre-Training: Stage-1 mixes text, speech-text interleaved, and ASR data, jointly optimizing text prediction, speech-text alignment, and speech perception.
- 3.2.2 Stage-1 Text-Speech Continued Pre-Training: The Stage-1 loss weights are a = 1.75, b = 0.25, c = 1.5, and d = 0.1, selected to balance preserved text capabilities with improved audio performance.
4 Post-Training
Post-training combines supervised fine-tuning and preference optimization to improve instruction following, multimodal reasoning, and interactive spoken behavior. Its data spans image, video, audio, vision-speech, and audio-visual interaction, while joint DPO optimizes text and speech outputs together.
- 4 Post-Training: Post-training uses SFT and DPO to produce instruction-following, multimodal reasoning, and interactive capabilities with semantically accurate and coherent responses.
- 4.1 Supervised Fine-Tuning: The curated SFT data covers image-text, video-text, audio understanding, vision-speech QA, and synchronized audio-visual understanding tasks.
- 4.1 Supervised Fine-Tuning: Specialized speech-to-speech and audio-visual interaction datasets target natural voice dialogue and multi-turn, context-dependent conversations.
- 4.1 Supervised Fine-Tuning: Human verification refines generated interaction data by correcting factual inconsistency, insufficient responses, ambiguity, linguistic infelicity, and irrelevance.
- 4.2 Reinforcement Learning: Joint DPO optimizes the text head and multiple audio heads simultaneously to improve cross-modal alignment and stability.
5 Training Infrastructures
The training infrastructure addresses multimodal data and model heterogeneity through modality-decoupled parallelism, chunked communication, and targeted efficiency optimizations. The resulting system maintains over 90% of text-only training throughput while reducing memory pressure and preserving numerical consistency.
- 5 Training Infrastructures: Over 90% of text-only training throughput is maintained in multimodal settings through independently optimized modality components and deterministic execution.The system decouples the LLM, vision encoder, and audio encoder while enforcing reproducibility and numerical consistency.
- 5.1 Multimodal Decoupling Framework: Modality-decoupled parallelism separates parallel strategies for heterogeneous encoders and the LLM, addressing differences in token distributions and computational workloads.The approach builds on prior heterogeneity-aware scheduling while independently configuring modality encoders and the LLM decoder.
- 5.1.1 Modality-Decoupled Parallelism: MDP co-locates modality encoders with the LLM decoder while combining HSDP, PP, DP, CP, EP, and InnerDP to map modality data across microbatches.InnerDP aligns modality-encoder data partitions with the decoder’s distributed dimensions.
- 5.1.1 Modality-Decoupled Parallelism: The four-phase MDP schedule loads and sorts microbatches, computes modality embeddings, runs LLM forward and backward passes, and backpropagates through modality encoders.Sorting by text sequence length reduces workload imbalance, while the ModalityBridge transfers embeddings and gradients between parallel layouts.
- 5.1.2 Chunk-based ModalityBridge: Chunk-based ModalityBridge processing reduces peak memory usage to 1/num_chunk of the original while maintaining bitwise numerical alignment.Chunked aggregation, hidden-size partitioning, and redistribution lower memory peaks during embedding and gradient transfers.
6 Inference and Deployment
Inference separates modality-specific processing from the LLM and uses an asynchronous streaming pipeline for low-latency interaction. Incremental inference, adaptive batching, speculative switching, and interruption handling reduce response delay while supporting real-time audio-visual input.
- 6 Inference and Deployment: The deployment framework places modality encoders and decoders on dedicated hardware while separating them from the LLM to reduce cross-modal resource contention.LLM deployment also uses Prefill-Decode disaggregation and Single Batch Overlap for ScMoE efficiency.
- 6 Inference and Deployment: The asynchronous pipeline concurrently executes VAD and frame sampling, audio-visual encoding, LLM prefilling and decoding, and audio decoding.Incremental inference and adaptive batching enable concurrent scheduling across streaming stages.
- 6 Inference and Deployment: Sparse-dense sampling uses dense video frames during speech and sparse frames otherwise to balance detailed analysis with high-level stream tracking.VAD determines whether audio and densely sampled frames should be supplied to the model.
- 6 Inference and Deployment: Speculative prefill-decode switching overlaps early decoding with silent-span detection, while delivery waits for confirmed turn completion and stops immediately on interruption.Interrupted output is truncated at a nearby natural breakpoint such as punctuation.
- 6 Inference and Deployment: Users can receive a model response within 100ms after endpoint detection by overlapping 600–700ms VAD detection with streaming prefill.Each request packet contains 1 second of audio and two corresponding video frames, which can be pref-filled immediately.
7 Evaluation
The evaluation compares LongCat-Flash-Omni with proprietary and open-source multimodal models across vision, audio, text, cross-modality, and audio-visual interaction capabilities.
- 7 Evaluation: The evaluation spans vision understanding, audio comprehension, text understanding and generation, cross-modality understanding, and audio-visual interaction.Comparisons include proprietary and open-source multimodal models.
7.1 Vision Capability Evaluation
LongCat-Flash-Omni is evaluated across image and video understanding benchmarks under specified comparison settings. It performs competitively on image understanding and achieves state-of-the-art video-to-text results, particularly on short-video tasks.
- 7.1.1 Image Understanding Evaluation: Image evaluation covers general domains, STEM and reasoning, multi-image understanding, and text recognition and document understanding.The benchmark suite includes MMBench, MathVista, BLINK, ChartQA, DocVQA, OCRBench, and OmniDocBench.
- 7.1.1 Image Understanding Evaluation: LongCat-Flash-Omni performs comparably to Gemini-2.5-Flash and outperforms open-source Qwen3-Omni on image understanding benchmarks.Its advantage is particularly pronounced on multi-image tasks, supported by interleaved image-text, multi-image, and video training data.
- 7.1.2 Video-to-Text Evaluation: Video evaluation covers short-video, long-video, and STEM-and-reasoning dimensions, with audio-visual understanding tested on VideoMME with and without audio.Video benchmarks use a maximum context length of 32K tokens for consistent comparison.
- 7.1.2 Video-to-Text Evaluation: LongCat-Flash-Omni achieves state-of-the-art performance on video-to-text tasks and surpasses all compared models by a significant margin on short-video understanding.On long-video tasks, it performs on par with leading models and achieves the best result among omni-modal models on VideoMME.
7.2 Audio Capability Evaluation
The evaluation examines speech recognition, text-to-speech, and speech continuation across staged base models, using error-rate, reconstruction, and continuation assessments.
- Speech recognition and synthesis: ASR evaluation covers Chinese and English speech recognition using SpeechIO and LibriSpeech benchmarks.The broader audio-capability evaluation also includes TTS, where generated speech tokens are reconstructed into waveforms and assessed with WER or CER.
- Evaluation scope: The audio-capability assessment systematically covers ASR, TTS, and speech continuation for base models from stages 1 through 4.These stages are evaluated to track audio capability during pre-training.
Speech Recognition and Translation
LongCat-Flash-Omni is evaluated on speech recognition, translation, broad audio understanding, and audio-driven text chat, showing strong results across these capabilities.
- Speech recognition and translation: LongCat-Flash-Omni consistently outperforms competing models on ASR benchmarks spanning multiple languages and datasets.The comparison includes Gemini-2.5-Pro, GPT-4o-Audio, Qwen3-Omni-Instruct, and Kimi-Audio.
- Speech recognition and translation: Together, the speech recognition and translation results indicate robust and comprehensive fundamental speech understanding.
- Audio understanding: LongCat-Flash-Omni outperforms most competing models across audio-understanding dimensions and reaches state-of-the-art performance on several benchmarks.The evaluation covers music, sound events, and speech understanding across MMAU, VocalSound, TUT2017, ClothoAQA, Nonspeech7k, CochlScene, and MELD.
- Audio-to-text chat: Audio-to-text chat evaluation reports strong performance across OpenAudioBench and VoiceBench subsets, with state-of-the-art results in several cases.The benchmarks test world knowledge, domain understanding, instruction following, and reasoning from audio instructions.
7.3 Text Capability Evaluation
Text evaluations test base and instruct models across general, reasoning, instruction-following, mathematical, and coding benchmarks, finding preserved or leading text capability after multimodal training.
- Base model evaluation: The base model performs on par with state-of-the-art base models despite its compact active/total parameter size.The evaluation reports no degradation in text capabilities after extensive multimodal training.
- Instruct model evaluation: The instruct-model evaluation spans general domains, instruction following, mathematical reasoning, general reasoning, and coding.Benchmarks include MMLU-family tasks, IFEval, mathematical datasets, reasoning tasks, and HumanEval+/MBPP+/LiveCodeBench.
- Instruct model evaluation: LongCat-Flash-Omni maintains superior text capability with consistently leading performance across evaluated domains.Against representative frontier chat models, it also shows no degradation relative to LongCat-Flash and improves in certain domains.
7.4 Cross-modality Evaluation
Cross-modality evaluation covers benchmarked understanding and end-to-end real-time audio-visual interaction, showing strong open-source performance and a substantial advantage over open-source interaction alternatives.
- Cross-modality understanding: The cross-modality comparison includes Gemini, Qwen, and other strong multimodal baselines, with some closed models evaluated through live applications.
- Cross-modality understanding: LongCat-Flash-Omni outperforms Gemini-2.5-Flash-non-thinking and matches Gemini-2.5-Pro-ThinkingBudget128 on cross-modality evaluation.It performs especially strongly on WorldSense, DailyOmni, and UNO-Bench, where it surpasses other open-source omni-modal models.
- Real-time audio-visual interaction evaluation: The evaluation addresses a gap between existing cross-modal benchmarks and real-world audio-visual interaction, where audio output is essential.The authors state that prior work had not systematically evaluated this form of real-time multimodal interaction.
- Real-time audio-visual interaction evaluation: The real-time interaction framework combines 200 three-minute dialogue sessions per model with ratings from 250 users and qualitative expert analysis.Sessions cover problem solving, entertainment, self-improvement, and emotional support.
- Real-time audio-visual interaction evaluation: LongCat-Flash-Omni scores 0.56 points above Qwen3-Omni, the current open-source state-of-the-art model, in naturalness and fluency.It ranks third overall, behind Doubao and GPT-4o, among the evaluated audio-visual interaction products.
- Real-time audio-visual interaction evaluation: Qualitative analysis finds strong paralinguistic understanding, relevance, and memory, including interpretation of emotions from facial and vocal cues.The qualitative results are presented in Table 16 alongside case studies in Figure 12.
8 Conclusion
LongCat-Flash-Omni unifies offline multimodal understanding with real-time audio-visual interaction across text, audio, image, and video. Evaluations report state-of-the-art omni-modal performance and competitive unimodal capabilities, while future work targets broader data, adaptive reasoning, and richer interaction.
- The model unifies robust offline multimodal understanding with real-time audio-visual interaction in one framework.
- A multi-stage early-fusion pretraining pipeline develops integrated multimodal representations while preserving unimodal strength.
- State-of-the-art performance on Omni-Bench and WorldSense accompanies matching or exceeding closed-source systems on key image, video, and audio tasks.
- Future work will expand training data, add adaptive thinking, refine streaming and generation, and explore embodied interactive intelligence.
9 Contributions
The contribution section supplies author-list metadata rather than technical contributions.
- Authors are listed alphabetically, and an asterisk marks people who have left the team.