LongCat-Flash-Omni Technical Report

Introduction

• A multi-stage pipeline progressively adds audio and visual data to text pretraining, using balanced multimodal data and early fusion to integrate modalities while maintaining unimodal performance.The approach is presented as a response to cross-modal heterogeneity and the risk of degrading single-modality capability.Source: §1 · pp. 4–5
• Human-in-the-loop interaction data addresses long-term memory and multi-turn dialogue, while vision-speech question-answering data transfers offline vision understanding to natural speech interactions.The vision-speech data is derived from existing vision-text corpora.Source: §1 · p. 4
• With up to 560B parameters (27B activated), the model achieves millisecond-level response latency using ScMoE, streaming modality encoders, synchronized chunk-wise processing, and efficient audio reconstruction.Its context window reaches 128K tokens, supporting long-term memory, multi-turn dialogue, and temporal reasoning across modalities.Source: §1 · pp. 4–5
• Modality-decoupled parallelism independently optimizes performance and memory for the LLM and modality encoders, sustaining more than 90% of pure-text training throughput.The training configuration also applies memory-reduction techniques to support system stability.Source: §1 · p. 4
• LongCat-Flash-Omni achieves state-of-the-art omni-modal benchmark results among open-source models, competitive unimodal performance, and low-latency, high-quality audio-visual interaction in subjective evaluations.The reported unimodal tasks include text, image, and video understanding, as well as speech comprehension and generation.Source: §1 · pp. 4–5
• The report releases the model and describes its training methodology and data strategies to support future research and development in omni-modal intelligence.The authors also frame the model as a unified framework for offline multimodal understanding and real-time audio-visual interaction.Source: §1 · p. 5

2 Architecture

• The model processes text, audio, images, video, and arbitrary combinations end to end, generating speech tokens in the LLM and reconstructing waveforms with a lightweight decoder.A vision encoder, audio encoder, and audio decoder each have approximately 600M parameters; modules are designed for efficient streaming inference.Source: §2 · p. 5, Figure 2 · p. 6

2.1 Vision Encoder

• LongCat-ViT unifies image and video encoding at native resolutions, combining a unified patchifier and transformer refinements with progressive resolution and modality adaptation.Inputs generally retain native resolution within a 576–5832 patch range; 2× pixel-unshuffle reduces high-resolution computation. Contrastive pretraining uses 14.6 billion samples, with video added in the final stage and frozen-model feature distillation weighted less over time.Source: §2.1 · pp. 5–7

2.2 Audio Tokenizer, Encoder, and Decoder

• Speech inputs shift from four-codebook tokens in pretraining stages 1–4 to continuous encoder features from stage 5, while generated speech remains four-codebook tokens decoded to waveforms.The codec operates at 16.67 Hz, and its decoder supports streaming with a three-frame look-ahead; the streaming encoder downsamples sequences by eight and uses FSMN layers, with one-frame look-ahead only in its final six layers.Source: §2.2 · p. 7

2.3 LLM Backbone

• LongCat-Flash-Omni retains and extends LongCat-Flash’s 560-billion-parameter MoE backbone, which averages 27B activated parameters and uses MLA, shortcut-connected MoE, and zero-computation experts.The model’s per-token activation ranges from 18.6B to 31.3B parameters.Source: §2.3 · pp. 7–8

2.4 Video Strategy and Streaming Audio-Visual Interaction

• Audio and visual streams are independently encoded, time-aligned, chunked into synchronized segments, and fed to the LLM for multimodal understanding.The architecture is designed to combine offline multimodal understanding with low-latency audio-visual interaction.Source: §2.4 · p. 8

2.4.1 Video Strategy

• Video strategy combines adaptive frame sampling, textual timestamps, and hierarchical token compression to balance temporal information and computational cost.Training samples short videos more densely to ensure at least 16 frames, while long videos are uniformly sampled under a frame cap; timestamp text precedes each frame, and compression includes temporal stride-2 convolution.Source: §2.4.1 · p. 8

2.4.2 Streaming Audio-Visual Interaction

• Chunk-wise synchronized audio-visual features are interleaved into the LLM, with denser video sampling during user input and sparser sampling during model responses.Audio-visual chunks are time-aligned and prefilled to reduce response latency; video is sampled at 2 FPS in one-second chunks during input and 0.5 FPS in two-second chunks during responses.Source: §2.4.2 · p. 8

3 Pre-Training

• Pre-training covers a 2.5-trillion-token multimodal corpus spanning seven data categories, alongside a training strategy described in the section.The categories are audio, generic image-text, OCR/grounding/GUI, STEM, multi-image, video, and long-context multimodal data.Source: §3.1 · p. 9, §3 · p. 9

3.1.1 Audio Data

• Audio curation filters tens of millions of hours of speech using VAD, ASR cross-validation, forced alignment, duration-to-text checks, and segment merging.Training samples are split at punctuation and randomly mask audio or text fragments; understanding data spans captioning, QA, scene and event detection, paralinguistics, and music, with translation, pseudo-labeling, and prompt variation for augmentation.Source: §3.1.1 · p. 9

3.1.2 Generic Image-Text Data

• Image-text curation combines multi-stage cleaning, SigLIP alignment filtering, detailed re-captioning, and diversity-aware cluster and concept resampling.Interleaved data is filtered and sampled for quality and diversity; educational video scenes are converted into synchronized key-frame and refined-text sequences to support academic reasoning.Source: §3.1.2 · pp. 9–10

3.1.3 OCR, Grounding and GUI Data

• OCR data spans documents, scene text, handwriting, math, and text-centric VQA, while grounding data provides localization and region-captioning examples with normalized coordinates.GUI data covers screenshot perception, element grounding, and mobile and desktop navigation planning with observations, reasoning, summaries, and actions.Source: §3.1.3 · p. 10

3.1.4 STEM Data

• The STEM corpus contains 15 million image-text pairs in multiple-choice and open-ended formats, spanning disciplines and academic levels from K12 to university.A multi-stage filtering pipeline targets factual accuracy, ambiguity reduction, and standardized formatting.Source: §3.1.4 · p. 10

3.1.5 Multi-Image Data

• Multi-image question-answering data targets coarse- and fine-grained visual skills through selected and augmented open-source examples and tool-synthesized images.The taxonomy includes capabilities such as emotion recognition, vehicle identification, and clock-based time calculation.Source: §3.1.5 · p. 10

3.1.6 Video Data

• Video pre-training draws on public corpora and curated data for classification, grounding, detection, captioning, and question answering.The in-house components include scene-based detailed captions, temporally grounded QA enriched with challenging questions, and action-recognition QA in multiple-choice and open-ended formats.Source: §3.1.6 · p. 11

3.1.7 Long-Context Multimodal Data

• Long-context multimodal data combines filtered samples, including videos longer than three minutes, with synthesized image-text interleavings and long-video QA.The video QA pipeline links continuous clips through captions before creating temporally grounded questions, supporting long-sequence cross-modal modeling and memory retention.Source: §3.1.7 · p. 11

3.2 Training Strategy

• The curriculum moves from text pretraining through speech-text and vision-language alignment to video reasoning, longer contexts, and direct processing of continuous audio.Text and speech differ in semantic density, while images and video add spatial and temporal structure; Stage-5 addresses information loss from discrete speech tokens.Source: §3.2 · p. 12

3.2.1 Stage-0 Text Pre-Training

• Approximately 16 trillion text tokens establish the Stage-0 foundation, with reasoning data progressively increased under a constant learning rate.The procedure follows the initial LongCat-Flash phase [Meituan, 2025a].Source: §3.2.1 · p. 12

3.2.2 Stage-1 Text-Speech Continued Pre-Training

• Stage-1 jointly trains text prediction, text-speech interleaving, and ASR tasks, using four-codebook speech tokens and audio heads to generate audio tokens.ASR data remains a small proportion because it contributes minimally to modality alignment; loss weights are a = 1.75, b = 0.25, c = 1.5, and d = 0.1, with 5.1 trillion tokens at a 2:1 text-to-audio ratio.Source: §3.2.2 · pp. 12–13

3.2.3 Stage-2 Multimodal Continued Pre-Training

• Stage-2 adds image-caption and interleaved image-text data, jointly training the ViT, vision projector, and LLM decoder for vision-language alignment.The stage uses more than 3 trillion tokens, keeps text-to-audio and text-to-vision ratios at 2:1, and sets the additional vision loss weight to 0.25.Source: §3.2.3 · p. 13

3.2.4 Stage-3 Multimodal Annealing Training

• Following pre-training, multimodal annealing continues training on curated higher-quality data at an annealed learning rate, incorporating video and broader image datasets while PPL-gap-based sampling raises weights for subsets lagging expected reference convergence.The annealing phase maintains a text:vision:speech token ratio of 2:1:1 and consumes 0.33 trillion tokens. High-value samples are isolated and relabeled into independent subsets.Source: §3.2.4 · p. 13

3.2.5 Stage-4 Context-Length Extension Training

• Stage-4 extends context from 8K to 32K and then 128K tokens, adjusting RoPE base frequency to preserve positional encoding and stable attention.The extensions use 100B and an additional 20B training tokens, respectively; native-resolution encoding preserves visual details and token efficiency, and the training mixture adds 25% long-context multimodal data while maintaining a 2:1:1 text-to-vision-to-speech ratio.Source: §3.2.5 · pp. 13–14

3.2.6 Stage-5 Audio Encoder Alignment Training

• Stage-5 freezes the LLM and trains only the audio encoder to align continuous speech inputs with its semantic space while preserving multimodal processing capabilities.The encoder is initialized from a speech-recognition model, the projector is randomly initialized, and training loss is computed only on the LLM response after varied task prompts and speech input.Source: §3.2.6 · p. 14

4 Post-Training

• Post-training uses SFT for multimodal instruction-following and spoken interaction, followed by DPO to improve behavioral alignment, coherence, and consistency.The stage aims to produce semantically accurate, perceptually natural, contextually coherent responses in offline and online interactions.Source: §4 · p. 14

4.1 Supervised Fine-Tuning

• SFT combines large-scale instruction data with curated speech-to-speech and audio-visual interaction data to strengthen multimodal instruction-following, reasoning, and interaction.The audio encoder is frozen while other modules update, preserving pretrained acoustic representations while learning multimodal alignment; pure-text reasoning, math, coding, tool-use, and dialogue data are also included.Source: §4.1 · p. 14, §4.1.3 · p. 16

4.1.1 High-quality and Diverse Instruction Data Curation

• 700K video-text SFT samples balance temporal reasoning, causal inference, and complex real-world visual understanding, alongside image-text data for captioning, VQA, OCR, grounding, and STEM reasoning.The image-text data also covers multi-image settings, document and chart comprehension, and agentic task execution.Source: §4.1.1 · pp. 14–15
• SFT curation reuses audio data for ASR, AST, paralinguistic understanding, captioning, and QA, and adds spoken visual QA plus time-synchronized audio-visual QA.Visual inputs paired with spoken prompts elicit speech answers; interleaved audio-visual chunks support reasoning over temporally aligned content.Source: §4.1.1 · p. 15

4.1.2 Multimodal Interaction Data Curation

• Interaction data targets natural, expressive speech-to-speech dialogue and multi-turn audio-visual conversation with immediate feedback, barge-in, memory, and co-reference resolution.Audio-visual samples are generated through model-driven QA and dialogue construction, then refined by human annotation and converted to speech for integration with video.Source: §4.1.2 · pp. 15–16

4.2 Reinforcement Learning

• Joint DPO optimizes the text head and all audio heads together to improve cross-modal alignment and stability, producing more coherent, natural responses [Rafailov et al., 2023].The text-head loss targets semantic quality, while audio-head losses address linguistic and pronunciation stability.Source: §4.2 · p. 17

4.2.1 Data Construction

• DPO data combines general safety, helpfulness, and style examples with SFT-prompt rollouts, using manual annotation and multimodal-model scoring to construct preference pairs.The model generates six candidate responses per prompt, and the prompt set spans the SFT modalities.Source: §4.2.1 · p. 17

4.2.2 Training Details

• DPO training runs for one epoch with batch size 256, equal text- and speech-head loss weights, and a KL-divergence regularizer weighted 0.1.Training uses cosine learning-rate decay from 1 × 10−6 to zero after a 0.03 warm-up fraction.Source: §4.2.2 · p. 17

5 Training Infrastructures

• Over 90% of text-only training throughput is sustained in multimodal training through component decoupling that enables independent performance and memory optimization while enforcing deterministic, reproducible runs.The infrastructure decouples the LLM, vision encoder, and audio encoder.Source: §5 · p. 17

5.1 Multimodal Decoupling Framework

• MDP fully decouples modality encoders from the LLM backbone at the distributed level, enabling independent scheduling and more efficient computational-resource use amid heterogeneous data and workloads.The report notes that the model’s parameter count makes the described FSDP approach impractical.Source: §5.1 · pp. 17–18

5.1.1 Modality-Decoupled Parallelism

• MDP co-locates encoders and decoder, uses HSDP and activation recomputation for encoder memory, and combines PP, ZeRO-1 DP, CP, and EP for the LLM decoder.InnerDP partitions modality data across microbatches; its size is the product of the decoder’s PP and CP dimensions, and encoder and decoder DP ranks map one-to-one.Source: §5.1.1 · p. 18

5.1.2 ModalityBridge

• Chunk-based ModalityBridge processing reduces peak memory to 1/num_chunk of the original while maintaining bitwise numerical alignment.The bridge transforms embedding and gradient formats between the differently parallelized encoders and decoder; chunking addresses memory pressure from gathering and scattering data at the inner_dp = 0 rank.Source: §5.1.2 · pp. 18–20

5.2 Performance Tuning

• The training system raises effective sequence length per EP rank to improve operator efficiency, while its overlap tuning allocates SMs to communication and compute and isolates PP communication across stages.Hardware benchmarking indicates that reducing CP substantially improves core-operator efficiency.Source: §5.2.1 · p. 20, §5.2.2 · p. 20

5.2.3 Kernel Optimizations

• Optimized Grouped GEMM achieves approximately 75% MFU in the evaluated scenarios; deterministic FA achieves approximately 0.8× the performance of the non-deterministic version, with semaphore-based synchronization ensuring deterministic backward-phase QK reduction.Source: §5.2.3 · p. 21

5.3 Memory Optimization Strategies

• Memory optimization targets a theoretical footprint of approximately 72 GB per device against an unoptimized requirement of approximately 137 GB, using recomputation, scheduling, and communication-memory measures.Dynamic expert recomputation frees MoE memory except down-projection inputs when a rank receives too many tokens, preventing imbalance-triggered crashes at modest overhead.Source: §5.3 · p. 21

5.4 Numerical Consistency

• Deterministic computation, communication, and data flow keep experiments reproducible with bitwise-aligned loss values.For features where bit alignment is not feasible, the framework analyzes numerical deviations and verifies implementations against a gold reference.Source: §5.4 · p. 22

6.1 Decoupled Framework

• Dedicated hardware for modality-specific encoders and decoders and the LLM reduces cross-modal resource contention, achieving lower latency and higher throughput than conventional hybrid deployment despite communication overhead.LLM deployment uses Prefill-Decode disaggregation and Single Batch Overlap for ScMoE, following LongCat-Flash.Source: §6.1 · p. 22

6.2 Asynchronous Streaming Pipeline

• The asynchronous serving pipeline concurrently executes VAD and frame sampling, audio-visual encoding, LLM prefilling and decoding, and audio decoding, with incremental inference and adaptive batching.VAD triggers dense video-frame sampling while the user is speaking, and sparse sampling otherwise, to support detailed analysis or track the stream’s high-level overview.Source: §6.2 · pp. 22–23, Figure 11 · p. 23
• Speculative decoding begins at t3 before silence-end detection at t4 to reduce first-packet latency; resumed speech triggers rollback to prefill and discards generated content.Audio is delivered only after end-of-turn detection at t4; an interruption at t5 immediately stops generation and truncates the LLM token sequence at a natural breakpoint such as punctuation.Source: §6.2 · p. 23
• Within 100ms after endpoint detection, users can receive a response by overlapping VAD detection (600–700ms) with streaming prefill of 1-second audio and two video frames per packet.Each packet can be prefilled immediately, without waiting for the user’s turn to finish.Source: §6.2 · p. 23

• The evaluation compares LongCat-Flash-Omni with proprietary and open-source multimodal models across several modality-specific and cross-modal tasks.Source: §7 · p. 23
• The evaluation covers vision understanding, audio comprehension, and text understanding and generation.Source: §7 · p. 23
• Table 5 evaluates image understanding; its asterisked values come from public reports, and GPT-4o results are omitted for RefCOCO and ScreenSpot-v2 because it does not support image grounding.Source: Table 5 · p. 24

7.1 Vision Capability Evaluation

• The comparison includes omni-models and vision-language models evaluated under their official configurations; Gemini-2.5-Pro’s thinking budget is limited to 128 tokens.GPT-4o and Seed-1.6 are assessed only on visual capabilities because they provide vision-understanding APIs only.Source: §7.1 · p. 24

7.1.1 Image-to-Text Evaluation

• Image evaluation covers general domains, STEM and reasoning, multi-image tasks, text recognition and documents, grounding and counting, and GUI understanding.Benchmarks include MMBench, MathVista, BLINK, ChartQA, RefCOCO, VisualWebBench, and others.Source: §7.1.1 · p. 24
• On image understanding benchmarks, LongCat-Flash-Omni performs comparably to Gemini-2.5-Flash and outperforms open-source Qwen3-Omni.Its advantage is especially pronounced on multi-image tasks, which the report links to training on high-quality interleaved image-text, multi-image, and video data.Source: §7.1.1 · p. 24

7.1.2 Video-to-Text Evaluation

• Video evaluation covers short videos, long videos, and STEM reasoning, with VideoMME tested without subtitles both with and without audio.All video benchmarks use a 32K-token maximum context; Gemini results are reported only for VideoMME with audio because official no-audio entries are unavailable.Source: §7.1.2 · p. 25
• LongCat-Flash-Omni achieves state-of-the-art video-to-text performance, leading significantly on short-video tasks and matching leading models on long-video tasks.It also scores best among omni-modal models on VideoMME; the report attributes its results to dynamic frame sampling, hierarchical token aggregation, and strong long-context modeling.Source: §7.1.2 · p. 25, Table 6 · p. 25

7.2.1 Base Model Evaluation

• Base models show competitive Chinese and English ASR performance despite relying on discretized audio features.Chinese speech recognition uses speechio02, while English recognition uses LibriSpeech.Source: §7.2.1 · p. 25, Table 7 · p. 26
• Base models generate robust speech from text, evaluated by CER for Chinese and WER for English against the input text.The evaluation uses discrete semantic-acoustic tokens as prompts, text as teacher-forcing guidance, and an audio decoder to reconstruct waveforms.Source: §7.2.1 · p. 25, Table 7 · p. 26
• Across pre-training stages, base models perform well on 1-shot speech continuation, with negligible accuracy disparity between text and speech outputs.The test uses 1,200 synthesized samples derived from CMMLU text and measures response accuracy for text and ASR-transcribed speech outputs.Source: §7.2.1 · pp. 25–26, Table 8 · p. 26

7.2.2 Instruct Model Evaluation

• The instruct model’s audio evaluation covers speech recognition and translation, audio understanding, and audio-to-text chat.Source: §7.2.2 · p. 26

Speech Recognition and Translation

• LongCat-Flash-Omni consistently outperforms competitors on ASR benchmarks and shows strong speech-to-text translation capabilities among evaluated models.Kimi-Audio defaults to ASR rather than executing the requested translation task.Source: Speech Recognition and Translation · p. 26, Table 9 · p. 27

Audio Understanding

• Across music, sound-event, and speech-comprehension benchmarks, LongCat-Flash-Omni outperforms most competitors and achieves state-of-the-art results on several.The evaluation covers seven benchmarks, including MMAU, VocalSound, ClothoAQA, and MELD.Source: Speech Recognition and Translation · p. 26, Table 10 · p. 27

Audio-to-Text Chat

• On OpenAudioBench and VoiceBench, LongCat-Flash-Omni performs strongly across subsets and reaches state-of-the-art results in several cases.The benchmarks assess world knowledge, domain-specific understanding, instruction following, and reasoning using their official evaluation code.Source: Speech Recognition and Translation · p. 27

7.3.1 Base Model Evaluation

• Base model evaluation compares general, reasoning, math, and coding performance against strong text-based models using the same protocol as [Meituan, 2025a].Benchmarks include MMLU [Hendrycks et al., 2021a], GPQA [Rein et al., 2023], GSM8K [Cobbe et al., 2021], and MBPP+ [Liu et al., 2024f].Source: §7.3.1 · p. 27
• The Base model performs on par with state-of-the-art base models despite its compact active/total parameter size, with no text-capability degradation after multimodal training.The authors attribute this to their training strategy.Source: §7.3.1 · p. 27

7.3.2 Instruct Model Evaluation

• Instruct evaluation covers general domains, instruction following, mathematical and general reasoning, and coding, including multi-turn instruction-following tests.The suite includes Meeseeks, which uses iterative feedback to simulate multi-turn interactions and assess self-correction.Source: §7.3.2 · p. 28
• LongCat-Flash-Omni Instruct maintains superior text capability, leading across domains and outperforming LongCat-Flash in some domains without text-capability degradation.The comparison includes DeepSeek-V3.1, Qwen3-235B-A22B, Kimi-K2, GPT-4.1, Claude Sonnet-4, Gemini2.5-Flash, and LongCat-Flash; models supporting thinking and non-thinking modes were evaluated in non-thinking mode for fairness.Source: §7.3.2 · p. 28

7.4.1 Cross-modality Understanding Evaluation

• UNO-Bench adds 1,880 human-crafted questions across 44 task types, with 98% requiring cross-modal reasoning, including multi-step open-ended items.It was manually annotated from in-house data to reduce contamination and represent real-world scenarios; the evaluation also uses an internally corrected OmniBench version because the public release has scoring deficiencies.Source: §7.4.1 · p. 29
• LongCat-Flash-Omni outperforms Gemini-2.5-Flash-non-thinking, performs comparably to Gemini-2.5-Pro-ThinkingBudget128, and leads open-source models on WorldSense, DailyOmni, and UNO-Bench.The authors describe its WorldSense and DailyOmni results as superior to other open-source omni-modal models and report exceptional performance among open-source models on UNO-Bench.Source: §7.4.1 · p. 29, Table 14 · p. 30

7.4.2 Real-time Audio-Visual Interaction Evaluation

• The proprietary interaction framework collected 200 approximately three-minute sessions per model across four scenario categories, then combined 250 users’ ratings with expert analysis across six factors.Users triple-annotated videos on a 0–3 naturalness-and-fluency scale; expert factors included real-timeness, human-likeness, paralinguistic understanding, relevance, accuracy, and memory capability.Source: §7.4.2 · pp. 30–31
• LongCat-Flash-Omni ranks third in end-to-end naturalness and fluency, scoring 0.56 points above Qwen3-Omni, the current SOTA open-source model.Among audio-visual interaction products, it ranks behind Doubao and GPT-4o but ahead of iFlytek Spark and StepFun.Source: Table 15 · p. 30, §7.4.2 · p. 31
• Qualitative analysis finds strengths in paralinguistic understanding, relevance, and memory, but gaps in real-timeness, human-likeness, and accuracy.The model may respond prematurely during pauses, produce pronunciation errors or robotic audio artifacts, miss text and numerical information, and over-agree while overlooking visual content.Source: Table 16 · p. 31, §7.4.2 · p. 32

8 Conclusion

• LongCat-Flash-Omni unifies robust offline multimodal understanding and real-time audio-visual interaction across text, audio, image, and video without sacrificing individual-domain performance.The report presents it as a cohesive framework for perceiving, integrating, and generating across modalities.Source: §8 · p. 32
• A multi-stage early-fusion pipeline, human-in-the-loop data, a 128K-token context window, and ScMoE with zero-computation experts support integrated reasoning, multi-turn capabilities, and real-time interaction.Lightweight modality encoders and a decoder also contribute to real-time audio-visual interaction.Source: §8 · p. 32
• The model achieves state-of-the-art performance on Omni-Bench and WorldSense and matches or exceeds closed-source systems on key image, video, and audio-comprehension tasks.Subjective assessments also confirm natural, low-latency, high-quality interactive experiences.Source: §8 · p. 32
• Future work will expand training-data diversity and scale, add adaptive thinking, refine streaming and generation, and explore embodied and interactive intelligence.The report anticipates that open release will accelerate multimodal research and inspire human-centered, AGI-oriented applications.Source: §8 · p. 32