PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCat-Flash Technical Report

Meituan LongCat Team, Bayan, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, Chengcheng Han, Chenguang Xi, Chi Zhang, Chong Peng, Chuan Qin, Chuyu Zhang, Cong Chen, Congkui Wang, Dan Ma, Daoru Pan, Defei Bu, Dengchang Zhao, Deyang Kong, Dishan Liu, Feiye Huo, Fengcun Li, Fubao Zhang, Gan Dong, Gang Liu, Gang Xu, Ge Li, Guoqiang Tan, Guoyuan Lin, Haihang Jing, Haomin Fu, Haonan Yan, Haoxing Wen, Haozhe Zhao, Hong Liu, Hongmei Shi, Hongyan Hao, Hongyin Tang, Huantian Lv, Hui Su, Jiacheng Li, Jiahao Liu, Jiahuan Li, Jiajun Yang, Jiaming Wang, Jian Yang, Jianchao Tan, Jiaqi Sun, Jiaqi Zhang, Jiawei Fu, Jiawei Yang, Jiaxi Hu, Jiayu Qin, Jingang Wang, Jiyuan He, Jun Kuang, Junhui Mei, Kai Liang, Ke He, Kefeng Zhang, Keheng Wang, Keqing He, Liang Gao, Liang Shi, Lianhui Ma, Lin Qiu, Lingbin Kong, Lingtong Si, Linkun Lyu, Linsen Guo, Liqi Yang, Lizhi Yan, Mai Xia, Man Gao, Manyuan Zhang, Meng Zhou, Mengxia Shen, Mingxiang Tuo, Mingyang Zhu, Peiguang Li, Peng Pei, Peng Zhao, Pengcheng Jia, Pingwei Sun, Qi Gu, Qianyun Li, Qingyuan Li, Qiong Huang, Qiyuan Duan, Ran Meng, Rongxiang Weng, Ruichen Shao, Rumei Li, Shizhe Wu, Shuai Liang, Shuo Wang, Suogui Dang, Tao Fang, Tao Li, Tefeng Chen, Tianhao Bai, Tianhao Zhou, Tingwen Xie, Wei He, Wei Huang, Wei Liu, Wei Shi, Wei Wang, Wei Wu, Weikang Zhao, Wen Zan, Wenjie Shi, Xi Nan, Xi Su, Xiang Li, Xiang Mei, Xiangyang Ji, Xiangyu Xi, Xiangzhou Huang, Xianpeng Li, Xiao Fu, Xiao Liu, Xiao Wei, Xiaodong Cai, Xiaolong Chen, Xiaoqing Liu, Xiaotong Li, Xiaowei Shi, Xiaoyu Li, Xili Wang, Xin Chen, Xing Hu, Xingyu Miao, Xinyan He, Xuemiao Zhang, Xueyuan Hao, Xuezhi Cao, Xunliang Cai, Xurui Yang, Yan Feng, Yang Bai, Yang Chen, Yang Yang, Yaqi Huo, Yerui Sun, Yifan Lu, Yifan Zhang, Yipeng Zang, Yitao Zhai, Yiyang Li, Yongjing Yin, Yongkang Lv, Yongwei Zhou, Yu Yang, Yuchen Xie, Yueqing Sun, Yuewen Zheng, Yuhuai Wei, Yulei Qian, Yunfan Liang, Yunfang Tai, Yunke Zhao, Zeyang Yu, Zhao Zhang, Zhaohua Yang, Zhenchao Zhang, Zhikang Xia, Zhiye Zou, Zhizhao Zeng, Zhongda Su, Zhuofan Chen, Zijian Zhang, Ziwen Wang, Zixu Jiang, Zizhe Zhao, Zongyu Wang, Zunhai Su

arXiv:2509.01322v2cs.CLcs.AIcs.DCcs.LG

TL;DR

LongCat-Flash addresses the need for cost-effective scaling of large language models and stronger real-world agentic capabilities. It combines dynamic-computation and shortcut-connected MoE architecture with scalable training and multi-stage post-training. The model was trained on over 20T tokens within 30 days, exceeded 100 TPS at $0.7 per million output tokens, and showed competitive non-thinking-model performance with strengths in agentic tasks.

  • Problem

    Scaling large language models cost-effectively while improving their ability to solve real-world agentic tasks requires advances in architecture, training strategies, and data strategy.

  • Method

    LongCat-Flash combines zero-computation experts, Shortcut-connected MoE, scalable training strategies, and multi-stage post-training for reasoning, coding, general capability, and agentic tool use.

  • Results

    LongCat-Flash completed pre-training on over 20T tokens within 30 days, exceeded 100 tokens per second (TPS) on H800 at $0.7 per million output tokens, and achieved competitive benchmark performance with strengths in agentic tasks.

  • Takeaways & Limitations

    The open-sourced 560B model provides a platform for research on efficient MoE architectures, high-quality data strategies, and agentic model development.

  • Takeaways & Limitations

    Operator precision differences can produce over 5 pp benchmark variation despite training-loss discrepancies of only 1e-3∼1e-4, leaving cost-effective precision-error evaluation open.

Abstract

from arXiv · show

We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming from the need for scalable efficiency, LongCat-Flash adopts two novel designs: (a) Zero-computation Experts, which enables dynamic computational budget allocation and activates 18.6B-31.3B (27B on average) per token depending on contextual demands, optimizing resource usage. (b) Shortcut-connected MoE, which enlarges the computation-communication overlap window, demonstrating notable gains in inference efficiency and throughput compared to models of a comparable scale. We develop a comprehensive scaling framework for large models that combines hyperparameter transfer, model-growth initialization, a multi-pronged stability suite, and deterministic computation to achieve stable and reproducible training. Notably, leveraging the synergy among scalable architectural design and infrastructure efforts, we complete model training on more than 20 trillion tokens within 30 days, while achieving over 100 tokens per second (TPS) for inference at a cost of \$0.70 per million output tokens. To cultivate LongCat-Flash towards agentic intelligence, we conduct a large-scale pre-training on optimized mixtures, followed by targeted mid- and post-training on reasoning, code, and instructions, with further augmentation from synthetic data and tool use tasks. Comprehensive evaluations demonstrate that, as a non-thinking foundation model, LongCat-Flash delivers highly competitive performance among other leading models, with exceptional strengths in agentic tasks. The model checkpoint of LongCat-Flash is open-sourced to foster community research. LongCat Chat: https://longcat.ai Hugging Face: https://huggingface.co/meituan-longcat GitHub: https://github.com/meituan-longcat

1 Introduction

LongCat-Flash targets scalable intelligence through computationally efficient architecture, scalable training, and systematic data and training strategies. It reports rapid large-scale training, low-cost high-throughput inference, and competitive performance across general, coding, and agentic evaluations.

  • Overview: LongCat-Flash combines computational efficiency and agentic capability through architectural innovations and a multi-stage training methodology.The model is designed to advance both directions using large-scale training and targeted capability development.
  • Scalable Architecture: 18.6 to 31.3 billion parameters activate per token, averaging approximately 27 billion out of 560 billion total, through context-dependent zero-computation experts.A PID-controlled expert bias maintains consistent average computation while allocating different budgets to tokens.
  • Scaling Results: 20T tokens were pretrained within 30 days, with 98.48% time availability without manual fault-resolution intervention.The reported result reflects combined architectural, training-strategy, and infrastructure efforts.
  • Inference Results: Over 100 tokens per second (TPS) on H800 was achieved at a cost of $0.7 per million output tokens.The report presents this as large-scale deployment efficiency compared with similarly sized models.
  • Evaluation: LongCat-Flash scores 86.5 on ArenaHard-V2, 39.5 on TerminalBench, and 67.7 on τ 2-Bench across general, coding, and agentic tool-use tasks.The model is described as competitive with leading non-thinking models while using fewer parameters and offering faster inference.
  • Evaluation: Meeseeks reports scores on par with frontier LLMs, while VitaBench reports superior performance than other LLMs on complex real-world business scenarios.The benchmarks target iterative multi-turn instruction following and real-world task proficiency, respectively.

2 Architecture

LongCat-Flash uses zero-computation experts to allocate variable computation by token while controlling average load, and combines this with load-balancing mechanisms for scalable MoE execution. The architecture preserves quality while supporting substantial variation in per-token computation.

  • 2.1 Zero-Computation Experts: Zero-computation experts return the input unchanged, allowing tokens to activate variable numbers of standard FFN experts according to contextual importance.The expanded pool contains N standard FFN experts and Z zero-computation experts, which add no computational cost.
  • 2.1 Zero-Computation Experts: Under matched budgets, the zero-expert variant dynamically activates 4.2B-7.0B parameters while maintaining an expected eight FFN experts.The baseline activates fixed 6B parameters per token, whereas the zero-expert variant shows consistent loss reduction.
  • 2.1.1 Computational Budget Control: A PID-based expert-bias update adjusts routing from recent utilization so each expert’s token allocation converges toward its target proportion.The update uses global-batch token counts, expected activated FFN experts, and a bias adaptation rate; large batches and decayed adaptation improve stability.
  • ScMoE Quality: ScMoE-enhanced and baseline training loss curves are virtually indistinguishable across MLA, MHA, and GQA configurations.The comparison covers 2.4B-16B, 3B-20B, and 15B-193B model configurations.
  • 2.1.1 Computational Budget Control: After approximately 20B tokens of adjustment, average expert activation converges to its expected value with fluctuations below 1%, while standard deviation remains high across tokens.The persistent standard deviation indicates substantially divergent computational resources among tokens.
  • 2.1.2 Load Balance Control: Device-level load balancing assigns zero-computation experts to an additional group and adjusts group frequencies toward the desired FFN-to-zero-computation ratio.The loss uses a balance factor, micro-batch token count, and indicator function to control device-level distribution.

2.2 Shortcut-Connected MoE

Shortcut-connected MoE addresses communication bottlenecks in expert-parallel execution by enlarging the computation-communication overlap window. The design preserves training quality while improving system efficiency in training and inference.

  • Motivation: Conventional expert parallelism routes tokens before computation, creating communication latency, device underutilization, and limited throughput.Shared-expert overlap remains constrained by the small computational window of one expert.
  • Architecture: ScMoE introduces a cross-layer shortcut that reorders execution so preceding-block dense FFN computation can overlap with MoE dispatch and combine communication.Fine-grained token-dimension partitioning enables the expanded overlap window during large-scale training.
  • Quality: ScMoE and baseline loss curves are nearly identical across MLA, MHA, and GQA settings, indicating no observed model-quality compromise.The reported configurations span 2.4B-16B, 3B-20B, and 15B-193B models.
  • Summary: ScMoE delivers system-level efficiency gains for both training and inference without sacrificing model quality.The reported gains are attributed to the quality-neutral architectural reordering rather than a quality trade-off.
  • Inference Efficiency: ScMoE reduces theoretical Time-Per-Output-Token by nearly 50% compared with leading models such as DeepSeek-V3.Single Batch Overlap also overlaps intra-node Tensor Parallelism communication with inter-node Expert Parallelism communication.

2.3 Variance Alignment Design for Scalability

LongCat-Flash addresses scaling instability by aligning variances in MLA and MoE components. Scale correction improves convergence, while variance compensation preserves MoE output variance after expert segmentation.

  • Variance misalignment in MLA and MoE modules can cause instability and degraded performance as models scale.
  • Variance Alignment for MLA: Scale-correction factors αq and αkv rescale MLA low-rank components against a full-model-dimension reference.
  • Variance Alignment for MLA: Scale correction improves convergence, producing lower loss on a 1B activated MoE model.
  • Variance Compensation for Expert Initialization: Expert segmentation reduces output variance through gating dilution and reduced expert dimensions, each by approximately a factor of m.
  • Variance Compensation for Expert Initialization: The compensation factor γ restores initialization variance to the pre-segmentation baseline, with γ = √m · m = m.

2.4 Model Information

LongCat-Flash uses multilingual BPE tokenization, adds targeted segmentation changes, and incorporates multi-token prediction to improve inference efficiency. Its configuration balances model capacity and computational efficiency through specified attention and expert dimensions.

  • Tokenizer: The tokenizer uses BPE trained on multilingual web pages, books, and source code for cross-domain performance.
  • Tokenizer: Enhanced CJK segmentation and independent digit tokenization target improved Chinese text handling and mathematical representation.
  • Multi-Token Prediction: Multi-Token Prediction is trained as an auxiliary objective with a single dense-layer head to enhance inference efficiency.
  • Model Configurations: The model has 28 layers, a 6144-dimensional hidden state, 64 attention heads, and 2048-dimensional FFN experts.

3 Pre-Training

LongCat-Flash combines transferred hyperparameters, model-growth initialization, and stability techniques to support scalable pre-training. Experiments report reduced hyperparameter-search cost, faster convergence after growth initialization, and competitive performance across diverse benchmarks.

  • 3.1.1 Hyperparameter Transfer: Hyperparameter transfer identifies optimal settings on a proxy model and maps them to the target architecture using theoretically motivated scaling rules.The transfer uses the width scaling factor s = n_target/n_proxy and Adam LR Full Align rules for Standard Parameterization.
  • 3.1.1 Hyperparameter Transfer: The transfer method significantly reduces computational costs for identifying initialization variances and learning rates for large-scale training.Experiments describe the resulting framework as robust and theoretically grounded for model scaling.
  • 3.1.2 Model Growth Initialization: Model growth stacks a half-scale model to initialize the target model, using layer stacking with expansion rate r = 2.The target transformation is constructed by stacking copies of the smaller model’s transformation.
  • 3.1.2 Model Growth Initialization: Model-growth initialization initially increases loss, then accelerates convergence and ultimately outperforms randomly initialized baselines.The reported pattern comes from extensive experiments, including a representative 6B activated-model case.
  • 3.1.2 Model Growth Initialization: Over-optimizing predecessor models may reduce token efficiency in target models, making growth timing an important constraint.The authors present this as experimental evidence and conjecture that growth operations may provide implicit regularization.
  • 3.1.3 Training Stability: Training stability is addressed through router, activation, and optimizer mechanisms, including hidden z-loss and gradient-RMS-guided epsilon selection.The supplied stability passages describe competing routing gradients, suppression of massive activations, and numerical-stability thresholds.
  • 3.6.2 Evaluation Results: LongCat-Flash achieves competitive performance across diverse benchmarks, matching DeepSeek-V3.1 Base across domains while using fewer parameters.It shows parity or superiority on many reasoning, math, and coding tasks, with modestly lower performance than Kimi K2 Base on general tasks.

4 Post-Training

LongCat-Flash uses multi-stage post-training to strengthen reasoning, coding, instruction following, and agentic tool use. Evaluations report strong performance across general, mathematical, coding, and agentic tasks.

  • Post-training framework: Multi-stage post-training targets reasoning, coding, agentic tool use, and general-purpose capabilities, while limited high-quality problem sets remain a bottleneck.The process is organized into reasoning and coding, agentic tool use, and general capability phases.
  • Agentic tool use: Agentic tasks require systematic environment interaction, including information processing, tool selection, and strategic user questioning.Task difficulty is attributed to information-processing, tool-set, and user-interaction complexity.
  • Instruction following: 89.65 on IFEval is the highest score, alongside best scores of 57.10 on COLLIE and 43.03 on Meeseeks-zh.These results cover complex instruction sets in both English and Chinese.
  • Mathematical reasoning: 96.40 on MATH500, 61.25 on AIME25, and 43.00 on BeyondAIME demonstrate strong mathematical reasoning, especially on challenging competition-level benchmarks.The reported results emphasize multi-step logical deduction and problem-solving.
  • Coding: 39.51 on TerminalBench ranks second, while 60.4 on SWE-Bench-Verified shows competitive coding performance.Foundational code-generation benchmarks remain an area for future optimization toward leading models.
  • Agentic tool use: 24.30 on VitaBench is the highest score, and LongCat-Flash outperforms larger models on τ 2-Bench in agentic tool use.The results indicate strength in highly complex scenarios.

5 Training Infrastructures

LongCat-Flash combines deterministic, fault-aware infrastructure with parallelism and communication-overlap techniques for scalable training. The system reports high availability while optimizing kernels, memory, and MoE communication.

  • Correctness and determinism: Deterministic operators and communication produce bitwise-aligned loss values across repeated training steps.The infrastructure also verifies operator precision and embeds online Silent Data Corruption detection.
  • Scalable training: 98.48% availability and near-linear scaling were achieved across tens of thousands of accelerators through model-system co-design and automated fault recovery.The system scales training while addressing stability and fault-detection challenges.
  • Limitations: Operator precision control is necessary but insufficient: loss discrepancies of 1e-3∼1e-4 can coincide with benchmark variation exceeding 5 pp.The paper identifies cost-effective evaluation of precision-error impact as an open challenge.
  • Kernel optimization: Grouped GEMM optimizations yield 5%–45% speedups, while fused GroupedGemmAdd reaches 3.12x to 3.86x speedup on its benchmark.The techniques overlap computation, memory I/O, and communication while reducing intermediate writes.
  • MoE communication: ScMoE overlaps dispatch and combine communication with computation, and chunking further increases the computation-communication overlap window.The architecture divides the MoE layer along the token dimension to improve overlap.
  • Reliability: Asynchronous checkpointing reduces stalls to 2∼4 seconds, while automated recovery reduces recovery time to <10 minutes.These mechanisms increase availability and limit fault-induced training loss.

6 Inference and Deployment

LongCat-Flash co-designs its architecture and inference system to overlap computation, communication, and scheduling overhead. Speculative decoding, cache-safe scheduling, and kernel optimizations support low latency and high throughput.

  • Computation-communication overlap: Single Batch Overlap uses a four-stage module-level pipeline to hide communication overhead within one batch.This strategy differs from Two Batch Overlap by keeping communication hidden inside a single batch.
  • Speculative decoding: Approximately 90% draft-token acceptance is achieved with a single MTP head, while a lightweight dense MTP layer balances draft quality and speed.C2T filtering further reduces target-verification cost.
  • Schedule overhead: TVD fuses target forward, verification, and draft forward into one CUDA graph, while overlapped scheduling hides CPU scheduling and synchronization.The scheduler is designed to maintain continuous GPU occupancy.
  • KV-cache scheduling: Mathematical induction guarantees safe future KV-cache allocation without knowing the current iteration’s accept length, while allocated cache size converges.The multi-step scheduler addresses speculative-decoding accept-length uncertainty.
  • Deployment efficiency: Nearly 100 tokens/s for action commands corresponds to $0.7 per million output tokens and constrains single-round tool-call latency below one second.The reported cost assumes an H800 price of $2 per hour.
  • Latency evaluation: At batch size 96, measured TPOT is approximately 26 ms, or about 61.5% of the theoretical value, comparable to DeepSeek-V3’s approximately 64%.The remaining gap is attributed mainly to small-operator overhead and communication-bandwidth loss.
  • System comparison: Model-system co-design yields significant theoretical improvements in throughput and latency compared with DeepSeek-V3 and Qwen3-235B-A22B under TBO scheduling.The comparison is based on theoretical limits reported in Table 7.

7 Conclusion

LongCat-Flash is a 560B-parameter MoE model combining dynamic computation, shortcut-connected MoE, stable large-scale training, and multi-stage agentic training. Its open-source release is intended to support community research.

  • Conclusion: LongCat-Flash combines context-aware dynamic computation and shortcut-connected MoE for efficient training and inference.The model also integrates stability strategies and multi-stage training for agentic capabilities.
  • Conclusion: The model’s multi-stage training pipeline targets complex tasks requiring iterative reasoning and environmental interaction.The checkpoint is released as an open-source model for community research.

8 Contributions

This section lists the paper’s authors and identifies authors who have left the team.

  • The authors are listed in alphabetical order.
  • Names marked with an asterisk indicate people who have left the team.
  • The section provides the complete author listing for the paper.

A.1 Statistics and Case Studies of Dynamic Routing

The routing analysis examines how many FFN experts are activated across benchmarks and how expert selection varies by token type and layer. It identifies computational bias toward English tokens and distinct selection patterns across layers.

  • A consistent computational bias favors English tokens over Chinese and mathematical tokens.
  • Figure 11 reports the average number of activated FFN experts across benchmarks.
  • Function words, numbers, and punctuation receive lower computational resources in the first layer, while Layer 28 shows less specialized allocation.
  • Expert selection patterns differ across layers and across the examined token cases.
Loading 2509.01322v2…