LongCat-Flash Technical Report

Scalable Architectural Design for Computational Efficiency

• Zero-computation experts dynamically activate 18.6–31.3B of 560B parameters per token, averaging about 27B, while Shortcut-connected MoE expands the computation-communication overlap window.An expert bias adjusted by a PID controller maintains consistent computation load; customized infrastructure supports large-scale training and high-throughput, low-latency inference.Source: §1 · p. 4, Figure 2 · p. 5

Effective Model Scaling Strategy

• LongCat-Flash’s scaling framework combines hyperparameter transfer, model-growth initialization, a stability suite, and deterministic computation to support stable, reproducible training.The stability suite uses router-gradient balancing, hidden z-loss, and tuned optimizer configurations; deterministic computation also enables detection of silent data corruption.Source: §1 · p. 4

Multi-Stage Training Pipeline for Agentic Capability

• A multi-stage pipeline builds agentic capability through reasoning-focused pretraining data, mid-training for reasoning and coding with context extended to 128k, and post-training with synthesized complex tasks.The multi-agent synthesis framework defines task difficulty across information processing, tool-set complexity, and user interaction, generating tasks requiring iterative reasoning and environmental interaction.Source: §1 · p. 4

Efficiency results

• 560B-parameter LongCat-Flash completes pre-training on over 20T tokens within 30 days, achieves 98.48% time availability, and exceeds 100 TPS on H800 at $0.7 per million output tokens.The report attributes these results to the synergy of architecture, training strategies, and infrastructure.Source: §1 · p. 4

Evaluation results

• 86.5 on ArenaHard-V2, 39.5 on TerminalBench, and 67.7 on τ 2-Bench accompany performance comparable to leading non-thinking models, including DeepSeek-V3.1 [DeepSeek-AI et al., 2025] and Kimi-K2 [Team et al., 2025].The report characterizes LongCat-Flash as using fewer parameters and offering faster inference than those models.Source: §1 · p. 4
• LongCat-Flash scores on par with frontier LLMs on Meeseeks and outperforms other LLMs on VitaBench’s real-world business tasks.Meeseeks evaluates multi-turn instruction-following through iterative feedback; VitaBench assesses complex real-world tasks.Source: §1 · p. 5

• Zero-computation experts [Jin et al., 2024] let tokens use variable computation according to contextual significance, while adaptive expert bias regulates average computational load.Source: §2 · p. 5
• Each layer integrates two MLA blocks and multiple heterogeneous FFN blocks, with a shortcut from the first MLA output directly to the MoE block [Cai et al., 2024].Source: §2 · p. 5
• The architecture refines both MLA and fine-grained FFN experts via variance alignment.Source: §2 · p. 5

2.1 Zero-Computation Experts

• At matched computation, the zero-expert variant activates 4.2B–7.0B parameters per token versus the baseline’s fixed 6B and consistently reduces validation loss.The variant maintains an expectation of 8 FFN experts, with less than 1% fluctuation; during training, the average remains near 8 while the standard deviation grows to 3.Source: Figure 3 · p. 6
• Zero-computation experts return the input without additional computation, allowing the router to vary activated FFN experts by token and allocate more resources to contextually important tokens.The mechanism is motivated by computational heterogeneity: easy tokens may need negligible computation, while difficult tokens may demand more.Source: §2.1 · pp. 5–6

2.1.1 Computational Budget Control

• After approximately 20B tokens of adjustment, average activated experts converged to the expected value with less than 1% fluctuation, while substantial per-token variation persisted.An expert-specific bias updated with a PID controller regulates token allocation toward target proportions, separately from the language-model objective; large batches and a decay schedule for µ improve stability.Source: §2.1.1 · pp. 6–7

2.1.2 Load Balance Control

• A device-level load-balance loss addresses sequence-level imbalance among expert-parallel groups by placing zero-computation experts in an additional group and steering the FFN-to-zero-expert ratio toward its target.The loss supplements corpus-level balancing and is designed to prevent extreme imbalance across EP groups.Source: §2.1.2 · p. 7

2.2 Shortcut-Connected MoE

• ScMoE overlaps the preceding block’s dense FFN with dispatch/combine communication in the current MoE layer, expanding the overlap window beyond shared-expert designs [Cai et al., 2024].Conventional expert parallelism routes tokens before computation, creating communication latency and device underutilization; shared-expert overlap is limited to one expert’s computation.Source: §2.2 · p. 8
• Nearly identical training loss curves show ScMoE does not compromise model quality across tested MLA, MHA, and GQA settings.The reported settings include 2.4B-16B MoE with MLA, 3B-20B with MHA [Vaswani et al., 2017], and 15B-193B with GQA [Ainslie et al., 2023], indicating the finding is orthogonal to attention mechanism choice.Source: §2.2 · p. 8
• ScMoE training fully parallelizes preceding-block computation with MoE dispatch and combine communication by partitioning operations into fine-grained token-dimension chunks.This expanded overlap window supports system-level training efficiency gains.Source: §2.2 · p. 8
• Nearly 50% lower theoretical TPOT than leading models such as DeepSeek-V3 is enabled by ScMoE’s Single Batch Overlap inference pipeline.It overlaps intra-node Tensor Parallelism communication on the dense FFN with inter-node Expert Parallelism communication, maximizing network utilization.Source: §2.2 · p. 8

2.3 Variance Alignment Design for Scalability

• Variance misalignment in specific modules can destabilize scaling and degrade performance, motivating alignment techniques for MLA and MoE blocks.Source: §2.3 · p. 8

2.3.1 Scale-Correction for MLA

• Scale-corrected MLA rescales low-rank paths against the full model dimension to address asymmetric-factorization variance imbalance, improving convergence (lower loss) on a 1B-activated MoE model [Liu et al., 2024a].Source: §2.3.1 · pp. 8–9, Figure 5 · p. 9

2.3.2 Variance Compensation for Experts Initialization

• Fine-grained expert segmentation [Liu et al., 2024a] makes performance sensitive to expert count, top-k, and m; gating dilution and reduced expert dimensions each lower variance by m, motivating γ = √m · m = m.This compensation preserves the MoE layer’s output variance at initialization, matching the pre-segmentation baseline.Source: §2.3.2 · p. 9

Tokenizer

• 131,072 tokens make up the BPE vocabulary, with enhanced CJK segmentation and independent digit tokenization.The tokenizer inherits GPT-4’s pre-tokenization framework and is trained on a multilingual corpus spanning web pages, books, and source code.Source: §2.4 · p. 10

Multi-Token Prediction

• >90% acceptance rate is achieved by the MTP head in evaluations.MTP is an auxiliary training objective introduced mid-training after rapid loss convergence, and uses a single dense layer rather than a MoE head.Source: §2.4 · p. 10

Model Configurations

• 560B total parameters use 28 layers and 512 FFN experts plus 256 zero-computation experts per layer, with 12 experts activated per token.Activation ranges from 18.6B to 31.3B parameters per token depending on context, averaging approximately 27B; the model has a 6144-dimensional hidden state.Source: §2.4 · p. 10

3.1.1 Hyperparameter Transfer

• Width-scaling hyperparameter transfer [Everett et al., 2024] selects initialization variance and learning rate on a proxy model, applies theoretically motivated scaling rules, and reduces the computational cost of large-model tuning.The method uses Adam LR Full Align rules for Standard Parameterization to adapt the proxy’s optimal initialization variance and learning rate to the target architecture.Source: §3.1.1 · pp. 10–11, Table 1 · p. 10

3.1.2 Model Growth Initialization

• Layer-stacking growth initializes the 28-layer target from a 14-layer predecessor (r = 2), producing an initial loss rise followed by accelerated convergence and better performance than random initialization.The authors conjecture that faster convergence of smaller models improves initialization and growth may regularize against parameter collapse; over-optimizing predecessors may hurt target token efficiency.Source: §3.1.2 · p. 11

3.1.3 Training Stability

• Training stability combines router monitoring, hidden z-loss to suppress extreme activations, and Adam ε=1e-16, chosen below critical gradient-RMS thresholds.Router weight similarity signals excessive load-balancing pressure, and the recommended gradient-norm ratio is Rg < 0.1; small hidden z-loss coefficients suppress massive activations without compromising training loss. Loss degrades rapidly when ε approaches gradient RMS, while further decreases below the threshold have negligible impact.Source: §3.1.3 · pp. 11–12, Figure 7 · p. 13

3.2 General Pre-Training

• General pre-training progressively increases high-quality reasoning data, with STEM and code reaching 70% of the final mixture and gradual code increases guided by external-validation perplexity.Stage 1 balances data quality and diversity using instance-level mixing, downsampling low-value domains and upsampling reasoning-rich domains.Source: §3.2 · p. 13

3.3 Reasoning and Coding Enhancement

• Mid-training strengthens reasoning and coding using retrieved and synthesized data, with knowledge-graph-based coverage, iterative difficulty and CoT refinement, and text/computation verification.Rule- and model-based filtering yields a final synthetic dataset of hundreds of billions of tokens.Source: §3.3 · p. 13

3.4 Long Context Extension

• Long-context training extends context from 8k to 32k with 80B tokens, then to 128k with another 20B, while raising RoPE base frequency and adding 25% long-context data to the mixture.The curated long-context corpus includes books, novels, and filtered repository-level code.Source: §3.4 · p. 14

3.5 Decontamination / 3.6.1 Evaluation Benchmarks and Configurations

• Training data is decontaminated against benchmark test sets, and evaluation compares the base model with open-source MoE baselines across general, reasoning, math, and coding tasks under identical pipelines.Web and code documents with 13-gram test-set overlap are removed; synthetic and QA data use semantic and lexical similarity filters. Benchmarks include MMLU, GPQA, GSM8K, and HumanEval+, among others.Source: §3.5 · p. 14, §3.6.1 · p. 14

3.6.2 Evaluation Results

• LongCat-Flash Base is competitive with leading base models, surpassing Llama-4-Maverick on nearly all benchmarks and matching DeepSeek-V3.1 across domains while outperforming it on many reasoning, math, and coding benchmarks.It is modestly behind Kimi K2 Base on general tasks but reaches parity or superiority in reasoning, math, and coding; Table 2 reports the benchmark comparison.Source: Table 2 · p. 15, §3.6.2 · p. 15

4.1 Reasoning and Coding

• Reasoning and coding data combine persona-guided math problem generation, verified solutions, filtered coding queries and tested software-engineering trajectories, plus difficulty-balanced logical reasoning tasks.Math personas span STEM disciplines and are selected for diversity with MinHash; answers are cross-checked across LLMs and scored for logical soundness. Coding sources include public datasets, GitHub snippets [Wei et al., 2024], forums, and Code Evol-Instruct [Luo et al., 2024]; Docker tests validate issue-fixing trajectories. Logic tasks include LogicPro [Jiang et al., 2025], PODA [Wang et al., 2025b], and Zebra-style puzzles.Source: §4.1 · p. 16

4.2 Agentic Tool Use

• Agentic task synthesis targets tool-set, information-processing, and user-interaction complexity, using a multi-agent framework and response selection to build cold-start training data.The ToolSetAgent enumerates 40 domains and 1,600 applications to construct 80,000 mock tools; selected tasks are retained for further post-training and exploration.Source: §4.2 · pp. 16–17

4.3 General Capability

• General-capability post-training covers constrained instruction following, long-context comprehension, and safety, including reverse-generated queries, knowledge-boundary-aware refusals, and categorized response policies.Safety data classifies queries into more than 40 categories and maps them to five response types, with human verification and evaluation.Source: §4.3 · p. 17

4.4.1 Evaluation Benchmarks and Configurations

• Evaluation compares LongCat-Flash with contemporary non-thinking chat models across benchmark categories, using category-specific scoring and sandboxed coding tests.General-domain scoring uses a model rather than exact match; instruction-following checks rule compliance, while GPQA-diamond uses a scoring model, DROP uses F1, and GraphWalks uses precision. Coding scores average pass-or-fail task results, with SWE-Bench-Verified runs limited to 100 iterations.Source: §4.4.1 · pp. 18–19

4.4.2 Evaluation Results

• LongCat-Flash demonstrates leading performance across domains, often outperforming contemporary models on challenging tasks with relatively fewer activated parameters.The evaluation covers general domains, instruction following, mathematics, general reasoning, coding, agentic tool use, and safety.Source: §4.4.2 · p. 19, Table 3 · p. 20
• LongCat-Flash scores 89.71 on MMLU and 90.44 on CEval; it ranks second on ArenaHard-V2 with 86.50 and achieves top-tier AIME25 (61.25) and BeyondAIME (43.00).It achieves the highest IFEval score (89.65), best COLLIE score (57.10) and Meeseeks-zh score (43.03), and scores 96.40 on MATH500.Source: §4.4.2 · p. 19, Table 3 · p. 20
• LongCat-Flash scores 89.30 on ZebraLogic and 79.06 on DROP, while GPQA-diamond (73.23) and GraphWalks (51.05) remain opportunities for improvement.The report highlights structured-data analysis in extremely long contexts as a particular improvement area.Source: §4.4.2 · p. 19, Table 3 · p. 20
• LongCat-Flash ranks second on TerminalBench with 39.51, scores 60.4 on SWE-Bench-Verified, and achieves the highest VitaBench score, 24.30; its safety performance is particularly strong on Harmful and Criminal risks.On Humaneval+ and MBPP+, the report describes performance as solid but with room for optimization.Source: §4.4.2 · pp. 19–20, Table 3 · p. 20

5 Training Infrastructures

• Near-linear scaling and 98.48% availability enabled LongCat-Flash training to finish within 30 days.All 20 faults were handled automatically without manual intervention; recovery time was reduced to <10 minutes.Source: §5 · p. 21, §5.4 · p. 23

5.1 Numerical Precision Control and Fault Detection

• Operator precision checks and in-place recomputation of sensitive FlashAttention Gradients detect numerical errors and potential silent data corruption.Recomputation intervals are manually adjustable to trade detection coverage against computational cost. Operator precision control alone is insufficient: loss discrepancies of 1e-3∼1e-4 coincided with benchmark variation larger than 5 pp, and cost-effective impact evaluation remains open.Source: §5.1 · p. 21

5.2 Kernel Optimization for Determinism and Performance

• Deterministic kernel redesign makes training computation reproducible while limiting performance costs across key backward-pass operations.The deterministic FAG kernel achieves 1.6x the performance of the original deterministic version and 0.95x that of the non-deterministic version; hierarchical ScatterAdd reaches performance parity with the non-deterministic version.Source: §5.2 · p. 21
• Optimized Grouped GEMM yields 5%–45% speedups, while fused GroupedGemmAdd achieves 3.12x to 3.86x speedup on its benchmark.Grouped GEMM uses pipelining, diagonal tiling, and bandwidth control to overlap computation with dispatch/combine communication; fused GEMM addition avoids intermediate write-backs and BF16-to-HBM precision loss.Source: §5.2 · p. 22

5.3 Distributed Strategy for Large-scale Training

• Expert parallelism and chunked ScMoE overlap dispatch/combine communication with dense FFN computation and with each other.Pipelined all-gather/reduce-scatter is the primary deterministic communication solution, reducing the proportion of time in non-overlapping dispatch/combine communication from 25.3% to 8.4%.Source: §5.3 · p. 22, Figure 8 · p. 22
• The V-ZB pipeline strategy balances memory use across stages and reduces peak training memory to less than 60GB.Post-validation achieves zero theoretical bubbles, and using backup data from the previous step during optimizer-state rollback preserves bitwise alignment.Source: §5.3 · pp. 22–23

5.4 Reliability and Observability

• Fine- and coarse-grained profiling and metric monitoring expose pipeline bubbles, inter-rank communication waits, and model-state changes.The platform tracks loss, weights, gradients, and activations to support rapid model-state assessment.Source: §5.4 · p. 23

6.1.1 Computation and Communication Orchestration

• SBO hides communication within a single batch using four stages, overlapping all-to-all dispatch and combine with Dense FFN, attention, and MoE computation.Wide expert parallelism, GPUDirect RDMA, custom kernels, and TP2 or TP4 instead of TP8 further reduce communication overhead and memory footprint.Source: §6.1.1 · p. 24

6.1.2 Speculative Decoding

• MTP achieves approximately 90% draft-token acceptance, while a lightweight single-dense-layer head and C2T filtering target draft cost and target-verification overhead.The single dense layer optimizes the latency–acceptance trade-off relative to ScMoE layers; C2T [Huo et al., 2025] filters tokens unlikely to be accepted before verification.Source: §6.1.2 · p. 24

6.1.3 Reducing KV Cache

• MLA with 64 heads reduces attention computation and KV-cache storage and bandwidth pressure, with an MQA-like structure sharing KV across heads.The structure aligns with WGMMA instruction shapes for hardware utilization.Source: §6.1.3 · p. 24

6.2.1 Minimize Schedule Overhead

• Fusing target forward, verification, and draft forward into a CUDA graph and scheduling multiple forward steps hides CPU scheduling and synchronization overhead.Induction establishes safe next-iteration KV-cache allocation without knowing the current accept length, while allocated cache size converges.Source: §6.2.1 · p. 25

6.2.2 Custom Kernel

• SwapAB maps weights as left-hand matrices to use flexible 8-element granularity, while custom NVLink Sharp communication kernels outperform NCCL and MSCCL++ across 4KB–96MB messages.The communication kernels support uniform and nonuniform token distributions using only four thread blocks; NCCL [NVIDIA] and MSCCL++ [Shah et al., 2025] are the baselines.Source: §6.2.2 · p. 26

6.2.3 Quantization

• Layer-wise mixed-precision quantization intersects extreme-activation screening with block-wise FP8 error analysis, yielding substantial accuracy improvements.The scheme uses activations per [1,128] blocks and weights per [128,128] blocks.Source: §6.2.3 · p. 26

6.3.1 Measured Performance

• LongCat-Flash achieves higher generation throughput and speed than DeepSeek-V3 across sequence lengths, while action-command generation reaches nearly 100 tokens/s at $0.7 per million output tokens.The comparison uses DeepSeek-V3-profile [2025a] and DeepSeek-V3-blog [2025b]; the cost assumes an H800 at $2 per hour.Source: Table 6 · p. 26, §6.3.1 · pp. 26–27

6.3.2 Theoretical Performance

• At batch size 96, measured TPOT is approximately 26 ms, about 61.5% of theoretical and on par with DeepSeek-V3 (~64%); despite slightly longer per-layer time, fewer layers yield lower overall latency.LongCat-Flash’s second MLA is overlapped by all-to-all combine, permitting some sequence-length increase without substantial latency increase.Source: §6.3.2 · pp. 27–28

• The 560B-parameter MoE combines context-aware dynamic computation and shortcut connections to enable high efficiency in training and inference.Source: §7 · p. 28
• Integrated strategies are designed to ensure stable large-scale training.Source: §7 · p. 28
• A multi-stage training pipeline cultivates agentic capabilities for complex tasks requiring iterative reasoning and environmental interaction.Source: §7 · p. 28
• The open-source release aims to advance research in efficient MoE architectures, high-quality data strategies, and agentic model development through community-driven innovation.Source: §7 · p. 28

A.1 Statistics and Case Studies of Dynamic Routing

• English tokens receive a computational bias over Chinese and mathematical tokens in average activated FFN experts across benchmarks.Figure 11 presents the benchmark-level averages for the LongCat-Flash base model.Source: §A.1 · p. 35, Figure 11 · p. 35
• In Layer 1, function words consistently receive fewer computational resources.The examples include articles, conjunctions, and prepositions.Source: §A.1 · p. 35, Table 8 · p. 36
• In Layer 1, numbers and punctuation marks consistently receive fewer computational resources.Table 8 gives per-token activated FFN experts across layers for several cases.Source: §A.1 · p. 35, Table 8 · p. 36
• Layer 28 shows less specialized feature allocation than Layer 1, although identifiable patterns remain.The layer-wise cases are presented in Table 8.Source: §A.1 · p. 35, Table 8 · p. 36
• In the Chinese text case, tokens preceding punctuation marks tend to receive fewer computational resources.This pattern is reported for the final layer, Layer 28.Source: §A.1 · p. 35, Table 8 · p. 36
• The authors hypothesize that shallow layers prioritize token-internal semantics, while deeper layers adjust resources according to predictive complexity.They interpret this as a possible shift from local feature processing toward global prediction optimization.Source: §A.1 · p. 35