PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Donghao Li, Dongjie Ji, Erhang Li, Fang Wei, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanting Chen, Guoai Cao, Guolai Meng, Guowei Li, Han Yu, Han Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoling Zhang, Haoming Luo, Haoran Wei, Haotian Yuan, Haowei Zhang, Haowen Luo, Haoyu Chen, Haozhe Ji, Hengqing Zhang, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J Yang, JQ Zhu, Jia Luo, Jia Song, Jia Yu, Jialiang Huang, Jialu Cai, Jian Liang, Jiangting Zhou, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jieyu Yang, Jin Chen, Jin Yan, Jingchang Chen, Jingli Zhou, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jingzi Zhou, Jinhua Zhu, Jiping Yu, Joseph Sun, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junmin Zheng, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Leyi Xia, Li Zhang, Liang Zhao, Lihua Guo, Lingxiao Luo, Linwang Ma, Linyan Zhu, Litong Wang, Liyu Cai, Liyue Zhang, Longhao Chen, MS Di, MY Xu, Max Mei, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Mingxu Zhou, Minmin Han, Ning Wang, Panpan Huang, Panpan Wang, Peixin Cong, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qiwei Jiang, Rui Tian, Ruifan Xu, Ruijie Lu, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqian Chen, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, Ruyi Chen, SH Liu, Shanghao Lu, Shangmian Sun, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoqing Wu, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Shuying Yu, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tun Wang, W Zhang, WL Xiao, Wangding Zeng, Wei An, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Yang, Wenlve Huang, Wenqing Hou, Wentao Zhang, Wenting Ma, Xi Gao, Xiang He, Xiangwen Wang, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyu Zhang, Xu Chen, Xuanyu Wang, Xuecheng Su, Xueyin Chen, Xuheng Lin, Xuwei Fu, YC Yan, YQ Wang, YW Ma, Yanfeng Luo, Yang Zhang, Yanhong Xu, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Xu, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Shao, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yijia Wu, Yiliang Xiong, Yiling Ma, Ying He, Ying Tang, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, YuKun Li, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuanhao Li, Yuduan Wang, Yuehan Yang, Yuer Xu, Yuhan Wu, Yuhao Meng, Yuheng Zou, Yukun Zha, Yunfan Xiong, Yupeng Chen, Yuping Lin, Yuqian Cao, Yuqian Wang, Yushun Zhang, Yuting Yan, Yutong Lin, Yuxian Gu, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhen Huang, ZF Wu, Zehao Wang, Zehua Zhao, Zehui Ren, Zekai Zhang, Zhangli Sha, Zhe Fu, Zhe Ju, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zheren Gao, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhixuan Chen, Zhiyu Wu, Zhizhou Ren, Zhongyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihua Qu, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Ziyi Wan, Zizheng Pan, Zongqing Yao

arXiv:2606.19348v1cs.CLcs.AI

TL;DR

Vanilla attention’s quadratic complexity constrains ultra-long-context processing, while long-horizon scenarios make efficient context support critical. DeepSeek-V4 combines CSA and HCA with other architecture and optimization changes; at one million tokens, Pro uses 27% of DeepSeek-V3.2’s single-token inference FLOPs and 10% of its KV cache.

  • Problem

    Vanilla attention’s quadratic complexity constrains ultra-long-context processing, while long-horizon scenarios make efficient support for these contexts critical.

    Source: §1 · p. 4

  • Method

    DeepSeek-V4 combines Compressed Sparse Attention and Heavily Compressed Attention for long-context efficiency, with mHC residual connections and Muon optimization.

    Source: §1 · p. 4, §2 · p. 7

  • Results

    At 1M-token context, DeepSeek-V4-Pro uses 27% of DeepSeek-V3.2’s single-token inference FLOPs and 10% of its KV cache.

    Source: §1 · p. 5

  • Takeaways & Limitations

    DeepSeek-V4 efficiently supports one-million-token contexts, supporting long-horizon tasks and further test-time scaling.

    Source: §1 · p. 4, §6 · p. 44

    The architecture remains relatively complex, and the principles underlying Anticipatory Routing and SwiGLU Clamping remain insufficiently understood.

    Source: §6 · p. 44

Abstract

from arXiv · show

We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold-Constrained Hyper-Connections (mHC) that enhance conventional residual connections; (3) and the Muon optimizer for faster convergence and greater training stability. We pre-train both models on more than 32T diverse and high-quality tokens, followed by a comprehensive post-training pipeline that unlocks and further enhances their capabilities. DeepSeek-V4-Pro-Max, the maximum reasoning effort mode of DeepSeek-V4-Pro, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks. Meanwhile, DeepSeek-V4 series are highly efficient in long-context scenarios. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token contexts, thereby making long-horizon tasks and further test-time scaling more feasible. The model checkpoints are available at https://huggingface.co/collections/deepseek-ai/deepseek-v4.

1. Introduction

DeepSeek-V4 targets the attention-efficiency bottleneck that limits test-time scaling and long-horizon tasks, introducing model, architecture, and training changes to support million-token contexts. The paper reports efficiency gains over DeepSeek-V3.2 and broad evaluation results for the Pro and Flash models.

1. Introduction

  • DeepSeek-V4 includes preview versions of Pro (1.6T parameters, 49B activated) and Flash (284B parameters, 13B activated), with the series supporting a context length of one million tokens.Source: §1 · p. 4
  • The series retains DeepSeekMoE and Multi-Token Prediction while adding hybrid CSA/HCA attention, mHC residual connections, and Muon optimization.CSA compresses KV caches along the sequence dimension before applying sparse attention; HCA compresses them more aggressively but retains dense attention. mHC upgrades conventional residual connections, while Muon supports faster convergence and improved training stability.Source: §1 · p. 4
  • Infrastructure optimizations target MoE execution, reproducible kernels, training recomputation and parallelism, shared-prefix KV reuse, and post-training FP4 quantization-aware training.They include a fused MoE kernel, tensor-level checkpointing, hybrid ZeRO for Muon, recomputation and fused kernels for mHC, two-stage contextual parallelism, and on-disk KV-cache storage.Source: §1 · p. 4
  • At 1M tokens, DeepSeek-V4-Pro uses 27% of DeepSeek-V3.2’s single-token inference FLOPs and 10% of its KV cache; Flash uses 10% of the FLOPs and 7% of the KV cache.FLOPs are measured in equivalent FP8 FLOPs. Routed expert parameters use FP4 precision, which the paper says could theoretically enable 1/3 greater efficiency on future hardware.Source: §1 · p. 5
  • DeepSeek-V4-Flash-Base, pretrained on 32T tokens, surpasses DeepSeek-V3.2-Base on most benchmarks, while Pro-Base, trained on 33T tokens, shows comprehensive superiority across four task areas.The reported areas are reasoning, coding, long-context, and world knowledge; both models natively support 1M-token contexts after pretraining.Source: §1 · p. 5
  • Post-training independently cultivates domain experts with SFT and GRPO-based reinforcement learning, then consolidates them into one model through on-policy distillation.The unified student optimizes reverse KL loss against the expert teachers.Source: §1 · p. 5

Summary of Core Evaluation Results

  • DeepSeek-V4-Pro-Max leads open models in broad knowledge and reasoning evaluations and excels on million-token long-context benchmarks, but trails frontier proprietary models on some evaluations.It significantly outperforms leading open models on SimpleQA and Chinese-SimpleQA, while slightly trailing Gemini-3.1-Pro on knowledge evaluations; on standard reasoning benchmarks it exceeds GPT-5.2 and Gemini-3.0-Pro but falls marginally short of GPT-5.4 and Gemini-3.1-Pro. DeepSeek-V4-Flash-Max achieves comparable reasoning performance with a larger thinking budget, but trails Pro on harder agent tasks.Source: Summary of Core Evaluation Results · pp. 5–6

2. Architecture

DeepSeek-V4 retains the Transformer and Multi-Token Prediction foundations while introducing mHC, hybrid attention, and Muon as key upgrades over DeepSeek-V3.

2. Architecture

  • mHC strengthens conventional residual connections in the Transformer architecture (Vaswani et al., 2017; Xie et al., 2026).Source: §2 · p. 6
  • The hybrid architecture combines Compressed Sparse Attention and Heavily Compressed Attention to improve long-context efficiency.Source: §2 · p. 7
  • Muon (Liu et al., 2025) is used as the optimizer, while DeepSeekMoE components receive only minor adjustments from DeepSeek-V3.Source: §2 · p. 7
  • The Multi-Token Prediction configuration remains identical to DeepSeek-V3 (DeepSeek-AI, 2024; Gloeckle et al., 2024; Li et al., 2024; Qi et al., 2020).Source: §2 · p. 7

2.1. Designs Inherited from DeepSeek-V3

DeepSeek-V4 carries forward DeepSeekMoE FFNs and MTP from earlier DeepSeek models, while revising several MoE design choices and leaving the MTP strategy unchanged.

2.1. Designs Inherited from DeepSeek-V3

  • DeepSeek-V4 retains DeepSeekMoE fine-grained routed and shared FFNs but revises affinity scoring, balancing, routing constraints, parallelism, and early-block FFNs.It changes the affinity activation from Sigmoid(·) to Sqrt(Softplus(·)), adds slight sequence-wise balance loss to auxiliary-loss-free balancing (DeepSeek-AI, 2024; Wang et al., 2024a), removes the routing-target-node constraint, and redesigns parallelism. Initial dense FFNs become hash-routed MoE layers, with target experts determined by input token ID (Roller et al., 2021).Source: §2.1 · p. 7
  • DeepSeek-V4 uses the same MTP modules and objectives as DeepSeek-V3, without modifying its validated strategy.The paper states that DeepSeek-V3 had validated this MTP strategy.Source: §2.1 · p. 7

2.2. Manifold-Constrained Hyper-Connections

DeepSeek-V4 uses Manifold-Constrained Hyper-Connections (mHC) to strengthen residual connections: compared with naive Hyper-Connections (HC), mHC constrains residual mappings to stabilize signal propagation while preserving expressivity. The section describes HC’s expanded residual stream, mHC’s constraints, and how its mappings are generated and constrained.

Standard Hyper-Connections

  • Standard HC expands the residual stream by n_hc and uses input, residual, and output mappings, but stacking layers frequently causes numerical instability that hinders scaling.The expanded stream is decoupled from the layer’s hidden size, so the actual layer input remains d-dimensional; n_hc is typically much smaller than d.Source: §2.2 · pp. 7–8

Manifold-Constrained Residual Mapping

  • mHC constrains each residual mapping to the manifold of doubly stochastic matrices, making it non-expansive and improving stability in forward and backward passes, including deep stacks.Its input and output mappings are also constrained to be non-negative and bounded to avoid signal cancellation.Source: §2.2 · p. 8

Dynamic Parameterization

  • mHC’s three mappings are dynamically generated from input-dependent and input-independent components after the residual state is flattened and RMS-normalized.Source: §2.2 · p. 8

Applying Parameter Constraints

  • The input and output mappings are bounded with a Sigmoid, while the residual mapping is projected onto the doubly stochastic manifold using Sinkhorn-Knopp with 20 iterations.Source: §2.2 · p. 8

2.3. Hybrid Attention with CSA and HCA

DeepSeek-V4 interleaves Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), compressing KV state at different scales and using sparse attention selectively to reduce long-context cost and make one-million-token contexts feasible.

2.3.1. Compressed Sparse Attention

  • CSA compresses every m KV entries into one, selects top-k compressed blocks, then performs shared-KV MQA for core attention.CSA also uses grouped output projection to reduce the cost of projecting its attention outputs.Source: §2.3.1 · pp. 9–11

2.3.2. Heavily Compressed Attention

  • HCA compresses every m′ (≫m) KV entries into one without sparse attention or overlap, reducing sequence length to 1.Like CSA, it uses shared-KV MQA and grouped output projection.Source: §2.3.2 · pp. 11–12

2.3.3. Other Details

  • Both attention types add per-head RMSNorm and partial RoPE; a sliding-window branch restores recent-token access, while learnable attention sinks let heads reduce total attention mass.RoPE is applied to the last 64 dimensions, including attention outputs with position −i to encode relative position; the window supplies uncompressed KV entries for recent tokens.Source: §2.3.3 · pp. 12–13

2.3.4. Efficiency Discussion

  • At 1M context, DeepSeek-V4 KV cache is approximately 2% of the BF16 GQA8 baseline with head dimension 128.The efficiency discussion attributes the reductions to hybrid attention and low-precision computation and storage: RoPE dimensions use BF16, other KV dimensions use FP8, and lightning-indexer attention uses FP4. It also reports substantial inference-FLOP and KV-cache advantages over DeepSeek-V3.2.Source: §2.3.4 · pp. 13–14

2.4. Muon Optimizer

DeepSeek-V4 uses Muon for most modules to improve convergence and training stability, with AdamW retained for specified components and a two-stage hybrid Newton–Schulz method for orthogonalization.

Muon Optimizer

  • Muon (Liu et al., 2025) updates most DeepSeek-V4 modules for faster convergence and improved training stability.The full Muon algorithm is summarized in Algorithm 1.Source: §2.4 · p. 14

Basic Configurations

  • AdamW remains for embeddings, prediction heads, specified mHC parameters, and RMSNorm weights; Muon updates all other modules.Muon uses weight decay, Nesterov, and RMS update rescaling to reuse AdamW hyperparameters; unlike Liu et al. (2025), it uses hybrid Newton–Schulz orthogonalization.Source: §2.4 · p. 14

Hybrid Newton-Schulz Iterations

  • Hybrid Newton–Schulz orthogonalization uses 10 iterations, switching coefficients after step 8 to stabilize singular values at 1.The first 8 steps use (3.4445, −4.7750, 2.0315) for rapid convergence; the final 2 use (2, −1.5, 0.5).Source: §2.4 · p. 14

Avoiding Exploding Attention Logits

  • Query and KV-entry RMSNorm prevents exploding attention logits, so the Muon optimizer omits QK-Clip (Liu et al., 2025).The DeepSeek-V4 attention architecture permits direct RMSNorm on attention queries and KV entries.Source: §2.4 · p. 14

3. General Infrastructures

This section presents infrastructure for efficient DeepSeek-V4 training and inference, including overlapping MoE communication and computation and kernels supporting its architecture. It also covers co-design of sparse-attention kernels and KV-cache storage, including on-disk reuse of shared-prefix state and differing strategies for compressed CSA/HCA and uncompressed SWA entries.

3.1. Fine-Grained Communication-Computation Overlap in Expert Parallelism

  • The fine-grained EP scheme achieves 1.50 ∼1.73× speedup for general inference workloads and up to 1.96× for latency-sensitive scenarios on NVIDIA GPUs and HUAWEI Ascend NPUs.The CUDA-based mega-kernel implementation, MegaMoE2, is open-sourced. Expert-wave scheduling overlaps computation, token transfer, and result sending; for DeepSeek-V4-Pro, each GBps of interconnect bandwidth suffices to hide communication for 6.1 TFLOP/s of compute.Source: §3.1 · pp. 15–16, Figure 5 · p. 15

3.2. Flexible and Efficient Kernel Development with TileLang

  • TileLang replaces most fine-grained Torch ATen operators with fused kernels, while Host Codegen reduces CPU-side validation overhead from tens or hundreds of microseconds to less than one microsecond per invocation.TileLang integrates Z3 for formal integer analysis, with compilation overhead limited to a few seconds; under conservative defaults, its accuracy- and reproducibility-oriented choices remain performance-competitive.Source: §3.2 · pp. 16–18

3.3. High-Performance Batch-Invariant and Deterministic Kernel Libraries

  • The kernel library provides end-to-end bitwise batch-invariant and deterministic kernels with minimal performance overhead, using dual-kernel attention and DeepGEMM for matrix multiplication.For deterministic backward passes, it uses separate per-SM attention-gradient buffers with a global summation, and token-order preprocessing plus buffer isolation for MoE.Source: §3.3 · pp. 18–19

3.4. Training Framework / 3.4.1. Efficient Implementation of Muon

  • To adapt the DeepSeek-V3 training foundation to DeepSeek-V4, the framework adds a hybrid ZeRO bucket strategy for Muon, which requires full gradient matrices for updates.Dense-parameter buckets are balanced across ranks and typically incur less than 10% padding memory overhead; MoE gradients are synchronized in BF16, halving communication volume, then summed locally in FP32.Source: §3.4 · p. 19, §3.4.1 · pp. 19–20

3.4.2. Cost-Effective and Memory-Efficient Implementation of mHC / 3.4.4. Extended Automatic Differentiation for Flexible Activation Checkpointing

  • mHC's additional memory and pipeline communication are managed with fused kernels, selective recomputation, and adjusted DualPipe, while tensor-level checkpointing automatically builds minimal recomputation graphs.The mHC optimizations limit wall-time overhead to 6.7% of the overlapped 1F1B pipeline stage; tensor-level checkpointing introduces no additional training overhead compared with manual implementation.Source: §3.4.2 · p. 20, §3.4.4 · p. 21

3.4.3. Contextual Parallelism for Long-Context Attention

  • For CSA/HCA context parallelism, ranks exchange boundary KV entries before compression, then all-gather and select/pad compressed entries; query-visible ranges use rules or CSA's top-k selector.This addresses variable compressed KV lengths across ranks and compression groups that straddle rank boundaries.Source: §3.4.3 · pp. 20–21

3.5. Inference Framework / 3.5.1. KV Cache Structure and Management

  • The inference framework uses a customized KV cache layout for hybrid attention, separating classical compressed KV entries from a fixed-size state cache for SWA and uncompressed tail tokens.The state-cache pool is dynamically assigned to sequences; the classical cache uses multiple blocks per request, while compression and sliding-window attention produce layer-dependent cache sizes and policies.Source: §3.5.1 · pp. 21–23, Figure 6 · p. 22

3.5.1. KV Cache Structure and Management

  • Variable tokens per block avoid performance degradation across layers, while cache-line-aware KV layout and kernel co-design allow block sizes in multiples of lcm(m, m′).For compression ratios m in CSA and m′ in HCA, the number of original tokens per block can be any multiple of their least common multiple.Source: §3.5.1 · p. 23

3.5.2. On-Disk KV Cache Storage

  • On-disk KV caching avoids repeated prefilling for shared-prefix requests, with separate management for compressed CSA/HCA entries and uncompressed SWA entries.For CSA/HCA, complete compressed blocks are read and reused, while tokens in the tail incomplete block are recomputed because uncompressed entries are not stored.Source: §3.5.2 · p. 23
  • SWA KV entries are approximately 8 times larger than compressed CSA/HCA entries, so three on-disk strategies trade storage overhead against recomputation.Full caching stores all SWA entries; periodic checkpointing stores the last n_win entries every p tokens; zero caching stores none and recomputes n_win · L tokens to restore the last n_win entries for an L-layer model. The deployment scenario determines the selected strategy.Source: §3.5.2 · pp. 23–24

4. Pre-Training

DeepSeek-V4 is pretrained on a diverse corpus exceeding 32T tokens, using long-context training setups and stability techniques for its Flash and Pro models. Base-model evaluations report broad gains for Flash over V3.2 and near-universal dominance for Pro over both predecessors.

4.1. Data Construction

  • The pretraining corpus exceeds 32T tokens and emphasizes diverse, high-quality sources, including math, code, multilingual material, and curated long documents.Web filtering removes batched auto-generated and templated content, while agentic data supports coding; preprocessing also packs documents to reduce truncation and uses sample-level attention masking.Source: §4.1 · p. 24

4.2.1. Model Setups

  • Flash uses 284B total parameters with 13B activated per token, while Pro uses 1.6T total with 49B activated; both interleave CSA and HCA in most layers.Flash uses sliding-window attention in its first two layers, while Pro uses HCA; both use MoE layers throughout, with six routed experts activated per token, and mHC expansion factor 4.Source: §4.2.1 · pp. 24–25

4.2.2. Training Setups

  • Both models use Muon for most parameters and AdamW for embeddings, prediction heads, and RMSNorm weights, while training sequence lengths grow from 4K to 1M.Flash trains on 32T tokens and Pro on 33T; both introduce sparse attention at 64K after dense-attention warmup, with Pro using a longer dense-attention stage.Source: §4.2.2 · pp. 25–26

4.2.3. Mitigating Training Instability

  • To mitigate MoE-related loss spikes, DeepSeek-V4 dynamically uses Anticipatory Routing and applies SwiGLU clamping during training.Anticipatory Routing computes routing indices with historical parameters and adds approximately 20% wall-clock overhead when active, while automatic detection triggers it after a loss spike; SwiGLU linear values are clamped to [−10, 10] and gate values capped at 10.Source: §4.2.3 · p. 26

4.3.1. Evaluation Benchmarks

  • Base-model evaluation covers world knowledge, language understanding and reasoning, coding and mathematics, and long-context processing.The benchmark set includes LongBench-V2 for long-context processing.Source: §4.3.1 · p. 27

4.3.2. Evaluation Results

  • DeepSeek-V4-Pro-Base improves across almost all categories and outperforms both V3.2-Base and Flash-Base, while Flash-Base surpasses V3.2-Base across many benchmarks.The reported advantages are especially evident for Flash in world knowledge and challenging long-context tasks, and for Pro in knowledge-intensive evaluations, long-context understanding, reasoning, and coding.Source: §4.3.2 · p. 27, Table 1 · p. 28

5. Post-Training · 5.1. Post-Training Pipeline · 5.2. Post-Training Infrastructures

DeepSeek-V4 replaces the prior mixed-RL stage with on-policy distillation to consolidate specialist capabilities, supported by post-training infrastructure for efficient long-context training and agentic tasks. The pipeline combines specialized training and model/tool interfaces with quantization, scalable rollout and teacher systems, and a production sandbox platform.

5.1. Post-Training Pipeline; 5.1.2. On-Policy Distillation

  • Multi-teacher on-policy distillation replaces mixed RL as the final-model merging stage, consolidating over ten domain experts into one student through on-policy trajectories and full-vocabulary logit alignment.The student learns from expert output distributions on its own generated trajectories, selecting the relevant expert by task; full-vocabulary reverse-KL distillation is used to improve gradient stability and faithful transfer.Source: §5.1 · p. 28, §5.1.2 · pp. 32–33

5.1.1. Specialist Training

  • Specialists are fine-tuned and trained with domain-specific GRPO, while distinct RL configurations support three reasoning-effort modes and rubric-guided generative reward evaluation.The actor also serves as the generative reward model, jointly learning evaluation and generation with limited diverse human annotations; the pipeline adds an XML-based |DSML| tool-call schema and Quick Instruction tokens that reuse KV cache for auxiliary tasks.Source: §5.1.1 · pp. 28, 29, 30, 32
  • Interleaved thinking preserves full reasoning history across user turns in tool-calling workflows, but discards prior-turn reasoning in general conversation to keep context concise.This addresses token waste and state reconstruction in complex agentic workflows; simulated tool interactions sent as user messages may not activate persistence, for which non-think models are recommended.Source: §5.1.1 · pp. 31–32

5.2. Post-Training Infrastructures; 5.2.1. FP4 Quantization-Aware Training

  • FP4 quantization-aware training targets MoE expert weights and the CSA indexer’s QK path; BF16 index scores yield 2× top-k-selector speedup with 99.7% KV-entry recall.Native FP4 weights are used during inference and RL rollout, aligning sampling with deployment while reducing memory loading and consumption; the infrastructure builds on DeepSeek-V3.2’s distributed training stack and rollout engine.Source: §5.2.1 · pp. 33–34, §5.2 · p. 33

5.2.2. Efficient Teacher Scheduling for Full-Vocabulary OPD

  • Full-vocabulary OPD scales to an effectively unbounded number of potentially trillion-parameter teachers by loading sharded weights and reconstructing logits from cached final-layer hidden states.Samples are grouped by teacher to keep at most one prediction head resident on-device; asynchronous loading and a specialized TileLang kernel support exact teacher–student KL computation.Source: §5.2.2 · p. 34

5.2.3. Preemptible and Fault-Tolerant Rollout Service; 5.2.4. Scaling RL Framework for Million-Token Context

  • A token-granular write-ahead log (WAL) and saved KV cache resume interrupted rollouts without restarting, avoiding the length bias of regenerating unfinished requests.For million-token RL and OPD, lightweight rollout metadata enables global shuffling and packing, while shared-memory loading and workload-dependent minibatches reduce CPU/GPU memory pressure and balance throughput with I/O overlap.Source: §5.2.3 · p. 34, §5.2.4 · p. 35

5.2.5. Sandbox Infrastructure for Agentic AI

  • DeepSeek Elastic Compute (DSec) unifies four sandbox execution substrates and supports hundreds of thousands of concurrent instances per cluster through layered image loading and density optimizations.Its Python SDK abstracts function calls, containers, microVMs, and full VMs; 3FS-backed on-demand storage supports image assembly and versioning, while memory reclamation and reduced container spinlock contention improve packing density.Source: §5.2.5 · pp. 35–36
  • DSec’s globally ordered trajectory logs record sandbox commands and results, enabling preemption-safe resumption, provenance tracking, and deterministic replay.On task resumption, cached results fast-forward completed commands and avoid re-executing non-idempotent operations.Source: §5.2.5 · p. 36

5.3. Standard Benchmark Evaluation

DeepSeek-V4-Pro-Max leads open models on knowledge and reasoning benchmarks, while the series shows strong agent performance and long-context retrieval. The evaluation covers standard benchmarks, agentic and formal-math settings, and contexts up to one million tokens.

5.3.1. Evaluation Setup

  • Codeforces evaluation averages expected ratings across 14 Division 1 contests (114 problems), sampling 10 of 32 candidate solutions per problem; formal-math submissions count only when the strict verifier Comparator accepts them.Formal-math evaluation used Lean v4.28.0-rc1 with up to 500 tool calls; code-agent evaluation allowed up to 500 interaction steps and 512K-token context.Source: §5.3.1 · pp. 36–37

5.3.2. Evaluation Results

  • DeepSeek-V4-Pro-Max sets a new open-model state of the art on knowledge and reasoning, including a 20-point SimpleQA-Verified lead over open baselines, while trailing leading proprietary models.It marginally outperforms Kimi and GLM across MMLU-Pro, GPQA, and HLE; it also outperforms prior open models across reasoning benchmarks and matches state-of-the-art closed models on many metrics.Source: §5.3.2 · p. 38, Table 6 · p. 38
  • DeepSeek-V4-Pro and Flash perform comparably to GPT-5.4 on coding competitions, while Pro-Max ranks 23rd among human Codeforces candidates; formal-math results reach 120/120 in the frontier regime.In the agentic formal-math setup, DeepSeek-V4 achieves state-of-the-art results, outperforming Seed Prover; with the compute-intensive hybrid formal-informal pipeline, it surpasses Aristotle and matches the best known results.Source: §5.3.2 · p. 39, Figure 8 · p. 40
  • DeepSeek-V4-Pro performs comparably to K2.6 and GLM-5.1 on code-agent tasks, while open models lag closed-source counterparts; Pro also performs well on MCPAtlas and Toolathlon.Flash underperforms Pro on coding tasks, particularly Terminal Bench 2.0; strong results on MCPAtlas and Toolathlon indicate generalization beyond internal frameworks.Source: §5.3.2 · p. 39
  • DeepSeek-V4-Pro outperforms Gemini-3.1-Pro on MRCR and CorpusQA, with retrieval stable within 128K tokens and remaining strong at 1M despite degradation beyond 128K.Claude Opus 4.6 remains ahead of Pro on MRCR.Source: §5.3.2 · pp. 39–40, Figure 9 · p. 40
  • Max mode outperforms High on the most challenging tasks, and scaling test-time compute improves the series over DeepSeek-V3.2; Pro is more token-efficient on HLE.Max uses longer contexts and reduced length penalties in RL; Figure 10 compares performance and cost on representative reasoning and agentic tasks.Source: Table 7 · p. 39, §5.3.2 · p. 40, Figure 10 · p. 41

5.4. Performance on Real-World Tasks

The section evaluates DeepSeek-V4 on practical API and chatbot use cases using proprietary internal metrics designed around real-world usage patterns. It reports results across Chinese writing, search, white-collar productivity, and coding tasks.

5.4.1. Chinese Writing

  • 62.7% versus 34.1%: DeepSeek-V4-Pro beats Gemini-3.1-Pro overall on concise, everyday Chinese functional-writing queries.The passage attributes Gemini’s losses primarily to its stylistic preferences overriding explicit user requirements.Source: §5.4.1 · p. 41
  • Against Gemini-3.1-Pro, DeepSeek-V4-Pro achieves a 60.0% win rate in creative-writing instruction following and 77.5% in writing quality, but Claude Opus 4.5 leads on the hardest prompts.For high-complexity constraints or multi-turn scenarios, Claude Opus 4.5 achieves a 52.0% win rate versus 45.9% for DeepSeek-V4-Pro.Source: §5.4.1 · p. 41

5.4.2. Search

  • DeepSeek-V4-Pro consistently outperforms DeepSeek-V3.2 in retrieval-augmented search across objective and subjective Q&A, with its largest gains in single-value search and planning & strategy.DeepSeek-V3.2 remains relatively competitive on comparison and recommendation tasks, where balanced, multi-perspective reasoning may be a weakness for V4-Pro.Source: §5.4.2 · p. 42
  • Agentic search consistently outperforms RAG, particularly on complex tasks, while costing only marginally more than standard RAG.DeepSeek-Chat’s thinking mode was optimized to maximize response accuracy within a predefined thinking budget; its non-think mode uses RAG and its thinking mode uses agentic search.Source: §5.4.2 · p. 42

5.4.3. White-Collar Task

  • 63% non-loss rate: DeepSeek-V4-Pro-Max outperforms Opus-4.6-Max across Chinese white-collar analysis, generation, and editing tasks.Human evaluations covered 30 advanced tasks across 13 industries and assessed task completion, instruction following, content quality, and formatting aesthetics.Source: §5.4.3 · p. 42, Figure 11 · p. 43
  • DeepSeek-V4-Pro-Max’s strengths include Task Completion and Content Quality, while it trails Opus on Instruction Following and needs improvement in summarization and slide design.It often anticipates implicit intent, adds insights and self-verification, and produces coherent long-form writing; it can miss formatting constraints and struggles to condense extensive inputs.Source: §5.4.3 · pp. 42–43, Figure 12 · p. 43

5.4.4. Code Agent

  • DeepSeek-V4-Pro significantly outperforms Claude Sonnet 4.5 and approaches Claude Opus 4.5 on a 30-task internal R&D coding benchmark.The evaluation set was filtered from approximately 200 challenging tasks submitted by 50+ engineers, spanning feature development, bug fixing, refactoring, and diagnostics.Source: §5.4.4 · p. 44, Table 8 · p. 44
  • 52% of 85 DeepSeek developers and researchers said V4-Pro was ready to be their primary coding model, 39% leaned yes, and fewer than 9% said no.Respondents noted trivial mistakes, misinterpretation of vague prompts, and occasional over-thinking.Source: §5.4.4 · p. 44

6. Conclusion, Limitations, and Future Directions

DeepSeek-V4 combines hybrid attention and infrastructure optimization to support million-token contexts, with strong open-model performance. The authors identify architectural complexity and gaps in understanding training stability, and outline directions for improving efficiency and expanding capabilities.

6. Conclusion, Limitations, and Future Directions

  • DeepSeek-V4-Pro-Max leads open models on knowledge benchmarks, approaches frontier proprietary models in reasoning, and delivers competitive agent capabilities; Flash-Max matches leading closed models in reasoning cost-efficiently.The conclusion describes Pro-Max as redefining the state of the art for open models and Flash-Max as maintaining a highly cost-efficient architecture.Source: §6 · p. 44
  • Combining CSA and HCA with infrastructure optimization enables efficient native million-token contexts and supports future test-time scaling, long-horizon tasks, and online learning.The authors frame this as a foundation for emerging paradigms and a new era of million-token contexts for open models.Source: §6 · p. 44
  • The architecture is relatively complex, and the principles behind Anticipatory Routing and SwiGLU Clamping for mitigating training instability remain insufficiently understood.Future work will seek simpler essential designs without sacrificing performance, study training stability, and strengthen internal metric monitoring for more predictive stable training.Source: §6 · p. 44
  • Future efficiency work will explore sparse embedding modules and low-latency architectures and system techniques for more responsive long-context deployment and interaction.Sparse embeddings are proposed as another sparsity dimension beyond MoE and sparse attention, with the aim of improving computational and memory efficiency without compromising capability.Source: §6 · p. 45
  • The authors will continue developing long-horizon, multi-round agentic tasks, incorporate multimodal capabilities, and improve data curation and synthesis for intelligence, robustness, and usability.The stated goal is to improve performance across an increasingly broad range of scenarios and tasks.Source: §6 · p. 45

Appendix · A. Author List and Acknowledgment · B. Evaluation Details

The supplied appendix material focuses on evaluation details, covering search methods and costs, model comparisons, writing tasks, and example research outputs.

B. Evaluation Details

  • Table 9 compares agentic search with retrieval-augmented search for DeepSeek-V4-Pro.The caption identifies the comparison but does not provide its results.Source: Table 9 · p. 55
  • Table 10 compares mean costs for agentic and retrieval-augmented search, noting that most agentic-search tool calls are parallel.The caption does not specify the cost values.Source: Table 10 · p. 55
  • Table 11 compares DeepSeek-V4-Pro and DeepSeek-V3.2 on search question-answering tasks.The caption gives the evaluation scope, not the comparative outcomes.Source: Table 11 · p. 55
  • Figure 14 shows an example output for comparing two regular investment strategies for the NASDAQ.The caption describes the task example, not its quality or correctness.Source: Figure 14 · p. 56
  • Figure 15 shows an example output for researching 2020–2025 Nobel Science Prizes and generating an analytical PDF report.The caption describes the example task, not its quality or correctness.Source: Figure 15 · p. 56
  • Table 12 analyzes DeepSeek-V4-Pro and Gemini-3.1-Pro in Chinese functional writing.The caption does not report which model performs better.Source: Table 12 · p. 57
  • Table 13 analyzes DeepSeek-V4-Pro and Gemini-3.1-Pro in Chinese creative writing.The caption does not report which model performs better.Source: Table 13 · p. 58
  • Table 14 compares DeepSeek-V4-Pro with Claude-Opus-4.5 on complex instruction following and multi-turn writing.The caption identifies the tasks but does not report comparative outcomes.Source: Table 14 · p. 58

Related papers

Loading 2606.19348v1…