PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V3 Technical Report

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan

arXiv:2412.19437v2cs.CLcs.AI

TL;DR

DeepSeek-V3 addresses the challenge of building a highly capable yet economical large MoE model. It combines efficient architectures, load balancing, FP8 training, and staged post-training, achieving leading open-source performance with comparable closed-source chat performance at 2.788M H800 GPU hours.

  • Problem

    DeepSeek-V3 targets strong model performance while reducing inference and training costs for a large MoE language model.

  • Method

    The model combines MLA, DeepSeekMoE, auxiliary-loss-free load balancing, Multi-Token Prediction, FP8 training, and post-training including SFT, RL, and reasoning distillation.

  • Results

    DeepSeek-V3 outperforms other open-source models, achieves performance comparable to leading closed-source models, and requires 2.788M H800 GPU hours for full training.

  • Takeaways & Limitations

    The results establish DeepSeek-V3 as a strong open-source model with economical training and broad capabilities in knowledge, code, mathematics, and reasoning.

  • Takeaways & Limitations

    Efficient inference requires a relatively large deployment unit, which may burden small teams, and generation speed still has room for improvement.

Abstract

from arXiv · show

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

1. Introduction

DeepSeek-V3 combines efficient MoE architectures and training innovations with extensive pre-training and post-training. It delivers strong open-source benchmark performance, comparable chat performance to leading closed-source models, and economical training.

  • The model combines MLA for efficient inference and DeepSeekMoE for cost-effective training with auxiliary-loss-free load balancing and Multi-Token Prediction.
  • The system also introduces FP8 training, computation-communication co-design, and reasoning distillation from DeepSeek-R1 to improve efficiency and reasoning performance.
  • DeepSeek-V3 is pre-trained on 14.8T tokens with stable optimization, extended to 128K context, and then refined through SFT and RL.
  • Its chat version outperforms other open-source models and performs comparably to GPT-4o and Claude-3.5-Sonnet across standard and open-ended benchmarks.
  • Training costs 2.788M H800 GPU hours overall, including 2.664M hours for pre-training and 0.1M hours for later stages.

2. Architecture

DeepSeek-V3 combines MLA and DeepSeekMoE for efficient inference and economical training, adding auxiliary-loss-free load balancing and multi-token prediction. Its architecture reduces inference memory, maintains balanced routing without token dropping, and preserves causal chains across future-token predictions.

  • Multi-Head Latent Attention: MLA reduces the KV cache during inference by caching only compressed latent vectors while maintaining performance comparable to standard MHA.MLA also uses low-rank query compression to reduce activation memory during training.
  • DeepSeekMoE: DeepSeekMoE uses finer-grained routed experts, isolated shared experts, and node-limited routing to reduce communication costs and nearly overlap computation with communication.Each token is sent to at most M nodes selected from nodes containing the highest-affinity routed experts.
  • Auxiliary-Loss-Free Load Balancing: Dynamic expert biases provide auxiliary-loss-free load balancing, while a very small sequence-wise balance loss prevents extreme imbalance within individual sequences.The bias is updated after each training step according to whether an expert is overloaded or underloaded.
  • Auxiliary-Loss-Free Load Balancing: DeepSeek-V3 maintains balanced loads throughout training and inference, so it does not drop tokens in either setting.The reported balance is achieved through the load-balancing strategy and deployment strategies for inference.
  • Multi-Token Prediction: Multi-Token Prediction sequentially predicts additional future tokens while preserving the complete causal chain at each prediction depth.The implementation uses sequential modules with shared embedding and output heads, extending the prediction scope beyond the next token.

3. Infrastructures

DeepSeek-V3’s infrastructure combines DualPipe scheduling, FP8 mixed-precision training, and communication-aware engineering to reduce pipeline and cross-node communication overhead. The design also exposes hardware limitations that constrain efficiency and motivate future accelerator support.

  • 3.1. Compute Clusters: The training system combines 16-way pipeline parallelism, 64-way expert parallelism across 8 nodes, and ZeRO-1 data parallelism on 2048 H800 GPUs.Nodes use NVLink and NVSwitch internally, while InfiniBand connects different nodes.
  • 3.2.1. DualPipe and Computation-Communication Overlap: DualPipe overlaps forward and backward computation with communication, reducing pipeline bubbles and addressing cross-node expert-parallelism overhead.It rearranges attention, all-to-all dispatch and combine, MLP, backward-input, backward-weight, and pipeline-communication components within paired chunks.
  • 3.2.1. DualPipe and Computation-Communication Overlap: Compared with ZB1P and 1F1B, DualPipe significantly reduces pipeline bubbles while increasing peak activation memory by only 1 PP times.It requires two model-parameter copies, but the large expert-parallel size limits the resulting memory increase.
  • 3.3. FP8 Training: FP8 training uses most GEMM operations in FP8 while retaining selected operations in higher-precision formats to balance computational efficiency and numerical stability.Fine-grained scaling and precise accumulation address FP8 dynamic-range and accumulation limitations; the framework also compresses cached activations and optimizer states.
  • 3.3. FP8 Training: FP8 communication quantizes activations before MoE dispatch and activation gradients before MoE down-projections, reducing communication bandwidth pressure.The strategy uses power-of-two scaling factors for relevant activations and gradients.
  • 3.5. Compute Hardware: Current hardware limits the efficiency of communication overlap and fine-grained FP8 quantization because communication consumes SMs and quantization requires frequent Tensor Core–CUDA core data movement.The authors recommend communication co-processors, native group scaling, fused FP8 casting with memory transfer, and direct transposed GEMM support.

4. Pre-Training

DeepSeek-V3’s pre-training combines a 14.8T-token corpus, long-context extension to 128K, and efficiency-oriented architecture and engineering choices. It reports strong benchmark performance and substantially reduced training cost, while exploring auxiliary-loss-free load balancing.

  • Training Data: The corpus contains 14.8T high-quality, diverse tokens, with increased mathematical and programming data, broader multilingual coverage, and reduced redundancy.Document packing is used for data integrity without cross-sample attention masking, and the tokenizer is optimized for multilingual compression efficiency.
  • Model Architecture: DeepSeek-V3 uses 61 Transformer layers with 256 routed experts per MoE layer, activating 8 experts per token and applying multi-token prediction with depth 1.Its architecture retains MLA and DeepSeekMoE, while the supplied hyperparameters specify the MoE routing and MTP depth.
  • Long-Context Extension: DeepSeek-V3 reaches 128K context through two-stage extension training, maintaining strong performance and consistent robustness on Needle In A Haystack tests.The context window is extended from 4K to 32K and then to 128K using YaRN and two additional 1000-step training phases.
  • Evaluation Results: DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base, while surpassing LLaMA-3.1 405B Base on most benchmarks.The reported advantages include multilingual, code, and math benchmarks, with stronger performance than Qwen2.5 72B on most Chinese benchmarks except CMMLU.
  • Training Efficiency: Training requires 180K H800 GPU hours per trillion tokens, enabled by efficient architectures and engineering optimizations.The training setup also investigates auxiliary-loss-free and batch-wise load balancing, whose flexibility supports expert specialization but may create efficiency challenges from within-batch imbalance.

5. Post-Training

DeepSeek-V3’s post-training combines domain-specific instruction data, reasoning-data distillation, alignment methods, and multi-token prediction, producing strong results across knowledge, long-context, code, math, Chinese, and open-ended evaluations.

  • 5.1. Instruction-Tuning Data: The post-training pipeline curates 1.5M multi-domain instruction instances and uses domain-specific data creation methods, including reasoning data generated with DeepSeek-R1.The R1-generated reasoning data is adjusted to balance accuracy against overthinking, poor formatting, and excessive length.
  • 5.3. Standard Evaluation: DeepSeek-V3 achieves top-tier or leading performance across English, long-context, code, math, Chinese, and open-ended evaluations, including competitiveness with frontier closed-source models.The model is especially strong on mathematical and algorithmic coding benchmarks, while remaining competitive on broad knowledge and generation tasks.
  • 5.3.2. Standard Evaluation: DeepSeek-V3 surpasses Qwen2.5 72B by approximately 10% in absolute scores on AIME, MATH-500, and CNMO 2024.This establishes a new state-of-the-art among non-o1-like models on the cited mathematics benchmarks.
  • 5.3.3. Open-Ended Evaluation: DeepSeek-V3 achieves an over-86% win rate against GPT-4-0314 on Arena-Hard and becomes the first open-source model to exceed 85% on that benchmark.It performs on par with Claude-Sonnet-3.5-1022 in this evaluation.
  • 5.2.1. Long-CoT Distillation: Long-chain-of-thought distillation improves LiveCodeBench and MATH-500 performance but substantially increases average response length.The authors select distillation settings to balance accuracy and computational efficiency, while reporting potential broader applications beyond mathematics and coding.
  • 5.2.3. Multi-Token Prediction: Multi-Token Prediction forecasts the next two tokens and supports speculative decoding, with the second predicted token accepted 85%–90% of the time.The method is intended to accelerate decoding while improving model performance.

6. Conclusion, Limitations, and Future Directions

DeepSeek-V3 combines strong open-source performance with economical training, while its deployment requirements remain a practical limitation and future work targets efficiency, data, reasoning, and architecture.

  • 6. Conclusion: DeepSeek-V3 is presented as the strongest open-source model, comparable to GPT-4o and Claude-3.5-Sonnet, while requiring 2.788M H800 GPU hours for full training.Its reported scale is 671B total parameters, with 37B activated per token, trained on 14.8T tokens.
  • 6.1. Limitations: Deployment requires a relatively large recommended unit, which may burden small teams despite an end-to-end generation speed more than twice that of DeepSeek-V2.The authors identify deployment as a remaining limitation and note that further improvement is possible.
  • 6.2. Future Directions: Future directions include improving training and inference efficiency, extending context length, overcoming Transformer limitations, scaling data, and deepening reasoning capabilities.The plans also include broader and more multidimensional model evaluation and improvement.

Research & Engineering

The research and engineering contributor list spans the paper’s authors, data annotation team, and business and compliance support, with authors ordered alphabetically by first name within each role.

  • Research & Engineering: The paper lists a large author team followed by contributors in data annotation, business and compliance, and related engineering-support roles.Names marked with * denote individuals who have departed from the team.
  • Research & Engineering: Within each role, authors are listed alphabetically by first name, and asterisks identify departed team members.This ordering convention applies to the contributor listings.

B. Ablation Studies for Low-Precision Training

Figure 10 compares BF16 and FP8 training loss curves after smoothing the results with an EMA coefficient of 0.9.

  • B. Ablation Studies for Low-Precision Training: Figure 10 compares training loss curves for BF16 and FP8, using Exponential Moving Average smoothing with coefficient 0.9.The supplied passage identifies the comparison and smoothing procedure but does not state an outcome.

B.1. FP8 v.s. BF16 Training

The FP8 mixed-precision framework is evaluated against BF16 training across small and large MoE baselines, with training curves reported in Figure 10. High-precision accumulation and fine-grained quantization keep relative error below 0.25%.

  • 0.25% relative error remains below this threshold with high-precision accumulation and fine-grained quantization across the evaluated FP8 training setups.The comparison uses approximately 16B-parameter and 230B-parameter MoE baselines trained at different token scales.

B.2. Discussion About Block-Wise Quantization

Block-wise quantization is unsuitable for activation gradients because token-correlated outliers remain difficult to manage. On a 16B-parameter MoE model, this approach causes model divergence during training.

  • Activation quantization uses different 1x128 and 128x1 groupings for forward and backward passes, unlike straightforward 128x128 block-wise quantization.
  • Model divergence occurs when block-wise activation-gradient quantization is used on a 16B-parameter MoE trained for around 300B tokens.The authors hypothesize that highly imbalanced activation gradients create token-correlated outliers that block-wise quantization cannot effectively manage.

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models

The 16B auxiliary-loss-free model develops greater expert specialization than the auxiliary-loss-based baseline across the Pile test set. Relative expert load compares actual usage with theoretically balanced usage.

  • Greater expert specialization appears across all layers in the auxiliary-loss-free model than in the auxiliary-loss-based baseline.The comparison uses expert-load measurements from the 16B models on the Pile test set.
  • Relative expert load measures actual expert load relative to the theoretically balanced expert load.
Loading 2412.19437v2…