PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, Ziwei Xie

arXiv:2405.04434v5cs.CLcs.AI

TL;DR

As language models scale, training demands more computing resources and inference throughput may decline. DeepSeek-V2 combines MLA and DeepSeekMoE; compared with DeepSeek 67B, it reports lower training costs and KV-cache use, higher maximum generation throughput, and stronger performance across almost all benchmarks.

  • Problem

    As LLM parameter counts grow, training requires more computing resources and inference throughput may decline, creating challenges for widespread adoption.

    Source: §1 · p. 4

  • Method

    DeepSeek-V2 combines MLA and DeepSeekMoE in a 128K-context MoE model, using low-rank joint key-value compression to target efficient inference.

    Source: §1 · p. 4, §5 · p. 21

  • Results

    42.5% training-cost savings, 93.3% KV-cache reduction, and 5.76 times maximum generation throughput accompany stronger performance than DeepSeek 67B on almost all benchmarks with 21B activated parameters.

    Source: §3.2.2 · p. 14, §5 · p. 21

  • Takeaways & Limitations

    The authors conclude that DeepSeek-V2 is the strongest open-source MoE model.

    Source: §5 · p. 21

    Because its data primarily consist of Chinese and English content, DeepSeek-V2 may have limited proficiency in other languages and should be used cautiously beyond those languages.

    Source: §5 · p. 21

Abstract

from arXiv · show

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

1. Introduction

DeepSeek-V2 combines MLA and DeepSeekMoE to target efficient inference and economical training, with 236B total parameters, 21B activated per token, and a 128K-token context. The paper reports top-tier open-source performance alongside lower training costs and KV-cache use.

1. Introduction

  • MLA uses low-rank joint key-value compression to reduce inference KV cache, while DeepSeekMoE uses fine-grained expert segmentation and shared-expert isolation for specialization and economical sparse training.The paper also devises mechanisms to control communication overhead and ensure load balance during expert-parallel training.Source: §1 · p. 4, Figure 2 · p. 5
  • DeepSeek-V2 is pretrained on a high-quality, multi-source corpus of 8.1T tokens, then its Chat versions receive SFT on 1.5M conversational sessions and GRPO alignment for human preference.The sessions cover domains including math, code, writing, reasoning, and safety.Source: §1 · p. 4
  • Compared with DeepSeek 67B, DeepSeek-V2 saves 42.5% of training costs, reduces KV cache by 93.3%, and raises maximum generation throughput to 5.76 times.Despite activating only 21B parameters, it is top-tier among open-source models and is described as the strongest open-source MoE language model.Source: §1 · p. 4
  • 38.9 length-controlled win rate on AlpacaEval 2.0, 8.97 overall score on MT-Bench, and 7.91 overall score on AlignBench are reported for DeepSeek-V2 Chat (RL).The English evaluations place Chat (RL) among top-tier open-source chat models; on Chinese AlignBench, it outperforms all open-source models and most closed-source models.Source: §1 · p. 5
  • DeepSeek-V2-Lite is released to facilitate further research and development on MLA and DeepSeekMoE; it has 15.7B total parameters, with 2.4B activated per token.Source: §1 · p. 5

2. Architecture

DeepSeek-V2 retains the Transformer block structure but replaces standard attention and FFNs with MLA and DeepSeekMoE. Its architecture targets efficient inference through compressed KV caching and economical, communication-aware MoE training.

2.1.2. Low-Rank Key-Value Joint Compression

  • MLA jointly compresses keys and values into a low-rank latent vector, reducing cached elements from MHA’s 2n_h d_h l to d_c l per token.During inference, the up-projections can be absorbed into the query and output projections, avoiding explicit key and value computation.Source: §2.1.2 · p. 7, §2.1.1 · p. 7

2.1.3. Decoupled Rotary Position Embedding

  • Because RoPE is incompatible with low-rank KV compression, MLA decouples positional queries and keys to preserve efficient inference.The decoupled key must also be cached, adding to MLA’s KV-cache contents.Source: §2.1.3 · p. 8

2.1.4. Comparison of Key-Value Cache

  • MLA uses KV cache equal to GQA with only 2.25 groups while achieving stronger performance than MHA.The comparison reports cache amounts per token; the table caption specifies its scope, and the prose states the comparison.Source: §2.1.4 · pp. 8–9, Table 1 · p. 9

2.2.1. Basic Architecture

  • DeepSeekMoE segments experts finely for specialization and isolates shared experts to mitigate redundancy among routed experts.The paper states that, with the same activated and total expert parameters, DeepSeekMoE can outperform conventional MoE architectures such as GShard (Lepikhin et al., 2021).Source: §2.2.1 · p. 9

2.2.2. Device-Limited Routing

  • Device-limited routing selects target experts on at most M devices to bound MoE communication costs.The method first selects M devices with the highest expert affinity scores, then selects top-K experts among those devices; performance is roughly aligned with unrestricted top-K when M≥3.Source: §2.2.2 · pp. 9–10

2.2.3. Auxiliary Loss for Load Balance

  • DeepSeek-V2 uses expert-, device-, and communication-balance losses to mitigate routing collapse and improve computation and communication efficiency.Device-limited routing bounds each device’s sending to M T hidden states, while the communication loss encourages it to receive around M T states.Source: §2.2.3 · pp. 10–11

2.2.4. Token-Dropping Strategy

  • To reduce computation wasted by residual imbalance, training drops each device’s lowest-affinity tokens after reaching its average budget, with capacity factor 1.0.Tokens in approximately 10% of training sequences are never dropped, allowing inference to enable dropping according to efficiency requirements while maintaining training–inference consistency.Source: §2.2.4 · p. 11

3. Pre-Training

DeepSeek-V2 was pretrained on an expanded, quality-filtered bilingual corpus and configured as a sparse MoE model, then extended to a 128K context window. The supplied evaluation reports top-tier open-source performance and improved training and inference efficiency versus DeepSeek 67B.

3.1.1. Data Construction

  • The pre-training corpus contains 8.1T tokens, with Chinese tokens approximately 12% more numerous than English tokens.The team recovered mistakenly deleted internet data, incorporated more Chinese and high-quality multi-source data, improved quality filtering, and removed contentious content to mitigate bias from specific regional cultures; the tokenizer is the same 100K-vocabulary BBPE tokenizer used for DeepSeek 67B.Source: §3.1.1 · pp. 11–12

3.1.2. Hyper-Parameters

  • Each MoE layer has 2 shared experts and 160 routed experts, with 6 routed experts activated per token; the model has 236B total parameters and 21B active per token.The model uses 60 Transformer layers, a 5120 hidden dimension, 128 attention heads, and 512-dimensional KV compression; added RMS Norm layers and scaling factors are used to stabilize training.Source: §3.1.2 · p. 12

3.1.3. Infrastructures

  • Training uses HAI-LLM with 16-way zero-bubble pipeline parallelism, 8-way expert parallelism, and ZeRO-1 data parallelism, without tensor parallelism.Computation of shared experts overlaps with expert-parallel all-to-all communication, and customized CUDA kernels accelerate communication, routing, and fused linear computations; experiments ran on H800 GPUs connected within nodes by NVLink and NVSwitch and across nodes by InfiniBand.Source: §3.1.3 · pp. 12–13

3.1.4. Long Context Extension

  • After initial pre-training with a 4K default context, YaRN extends the context window to 128K; 1,000 additional training steps use 32K sequences.The model performs well on Needle In A Haystack tests across context window lengths up to 128K, despite additional training only at sequence length 32K.Source: §3.1.4 · p. 13, Figure 4 · p. 13

3.2.1. Evaluation Benchmarks

  • Evaluation covers English and Chinese benchmarks, using perplexity-based, generation-based, and language-modeling-based evaluation; Pile-test uses Bits-Per-Byte for tokenizer-fair comparison.The benchmark suite spans multiple-choice, reasoning, question answering, reading comprehension, reference disambiguation, language modeling, Chinese understanding, math, code, and standardized exams.Source: §3.2.1 · pp. 13–14

3.2.2. Evaluation Results

  • Under a shared evaluation setting, DeepSeek-V2, with only 21B activated parameters, significantly outperforms DeepSeek 67B on almost all benchmarks and achieves top-tier performance among open-source models.Against Qwen1.5 72B, it has advantages on the majority of English, code, and math benchmarks, while Qwen performs better on Chinese multi-subject multiple-choice tasks. It outperforms Mixtral 8x22B on MMLU and has comparable code and math performance; versus LLaMA3 70B, it has a slight basic-English gap but comparable code and math capability and stronger Chinese performance.Source: §3.2.2 · pp. 14–16

3.2.3. Training and Inference Efficiency

  • 42.5% fewer training costs: per trillion tokens on H800 GPUs, DeepSeek-V2 requires 172.8K GPU hours versus DeepSeek 67B’s 300.6K.On a single node with 8 H800 GPUs, generation throughput exceeds 50K tokens per second, or 5.76 times DeepSeek 67B’s maximum; prompt input throughput exceeds 100K tokens per second.Source: §3.2.3 · p. 16

4. Alignment

DeepSeek-V2 alignment combines quality-improved supervised fine-tuning with staged reinforcement learning for reasoning and human preferences. RL improves math, coding, and open-ended generation, while the authors note a trade-off with some standard benchmarks.

4.1. Supervised Fine-Tuning

  • SFT uses 1.5M curated instances—1.2M for helpfulness and 0.3M for safety—with improved data quality, and trains for two epochs at a 5 × 10−6 learning rate.The authors evaluate SFT chat mainly with generation benchmarks, plus MMLU, ARC, IFEval, LiveCodeBench, and open-ended conversation benchmarks.Source: §4.1 · p. 16

4.2. Reinforcement Learning

  • RL uses GRPO, which avoids a critic model by estimating a baseline from group scores, and proceeds from code/math reasoning alignment to multi-reward human-preference alignment.The second stage combines helpfulness, safety, and rule-based rewards.Source: §4.2 · p. 17
  • Reward models use compiler-feedback code preferences and ground-truth math preferences; hybrid parallelism, large-batch vLLM inference, and CPU offloading balance RL speed and memory.The reward models are initialized from DeepSeek-V2 Chat (SFT) and trained with point-wise or pair-wise loss; the authors report RL helps the model select correct, satisfactory responses.Source: §4.2 · pp. 17–18

4.3. Evaluation Results

  • DeepSeek-V2 Chat (RL) further improves math and code over SFT; SFT surpasses Qwen1.5 72B Chat on almost all English, math, and code benchmarks, with a Chinese multiple-choice caveat.SFT also outperforms Mixtral 8x22B Instruct on most benchmarks except NaturalQuestions and IFEval; it has similar code and math performance to LLaMA3 70B Chat, which does better on MMLU and IFEval.Source: §4.3 · p. 18, Table 3 · p. 19
  • On English open-ended benchmarks, RL beats SFT, Mistral 8x22B Instruct, and Qwen1.5 72B Chat on both; it is competitive with LLaMA3 70B Instruct on MT-Bench and beats it on AlpacaEval 2.0.AlpacaEval 2.0 uses length-controlled win rate.Source: §4.3 · p. 18, Table 4 · p. 19
  • On AlignBench, RL slightly tops SFT; SFT beats open-source Chinese models, while both beat GPT-4-0613 and ERNIEBot 4.0, and RL leads in language understanding but trails giant models in reasoning.RL's Chinese language understanding also exceeds GPT-4-Turbo-1106-Preview.Source: §4.3 · pp. 18–19, Table 5 · p. 20

4.4. Discussion

  • Fewer than 10K SFT instances significantly reduce IFEval performance, and the authors emphasize that both sufficient data quantity and data quality matter, especially for writing and open-ended tasks.They note that required data may decrease as model size grows but cannot be eliminated.Source: §4.4 · p. 20
  • RL improves open-ended generation but can cause an alignment tax on standard benchmarks such as BBH; online preference alignment outperforms offline in these experiments, though the comparison may vary by context (Ouyang et al., 2022).The authors report mitigating the trade-off through data processing and training strategies, and leave broader online-versus-offline analysis for future work.Source: §4.4 · pp. 20–21

5. Conclusion, Limitation, and Future Work … B.1. Model Description

DeepSeek-V2 combines strong open-source performance with lower training and inference costs, while acknowledging limitations and outlining future development goals. The appendix describes DeepSeek-V2-Lite’s architecture and training, alongside acknowledgments of contributors.

5. Conclusion, Limitation, and Future Work

  • Compared with DeepSeek 67B, DeepSeek-V2 is significantly stronger, saves 42.5% of training costs, reduces KV cache by 93.3%, and raises maximum generation throughput to 5.76 times.With 21B activated parameters, it is described as top-tier among open-source models and the strongest open-source MoE model.Source: §5 · p. 21
  • DeepSeek-V2 and its chat versions may hallucinate or generate non-factual information, lack post-training knowledge updates, and have limited proficiency beyond Chinese and English.The paper advises caution in scenarios beyond Chinese and English because the data primarily consist of those languages.Source: §5 · p. 21
  • Future work targets economical scaling toward GPT-4-level performance, improved helpfulness, honesty and safety with less human supervision, and support for multiple modalities beyond text.The paper also states a long-term aim of progressively approaching artificial general intelligence.Source: §5 · p. 21

Data Annotation

  • The acknowledgments credit Huazuo Gao and Wangding Zeng with key MLA innovations and thank Jianlin Su for discussion of position embedding.Authors within each role are listed alphabetically by first name.Source: Data Annotation · p. 28

B.1. Model Description

  • DeepSeek-V2-Lite has 15.7B total parameters, with 2.4B activated per token, and combines MLA with DeepSeekMoE across 27 layers.It uses 2 shared and 64 routed experts per MoE layer, activates 6 routed experts per token, and leaves queries uncompressed; all FFNs except the first layer are MoE layers.Source: §B.1 · p. 29
  • DeepSeek-V2-Lite was pretrained from scratch on 5.7T tokens, using 4K maximum sequence length and a constant batch size of 4608 sequences.It uses AdamW, a warmup-and-step-decay learning-rate schedule, and pipeline parallelism; after pretraining it receives long-context extension and SFT to produce DeepSeek-V2-Lite Chat.Source: §B.1 · pp. 29–30
  • DeepSeek-V2-Lite Chat is produced by applying long-context extension and supervised fine-tuning after pretraining.The model’s training setup uses expert-level balance loss but not device-level or communication balance loss.Source: §B.1 · p. 30

B.2. Performance Evaluation · C. Full Formulas of MLA

DeepSeek-V2-Lite outperforms the authors’ previous small-size base and chat models, while the MLA formulas describe inference-time caching and an optimization that avoids recomputing keys and values for each query.

B.2. Performance Evaluation

  • DeepSeek-V2-Lite shows overwhelming performance advantages over the authors’ previous small-size base models, especially in reasoning, coding, and math.The comparison is presented in Table 6; no table values are supplied here.Source: §B.2 · p. 30
  • DeepSeek-V2-Lite Chat outperforms the authors’ previous small-size chat models by a large margin.The comparison is presented in Table 7; no table values are supplied here.Source: §B.2 · p. 30

C. Full Formulas of MLA

D. Ablation of Attention Mechanisms · D.1. Ablation of MHA, GQA, and MQA

On four hard benchmarks, 7B dense models using MHA, GQA, and MQA were compared under matched training and parameter conditions; MHA showed significant advantages over GQA and MQA.

D.1. Ablation of MHA, GQA, and MQA

  • MHA showed significant advantages over GQA and MQA on four hard benchmarks.The evaluation results are presented in Table 8.Source: §D.1 · p. 31
  • The three models were trained on 1.33T tokens with the same architecture except for attention, and their parameter counts were aligned to around 7B by adjusting layer counts.Source: §D.1 · p. 31

D.2. Comparison Between MLA and MHA

Across two matched MoE scales, MLA outperforms MHA on four hard benchmarks while using substantially less KV cache. The comparison covers small and large models trained on different token counts.

D.2. Comparison Between MLA and MHA

  • The comparison spans about 16B-parameter MoE models trained on 1.33T tokens and about 250B-parameter MoE models trained on 420B tokens.Two models at each scale share the same architecture except for their attention mechanisms.Source: §D.2 · p. 31
  • MLA outperforms MHA on four hard benchmarks and uses 14% of MHA’s KV cache for small MoE models and 4% for large MoE models.The evaluation compares matched architectures that differ only in their attention mechanisms.Source: §D.2 · p. 31, Table 9 · p. 32

E. Discussion About Pre-Training Data Debiasing

DeepSeek-V2 filters contentious pre-training content, including values influenced by regional cultures, to reduce subjective bias, and observes slightly worse performance on culturally specific test sets. The authors attribute its underperformance on MMLU Humanity-Moral to this debiasing effort, while noting low agreement between annotators and the test labels.

E. Discussion About Pre-Training Data Debiasing

  • DeepSeek-V2 filters contentious pre-training content, including regionally influenced values, to avoid unnecessary subjective bias on controversial topics.The authors report that this approach is associated with slightly worse performance on test sets closely tied to specific regional cultures.Source: E. Discussion About Pre-Training Data… · p. 32
  • On MMLU Humanity-Moral, DeepSeek-V2 lags behind competitors such as Mixtral 8x22B, despite comparable or superior performance on most MMLU test sets.The authors describe Humanity-Moral as mainly associated with American values.Source: E. Discussion About Pre-Training Data… · p. 32
  • Three annotators independently labeled 420 MMLU Humanity-Moral scenarios, and their agreement with each other and the ground-truth labels was low.The authors attribute DeepSeek-V2’s abnormal performance on value-sensitive test sets to their pre-training corpus debiasing efforts.Source: E. Discussion About Pre-Training Data… · p. 32

F. Additional Evaluations on Math and Code

DeepSeek-V2 Chat (RL) performs strongly on Chinese math and live coding evaluations, with the supplied results highlighting its SC-Math6 standing and LiveCodeBench Pass@1 performance.

F. Additional Evaluations on Math and Code

G. Evaluation Formats

This section documents benchmark-specific evaluation formats through examples in Tables 12–37, including reasoning and reading-comprehension prompts and a special WinoGrande scoring procedure.

G. Evaluation Formats

  • Tables 12–37 illustrate benchmark-specific formats across tasks including AGIEval, ARC, BBH, C-Eval, GSM8K, and HumanEval.The tables are presented as examples for each benchmark.Source: G. Evaluation Formats · p. 33, Table 12 · p. 34, Table 13 · p. 35
  • Examples include arithmetic and algebra problems answered with explicit step-by-step reasoning and a final answer.The sample solutions show intermediate calculations or equations before stating the answer.Source: G. Evaluation Formats · pp. 43, 45
  • A reading-comprehension example pairs a long article with questions about its purpose, answerability, and interpretation.The questions refer to the article about earthworms.Source: G. Evaluation Formats · p. 49
  • WinoGrande uses multiple prefixes and one completion, selecting the predicted prefix with the lowest completion perplexity.This evaluation-specific selection rule is noted in the example caption.Source: Table 35 · p. 50

Related papers

Loading 2405.04434v5…