PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, Ziwei Xie

arXiv:2405.04434v5cs.CLcs.AI

TL;DR

Scaling language models increases computing demands for training and can reduce inference throughput. DeepSeek-V2 combines MLA and DeepSeekMoE with large-scale pre-training, SFT, and RL, achieving strong open-source performance with economical training and efficient inference. Compared with DeepSeek 67B, it saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts maximum generation throughput to 5.76 times.

  • Problem

    Increasing language-model parameters improves capabilities but increases training-resource requirements and can reduce inference throughput, constraining broader use.

  • Method

    DeepSeek-V2 combines MLA and DeepSeekMoE with an 8.1T-token pre-training corpus, supervised fine-tuning, and reinforcement learning.

  • Results

    With 21B activated parameters, DeepSeek-V2 achieves top-tier open-source performance; compared with DeepSeek 67B, it saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts maximum generation throughput to 5.76 times.

  • Takeaways & Limitations

    DeepSeek-V2 demonstrates that a sparse MoE model can combine strong open-source performance with economical training and efficient inference.

  • Takeaways & Limitations

    The model may lack ongoing knowledge updates, generate unverified or hallucinated information, and have limited proficiency beyond Chinese and English.

Abstract

from arXiv · show

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.

1. Introduction

DeepSeek-V2 addresses the resource costs of scaling language models with an economical, efficient MoE architecture and broad training and evaluation pipeline. Its MLA and DeepSeekMoE designs support efficient inference, economical training, and strong open-source performance.

  • 236B total parameters, with 21B activated per token, enable DeepSeek-V2 to support a 128K-token context length.
  • MLA uses low-rank key-value joint compression to reduce the inference-time KV cache, while DeepSeekMoE uses sparse architecture for economical training.
  • 8.1T tokens form the high-quality, multi-source pre-training corpus, followed by SFT and RL to develop chat models.
  • DeepSeek-V2 achieves top-tier performance among open-source models and becomes the strongest open-source MoE language model despite activating only 21B parameters.
  • 38.9 length-controlled win rate on AlpacaEval 2.0, 8.97 on MT-Bench, and 7.91 on AlignBench are reported for DeepSeek-V2 Chat (RL).
  • DeepSeek-V2-Lite is released with MLA and DeepSeekMoE, using 15.7B total parameters and 2.4B activated parameters per token.

2. Architecture

DeepSeek-V2 remains a Transformer while redesigning attention and FFNs with MLA and DeepSeekMoE to improve inference efficiency and economical sparse computation. MLA compresses key-value states, while DeepSeekMoE specializes experts and manages device communication.

  • Attention: MLA uses low-rank key-value joint compression to reduce the inference-time KV-cache bottleneck.It compresses keys and values into a latent vector and reconstructs them through up-projection matrices.
  • Attention: Decoupled RoPE uses additional multi-head queries and keys because standard RoPE is incompatible with low-rank KV compression.Without decoupling, keys for all prefix tokens would need recomputation during inference.
  • Attention: MLA requires KV cache comparable to GQA with 2.25 groups while achieving stronger performance than MHA.For DeepSeek-V2, the KV compression dimension is set to 4d_h, with an additional decoupled RoPE key component.
  • Feed-Forward Networks: DeepSeekMoE segments experts finely for specialization and isolates shared experts to reduce redundancy among routed experts.With the same activated and total expert parameters, it can outperform conventional MoE architectures such as GShard.
  • Routing and Load Balance: Device-limited routing bounds communication, while communication balance loss encourages each device to receive around M_T hidden states.The design addresses higher communication costs caused by fine-grained expert segmentation and multiple activated experts.

3. Pre-Training

DeepSeek-V2 is pretrained on an expanded, higher-quality bilingual corpus and uses sparse MoE training with efficiency-oriented infrastructure and long-context extension. Evaluations cover English and Chinese benchmarks, showing strong open-source performance alongside training and inference efficiency.

  • Data construction: 8.1T tokens form DeepSeek-V2’s high-quality, multi-source pre-training corpus, with Chinese tokens approximately 12% more numerous than English tokens.The corpus extends prior data, especially Chinese data, while improving data quality and recovering mistakenly deleted internet data through optimized cleaning.
  • Model configuration: 60 Transformer layers and 5120 hidden dimensions define the model, with 160 routed experts and 6 activated experts per token in each MoE layer.Each MoE layer also contains 2 shared experts, while most FFNs are replaced with MoE layers.
  • Training infrastructure: 16-way pipeline parallelism, 8-way expert parallelism, and ZeRO-1 data parallelism support training on NVIDIA H800 GPU clusters.The training framework also recomputes some operators to save activation memory.
  • Long-context evaluation: 128K context evaluation remains robust after additional training at 32K sequence length, with strong performance across all NIAH context lengths up to 128K.YaRN extends the default context window from 4K to 128K, and the model is evaluated on English and Chinese benchmarks.
  • Evaluation and efficiency: 21B activated parameters deliver top-tier open-source benchmark performance while reducing training cost and KV-cache requirements relative to DeepSeek 67B.The supplied passages report broad benchmark comparisons and efficiency gains, including 42.5% lower training costs and substantially smaller deployed KV cache.

4. Alignment

DeepSeek-V2 Chat is aligned through supervised fine-tuning and reinforcement learning, with evaluations showing strong performance across standard, English open-ended, and Chinese benchmarks. The discussion also identifies data requirements, alignment trade-offs, and uncertainty about online versus offline preference alignment.

  • Supervised Fine-Tuning: 1.5M instruction-tuning instances cover helpfulness and safety, supporting DeepSeek-V2 Chat’s supervised fine-tuning.The dataset contains 1.2M helpfulness and 0.3M safety instances.
  • Reinforcement Learning: Two-stage reinforcement learning first performs reasoning alignment, then human preference alignment.The reasoning stage targets code and mathematics; the preference stage uses helpfulness, safety, and rule-based rewards.
  • Standard Benchmarks: DeepSeek-V2 Chat (SFT) substantially improves GSM8K, MATH, and HumanEval over the base model, while RL further improves mathematics and coding performance.SFT data includes substantial math- and code-related content.
  • Open-Ended Generation: DeepSeek-V2 Chat (RL) significantly outperforms its SFT version on English open-ended benchmarks and surpasses Mistral 8x22B Instruct and Qwen1.5 72B Chat on both benchmarks.It is competitive with LLaMA3 70B Instruct on MT-Bench and outperforms it on AlpacaEval 2.0.
  • Open-Ended Generation: DeepSeek-V2 Chat models achieve top-tier Chinese open-ended performance, with RL outperforming all evaluated models in Chinese language understanding while reasoning still lags behind giant models.The cited evaluation uses AlignBench, whose overall score is ranked by GPT-4-0613.
  • Discussion: Fewer than 10K SFT instances cause a significant IFEval decline, while reinforcement learning improves open-ended scores but can reduce performance on benchmarks such as BBH.The authors describe their resulting balance between standard and open-ended benchmarks as a tolerable trade-off.

5. Conclusion, Limitation, and Future Work

DeepSeek-V2 combines strong performance with economical training and efficient inference, while acknowledging standard LLM limitations and outlining continued open-source development.

  • Conclusion: 42.5% lower training costs, 93.3% smaller KV cache, and 5.76 times higher maximum generation throughput are reported versus DeepSeek 67B.The model supports a 128K context length and uses 21B activated parameters.
  • Conclusion: DeepSeek-V2 achieves top-tier performance among open-source models with only 21B activated parameters and is described as the strongest open-source MoE model.
  • Limitation: The paper acknowledges that the models lack ongoing knowledge updates, may generate non-factual information, and may hallucinate.These limitations are shared with other LLMs.
  • Limitation: Because the data primarily contain Chinese and English content, proficiency in other languages may be limited and requires caution beyond those languages.
  • Future Work: The authors state that DeepSeek will continue investing in open-source large models while progressively pursuing artificial general intelligence.
  • Future Work: Future work targets scaling MoE models while maintaining economical training and inference costs, with a stated goal of GPT-4-level performance in an upcoming release.

B.2. Performance Evaluation

DeepSeek-V2-Lite shows strong performance against earlier small-size models, while the appendix presents MLA’s computation process and inference optimization.

  • B.2. Performance Evaluation: DeepSeek-V2-Lite exhibits overwhelming performance advantages over previous small-size base models, especially in reasoning, coding, and math.
  • B.2. Performance Evaluation: DeepSeek-V2-Lite Chat outperforms previous small-size chat models by a large margin.
  • MLA: MLA’s inference optimization absorbs projection matrices through associativity, avoiding per-query key and value reconstruction and its associated computational overhead.

D.1. Ablation of MHA, GQA, and MQA

Controlled 7B comparisons show that MHA performs significantly better than GQA and MQA on four hard benchmarks under matched training and architectural conditions.

  • D.1. Ablation of MHA, GQA, and MQA: MHA demonstrates significant advantages over GQA and MQA on four hard benchmarks.The models were trained on 1.33T tokens and aligned to around 7B parameters by adjusting layer counts.
  • D.1. Ablation of MHA, GQA, and MQA: The comparison isolates attention mechanisms by keeping the broader architecture shared while adjusting layers to match approximately 7B parameters.

D.2. Comparison Between MLA and MHA

The MLA comparison evaluates matched MoE models across two scales and reports better performance than MHA with substantially less KV cache.

  • D.2. Comparison Between MLA and MHA: The MLA-versus-MHA evaluation trains and evaluates small and large MoE models across two parameter scales.The small models contain about 16B total parameters, while the large models contain about 250B total parameters.
  • D.2. Comparison Between MLA and MHA: DeepSeek-V2 shows better performance than MHA while requiring a significantly smaller amount of KV cache.

E. Discussion About Pre-Training Data Debiasing

DeepSeek-V2 filters contentious, regionally influenced content to reduce unnecessary subjective biases, but this choice affects performance on culturally specific evaluations. Human-annotation analysis suggests the Humanity-Moral benchmark itself has low label agreement.

  • Pre-training data debiasing: Filtering regionally influenced contentious content aims to avoid unnecessary subjective biases but slightly reduces performance on culturally specific test sets.On MMLU, DeepSeek-V2 lags behind Mixtral 8x22B on the Humanity-Moral subset despite comparable or superior performance on most other subsets.
  • Humanity-Moral analysis: 420 Humanity-Moral scenarios received independent annotations from three well-educated human annotators.The analysis compared annotator judgments with one another and with the ground-truth label.
  • Humanity-Moral analysis: Low agreement among annotators and the ground-truth label indicates that Humanity-Moral answers can be contentious across regional cultures.

F. Additional Evaluations on Math and Code

Additional evaluations report strong Chinese mathematical performance and strong coding performance for DeepSeek-V2 Chat (RL), including live coding tasks. The SC-Math6 evaluation covers Chinese math problems, while LiveCodeBench uses questions from a defined recent period.

  • Math: DeepSeek-V2 Chat (RL) outperforms all Chinese LLMs on SC-Math6, including both open-source and closed-source models.SC-Math6 consists of thousands of Chinese math problems.
  • Evaluation caveat: Table 10 attributes inconsistent Humanity-Moral performance to debiasing efforts after finding low agreement among annotators and the ground-truth label.Three annotators independently assessed 420 moral scenarios.
  • Code: DeepSeek-V2 Chat (RL) demonstrates considerable LiveCodeBench proficiency, with Pass@1 surpassing some giant models.LiveCodeBench questions were selected from September 1st, 2023 to April 1st, 2024.

G. Evaluation Formats

The paper specifies benchmark evaluation formats across language, reasoning, mathematics, coding, and reading-comprehension tasks. It also provides example prompts, solutions, and answer formats for many benchmarks.

  • Code evaluation: LiveCodeBench evaluation uses questions selected from September 1st, 2023 to April 1st, 2024.
  • Mathematical reasoning: The examples include step-by-step mathematical and quantitative reasoning problems with explicit final answers.Shown tasks include algebra, logarithms, domains, arithmetic, word problems, and systems of equations.
  • Reading and word problems: Several examples use natural-language questions followed by reasoning traces and concise answers for reading comprehension and elementary word problems.The supplied formats cover passages, titles, answerability questions, ages, counts, costs, and proportions.
  • Task-specific format: Table 35 specifies that WinoGrande has multiple prefixes but one completion, with the predicted prefix chosen by the completion’s lowest perplexity.
Loading 2405.04434v5…