PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, Yuheng Zou

arXiv:2401.02954v1cs.CLcs.AIcs.LG

TL;DR

Prior scaling-law studies reached varying conclusions and incompletely specified hyperparameters, leaving uncertainty about how to scale open-source language models efficiently. DeepSeek LLM revisits scaling behavior, develops models and training methods guided by those findings, and reports strong benchmark and open-ended evaluation results. DeepSeek 67B surpasses LLaMA-2 70B across various benchmarks and outperforms GPT-3.5 in Chinese and English open-ended evaluations.

  • Problem

    Earlier scaling-law studies reached varying conclusions and often lacked complete hyperparameter descriptions, leaving the general applicability and optimality of scaling strategies uncertain.

  • Method

    The paper calibrates scaling laws through extensive experiments, models compute-dependent batch size and learning rate, trains open-source models on 2 trillion English and Chinese tokens, and applies staged instruction-data training.

  • Results

    DeepSeek LLM surpasses LLaMA-2 70B across various benchmarks, particularly in code, mathematics, and reasoning, while DeepSeek 67B Chat outperforms GPT-3.5 in Chinese and English open-ended evaluations.

  • Takeaways & Limitations

    DeepSeek LLM is presented as a long-term project for advancing open-source language models through scaling-law-guided training and evaluation.

  • Takeaways & Limitations

    DeepSeek Chat may generate unverified information and hallucinations, while its Chinese data is not exhaustive and its proficiency in other languages remains delicate.

Abstract

from arXiv · show

The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.

1. Introduction

Open-source LLM research has emphasized fixed-size models despite unresolved scaling-law questions. DeepSeek LLM investigates scaling, trains open-source 7B and 67B models, and reports strong benchmark and open-ended evaluation results.

  • Motivation: Open-source research has largely focused on fixed-size 7B, 13B, 34B, and 70B models while giving less attention to scaling laws.Prior scaling studies also reached varying conclusions about model–data allocation and did not adequately address hyperparameters.
  • Project scope: DeepSeek LLM investigates scaling behavior and applies its findings to two commonly used large-scale open-source configurations.The project adopts a long-term perspective toward advancing open-source language models.
  • Training approach: 2 trillion tokens support pre-training, alongside an architecture based generally on LLaMA with a multi-step learning-rate scheduler.The scheduler replaces cosine scheduling while maintaining performance and facilitating continual training.
  • Evaluation: DeepSeek LLM surpasses LLaMA-2 70B across various benchmarks, especially in code, mathematics, and reasoning.The reported evaluation covers both base and chat models.
  • Evaluation: After SFT and DPO, DeepSeek 67B Chat outperforms GPT-3.5 in Chinese and English open-ended evaluations.Safety evaluation also indicates that DeepSeek 67B Chat can provide harmless responses in practice.

2. Pre-Training

DeepSeek LLM’s pre-training pipeline combines curated multilingual data, LLaMA-like architecture with modified macro design, and empirically selected training hyperparameters. The system emphasizes scalable training, continual-training reuse, and efficient evaluation infrastructure.

  • Data: Data preparation uses deduplication, filtering, and remixing to improve diversity, information density, and representation of underrepresented domains.The pipeline targets a richer and more balanced dataset.
  • Data: Deduplicating across 91 Common Crawl dumps eliminates four times more documents than deduplicating within a single dump.The comparison concerns duplicate removal across the corpus versus within one dump.
  • Tokenizer: The tokenizer uses BBPE with category-aware pre-tokenization, digit-level number splitting, a 100000-token conventional vocabulary, and 15 added special tokens.The resulting vocabulary contains 100015 tokens and was trained on an approximately 24 GB multilingual corpus.
  • Architecture: DeepSeek LLM largely follows LLaMA’s Pre-Norm, RMSNorm, SwiGLU, and Rotary Embedding design, while the 67B model uses GQA to reduce inference cost.The 67B model uses depth expansion rather than widening FFN intermediate layers.
  • Architecture: DeepSeek LLM 7B has 30 layers and DeepSeek LLM 67B has 95 layers, supporting parameter consistency and model pipeline partitioning.This macro-design choice differs from other open-source configurations while facilitating training and inference.
  • Hyperparameters: A multi-step scheduler reaches its maximum after 2000 warmup steps, then falls to 31.6% after 80% of tokens and 10% after 90%.The schedule uses gradient clipping at 1.0 and balances continual-training reuse against model performance.
  • Hyperparameters: Despite different loss-reduction trends, multi-step and cosine scheduling produce essentially consistent final performance.The multi-step schedule was selected as the default because first-phase training can be reused when the training scale changes.
  • Infrastructure: HAI-LLM integrates data, tensor, sequence, and pipeline parallelism with flash attention and ZeRO-1 for large-model training.Asynchronous checkpoints limit worst-case training loss from occasional failures to no more than five minutes, and evaluation uses vLLM or continuous batching.

3. Scaling Laws

The paper revisits scaling laws by calibrating hyperparameters and model/data allocation with more precise compute measurements. Its experiments show that data quality changes allocation, while small-scale results can predict the performance of much larger models.

  • Motivation: Earlier scaling-law studies reached varying conclusions about model/data allocation and often incompletely described hyperparameter settings, motivating a reassessment.The paper frames this reassessment as necessary for efficient compute scaling.
  • Scaling Laws for Hyperparameters: Power-law fits relate compute budget C to near-optimal batch size B and learning rate η, with B increasing and η decreasing as C grows.Near-optimal models were those within 0.25% of minimum generalization error, and the fitted parameters occupied a broad band.
  • Limitations: The hyperparameter analysis does not yet account for factors beyond compute budget C, and optimal parameter spaces vary slightly across model/data allocations at equal compute.The authors identify further research on hyperparameter selection and training dynamics as necessary.
  • Model and Data Scaling: Non-embedding FLOPs/token M replaces parameter count as the model-scale measure, yielding the more precise compute relation C = MD.M includes attention-operation cost but excludes vocabulary computation, reducing approximation errors in model-scale estimation.
  • Model and Data Scaling: The resulting IsoFLOP analysis estimates optimal model/data allocation and predicts the performance of DeepSeek LLM 7B and 67B.The scaling curves are fitted using optimal allocation for each compute budget.
  • Performance Prediction: Small-scale experiments accurately predict model performance at a 1000× compute budget, providing guidance for larger-scale training.The prediction is evaluated using the validation-set bits-per-byte performance scaling curve.
  • Scaling Laws with Different Data: Higher-quality training data increases the model-scaling exponent and decreases the data-scaling exponent, directing more additional compute toward model scaling.The paper also reports that high-quality data can support training larger models at the same data scale.

4. Alignment

DeepSeek’s alignment pipeline combines supervised fine-tuning and Direct Preference Optimization with multilingual helpfulness and harmlessness data. The authors report that DPO improves open-ended generation while standard benchmark performance changes little.

  • Alignment data: 1.5 million instruction instances cover English and Chinese helpfulness and harmlessness topics.The helpfulness portion includes general language, mathematics, and coding data, while safety data covers sensitive topics.
  • Alignment pipeline: The alignment pipeline contains two stages: supervised fine-tuning followed by preference optimization.The 7B model was fine-tuned for four epochs, whereas the 67B model used two because overfitting was more serious.
  • SFT effects: Math SFT data increased repetition, with weaker models struggling to learn recurring reasoning patterns.The authors evaluated repetition using 3,868 Chinese and English prompts and attributed the issue to similar patterns in mathematical reasoning data.
  • Preference optimization: DPO uses multilingual helpfulness and harmlessness preference data to further align DeepSeek Chat models.The helpfulness prompts include creative writing, question answering, and instruction following.
  • DPO effects: DPO strengthens open-ended generation while producing little difference across standard benchmark performance.The DPO training used one epoch with a learning rate of 5e-6 and batch size of 512.

5. Evaluation

DeepSeek models are evaluated across multilingual understanding, reasoning, mathematics, coding, knowledge, and open-ended generation tasks using task-specific protocols. DeepSeek 67B improves over LLaMA2 70B on several challenging domains, while its chat variants perform strongly in Chinese and English open-ended evaluations.

  • Evaluation scope: The evaluation covers English and Chinese multiple-choice, generation, language-modeling, mathematics, coding, reasoning, and comprehension benchmarks.The benchmark suite includes MMLU, C-Eval, CMMLU, GSM8K, MATH, HumanEval, MBPP, AGIEval, and Pile-test.
  • Evaluation protocols: Perplexity-based evaluation selects the lowest-perplexity option for multiple-choice tasks, with unconditional or length normalization depending on the dataset.The framework applies unconditional normalization to ARC and OpenBookQA and length normalization elsewhere.
  • Evaluation protocols: Generation-based evaluation lets models produce free text and parses the answers using greedy decoding.This protocol is used for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.
  • Base model results: DeepSeek 67B achieves considerably better performance than LLaMA2 70B on MATH, GSM8K, HumanEval, MBPP, BBH, and Chinese benchmarks.DeepSeek models remain comparable to LLaMA2 models on English language-understanding benchmarks despite using a bilingual corpus.
  • Chat-model results: Fine-tuning improves most chat-model tasks, with HumanEval and GSM8K scores improving by over 20 points.The authors report declines on some cloze or sentence-completion tasks such as HellaSwag, while reasoning gains are described as learning the correct reasoning format.
  • Chinese open-ended evaluation: DeepSeek 67B Chat surpasses ChatGPT and other baselines on AlignBench, with the DPO model improving across almost all metrics.The results indicate strong performance across Chinese tasks, including advanced reasoning and mathematical calculation.
  • English open-ended evaluation: 8.76 is the average MT-Bench score for DeepSeek LLM 67B Chat DPO, behind only GPT-4.The base chat model scores 8.35, comparable with GPT-3.5-turbo, and DPO further improves the average score.

5.3. Held-Out Evaluation

Held-out evaluations examine coding, mathematics, instruction following, safety, and alignment-related behavior beyond conventional benchmarks. Results indicate substantial size-related differences, improved capabilities after training stages, and limits to gains from multiple-choice data.

  • Held-out evaluation uses recent test sets to reduce risks from data contamination and benchmark overfitting.
  • Coding evaluation: 126 LeetCode problems with over 20 test cases each measure coding through a pass@1-style metric, including out-of-domain contest problems.
  • Large models show a significant performance gap over small models on held-out datasets, while DeepSeek 7B remains relatively commendable on held-out tasks despite weaker standard-benchmark results.
  • Safety evaluation: The safety evaluation combines expert-designed categories, manually constructed cases, and human review of safe, unsafe, and refused answers.
  • Safety evaluation: The Do-Not-Answer evaluation reports a score of 97.8 for DeepSeek 67B Chat, higher than ChatGPT and GPT-4.
  • Training effects: The second training stage preserves code and math proficiency while reducing repetition and improving instruction following.
  • Training effects: Adding 20M multiple-choice examples improves Chinese and English multiple-choice benchmarks but not generative evaluations such as TriviaQA and ChineseQA.
  • Training effects: Adding a system prompt slightly degrades 7B performance but substantially improves 67B performance, which the authors attribute to stronger prompt understanding in larger models.

6. Conclusion, Limitation, and Future Work

The paper introduces DeepSeek LLM as a long-term open-source effort grounded in scaling-law analysis, large bilingual data, and extensive training experiments. It reports broad model capabilities while acknowledging factuality, updating, Chinese-data, and multilingual limitations.

  • DeepSeek LLMs are open-source models trained from scratch on 2 trillion English and Chinese tokens.
  • The paper calibrates prior scaling laws, proposes a model/data allocation strategy, and predicts near-optimal batch size and learning rate for a given compute budget.
  • The authors relate scaling-law behavior to data quality and use the analysis to select pre-training hyperparameters and evaluate the resulting models.
  • Limitations and future work: DeepSeek Chat may lack post-training knowledge updates, generate non-factual information, and hallucinate; limited Chinese coverage and predominantly Chinese-English data constrain some topics and languages.
  • DeepSeek LLM is presented as a long-term project committed to advancing open-source language models.
  • Limitations and future work: Future work includes larger datasets, technique reports for code intelligence and Mixture-of-Experts, and continued study of helpful, honest, and safe alignment.

A.2. Different Model Scale Representations

The appendix compares three model-scale representations for fitting performance scaling curves. Differences in optimal model/data allocation are small at high compute budgets but more visible at low budgets, with FLOPs-based scaling producing the most accurate predictions.

  • At higher compute budgets, the three representations produce similar optimal model/data allocations, whereas lower budgets show noticeable deviations.
  • Using 6N_1 tends to overestimate large-model performance, while 6N_2 tends to underestimate it.
  • M, the non-embedding FLOPs/token representation, achieves the most accurate predictions.

A.3. Benchmark Metrics Curves

Figure 7 tracks DeepSeek LLM Base benchmark metrics across training steps. The reported curves improve consistently throughout training, with further gains expected if training continues.

  • Figure 7 presents benchmark metric curves for DeepSeek LLM Base across different training steps.
  • Benchmark performance improves consistently from the beginning to the end of training.
  • The authors expect performance to improve further with continued training.

A.4. Comparison with Code or Math Specific Models

DeepSeek LLM 67B is compared with code- and math-specific models. It achieves similar performance to CodeLlama and shows strong performance across multilingual math benchmarks, including program-based solving.

  • Code-specific models: DeepSeek LLM 67B achieves similar performance to CodeLlama despite using less code data.The paper also notes broader capabilities beyond code.
  • Math-specific models: DeepSeek 67B exhibits exceptional performance across math benchmarks in different languages.The cited benchmarks include GSM8K, MATH, MGSM-zh, and CMath.
  • Math-specific models: Program-based math solving performs better than chain-of-thoughts for DeepSeek LLM.

A.5. Benchmark Results w/ DPO Stage

The DPO-stage evaluation compares benchmark metrics before and after preference optimization. The reported results indicate that DPO does not significantly affect an LLM's fundamental capability.

  • DPO does not significantly impact the fundamental capability of an LLM.This conclusion is based on the benchmark results presented in Table 17.

A.6. Evaluation Formats

The paper presents evaluation formats and worked examples across many benchmarks. These examples span language understanding, reasoning, mathematics, coding, and other evaluation tasks.

  • Tables 18–40 present example evaluation formats for benchmarks including AGIEval, ARC, BBH, C-Eval, C3, CCPM, and CMATH.
  • The examples include step-by-step solutions for arithmetic, algebra, logarithms, determinants, and word problems.Worked examples include determinant multiplication, weight lifting, equation systems, and logarithms.
  • Additional formats cover GSM8K, HellaSwag, HumanEval, MATH, MBPP, MMLU, NaturalQuestions, OpenBookQA, PIQA, RACE, TriviaQA, and WinoGrande.For WinoGrande, the predicted prefix is selected using the lowest perplexity of the completion.
Loading 2401.02954v1…