PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V3 Technical Report

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan

arXiv:2412.19437v2cs.CLcs.AI

TL;DR

DeepSeek-V3 faces inefficient communication overhead in cross-node expert parallelism. It combines established attention and MoE architectures with additional training strategies, and evaluations report leading open-source performance with results comparable to leading closed-source models.

  • Problem

    Cross-node expert parallelism in DeepSeek-V3 incurs communication overhead that yields an approximately 1:1 computation-to-communication ratio.

    Source: §3.2.1 · p. 12

  • Method

    DeepSeek-V3 combines MLA and DeepSeekMoE with auxiliary-loss-free load balancing and a multi-token-prediction training objective.

    Source: §6 · p. 35

  • Results

    Across comprehensive benchmarks, DeepSeek-V3-Base is reported as the strongest open-source base model, while its chat version outperforms open-source peers and performs comparably to leading closed-source models.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The report gives full-training compute for DeepSeek-V3 as 2.788M H800 GPU hours.

    Source: §1 · p. 5

    Its recommended deployment unit is relatively large, which may burden small-sized teams.

    Source: §6 · p. 35

Abstract

from arXiv · show

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

C Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models 48 · 1. Introduction

DeepSeek-V3 combines a 671B-parameter MoE model with efficient architectures, training methods, and post-training to achieve strong open- and closed-model comparisons at economical training cost. Its evaluation results and training approach are detailed below.

1. Introduction

  • DeepSeek-V3 has 671B total parameters, with 37B activated per token, and was pretrained on 14.8T tokens without irrecoverable loss spikes or rollbacks.Its context length was extended in two stages, first to 32K and then to 128K.Source: §1 · p. 4

Architecture: Innovative Load Balancing Strategy and Training Objective

  • DeepSeek-V3 combines MLA for efficient inference and DeepSeekMoE for cost-effective training with auxiliary-loss-free load balancing and multi-token prediction.The report says multi-token prediction improves evaluation performance and can support speculative decoding for inference acceleration.Source: §1 · p. 4, Architecture: Innovative Load… · p. 5

Pre-Training: Towards Ultimate Training Efficiency

  • FP8 training, DualPipe, cross-node communication kernels, and memory optimization enable high training efficiency and near-full computation-communication overlap without costly tensor parallelism.The report states that FP8 support accelerates training and reduces GPU memory use.Source: §1 · p. 4, Pre-Training: Towards Ultimate… · p. 5

Summary of Core Evaluation Results

  • DeepSeek-V3-Base is described as the strongest open-source base model, while its chat version outperforms other open-source models and is comparable to GPT-4o and Claude-3.5-Sonnet.The report also highlights strength on code and math benchmarks.Source: §1 · p. 5

Pre-Training: Towards Ultimate Training Efficiency

Post-Training: Knowledge Distillation from DeepSeek-R1

  • Post-training distills reasoning capabilities from a DeepSeek-R1-series model into DeepSeek-V3, improving reasoning while controlling output style and length.The pipeline incorporates R1 verification and reflection patterns.Source: Post-Training: Knowledge Distillation… · pp. 5–6

Summary of Core Evaluation Results

  • DeepSeek-V3 achieves 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA; it leads open-source models in factuality and coding-competition results, and non-long-CoT models on math benchmarks.It trails GPT-4o and Claude-Sonnet-3.5 on English SimpleQA but surpasses them on Chinese SimpleQA; it outperforms o1-preview on some math benchmarks, while Claude-Sonnet-3.5 is slightly ahead on engineering tasks.Source: Summary of Core Evaluation Results · p. 6

2. Architecture

DeepSeek-V3 retains the Transformer framework, combining MLA for efficient inference and DeepSeekMoE for economical training. It adds auxiliary-loss-free load balancing and a multi-token prediction training objective.

2.1.1. Multi-Head Latent Attention

  • MLA jointly compresses attention keys and values so only the latent vector is cached, reducing KV cache while maintaining performance comparable to standard MHA (Vaswani et al., 2017).DeepSeek-V3 also low-rank compresses queries to reduce activation memory during training.Source: §2.1.1 · pp. 7–8

2.1.2. DeepSeekMoE with Auxiliary-Loss-Free Load Balancing

  • DeepSeekMoE uses finer-grained and shared experts with sigmoid affinities; for load balance, per-expert routing biases adapt to batch overload while gating uses unmodified affinities, outperforming pure auxiliary-loss balancing (Wang et al., 2024a).The bias changes top-K routing only; the gating values still come from the original affinity scores.Source: §2.1.2 · pp. 8–9
  • A tiny sequence-wise balance loss guards per-sequence imbalance, node-limited routing caps each token at M nodes for near-full computation-communication overlap, and tokens are never dropped in training or inference.DeepSeek-V3 also adopts a restricted routing mechanism to limit communication costs during training.Source: §2.1.2 · pp. 9–10

2.2. Multi-Token Prediction

  • MTP sequentially predicts multiple future tokens at each position while preserving the complete causal chain; its depth-wise cross-entropy objective is reported to enhance evaluation performance.At inference, the MTP modules can be discarded or repurposed for speculative decoding.Source: §2 · p. 6, §2.2 · pp. 10–11

3. Infrastructures … 3.3. FP8 Training

DeepSeek-V3’s infrastructure combines distributed training and communication-overlap techniques with a fine-grained FP8 training framework designed to reduce overhead while maintaining accuracy. The framework’s relative loss error remained below 0.25% versus BF16 in tests on two model scales trained for approximately 1 trillion tokens.

3.1. Compute Clusters

  • Training uses 2048 H800 GPUs, HAI-LLM, 16-way pipeline parallelism, 64-way expert parallelism across eight nodes, and ZeRO-1 data parallelism.Within nodes, GPUs connect through NVLink and NVSwitch; InfiniBand connects nodes. DualPipe, optimized all-to-all kernels, and memory optimizations enable training without tensor parallelism.Source: §3.1 · p. 11, §3.2 · p. 12

3.2.1. DualPipe and Computation-Communication Overlap

  • DualPipe overlaps forward and backward computation with communication to reduce pipeline bubbles and hide all-to-all and pipeline-parallel communication.It uses bidirectional scheduling, feeding micro-batches from both pipeline ends; the reported computation-to-communication ratio is approximately 1:1.Source: §3.2.1 · p. 12, Figure 4 · p. 12

3.2.2. Efficient Implementation of Cross-Node All-to-All Communication

  • Cross-node kernels overlap InfiniBand and NVLink transfers, allowing each token to select an average of 3.2 experts per node while using 20 SMs.Tokens are dispatched to at most four nodes; DeepSeek-V3’s eight routed experts can scale to a maximum of 13 at the same communication cost.Source: §3.2.2 · pp. 13–14

3.2.3. Extremely Memory Saving with Minimal Overhead

  • Memory-saving techniques recompute RMSNorm and MLA up-projections, store EMA parameters in CPU memory asynchronously, and share embedding and output-head parameters and gradients.Recomputation reduces activation storage with minor overhead, while CPU EMA updates add no memory or time overhead.Source: §3.2.3 · p. 14

3.3. FP8 Training

  • Below 0.25% relative loss error versus BF16 was maintained by FP8 training across two model scales trained for approximately 1 trillion tokens.The tests ran for approximately 1 trillion tokens; the reported error remained within the acceptable range of training randomness.Source: §3.3 · p. 15

3.3.1. Mixed Precision Framework

  • Most compute-intensive GEMMs use FP8, while embeddings, output heads, MoE gating, normalization, and attention retain BF16 or FP32 for stability.FP8 Fprop, Dgrad, and Wgrad theoretically double computational speed versus BF16; FP8 Wgrad also allows activations to be stored in FP8 for backward computation.Source: §3.3.1 · pp. 15–16

3.3.2. Improved Precision from Quantization and Multiplication

  • Fine-grained quantization and FP32 accumulation address FP8 errors; setting the accumulation interval to 128 elements significantly improves precision without substantial overhead.Activation tiles are 1×128 and weight blocks 128×128; the H800’s limited-precision accumulation produced nearly 2% maximum relative error in a preliminary K = 4096 test. The method uses E4M3 for all tensors and computes scales online.Source: §3.3.2 · pp. 16–18

3.3.3. Low-Precision Storage and Communication

  • Low-precision storage and communication use BF16 for AdamW moments and FP8 for cached activations and MoE dispatch, while retaining FP32 master weights and gradients.Some activation paths use E5M6 or recomputation, and forward and backward combine components remain BF16 to preserve precision in critical pipeline parts.Source: §3.3.3 · p. 18

3.4. Inference and Deployment · 3.5. Suggestions on Hardware Design

DeepSeek-V3 separates prefilling and decoding deployments to balance online-service SLOs and throughput. Its hardware suggestions target communication offload and improved support for FP8 computation and quantization.

3.4. Inference and Deployment

  • DeepSeek-V3 separates prefilling and decoding deployment to meet online-service SLOs while maintaining throughput.The deployment uses an H800 cluster with intra-node NVLink and cluster-wide InfiniBand connectivity.Source: §3.4 · p. 18

3.4.1. Prefilling

  • Prefilling uses TP4/SP with DP8 for attention and EP32 for MoE, plus redundant experts to balance GPU token loads.The deployment sets 32 redundant experts, one additional expert per GPU, and periodically adjusts placement from online load statistics; two micro-batches are overlapped to hide communication. A dynamic routing strategy with more resident experts is under exploration.Source: §3.4.1 · p. 19

3.4.2. Decoding

  • Decoding uses TP4/SP with DP80 for attention and EP320 for MoE, with direct IB point-to-point transfers and IBGDA (NVIDIA, 2022) to reduce latency.Two-micro-batch overlap is also being explored; because decoding attention is time-dominant and MoE batches are usually within 256 tokens, only a small portion of SMs can be allocated to dispatch, MoE, and combine.Source: §3.4.2 · pp. 19–20

3.5. Suggestions on Hardware Design · 3.5.1. Communication Hardware

  • Hardware suggestions based on the all-to-all and FP8 training implementations include offloading communication from SMs and unifying IB and NVLink access.The current communication implementation allocates 20 of the H800’s 132 SMs and leaves tensor cores under-utilized; proposed co-processors would expose read, write, multicast, and reduce across the unified network domain.Source: §3.5 · p. 20, §3.5.1 · p. 20

3.5.2. Compute Hardware

  • Future Tensor Cores should use higher-precision FP8 GEMM accumulation; Hopper’s 14-bit accumulation limitation is partially mitigated with FP32 accumulation in CUDA cores.The implementation accumulates results from 128 FP8×FP8 multiplications in FP32, a compromise rather than a hardware fix.Source: §3.5.2 · pp. 20–21
  • Future chips should support fine-grained tile- and block-wise quantization within Tensor Cores to avoid repeated data movement between Tensor Cores and CUDA cores.The proposal is to pass scaling factors into Tensor Cores and perform group-scaled MMA, accumulation, and dequantization there.Source: §3.5.2 · p. 21
  • Future chips should fuse FP8 casting with TMA access during activation transfer, and support warp-level casting or near-memory casting.The current workflow writes quantized FP8 activations to HBM before reading them again for MMA; near-memory casting is estimated to reduce off-chip memory access by roughly 50%.Source: §3.5.2 · p. 21
  • Future chips should support transposed matrix reads from shared memory before MMA to reduce memory operations in transposed GEMM workflows.The proposed support would streamline the workflow of dequantizing, transposing, and re-quantizing stored activations, especially when combined with fused FP8 conversion and TMA access.Source: §3.5.2 · p. 21

4. Pre-Training

DeepSeek-V3’s pre-training combines a 14.8T-token multilingual corpus and a 671B-parameter MoE model with staged extension to 128K context. Evaluations report strong base-model results and support for multi-token prediction and auxiliary-loss-free balancing.

4.1. Data Construction

  • The 14.8T-token corpus expands mathematical and programming data and multilingual coverage, reduces redundancy, and uses document packing and document-level FIM.The tokenizer uses byte-level BPE with a 128K vocabulary and randomized splitting of some punctuation-and-line-break tokens to mitigate token-boundary bias. FIM uses the PSM framework at a rate of 0.1.Source: §4.1 · pp. 21–22

4.2. Hyper-Parameters

  • The 61-layer model has 671B total parameters, with 37B activated per token, and uses 256 routed experts with 8 activated per token.Pre-training uses 4K sequences on 14.8T tokens; the learning rate decays from 2.2 × 10^-4 to 2.2 × 10^-5 before the final 500B tokens, which use two constant rates. MTP predicts one additional token, with its loss weight changing from 0.3 to 0.1.Source: §4.2 · pp. 22–23

4.3. Long Context Extension

  • Two 1,000-step YaRN phases extend the context window from 4K to 32K and then 128K; following supervised fine-tuning, the model performs robustly on NIAH through 128K.The phases use sequence lengths of 32K and 128K, respectively, with learning rate 7.3 × 10^-6; Figure 8 reports the NIAH results.Source: §4.3 · p. 23, Figure 8 · p. 23

4.4.1. Evaluation Benchmarks

  • Evaluation covers primarily English and Chinese benchmarks plus a multilingual benchmark, using perplexity, generation, or language-modeling evaluation as appropriate.Pile-test uses Bits-Per-Byte to support fair comparison across models with different tokenizers.Source: §4.4.1 · p. 24

4.4.2. Evaluation Results

  • DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base, and surpasses LLaMA-3.1 405B Base on most benchmarks.It shows particular advantages over Qwen2.5 72B Base on English, multilingual, code, and math tasks, and over LLaMA-3.1 405B Base on multilingual, code, and math tasks. Training requires 180K H800 GPU hours per trillion tokens.Source: §4.4.2 · pp. 25–26, Table 3 · p. 25

4.5.1. Ablation Studies for Multi-Token Prediction

  • The 1-depth MTP strategy consistently improves performance on most benchmarks across small and large MoE ablations, with inference costs unchanged because the MTP module is discarded.The ablations compare 15.7B-parameter models trained on 1.33T tokens and 228.7B-parameter models trained on 540B tokens, keeping data and other architectures the same.Source: §4.5.1 · p. 26, Table 4 · p. 26

4.5.2. Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy

  • The auxiliary-loss-free strategy consistently improves performance on most benchmarks versus auxiliary-loss-based balancing in small and large MoE ablations.The comparisons remove auxiliary losses while holding training data and other architectures constant; the baselines use sigmoid gating with top-K affinity normalization.Source: §4.5.2 · pp. 26–27, Table 5 · p. 27

4.5.3. Batch-Wise Load Balance VS. Sequence-Wise Load Balance

  • Batch-wise balancing allows greater expert specialization than sequence-wise balancing, and batch-wise auxiliary loss matches auxiliary-loss-free validation losses in the reported 1B and 3B experiments.At 1B, validation losses are 2.258 for sequence-wise auxiliary loss and 2.253 for both batch-wise auxiliary loss and auxiliary-loss-free balancing; at 3B, they are 2.085 and 2.080, respectively. Large micro-batches and redundant expert deployment address potential load-imbalance challenges.Source: §4.5.3 · pp. 27–28, Figure 9 · p. 28

5. Post-Training

DeepSeek-V3’s post-training combines supervised fine-tuning, reward-guided reinforcement learning, and multi-token prediction. Evaluations report strong performance across standard and open-ended benchmarks, while ablations examine reasoning-data distillation and self-rewarding.

5.1. Supervised Fine-Tuning

  • The SFT pipeline uses 1.5M cross-domain instances and expert-generated reasoning data to balance R1’s accuracy with clearer, more concise responses.After expert-model SFT and RL, rejection sampling curates final SFT data that retains R1’s strengths while producing concise responses.Source: §5.1 · pp. 28–29

5.2.1. Reward Model; 5.2.2. Group Relative Policy Optimization; 5.4.2. Self-Rewarding

5.3.2. Standard Evaluation

  • DeepSeek-V3 is the best-performing open-source chat model and is competitive with frontier closed-source models, including GPT-4o and Claude-3.5-Sonnet.The comparison evaluates a range of instructed-model benchmarks and representative open- and closed-source baselines.Source: §5.3.1 · p. 30, Table 6 · p. 31, §5.3.2 · p. 31
  • DeepSeek-V3 is competitive with top-tier models on MMLU, closely trails Claude-Sonnet 3.5 on MMLU-Pro, and ranks just behind Claude 3.5 Sonnet on GPQA-Diamond.It scores 91.6 F1 on 3-shot DROP, leads on LongBench v2, and closely trails GPT-4o on FRAMES; SimpleQA is a relative weakness, while C-SimpleQA is strong.Source: §5.3.2 · p. 32
  • DeepSeek-V3 beats all baselines on HumanEval-Mul and LiveCodeBench, while Qwen2.5 72B is about 10% behind on AIME, MATH-500, and CNMO 2024 absolute scores.On Chinese SimpleQA it exceeds Qwen2.5-72B by 16.4 points; on C-Eval and CLUEWSC, the models perform similarly.Source: §5.3.2 · pp. 32–33

5.3.3. Open-Ended Evaluation; 5.3.4. DeepSeek-V3 as a Generative Reward Model

  • DeepSeek-V3 wins over GPT-4-0314 at over 86% on Arena-Hard and exceeds 85% as the first open-source model; it also outperforms DeepSeek-V2.5-0905 by 20% on AlpacaEval 2.0.On RewardBench, its judgment ability is on par with the best GPT-4o-0806 and Claude-3.5-Sonnet-1022 versions, and voting is used to provide self-feedback on open-ended questions.Source: §5.3.3 · p. 33, §5.3.4 · p. 33, Table 8 · p. 34

5.4.1. Distillation from DeepSeek-R1

  • Distillation from DeepSeek-R1 improves LiveCodeBench and MATH-500 performance but substantially increases average response length, motivating a balance between accuracy and computational efficiency.The experiments use DeepSeek-V2.5, comparing short-CoT data with expert-checkpoint-generated data; the authors identify broader cognitive tasks as a direction for future exploration.Source: Table 9 · p. 34, §5.4.1 · p. 34

5.4.3. Multi-Token Prediction Evaluation

  • With speculative decoding, predicting two tokens rather than one yields 85%–90% acceptance for the second token and 1.8 times TPS across generation topics.This evaluation reports consistent reliability across varied generation topics.Source: §5.4.3 · p. 35

6. Conclusion, Limitations, and Future Directions

DeepSeek-V3 combines strong open-source performance with economical training, while the authors identify deployment constraints and outline directions for improving model efficiency, capabilities, and evaluation.

Conclusion

  • DeepSeek-V3 is described as the strongest available open-source model, comparable to GPT-4o and Claude-3.5-Sonnet, and required 2.788M H800 GPU hours for full training.The 671B-parameter MoE model activates 37B parameters and was trained on 14.8T tokens; its training uses FP8 and engineering optimizations, and post-training distills reasoning from DeepSeek-R1 models.Source: §6 · p. 35

Limitations

  • DeepSeek-V3’s recommended deployment unit may burden small teams, and its generation speed, though more than twice DeepSeek-V2’s, could improve further; advanced hardware may address these limits.Source: §6 · p. 35

Future Directions

  • The authors plan to refine architectures for better training and inference efficiency, support for infinite context, and expanded modeling beyond Transformer’s architectural limits.Source: §6 · p. 35
  • The authors plan to improve training-data quantity and quality, incorporate additional training signals, and scale data across more dimensions.Source: §6 · p. 36
  • The authors aim to strengthen reasoning by expanding models’ reasoning length and depth, improving intelligence and problem-solving.Source: §6 · p. 36
  • The authors plan broader, multidimensional evaluation to reduce fixed-benchmark optimization that can mislead assessments of model capabilities.Source: §6 · p. 36

Appendix · A. Contributions and Acknowledgments · B. Ablation Studies for Low-Precision Training

The low-precision training ablation includes a loss-curve comparison between BF16 and FP8 training.

B. Ablation Studies for Low-Precision Training

  • Figure 10 compares BF16 and FP8 training loss curves smoothed with EMA coefficient 0.9.Source: Figure 10 · p. 47

B.1. FP8 v.s. BF16 Training

The FP8 mixed-precision framework is validated against BF16 training on baseline MoE models at two scales, with relative error below 0.25% using high-precision accumulation and fine-grained quantization.

B.1. FP8 v.s. BF16 Training

  • The comparison covers baseline MoE models of approximately 16B parameters trained on 1.33T tokens and approximately 230B parameters trained on around 0.9T tokens.Source: §B.1 · p. 47
  • Relative error remains below 0.25% in the FP8-versus-BF16 comparison with high-precision accumulation and fine-grained quantization.Source: §B.1 · p. 47

B.2. Discussion About Block-Wise Quantization

The discussion tests block-wise quantization of Dgrad tensors as a simpler alternative to pass-specific activation groupings. On an approximately 16B-parameter MoE trained for around 300B tokens, block-wise quantization of activation gradients caused divergence, which the authors hypothesize stems from token-correlated outliers.

B.2. Discussion About Block-Wise Quantization

  • Block-wise quantization of activation gradients caused divergence in an approximately 16B-parameter MoE trained for around 300B tokens.The experiment quantized all Dgrad-associated tensors on a 128x128 block basis.Source: §B.2 · pp. 47–48
  • The authors hypothesize that token-imbalanced activation gradients create token-correlated outliers that block-wise quantization cannot effectively manage.The proposed approach was intended to simplify backward quantization by requiring only transposition, but the outliers were not handled effectively.Source: §B.2 · pp. 47–48

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models

On the Pile test set, the 16B auxiliary-loss-free model shows greater expert specialization than the auxiliary-loss-based baseline across all layers.

C. Expert Specialization Patterns of the 16B Aux-Loss-Based and Aux-Loss-Free Models

Related papers

Loading 2412.19437v2…