PaperlayerView exact revision on arXiv

Source-linked AI summary

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, Wenfeng Liang, Ying He, Yuqing Wang, Yuxuan Liu, Y. X. Wei

arXiv:2505.09343v2cs.DCcs.AIcs.AR

TL;DR

The paper addresses hardware limitations in memory, computation, and interconnection bandwidth as LLM workloads scale. It analyzes DeepSeek-V3’s hardware-aware model and infrastructure co-design, reporting faster inference and discussing hardware directions for future systems. The paper concludes that hardware–software co-design provides a roadmap for scalable, efficient, and robust AI systems.

  • Problem

    Rapidly scaling LLMs face constrained memory capacity, computational efficiency, and interconnection bandwidth, while longer contexts and deeper reasoning increase inference demands.

  • Method

    The paper analyzes the interaction between DeepSeek-V3’s model architecture, low-precision computation, parallelism, networking, and hardware design, including its practical limitations.

  • Results

    MTP achieves an 80% to 90% acceptance rate for the second subsequent token and increases generation TPS by 1.8x.

  • Takeaways & Limitations

    DeepSeek-V3 provides a hardware–software co-design roadmap for improving the scalability, efficiency, and robustness of large-scale AI systems.

  • Takeaways & Limitations

    LogFMT is not employed because encode/decode overhead fused with all-to-all communication can reach 50%∼100%.

Abstract

from arXiv · show

The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model co-design can effectively address these challenges, enabling cost-efficient training and inference at scale. This paper presents an in-depth analysis of the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting key innovations such as Multi-head Latent Attention (MLA) for enhanced memory efficiency, Mixture of Experts (MoE) architectures for optimized computation-communication trade-offs, FP8 mixed-precision training to unlock the full potential of hardware capabilities, and a Multi-Plane Network Topology to minimize cluster-level network overhead. Building on the hardware bottlenecks encountered during DeepSeek-V3's development, we engage in a broader discussion with academic and industry peers on potential future hardware directions, including precise low-precision computation units, scale-up and scale-out convergence, and innovations in low-latency communication fabrics. These insights underscore the critical role of hardware and model co-design in meeting the escalating demands of AI workloads, offering a practical blueprint for innovation in next-generation AI systems.

1 Introduction

Rapidly growing LLM scale and inference demands are increasing pressure on computational resources, while large training clusters remain costly. The paper examines hardware–model co-design in DeepSeek-V3 to derive practical directions for scalable, cost-efficient AI systems.

  • 1 Introduction: Increasing model size, training data, and computational resources has driven substantial improvements in model performance.
  • 1 Introduction: Longer contexts and deeper reasoning increase the need for faster and more efficient inference.
  • 1 Introduction: Large training clusters with tens or hundreds of thousands of GPUs or TPUs enable state-of-the-art models but impose major costs on smaller organizations.
  • 1 Introduction: DeepSeek-V3 achieves state-of-the-art performance using 2,048 NVIDIA H800 GPUs, illustrating cost-effective training with existing hardware resources.
  • 1 Introduction: The paper studies hardware architecture and model design together rather than repeating DeepSeek-V3’s detailed architectural and algorithmic specifications.
  • 1 Introduction: Its stated focus covers hardware-driven model design, hardware–model dependencies, and future hardware directions for scalable, cost-efficient AI systems.

2 Design Principles for DeepSeek Models

DeepSeek-V3 uses hardware-aware model and infrastructure choices to address memory efficiency, cost-effectiveness, and inference speed. Its design combines MLA, DeepSeekMoE, FP8 mixed precision, speculative decoding, and a multi-plane network.

  • 2 Design Principles for DeepSeek Models: DeepSeek-V3 aligns each design decision with hardware constraints to optimize performance and cost efficiency.
  • 2 Design Principles for DeepSeek Models: MLA reduces memory consumption by compressing KV caches, while DeepSeekMoE supports sparse computation.
  • 2 Design Principles for DeepSeek Models: FP8 mixed-precision training lowers computational costs without compromising model quality.
  • 2 Design Principles for DeepSeek Models: A Multi-Token Prediction Module based on speculative decoding improves inference speed by increasing generation speed.
  • 2 Design Principles for DeepSeek Models: A Multi-Plane two-layer Fat-Tree network replaces a traditional three-layer Fat-Tree topology to reduce cluster networking costs.
  • 2 Design Principles for DeepSeek Models: These innovations target three scaling challenges: memory efficiency, cost-effectiveness, and inference speed.

2.1 Memory Efficiency

LLM memory demands are growing faster than high-speed memory capacity, and inference becomes memory-bandwidth-bound through KV caching. DeepSeek-V3 addresses this with MLA and FP8, while long-context decoding remains challenging.

  • 2.1 Memory Efficiency: LLM memory demands increase by more than 1000% per year, whereas HBM capacity typically grows by less than 50% annually.
  • 2.1 Memory Efficiency: FP8 reduces weight memory consumption by half compared with BF16 weights, alleviating memory constraints.
  • 2.1 Memory Efficiency: KV caching avoids recomputation across multi-turn requests but shifts inference toward memory-bandwidth-bound GEMV operations.
  • 2.1.2 Reducing KV Cache with MLA.: MLA compresses all attention heads’ KV representations into a smaller latent vector, so inference caches only that vector.
  • 2.1.2 Reducing KV Cache with MLA.: DeepSeek-V3’s MLA largely reduces BF16 KV-cache size compared with models using GQA.
  • 2.1.3 Future Directions and Perspectives on Resource-Efficient Tech-: Quadratic complexity in Transformer autoregressive decoding remains a challenge for extremely long contexts despite KV-cache reduction.

2.2 Cost-Effectiveness of MoE Models

DeepSeekMoE scales total parameters while activating only a subset per token, reducing computational requirements relative to dense models. This supports cost-efficient training and inference under limited hardware resources.

  • 2.2 Cost-Effectiveness of MoE Models: DeepSeekMoE scales total parameters while selectively activating only a subset of expert parameters for each token.DeepSeek-V3 has 671B total parameters and 37B activated per token, while DeepSeek-V2 has 236B and 21B respectively.
  • 2.2 Cost-Effectiveness of MoE Models: Dense models such as Qwen2.5-72B and LLaMa3.1-405B require all parameters to be active during training.
  • 2.2.1 Reducing Computational Requirements for Training.: 250 GFLOPS per token is reported for DeepSeek-V3, compared with 394 GFLOPS for Qwen2.5-72B and 2448 GFLOPS for LLaMa3.1-405B.The comparison assumes a sequence length of 4096.
  • 2.2 Cost-Effectiveness of MoE Models: MoE’s subset activation reduces memory and computational demands in single-request inference scenarios.
  • 2.2 Cost-Effectiveness of MoE Models: Nearly 20 TPS is achieved for complete DeepSeek-V3 inference on a low-cost server with a consumer GPU costing approximately $10,000.

2.3 Increasing Inference Speed

DeepSeek-V3 improves inference speed by overlapping communication with computation and by using algorithmic techniques that reduce sequential decoding bottlenecks. These designs target both throughput and latency, while hardware bandwidth remains a fundamental limit for MoE inference.

  • Overlapping Computation and Communication: Dual micro-batch overlap decouples MLA and MoE computation from dispatch communication, allowing the two micro-batches to execute these stages concurrently.This design intentionally overlaps communication latency with computation to maximize throughput.
  • Inference Speed Limits: MoE inference speed is ultimately bounded by interconnection bandwidth because expert parallelism requires all-to-all token routing across devices.Efficient expert placement reduces computation bottlenecks but makes network communication a key constraint.
  • Inference Speed Limits: 14.76 ms TPOT, equivalent to 67 tokens per second, is the theoretical upper limit calculated for the CX7 InfiniBand system.Practical performance is lower because of communication overhead, incomplete bandwidth utilization, latency, and computational inefficiencies.
  • Inference Speed Limits: 0.82 ms TPOT, or approximately 1200 tokens per second, is the theoretical upper limit under perfect overlap with a high-bandwidth scale-up network.The estimate is idealized and excludes the substantial GPU-efficiency drop that occurs at small batch sizes.
  • Multi-Token Prediction: Multi-Token Prediction generates candidate tokens cheaply and verifies them in parallel, reducing the sequential bottleneck of autoregressive decoding.The approach is designed to improve inference speed without compromising accuracy.
  • Multi-Token Prediction: An MTP module achieves an 80% to 90% acceptance rate for the second subsequent token and increases generation TPS by 1.8x.Each module uses a lightweight single layer, slightly reducing throughput while improving end-to-end generation latency.

2.4 Technique Validation Methodology

The paper validates each acceleration technique empirically while reducing the cost of full-scale ablation through a hierarchical validation pipeline.

  • Technique Validation Methodology: Each technique is validated on small-scale models, minimally tuned at large scale, and then integrated into a comprehensive training run.The validated techniques include MLA, FP8 mixed-precision computation, and network co-designed MoE gate routing.

3 Low-Precision Driven Design

DeepSeek-V3 uses FP8 mixed-precision training and low-precision communication to reduce memory and communication costs, while exposing hardware limitations that constrain efficiency and training stability. The paper proposes hardware support for more precise accumulation, native fine-grained quantization, and compression operations.

  • FP8 Mixed-Precision Training: FP8-compatible MoE training uses FP8 forward and backward computation with tile-wise 1x128 activation quantization and block-wise 128x128 weight quantization.The framework was developed through infrastructure–algorithm collaboration and its fine-grained FP8 GEMM implementation was open-sourced in DeepGEMM.
  • Limitations: FP8 training is limited by constrained Hopper Tensor Core accumulation precision and by dequantization overhead from fine-grained quantization.The latter requires frequent data movement between Tensor Cores and CUDA Cores, reducing computational efficiency.
  • Suggestions: Hardware should support configurable higher-precision accumulation and native fine-grained quantization inside Tensor Cores.These changes aim to trade performance against accuracy and avoid repeated partial-result transfers for scaling-factor multiplication.
  • Low-Precision Communication: Fine-grained FP8 quantization reduces MoE dispatch communication volume by 50% compared with BF16, while combine communication remains higher precision for accuracy requirements.The paper also evaluates FP8, E5M6, and mixed FP8-BF16 formats for further reduction.
  • Alternative Formats: LogFMT-8Bit achieves superior training accuracy to E4M3 and E5M2 in the reported dense-model validation, while LogFMT-10Bit is similar to BF16 for the combine stage.Despite these results, LogFMT is not deployed because encode/decode overhead can reach 50%∼100%.
  • Future Hardware: Dedicated hardware compression and decompression units for FP8 or custom formats could reduce bandwidth requirements and streamline communication pipelines.The paper identifies bandwidth-intensive MoE training as a particularly relevant use case.

4 Interconnection Driven Design

DeepSeek-V3 co-designs parallelism, expert routing, and communication mechanisms around the bandwidth asymmetry of H800 scale-up and scale-out networks. The design reduces inter-node traffic but shifts work to communication pipelines, motivating unified networks and dedicated hardware support.

  • Hardware Constraints: H800 SXM reduces NVLink bandwidth from 900 GB/s to 400 GB/s and compensates with eight 400G InfiniBand NICs per node.The reduced intra-node bandwidth creates a challenge for high-performance workloads while strengthening scale-out capability.
  • Hardware-Aware Parallelism: DeepSeek-V3 avoids tensor parallelism during training under limited NVLink bandwidth and uses DualPipe to overlap attention, MoE computation, and MoE communication.Tensor parallelism can still be selectively used during inference to improve TTFT and TPOT.
  • Hardware-Aware Parallelism: The H800 scale-up and scale-out bandwidth disparity is approximately 4:1, motivating model co-design that balances NVLink and InfiniBand utilization.The passage reports 200GB/s NVLink bandwidth versus 50GB/s per 400Gbps InfiniBand NIC before effective-bandwidth adjustments.
  • Expert Routing: Node-Limited Routing groups 256 routed experts into 8 node-local groups and restricts each token to at most 4 nodes, reducing InfiniBand communication requirements.This strategy enhances effective communication bandwidth during training.
  • Communication Pipeline Costs: Node-Limited Routing complicates communication kernels because SM threads handle both network message processing and NVLink forwarding.During training, these communication-related operations can consume up to 20 SMs, leaving fewer resources for computation.
  • Future Interconnects: Future hardware should unify intra-node and inter-node communication with dedicated co-processors for traffic management and forwarding between NVLink and InfiniBand.The proposal is intended to reduce software complexity and maximize bandwidth utilization.
  • Future Interconnects: Dedicated hardware should support forwarding, broadcast, reduction, memory-layout handling, and synchronization across scale-up and scale-out networks.Hardware synchronization could replace higher-latency software mechanisms for memory consistency and out-of-order packet handling.

4.5 Bandwidth Contention and Latency

Current interconnects cannot dynamically prioritize competing traffic, so KV-cache transfers can contend with EP communication and cause latency spikes. The paper proposes traffic prioritization, tighter NIC integration, and dedicated CPU–GPU fabrics to reduce these bottlenecks.

  • Bandwidth Contention and Latency: KV-cache transfers from CPU memory can saturate PCIe while IB carries EP traffic, causing contention and latency spikes.The paper proposes assigning different priorities to EP, TP, and KV-cache traffic.
  • Bandwidth Contention and Latency: The eight-plane fat-tree assigns each GPU–NIC pair to one plane, forwarding cross-plane traffic through another NIC and PCIe or NVLink.This forwarding path introduces an intra-node dependency for cross-plane communication.
  • Bandwidth Contention and Latency: An ideal multi-plane NIC would connect multiple physical ports to distinct planes and let one QP use all ports concurrently.Native out-of-order placement inside the NIC is required for this design.
  • Bandwidth Contention and Latency: Integrating NICs into the I/O die and connecting CPUs and GPUs with dedicated high-bandwidth fabrics could reduce latency and PCIe contention.The proposed alternatives include NVLink or similar scale-up interconnects.

5 Large Scale Network Driven Design

DeepSeek-V3 uses a multi-plane two-layer fat-tree to scale communication while limiting cost and latency. Experiments show performance comparable to single-plane multi-rail networks, although current hardware imposes cross-plane forwarding and RoCE faces congestion and scalability limitations.

  • 5.1 Network Co-Design: Multi-Plane Fat-Tree: The deployed MPFT network uses eight GPU–NIC pairs per node and 400 Gbps switches, theoretically supporting up to 16,384 GPUs in two layers.A separate RoCE NIC serves the storage network.
  • 5.1.1 Advantages of Multi-Plane Fat-Tree Network: MPFT provides independent traffic planes, fault isolation, load balancing, and scalability while reducing cost relative to three-layer fat trees.The topology supports over 10k endpoints using a two-layer fat tree.
  • 5.1.1 Advantages of Multi-Plane Fat-Tree Network: Current 400G NDR InfiniBand requires intra-node forwarding for cross-plane communication, adding inference latency.Scale-up and scale-out convergence could significantly reduce this overhead.
  • 5.1.2 Performance Analysis: MPFT and MRFT show nearly identical all-to-all performance, with negligible latency differences in 16-GPU tests.NCCL PXN optimizes multi-plane forwarding through NVLink.
  • 5.1.2 Performance Analysis: Each GPU exceeds 40GB/s in EP communication on MPFT, while DeepSeek-V3 training performance on 2,048 GPUs remains nearly identical to MRFT.The measured differences fall within normal fluctuations and measurement error.
  • 5.2.1 IB or RoCE: IB offers lower latency than RoCE but costs more and typically supports fewer switch ports, restricting scalability for large clusters.RoCE remains attractive on cost but does not yet meet large-scale AI latency and scalability demands.
  • 5.2.2 Recommendations for RoCE Improvements: RoCE’s ECMP routing can concentrate deterministic LLM-training flows on the same links, causing severe congestion in collective communication.Adaptive routing can spray packets across paths, while limited priority queues complicate mixed EP and DP traffic.
  • 5.2.2 Recommendations for RoCE Improvements: IBGDA lets GPUs manage RDMA control-plane operations directly, removing GPU–CPU communication overhead and distributing small-packet work across GPU threads.This addresses control-plane bottlenecks when sending many small packets.

6 Discussion and Insights for Future Hardware Architecture Design

The discussion extends DeepSeek-V3’s concrete hardware lessons into future architecture directions covering reliability, CPU–GPU integration, memory bandwidth, intelligent interconnects, and in-network communication. These proposals target bottlenecks exposed by increasingly large and communication-intensive AI workloads.

  • 6 Discussion and Insights for Future Hardware Architecture Design: The paper connects earlier hardware limitations with broader recommendations for future AI-oriented architecture design.The discussion explicitly builds on scale-up, low-precision, network-convergence, and multi-plane analyses.
  • Reliability: Interconnect disconnections and single-device failures can disrupt communication-heavy EP workloads or force costly restarts of long-running training jobs.The paper identifies IB, NVLink, node, GPU, and ECC-memory failures as risks.
  • Reliability: Checksum validation and hardware-accelerated redundancy checks are proposed to detect silent corruption beyond traditional ECC.Vendor diagnostic toolkits would support ongoing system-integrity verification.
  • CPU and Memory: PCIe can bottleneck parameter, gradient, and KV-cache transfers, motivating direct CPU–GPU interconnects or integrating both processors into the scale-up domain.Suggested fabrics include NVLink and Infinity Fabric.
  • CPU and Memory: 160 lanes of PCIe 5.0 require over 640 GB/s per node, translating to approximately 1 TB/s of memory bandwidth for sustained transfers.The paper presents this as a significant challenge for conventional DRAM.
  • CPU and Memory: Latency-sensitive workloads require high single-core CPU performance and sufficient CPU cores per GPU to avoid control-side bottlenecks.Chiplet architectures also need additional cores for cache-aware partitioning and isolation.
  • Interconnects: Future interconnects should combine low latency with intelligent traffic management, including co-packaged optics, lossless flow control, and endpoint-driven congestion control.Naive credit-based flow control can cause head-of-line blocking.
  • Communication Semantics: Memory-semantic communication currently incurs ordering-related RTT latency, so hardware support for acquire/release semantics and receiver-side in-order delivery is proposed.Candidate mechanisms include sequence-number buffering and region-based acquire/release.

7 Conclusion

The paper presents DeepSeek-V3 as evidence that hardware–software co-design can improve the scalability, efficiency, and robustness of large-scale AI systems. It concludes with a roadmap for AI-optimized hardware as workloads continue growing in complexity and scale.

  • 7 Conclusion: DeepSeek-V3 exemplifies hardware–software co-design aimed at advancing scalability, efficiency, and robustness in large-scale AI systems.The paper frames its recommendations as a roadmap for next-generation AI-optimized hardware.
Loading 2505.09343v2…