Source-linked AI summary
Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration
Sherry Xu, Marco Heddes, Jackson Peng, Tom Savell, Monica Tang, Prashant Ranjan, Jesse Benson, Ofer Dekel, Saurabh Dighe, Anupama Kurpad, Artour Levin, Matthew Mattina, George Petre, Cheng Tang, Yuan Yu, Li Zhang, Torsten Hoefler
TL;DR
Large-scale AI inference is driving substantial compute demand while stressing data management, memory bandwidth, and communication. The paper introduces Maia 200 and the SDLA architecture, which separates control from dataflow and explicitly orchestrates localized memories and movement engines. Maia 200 reports 30% lower cost and 15% lower energy than other AI accelerators in Microsoft’s fleet, alongside high utilization and communication efficiency.
Problem
Large-scale AI workloads make storing, moving, and converting data a central modern computer-architecture challenge while consuming growing compute capacity.
Method
Maia 200 implements SDLA, separating control and dataflow so software can asynchronously orchestrate specialized memories, data-movement engines, and synchronization.
Results
30% cost and 15% energy savings are reported versus any other AI accelerator in Microsoft’s fleet.
Takeaways & Limitations
SDLA combines explicit dataflow orchestration with local data access to support efficient, highly parallel AI inference systems.
Abstract
from arXiv · showhide
We introduce Maia 200, an advanced AI accelerator delivering high performance-10 145 Tflop/s FP4 and 5072 Tflop/s FP8 within a 750W TDP and 7 TB/s HBM bandwidth. Maia exemplifies a new class of Software Defined Locally Accessed Dataflow Architectures (SDLA), which explicitly program dataflow engines to orchestrate highly specialized memories and data movement engines. This approach shifts the focus from today's thread-centric to data-movement-centric architecture, improving efficiency and scalability. Our taxonomy of data management, inspired by Flynn's classification, highlights how SDLA addresses challenges in modern AI computing. Maia 200 achieves significant cost and energy savings while supporting massive parallelism for AI inference workloads, making it a compelling solution for next-generation high-performance computing systems.
I. INTRODUCTION
Maia 200 targets the growing demands of large-scale AI inference with a data-movement-centric architecture called Software Defined Locally Accessed Dataflow Architecture (SDLA). SDLA separates control and dataflow while combining explicit orchestration with localized, specialized memories.
- Motivation: 1.2 trillion tokens generated daily motivates efficient high-performance systems for large-language-model inference.The paper frames inference as a dominant consumer of worldwide compute cycles.
- Maia 200: 10 145 Tflop/s FP4 and 5072 Tflop/s FP8 per chip are delivered within a 750W TDP and 7 TiB/s HBM bandwidth.A distributed 6144-chip system reaches 62 exaflop/s FP4 throughput, 43 PiB/s memory, and 8.6 PiB/s Ethernet bandwidth.
- SDLA: SDLA explicitly separates parallel control and data instruction streams to orchestrate specialized memories and data-movement engines with localized data access.This shifts architectural emphasis from threads toward data movement.
- Taxonomy: The data-management taxonomy classifies processors by local versus global data access and by integrated load/store versus separately orchestrated data movement.Its four quadrants include LSGA, LSLA, SDGA, and SDLA examples.
- SDLA: SDLA separates dataflow execution from control, enabling fine-grained asynchronous orchestration and overlap of computation with data movement.Distributed memories also support right-sized placement and specialized movement and type-conversion engines.
III. THE MAIA 200 SYSTEM
Maia 200 is an SDLA accelerator optimized through broad co-design for Microsoft’s inference workloads while retaining support for future workload shifts. Its architecture targets lower total cost and energy consumption without overspecializing to a single problem.
- System goals: Maia 200 is Microsoft’s second-generation AI accelerator implementing an SDLA dataflow architecture for extreme-scale transformer-based LLM inference.The design addresses cost and energy consumption across the worldwide fleet.
- Co-design: Extreme co-design of hardware, software, deployment, and operations delivers lower TCO and energy consumption for a defined workload.The architecture is explicitly optimized around the complete system rather than hardware alone.
- System goals: Maia 200 is designed to support future workload shifts without becoming overspecialized.
- Architecture: Specialized small memories attached to functional units improve silicon and energy efficiency.
- Architecture: Hierarchical organization of memories and compute units exploits workload and programming locality.
- System integration: The programmable data path extends from local SRAM or HBM to remote SRAM or HBM across the network.The chip uses a 3nm process, more than 140 billion transistors, and a 750W SoC TDP in a scalable system architecture.
A. Inference workload challenges at scale
Large-LLM inference must meet soft real-time latency requirements while handling distinct prefill and decode phases, large memory demands, and increasingly long contexts. Maia 200 addresses these pressures with a cluster-scale architecture supporting phase specialization and high network bandwidth.
- Latency requirements: 300–4000 ms to first token and 20–30 ms per token illustrate the soft real-time requirements of large-LLM inference.Inference is generally divided into prefill and decode phases.
- Memory and batching: Decode requires loading a separate KV cache for each batch entry, while batch size is limited by service-level agreements, memory capacity, and parallelization.Lower-precision FP4 or FP8 can be used with fine-grained scaling factors.
- Scale: 62 exaflop/s FP4 performance and 8.6 PiB/s network connectivity support cluster-scale inference and specialization across workload phases.The system is designed as a petaflop/s-class accelerator cluster for multiple simultaneous requests.
- Future workloads: Reasoning, multimodal, and retrieval-augmented workloads require context windows ranging from hundreds of thousands to millions of tokens and extreme bandwidth.
- Future workloads: Growing models increase compute power and memory-bandwidth requirements per token, while diffusion models require additional compute across many iterations.
B. The Maia 200 System on Chip
The Maia 200 SoC combines specialized memories, programmable data-movement engines, separate control and data networks, and high-bandwidth connectivity. Its hierarchical tile-and-cluster organization implements SDLA principles across computation, memory, synchronization, and networking.
- SoC features: Maia 200 includes specialized memories, programmable data-movement accelerators, narrow datatypes, high HBM bandwidth, flexible tensor cores, and power management.These features support efficient AI inference and training across multiple deployment configurations.
- Networking: The NoC is split into logical data and control networks, with QoS on the data NoC to prioritize traffic from sources such as NICs or HBM.
- SDLA innovations: Separate control and data management allow Maia 200 to optimize specialized DMA, synchronization, and processing units.
- Data movement: Specialized DMA engines support strided 1D-, 2D-, 3D-, and 4D-tensor operations, scatter/gather, datatype casting, and sparsity support.
- Organization: Four clusters contain nine or ten tiles each, with tile tensor and vector processors, local memories, DMA and synchronization engines, and control processors.Cluster and device networks connect these units to HBM, PCIe, and integrated NICs.
1) Explicit Scratchpad versus Caches for SoC SRAM:
Maia 200 uses explicitly programmed scratchpad memories rather than relying solely on caches, trading programming simplicity for control over data placement and movement. This approach reduces memory overhead while supporting efficient AI data transfers.
- Explicit Scratchpad versus Caches for SoC SRAM:: The paper frames scratchpads as a tradeoff: they require more programming control, while caches simplify initial implementation and non-performance-critical code.Cache behavior can be complex and non-portable when optimizing replacement, associativity, and cache-line sizes.
- Explicit Scratchpad versus Caches for SoC SRAM:: Scratchpad memories let programmers place and pack data at word granularity, avoiding cache-line sharing for fine-grained, non-contiguous data.Explicit management also removes the need to reason about replacement schemes and hidden prefetch streams.
- Explicit Scratchpad versus Caches for SoC SRAM:: Cache hardware adds tag arrays and address-remapping logic, creating area, latency, and dynamic-energy overheads.The cited analysis reports roughly 30–35% area and energy overhead for this logic, 10–15% higher access latency, and up to 43% overall area-time and dynamic-energy overhead in practice.
- Explicit Scratchpad versus Caches for SoC SRAM:: Maia 200 spends less than 20% of chip space on memory using its explicitly programmed SDLA memory architecture.The SRAMs and HBM are ECC protected, while specialized DMA engines occupy negligible die area.
- Explicit Scratchpad versus Caches for SoC SRAM:: Maia 200 combines software-managed scratchpads with compiler support so programmers can choose higher-level interfaces or explicit C/C++ control for efficiency.The paper states that its focus is hardware and system architecture rather than describing the software stack.
2) Dataflow Programming and Instructions:
Maia 200 implements dataflow programming through specialized compute and movement engines governed by a dataflow instruction set. Hierarchical control processors configure these operations ahead of time so execution proceeds asynchronously.
- Dataflow Programming and Instructions:: Maia 200 combines Tile Tensor Units, Tile Vector Processors, and DMA engines for specialized computation and data movement.TTUs support fixed dataflow functions such as matrix multiplication and convolutions, TVPs run C/C++ programs, and DMA engines move data between memories.
- Dataflow Programming and Instructions:: DISA instructions define dataflow preconditions, invoke a macro-instruction, and specify postconditions for execution engines.Macro-instructions can be complete programs, as in TVPs, or fixed functions, as in TTUs and DMA engines.
- Dataflow Programming and Instructions:: Hierarchical control processors set up dataflow instructions and semaphores ahead of time while the dataflow program executes asynchronously.The control program usually runs ahead, enabling optimal dataflow for oblivious programs without control-induced delays.
3) Connecting Memories and Compute Engines:
Maia 200 connects compute engines to directly attached, specialized memories and numerous DMA paths. The design sizes and formats these resources for overlapped execution, reuse, and efficient data conversion.
- Connecting Memories and Compute Engines:: Directly attached memories are sized using Little’s Law to sustain pipelined, overlapped loading and computation while maximizing reuse.This design also minimizes data movement and addressing overheads.
- Connecting Memories and Compute Engines:: Each tile provides 3 MiB of specialized SRAM for TTU operands.The supplied passage identifies this SRAM as the storage attached to each tile’s tensor-multiply workload.
- Connecting Memories and Compute Engines:: At 2 GHz, each TTU provides 2024 GiB/s input bandwidth per operand and up to 4096 GiB/s output bandwidth.An additional 4096 GiB/s read bandwidth supports accumulation, and the memories hold more than 700 cycles of TTU work.
- Connecting Memories and Compute Engines:: Maia 200 uses many DMA engines to move data among on-chip SRAM, HBM, and remote memories over the network.NoC quality-of-service priorities are assignable per command, and Sync blocks can notify one another across the SoC.
- Connecting Memories and Compute Engines:: The Reshaper converts between storage and compute formats, including sparse 8:16 and 4:16 tensors and dense tensors at full bandwidth.Conversions can use stochastic rounding or round-to-nearest while storage formats reduce transported and stored bits.
- Connecting Memories and Compute Engines:: Each TTU supports BF16, FP8, FP6, and FP4, with 8192, 32 768, 32 768, and 65 536 MACs, respectively.FP6 matches FP8 performance while reducing energy through gating.
- Connecting Memories and Compute Engines:: The supported low-precision formats include E4M3 and E5M2 for FP8, E2M3 and E3M2 for FP6, and E1M2 and E2M1 for FP4.Maia 200 also composes these formats into OCP-compliant MXFP types with E8M0 scaling factors and groups of 32.
5) Software-Defined Dataflow Control:
Maia 200 completes its SDLA design with hierarchical control processors that program and coordinate dataflow engines. This control operates ahead of execution and supports data-dependent handoffs within the tile hierarchy.
- Software-Defined Dataflow Control:: Maia 200 uses SoC-, Cluster-, and Tile-level control processors running C/C++ programs to orchestrate the data path.The processors generally run slightly ahead, configuring engines before use, while staging memories preserve enough queued work to avoid idle units.
- Software-Defined Dataflow Control:: Data-dependent results can pass quickly from the data path to the control path within a tile.For mixture-of-experts routing, the TVP evaluates TTU output in TSRAM and hands the result to the TCP to initiate preconfigured data movement.
- Software-Defined Dataflow Control:: The paper notes that Maia 200’s management functionality is complex, including control-processor configuration and kernel launching.These details are explicitly treated as less scientifically interesting than the performance-relevant aspects highlighted by the paper.
- Software-Defined Dataflow Control:: Maia 200 uses separate frequency domains for its four clusters and Global NoC in a flexible DVFS system.This enables configurations with high compute and memory clocks for prefill or lower compute and high memory clocks for token generation.
7) Roofline Matrix Multiply Performance:
Maia 200 sustains high roofline efficiency for bf16 and fp8 matrix multiplication across compute-bound and memory-bound regimes. Its programmable data movement and integrated network engines support explicit, asynchronous movement across local and remote memories.
- bf16 matrix multiplication: 99.69% of peak bf16 performance is achieved in the compute-bound regime through asynchronous data movement and explicit SDLA programming.Instructions and data loading overlap, with semaphores synchronizing busy execution units.
- bf16 matrix multiplication: 51.4% of peak bandwidth is achieved in the memory-bound bf16 regime, with higher than 50% bandwidth for operands exceeding 113.5 MiB.Larger matrices quickly become compute bound.
- fp8 matrix multiplication: 96% of peak fp8 performance is achieved in the compute-bound regime, while memory-bound fp8 multiplication reaches up to 56% of peak.The fp8 study uses the same matrix sizes and experimental methodology across arithmetic intensities from 95 to 4600.
- networked data movement: Maia 200 integrates 28 400 Gbps Ethernet-based AI Network Controllers providing 1.4 TB/s full-duplex bandwidth and programmable DMA access to remote SRAM and HBM.The controllers also provide synchronization and seamless data movement within the SDLA microarchitecture.
- networked data movement: Receiver-driven messaging exposes remote-memory access through send and receive buffers, implemented with RDMA in the AI Network Controllers.The scheme is optimized for collective communication libraries to minimize message-matching state.
1) Collective Communication Performance:
Maia 200 evaluates direct-connect and ring collective algorithms for production-relevant Allgather workloads across chips. On eight chips, measured performance reaches close to architectural speed-of-light limits.
- Collective algorithms: Direct-connect sends data pieces directly to target endpoints, whereas ring pipelines chunks through a logical one-dimensional ring.The two algorithms differ in message depth and communication path dependence.
- Network topology: The estimated maximum small-message latency through the switched network is around 4us.Tray links provide different directional bandwidths, while switched links can flexibly connect the system.
- Collective communication performance: Allgather collects data from all chips, with each chip receiving R = N(P −1) Bytes for communicator size P and per-process volume N.The operation is central to Fully Sharded Data Parallel LLM inference and 3D parallelism using narrow datatypes.
- Collective communication performance: 78% of the latency bound and 94% of the bandwidth bound are achieved on eight Maia 200 chips for production-relevant data sizes.The bounds are defined by architectural speed-of-light limits.
2) Azure integration:
Maia 200 is integrated into Azure datacenters and evaluated on an end-to-end Qwen 2.5 7B inference workload. The study emphasizes memory-bound token generation and reports strong practical SDLA performance beyond isolated kernels.
- Azure integration: Maia 200 supports liquid-cooled deployment and air-cooled datacenters through an integrated heat exchanger.It uses standard Ethernet cabling and switches for networking.
- End-to-end inference: The Qwen 2.5 7B evaluation runs on one Maia 200 chip using SDLA features across a 28-layer transformer with 3584-dimensional inner layers.The model includes 28 attention heads with grouped query attention and four KV heads.
- End-to-end inference: The token-generation analysis is fully memory bound at S=1 and loads 112 small matrices, 84 medium-sized matrices, and one approximately 1 GB matrix.The workload uses 16 384 previous tokens in the analyzed long-generation case.
- Results: Maia 200 performs well on complex end-to-end practical inference workloads as well as near-peak matrix multiplication and collective operations.The implementation was validated for correctness against existing GPU inference solutions.
- Results: SDLA enables explicit management of memory allocation and movement and asynchronous control codes targeting near-100% utilization of execution units.The paper presents this capability as a basis for further optimizations that inline operators and minimize data movement.
V. RELATED WORK
The paper positions SDLA as an extension of accelerator design toward software-defined dataflow and explicit data management. Its conclusion connects the architecture to specialized memories, movement engines, and production deployment for AI and HPC.
- Related accelerators: SDLA extends the accelerator lineage toward software-defined dataflow, while related systems include GPUs, TPUs, Trainium, Inferentia, and MTIA.The paper frames Maia 200 as part of the broader development of AI-specific accelerators.
- Architectural positioning: Unlike GPU approaches that combine specialized movement engines with SIMT or warp systems, SDLA makes memory orchestration explicit and localized.The paper contrasts SDLA with NVIDIA’s TMA and its interaction with GPU warp mechanisms.
- Programming model: Manual scheduling practices such as Flash Attention motivate programming approaches that adjust kernels to each GPU architecture.Frameworks including Triton and DaCe are presented as attempts to support this direction.
- Discussion and conclusions: SDLA combines specialized data movement through distributed memories and networks with specialized control through explicit DMA, synchronization, and control units.The conclusion presents these components as a highly efficient architecture deployed in Microsoft’s production fleet.