Source-linked AI summary
Balancing Generality and Specialization: A Survey on AI Datacenter Hardware Architecture
Yufeng Gu, Jiazhen Wang, Reetuparna Das
TL;DR
AI datacenter design must balance specialization and generality as workloads engage accelerators, interconnects, and facility infrastructure. This survey synthesizes accelerator architectures, communication fabrics, and generational changes; it concludes that compute-engine choices alone cannot capture the constraints on performance and hardware reuse.
Problem
AI workloads distribute computation and model state across devices, making execution dependent on accelerator resources, communication, and supporting infrastructure.
Source: §1 · p. 1Method
The survey classifies industrial accelerators, analyzes scale-up interconnects and collective communication, and traces architectural evolution and deployment constraints.
Source: §1 · p. 3, §5.2 · p. 20, §6.3 · p. 27Results
Specialization cannot be assessed from compute engines alone: data placement, communication overhead, power, and thermal constraints also affect performance and hardware reuse.
Source: §7 · p. 28Takeaways & Limitations
Architectural choices link efficiency with system flexibility and longevity as workloads evolve.
Source: §7 · p. 28Hard-wiring a model means replacing the base model requires new silicon, making hardware reuse dependent on continued demand for that model.
Source: §6.5 · p. 28
Abstract
from arXiv · showhide
Rapidly growing AI workloads are driving large investments in AI datacenters. This survey classifies industrial AI accelerators into four architectural categories and compares their compute and memory organizations. It examines how node-, rack-, and pod-scale interconnects support collective communication, and traces architectural evolution across accelerator generations. The analysis connects advances in arithmetic throughput with changes in precision, data delivery, execution coordination, communication, power delivery, and cooling. It also discusses future design challenges arising from workload diversity, data movement, infrastructure constraints, and model evolution, showing how the trade-off between generality and specialization extends from individual accelerators to datacenter-scale systems. GitHub: github.com/Yufeng98/AI-datacenter
1 Introduction
AI datacenter architecture must be understood across accelerators, interconnects, and facility infrastructure because large-model execution distributes computation and model state across connected devices. The survey organizes this landscape around the balance between specialization and generality, tracing architectural choices and their implications across hardware generations.
1 Introduction
- Large-model execution distributes computation and model state across devices, while interconnects, power delivery, and cooling shape operation across servers, racks, pods, and datacenters.Scale-up fabrics may span multiple servers, racks, or an entire pod, so physical levels do not impose rigid communication boundaries.Source: §1 · p. 1
- Model parameter counts grow faster than per-accelerator compute, HBM resources, and scale-up bandwidth, motivating analysis beyond arithmetic units.Across selected observations, parameter counts grow at approximately 248% annually, versus 67% for dense compute, 47% for HBM bandwidth, 42% for HBM capacity, and 30% for scale-up interconnect bandwidth.Source: §1 · p. 1
- The survey groups industrial accelerators into four categories by dominant execution and data-movement organization, then compares their compute engines and memory systems.The categories are GPUs, NPUs, spatial dataflow architectures, and compute-in-memory architectures; designs within a category can still vary in operations, memory organization, and general-purpose resources.Source: §1 · pp. 1–3, Table 1 · p. 2
- The generational analysis connects rising arithmetic throughput with changes in precision, operand delivery, execution coordination, communication, memory, and cooling.The survey’s Tensor Core case study examines how operand delivery and coordination support overlap between data movement and computation [121].Source: §1 · p. 3
- The survey’s four contributions cover accelerator taxonomy, scale-up fabrics and collective communication, generational change, and future design challenges.Its scope includes workload evolution, memory and communication constraints, power delivery, cooling, flexibility, and deployment longevity.Source: §1 · p. 3
2 Background and Related Works
The background situates industrial AI datacenters within rapidly rising compute and energy demands, then compares model growth with hardware scaling. It traces accelerator objectives and positions the survey’s combined architectural and infrastructure scope against prior work.
2.1 The Rise of Industrial-Scale AI Infrastructure
- Datacenter electricity demand is a first-order architectural constraint amid rapidly growing AI compute infrastructure.Global datacenter electricity consumption reached approximately 415 TWh in 2024 and could reach about 945 TWh by 2030; U.S. datacenters accounted for approximately 4.4% of electricity use in 2023, with a projected 6.7%–12% share by 2028.Source: §2.1 · p. 4
2.2 Why Specialization Is Necessary
- 248%/yr model-parameter growth exceeds dense FP16/BF16 throughput growth (67%/yr), HBM bandwidth growth (47%/yr), HBM capacity growth (42%/yr), and NVLink bandwidth growth (30%/yr).The comparison uses selected observations and total model parameters, rather than active parameters; specialized arithmetic, memory reuse, and interconnects address different system bottlenecks.Source: §2.2 · p. 4, Figure 2 · p. 5
2.3 A Decade of Datacenter AI Accelerators
- Accelerator development spans programmable GPUs and custom provider accelerators tailored to changing training, inference, and serving objectives.The timeline includes provider disclosures from Google in 2016, AWS in 2018, and Meta and Microsoft in 2023; workload targets also shift across generations.Source: §2.3 · pp. 5–6, Figure 3 · p. 6
2.4 Related Work
- The survey combines an industrial accelerator taxonomy with scale-up collective communication, generational architecture, facility constraints, and workload-driven design challenges.Prior surveys cover overlapping accelerator, infrastructure, and communication topics, while this survey connects node-, rack-, and pod-scale fabrics with mapped collective operations and links power and cooling to accelerator, rack, and facility design.Source: §2.4 · p. 6
3 Architectural Categories of AI Accelerators
The survey organizes industrial AI accelerators by their dominant execution and data-movement models, then compares compute and memory choices that balance programmability, specialization, and data locality.
3 Architectural Categories of AI Accelerators
- Four categories distinguish GPU, heterogeneous NPU, spatial dataflow, and compute-in-memory architectures by their dominant execution and data-movement models.GPUs combine SIMT and matrix units; NPUs integrate heterogeneous engines around scratchpads; spatial designs map computation across distributed resources; compute-in-memory places arithmetic within or near memory.Source: §3 · p. 7, Figure 4 · p. 7
- Spatial dataflow architectures expose operator placement, local-memory allocation, and inter-operator movement to the compiler, reducing coordination overhead and memory traffic while improving locality.The three substyles are PE arrays, reconfigurable architectures, and functional-slice streaming processors; PE arrays use compiler-managed transfers across programmable processing elements.Source: §3 · pp. 9–10
3.1 Compute Engine
- Compute engines balance programmability against specialization, with workload computational intensity and training or inference demands shaping the required compute-throughput and memory-bandwidth mix.NPUs combine matrix engines for GEMM with vector and scalar units for other tasks; systolic arrays reuse data between neighboring processing elements.Source: §3 · p. 9, §3.1 · p. 10
3.2 Memory Hierarchy
- On-chip memory ranges from hardware-managed caches to software-managed scratchpads, trading programming simplicity for more predictable placement, movement, and performance.GPUs combine caching with explicitly managed shared memory, while compiler-managed locality can improve predictability; compute-in-memory instead performs arithmetic in or near memory to reduce data movement, especially for bandwidth-bound workloads.Source: §3 · p. 10, §3.2 · pp. 11–12
4 Scale-up Interconnect
Scale-up systems use different topologies to balance bandwidth, latency, scalability, and implementation cost across node, rack, and pod tiers. Collective schedules must then fit both the operation and the fabric.
4 Scale-up Interconnect
- Scale-up fabrics connect accelerators within servers, racks, or pods, while scale-out networks connect those domains across clusters; intra-cluster communication efficiency can strongly influence workload performance.The surveyed tiers range from node groups of typically 4–8 accelerators to rack groups of 64–256 and pod fabrics scaling to thousands.Source: §4 · p. 13
4.1 Node Scale
- At node scale, PCIe switching offers flexibility and easier integration but lower bandwidth than purpose-built fabrics; direct all-to-all links provide single-hop bandwidth at growing wiring and packaging complexity.The figure contrasts switched PCIe attachment, an eight-accelerator physical complete graph, and a dedicated switched fabric with logical any-to-any reachability.Source: §4.1 · p. 13, Figure 6 · p. 13
4.2 Rack Scale
- At rack scale, switched all-to-all fabrics extend connectivity to larger domains, while torus topologies bound node degree and trade uniform all-to-all reachability for scalability and greater network diameter.Dragonfly groups use dense local links and fewer global links to reduce network diameter without the quadratic wiring cost of a full mesh.Source: §4.2 · p. 14, Figure 7 · p. 14
4.3 Pod Scale
- Pod-scale designs use deeper hierarchies, including Boardfly, UB-Mesh, and Ethernet-switched connectivity, to combine local and global links across larger domains.Maia’s two-tier design uses a switchless fully connected quad locally and Ethernet switches to extend the scale-up domain across racks.Source: §4.3 · pp. 14–15, Figure 8 · p. 15
4.4 Collective Communications
- Collective operations specify endpoint data transformations, but the best schedule depends on operation, message size, rank count, topology, link asymmetry, and contention.Rings suit large reductions on uniform links but have rank-linear startup stages; trees reduce dependent stages to logarithmic depth, while fabric-aware mappings can avoid congestion.Source: §4.4 · pp. 16–17
5 Architecture and System Evolution across Generations
Across accelerator generations, rising arithmetic throughput is accompanied by changes in precision, data delivery, execution coordination, memory, interconnects, power, and cooling. These changes reflect the need to support growing AI workloads across both devices and larger systems.
5.1 Lower Precision and Structured Sparsity
- AI-optimized formats BF16 and TF32 retain FP32-comparable 8-bit exponents while reducing mantissa width; sparsity-aware matrix engines skip zero multiplications, with up to ∼2× throughput when the 2:4 constraint is met.Block-scaled MXFP and NVFP share scale factors across groups of values.Source: §5.1 · pp. 18–19
5.2 Data Path and Control Flow
- Tensor Core generations expand data-loading flexibility and execution coordination, from warp-level mechanisms and asynchronous copies to warp-group, SM-cluster, and multi-SM operations.The case study traces Volta through Blackwell, including Ampere’s asynchronous copy pipeline, Hopper’s TMA and direct SM-to-SM movement, and Blackwell’s TMEM and SM-pair execution.Source: §5.2 · pp. 20–22
5.3 Memory Hierarchy and Scale-up Interconnect
- HBM capacity and bandwidth have grown substantially, while larger shared caches increase on-chip reuse; scale-up fabrics and topologies also evolve to connect more accelerators.Across listed GPU generations, HBM capacity rises approximately 13.5–18× and bandwidth 23–27×; shared-cache capacity rises approximately 32× for NVIDIA and 64× for AMD GPUs.Source: §5.3 · pp. 22–23
- Scale-up topology choices trade incremental, low-radix connectivity against switched fabrics that reduce path length and placement sensitivity.Trainium 3 uses a switched all-to-all fabric, while TPU 8i’s Boardfly reports seven-hop maximum paths versus 16 hops for the compared 3D torus.Source: §5.3 · pp. 23–24
5.4 Power Consumption and Cooling
- Datacenter electricity demand and deployment scale raise infrastructure requirements beyond individual racks, including power procurement and grid interconnection.The IEA estimates 415 TWh of datacenter electricity use in 2024 and projects 945 TWh by 2030; large deployments can require campus-scale provisioning.Source: §5.4 · pp. 24–25
- Accelerator power growth has driven changes in cooling, from air-cooled cards to planar modules and rack-scale direct liquid cooling.The cited examples span about 300 W for earlier devices, up to 1100 W for B300 SXM, and up to 1400 W per GPU for GB200 rack-scale DLC.Source: §5.4 · pp. 24–25
6 Challenges of Future AI Datacenter Design
Future AI datacenter design must account for changing workloads, data movement, network behavior, power and cooling constraints, and model evolution. The paper examines how specialization can improve efficiency while limiting flexibility as workloads and infrastructure demands change.
6.1 Workload Specialization and Fleet Flexibility
- Workload mix can shift while installed capacity remains fixed, so specialized resource pools may improve efficiency yet leave one pool underused as another queues.Training, prefill, and decode have different resource demands, and inference workloads vary with model type, context, and serving techniques.Source: §6.1 · p. 26
6.2 Data Placement and Movement
- Keeping data near compute can reduce transfers, but limited nearby memory means larger models, long contexts, or higher concurrency may require off-chip movement or network exchange.SRAM capacity can be exhausted; offloading adds transfers, while distributing computation requires communication of activations or partial results.Source: §6.2 · p. 26
6.3 Communication Across Scale-Up and Scale-Out Networks
- Communication limits depend on workload mapping: large training exchanges may be bandwidth-bound, while repeated short decoding exchanges remain sensitive to startup, path, and synchronization delays.Topology changes can reduce path lengths, but collective performance also depends on schedule and placement; scale-up and scale-out links serve different system domains.Source: §6.3 · p. 27
6.4 Power Delivery and Cooling from Rack to Facility
- Power delivery and cooling can limit usable compute even when total power is sufficient, and synchronized workload peaks can burden infrastructure differently from average power.Rack power delivery, coolant temperature, and the timing and magnitude of power swings affect how much activity infrastructure can sustain.Source: §6.4 · pp. 27–28
6.5 Architectural Specialization and Model Evolution
- Reconfigurable architectures support algorithmic change only within their physical resources and supported operations, while hard-wired models can be difficult to repurpose when demand shifts.Taalas HC1 retains configurable context lengths and adapters, but replacing its hard-wired base model requires new silicon.Source: §6.5 · p. 28
7 Conclusion
The survey compares industrial AI datacenter architectures across accelerator organization, interconnects, and architectural evolution. Its comparisons show that specialization involves trade-offs across compute, memory, communication, power, thermal limits, and system flexibility.
7 Conclusion
- Across surveyed systems, specialized arithmetic coexists with general-purpose execution resources and varied memory-management strategies.The survey examines accelerator organization, scale-up interconnects, and architectural evolution.Source: §7 · p. 28
- Interconnect analysis relates collective communication to node-, rack-, and pod-scale fabrics, while generational studies link throughput gains with changes in operand delivery, memory, communication, and cooling.Source: §7 · p. 28
- Specialization cannot be assessed from the compute engine alone; memory traffic, data placement, communication overhead, power, and thermal constraints also shape usable system capacity.The comparisons also connect architectural choices to hardware reuse, system flexibility, and longevity as workloads evolve.Source: §7 · p. 28