PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing the Scalability and Performance of Large-Scale AI Training Under Multi-Tenancy

Jacopo Raffi, Thomas Pasquali, Lorenzo Piarulli, Filippo Spiga, Marco Faltelli, Andreas Herten, Domenico Siracusa, Daniele De Sensi, Flavio Vella

arXiv:2609.00817v2cs.DC

TL;DR

Large-scale distributed AI training remains difficult to characterize because communication, topology, parallelization, and multi-tenant interference interact across systems. The paper uses DLNetBench to evaluate these factors across scales and architectures. It finds that communication becomes highly relevant beyond node and rack boundaries, while strategy and topology-aware placement shape efficiency and slowdown under contention.

  • Problem

    Existing studies provide limited evidence about distributed AI training under combined large-scale, diverse parallelization, complex topology, and multi-tenant conditions.

  • Method

    The paper uses DLNetBench to reproduce strategy-specific collectives and synchronization while controlling computation, placement, scale, and network interference across systems.

  • Results

    Communication accounts for 30% to over 70% of runtime at small supercomputer scales, with FSDP sustaining high efficiency while pure DP degrades sharply as scale increases.

  • Takeaways & Limitations

    Robust large-scale training requires jointly considering parallelization strategy, communication pattern, interconnect characteristics, and topology-aware placement.

  • Takeaways & Limitations

    The evaluation uses a limited set of fixed-parameter models, constraining analysis of compute-to-communication balance and achievable overlap.

Abstract

from arXiv · show

Characterising AI workload performance on modern HPC systems requires understanding both their scalability in isolation and their behaviour under concurrent execution. However, the interplay among parallelisation strategies, network congestion, compute capability, and interconnect technologies remains poorly understood. This work investigates the performance and scalability of AI models up to 2400 GPUs. We quantify the communication overheads and their impact across different interconnects by evaluating scale-up, scale-out, and rack-scale configurations under multiple allocation schemes. Finally, we study how multiple concurrent training jobs interfere with each other by designing a realistic noise model. We design a benchmark suite of AI models to evaluate the performance of five distinct parallelisation strategies across different supercomputing clusters, including Alps, Leonardo, LUMI, JUPITER, NVL72 GB300, and DGX A100. Our work provides a systematic characterization of the scalability and execution efficiency of distributed AI training, while offering key insights into performance behavior under realistic multi-tenant scenarios.

I. INTRODUCTION

Large-scale distributed AI training is difficult to characterize because communication, topology, compute, parallelization, placement, and concurrency interact. The paper addresses this gap with DLNetBench, a controlled framework for systematic cross-system analysis.

  • Communication and synchronization increasingly limit scalability as AI training spans hundreds to thousands of GPUs.
  • Prior studies usually isolate individual factors, leaving realistic multi-tenant behavior across diverse systems and parallelization strategies insufficiently understood.
  • DLNetBench reproduces strategy-specific collective operations and synchronization while approximating computation–communication balance with hardware-aware timing models.
  • The methodology jointly controls communication structure, computation time, and network interference to compare systems, placements, and configurations.
  • Five parallelization strategies cover replicated, sharded, pipeline, tensor, and expert-parallel training designs.

III. SYSTEMS DESCRIPTION

The experiments span node-, rack-, and supercomputer-scale platforms, with system characteristics and associated benchmark strategies documented for cross-system evaluation.

  • The study evaluates node-level DGX A100 and LUMI-G, rack-scale NVIDIA GB300 NVL72, and supercomputers including Alps, JUPITER, Leonardo, and LUMI.
  • Table II associates the DLNetBench parallelization strategies with the models used in the results analysis.

A. NVIDIA DGX A100

The systems description contrasts node-, rack-, and supercomputer-scale interconnects, emphasizing how topology and bandwidth shape the experimental platforms.

  • A. NVIDIA DGX A100: DGX A100 provides eight GPUs in a switched all-to-all NVLink/NVSwitch fabric with uniform node-local communication characteristics.The platform can sustain up to 4.8 TB/s of aggregate bidirectional GPU-to-GPU traffic when all GPUs communicate concurrently at peak rate.
  • B. NVIDIA GB300 NVL72: GB300 NVL72 integrates 72 GPUs in one liquid-cooled NVLink domain with full-mesh connectivity and 130 TB/s aggregate NVLink bandwidth.
  • C. Alps: Alps combines four-GPU GH200 nodes using all-to-all NVLink with inter-node Dragonfly connectivity over Slingshot-11.

D. Leonardo

The paper describes Leonardo, LUMI, and JUPITER as large GPU systems with distinct node architectures and Dragonfly-based or InfiniBand interconnects, while defining experimental allocation and feasibility constraints.

  • D. Leonardo: Leonardo’s Booster partition contains 3,456 nodes and 13,824 A100 GPUs connected through NVLink and InfiniBand HDR200.Its Dragonfly+ network uses groups of 180 nodes organized as two-level fat trees.
  • E. LUMI: LUMI-G models each MI250X module’s two GCDs as separate devices, yielding an eight-GPU software-visible node.Each GCD connects to a CPU NUMA domain through a 288 Gb/s Infinity Fabric link.
  • F. JUPITER: JUPITER nodes contain four GH200 superchips with direct GPU and CPU connections, while 25 Dragonfly+ groups form the overall topology.
  • IV. EXPERIMENTAL SETUP: Experimental sections define baseline and concurrent configurations, job placement, and metrics such as slowdown.
  • GPU counts per strategy: Hybrid strategies require GPU counts in multiples of their replica sizes, whereas pure data parallelism can harm convergence through global batch-size growth.

2) Allocation patterns:

The study samples realistic allocation patterns by varying job sizes, strategy diversity, network locality, and placement across a hierarchical topology. Measurements cover valid strategy–GPU-count–placement configurations in isolation and through normal SLURM allocation.

  • Allocation patterns:: Tier-sampled patterns model skewed workloads with small jobs dominant, medium jobs less frequent, and large jobs rare within a GPU budget.Small jobs use 8 GPUs with 75% probability; medium jobs use 16, 32, 64, or 512 GPUs with 20%; large jobs use 1024 GPUs with 5%.
  • Allocation patterns:: Strategy assignments are sampled by mixture entropy to cover low, medium, and high diversity without enumerating the exponentially large assignment space.Entropy is zero when all slots use one strategy and maximal when GPU usage is balanced across strategies.
  • Allocation patterns:: Placement classes distribute GPUs across a hierarchy of nodes, L1 switches, and groups, distinguishing increasingly broad network locality.The hierarchy groups 4 or 8 GPUs per node, roughly 10–16 nodes per L1 switch, and approximately 120–240 nodes per group.
  • Allocation patterns:: The methodology samples locality bins from predominantly INTRA-L1 to predominantly INTER-GROUP placements for each strategy assignment.A mean locality score partitions placements into three bins over the range [1, 3].
  • Allocation patterns:: Each configuration is measured after warmup with strategy- and system-dependent iterations, recording total runtime and communication synchronization costs.Blocking collectives are timed directly, while non-blocking collectives record time at synchronization barriers.
  • Allocation patterns:: Valid strategy, GPU-count, and placement triples are measured both in isolation and through ordinary SLURM allocation before concurrent workloads are introduced.The isolated experiments span at most three Dragonfly groups.

1) Baselines:

The baseline methodology measures communication relevance, throughput, and slowdown across systems and allocations. It establishes isolated performance before analyzing concurrent execution and reports communication-sensitive behavior across hardware and strategies.

  • Baselines:: Communication relevance is measured as the fraction of total runtime spent in communication, serving as a proxy for network sensitivity.The metric includes blocking collective time and synchronization time that overlapping computation could not hide.
  • Baselines:: Throughput is reported in samples per second, with global batch size determined by local batch size and the number of model replicas.Each rank reports throughput from its iteration runtime.
  • Baselines:: Slowdown σ compares baseline throughput with concurrent throughput, while (σ −1) × 100 represents percentage throughput loss.σ = 1 indicates no degradation, and baseline throughput is the geometric mean of iteration throughputs.
  • Baselines:: The experiments reserve up to three Dragonfly(+) groups on large systems, including 2400 GPUs on JUPITER, 1860 on Leonardo, 1320 on Alps, and 2720 GPU modules on LUMI.Some faulty or underperforming nodes were excluded according to administrator guidance.
  • Baselines:: Communication overhead is limited on DGX A100 but increases substantially on NVL72 at larger scales, reaching approximately 41–58% in some configurations.DGX A100 remains below roughly 10% communication time, whereas NVL72 shows greater variability.
  • Baselines:: FSDP remains below 10% communication, hybrid strategies generally occupy 30–50%, and DP often exceeds 70% at larger scales.FSDP overlaps IALLGATHER with computation, while DP relies on global IALLREDUCE operations whose communication fraction grows with scale.
  • Baselines:: Communication overhead is minimal within nodes but grows sharply across nodes, while systems with lower compute-to-network balance can overlap communication better.This observation directly frames the study’s first research question.

B. Scalability and Parallel Efficiency

Near-ideal scaling occurs when communication is negligible or well provisioned, but efficiency becomes strategy- and interconnect-dependent on supercomputers. Compute capability and interconnect bandwidth dominate throughput, while placement has little effect on scaling trends.

  • Scalability and Parallel Efficiency: Near-ideal scaling is achievable only when communication remains negligible or perfectly provisioned.The scalability analysis compares measured efficiency against ideal scaling.
  • Scalability and Parallel Efficiency: DGX A100 achieves ≥99% efficiency for all strategies, while NVL72 preserves similar behavior except for DP, which falls to 80–85% at 32–64 GPUs.The DP decline is associated with increasing participants in global IALLREDUCE communication.
  • Scalability and Parallel Efficiency: FSDP’s REDUCESCATTER operations remain confined to small 8-GPU groups, while hybrid strategies exploit intra-node tensor parallelism and favorable compute-to-communication ratios.DP+PP+EP adds eight independent 8-GPU all-to-all communicators per newly added replica because the model uses eight pipeline stages.
  • Scalability and Parallel Efficiency: Configurations spanning fewer groups achieve higher efficiency than scaled-out configurations spanning 17 or 18 groups.The comparison is reported for purple-rhombus versus purple-circle configurations and for DP+PP+TP runs.
  • Scalability and Parallel Efficiency: Absolute throughput follows raw compute capability, while JUPITER and Alps show broadly comparable throughput across scales despite differing interconnects.Across systems, placement classes produce curves that remain close to one another.
  • Scalability and Parallel Efficiency: Network placement has little to no impact on scaling trends across the evaluated systems.The reported interpretation is that bandwidth saturation masks additional latency from longer network distances.

C. Impact of Network Congestion and Multi-Tenancy

Concurrent execution exposes system- and placement-dependent slowdown patterns. Congestion is limited within tightly coupled fabrics but can become severe across broader network placements, with regular allocations reducing slowdown in some JUPITER configurations.

  • Impact of Network Congestion and Multi-Tenancy: Slowdown distributions classify each co-scheduled configuration by strategy, model, GPU scale, placement, and reservation status.The figures use violin plots and boxplots, with medians, means, interquartile ranges, and whiskers summarizing slowdown variation.
  • Scale-Up Systems:: DGX A100 slowdowns are generally below 10–15%, but extreme outliers reach 21× for some communication-heavy workloads.The outliers are attributed to transient interference or system noise rather than typical congestion behavior.
  • Scale-Up Systems:: NVL72 slowdowns remain below 1.5%, with rare exceptions reaching 4%, indicating negligible concurrent degradation.The NVSwitch-based fabric sustains concurrent workloads despite increased communication demand from higher compute performance.
  • JUPITER:: A regular 4-nodes-per-switch allocation reduces average slowdown for 224-GPU DPT on LLaMA3-70B from approximately 15% to 5%.The comparison concerns standard versus application-specific inter-group allocation.
  • Multi-Tenancy: Production SLURM runs usually show greater slowdown variance than reservations, but DP+PP is an exception with approximately 25% of reserved runs showing higher slowdowns.The paper attributes this exception to pipeline peer-to-peer messages becoming vulnerable to delays under high interconnect utilization.

3) Leonardo:

On Leonardo, network congestion usually has lower overall impact because A100 compute reduces the relative importance of communication, but large allocations and communication-heavy strategies remain vulnerable. DP also exhibits high variance linked to unstable ALLREDUCE behavior, while LUMI’s hybrid strategies show no significant slowdown under the described placement.

  • Leonardo congestion: Leonardo generally shows lower congestion impact than JUPITER because its A100 GPUs make communication performance less relevant.The lower-performing GPUs reduce the relative contribution of communication to total execution time.
  • Leonardo congestion: 106× slowdowns occur for some 8-GPU DP runs with inter-group allocation.The variance is associated with inter-group placement and congestion exposure.
  • Leonardo congestion: 65% average slowdowns affect 1K-GPU DP+PP+EP runs without reservation across more than 10 Dragonfly+ groups.Slowdowns appear when allocations exceed 3 groups, with Leonardo offering half JUPITER’s interconnect bandwidth and DPE spending nearly half its runtime communicating.
  • ALLREDUCE variability: On Alps, DP slowdown variance is consistently high because ALLREDUCE runtimes are variable, making absolute slowdown values difficult to interpret.The unstable communication baseline prevents reliable interpretation of reported absolute slowdowns.
  • Alps results: On Alps, FSDP can reach 60% slowdown in eight isolated runs despite communication comprising only 3–7% of runtime, whereas DP+PP and DP+PP+TP remain comparatively robust.The FSDP cases involve orders-of-magnitude increases in communication time; DP+PP and DP+PP+TP spend 40%–60% of runtime communicating without significant congestion effects.
  • LUMI comparison: On LUMI, hybrid strategies show no significant slowdown, while DP retains high variance because RCCL had not reached a stable regime.LUMI’s eight GPUs per node keep intra-replica communication on the scale-up network, limiting scale-out traffic for hybrid strategies.

VI. RELATED WORK

The paper situates its contribution in prior work on communication overhead, topology, congestion, and hardware utilization, then synthesizes results showing that distributed-training performance depends jointly on strategy, topology, placement, and contention. The study identifies bounded mitigation from topology-aware placement, while collective-library instability remains a distinct source of variance and the evaluation is limited by fixed models.

  • Prior work: Prior studies examine communication overheads, topology-aware optimization, congestion, and hardware utilization, but usually isolate only a subset of systems or workload factors.The paper identifies limited evidence for combined behavior across systems, scales, network conditions, and workloads.
  • Cross-system findings: Communication overhead is minimal within nodes or racks but reaches 30% to over 70% of runtime when workloads cross node boundaries.This transition varies by strategy: FSDP overlaps communication with computation, pure DP degrades with scale, and hybrid strategies depend on topology mapping.
  • Placement: Placement has little impact on aggregate throughput scaling in isolation, which is governed mainly by compute capability and interconnect bandwidth.This decoupling breaks down under contention, where topology-aware placement becomes relevant.
  • Multi-tenancy: Multi-tenancy effects are uneven: cross-group traffic can cause severe variable slowdowns, while unstable NCCL or RCCL convergence produces variance that placement cannot mitigate.Topology-aware allocation can reduce congestion-driven slowdowns several-fold, but reserved scheduling does not universally reduce variance for pipeline-parallel jobs.
  • Conclusion: The paper concludes that robust scaling requires jointly considering parallelization strategy, communication pattern, interconnect, and topology-aware placement.No single factor predicts performance both in isolation and under multi-tenancy.
  • Open aspects: The evaluation is constrained by a limited set of fixed-parameter models, motivating broader models, scales, DLNetBench settings, libraries, and controlled allocations.The authors also identify collective-library convergence as requiring further study to separate noise from genuine contention.

APPENDIX

The appendix documents DLNetBench’s algorithmic building blocks for FSDP and pipeline-based parallel strategies. These pseudocode fragments specify the ordering of computation, communication, synchronization, and stage-to-stage transfers.

  • Appendix contents: The appendix lists experimental setup parameters alongside algorithmic constants and operation sequences used by DLNetBench.The provided appendix includes the experimental-setup table and pseudocode labels for the parallelization strategies.
  • FSDP: FSDP begins with an all-gather, overlaps subsequent all-gathers with forward computation, and uses reduce-scatter during backward execution.Inter-replica all-reduce is added when multiple replicas are present.
  • DP+PP: DP+PP executes microbatch forward and backward computation across pipeline stages with sends and receives between neighboring stages.The final step performs all-reduce for data-parallel gradient synchronization.
  • DP+PP+TP: DP+PP+TP inserts tensor-parallel all-reduces within attention and MLP computation while retaining pipeline transfers between stages.The algorithm performs data-parallel gradient synchronization after the backward pipeline operations.
  • DP+PP+EP: DP+PP+EP uses expert-parallel all-to-all operations around attention and MLP computation, followed by pipeline communication and gradient synchronization.The pseudocode distinguishes non-expert and data-parallel gradient reductions.
Loading 2609.00817v2…