PaperlayerView exact revision on arXiv

Source-linked AI summary

Sharing a Fabric with Collective Communication: Two Storage Penalties in Deep Learning Training

Chen Wang, Wenzhao Wu, Hyojin Kim, Jae-Sung Yeom

arXiv:2609.06506v2cs.DCcs.PF

TL;DR

Shared-fabric storage in distributed deep-learning training produces both heavy-tailed DataLoader stalls and, when storage shares a traffic class with collectives, all-reduce contention. The paper evaluates DYAD, a staging system for node-local storage, and finds that DYAD’s DataLoader stall tail disappears by the second epoch once its cache is warm.

  • Problem

    Storage reads over the shared fabric can create heavy-tailed DataLoader latency, while storage and collective traffic sharing a traffic class can also stall all-reduce.

    Source: §II · pp. 2–3, A. Mitigations · p. 3

  • Method

    DYAD stages training samples across node-local servers, with consumers locating owning nodes through a distributed key-value store and retrieving data via RDMA.

    Source: §III · p. 3

  • Results

    By the second epoch, DYAD has 0% of iterations exceeding 1 s of DataLoader wait, down from 2.93% in the first epoch.

    Source: B. Multi-Epoch Behavior · p. 5

  • Takeaways & Limitations

    Once the cache is warm, samples are served from node-local NVMe or retrieved from their owner by RDMA rather than fetched from Lustre.

    Source: B. Multi-Epoch Behavior · p. 5

    DYAD is not immune to fabric contention on first touch: its owner rank still fetches byte ranges from Lustre over the kfabric path used by RCCL.

    Source: A. Per-Iteration Interference · p. 5

Abstract

from arXiv · show

Distributed DL training on HPC systems often shares one network fabric between NCCL/RCCL collective communication and parallel-filesystem I/O. Using a real GNN training workload on a Slingshot-11 system, we show that this sharing imposes two distinct costs. The primary cost is heavy-tailed DataLoader stalls: the typical DataLoader wait is just 15 ms at steady state, yet spikes to multiple seconds in 28% of Lustre iterations and 12% of VAST iterations. The secondary cost is traffic-class contention on collective communication: Lustre I/O stalls the all-reduce by up to 145$\times$ in an isolated benchmark. The two costs arise from different mechanisms. I/O stall latency affects any storage path that traverses the shared fabric, whereas all-reduce network contention occurs only when storage and collective communication share the same traffic class. Their common root cause is that storage I/O traverses the shared fabric. This work shows that node-local NVMe staging via DYAD (Our code is publicly available at https://github.com/flux-framework/dyad) eliminates both effects by keeping storage I/O off that path. Across a full training epoch, DYAD achieves a 7.4 times speedup over direct Lustre reads and a 1.06 times speedup over VAST. By the second epoch, once the local cache is fully warmed, DataLoader stalls are eliminated entirely, allowing DYAD to reach a 1.31 times speedup over VAST.

I. INTRODUCTION

The section motivates studying deep-learning training across heterogeneous storage tiers and shows a large epoch-time gap alongside distinct DataLoader and all-reduce spikes.

I. INTRODUCTION

  • Modern HPC centers pair parallel file systems with all-flash storage to serve different I/O patterns, making data-access performance harder to predict and optimize.The study investigates deep-learning training in this heterogeneous storage environment.Source: §I · p. 1
  • MILAN predicts protein–ligand binding affinity with equivariant graph attention networks, and this study trains it on SAIR using four nodes and 16 AMD MI300A GPUs.SAIR contains about 5.2 million samples, totaling approximately 687 GB including the graph-edge cache.Source: §I · p. 1
  • A steady-state iteration takes about 191 ms at the median, with 15 ms of exposed DataLoader wait despite workers fetching about 18 MB per batch.A single worker’s network-induced fetch delay can grow from milliseconds to seconds and stall the entire batch.Source: §I · p. 1
  • DataLoader stalls are dense and sustained on Lustre, while all-reduce communication spikes are less frequent but still severe.The two spike patterns arise from different mechanisms, so identifying which drives wall time matters for choosing a fix.Source: §I · pp. 1–2
  • 11,843.9 s on Lustre versus 1,693.9 s on VAST makes the identical workload a 7× full-epoch gap.Figure 1 shows per-iteration DataLoader wait and backward-pass plus all-reduce time across 5,120 iterations.Source: §I · p. 1, Figure 1 · p. 2

II. TWO COSTS WITH SHARED-FABRIC STORAGE

Shared-fabric storage creates a heavy-tailed DataLoader cost and a separate traffic-class contention cost for all-reduce; measurements show the DataLoader tail dominates epoch-time excess.

II. TWO COSTS WITH SHARED-FABRIC STORAGE

  • Lustre and RCCL use the kfabric face and share TC1+, whereas VAST uses Ethernet and a separate TC0.The fabric allocates injection credits by traffic class, and traffic classes do not borrow credits from one another.Source: §II · p. 2
  • The isolated all-reduce benchmark compares with-I/O and without-I/O latency over four storage tiers using 16 GPUs across four nodes.It runs 200 warmup and 100 measured iterations before repeating the sequence with I/O enabled.Source: §II · p. 2
  • 145× is Lustre’s worst-case all-reduce interference factor, compared with 24× for VAST; shm and XFS show no detectable interference.Lustre and RCCL share a traffic class, while VAST’s separate traffic class substantially reduces interference.Source: Table II · p. 2, §II · pp. 2–3
  • 28.2% of Lustre iterations and 11.5% of VAST iterations exceed 1 s of DataLoader wait, despite median waits of 3 ms and 15 ms, respectively.The tail persists with VAST even though its storage traffic occupies a separate traffic class.Source: Table III · p. 3, §II · p. 3
  • DataLoader stalls account for about 93% of Lustre’s total excess iteration time, while all-reduce spikes account for about 7%.For VAST, DataLoader wait excess totals 1,608 s versus 119 s for all-reduce excess.Source: §II · p. 3

A. Mitigations

Traffic-class isolation reduces all-reduce contention but leaves DataLoader stalls; node-local NVMe removes storage reads from the network fabric and addresses both costs.

A. Mitigations

  • Traffic-class isolation substantially reduces all-reduce spikes but does not fix heavy-tailed DataLoader latency when storage still traverses the fabric.VAST’s separate traffic class avoids competing with RCCL, but its storage path remains on the shared network fabric.Source: A. Mitigations · p. 3
  • Node-local NVMe routes storage reads through PCIe rather than the network fabric, avoiding both traffic-class contention and fabric-credit exhaustion.Because one local NVMe may not hold the entire dataset, a transparent staging layer must distribute the working set across node-local devices.Source: A. Mitigations · p. 3
  • The workload transfers about 18 MB per 64-sample batch, and median DataLoader wait is only 3–15 ms, so sustained bandwidth is not the limiting factor.The proposed approach stages the working set on node-local NVMe to remove its reads from the shared network path.Source: A. Mitigations · p. 3

III. DYAD

DYAD adapts producer-consumer streaming for training by fetching samples lazily and at byte-range granularity, hiding transfers within GPU computation. This addresses the mismatch between file-level staging and small, one-time-per-epoch sample reads.

III. DYAD

  • DYAD uses a producer-consumer file-streaming system: node services and a client library locate the shard-owning node, which serves sample byte ranges.In this workload, dataset shards are statically partitioned across nodes, with each rank serving its assigned data.Source: §III · p. 3
  • File-level staging is ill-suited to samples accessed once per epoch; DYAD instead hides transfers in the GPU compute phase before the next forward pass.The workload’s compute window is about 144–214 ms, during which the DataLoader prefetches the next batch.Source: §III · p. 3
  • Lazy staging avoids a ∼20-minute pre-run transfer, while fine-grained fetches transfer only the ∼18 MB needed for a batch within the compute window.Eager pre-staging could also transfer data that a run never accesses.Source: §III · p. 3, Figure 3 · p. 4
  • A byte-range API fetches arbitrary spans into a lazy write-through cache of 64 KB blocks on node-local NVMe.On a cache miss, DYAD fetches a block-aligned span from Lustre; a bitmap tracks resident blocks, with locking limited to bitmap updates.Source: §III · pp. 3–4
  • One-time dataset flattening precomputes each sample’s offset and length, reducing origin-side access to a single pread() without per-sample HDF5 metadata resolution.The preprocessing takes minutes and is required once per dataset.Source: §III · p. 4

IV. EVALUATION

Across a full single training epoch, DYAD staged from Lustre outperforms direct reads from both Lustre and VAST. Its first-epoch advantage reflects reduced per-sample I/O overhead and Lustre contention, despite no cache reuse.

IV. EVALUATION

  • In epoch one, every sample is accessed once, so the NVMe cache provides no reuse benefit and transfers approximately the same data volume as direct Lustre reads.The 64 KB block granularity may cause marginal overfetch.Source: §IV · p. 4
  • 1.06× and 7.4×: DYAD is faster than direct VAST and Lustre reads, respectively, over a full single epoch.The evaluation processes 5,120 iterations and ∼5.24M samples on 4 nodes and 16 GPUs; the configurations process the same global sample sequence.Source: Table IV · p. 4, §IV · p. 4
  • Flattened sample access and single-file ownership reduce per-sample I/O overhead and Lustre contention, helping batch prefetch finish within the GPU compute window.Only 2.93% of DYAD iterations exceed 1 s in DataLoader wait, versus 28.24% for Lustre and 11.48% for VAST; DYAD’s worst-case all-reduce time is 10.2 s, versus 53.8 s for Lustre and 3.3 s for VAST.Source: §IV · p. 4

A. Per-Iteration Interference

Across the full epoch, Lustre has more frequent and longer iteration stalls than VAST or DYAD. Tail distributions show that DataLoader wait and backward-pass-plus-all-reduce have distinct patterns.

A. Per-Iteration Interference

  • 28.0% of Lustre iterations exceed 1 s, versus 9.9% for VAST and 4.7% for DYAD, across all 5,120 iterations.Lustre’s worst iteration takes 54.9 s, compared with 7.2 s for VAST and 10.3 s for DYAD.Source: A. Per-Iteration Interference · p. 4, Figure 4 · p. 5
  • The tail plots show the fraction of iterations exceeding each value, separately for DataLoader wait and backward pass plus all-reduce.Both distributions use log-log axes.Source: Figure 5 · p. 5
  • DYAD has no persistent 100 ms–10 s DataLoader-wait shoulder like VAST, while Lustre’s DataLoader tail extends an order of magnitude further.At the 1 s line, backward-pass-plus-all-reduce exceeds that threshold in 1.54% of Lustre iterations, 0.61% of VAST iterations, and 0.37% of DYAD iterations.Source: A. Per-Iteration Interference · p. 5

B. Multi-Epoch Behavior

DYAD’s local cache is fully populated by the second epoch, eliminating Lustre-origin fetches and the DataLoader stall tail. Epoch time improves further relative to direct storage reads once the cache is warm.

B. Multi-Epoch Behavior

  • By epoch two, all needed samples reside on owner-node NVMe; DataLoader waits over 1 s fall from 2.93% in epoch one to 0% in epochs two and three.Mean DataLoader wait is 90.4 ms in epoch one, then 13.8 ms and 14.1 ms; non-local samples are retrieved from their owner via RDMA.Source: B. Multi-Epoch Behavior · p. 5
  • 1.31× and 1.32×: DYAD epochs two and three are faster than direct VAST reads, and 9.2× and 9.3× faster than direct Lustre reads.Epoch times are 1,293.3 s and 1,280.1 s, respectively.Source: B. Multi-Epoch Behavior · p. 5

A. Related Work

Prior storage systems provide shared or cached storage, but the described designs do not address globally shuffled, sample-per-request access together with the interaction between DataLoader stalls and collective-communication interference. DYAD instead flattens HDF5 data into a byte-offset index, avoiding repeated metadata resolution on first touch.

A. Related Work

  • Existing burst-buffer and node-local storage systems commonly operate at whole-file or POSIX-call granularity, unlike globally shuffled, sample-per-request training access.The related-work discussion also notes that these systems do not study the interdependence or relative magnitude of DataLoader stalls and collective-communication interference.Source: A. Related Work · pp. 5–6
  • Several DL-training caching systems optimize throughput or cache efficiency, but cache efficiency alone does not remove external interference unless storage traffic leaves the shared fabric.The cited approaches include cluster-wide caching, RDMA sample serving, coordinated loading, distributed caching, and local prefetching.Source: A. Related Work · p. 6
  • HDF5 Cache VOL caches HDF5 objects, whereas DYAD flattens the dataset into a byte-offset index to avoid repeated group-hierarchy and object-header resolution on first touch.Under global epoch shuffling, MILAN touches millions of distinct groups once per epoch; DYAD resolves metadata offline.Source: A. Related Work · p. 6

B. Conclusion

The conclusion identifies shared-fabric storage I/O as the source of two distinct training costs: heavy-tailed DataLoader stalls and traffic-class contention on all-reduce. DYAD stages data on node-local NVMe; it outperforms both direct Lustre and VAST reads, with a larger VAST speedup after the cache warms.

B. Conclusion

  • 7.4× and 1.06×: DYAD is faster than direct Lustre reads and VAST, respectively, in the first epoch.After the cache warms, DYAD reaches 1.3× speedup over VAST in the second and third epochs.Source: B. Conclusion · p. 6
  • Heavy-tailed DataLoader stalls account for ∼93% of Lustre’s epoch-time excess, while all-reduce traffic-class contention accounts for ∼7%.VAST also has 11.5% of iterations stall beyond 1 s despite traffic-class isolation.Source: B. Conclusion · p. 6
  • Lustre I/O stalls all-reduce by up to 145× in an isolated benchmark, and traffic-class isolation substantially reduces this secondary cost.Fixing only that effect leaves the primary DataLoader-stall cost high.Source: B. Conclusion · p. 6
  • DYAD uses a lazy block-granularity byte-range cache over flattened, byte-offset-indexed samples, staging only from Lustre to node-local NVMe.Once warm, the cache stops first-touch fetches; node-local NVMe access via PCIe is invisible to the fabric.Source: B. Conclusion · p. 6
Loading 2609.06506v2…