PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing and Mitigating Data Stalls in DNN Training

Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, Vijay Chidambaram

arXiv:2007.06775v3cs.DCcs.LGcs.OS

TL;DR

DNN training pipelines can be bottlenecked by data fetching and preprocessing, yet their impact has been relatively unexplored. The paper measures and predicts these data stalls and develops CoorDL to mitigate them, accelerating training across distributed and hyperparameter-search settings.

  • Problem

    The impact of fetching data from storage and preprocessing it in memory on DNN training time has been relatively unexplored.

  • Method

    The paper analyzes data stalls, uses DS-Analyzer for measurement and predictive what-if analysis, and develops CoorDL to coordinate data fetching and preprocessing.

  • Results

    Data stalls account for up to 65% of training time, while CoorDL accelerates training by up to 15× for two-server distributed training and 5.3× for audio hyperparameter search.

  • Takeaways & Limitations

    Data stalls can waste faster GPU performance, making slower GPUs without stalls potentially more economical and enabling more efficient accelerator assignment.

  • Takeaways & Limitations

    Reusing preprocessed data across epochs may reduce accuracy, while storing it can cause memory overflow or fetch stalls because preprocessed items are 5–7× larger.

Abstract

from arXiv · show

Training Deep Neural Networks (DNNs) is resource-intensive and time-consuming. While prior research has explored many different ways of reducing DNN training time, the impact of input data pipeline, i.e., fetching raw data items from storage and performing data pre-processing in memory, has been relatively unexplored. This paper makes the following contributions: (1) We present the first comprehensive analysis of how the input data pipeline affects the training time of widely-used computer vision and audio Deep Neural Networks (DNNs), that typically involve complex data preprocessing. We analyze nine different models across three tasks and four datasets while varying factors such as the amount of memory, number of CPU threads, storage device, GPU generation etc on servers that are a part of a large production cluster at Microsoft. We find that in many cases, DNN training time is dominated by data stall time: time spent waiting for data to be fetched and preprocessed. (2) We build a tool, DS-Analyzer to precisely measure data stalls using a differential technique, and perform predictive what-if analysis on data stalls. (3) Finally, based on the insights from our analysis, we design and implement three simple but effective techniques in a data-loading library, CoorDL, to mitigate data stalls. Our experiments on a range of DNN tasks, models, datasets, and hardware configs show that when PyTorch uses CoorDL instead of the state-of-the-art DALI data loading library, DNN training time is reduced significantly (by as much as 5x on a single server).

1 INTRODUCTION

DNN training depends on a data pipeline that fetches and preprocesses data before GPU computation, but its effect on training time has been relatively unexplored. The paper analyzes data stalls, develops DS-Analyzer, and introduces CoorDL to mitigate them.

  • Motivation: DNN training uses storage, CPU, and GPU resources, yet prior optimization work largely overlooked the input data pipeline.The pipeline fetches data from storage and preprocesses it in memory before GPU processing.
  • Motivation: Data stalls occur when fetching or preprocessing fails to keep GPUs continuously supplied with data.The paper distinguishes fetch stalls caused by I/O from prep stalls caused by CPU preprocessing.
  • Contributions: The study comprehensively analyzes data stalls across nine DNN models, three domains, four datasets, and varied hardware and pipeline factors.Factors include storage media, cache capacity, CPU threads, and GPU generation.
  • Contributions: DS-Analyzer uses differential comparisons between runs to identify data-stall bottlenecks and answer predictive what-if questions.Examples include estimating the effect of increasing DRAM capacity.
  • Contributions: CoorDL combines specialized caching, partitioned caching, and coordinated preprocessing to mitigate data stalls without changing cluster infrastructure.It is implemented as a user-space library on top of DALI and accelerates training by up to 5× on a single server over DALI.

2 BACKGROUND

DNN training repeatedly processes shuffled and randomly transformed data through an extract-transform-load pipeline before GPU computation. These requirements support training quality, while DALI provides a pipelined, GPU-accelerated data-loading baseline.

  • DNN Training: DNNs learn higher-level features from input data over epochs, with hyperparameter search preceding training to target accuracy.Hyperparameter search runs jobs with different learning settings and replaces poor performers.
  • DNN ETL Requirements: Each training epoch applies an extract-transform-load process before accelerator computation.The pipeline imposes data-ordering and preprocessing requirements for model convergence and accuracy.
  • DNN ETL Requirements: Every epoch must shuffle the dataset, process each item exactly once, and apply random transformations rather than reusing identical transformed items.The paper follows these requirements in all experiments because relaxing them can affect SGD convergence.
  • DALI: DALI is a drop-in data loader that accelerates preprocessing on GPUs and pipelines fetching and preprocessing with GPU computation.The authors empirically found DALI stronger than the default loaders in PyTorch, TensorFlow, and MxNet.

3 DATA STALLS IN DNN TRAINING

DNN iterations fetch and preprocess minibatches before GPU computation, with these stages pipelined across CPU resources and accelerator work. Data stalls arise when fetch or preparation cannot sustain GPU demand and thereby leave GPUs idle.

  • Training Pipeline: Each iteration fetches a minibatch, preprocesses it, computes predictions and loss on the GPU, and updates model weights.Image preprocessing can include decompression, cropping, resizing, and flipping.
  • Pipeline Concurrency: Data preparation and GPU computation are pipelined, so later minibatches are fetched and preprocessed while the GPU handles the current minibatch.Multiple CPU cores support the preparation stages.
  • Data Stalls: A fetch stall makes training I/O-bound, whereas a prep stall makes it CPU-bound; both leave the GPU idle.Fetch rate depends primarily on storage media, while preprocessing rate depends on operations and available CPU cores.
  • Data Stalls: Data stalls appear when GPU processing rate G exceeds min(F, P), where F is fetch rate and P is preprocessing rate.The condition identifies whether the pipeline cannot supply data as quickly as the GPU consumes it.
  • Data Stalls: Reported fetch and prep stalls are unmasked critical-path time despite being pipelined with computation.The paper treats both forms of stall as idle GPU time that should be minimized.

4 ANALYZING DATA STALLS

The analysis evaluates data stalls across DNN models, datasets, and server configurations, showing that storage access, caching, CPU preprocessing, and workload coordination can substantially limit training.

  • Experimental scope: The study analyzes nine DNN models across three tasks and four datasets while varying storage, cache size, CPU threads, and GPU generation.Experiments use two server SKUs with 24 CPU cores, 500GiB DRAM, and eight GPUs each.
  • Fetch stalls: 10–70% of epoch time is spent on blocking I/O when only 35% of the dataset is cached, despite pipelining and prefetching.Fetch stalls occur because the compute rate exceeds the fetch rate.
  • Caching: The OS Page Cache fetches 85% rather than the expected 65% of a 146GiB dataset per epoch because of thrashing.The resulting 20% excess fetch is identified as thrashing-induced overhead.
  • Distributed training: Distributed training lacks cache coordination, causing each server to fetch 45GiB per epoch and leaving ResNet50 stalled on I/O for 75% of epoch time.The example uses two servers with a combined 150GiB cache for ImageNet-1K.
  • Hyperparameter search: Hyperparameter search creates 7× read amplification and slows ResNet18 by 2× when eight single-GPU jobs share a 35% cache.Concurrent jobs access the same dataset independently, producing redundant I/O.
  • Preprocessing: Prep-stall requirements range from 3–4 CPU cores per GPU for ResNet50 to as many as 24 for lighter models such as ResNet18 or AlexNet.DALI can reduce prep stalls, but on V100 it still leaves 50% prep stalls with three CPU cores per GPU and GPU preprocessing.

5 DS-ANALYZER: PREDICTIVE ANALYSIS

DS-Analyzer measures data-pipeline rates and predicts how hardware or cache changes affect DNN training. Its cache-size analysis identifies when fetch stalls disappear and additional memory stops helping.

  • Pipeline-rate measurement: DS-Analyzer measures GPU ingestion, CPU preparation, cache-fetch, and storage-fetch rates to analyze DNN data stalls.These rates describe the pipeline’s compute, preprocessing, caching, and storage stages.
  • Pipeline-rate measurement: DS-Analyzer isolates ingestion rate by running training iterations with synthetic data pre-populated at the GPUs.This estimates the maximum training speed without data-loading delays.
  • Pipeline-rate measurement: DS-Analyzer measures preparation with cached data and disabled GPU computation, then measures storage throughput with cold-cache data loading.Cache-fetch rate is approximated separately using a memory-bandwidth microbenchmark.
  • What-if analysis: DS-Analyzer predicts effective fetch rate for a chosen cache fraction by modeling cached and uncached dataset access.Under an efficient cache, the model assumes cached items generate hits across epochs.
  • What-if analysis: 4% maximum prediction error separates DS-Analyzer’s predicted training speed from empirical results for AlexNet on ImageNet1K.The predicted speed is computed from the minimum of fetch, preparation, and GPU-ingestion rates.
  • What-if analysis: 50% cache capacity is sufficient to eliminate fetch stalls in the AlexNet example; larger caches provide no further benefit because training becomes CPU-bound.At lower cache sizes, the training configuration remains I/O-bound.

6 MITIGATING DATA STALLS

The paper mitigates data stalls with caching, coordinated remote caches, and coordinated preprocessing. These techniques target cache misses and redundant fetch or preparation work in single-server, distributed, and hyperparameter-search settings.

  • Mitigation techniques: CoorDL combines DNN-aware caching, coordinated remote MinIO caches, and coordinated preprocessing to reduce data-stall sources.The techniques target cache misses and redundant data movement or preparation.
  • The MinIO cache: MinIO keeps cached items without replacement, matching DNN training’s random-within-epoch and repetitive-across-epoch access pattern.This avoids the thrashing caused by replacement policies that evict items likely to be reused in later epochs.
  • The MinIO cache: 20% more misses occur with the OS Page Cache than with MinIO because page-cache replacement causes thrashing.MinIO incurs only capacity misses per epoch in the illustrated access pattern.
  • Partitioned MinIO caching: MinIO alone is inefficient for distributed training because changing random dataset shards create local-cache misses and storage I/O.Partitioned MinIO caching addresses this by coordinating cache contents across servers.
  • Coordinated prep: Coordinated preprocessing lets concurrent hyperparameter-search jobs reuse minibatches while preserving one full dataset sweep per epoch.Minibatches are staged briefly in shared memory, and one coordinated sweep eliminates redundant fetch and preprocessing.
  • Coordinated prep: Preprocessed data cannot simply be reused across epochs because random transformations support learning and preprocessed items are 5–7× larger than raw items.The resulting memory and storage costs make naïve cross-epoch reuse unsuitable for large datasets.
  • Implementation: CoorDL can serve as a drop-in replacement for the default PyTorch dataloader.

7 EVALUATION

CoorDL is evaluated against DALI across single-server, distributed, and hyperparameter-search workloads spanning nine models, three tasks, four large datasets, and multiple server configurations. It improves training speed by reducing cache misses, coordinating distributed caching, and eliminating redundant preprocessing.

  • Single-server training: Up to 1.5× speedup is achieved on ImageNet-22k, while ResNet50 on OpenImages reaches 2.1× over DALI-seq and 1.53× over DALI-shuffle.These single-server results use MinIO to reduce cache misses and thrashing.
  • Single-server training: 225 GB of I/O results from CoorDL’s 35% minimum cache-miss rate, versus 422 GB for DALI-seq and 340 GB for DALI-shuffle.The corresponding baselines incur 66% and 53% cache misses, respectively.
  • Multi-server distributed training: Up to 15× higher distributed-training throughput is obtained for AlexNet on OpenImages across two HDD-based servers and 16 GPUs.Aggregated memory fully caches the dataset, moving training from I/O-bound to GPU-bound.
  • Multi-server distributed training: 1.3× and 2.9× speedups are reported for distributed ShuffleNet and Audio-M5 workloads on SSD-based servers.The gains are lower than on HDDs because SSDs have higher random-read throughput.
  • Hyperparameter search: 5.6× faster audio-model training is achieved during eight-job hyperparameter search, alongside a disk-I/O reduction from 3.5TB to 550GB.Less-complex models gain 3× because coordinated preprocessing addresses their original CPU-bound behavior.
  • Hyperparameter search: Coordinating both fetch and preprocessing mitigates prep-dominated stalls that fetch-only coordination does not address.This comparison distinguishes CoorDL from approaches that coordinate only data fetching.
  • Training to accuracy: 4× less time to target accuracy reduces ResNet50 training from two days to 12 hours at 75.9% accuracy.The experiment uses 16 GPUs across two HDD-based servers and partitioned caching.
  • DGX-2 evaluation: 1.5×–2.5× acceleration over DALI is obtained on DGX-2 hyperparameter search by eliminating redundant preprocessing.The dataset fits in DGX-2 memory, so the remaining stalls arise from CPU-GPU imbalance rather than fetches.

8 DISCUSSION

The discussion identifies practical ways to reduce data stalls while emphasizing trade-offs involving storage, GPU memory, CPU capacity, and convergence. It also highlights hardware-cost implications and possible disaggregation of data preparation.

  • Decoded cache to reduce pre-processing overhead: Decoded caching could reduce preprocessing overhead because decoding is the most expensive preparation operation, but decoded data increases dataset size by 5–7×.CoorDL currently caches raw encoded items; enabling decoded caching across epochs remains a future direction.
  • Automatic prep offload to GPUs: GPU preprocessing requires a model- and batch-size-dependent split because it consumes scarce GPU memory and may interfere with learning computations.The paper describes current split selection as manual trial-and-error and proposes automation based on GPU and CPU utilization.
  • Minibatch as a service: Minibatch-as-a-service could disaggregate learning from data management by centrally preprocessing minibatches on idle cluster servers.This approach is especially suited to production clusters with high-bandwidth Ethernet and shared datasets or preprocessing pipelines.
  • Cost-performance tradeoff of upgrading hardware: Data stalls can make slower, less expensive GPUs more economical than faster GPUs whose capabilities are underutilized by stalls.The paper connects this assignment strategy to maximizing GPU utilization in multi-tenant clusters.
  • Trade-off between convergence rate and epoch time for other SGD variants: Relaxing random preprocessing or per-epoch shuffling may reduce epoch time but could prolong convergence or affect some models’ accuracy.The paper identifies this convergence-rate versus epoch-time trade-off as a future direction for other SGD variants.

9 RELATED WORK

The paper distinguishes its focus on comprehensive data-stall analysis and single-server optimization from prior work on remote caching, distributed fetch, model search, and specialized hardware. It positions CoorDL as complementary to these approaches by optimizing fetch and preparation with existing servers.

  • Optimizing remote storage via local caching: Quiver caches remote data locally and handles fetch stalls, whereas this work starts from locally available data and also optimizes preparation stalls.The paper presents its setting as a performance baseline beyond Quiver’s best case.
  • Partitioned caching: Cerebro targets distributed model search and does not improve single-server DNN training, while CoorDL targets the latter scenario.The paper describes its analysis and CoorDL as broader than Cerebro’s specific model-search setting.
  • Redundancy in DNN training: Prior redundancy-reduction work addresses model search or stores preprocessing results across epochs, whereas CoorDL aims to eliminate redundancy while preserving online preprocessing and accuracy.The cited comparison concerns settings where GPUs are not shared between jobs.
  • Hardware solutions to fetch stalls: Fast storage hardware may mask fetch stalls but may not help models bottlenecked by preparation stalls; this work instead uses commodity servers.The paper frames CoorDL as an alternative to relying on expensive specialized storage solutions.
  • Optimizing DNN training time: This paper adds data stalls as a training-time optimization target alongside specialized hardware, parallel training, memory, communication, and operator optimizations.The related-work discussion places data stalls within the broader spectrum of DNN training optimizations.
  • Domain specific caching: The paper follows domain-aware caching by first analyzing DNN access patterns and then devising a caching policy based on those observations.The comparison connects the work to informed prefetching and caching in databases and file systems.

10 CONCLUSION

The conclusion reports that data stalls can occupy a substantial share of DNN training time and motivates CoorDL as a coordinated caching and preprocessing library. It also reports substantial speedups across distributed training and hyperparameter search settings.

  • 10 CONCLUSION: Up to 65% of training time is accounted for by data stalls in the paper’s study of several DNNs.The conclusion presents this as the central finding of its detailed data-stall analysis.
  • 10 CONCLUSION: CoorDL uses insights from the study to coordinate data fetching and preprocessing for mitigating data stalls.The conclusion describes the underlying techniques as simple and intuitive for production adoption.
  • 10 CONCLUSION: Up to 15× acceleration is reported for distributed training across two servers, and 5.3× for hyperparameter search on the audio model.These are the conclusion’s reported peak speedups for the two named settings.
  • Supplementary material: The supplementary material adds analysis of preparation stalls and data-pipeline rates across CPU cores per GPU, models, datasets, and GPU counts.The additional experiments include increasing CPU cores per GPU beyond three.

A.1 Data pipeline rates

The rate analysis models the data pipeline through GPU ingestion, preparation, and fetching rates, showing that stalls arise when pipeline supply cannot match GPU demand. Changing GPU count can shift the bottleneck between fetching and preparation.

  • A.1 Data pipeline rates: P is the better of CPU- and GPU-based preparation rates, and pipeline speed is min(P, F).The analysis defines G as GPU rate, P_g and P_c as GPU- and CPU-preparation rates, and F as effective fetch rate.
  • A.1 Data pipeline rates: Data stalls exist when G > min(P, F), and the rate graphs show prominent stalls across several models and configurations.The fetch-rate interpretation assumes an efficient in-memory cache; OS Page Cache produces fetch rates 20% lower.
  • A.1 Data pipeline rates: Reducing GPU count lowers GPU ingestion rate while storage bandwidth remains constant, allowing fetch rate to catch up and shifting the bottleneck to preprocessing.This describes the direction of bottleneck movement as GPU parallelism decreases.
  • A.1 Data pipeline rates: Several models experience fetch or preparation stalls across GPU configurations and datasets, including computationally expensive ResNet50 and VGG.The cited takeaways attribute stalls in these models to costly preprocessing despite state-of-the-art data pipelines.
  • A.1 Data pipeline rates: GPU-based preprocessing can hurt ResNet50 and VGG by interfering with GPU computation.The rate-graph takeaways identify this as a model-specific consequence of using GPU preprocessing.

A.2 Training on servers with high CPU count

On high-CPU servers, training can remain limited by preprocessing, while DALI improves image decoding but GPU preprocessing can consume memory and interfere with compute-heavy models.

  • Training on servers with high CPU count: With 8 vCPUs per GPU, ResNet18 still has 37% prep stalls despite higher preprocessing capacity.CPU preprocessing scales linearly only up to the number of physical cores; increasing threads from 32 to 64 raises preprocessing speed by just 30%.
  • Training on servers with high CPU count: The pipeline analysis distinguishes fetch and preprocessing stalls by comparing GPU ingestion with fetch and preprocessing rates.Across the figure-based configurations, the relevant stall occurs when the black ingestion-rate bar exceeds the limiting fetch or preprocessing bar.
  • Comparing PyTorch DL with DALI: DALI accelerates image classification over native PyTorch partly through its optimized nvJPEG decoding library.The comparison uses fully cached ImageNet-1K data and evaluates CPU- and GPU-based DALI pipelines.
  • Comparing PyTorch DL with DALI: GPU-based preprocessing can hurt compute-heavy models such as ResNet50 because it competes with GPU computation.GPU preprocessing also requires 2–5GB of additional GPU memory, which may be unavailable for some models and GPUs.

B.1 Evaluation with ImageNet22k

CoorDL improves ImageNet-22K training and related workloads by increasing cache effectiveness, reducing disk I/O, and coordinating preprocessing across concurrent jobs.

  • Evaluation with ImageNet22k: 20% higher cache hits yield 1.5× faster ShuffleNet and 1.4× faster AlexNet and ResNet18 training with ImageNet-22K.The comparison is against DALI-shuffle on Config-SSD-V100.
  • Evaluation with ImageNet22k: Distributed ImageNet-22K training reaches 1.3× speedup for AlexNet, 1.33× for ShuffleNet, and 1.12× for ResNet18 on two servers.Fetch stalls are lower than for OpenImages because ImageNet-22K has smaller average images.
  • Evaluation with ImageNet22k: CoorDL achieves up to 2.5× speedup for eight concurrent hyperparameter-search jobs.The evaluation uses Config-SSD-V100 across seven image-classification models.
  • Evaluation with ImageNet22k: CoorDL reduces disk I/O by 47% versus DALI-seq and 33% versus DALI-shuffle by reducing cache thrashing.The OpenImages experiment uses a server caching 65% of the dataset.
  • Evaluation with ImageNet22k: Coordinated preprocessing accelerates concurrent jobs by reusing prepared minibatches while adding 5GB of process memory.For ImageNet-1K, the reported speedups are 1.9× on AlexNet and 1.2× on ResNet50.

C.2 Evaluation

Py-CoorDL reduces training and hyperparameter-search time by combining cache-aware data access with coordinated preprocessing, with benefits depending on storage speed and preprocessing cost.

  • C.2 Evaluation: 2.1×–3.3× lower per-epoch time is achieved on hard drives through sequential reads and fewer cache misses.MinIO reduces cache misses by 20% relative to the page-cache LRU scheme.
  • C.2 Evaluation: On SSDs, a 20% reduction in store misses translates to only 7% lower training time because CPU preprocessing is the bottleneck.The SSD throughput is 500MBps, while preprocessing throughput is around 327MBps.
  • C.2 Evaluation: Coordinated preprocessing reduces data-stall time as concurrent jobs increase by reusing work across jobs.The microbenchmark compares four jobs with six workers each against eight jobs with three workers each.
  • C.2 Evaluation: On hard drives, coordinated preprocessing provides up to 2.5× speedup, while adding MinIO raises the effective speedup to nearly 5.5×.The improvements correspond to reduced disk accesses, fewer storage misses, and fewer random accesses.
  • C.2 Evaluation: On SSDs, coordinated preprocessing speeds hyperparameter search, but adding MinIO provides little additional benefit because I/O is inexpensive.The bottleneck shifts from storage throughput to CPU preprocessing when SSD throughput exceeds preprocessing throughput.
Loading 2007.06775v3…