Source-linked AI summary
Faster Neural Network Training with Data Echoing
Dami Choi, Alexandre Passos, Christopher J. Shallue, George E. Dahl
TL;DR
Upstream input stages can leave accelerators idle when they cannot keep pace with SGD. Data echoing repeats earlier-stage outputs to reclaim capacity, and the paper reports reduced upstream computation across workloads alongside a 3.25-fold walltime decrease in one ImageNet setting.
Problem
When upstream pipeline stages cannot supply images as quickly as the SGD step consumes them, the accelerator is partly idle.
Source: §1 · p. 2Method
Results
Across datasets and architectures, echoing reduced upstream computation while reaching competitive out-of-sample error and matching well-tuned baselines’ final error; ResNet-50 on ImageNet had a factor of 3.25 decrease in walltime when training data were read over a network.
Source: §1 · p. 2, §4 · p. 9Takeaways & Limitations
Data echoing is an alternative to optimizing the pipeline or adding workers, though its speedup depends on the model, dataset, batch size, and shuffling.
Source: §4 · p. 9The analysis assumes upstream processing takes at least as long as downstream processing, the setting that motivates data echoing.
Source: §2 · p. 3
Abstract
from arXiv · showhide
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.
1. Introduction
When upstream input work cannot keep pace with SGD, accelerators can sit idle; data echoing reuses available data to reduce upstream computation. The introduction reports competitive performance across varied workloads and distinguishes echoing from related data-reuse approaches.
1. Introduction
- When upstream stages process images more slowly than the SGD step consumes them, the accelerator is partly idle.Slow disk or cloud transfer, time-consuming preprocessing, or too few CPU threads can cause this mismatch.Source: §1 · p. 2
- Data echoing repeats outputs from an earlier pipeline stage after a bottleneck, reclaiming idle accelerator capacity instead of waiting for more data.The paper focuses on reducing examples read from disk as a reproducible proxy for walltime when upstream computation is the bottleneck.Source: §1 · p. 2
- Across datasets and architectures, data echoing reduced upstream computation to reach competitive out-of-sample error and matched well-tuned baselines’ final error.The introduction also reports a walltime speedup in practice and says echoing can support a wide range of factors and insertion points.Source: §1 · p. 2
1.1. Related work
- Prior work differs: experience replay usually does not control repetitions, local SGD targets communication overhead, and example subsampling uses less accelerator computation per example.Earlier minibatch-persistency and batch-augmentation studies did not tune optimization hyperparameters, leaving comparison with well-tuned baselines unresolved.Source: §1.1 · p. 3
2. Data Echoing
Data echoing inserts a stage that repeats outputs from an earlier pipeline stage, allowing each upstream step to serve multiple downstream SGD updates. Its potential speedup depends on timing, insertion point, and how repeated data are shuffled and reused.
2. Data Echoing
- An echoing stage repeats outputs from the preceding pipeline stage; the echoing factor e specifies how many times each data item is repeated.The stage can optionally shuffle its outputs, which may require additional memory.Source: §2 · p. 3
- When upstream processing takes at least as long as downstream processing, each echoing factor e ≤ R adds downstream steps without increasing completion time per upstream step.R is the upstream-to-downstream processing-time ratio; the analysis assumes upstream processing is the slower side.Source: §2 · p. 3
- If echoing leaves the required number of SGD updates unchanged, training time falls by a factor of e; less valuable repeated data can require more updates and reduce that speedup.The realized speedup may therefore be smaller than e.Source: §2 · pp. 3–4
- Echoing just before the SGD update can exploit the most idle downstream time, while earlier insertion points may offer different trade-offs.Before batching, example echoing diversifies examples across batches but may duplicate examples within a batch; after batching, batch echoing repeats batches.Source: §2 · p. 4
- Echoing before augmentation lets repeated data receive different transformations, and dropout can also make repeated data appear different during the SGD update.Echoing after augmentation does not provide that new augmentation for each repetition.Source: §2 · p. 4
- A post-echo shuffle buffer makes nearby SGD updates more likely to use different data, but the assumed buffer is only a relatively small fraction of the augmented dataset.A larger buffer more closely approximates loading the whole training set before random sampling, but requires memory.Source: §2 · p. 4
3. Experiments
Across language modeling, image classification, and object detection workloads, the experiments test data-echoing variants using fresh examples as a proxy for upstream work. They examine echoing placement, factor, batch size, and shuffling, as well as predictive performance.
3.1. Data echoing can reduce the number of fresh examples required for training
- With echoing factor 2, all but one workload required fewer fresh examples than baseline; batch echoing on ResNet-50 required about the same, and earlier insertion required fewer.The experiments found no negative interaction with batch normalization for ResNet-50 or SSD.Source: §3.1 · p. 5, Figure 3 · p. 6
3.2. Data echoing can reduce training time
- For ResNet-50 on ImageNet, data echoing provided up to a speedup factor of 3.25 at echoing factor 5, with walltime speedup proportional to fresh examples read over the network.The walltime reduction was slightly lower than the fractional reduction in fresh examples because the implementation incurred slight overhead.Source: §3.2 · p. 6, Figure 4 · p. 7
3.3. Data echoing can be useful up to a reasonable upper bound on the echoing factor
- For Transformer on LM1B, example echoing at factor 16 still required fewer fresh examples than baseline, while larger batches supported larger useful echoing factors.At batch size 1024, the maximum useful factor was between 4 and 8; at batch size 4096, it was above 16.Source: §3.3 · p. 6, Figure 5 · p. 7
3.4. Data echoing as batch size increases
- As batch size increased, batch echoing stayed as effective or improved relative to baseline, while example echoing stayed as effective or worsened.The authors suggest duplicate examples within larger batches may explain the latter pattern; additional shuffling could reduce duplicates at a memory cost.Source: §3.4 · p. 6, Figure 6 · p. 7
3.5. Data echoing performs better with more shuffling
- More shuffling improved batch echoing, and larger shuffle buffers improved example echoing, although larger buffers did not help the baseline.The authors attribute the example-echoing result to fewer duplicate examples within each batch.Source: §3.5 · pp. 7–8, Figure 7 · p. 8
3.6. Data echoing does not harm predictive performance
- All data-echoing variants achieved at least the baseline’s best out-of-sample performance on Transformer-LM1B and ResNet-50-ImageNet.The comparison used an echoing factor of 4 and fixed budgets of fresh examples.Source: §3.6 · p. 8, Figure 8 · p. 8
4. Conclusion
Data echoing reduced the input examples read across the considered workloads, and the conclusion reports a 3.25-fold wall-time decrease for network-fed ResNet-50 on ImageNet. Its speedup depends on workload and pipeline conditions, with the upstream-to-downstream processing-time ratio offered as a useful setting guide.
4. Conclusion
- Across the considered workloads, at least one echoing variant reduced the number of examples read, including Transformer on Common Crawl, which reached target performance using only a subset of baseline examples.Echoing after augmentation also reduced disk reads, making it appealing for image datasets with expensive CPU augmentation.Source: §4 · p. 8
- 3.25-fold lower wall time was measured for ResNet-50 on ImageNet when training data were read over a network.The conclusion notes that echoing before augmentation typically provides the greatest speedup when input reading is the bottleneck.Source: §4 · p. 9
- Speedup depends on architecture, dataset, batch size, and shuffling; setting the echoing factor to the upstream-to-downstream processing-time ratio maximized potential speedup and worked well in the experiments.The authors present data echoing as an alternative to optimizing the pipeline or adding workers, options that may not be possible or desirable.Source: §4 · p. 9
A. Best training curves for all experiments
The figures show the best training curves across experiments, with solid lines giving means over the best trials from five independent hyperparameter searches, shaded regions showing one standard deviation, and dashed lines marking target metric values.
A. Best training curves for all experiments
- Figures 9, 10, and 11 show best training curves for experiments in Sections 3.1, 3.2, and 3.3, respectively.Solid lines represent means over the best trials from five independent hyperparameter searches, and shaded regions represent one standard deviation.Source: Figure 9 · p. 11, Figure 10 · p. 12, Figure 11 · p. 12
- Figures 12 and 13 show best training curves for Transformer on LM1B and ResNet-50 on ImageNet, respectively.Each curve summarizes the best trial from five independent hyperparameter searches; shaded regions indicate one standard deviation and dashed lines indicate target metric values.Source: Figure 12 · p. 13, Figure 13 · p. 14
- Figures 14 and 15 show best training curves for batch echoing and example echoing, respectively.Both figures mark the target metric with a dashed line and show one standard deviation around the mean over the best trials from five independent hyperparameter searches.Source: Figure 14 · p. 14, Figure 15 · p. 15