PaperlayerView exact revision on arXiv

Source-linked AI summary

Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

Cody Coleman, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Re, Matei Zaharia

arXiv:1806.01427v2cs.LGstat.ML

TL;DR

Deep-learning optimizations can improve speed while changing accuracy, complicating comparisons based only on throughput. The paper analyzes DAWNBENCH and MLPERF v0.5 using time-to-accuracy (TTA), finding that TTA is stable, TTA-optimized models generalize nearly as well as standard models, and optimized systems still have hardware and communication bottlenecks.

  • Problem

    Speed-oriented ML optimizations can affect accuracy on unseen data, while throughput-only metrics do not jointly evaluate runtime and generalization.

    Source: §1 · p. 1

  • Method

    The paper analyzes DAWNBENCH and MLPERF v0.5 entries to evaluate TTA behavior and identify bottlenecks in highly optimized training systems.

    Source: §1 · p. 2, §3.2 · p. 3, §4.1 · p. 4

  • Results

    TTA is stable across multiple tasks, and models optimized for it generalize nearly as well as off-the-shelf models.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Highly optimized entries still leave significant room for improvement because they underuse hardware and distributed training can spend more than half of its time communicating.

    Source: §7 · p. 10

    The authors could not make a completely fair apples-to-apples comparison of AmoebaNet and ResNet-50 scaling on GPUs.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model's accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBench, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset---a combined metric called time-to-accuracy (TTA). In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries. We show that TTA has a low coefficient of variation and that models optimized for TTA generalize nearly as well as those trained using standard methods. Additionally, even though DAWNBench entries were able to train ImageNet models in under 3 minutes, we find they still underutilize hardware capabilities such as Tensor Cores. Furthermore, we find that distributed entries can spend more than half of their time on communication. We show similar findings with entries to the MLPERF v0.5 benchmark.

1 Introduction

ML training optimizations can improve speed while changing generalization, making throughput-only evaluation inadequate. DAWNBENCH introduced time-to-accuracy (TTA), and this paper analyzes its stability, generalization, and system bottlenecks across DAWNBENCH and MLPERF.

1 Introduction

  • TTA combines end-to-end training time with near-state-of-the-art accuracy on held-out data, addressing the limits of throughput-only metrics.Earlier metrics such as time-per-epoch and time-to-training-loss do not jointly capture runtime and final accuracy on unseen data.Source: §1 · p. 1
  • The study evaluates professionally optimized, mostly open-source DAWNBENCH and MLPERF v0.5 entries to characterize TTA and identify bottlenecks.DAWNBENCH entries reached 93% top-5 ImageNet accuracy in under 3 minutes with rolling submissions.Source: §1 · pp. 1–2
  • TTA is relatively stable across image classification, object detection, and machine translation, despite stochastic training procedures.The conclusion reports a coefficient of variation below 14% across these tasks.Source: §1 · p. 2
  • Models optimized for TTA generalize nearly as well as off-the-shelf models, despite accuracy being measured on a fixed held-out set.This addresses the concern that repeated benchmark tuning could implicitly adapt models to the evaluation set.Source: §1 · p. 2
  • Distributed training often spends more than half of total time on communication, including on cloud infrastructure and optimized on-premise deployments.Communication is therefore a major bottleneck in highly optimized entries.Source: §1 · p. 2
  • Top-performing entries can underutilize hardware capabilities such as Tensor Cores by up to 10×, while ReLUs can become performance bottlenecks.These findings indicate that substantial optimization opportunities remain even after achieving very short ImageNet training times.Source: §1 · pp. 1–2

2 Background: ML Training

ML training seeks models that generalize to unseen data, so faster computation can trade off against solution quality. This background describes stochastic training, validation-based evaluation, and speed–generalization trade-offs underlying the benchmark analysis.

2 Background: ML Training

  • ML training differs from pure optimization because models must perform well on unseen data rather than only fit training data.High-capacity models can overfit training data and generalize worse than simpler models.Source: §2 · p. 2, Figure 1 · p. 2
  • A held-out validation dataset measures generalization during training, and systems commonly stop when validation performance plateaus.Repeated experimentation can still overfit this held-out data through hyperparameter tuning.Source: §2 · p. 2
  • Stochastic Gradient Descent and variants such as Adam update model weights over randomly sampled minibatches across repeated dataset passes called epochs.Random initialization, data traversal, and operation ordering can cause repeated trials to reach the same validation accuracy in different numbers of epochs.Source: §2 · pp. 2–3
  • Optimizations that increase hardware efficiency can alter convergence, so faster processing does not necessarily preserve final model quality.Larger updates can slow convergence, while naive reduced precision can prevent convergence; loss scaling can restore convergence at 16 bits.Source: §2 · p. 2

3 Overview of Benchmarks

This section defines the DAWNBENCH and MLPERF benchmark settings analyzed, their scope and exclusions, and the optimized entries spanning hardware, software, and compute scales.

3.1 DAWNBENCH Overview

  • DAWNBENCH measures training time and cloud cost to specified accuracy, alongside single-item inference latency and cost across deep learning workloads.Its initial tasks included ImageNet, CIFAR10, and SQuAD.Source: §3.1 · p. 3
  • The analysis focuses on reproducible DAWNBENCH ImageNet training submissions targeting 93% top-5 accuracy, excluding SQuAD and CIFAR10.SQuAD submissions lacked sufficient code and CIFAR10 was judged not representative of production workloads.Source: §3.1 · p. 3

3.2 MLPERF Overview

  • MLPERF v0.5 evaluates time-to-accuracy across image classification, object detection, translation, and recommendation using fixed models and optimization algorithms.Submissions could choose SGD hyperparameters and report only a subset of tasks.Source: §3.2 · p. 3
  • The analysis excludes MLPERF reinforcement-learning entries because game simulation was the bottleneck and recommendation results because that task did not reflect production usage and was being replaced.The recommendation model and dataset were also being replaced for the next MLPERF version.Source: §3.2 · pp. 3–4

3.3 Summary of Entries

  • Entries came from organizations including Google, NVIDIA, and Intel and used GPUs, TPUs, or CPUs with TensorFlow, PyTorch, Caffe, MXNet, or Big DL.The analyzed set included pre-February 2019 submissions reproducible on public cloud infrastructure or supported by sufficient analysis information.Source: §3.3 · p. 4
  • Compute scale ranged from 2 to 640 processors, while speedups over reference implementations ranged from 1.6× to over 1,400×.In MLPERF, accelerator entries used mixed precision and large batch sizes; DAWNBENCH allowed additional techniques such as progressive resizing and new architectures.Source: §3.3 · p. 4

4 Analysis of Time-to-Accuracy

The analysis finds that TTA is generally stable, preserves generalization on unseen data, and captures training-system performance better than throughput or peak FLOPS alone.

4.1 Variability of Time-to-Accuracy

  • Several DAWNBENCH ImageNet entries using progressive resizing failed to consistently reach the target accuracy, making validation convergence less robust.This finding qualifies the otherwise low observed TTA variability.Source: §4.1 · p. 4
  • TTA remains largely stable despite stochastic training, with coefficient of variation below 14% for MLPERF and up to 12.2% across reproduced DAWNBENCH entries.For DAWNBENCH entries without novel statistical optimizations, the maximum coefficient of variation is 4.5%.Source: §4.1 · p. 4, §4 · p. 4
  • Most TTA variation comes from differing epochs to target rather than time per epoch, with epoch-count variation up to 45× higher in selected large-scale MLPERF entries.Selected entries had time-per-epoch coefficients of variation below 3%.Source: §4.1 · p. 4, Table 4 · p. 5

4.2 Generalization of Optimized Models

  • Models optimized for TTA generalize nearly as well as reference or pretrained models on unseen image-classification and translation data.Optimized GNMT and Transformer models generalized as well as reference implementations despite being over 50× faster.Source: §4.2 · pp. 4–5, Table 5 · p. 6, Table 6 · p. 6

4.3 Comparison to Alternative Metrics

  • Throughput is insufficient as a performance metric because throughput and TTA speedups can differ by up to 3× when larger batches increase epochs to convergence.Table 7 reports a transformer example of 10.9× throughput speedup versus 3.6× TTA speedup.Source: §4.3 · p. 5, Table 7 · p. 6
  • Peak device FLOPS is also a poor proxy: system scale does not correlate with speedup, accelerators are underutilized, and communication can add more than 2× overhead.The analysis reports that distributed entries can spend more than half their time communicating.Source: §4.3 · pp. 5–6

5 Hardware Utilization and Scaling

The analysis finds that distributed training scales poorly across servers because communication becomes a major overhead, while single-worker entries substantially underutilize available hardware through memory-bound and low-throughput kernels.

5.1 Scaling of Distributed Training

  • Within a server, time-per-epoch and TTA scale almost linearly across image classification, object detection, and language translation workloads.The study evaluates scaling by training the same models with different accelerator counts and measuring end-to-end behavior.Source: §5.1 · p. 6
  • Across multiple servers, time-per-epoch and TTA scale worse than linearly, with TTA suffering additionally because larger minibatches require more epochs to reach the same accuracy.Time-per-epoch shows up to a 38.9% gap from linear scaling, with similar behavior for SSD and Mask R-CNN across AWS and private-cloud deployments.Source: §5.1 · p. 6
  • Communication can consume more than half of distributed-training time, reaching 71.15% on 32 DGX-1 servers and 77.46% on AWS p3.16xlarge instances.Within a DGX-1, overhead is 17.82%, compared with 53% across servers, reflecting the bandwidth difference between NVLink and cross-server interconnects.Source: §5.1 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

5.2 Single-worker Utilization

  • Roofline analysis shows that every analyzed DAWNBENCH and MLPERF entry achieves throughput far below peak device throughput, indicating severe hardware underutilization.The analysis instruments training code and profiling tools to estimate operational intensity and throughput for each entry or image size.Source: Figure 7 · p. 8, Figure 8 · p. 8, §5.2 · p. 9
  • FP16 ResNet-50 kernels use Tensor Cores poorly, while FP32 training reaches 7.6 Teraflops versus a 15.7-Teraflop peak and is limited by memory-bound BatchNorm and ReLU kernels.The reported time-averaged Tensor Core utilization is 0.71 on nvprof’s 1–10 scale, and removing BatchNorm and ReLU improves FP32 throughput by about 20%.Source: §5.2 · p. 9, Figure 9 · p. 9, Figure 11 · p. 9
  • Kernel fusion and hardware-aware model co-design are proposed to reduce memory traffic from memory-bound operations such as BatchNorm and ReLU.Fusing the operations can avoid intermediate DRAM reads, while alternative operations could preserve similar statistical effects with less memory pressure.Source: §5.2 · p. 9

6 Related Work

Prior benchmarks often emphasize throughput or older hardware, whereas this work studies optimized, open-source distributed deep-learning systems on modern accelerators and evaluates both speed and generalization.

Benchmarking DL Training

  • Throughput-only metrics cannot fully characterize deep-learning systems because training optimizations can affect the generalizability of the final model.End-to-end evaluation must reason about both runtime and the quality of predictions on unseen data.Source: §6 · p. 10
  • Unlike earlier benchmarks, this analysis covers optimized distributed training on state-of-the-art hardware, including systems with FP16 support and Tensor Cores.The study reports Tensor Core underutilization and communication overheads as high as 71% even in optimized on-premise deployments.Source: §6 · p. 10

High Performance DL

  • DAWNBench and MLPERF provide open-entrant benchmarks with open-source entries for optimizing time-to-accuracy across multiple tasks.The open-source entries enable direct analysis of TTA behavior and the workloads produced by heavily optimized training systems.Source: §6 · p. 10
  • This work extends prior uses of TTA by analyzing its stability and finding that models optimized for it generalize well.Earlier systems used TTA as an optimization metric but generally did not study the metric itself in detail.Source: §6 · p. 10

7 Conclusion

The paper analyzes DAWNBench and MLPERF v0.5 entries to characterize time-to-accuracy and the behavior of highly optimized training systems. It finds that TTA is stable, preserves near-comparable generalization, and still leaves substantial hardware-optimization opportunities.

7 Conclusion

  • TTA has a low coefficient of variation below 14% across image classification, machine translation, and object detection.The result indicates that TTA is usually stable to randomness in ML training across these tasks.Source: §7 · p. 10
  • Models optimized for TTA generalize nearly as well as unoptimized models despite accuracy being measured on a fixed held-out evaluation set.The analysis uses DAWNBench entries and corroborates findings with MLPERF v0.5 entries.Source: §7 · p. 10
  • Entries highly optimized for TTA still underutilize available hardware, leaving significant room for further improvement.The conclusion follows the paper’s broader analysis of hardware utilization and bottlenecks in DAWNBench and MLPERF entries.Source: §7 · p. 10

Related papers

Loading 1806.01427v2…