Source-linked AI summary
MLPerf Training Benchmark
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang, Atsushi Ike, Bill Jia, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, Matei Zaharia
TL;DR
ML training requires benchmarks that account for quality effects, stochastic variation, and diverse software and hardware systems. MLPerf addresses these challenges with representative workloads, reference implementations, rules, and timing procedures, and reports rapid performance and scaling improvements across two rounds while noting framework-specific limitations.
Problem
ML training benchmarks must account for optimizations that affect quality, stochastic run-to-run variation, and diverse software implementations that complicate fair comparison.
Method
MLPerf combines diverse benchmarks with precisely defined reference implementations, equivalent-submission and hyperparameter rules, and timing rules that reduce stochasticity effects.
Results
Across two rounds on unchanged underlying hardware, MLPerf results show rapid performance and scaling improvement in implementations and software stacks.
Takeaways & Limitations
MLPerf provides a maintained benchmark suite spanning industrial and academic use cases, with realistic data and portable hyperparameters informing system comparisons.
Takeaways & Limitations
Frameworks can implement mathematically different training procedures and interpret interfaces differently, complicating cross-framework comparisons and portability.
Abstract
from arXiv · showhide
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.
1 INTRODUCTION
MLPerf addresses the need for fair, reproducible, affordable, and broadly useful measurement as ML systems and workloads diversify. It combines a representative benchmark suite with reference implementations, equivalence rules, stochasticity-aware timing, open submissions, and ongoing maintenance.
- 21?
- MLPerf targets fair measurement, reproducibility, commercial and research use, and affordable participation.
2 BACKGROUND
ML benchmarking requires end-to-end evaluation because training optimizations, stochasticity, and diverse software and hardware can change both model quality and time to solution. Existing benchmarks span low-level operations and whole-model throughput, but do not fully capture these challenges or ensure fair comparisons across systems.
- Benchmarking challenges: End-to-end ML benchmarks must evaluate training to target quality because operation, implementation, and numerical-representation choices affect the resulting model.Unlike traditional compute benchmarks, ML systems must train a selected model to state-of-the-art quality before performance comparisons are meaningful.
- Benchmarking challenges: Accuracy differences between precision choices may emerge only after tens of epochs, so microbenchmarks cannot reliably predict full-session training outcomes.Some lower-precision representations never match full-precision validation error, despite improving immediate performance metrics.
- Benchmarking challenges: Large-scale data parallelism and minibatches require optimizer changes that alter learning dynamics and the iterations needed to reach target accuracy.These effects mean higher system utilization or lower nominal training time does not alone determine time to solution.
- Benchmarking challenges: Stochastic training causes run-to-run variation in accuracies and epochs to target, while floating-point and asynchronous-update differences make comparisons difficult to reproduce.Figure 1 reports considerable MiniGo variability even when the random seed is fixed.
- System diversity: Frameworks and libraries can implement mathematically equivalent operations differently, producing numerical differences that complicate portability and fair benchmarking.Programming-interface differences also affect padding and augmentation order, while interoperability tools have limited support.
- Prior benchmarks: Prior benchmarks measure kernels, model throughput, or narrower hardware and workload settings, whereas MLPerf follows an end-to-end time-to-train approach for more-diverse tasks and fairer evaluation.DAWNBench established a related end-to-end approach; MLPerf extends it with rules and mechanisms for fair comparisons in its Closed division.
3 MLPERF TRAINING BENCHMARK
MLPerf Training defines a compact, representative suite of seven modern ML benchmarks and measures end-to-end time to reach task-specific quality targets. Its rules specify workloads, timing boundaries, and system operations needed for fair training comparisons.
- 3.1 Benchmark Suite: Seven benchmarks span vision, language, recommendation, and reinforcement learning while representing diverse compute motifs and commercial or research relevance.The suite was selected using feedback from commercial and academic organizations and was designed to remain affordable.
- 3.1.1 Image Classification: ResNet-50 v1.5 fixes model architecture, initialization, optimizer schedule, data augmentation, and downsampling choices to make image-classification comparisons meaningful.The specification addresses differences among ResNet-50 implementations in training-framework repositories.
- 3.1.2 Object Detection and Segmentation: COCO-based detection and segmentation use Mask R-CNN-style workloads with higher-resolution inputs and measure both tasks using mAP.These workloads include operations such as upscaling, ROIalign, NMS, and sorting, which distinguish them from image classification.
- 3.1.3 Translation: Translation benchmarks use WMT EN-DE and Bleu-based quality measurement, covering both Transformer and recurrent GNMT architectures.GNMT represents recurrent applications, while Transformer represents attention-based translation models.
- 3.2 Time-to-Train Performance Metric: MLPerf measures time to train to a defined quality target, combining system speed and accuracy while retaining end-to-end data-pipeline and validation costs.Timing starts when training or validation data is first accessed and stops when the validation target is achieved; initialization overhead is excluded.
4 BENCHMARKING PROCESS
MLPerf organizes submissions into reviewed divisions and categories that balance direct workload comparison with innovation and practical system availability. Reports include performance together with system scale because training systems differ widely in cost and power context.
- 4.1 Submission and Review: MLPerf runs submission and review periods for successive rounds, with v0.5 and v0.6 results published after compliance checks.Submissions include system descriptions, logs, code, and libraries needed for reproduction, and the materials are publicly available.
- 4.2.1 Submission Divisions: Closed submissions require mathematically equivalent models, initialization, optimization, schedules, and data processing, while open submissions permit architectural and optimization differences.Both divisions use the reference data set and quality metric, but the closed division restricts hyperparameter modification to support direct comparison.
- 4.2.2 System Categories: Available, preview, and research categories distinguish systems by production availability and intended use, separating shipping products from prototypes and larger-scale configurations.Preview systems must meet available-category criteria within 60 days or by the next submission cycle and then be submitted as available.
- 4.2.3 Reporting Scale: Performance reports include processor or accelerator configuration because ML training spans multiple orders of magnitude in system power draw and cost.For cloud systems, MLPerf derives cloud scale from host processors, host memory, and accelerator number and type, and reports that it correlates closely with cost across three providers.
- 4.2.4 Reporting Results: MLPerf reports time to train for each benchmark rather than a suite-wide summary score because no weighting scheme is universally representative across its diverse applications.Figure 3 compares fastest 16-chip entries between v0.5 and v0.6 and separately shows quality-target increases.
5 RESULTS
Across two MLPerf Training rounds on unchanged hardware, best 16-chip performance improved while the fastest overall results required substantially more chips.
- 1.3× average improvement in best performance results for a 16-chip system occurred between v0.5 and v0.6.
- The number of chips necessary to produce the fastest time to solution increased by as much as 5.5× from v0.5 to v0.6.
- 5.5× average increase in the number of chips necessary to produce the best overall performance result occurred between the two rounds.
6 CONCLUSIONS
MLPerf Training provides standardized, representative benchmarks and procedures for comparing ML systems, while its results reveal practical lessons about data, hyperparameters, optimizers, and ongoing benchmark evolution.
- MLPerf Training represents industrial and academic use cases through a widely used benchmark suite.
- Precise model architectures and training procedures enable comparisons for equivalent workloads.
- Reference implementations and rules address stochastic training, quality effects of optimizations, and workload variation across system scales.
- Realistic data-set sizes are critical for realistic memory-system behavior, and startup time should be excluded when smaller-than-industrial data sets are benchmarked.
- Small hyperparameter changes can substantially change performance, while hyperparameters are relatively portable at similar system scales.
- Frameworks exhibit subtle optimizer-algorithm variations that affect convergence.
- MLPerf maintains and updates its suite as the field evolves, including changes to ResNet-50, GNMT, and MiniGo in v0.6.
A.1 Abstract
The artifact describes a workflow for reproducing Nvidia’s MLPerf v0.5 image-classification submissions and points readers to broader result repositories, while noting that the results lack independent reproducibility verification.
- The artifact documents how to run Nvidia’s v0.5 image-classification submissions on a single-node DGX-1 system.
- Additional details for DGX-2 and multi-node systems are provided in official MLPerf results repositories.
- MLPerf v0.5 and v0.6 training results from other tasks and submitters are also available.
- The results have not been independently verified for reproducibility.
A.2 Artifact check-list (meta-information)
The checklist identifies the artifact’s MLPerf program, ResNet-50/ImageNet workload, NVIDIA DGX hardware, time-to-train metric, reproducible workflow materials, and licensing details.
- The artifact uses MLPerf with ResNet-50 v1.5 on ImageNet and NVIDIA DGX-1 or DGX-2 hardware.
- Time-to-Train measures minutes to reach the 74.9% Top-1 accuracy threshold for v0.5.
- Execution ends once the accuracy threshold is reached and produces an MLPerf-compliant log with timestamps and evaluation accuracy.
- The workflow includes a shell script, uses MXNet, and requires approximately 300 GB of disk space.
- Workflow preparation takes approximately 2 hours, experiments approximately 8 hours, and the code is publicly available under Apache License 2.0.
- The artifact provides an archival DOI and links to the official TensorFlow implementation repository.
A.3.1 How to access
The MLPerf v0.5 training results are available on GitHub, with an Nvidia ResNet-50 submission README providing implementation-specific details. One reported result lists a Time-to-Train of 134.6 minutes.
- MLPerf v0.5 training results are available on GitHub.The cited passage provides the repository URL.
- Nvidia’s v0.5 ResNet-50 submission includes a README with implementation details.The cited passage directs readers to the submission README.
- 134.6 minutes is the reported Time-to-Train.