PaperlayerView exact revision on arXiv

Source-linked AI summary

MLPerf Inference Benchmark

Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, Yuchen Zhou

arXiv:1911.02549v2cs.LGcs.PFstat.ML

TL;DR

The rapid growth and architectural diversity of ML inference make fair, representative, and reproducible benchmarking difficult. MLPerf Inference establishes a community-developed benchmark method with realistic scenarios, quality targets, and flexible rules; its first evaluation produced over 600 submissions and revealed four orders of magnitude of performance variation.

  • Problem

    Diverse ML hardware, software, models, and deployment conditions create a need for architecture-neutral, representative, and reproducible inference benchmarks.

  • Method

    MLPerf Inference combines realistic scenarios, predefined accuracy and performance metrics, quality targets, and rules allowing flexible implementations across systems.

  • Results

    Over 600 submissions from 14 organizations were received, 595 were cleared as valid, and measured performance varied across four orders of magnitude.

  • Takeaways & Limitations

    MLPerf provides a comprehensive framework for comparing inference accelerators across use cases where an accelerator may perform differently by scenario.

Abstract

from arXiv · show

Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.

I. INTRODUCTION

ML inference has become a critical, rapidly expanding workload, but diverse systems and inconsistent prior benchmarks make fair, reproducible comparison difficult. MLPerf Inference addresses this with community-developed scenarios, quality targets, permissive implementation rules, and modularity for evolving models.

  • Motivation: Inference has become a critical workload as ML applications generate enormous query and translation volumes.Models can serve as many as 200 trillion queries and perform over 6 billion translations daily.
  • Motivation: Over 100 companies target specialized inference chips, while systems trade off latency, throughput, power, and model quality across many hardware-software combinations.These combinations make evaluating inference performance nearly intractable.
  • Benchmarking gap: Prior benchmarks lack industry-wide consensus on representative models, metrics, tasks, and rules.Existing academic and industrial efforts made substantial contributions but were developed without broad ML-system-designer input.
  • Contribution: MLPerf Inference is a jointly developed standard benchmark suite and method for fairly measuring ML hardware, software, and services.Researchers and developers from more than 30 organizations contributed to its methodology.
  • Method: The benchmark combines representative reproducible workloads, realistic scenarios, model-quality and tail-latency targets, and semantic-level rules that preserve implementation freedom.Closed and open divisions respectively enforce stricter comparability or permit model changes and alternative quality-performance targets.
  • Adaptability: The modular design allows models and tasks to change frequently while preserving scenarios, target qualities, and infrastructure.This design reduces the cost of adding new models and supports continued benchmark maintenance.
  • Impact: More than 600 submissions from 14 organizations were tested, with 595 cleared as valid and performance spanning four orders of magnitude.The submissions covered varied tasks, frameworks, and platforms from embedded devices and smartphones to data centers.

II. INFERENCE-BENCHMARKING CHALLENGES

Inference benchmarking must represent diversity in models, deployment scenarios, data arrival patterns, and system stacks. These dimensions create different accuracy, latency, memory, complexity, and throughput tradeoffs that defeat one-size-fits-all evaluation.

  • Overview: A useful ML benchmark must address model diversity, deployment-scenario variety, and inference-system diversity.These are identified as three critical benchmarking challenges.
  • Diversity of Models: No single image-classification model is optimal because models trade off accuracy, memory requirements, and computational complexity.The model space forms a Pareto frontier rather than one universally best choice.
  • Deployment-Scenario Diversity: Quality requirements differ by application, with pedestrian detection requiring higher accuracy than labeling animals in photographs.Inference QoS requirements also range from effectively unconstrained offline processing to millisecond-scale real-time applications.
  • Deployment-Scenario Diversity: Offline processing can batch readily available data, whereas real-time applications handle constant data flow and may not benefit from batching.On-device inference latency alone therefore fails to reflect real-world requirements.
  • Inference-System Diversity: The many combinations of models, data sets, frameworks, libraries, systems, and platforms complicate systematic and reproducible benchmarking.Figure 2 characterizes substantial complexity on both the hardware and software sides.

C. Inference-System Diversity

Inference-system diversity spans software frameworks, optimized libraries, runtimes, hardware platforms, tasks, and reference models. MLPerf responds with representative workloads and flexible, mathematically equivalent implementations to preserve comparison across this design space.

  • Inference-System Diversity: ML inference combines numerous software frameworks, optimized libraries, and runtimes, each with implementation-specific idiosyncrasies.Common examples include TensorFlow and PyTorch frameworks, cuDNN and MKL libraries, and TensorRT and ONNX Runtime.
  • Inference-System Diversity: Hardware ranges from CPUs and GPUs to DSPs, FPGAs, ASICs, and analog or mixed-signal accelerators.Hardware-specific features and constraints affect performance depending on the model and scenario.
  • Benchmark Design: MLPerf selects representative workloads and describes benchmark principles to cover diverse systems while supporting robust, flexible evaluation.The approach is based on community input and includes benchmarks, quality targets, and scenarios.
  • Benchmark Design: MLPerf defines high-level tasks with canonical reference models and accepts mathematically equivalent implementations.This semantic-level design gives systems implementation freedom while retaining relevant comparisons.
  • Representative Workloads: The initial suite covers vision and language tasks using mature reference models spanning architectures such as CNNs and RNNs.The selected tasks are intended to serve systems from edge devices to cloud data centers.
  • Representative Workloads: Image classification uses ResNet-50 v1.5 and MobileNet-v1-1.0-224 to represent heavier, more accurate and lighter, faster model choices.MobileNet-v1 reduces parameters by 6.1× and operations by 6.8× versus ResNet-50 v1.5.
  • Representative Workloads: Object detection uses lightweight and heavyweight COCO models, including MobileNet-v1 and SSD with a ResNet-34 backbone.The models target constrained devices and higher-resolution automotive or industrial settings.
  • Representative Workloads: Translation uses the WMT16 EN-DE data set, SacreBLEU scoring, and a GNMT reference model with an RNN architecture.This adds a language workload to the suite alongside vision tasks.

B. Robust Quality Targets

MLPerf establishes application-specific quality targets because accuracy, latency, and TCO tradeoffs vary across inference workloads. It evaluates realistic usage scenarios with distinct latency and throughput metrics.

  • Quality targets: Per-model quality targets constrain inference systems to preserve application-appropriate accuracy while balancing latency, TCO, and throughput.The paper notes that sacrificing 1% accuracy for 50% lower TCO may suit cat-photo identification but is less appropriate for pedestrian detection.
  • Quality targets: MobileNet and SSD-MobileNet could not meet the required accuracy without retraining, making low-accuracy benchmarking unrepresentative.MLPerf addressed this with quantization-friendly weights, provided 8-bit implementations, and a reduced SSD-MobileNet quality requirement.
  • Realistic scenarios: MLPerf defines four scenarios—single-stream, multistream, server, and offline—to represent distinct inference applications and workload behaviors.The scenarios emulate mobile devices, autonomous vehicles, robotics, and cloud-based setups.
  • Realistic scenarios: Single-stream measures 90th-percentile latency for sequential one-sample queries, whereas multistream measures supported streams under a QoS requirement.Multistream sends N-sample queries at fixed intervals and permits no more than 1% of queries to produce skipped intervals.
  • Realistic scenarios: Server measures QPS under random Poisson arrivals and latency bounds, while offline measures throughput in samples per second without latency constraints.Server limits latency-bound violations to 1% for vision tasks and 3% for translation.
  • Realistic scenarios: Latency constraints in multistream and server scenarios restrict batching optimizations because arrivals and tail latency must satisfy QoS requirements.Server systems may process inferences with suboptimal batch sizes because query arrivals are random.

D. Statistically Confident Tail-Latency Bounds

MLPerf sizes benchmark runs to make tail-latency measurements statistically reliable and representative of steady-state behavior. Query requirements depend on latency percentiles, confidence, margin, and scenario-specific constraints.

  • Statistical requirements: Each task and scenario requires a minimum query count to obtain statistically robust measurements of steady-state system behavior.The count is determined by the tail-latency percentile, desired margin, and confidence interval.
  • Statistical requirements: MLPerf uses a 99% confidence bound and sets the margin to one-twentieth of the difference between the tail-latency percentage and 100%.The margin is intended to remain relatively small compared with the distance from the tail-latency percentile to 100%.
  • Statistical requirements: Equation 2 determines the number of queries required for a statistically valid latency-bound throughput measurement.The paper relates this calculation to sample-size determination for an electoral poll with an infinite electorate.
  • Scenario requirements: The single-stream scenario requires 1,024 queries, while offline requires 1 query containing at least 24,576 samples.The single-stream count keeps benchmark runtime short enough for embedded platforms and smartphones.
  • Scenario requirements: A 99th-percentile guarantee requires 262,742 queries, rounded to 270K; machine translation with a 97th-percentile guarantee requires 90K queries.For multistream, total samples equal N×270K when each query processes N samples.
  • Run protocol: Multistream and server runs balance repeatability against duration, with five server runs reported by their minimum and one run for other scenarios.Multistream runs can last 2.5–7.0 hours, and the authors expect to revisit this choice.
  • Run protocol: All benchmarks run for at least 60 seconds to capture equilibrium behavior in power-management and DVFS systems.This minimum is particularly relevant to single-stream benchmarks with few queries.

IV. INFERENCE SUBMISSION SYSTEM

MLPerf separates fixed benchmark-owned components from the submitter-owned system under test to preserve comparability across architectures. Its rules permit implementation diversity while restricting techniques that undermine equivalence or production relevance.

  • System boundary: A submission contains the SUT, LoadGen, dataset, and accuracy script, with MLPerf fixing the latter three components while submitters implement the SUT.This boundary supports comparability across architectures while allowing engineering discretion inside the SUT.
  • Equivalence rules: Model-equivalence rules use an explicit blacklist to permit efficient reimplementation while encouraging architectural diversity.The rules define disallowed techniques and provide examples of allowed optimizations.
  • Equivalence rules: Allowed optimizations include data rearrangement, alternate numerical representations, equivalent transformations, sparsity, operation fusion, and dynamic batch-size switching.These techniques may change implementation while preserving mathematical equivalence or approved approximation behavior.
  • Neutrality: MLPerf permits untimed preprocessing so implementations can transform inputs into system-specific forms without reducing architecture neutrality.The paper acknowledges that no vendor- or application-neutral preprocessing exists.
  • Neutrality: Quantization supports architecture neutrality across formats including INT4, INT8, FP16, bfloat16, and FP32.Submitters register numerical formats in advance to guide accuracy-target discussions.
  • Prohibited techniques: Retraining, pruning, caching, and benchmark- or dataset-aware optimizations are prohibited to maintain comparability and production relevance.The benchmark also forbids exploiting predetermined query traffic or performance and accuracy dataset statistics.
  • System workflow: Figure 3 places LoadGen between sample loading, query issuance, result processing, and accuracy-log verification.The sequence runs from SUT sample loading through benchmark results returned to LoadGen and logs consumed by the accuracy script.

B. Load Generator

The LoadGen generates scenario-specific traffic, measures SUT performance, and supports accuracy validation and compliance. Its decoupled design centralizes technical rules while enabling multiple languages and future scenarios.

  • LoadGen operation: LoadGen generates query traffic for the four scenarios and records SUT queries, responses, logs, debugging information, and postprocessing data.Its behavior is controlled by a configuration file read at the start of each run.
  • LoadGen operation: At startup, LoadGen requests dataset loading, waits for the SUT readiness signal, and then sends queries according to scenario rules.Compilation, cache warmup, and preprocessing may occur as untimed operations before the first query.
  • Operating modes: Accuracy mode processes the complete dataset to verify quality, whereas performance mode uses enough samples to determine system performance.Accuracy results are written to logs for checking against the required target.
  • Extensibility: Decoupling LoadGen from benchmarks lets new scenarios and experiments roll out across models and SUTs without extra submitter effort.It also centralizes rules for query arrivals, timing, and accuracy, facilitating compliance and auditing.
  • Extensibility: Language bindings for Python, C, and C++ allow benchmark implementations in multiple languages, while the interface can support future modes such as multitenancy.A multitenancy mode would continuously serve multiple models while maintaining QoS constraints.
  • Measurement scope: LoadGen measures holistic end-to-end SUT performance rather than the performance of an individual component.This matches inference engines’ role as black-box components within larger systems.
  • Validation: Publicly available datasets and LoadGen compliance features support reproducible participation and validation of submissions.LoadGen can self-check whether its source code was modified during submission, and experimental configurations expose nondefault features.
  • Validation: Submissions are organized into divisions and categories, peer reviewed for validity, and reported through a defined evaluation process.The paper describes this process as part of the benchmark’s submission and reporting methodology.

A. Result Submissions, Divisions, and Categories

MLPerf Inference supports flexible submissions across specialized systems while preserving comparability through standardized divisions, validation, and public review. The benchmark avoids reducing diverse task capabilities to a single summary score.

  • Result submissions: Submissions include performance scores, benchmark code, system descriptions, and LoadGen logs uploaded publicly for peer review and validation.The records cover configuration characteristics and performance and accuracy runs for task–scenario combinations.
  • Result submissions: MLPerf Inference permits submissions for a subset of tasks and scenarios because ML systems are often highly specialized.Unlike benchmarks requiring every component, this approach allows specialized systems to report relevant capabilities.
  • Divisions: The closed division standardizes models, data sets, quality targets, and reference-equivalent implementations for cross-system comparison.It permits calibration for quantization but prohibits retraining.
  • Divisions: The open division allows model, architecture, preprocessing, postprocessing, and quality-target changes, so open submissions are generally not directly comparable.It is intended to foster innovation in algorithms, optimization, and hardware/software co-design.
  • Validation: The validation suite uses experiments to check compliance and reproducibility despite proprietary inference engines and quantization flows.Accuracy, caching, random-seed, and custom-data-set tests address common sources of invalid results.
  • Evaluation policy: MLPerf Inference provides no summary score because weighting diverse ML tasks is subjective and system priorities differ.A single number could be driven by submitters’ customer-specific needs rather than a neutral weighting.

VI. BENCHMARK ASSESSMENT

The benchmark assessment shows broad workload coverage and realistic scenario effects across submitted systems. Latency constraints substantially alter throughput, with the magnitude depending on both system and model.

  • Assessment scope: More than 600 submissions from 14 organizations covered ML tasks, scenarios, hardware architectures, and software runtimes after extensive review.The approved results formed a large public corpus of inference-performance data.
  • Assessment scope: The initial benchmark selected representative workloads, scenarios, permissive rules, and a method that can evolve as benchmarks change.These design goals target reproducibility, realistic evaluation, broad capability coverage, and adaptability.
  • Workload coverage: ResNet-50 v1.5 was the most popular model but appeared just under three times as often as GNMT, the least popular model.The relatively small spread suggests broad interest across the selected models.
  • Scenario usage: Server scenarios deliver less throughput than offline scenarios because latency constraints make large batches harder to form.The resulting suboptimal batching exposes performance considerations that unconstrained studies can overlook.
  • Scenario usage: System B loses about 50% or more throughput across three models, whereas system A loses as much as 40% for NMT and approximately 10% for vision models.Differences may reflect low-batch-size architecture, dynamic batching, or both.
  • Scenario usage: NMT server throughput falls 39–55%, while ResNet-50 v1.5 falls 3%–35% and MobileNet-v1 averages under 10%.The results indicate that latency effects vary substantially by network type and extrapolate poorly across models.

C. Processor Types and Software Frameworks

MLPerf Inference evaluates a wide range of processor architectures, software stacks, and deployment scales. Its submissions span specialized hardware and diverse optimization approaches, demonstrating broad applicability.

  • Processor coverage: Submissions covered CPUs, GPUs, DSPs, FPGAs, and ASICs, indicating that the benchmark can evaluate most processor architectures.The reported coverage includes both programmable and fixed-function platforms.
  • System range: The smallest and largest inference systems differed by four orders of magnitude, or about 10,000×, in performance.The range extended from mobile and edge devices to cloud computing.
  • System range: Large performance differences of 100× appeared for MobileNet-v1 single-stream, ResNet-50 v1.5 single-stream, and SSD-MobileNet-v1 offline.These models span systems from low-power embedded devices to high-performance servers.
  • Applicability: The broad performance range suggests the initial tasks represent multiple use cases and market segments, while the method is widely applicable.This conclusion is based on the range and diversity of submitted systems and scenarios.
  • Open-division innovation: The open division received 429 results featuring 4-bit quantization, alternative models, and high throughput under tighter latency bounds.Participants could change models and quality targets under the less restrictive rules.
  • Open-division innovation: Multiple accelerators operated concurrently in some mobile multistream submissions, illustrating industry efforts to push chipset performance limits.Such concurrent use was described as rare in conventional mobile situations.

A. Models: Breadth vs. Use-Case Depth

MLPerf balances model and application breadth against realistic use-case depth, while exposing dataset and measurement limitations. Results also show that simple operation counts can misrepresent measured inference difficulty.

  • A. Models: Breadth vs. Use-Case Depth: The benchmark implemented 4 versions of each benchmark, 20 in total, but limited resources prevented including speech recognition, recommendation, and newer model families.The authors aimed to add models such as Transformers, BERT, and DLRM later.
  • A. Models: Breadth vs. Use-Case Depth: Latency-bounded throughput combines latency and throughput because excellent unconstrained throughput can coexist with poor latency-constrained performance.Server scenarios impose latency bounds and nonuniform arrivals, making large batches harder to form.
  • C. Data Sets: Public vs. Private: The SSD-large benchmark uses upscaled 1,200x1,200 COCO images because suitable public large-image detection data sets lack ideal annotations.The authors identify a need for larger, better-quality public data sets and more data-set generators.
  • D. Performance: Modeled vs. Measured: SSD-ResNet-34 requires 175× more operations per image than SSD-MobileNet-v1 but achieves only 50–60× lower throughput.The consistent 3× gap between operation-count and throughput ratios shows that network structure affects performance.
  • D. Performance: Modeled vs. Measured: Reviewers found about 40 issues among approximately 180 closed-division results and released only 166.Problems included unmet quality targets, latency bounds, query requirements, and rule misinterpretations.

IX. CONCLUSION

MLPerf Inference defines a comprehensive framework for evaluating inference performance across use cases through consensus metrics, accuracy targets, realistic scenarios, and standardized methods. Its maintenance process acknowledges that ML tasks and models continue to evolve.

  • Performance metrics: MLPerf Inference measures performance with latency, latency-bounded throughput, throughput, and maximum inferences per query under predefined accuracy targets.
  • Performance metrics: Latency-bounded throughput captures data-center inference constraints and extends a metric commonly used for data-center CPUs to ML accelerators.
  • Accuracy/performance tradeoff: Industry and academic experts set accuracy and tolerable-degradation thresholds, standardizing accuracy/performance tradeoffs for distributed evaluation and optimization.
  • Evaluation of AI inference accelerators: MLPerf evaluates accelerators across server, single-stream, multistream, and offline scenarios, recognizing that optimization benefits vary by use case.
  • Evaluation of AI inference accelerators: Batching in server, multistream, and offline scenarios across five networks exposes additional optimization opportunities for AI-system development and research.
  • Maintenance: Because ML is still evolving, MLPerf updates tasks and models while retaining the established benchmarking method.
Loading 1911.02549v2…