PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

Yu Emma Wang, Gu-Yeon Wei, David Brooks

arXiv:1907.10701v4cs.LGcs.PFstat.ML

TL;DR

Deep-learning benchmarking needs broader coverage to capture interactions among evolving models, hardware, and software. The paper introduces ParaDnn, combines it with six real-world models, and evaluates TPU, GPU, and CPU platforms. It finds platform-specific strengths, identifies TPU bottlenecks, and quantifies specialized software-stack improvements.

  • Problem

    Existing benchmarks provide limited coverage for systematically analyzing interactions among model attributes, hardware choices, and software support.

  • Method

    The paper uses ParaDnn to generate parameterized FC, CNN, and RNN training models, combines them with six real-world models, and benchmarks TPU, GPU, and CPU platforms.

  • Results

    The platforms have unique strengths for different models, while the study reveals TPU bottlenecks and performance improvements from specialized software stacks.

  • Takeaways & Limitations

    Broad parameterized benchmarking provides architectural and system-design insights beyond conclusions drawn from a few fixed models.

  • Takeaways & Limitations

    The study evaluates training but not inference, cloud overhead, multi-node systems, accuracy, or convergence.

Abstract

from arXiv · show

Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware specialization to improve performance. To systematically benchmark deep learning platforms, we introduce ParaDnn, a parameterized benchmark suite for deep learning that generates end-to-end models for fully connected (FC), convolutional (CNN), and recurrent (RNN) neural networks. Along with six real-world models, we benchmark Google's Cloud TPU v2/v3, NVIDIA's V100 GPU, and an Intel Skylake CPU platform. We take a deep dive into TPU architecture, reveal its bottlenecks, and highlight valuable lessons learned for future specialized system design. We also provide a thorough comparison of the platforms and find that each has unique strengths for some types of models. Finally, we quantify the rapid performance improvements that specialized software stacks provide for the TPU and GPU platforms.

1. INTRODUCTION

The paper argues that existing deep-learning benchmarks are too narrow for understanding interactions among models, hardware, and software. It introduces ParaDnn and combines it with real workloads to benchmark platforms and expose architecture and software insights.

  • Findings: The paper reports that no platform is best for all scenarios, with different platforms offering advantages for different models.It also examines TPU architectural bottlenecks, cross-platform differences, and specialized software stacks.
  • Benchmarking gap: Existing benchmarks use small, seemingly arbitrary model collections, so single-model studies can miss important hardware bottlenecks.Transformer trains 3.5× faster on TPU than GPU, yet does not reveal the TPU memory-bandwidth bottleneck for FCs exceeding 4k nodes.
  • ParaDnn: ParaDnn generates thousands of parameterized FC, CNN, and RNN models across almost six orders of magnitude in parameter size.Its range exceeds existing benchmarks and spans from 10k to nearly a billion parameters.
  • Study scope: The study combines ParaDnn with six real-world models to compare TPU, GPU, and CPU platforms across diverse workloads.The benchmark examines platform differences and interactions among model attributes, hardware choices, and software support.
  • Limitations: The study focuses on training and does not evaluate inference, multi-GPU platforms, or 256-node TPU systems.These omitted settings may lead to different conclusions and are left for future work.

2. DEEP LEARNING BENCHMARKING

ParaDnn complements fixed real-world and micro-benchmark suites with parameterized end-to-end FC, CNN, and RNN workloads. Its configurable model and dataset attributes cover a broad design space while six real models anchor the benchmark in current applications.

  • Existing benchmarks: Existing real-world suites contain only a handful of current models, while micro-benchmarks do not provide broad end-to-end coverage.Because deep-learning models evolve rapidly, fixed suites may become obsolete and fail to reveal interactions among model attributes.
  • ParaDnn Models: ParaDnn generates end-to-end FC, CNN, and RNN models and parameterizes their architectures, datasets, and training batch sizes.The suite sweeps variables such as layers, nodes, filters, embedding size, sequence length, vocabulary size, and datasets.
  • ParaDnn Models: ParaDnn CNNs use residual-network structures with four block groups followed by a fully connected layer.Blocks can be residual or bottleneck variants, and the benchmark sweeps blocks per group and minimum filters.
  • Real-World Models: The benchmark includes Transformer, ResNet-50, RetinaNet, DenseNet, MobileNet, and SqueezeNet as real-world workloads.These models cover translation, image classification or detection, and different model sizes and architectures.
  • Real-World Models: ParaDnn spans 10k to nearly a billion trainable parameters, exceeding the range of the real workloads shown as individual points.Transformer is the largest real FC, RetinaNet the largest real CNN, and SqueezeNet and MobileNet represent smaller mobile-oriented models.

3. HARDWARE PLATFORMS

The study compares a Skylake CPU, an NVIDIA V100 GPU, and a Cloud TPU configuration using minimally available platform units. Their architectures differ in compute, memory, and distribution characteristics relevant to model support and batch size.

  • Hardware Platforms: The CPU is a 16-core, 32-thread Skylake instance with 120 GB memory and 2 TFLOPS peak performance.It has the largest memory and lowest peak FLOPS among the three platforms.
  • Hardware Platforms: The GPU is a single NVIDIA V100 SXM2 node with 16 GB memory, 900 GB/s bandwidth, and 125 TFLOPS mixed-precision peak performance.The DGX-1 platform contains eight V100 packages connected by 300 GB/s NVLink 2.0, but this study measures one node.
  • Hardware Platforms: The TPU configuration is a Cloud TPU board containing four TPU packages, with computation distributed across packages automatically.The comparison uses one V100 package and one TPU board because they are the minimal available units.
  • Hardware Platforms: TPU data parallelism splits each batch across eight cores while keeping a complete model copy on every core.Per-core memory limits maximum model size, whereas total board memory limits maximum batch size.
  • Comparison rationale: Multi-GPU and multi-node TPU performance is outside the study’s scope because multi-GPU performance depends substantially on user implementation.Consequently, the reported conclusions apply to the selected single-package GPU and single-board TPU configurations.

4. TPU ARCHITECTURAL IMPLICATIONS

The TPU analysis identifies bottlenecks in computation utilization, memory bandwidth, communication, and host-device data balance, while comparing TPU generations. ParaDnn exposes how model attributes and workloads shape these effects.

  • 4.1 FLOPS Utilization: TPU utilization rises with batch size and model width, while model depth is under-exploited, suggesting model pipelining opportunities.FC utilization increases with nodes, CNN utilization with filters, and RNN utilization with embedding size.
  • 4.2 Roofline Model Analysis: Memory bandwidth bottlenecks many models; even compute-bound ResNet-50 contains 13% memory-bound operations.Large fused MatMuls are the principal compute-bound operations, while many others remain bandwidth-constrained.
  • 4.4 Host-Device Balance: Data quantization can shift compute-bound workloads into data-infeed-bound workloads, making host-device balance important.Resolving infeed bottlenecks can improve performance by at least 34%; RetinaNet, ResNet-50, and SqueezeNet are affected in bfloat16.
  • 4.5 TPU v3: TPU v3 improves compute-bound operations by about 2.3× and can speed memory-bound operations by up to 3× over v2.The memory-bound improvement reflects doubled memory capacity, larger batch size, and increased memory bandwidth.

5. CROSS-PLATFORM COMPARISON

Across TPU, GPU, and CPU platforms, no single platform is best for every workload: strengths depend on model type, batch size, size, and computational regularity. TPU favors large batches and CNNs, GPU offers flexibility and large-FC performance, while CPU supports irregular RNNs and the largest models.

  • TPU has the highest training throughput and is highly optimized for large batches and CNNs.
  • GPU provides better flexibility for irregular computations and small batches, while large FC models benefit from its memory system and bandwidth.
  • CPU achieves the highest FLOPS utilization for RNNs and supports the largest models because it has the greatest memory capacity.
  • Fully-Connected DNNs: Large batch sizes improve TPU-over-GPU speedups because TPU systolic arrays need large batches for full utilization.
  • Fully-Connected DNNs: GPU is the best platform for large FC models, whereas models with large batch sizes perform best on TPU.
  • CNNs: All CNNs perform better on TPU, and larger CNNs show higher TPU-over-GPU speedups, consistent with TPU optimization for spatial weight reuse.
  • The Google ResNet-50 implementation reaches 6.2× speedup over GPU, compared with 4.2× using NVIDIA’s reported implementation, highlighting software-stack effects.
  • RNNs: TPU and GPU achieve less than 26% and 9% FLOPS utilization for RNNs, respectively, while CPU reaches up to 46%.

6. SOFTWARE STACK ADVANCES

Software-stack updates substantially improve deep-learning platform performance, especially on TPU, while quantization increases capacity and can relieve memory pressure. Gains vary by model family and software version, with RNNs benefiting particularly late in the TensorFlow sequence.

  • TensorFlow Versions and TPU Performance: TensorFlow 1.7 to 1.12 improves performance for all ParaDnn models, although TensorFlow 1.8 temporarily regresses FC and CNN performance.
  • TensorFlow Versions and TPU Performance: TensorFlow 1.11 produces 10× speedup for RNN and 7.5× for LSTM and GRU, while Transformer, ResNet-50, and RetinaNet improve continuously across updates.
  • The 90th-percentile TPU speedup over seven months reached 7× for FC, 1.5× for Residual CNN, 2.5× for Bottleneck CNN, 9.7× for RNN, and 6.3× for LSTM and GRU.
  • Quantization: bfloat16 enables 90th-percentile speedups up to 1.8× for FC and Bottleneck CNN and 1.3× for Residual CNN.
  • CUDA Versions and GPU Performance: Bitwidth reduction can speed up CNNs by more than 2× by lowering memory traffic and enabling larger batch sizes.
  • CUDA Versions and GPU Performance: CUDA 9.2 speeds up ResNet-50 by 8%, compared with less than 1% for other real workloads.

7. LIMITATIONS OF THIS WORK

The study evaluates single-platform training under constrained experimental conditions and leaves several system-level and outcome-level questions for future work. Its conclusions may differ for inference, multi-node systems, accuracy, convergence, or broader training settings.

  • The study does not evaluate deep-learning inference, cloud overhead, multi-node systems, accuracy, or convergence.
  • NVIDIA’s eight-node DGX-1 and Google’s 256-TPU systems are outside the study’s scope.
  • Multi-node evaluation introduces node counts, inter-node bandwidth, topology, synchronization, and more acute cloud-system overhead.
  • Extrapolating training throughput to time-to-accuracy remains an open question because very large batches can scale sub-linearly and optimal batch size depends on model and optimizer.

8. RELATED WORK

Prior deep-learning benchmarks either use small collections of current real-world models or focus on lower-level microbenchmarks. ParaDnn addresses this space with parameterized end-to-end workloads intended to cover broader and evolving model designs.

  • Existing real-world benchmark suites contain only a handful of popular models and may become obsolete as deep-learning models evolve rapidly.
  • Microbenchmark suites provide lower-level evaluations, whereas ParaDnn targets large end-to-end models with parameterized attributes.
  • ParaDnn is presented as the first parameterized benchmark suite for deep learning in the literature.

9. CONCLUSION

The paper presents a comprehensive analysis of deep learning training hardware and software using ParaDnn and six real-world models to compare TPU, GPU, and CPU platforms. It identifies specialized-system design insights and TPU bottlenecks while motivating further research.

  • The study comprehensively benchmarks deep neural network training hardware and software across TPU, GPU, and CPU platforms.
  • ParaDnn, combined with six real-world models, enables comparisons across diverse end-to-end deep learning workloads.
  • The analysis presents TPU architectural bottlenecks and observations that inform specialized hardware and software design.
  • The paper's findings motivate further work in specialized deep learning hardware and software.
Loading 1907.10701v4…