PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Datacenter Performance Analysis of a Tensor Processing Unit

Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, Doe Hyun Yoon

arXiv:1704.04760v1cs.ARcs.LGcs.NE

TL;DR

Datacenter neural-network inference demands low response time, but evidence was limited on whether custom domain-specific hardware could deliver major cost-energy-performance gains over contemporary CPUs and GPUs. The paper evaluates a production-workload TPU against a Haswell CPU and K80 GPU, finding the TPU averages 15X–30X higher inference speed and 30X–80X higher TOPS/Watt.

  • Problem

    Datacenter neural-network inference emphasizes response time, motivating evidence on whether custom ASICs can improve cost-performance over conventional CPUs and GPUs.

  • Method

    The paper compares a production-workload TPU with contemporary Haswell CPU and K80 GPU platforms across MLP, CNN, and LSTM inference applications.

  • Results

    15X–30X faster inference and 30X–80X higher TOPS/Watt are achieved by the TPU than contemporary GPU and CPU products.

  • Takeaways & Limitations

    A deterministic execution model, large software-managed memory, and 8-bit matrix multiplication make the TPU a small, low-power accelerator suited to latency-constrained inference.

  • Takeaways & Limitations

    The comparison does not establish performance against newer GPUs such as the P40, which was unavailable and not contemporary with the evaluated platforms.

Abstract

from arXiv · show

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.

1. Introduction to Neural Networks

Neural networks use layered nonlinear transformations and are developed through training before production inference, where quantization can make computation substantially more efficient. The paper evaluates a TPU for inference on six representative applications spanning popular NN types and reports major latency, throughput, and efficiency advantages over contemporary CPUs and GPUs.

  • Neural-network fundamentals: Neural networks apply nonlinear functions to weighted sums, with each layer feeding its outputs to the next; deep networks use extra and larger layers.These structures target brain-like functionality and benefit from large cloud datasets.
  • Training, inference, and quantization: Training determines network weights for development, while inference uses those weights for production prediction; training is usually performed in floating point.Quantization converts computations to lower-precision representations for inference.
  • Training, inference, and quantization: 6X less energy and 6X less area characterize eight-bit integer multiplies versus IEEE 754 16-bit floating-point multiplies.Integer addition offers 13X lower energy and 38X lower area.
  • NN application types: Three popular NN types are MLPs, CNNs, and RNNs, differing respectively in fully connected, spatially local, and recurrent weighted-sum structures.The paper’s workload includes applications from these NN categories.
  • Paper scope and previewed results: 15X - 30X faster inference is reported for the TPU than for the K80 GPU and Haswell CPU across the paper’s evaluated applications.The six applications represent 95% of the TPU’s workload; user-facing inference emphasizes response time over throughput, contributing to K80 underutilization.

2. TPU Origin, Architecture, and Implementation

The TPU emerged from datacenter-scale inference demands and was designed as a flexible PCIe coprocessor that runs whole neural-network inference models. Its architecture centers on a 256x256 8-bit MAC matrix unit, large on-chip buffering, and systolic execution to sustain computation while reducing data movement.

  • Motivation: A 2013 projection that voice-search speech-recognition DNNs would require datacenters to double shifted attention toward deploying custom inference hardware.Earlier discussions had concluded that special-purpose hardware could use excess datacenter capacity virtually for free.
  • Implementation: The TPU was implemented as a PCIe coprocessor, with the host sending instructions, to accelerate deployment, simplify design and debugging, and avoid tight CPU integration.This design let it plug into existing servers like a GPU.
  • Architecture: The TPU targets whole-model inference and centers on a 256x256 matrix unit performing 8-bit multiply-and-adds, with 16-bit products collected in 4 MiB of 32-bit accumulators.The matrix unit can perform either matrix multiplication or convolution and reads and writes 256 values per clock cycle.
  • Memory system: The design stages weights from 8 GiB off-chip Weight Memory through a four-tile Weight FIFO and stores intermediate results in a 24 MiB on-chip Unified Buffer.A programmable DMA controller transfers data between CPU host memory and the Unified Buffer.
  • Execution model: Systolic execution reduces Unified Buffer reads and writes because SRAM access consumes more power than arithmetic, while a four-stage instruction pipeline overlaps operations to keep the matrix unit busy.Layer dependencies can nevertheless create delay slots in which the matrix unit waits for explicit synchronization.

3. CPU, GPU, and TPU Platforms

The study benchmarks six production neural-network applications representing 95% of TPU datacenter use on contemporaneous, reliability-qualified CPU, GPU, and TPU platforms. It uses an 18-core dual-socket Haswell server and a four-card Nvidia K80 configuration with up to eight dies, reporting most results per die.

  • Workload: Six production applications represent 95% of TPU use in the datacenters and form the paper’s workload.One CNN derives from widely used Inception V2.
  • Platform selection: Benchmark platforms were available in 2015, included SECDED protection for internal SRAM and external DRAM, and were sensibly configured for deployment.These requirements excluded some alternatives, such as Nvidia’s Maxwell GPU.
  • CPU platform: The CPU platform is an 18-core, dual-socket Intel Haswell processor fabricated in a 22nm process and serving as the host for GPUs and TPUs.The CPU and GPU are both about 600 mm^2 dies.
  • GPU platform: The benchmarked GPU platform uses four Nvidia K80 cards containing up to eight dies, with Boost mode reaching 875 MHz.K80 Boost mode is software-controlled and lasts at least hundreds of milliseconds, affecting power and cooling provisioning.
  • Reporting convention: Because benchmarked servers contain 2 to 8 dies, results are usually normalized per die, with whole-system results shown occasionally.Per-die results appear in Figures 5-8, Figures 10-11, and Tables 3, 4, and 6; Figure 9 shows a whole system.

4. Performance: Rooflines, Response-Time, and Throughput

The TPU roofline analysis shows that most applications approach their computation- or memory-bandwidth ceilings, while CPU/GPU inference is constrained below its ceilings by response-time limits. Including host overhead, the TPU substantially outperforms Haswell and K80, especially under the workload’s actual application mix.

  • Roofline analysis: 1350 operations per byte of weight memory fetched is the TPU roofline’s ridge point, with MLPs and LSTMs memory bound and CNNs computation bound.Five of six applications reach the roofline ceiling; CNN1 is an exception, achieving only 14.1 TOPS versus CNN0’s 86 TOPS despite high operational intensity.
  • Roofline analysis: The TPU roofline redefines operational intensity as integer operations per byte of weights read because quantized NN weights generally do not fit on-chip.This adapts the model from floating-point operations and cache-resident assumptions to the TPU’s quantized inference workload.
  • Response-time limits: A 7 ms 99th-percentile response-time limit restricts Haswell and K80 MLP0 throughput to 42% and 37%, respectively, of their unconstrained maxima.Larger batches could increase throughput but violate the latency requirement, wasting the processors’ potentially higher throughput.
  • Relative inference performance: 14.5 times as fast is the TPU die’s geometric-mean performance relative to Haswell, versus 1.1X for K80 and 13.2 times as fast relative to K80, including host overhead.Using the actual workload mix raises TPU performance to 29.2X relative to Haswell and 15.3 times as fast relative to K80.

5. Cost-Performance, TCO, and Performance/Watt

Because chip prices are unavailable, the paper uses performance/Watt as a proxy for performance/TCO. Relative to Haswell and K80 servers, the TPU delivers substantially higher total and incremental performance/Watt.

  • Cost-Performance, TCO, and Performance/Watt: Power correlates with TCO, so performance/Watt serves as the paper’s proxy for performance/TCO when chip prices cannot be published.The paper reports Watts per server instead of negotiated chip prices.
  • Cost-Performance, TCO, and Performance/Watt: Figure 9 reports total performance/Watt including host CPU power and incremental performance/Watt after subtracting host CPU server power.Both geometric and weighted means are shown relative to the Haswell CPU.
  • Cost-Performance, TCO, and Performance/Watt: 1.2–2.1X total-performance/Watt and 1.7–2.9X incremental-performance/Watt characterize the K80 server relative to Haswell.The incremental calculation omits Haswell server power.
  • Cost-Performance, TCO, and Performance/Watt: 17–34 times better total-performance/Watt and 41–83 times better incremental-performance/Watt characterize the TPU server relative to Haswell.The TPU reaches 14–16 times the K80’s total performance/Watt and 25–29 times its incremental performance/Watt.

6. Energy Proportionality

The TPU has the lowest measured power but poor energy proportionality, consuming nearly full-load power even at low utilization; Haswell scales power most proportionally among the compared chips.

  • Measurement methodology: Power was measured as offered workload utilization varied from 0% to 100% in 10% workload increments, using equal batch sizes.The experiment plotted both incremental and total power for the GPU and TPU configurations.
  • Measured power and proportionality: 118W per die total and 40W per die incremental make the TPU the lowest-power design measured.These values correspond to TPU+Haswell/2 total power and TPU incremental power, respectively.
  • Measured power and proportionality: At 10% load, the TPU uses 88% of its 100%-load power, indicating poor energy proportionality.The paper attributes this partly to the short design schedule, which prevented inclusion of many energy-saving features.
  • Measured power and proportionality: At 10% load, Haswell uses 56% of its 100%-load power, making it the best in energy proportionality among the compared chips.The K80 is closer to Haswell than the TPU, using 66% of its full-load power at 10% load.

7. Evaluation of Alternative TPU Designs

The TPU design model closely matches hardware counters and shows that memory bandwidth, rather than clock rate, is the dominant performance lever across the evaluated applications. A hypothetical TPU' with faster logic and GDDR5 would substantially improve performance efficiency, with manageable area and power costs and host-overhead limitations.

  • Model validation: 8% average difference validates the TPU performance model against hardware performance counters.The model was used to vary memory bandwidth, clock rate, accumulator count, and matrix multiply unit size.
  • Parameter sensitivity: 3X average performance improvement results from scaling memory bandwidth by 4X, while scaling clock rate provides little average benefit.MLPs and LSTMs are memory bound, whereas CNNs are compute bound.
  • Memory capacity: 14 MiB of Unified Buffer suffices for the six applications, versus the original 24 MiB capacity.The improved allocator reduced the largest application’s requirement from the full capacity to 14 MiB, leaving margin for larger models.
  • Hypothetical TPU': More than 5X higher Weight Memory bandwidth from a GDDR5 interface shifts the roofline ridge point from 1350 to 250.The hypothetical TPU' also assumes a 50% clock-rate increase enabled by more aggressive logic synthesis and block design.
  • Costs and limitations: 10% larger die and roughly 900 Watts of system power result from replacing DDR3 with GDDR5, while reducing the Unified Buffer to 14 MiB can recover 10% area.The GDDR5 configuration requires doubling memory channels from two to four; host interaction overhead reduces TPU' means from 2.6 to 1.9 and 3.9 to 3.2.
  • Hypothetical TPU': 31X - 86X total performance/Watt/die over Haswell and 25X - 41X over K80 are projected for TPU'.The incremental metric reaches 69X - 196X over Haswell and 42X - 68X over K80.

8. Discussion

The discussion rebuts assumptions about inference workloads, accelerator fit, and evaluation while identifying limitations in current NN-hardware research and opportunities for further TPU tuning. It emphasizes that response time, workload diversity, and application complexity materially affect architectural conclusions and summary metrics.

  • Fallacies and pitfalls: Response-time demands in datacenter NN inference were stronger than expected, challenging the assumption that applications value throughput equally or can simply accumulate larger batches.The paper notes that some developers expected large batches or looser latency requirements, despite strong response-time demands.
  • Fallacies and pitfalls: The K80’s throughput-oriented architecture is a poor match for NN inference, helping explain why it is only slightly faster than Haswell and much slower than the TPU.The K80 relies on high-bandwidth DRAM and thousands of threads to achieve high throughput.
  • Fallacies and pitfalls: 15% of ISCA 2016 papers addressed NN accelerators, yet all nine examined CNNs even though CNNs represented only about 5% of the datacenter NN workload.The discussion argues that architects have neglected important NN tasks beyond CNNs.
  • Fallacies and pitfalls: 75X: TPU Inferences Per Second varies from 360,000 IPS for 4-layer MLP1 to 4,700 IPS for 89-layer CNN1, making IPS a poor single summary metric.IPS primarily reflects the complexity of the typical inference, including the number, size, and type of NN layers.
  • Fallacies and pitfalls: 1.4X: K80 Boost mode increased LSTM1 performance while raising power by 1.3X, producing only a 1.1X performance-per-Watt gain.The clock rate rose by up to 1.6X, from 560 to 875 MHz, but Boost mode had a minor impact on the energy-speed analysis.
  • Fallacies and pitfalls: ~3.5X: Efficient AVX2 integer support improved CPU performance for one 8-bit DNN, but presenting all CPU results in floating point avoided a separate exception and roofline.The paper originally had 8-bit results for only one DNN because efficiently using AVX2 integer support required significant work.
  • Fallacies and pitfalls: CNN1 TPU performance could improve through software work, such as aggregating short convolution batches into a deeper batch from 32 to 128 for four fully connected layers.The proposed reorganization would better match CNN1 to the TPU hardware.

9. Related Work

Prior NN accelerators used custom ASICs, systolic arrays, reconfigurable logic, and dataflow techniques to improve performance, efficiency, or latency. The TPU differs from contemporary alternatives in its deployed scale, hardware resources, and measured CNN performance.

  • Prior NN accelerators: 25 years of prior work included custom NN ASICs such as CNAPS and Synapse-1, which used multiplier arrays and custom systolic multiply-accumulate chips.CNAPS used a 64-SIMD array of 16-bit by 8-bit multipliers, while Synapse-1’s MA-16 performed sixteen 16-bit multiplies at a time.
  • Prior NN accelerators: 25 times faster inference and 20 times faster training were reported for T0-accelerated SPERT-II workstations versus a SPARC-20 workstation.The 40-MHz T0 added vector instructions to MIPS and used an eight-lane vector unit supporting up to sixteen 32-bit arithmetic results per cycle.
  • Architectural alternatives: DianNao minimized memory accesses, while related designs pursued on-chip weight storage, CNN-specific acceleration, sparsity, pruning, quantization, and lower-precision or analog computation.The original DianNao used 64 16-bit integer MACs and 44 KB of on-chip memory; DaDianNao added eDRAM for 36 MiB of weights, and Minerva reduced power by 8X.
  • Benchmark comparison: The Fathom benchmark’s GPU-faster-than-CPU result differed from this paper because its systems were not server-class, its CPU had four cores, AVX was unused, and it lacked a response-time cutoff.These methodological differences affect the comparison with the TPU’s server-class, latency-sensitive evaluation.
  • Catapult comparison: Catapult deployed 3,926 18-bit MACs, 5 MiB of on-chip memory, 11 GB/s bandwidth, and 25 Watts, versus the TPU’s 65,536 8-bit MACs, 28 MiB, 34 GB/s, and 40 Watts.Catapult chose FPGAs over GPUs to reduce power and latency-mapping risk, while retaining reconfigurability for other workloads.
  • Catapult comparison: 2.3X as fast as a 2.1 GHz, 16-core, dual-socket server was reported for Catapult V1 on CNNs, while a current TPU ran CNNs 40X to 70X versus a somewhat faster server.Catapult V2 performance was projected at 7X, or perhaps 17X with more careful floorplanning.

10. Conclusion

The TPU’s success stems from domain-specific hardware, substantial software-controlled memory, deterministic execution, and flexibility for neural-network inference. Its order-of-magnitude advantages suggest it may become an archetype for future domain-specific architectures.

  • Performance interpretation: Four of six neural-network applications are memory-bound, yet low utilization of the TPU’s large resource can still deliver cost-effective performance.The Roofline performance model motivates the paper’s “Cornucopia Corollary” to Amdahl’s Law.
  • Hardware specialization: 25 times as many MACs and 3.5 times the on-chip memory let the TPU use less than half the K80’s power.The TPU uses 65,536 8-bit MACs and 28 MiB of on-chip memory versus the K80’s 2,496 32-bit MACs and 8 MiB.
  • Workload composition: About 5% of representative neural-network demand consists of CNNs, suggesting greater architectural attention should go to MLPs and LSTMs.The paper compares this pattern with historical overemphasis on floating-point performance despite integer-dominated mainstream workloads.
  • Inference requirements: 99th-percentile latency deadlines matter because inference applications are often user-facing, favoring architectures optimized for response time rather than throughput.The K80 excels at training but is on average only a little faster than Haswell for this inference workload.
  • Implications: Order-of-magnitude differences between commercial products are rare, potentially making the TPU an archetype for domain-specific architectures.The authors expect successors to raise the performance bar further.
Loading 1704.04760v1…