PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Deep Learning Accelerator Enabled GPUs

Md Aamir Raihan, Negar Goli, Tor Aamodt

arXiv:1811.08309v2cs.MScs.AR

TL;DR

Deep neural networks use computationally costly matrix operations, and NVIDIA's tensor-core design had not been publicly described. This paper characterizes Volta and Turing tensor cores, models Volta tensor cores in GPGPU-Sim, and enables CUTLASS support; the model achieves 99.6% IPC correlation versus a Titan V GPU.

  • Problem

    Deep neural networks require many computationally costly matrix operations, and the underlying design of NVIDIA tensor cores had not been publicly described.

    Source: §I · p. 1

  • Method

    The paper characterizes tensor cores in Volta and Turing, uses its analysis to model Volta tensor cores in GPGPU-Sim, and adds support for running CUTLASS.

    Source: §I · p. 1, §III · p. 3, §VII · p. 12

  • Results

    99.6% IPC correlation was achieved by the Volta tensor-core model versus a Titan V GPU.

    Source: §I · p. 1, §VII · p. 12

  • Takeaways & Limitations

    The work is presented as a promising starting point for further microarchitectural investigation of machine-learning workloads.

    Source: §VII · pp. 12–13

Abstract

from arXiv · show

The efficacy of deep learning has resulted in its use in a growing number of applications. The Volta graphics processor unit (GPU) architecture from NVIDIA introduced a specialized functional unit, the "tensor core", that helps meet the growing demand for higher performance for deep learning. In this paper we study the design of the tensor cores in NVIDIA's Volta and Turing architectures. We further propose an architectural model for the tensor cores in Volta. When implemented a GPU simulator, GPGPU-Sim, our tensor core model achieves 99.6\% correlation versus an NVIDIA Titan~V GPU in terms of average instructions per cycle when running tensor core enabled GEMM workloads. We also describe support added to enable GPGPU-Sim to run CUTLASS, an open-source CUDA C++ template library providing customizable GEMM templates that utilize tensor cores.

I. INTRODUCTION

Deep neural networks rely on computationally costly matrix operations, while tensor cores provide specialized GPU hardware for these operations. The paper investigates tensor cores in Volta and Turing, models Volta tensor cores in GPGPU-Sim, and adds support for running CUTLASS.

I. INTRODUCTION

  • Deep neural networks require many multidimensional matrix computations, motivating efforts to accelerate these operations and the development of specialized hardware.The paper notes that Volta introduced tensor cores for this purpose and that Turing also includes them.Source: §I · p. 1
  • The paper investigates Volta and Turing tensor cores because NVIDIA had not publicly described their underlying design.The investigation includes how threads cooperate to transfer matrix inputs and how tensor operations execute.Source: §I · p. 1
  • Microbenchmark observations informed a proposed Volta tensor-core microarchitectural model and functional and timing changes to GPGPU-Sim.The paper describes the model as consistent with the characteristics revealed by the microbenchmarks.Source: §I · p. 1
  • The authors added support for applications built with CUTLASS to run on GPGPU-Sim.They say the simulator changes should provide a baseline for comparing alternative approaches and ease study of custom-kernel characteristics in frameworks such as PyTorch.Source: §I · p. 1
  • 99.6% IPC correlation was demonstrated for the modified GPGPU-Sim on tensor-core-enabled kernels generated with CUTLASS.The paper reports this as a measure of the simulator's accuracy.Source: §I · p. 1

A. Volta Microarchitecture

Volta's SM is divided into four sub-cores, each with tensor cores and associated scheduling and register resources. The WMMA API exposes warp-wide matrix multiply-accumulate operations, while tile sizes and the mapping of tile elements to threads are constrained or specified as described in the paper.

A. Volta Microarchitecture

  • Each Volta sub-core has two tensor cores, one warp scheduler, one dispatch unit, and a 64 KB register file.The Titan V has 24 more SMs than prior-generation GPUs, while its SM has the same number of registers as Pascal.Source: A. Volta Microarchitecture · p. 2, Figure 1 · p. 2
  • Volta adds twice as many scheduling units per SM as Pascal, separate integer and FP32 cores, and a different way to execute divergent paths within a warp.Both paths following a branch can be executed by threads within a single warp in an interleaved fashion.Source: A. Volta Microarchitecture · p. 2

B. Warp Matrix Function (WMMA) API

  • The WMMA API lets all threads in a warp cooperatively compute D = A × B + C on matrix tiles.The API describes tiles using M × N × K, where A is M × K, B is K × N, and C and D are M × N.Source: B. Warp Matrix Function (WMMA) API · p. 2
  • CUDA 9.0 supports only 16 × 16 × 16 WMMA tiles, while later versions allow additional flexibility.Source: B. Warp Matrix Function (WMMA) API · p. 2
  • WMMA fragments map tile elements to individual threads' registers, but NVIDIA leaves the mapping unspecified.The paper notes that current GPUs use a more sophisticated arrangement than a simple eight-element fragment per thread.Source: B. Warp Matrix Function (WMMA) API · p. 2
  • The WMMA API provides functions to load and store matrix portions and perform a warp-synchronous matrix multiply-accumulate; CUDA libraries also expose tensor cores.The named libraries include cuBLAS, cuDNN,, and CUTLASS,.Source: B. Warp Matrix Function (WMMA) API · p. 2

C. PTX Instruction Set

The section describes the PTX interface and tensor-core operations, then connects them to the paper’s low-level analysis of tensor-core behavior.

C. PTX Instruction Set

  • NVIDIA’s toolchain compiles CUDA into CPU host code and GPU device code, with the latter compiled to device-specific SASS.Source: C. PTX Instruction Set · pp. 2–3
  • Three PTX instructions expose tensor-core operations, with qualifiers specifying synchronization, layout, fragment shape, and operand precision.For Volta, A and B must be FP16, while C may be FP16 or FP32; Turing also supports integer modes.Source: C. PTX Instruction Set · p. 3
  • The wmma.load instructions place operand matrices A, B, and C into warp-distributed registers before a matrix-multiply operation.The corresponding memory addresses are supplied for each operand, and the registers represent thread fragments.Source: C. PTX Instruction Set · p. 3
  • The stride operand in wmma.load and wmma.store specifies the beginning of each row or column for strided tile access.Source: C. PTX Instruction Set · p. 3
  • The wmma.mma instruction computes D = A × B + C using operand registers and stores the result in general-purpose registers.Source: C. PTX Instruction Set · p. 3

D. Tensor Core

  • A tensor core performs one 4 × 4 matrix-multiply-and-accumulate per cycle, while a 16 × 16 WMMA operation may require 64 core operations.The core supports FP16 and mixed-precision modes; mixed precision uses FP16 A and B matrices with an FP32 accumulation matrix.Source: D. Tensor Core · p. 3

III. DEMYSTIFYING NVIDIA’S TENSOR CORES

  • The low-level analysis examines fragment decoding, tensor-core data access, and timing, extending Jia et al.’s mixed-precision, column-major operand-mapping analysis.The paper also describes threadgroups within a warp as part of its analysis.Source: §III · pp. 3–4, Figure 4 · p. 4, Figure 5 · p. 4

A. Microbenchmarks

The paper uses microbenchmarks to examine how tensor-core data moves and how long operations take, including how fragments map to threads and instructions.

A. Microbenchmarks

  • Two microbenchmark types examine data movement into and out of tensor cores and the time tensor cores take to perform operations.Source: A. Microbenchmarks · p. 4

1) Fragment to thread mapping:

  • A GEMM-kernel microbenchmark initializes matrix elements with distinct values and prints each thread’s loaded segment to reveal the fragment-to-thread mapping.The described code operates on 16 × 16 matrices.Source: 1) Fragment to thread mapping · p. 4

2) Analyzing machine instructions:

  • The analysis maps wmma.mma PTX instructions to multiple HMMA SASS instructions and uses clock-register reads around HMMA instructions to measure timing.The timing microbenchmark places reads before the first and after the nth HMMA instruction; radare2 replaces other HMMA operations with NOPs for the data-access analysis.Source: 2) Analyzing machine instructions · p. 4

B. Operand matrix element mapping

1) Volta Tensor Cores:

The analysis maps Volta operand elements to warp threads and registers, then finds that Turing uses a simpler mapping with each operand element loaded once. At the machine-ISA level, tensor-core loads and stores use ordinary SASS memory instructions, while matrix multiply-accumulate uses HMMA instructions.

1) Volta Tensor Cores:

  • On Volta, each operand-matrix segment is loaded by two threadgroups, so every A and B element is loaded by two warp threads.A segment is a 4 × 16 sub-matrix; for example, threadgroups 0 and 2 load the first four rows of A.Source: 1) Volta Tensor Cores · p. 4
  • Volta’s A-row-major mapping matches B-column-major, while C uses 8 × 4 segments and a distribution dependent on precision rather than layout.A row-major loads use two coalesced 128-bit instructions; column-major loads use four coalesced 64-bit instructions.Source: 1) Volta Tensor Cores · p. 4

2) Turing Tensor Cores:

  • Turing supports 1-, 4-, and 8-bit modes and multiple tile sizes, and each operand element is loaded only once.The 1-bit mode was not analyzed because it did not appear to work on the authors’ system.Source: 2) Turing Tensor Cores · pp. 4–5

C. Machine ISA interface

  • On Volta, wmma.load and wmma.store expand into ordinary SASS load and store instructions, with A/B load widths depending on layout.A/B loads use either four 64-bit or two 128-bit instructions; the paper infers that tensor cores access fragments from the normal register file.Source: C. Machine ISA interface · p. 5
  • Volta implements wmma.mma with HMMA instructions, whose operands are encoded as register pairs using the higher register identifier.The authors infer the register pairing by comparing HMMA operands with the registers used by loads and stores.Source: C. Machine ISA interface · p. 5

1) Volta Tensor Cores:

Volta and Turing lower WMMA matrix multiply-accumulate operations into different numbers and groupings of HMMA instructions. The reported latencies vary by precision mode and architecture.

1) Volta Tensor Cores:

  • On Volta, one mixed-precision wmma.mma expands to 16 HMMA instructions arranged as four sets of four steps; FP16 uses four sets of two steps.The mixed-precision latency is ten cycles lower than FP16 on Volta.Source: 1) Volta Tensor Cores · pp. 5–6

2) Turing Tensor Cores:

  • On Turing, wmma.mma expands to four HMMA instructions in most modes, but to one HMMA instruction in 4-bit mode.The reported Turing latency is highest in 4-bit mode and fastest in 8-bit mode.Source: 2) Turing Tensor Cores · p. 6
  • 99 cycles on Turing versus 54 cycles on Volta for mixed-precision wmma.mma with a 16 × 16 × 16 tile.The Turing value is reported in Table I and the Volta value in Figure 9a.Source: 2) Turing Tensor Cores · p. 6, Table I · p. 6

D. HMMA Instruction Analysis

The analysis breaks Volta HMMA execution into sets and steps, describing how threadgroups multiply operand subtiles and accumulate results into output subtiles. The operation’s subtile sizes and step counts differ by precision mode.

1) Volta:

  • Each Volta HMMA set performs a threadgroup-level product of operand subtiles and accumulates it into a C subtile.The analysis examines the work performed within a set and within its individual steps.Source: 1) Volta · pp. 6–7
  • The mixed-precision example organizes threadgroup work across four HMMA sets.The figure labels the panels as elements accessed in each set and in each mixed-precision step.Source: 1) Volta · p. 7
  • The accompanying figure also illustrates HMMA instruction analysis for Volta’s Titan V.Its caption identifies the figure as HMMA instruction analysis.Source: Figure 10 · p. 7
  • In mixed-precision mode, a set multiplies a 4 × 4 A subtile by a 4 × 8 B subtile and accumulates into a 4 × 8 C subtile.For the first set, threadgroup 0 uses the first four rows of A and the first four rows and eight columns of B.Source: 1) Volta · p. 7
  • A mixed-precision Volta set contains four steps, each operating on a 2 × 4 A subtile.The passage introduces the per-step A subtile size but does not specify the remainder of that step’s operation.Source: 1) Volta · p. 7
  • In FP16 mode, each Volta set contains two steps, each multiplying 4 × 4 A and B subtiles and accumulating into C.The described step operation applies to every threadgroup.Source: 1) Volta · p. 7

2) Turing:

Turing HMMA instructions lack Volta’s explicit STEP annotation, and the authors suggest that similar sequencing may instead be handled by the microarchitecture. The subtile products vary with precision mode.

2) Turing:

  • Turing HMMA instructions lack Volta’s STEP annotation, and the authors suggest similar steps may be sequenced by a microarchitectural state machine.The latency results do not suggest increased parallelism.Source: 2) Turing · p. 7
  • For FP16 and mixed precision, Turing multiplies an 8 × 8 subtile by a 16 × 8 or 8 × 16 subtile, depending on tile configuration.The passage gives 16 × 16 × 16 and 32 × 8 × 16 examples using 16 × 8 A by 8 × 8 B, and an 8 × 32 × 16 example using 8 × 8 A by 8 × 16 B.Source: 2) Turing · p. 7
  • For 8-bit mode, Turing multiplies an 8 × 16 A subtile by a 16 × 8 B subtile.The paper states that 4-bit wmma.mma uses a single HMMA instruction and is omitted from the figure.Source: 2) Turing · p. 7

E. Discussion

Volta organizes threadgroups into octets that can compute independently, even though their operand subtiles may be loaded by different threadgroups. Across sets and steps, octets form outer products of input subtiles to build partial results.

E. Discussion

  • Each octet pairs threadgroup X with threadgroup X+4; four octets in a warp each read an 8 × 16 A subtile, a 16 × 8 B subtile, and an 8 × 8 C subtile.Operand elements are loaded twice by different threadgroups, allowing octets to work independently.Source: E. Discussion · pp. 7, 9, Table II · p. 9
  • Across sets and steps, octets compute outer products of 4 × 4 input subtiles, producing partial-result subtiles.A threadgroup may need a B subtile loaded by its paired threadgroup, so threadgroups cannot work independently even though octets can.Source: E. Discussion · p. 9, Table III · p. 9

IV. A TENSOR CORE MICROARCHITECTURE

The proposed Volta tensor-core microarchitecture uses two tensor cores per sub-core to meet the inferred throughput demand of a warp. Its buffers and FEDP pipeline support one 4 × 4 matrix multiplication per tensor core each cycle.

IV. A TENSOR CORE MICROARCHITECTURE

  • 32 FEDPs per warp per cycle against 16 FEDPs per tensor core implies two tensor cores per sub-core for full throughput; a microbenchmark supports this estimate.The Titan V SM has eight tensor cores, while the microbenchmark indicates only four warps concurrently execute on one SM, suggesting each warp uses two tensor cores.Source: §IV · p. 9
  • The authors suggest tensor-core operations may not co-issue with INT and FP32 arithmetic because they may use those cores’ register-file access ports.The proposed explanation is tied to register access sharing, rather than a demonstrated hardware cause.Source: §IV · p. 9
  • The proposed design assigns 16 SIMD lanes to each tensor core, with eight per octet and four per threadgroup, and buffers operands before computation.A and C operands use separate threadgroup buffers, while B operands use a shared buffer.Source: §IV · p. 9, Figure 13 · p. 10
  • Sixteen FP16 FEDP units use a four-stage multiply-and-accumulate pipeline, allowing each tensor core to complete one 4 × 4 matrix multiplication per cycle.Multiplication is parallel in the first stage, followed by three accumulation stages.Source: §IV · p. 10

V. MODELING AND EVALUATION

The authors modeled Volta tensor-core operations in GPGPU-Sim and extended the simulator to run CUTLASS. The simulator’s CUTLASS-kernel IPC measurements correlate 99.60% with a Titan V.

A. Modelling Tensor Cores

  • GPGPU-Sim models wmma.load, wmma.mma, and wmma.store at the PTX level, with functional and timing models for tensor-core operations.The simulator was extended with half-precision support; the wmma.mma model supports all 32 Titan V configurations and is integrated with operand collection and hazard checking.Source: A. Modelling Tensor Cores · p. 10
  • The wmma.load and wmma.store functional models support all operand-layout combinations, and their timing model matches Titan V’s number of coalesced memory transactions.The functional model follows the operand-element-to-thread mapping described in the paper.Source: A. Modelling Tensor Cores · p. 10
  • For WMMA matrix-multiply kernels, GPGPU-Sim’s cycle counts track Titan V with a standard deviation below 5% as matrix size varies.This agreement is reported despite implementation of the model at the PTX level.Source: A. Modelling Tensor Cores · p. 10, Figure 14 · p. 11

B. CUTLASS

  • 99.60% IPC correlation was measured between GPGPU-Sim and a Titan V for a tensor-core-enabled CUTLASS kernel.The comparison is reported for CUTLASS-developed workloads; Figure 14 presents simulated-versus-hardware performance comparisons.Source: Figure 14 · p. 11, B. CUTLASS · p. 12
  • The CUTLASS extension added missing API calls and PTX instruction definitions, and the modified simulator ran around 680 library test cases.CUTLASS is an open-source CUDA C++ template library for efficient linear algebra and fused matrix-multiply kernels.Source: B. CUTLASS · p. 12

C. Profiling Tensor Cores

On a Titan V, tensor cores provide about 3–6× the performance of an SGEMM kernel and about 3× that of an HGEMM kernel. The paper also characterizes their throughput and instruction latencies, relates its findings to prior work, and concludes that its model can support further microarchitectural investigation.

C. Profiling Tensor Cores

  • 109.6 TFLOPs in FP16 and 108.7 TFLOPs in mixed precision were measured for a repeated-wmma.mma kernel; an 8192 × 8192 FP16 GEMM reached around 96 TFLOPs.The repeated-operation kernel had computational intensity on the order of 10^8; NVIDIA documentation gives a 125 TFLOPs theoretical peak.Source: C. Profiling Tensor Cores · p. 12
  • For a 1024 × 1024 shared-memory WMMA kernel, minimum latencies were 125 cycles for wmma.load, 120 for wmma.store, and 70 for wmma.mma.The profiled latency distributions also showed occasional high latencies, possibly from warp scheduling and memory traffic.Source: C. Profiling Tensor Cores · p. 12
  • Shared memory reduces median wmma.load latency by more than 100× on larger matrices.The paper plots median instruction latency versus matrix size, with a logarithmic axis for wmma.load.Source: C. Profiling Tensor Cores · p. 12
  • Tensor cores provide about 3–6× the performance of an SGEMM kernel and about 3× that of an HGEMM kernel.The cuBLAS GEMM kernel gains more than the WMMA GEMM implementation; the paper attributes this to cuBLAS optimizations including avoiding shared-memory bank conflicts and software pipelining.Source: C. Profiling Tensor Cores · p. 12

VI. RELATED WORK

  • The analysis extends prior Volta work by examining both operating modes, identifying octets, and finding that Turing’s operand-to-thread mapping differs from Volta’s.The paper also presents a methodology for uncovering these details using microbenchmarks.Source: §VI · p. 12

VII. CONCLUSION

  • The authors conclude that their Volta model agrees well with hardware and that the work may serve as a starting point for further microarchitectural investigation of machine-learning workloads.They also report enabling CUTLASS on GPGPU-Sim.Source: §VII · pp. 12–13
Loading 1811.08309v2…