Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking

Related work

• Prior independent research used microbenchmark-based dissection to expose NVIDIA instruction encoding and memory-hierarchy geometry and features.These efforts address the dearth of public microarchitectural-level information on novel NVIDIA GPUs.Source: Summary · p. 5

Contribution

• The report presents NVIDIA Volta’s microarchitectural details, discovered through microbenchmarks and instruction-set disassembly.This continues prior microbenchmark-based research on NVIDIA architectures, including Kepler, Maxwell, and Pascal.Source: Summary · p. 5
• It quantitatively compares the Volta findings against the predecessor architectures Kepler, Maxwell, and Pascal.The work responds to rapid NVIDIA architectural progression and manufacturers’ reluctance to disclose low-level details.Source: Summary · p. 5

Why these details matter

• Bare-metal peak performance remains inaccessible to plain CUDA and generally to inline PTX without deeper hardware understanding, unlike NVIDIA libraries such as cuBlas and cuDNN.The authors suggest NVIDIA’s optimized libraries may use unpublished low-level SASS code and unavailable assemblers.Source: Why these details matter · p. 7
• Prior studies [13] found efficiency gaps in NVCC machine code and showed that instruction encoding and microarchitectural knowledge are needed to realize full potential.These findings concern architectures preceding Volta.Source: Why these details matter · p. 7
• For Volta, NVCC-emitted code still contains performance-improvement opportunities exploitable through a minimal example.This extends prior research on NVIDIA GPU architectures.Source: Why these details matter · p. 7
• The described optimizations require substantial human effort and may be worthwhile mainly for tight computational kernels that dominate an application’s execution time.The authors identify this effort-to-gain tradeoff as a limitation of their approach.Source: Why these details matter · p. 7

• Volta-specific optimizations require repeating the discovery work for each future GPU architecture.Source: §6 · p. 8
• CUDA libraries and NVCC generally provide portability and efficiency, with new-generation performance gains requiring little or no extra development effort.Source: §6 · p. 8
• The motivating kernel multiplies 8-value slices from matrices A and B and accumulates products into an 8×8 register-mapped tile.Source: §6 · p. 8
• On Volta, two 64-bit register banks limit each instruction to 64 bits per bank per cycle, so FFMA instructions sourcing three values from one bank are serialized by conflicts.Source: §6 · p. 8
• Binary-level rewriting uses a better register mapping with no conflicts while leveraging register-reuse caches, compared with the NVCC 9.0 mapping shown for the motivational example.The changes are visible in Figure 1.Source: §6 · p. 9, Figure 1.1 · p. 9
• With 128 threads, the binary-level changes improve performance, although the supplied passage does not provide the resulting values.Source: §6 · p. 9

• 132.05 to 152.43 GFlops/s per SMX (+15.4%) after improving register mapping and register-cache reuse to avoid bank conflicts.Table 1.1 compares NVCC-generated machine code with the improved code.Source: §8 · p. 10, Table 1.1 · p. 11
• On Volta, a third input register from the other bank can avoid conflicts even when row and column input registers share a bank.Maxwell and Pascal use 32-bit-wide banks that allow only one value per bank per clock, preventing register mapping from eliminating such conflicts.Source: §8 · p. 10
• Discovering instruction encoding and studying bank-conflict behavior are primary contributions because they enable machine-code patching that eliminates conflicts.Source: §8 · p. 10
• The improved machine code combines register mapping that avoids bank conflicts with maximal reuse of register caches.Source: Table 1.1 · p. 11

Instructions

• Volta encodes each instruction and its corresponding control information in one 128-bit word, unlike Pascal and Maxwell’s separate 64-bit words.The Volta format represents a substantial departure from the prior architectures’ encoding scheme.Source: Instructions · p. 13
• In nvdisasm’s display, the first 64-bit word contains only instruction information, while the second contains both instruction and control information.The 128-bit encoding is shown as two 64-bit words.Source: Instructions · p. 13
• At least 91 bits encode the instruction, at least 23 bits encode control information, and the remaining 14 bits appeared unused in experiments.These allocations were inferred from exhaustive instruction disassembly.Source: Instructions · p. 13

2.1 Control information

• Volta stores one instruction and its associated control information in a single 128-bit word, whereas pre-Volta bundles interleave control words with three instructions on Pascal/Maxwell or seven on Kepler.Volta places control information before and after the instruction encoding within the 128-bit word.Source: §2.1 · p. 14
• Kepler, Pascal, and Maxwell encode control information in separate bundled control words, with Kepler using seven 8-bit sections and Pascal/Maxwell using three 21-bit sections.These control words determine scheduling of the instructions within each bundle.Source: §2.1 · p. 14
• Volta, Pascal, and Maxwell organize control sections into six fields covering reuse, wait and read/write barriers, yield, and stall cycles.Volta control sections contain one 21-bit section preceded by two zero bits, with instruction encoding surrounding the control information.Source: §2.1 · p. 14

2.1. CONTROL INFORMATION

• Reuse flags select four register reuse caches corresponding to four source-operand slots, mitigating register bank conflicts for subsequent instructions.The least significant reuse bit controls the first source slot, and the most significant bit controls the fourth.Source: §2.1 · p. 15
• Dependency barriers track variable-latency memory and shared-resource instructions, stalling consumers until earlier results become available.Volta, Pascal, and Maxwell provide six barriers; wait masks can represent dependencies on multiple barriers, while read barriers protect against write-after-read hazards.Source: §2.1 · p. 15
• Volta’s 4-bit stall field specifies a delay of 0 to 15 cycles before issuing the next instruction.Unlike Pascal and Maxwell, Volta has one dispatcher per processing block, and generated code shows no dual issue.Source: §2.1 · p. 15
• The yield flag balances warp workload by favoring the current warp when set and switching to another warp when cleared.Switching warps costs one extra cycle and makes the next instruction’s register-reuse flags ineffective.Source: §2.1 · p. 15

2.2 Scheduler

• Volta partitions each streaming multiprocessor into four processing blocks, and each warp can access only the units within its assigned block.Instructions from the same warp are allocated to a specific processing block.Source: §2.2 · p. 16
• When FFMA instructions are processed on a single processing block, measured throughput is lower than when warps use different blocks.The benchmark varies the two active warp indices while running FFMA sequences on one Volta SM; Table 2.1 reports throughput in GFLOPS.Source: §2.2 · p. 16, Table 2.1 · p. 16
• Volta maps warps to schedulers using scheduler id = warp id%4.This mapping is tested with two active warps issuing FFMA instructions while six of eight invoked warps remain idle.Source: §2.2 · p. 16
• At least 128 threads are required to fully use Volta’s processing units.Source: §2.2 · p. 16

2.3 Instruction encoding

• Volta uses more instruction-encoding bits than previous architectures, with opcodes ranging from 10 to 13 bits.Source: §2.3 · p. 16
• Unlike Pascal, Maxwell, and Kepler, Volta places the opcode in the least significant bits of the first 64-bit instruction-bundle part.Previous architectures organize the opcode in the most significant instruction bits.Source: §2.3 · p. 16

2.3. INSTRUCTION ENCODING

• Volta operands can be registers, constant, shared, or global memory addresses, or immediate values.Predication uses 4 bits: one negation flag and three bits encoding a predicate-register index.Source: §2.3 · pp. 16–17

3.1.1 Latency and bandwidth

• 28-cycle L1 hit latency on Volta versus 84 cycles on Pascal, while V100 L1 bandwidth reaches 109.1 bytes per cycle per SM versus 31.3 on P100.These measurements support the improved latency and bandwidth of Volta’s combined L1 data-cache and shared-memory subsystem.Source: §3.1.1 · p. 22, §3.1 · p. 23, Table 3.2 · p. 23

3.1.2 Geometry

• Volta’s L1 data-cache measurements fall 7 KiB short of its nominal configured size, a discrepancy the authors cannot explain and do not attribute to ECC.The authors conjecture that a newly applied replacement policy may cause the discrepancy.Source: §3.1.2 · p. 24, Table 3.3 · p. 24

3.2. L2 CACHE

• When saturated, Volta’s L1 replacement policy evicts four low-priority cache lines from different sets before preserving higher-priority lines, better protecting large arrays than LRU.The findings support the claim that Volta narrows the gap between explicitly managed shared memory and direct device-memory access.Source: §3.2 · p. 25

3.2 L2 cache

• Volta’s unified L2 cache is 16-way set-associative, has 6,144 KiB capacity and 64 B lines, and averages 193 clock cycles of latency.The L2 cache serves data, instructions, and constant memory, as in previous GPU generations.Source: Memory hierarchy · p. 19, Figure 3.2 · p. 22, §3.2 · p. 25

3.3 Instruction caches

• Volta’s three instruction-cache levels provide better inverse throughput than predecessors at the second and third levels, while aggregate L0 throughput remains unchanged despite compiler-inserted stalls.Volta’s doubled scheduler count offsets the 2-cycle per-scheduler stalls introduced by NVCC’s new partitioning method.Source: Figure 3.3 · p. 27, §3.3 · p. 28

3.3. INSTRUCTION CACHES

• Volta’s L0 instruction cache is private to each scheduler, L1 is private to each SM, and unified L2 is shared across all SMs; Volta lacks Pascal, Maxwell, and Kepler’s L1.5.The measured L0 and L1 caches are respectively 3-way associative with 16 sets and 256 B lines, and 8-way associative with 32 sets and 512 B lines.Source: §3.3 · p. 28, Figure 3.5 · p. 30

3.4 Constant memory and cache

• Volta has three constant-cache levels: two private SMX levels and a unified L2 shared across SMXs for instruction and data.Its L1 constant cache uses a non-LRU replacement policy, and its L1.5 constant cache coincides with the L1 instruction cache.Source: §3.4 · pp. 30–31

3.4. CONSTANT MEMORY AND CACHE

• Volta constant-memory accesses broadcast one address to all warp threads simultaneously, whereas divergent addresses are serialized.The constant-memory hierarchy did not change significantly from previous generations.Source: §3.4 · pp. 30–31
• Constant-memory load latency depends on whether data is found in L1, L1.5, or L2 and on the number of distinct locations referenced.Hardware broadcasts accesses targeting the same location within a warp.Source: Figure 3.7 · p. 32

3.5 Register banks

• Volta’s register file has two 64-bit banks, unlike the four 32-bit banks of previous architectures, and a register maps to bank index modulo 2.The wider banks make conflict-free code easier for the compiler to generate.Source: §3.5 · p. 32
• Register-bank conflicts occur when all three 32-bit FFMA source registers reside in the same 64-bit bank.In the benchmark, R97 and R99 occupy bank 1, so RX causes a conflict when it also maps to bank 1.Source: §3.5 · p. 32, Figure 3.8 · p. 33

3.6. SHARED MEMORY

• Volta shared memory has the lowest measured latency among the examined GPUs, while latency generally increases with warp bank conflicts.Kepler avoids degradation for two-thread conflicts but degrades when more than two threads access the same bank.Source: §3.6 · p. 33, Figure 3.9 · p. 34
• Volta and Pascal show significantly increased theoretical and measured shared-memory bandwidth, driven by their greater numbers of streaming multiprocessors.Bandwidth was measured with a high-pressure benchmark in which threads load from and store to shared memory around a simple calculation.Source: §3.6 · p. 33, Figure 3.10 · p. 35

3.7 Global memory

• Volta achieves higher actual and theoretical global-memory bandwidth than Pascal, with an actual-to-theoretical ratio of 83.3% versus 69.6%.Volta and Pascal use HBM2 memory, which provides higher bandwidth than GDDR5-based GPUs.Source: §3.7 · p. 34, Figure 3.11 · p. 35

3.8 TLBs

• Volta’s L1 data cache is virtually indexed, whereas its L2 data cache is physically indexed and therefore accesses L2 involve the TLBs.Scanning arrays beyond L1 TLB coverage produced no TLB misses when accesses were served by L1, but L2 accesses used the TLBs with L1 disabled.Source: §3.8 · p. 36
• Volta has two TLB levels: 2 MiB L1 entries with 32 MiB coverage and an L2 TLB covering about 8192 MiB.The L2 TLB coverage is larger than on previous architectures.Source: §3.8 · p. 36, Figure 3.12 · p. 37

4.1 Native instructions

• Volta generally improves instruction latency over Pascal: most integer and single-precision instructions take 4 cycles, double-precision instructions 8 cycles, and half-precision instructions 6 cycles.The dependent issue latency of core FMA operations was reduced to 4 clock cycles on Volta.Source: §4.1 · pp. 39–40, Table 4.1 · p. 40

4.2 Atomic operations

• Shared-memory atomics have lower latency on Volta than on Pascal and Maxwell, but global-memory atomic latency shows no obvious improvement.The shared-memory improvement is attributed to Volta’s improved shared-memory latency.Source: §4.2 · p. 40, Table 4.2 · p. 41

4.3 Tensor cores

• Volta tensor-core execution maps a 16 × 16 × 16 half-precision multiplication across four HMMA instruction sets and eight thread groups, each computing 32 output elements.Each HMMA set contains four STEP0–STEP3 instructions, while register values are shared within and among thread groups.Source: §4.3 · pp. 43, 46

4.4 Floating-point performance

• 90.2% of theoretical throughput is achieved for half-precision cuBLAS matrix multiplication on V100 using tensor cores, while single- and double-precision benchmarks approach peak performance.Measurements used cuBLAS on a V100 running at 1,380 MHz.Source: §4.4 · p. 50, Figure 4.8 · p. 51

Volta boards with NVIDIA NVLink support

• V100 boards were available as SXM2 models supporting NVLink 2.0 or PCIe models supporting PCIe Gen 3, with prior chapter data referring to PCIe boards.The chapter compares V100 NVLink, V100 PCIe, and P100 NVLink GPUs at their respective highest peak graphics clock frequencies.Source: Volta boards with NVIDIA NVLink support · p. 53

5.1 Peer-to-peer and host communication

• V100 NVLink GPUs deliver significantly higher peer-to-peer bandwidth than P100 NVLink and PCIe boards, while interconnect and generation produce substantially similar latencies.Measurements used peer-to-peer communication for NVLink GPUs connected by two sub-links.Source: Table 5.1 · p. 54, §5.1 · p. 54
• Host-to-device and device-to-host bandwidth measurements are identical when one GPU communicates with its closest CPU through the shared x16 Gen 3 PCIe path.All GPUs were connected to one PCIe switch communicating with one CPU.Source: Table 5.2 · p. 54, §5.1 · p. 54

5.2 Arithmetic throughput and memory bandwidth

• The V100 NVLink board’s 1,530 MHz peak graphics clock, versus 1,380 MHz for V100 PCIe, explains higher floating-point, shared-memory, and multi-block atomic throughput.The NVLink V100 achieved the highest absolute atomic-operation bandwidth among the examined GPUs.Source: §5.2 · p. 54, §5.3 · p. 55, Figure 5.1 · p. 56

5.3. LATENCY

• Global memory bandwidth does not differ between V100 PCIe and NVLink versions, despite NVLink’s higher floating-point and shared-memory performance.Floating-point throughput and bandwidth were measured with cuBLAS 9.0 while varying matrix sizes from 32 to 8192.Source: §5.2 · p. 54, §5.3 · p. 55, Table 5.3 · p. 55

5.3 Latency

• 405 versus 391 clock cycles make global-memory access appear slower on V100 NVLink, but real latency is marginally faster: 26.5 versus 28.3 ns.L1, L2, constant-cache, shared-memory, and atomic-operation latencies are the same because Volta’s microarchitecture is unchanged across board versions.Source: §5.3 · p. 55

Conclusions

• Volta substantially changes its instruction encoding, memory hierarchy, and processing-unit behavior compared with previous architectures.Source: Conclusions · p. 57
• The scheduler-to-core ratio grows from 1:48 on Kepler to 1:16 on Volta, facilitating higher instruction throughput.Source: Conclusions · p. 57
• A newly introduced L0 instruction cache mitigates the penalty of Volta’s larger instruction size.Source: Conclusions · p. 57
• Volta’s improved L1 cache provides lower latency and higher bandwidth, while its replacement policy reduces misses without shared memory and redesigned register banks mitigate conflicts.Source: Conclusions · p. 57
• The findings enable binary-level code optimization and customized Volta assemblers that produce tighter-scheduled, higher-performance code.Source: Conclusions · p. 57
• Disclosed memory-hierarchy information lets developers choose cache-matched working sets to reduce miss rates and improve performance.Source: Conclusions · p. 57

• The appendix lists opcodes for common instructions.Source: Appendix · p. 59
• These opcodes are given as encoded in Volta’s instruction encoding.Source: Appendix · p. 59

Floating point instructions

• The appendix compares Volta’s instruction encoding with Pascal’s encoding and includes floating-point instructions.Source: Appendix · p. 59