PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting GPU Memory Hierarchy through Microbenchmarking

Xinxin Mei, Xiaowen Chu

arXiv:1509.02308v2cs.ARcs.DC

TL;DR

GPU memory-hierarchy details are poorly disclosed even though memory efficiency is central to GPU performance. The paper develops fine-grained microbenchmarks for Fermi, Kepler, and Maxwell, revealing cache behavior and reporting Maxwell’s advantage under shared-memory bank conflict.

  • Problem

    GPU vendors disclose limited memory-hierarchy details, leaving cache structures, throughput, and latency insufficiently characterized for memory-efficiency analysis.

  • Method

    The paper applies reproducible fine-grained microbenchmarks, including pointer chasing, to caches, global/shared-memory throughput and latency, and bank-conflict behavior across three NVIDIA generations.

  • Results

    Maxwell is highly optimized to avoid long latency under shared-memory bank conflict, alongside newly characterized TLB, texture-cache, and data-cache properties.

  • Takeaways & Limitations

    The study provides comprehensive, up-to-date information for understanding and comparing recent NVIDIA GPU memory hierarchies.

  • Takeaways & Limitations

    The fine-grained P-chase requires more memory than available hardware to store one round of the smallest Fermi L2-cache accesses.

Abstract

from arXiv · show

Memory access efficiency is a key factor in fully utilizing the computational power of graphics processing units (GPUs). However, many details of the GPU memory hierarchy are not released by GPU vendors. In this paper, we propose a novel fine-grained microbenchmarking approach and apply it to three generations of NVIDIA GPUs, namely Fermi, Kepler and Maxwell, to expose the previously unknown characteristics of their memory hierarchies. Specifically, we investigate the structures of different GPU cache systems, such as the data cache, the texture cache and the translation look-aside buffer (TLB). We also investigate the throughput and access latency of GPU global memory and shared memory. Our microbenchmark results offer a better understanding of the mysterious GPU memory hierarchy, which will facilitate the software optimization and modelling of GPU architectures. To the best of our knowledge, this is the first study to reveal the cache properties of Kepler and Maxwell GPUs, and the superiority of Maxwell in shared memory performance under bank conflict.

1 INTRODUCTION

GPU memory bottlenecks constrain the utilization of otherwise substantial computational capability, while vendor secrecy leaves many hierarchy details unknown. The paper addresses this gap with reproducible fine-grained microbenchmarks across Fermi, Kepler, and Maxwell GPUs.

  • Motivation: NVIDIA’s Fermi, Kepler, and Maxwell generations improved memory efficiency over earlier hardware, but memory bottlenecks remained a primary limitation.The paper notes that NVIDIA provides very limited information about these memory systems, leaving many details unknown publicly.
  • Approach: The study applies cache, throughput, and latency microbenchmarks to three recent NVIDIA GPU generations.Its results are based on many experimental rounds and are described as reproducible.
  • Contributions: The fine-grained P-chase benchmark reveals GPU cache characteristics that differ from those of CPUs.The study uses cross-generation comparisons to illuminate the evolution of GPU memory designs.
  • Contributions: The paper reports unequal L2 TLB sets, 2D spatial-locality mapping in the texture L1 cache, and a non-traditional L1 data-cache replacement policy.It also benchmarks global and shared-memory throughput and access latency, including shared-memory bank-conflict effects.
  • Contributions: Maxwell is reported to be highly optimized to avoid long latency under shared-memory bank conflict.The paper presents this as the first verification of that property in its study.

2 RELATED WORK

Prior GPU memory studies used characterization, optimization algorithms, and microbenchmarks to investigate memory structures, throughput, latency, and access efficiency. This paper extends that line of work with a fine-grained GPU-oriented approach and broader coverage of recent architectures.

  • Prior characterization and optimization: Existing studies linked limited application performance to GPU memory bottlenecks and localized causes of low memory throughput to particular memory spaces.Other work proposed data-mapping and memory-management algorithms to improve memory-access efficiency.
  • Microbenchmark methods: P-chase originated for CPU hardware, while later variants explored multicore cache coherence and early GPU memory hierarchies.The GPU’s different multithreading model prevents directly applying the multithreaded CPU method.
  • Microbenchmark methods: Prior GPU studies used P-chase and footprint experiments to examine global-memory TLBs and other GPU caches.Fermi work found non-LRU L1 data-cache replacement and non-integer L2 associativity.
  • This study’s extension: The present study’s fine-grained P-chase microbenchmarks obtain L1 cache replacement-policy information beyond earlier measurements.The authors state that their experimental results coincide with prior Fermi findings on non-LRU replacement and non-integer L2 associativity.
  • This study’s extension: Earlier throughput work emphasized bandwidth, whereas this study focuses on factors affecting effective throughput and also includes memory-access latency.The comparison covers global and shared memory.

3 OVERVIEW OF GPU MEMORY HIERARCHY

The study focuses on global, shared, and texture memory and examines their caches, throughput, latency, and shared-memory bank conflicts. Recent NVIDIA GPUs use cached global memory, multi-level caching, TLBs, and generation-specific shared-memory organizations.

  • Scope: The study limits its scope to global, shared, and texture memory among CUDA’s six memory spaces.It examines cache mechanisms, global/shared-memory throughput and latency, and shared-memory bank-conflict effects.
  • Cache organization: Cached global and texture memory use two cache levels, with per-SM L1 caches and an off-chip L2 cache shared across SMs.The L2 cache is unified for instruction, data, and page-table access.
  • Address translation: The TLB caches page-table entries, and a TLB miss accesses global memory to search the page table, causing significant access latency.The page table is usually stored in global memory.
  • Memory-space properties: Global memory is read-and-write, texture memory is read-only, and both are accessible to all threads in a kernel function.Shared memory is read-write and located within the SM.
  • Generation differences: Modern GPU global-memory access is cached, unlike the uncached global-memory access of the older Tesla device.On Maxwell, the L1 data, texture on-chip, and read-only data caches are combined in one physical space.
  • Generation differences: Fermi and Kepler share 64 KB of memory space between shared memory and the L1 data cache, whereas Maxwell provides 96 KB of dedicated shared memory.Modern GPUs also have larger shared-memory spaces and more banks than the Tesla device described for comparison.

4 CACHE STRUCTURES

The paper replaces conventional GPU-cache assumptions with a fine-grained P-chase that records individual access latencies, revealing cache organizations and policies that often differ from CPU designs.

  • Why typical P-chase fails: Traditional P-chase assumptions and average latency are ineffective for GPU caches because irregular mapping and non-LRU replacement can produce variable, non-periodic access patterns.The measured average latency can vary for a fixed (N, s) pair when replacement behavior is nontraditional.
  • Microbenchmark design: Fine-grained P-chase records every single data-access latency using shared memory, preserving detailed access-process information without affecting the data cache.The benchmark uses a single thread and single CTA; shared memory stores the latency sequence.
  • Microbenchmark design: The two-stage procedure estimates cache size and line size, tests replacement behavior, and derives associativity by progressively overflowing the cache.Cache size is found by increasing N until the first miss; line size is identified by a sudden miss increase at N = C + b + 1.
  • Texture and read-only caches: Fermi and Kepler texture L1 caches contain 4 sets of 96 lines with 32-byte lines, while 128 successive bytes map across 4 successive sets for 2D locality.The set selection uses address bits 7-8 rather than the traditional bits 5-6.
  • Texture and read-only caches: The GTX780 read-only cache matches the texture L1 structure with 4 sets, 32-byte lines, and 96 lines per set; GTX980 differs in having rather random data mapping.On GTX780, arrays larger than 12.5 KB produce a miss on each access under the tested configuration.
  • Translation look-aside buffer: Recent GPUs share a 16-way fully associative L1 TLB with 2 MB pages, while the inferred L2 TLB has one 17-entry set and six 8-entry sets using LRU replacement.The piecewise-linear miss-rate increase indicates unequal L2 TLB sets.
  • L1 data cache: Fermi’s 16 KB L1 data cache has 128 lines across four cache ways and uses a non-LRU replacement policy that yields non-periodic access behavior.Cache way 2 is three times more likely to be replaced than the other three ways in the reported experiment.
  • L1 data cache: For sequential loading, Fermi’s non-LRU policy reduces cache misses relative to conventional LRU behavior in the demonstrated access pattern.The listed accesses would all be misses under the LRU policy, but some become hits under the measured policy.

5 GLOBAL MEMORY

The paper benchmarks global-memory throughput and latency across Fermi, Kepler, and Maxwell using controlled microbenchmarks. Throughput depends on available memory requests and ILP, while latency varies substantially by cache and page-table access pattern.

  • Global Memory Throughput: Global-memory throughput converges as CTA count, CTA size, and ILP increase, because active-warps availability limits bandwidth utilization.Larger ILP saturates throughput faster across all three devices.
  • Global Memory Throughput: GTX780 achieves the highest global-memory throughput but converges most slowly, requiring the most memory requests to hide pipeline latency.The result is associated with GTX780’s highest bus width.
  • Global Memory Latency: The fine-grained P-chase constructs distinct global-memory patterns, including TLB/page-table misses, L1-TLB hits with cache misses, and cache hits with TLB misses.The experiment records latency across six access patterns while varying cache state and stride.
  • Global Memory Latency: Maxwell and Kepler exhibit a P6 page-table context-switching pattern, while GTX980 has especially long cold-cache-miss and page-table-switching latencies.Except for these rare patterns, GTX980’s latency is close to Kepler’s.
  • Global Memory Latency: GTX780 generally has the shortest global-memory latencies for access patterns P2-P5, nearly half those of Fermi.GTX980 resembles GTX780 for P1-P4 but has substantially longer cold-cache-miss latency.

6 SHARED MEMORY

The study measures shared-memory throughput and latency under varying occupancy, ILP, and bank-conflict conditions. Maxwell achieves the strongest overall shared-memory performance and substantially reduces bank-conflict penalties.

  • 6.1 Shared Memory Throughput: Shared-memory throughput depends mainly on active warps per SM and ILP, increasing with occupancy until reaching a threshold.The benchmark varies CTA size, CTAs per SM, and ILP across the three GPU platforms.
  • 6.1 Shared Memory Throughput: 137.41 GB/s is GTX980’s peak shared-memory throughput, about 83.9% of theoretical bandwidth, compared with 34.90 GB/s for GTX560Ti and 83.81 GB/s for GTX780.GTX980 peaks at 16 active warps; GTX560Ti and GTX780 peak at 16 and 32 active warps, respectively.
  • 6.1 Shared Memory Throughput: Kepler’s throughput is constrained because its latency requires about 94 active warps at ILP = 1, while the device permits at most 64 concurrent warps.The paper identifies this gap as the main reason GTX780 underperforms its designed throughput.
  • 6.2 Shared Memory Latency: Kepler’s 8-byte mode avoids the 2-way conflict at stride 6 that occurs in its 4-byte mode, because 32 threads map to 32 distinct banks.At stride 2, neither Kepler mode conflicts; at stride 4, both modes show a 2-way conflict.
  • 6.2 Shared Memory Latency: Shared-memory latency increases almost linearly with potential bank conflicts, but Maxwell’s longest latency remains at the L1 data-cache latency level.Fermi and Kepler can become slower than regular global-memory access under 32-way conflicts.
  • 6.2 Shared Memory Latency: Maxwell delivers the best shared-memory performance, with 2x size, 2x memory access speedup, and optimization against long bank-conflict latency.The paper connects this improvement to faster and more efficient GPU computations.

7 CONCLUSIONS

Across Fermi, Kepler, and Maxwell, the study observes an evolution toward larger and more optimized GPU memory hierarchies. Maxwell retains good performance while being more economical.

  • 7 CONCLUSIONS: Kepler and Maxwell significantly enhance memory capacity compared with Fermi, while Kepler emphasizes performance-oriented design elements.The conclusion frames these changes as part of an evolution in NVIDIA GPU memory hierarchies.
  • 7 CONCLUSIONS: Maxwell’s optimized memory hierarchy retains good performance while being more economical.
Loading 1509.02308v2…