PaperlayerView exact revision on arXiv

Source-linked AI summary

SnakeCharmer: Orchestrating Utility, Timeliness, and Placement in GPU Prefetching

Yuhan Tang, Tiejun Li, Jianmin Zhang, An Hu, Guangyu Sun, Sheng Ma, Yiqun Lang, Hanqing Li, Siqing Fu

arXiv:2610.30990v1cs.AR

TL;DR

GPU thread-level parallelism cannot always hide long memory latency, and address prediction alone does not ensure a prefetch becomes beneficial. SnakeCharmer adds backend control over prefetch utility, timeliness, and placement. On an A100 GPU model, it improves performance and energy efficiency over both Snake and APOGEE.

  • Problem

    Address-prediction-based GPU prefetching can identify future addresses, but concurrent resource contention means accuracy alone does not ensure performance improvement.

    Source: §I · p. 1

  • Method

    SnakeCharmer separates address prediction from realization and jointly controls whether, when, and where candidates are issued and retained.

    Source: §I · p. 2

  • Results

    On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 12.8%/20.8% over Snake and 1.7%/3.2% over APOGEE.

    Source: §VII · p. 13

  • Takeaways & Limitations

    The backend-control framework also applies to a different frontend predictor, APOGEE, through lightweight frontend-specific adaptations.

    Source: §I · p. 2

Abstract

from arXiv · show

GPUs rely on massive thread-level parallelism to tolerate long memory latency, yet they still suffer severe stalls when many warps simultaneously wait for data from the memory system. Address-prediction-based GPU prefetchers can mitigate such stalls, but accurate prediction alone does not guarantee performance improvement: a predicted prefetch becomes beneficial only if it is worth issuing, arrives at the right time, and is retained without harmful interference. Because these requirements interact, optimizing any single dimension in isolation can simply move the bottleneck to another. We capture these requirements with a predicted-prefetch-to-realized-benefit perspective and propose SnakeCharmer, a unified backend prefetch-control framework for address-prediction-based GPU prefetchers. SnakeCharmer separates address prediction from prefetch realization: the frontend predictor generates prefetch candidates, while the backend control path jointly controls utility, timeliness, and placement by determining whether each candidate should be issued, when it should be issued, and where the returned data should be retained. SnakeCharmer realizes this control path with zero-value-aware filtering for utility, hierarchical feedback-driven distance tuning for timeliness, and a register-assisted prefetch buffer for placement. We design and implement SnakeCharmer primarily on top of Snake, and adapt the same backend-control principles to APOGEE to demonstrate that SnakeCharmer's benefits are not limited to a single frontend predictor. On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 12.8%/20.8% over Snake and by 1.7%/3.2% over APOGEE, with modest hardware overhead.

I. INTRODUCTION

GPU thread-level parallelism is increasingly insufficient to hide long memory latency, and accurate address prediction alone does not guarantee useful prefetching. SnakeCharmer treats prefetch realization as a joint control problem spanning utility, timeliness, and placement.

I. INTRODUCTION

  • As compute throughput outpaces memory-system latency, many warps can wait simultaneously, leaving the SM without ready warps and degrading performance.Massive thread-level parallelism therefore cannot fully hide long-latency accesses.Source: §I · p. 1
  • Accurate address prediction is insufficient because each candidate must be worth issuing, arrive at the right time, and be retained without displacing demand-fetched data.Utility, timeliness, and placement interact, so optimizing one dimension alone can move the bottleneck to another.Source: §I · pp. 1–2, Figure 1 · p. 2
  • SnakeCharmer separates candidate generation from realization and jointly controls whether, when, and where prefetches are handled.It is designed as a predictor-agnostic backend for address-prediction-based GPU prefetchers.Source: §I · p. 2
  • On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 12.8%/20.8% over Snake and 1.7%/3.2% over APOGEE.The framework's benefits extend beyond the Snake frontend.Source: §I · p. 2
  • Its backend uses zero-value-aware filtering, feedback-driven distance tuning, and register-assisted buffering, and its principles are adapted to APOGEE.The framework is implemented primarily on Snake, with lightweight frontend-specific modifications for APOGEE.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

GPU prefetching can supplement reactive warp scheduling, but address prediction alone does not ensure useful prefetches. Snake measurements expose utility, timeliness, and placement gaps that can prevent predictions from becoming realized benefits.

A. GPU Hardware Prefetching

B. Quantifying the Gap: Utility, Timeliness, and Placement

  • Address prediction alone does not guarantee speedup because prefetches can consume resources, arrive mistimed, or displace demand data from limited cache capacity.The section motivates backend control over whether, when, and where predicted candidates are handled.Source: B. Quantifying the Gap: Utility,… · pp. 3–4
  • At 4B granularity, 36.8% of L2 prefetch traffic and 31.9% of demand traffic is zero-valued, identifying low-utility prefetch traffic in sparse applications.The analysis defines blocks at 32B-sector granularity unless otherwise specified; Fig. 3b reports the 4B-granularity fractions.Source: B. Quantifying the Gap: Utility,… · p. 3
  • Among Snake prefetches eventually consumed by demand loads, 48.3% arrive on time, 28.7% too early, and 23.0% too late.Prefetches never consumed are excluded from this distribution and captured separately by the accuracy metric.Source: Figure 4 · p. 3, B. Quantifying the Gap: Utility,… · p. 3
  • A fixed prefetch distance can produce different time-to-use intervals across applications because scheduling, dependencies, computation, and contention vary.This variability motivates runtime feedback to adjust prefetch distance.Source: B. Quantifying the Gap: Utility,… · p. 3
  • 34.0% of all L1 victim-sector evictions on average are prefetch-induced demand evictions, indicating pressure on the contended L1 cache.The metric counts demand-fetched victim sectors displaced by incoming prefetch sectors.Source: Figure 5 · p. 3, B. Quantifying the Gap: Utility,… · pp. 3–4

C. Motivation for SnakeCharmer

SnakeCharmer targets three interacting sources of ineffective prefetching: low utility, poor timeliness, and harmful placement. Idle register capacity offers a possible destination for prefetched data outside the contended L1 cache.

C. Motivation for SnakeCharmer

  • Sparse applications can repeatedly prefetch zero-valued regions, motivating filtering before candidates consume cache, queue, MSHR, and bandwidth resources.Contiguous zero-valued regions can occur in deep-learning workloads through activation functions such as ReLU.Source: C. Motivation for SnakeCharmer · p. 4
  • Fixed prefetch distances map to varying time-to-use intervals under warp scheduling, memory dependences, intervening computation, and memory-system contention.The proposed response is runtime feedback that adapts distance to dynamic consumption behavior.Source: C. Motivation for SnakeCharmer · p. 4
  • Prefetching into L1 can increase contention and evict useful demand data, while idle registers can buffer prefetched blocks outside that cache.The paper reports at least 8 KB of idle register capacity across evaluated applications, averaging 21.0 KB per kernel under a 256 KB-per-SM assumption.Source: C. Motivation for SnakeCharmer · p. 4, Figure 6 · p. 4

III. SNAKECHARMER DESIGN

SnakeCharmer combines candidate filtering, timeliness feedback, and register-assisted buffering in a backend control path. Its Snake+ frontend tracks stride patterns and adds zero-value confidence to suppress likely low-utility candidates.

III. SNAKECHARMER DESIGN

  • SnakeCharmer combines Snake+ candidate generation, SCC confidence and timeliness decisions, and PRT buffering in idle registers to reduce L1 interference.The added structures total 2.7 KB per SM; prefetch generation and SCC decisions are off the demand critical path, with PRT lookup the only demand-side access.Source: §III · p. 4, Figure 7 · p. 4
  • SnakeCharmer checks PRT first for issued prefetches and later demand loads, buffering returned data in idle registers when an entry is available and otherwise using L1.PRT is configured for at most 8 KB of idle register capacity; it is partially enabled or disabled when less capacity or none is available.Source: §III · pp. 4–5

A. Snake+ Prefetcher

  • Snake+ learns inter-warp, inter-PC, and intra-warp stride patterns, with maximum distances of 63, 10, and 8, respectively.Its Head Table tracks each warp's recent load PC and address, while the SC-Tail Table stores stride patterns and training confidence.Source: A. Snake+ Prefetcher · p. 5
  • Snake+ suppresses a candidate only when its source pattern's 2-bit zero-value confidence counter reaches 3, and resets that counter after a 128-cycle suppression timer.Counters increase for all-zero data and decrease otherwise; the timeout supports recovery when the data phase changes.Source: A. Snake+ Prefetcher · p. 5
  • Zero-value filtering acts before issuance, complementing reactive zero-compressed caching by avoiding cache accesses and cache-side resource use for likely-zero candidates.Zero-compressed caching identifies zero-valued data after a request has entered the cache hierarchy.Source: A. Snake+ Prefetcher · p. 5

B. SnakeCharmer Controller

The SnakeCharmer Controller adjusts prefetch timing using feedback and gates candidates, while the Prefetch Remapping Table retains prefetched data in idle registers rather than always placing it in L1.

B. SnakeCharmer Controller

  • The AWT hierarchy learns local and global best prefetch distances from on-time, too-early, and too-late feedback, updating the best distance every 15 events.L1-AWT learns a Local Best Dist after 31 feedback samples; L2-AWT supplies a Global Best Dist for cold-start or infrequently executed patterns.Source: B. SnakeCharmer Controller · p. 5
  • The SGU gates each candidate using a total score based on training confidence and distance deviation, prioritizing higher confidence and smaller deviations.The score combines Sconf and Sdist; a total of at least 4 triggers issue.Source: B. SnakeCharmer Controller · pp. 5–6
  • The 64-entry PEB records prefetched blocks evicted before first use, distinguishing too-early but eventually demanded prefetches from never-used ones.Hits in the PRT or L1 indicate on-time use, PEB hits indicate too-early use, and MSHR hits indicate too-late use.Source: B. SnakeCharmer Controller · p. 6

C. Prefetch Remapping Table

  • The PRT stores prefetched data in reserved idle physical registers, allowing later demand loads to read it directly from the register file.The PRT is metadata-only; the prefetched data resides in registers, with returned data written through the existing LD/ST fill/writeback path.Source: C. Prefetch Remapping Table · p. 6
  • PRT capacity is limited to available idle registers, and if none are available, the PRT is disabled; consumed data is subsequently inserted into L1.Each PRT entry tracks four 32-byte slots through INVALID, WAITING, READY, and CONSUMED states; consumed data is read from the RF before its L1 refill.Source: C. Prefetch Remapping Table · p. 7

IV. EXPERIMENTAL SETUP

The evaluation uses cycle-level simulation of A100 and V100 GPUs, with reported results generally based on A100. Workloads are selected for sufficient Snake prediction accuracy and coverage, and the study compares SnakeCharmer configurations with Snake and APOGEE.

Simulation methodology

  • The study evaluates SnakeCharmer with AccelSim cycle-level simulation on modeled A100 and V100 GPUs, using AccelWattch for power estimates.Unless otherwise specified, reported results use A100; the simulator models banked register files.Source: §IV · p. 7
  • Kernel execution from launch to completion defines the region of interest, and results include cold-start effects because no warmup is discarded.For each application, results aggregate all kernels.Source: §IV · p. 7

Applications

  • The 14 selected applications have ideal Snake prefetch accuracy above 50% and coverage above 30%, focusing evaluation on workloads with address-prediction potential.High sparsity means more than 20% of demand-load response 4-byte units are zero-valued.Source: §IV · p. 8, Table II · p. 8

Evaluated Systems

  • Ten configurations compare Snake mechanisms and combinations, complete SnakeCharmer on Snake, and SnakeCharmer replacing APOGEE’s backend while retaining its FOA/TIA frontend.The zero-compressed cache is enabled in every configuration.Source: §IV · p. 8

V. EVALUATION

On A100, the full SnakeCharmer configuration achieves the strongest average prefetch coverage and accuracy among the evaluated systems. Combining backend mechanisms improves these measures, and SnakeCharmer also improves APOGEE-generated prefetches, though frontend opportunity limits the gains.

A. Coverage and Accuracy

  • 20.6% coverage and 40.4% accuracy for SK+ZF, 19.9% and 40.8% for SK+SCC, and 16.9% and 31.6% for SK+PRT show gains from each mechanism alone.For high-sparsity applications, SK+ZF reaches 23.7% coverage and 45.3% accuracy.Source: A. Coverage and Accuracy · p. 8
  • 27.9%, 27.1%, and 26.4% average coverage, with 52.2%, 44.9%, and 41.6% accuracy, respectively, result from combining each pair of mechanisms.The combinations are SK+ZF+SCC, SK+SCC+PRT, and SK+PRT+ZF; the results show that single-dimension control leaves other dimensions as bottlenecks.Source: A. Coverage and Accuracy · p. 8
  • 7.0% average coverage and 26.5% accuracy for APG+SC exceed APG’s 4.7% and 13.7%, respectively, but gains remain modest because APOGEE exposes fewer sufficiently accurate, high-coverage candidates.When the frontend cannot provide enough accurate candidates, address generation remains the dominant limitation rather than backend control.Source: A. Coverage and Accuracy · pp. 8–9
  • Figure 12 reports prefetch coverage and accuracy for evaluated systems on A100: coverage is H/(H + M), the fraction of baseline demand misses eliminated by prefetches; accuracy is H/F, the fraction of prefetch fills consumed by demand loads.Source: Figure 12 · p. 9

B. Performance and Energy Efficiency Analysis

SnakeCharmer improves performance and energy efficiency over Snake, with additional gains over APOGEE, while mechanism analyses report improved prefetch timing and reduced memory traffic alongside modest register-file bank-conflict increases.

B. Performance and Energy Efficiency Analysis

  • 10.6%, 9.9%, and 9.1% average performance gains for two-mechanism combinations compare with 12.8% for the complete design.The combinations are SK+ZF+SCC, SK+SCC+PRT, and SK+PRT+ZF, respectively.Source: B. Performance and Energy Efficiency… · p. 9
  • 1.7% average performance and 3.2% energy-efficiency gains result when SnakeCharmer replaces APOGEE’s original backend control path.The gains are smaller than with Snake because APOGEE exposes fewer sufficiently accurate and high-coverage prefetch opportunities.Source: B. Performance and Energy Efficiency… · p. 9
  • 12.8% higher average IPC and 20.8% higher average performance per watt for SK+SC over SK are the main overall gains.The comparison covers all applications on A100 and is normalized to SK.Source: B. Performance and Energy Efficiency… · p. 9, Figure 13 · p. 10

C. Mechanism Analysis

  • 6.1% higher register-file bank conflicts on average, with increases up to 15.2%, accompany SK+SC over SK.The PRT reuses idle registers and holds a limited number of prefetched blocks.Source: C. Mechanism Analysis · p. 9, Figure 14 · p. 10
  • 88.3% of consumed prefetched blocks are on time with SK+SC, compared with 48.3% for SK; SK has 28.7% too-early and 23.0% too-late prefetches.The reported distribution counts only prefetched blocks eventually consumed by demand loads.Source: C. Mechanism Analysis · p. 9, Figure 15 · p. 10
  • 23.6% lower L2 traffic and 15.1% lower off-chip memory traffic on average accompany SK+SC versus SK.The paper attributes the reductions mainly to improved utility and timeliness reducing unnecessary prefetches.Source: Figure 16 · p. 10, C. Mechanism Analysis · p. 10
  • 4.6% false positives and 20.9% false negatives characterize the zero-value confidence filter in SK+SC.The conservative policy rarely suppresses useful non-zero prefetches but requires repeated zero-value evidence before disabling a pattern.Source: Figure 18 · p. 10, C. Mechanism Analysis · p. 10

D. Sensitivity Analysis

Sensitivity studies show that SnakeCharmer outperforms Snake across the evaluated cache-capacity and bandwidth settings. The section also examines design choices for zero-value caching, timing, feedback storage, and register-assisted placement.

Zero-Compressed Cache Capacity

  • The zero-compressed cache reserves part of aggregate L1 and L2 capacity for zero-value metadata, balancing zero-block support against capacity for regular data.Smaller fractions limit zero-value representation, while larger ones reduce capacity available to regular data.Source: D. Sensitivity Analysis · p. 10, Figure 19 · p. 11

Different GPU Architectures

  • On V100, SnakeCharmer improves performance by 19.0% on average over Snake, exceeding its A100 gain.The passage attributes the larger gain to V100's more constrained cache hierarchy and greater vulnerability to cache contention and long-latency accesses.Source: D. Sensitivity Analysis · p. 11, Figure 20 · p. 11

SGU Threshold

  • The SGU threshold is 4 on A100 and 5 on V100, with maximum losses of 2.39% and 1.58%, respectively, versus per-application optimal thresholds.Lower thresholds issue more prefetches and may increase pollution and bandwidth waste; higher thresholds may suppress useful prefetches.Source: D. Sensitivity Analysis · p. 11, Figure 21 · p. 11

PEB Entries

  • Using 64 PEB entries yields gains of 9.1% on A100 and 9.6% on V100, with maximum losses of 1.5% and 0.3% versus per-application optimal sizes.Although 128 entries maximize performance on both architectures, 64 balances performance against storage overhead.Source: D. Sensitivity Analysis · p. 11, Figure 22 · p. 11

AWT Sampling Window Size

  • AWT sampling windows from 11 to 15 keep average performance variation below 1% and individual-application variation below 1.5%; the chosen window is 15.The evaluation uses this window size on A100 and V100.Source: D. Sensitivity Analysis · p. 11, Figure 23 · p. 11

Register Availability and PRT

  • A 64-entry PRT uses at most 8 KB of idle registers, while additional entries beyond available idle capacity are disabled.More entries generally improve performance by buffering more prefetched data, but the benefit diminishes once idle register capacity is reached.Source: D. Sensitivity Analysis · p. 11, Figure 24 · p. 11

Cache Capacity and Bandwidth

  • At least 12.2% and 11.2% performance improvements over Snake hold across evaluated cache-capacity and bandwidth settings, with larger gains under tighter resources.The larger gains occur at smaller cache capacities and lower bandwidth, where prefetch-induced cache pollution and traffic are more costly.Source: D. Sensitivity Analysis · pp. 11–12, Figure 25 · p. 12

E. Upper-Bound and Oracle Studies

Oracle studies compare SnakeCharmer's backend mechanisms with idealized utility, timing, and placement controls, as well as a broader L1 upper bound. The measured gaps reflect limits in prediction coverage, finite mechanisms, and hardware resources.

Upper-Bound and Oracle Studies

  • SK+SC has absolute performance gaps of 9.7% to O-SC and 19.8% to O-L1.O-SC combines backend oracles; O-L1 assumes all future global accesses hit at the L1 access point and is not constrained by Snake's prediction stream.Source: Figure 26 · p. 12, E. Upper-Bound and Oracle Studies · p. 12
  • SK+ZF trails O-ZF by 3.4%, SK+SCC trails O-TIME by 4.5%, and SK+PRT trails O-PLACE by 3.7% in absolute performance.The oracle variants respectively provide perfect zero-block filtering, ideal arrival timing, and unlimited prefetch placement space.Source: E. Upper-Bound and Oracle Studies · p. 12
  • The remaining gaps mainly reflect incomplete Snake prediction coverage, finite hardware mechanisms, and contention for L1 capacity, MSHRs, cache ports, and bandwidth.The oracle studies are not practical designs; most use the same prediction stream generated by the Snake frontend.Source: E. Upper-Bound and Oracle Studies · p. 12

F. Cost Analysis

SnakeCharmer's cost analysis reports modest storage and area overhead, while the related-work discussion distinguishes its backend control from software-managed staging, scheduling-aware prefetching, and prior register-based approaches.

F. Cost Analysis

  • SnakeCharmer adds 2.7 KB per SM (0.6% of A100 on-chip SRAM) and 0.04 mm2 (less than 0.005% of die area); its structures are one-cycle accessible.Only a one-cycle PRT lookup is on the demand side; SnakeCharmer reduces GPU power by 6.6% versus Snake.Source: F. Cost Analysis · p. 12, Table III · p. 12

VI. RELATED WORK

  • Unlike software-managed staging –, SnakeCharmer transparently controls hardware-generated candidates; dotf and sto benefit despite already using software-managed prefetching.Software-managed techniques require explicit movement, synchronization, and shared-memory capacity, and work best when access patterns or reuse are known ahead of time.Source: §VI · p. 12
  • Scheduling-aware prefetchers – couple candidate generation to warp or CTA scheduling, so integration would require scheduler-interface redesign rather than backend replacement.The paper focuses on address-prediction-based prefetchers such as Snake and APOGEE.Source: §VI · pp. 12–13
  • Prior register-capacity designs target demand or victim data,, while spare-register prefetching depends on identifiable loop-based load pairs rather than a hardware-managed buffer.SnakeCharmer coordinates register-assisted placement with utility filtering and timeliness control.Source: §VI · p. 13
  • GPU warp interleaving makes logical prefetch distance an unstable proxy for consumption time, so SnakeCharmer learns timing by warp and load-PC context.This adapts feedback-driven distance mechanisms used by CPU prefetchers, to GPU execution.Source: §VI · p. 13
  • With APOGEE's FOA/TIA frontend unchanged, replacing its backend with SnakeCharmer improves performance by 1.7%.The two-level tuner learns pattern-local and global timing behavior while coordinating with zero-value filtering and PRT-based placement.Source: §VI · p. 13
  • SnakeCharmer adds proactive zero-value-aware filtering to zero-value caching –, suppressing likely-zero prefetches before they consume queues, MSHRs, bandwidth, or cache resources.Zero-value caching compactly represents all-zero blocks, while LazyGPU also combines it with lazy execution.Source: §VI · p. 13

VII. CONCLUSION

SnakeCharmer is a predictor-agnostic backend for address-prediction-based GPU prefetchers that jointly controls utility, timeliness, and placement. On an A100 GPU model, it improves performance and energy efficiency over both Snake and APOGEE.

VII. CONCLUSION

  • SnakeCharmer is a predictor-agnostic backend for address-prediction-based GPU prefetchers.It jointly controls utility, timeliness, and placement through zero-value-aware filtering, feedback-driven distance tuning, and register-assisted buffering.Source: §VII · p. 13
  • On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 12.8%/20.8% over Snake.Source: §VII · p. 13
  • On an A100 GPU model, SnakeCharmer improves performance/energy efficiency by 1.7%/3.2% over APOGEE.Source: §VII · p. 13
Loading 2610.30990v1…