PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheFlex: Direct Software-Managed Access to Higher-Level Cache for Scalable Vector Support

Jingqun Zhang, Maohua Nie, Jiayi Wang, Weihang Li, Rishi Sappidi, Shwet Chitnis, Ang Li

arXiv:2610.12940v1cs.AR

TL;DR

Wider SIMD lanes increase compute throughput, but L1 capacity constrains GEMM performance while private L2 is not directly software-accessible. CacheFlex converts configurable L2 ways into a software-managed SPM. At VL = 16, it achieves a 1.44× geometric-mean kernel speedup with 40% lower memory-subsystem energy, and CacheFlex with FlashAttention speeds full layers by 1.64× on LLaMA and 1.78× on BERT at T=4096.

  • Problem

    The paper asks how to provide software-managed access to private L2 while retaining out-of-order execution and OS support.

    Source: §I · p. 1, C. Opportunity: Direct Access to L2 · p. 3

  • Method

    CacheFlex dynamically converts configurable private L2 ways into a software-managed SPM, with cross-domain memory ordering for bridge operations.

    Source: §I · p. 1, 3) Decoupled retirement · p. 5

  • Results

    At VL = 16, CacheFlex achieves a 1.44× geometric-mean speedup and 40% lower memory-subsystem energy on kernels; with FlashAttention at T=4096, layer speedup is 1.64×/1.78× on LLaMA/BERT.

    Source: D. Expected Results and Validation · p. 15

  • Takeaways & Limitations

    CacheFlex uses existing L2 SRAM to improve memory-subsystem energy efficiency and, in capacity-limited high-vector-length workloads, performance, with 2.02% area overhead relative to L2.

    Source: §VII · p. 12

Abstract

from arXiv · show

Wider vector/SIMD lanes multiply compute throughput, but cache improvements remain challenging. We identify L1 data cache capacity as the most impactful parameter for performance and efficiency. L1 capacity is constrained by pipeline coupling and latency requirements; private L2 offers an order of magnitude more capacity but is inaccessible from software. This work proposes CacheFlex, an ISA and microarchitecture co-design that enables software-managed, direct access to private L2 SRAM by allowing runtime conversion of parts of L2 into a scratchpad memory (SPM). Through a lightweight ISA extension, software explicitly controls data placement and lifetime within the L2. CacheFlex integrates into an out-of-order pipeline with cross-domain memory consistency and OS context-switch support, with merely 2.02% area overhead relative to the L2 in a post-layout 12 nm implementation. We prototype CacheFlex based on the Armv8-A SVE ISA, model it in gem5, and evaluate it using optimized general matrix multiplication (GEMM) kernels with diverse model-derived shapes, FlashAttention-based fused attention, and full-layer workloads from LLaMA-3.2-1B and BERT-Base in the prefill phase. As the vector length (VL) scales from VL = 4 (512-bit) to VL = 16 (2048-bit), CacheFlex's kernel-level geometric-mean speedup and memory-subsystem energy reduction grow from 1.12x and 26% to 1.44x and 40%, respectively, bridging the widening gap between per-instruction data demand and memory performance. On full transformer layers combining CacheFlex with FlashAttention, the design delivers up to 1.78x speedup and 45% memory-subsystem energy savings (BERT-Base, sequence length T=4096, VL = 16; LLaMA-3.2-1B: 1.64x and 38%). CacheFlex is complementary to hardware prefetching, and its benefit grows as vector width and peak compute throughput increase for compute-intensive AI kernels.

I. INTRODUCTION

CacheFlex targets L1 capacity, identified as the main bottleneck as SIMD widths grow, by making private L2 SRAM directly software-accessible as an SPM. It preserves out-of-order execution and OS support while improving kernel and full-layer performance with low area overhead.

Motivation

  • Directly enlarging L1 is impractical, while private L2 offers substantially more storage but lacks direct software access.The introduction frames on-demand L2 access as a design challenge that must retain OoO execution and OS support.Source: §I · p. 1

CacheFlex architecture

  • CacheFlex converts configurable private L2 ways into a software-managed SPM, enabling explicit control of data placement and lifetime while retaining out-of-order execution and OS compatibility.The dedicated SPM load/store unit separates SPM accesses from cross-domain transfers, allowing data movement to overlap computation.Source: §I · pp. 1–2

Motivation

  • Widening SIMD vectors makes L1 capacity the primary GEMM throughput bottleneck, driving persistent L1–L2 data movement and energy overhead.The paper identifies L1 capacity as the most impactful design parameter for performance and efficiency.Source: §I · pp. 1–2

Evaluation

  • At VL = 16, CacheFlex achieves a 1.44× geometric-mean speedup and 40% memory-subsystem energy reduction across kernels, with up to 1.78× speedup and 45% energy savings on full transformer layers.The implementation adds 2.02% area overhead relative to the L2.Source: §I · p. 2

II. MOTIVATION

Wider SIMD lanes raise peak compute throughput, but cache capacity can limit attained GEMM throughput. Multi-level tiling targets reuse at different cache levels, and measurements identify L1 capacity—not bandwidth—as the key constraint in the evaluated configuration.

A. SIMD Throughput Scaling

  • Widening SIMD lanes increases peak throughput, but without a proportional memory upgrade, cache misses can cause pipeline stalls and limit attained throughput.The paper motivates analysis of the memory hierarchy to identify the primary bottleneck.Source: A. SIMD Throughput Scaling · p. 2

B. GEMM: Cache Capacity and Tiling

  • GEMM kernels use packed contiguous panels and multi-level blocking to target reuse, with smaller working panels for L1 and larger panels for higher-level caches.The innermost output-stationary C tile is held in vector registers; larger cache capacity permits bigger panels and reduces panel swaps and memory traffic.Source: B. GEMM: Cache Capacity and Tiling · pp. 2–3, Figure 1 · p. 2
  • With 64 KB L1, increasing SIMD width from 512-bit to 2048-bit yields only a 1.6× attained-throughput gain as utilization falls from 72% to 29%; 512 KB L1 raises 2048-bit utilization to 45%.Doubling L1 bandwidth from 320 to 640 GB/s yields no measurable improvement; CacheFlex matches or exceeds the 512 KB L1 configuration at each evaluated width.Source: B. GEMM: Cache Capacity and Tiling · p. 3, Figure 2 · p. 3

C. Opportunity: Direct Access to L2

L1 capacity is difficult to scale because of its close integration with the core, even though private L2 contains substantially more SRAM. CacheFlex exposes configurable L2 ways as a software-managed SPM through an ISA extension and access paths for regular-to-SPM transfers.

C. Opportunity: Direct Access to L2

  • L1 capacity is difficult to increase without latency, area, and power costs, while private L2 holds 256 KB to 2 MB of SRAM but conventional accesses incur translation, tag lookup, and line-fill overheads.Direct access to L2 SRAM can avoid those costs for repeated SPM accesses after a coherent bridge copy.Source: C. Opportunity: Direct Access to L2 · p. 3

III. CACHEFLEX ARCHITECTURE

  • CacheFlex converts a configurable subset of L2 ways into a thread-private SPM whose cache-line slots use way and set indices as SPM addresses.Pure SPM accesses bypass address translation, associative tag matching, and coherence probes.Source: §III · p. 3, C. Opportunity: Direct Access to L2 · p. 3

A. ISA Extension

  • The ISA separates privileged SPM management from user-level SPM loads, stores, and bridge instructions for moving data between regular memory and SPM.spm_cp and spm_wb provide bridge paths without round-trip register transfers.Source: A. ISA Extension · pp. 3–4, Table I · p. 4

B. Microarchitecture Overview

CacheFlex adds direct, software-managed access to configurable L2 ways while retaining conventional cache ways. Dual-LSQ coordination preserves ordering across SPM and regular memory during bridge operations.

B. Microarchitecture Overview

  • CacheFlex adds SPM control logic, a dedicated bank-arbitrated L2 read path, and an SPM coherence state; remaining L2 ways continue as a conventional cache.SPM accesses use a dedicated SPM LSQ, while bridge instructions enter both LSQs.Source: B. Microarchitecture Overview · p. 4, Figure 4 · p. 4

C. Cross-Domain Memory Ordering

  • Bridge operations enter both LSQs so each memory domain enforces its own ordering while coordinating transfers across the boundary.Pure SPM operations use only the SPM LSQ; regular bridge operands retain normal translation and permission checks.Source: C. Cross-Domain Memory Ordering · pp. 4–5
  • Dual-LSQ coordination holds or replays conflicting younger SPM accesses for spm_cp and blocks overlapping regular accesses for spm_wb until installation is visible.Bridge operations do not forward data to younger regular loads.Source: 1) Coordination for spm_cp (regular… · p. 5, 2) Coordination for spm_wb (SPM… · p. 5, Figure 5 · p. 5
  • Bridge instructions retire at commit without waiting for installation, while destination-side ordering state remains active until completion.Barriers wait on retained state when their completion semantics require it; SPM regions are thread-private and non-coherent.Source: 3) Decoupled retirement · pp. 5–6

D. In-Cache Data Transactions

CacheFlex moves data between regular memory and SPM with lazy, demand-driven copies rather than bulk transfers. Direct SPM accesses bypass regular-cache lookup and coherence, while bridge transfers use the regular memory path.

Lazy, demand-driven copying

  • Demand-driven spm_cp fills individual SPM slots only when software requests a copy, avoiding bulk pre-population and interconnect traffic bursts.The regular hierarchy continues serving the coherent source during a copy, with LSQs enforcing necessary ordering.Source: D. In-Cache Data Transactions · p. 6

Regular-to-SPM transfer

  • spm_cp obtains the latest regular-memory line through the L1-to-L2 path and installs it in the SPM slot without returning it to a register.The source remains in place; a valid destination cache line is evicted before lazy conversion completes.Source: D. In-Cache Data Transactions · p. 6
  • An L1 hit supplies the SPM slot without allocating or evicting the source, while an L2 hit copies the resident line through the L2 pipeline.For a full miss, the fill is routed directly into the SPM slot rather than installed on the regular-memory side.Source: D. In-Cache Data Transactions · p. 6

SPM-to-regular transfer

  • spm_wb pairs an SPM-side read with a coherent regular-side write, whose completion follows visible installation through the normal coherence path.The destination write obtains writable ownership, and conflicting copies are handled by the existing coherence protocol.Source: D. In-Cache Data Transactions · p. 6

SPM accesses

  • spm_ld and spm_st access L2 SRAM directly using physical way/slot addresses, bypassing translation, regular-cache tags, and coherence probes.Within the SPM domain, ordering is handled by the SPM LSQ.Source: D. In-Cache Data Transactions · pp. 6–7

E. Cache Coherence Compatibility

CacheFlex uses conventional coherence behavior when SPM slots encounter resident cache lines and when data is written back to regular memory. OS support drains and saves SPM state at context switches, with policy options for retaining or spilling contents.

E. Cache Coherence Compatibility

  • When an SPM access first touches a converted way, a resident line is flushed through the standard coherence path: dirty lines are written back and clean lines invalidated.Deallocation drains outstanding accesses and bridge operations; live contents can be spilled or discarded, and ways return lazily to cache mode.Source: E. Cache Coherence Compatibility · p. 7

F. OS Compatibility

  • The OS manages SPM through privileged allocation operations and CSRs for mode, slot bounds, and a page-aligned backing region.User code accesses CacheFlex through an OS- or runtime-provided interface.Source: F. OS Compatibility · p. 7
  • At a context switch, older SPM accesses and bridge operations drain before the OS saves SPM state; contents may remain resident or be spilled to backing memory.An owner-thread ID prevents other threads from accessing resident SPM data, and ownership conflicts are resolved before reassignment.Source: F. OS Compatibility · p. 7

IV. EXPERIMENTAL METHODOLOGY

The evaluation models an Armv8-A out-of-order core and compares CacheFlex with the same core, cache, and memory parameters. It tests optimized GEMM kernels, alternative reuse mechanisms, and transformer-layer workloads, while combining simulation with physical implementation for energy and area estimates.

Simulation infrastructure and baseline

  • The gem5 evaluation uses a single-core Armv8-A out-of-order configuration to isolate CacheFlex's effects on private-hierarchy data placement and movement.The baseline and CacheFlex share core, cache, and memory parameters; CacheFlex adds its ISA, SPM structures, way reconfiguration, and read port.Source: §IV · p. 7, Table II · p. 7

Kernel-level GEMM analysis

  • Seven model-derived GEMM shapes cover compute-bound and memory-sensitive regimes, with independently swept tiling across VL = 4, 8, and 16.Each design point reports its highest-throughput tiling.Source: §IV · p. 7, Table III · p. 8

Alternative mechanisms and layer-level evaluation

  • Comparisons include hardware and software prefetching, PLDL2KEEP, and idealized L2 pinning with non-temporal loads, alongside prefill layers from LLaMA-3.2-1B and BERT-Base.Layer-level evaluation includes FlashAttention-like fused operators; longer BERT sequences use BERT-Base-shaped encoder layers.Source: §IV · p. 8

Layer-level evaluation

  • FlashAttention places selected streaming K/V tiles in SPM, while other operands remain in the conventional cache hierarchy.Full-layer time and memory-subsystem energy combine constituent operators using their model-defined multiplicities.Source: §IV · p. 8

Energy and area

  • Memory-subsystem energy combines simulated access counts with RTL-derived per-access energy and leakage, while area comes from a post-place-and-route layout.The reported subsystem includes cache, SPM, and DRAM; DRAM energy uses gem5's integrated power model.Source: §IV · p. 8

V. RESULTS

CacheFlex preserves GEMM throughput as vector length increases, where fixed L1 capacity increasingly limits the baseline. Comparisons with prefetching and L2-pinning alternatives show the value of direct SPM access and its dedicated read path.

A. Kernel-Level GEMM Evaluation

  • As VL rises, fixed L1 capacity pushes the baseline into a capacity-limited regime, while CacheFlex degrades more gradually across shapes.At VL = 4, latency hiding keeps the baseline and CacheFlex at similar throughput; the main difference is energy from avoiding repeated cache-hierarchy overheads.Source: A. Kernel-Level GEMM Evaluation · pp. 8–9
  • At VL = 16, CacheFlex reaches a geometric-mean speedup of 1.44× and 40% memory-subsystem energy reduction; speedup grows monotonically with VL, though individual shapes may saturate under off-chip bandwidth limits.Source: A. Kernel-Level GEMM Evaluation · p. 9

B. The Capacity Wall and the Decoupled SPM Datapath

  • Hardware prefetch improves W3/W5 over no prefetch by +46%/+29%, but cannot reduce the L1–L2 swap traffic that caps reuse.The evaluated L1 software-prefetch configurations add negligible benefit over the hardware-prefetch baseline.Source: B. The Capacity Wall and the Decoupled… · pp. 9–10
  • PLDL2KEEP cannot pin L2 lines or bypass L1, leaving it exposed to conflicts and the same L1–L2 traffic limitation.Avoiding conflicts requires conservative use of L2 capacity, which can produce suboptimal tiling.Source: B. The Capacity Wall and the Decoupled… · p. 10
  • CacheFlex outperforms idealized L2 pinning with non-temporal loads by 1.63×/1.34× on W3/W5, respectively.Unlike spm_ld, non-temporal loads still traverse the L1 pipeline and incur address translation, tag lookup, and coherence overheads.Source: Figure 8 · p. 9, B. The Capacity Wall and the Decoupled… · p. 10
  • Sharing regular L2 read bandwidth still leaves CacheFlex 1.43×/1.19× faster than the pinning upper bound on W3/W5; dedicated read bandwidth adds 1.14×/1.13×.The reported gains stack: most comes from bypassing L1 and accessing larger L2 directly, with a further contribution from the dedicated read path.Source: Figure 8 · p. 9, B. The Capacity Wall and the Decoupled… · p. 10

C. Full-Layer Evaluation

In full-layer prefill workloads, CacheFlex and FlashAttention combine to improve performance, particularly at long sequence lengths and wider vectors. CacheFlex also reduces memory-subsystem energy, while its attention-kernel gains depend on the K+V working set relative to L2 capacity.

C. Full-Layer Evaluation

  • At T=256 and VL = 16, CacheFlex GEMM reaches 1.61× speedup on LLaMA and 1.57× on BERT over baseline.At short sequence lengths, FFN and projection GEMMs dominate layer time; at VL = 4, CacheFlex GEMM reaches 1.04–1.08×.Source: C. Full-Layer Evaluation · p. 10
  • For FlashAttention on LLaMA, CacheFlex gains at most 3.2% when the per-KV-head K+V working set fits in L2, and 18.3% at VL = 16 when it reaches L2 capacity.At T=4096, the working set is twice L2 capacity and gains remain positive at 9.7% for VL = 4 and 15.2% for VL = 16.Source: Table IV · p. 10, C. Full-Layer Evaluation · pp. 10–11
  • CacheFlex stages selected K/V tiles in SPM for reuse across query blocks, complementing FlashAttention’s avoidance of score-matrix materialization.In unfused attention at T=4096, the 32 MB score matrix dwarfs L2 and SPM, so CacheFlex provides negligible speedup on attention itself.Source: C. Full-Layer Evaluation · p. 10
  • At T=4096, CF+FA reaches 1.19×/1.34× speedup at VL = 4 and 1.64×/1.78× at VL = 16 over baseline on LLaMA/BERT.At VL = 16, CacheFlex adds about 34% on LLaMA over FlashAttention alone; BERT shows the same trend.Source: C. Full-Layer Evaluation · p. 10, Figure 9 · p. 11
  • On LLaMA, CF+FA reduces memory-subsystem energy by 13.1% at T=256 and 28.3% at T=4096 for VL = 4, and by 34.6% and 38.4% for VL = 16.Energy savings remain substantial even when throughput gains are modest.Source: C. Full-Layer Evaluation · p. 11

D. Area Overhead

CacheFlex repurposes existing L2 capacity rather than adding SRAM, with a measured area overhead of 2.02% relative to the L2 baseline. Prior designs provide related cache–scratchpad and configurable-memory approaches, but differ in their storage, access, or execution models.

D. Area Overhead

  • CacheFlex adds 8,377 µm2 of silicon and increases total area overhead relative to the L2 baseline by 2.02%, without adding SRAM arrays.The SPM load/store queue occupies 7,084 µm2 and L2 SPM control logic adds 1,293 µm2.Source: D. Area Overhead · p. 11

VI. RELATED WORK

  • CacheFlex combines reclaimed private-L2 capacity, explicit bridge operations, dual-LSQ cross-domain ordering, and wide-vector operand reuse.Earlier hybrid designs include shared SRAM cache–scratchpad systems, dynamically divided local SRAM, and separate scratchpads, but differ in CPU execution or memory-management support.Source: §VI · pp. 11–12
  • Other configurable-memory designs repartition or reconfigure on-chip storage, but CacheFlex converts selected private-L2 ways at runtime into an untagged, directly addressed per-thread scratchpad.Examples include Smart Memories, software-controlled cache-way repartitioning, fixed L2 cache sizes, and coarse Flat, Cache, or Hybrid modes.Source: §VI · p. 12
  • Cache partitioning controls capacity or placement while leaving lines replacement-managed; CacheFlex instead guarantees residency for selected reused tiles.Cache compression increases effective capacity, while non-temporal hints reduce pollution but do not provide that residency guarantee.Source: §VI · p. 12
  • Prefetching hides latency but leaves lines replacement-eligible, while cache pinning retains lines through the ordinary tagged-cache access path.UVE streams operands into vector registers but exposes no separately addressable cache-capacity region.Source: §VI · p. 12
  • AMU also partitions L2 into a scratchpad on an out-of-order core, but targets far-memory latency and uses software disambiguation rather than CacheFlex’s hardware dual-LSQ ordering.CacheFlex also virtualizes SPM state per thread.Source: §VI · p. 12
  • CacheFlex is complementary to prefetching: prefetchers can stage coherent source data before spm_cp copies it into the SPM.The related work also identifies orthogonal near-data approaches and algorithm-level data-placement work such as FlashAttention.Source: §VI · p. 12

VII. CONCLUSION

As vector length grows, fixed L1 capacity weakens latency hiding and limits the conventional cache hierarchy’s ability to provide software-controlled, guaranteed-residency access to private L2. CacheFlex repurposes private L2 as a scratchpad while preserving OoO support, improving energy efficiency and high-vector-length performance across the evaluated workloads.

VII. CONCLUSION

  • As vector length scales, per-instruction data demand outgrows fixed L1 capacity and weakens the OoO CPU’s tolerance for L1-capacity overflow.The resulting limitation is the lack of direct, software-controlled private-L2 access with guaranteed residency.Source: §VII · p. 12
  • CacheFlex repurposes existing private L2 SRAM as a software-managed scratchpad, preserves OoO execution support, and incurs 2.02% area overhead relative to L2.It improves memory-subsystem energy efficiency and performance in the capacity-limited high-vector-length regime across GEMM and FlashAttention-based workloads.Source: §VII · p. 12
  • The paper concludes that exposing existing L2 capacity as a software-managed resource is practical for sustaining CPU efficiency as vector throughput scales.It identifies graph analytics, sparse linear algebra, bandwidth-bound kernels, matrix engines, and OS-level SPM scheduling as possible extensions.Source: §VII · p. 12

APPENDIX ARTIFACT APPENDIX

The appendix documents artifact requirements and reproduction steps, then lists expected validation outcomes for the paper’s measurements and reruns.

A. Availability and Requirements

  • The artifact includes the CacheFlex gem5 model, instruction support, benchmarks, experiment scripts, and paper measurements; inputs are deterministic and need no external datasets or model weights.An x86-64 Linux host with Python 3.11 or 3.12 is required.Source: A. Availability and Requirements · p. 15

B. Generate the Paper Figures

C. Rerun the Simulations

  • Full simulation reruns require the Arm GNU Toolchain, QEMU, gem5, and GCC/G++ 10–14, and take roughly 1–2 days at -j 8 after setup.The optional tiling sweep is not needed to reproduce the paper results.Source: C. Rerun the Simulations · p. 15

D. Expected Results and Validation

Related papers

Loading 2610.12940v1…