Source-linked AI summary
Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE
Jie Ren, Ivy Peng
TL;DR
Growing memory demand and cost make multi-tier memory attractive, but using it effectively requires understanding workload memory behavior. The paper implements high-resolution characterization with ARM SPE and studies eight CloudSuite and LLM workloads. It reports migration benefits for workloads identified as suitable, while Llama and Memcached show no benefit under the tested policy.
Problem
Memory-intensive cloud and data-center workloads increase demand and cost, while ARM platforms remain relatively underexplored for fine-grained memory characterization.
Source: §1 · p. 1Method
The paper implements ARM SPE-based characterization, analyzes eight CloudSuite and LLM workloads, and evaluates workload-focused metrics with an SPE-guided data-migration runtime.
Source: §1 · pp. 1–2Results
Page ranking and connected components improve under SPE-guided migration, while Llama and Memcached show no benefit with the tested policy.
Source: §4.3.3 · p. 10Takeaways & Limitations
Latency distributions and longest idle periods help assess workload suitability for tiered-memory placement and runtime migration.
Source: §4.3.3 · p. 10, §6 · p. 10
Abstract
from arXiv · showhide
Memory demands from data analytics and LLM workloads grow rapidly, driving up memory costs in cloud and data centers. Multi-tier memory has emerged as a promising solution to balance performance and cost. However, effectively leveraging it requires a deep understanding of workload memory behaviors. This work introduces a high-resolution memory characterization methodology for analyzing workload-specific opportunities for multi-tier memory usage. In response to the increasing deployment of ARM-based processors in data centers, we implement our methodology using the recent Statistical Profiling Extension (SPE) for ARM. By characterizing eight memory-intensive workloads from CloudSuite and LLM applications, we identify region-specific access patterns, evaluate the effectiveness of cloud-oriented cache hierarchies, and propose new metrics based on memory idle periods and latency profiles to determine suitable data placement for tiered memory.
1 Introduction
Rising memory demands and costs make multi-tier memory attractive, but effective placement requires detailed knowledge of workload access patterns. This work addresses the relative lack of fine-grained ARM memory characterization with an SPE-based method and workload-focused metrics.
1 Introduction
- Multi-tier memory pairs fast, expensive memory with a larger, more affordable but slower tier to balance cost and performance.Effective management requires understanding workload access patterns and hardware–software interactions.Source: §1 · p. 1
- ARM platforms have been underexplored for fine-grained memory characterization because most profiling tools target x86 and earlier ARM implementations offered limited precise sampling.ARM SPE makes detailed memory analysis feasible.Source: §1 · p. 1
- The paper illustrates region-specific GPT-2 L2-hit patterns across weights, activations/gradients, and optimizer states during training and forward-only phases.Forward-only intervals include validation-loss computation and text generation.Source: §1 · p. 1, Figure 1 · p. 2
- The work provides a high-resolution ARM characterization method and SPE implementation, characterizes eight CloudSuite and LLM workloads, and evaluates accuracy and overhead against DAMON.The method analyzes workload characteristics and memory-hierarchy utilization.Source: §1 · p. 1
- The proposed workload-focused metrics assess multi-tier memory feasibility, and experiments evaluate their usefulness with an SPE-guided data-migration runtime.The metrics include latency profiles and longest idle periods.Source: §1 · p. 2
2 Characterization Methodology
The methodology profiles workload access patterns and their use of the memory hierarchy to inform prefetching, data placement, and cache or memory configuration. It uses ARM SPE for high-resolution, per-thread memory characterization and validates the resulting tool against DAMON.
2 Characterization Methodology
- The characterization targets comprehensive access patterns, hierarchy utilization tied to memory regions, per-thread concurrency, high temporal and spatial resolution, low overhead, and language-agnostic profiling.These goals support deriving prefetching and placement strategies and tailoring cache and memory configurations.Source: §2 · p. 2
2.1 An ARM SPE-Based Design and Implementation
- ARM SPE samples instructions at a configurable period and can record memory addresses, cache and TLB events, data sources, and latency through the execution pipeline.The tool separates samples by core for per-thread analysis, supports 200 ms profiling windows and byte-addressable spatial resolution, and uses L1D load misses at period 1,024 with a 1 MiB buffer in this work.Source: §2.1 · p. 3
- The tool correlates cache and memory hierarchy utilization with specific memory regions and measures DRAM issue, translation, and total latency.The reported latency for DRAM samples is an SPE operation-latency measurement that can include delays beyond DRAM service time.Source: §2.1 · p. 3
2.2 Validation
- SPE captures STREAM’s per-thread streaming regions and RandomAccess’s normal access distribution, while DAMON loses moderate- and low-frequency RandomAccess detail.In MASIM, SPE resolves three address-range access phases; DAMON does not clearly resolve the transitions under the tested configuration.Source: §2.2 · pp. 3–4
- 1.0 precision is maintained across all three benchmarks; recall is 0.5 for RandomAccess, 0.6 for MASIM, and 0.02 for STREAM, while DAMON’s metrics vary.The top-50-region selection limits recall for uniformly accessed STREAM regions.Source: Figure 4 · p. 4, §2.2 · p. 4
- SPE overhead is below 0.6% for MASIM’s 64 KB, 1 GB, and 10 GB configurations, but 3.6–4.5% for 100 GB; DAMON incurs about 1.5–6.1%.SPE overhead is consistent across the tested sample periods for the three smaller configurations; in 100 GB, it decreases with larger periods.Source: §2.2 · p. 4, Figure 5 · p. 5
3 Experimental Methodology
The experiments use a data-center-grade ARM testbed to characterize eight workloads spanning LLMs, graph analytics, and data processing. The workload configurations include distinct models, datasets, thread counts, and measured outcomes.
3 Experimental Methodology
- Eight workloads are evaluated on an ARM Ampere Altra Max testbed with 128 Neoverse N1 cores, 512 GB DDR4, and a 16 MB shared system-level cache.The testbed runs Ubuntu 22.04 and Linux kernel 5.15; the authors also report similar traces on AmpereOne.Source: §3 · p. 5, Table 1 · p. 5
- LLM inference uses Llama 8B with batch size 8 on 16 threads, while GPT-2 124M pretraining runs with sequence length 1,024 and 1,192 batches.A separate Qwen3-30B-A3B inference setup uses 2,000 prompt tokens, generates 512 tokens, and runs on 128 threads.Source: §3 · p. 5
- Graph workloads run PageRank, Triangle Counting, and Connected Components on a Twitter dataset; data-processing workloads include Memcached, Cassandra, and Spark MLlib.The evaluations measure workload-specific outcomes including latency, throughput, and execution time.Source: §3 · p. 5
4 Memory-intensive Workloads
The section characterizes eight workloads through access patterns, cache and memory-hierarchy use, and region-level latency and idle-period profiles. It uses those profiles to assess multi-tier memory opportunities and reports that migration outcomes align with workload-specific metrics.
4 Memory-intensive Workloads
- The study applies temporal and spatial access-pattern analysis, including strided, sparse, and block patterns, to eight workloads with multi-tier-memory systems in view.Its heatmaps filter sampled L1D load misses; the characterization also examines hierarchy utilization and memory-region profiles.Source: §4 · p. 5, §4.1 · p. 5
4.1 Workload Insights.
- Workload heatmaps showed GPT-2 parameter accesses reversing traversal direction between forward and backward passes, sparse forward-pass activations, and dense backward-pass gradient blocks; Qwen3 KV-cache accesses formed repeated ascending bands.Cassandra, Memcached, and In-Memory Analytics also showed strided patterns, while graph workloads showed block patterns linked to frequently accessed data structures.Source: §4.1 · pp. 5–7
4.2 Utilization of Cache and Memory Hierarchy.
- Private L2 was the most effectively used cache; most workloads also used the shared LLC, except Llama, while stride patterns hit LLC more than sparse or block patterns.The tested system had 16 MB LLC versus 128 MB aggregate private L2; Memcached showed peer-cluster sharing in both phases and peer-CPU sharing only after its initial set-only phase.Source: §4.2 · p. 7
4.3 Memory Region-based Characterization.
- Two hot PageRank regions with different DRAM-latency profiles illustrate why access counts alone can miss placement priorities: the smaller region had a tail up to 2,500 cycles and was prioritized for fast memory.Across workloads, Memcached, Triangle Counting, Page Rank, and Connected Components had longer latency tails than Llama and GPT2, while Llama had more samples near 4,096 cycles.Source: §4.3 · p. 7
4.3.1 Region-Latency Distribution.
- Latency-weighted ranking found highly skewed DRAM latency in In-Memory Analytics, where 50% was attributed to 0.7% of regions, versus 3% in Cassandra and about 5.2% in GPT2 training.Llama, Triangle Counting, and Memcached had more even distributions; the authors state that block-dominated workloads are less feasible for tiering than highly skewed workloads.Source: §4.3.1 · pp. 7–8
4.3.2 Idle Period Distribution.
- A longest observed idle period above 30% of execution time served as an adaptable migration-candidate heuristic; Llama had few such regions, and Memcached had less than 5%.Together with latency concentration, the profiles suggested fewer runtime-migration opportunities for Llama and Memcached, while GPT2 had concentrated latency and some long observed gaps.Source: §4.3.2 · pp. 8–9
4.3.3 SPE-guided Optimization on Memory Tiers.
- Cassandra achieved the highest performance improvement, while Page Ranking and Connected Components improved and Llama and Memcached saw no benefit under the tested migration policy.Memcached suffered a large performance loss in its built-in 99th-percentile latency; the runtime made migration decisions periodically and migrated selected regions every second.Source: §4.3.3 · p. 10
5 Related Works
Prior work profiles memory through OS mechanisms and PMU-based sampling, with differing levels of detail and overhead. The paper contrasts those approaches with ARM SPE's sample-level hierarchy, data-source, and latency information.
OS-based profiling
- OS-based methods range from costly page-poisoning fault handlers to PTE scans that miss repeated accesses between scans and lightweight queue-based methods with limited information.DAMON trades accuracy for performance within an overhead budget.Source: §5 · p. 10
PMU-based profiling
- PMU-based approaches include Intel PEBS, AMD IBS, and ARM SPE; PEBS has been used for NUMA bottleneck analysis, hotness identification, and traces, while IBS provides richer per-sample data.The passage notes that sampling bias and overhead remain challenges.Source: §5 · p. 10
ARM SPE
- ARM SPE combines supported cache and TLB events, data-source information, and operation latency in each sample record, enabling memory characterization on ARM platforms.Source: §5 · p. 10
6 Conclusions
The paper presents ARM SPE-based high-resolution memory characterization for data centers and proposes latency-profile and longest-idle-period metrics for evaluating multi-tier opportunities. Its evaluation across eight workloads highlights the method's usefulness for workload and memory optimization.
6 Conclusions
- The methodology uses ARM SPE to characterize memory behavior amid increasing ARM processor deployment in data centers.Source: §6 · p. 10
- Latency profiles and longest idle periods are proposed as metrics for assessing workload opportunity and migration feasibility in multi-tier memory.Source: §6 · p. 10
- Evaluation on an ARMv8 platform across eight CloudSuite and LLM workloads highlights the approach's usefulness for studying cloud-oriented cache hierarchies and guiding memory optimizations.Source: §6 · p. 10