System Evaluation of the Intel Optane Byte-addressable NVM

1 INTRODUCTION

• Large datasets, deep neural networks, and graph workloads increase memory-capacity demands, while scaling DRAM can be costly in power and cost and constrained by density.DRAM refresh power scales with capacity, and its low density complicates large-capacity systems within area constraints.Source: §1 · p. 1
• The study evaluates the first commercially available byte-addressable NVM using hardware experiments and modeling, including eight memory configurations and graph workloads.The authors distinguish this realistic hardware evaluation from earlier simulation- and emulation-based studies.Source: §1 · p. 1
• For graph applications, DRAM-cached NVM brings performance close to DRAM, while directly replacing DRAM with NVM can reduce performance by more than the DRAM–NVM bandwidth and latency gaps.Using local NVM on one socket may be more efficient than using DRAM on two sockets for some workloads.Source: §1 · p. 1
• Bandwidth spilling achieves 2.0x bandwidth and enables larger problems than DRAM-cached NVM; write isolation saves up to 3.9x energy and improves bandwidth by 3.1x versus DRAM-cached NVM.The study also identifies benefits from adapting traffic distributions between DRAM and NVM.Source: §1 · p. 2
• A roofline model examines power efficiency across read-only traffic distributions; a balanced DRAM/NVM distribution could improve performance and power efficiency.The paper reports that adapted traffic distributions can improve power and energy efficiency at different arithmetic intensities.Source: §1 · p. 2

2 ARCHITECTURE

• Each Purley socket has two memory controllers and six channels connected to DRAM DIMMs and NVDIMMs; platform NVDIMMs have capacities of 128, 256, or 512 GB.The 128-GB NVDIMM has the lowest cost per byte among the listed capacities.Source: §2 · p. 2
• NVDIMMs use DDR-T and 256-byte media transfers behind 64-byte CPU requests; modifying fewer than 256 bytes still incurs the energy and overhead of a 256-byte write.A small DRAM buffer caches media data, while an NVDIMM controller translates addresses and caches the address indirection table.Source: §2 · pp. 2–3
• DRAM can cache only PMM attached to the same memory controller, so channel-sharing configurations cannot achieve the aggregate bandwidth of DRAM and NVM.In Memory mode, this locality restriction also means DRAM on one socket cannot cache accesses to PMM on the other.Source: §2 · pp. 2–3
• App Direct exposes DRAM and PMM separately, while Memory mode presents PMM as NUMA nodes and uses DRAM as a transparent cache; Memory mode requires no application changes.App Direct can require applications to select data structures and change allocation sites, though exposing PMM as kernel NUMA nodes permits standard NUMA controls.Source: §2 · pp. 2–3
• PMM namespace metadata occupies 64 bytes per page, reaching 24 GB even with the platform’s smallest 128-GB NVDIMMs; storing it in PMM can severely impact performance.Users can store the metadata in DRAM or PMM.Source: §2 · p. 3
• Optane modules include DRAM caches and address-translation and write-leveling controllers that consume active and static power; acceptable PMM performance also requires some DRAM to remain powered.The study stores namespace metadata in DRAM to avoid significant slowdown and evaluates power and energy across memory configurations.Source: §2 · p. 3

3 METHODOLOGY

• The testbed has 192 GB DRAM and 1.5 TB NVM, with 168 GB DRAM available to applications after storing PMM page metadata in DRAM.The bus runs at 2400 GT/s, supporting a stated peak bandwidth of 19.2 GB/s per channel or 230.4 GB/s per platform.Source: §3 · p. 3
• MLC measures latency and bandwidth, STREAM includes an accumulation kernel, and PCM collects memory and CPU power and energy per socket.The authors also develop benchmarks for roofline, power-line, and arch-line models across DRAM/NVM traffic distributions.Source: §3 · p. 3
• The evaluation uses BFS, betweenness centrality, triangle counting, connected components, and PageRank from GAP and Ligra on generated graphs.Experiments generally use 96 threads across two sockets; single-socket runs use 48 threads and one socket’s memory to eliminate NUMA effects.Source: §3 · pp. 3–4
• Latency measurements vary sequential and random read-buffer sizes, against capacities of 96 GB DRAM per socket and 192 GB across the system.The methodology includes eight memory configurations.Source: Figure 3 · p. 4, §3 · p. 4

4.1 Latency

• PMM read latency is 174 ns sequentially and 302 ns randomly, versus 79 ns and 87 ns for DRAM; NUMA effects increase latency by 1.2 to 1.8 times.MemoryMode-local stays close to DRAM-local when data fits in one socket, but its latency approaches remote PMM once data exceeds that capacity.Source: §4.1 · p. 4

4.2 Bandwidth

• 104 GB/s DRAM bandwidth exceeds PMM's 39 GB/s for sequential reads, while PMM write bandwidth is 12.1 GB/s; mixed reads and writes widen the DRAM-PMM gap to 4.1–12.5 times.Local PMM beats remote DRAM for read-only traffic above 14 threads, but remote PMM bandwidth can fall below 1 GB/s beyond three threads.Source: Figure 4 · p. 5, §4.2 · p. 5
• At 24 threads, MemoryMode-local with non-temporal writes reaches only 47% and 64% of DRAM-local bandwidth in the two tested write patterns.Without non-temporal writes, MemoryMode sustains 80 to 88% of DRAM bandwidth in the other reported patterns; cache conflicts also reduce MemoryMode-local bandwidth for data smaller than single-socket DRAM capacity.Source: §4.2 · p. 5

4.3 Power and Energy

• PMM configurations reduce dynamic power by 4–29 times versus DRAM and can deliver up to 47% higher power efficiency on read-only workloads.PMM power efficiency is 20% lower than DRAM-local for write-only workloads.Source: Figure 6 · p. 6, §4.3 · p. 6
• For the 1 read : 1 write workload, static energy accounts for 95% of total memory energy, limiting the total-energy benefit of low-dynamic-power PMM.DRAM DIMMs cannot be powered off, and longer PMM execution times let static energy dominate for bandwidth-bound workloads.Source: Figure 8 · p. 6, §4.3 · p. 7

4.4 Graph Applications

• 2–18x slowdown occurs for graph applications using PMM without DRAM caching on inputs smaller than DRAM capacity; MemoryMode stays near DRAM performance.Triangle counting is less sensitive than BFS, and DRAM-PMM interleaving improves on PMM alone by about 2x.Source: Figure 9 · p. 7, §4.4 · p. 7
• At the largest graph input, the MemoryMode performance gap over PMM and DRAM-PMM interleaving narrows to 2x–6x from as much as 18x on small inputs.DRAM-PMM interleaving adds 192 GB of capacity relative to MemoryMode, allowing some applications to trade a slight slowdown for a larger problem.Source: §4.4 · pp. 7–8, Figure 10 · p. 8, Figure 11 · p. 8
• Using two sockets can slow some graph applications, while compute-intensive triangle counting can benefit from their added throughput.Dual-socket execution gives less than 20% speedup on small inputs; low-compute-intensity BFS is more sensitive to remote-socket overhead.Source: Figure 12 · p. 8, §4.4 · p. 8

5.1 Bandwidth Spilling

• Bandwidth spilling allocates blocks round-robin across sockets, spilling from DRAM to NVM when DRAM is exhausted to distribute traffic and address inter-socket bottlenecks.Its analytical model uses the DRAM traffic portion and peak DRAM and PMM bandwidths to guide achievable bandwidth.Source: §5 · p. 9, §5.1 · p. 9

5.2 Write Isolation

• Write isolation places write-intensive data structures in DRAM and splits persistent structures across PMM on two sockets, with thread affinity to mitigate inter-socket accesses.Fine-grained policies require application modifications; their performance trade-off against Memory mode depends on data size and the complexity of explicit data management.Source: §5 · p. 9, §5.2 · p. 9

5.1 Bandwidth Spilling

• At data sizes above 1 TB, bandwidth spilling sustains 76–97 GB/s, about 2x the best Memory mode performance, and supports 1.54 TB—20% more data than Memory mode.Figure 13 compares App Direct bandwidth under spilling with Memory mode optimizations; the model assumes read traffic because PMM writes dramatically lower performance.Source: §5.1 · p. 9, Figure 13 · p. 9

5.2 Write Isolation

• At the largest data size, write isolation sustains 83 GB/s versus 27 GB/s for Memory mode, bypassing dirty-cache-eviction throttling from writes to PMM.Memory mode achieves 46–89% of DRAM bandwidth at medium data sizes; write isolation begins outperforming it above 32 GB.Source: §5.2 · pp. 9–10, Figure 14 · p. 10
• Write isolation reduces energy cost by up to 3.9x versus MemoryMode and 8.4x versus PMM.Figure 15 partitions energy per gigabyte into CPU and memory energy; MemoryMode’s CPU share rises from 55% at small data sizes to 74% at the largest, attributed mostly to prolonged execution.Source: §5.2 · p. 10, Figure 15 · p. 10

5.3 Tra￿c Distribution

• For memory-intensive workloads with arithmetic intensity below 21, memory power rises with DRAM traffic share; at intensity 2–3, all-DRAM traffic uses about 95 W versus 54 W with all traffic to NVM.Figure 16 plots one-socket memory power against traffic distribution and arithmetic intensity; the 54 W includes static power from DRAM DIMMs.Source: §5.3 · p. 10, Figure 16 · p. 10
• At low compute intensity, directing 10% of traffic to NVM saves up to 40 W versus all-DRAM traffic, which uses over 480 W on the platform.Figure 17a reports total platform power; the all-DRAM distribution shows no power peak, possibly due to platform power capping.Source: §5.3 · p. 11, Figure 17 · p. 11
• Performance becomes compute-limited at arithmetic intensity 20–21; above 21, mixed DRAM–NVM traffic distributions can outperform full distribution to either memory, while 10% or 90% NVM traffic yields higher energy efficiency.The exploration uses read traffic only because writes to NVM severely reduce performance.Source: §5.3 · pp. 10–11, Figure 17 · p. 11

• Earlier prototype work explored STT-RAM, RRAM, and phase-change memory; Optane DC PMM is described as the first commercially available hardware providing enormous memory capacity.Source: §6 · p. 11
• Earlier heterogeneous-memory studies placed performance-critical data statically or at runtime, while many evaluations relied on emulators or cycle-accurate simulators because hardware was unavailable.Source: §6 · p. 11
• Prior Optane DC PMM work studied in-memory databases and NVM-specific file systems [13], and optimized Galois to mitigate NUMA effects in Memory mode [6].Source: §6 · p. 11
• This study adds evaluation of power and energy efficiency across memory configurations and fine-grained traffic control between NVM and DRAM.Source: §6 · p. 11

• DRAM-cached NVM could bring reasonable performance for large graphs, while fine-grained policies can yield higher bandwidth and lower energy cost than coarse-grained configurations.The evaluation covers Intel Optane byte-addressable NVM and provides guidance for selecting among eight memory configurations.Source: §7 · pp. 11–12