PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Guide to the Behavior and Use of Scalable Persistent Memory

Jian Yang, Juno Kim, Morteza Hoseinzadeh, Joseph Izraelevitz, Steven Swanson

arXiv:1908.03583v1cs.DCcs.PF

TL;DR

Prior persistent-memory research lacked detailed evidence about how real scalable NVDIMMs behave. This paper measures Intel’s 3D XPoint DIMM across micro- and macro-level workloads, derives programming guidelines, and reevaluates software and emulation results. It finds that real-device behavior is more nuanced than expected and that emulation results do not have a simple relationship to real hardware.

  • Problem

    Before Intel’s 3D XPoint DIMM, researchers lacked detailed knowledge of real scalable NVDIMM capabilities, limitations, and behavior when designing persistent-memory systems.

  • Method

    The paper evaluates 3D XPoint DIMMs with microbenchmarks, benchmarks, and applications, then uses the results to develop guidelines and compare real hardware with emulation methods.

  • Results

    Real 3D XPoint performance depends strongly on access size, access type, pattern, and concurrency, while emulated and real persistent-memory results differ substantially and lack a simple relationship.

  • Takeaways & Limitations

    Programmers should use concrete device-specific guidelines, and prior emulation-based performance conclusions should not be assumed to represent real 3D XPoint hardware.

  • Takeaways & Limitations

    Several guidelines depend on current 3D XPoint microarchitectural characteristics that may change in future implementations.

Abstract

from arXiv · show

After nearly a decade of anticipation, scalable nonvolatile memory DIMMs are finally commercially available with the release of Intel's 3D XPoint DIMM. This new nonvolatile DIMM supports byte-granularity accesses with access times on the order of DRAM, while also providing data storage that survives power outages. Researchers have not idly waited for real nonvolatile DIMMs (NVDIMMs) to arrive. Over the past decade, they have written a slew of papers proposing new programming models, file systems, libraries, and applications built to exploit the performance and flexibility that NVDIMMs promised to deliver. Those papers drew conclusions and made design decisions without detailed knowledge of how real NVDIMMs would behave or how industry would integrate them into computer architectures. Now that 3D XPoint NVDIMMs are actually here, we can provide detailed performance numbers, concrete guidance for programmers on these systems, reevaluate prior art for performance, and reoptimize persistent memory software for the real 3D XPoint DIMM. In this paper, we explore the performance properties and characteristics of Intel's new 3D XPoint DIMM at the micro and macro level. First, we investigate the basic characteristics of the device, taking special note of the particular ways in which its performance is peculiar relative to traditional DRAM or other past methods used to emulate NVM. From these observations, we recommend a set of best practices to maximize the performance of the device. With our improved understanding, we then explore the performance of prior art in application-level software for persistent memory, taking note of where their performance was influenced by our guidelines.

1 Introduction

The paper evaluates Intel’s commercially available 3D XPoint DIMM to replace assumptions about persistent memory with measurements of real hardware. It characterizes device behavior, develops programming guidelines, and tests their implications for persistent-memory software.

  • 3D XPoint DIMMs provide commercially available, scalable, byte-addressable storage that survives power outages.
  • The authors characterize 3D XPoint DIMMs with microbenchmarks, benchmarks, and applications, finding that many prior NVDIMM assumptions are incorrect.
  • 3D XPoint performance depends strongly on access size, read-versus-write type, access pattern, and concurrency, unlike the simpler “slower, persistent DRAM” model.
  • The paper derives concrete programmer guidelines, examines their consequences, and applies them to NVMM-aware software packages and prior published results.
  • Emulation methods using custom hardware, NUMA effects, or persistent DRAM are inaccurate, so their results may not reflect real 3D XPoint performance.

2 Background and Methodology

The paper explains 3D XPoint DIMM architecture, persistence mechanisms, operating modes, and the experimental configurations used to evaluate persistent-memory behavior.

  • The 3D XPoint DIMM is a scalable NVDIMM that offers higher density and persistence than DRAM, with lower latency and higher read bandwidth than externally connected storage devices.
  • The ADR domain includes the integrated memory controller’s write-pending queues, ensuring updates reaching those queues are flushed to 3D XPoint media during a power failure.
  • The DIMM’s 256-byte media granularity converts smaller requests into larger accesses, causing write amplification, while the XPBuffer merges adjacent writes.
  • In Memory mode, 3D XPoint expands volatile main memory while DRAM acts as a transparent 64 B direct-mapped cache.
  • In App Direct mode, 3D XPoint appears as separate persistent memory that applications and file systems can access with load and store instructions.
  • Experiments use App Direct namespaces, with six interleaved local DIMMs as the Optane baseline and additional remote, DRAM, and non-interleaved configurations.

3 Performance Characterization

The paper systematically characterizes 3D XPoint DIMM performance across latency, tail behavior, bandwidth, access patterns, access sizes, concurrency, and interleaving. The results show behavior substantially more complex than slightly slower persistent DRAM, motivating targeted measurement and programming guidance.

  • Measurement Approach: The study combines LATTester with broad parameter sweeps and targeted follow-up experiments, collecting over ten thousand data points.The measurements cover access patterns, operations, access and stride sizes, power budget, NUMA configuration, concurrency, and interleaving.
  • Latency: 3D XPoint read latency is 2×–3× higher than DRAM, with an 80% random-vs-sequential gap versus 20% for DRAM.For writes, cached writes with clwb have lower latency than non-temporal stores, while DRAM and 3D XPoint show similar store latency.
  • Tail Latency: Rare 3D XPoint latency spikes affect 0.006% of accesses and reach 2 orders of magnitude above common-case access latency.Outliers become more numerous for stores concentrated in a hotspot, while the number of outliers decreases as hotspot size increases and they do not occur for DRAM.
  • Bandwidth: Interleaving across six DIMMs improves peak read and write bandwidth by 5.8× and 5.6×, respectively, but introduces a performance dip at 4 KB from iMC contention.The speedups match the number of DIMMs and expose per-DIMM bandwidth limitations.
  • Latency Under Load: Under load, 3D XPoint reaches its queuing-related latency wall earlier than DRAM and shows greater sensitivity to sequential versus random access patterns.DRAM performance remains comparatively consistent between those access patterns.

4 Comparison to Emulation

Real 3D XPoint behavior differs substantially from persistent-memory emulation, and those differences can reverse software-design conclusions. The paper therefore compares emulation methods with real hardware and revisits RocksDB results.

  • 4.1 Microbenchmarks in Emulation: The study evaluates emulation mechanisms including software, hardware, NUMA-based, and DRAM-persistence approaches against real 3D XPoint.The comparison uses microbenchmarks and a case study of how those differences affect research results.
  • 4.1 Microbenchmarks in Emulation: Emulation methods deviate drastically from real 3D XPoint performance and miss its sequential-access preference and read/write asymmetry.They also produce inaccurate latency and bandwidth estimates.
  • 4.2 Case Study: Optimizing RocksDB: 19% better performance favored fine-grained persistence when DRAM emulated 3D XPoint, but real 3D XPoint reversed the result: FLEX performed 10% better.The RocksDB evaluation used SET throughput with 20-byte keys, 100-byte values, and synchronization after each SET.
  • 4.3 Discussion: Differences between emulated and real persistent memory can alter experimental conclusions, with no simple relationship between emulated and real-hardware results.The paper concludes that future designs should be validated on real hardware and previously discarded ideas reconsidered.

5 Best Practices for 3D XPoint DIMMs

The paper distills its characterization results into four principles for building and tuning 3D XPoint systems, including avoiding small random accesses and using suitable persistence instructions.

  • 5 Best Practices for 3D XPoint DIMMs: The guidelines derive from characterization experiments and are accompanied by implementation examples and case studies.The paper presents four principles for building and tuning 3D XPoint-based systems.
  • 5.1 Avoid small random accesses: Avoid random accesses smaller than 256 B.This is presented as the first guideline for 3D XPoint systems.
  • 5.2 Use non-temporal stores for large writes: Use non-temporal stores for large transfers when possible, while controlling cache evictions.The guideline combines store selection with explicit cache-eviction control.

3. Limit the number of concurrent threads accessing a 3D XPoint DIMM.

Small, poorly localized stores are inefficient on 3D XPoint because updates occur at 256 B granularity. The paper uses EWR and application case studies to motivate locality and logging guidance.

  • 5.1.1 Characterizing small stores: EWR is the ratio of bytes issued by the iMC to bytes written to 3D XPoint media, and it is the inverse of write amplification.Values below one indicate that the DIMM writes more data internally than the application requested.
  • 5.1.1 Characterizing small stores: 64-byte random non-temporal stores achieve EWR 0.25, while 256-byte accesses achieve EWR 0.98 with one thread.Small stores generally have EWR below one.
  • 5.1.1 Characterizing small stores: The XPBuffer combines 64 B accesses into 256 B internal writes, making 256-byte updates efficient when locality is sufficient.Its capacity is approximately 16 KB, and reads also compete for that space.
  • 5.1.1 Characterizing small stores: Avoid small stores; when unavoidable, limit the working set to 16 KB per 3D XPoint DIMM.The guidance follows from the XPBuffer capacity and the efficiency of localized accesses.
  • 5.1.2 Case Study: The NOVA filesystem: NOVA-datalog embeds sub-page write data in the log, converting random writes into sequential writes while preserving atomic file updates.It addresses small, poorly localized log entries and inefficient copy-on-write updates.
  • 5.1.2 Case Study: The NOVA filesystem: NOVA-datalog improves write performance by 7× for 64-byte writes and 6.5× for 256-byte writes, with a slight increase in read latency.Its gains reflect elimination of the discontinuity between log-entry writes and file-data writes.

5.2 Use non-temporal stores for large writes

Persistence-instruction choice and cache-eviction timing substantially affect 3D XPoint performance. The paper compares normal stores, flushes, fences, and non-temporal stores to derive tuning guidance.

  • 5.2 Use non-temporal stores for large writes: Flushing after each 64 B store raises EWR from 0.26 to 0.98 and improves bandwidth for accesses larger than 64 B.Proactive cleaning keeps the access stream sequential by avoiding nondeterministic natural evictions.
  • 5.2 Use non-temporal stores for large writes: Non-temporal stores have lower latency above 512 B and highest bandwidth above 256 B.They bypass the cache and avoid an additional read of the cache line from 3D XPoint memory.
  • 5.2 Use non-temporal stores for large writes: Bandwidth peaks at a 256 B write size, while flushing writes larger than 8 MB after completion degrades performance.The degradation reflects cache-capacity invalidations and higher EWR.
  • 5.2 Use non-temporal stores for large writes: Micro-buffering can improve by using normal stores for small objects with immediate cache-line flushes instead of using only non-temporal stores.The paper evaluates this choice across object sizes in transactional persistent-object updates.

5.3 Limit the number of concurrent threads accessing a 3D XPoint DIMM

Concurrent access to a single 3D XPoint DIMM creates contention in both the XPBuffer and integrated memory controller, reducing bandwidth. Limiting and balancing writers across DIMMs improves performance.

  • Contention mechanisms: Concurrent threads targeting one DIMM reduce store performance because limited XPBuffer buffering and DIMM store capacity increase contention.XPBuffer contention increases evictions and write backs to 3D XPoint media.
  • Contention mechanisms: 0.98 to 0.62: sequential non-temporal-store EWR falls when concurrency rises from one thread to eight threads.The eight threads write to private regions of the DIMM.
  • Contention mechanisms: Increasing the number of DIMMs targeted per thread lowers per-DIMM bandwidth because integrated-memory-controller queue capacity becomes contended.EWR remains close to 1, implicating the iMC rather than XPBuffer capacity in this experiment.
  • Contention mechanisms: The iMC write-pending queue cannot queue more than 256 B from one thread, and slow DIMM draining can cause head-of-line blocking.More writers increase contention and the chance that processors wait for earlier stores to complete.
  • Access distribution: Random 4 KB accesses can cause nonuniform DIMM contention, while larger accesses distribute traffic across DIMMs and 24 KB or 48 KB writes can align perfectly.A 4 KB page buffer may therefore perform poorly in this configuration.
  • Practical guideline: Pinning writer threads to non-interleaved DIMMs balances load, and Multi-NVDIMM-aware NOVA improves FIO bandwidth by 3–34%.The optimization matches threads to NVDIMMs to maximize per-DIMM bandwidth.

5.4 Avoid mixed or multi-threaded accesses to remote NUMA nodes

Remote 3D XPoint memory is especially sensitive to mixed and multithreaded access, with bandwidth gaps far exceeding those observed for DRAM. Systems should distribute work locally and avoid cross-socket traffic.

  • NUMA effects: Remote 3D XPoint memory has substantially larger NUMA effects than DRAM, especially for mixed load-store accesses with multiple threads.Remote memory reaches 59.2% of local read bandwidth and 61.7% of local write bandwidth at optimal thread counts.
  • Application impact: Evenly distributing load across NVDIMMs improves NOVA performance by an average of 17% on FIO.The optimization makes NOVA aware of multiple NVDIMMs.
  • NUMA effects: The local-to-remote bandwidth gap can exceed 30× for 3D XPoint under the same workload, versus at most 3.3× for DRAM.The large gap appears when thread count increases or workloads mix reads and writes.
  • Access patterns: Remote performance drops much faster than local performance as store intensity rises in multithreaded workloads.Single-threaded local and remote bandwidth is similar, but remote performance degrades more quickly with stores.
  • Application impact: Migrating PMemKV to remote 3D XPoint memory causes a 75% performance loss, compared with 8% for DRAM, on a mixed overwrite workload.The remote 3D XPoint loss appears beyond two threads during repeated read-modify-write operations.

6 Discussion

The guidelines provide a practical starting point for tuning 3D XPoint systems, while their portability is bounded by implementation-specific behavior and evaluation in App Direct mode. The analysis also identifies mechanisms that may recur in future persistent memories.

  • Scope and applicability: The guidelines are a starting point for 3D XPoint systems but reflect one implementation of one persistent-memory technology.Their applicability to other technologies and future Intel 3D XPoint versions remains an open question.
  • Scope and applicability: The study evaluates the guidelines only in App Direct mode, where programmers directly access persistent memory rather than relying on Memory Mode’s large DRAM cache.The DRAM cache in Memory Mode mitigates most or all of the studied effects.
  • Broader implications: The analysis offers a roadmap for performance problems involving interleaving, buffering, instruction choice, concurrency, and cross-core interference.These issues may have analogues in future persistent memories even if their exact pathologies differ.
  • Future implementations: Changing XPBuffer or iMC write-pending-queue sizes could reduce the importance of concurrency and 256 B write-granularity guidelines, but larger structures increase ADR energy requirements.Extending ADR to the last-level cache could eliminate the associated power-failure limitation.
  • Future implementations: Battery-backed DRAM would invalidate most guidelines, although non-temporal stores remain more efficient for large transfers under cache-coherency restrictions.This comparison shows that guideline relevance depends strongly on the underlying memory technology.

7 Related Work

Related work spans early evaluations of real 3D XPoint devices and a broad body of persistent-memory programming research. Much prior work developed abstractions and systems before real 3D XPoint hardware was widely available.

  • Real-device studies: Early studies of commercially available 3D XPoint devices examined logging mechanisms and general performance characteristics.The authors expected more results as the devices became widely available.
  • Persistent-memory systems: Persistent-memory programming research has explored transactional abstractions for enforcing a consistent persistent state.These efforts preceded widespread evaluation on real 3D XPoint media.
  • Persistent-memory systems: Prior work also developed NVM data structures for data storage and transaction processing, alongside recent in-memory database research.The related systems span multiple layers of the system stack.

8 Conclusion

The paper characterizes Intel’s 3D XPoint DIMMs through micro- and macro-level benchmarks and extracts actionable programming guidelines. It finds performance between traditional storage and memory, alongside pathologies and pitfalls programmers must avoid.

  • The paper evaluates Intel’s 3D XPoint DIMMs using both micro- and macro-level benchmarks.
  • 3D XPoint DIMMs exhibit performance characteristics between traditional storage and memory devices.
  • The devices present performance pathologies and pitfalls that programmers must avoid, motivating actionable usage guidelines.
Loading 1908.03583v1…