PaperlayerView exact revision on arXiv

Source-linked AI summary

Basic Performance Measurements of the Intel Optane DC Persistent Memory Module

Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R. Dulloor, Jishen Zhao, Steven Swanson

arXiv:1903.05714v3cs.DCcs.PF

TL;DR

Before Optane DC PMMs, researchers lacked detailed knowledge of how real NVDIMMs would behave, perform, and integrate into architectures. This paper measures Optane DC PMM across fundamental characteristics, persistence costs, access patterns, system modes, file systems, and user-space applications. The experiments find that Optane DC creates a distinct memory tier, accelerates storage applications as integration increases, and can provide greater gains through NVMM-aware software, although application-managed persistence requires crash-consistency work.

  • Problem

    Earlier research lacked detailed knowledge of real NVDIMM behavior, performance, and architectural integration.

  • Method

    The paper conducts broad performance experiments measuring Optane DC PMM across system modes, file systems, and NVMM-aware user-space applications.

  • Results

    Optane DC creates a new memory tier between DRAM and storage, while deeper storage-stack integration and NVMM-aware applications improve performance.

  • Takeaways & Limitations

    Cached Optane DC can expand memory capacity with comparable performance for many explored applications, while uncached Optane DC can accelerate storage workloads.

  • Takeaways & Limitations

    Application-managed Optane DC persistence requires the application to ensure crash consistency rather than relying on the file system.

Abstract

from arXiv · show

Scalable nonvolatile memory DIMMs will finally be commercially available with the release of the Intel Optane DC Persistent Memory Module (or just "Optane DC PMM"). This new nonvolatile DIMM supports byte-granularity accesses with access times on the order of DRAM, while also providing data storage that survives power outages. This work comprises the first in-depth, scholarly, performance review of Intel's Optane DC PMM, exploring its capabilities as a main memory device, and as persistent, byte-addressable memory exposed to user-space applications. This report details the technologies performance under a number of modes and scenarios, and across a wide variety of macro-scale benchmarks. Optane DC PMMs can be used as large memory devices with a DRAM cache to hide their lower bandwidth and higher latency. When used in this Memory (or cached) mode, Optane DC memory has little impact on applications with small memory footprints. Applications with larger memory footprints may experience some slow-down relative to DRAM, but are now able to keep much more data in memory. When used under a file system, Optane DC PMMs can result in significant performance gains, especially when the file system is optimized to use the load/store interface of the Optane DC PMM and the application uses many small, persistent writes. For instance, using the NOVA-relaxed NVMM file system, we can improve the performance of Kyoto Cabinet by almost 2x. Optane DC PMMs can also enable user-space persistence where the application explicitly controls its writes into persistent Optane DC media. In our experiments, modified applications that used user-space Optane DC persistence generally outperformed their file system counterparts. For instance, the persistent version of RocksDB performed almost 2x faster than the equivalent program utilizing an NVMM-aware file system.

How to Use this Document

The document directs readers to sections according to their interests and marks key facts or findings as “observations.” It also notes that limited public information about Intel’s design may constrain answers about the data or methodology.

  • Readers interested in Optane DC memory’s basic characteristics should focus on Section 3.
  • Application developers using large amounts of memory should read Section 4 to assess Optane DC memory as a large main-memory device.
  • File-system and storage researchers should consult Section 5 to examine Optane DC memory’s effects on file systems.
  • The document identifies key facts or findings in boxed “observations.”These observations are generally selected for broad usefulness or because they represent key device statistics.
  • Many aspects of Intel’s design remain unavailable publicly, limiting the information the authors may provide about data or methodology.

Versions

The document progresses through versions from an initial release with embargoed results to a full release and then an updated release with clarifications, experiments, and errata. The update corrected the Memory Mode cache-line size from 4 KB to 64 bytes.

  • The document identifies itself as Version 1.0.1.
  • Version 0.1.0, dated 3/13/2019, was the initial release and still contained results under embargo.
  • Version 1.0.0, dated 4/3/2019, was the first full release, with all results released from embargo.
  • The updated release corrected the Memory Mode data-movement cache-line size from 4 KB to 64 bytes.

Executive Summary

This report evaluates Optane DC PMMs as expanded main memory, persistent storage, and user-space persistent memory, establishing their performance trade-offs and practical benefits. Results show lower raw bandwidth than DRAM but substantially greater capacity and strong gains when software is optimized for persistent-memory access.

  • Scope and contribution: Optane DC PMMs provide scalable, byte-addressable nonvolatile memory, and this report evaluates their behavior across main-memory, storage, and persistent-memory configurations.The report aims to help architecture and systems researchers develop intuition about the technology and presents an executive snapshot alongside broader measurements.
  • Basic device characteristics: Optane DC’s 256 B internal block size makes smaller accesses inefficient and causes write amplification for stores below that granularity.Smaller accesses have the same latency as a 256 B access, while smaller stores write at least 256 B, consuming bandwidth, energy, and device endurance.
  • Optane DC as main memory: In Memory Mode, a DRAM cache largely preserves performance for small-footprint applications, while larger workloads may slow relative to DRAM but fit in substantially more memory.A socket can hold 192 GB of DRAM versus 1.5 TB of Optane DC, enabling workloads that cannot fit solely in DRAM.
  • Optane DC as persistent storage: In App Direct mode, optimized file systems improve persistent-storage performance, with NOVA outperforming legacy file systems and Kyoto Cabinet improving by almost 2× under NOVA-relaxed.Optane DC improves basic storage performance over SATA and Optane SSDs, while NOVA provides better scalability.

1 Introduction

This report addresses the lack of detailed evidence about real nonvolatile DIMMs by measuring Intel Optane DC PMM performance across fundamental characteristics, system roles, and applications.

  • Prior NVDIMM research proposed programming models, file systems, libraries, and applications without detailed knowledge of real-device behavior.
  • The report provides measurements exploring Optane DC PMM’s fundamental performance and system integration now that the devices are available.
  • The study examines basic device characteristics, persistence instructions, application performance as memory and storage, system software, and custom software.
  • Experiments cover a wide range of applications, benchmark suites, and microbenchmarks, using over 330 hours of machine time.

2 Background and Methodology

The paper characterizes Optane DC PMM hardware, persistence mechanisms, operating modes, evaluation platform, and configurations for memory and storage experiments.

  • Background: Optane DC PMM creates a byte-addressable, persistent memory tier between volatile DRAM and block-based storage.
  • Background: The iMC lies within Intel’s ADR domain, but processor caches do not; stores become persistent after reaching the iMC.
  • Configurations: Memory mode uses adjacent DRAM as a direct-mapped write-back cache and exposes a larger volatile memory region to the operating system.
  • Configurations: App Direct mode exposes Optane DC directly as persistent storage, with file systems and applications using load/store access and ordering facilities.
  • Evaluation: The evaluation uses a dual-socket platform and examines Optane DC as main memory and as persistent storage across six system configurations.

3 Basic Performance

Optane DC has higher latency, asymmetric and access-pattern-sensitive bandwidth, and distinctive scaling behavior; sequential and sufficiently large accesses improve utilization.

  • Read latency: 305 ns is the random-read latency of Optane DC memory, about 3× slower than local DRAM.
  • Read latency: 2× better latency occurs for sequential Optane DC accesses, indicating merging of adjacent requests into a single 256-byte access.
  • Maximum bandwidth: 39.4 GB/sec is the maximum read bandwidth and 13.9 GB/sec the maximum write bandwidth across six interleaved Optane DC PMMs.
  • Maximum bandwidth: Optane DC reads scale with thread count to maximum throughput at 17 threads, while four threads saturate write bandwidth.
  • Access size and bandwidth: 256 byte or larger accesses are preferred to fully utilize Optane DC device bandwidth.
  • Performance under load: Optane DC is more vulnerable than DRAM to mixed read-write workloads and irregular access patterns.
  • Performance under load: 4× higher bandwidth occurs for sequential access because request merging avoids write amplification from the 256-byte media access size.

4 Optane DC as Main Memory

As main memory, cached Optane DC preserves performance for small footprints but larger or bandwidth-saturating workloads can slow relative to DRAM.

  • SPEC CPU: Cached Optane DC is almost as fast as DRAM for relatively small memory footprints, showing that the DRAM cache can hide Optane DC’s limitations.
  • SPEC CPU: Uncached Optane DC is significantly slower in rate tests because multiple workload copies saturate memory bandwidth.
  • PARSEC: PARSEC performance differences appear only for facesim, fluidanimate, raytrace, freqmine, and x264; other footprints fit in the last-level cache.
  • PARSEC: Optane DC’s lower bandwidth can saturate in real-world multithreaded applications, creating overhead when used uncached as main memory.

5 Optane DC as Persistent Storage

Optane DC is evaluated as persistent storage beneath conventional and NVMM-aware file systems, with DAX and non-DAX configurations compared across memory and SSD-backed devices. Real Optane DC latency exposes file-system design trade-offs, while caching, copy-on-write, and in-place updates strongly affect performance.

  • Configurations and goals: The evaluation compares seven file-system configurations across three memory configurations and two compatible SSD configurations.The study examines conventional, DAX-enabled, and purpose-built NVMM file systems using multiple storage substrates.
  • File operation latency: Non-DAX file systems have the longest latencies on SATA SSD, but their write performance is best on Optane SSD, indicating unoptimized write and fsync paths for memory storage.This comparison comes from the measured file-operation latency experiment.
  • File operation latency: NOVA and NOVA-Relaxed outperform conventional file systems for create and append operations, while NOVA’s page-level copy-on-write increases write latency, especially for 512-byte writes.NOVA-Relaxed recovers performance by allowing in-place file-data writes while retaining metadata consistency.
  • File operation latency: Optane DC raises memory-storage operation latencies by 3.3% to 156% relative to DRAM, depending on the file system.The increase spans XFS-DAX at the low end and NOVA at the high end.
  • File operation latency: Non-DAX file systems can outperform DAX systems on Optane DC because the DRAM page cache hides read latency.DAX systems access persistent memory directly, whereas non-DAX systems retain the page-cache layer.

5.2 FIO Bandwidth

FIO and Filebench measure bandwidth and application-like throughput across file systems, storage types, and workloads. NOVA-family systems are strongest when workloads contain substantial write traffic, while small random writes make real Optane DC differ sharply from DRAM emulation.

  • FIO methodology: FIO measures sequential and random 4 KB reads and writes for one to sixteen threads using 512 MB per-thread files.Write workloads issue fsync() after every 4 KB of file data and run for 30 seconds.
  • FIO results: NOVA and NOVA-Relaxed generally outperform other file systems and scale better in 4 KB read and write bandwidth tests.DAX-enabled systems benefit from bypassing the page cache, while NOVA-family systems show the strongest scaling.
  • Filebench results: Small random writes can cause drastic differences between DRAM emulation and real Optane DC, especially for NOVA and NOVA-Relaxed on fileserver.The fileserver workload generates many small random writes that cause write amplification on Optane DC PMMs.
  • Filebench results: Filebench workloads show similar read-intensive performance across file systems, but NOVA and NOVA-Relaxed lead when write traffic increases.The evaluated workloads include fileserver, varmail, webproxy, and webserver patterns.

5.4 RocksDB

The RocksDB experiment examines a write-intensive workload whose LSM-tree and WAL design still depend heavily on sync operations. NOVA-type file systems improve throughput by making sync fast, while user-space persistence can bypass kernel and file-system overhead.

  • Design and workload: RocksDB uses an LSM-tree with a DRAM memtable, write-ahead log, SSTables, and compaction to organize persistent key-value data.Memtables absorb inserts before flushing, while compaction merges sorted files across levels.
  • Results: Frequent syncs impose substantial flushing costs on non-NVMM file systems, while NOVA-type systems improve performance through fast synchronization.The comparison is conducted on a write-dominant RocksDB workload.
  • Design and workload: The benchmark measures SET throughput with 20-byte keys, 100-byte values, and database synchronization after every SET operation.This workload makes WAL appends and sync operations important to NVMM file-system performance.

5.6 Kyoto Cabinet

Kyoto Cabinet updates a single memory-mapped database file using load/store operations, undo logging, and msync persistence. Its write-dominant HashDB benchmark favors NOVA-Relaxed, whose in-place updates benefit workloads concentrated in one B+tree file.

  • Design and workload: Kyoto Cabinet memory maps database metadata, updates it with load/store instructions, persists changes with msync, and uses write-ahead logging for failure atomicity.The database is stored in a single file.
  • Results: NOVA-Relaxed performs best on Kyoto Cabinet because its fast sync mechanism improves write-dominant workloads.This result parallels the RocksDB and Redis experiments.
  • Design and workload: HashDB transactions append undo-log records, update target records in place, flush changes with msync, and truncate the WAL during commit.The benchmark runs one million random SET transactions with 8-byte keys and 1024-byte values.
  • Related application result: MySQL TPC-C throughput remains surprisingly stable across file systems and storage devices because its buffer pool and checkpointing avoid frequent persistence writes.The experiment uses ten warehouses and five-minute runs.
  • Related application result: NOVA-Relaxed’s in-place update optimization significantly improves SQLite when randomly distributed writes modify a single B+tree file.SQLite’s benchmark inserts 100-byte values into one table using WAL mode.

5.9 LMDB

This section evaluates database workloads including LMDB and MongoDB under write- and read-dominant access patterns. The reported observations emphasize file-system choice, caching, and workload characteristics.

  • LMDB memory-maps its entire database and uses copy-on-write with frequent msync calls to provide atomicity.
  • LMDB throughput is measured with synchronous sequential SET operations using 16-byte keys and 96-byte values.
  • MongoDB evaluation uses YCSB-A with 50% reads and 50% updates and YCSB-B with 95% reads and 5% updates.
  • Applications generally perform slower on real Optane DC than on emulated persistent memory, with the gap growing as the file system becomes faster.
  • Block-oriented file systems are not necessarily slower than DAX file systems, particularly for read-oriented workloads, because DRAM page caching can hide Optane DC read latency.
  • Native NVMM file systems such as NOVA and NOVA-Relaxed generally outperform adapted file systems, especially for applications with frequent sync operations.

6 Optane DC as Persistent Memory

Optane DC persistent memory exposes persistent storage through load/store access, either through NVMM-aware file systems or application-level libraries. Across Redis, RocksDB, MongoDB, PMemKV, and WHISPER, deeper integration often improves performance, though Optane-specific latency remains visible.

  • Optane DC persistent memory lets applications access persistent storage with loads and stores, using cache-line flushes to make writes persistent.
  • 6.1 Redis-PMEM: Redis using PMDK for user-space persistence outperforms Redis backed by NOVA-Relaxed, showing the benefit of bypassing the file system.
  • 6.2 RocksDB-PMEM: RocksDB with a persistent crash-consistent memtable outperforms volatile memtables backed by write-ahead logging, achieving a 73% gain on PM-Optane.
  • 6.3 MongoDB-PMEM: PMem storage provides similar MongoDB performance to WiredTiger for both write-dominant YCSB-A and read-dominant YCSB-B workloads.
  • 6.4 PMemKV: PMemKV on PM-Optane increases latency by 2% to 15% for sequential reads and 45% to 87% for random reads relative to PM-LDRAM.
  • 6.6 Summary: Performance improves as Optane DC becomes more integrated into the storage stack, and application-level software modifications may provide significant benefits.

7 Conclusion

The conclusion characterizes Optane DC PMM as a distinct memory tier between DRAM and storage. It reports useful cached-mode performance and faster-than-storage access while emphasizing that understanding the technology remains incomplete.

  • Optane DC PMM creates a new memory tier between DRAM and storage with performance properties different from currently deployed media.
  • Cached-mode Optane DC provides comparable performance to DRAM for many evaluated applications while greatly increasing total available memory.
  • The report states that further work is necessary because important questions about these new memory devices remain unanswered.

A Observations

The observations show that Optane DC performance depends strongly on access pattern, access size, caching, and software integration. Sequential and cached access can mitigate costs, while writes, random access, and persistent-data workloads expose larger gaps versus DRAM.

  • 305 ns: random Optane DC load latency is about 3× slower than local DRAM.
  • 2×: Optane DC latency is significantly better for sequential access, consistent with merging adjacent requests into a 256-byte access.
  • For small accesses, clflushopt and clwb outperform clflush and non-temporal stores, demonstrating the utility of persistent-memory ISA support.
  • 39.4 GB/sec read and 13.9 GB/sec write bandwidth are achieved with six interleaved Optane DC PMMs.
  • Optane DC reads scale with thread count to 17 threads, while four threads suffice to saturate write bandwidth.
  • 256-byte or larger accesses are preferred to fully utilize application-level Optane DC bandwidth.
  • Optane DC is more sensitive than DRAM to access patterns and is vulnerable to workloads mixing reads and writes.
  • 4×: Optane DC bandwidth is significantly higher for sequential access because merged requests avoid the write-amplification cost of 256-byte accesses.
Loading 1903.05714v3…