PaperlayerView exact revision on arXiv

Source-linked AI summary

SplitFS: Reducing Software Overhead in File Systems for Persistent Memory

Rohan Kadekodi, Se Kwon Lee, Sanidhya Kashyap, Taesoo Kim, Aasheesh Kolli, Vijay Chidambaram

arXiv:1909.10123v1cs.OScs.PF

TL;DR

Persistent-memory file systems still incur substantial software overhead, especially for writes and appends. SplitFS splits data handling into user space and metadata handling into ext4 DAX, using relink and selectable consistency modes. It reduces overhead by up to 4× versus NOVA and 17× versus ext4 DAX, while improving application performance by up to 2× on reported workloads.

  • Problem

    Persistent-memory file systems retain significant software overhead for operations such as appending 4 KB blocks, despite prior efforts to reduce it.

  • Method

    SplitFS uses a user-space library for data operations, ext4 DAX for metadata, and relink for efficient appends and atomic data operations.

  • Results

    SplitFS reduces software overhead by up to 4× versus NOVA and 17× versus ext4 DAX, and improves application performance by up to 2× on many workloads.

  • Takeaways & Limitations

    SplitFS provides selectable consistency modes and outperforms state-of-the-art PM file systems across micro-benchmarks and several applications.

  • Takeaways & Limitations

    Metadata-heavy workloads can reduce SplitFS performance by less than 15% compared with NOVA and ext4 DAX.

Abstract

from arXiv · show

We present SplitFS, a file system for persistent memory (PM) that reduces software overhead significantly compared to state-of-the-art PM file systems. SplitFS presents a novel split of responsibilities between a user-space library file system and an existing kernel PM file system. The user-space library file system handles data operations by intercepting POSIX calls, memory-mapping the underlying file, and serving the read and overwrites using processor loads and stores. Metadata operations are handled by the kernel PM file system (ext4 DAX). SplitFS introduces a new primitive termed relink to efficiently support file appends and atomic data operations. SplitFS provides three consistency modes, which different applications can choose from, without interfering with each other. SplitFS reduces software overhead by up-to 4x compared to the NOVA PM file system, and 17x compared to ext4-DAX. On a number of micro-benchmarks and applications such as the LevelDB key-value store running the YCSB benchmark, SplitFS increases application performance by up to 2x compared to ext4 DAX and NOVA while providing similar consistency guarantees.

1 Introduction

SplitFS reduces persistent-memory file-system overhead by assigning data operations to a user-space library and metadata operations to ext4 DAX. It targets substantial append overhead while preserving selectable consistency guarantees and improving performance on many workloads.

  • Motivation: 3.5–12.4× overhead remains for file appends across persistent-memory file systems.A 4 KB write to PM takes 671 ns, illustrating that software overhead can dominate the operation.
  • Architecture: SplitFS assigns data operations to a user-space library and metadata operations to the kernel PM file system ext4 DAX.The design intercepts POSIX calls and separates common-case data handling from metadata handling.
  • Results: SplitFS reduces software overhead by up to 4× versus NOVA and 17× versus ext4 DAX.Reads and overwrites use memory mapping with processor loads and stores, while relink minimizes copying and kernel traps for appends.
  • Mechanisms: Relink supports efficient file appends and atomic data operations by moving extents without physical data movement.It is built on ext4 DAX’s swap_extents ioctl and uses ext4 journaling for atomicity.
  • Results: SplitFS outperforms ext4 DAX by up to 2× and NOVA by 10%–2× on several application workloads.These workloads include LevelDB, Redis, and SQLite benchmarks such as YCSB and TPCC.
  • Contributions: SplitFS contributes a split architecture, the relink primitive, an implementation, and experimental evidence across workloads.The system is publicly available and supports multiple consistency modes.

2 Background

Persistent memory offers durable, near-DRAM access through processor loads and stores, but its latency, bandwidth, and endurance differ from DRAM. PM file systems add naming and atomicity while trading off overhead, write I/O, and guarantees.

  • 2.1 Persistent Memory: PM provides durability and performance close to DRAM while being accessed through processor loads and stores.It offers 8-byte atomic stores that become persistent at the PM controller.
  • 2.1 Persistent Memory: PM random-read latency is 3.7× higher than DRAM, with one-third the read bandwidth and close to one-sixth the write bandwidth.These characteristics define PM’s distinct performance profile relative to DRAM.
  • 2.1 Persistent Memory: PM is expected to have limited write endurance of about 10^7 write cycles.Write endurance is an additional constraint beyond latency and bandwidth.
  • 2.2 Direct Access: DAX and mmap() provide low-latency PM access but do not provide naming or atomicity for operations.Applications otherwise must impose their own structure and semantics on mapped bytes.
  • 2.3 PM File Systems: PM file systems trade off software overhead, write I/O, and operation guarantees.NOVA offers strong atomicity guarantees, while PMFS provides weaker data-operation guarantees for better performance on some workloads.
  • 2.3 PM File Systems: Strata can write append data twice when it cannot coalesce private-log data, increasing PM wear-out by up to 2×.All discussed PM file systems still suffer significant write-operation overhead.

3 SplitFS: Design and Implementation

SplitFS divides file-system responsibilities between a user-space library handling data operations and an in-kernel PM file system handling metadata. It combines memory mapping, staging, relink, and selectable consistency modes to reduce overhead while preserving crash consistency.

  • Design Goals: SplitFS aims to reduce software overhead, data copying, and PM write I/O while reusing ext4 DAX and requiring no application modifications.Its design goals also include low implementation complexity and flexible guarantees.
  • Modes and Guarantees: SplitFS provides POSIX, sync, and strict modes, allowing concurrent applications to choose different crash-consistency guarantees while preserving file-system integrity across crashes.Across all modes, appends are atomic when followed by fsync().
  • Design Overview: SplitFS combines its techniques to provide strong guarantees at low software overhead.Table 4 summarizes the techniques and their benefits.
  • Split Architecture: The split architecture handles reads and writes in user space while routing metadata operations such as open() and fsync() to the kernel file system.U-Split serves data operations directly, while K-Split handles metadata operations and crash consistency.
  • Data Operations: Reads and overwrites use memory-mapped regions, while appends and strict-mode overwrites are staged and later relinked into their original files.SplitFS maintains a collection of mappings because one logical file can span the original file and staging files.
  • Relink: Relink logically moves PM blocks from a staging file to a target file without copying data, reducing write amplification during fsync().It atomically transfers extents and uses ext4 journaling; block-aligned operations avoid data copying.

4 Discussion

The discussion identifies implementation trade-offs involving page faults, huge pages, critical-path work, DRAM staging, and legacy applications. These observations define practical conditions under which SplitFS’s benefits are realized.

  • Page Faults: Page faults consume significant open() time because persistent memory’s low device latency makes fault overhead more visible.SplitFS uses MAP_POPULATE to pre-fault mapped pages before later reads and writes.
  • Huge Pages: Huge pages are fragile: fragmentation can prevent their creation, and without them read performance dropped by 50% in many workloads.SplitFS reuses huge pages created at workload startup through its collection-of-mappings technique.
  • Critical Path: Simplifying the critical path is crucial; SplitFS pre-allocates and pre-faults resources, reuses mappings, and performs heavyweight work in background threads where possible.These techniques reduce work during data operations.
  • DRAM Staging: DRAM staging was less effective because copying data from DRAM to PM during fsync() outweighed its lower allocation cost.The authors attribute this to similar PM and DRAM performance.
  • Legacy Applications: LevelDB spent 60–80% of its time in POSIX calls on current PM file systems, while SplitFS reduced that share to 46–50%.Further software-overhead reductions had negligible runtime impact because most remaining time was spent in application code.

5 Evaluation

SplitFS is evaluated against state-of-the-art persistent-memory file systems using microbenchmarks, utilities, databases, and application workloads. The evaluation examines performance, overhead, consistency-related behavior, and resource costs.

  • The evaluation compares SplitFS with ext4 DAX, NOVA, and PMFS across system calls, access patterns, applications, and resource overheads.
  • The study evaluates SplitFS using microbenchmarks, three utilities, two key-value stores, and an embedded database.
  • The experiments also measure the compute and storage overheads incurred when using SplitFS.

5.1 Experimental Setup

SplitFS is evaluated on Intel Optane persistent memory using a range of real-world applications and workloads, with repeated measurements and correctness and recovery checks. The setup includes Redis, LevelDB, SQLite, tar, git, and rsync.

  • Experimental platform: The experiments run on a 2-socket, 96-core machine with 768 GB PMM, 375 GB DRAM, and a 32 MB LLC.
  • Workloads: The evaluation uses Redis, LevelDB, SQLite, tar, git, and rsync as application workloads.
  • Correctness: File-system states produced by ext4 DAX and SplitFS are equivalent across the evaluated microbenchmarks and applications.
  • Recovery: Strict-mode recovery replayed up to 18,000 log entries in about 3 seconds, while a worst-case 2M-entry replay took 6 seconds on emulated PM.

5.4 SplitFS system call overheads

SplitFS accelerates data operations by serving reads and overwrites in user space, while metadata operations remain slower because they also require SplitFS-specific setup and kernel handling. Its append optimizations progressively reduce copying and improve throughput.

  • System call overheads: Writes on SplitFS are 3–4× faster than on ext4 DAX, while metadata operations such as open() and close() are slower.The metadata cost includes setting up user-space data structures and performing the operation on ext4 DAX.
  • Consistency modes: Stronger consistency modes generally increase syscall latency because SplitFS performs additional work such as strict-mode logging.
  • Performance breakdown: SplitFS increases sequential overwrite performance by more than 2× compared to ext4 DAX through user-space processor stores.Staging files and relink have negligible impact because this workload performs no appends.
  • Performance breakdown: Introducing staging files improves append performance by about 2×, while relink increases application throughput by 5× by eliminating fsync() data copies.

5.6 Performance on different IO patterns

Across five file-access patterns, SplitFS is compared with systems providing matching consistency guarantees. It reduces execution time or increases performance most clearly on write-heavy workloads, while strict-mode gains are tied to more efficient logging.

  • POSIX mode: SplitFS reduces ext4 DAX execution times by at least 27% and up to 7.85× in POSIX mode.The largest reported improvement is for appends; read-heavy workloads offer fewer optimization opportunities.
  • Sync mode: Compared to PMFS, SplitFS improves write-workload performance by up to 2.89× and read-workload performance by up to 56% in sync mode.
  • Strict mode: SplitFS improves performance over NOVA by up to 5.8× on random writes in strict mode.The improvement stems from half as many log writes and fence operations as NOVA.
  • Software overhead: File-system software overhead is defined as syscall service time minus time spent accessing data on persistent memory.The evaluation compares overheads across LevelDB YCSB and SQLite TPCC workloads.

5.8 Performance on data-intensive workloads

SplitFS outperforms other PM file systems on data-intensive workloads across POSIX, sync, and strict consistency modes, with the largest gains on write-heavy workloads. Its evaluation covers applications including LevelDB/YCSB, Redis, and SQLite/TPCC.

  • Overall results: Up to 2.70× higher throughput distinguishes SplitFS from other PM file systems on data-intensive workloads with similar consistency guarantees.The workloads include LevelDB with YCSB, Redis with 100% writes, and SQLite with TPCC; throughput is measured in KOps/s.
  • POSIX mode: SplitFS outperforms ext4 DAX across all POSIX-mode workloads, with write-heavy workloads benefiting most.Reported examples include RunA at 2×, LoadA at 89%, LoadE at 91%, and Redis at 27%.
  • POSIX mode: Read-dominated workloads also improve over ext4 DAX, but their performance margin is lower.
  • Sync and strict mode: SplitFS outperforms PMFS and NOVA (relaxed) in sync mode and NOVA (strict) in strict mode on all data-intensive workloads.Write-heavy workloads show the biggest boost; for example, RunA improves by 2× over NOVA (relaxed) and 30% over PMFS in sync mode, and 2× over NOVA (strict) in strict mode.
  • Comparison with Strata: The Strata comparison uses smaller-scale LevelDB YCSB workloads on DRAM-emulated persistent memory rather than Intel DC Persistent Memory.The evaluation injects a 220 ns delay on every read() system call to emulate PM access latency.

5.9 Performance on metadata-heavy workloads

SplitFS performs well on data-intensive workloads but incurs overhead on metadata-heavy utilities, where user-space servicing opportunities are limited. Its resource use includes dedicated metadata memory and a background thread for deferred work.

  • Metadata-heavy workloads: 13% is the maximum overhead SplitFS experiences on metadata-heavy workloads such as git, tar, and rsync.These workloads represent SplitFS’s worst-case scenarios because they provide few opportunities to service system calls in userspace.
  • Resource consumption: SplitFS consumes memory for file-related metadata and CPU time for background metadata-management threads.The background work moves some expensive tasks off the application’s critical path.
  • Memory usage: SplitFS uses a maximum of 100MB for its own metadata and an additional 40MB in strict mode for atomicity data structures.
  • CPU utilization: A background thread can occasionally increase CPU consumption by 100% while handling deferred tasks such as staging-file allocation and file closures.The thread uses one physical machine thread.

6 Related Work

SplitFS builds on prior user-space and persistent-memory file-system designs while combining user-space data handling with ext4 DAX metadata management. Its distinguishing mechanisms are transparent processor-based data access, relink-based append optimization, and selectable strong consistency semantics.

  • Aerie: Aerie also uses a split user-space library and kernel component, but SplitFS uses ext4 DAX for metadata and avoids leases.SplitFS instead makes most operations immediately visible.
  • Strata: SplitFS differs from Strata by relinking appends into shared files instead of copying data from a private log, while relying on ext4 DAX for metadata functionality.Strata writes data to a process-private log and reimplements substantial VFS functionality in userspace.
  • Quill and FLEX: Quill and FLEX share SplitFS’s transparent conversion of read and overwrite POSIX calls into processor loads and stores.Unlike those systems, SplitFS can provide synchronous, atomic operations when required and uses a different append strategy.
  • PM file systems: PM file systems such as SCMFS, BPFS, and NOVA reduce software overhead but still incur kernel traps for operations.SplitFS’s relink resembles BPFS short-circuit paging, but uses ext4 journaling rather than an atomic 8-byte write for atomicity.
  • Kernel bypass: SplitFS follows kernel-bypass designs that separate control and data paths, while retaining a PM file system kernel component for metadata operations.

7 Conclusion

SplitFS uses a split architecture that handles data operations in userspace and metadata operations through ext4 DAX, while offering three consistency modes. The design combines ext4 DAX’s maturity with the performance and guarantees of state-of-the-art PM file systems.

  • Conclusion: SplitFS handles data operations entirely in userspace and routes metadata operations through ext4 DAX.It requires adding only a single system call to ext4 DAX.
  • Conclusion: Three consistency modes let applications choose different guarantees while running concurrently.
  • Conclusion: SplitFS combines ext4 DAX’s maturity and continued development with the performance and strong guarantees of state-of-the-art PM file systems.
Loading 1909.10123v1…