PaperlayerView exact revision on arXiv

Source-linked AI summary

Callosal: Rethinking the Kernel Page Cache for Disaggregated Storage and Memory

James McMahon, Amit Samanta, Seung-seob Lee, Ryan Stutsman, Anurag Khandelwal

arXiv:2610.36347v1cs.OS

TL;DR

Shared file caches in distributed systems lack coherence, complicating applications that mutate shared files. Callosal extends Linux's page cache with cross-machine coherence over RDMA-accelerated shared memory. It outperforms NFS and CephFS on reported workloads and scales on a file-backed key-value store.

  • Problem

    Distributed storage does not keep file caches coherent across compute nodes, complicating applications that concurrently access and mutate shared files.

    Source: §1 · p. 1

  • Method

    Callosal extends the Linux kernel page cache with cross-machine coherence using RDMA and an in-network directory over shared DSM.

    Source: §1 · p. 2

  • Results

    CallosalFS outperforms NFS by around 4.6× on mixed read/write benchmarks when contention is not extreme and scales 1.72× from one to four nodes on a key-value store.

    Source: §8 · p. 13

  • Takeaways & Limitations

    Callosal provides a transparent, consistent, and scalable page cache for filesystems designed for disaggregated block storage.

    Source: §8 · p. 13

    The prototype does not implement complete writeback or durability, though it passes dirty blocks to CallosalFS for writeback.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

Classic distributed filesystems like NFS centralize filesystem operations on a server and use custom, filesystem-specific approaches for managing cache consistency across client nodes. This limits scalability, and it leads to weak consistency, ruling out applications that concurrently access and mutate shared files. Our key insight is that these legacy designs are now too storage-centric; with today's large memories, OS page caches do most of the "heavy lifting" in accessing data, not the filesystem and block storage layers. In response to this, we present Callosal, a page cache for a cluster of machines that is centered on shared disaggregated memory. Rather than serializing operations through a filesystem server and using ad hoc caching like NFS, Callosal is built around the existing Linux page cache paths and is accelerated by RDMA and programmable switches. Callosal delivers the strong consistency that programs expect with local filesystems (like in ext4 and tmpfs) while avoiding the additional network access delays and bottlenecks that host-based solutions incur. We compare Callosal against NFS, CephFS, and local filesystems; often Callosal can provide caching performance similar to local filesystems while also scaling across nodes. For mixed read/write workloads, on average Callosal has 4.6x higher throughput than NFS when contention is not extreme, and it offers strong consistency guarantees like CephFS, while outperforming it. For a file-backed key-value store with 1% writes and uniform access, Callosal scales 1.7x across 4 nodes, while CephFS collapses across nodes.

1 Introduction

Distributed storage leaves shared file caches incoherent, complicating applications that mutate shared files; existing systems trade weak consistency for performance or stronger semantics for coordination overhead. Callosal instead coordinates Linux page caches directly using RDMA and in-network coherence.

1 Introduction

  • Distributed storage lacks coherent file caches across compute nodes, complicating applications that maintain shared mutable state in files.A file-backed key-value store may divide key ownership among workers while allowing each worker to read keys updated by others.Source: §1 · p. 1
  • Growing DRAM capacities make the kernel page cache the primary performance and consistency focus for many workloads, including those backed by remote storage.Linux brings data into the page cache before it is accessed, modified, and synchronized.Source: §1 · p. 1
  • NFS centralizes operations and uses ad hoc client caching for weak consistency, while CephFS provides stronger POSIX-like semantics with coordination overheads that limit shared-file performance.The tradeoff matters for workloads mixing reads and writes to shared files.Source: §1 · p. 1, Figure 1 · p. 1
  • Callosal treats cached file-backed pages as shared objects, aligning distributed coordination with the dominant file-I/O path and avoiding heavyweight filesystem-level protocols.Page-ownership invariants also simplify safe writes to shared block storage and can support a filesystem design without centralization.Source: §1 · p. 2
  • Callosal extends the Linux kernel page cache with cross-machine coherence, coordinating file-backed pages through RDMA and an in-network directory while preserving standard POSIX access.Nodes cache and modify pages independently; coordination is invoked when they concurrently access overlapping file regions.Source: §1 · p. 2
  • Callosal avoids moving generic Linux page-cache structures into DSM, instead limiting remote DSM operations to cases requiring consistency or where metadata semantics cannot avoid communication.Its in-network directory targets active sharers when coordination is necessary.Source: §1 · p. 2

2. A decentralized design for strongly consistent shared

Callosal and CallosalFS provide coherent file caching across four nodes, outperforming NFS and CephFS on reported shared-file workloads while retaining consistent behavior.

2. A decentralized design for strongly consistent shared

  • CallosalFS outperforms NFS by 4.6× on average for DRWOM mixed read/write sharing with 5% writes, and improves throughput by an order of magnitude over CephFS on multiple workloads.On the file-backed key-value store, CallosalFS provides correct behavior unlike NFS and substantially higher throughput than CephFS.Source: §2 · p. 2
  • On controlled page-sharing benchmarks, Callosal matches or exceeds NFS for read-heavy workloads while providing stronger consistency guarantees.The evaluation uses four nodes connected by a Barefoot Tofino switch running the MIND coherence protocol.Source: §2 · p. 2
  • CallosalFS demonstrates coherent file caching on a shared-mutable-state key-value store, with correct behavior unlike NFS and much higher throughput than CephFS.The result is reported for workers sharing access to mutable state.Source: §2 · p. 2

2 Background and Motivation

Distributed filesystems vary in consistency, scalability, and transparency, while Linux page caches are central to file I/O. MIND provides the network-managed coherent memory substrate used by Callosal.

2.1 Distributed Filesystem Designs

  • NFS provides weak close-to-open consistency and can bottleneck at its centralized server, whereas CephFS offers strong consistency using leases but shows throughput collapse on evaluated shared-file workloads with writes.Assise uses client-local NVM and userspace RDMA replication but requires a specialized library and depends on NVM hardware.Source: §2.1 · p. 3

2.2 Page Cache in Modern Filesystems

  • Linux filesystem operations pass through cached pages, so synchronization for operations affecting common file pages is handled primarily in the page cache.For networked filesystems, caching can also avoid slower network traversals.Source: §2.2 · p. 3
  • Local filesystems trade strict serialization for fast reads, and Callosal's hypothesis is that direct synchronization of distributed cached pages can retain scalable reads with strong shared-file semantics.The passage notes that reads may observe partially completed writes in local filesystems; enforcing serialization would require page locks on reads.Source: §2.2 · p. 3

2.3 Cache-coherent Shared Memory in MIND

  • MIND uses a programmable switch as a cache directory to track permissions and sharers, issue invalidations, and fetch data from disaggregated memory when no compute-node cache holds it.Compute nodes request read or write permissions through kernel cache-controller logic.Source: §2.3 · p. 3

3 Callosal Overview

Callosal preserves Linux's local page-cache abstraction while using shared DSM to coordinate pages across machines. It provides real-time ordering, but does not make concurrent reads and writes linearizable.

3 Callosal Overview

  • Callosal integrates with conventional read and write system calls, uses MSI-style invalidations through an in-network directory, and avoids centralized nodes for file state, coherence, or storage management.It preserves most generic Linux page-cache code and interfaces, and its approach is compatible with different remote block stores.Source: §3 · p. 4
  • Callosal's page-cache-centered design aims to combine efficiency and strong consistency by using RDMA and in-network coherence rather than storage-device-centric coordination.Its design is motivated by larger DRAM capacities and advances in low-latency networking and in-network processing.Source: §3 · p. 4

3.1 Operation Overview

  • Callosal preserves each node's local page cache and uses shared DSM memory to synchronize file data across nodes, avoiding remote-access overhead for most locally handled operations.Nodes maintain a consistent view of pages through MIND's shared virtual address space.Source: §3 · p. 4, §3.1 · p. 4
  • A node requests a page through MIND when it is absent locally; writes require a writable copy, and MIND serializes conflicting concurrent writes to the same page.If no node caches the requested page, it is fetched from storage into the shared address space.Source: §3.1 · p. 4

3.2 Consistency Model

  • Callosal guarantees that reads or writes started after a completed write observe its effects, preserving real-time ordering.This is strong consistency, but not linearizability or atomicity for concurrent operations; cross-node writes may interleave at page granularity.Source: §3.2 · p. 5

4 Callosal Design

Callosal extends Linux’s page cache with cross-machine coherence while keeping local filesystem paths largely intact. Its read, write, invalidation, directory, and metadata mechanisms coordinate shared file access at page granularity.

4 Callosal Design

  • Remote locks integrated with local page-cache locks coordinate inode and page access while preserving generic filesystem behavior and deadlock-freedom guarantees.Callosal directs inode synchronization through filesystem code; depending on the operation, synchronization is local or cross-node.Source: §4 · p. 5

4.1 File Reads

  • Cached up-to-date pages are read locally without page locks, while uncached reads acquire synchronized page locks and fetch a read-only copy through MIND.MIND’s SWMR invariant ensures reads issued after a write completes in real time observe its effects.Source: §4.1 · pp. 5–6

4.2 File Writes

  • Writable local pages use synchronized locks to serialize same-page writes; uncached or read-only pages require a remote permission upgrade before writing.File-size synchronization also lets concurrent appends extend the file atomically without interleaving appended data with concurrent writes.Source: §4.2 · p. 6

4.3 Page Invalidation

  • MIND invalidations prevent stale cached pages from being read, and its SWMR property globally linearizes writes to each page.A read beginning after a write processes invalidations before proceeding.Source: §4.3 · p. 6

4.4 In-network Coherence Directory for RDMA

  • MIND’s in-network cache directory orders coherence operations at line rate, resolves page locations, targets relevant sharers, and avoids centralized-coordinator hotspots.The switch’s state can support ownership reconstruction during fault recovery.Source: §4.4 · p. 6

4.5 Metadata Synchronization and Durability

  • Shadow inodes in MIND track important fields, including file size, while inode locks synchronize writes and serialize size changes.Other fields such as mtime can be incorporated, but are not currently included; distributed fsync is not fully implemented.Source: §4.5 · p. 7

5 Implementation

The implementation integrates Callosal with Linux and MIND while retaining standard filesystem interfaces. It supports synchronized page operations, but the prototype leaves complete writeback and durability unimplemented and statically allocates much of its DSM.

5 Implementation

  • Callosal bridges each node’s page cache with MIND’s coherence protocol, using MIND to synchronize local caches rather than as the cache itself.CallosalFS builds on a skeleton filesystem kernel module and comprises about 3,700 lines of C.Source: §5 · p. 7

5.1 Kernel Modifications

  • Callosal adds 250 lines of changes to Linux 4.15, mainly exporting kernel functions and adding callbacks for inode locking and size access.The callbacks allow filesystems to control and synchronize these common kernel paths.Source: §5.1 · p. 7

5.2 CallosalFS Implementation

  • CallosalFS is a synthetic filesystem with initially zeroed file pages that synchronizes page data, inode locks, and file size.It uses spinlocks and read-write semaphores, and tracks shared structures needed for cases such as truncation.Source: §5.2 · p. 7

5.3 MIND Coherence Protocol Modifications

  • Callosal repurposes MIND address bits to locate shadow file and inode pages, and adds per-node permission tracking, locks, and invalidation queues.The additional queues let I/O operations block other operations, unlike MIND’s immediate handling of invalidations.Source: §5.3 · pp. 7–8

5.4 Syncing Remote Operations with Local Reads

  • Concurrent local reads can overlap with a remote refresh and observe an inconsistent mixture of bytes, so readers validate a page version after copying and retry if it changed.The paper reports that retries occur exceedingly rarely in its experiments.Source: §5.4 · p. 8

5.5 Limitations of the Prototype

  • The prototype supports efficient reads, writes, and appends, but complete writeback and durability are unimplemented.Dirty blocks are passed to CallosalFS for writeback.Source: §5.5 · p. 8
  • Much of the prototype’s DSM and per-filesystem state is preallocated or statically divided, so its performance assessment does not include dynamic memory allocation or reallocation.The paper notes that a simple allocator on MIND could address the DSM allocation limitation.Source: §5.5 · p. 8

6 Evaluation

The evaluation compares CallosalFS with distributed and local filesystems across contention levels, read/write mixes, and applications. Results show that CallosalFS often scales across nodes while preserving strong consistency, with performance depending on workload contention.

6 Evaluation

  • The evaluation uses fxmark microbenchmarks and application workloads, comparing CallosalFS with NFS, CephFS, ext4, and tmpfs on a four-server cluster.The setup scales threads on one node before adding compute nodes; ext4 and tmpfs are single-node baselines.Source: §6 · pp. 8–9

6.1 Read Throughput

  • Across DRBL, DRBM, and DRBH, CallosalFS scales well on reads and often matches local-filesystem performance; CephFS collapses when clients read different blocks of one shared file.For DRBH, CallosalFS consistently outperforms NFS within and across nodes.Source: Figure 8 · p. 9, §6.1 · p. 9

6.2 Write Throughput

  • For private-file writes, NFS can outperform CallosalFS, while CallosalFS adds synchronization for strong consistency; CephFS does not scale linearly across nodes.For writes to separate blocks in one shared file, CallosalFS permits cross-node concurrency, whereas CephFS performance collapses as the workload scales.Source: Figure 9 · p. 9, §6.2 · p. 10
  • Under DWOH, CallosalFS reaches 136 Kops across four nodes, far below NFS's 3.2 Mops but up to two orders of magnitude above CephFS.The workload repeatedly overwrites one shared block; CallosalFS's throughput falls as global locking, invalidation, and fetching add cost, while NFS does not serialize conflicting writes.Source: Figure 9 · p. 9, §6.2 · p. 10

6.3 Mixed Read/Write Throughput

  • For DRWOM, CallosalFS outperforms NFS at all read/write ratios, averaging 4.6× higher throughput for 10–30% writes; at 30% writes it performs about 15.6 Mops, while providing strong consistency for reads.Source: Figure 10 · p. 10, §6.3 · pp. 10–11

6.4 Producer/Consumer Queue Application

  • In the producer-consumer queue, CallosalFS performance rises with buffer size and exceeds CephFS by more than an order of magnitude in the best case.Larger buffers can amortize coherence costs and reduce contention as the buffer spans pages.Source: Table 2 · p. 11, §6.4 · p. 11

6.5 Key-Value Store Application

  • With 1% writes and uniform access, CallosalFS reaches 1.72× its single-node throughput at four nodes, while CephFS throughput collapses across nodes.NFS's higher throughput reflects stale query results rather than correct key-value-store behavior.Source: Figure 11 · p. 11, §6.5 · p. 12

7 Related Work

Related systems pursue distributed memory or low-latency storage and filesystems through different interfaces, hardware, and consistency granularities. Callosal is distinguished by its kernel-space page-cache integration and use of commodity RDMA with an in-network coherence directory.

Memory and Disaggregation

  • MIND and Concordia use in-network coherence directories to route page requests and multicast invalidations; Callosal reuses these optimizations for file-page coherence.Earlier DSM work also used application-object semantics to accelerate object operations and reduce false sharing.Source: §7 · p. 12

Low-latency Distributed Filesystems

  • Several low-latency filesystems trade transparency, granularity, or hardware requirements: Assise, Octopus, and Ethane require NVM or user-level components, while Lustre operates at coarser granularity.Callosal is a purely kernel-space approach; Assise's file-granularity leases require costlier invalidations than Callosal's page-granularity sharing.Source: §7 · p. 12
  • DPC uses hardware-coherent CXL memory mappings, whereas Callosal targets commodity RDMA and programmable switches and integrates into Linux's page cache.blk-switch and i10 target block storage rather than distributed filesystems.Source: §7 · p. 12

8 Conclusion

Callosal is a kernel page-cache approach for shared disaggregated storage using network-accelerated DSM. With CallosalFS, it outperforms NFS on mixed workloads when contention is not extreme, outperforms CephFS on multiple workloads, and scales on a key-value store with 1% writes and uniform access.

8 Conclusion

  • CallosalFS scales 1.72× from one to four nodes on a key-value store with 1% writes and uniform access, without sacrificing consistency.The conclusion also reports around 4.6× higher throughput than NFS on mixed read/write benchmarks when contention is not extreme, and an order-of-magnitude advantage over CephFS on multiple workloads.Source: §8 · p. 13

AI Disclosure

The authors report using AI tools for language editing, experimental-script and benchmark-harness debugging, data-analysis scripts, and code for the prototype key-value store and related scripts.

AI Disclosure

  • The authors state that they specified the key-value store design, synchronization approach, and workload, manually reviewed generated code, and did not use AI code generation for the filesystem implementation.They take responsibility for the work's accuracy, integrity, and originality.Source: AI Disclosure · p. 13
Loading 2610.36347v1…