Source-linked AI summary
CFS: A Distributed File System for Large Scale Container Platforms
Haifeng Liu, Wei Ding, Yuan Chen, Weilong Guo, Shuoran Liu, Tianpeng Li, Mofei Zhang, Jianxing Zhao, Hongyin Zhu, Zhengyi Zhu
TL;DR
CFS targets the need for persistent, shared storage in container platforms while addressing performance costs associated with strong POSIX semantics. It combines scenario-aware replication, memory-based metadata placement, and relaxed POSIX semantics; in TreeCreation, Ceph’s lead narrows as more clients join.
Problem
Container workloads need persistent storage shared across containers, while strong POSIX semantics and inode–dentry atomicity can limit metadata-operation performance.
Source: §1 · pp. 1–2Method
CFS is a distributed file system for large-scale container platforms, using write-scenario-specific replication, memory-based metadata placement, and POSIX APIs with relaxed semantics and metadata atomicity.
Source: §1 · p. 2Results
In TreeCreation, Ceph performs better than CFS with one client, but the performance gap narrows as more clients join.
Source: §4.2 · p. 10Takeaways & Limitations
Memory-usage-based metadata placement avoids metadata rebalancing during capacity expansion.
Source: §1 · p. 2Some POSIX-standard operations, including xattr, fcntl, ioctl, mknod, and readdirplus, were still being implemented.
Source: §6 · p. 12
Abstract
from arXiv · showhide
We propose CFS, a distributed file system for large scale container platforms. CFS supports both sequential and random file accesses with optimized storage for both large files and small files, and adopts different replication protocols for different write scenarios to improve the replication performance. It employs a metadata subsystem to store and distribute the file metadata across different storage nodes based on the memory usage. This metadata placement strategy avoids the need of data rebalancing during capacity expansion. CFS also provides POSIX-compliant APIs with relaxed semantics and metadata atomicity to improve the system performance. We performed a comprehensive comparison with Ceph, a widely-used distributed file system on container platforms. Our experimental results show that, in testing 7 commonly used metadata operations, CFS gives around 3 times performance boost on average. In addition, CFS exhibits better random-read/write performance in highly concurrent environments with multiple clients and processes.
1 INTRODUCTION
Containerized applications need persistent shared storage, but existing systems may not suit mixed file sizes, access patterns, write scenarios, metadata scale, and performance needs. CFS addresses these needs with a general-purpose storage engine, scenario-aware replication, utilization-based metadata placement, and relaxed POSIX semantics and metadata atomicity.
1 INTRODUCTION
- Container workloads need persistent storage because containers may close, files may be shared across containers, and services may share storage resources.This need is especially relevant to stateful applications.Source: §1 · p. 1
- Mixed workloads combine files from kilobytes to hundreds of gigabytes with sequential or random access, while many systems optimize for only large or small files.Existing systems may also use one replication protocol across different write scenarios.Source: §1 · p. 2
- A single metadata node can become a performance or storage bottleneck under heavy client access, while some distributed metadata approaches require rebalancing during capacity expansion.Such migration can degrade read/write performance.Source: §1 · p. 2
- Even relaxed POSIX semantics can leave inode–dentry atomicity as a limit on metadata-operation performance.CFS relaxes this requirement alongside POSIX consistency semantics.Source: §1 · p. 2
- CFS provides a general-purpose storage engine for large and small files, uses different replication protocols for append and overwrite, and relaxes POSIX semantics and inode–dentry atomicity to improve performance.The system is designed for large-scale container platforms.Source: §1 · p. 2
2 DESIGN AND IMPLEMENTATION
CFS separates metadata, file contents, and resource management, with volumes grouping metadata and data partitions for access by containers. Its metadata subsystem stores file metadata in memory across meta nodes and uses replication and recovery mechanisms.
2 DESIGN AND IMPLEMENTATION
- CFS separates metadata and data subsystems and uses a resource manager to manage the file system for container-hosted client processes.A volume groups metadata and data partitions and can be mounted by multiple containers for shared access.Source: §2 · p. 2
2.1.1 Internal Structure.
- The metadata subsystem is a distributed in-memory datastore whose meta partitions store file inodes and dentries in separate B-trees indexed by inode ID and by parent inode ID plus name.The inode and dentry trees support fast lookup.Source: §2.1 · p. 3, §2.1.1 · p. 3
2.1.2 Raft-based Replication.
- Metadata replication is performed by partition, with MultiRaft maintaining strong consistency and reducing heartbeat traffic relative to original Raft.Snapshots and logs persist in-memory meta partitions for backup and recovery.Source: §2.1.2 · p. 3, §2.1.3 · p. 3
2.1.3 Failure Recovery.
- A failure during a metadata operation can leave an orphan inode without a dentry, which may be difficult to free.The client retries failed operations until success or the maximum retry limit is reached.Source: §2.1.3 · p. 3
2.2 Data Storage
CFS supports sequential and random access for large and small files through an extent-based data subsystem. It uses scenario-specific replication and a committed-offset approach to simplify recovery after sequential-write failures.
2.2 Data Storage
- The data subsystem is optimized for large and small files accessed sequentially or randomly.Files at or below a configurable threshold—128 KB by default—are treated as small files.Source: §2.2 · p. 3, §2.2.1 · p. 4
2.2.1 Internal Structure.
- The data subsystem consists of data nodes and partitions; each partition contains metadata and an extent store built from storage units called extents.The CRC of each extent is cached in memory for data-integrity checks.Source: §2.2.1 · p. 3
2.2.3 Small File Storage and Punch Holes.
- For small files, CFS aggregates contents in shared extents and records each file’s physical offset in metadata; punch-hole deletion asynchronously frees disk space without requiring garbage collection.Large-file extents can instead be removed directly from disk.Source: §2.2.3 · p. 4
2.2.4 Scenario-Aware Replication.
- CFS uses primary-backup replication for sequential appends and MultiRaft-based replication for overwrites, matching strongly consistent protocols to write patterns.The design balances performance and code reusability; MultiRaft’s extra log I/O can affect read-after-write performance, but the authors say overwrites are usually much less frequent than sequential writes.Source: §2.2.4 · p. 4
2.2.5 Failure Recovery.
- For sequential writes, CFS tracks the largest offset committed by all replicas and resends only the uncommitted remainder, avoiding recovery of inconsistent replica portions.The authors say this simplifies maintaining strong replication consistency.Source: §2.2.5 · pp. 4–5
2.3 Resource Manager
The resource manager tracks cluster status and distributes metadata and data using utilization-based placement. It also adds partitions as volumes fill and splits metadata partitions to maintain unique inode ranges.
2.3.1 Utilization-Based Placement.
- Utilization-based placement selects partitions on nodes with lower memory or disk use and does not require rebalancing when storage nodes are added.The authors also state that uniform distribution may reduce simultaneous client access to the same node, potentially improving performance stability.Source: §2.3.1 · p. 5
2.3.2 Meta Partition Splitting.
- When a metadata partition nears its inode and dentry capacity, CFS splits its inode range so the original and new partitions receive distinct ranges.The original range ends at a cutoff above the largest inode ID already used; the new partition starts at the next ID.Source: §2.3.2 · p. 5
2.3.3 Exception Handling.
- When a partition times out, its remaining replicas are marked read-only; when it becomes unavailable, its data is eventually migrated manually to a new partition.Multiple failures reported by the node identify unavailability.Source: §2.3.3 · p. 6
2.4 Client
The CFS client caches partition assignments and metadata to reduce communication with resource managers and metadata nodes, and caches the most recently identified data-node leader to reduce retries.
2.4 Client
- The client process runs entirely in user space with its own cache and is integrated with FUSE to provide a file-system interface.The source marks the FUSE reference with footnote 6.Source: §2.4 · p. 6
- The client caches available metadata and data partition addresses, obtained at startup, and periodically synchronizes them with the resource manager.This reduces communication with the resource manager.Source: §2.4 · p. 6
- The client caches returned inodes and dentries when creating files, plus data partition, extent, and offset information after a successful write.On read/write open, cached metadata is forced to synchronize with the metadata node.Source: §2.4 · p. 6
- The client caches the most recently identified leader to minimize retries when a read is sent to replicas after the leader may have changed.Because leaders do not change frequently, this cache minimizes retries in most cases.Source: §2.4 · p. 6
2.5 Optimizations
CFS describes two cluster optimizations: limiting heartbeat exchanges among Raft groups and using non-persistent client connections to the resource manager.
2.5.1 Minimizing Heartbeats.
- Raft sets limit heartbeat exchanges to nodes within the same set, addressing communication overhead when many partitions span metadata and data nodes.CFS prefers replicas for a new partition from the same Raft set.Source: §2.5.1 · p. 6
2.5.2 Non-Persistent Connections.
- Non-persistent connections between clients and the resource manager prevent its overload when tens of thousands of clients access one CFS cluster.The overload risk arises if all client connections are kept alive.Source: §2.5.2 · p. 6
2.6 Metadata Operations
CFS relaxes inode–dentry atomicity because utilization-based placement can put them on different metadata nodes, making operations distributed transactions. Its workflows manage the resulting possibility of orphan inodes.
2.6 Metadata Operations
- CFS relaxes inode–dentry atomicity while requiring each dentry to remain associated with at least one inode, reducing distributed-transaction overhead from metadata stored on separate nodes.A failure can still create orphan inodes; workflows minimize their likelihood, and administrators can use fsck to repair files if they accumulate.Source: §2.6 · p. 6
2.6.1 Create.
- File creation allocates an inode before its dentry; if dentry creation fails, the client unlinks the inode and lists it as an orphan for later eviction.The inode and dentry need not reside on the same metadata node; an orphan inode has no associated dentry.Source: §2.6.1 · pp. 6–7
2.6.2 Link.
- Linking increments the inode’s nlink before creating a dentry on the target parent’s metadata partition, and reverses the increment if dentry creation fails.nlink is the number of associated links.Source: §2.6.2 · p. 7
2.6.3 Unlink.
- Unlinking deletes the dentry before decrementing nlink; at the file or directory threshold, the inode is listed as an orphan for later eviction.If decrement retries fail, the inode may become orphaned and require manual resolution.Source: §2.6.3 · p. 7
2.7 File Operations
CFS file operations use relaxed consistency, ensuring sequential consistency for file and directory operations without leasing to prevent concurrent writes. The client handles sequential and in-place random writes, asynchronous deletion, and reads through a Raft leader.
2.7 File Operations
- CFS ensures sequential consistency for file and directory operations but has no leasing mechanism to prevent multiple clients from writing the same file or directory.Applications must maintain a more restrictive consistency level if necessary.Source: §2.7 · p. 7
2.7.1 Sequential Write.
- Sequential writes send fixed-size packets to a data-node leader, which replicates them in primary-backup order; the client updates its cache on commit and synchronizes with metadata periodically or on fsync().The client selects an available data partition from its cache; packets include replica addresses, extent id, offset, and content.Source: §2.7.1 · p. 7
2.7.2 Random Write.
- Random writes are in-place: the client separates appended portions from overwritten portions, writing the former sequentially while the file’s data-partition offset stays unchanged for the latter.The client determines the portions using the original-data and new-data offsets.Source: §2.7.2 · p. 7
2.7.3 Delete.
- Deletion is asynchronous: the metadata node updates nlink and marks the inode deleted at its threshold, while a separate process later clears it and deletes the data content.The threshold is 0 for a file and 2 for a directory.Source: §2.7.3 · p. 7
2.7.4 Read.
- Reads occur only at the Raft leader, and the client constructs requests from cached partition, extent, and offset information.The primary-backup group leader and Raft group leader may differ.Source: §2.7.4 · p. 8
3 DISCUSSION OF DESIGN CHOICES
CFS combines centralized management with a separate metadata cluster, choosing in-memory metadata placement for scalability and deployment flexibility. Its storage replicas remain strongly consistent, while the POSIX-facing file system selectively relaxes consistency semantics.
3.1 Centralization vs. Decentralization
- CFS uses a separate metadata cluster to reduce the resource manager’s metadata bottleneck while retaining centralized design for simplicity.The resource manager’s scalability can still be limited by memory and disk space, though the authors report this has not been an issue in their experience.Source: §3.1 · p. 8
3.2 Separate Meta Node vs. Metadata on Data Node
- CFS keeps metadata in memory on separate meta nodes, allowing memory-intensive metadata machines and disk-intensive data machines or co-location.Metadata placement based on memory usage avoids metadata rebalancing during capacity expansion.Source: §3.2 · p. 8
3.3 Consistency Model and Guarantees
- Storage replicas use strong consistency through primary-backup or Raft protocols, chosen to avoid overwrite performance compromises or Raft log-write amplification.At the file-system layer, CFS provides POSIX-compliant APIs but selectively relaxes POSIX consistency and inode–dentry atomicity to improve performance.Source: §3.3 · p. 8
- CFS guarantees consistency for modifications to non-overlapping file portions, but none when clients modify the same portion.This design reflects the authors’ stated container setting, where applications seldom rely on full strong consistency and independent jobs rarely write to a shared file.Source: §3.3 · p. 9
4 EVALUATION
Against Ceph, CFS’s metadata performance improves with concurrency, while its large-file advantage is strongest for random reads and writes. CFS also outperforms Ceph on small-file reads and writes in the evaluated workloads.
4.1 Experiment Setup
- The comparison uses Ceph and CFS deployments on 10-machine clusters and tests seven commonly used metadata operations with mdtest.Large-file tests use fio with direct I/O on separate 40 GB files; the Ceph queue and thread settings were tuned to 6 and 4, respectively.Source: §4 · p. 9, §4.1 · p. 9, §4.2 · p. 9
4.2 Metadata Operations
- About 3 times average performance boost: at 8 clients with 64 processes each, CFS outperforms Ceph in 6 of 7 metadata tests.With one client and one process, Ceph leads in 5 of 7 tests; CFS catches up as client and process counts rise.Source: Table 3 · p. 9, §4.2 · p. 9
- DirStat and TreeRemoval favor CFS through batched, client-cached inode retrieval and less locality-related queuing; Ceph leads TreeCreation initially, but its edge narrows as clients increase.Ceph’s TreeCreation advantage with few clients is attributed to directory locality; rising metadata-server pressure and proxy redirects add overhead.Source: §4.2 · p. 10
4.3 Large Files
- CFS has higher IOPS in random read/write tests above 16 processes and an advantage in multi-client random tests, while sequential performance is similar.The authors attribute the random-workload advantage to in-memory metadata caching and in-place overwrites that avoid metadata updates.Source: Figure 8 · p. 11, §4.3 · p. 11, Figure 9 · p. 11
4.4 Small Files
- CFS outperforms Ceph in both small-file read and write tests for files from 1 KB to 128 KB.CFS stores metadata in memory for reads, and clients send small-file writes directly to data nodes without requesting new extents from the resource manager.Source: Figure 10 · p. 11, §4.4 · pp. 11–12
5 RELATED WORK
The related-work discussion contrasts CFS with distributed file systems and cloud storage systems along metadata placement, file-size specialization, and storage design. It also describes systems whose hardware and storage-tier techniques are orthogonal to CFS.
5 RELATED WORK
- GFS and HDFS target large sequential files and use a single metadata master, unlike CFS’s separate metadata subsystem.MooseFS also uses a single master, while Ceph, GlusterFS, and other systems are described as distributed file systems integrated with Kubernetes.Source: §5 · p. 12
- CFS shares Haystack’s in-memory metadata approach but separates metadata from contents, stores physical offsets, and uses punch holes for deletion.The discussion also distinguishes CFS from file-grouping designs and notes that Haystack does not guarantee strong replica consistency during deletion.Source: §5 · p. 12
- Azure Storage uses an extra partition layer for random writes, while AWS EFS and Azure were not evaluated because they are not open source.PolarFS and OctopusFS use techniques the authors characterize as orthogonal and potentially adoptable by CFS.Source: §5 · p. 12
6 CONCLUSIONS AND FUTURE WORK
CFS combines scenario-aware storage and replication, distributed metadata placement, and relaxed POSIX APIs. Most POSIX-standard operations are implemented, while several remain in progress and further performance and interface work is planned.
6 CONCLUSIONS AND FUTURE WORK
- CFS combines scenario-aware replication, distributed metadata placement, and relaxed POSIX APIs in a general-purpose storage engine.The design accommodates different file-access patterns and aims to improve system performance.Source: §6 · p. 12
- Most POSIX-standard operations are implemented, while xattr, fcntl, ioctl, mknod, and readdirplus remain in progress.Source: §6 · p. 12
- Planned work includes using Linux page cache, improving file locking and cache coherency, supporting RDMA, and developing a kernel-space POSIX interface to eliminate FUSE overhead.Source: §6 · p. 12