Source-linked AI summary
HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases
Salman Niazi, Mahmoud Ismail, Steffen Grohsschmiedt, Mikael Ronström, Seif Haridi, Jim Dowling
TL;DR
Distributed file-system metadata can limit scalability, and conventional wisdom held that fully normalized metadata in a distributed database was too costly. HopsFS uses a NewSQL database with multiple stateless namenodes and a protocol for large subtree operations. It reports higher throughput than HDFS, no downtime during failover, and greater metadata capacity, with hotspot handling remaining a limitation.
Problem
Metadata management is a scalability bottleneck in large hierarchical file systems, while conventional wisdom held that fully normalized metadata in a distributed database was too expensive in throughput and latency.
Source: §1 · p. 1Method
HopsFS stores normalized metadata in a distributed NewSQL database accessed by parallel stateless namenodes, and splits oversized subtree operations into smaller transactions under distributed locking.
Source: §1 · pp. 1–2Results
16 times higher throughput than HDFS on Spotify’s workload and 37 times on a more write-intensive workload; HopsFS also reports no downtime during failover.
Source: §1 · p. 2Takeaways & Limitations
HopsFS scales metadata capacity to 37 times HDFS and makes metadata easier to extend, analyze, and export to external systems.
Source: §10 · p. 13A hot inode’s operation throughput is limited by its database shard, and HopsFS has no built-in runtime hotspot identification.
Source: §4.2.1 · p. 5
Abstract
from arXiv · showhide
Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS' single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS' capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
1 Introduction
Distributed file-system metadata is a scalability bottleneck, while static namespace sharding complicates cross-shard operations and administration. HopsFS moves normalized metadata to a NewSQL database and distributes request handling across stateless namenodes, using transactions and a subtree protocol for large operations.
1 Introduction
- Distributed file-system metadata is a scalability bottleneck, and static namespace sharding complicates cross-shard operations such as moves and shard administration.The shard-management burden grows as namespace shards change in size over time.Source: §1 · p. 1
- Recent NewSQL performance and scalability improvements reopen the possibility of storing distributed file-system metadata in a commodity database, despite conventional concerns about throughput and latency.The conventional view held that fully normalized metadata in a distributed database was too expensive.Source: §1 · p. 1
- HopsFS stores normalized metadata in highly available, in-memory NDB and serves it through redundant stateless namenodes that read and update the database in parallel.It decouples metadata storage from metadata management.Source: §1 · p. 1
- HopsFS uses distributed transactions, batching, write-ahead caches, distribution-aware database techniques, and inode hints to improve file-system operation performance.On a path-resolution cache hit, the inode hints cache can reduce database round trips from N to 1 for a path of depth N.Source: §1 · p. 1
- Large subtree operations that exceed database transaction limits are isolated with application-level distributed locks, then split into smaller transactions executed in parallel.The protocol preserves namespace consistency if the executing namenode fails.Source: §1 · p. 2
- 16 times higher throughput than HDFS on Spotify’s real-world workload, and 37 times on a more write-intensive workload; HopsFS also has no downtime during failover.HopsFS is described as a drop-in replacement for HDFS.Source: §1 · p. 2
2 Background
HDFS centers metadata service on an active namenode, while NDB provides distributed storage and transaction capabilities. The background also describes constraints of HDFS’s monolithic metadata architecture and NDB’s replication and isolation model.
2.1 Hadoop Distributed File System
- HDFS stores metadata in the active namenode’s Java heap, with a standby, journal quorum, and ZooKeeper coordinating high availability and failover.Large operations may release the global lock before journaling, so a failover can lose changes not yet logged.Source: §2.1 · p. 2, Figure 1 · p. 2
- HDFS metadata capacity is constrained by JVM heap and garbage collection, while memory-optimized structures make metadata difficult to modify or export.Spotify’s 60-petabyte cluster metadata fits in 140 gigabytes of heap; the paper identifies JVM garbage collection and the monolithic namenode as scalability bottlenecks.Source: §2.1 · p. 2
2.2 Network Database (NDB)
- NDB is MySQL Cluster’s shared-nothing, replicated, in-memory relational storage engine, with datanode- and cluster-level recovery mechanisms.Cluster-level recovery uses global checkpointing; the default global epoch advances every 2 seconds.Source: §2.2 · p. 3
- NDB supports application-defined partitioning and distribution-aware transactions that can start on the shard containing the required data.Partition-pruned scans can read data locally; incorrect hints add network traffic but do not otherwise affect correct operation.Source: §2.2 · p. 3
2.2.1 NDB Data Replication and Failure Handling
- NDB groups replicas by replication degree R, so each node group can tolerate failures while at least one datanode survives in that group.With default replication degree two, a 12-node cluster has six node groups and can tolerate six failures if one datanode survives in each group; increasing replication can reduce throughput.Source: §2.2.1 · p. 3
2.2.2 Transaction Isolation
- NDB provides read-committed isolation, which prevents dirty reads but permits phantom and fuzzy reads; row-level locks can isolate conflicting transactions.The supported lock types include exclusive, shared, and read-committed locks.Source: §2.2.2 · p. 3
3 HopsFS Overview
HopsFS separates request handling from metadata storage, allowing multiple stateless namenodes to access an external distributed database. Clients can distribute operations across namenodes, while HDFS v2.x clients remain compatible but do not do so.
3 HopsFS Overview
- HopsFS uses multiple stateless namenodes to process requests against metadata in an external distributed database, with a per-namenode DAL driver abstracting database operations.A leader namenode coordinates housekeeping and membership management uses the database as shared memory.Source: §3 · p. 3
- Clients may select namenodes randomly, round-robin, or stickily; HDFS v2.x clients are compatible but assume a single active namenode and therefore do not distribute operations.The leader namenode load-balances datanode block reports across alive namenodes.Source: §3 · p. 3
- Spotify’s HDFS workload has list, read, and stat operations accounting for approximately 95% of metadata operations.The table reports relative frequencies for selected file-system operations.Source: Table 1 · p. 4
4 HopsFS Distributed Metadata
HopsFS distributes normalized file-system metadata across a NewSQL database, using partitioning and database operations designed around common file-system workloads. Its partitioning supports efficient directory and file access, while configurable handling of hotspots addresses uneven workloads with trade-offs.
4 HopsFS Distributed Metadata
- HopsFS designs metadata partitioning around common file-system operations and avoids costly full-table and index scans where possible.List, stat, and file-read operations account for about 95% of Spotify’s HDFS operations; reads and directory listings use primary-key lookups and partition-pruned index scans.Source: §4 · p. 4, Figure 2 · p. 4
4.1 Entity Relation Model
- The relational model stores directory and file inodes separately from file-related blocks and replica metadata, with lifecycle tables for replica states.Inode records contain parent references and individual path components; the file-related entities use the inode foreign key as a partition key.Source: §4.1 · p. 4
4.2 Metadata Partitioning
- Partitioning by parent inode places directory children together for listing, while partitioning file-related metadata by file inode keeps it together for file operations.For example, the children of /user reside on one shard, while metadata for /user/foo.txt resides on another.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
4.2.1 Hotspots
- Hot inodes can bottleneck their database shard, and HopsFS has no built-in runtime hotspot detection.To spread top-level operations, HopsFS pseudo-randomly partitions children by name; additional levels can be configured at the cost of slower top-level move and listing operations.Source: §4.2.1 · p. 5
5 HopsFS Transactional Operations
HopsFS implements file-system operations as database transactions, using locking, cached metadata, and batched operations to support concurrent access. It accelerates path resolution with inode hints and uses ordered locking to manage conflicts.
5 HopsFS Transactional Operations
- HopsFS distinguishes inode operations on a known file, directory, or block from subtree operations on an unknown, potentially very large number of inodes.Examples of subtree operations include recursive delete, move, chmod, and chown on non-empty directories.Source: §5 · p. 5
- Because NDB's read-committed isolation is weaker than HDFS's global-lock semantics, HopsFS uses row-level locks and reorders inode operations to avoid deadlocks and lock upgrades.Operations can proceed in parallel unless they lock the same inodes; metadata is acquired in a consistent root-to-leaf order.Source: §5 · p. 5
5.1 Inode Hint Cache
- Inode hints let namenodes batch path-component lookups using cached primary keys, avoiding a database round trip for each path component when the hints hit.Invalid hints trigger recursive path resolution and cache repair; moves, which can invalidate entries, are less than 2% of operations in typical Hadoop workloads.Source: §5.1 · pp. 5–6, §5.1.1 · p. 6
5.1.1 Cache Consistency
- The inode hint cache stores primary keys rather than whole inodes, and invalid entries fall back to recursive resolution before the cache is repaired.Sticky client behavior can improve temporal locality and cache hit rates.Source: §5.1.1 · p. 6
5.2 Inode Operations
- Inode operations use a pessimistic concurrency model and pass through lock, execute, and update phases; conflicting inode or subtree operations are serialized.The authors chose pessimistic concurrency because it performs well at medium to high resource utilization, common in many HDFS clusters.Source: §5.2 · p. 6
5.2.1 Lock Phase
- During the lock phase, HopsFS batches path reads and locks the final inode component, using shared locks for reads and exclusive locks for modifications.Batched reads remove N-1 database round trips for a path of depth N; related metadata is read in a predefined order using partition-pruned scans.Source: §5.2.1 · p. 6, Figure 4 · p. 6
5.2.2 Per-Transaction Cache
- A per-transaction cache holds database reads and updates until completion, reducing repeated database round trips while row-level locks protect its consistency.The final update phase sends changed and new metadata to the database in batches, then commits or rolls back the transaction.Source: §5.2.2 · pp. 6–7, §5.2.3 · p. 7
6 Handling Large Operations
Large subtree operations cannot fit in a single transaction when they involve millions of inodes. HopsFS isolates the subtree, quiesces ongoing work, and processes the operation in batches while preserving namespace consistency.
6 Handling Large Operations
- HopsFS applies large subtree operations incrementally because operations on millions of inodes exceed transaction limits; its protocol provides HDFS-equivalent consistency and avoids metadata inconsistencies on failed deletes.The protocol uses application-level distributed locks and parallel batches; unlike failed HDFS deletes, failed HopsFS deletes do not leave metadata inconsistencies.Source: §6 · p. 7, §6.1 · p. 7
6.1 Subtree Operations Protocol
- The protocol locks the subtree root, waits for active operations to finish by traversing and write-locking descendants, then runs batched transactions in parallel.If quiescing fails because of concurrent operations, HopsFS retries with exponential backoff; Figure 5 depicts the descent to quiesce and upward batched delete.Source: §6.1 · p. 7, Figure 5 · p. 7
6.2 Handling Failed Subtree Operations
- After a namenode failure, unfinished deletes leave remaining inodes connected so clients can resubmit the operation; other subtree operations update root inodes in a single transaction.Locks owned by dead namenodes can be cleared by another namenode when encountered.Source: §6.2 · pp. 7–8
6.3 Inode and Subtree Lock Compatibility
- Inode and subtree locks are compatible: a subtree flag implicitly locks descendants, and inode operations encountering that flag abort voluntarily.Source: §6.3 · p. 8
7 HopsFS Evaluation
In a Spotify workload experiment, HopsFS outperformed HDFS in throughput, including under a hotspot workload. The authors also report substantially greater metadata capacity for HopsFS, with higher memory requirements per file.
7.1 Experimental Setup
- The evaluation used zero-length files to isolate metadata performance and generated workloads from Spotify’s operational workload statistics.The Spotify cluster described had 13 million directories and 218 million files; experiments used on-premise servers, with HDFS deployed on five servers.Source: §7.1 · p. 8, §7.2 · p. 8, §7 · p. 8
7.2 Industrial Workload Experiments
- 1.25 million operations per second gave HopsFS 16 times the throughput of HDFS on the Spotify workload using 60 namenodes and 12 NDB nodes.Source: §7.2 · p. 8, Figure 6 · p. 9
7.2.1 Hotspots
- 3 times the throughput of HDFS was achieved by HopsFS under a workload where all operation paths shared a common ancestor.The hotspot workload's scalability was limited by the shard holding /shared-dir; HDFS performance was unaffected by the hotspot.Source: §7.2.1 · p. 9
7.3 Metadata (Namespace) Scalability
- HopsFS could store up to 17 billion files using 24 TB of metadata, about 37 times HDFS's capacity.The paper notes that HopsFS requires about 1.5 times more memory than HDFS to store highly available metadata.Source: Table 3 · p. 9, §7.3 · p. 9
7.4 FS Operations’ Raw Throughput
HopsFS outperforms HDFS across file-system operations, while subtree operations trade performance for common-operation throughput. Under concurrent Spotify workloads, HopsFS can keep operation latency lower, although individual latency comparisons vary by operation.
7.4 FS Operations’ Raw Throughput
- HopsFS outperforms HDFS for every file-system operation, with the largest gains on the most common operations.Figure 7 represents each HopsFS throughput increase as five namenodes are added and shows HDFS’s maximum with a five-server namenode.Source: §7.4 · p. 10, Figure 7 · p. 10
7.4.1 Subtree Operations
- HDFS outperforms HopsFS on large-subtree move and delete operations, an accepted trade-off given their low frequency in typical industrial workloads.HopsFS reads large amounts of data over the network and runs the work in small transaction batches; move time rises less rapidly because it does not update every inner node or leaf.Source: §7.4.1 · p. 10
7.5 Operational Latency
- For concurrent Spotify-workload clients, HopsFS has lower average operation latency than HDFS, though an unloaded HDFS namenode has lower single-operation latency.HDFS namespace updates block other operations, while its RPC queues can increase end-to-end latency as clients wait; HopsFS handles more concurrent clients while keeping latency low.Source: §7.5 · p. 10, Figure 8 · p. 10
- At 50% load, HopsFS 99th-percentile touch-file latency is 100.8 ms versus HDFS’s 101.8 ms; read-file latency is 8.6 ms versus 1.5 ms.The comparison uses the Spotify workload in a non-overloaded cluster.Source: §7.5 · p. 10
- At 50% load, HopsFS 99th-percentile ls-dir latency is 11.4 ms versus HDFS’s 0.9 ms; stat-dir latency is 8.5 ms versus 1.5 ms.The comparison uses the Spotify workload in a non-overloaded cluster.Source: §7.5 · p. 10
7.6 Failure Handling
The section evaluates failure handling across namenodes, database datanodes, and journal nodes, as well as block-report processing. HopsFS clients can retry operations through surviving namenodes, while recovery behavior and tolerable failures depend on the affected component.
7.6.1 Namenodes failure
- HopsFS clients transparently re-execute failed operations on remaining namenodes, avoiding the observed 8–10 seconds of HDFS metadata downtime during failover.In the experiment, HopsFS namenodes failed round-robin and clients were fixed in number; sticky selection caused throughput to fall gradually as more namenodes restarted.Source: §7.6.1 · p. 11, Figure 10 · p. 11
7.6.2 Failure of NDB Datanodes or Journal Nodes
- HDFS tolerates up to ⌈N/2⌉−1 journal-node failures, while HopsFS tolerates NDB datanode failures only if each node group retains a live datanode.With three journal nodes, HDFS tolerates one failure; with default replication degree two, four- and twelve-node NDB clusters tolerate up to two and six failures, respectively, in disjoint replica groups.Source: §7.6.2 · p. 11
- NDB coordinator failover hands transactions to another coordinator, and failed participant transactions are automatically retried by the namenode.The default heartbeat timeout gives an upper bound of six seconds after three missed heartbeats; participant failures use a 1200 ms inactive-transaction timeout.Source: §7.6.2 · p. 11
7.7 Block Report Performance
- HopsFS block-report processing is functional but does not reach HDFS throughput because namenodes read substantial metadata over the network.Source: §7.7 · p. 11
- With 30 namenodes processing reports containing 100,000 blocks, HopsFS handles 30 reports per second versus HDFS’s 60.HopsFS persists block-location mappings, so full block reports are needed less often; at 512 MB blocks and six-hour report intervals, the authors project it can handle an exabyte cluster.Source: §7.7 · pp. 11–12
8 Related Work
Related work includes database-backed file systems, distributed metadata architectures, and middleware for small-file workloads. The paper contrasts their metadata organization, subtree support, transaction handling, and operational trade-offs with HopsFS.
8 Related Work
- Earlier relational-database file systems include InversionFS and WinFS; Gunawi showed relational databases can improve some operations such as fsck.Source: §8 · p. 12
- CalvinFS, CassandraFS, and GiraffaFS use denormalized metadata, which affects subtree operations; GiraffaFS supports moves only within one directory.CalvinFS uses CalvinDB’s two-phase lock-set protocol for large transactions, but did not experimentally demonstrate viability for directories with millions of files; transaction operation limits remain a constraint.Source: §8 · p. 12
- IndexFS and ShardFS use local LevelDB instances for small-file metadata and delegate large files to an underlying file system, without atomic operations spanning both systems.IndexFS caches inode information at clients and targets hot directories or files; ShardFS caches at metadata servers, while HopsFS load-balances a user-configurable number of top-level directories.Source: §8 · p. 12
- Other systems shard inodes by random or hashed identifiers or paths; Ceph dynamically partitions the tree and hashes hot-spot directories across metadata servers.Source: §8 · p. 12
- The architecture supports pluggable NewSQL engines such as MemSQL and SAP Hana, which support cross-partition transactions and application-defined partitioning; VoltDB is excluded because it serializes cross-partition transactions.Source: §8 · p. 12
9 External Metadata Implications
HopsFS’s database-backed metadata can support analytics and extensions beyond file-system operations. The paper describes replication for analysis, integrity-preserving metadata extensions, and Elasticsearch-based free-text search.
9 External Metadata Implications
- Administrators can asynchronously replicate HopsFS metadata to a backup cluster or MySQL slave for complex analytics without affecting the active cluster.Source: §9 · p. 12
- Additional tables can extend metadata while foreign keys to inodes preserve integrity; the authors have used this approach for extended attributes and erasure coding.Source: §9 · p. 12
- An eventually consistent protocol replicates extended HopsFS metadata to Elasticsearch for free-text search across the namespace with sub-second latency.Source: §9 · p. 12
10 Summary
HopsFS is a production-grade distributed hierarchical file system that stores metadata in an external NewSQL database. It scales capacity and throughput beyond HDFS while providing lower average latency for many concurrent clients and no downtime during failover.
10 Summary
- HopsFS stores metadata in an external NewSQL database and, to the authors’ knowledge, is the first production-grade distributed hierarchical file system to do so.It is open source and highly available.Source: §10 · pp. 12–13
- 37 times more metadata than HDFS and 16 times HDFS’ throughput on a Spotify workload trace are supported by HopsFS.It scales out in capacity and throughput by adding namenodes and database nodes.Source: §10 · p. 13
- HopsFS has lower average latency for many concurrent clients, no downtime during failover, and supports pluggable database storage engines.The metadata can also be extended, analyzed, and used by applications.Source: §10 · p. 13