Source-linked AI summary
MetaDB: Putting File-System Structure Back into Distributed Databases
Zebin Ren, Biao Cao, Kang Chen, Zhihui Yin, Qiushi Chen, Jielong Jian, Jiwei Xiong, Xinxing Wang, Yan Xing, Liguo Duan, Ran Zheng, Xianjun Meng, Zhenyu Hou
TL;DR
General-purpose databases overlook metadata workloads’ tree structure and differentiated update semantics, despite serving as shared metadata layers in production storage systems. MetaDB co-designs a distributed database around metadata structures and semantics; it reports 2.2×–4.1× the throughput of the previous general-purpose database and halves metadata-machine requirements.
Problem
General-purpose databases overlook metadata workloads’ structural and semantic properties, even as distributed databases serve as shared metadata layers for production storage systems.
Source: §1 · p. 1Method
MetaDB co-designs a metadata-aware distributed database around file-system structures, semantics, and workload dynamics.
Source: §1 · p. 2Results
2.2×–4.1× the throughput of the previous general-purpose database and approximately 50% fewer metadata machines are reported for MetaDB.
Source: §1 · p. 2, §10 · p. 12Takeaways & Limitations
The results suggest that exploiting workload-specific structure inside the database is a design principle for performance and efficiency in storage-system metadata management.
Source: §10 · p. 12
Abstract
from arXiv · showhide
Storage systems in production increasingly rely on distributed databases for metadata management, yet our experience at Baidu shows that treating a general-purpose database as a black box leaves substantial performance and efficiency untapped. File-system metadata exhibits directory-tree locality, differentiated consistency requirements, and clustered deletions that general-purpose databases largely ignore. We thus propose MetaDB, a metadata-aware distributed transactional database that serves as the shared metadata layer for Baidu's large-scale storage systems. MetaDB addresses three key mismatches between general-purpose databases and metadata workloads: Data Model Mismatch, Transaction Model Mismatch, and Storage Implementation Mismatch. To bridge these gaps, MetaDB combines (1) tree-topology-induced sharding and execution, (2) semantic-aware off-critical-path processing, and (3) a lifecycle-aware storage engine with in-memory MVCC and locality-aware compaction. MetaDB achieves 2.2x-4.1x the throughput of our previous general-purpose database and halves the number of metadata machines. Deployed in production at Baidu for over five years, MetaDB unifies metadata management across the company's storage fleet and scales to over 1,000 metadata nodes, tens of trillions of metadata records, and exabyte-scale storage in a single cluster.
1 Introduction
Baidu’s production experience showed that general-purpose databases leave performance untapped because they overlook file-system metadata’s structure, semantics, and dynamics. MetaDB co-designs database techniques for these workloads and reports higher throughput, fewer metadata machines, and large production scale.
1 Introduction
- Baidu built a unified general-purpose distributed database as the shared metadata layer across its storage systems, but production experience exposed performance bottlenecks when it was treated as a black box.MetaDB was designed around metadata-specific structures, semantics, and dynamics.Source: §1 · pp. 1–2
- Tree-topology-induced sharding aligns shards with directory boundaries, while storage-node transaction-logic pushdown reduces multi-round coordination to one remote execution.Policy/size-based sharding keeps single-directory operations within a shard.Source: §1 · p. 2
- Semantic-aware off-critical-path processing asynchronously aggregates hotspot parent-directory attributes and updates auxiliary quotas and indexes without strict timeliness requirements.Read-path reconciliation preserves the latest visible state for parent attributes.Source: §1 · p. 2
- A lifecycle-aware storage engine combines in-memory MVCC with locality-aware range compaction to avoid flushing short-lived tombstones and promptly purge localized ones.The design is intended to sustain low listdir latency.Source: §1 · p. 2
- 2.2×–4.1× the throughput of the previous general-purpose database, with approximately 50% fewer metadata machines, is reported for MetaDB.It has been deployed for over five years and a single cluster scales to over 1,000 metadata machines, tens of trillions of records, and exabyte-scale storage.Source: §1 · p. 2
- MetaDB addresses mismatches in tree topology, transaction semantics, and operation dynamics with three corresponding designs.The paper evaluates these designs on production metadata workloads and reports performance improvements and resource savings over general-purpose databases.Source: §1 · p. 2
2 Background and Motivation
Baidu’s metadata services support both flat and hierarchical namespaces, and evolved from separate systems to a unified general-purpose database. Production experience with that database exposed mismatches between database designs and metadata’s tree locality, consistency requirements, and deletion patterns.
2.1 Metadata Management at Baidu
- Object storage offers flat and hierarchical namespace services, while CFS provides a POSIX-compliant hierarchical namespace; both Mantle and CFS need a scalable layer for directory trees.Mantle’s disaggregated design sends every metadata operation to the backend, whereas CFS clients can cache directory metadata and physical layout information.Source: §2.1 · p. 2
2.2 Metadata Databases Prior to MetaDB
- Baidu evolved from separate metadata systems with static sharding to a unified distributed transactional database, whose production bottlenecks motivated MetaDB.The earlier systems faced inflexible scaling, missing cross-shard transactional guarantees, management overhead, and hotspot risks; the unified database added dynamic sharding, replication, and snapshot isolation.Source: §2.2 · pp. 2–3
2.3.1 Data Model Mismatch: Flat Sharding vs. Tree Topol-
- Over 97% of directory-tree-modifying operations are single-directory operations, but flat sharding can split large directories across shards and turn localized updates into distributed transactions.Directory sizes across five users suggest aligning shard boundaries with directory boundaries is viable; Figure 1 shows the operation distribution and Figure 2 the directory-size distribution.Source: §2.3.1 · p. 3, Figure 1 · p. 3, Figure 2 · p. 3
2.3.2 Transaction Model Mismatch: Rigid ACID vs. Relaxed Semantics.
- Applying strict atomicity to all metadata fields creates parent-record contention and multi-shard coordination, although quota and index updates can tolerate minutes of delay.For same-directory creates, throughput drops from ∼60K QPS without contention to ∼6K QPS; production quota records serve post-paid billing and monitoring with hourly settlement.Source: §2.3.2 · p. 3
- During bulk deletions, persisted MVCC tombstones accumulate and degrade range scans, while conventional full-shard compaction is too slow and resource-intensive to purge localized deletions promptly.For a 1,000-item listdir, latency rises from < 100 ms to ∼6 seconds during deletion; metadata transactions are exceptionally short (< 10 ms).Source: §2.3.2 · p. 4
3 MetaDB Design
MetaDB organizes metadata around directory-tree structure and co-designs its data model, cluster architecture, and processing mechanisms for metadata workloads. Its three design pillars target data-model, transaction-model, and storage-implementation mismatches.
3.2 Architecture
- Proxies parse and route requests through MetaCache, while the Manager oversees cluster status and dynamic sharding; backend nodes replicate through Raft and persist data in in-memory MVCC plus RocksDB.MetaCache consults the Manager on a cache miss; backend nodes apply operations after Raft consensus.Source: Figure 6 · p. 4, §3.2 · pp. 4–5
3.1 Data Model
- MetaTable flattens the directory tree using (PID,name) keys and a special _attr_ entry, colocating each directory’s attributes with its children; QuotaTable and SecondaryIndexTable hold auxiliary metadata.QuotaTable tracks per-user resource usage, while SecondaryIndexTable maps data-plane files to metadata entries.Source: §3.1 · p. 4
3.3 Addressing the Mismatches
- MetaDB targets the three identified mismatches with tree-topology-induced sharding and execution, semantic-aware off-critical-path processing, and a lifecycle-aware storage engine.The storage design combines in-memory MVCC with locality-aware range compaction.Source: §3.3 · p. 5
4 Tree-Topology-Induced Sharding and Execution
MetaDB aligns shard boundaries with directory-tree structure and executes single-shard transaction logic at backend nodes. This keeps common directory operations local while reducing coordination overhead, with a fallback for exceptionally large directories.
4 Tree-Topology-Induced Sharding and Execution
- Over 96% of directory-entry updates are single-directory operations, motivating MetaDB’s design to colocate directory data and execute these operations locally.The conventional size-based sharding can separate a parent directory from its children, turning such updates into distributed transactions.Source: §4.1 · p. 5, §4 · p. 5
4.1 Sharding Policy
- MetaDB splits shards at group_key boundaries, using the parent directory identifier to keep a directory’s attributes and direct children together and avoid distributed transactions for single-directory operations.Jumbo shards are monitored and may be isolated on under-utilized nodes; operators can manually split one that still exceeds a machine’s capacity, at the cost of possible distributed transactions.Source: §4.1 · p. 5
4.2 Single-Shard Transaction Logic Pushdown
- At least two network round-trips can arise when a coordinator executes a directory operation’s read-modify-write sequence; MetaDB instead performs the full sequence locally on the backend node.The local execution depends on the sharding policy keeping the directory’s entries and attributes colocated.Source: §4.2 · p. 5
5 Semantic-Aware Off-Critical-Path Processing
MetaDB moves metadata updates with different consistency needs off the transactional critical path. It asynchronously handles auxiliary metadata and aggregates parent-directory attributes, while reconciling pending attribute changes on reads.
5 Semantic-Aware Off-Critical-Path Processing
- MetaDB removes aggregatable directory attributes and auxiliary metadata from the transaction critical path to address lock contention and unnecessary multi-shard coordination.The design uses asynchronous off-critical-path processing for fields with different consistency requirements.Source: §5 · p. 6
5.1 Decoupled Asynchronous Auxiliary Updates
- Quota and index updates usually reside on different shards, so atomically updating them with primary metadata requires multi-shard transactions; MetaDB instead queues and batches these auxiliary updates asynchronously.Requests are first persisted in the Raft log, then enqueued after the primary update; timestamps and replay after failure prevent duplicate or lost updates.Source: §5.1 · p. 6
5.2 Conflict-Free Asynchronous Attribute Aggregation
- Concurrent changes to a directory can contend on its shared attributes; MetaDB aggregates num_children deltas and selects the committed modify_time with the largest timestamp.Attribute changes are logged and queued for background batch updates, while reads merge persisted values with pending changes to return up-to-date attributes.Source: §5.2 · pp. 6–7
6 Lifecycle-Aware Storage Engine
MetaDB redesigns storage for metadata’s short transactions and clustered deletions. In-memory MVCC avoids persisting redundant MVCC tombstones, while localized compaction targets deletion-heavy key ranges.
6 Lifecycle-Aware Storage Engine
- MetaDB combines in-memory MVCC with locality-aware compaction to address tombstone accumulation from consecutive deletions and protect listdir range-scan latency.The storage design is tailored to the metadata lifecycle described for the cloud object-storage workload.Source: §6 · p. 7
6.1 In-Memory MVCC
- MetaDB retains committed versions and MVCC tombstones in memory, flushing only the latest committed state; deletions persist an LSM-tree tombstone, avoiding accumulated MVCC tombstones on disk.This avoids the double-tombstone penalty and reduces read amplification; the in-memory table is retained beyond the 5-second transaction timeout before flushing, and Raft log entries support recovery if a node fails before persistence.Source: §6.1 · p. 7
6.2 Locality-Aware Compaction
- Consecutive deletion runs create LSM-tree tombstones that penalize listdir scans; MetaDB tracks deletion-heavy key ranges and proactively compacts only affected areas after a threshold is crossed.This targets tombstone cleanup without waiting for global level saturation or compacting an entire shard, supporting stable range-scan latency.Source: §6.2 · p. 7
7 Evaluation
Evaluation combines synthetic microbenchmarks with production traces to test MetaDB’s design mechanisms and performance. Production-cluster tests report higher throughput and lower latency than the general-purpose database.
7.2 Effectiveness of Tree-Topology-Induced Sharding and Execution
- Policy/size-based sharding sustained 6.0K QPS after size-based sharding fell to 0.4K QPS; pushdown at 256 clients delivered 33.9% lower latency and 28.8% higher throughput.Pushdown achieved 25.3% lower latency and 10.0% higher throughput with one client; each client used standalone directories to avoid directory-attribute lock contention.Source: Figure 10 · p. 8, §7.2 · p. 8, Figure 11 · p. 8
7.3 Effectiveness of Semantic-Aware Off-Critical-Path Processing
- 10.7× throughput for asynchronous attribute aggregation over lock-based execution was observed under concurrent same-directory mkdir; at 16K QPS, pending quota updates implied less than 10 ms of update lag.The queue peaked below 100 pending items over 120 seconds, while quota and index updates can tolerate minutes of latency.Source: Figure 12 · p. 8, §7.3 · pp. 8–9, Figure 13 · p. 8
7.4 Effectiveness of Lifecycle-Aware Storage Engine
- With both storage optimizations, tombstones stayed below 2M and listdir latency peaked at 0.7 s during sustained bulk deletion.Baseline and MEM-MVCC listdir latency fluctuated around 5 s after sustained deletions; locality-aware compaction alone kept it between 1 s and 3 s.Source: Figure 14 · p. 9, §7.4 · p. 9, Figure 15 · p. 9
- In-memory MVCC configurations produced about 60% fewer LSM-tree writes, while compaction reads were lower than with Baseline and Loc.Comp.Loc.Comp had the highest compaction reads because compaction was triggered more frequently.Source: Figure 16 · p. 9, §7.4 · p. 9, Figure 17 · p. 9
7.5 Overall Performance Stress Testing
- 2.2×–4.1× throughput and 23.1%–75.0% lower latency than GP-DB were measured in a production-cluster test of five metadata operations.The test used one 10-BE-node cluster, 20 clients with 50 threads each, and mdtest.Source: Figure 18 · p. 10, §7.5 · p. 10
7.6 MetaDB in Production
- Nine representative deployments span BOS, CFS, and Mantle; BOS’s largest cluster had ∼500 nodes, 1.3 to 1.8 trillion records, and peaks of 2M read and 0.7M write QPS.The deployment table summarizes nine representative MetaDB deployments.Source: Table 1 · p. 10, §7.6 · p. 10
- Multi-shard directory-modification transactions fell from 100% to 0.79%–4.37% across four Mantle and CFS clusters.Remaining multi-shard transactions were higher in Mantle because path resolution uses additional index entries on other shards; in CFS, they came from cross-directory rename.Source: Table 2 · p. 10, §7.6 · p. 10
- On a high-deletion-throughput BOS node, tombstones peaked at ∼6M and varied around 3M; production listdir latency decreased from around 8–18 ms to 0.5–4 ms.The production latency observations span two CFS clusters over 21 days and are not intended as strict performance comparisons.Source: Figure 19 · p. 10, §7.6 · pp. 10–11, Figure 20 · p. 11
8 Experiences and Lessons
MetaDB’s experience favors controlled integration of directory semantics with database capabilities, while keeping the implementation focused and management scalable.
Decoupling vs. Controlled Coupling?
- Integrating directory indexing through internal tables and native transactions replaced cross-system two-phase commits and reused database replication and recovery.The earlier separation also required its own replication, failure detection, and recovery; integration saved roughly six months of work for a ten-engineer team.Source: §8 · p. 11
Building from Scratch vs. Modifying General-Purpose Distributed Databases?
- MetaDB was built from scratch with RocksDB rather than deeply modifying a large general-purpose database, yielding a ∼150K-line codebase.The trade-off is implementing some features that general-purpose databases already provide, a cost the authors say has remained manageable because metadata requirements and access patterns changed little.Source: §8 · p. 11
SQL Interface vs. Dedicated API?
- A lightweight direct-execution framework maps ∼20 fixed SQL patterns to storage paths, avoiding the full general SQL processing pipeline.Keeping SQL eased storage-service migration to MetaDB.Source: §8 · p. 11
Single Manager vs. Two-Tier Management?; Central Services: Simplicity vs. Availability?
- Independent Cells with Local Managers keep per-shard heartbeats and scheduling off the Global Manager; MetaCache and a Hybrid Logical Clock address central-service risks.MetaCache serves routing lookups, while the clock lets each node generate timestamps locally instead of relying on a write-blocking TSO.Source: §8 · pp. 11–12
9 Related Work
Related work spans distributed file-system metadata services, database-backed metadata systems, and techniques for asynchronous indexing and LSM-tree deletion.
Distributed Metadata Management
- Distributed metadata systems place metadata by subtree or directory, split large directories, separate directory and file metadata, or colocate parent and child metadata.Examples include CephFS, Lustre DNE, IndexFS, LocoFS, and InfiniFS.Source: §9 · p. 12
Distributed Metadata Management; Distributed Databases; Asynchronous Auxiliary Metadata Maintenance
- Database-backed systems differ in transaction and sharding choices; MetaDB specializes sharding, transaction processing, and storage behavior around metadata semantics.HopsFS [31] uses row locks and two-phase commit; Tectonic does not guarantee atomic cross-directory rename. Prior asynchronous-index work focuses mainly on indexing or scalability, unlike MetaDB’s transaction-path restructuring.Source: §9 · p. 12
LSM-Tree-Based KV Store Deletion
- Prior deletion approaches use latency bounds or retention windows, whereas MetaDB targets deletion hotspots by density to avoid time-driven compaction overhead.The cited retention strategies require intrusive engine modifications and incur high write amplification; marked deletion’s LSM-tree effects are discussed in.Source: §9 · p. 12
10 Conclusion
MetaDB co-designs a distributed database around file-system metadata properties and reports production gains in performance and efficiency. Its results support exploiting workload-specific structure inside the database for metadata management.
10 Conclusion
- MetaDB achieves 2.2×–4.1× the previous general-purpose database’s throughput, halves production metadata machines, and sustains stable low-latency range scans under continuous bulk deletions.The paper presents workload-specific structure as a key design principle for performance and efficiency in storage-system metadata management.Source: §10 · p. 12
A Performance Comparison with TiDB
Against TiDB, MetaDB reports higher throughput for create with index updates and same-directory concurrent mkdir, and lower listdir latency during consecutive bulk deletions.
A Performance Comparison with TiDB
- 85.6K QPS with 256 clients was 2.2× TiDB’s throughput for create with index updates, with MetaDB updating indexes asynchronously and TiDB atomically.Each client sent four concurrent requests.Source: A Performance Comparison with TiDB · p. 16
- 36.2× the peak throughput of TiDB was achieved by MetaDB for concurrent mkdir operations modifying the same parent-directory row.Each client sent four concurrent requests under the same directory.Source: A Performance Comparison with TiDB · p. 16
- During object deletions at 5K QPS, MetaDB listdir latency stayed below 0.7 s, while TiDB peaked at 2.5 s and remained between 1 s and 2.5 s after 30 minutes of accumulated tombstones.Source: A Performance Comparison with TiDB · p. 16