PaperlayerView exact revision on arXiv

Source-linked AI summary

VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets

Cong Wang, Zelin Liu, Yang Luo Ran Zhang, Zhijian Guo, Hui Zhang, Fan Yu, Yanfei Cao, Naijie Gu, Jun Yu

arXiv:2608.22795v1cs.CV

TL;DR

Diverse AI datasets and storage formats can make data loading inefficient as hardware accelerates model computation. VersaDB unifies these datasets with separated page-based storage, B+ tree indexing, sharding, hierarchical metadata, and conversion APIs, achieving up to 5.35x acceleration while showing architecture- and dataset-specific limitations.

  • Problem

    AI datasets span modalities and storage formats, while existing processing architectures can provide inefficient or inconsistent data access as training throughput demands increase.

  • Method

    VersaDB combines separated raw and blob storage, page-based access, B+ tree indexing, two-level sharding, hierarchical metadata, and dataset-conversion APIs.

  • Results

    VersaDB maintained performance advantages across most tested datasets, tasks, parallelism levels, and environments, with acceleration ranging from 1.06x to 5.35x except for The Pile in Machine Environment 2.

  • Takeaways & Limitations

    VersaDB provides a unified AI dataset storage system with strong performance and resource management across dataset categories and machine environments.

  • Takeaways & Limitations

    Performance on The Pile dataset in the AARCH64 architecture was not ideal, motivating further AARCH64 optimization.

Abstract

from arXiv · show

The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.

I. Introduction W

VersaDB addresses the difficulty of efficiently reading diverse, large AI datasets by separating data types, indexing storage, and organizing metadata. It combines these mechanisms with sharding, dynamic access features, and format-conversion APIs, achieving up to 5.35x acceleration in evaluated tasks.

  • AI datasets increasingly combine multiple modalities and storage formats, while faster AI hardware makes data loading a potential training bottleneck.
  • Existing solutions have limitations including restricted random access, framework incompatibility, and sequential reading and writing constraints.
  • VersaDB separates structured raw data from unstructured blob data and uses page-based storage for direct random access through page offsets.
  • A separate B+ tree-based index file accelerates data access, while hierarchical metadata records global, shard, and page-level information.
  • VersaDB supports two-level sharding, dynamic field extension, diverse data types, hierarchical locking, and reading without affecting write operations.
  • 5.35x acceleration was achieved under default conditions across pure reading, shuffle reading, and preprocessing tests on image, text, and audio datasets.

II. Background

AI training systems rely on varied frameworks, data formats, and distributed or multimodal workflows, creating demands for efficient unified storage. VersaDB is presented as a hierarchical storage and management framework targeting these requirements.

  • AI training loads batches into host memory, parses modality-specific formats, and preprocesses data before model execution.
  • Its workflow parses input according to a user-defined schema, separates structured and unstructured data, generates statistics and shard metadata, and constructs indexes.
  • PyTorch, TensorFlow, and MindSpore provide distinct data-processing workflows, while JSON, BIN, CSV, and TFRecord offer different storage trade-offs.
  • Distributed training divides data across compute cards, and multimodal training combines information from diverse data types through feature-level or decision-level fusion.
  • VersaDB uses a hierarchical framework with a binary data file, a separate index file, schema and metadata management, sharding, dynamic pages, and parallel operators.

IV. hierarchical file structure

VersaDB’s file structure separates metadata, structured records, and unstructured objects while pairing page-based data storage with an external B+ tree index. This design supports direct page access, blob localization, and sequential or random reading.

  • The data file contains a header with metadata, a raw data section for structured records, and a blob data section for unstructured content.
  • Raw data uses pages as minimal storage units with schema-defined contents and direct random access through page offsets.
  • Blob storage uses a header offset table, with each variable-length entry containing a size field followed by its data.
  • The external index file uses B+ tree leaf key-offset pairs, separator keys, and linked leaves to support retrieval and range queries.
  • The architecture combines structured and unstructured storage separation, multilevel indexing, and fine-grained metadata for sequential and random access.
  • FileWriter, FileReader, and VersaPage provide buffered writing, multithreaded sequential or random reading, prefetching, field-selective access, and LRU page caching.
  • Dataset conversion modules apply format-specific parsing within a shared write pipeline, including parallel image decoding and streaming TFRecord parsing.

VI. Schema and Metadata Management

VersaDB manages schemas and dataset metadata hierarchically to support structured access, schema evolution, distributed consistency, and statistics. Its shard system partitions data physically and maps records logically for distribution and load balancing.

  • Schema Management: VersaDB schemas encode field names, types, and shapes, supporting primitive and complex types through a composable hierarchy.
  • Schema Management: Binary schema encoding and recursive validation reduce schema overhead and process nested structures according to their hierarchy.
  • Schema Management: Schema evolution records additions, deletions, and modifications in versioned header metadata to resolve older schemas during reads.
  • Metadata Management: Metadata is organized at global, shard, and page levels, with compact encoding, lazy loading, and LRU caching.
  • Metadata Management: Metadata updates use two-phase commit and write-ahead logging to provide atomic updates and crash recovery.
  • Statistic Management: Statistics use global, shard, and page counters, distributed aggregation, and HyperLogLog cardinality estimation to reduce memory usage with bounded error.
  • Shard Management: VersaDB uses fixed-size physical shards and a distributed hash table that maps record keys to shards for data distribution and load balancing.
  • Shard Management: Shard writes use locking for consistency, while MVCC permits concurrent reads without blocking writes.

B. Statistic Management

VersaDB manages statistics hierarchically across global, shard, and page levels while supporting indexing for structured and unstructured data. Its indexing and aggregation designs trade immediate global consistency or update cost for scalable query and write performance.

  • Statistic Management: Hierarchical counters track statistics at global, shard, and page levels through distributed aggregation.Statistics are generated during file creation and updated during writes, with HyperLogLog supporting memory-efficient cardinality estimation.
  • Statistic Management: Eventual consistency makes local shard statistics immediate while global counters are consolidated asynchronously.The design improves write performance while allowing statistical queries to eventually reflect the complete dataset.
  • Statistic Management: Hybrid indexing combines B+ trees for structured data with inverted indexes for unstructured content.The system supports both range-based and exact-match queries through field-specific index structures.
  • Statistic Management: LSM-based index updates buffer new entries in memory before background compaction into disk-based structures.Bloom filters at each index level reduce unnecessary disk accesses during query processing.
  • Statistic Management: Vector timestamps maintain consistent index views across distributed nodes and help detect conflicts during concurrent modifications.Range partitioning enables parallel query processing across nodes while maintaining global consistency.

VIII. Page Management

VersaDB organizes data into dynamically managed pages within shards, combining page-level metadata, compression, indexing, buffering, and execution operators. Its sampling, shuffling, and pipeline mechanisms target efficient access and processing for AI workloads.

  • VIII. Page Management: Pages use hybrid fixed-header and variable-length storage, with validity bitmaps, checksums, metadata, and slot-based record locations.When pages exceed thresholds, records are redistributed while preserving links for sequential access and data locality.
  • VIII. Page Management: Page-level LZ4 compression is combined with column-specific schemes to balance storage efficiency, access speed, and computational overhead.The outer layer compresses whole pages, while the inner layer adapts compression to data characteristics.
  • VIII. Page Management: B+ trees organize page metadata and physical locations, while a clock-based buffer pool caches frequently accessed pages.Dirty pages are persisted before clean pages during memory pressure.
  • VIII. Page Management: Row-group storage clusters similar structured records, reducing I/O for column-specific scans and providing statistics for query planning.Page-level record distributions and value ranges inform optimizer access decisions.
  • VIII. Page Management: Sampling uses a fixed-size reservoir with weighted selection probabilities to achieve O(1) space complexity and representative category coverage.The implementation processes category data and extends balanced samples before returning them.
  • VIII. Page Management: Category-aware shuffling reconstructs a sequence by interleaving independently permuted category buffers while preserving relative proportions.The modified distributed Fisher-Yates approach achieves O(n) time complexity.
  • VIII. Page Management: Prefetch, process, and write stages overlap I/O with computation through lock-free queues, with adaptive read-ahead and SIMD transformations where applicable.This pipeline architecture targets execution efficiency across data-processing stages.

B. Parallelization Support

VersaDB supports parallel execution through work stealing, hierarchical locking, timestamp ordering, and shard-grouped write-ahead logging. The evaluation compares these capabilities across AI data structures, datasets, machines, and resource-related metrics.

  • B. Parallelization Support: Work stealing dynamically balances parallel writes when a writer exhausts its local task queue.Each writer maintains a local queue and steals tasks from other queues with minimal synchronization overhead.
  • B. Parallelization Support: Hierarchical locks operate at shard level for writes and page level for reads, enabling concurrent access to different pages within a shard.Timeout-based lock abortion prevents deadlocks when requests cannot be satisfied within the specified window.
  • B. Parallelization Support: Multi-version timestamp ordering detects conflicts by comparing write timestamps during parallel operations.Write-ahead logs are grouped by shard and flushed with group commit to support durability while optimizing I/O.
  • B. Parallelization Support: The evaluation compares LMDB, TFRecord, VersaDB, and raw loading across sequential access, random access, preprocessing, and resource utilization.Each storage method is read through its corresponding AI training framework.
  • B. Parallelization Support: Experiments use two distinct hardware configurations and datasets spanning computer vision, NLP, video, multimodal, and audio domains.These settings broaden evaluation across hardware and modality categories.

B. Performance analysis

VersaDB is evaluated on eleven datasets across pure reading, shuffle reading, and preprocessing tasks in two machine environments. It generally outperforms LMDB and TFRecord, with gains reaching 5.35x and increasing for more complex data types.

  • B. Performance analysis: Across eleven datasets, VersaDB is tested on pure reading, shuffle reading, and preprocessing in two machine environments.Preprocessing is omitted for Laion, Youtube8M, and The Pile because these large-scale datasets are typically preprocessed and dimensionality-reduced.
  • B. Performance analysis: 5.35x speedup is achieved in Machine Environment 2 for shuffle reading on the VOC dataset versus LMDB and TFRecord.In that environment, VersaDB fails to maintain superior performance only on The Pile.
  • B. Performance analysis: In Machine 1, VersaDB speedups range from 1.2x to 3.9x for pure reading, 1.06x to 4.3x for shuffle reading, and 1.09x to 1.87x for preprocessing.These ranges compare VersaDB with LMDB and TFRecord across the tested datasets.
  • B. Performance analysis: In Machine Environment 2, VersaDB speedups range from 1.1x to 2.49x for pure reading, 1.09x to 5.35x for shuffle reading, and 1.06x to 4.12x for preprocessing.The reported ranges exclude The Pile for the discussion of performance improvements.
  • B. Performance analysis: Performance advantages increase across datasets of increasing complexity, indicating particular strength with sophisticated data types and multimodal datasets.The paper attributes these benefits to efficient data organization and processing mechanisms.

C. Comparison of Resource consumption

VersaDB generally reduces memory consumption and maintains lower reading times across parallelism levels, though gains vary by dataset and environment.

  • Memory consumption: VersaDB reduced memory consumption across several datasets in Machine Environment 1, including 93.02% for ImageNet and 91.2% for SQuAD.The reported reductions also included COCO, Kitti, LibriTTS, LJSpeech, VOC, and WikiText.
  • Memory consumption: 30.6% of TFRecord’s peak memory accompanied 4.4x acceleration on Laion, while The Pile reached 130% of TFRecord’s memory.These results show that memory and acceleration outcomes can diverge by dataset.
  • Memory consumption: VersaDB reduced memory consumption on COCO, ImageNet, Kitti, Laion, and VOC in Machine Environment 2, but The Pile reached 160% of TFRecord.The reported reductions were 49.8%, 81.05%, 78.19%, 22.39%, and 79.51%, respectively.
  • Parallel performance: VersaDB had lower times than LMDB and TFRecord across all five tested parallelism levels in Machine Environment 1.Its acceleration remained high for several datasets, but approached TFRecord on ImageNet in some parallel environments.
  • Parallel performance: VersaDB had shorter run times across all five parallelism levels and datasets except The Pile in Machine Environment 2.Acceleration remained high except on ImageNet and The Pile, with VersaDB and TFRecord converging on ImageNet as parallelism increased.
  • Parallel performance: The authors attribute parallel-performance gains to page-based organization and independent worker scheduling that minimize thread contention.This explanation is presented as an architectural reason for VersaDB’s performance across parallel environments.

E. Data parallel reading

VersaDB uses distributed data parallelism across sharded datasets and achieves substantial multi-node speedups on text, image, and audio workloads.

  • Experimental setup: VersaDB was evaluated with data parallelism on Kitti, LJSpeech, and WikiText using 2, 4, and 8 nodes.The number of VersaDB shards matched the number of nodes on an eight-GPU NVIDIA 3090 cluster.
  • Results: 40x speedup was achieved on LJSpeech with eight nodes, alongside up to 5.3x on Kitti and 2.3x on WikiText.These results are reported in the distributed-node performance comparison.
  • Architecture: VersaDB separates scalar and blob data to support storage and retrieval strategies tailored to different data types.Scalar data includes labels and metadata, while blob data includes images and audio.
  • Architecture: VersaDB uses adaptive indexing that dynamically optimizes access according to data characteristics, especially for random access patterns.This indexing mechanism is contrasted with LMDB’s B+ tree reliance.
  • Architecture: VersaDB’s flexible schema organizes complex data structures differently from TFRecord’s fixed serialization and LMDB’s key-value paradigm.The schema is intended to tailor representation and access patterns to specific data types.
  • Results: The distributed comparison is summarized in a table of performance under different numbers of nodes.The cited table is the reported source for the multi-node comparison.

B. Resource management

VersaDB emphasizes memory-efficient multimodal data management while maintaining strong performance across tasks, machines, parallelism levels, and distributed settings, with identified architecture-specific limitations.

  • Memory management: Selective loading, dynamic page replacement, and scalar-blob separation are identified as mechanisms supporting memory efficiency.These strategies target data-component loading and memory allocation according to usage patterns.
  • Practical consequence: Reduced memory usage is presented as enabling larger datasets to run on existing infrastructure with lower hardware requirements and costs.This is the paper’s stated practical consequence of its resource-efficiency improvements.
  • Overall performance: VersaDB maintained performance advantages across Pure Reading, Shuffle Reading, and Preprocessing, with acceleration from 1.06x to 5.35x under default settings.The exception was The Pile in Machine Environment 2.
  • Overall performance: VersaDB showed performance advantages across tested parallelism levels and distributed environments, except for The Pile in Machine Environment 2.The evaluation covered NLP, computer vision, multimodal, and audio datasets.
  • Memory management: VersaDB generally used less memory than LMDB and TFRecord in Machine Environment 1, but used more memory for audio and NLP datasets in Machine Environment 2.This environment-dependent pattern qualifies the overall resource-management result.
  • Overall conclusion: VersaDB is described as adaptable across dataset categories, machine environments, and architectures while maintaining strong performance and resource management.This conclusion summarizes the reported evaluation scope.
  • Limitations: On AARCH64, VersaDB consumed more memory than LMDB and TFRecord for NLP and audio datasets; on x86-64, this occurred only for The Pile.This is an architecture-specific memory limitation.
  • Limitations: Performance on The Pile in AARCH64 was not ideal, motivating future optimization for consistent performance across mainstream architectures.The authors identify this as a future-work priority.
Loading 2608.22795v1…