PaperlayerView exact revision on arXiv

Source-linked AI summary

FFSlim: An Efficient and Lightweight Format for Multi-modal Data Storage and Retrieval

Long Yang, Yu Mao, Yuchen Shao, Yumiao Zhao, Yaqi Li, Xuan Liu, Xiaolong Shen, Tao Yu, Gezi Li, Jing Wang, Chengcheng Wan, Liang Shi

arXiv:2608.27865v1cs.PF

TL;DR

Multi-modal storage formats poorly fit datasets in which one media object has multiple associated texts, contributing to storage, caching and indexing inefficiencies. FFSlim reorganizes data at Media2Texts granularity with adaptive retrieval and indexing. It reports higher read and write throughput, minimal resource overhead, and reduced end-to-end training time across seven models.

  • Problem

    Existing formats flatten one-to-many multi-modal datasets into independent media–text pairs, producing redundant storage, cache-unfriendly access and indexing overhead.

  • Method

    FFSlim stores each media once with associated texts, retrieves complete Media2Texts objects for cache reuse, and adaptively selects lightweight indexing structures.

  • Results

    FFSlim achieves average read and write throughput improvements of 2.07× and 8.26×, respectively, while maintaining minimal storage and index overhead.

  • Takeaways & Limitations

    FFSlim reduces total training time by 5.36%–14.18% across seven diverse multi-modal models compared with the best-performing baseline.

  • Takeaways & Limitations

    When the media-to-text ratio is small, complete-object retrieval introduces read amplification; indexing and cache behavior also vary with dataset characteristics.

Abstract

from arXiv · show

With the rapid expansion of large-scale media-text corpora, multi-modal datasets increasingly require efficient storage and retrieval. Existing formats such as Files, TDP, and FFRecord work adequately for uni-modal data but expose fundamental limitations in multi-modal settings, including storage redundancy, massive small-file overheads, cache-unfriendly layouts, and heavy index structures. These issues jointly inflate storage and memory usage and make I/O the dominant bottleneck in real training workloads. We present FFSlim, a lightweight format for storing and retrieving multi-modal data. FFSlim improves storage efficiency and loading throughput through three components: a unified file format that removes media duplication and avoids small-file proliferation; an adaptive retrieval mechanism that enables low-overhead pair-level access and accelerates repeated media loading; and a redundancy detection and aggregation module that converts existing datasets into the FFSlim layout. The experimental results demonstrate that FFSlim achieves 2.07x and 8.26x higher data loading and write throughput on average than the strongest baseline, with minimal storage and index overhead. Consequently, these underlying I/O accelerations enable FFSlim to reduce end-to-end training time by 5.36%-14.18% across seven diverse multi-modal models.

I. INTRODUCTION

Multi-modal training faces severe I/O bottlenecks because existing storage formats mismatch the one-to-many structure of real datasets. FFSlim addresses this mismatch with unified Media2Texts storage, adaptive retrieval and indexing, achieving higher throughput and lower system overhead.

  • Data I/O has become a severe bottleneck in multi-modal training as accelerator performance outpaces storage-device performance.
  • FFSlim stores each media once with its associated texts at Media2Texts granularity, avoiding repeated media storage and enabling cache reuse across samples.
  • FFSlim uses a single large file and adaptive sample-level indexing to eliminate massive small-file overhead and avoid complex one-size-fits-all indexes.
  • 5.36%–14.18% lower total training time is reported across seven diverse multi-modal models compared with the best-performing baseline.
  • Existing formats assume one-to-one pairing, whereas multi-modal datasets commonly associate one media object with multiple texts across image–text, video–text and audio–text data.

II. EXISTING STORAGE FORMATS AND THEIR STRUCTURAL MISMATCH

Existing multi-modal storage formats organize data around independent media–text pairs despite datasets’ one-to-many structure. This structural mismatch contributes to redundant storage, small-file proliferation, cache-unfriendly layouts and indexing overhead.

  • II. EXISTING STORAGE FORMATS AND THEIR STRUCTURAL MISMATCH: The paper evaluates Files, TDP and FFRecord as representative multi-modal storage formats across seven representative datasets.
  • A. Overview of Multi-modal Datasets: Multi-modal datasets span image–text, video–text and other modality combinations, with diverse large-scale corpora supporting vision, language and speech tasks.
  • B. Existing methods: Figure 2 compares the data pipelines used by the three commonly used storage formats.
  • B. Existing methods: Files stores each sample’s media and text as separate files, creating a very large collection of heterogeneous small files and computing indices during training.
  • B. Existing methods: TDP stores text and alignment metadata in one table while retaining individual media files, using in-memory paths and metadata to reduce media duplication.
  • B. Existing methods: FFRecord serializes each media–text pair into a single large file and maintains an in-memory offsets array for random sample access.
  • B. Existing methods: All three formats use the media–text pair as their storage and access unit, flattening datasets instead of representing one-to-many Media2Texts relationships.
  • B. Existing methods: Pair-level granularity creates a structural mismatch with actual multi-modal organization and is closely related to four practical bottlenecks.

C. From Structural Mismatch to System-Level Inefficiencies

Pair-level formats fail to match the one-to-many structure of multi-modal data, producing redundant storage and inefficient access. Existing alternatives trade storage improvements against indexing, caching, or random-access costs.

  • System-Level Trade-offs: No existing format addresses storage redundancy, small-file overhead, cache-unfriendly layouts, and indexing overhead simultaneously.The comparison identifies a shared design failure: existing formats do not account for multi-modal structural characteristics.
  • Storage Redundancy: One-to-many data makes pair-level representations store each media object repeatedly, causing storage consumption to grow approximately linearly with the media-to-text ratio.For a 1:N dataset, each media object is stored N times, and media size dominates text size.
  • Storage Redundancy: Files and FFRecord use significantly more storage than TDP because they duplicate media across media–text pairs.TDP replaces duplicated media with lightweight links by separating modalities.
  • Cache-Unfriendly Data Layouts: Pair-level formats prevent caches from recognizing repeated media, whereas TDP enables shared-media reuse through references.Files and FFRecord store redundant copies as distinct files or serialized objects from the cache’s perspective.
  • Cache-Unfriendly Data Layouts: On Flickr30k, TDP throughput rises with cache capacity and approaches FFRecord at roughly 70% dataset-cache coverage, while Files and FFRecord remain nearly unchanged.The result indicates that default caching does not exploit latent media reuse for Files and FFRecord.

3) Indexing Overhead:

Indexing and file granularity create distinct scalability barriers. Pair-level metadata consumes memory, while massive numbers of small files degrade random access and filesystem management.

  • Indexing Overhead: Pair-level indexes scale with the number of media–text pairs rather than the number of media objects, increasing memory requirements for one-to-many datasets.Additional text and alignment metadata can amplify this overhead.
  • Indexing Overhead: TDP and FFRecord provide fast lookup but incur substantial indexing memory overhead through in-memory text, alignment data, or sample-level offsets.This overhead can limit scaling to very large datasets.
  • Large Numbers of Small Files: Files produces mS files for S samples with m modalities, making filesystem and metadata management problematic at web scale.A Files-style representation of WebLI would exceed 200 billion files.
  • Large Numbers of Small Files: FFRecord avoids the small-file problem with one large file, but retains pair-level layout and duplicates media.TDP reduces file counts by aggregating text while retaining separate media files.
  • Random Access: Across seven datasets, FFRecord achieves substantially higher sample-level throughput than Files and TDP under zero cache capacity.This comparison confirms that many small files can severely degrade random-access performance.

III. DESIGN OF FFSLIM

FFSlim is organized around Media2Texts objects to align storage with multi-modal one-to-many structure. Its ULHFF format combines deduplicated media, contiguous storage, and dataset-adaptive indexing.

  • System Architecture: FFSlim comprises ULHFF, Adaptive Sample Retrieval, and a conversion tool that respectively store data, retrieve pairs, and convert existing datasets.The components are tightly coupled around Media2Texts-granularity storage.
  • ULHFF: ULHFF stores each media object once with all associated texts, removing cross-modal redundancy and enabling cache reuse across samples.The design exploits the typical asymmetry in which media objects are much larger than text entries.
  • ULHFF: ULHFF organizes all data into a single contiguous file containing a header and M serialized Media2Texts objects.The layout avoids filesystem overhead from many small files and exposes a predictable byte stream.
  • Indexing: The header records pairing type, logical pair and object counts, object offsets, and fixed-ratio metadata when applicable.Object offsets enable direct offset-based retrieval, while fixed ratios support constant-time arithmetic indexing.
  • Indexing: AVLplus indexes Media2Texts objects rather than pairs, providing O(log M) lookup with memory proportional to the number of objects.This trades modest lookup complexity for substantially lower index memory on large hybrid datasets.

B. ASR: Adaptive Sample Retrieval

ASR bridges Media2Texts-level storage and pair-level training by locating objects adaptively and extracting requested pairs. The conversion tool deduplicates and aggregates legacy datasets into indexed ULHFF files.

  • ASR Retrieval: ASR retrieves complete Media2Texts objects, then extracts the requested media–text pair after an offset-based read.Loading extra text entries is expected to add little cost when media dominates object size.
  • Adaptive Indexing: 63.64–85.82% lower indexing memory is achieved by AVLplus versus hash-based designs for hybrid one-to-many datasets, while retaining logarithmic-time lookup.ASR selects indexing structures according to the dataset’s pairing paradigm.
  • ASR Retrieval: When a requested object is already cached, ASR can serve the read from the OS page cache; otherwise it fetches the object from storage.This preserves pair-level access while exploiting object-level locality.
  • Adaptive Indexing: AVLplus maps a logical sample ID to its containing object and intra-object offset through height-balanced range search in O(log M) time.M denotes the number of Media2Texts objects.
  • Conversion Tool: The conversion tool detects duplicate media, aggregates associated texts into Media2Texts objects, and builds dataset-specific metadata and indexes.It produces a self-contained ULHFF file used by ASR during training.

D. Implementation and Training Integration

FFSlim integrates with standard deep learning workflows through conversion and training stages, supporting pair-level access and diverse multi-modal training strategies. Its implementation uses adaptive indexing and Media2Texts objects to manage retrieval and caching.

  • FFSlim provides a practical storage and retrieval subsystem that integrates with existing deep learning workflows.
  • Conversion and training workflow: During conversion, FFSlim detects duplicate media, aggregates associated texts into Media2Texts objects, serializes them, and generates a dataset-specific lightweight index.
  • Conversion and training workflow: During training, standard DataLoader requests are resolved from logical sample IDs to Media2Texts objects through adaptive sample retrieval.
  • Training-strategy support: FFSlim supports training regimes that either sample all texts associated with each media or sample a single text per media per epoch.
  • Evaluation scope: The evaluation covers loading, writing, storage, index memory, cache sensitivity, indexing tradeoffs, end-to-end training, and ablations across seven representative datasets.

A. Experimental Setup

The experiments compare FFSlim with Files, TDP, and FFRecord across seven multi-modal datasets under fully randomized access. They measure loading and writing throughput alongside storage, index memory, cache sensitivity, and end-to-end efficiency.

  • Metrics: Data loading performance measures samples loaded per second under fully randomized access, averaged across one training epoch.
  • Metrics: Write performance measures throughput for writing the entire dataset, reported as the average when copying it to a parallel file system.
  • Metrics: Storage consumption includes on-disk data and estimated filesystem metadata overhead, while index footprint measures runtime memory used by in-memory indexes.
  • Workloads and baselines: The evaluation benchmarks FFSlim against raw files, TDP, and FFRecord across seven image–text, video–text, and audio–text datasets.
  • Cache configuration: The default effective OS page-cache capacity is x% of dataset size, with x swept from 10% to 100% across ten configurations.
  • Throughput results: FFSlim achieves the highest loading throughput, averaging 2.07× over the best-performing baseline, and the highest write throughput, averaging 8.26× over that baseline.
  • Dataset-dependent comparisons: Higher media-to-text ratios increase FFSlim’s gains over FFRecord on GQA and MSRVTT, while TDP’s relative advantage is larger for datasets with few large media objects.

C. Storage Consumption

FFSlim minimizes storage and index-memory overhead by storing each media object once with its associated texts and using lightweight, dataset-adaptive indexing. Across seven datasets, it has the smallest storage footprint and substantially lower index memory than the baselines.

  • Storage organization: FFSlim stores multiple text entries with one shared media object and organizes the dataset as a single file, reducing redundancy and metadata overhead.
  • Storage consumption: 2.09% average storage reduction versus the best-performing baseline gives FFSlim the smallest storage footprint across all seven datasets.
  • Baseline comparison: FFRecord averages 8.28× FFSlim’s storage consumption, while TDP averages 1.02× and Files averages 8.58×.
  • Index-memory motivation: Index memory matters because datasets exceed memory capacity and excessive indexing can trigger OS-level swapping that increases training time.
  • Index-memory efficiency: FFSlim’s index averages 0.0043% of dataset size and consumes 4.3% of memory when the dataset is 1000× larger than memory.
  • Index-memory comparison: FFSlim reduces index memory by 72.23% versus FFRecord and 99.26% versus TDP on average across seven datasets.

E. Impact of Cache Capacity

FFSlim maintains strong loading performance across cache capacities and reduces index memory with little performance loss through adaptive index selection. In end-to-end training, its faster random access translates into shorter epochs than TDP.

  • Cache-capacity sensitivity: FFSlim consistently outperforms all three baselines on Flickr30k across different cache-capacity settings.
  • Cache-capacity sensitivity: 2.25× average loading improvement over the best-performing baseline on Flickr30k shows effectiveness across a wide range of cache capacities.
  • Baseline cache behavior: Files and FFRecord remain stable across cache capacities, whereas TDP improves as cache increases and surpasses FFRecord above 70% of dataset size.
  • Adaptive-index tradeoff: The adaptive index reduces memory usage by 87.24% on average while incurring only a 0.26% average performance loss versus the hash index.
  • Adaptive-index tradeoff: For hybrid one-to-many datasets, AVLplus Tree with a Media2Texts offset array trades 0.51% and 1.35% performance drops for 63.64% and 85.82% memory reductions.
  • End-to-end training: FFSlim reduces total epoch training time by 5.36%–14.18% compared with TDP across the evaluated end-to-end workloads.

H. Ablation Study

The ablation study shows that both caching and unified formatting substantially improve data loading, while dataset characteristics determine how storage and retrieval overhead emerge.

  • Component ablations: 59.45% average performance drop follows cache removal across seven datasets, showing the dominant contribution of reuse within cached Media2Texts objects.Removing the format causes a further 79.44% degradation by reintroducing many small files and slowing retrieval.
  • Component ablations: 79.44% further performance degradation follows format removal, because unified storage prevents small-file overhead during sample retrieval.
  • Dataset characteristics: Increasing media count makes Files and TDP generate enormous numbers of media files, whereas FFRecord and FFSlim serialize datasets into one large file.TDP also grows fastest in index memory because it builds sample-level indexes and embeds text content directly.
  • Dataset characteristics: For a 1 : N media-to-text ratio, Files and FFRecord duplicate media nearly N×, while TDP and FFSlim maintain almost no redundancy.Higher ratios also increase cache value and loading performance for TDP and FFSlim, whereas Files and FFRecord remain cache-unfriendly.
  • Dataset characteristics: As media size increases, TDP’s read performance approaches FFSlim, but Files continues suffering from text-generated small files.
  • Caching limitation: FFSlim currently relies on the default page cache, whose LRU policy and per-node maintenance do not fit irregular multimodal access patterns.The paper proposes a future modality-aware, centrally managed caching framework for constrained memory budgets.

VI. RELATED WORK

Related work addresses deep-learning I/O through serialization, file-system design, prefetching, and cache management. FFSlim instead redesigns multimodal layout and retrieval around one-to-many Media2Texts structure, while remaining complementary to these systems.

  • Dataset serialization: Serialization formats such as TFRecord, WebDataset, and MDS use coarse-grained shuffling and sequential streaming to increase throughput, but may compromise strict global randomness.
  • Multimodal storage formats: Existing formats flatten multimodal datasets into isolated media–text pairs, ignoring one-to-many Media2Texts relationships and creating I/O inefficiencies and overhead.
  • I/O acceleration systems: Deep-learning I/O acceleration spans file-system design, prefetching mechanisms, and cache management as three orthogonal dimensions.
  • File-system optimizations: Distributed file systems reduce small-file metadata pressure through redesigned directory semantics, scalable key-value or serverless backends, and reduced synchronization bottlenecks.These approaches include specialized AI parallel file systems such as 3FS for extreme raw throughput.
  • Prefetching optimizations: PyTorch and storage systems use multithreading and asynchronous I/O to overlap prefetching with computation, masking storage latency and preventing GPU starvation.
  • Cache management: Traditional LRU caching can thrash under epoch-level random reshuffling, motivating workload-aware policies such as Quiver’s benefit-based eviction and CoorDL’s static pinning.
  • Relationship to FFSlim: FFSlim focuses on data layout format and can integrate with backend storage and runtime optimizations to amplify their benefits.
Loading 2608.27865v1…