Source-linked AI summary
Benchmarking Storage Systems for Machine Learning Workloads Using NIO Bench
Jonathan W. Morris, Ionut Mistreanu, Connor Louie
TL;DR
Machine-learning benchmarks often underdescribe file-system I/O, motivating NIO Bench’s broader characterization of storage behavior across modern architectures. The framework combines phase-aware Python hooks with fork-following strace on a Ceph-backed Kubernetes cluster, finding front-loaded and bursty I/O, extreme byte concentration, and cache-miss read tail latency as the primary bottleneck.
Problem
Existing ML benchmarks focus primarily on computational performance, while file-system I/O behavior and modern architecture-specific storage interactions remain insufficiently characterized.
Method
NIO Bench benchmarks six ML architectures using Python-level tracing for semantic phase context and strace for complete syscall coverage, including DataLoader workers.
Results
Across the evaluated workloads, I/O concentrates in preparation and checkpoint-related phases, training is compute-bound after staging, fewer than 10% of files account for over 90% of bytes, and cache-miss read tail latency is the primary bottleneck.
Takeaways & Limitations
ML-oriented storage systems should prioritize data prefetching, page-cache support, and efficient handling of bursty checkpoint writes.
Abstract
from arXiv · showhide
Machine learning training workloads place unique demands on storage systems, yet most existing benchmarks focus on computational throughput rather than file system I/O behavior. We present a benchmarking framework, Neural I/O Benchmark (NIO Bench), that characterizes storage access patterns across six diverse ML model architectures: Language Transformers, Vision Transformers, Diffusion Models, Spiking Neural Networks, Artificial Neural Networks, and Reinforcement Learning. Our framework employs a two-layer tracing approach combining Python-level I/O hooks for semantic phase context with Linux strace for complete syscall coverage including DataLoader worker subprocesses. We evaluate all six models on a Nautilus Kubernetes cluster with Ceph distributed file system. Our results reveal that I/O is heavily concentrated in data preparation, model loading, and model checkpointing. We also found that training is compute-bound rather than data-bound once data is staged, and that storage access follows an extreme power law where fewer than 10% of files account for over 90% of bytes transferred, and that read tail latency from cache misses on distributed storage is the primary storage bottleneck. These findings suggest that storage systems optimized for ML should prioritize aggressive data prefetching, page cache pinning, and efficient handling of bursty checkpoint writes.
I. INTRODUCTION
NIO Bench addresses the limited characterization of file-system I/O in ML benchmarks by covering contemporary architectures and combining semantic and syscall-level tracing. It aims to reveal architecture-specific access patterns that can guide ML storage-system design.
- Understanding how architectures interact with storage is presented as a prerequisite for designing better ML storage systems.
- NIO Bench characterizes I/O across diverse ML architectures on a Kubernetes cluster and identifies common and model-specific access patterns.
- Existing ML benchmarks primarily measure computational performance rather than file-system I/O behavior.
- Existing I/O-focused work covers older architectures, while modern transformers, diffusion models, and spiking networks remain underrepresented.
- Its two-layer tracing captures application-level semantics and complete syscall-level I/O, including activity missed by application-only instrumentation.
III. BENCHMARK DESIGN
The benchmark spans six model architectures selected to represent distinct modern ML I/O behaviors, while addressing the complementary blind spots of application-level and syscall-level tracing.
- Model Selection: Six model architectures span distinct ML I/O behaviors, including language transformers, artificial neural networks, spiking networks, and vision transformers.The selected workloads include Llama fine-tuning, an in-memory tabular baseline, N-MNIST processing, and COCO image classification.
- Model Selection: The Language Transformer pipeline includes staging, tokenization, sharding, checkpointed training, and merged model export.
- Model Selection: Vision Transformer training produces millions of small JPEG reads through DataLoader worker subprocesses each epoch.
- Tracing Challenge: Application-level tracing misses parallel worker reads, whereas syscall tracing captures them without identifying the responsible training phase.
- Tracing Challenge: The benchmark therefore uses a two-layer tracing architecture combining semantic application instrumentation with complete syscall coverage.
1) Layer 1: Python IOHooks:
The first tracing layer records semantically tagged application I/O, while the second follows all processes to capture complete syscall activity and align both views.
- Layer 1: Python IOHooks:: Python hooks wrap file and checkpoint operations and tag each event with its current training phase.Recorded fields include timestamps, sequence numbers, paths, byte counts, and process or thread identifiers.
- Layer 2: Syscall Tracing: strace with -f captures file-related syscalls from the training process and DataLoader workers.
- Layer 2: Syscall Tracing: The two layers are synchronized with markers recorded at the start and end of each run.
3) Phase-Aware Collection:
Phase markers organize I/O collection across training stages, and experiments run individually on a Ceph-backed Nautilus Kubernetes environment with realistic, time-bounded settings.
- 3) Phase-Aware Collection:: Phase markers distinguish data staging, tokenization, model loading, training, validation, checkpointing, and model export.This enables comparison of sequential sharding writes with bursty checkpoint writes.
- 3) Phase-Aware Collection:: Each model run produces a JSON trace containing timestamped events, with optional strace logs parsed into structured JSON.
- Experimental Environment: Experiments ran on the Nautilus Kubernetes Cluster using Ceph distributed storage through a Kubernetes Persistent Volume Claim.The Ceph backend introduces network latency on each I/O operation relative to local NVMe storage.
- Experimental Environment: Models were run individually with Python-level and strace-level tracing enabled.
- Experimental Environment: Training hyperparameters approximated realistic workloads while keeping total runtime near 2 hours across six models.
V. RESULTS
Training dominates total runtime, while I/O appears in sparse bursts rather than continuously. Python-level tracing identifies these events but substantially understates the bytes transferred during large writes.
- ∼19,000 of ∼22,000 total seconds are spent in training epochs, while all I/O-intensive phases together account for less than 15% of runtime.The measured I/O phases include data staging, checkpointing, model loading, tokenization, and export.
- 6–12 operations per second appear as sparse Python-level spikes during checkpoint saves and model exports, with throughput near zero between spikes.
- 2–3.5 GB/s bursts occur during Diffusion checkpoint saves, while the LT model export reaches 5.3 GB/s.These bursts are visible in strace-level byte measurements.
- Each torch.save call appears as one Python operation despite transferring gigabytes, so application-level tracing understates transferred data.
C. Syscall Distribution by Model
Syscall activity varies sharply across architectures, with ViT generating the most activity through DataLoader workers. Checkpointing is also a dominant component of Python-level file operations across models.
- ∼12 million syscalls make ViT an order of magnitude higher than any other model, driven by DataLoader workers reading 118K JPEG images per epoch.Read and lseek calls account for approximately 6M and 4M of these syscalls, respectively.
- ViT shows only ∼13 Python-level operations despite its 12 million strace syscalls, because DataLoader worker I/O is invisible to the Python trace.
- ∼3M syscalls come from SNN reading N-MNIST binaries, while Diffusion and ANN each generate 1–1.5M syscalls.
- RL produces negligible I/O because its training data is generated in memory.
- 50–95% of traced file operations are checkpointing operations across models, with SNN highest at 95.2% and ANN lowest at 50.0%.
- 65 unique files are written during checkpoint_save across all models, making it the most write-intensive phase.
E. Syscall Latency Analysis
Most syscalls complete quickly from the Linux page cache, but rare cache-miss reads from Ceph create a strongly skewed latency distribution. Other operations remain comparatively stable, with write spikes associated with flushes.
- 0.41 ms mean read latency is 13× higher than the 0.03 ms P95 because rare Ceph cache-miss reads skew the average.Most reads are served quickly from the page cache, while misses fetch data over the network from Ceph OSDs.
- ∼0.02 ms is the consistent latency for lseek, fstat, and close syscalls.
- 0.045 ms is the average write latency, with spikes during fsync or large checkpoint flushes.Writes are buffered by the kernel page cache.
- 0.045 ms is the slightly higher openat P95, attributed to occasional metadata server lookups.
F. Byte Concentration
Bytes transferred are extremely concentrated in a small subset of files, while most touched files contribute negligible data. Checkpoint files are also associated with many short-lived metadata operations.
- Over 90% of bytes come from the top 10% of files, forming an extreme concentration pattern.The bottom 70% of files account for nearly 0% of transferred bytes.
- Checkpoint saves, dataset archives, and Arrow shards are the dominant high-byte files.
- Approximately 100,000 temporary files are created, written, and renamed or deleted within milliseconds.
- Slow create, rename, or delete operations can bottleneck checkpointing.
A. Key Findings
The benchmark identifies front-loaded I/O, power-law byte concentration, and distributed-storage read latency as central storage-system concerns for ML workloads. It therefore motivates prefetching, hot-file placement, checkpoint-aware writes, and efficient metadata handling.
- I/O is front-loaded: I/O is front-loaded: data preparation produces the highest file-operation throughput, while training becomes compute-bound after staging.
- Byte distribution follows a power law: Fewer than 10% of files account for over 90% of bytes transferred.
- Read tail latency is the bottleneck: Read tail latency is the primary bottleneck because Ceph cache misses add 10–100× latency.
- Data prefetching: Pre-staging and pinning datasets in fast storage targets the front-loaded I/O burst.
- Checkpoint aware write path: Asynchronous buffered checkpoint writes can reduce metadata overhead from atomic save-to-temp-then-rename operations.
- Hot-file tiering: Hot-file tiering places the small number of byte-dominant files on the fastest storage tier.
- Efficient metadata operations: Fast create, rename, and delete operations are needed because checkpointing generates many short-lived temporary files.
VII. FUTURE WORK
Future work extends NIO Bench across storage backends, workload settings, model scales, and GPU measurements. These directions test whether the reported patterns generalize beyond the current evaluation.
- Running identical workloads on Lustre, local NVMe, HDFS, and Amazon S3 would isolate storage-architecture effects.
- Inference and multi-GPU training would characterize read-heavy inference and cross-node parameter-synchronization I/O.
- Scaling to 70B+ parameter LLMs would test whether front-loaded I/O, power-law byte concentration, and bursty checkpoints persist at production scale.
- Collecting GPU utilization over time could reveal whether GPUs are data-starved during training.