Source-linked AI summary
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
Maximilian Böther, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
TL;DR
Online foundation-model pipelines use stateful n-to-m transformations that make deterministic loading, topology changes, and efficient checkpoint resumption difficult. Zephon uses lanes and serialized ordering decisions to support these pipelines, with bounded-cost resumption. It reports near-linear scaling with Grain and much smaller GPU-count score spreads than loading without elastic determinism.
Problem
Method
Zephon partitions the stream into deterministic lanes and serializes ordering decisions while parallelizing stateless work on interchangeable backends.
Source: §4.2 · p. 4, §10 · p. 12Results
Zephon scales near-linearly with Grain and limits GPU-count score spread to 0.061 pp on Core v1 and 0.007 pp on FineWeb, versus 0.922 pp and 0.555 pp without elastic determinism.
Source: §8.1.2 · p. 11, §8.3 · p. 12Takeaways & Limitations
Zephon supports more reliable ablations while reducing compute spend through higher, more stable VLM throughput and low startup time on trillion-token-scale text corpora.
Source: §10 · p. 12Elastic comparisons require the same semantic input stream and operator graph, and supported executions require G_e|L and m_e|q.
Source: §6.2 · p. 7
Abstract
from arXiv · showhide
Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends. Achieving this is difficult because modern foundation model data pipelines tokenize, pack, and mix samples online, introducing stateful n-to-m transformations that break sample indexing. Existing data loaders largely assume indexable 1-to-1 pipelines, and the common workaround of offline materialization is expensive and, for some modalities such as video, infeasible. We present Zephon, a data loader for foundation models that supports online, stateful pipelines while providing elastic determinism and efficient resumption from checkpoints. It partitions the global stream into topology-independent lanes, serializes ordering decisions while parallelizing stateless work on interchangeable backends, and checkpoints only bounded in-flight state so recovery cost does not grow with training progress. We evaluate Zephon on text and vision-language workloads and show that it achieves competitive throughput while providing a combination of guarantees that no existing loader offers for online, stateful pipelines.
1 Introduction
Foundation-model ablations need deterministic data ordering because costly comparisons are confounded by changes in training inputs. Zephon targets online, stateful pipelines with topology-independent batches and bounded-cost resumption.
1 Introduction
- Data-order variation confounds ablation results and impedes debugging, while even smaller foundation-model ablations can consume tens of thousands of H100 GPU-hours.The cited 1.5-orders-smaller example costs roughly 30 k to 50 k H100 GPU-hours.Source: §1 · pp. 1–2
- Elastic determinism requires identical global batches across GPU topologies, and checkpoint recovery must avoid replay costs that grow with training progress.Topology can change when jobs are preempted or allocations scale; replays from the beginning leave GPUs idle.Source: §1 · p. 2
- Stateful n-to-m pipelines spread progress across in-flight state and reordered records, making exact restoration across topology or backend changes difficult.Tokenization can split documents, packing merges sequences, and mixing reorders samples.Source: §1 · p. 2
- Offline materialization makes pipelines indexable but is prohibitively expensive to repeat for changing petabyte-scale corpora and infeasible for some modalities such as video.Teams using such modalities may build proprietary loading infrastructure.Source: §1 · p. 2
- Zephon partitions the stream into lanes and serializes stateful ordering decisions while running stateless transformations in parallel on interchangeable backends.The paper introduces accumulators and per-offset contributor tracking for operators that reorder, merge, or spawn samples.Source: §1 · p. 2
- Zephon checkpoints stateful pipelines with bounded cost and provides an open-source, composable loader for foundation-model operators.The paper presents it as the first loader to combine elastic determinism and efficient resumption for online, stateful n-to-m pipelines.Source: §1 · p. 2
2 Stateful 𝑛-to-𝑚data loading pipelines
Foundation-model preprocessing combines cardinality-changing operators with state that carries across inputs. These properties break indexability and make resumption dependent on more than a sample offset.
2 Stateful 𝑛-to-𝑚data loading pipelines
- A representative VLM pipeline processes image-caption pairs, interleaved documents, and text-only samples through decoding, transforms, templating, tokenization, shuffling, packing, and batching.Source: §2 · p. 3
- Pipelines combine one-to-one decoding and augmentation with splitting, packing, and filtering that change record cardinality.Tokenization can be 1-to-n, packing n-to-1, and filtering 1-to-0.Source: §2 · p. 3
- Packing bins, shuffle buffers, and mixture counters make later outputs depend on the history of prior inputs.Source: §2 · p. 3
- A pipeline is indexable only when the source records for step t can be located without evaluating the preceding input prefix.Data-dependent splitting and accumulated operator state break this property and complicate resumption from an arbitrary offset.Source: §2 · p. 3
3 Requirements overview
Non-indexable loaders must preserve logical output order while meeting throughput, composability, elastic-determinism, and checkpoint-recovery requirements. These demands rule out relying on static sample offsets and partitioning alone.
3 Requirements overview
- High-throughput loading must decouple parallel execution from logical output order and support different execution substrates.Data-dependent output boundaries make straightforward static partitioning unavailable.Source: §3 · p. 3
- A loader must let teams compose custom operators while remaining independent of file format, storage backend, and training framework.Source: §3 · p. 3
- For a fixed logical pipeline, elastic determinism requires the same ordered global batches across operator parallelism, execution substrate, and data-parallel degree changes.The requirement applies at constant global batch size, including for non-indexable pipelines without static sample indices.Source: §3 · p. 3
- Checkpoint recovery for non-indexable pipelines must have bounded cost regardless of training progress, rather than replaying the pipeline from the start.Progress is entangled across operator state such as bins, buffers, and counters.Source: §3 · p. 3
4 Zephon’s architecture and abstractions
Zephon separates curriculum selection from preprocessing and expresses loading as composable operators over lane-specific streams. Its abstractions cover user-built pipelines, data access, and curriculum chunking.
4 Zephon’s architecture and abstractions
- A WorkSource defines the training curriculum, while composable pipeline operators transform its data into training batches.Source: §4 · p. 3
4.1 User interface
- Zephon offers a Python builder-pattern interface supporting user-defined functions and arbitrary payloads.Users specify a WorkSource and attach an operator chain to build a pipeline.Source: §4.1 · p. 3
4.2 Component overview
- The global stream is partitioned into fixed lanes, each with a deterministic, non-overlapping slice and independent operator state.The WorkSource emits chunks of pointers per lane; Fetch reads their payloads before records flow through the user-defined operator graph.Source: §4.2 · p. 4, Figure 3 · p. 4
4.3 Data model
- Zephon models datasets as shards and addresses each sample by dataset, shard, and within-shard position.The shard-based abstraction is not tied to one file format.Source: §4.3 · p. 4
4.4 WorkSource
- The WorkSource protocol produces deterministic chunk streams for each lane and decouples curriculum specification from downstream preprocessing.The current implementation uses file-system paths, but the protocol can support other backends such as a centralized sample-management server.Source: §4.4 · p. 4
- Chunks contain fixed-size pointer sequences weighted across mixture components, with deterministic interleaving and configurable within-component ordering.A WorkSource can adjust mixture weights between chunks, supporting dynamic curricula.Source: §4.4 · p. 4
5 Execution model
Zephon compiles pipelines into runner-assigned stages, using stateful accumulators to make ordering decisions while parallel workers perform stateless transformations. Bounded queues, interchangeable runners, and user contracts support deterministic execution across supported configurations.
5 Execution model
- Zephon compiles the operator graph into stages with assigned runners, then combines stateful accumulators with parallel stateless workers.The engine orchestrates data flow, while runners provide worker backends.Source: §5 · p. 4, §5.2 · p. 5, §5.3 · p. 5
5.1 Engine and pipeline execution
- Pull-driven stages connect the training loop to the pipeline, while per-operator pumps push records internally and prefetch buffers overlap computation with data movement.Bounded inter-stage queues apply backpressure when downstream stages slow, preventing unbounded memory growth.Source: §5.1 · pp. 4–5
5.2 Stage execution and the operator pump
- Each operator’s accumulator reorganizes incoming microbatches before parallel workers transform them, and a reorder buffer restores dispatch order.For packing, the accumulator holds partially filled bins until they reach the target length; it may not mix records across lanes.Source: §5.2 · p. 5
5.3 The runner
- Zephon currently supports process and thread runners that co-locate data loading with training; elastic determinism under disaggregated preprocessing remains future work.The paper describes disaggregated preprocessing as mostly used in recommendation systems and proprietary generative video models.Source: §5.3 · pp. 5–6
5.3.1 The process runner.
- The process runner coalesces large microbatch payloads into shared-memory regions by dtype, reducing per-payload allocations and file-descriptor pressure.The pipe carries metadata so receivers can reconstruct payloads as zero-copy views.Source: §5.3.1 · p. 6
5.3.2 The thread runner.
- The thread runner avoids serialization and shared-memory management; synchronous future results are buffered locally to prevent a full result queue from deadlocking the pump.Threads can parallelize operators that release the GIL, and free-threaded Python makes threads viable for all operators.Source: §5.3.2 · p. 6
5.4 User obligations and system guarantees
- Custom stateful operators must keep state per lane, base accumulator decisions on input order, and seed randomness from microbatch contents; Zephon probes operators for determinism and metadata handling.Subject to the trainer-consumption contract, Zephon guarantees identical global batch sequences across supported execution configurations.Source: §5.4 · p. 6
6 Deterministic and elastic execution
Zephon defines elastic determinism as invariance of ordered global batches across supported physical executions of the same semantic pipeline. Lane-local ordering, round-robin multiplexing, and trainer accumulation preserve each lane’s contribution as topology changes.
6.1 Determinism by induction
- Induction establishes pipeline determinism: accumulators make ordering decisions, while workers and order-preserving queues carry the resulting sequence forward.Workers may finish out of order, but results are restored by dispatch sequence number before forwarding.Source: §6.1 · p. 7
6.2 Elastic determinism
- Supported executions preserve the same global batch sequence when they retain the semantic input streams and operator graph and satisfy the lane and microbatch divisibility conditions.The conditions are data-parallel degree G dividing L lanes, and training microbatch size m dividing q=N/L.Source: §6.2 · p. 7
- Lane outputs are topology-independent, and round-robin multiplexing restores lane order when processing rates differ; each optimizer step consumes q records from every lane.Changing lane-to-replica assignment changes where contributions are produced, not the logical global batch.Source: §6.2 · p. 7
- Identical global batches do not guarantee bit-identical losses or model states because GPU topology can change floating-point accumulation and reduction order.Zephon’s guarantee ends at the ordered global batches.Source: §6.2 · p. 7
7 Checkpointing and resumption
Zephon checkpoints bounded in-flight state and resumes by replaying those chunks, rather than replaying the pipeline from the beginning. Contributor tracking and flush-based segments let it determine when chunks are safe to evict, including when operators reorder records or retain history-dependent state.
7 Checkpointing and resumption
- Checkpoints save pointers for in-flight chunks alongside WorkSource state, bounding stored state by backpressure rather than training progress.Only chunks whose samples entered the pipeline but have not all been delivered need to be saved and replayed.Source: §7 · p. 7
7.1 Cursors and contributors
- Cursors identify records within lanes, while contributor references track every source record in a derived output and mark when its last child is delivered.A packed record has a primary replay cursor but can carry contributors from multiple source records.Source: §7.1 · pp. 7–8
7.2 Replay
- On resume, Zephon replays in-flight chunks through the deterministic pipeline, filters the already-delivered prefix by cursor, then fetches new chunks.A global checkpoint lets ranks restore whichever lanes they are assigned after a topology change.Source: §7.2 · p. 8
7.3 Eviction
- For non-monotone pipelines, chunk order cannot determine completion because records may combine data across chunks and arrive out of order.Zephon tracks per-offset completion and evicts a chunk only when all its offsets are closed; dropped last-child records require tombstones to close their offsets.Source: §7.3 · p. 8
- Completion bitmaps alone cannot restore history-dependent accumulators, so Zephon inserts flush sentinels every K chunks to emit buffered data and reset state.A smaller K permits more frequent chunk eviction but can increase padding waste by flushing partially filled packing bins.Source: §7.3 · pp. 8–9
- Batch is the exception to immediate flush handling: it may finish a partial batch because it preserves cursor order and has no history-dependent state beyond its buffer.The replay filter precedes Batch so it rebuilds from an empty buffer at the batch-aligned cut.Source: §7.3 · p. 9
- A segment is safe to evict only after all its offsets close and the replay cursor has moved beyond it; monotone pipelines can instead evict by watermark.Non-monotone segments are evicted atomically because chunks within a segment share accumulator history.Source: §7.3 · pp. 8–9
8 Evaluation
Evaluation measures throughput and recovery costs, then tests whether GPU-count changes affect downstream scores and VLM loss trajectories. Across these experiments, Zephon combines competitive throughput with topology-independent batch ordering and bounded checkpoint state.
8.1 Throughput
- Benchmarks use raw data-loader throughput, with local NVMe storage and three random seeds, to measure pipeline performance rather than cloud-storage bandwidth.The evaluation compares fetch-and-batch workloads with a non-monotone VLM pipeline.Source: §8.1 · p. 9
8.1.1 Fetch-and-batch.
- 16 k samples/s on VLM data gave Zephon’s process runner 2.7× the best baseline; text throughput was sufficiently high for stall-free training.Figure 6 reports fetch-and-batch throughput; the text process runner reached 59 k samples/s, while disabling SHM coalescing reduced VLM throughput from 16 k to 2.4 k samples/s.Source: Figure 6 · p. 10, §8.1.1 · p. 10
- Disabling deterministic result reordering raised process-runner peaks from 59 k/16 k to 62.1 k/20.5 k samples/s for text/VLM, while removing further guarantees cost 16%/22%.The stripped pipeline could neither order output nor checkpoint.Source: §8.1.1 · p. 10
- On Python 3.14t, Zephon’s thread runner reached 59 k samples/s on VLM data, surpassing its process runner’s 27 k by 2.2×.On text, its thread runner improved from 34 k to 140 k samples/s.Source: §8.1.1 · p. 10
8.1.2 VLM pipeline
- At DOP 32, Zephon and Grain reached 29.9 k and 29.8 k utps per GPU; at DOP 64, Grain led Zephon, 28.7 k to 26.6 k.Both systems delivered over 23× InternalDs throughput at DOP 64; Zephon maintained elastically deterministic output ordering.Source: Figure 7 · p. 11, §8.1.2 · p. 11
8.1.3 Time-to-first-batch.
- 7.61 s VLM-pipeline time-to-first-batch with Zephon’s process runner was 17% lower than Grain’s 9.21 s and 13× faster than InternalDs.InternalDs required 101.17 s.Source: §8.1.3 · p. 11
8.2 Checkpointing analysis
- 29.8/30.3 k in-flight records per rank were retained at steps 1,000/10,000, while even worst-case flushing increased token waste by less than 0.37 pp.More lanes per rank increased in-flight state and recovery cost.Source: §8.2 · p. 11
8.3 Elastic determinism
- 0.061 pp on DCLM Core v1 and 0.007 pp on FineWeb were the score spreads across GPU counts with Zephon, versus 0.922 pp and 0.555 pp without elastic determinism.Both Zephon spreads were below the stated meaningful-difference reference levels.Source: Figure 8 · p. 12, §8.3 · p. 12
9 Related work
Foundation-model data work spans filtering, mixing, curation, and processing, while prior systems address different data-loading and determinism needs. Related approaches cover indexed pipelines, recovery, stream processing, and deterministic parallelism.
9 Related work
- Foundation-model data research explores filtering, mixing, and curating or processing training data.Source: §9 · p. 12
- Prior data loaders target operator graphs, specific formats, disaggregated loading, data sharing, or throughput, sometimes trading determinism for throughput.MinatoLoader explicitly makes that trade-off; Mosaic Streaming and Grain support elastic determinism for indexable pipelines, while Grain also supports deterministic resumption for stateful pipelines under fixed execution configurations.Source: §9 · p. 12
- EasyScale and Tenplex preserve sample order across GPU-count changes but assume indexable datasets; Gemini accelerates checkpoint recovery under fixed resources without addressing ordering.Source: §9 · p. 12
- Stream-processing systems checkpoint stateful pipelines, but Zephon’s data-loading pipeline is a simpler single-input operator chain without joins or triggers.The paper says Zephon could also serve stateful n-to-m dataflow graphs that require reproducible ordered output.Source: §9 · p. 12
- Deterministic parallelism has been studied at the language, compiler, and operating-system levels.The cited examples include DPJ, CoreDet, Kendo, DThreads, and Determinator [8].Source: §9 · p. 12
10 Conclusion
Zephon combines elastic determinism and efficient resumption for online, stateful n-to-m foundation-model pipelines. The authors report more reliable ablations and reduced compute spend, alongside higher, more stable VLM throughput and low startup time on trillion-token-scale text corpora.
10 Conclusion
- Zephon is presented as the first loader to combine elastic determinism and efficient resumption for online, stateful n-to-m foundation-model pipelines.It serializes ordering decisions in accumulators while parallelizing stateless work on interchangeable backends.Source: §10 · p. 12
- Zephon’s higher and more stable VLM throughput and low startup time on trillion-token-scale text corpora are reported to reduce compute spend.Source: §10 · p. 12
- The authors report that Zephon supports more reliable ablations.Source: §10 · p. 12