ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development

• LFM training spans pre-training, post-training, and concurrent evaluation, while checkpoints support state resumption and transfers to downstream tasks.The pipeline is resource-intensive and time-consuming because of model size and training-data scale.Source: §1 · p. 1
• Changes in parallelism after failures, resource shifts, configuration changes, or transitions between stages require resharding saved checkpoints to minimize overhead and maximize ETTR.Source: §1 · p. 1
• Different users choose varied training frameworks and storage backends, making customized checkpoint modules and workflows complex to develop and maintain.Source: §1 · p. 2
• ByteCheckpoint separates tensor-shard metadata from values in a global file and stores dataloader states according to whether they are replicated or sharded.Source: §1 · p. 2
• Framework-specific planners generate unified saving and loading plans for an agnostic I/O engine, while full-stack optimizations support training a 405B LFM on 8,960 GPUs.The same saving and loading steps serve different frameworks and storage backends.Source: §1 · p. 2
• Compared with PyTorch DCP and Megatron Distributed Checkpoint, ByteCheckpoint reduces checkpoint stalls by 12.13× to 161.50× and makes saving and loading 6.05× and 3.88× faster on average, respectively.Source: §1 · p. 2

2.1 LFM development

• Checkpoints are needed across pre-training, post-training, and evaluation, and preserve GPU and CPU training states in persistent storage.GPU states include model parameters and optimizer information; CPU states include dataloader, RNG, training-step, and learning-rate-scheduler state.Source: §2.1 · pp. 2–3

2.2 Checkpoint Resharding Scenarios

• Resharding supports training resumption after resource or configuration changes, cross-stage transitions to post-training, and evaluation on separately assigned resources.The platform recorded 1,870 pre-training-resumption, 13,080 cross-stage-reconfiguration, and 19,844 evaluation resharding instances over six months.Source: §2.2 · p. 3
• 19,844 evaluation resharding instances were recorded, compared with 13,080 for cross-stage reconfiguration and 1,870 for pre-training resumption over six months.The counts cover training text, image, and video generation models on the AI platform.Source: §2.2 · p. 3

2.3 LFM Checkpointing Requirements

• Offline resharding scripts require separate jobs to download, transform, and re-upload checkpoints, delaying training or evaluation and producing parallelism-specific checkpoints that increase storage overhead.Table 1 is described as reporting average completion times for offline resharding jobs under different scenarios, but its values are not supplied here.Source: §2.3 · pp. 3–4, Table 1 · p. 4
• Load-time resharding retrieves necessary data from existing distributed checkpoints during loading, requiring checkpoint storage representation independent of specific parallelism.The section also identifies a need for generic workflows because framework-specific checkpoint modules and storage choices complicate consistent saving, loading, and transfer.Source: §2.3 · p. 4
• Large-scale checkpointing must maintain I/O performance: massive storage requests can overload metadata services, while naive collective communications can cause timeouts and job failures.The paper identifies performance analysis and error detection across workers and machines as increasingly challenging at scale.Source: §2.3 · p. 4
• Prior systems either assume consistent parallelism or have limits in resharding support, framework coverage, I/O performance, and scalability; ByteCheckpoint targets these gaps with decoupled storage and full-stack optimizations.The paper also proposes generic saving/loading workflows for multiple frameworks and storage backends.Source: §2.3 · pp. 4–5

3.1 Unified Architecture

• The unified API abstracts sharding, plan generation, and I/O, while separate Planner, Engine, and Storage I/O layers accommodate different frameworks and storage backends.Users call bytecheckpoint.save and bytecheckpoint.load; the Planner generates plans, the Engine executes them, and Storage I/O handles backend-specific operations.Source: §3.1 · p. 5

3.2 Decoupled Checkpoint Representation

• ShardMeta records each tensor shard’s position in the original tensor, allowing checkpoint representation to remain independent of training parallelism.The global metadata file consolidates shard metadata; dataloader states are separated into replicated and sharded states, with sharded states stored in individual files.Source: §3.2 · p. 6
• Irregular tensor shards are decomposed into regular tensors represented by multiple ShardMeta entries, avoiding the communication overhead of gathering complete tensors before saving.The decomposition slightly increases metadata size and may require additional lookup steps during loading.Source: §3.2 · pp. 6–7

3.3 Checkpoint Resharding Workflow

• During load-time resharding, planners match saved tensor segments to new shard specifications, distribute optimized load plans, and execute loading across ranks.Dataloader resharding copies token buffers when only non-DP parallelism changes, and splits or merges them when the DP degree changes.Source: §3.3 · p. 7

4.1 Optimized Plan Generation

• Worst-Fit workload assignment balances saving across ranks, loading avoids duplicate tensor reads across DP groups, and caching reuses save plans and metadata within a training session.For a 405B transformer across 8,960 GPUs, saving-plan generation takes 62 seconds; caching makes planning a one-time cost.Source: §4.1 · p. 8

4.2 Fully Asynchronous Engine Pipeline

• The Engine pipelines file reading, deserialization, H2D copies, and inter-GPU communication; its saving pipeline similarly overlaps D2H copies, serialization, and uploads.Pinned CPU memory and Ping-Pong buffering are used to accelerate D2H copies and mitigate their training impact.Source: §4.2 · p. 8, Figure 10 · p. 8

4.3 High Performance Read/Write

• Multi-threaded random reads from HDFS raise tested single-file read speed from 400 MB/s to 2–3 GB/s, while concurrent sub-file writes can reach 3 GB/s without network congestion.For writes, metadata-level concatenation rejoins fixed-size sub-files after upload.Source: §4.3 · p. 8

4.4 Prefetching Dataloader States

• Prefetching lets dataloader read workers prepare state before checkpointing, reducing state-gathering delays that otherwise pause training.With four workers and about 1GB of state, collection typically takes around 8 seconds; prefetching achieves near-zero gathering delay.Source: §4.4 · p. 9

5.1 Storage Support

• The HDFS deployment rewrites components in C++, adds NNProxy for NameNode metadata requests, and uses SSD-to-HDD cooldown to manage checkpoint storage.The optimized HDFS supports hundreds of PBs, 10 TB/s read/write bandwidth, and approximately 100,000 metadata QPS.Source: §5.1 · p. 9

5.2 Collective Communication

• At 8,960 GPUs, NCCL channel initialization and GPU-memory use caused delays and sometimes OOM errors; ByteCheckpoint moved planning communication to gRPC and a hierarchical tree topology.The tree reduces coordinator burden from centralized gathering and scattering at tens of thousands of GPUs.Source: §5.2 · p. 9

5.3 Monitoring and Analysis

• Rank-level phase visualizations and storage-client and cluster metrics help identify stragglers, slow I/O, upload failures, and capacity-related availability risks.The visualizations include topology-wide saving-time heat maps and per-rank phase breakdowns.Source: §5.3 · p. 10, Figure 11 · p. 10, Figure 12 · p. 10

6.1 Checkpointing Efficiency

• Checkpoint stalls fall by 13.06×–161.50×, saving averages 6.05× faster, loading improves by 1.41×–8.80×, and average ETTR improves by 1.16×–1.29× versus baselines.The saving-time speedup increases from 2.21× at 2,400 GPUs to 8.87× at 4,800 GPUs; model and optimizer resharding averages 3.64× acceleration.Source: Table 4 · p. 11, §6.1 · pp. 11–12

6.2 Microbenchmarks

• Microbenchmarks report average saving time reduced from 48.3 seconds to 19.27 seconds, and ByteCheckpoint’s irregular-tensor processing has 0.20s average blocking time, 25.15× shorter than FSDP.Asynchronous saving yields 1.45× and 1.81× speedups for tGPT 13B and tGPT 30B, respectively; loading times for those models are 1.53× and 1.58× shorter than baseline.Source: §6.2 · p. 12, Table 5 · p. 12, Table 6 · p. 12

6.3 Correctness Verification

• After PP and TP resharding, tGPT 13B’s normalized loss curve matches the prior phase and continues its decline; unit tests also confirm resharding across parallelism configurations.DP and hybrid resharding results are reported in Appendix E.Source: §6.3 · p. 12, Figure 13 · p. 13
• After resuming a 175B production model without changing parallelism, normalized loss before and after resuming is exactly the same.Figure 14 shows the production run across 2,080 H800 GPUs over several days.Source: §6.3 · p. 12, Figure 14 · p. 13

6.4 Checkpointing in Real LFM Production

• Production runs maintained average checkpoint stalls under 600ms, including at 8,960 GPUs, with end-to-end checkpointing typically completed within a few seconds.The reported deployments trained transformer-based image- and text-generation models on H800 GPUs.Source: §6.4 · p. 12, Table 8 · p. 13
• Monitoring exposed sequential dataloader-file uploads and serial HDFS metadata concatenation as bottlenecks; concurrent upload and concatenation changes addressed these issues.For a 7B vision transformer, dataloader-state uploads accounted for 73.16% of total saving time.Source: §6.4 · p. 13

7 Conclusion

• Checkpoint stalls fall by up to 161.50× and end-to-end checkpointing completion time shortens by up to 9.96× versus state-of-the-art baselines.The loading (resharding) procedure is also 3.88× faster on average.Source: §7 · p. 13
• ByteCheckpoint uses a unified checkpoint representation and workflow to enable efficient load-time resharding across multiple training frameworks and storage backends.The system also includes full-stack I/O and scalability optimizations and performance monitoring and analysis tools.Source: §7 · p. 13
• The authors present ByteCheckpoint as a source of practical experience for real-world LFM checkpointing and insights for future research.Source: §7 · p. 13

A More Background on Maintaining Resharding Scripts

• Offline GPU-state resharding scripts must cover model and optimizer components and their differing behavior under parallelism strategies.The largest production script contains 3193 lines of Python code.Source: A More Background on Maintaining… · p. 18
• Tensor Parallelism shards GEMM operators in attention and MLP blocks along different dimensions, while replicating operators such as LayerNorm across GPUs.Source: A More Background on Maintaining… · p. 18
• Under hybrid 3D parallelism, TP-sharded optimizer tensors are flattened and merged before being sharded for the designated Data Parallelism degree.Source: A More Background on Maintaining… · p. 18
• GQA and MLA can change operator tensor layouts, requiring corresponding resharding support in the scripts.The passage gives the attention-block query-key-value projection GEMM as an example.Source: A More Background on Maintaining… · p. 18

B Efficient Integrity Guarantee

• A complete checkpoint spans files stored by multiple workers, and one worker failure can corrupt the entire checkpoint.A barrier is used to provide atomic save/load operations across workers.Source: B Efficient Integrity Guarantee · p. 18
• At approximately 10,000 GPUs, the framework barrier can stall training for about 20 seconds per checkpoint operation.The approach synchronizes workers until all checkpoint saving or loading operations finish.Source: B Efficient Integrity Guarantee · p. 18
• ByteCheckpoint reimplements the barrier with gRPC tree-based communication and asynchronous integrity checks to eliminate blocking time.Its I/O workers also retry uploads and downloads and log the failure stage for incomplete checkpoint tasks.Source: B Efficient Integrity Guarantee · p. 18

C Average ETTR Calculation

• The wasted-time calculation assumes failures are evenly distributed within a checkpoint interval, from just after a completed save to just before one.Source: C Average ETTR Calculation · p. 18
• Average wasted time is derived using per-iteration training time, checkpoint interval, end-to-end save time, and loading (resharding) time.Source: C Average ETTR Calculation · p. 18
• Table 4's end-to-end ETTR results are averaged across standard loading and resharding settings.Source: C Average ETTR Calculation · p. 18

D Checkpointing Overhead Breakdown

• Planning communication overhead increases with training scale and causes significant stalls, but caching makes it a one-time cost per resumed training session.Other phases are averaged across checkpointing steps.Source: D Checkpointing Overhead Breakdown · p. 18
• Pinned-memory CPU pooling makes D2H blocking almost negligible, while an asynchronous engine overlaps serialization, shared-memory dumping, and HDFS uploading.Load balancing also enables parallel uploads within each DP group; at 4800 GPUs, model-state upload is 3.03× faster than at 2400 GPUs.Source: D Checkpointing Overhead Breakdown · p. 18
• Table 9 breaks down rank 0's end-to-end checkpoint saving time across procedure phases.Costs for phases other than planning are averaged over checkpointing steps.Source: D Checkpointing Overhead Breakdown · p. 18, Table 9 · p. 19

E More Resharding Correctness Experiments

• The normalized loss curves for DP and hybrid resharding correctness are shown in Fig. 16; when global batch size is also increased, the post-resharding loss curve declines more rapidly.Source: E More Resharding Correctness… · p. 19
• The normalized data sample length is identical before and after restarts, demonstrating bit-wise correct resuming of dataloader states with a fixed RNG state.The normalized sample-length curve is used to evaluate whether restarts preserve the same data-sampling trajectory.Source: E More Resharding Correctness… · p. 19

Checkpointing frameworks

• DCP supports FSDP resharding but not TP or PP, DeepSpeed-UCP uses offline resharding scripts, and Megatron MCP extends DCP's storage options.The passage characterizes these frameworks as limited in parallelism and framework support, with I/O performance that does not scale well for large training.Source: F Related Work · p. 19

Checkpoint representation

• ByteCheckpoint adopts DCP's metadata-separated representation, adapting it for irregular tensor sharding, and exports Safetensors checkpoints for Hugging Face compatibility.DCP metadata includes global shape and offset information for automatic resharding; Tenplex's Parallelizable Tensor Collection represents and transforms tensor states across parallelism configurations.Source: F Related Work · p. 20

Storage systems for LFM

• LLaMA 3.1 identifies Tectonic as Meta's checkpoint-storage backbone, while DeepSeek's FireFlyer uses its custom 3FS distributed file system.The cited LLaMA 3.1 report notes high-frequency checkpoint writes as a Tectonic challenge.Source: F Related Work · p. 20

Elastic training systems

• The cited elastic-training systems are limited to specific parallelism strategies, whereas ByteCheckpoint's representation is described as strategy-agnostic for production training.The passage says adopting this representation could enhance the flexibility of elastic training systems.Source: F Related Work · p. 20