Source-linked AI summary
SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs
Ruijia Yang, Shiyuan Lin, Yulong Ao, Zhiyu Li, Yingli Zhao, Xianduo Li, Yonghua Lin, Zeyi Wen
TL;DR
Host-resident layer streaming supports full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks contend for shared host resources. SlideDP coordinates host state and pipeline work across ranks, and its matched-batch sweeps report higher throughput than existing offloading systems.
Problem
Host-resident layer streaming addresses GPU-memory limits, but data-parallel ranks compete for shared host resources as replicated transfers amplify traffic.
Source: §1 · p. 1, §2.4 · p. 4Method
SlideDP is a synchronous data-parallel runtime that coordinates parameter delivery, gradient aggregation, and CPU updates around one authoritative host state.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Communication and activation policies should be guided by shared-resource demands and pipeline dependencies, and validated through measurements.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems. It maintains one authoritative host state, decouples communication routes from state layout, and pipelines parameter delivery, gradient aggregation, and CPU updates across ranks and chunks. An analytical step-time model characterizes resource bottlenecks and pipeline exposure; runtime measurements guide communication, chunking, and activation policies under a GPU memory budget. In matched-batch sweeps, SlideDP achieves geometric-mean throughput ratios of 1.46-2.64$\times$ over SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B at a smaller batch size. With a larger batch, it processes over 1M tokens per step and exceeds FSDP2's measured peak throughput by 11.2%. Separately, it supports 256K-token sequences for the same model and fine-tunes Qwen2.5-72B on four RTX 4090 GPUs. Project page: https://github.com/RegiaYoung/SlideDP.
1 Introduction
Host-resident layer streaming addresses the GPU-memory demands of full-parameter fine-tuning, but shared-host data-parallel ranks contend for fixed CPU, DRAM, and host–device resources. SlideDP coordinates state movement and updates across ranks, models pipeline exposure, and selects communication and activation policies using measurements.
1 Introduction
- SlideDP is a synchronous data-parallel runtime that coordinates parameter delivery, gradient aggregation, and layer-wise CPU updates around one authoritative host state.Chunking overlaps parameter conversion with transfers and gradient return with CPU updates while preserving parameter-version and update dependencies.Source: §1 · pp. 1–2
- Shared-host ranks can multiply transfer and gradient-reduction demand, while strong scaling shrinks the computation window available to hide remaining host work.Communication-route costs also vary with topology and workload, and spare GPU memory can reduce different sources of exposed pipeline work.Source: §1 · pp. 1–2
- Its analytical step-time model characterizes host- and GPU-bound execution and explains when reduced traffic shortens a step or shrinking compute windows expose host work.The model incorporates shared-resource demands and cross-layer, cross-rank dependencies, including readiness, buffer reuse, and parameter versions.Source: §1 · p. 2
- AutoPolicy selects communication routes, chunk sizes, and activation layouts using staged profiling and measured costs calibrated in a mini-pipeline.Elastic Checkpointing provides per-layer choices for retention, offloading, and recomputation.Source: §1 · p. 2
- 1.46–2.64× geometric-mean throughput ratios over SlideFormer [35], MegaTrain [37], and ZeRO-Offload [28] in matched-batch sweeps.SlideDP also reaches approximately 85–92% of compute-only reference throughput projected from single-GPU measurements across 4B–72B models on four H100s.Source: §1 · p. 2
2 Background and Motivation
The background traces how full-parameter training states, data-parallel ownership, and host-resident streaming create a coordination problem on shared hosts. Its observations connect rank-scaled resource demand and shrinking computation windows to route and memory-allocation choices.
2.1 Training States in Full-Parameter Fine-Tuning
- Mixed-precision Adam requires approximately 16N bytes for model states, while activations scale with layer count, hidden dimension, sequence length, and batch size.Activation checkpointing and other methods reduce individual footprint components, but do not decide how persistent states and transient execution data are placed and coordinated.Source: §2.1 · pp. 2–3
2.2 GPU-Centric Data Parallelism and Offloading
- DP preserves identical layer computation across ranks, whereas ZeRO and FSDP partition training states but add parameter materialization and gradient synchronization.With CPU or NVMe offload, transfers and host optimizer work can contend with shared PCIe, memory bandwidth, and CPU resources, so state placement alone does not determine step time.Source: §2.1 · p. 3, §2.2 · p. 3
2.3 Host-Memory-Centric Layer Streaming
- Host-memory-centric streaming keeps persistent parameters and optimizer states in host memory, but synchronous DP must coordinate their delivery, gradient aggregation, and updates across ranks.RoundPipe's overlap relies on asynchronous updates and delayed parameter versions; standard synchronous fine-tuning restores the optimizer-update dependency and can reintroduce stalls.Source: §2.3 · pp. 3–4
2.4 Data Parallelism Is Not Free on a Shared Host
- Direct DP extensions can replicate each layer transfer across ranks and return rank-wise gradients for host reduction, adding demand on shared host resources.The resulting need is to coordinate state movement, scheduling, and memory allocation rather than optimize each component independently.Source: §2.4 · pp. 4–5
- On an A800 node, per-GPU cast-and-copy bandwidth falls from 15.6 to 8.7 GB/s with two concurrent GPUs; on the H100 host, doubling CPU Adam cores from 16 to 32 improves speed only 8.5–10.0%.These measurements show that concurrent users and CPU workers do not proportionally increase shared-host service capacity.Source: §2.4 · p. 4, Figure 2 · p. 4
- For Qwen3-14B at fixed global batch 256 on A800s, SlideDP step time rises from 29.6 to 15.3 s as GPUs increase from four to eight, versus SlideFormer remaining near 43 s.The matched compute reference falls from 27.2 to 14.4 s, illustrating the shrinking strong-scaling computation window.Source: §2.4 · p. 4
- Sharded delivery is 17% faster than replicated on bridged 4×A800 at 16 sequences per GPU, while replicated is 26% faster after NVLink bridges are removed.In GPU-bound eight-GPU workloads, the routes differ by under 2.5%, so route choice depends on both topology and workload regime.Source: §2.4 · p. 5
- At 32 sequences per GPU, Qwen3-14B uses 15.0 GiB of peak CUDA reserved memory on an 80-GB H100, leaving headroom for policies that reduce different exposed work.Retaining activations avoids offload and reload traffic, while retaining intermediate results reduces recomputation; their value depends on the exposed pipeline bottleneck.Source: §2.4 · p. 5
3 Design of SlideDP
SlideDP coordinates synchronous multi-GPU layer streaming around one authoritative host state while separating communication routes from state ownership. Its pipeline and measurement-guided policies manage shared-resource demand, activation tradeoffs, and GPU-memory feasibility.
3.1 Overview
- SlideDP coordinates synchronous multi-GPU layer streaming around one authoritative host state, with temporary layer copies in bounded, reusable GPU windows.The runtime jointly schedules parameter delivery, gradient aggregation, buffer reuse, and layer updates while preserving synchronous data-parallel semantics.Source: §3.1 · p. 5
3.2 Shared Host State and Communication Routes
- One host-held FP32 master and Adam state is updated once; routes independently choose parameter delivery and gradient reduction under topology-dependent resource tradeoffs.All ranks use the same parameter version, aggregate gradients across ranks, and wait for the update before next-iteration dispatch.Source: §3.2 · p. 5
3.3 Cross-Rank Pipelining and Chunked Overlap
- Layer- and chunk-level pipelines overlap movement, GPU work, gradient return, and CPU LayerAdam, moving updates out of the step-end serial tail.Per-layer version gates and buffer-completion signals preserve synchronous execution and bound in-flight storage; smaller chunks trade shorter service paths for more launches and queue operations.Source: §3.3 · pp. 5–6
3.4 Modeling Shared-Host Scaling
- The step-time model combines resource service demands with operation dependencies, separating compute and host bottlenecks from exposed pipeline delay.Under strong scaling, the computation window can shrink while host service remains nearly constant; under weak scaling, throughput may grow superlinearly while H2D is limiting, until shared host capacity or all-gather takes over.Source: §3.4 · pp. 6–7
3.5 Elastic Checkpointing
- Elastic Checkpointing varies checkpoint granularity, activation retention or offload, and recomputation to trade GPU memory against transfers, replay, and exposed execution work.AutoPolicy compares measured candidates using pipeline effects and accounts for transient replay peaks before allocating layers.Source: §3.5 · p. 7
3.6 Measurement-Guided Policy Selection
- AutoPolicy probes routes and chunks, then measures checkpoint candidates and allocates layer layouts under GPU-memory and optional host-memory or activation-transfer budgets.Short full-model runs validate feasibility and step time; reserved-memory feedback supports bounded re-solves, retaining the fastest validated feasible configuration.Source: §3.6 · p. 8
4 Evaluation
Across three multi-GPU platforms, the evaluation measures throughput, memory use, shared-host runtime behavior, scaling, policy selection, MoE support, and training-loss agreement. SlideDP sustains throughput across larger and longer workloads while selected runtime policies address changing hardware and workload constraints.
4.1 Experimental Setup
- The evaluation uses RTX 4090, A800, and H100 systems, mixed-precision full-parameter fine-tuning, and matched-batch comparisons against offloading and GPU-resident baselines.Throughput is global tokens per step divided by steady-state step time; policy-selection overhead is reported separately.Source: §4.1 · p. 9
4.2 End-to-End Throughput
- On H100, SlideDP sustains 1.8–2.0 PFLOPS across 4B–72B models and exceeds FSDP2 by 11.2% for Qwen3-14B; it also completes Qwen3-14B at 256K tokens.At 72B, SlideDP reaches 1.88× ZeRO-Offload throughput with a fourfold larger batch; at 256K, it reaches 1.86 PFLOPS.Source: §4.2 · p. 10
4.3 Memory Efficiency and Capacity
- With full-layer checkpointing and activation offload, SlideDP cuts Qwen3-14B peak reserved GPU memory by 51–59% versus ZeRO-Offload and 29–42% versus MegaTrain.On RTX 4090, Qwen3-8B host PSS is 106–109 GiB for SlideDP versus 183 GiB for ZeRO-Offload and 211–212 GiB for MegaTrain; Figure 12 estimates 276K-token capacity for 14B models.Source: §4.3 · p. 10
4.4 Shared-Host Runtime Mechanisms
- With 64 MiB chunks, conversion plus H2D falls from 127.8 to 50.8 ms under contention, while Qwen3-8B throughput rises 7.7% on four RTX 4090s.The four-GPU comparison keeps layer-wise prefetching and CPU allocation fixed.Source: §4.4 · p. 11
4.5 Understanding Shared-Host Scaling
- For Qwen3-32B, SlideDP reaches 7,963 tokens/s on eight A800s with 95–98% weak-scaling efficiency; for Qwen3-14B, sharded delivery gains 16.8–32.7% with NVLink bridges in host-bound workloads.Without NVLink bridges, replicated delivery instead gains 19.2–26.0%; in GPU-bound eight-GPU workloads, route combinations remain within 2.5%.Source: §4.5 · pp. 11–12
4.6 Measurement-Guided Policy Selection
- Policy selection raises Qwen3-14B H100 throughput at batch 16 by 25.4%; selected layouts also gain 12.4% at batch 32 by using 76.6 GiB instead of 15.0 GiB.On four H100s, a separate selection run takes 20.3 minutes and has an estimated break-even of 1,257 steps.Source: §4.6 · p. 12
4.7 MoE Training and Correctness
- On four H100s, SlideDP reaches 44.7K and 27.3K tokens/s for two MoE models, while 200-step Qwen3-14B loss comparisons with FSDP2 show maximum absolute differences of 5.2 × 10−4 and 6.3 × 10−4.The loss differences are for F-r1 and the selected layout, respectively.Source: §4.7 · p. 12, Table 4 · p. 12
5 Conclusion
SlideDP is a synchronous data-parallel runtime for full-parameter LLM fine-tuning with shared host resources. Shared host state, cross-rank pipelining, an analytical model, and measurement-guided policies coordinate communication and memory use to sustain throughput across workstations and multi-GPU servers.
5 Conclusion
- SlideDP coordinates parameter delivery, gradient aggregation, and layer-wise CPU updates through shared state and cross-rank pipelining, reducing redundant host work while overlapping remaining transfers and updates with GPU computation.Measurements guide communication and activation policies for each workload and topology.Source: §5 · p. 13