Source-linked AI summary
TRANSIT: Transparent Scale-in for Multi-Node LLM Training
Hyungyo Kim, Nicholas Satchanov, Hrishi Shah, Gaohan Ye, Jiaqi Lou, Robert Walkup, Shweta Salaria, I-Hsin Chung, Hubertus Franke, Seetharami Seelam, Apoorve Mohan, Nam Sung Kim
TL;DR
GPU memory limits, infrastructure constraints, and network costs make multi-node LLM training on a fixed GPU allocation challenging. TRANSIT transparently extends GPU memory with CPU DRAM using runtime-guided UVM policies. It outperforms framework-managed offloading and supports scale-in while retaining high per-GPU throughput.
Problem
GPU memory limits force scale-out, while limited availability of the exact requested GPU count can leave training jobs waiting to start.
Source: §1 · p. 1Method
TRANSIT uses a user-space interposition layer and runtime-guided UVM policies to use CPU DRAM as an extension of GPU memory without modifying training applications or frameworks.
Source: §1 · pp. 2–3Results
TRANSIT maintains over 90% of baseline per-GPU TFLOPS with up to 50% fewer GPUs and outperforms framework-managed offloading techniques.
Source: §1 · p. 3Takeaways & Limitations
Scale-in can reduce collective participants and network traffic while lowering cluster resource pressure.
Source: §1 · p. 3, §7 · p. 13TRANSIT-Advise installs zero-copy mappings for all spilled data, leaving little opportunity for prefetching; efficient strategies must also account for computation, communication, and offloading costs.
Source: §5 · p. 12
Abstract
from arXiv · showhide
TRANSIT is a transparent scale-in framework to enable multi-node model training on fewer GPUs while maintaining training efficiency by transparently leveraging CPU DRAM as an extension of GPU memory during distributed training. It achieves this through a user-space interposition layer, requiring no modifications to the application, training framework, cluster scheduler, device driver, or operating system. Furthermore, TRANSIT achieves higher efficiency by leveraging a zero-copy data path for CPU-GPU transfers. We evaluate TRANSIT on dense and MoE models across scales up to 64 NVIDIA H100 GPUs and multiple parallelism configurations over a RoCE network. Our evaluation shows that TRANSIT can: (a) outperform state-of-the-art framework-managed offloading techniques, achieving up to 68%, 59%, and 42% higher per-GPU throughput than TorchTitan, ZeRO-Offload, and ZeRO-Infinity, respectively, (b) enables training with 50% fewer GPUs while maintaining over 90% of baseline per-GPU throughput, (c) lower per-node network traffic by up to 33%, and (d) improve per-GPU throughput by up to 35% in communication-bound settings.
1 Introduction
GPU memory limits drive scale-out, coupling memory capacity to network and synchronization costs while exact-GPU scheduling can delay jobs. TRANSIT uses CPU DRAM and runtime-guided UVM policies to scale in transparently; it outperforms framework-managed offloading and preserves per-GPU efficiency with fewer GPUs.
Motivation
- GPU memory limits push practitioners to scale out, increasing collective communication and network sensitivity, while jobs may wait when their exact requested GPU count is unavailable.GPU shortages, procurement overhead, export controls, and budgets can also leave organizations without enough capacity to start training.Source: §1 · p. 1
Prior Art
- Prior offloading systems require framework-specific integration and often move data coarsely, while prior UVM optimizations rely on privileged system changes or lack demonstrated distributed scale.The stated gap is an approach that is transparent, efficient, deployable, and scalable across distributed multi-node training.Source: §1 · p. 2
Requirements and Approach
- TRANSIT is designed to preserve training efficiency, avoid changes to applications and system software, work in multi-node settings, and minimize CPU–GPU data movement.Its user-space shared library is injected at job launch, and reducing the minimum node count also reduces participants in collective communication and per-node network traffic.Source: §1 · pp. 2–3
Our Approach
- TRANSIT transparently redirects GPU allocations to UVM-backed memory and uses runtime activity records to guide prefetching and zero-copy access to CPU DRAM.Frequently reused data can remain in GPU memory while less frequently accessed data is accessed directly from host memory, reducing unnecessary migrations and page faults.Source: §1 · p. 2
Key Results
- TRANSIT achieves 68%, 59%, and 42% higher per-GPU TFLOPS than TorchTitan, ZeRO-Offload, and ZeRO-Infinity, respectively, and maintains over 90% baseline per-GPU TFLOPS with up to 50% fewer GPUs.The results also report up to 3.4× higher throughput than reactive page-fault-driven UVM and cluster-scale benefits in trace simulation.Source: §1 · p. 3
2 Background and Motivation
Distributed LLM training pools memory across GPUs using multiple parallelism strategies, but adds communication and synchronization overhead. Scale-out exposes jobs to stragglers and failures; local CPU DRAM can offer faster small-transfer access than remote GPU HBM.
2.1 Distributed LLM Training
- Tensor, pipeline, fully sharded data, ZeRO, and expert parallelism distribute model state and computation across devices, pooling memory while adding communication and coordination overhead.These strategies are commonly composed into hybrid multi-dimensional configurations.Source: §2.1 · p. 3
2.2 Scale-Out vs. Stragglers and Failures
- A single slow node can gate synchronized training; limiting one node's NIC to 100 Gbps while others use 400 Gbps reduces Llama4-Scout throughput by 50% on 64 H100 GPUs.Using fewer nodes also reduces exposure to stragglers and failures by involving fewer participants and less shared-network traffic.Source: §2.2 · p. 3
2.3 Local DRAM vs. Remote GPU HBM Access
- Local CPU DRAM access is 4–6× faster than remote GPU HBM for small transfers, narrowing to 2–3× at larger sizes on the evaluated H100 setup.The comparison uses PCIe Gen5 for local CPU–GPU access and a 400 Gbps ConnectX-7 NIC for remote access; remote accesses still incur coordination and potential congestion costs.Source: §2.3 · p. 4, Figure 1 · p. 4
3 TRANSIT Overview
TRANSIT extends GPU memory with CPU DRAM through a transparent, user-space framework built on UVM. Runtime observations guide policies that make data movement more efficient and stabilize data placement under memory oversubscription.
3.1 Design
- TRANSIT combines a memory allocation interceptor, activity recorder, and optimization policies to guide placement from runtime observations while remaining transparent.The recorder captures kernel launches and migration history during warm-up.Source: §3.1 · pp. 4–5
3.2 Opportunity for Prefetch and Zero-copy
- Under UVM, page migration faults on CPU-resident data and transfers it to GPU memory, while zero-copy directly accesses host-resident data without migration.Page migration suits high temporal reuse; zero-copy suits low reuse or streaming access.Source: §3.2 · p. 5, Figure 3 · p. 5
- LLM training exhibits spatial and temporal locality that runtime-guided prefetching and zero-copy can exploit to reduce redundant movement.Contiguous tensor accesses create spatial locality, while reuse varies across tensors and training phases.Source: §3.2 · pp. 5–6
3.3 Implementation
- TRANSIT intercepts CUDA memory allocations and redirects them to UVM-backed allocations, enabling transparent memory oversubscription without changing program semantics.The shared library is injected with LD_PRELOAD, without modifying or recompiling the original program.Source: §3.3 · p. 6
3.4 Policy Exploration
- Default reactive UVM can incur fine-grained migrations with poorly amortized overhead and thrashing under oversubscription.TRANSIT-Prefetch coalesces migrations; TRANSIT-Advise uses zero-copy to reduce unnecessary movement and stabilize placement.Source: §3.3 · p. 6, §3.4 · p. 6
- TRANSIT-Prefetch reduces page faults by 97% while increasing average PCIe bandwidth utilization by 2.4×.It coalesces small migrations into larger transfers and prefetches data before kernel execution.Source: Figure 5 · p. 6, §3.4 · p. 7
- TRANSIT-Advise reduces PCIe transfer volume by 3× compared to TRANSIT-Base and improves effective bandwidth utilization by 2.5×.It eliminates page fault events by retaining mappings for host-resident data and enabling zero-copy access.Source: Figure 6 · p. 7, §3.4 · p. 8
4 Evaluation
The evaluation measures TRANSIT’s scale-in performance across models, parallelism strategies, bandwidth settings, and cluster scenarios. It reports throughput and communication results at the training-job and simulated cluster levels.
4.1 Setup
- The evaluation uses TorchTitan on four dense and MoE models, multiple parallelism configurations, and two H100 clusters; it measures per-GPU TFLOPS and per-node network traffic.It also emulates bandwidth limitation by capping NICs at 100 Gbps; Figure 7 compares offloading variants across GPU scales.Source: §4.1 · pp. 8–9, Figure 7 · p. 9
4.2 Scale-in Performance
- At 50% fewer GPUs, TRANSIT-Advise maintains up to 90% of baseline per-GPU TFLOPS, and at 16–32% fewer GPUs it exceeds the baseline by 3–4%.For Llama3-70B, TRANSIT-Prefetch and TRANSIT-Advise improve per-GPU TFLOPS over TRANSIT-Base by 1.04–1.99× and 1.10–3.43×, respectively.Source: §4.2 · p. 9
- At 16 GPUs, TRANSIT-Advise achieves 433 per-GPU TFLOPS versus 258 for TorchTitan-Offload, 272 for ZeRO-Offload, and 305 for ZeRO-Infinity.The comparison uses a 24-GPU no-offloading baseline of 418 per-GPU TFLOPS; Figure 8 also shows data offloaded per GPU.Source: §4.2 · p. 9, Figure 8 · p. 9
- Across FSDP, FSDP+TP, and TP+PP, TRANSIT-Advise retains over 90% of baseline per-GPU throughput at half the GPU count.For FSDP+TP, scaling from 4 to 2 nodes reduces per-node communication volume by 33%; TP+PP traffic remains largely invariant because pipeline activation exchanges dominate.Source: §4.2 · p. 10, Figure 9 · p. 10
- With FSDP+TP at 100 Gbps, TRANSIT-Advise reaches 161 TFLOPS at 8 GPUs versus 112 TFLOPS for the same-bandwidth no-offloading baseline, a 44% improvement.For FSDP+TP, bandwidth sensitivity 𝜌 falls from 2.1 to 1.5 when scaling from 16 to 8 GPUs; for full FSDP, it falls from 1.6 at 20 GPUs to 1.3 at 8 GPUs.Source: §4.2 · p. 10, Figure 10 · p. 10
4.3 Cluster-level Simulation
- The cluster simulator replays production traces and applies 25–50% node-count reductions only to eligible multi-node jobs, modeling placement transitions and reporting throughput, makespan, queue time, rack use, and GPU utilization.The simulated cluster has 32 GPUs per rack, with 4- or 8-GPU nodes and locality-aware placement.Source: §4.3 · pp. 10–11, Figure 12 · p. 11
- On the full 72-rack Seren trace, TRANSIT reduces inter-rack traffic by 15–49% and average queue time by up to 6.8% and 11.2% at 25% and 50% node reduction, respectively.On the lightly loaded 76-rack Kalos trace, rack traffic falls by 24.2% and 49.2%, while queue-time benefits do not materialize and idle GPU time changes by less than 1%.Source: §4.3 · p. 11, Figure 11 · p. 11
- On a 32-rack cluster with the Seren trace, 50% node reduction yields 5.2% higher eligible-job throughput, 82.4% lower eligible queue time, and 38.2% less rack traffic than the 72-rack baseline.On a reduced 32-rack Kalos cluster, queue time falls by 85.2% and 95.3% at 25% and 50% node reduction, respectively.Source: §4.3 · p. 12
5 Discussion
The discussion identifies constraints in combining TRANSIT’s zero-copy and prefetch mechanisms, then describes an expanded optimization space and its open challenges.
Potential improvements to TRANSIT
- TRANSIT-Advise’s zero-copy mappings for all spilled data eliminate migrations, leaving little opportunity for TRANSIT-Prefetch to contribute in the current design.Combining them would require selectively choosing which data uses zero-copy and which remains migration-based.Source: §5 · p. 12
- Choosing between migration and zero-copy is difficult because the shim lacks fine-grained reuse information, and existing profiling interfaces do not expose the required GPU access tracking.Deeper runtime or driver integration would be required to capture that information.Source: §5 · p. 12
Co-optimizing communication, compute, and memory offloading
- Transparent offloading expands parallelization choices beyond configurations that fit aggregate GPU memory, but optimizing them requires balancing compute, communication, and data-movement costs.Finding efficient strategies for this enlarged design space remains future work.Source: §5 · p. 12
6 Related Work
Related work addresses memory offloading and communication optimization through approaches with different integration and resource-allocation assumptions. The paper positions TRANSIT against framework-integrated, framework-managed, and communication-focused systems.
Training LLMs with data offloading
- Fine-grained offloading systems use runtime profiling and scheduling across memory tiers but require deep framework integration, while coarse-grained framework-managed approaches trade efficiency for deployment simplicity.The latter include ZeRO-Offload [41], ZeRO-Infinity, and TorchTitan.Source: §6 · p. 12
- Prior transparent UVM offloading work adds custom prefetching or migration policies, but driver modifications or kernel-level components limit deployability in environments with restricted administrative control.Forest modifies the UVM driver, while DeepUM adds a custom kernel-space driver and runtime integration.Source: §6 · p. 12
Communication optimizations for distributed LLM training
- Communication-optimization systems reduce traffic, alter parallelism or placement, or overlap communication and computation, but typically assume fixed parallelism and resource allocation.They do not address scaling GPU count to meet memory capacity when that scale-out exacerbates network bottlenecks.Source: §6 · p. 13
7 Conclusion
The paper concludes that TRANSIT uses host DRAM to train distributed LLMs on fewer GPUs and improves data movement through runtime-guided policies. The reported results include better offloading performance and lower network traffic, queueing delay, and cluster resource pressure.
7 Conclusion
- TRANSIT enables distributed LLM training on fewer GPUs by using CPU DRAM as an extension of GPU memory.It uses a user-space interposition layer without modifying applications, frameworks, schedulers, runtimes, kernels, or device drivers.Source: §7 · p. 13
- Runtime-guided data movement policies, including fused prefetching and zero-copy access, improve CPU–GPU data movement efficiency; zero-copy avoids migration thrashing and preserves per-GPU efficiency under scale-in.The conclusion reports that TRANSIT outperforms default UVM and framework-managed offloading.Source: §7 · p. 13
- The reported system-level benefits include reduced network traffic, queueing delay, and cluster resource pressure.Source: §7 · p. 13