Efficient Training on Multiple Consumer GPUs with RoundPipe

Yibin Luo; Shiwei Gao; Huichuan Zheng; Youyou Lu; Jiwu Shu

Source-linked AI summary

Efficient Training on Multiple Consumer GPUs with RoundPipe

Yibin Luo, Shiwei Gao, Huichuan Zheng, Youyou Lu, Jiwu Shu

arXiv:2604.27085v1Apr 29, 2026cs.DCcs.AIcs.LG

TL;DR

Consumer-GPU fine-tuning is limited by memory, PCIe communication, and pipeline weight binding that creates bubbles around uneven stages. RoundPipe decouples stages from GPUs through dynamic round-robin dispatch and supporting scheduling, synchronization, and partitioning designs. Across consumer and data-center GPU servers, it reports higher throughput and longer supported sequences, including 235B-model LoRA fine-tuning on 24 GB GPUs.

Problem
Consumer-GPU pipeline schedules bind uneven stages to fixed devices, so stage imbalance and slow PCIe communication limit efficient large-model fine-tuning.
Method
RoundPipe treats GPUs as stateless workers, uses asymmetric stage splitting with round-robin dispatch, and adds transfer scheduling, distributed event synchronization, and automated partitioning.
Results
RoundPipe delivers up to 2.16× higher throughput and 7.3× longer sequences on 8×RTX 4090 servers, and enables LoRA fine-tuning of a 235B MoE model on 24 GB GPUs.
Takeaways & Limitations
RoundPipe’s 4090 throughput reaches at least 76% of existing A800 solutions across all evaluated models, bridging part of the consumer–data-center performance gap.
Takeaways & Limitations
Existing schedules can reach bubble ratios up to 30% because GPU-count-multiple stage constraints exacerbate imbalance from compute-heavy components.

Abstract

from arXiv · show

Fine-tuning Large Language Models (LLMs) on consumer-grade GPUs is highly cost-effective, yet constrained by limited GPU memory and slow PCIe interconnects. Pipeline parallelism combined with CPU offloading mitigates these hardware bottlenecks by reducing communication overhead. However, existing PP schedules suffer from an inherent limitation termed the weight binding issue. Binding uneven model stages (e.g., the LM head is large) to GPUs limits the pipeline's throughput to that of the GPU with the heaviest load, leading to severe pipeline bubbles. In this paper, we propose RoundPipe, a novel pipeline schedule that breaks the weight binding constraint on consumer GPU servers. RoundPipe treats GPUs as a pool of stateless execution workers and dynamically dispatches computation stages across devices in a round-robin manner, achieving a near-zero-bubble pipeline. To ensure training correctness and system efficiency, RoundPipe integrates a priority-aware transfer scheduling engine, a fine-grained distributed event-based synchronization protocol, and an automated layer partitioning algorithm. Evaluations on an 8$\times$ RTX 4090 server demonstrate that RoundPipe achieves 1.48--2.16$\times$ speedups over state-of-the-art baselines when fine-tuning 1.7B to 32B models. Remarkably, RoundPipe enables LoRA fine-tuning of the Qwen3-235B model with 31K sequence length on a single server. RoundPipe is publicly available as an open-source Python library with comprehensive documentation.

1 Introduction

RoundPipe addresses consumer-GPU memory and communication constraints by decoupling pipeline stages from GPUs and dispatching them dynamically, while adding mechanisms for efficient and correct execution. Evaluations report substantial throughput, sequence-length, and model-scale gains.

Motivation: Consumer GPUs reduce fine-tuning cost but face limited VRAM and slow PCIe interconnects that restrict model scalability and training efficiency.An 8B model can require 128 GB for model states, compared with 24 GB on an RTX 4090.
Motivation: Existing pipeline schedules bind stage weights and computation to specific GPUs, so the slowest stage creates structural or imbalance bubbles.For Llama-3.1-8B, pipeline bubbles can reach 30%.
RoundPipe: RoundPipe treats GPUs as stateless workers and dynamically dispatches stages in round-robin order, allowing asymmetric stages to be pipelined with almost no bubbles.Asymmetric splitting can combine three layers into a forward stage or one layer into a backward stage.
System Design: RoundPipe overlaps parameter transfers with critical-path activation transfers, executes optimizer updates asynchronously, and enforces layer-level ordering with distributed events.These designs target transfer blocking and race conditions without reintroducing pipeline-stalling barriers.
System Design: RoundPipe automatically computes a near-optimal asymmetric partition in O(L^3) complexity, avoiding manual stage-balancing decisions.The algorithm evaluates contiguous subsequence-sum candidates and uses greedy constrained partitioning.
Evaluation: On 8×RTX 4090 servers, RoundPipe delivers up to 2.16× higher throughput and 7.3× longer sequences across 1.7B to 235B models.It also enables LoRA fine-tuning of a 235B MoE model on 24 GB GPUs.

2 Background and Motivation

Consumer-GPU training is constrained by memory pressure, activation storage, and low-bandwidth communication. Existing offloading and pipeline schedules mitigate some constraints but retain communication costs, structural bubbles, or severe imbalance from uneven stage execution.

Memory Pressure: Consumer-GPU training must accommodate large model states and activations within limited memory, with activation footprints increasing linearly with sequence length.A single 16k-token LLaMA-3.1-8B sequence generates 68 GB of activations.
Memory Pressure: Activation recomputation stores layer inputs and recomputes intermediate activations before backward propagation, reducing memory requirements.On an RTX 4090, recomputing a transformer layer is 2.37×–5.75× faster than reloading its activations from host memory.
Parallelism with Offloading: Data-parallel offloading methods distribute model states but require full-parameter exchanges for each forward and backward computation, making communication a major PCIe bottleneck.A previous study reports DeepSpeed spending about 70% of training time on communication on consumer GPU servers.
Parallelism with Offloading: Pipeline-parallel offloading reduces communication by passing activations and gradients between GPUs while loading each stage’s weights to its assigned device.This approach partitions model layers into stages stored in DRAM and assigned to different GPUs.
Pipeline Bubbles: Pipeline schedules exhibit structural bubbles from forward-backward dependencies and imbalance bubbles when stage latencies differ.Existing schedules struggle to mitigate both bubble types simultaneously.
Pipeline Bubbles: Existing schedules constrain stage counts to GPU-count multiples, forcing a trade-off between structural bubbles and load imbalance from compute-heavy components such as the LM head.The overall bubble ratio can reach up to 30% in current pipeline schedules.

3 Introuding RoundPipe

RoundPipe decouples pipeline stages from fixed GPUs, then combines round-robin dispatch with asymmetric partitioning to reduce structural and imbalance bubbles. Its analysis shows near-zero-bubble execution while PCIe transfers overlap computation.

3.1 Computation Dispatch Paradigm: Existing fixed-weight schedules leave GPUs underutilized when flexible partitions assign unequal numbers of stages, increasing structural bubbles despite reducing imbalance.For Llama-3.1-8B, pipeline bubbles can reach up to 30%.
3.1 Computation Dispatch Paradigm: RoundPipe dispatches computation stages dynamically across a stateless GPU worker pool, allowing flexible pipelines without fixed stage-to-GPU binding.Model states and activations remain on the host and are transferred to whichever GPU is ready to execute a stage.
3.2 RoundPipe Schedule: RoundPipe combines round-robin dispatch with asymmetric forward and backward stage splitting, enabling flexible stage counts and almost bubble-free forward-backward pipelining.Forward and backward stages are dispatched as one continuous sequence across GPUs.
3.2 RoundPipe Schedule: Asymmetric splitting balances stages by using separate forward and backward partitions and fusing forward computation for the first backward-stage layers.This bridges the faster forward and slower recomputed backward phases and eliminates phase-boundary bubbles.
3.2 RoundPipe Schedule: Asynchronous optimizer updates continue round-robin assignment across iteration boundaries without pipeline flush, eliminating warm-up and cool-down bubbles at those boundaries.The schedule resumes from the previous iteration’s stage position.
3.3 Benefits and Tradeoff Analysis: RoundPipe’s bubble ratio is smaller than looped schedules because it uses around 4/3× more stages and permits better time balance.Its PCIe transfer time can be entirely overlapped by computation with batch sizes as small as B = 8 for dense models and B = 80 for MoE models.

4 Design and Implementation

RoundPipe implements its dispatch schedule through a controller, GPU workers, and an optimizer worker, while overlapping transfers and computation. Event-based consistency and automated partitioning address correctness and load-balancing challenges without blocking the pipeline.

4.1 System Architecture: RoundPipe separates control and data planes through a controller, an optimizer worker, and one GPU worker per available GPU.The controller schedules tasks and ordering while workers execute hardware operations and asynchronous updates concurrently.
4.1 System Architecture: The forward_backward() API orchestrates complete pipelined execution across GPUs, while step() dispatches gradient processing and optimizer updates asynchronously.GPU workers compute the next iteration concurrently with the optimizer worker’s updates.
4.1.2 Challenges: Simple compute-transfer overlap suffers head-of-line blocking because large parameter and gradient transfers delay critical-path activation transfers.RoundPipe addresses this with priority-aware scheduling that places non-critical transfers in idle windows.
4.3 Parameter Consistency: Parameter consistency requires preserving staleness-1 asynchronous updates while optimizer and GPU workers access separate model copies concurrently.The system enforces ordering constraints for weight copies, gradient copies, GPU transfers, and optimizer steps.
4.2 Data Transfer Overlap: RoundPipe uses four dedicated communication streams per device to overlap activation transfers with parameter and gradient transfers while avoiding compute blocking.Activation transfers are scheduled early or delayed, and lower-priority transfers occupy idle intervals.
4.3 Parameter Consistency: Blocking synchronization would reintroduce pipeline bubbles, so RoundPipe offloads copies to the optimizer worker and coordinates them with point-to-point threading events.Four dependency edges preserve the first four ordering constraints, while the optimizer worker’s sequential execution enforces the fifth.
4.4 Stage Partitioning: RoundPipe’s automated partitioning algorithm searches O(L^2) candidate maximum stage times and solves each partitioning case greedily in O(L), yielding O(L^3) complexity.It respects contiguous-layer and GPU-memory constraints while minimizing the maximum stage time.

5 Evaluation

RoundPipe is evaluated on consumer- and datacenter-grade GPU servers across model sizes, sequence lengths, and GPU counts. It improves throughput, sequence capacity, scaling, and pipeline utilization through dynamic dispatch, offloading, and asynchronous coordination.

Experimental Setup: RoundPipe is evaluated on 8×RTX 4090 and 8×A800 servers using five models from 1.7B to 235B parameters.Experiments report throughput and maximum sequence length, with full-parameter training for four models and LoRA fine-tuning for Qwen3-235B-A22B.
End-to-End Performance on 4090: 1.48–2.16× higher training throughput is achieved over the fastest existing systems for 1.7–32B models on RTX 4090 GPUs.RoundPipe and RoundPipe-sync achieve the highest throughput across all five models, while RoundPipe supports LoRA fine-tuning of the 235B model on 24GB GPUs.
End-to-End Performance on 4090: 4.7–7.3× longer maximum sequences are supported than the next-best baseline, excluding Megatron-TP because its PCIe throughput is impractical.RoundPipe stores stage-boundary activations in host memory and recomputes layer-internal activations on demand, while avoiding TP’s heavy communication overhead.
End-to-End Performance on A800: 1.19–5.62× larger maximum sequence lengths are achieved on A800 servers, reaching 192K–288K tokens for smaller models.RoundPipe stores model states and stage-boundary activations in CPU memory, while larger models remain limited by GPU memory under competing methods.
Scalability: Near-linear throughput scaling from 1 to 8 GPUs is reported, while maximum sequence lengths remain invariant across GPU counts.The five models retain maximum sequence lengths of 73K, 49K, 39K, 28K, and 31K from 1 to 8 GPUs.
Schedule and Ablation Studies: Throughput decreases smoothly as sequence length grows over two orders of magnitude, while asynchronous optimizer updates reduce the absolute bubble ratio below 4.5%.RoundPipe-sync reduces bubbles by 23%–55% relative to the best baseline; the remaining asynchronous idle time comes from bounded stage-execution imbalance.
Schedule and Ablation Studies: The consistency protocol removes 2.6–14 seconds of per-iteration overhead associated with blocking copies and supports asynchronous optimizer updates.The overhead grows roughly with trainable-parameter size, while Qwen3-235B-LoRA benefits less because LoRA updates fewer weights.

6 Related Work

Prior pipeline schedules reduce bubbles through weight stashing, delayed updates, or looped execution, but these approaches trade memory or remain constrained by stage placement. RoundPipe instead uses heterogeneous memory to decouple stages from GPUs.

Pipeline Parallelism: Asynchronous and backward-splitting schedules reduce pipeline bubbles but trade additional memory consumption for efficiency.Looped pipeline methods increase utilization, while RoundPipe takes a different approach to avoid GPU memory overhead.
RoundPipe: RoundPipe combines flexible stage partitions with heterogeneous-memory offloading to decouple stages from GPUs.Its schedule is synchronous with flexible partitions and asynchronous without GPU memory overhead.
Memory Offloading: Existing offloading systems target weights, optimizer states, activations, or tensor-granularity transfers, but are predominantly designed for other execution settings.RoundPipe extends this line of work with computation dispatch across consumer GPU workers.

7 Conclusion

RoundPipe introduces a pipeline schedule for large-model training on consumer GPU servers. Its computation dispatch paradigm, asymmetric splitting, and round-robin dispatch decouple stages from GPUs and improve pipeline efficiency.

Conclusion: RoundPipe is a pipeline-parallel training system designed for large models on consumer GPU servers.The conclusion frames the system around a new schedule rather than a new model or workload.
Conclusion: The Computation Dispatch Paradigm decouples stages from GPUs while preserving full compute-bound throughput.RoundPipe builds on this paradigm with asymmetric stage splitting and round-robin dispatch.
Conclusion: Asymmetric stage splitting and round-robin dispatch mitigate stage imbalance and improve pipeline efficiency.The conclusion reports performance gains from the resulting schedule.

B Recomputation Analysis Details

The appendix provides derivations for activation-size-related conclusions in Section 2.1 and Figure 2.

Appendix Scope: The appendix derives the activation-size-related conclusions presented in Section 2.1 and Figure 2.It serves as supporting analysis for the paper’s activation-memory discussion.

B.1 Activation Size

The appendix derives per-layer activation storage for GQA transformer components under 16-bit storage assumptions and reports the resulting full-model footprint for LLaMA-3.1-8B.

The activation analysis assumes network states and activations use 16-bit floating point, requiring 2 bytes per element.
Each transformer layer comprises attention, MLP, and two layer norms whose activation storage is summed.
The attention block stores shared QKV inputs and FlashAttention’s Q, K, and V tensors.
The MLP stores inputs for its linear layers and SwiGLU, requiring 2𝑠𝑏ℎ + 6𝑠𝑏𝑚𝐸act bytes.
68 GB of activations are generated by all 32 layers when training LLaMA-3.1-8B with one 16k-token sequence.

B.2 Activation Recompute v.s. Reload Analysis

The analysis characterizes the computation and evaluates activation recompute and reload time using specified micro-batch, sequence-length, model, and RTX 4090 configurations.

A transformer layer’s forward/recompute pass includes four self-attention projections, attention computation, and three FFN projections.
Dense transformer models are represented by setting 𝐸act = 1.
Figure 2 calculates activation recompute and reload time using micro-batch size 4, sequence length 2048, Table 3 model configurations, and RTX 4090 specifications.

C Roofline Analysis Details

The roofline analysis accounts for full-duplex PCIe transfer of layer parameters and activations, with effective transfer time determined by the larger direction.

Layer data movement uploads parameters and input activations while downloading output activations.
Because PCIe is full-duplex, effective transfer time is governed by the larger of the upload and download directions.
The output activation download volume is 2𝑏𝑠𝐻 bytes, and the upload side exceeds it for the dense forward pass.

C.2 OI of a Mixture-of-Experts Layer

The roofline analysis compares dense and MoE operational intensity, then extends the comparison to backward execution with activation recomputation and PCIe data movement.

C.2 OI of a Mixture-of-Experts Layer: MoE attention matches dense layers, but tokens use 𝐸act active experts while transfers cover all 𝐸total experts.
C.2 OI of a Mixture-of-Experts Layer: MoE layers have lower operational intensity than comparable dense layers because all expert weights transfer while only active experts contribute FLOPS.
C.2 OI of a Mixture-of-Experts Layer: The analysis counts only matrix-multiplication FLOPS, treating elementwise operations as negligible relative to matrix operations.
C.3 Backward Pass Has Even Higher OI: Backward execution has approximately 3× the forward FLOPS but less than 2× the data movement, making its operational intensity strictly higher.
C.2 OI of a Mixture-of-Experts Layer: At sequence length 2048, dense-model OI exceeds the GPU ridge point at batch size 8, while MoE models cross it below batch size 100.