Source-linked AI summary
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica
TL;DR
Serving the many task-specific variants produced by pretraining and LoRA fine-tuning remains difficult because concurrent adapters create batching and memory-management challenges. S-LoRA separates shared and adapter computation, manages adapters and KV caches through unified memory techniques, and reports higher throughput with substantially more served adapters. These capabilities support scalable serving of customized fine-tuned models.
Problem
Serving numerous LoRA adapters concurrently remains unexplored despite the need to support many task-specific fine-tuned variants at scale.
Method
S-LoRA separates batchable base-model computation from individual LoRA computation and combines unified paging, heterogeneous batching, and multi-GPU tensor parallelism.
Results
Up to 4× higher throughput than vLLM with naive LoRA serving and several-orders-of-magnitude more served adapters are reported.
Takeaways & Limitations
S-LoRA enables serving thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
Takeaways & Limitations
The evaluation uses a non-official Llama-70B architecture without group-query attention and compares against PEFT without continuous batching or PagedAttention.
Abstract
from arXiv · showhide
The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. The code is available at https://github.com/S-LoRA/S-LoRA
1 INTRODUCTION
LoRA makes it practical to create many task-specific variants of one base LLM, but serving these variants concurrently at scale remains challenging. S-LoRA addresses this by batching shared base-model computation while managing heterogeneous adapters and memory efficiently.
- LoRA updates only low-rank additive matrices, reducing fine-tuning parameters while retaining performance comparable to full-weight fine-tuning.
- S-LoRA separates batchable base-model computation from individual LoRA computations to exploit shared computation across requests.
- Serving many fine-tuned variants concurrently is difficult because GPU memory is limited and dynamic adapter and KV-cache allocation can cause fragmentation and I/O overhead.
- Unified Paging manages dynamic adapter weights and KV-cache tensors in a unified memory pool to reduce fragmentation and increase batch size.
- S-LoRA combines heterogeneous-batching CUDA kernels with a tensor-parallelism strategy designed to minimize communication overhead across GPUs.
- 4× throughput improvement over naive vLLM LoRA serving is reported, alongside several-orders-of-magnitude growth in served adapters.
2 BACKGROUND
LoRA adapts pretrained language models with small trainable low-rank updates, while LLM serving remains constrained by model scale, autoregressive decoding, KV-cache memory, and dynamic requests.
- LoRA freezes pretrained weights and adds trainable low-rank matrices, often reducing trainable parameters by 10000× versus full fine-tuning while retaining comparable accuracy.
- LoRA inference can merge low-rank matrices into base-model weights, eliminating additional inference overhead for a single adapter.
- LLMs span billions to trillions of parameters, creating substantial computational and memory demands during serving.
- Autoregressive decoding generates tokens sequentially and stores preceding hidden states in a KV cache, making decoding more memory-intensive than computation-intensive.
- Online serving must accommodate dynamically arriving requests with varying sequence lengths, motivating fine-grained token-level batching approaches such as Orca.
- Models exceeding one GPU's memory capacity or requiring stringent latency targets can be parallelized across multiple GPUs using tensor, sequence, or pipeline parallelism.
3 OVERVIEW OF S-LORA
S-LoRA combines batching, scheduling, unified paging, and tensor-parallel execution to serve many LoRA adapters concurrently.
- S-LoRA decomposes base-model and adapter computation, and uses adapter clustering and admission control when scheduling requests.
4 BATCHING AND SCHEDULING
S-LoRA batches shared base-model computation while executing heterogeneous adapter computations separately, storing inactive adapters in host memory and loading active ones into GPU memory.
- S-LoRA targets online, high-throughput serving of many LoRA adapters simultaneously.
- Merging each adapter into the base model creates multiple full-model copies and misses batching opportunities for concurrent multi-adapter inference.
- Computing xAB on the fly avoids weight duplication and enables batching of the more costly xW operation despite added xAB computation.
- Heterogeneous sequence lengths and adapter ranks make naive batched GEMM inefficient because padding causes poor hardware utilization.
- Custom CUDA kernels execute adapter computations without padding while supporting varying sequence lengths and adapter ranks.
- S-LoRA stores all adapters in main memory and fetches only those needed by the current batch into GPU memory.
- Adapter clustering reduces active-adapter diversity, freeing memory for KV cache allocation and potentially enabling larger batches and higher throughput.
- Admission control drops requests when system capacity cannot satisfy the service-level objective.
5 MEMORY MANAGEMENT
S-LoRA addresses memory fragmentation and adapter-transfer latency when serving many adapters concurrently by jointly managing adapter weights and KV caches, then prefetching upcoming adapters. Custom kernels support heterogeneous batched LoRA computation over the resulting non-contiguous layout.
- Unified Paging: S-LoRA stores adapter weights in main memory and dynamically loads those needed by the active batch into GPU memory.This addresses limited GPU capacity when serving many adapters.
- Unified Paging: Unified Paging jointly manages variable-size adapter weights and KV caches in a unified, paged memory pool to reduce fragmentation.Adapter ranks vary across requests, while KV-cache sizes vary with sequence length; both share a hidden dimension H.
- I/O Management: Prefetching predicts adapters for the next batch from the waiting queue and overlaps their loading with current decoding to reduce adapter-swapping I/O time.The prediction keeps most adapters needed by the next batch in available memory before execution.
- Heterogeneous Batching: Custom CUDA kernels batch LoRA computations with varying ranks and sequence lengths despite non-contiguous adapter storage.MBGMM handles prefill, while MBGMV handles decode; the latter extends support for non-contiguous memory and multiple ranks in a batch.
6 TENSOR PARALLELISM
S-LoRA introduces tensor-parallel partition strategies for batched LoRA inference that align adapter computation with the base model across multiple GPUs. The design limits communication overhead and avoids replicated weight matrices.
- Design Motivation: S-LoRA designs tensor-parallel strategies for batched LoRA inference across multiple GPUs, addressing the extra matrices and multiplications introduced by adapters.The strategy supports multi-GPU inference of large transformer models.
- Partition Strategy: The partition strategy aligns LoRA inputs and outputs with Megatron-LM’s base-model partitioning to avoid unnecessary communication and fuse some communications.The base model column-partitions W1 and row-partitions W2, with all-reduce accumulating distributed partial sums.
- Partition Strategy: For the added LoRA computation, A1 and B1 are column-partitioned, while A2 and B2 use row and column partitioning, respectively.An all-gather collects intermediate results, and an all-reduce sums the second adapter path before combining it with the base-model result.
- Communication Fusion: The strategy fuses an all-gather for one LoRA matrix multiplication with the final all-reduce, requiring only a single all-reduce to accumulate the final result.The passage presents this communication arrangement as a previously unstudied parallelization strategy.
- Cost Analysis: The added LoRA communication is negligible relative to the base model because r ≪ h, while all weight matrices are partitioned without replication.The design schedules communication on small LoRA intermediates and fuses it with base-model communication.
7 EVALUATION
S-LoRA is evaluated on synthetic and real workloads across Llama models and diverse GPU configurations. It serves thousands of adapters with higher throughput and stable scaling than baselines and ablated variants.
- End-to-End Results: S-LoRA can serve 2,000 adapters with minimal added LoRA-computation overhead, whereas vLLM-packed serves fewer than 5 under GPU-memory constraints.vLLM-packed also misses batching opportunities, while PEFT handles many adapters but performs significantly worse.
- Variant Comparison: S-LoRA achieves higher throughput and lower latency than S-LoRA-bmm and S-LoRA-no-unify-mem as adapter counts increase.The comparison attributes the advantage to the unified memory pool and custom kernels.
- Variant Comparison: After adapter counts reach a threshold such as 100, S-LoRA throughput no longer decreases because the number of activated adapters per batch remains unchanged.Scaling is therefore constrained by available main memory rather than additional adapter-count overhead.
- Real Workload Trace: S-LoRA preserves the same strong performance pattern on real workload traces as on synthetic workloads.Figure 7 reports throughput and attainment for traces derived from LMSYS Chatbot Arena.
- Multi-GPU Scaling: More than 2x throughput growth occurs when transitioning from 2 GPUs to 4 GPUs in the evaluated memory-bound setting.The reported explanation is that additional GPUs alleviate memory constraints; LoRA communication adds small overhead relative to computation.
8 RELATED WORK
Related work spans system techniques for batching, memory, kernels, and parallelism, as well as parameter-efficient fine-tuning and general model serving. S-LoRA is positioned alongside concurrent adapter-serving work while emphasizing distinct memory-management and tensor-parallelism designs.
- System Techniques: Prior transformer-serving systems improve batching, memory efficiency, GPU kernels, model parallelism, and parameter sharing.These system techniques motivate the serving-system context for S-LoRA.
- Concurrent Work: Punica also decomposes base-model and adapter computation, while S-LoRA differs through novel memory management and tensor parallelism.S-LoRA additionally supports batching different ranks and non-contiguous memory in some CUDA kernels.
- Algorithm Techniques: Algorithmic approaches such as quantization, sparsification, and architectural improvements reduce memory consumption or accelerate computation with minor model-quality compromise.These methods are presented as complementary to system-level optimization.
- Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning methods include LoRA, Prefix-tuning, P-Tuning, Prompt tuning, AdaLoRA, and (IA)3.The paper focuses specifically on LoRA.
- General-Purpose Model Serving: General-purpose serving research addresses batching, caching, and model placement for individual and multiple-model deployments.The cited systems include Clipper, TensorFlow Serving, Nexus, InferLine, Clockwork, DVABatch, and REEF.
9 CONCLUSION
S-LoRA serves thousands of LoRA adapters from one machine with substantially higher throughput than existing systems. Its unified memory pool, tensor parallelism, adapter batching, and CUDA kernels support large-scale customized fine-tuning services.
- Conclusion: S-LoRA serves thousands of LoRA adapters from a single machine with much higher throughput than existing systems.The conclusion attributes this capability to several system-design innovations.
- Conclusion: Unified memory pooling, tensor parallelism, adapter batching, and CUDA kernels are the principal designs enabling S-LoRA.Future extensions include additional adapter methods, fused kernels, and multiple CUDA streams.
- Conclusion: S-LoRA enables large-scale customized fine-tuning services for deploying models tailored to diverse requirements.This is the paper's stated practical consequence.
A.1 Analysis of PEFT
PEFT's limited batching and memory support constrain batch size and throughput, especially as adapter counts or request rates grow. The appendix quantifies these bottlenecks and their latency consequences.
- PEFT Analysis: PEFT accommodates a maximal batch size of 6 on A10G S1, compared with 30 for S-LoRA.The passage attributes the difference to PEFT's lack of KV-cache support.
- PEFT Analysis: PEFT reaches only 0.17 request/second at the largest tested adapter count because it lacks batching across different adapters.Its average latency also explodes when request rates exceed system capacity.
- PEFT Analysis: Without continuous batching, shorter PEFT requests wait for longer requests in the same batch.Together with smaller maximal batches, this produces low throughput even with one adapter.
- PEFT Analysis: PEFT fails to process with low latency even at the lowest tested request rate.The appendix reports this result in the request-rate analysis.
A.2 Experiments for adapter clustering.
The adapter clustering experiments find a small but observable impact on throughput and SLO attainment, especially at larger α and cv. The section also describes admission control based on reward and serving the most recent requests under a monotonic-reward condition.
- Adapter clustering: Adapter clustering prioritizes requests whose adapters are already in the batch once d adapters are represented, while allowing other requests to fill remaining capacity.The algorithm uses FCFS order, or early-abort order when enabled, and treats d as the number of clusters.
- Adapter clustering: The impact of cluster count on throughput and SLO attainment is small but observable, especially for larger α and cv.Figures 11 and 12 evaluate different cluster counts under varying α and cv settings.
- Adapter clustering: Generally, a small d can result in better performance, while fluctuations for small d may reflect scheduler overhead and random noise.
- Admission control: The admission-control formulation uses a reward function mapping first-token latency to [0, 1] and constrains the number of requests with positive reward to l.A reward of 0 means the user gives up, while 1 means complete satisfaction with latency.
- Admission control: When reward derivatives are nonincreasing, the optimal policy serves the most recent l queued requests in arrival order.In practice, l can be estimated by simulating an optimized FCFS strategy, while S-LoRA uses a heuristic based on periodically fetched minibatches and moving-average request rates.
B.1 Proof of Theorem B.1
The proof first shows that replacing older served requests with more recent ones preserves the service count and increases cumulative reward. It then establishes that serving those recent requests in arrival order is optimal by using reward concavity.
- Recent-request selection: Any strategy serving l requests can be transformed into one serving the most recent l requests with larger cumulative reward.The transformation replaces an older served request with an unserved newer request while preserving the number of served queries and the constraint.
- Recent-request selection: Replacing an older request with a more recent one increases reward because the newer request has lower serving latency.
- Ordering: Among the most recent l requests, serving them in arrival order does not decrease total reward.The proof compares an out-of-order pair and shows that swapping their service times preserves or improves reward.
- Ordering: Concavity of the reward function establishes the inequality needed for the pairwise swap argument.