PaperlayerView exact revision on arXiv

Source-linked AI summary

SplitLite: Low-Rank Residual Compression for Split Learning

Tao Li, Yulin Tang, Qi Guo, Xianhao Chen

arXiv:2608.23018v1cs.LGcs.AI

TL;DR

On-device LLM federated fine-tuning is limited by device computation and the communication cost of exchanging high-dimensional split-learning activations and gradients. SplitLite uses LoRA-induced low-rank adjacent-epoch residuals, transmitting quantized truncated-SVD factors, and reports large communication reductions without performance degradation.

  • Problem

    On-device LLM federated fine-tuning faces limited device resources and prohibitive communication from repeated high-dimensional activation and gradient exchanges in split learning.

  • Method

    SplitLite exploits LoRA-induced rank-2r activation and rank-4r gradient residuals between adjacent epochs, transmitting their quantized truncated-SVD factors.

  • Results

    Up to 93.5% activation uplink and 83.7% total communication cost reductions are achieved on GLUE without degrading fine-tuning performance.

  • Takeaways & Limitations

    Low-rank residual compression reduces bidirectional cut-layer traffic for split federated LoRA fine-tuning of on-device LLMs without auxiliary learning.

Abstract

from arXiv · show

Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL) has emerged as a promising solution, which offloads the primary training workload to a powerful server. However, SL requires exchanging high-dimensional activations and gradients between clients and the server, resulting in prohibitive communication costs. To overcome this challenge, we propose SplitLite, a communication-efficient split federated LoRA fine-tuning method that exploits the low effective rank structure of consecutive-epoch activation and gradient residuals. Our key finding is that, when LoRA uses rank $r$ updates in parameter space, the activation and gradient residuals of the same data sample between adjacent epochs also exhibit effective rank-$2r$ and rank-$4r$ structures, respectively. By revealing this property, SplitLite transmits only quantized truncated singular value decomposition (SVD) residual factors, thereby significantly reducing both activation uplink and gradient downlink traffic. Extensive experiments on the GLUE benchmark across a series of advanced on-device LLMs demonstrate that our method reduces activation uplink communication costs by up to 93.5\% and total communication costs by up to 83.7\%, without performance degradation.

I. INTRODUCTION

Federated fine-tuning of on-device LLMs is constrained by device resources and split learning’s high-dimensional cut-layer communication. SplitLite addresses this bottleneck by exploiting low-rank adjacent-epoch residuals and compressing their transmission.

  • Motivation: Conventional federated learning is impractical for LLMs on edge devices because clients must train the entire model locally.LoRA reduces trainable parameters by updating lightweight adapters while freezing the backbone, but split learning is used to offload primary computation to a server.
  • Communication bottleneck: High-dimensional activation and gradient exchanges at the split cut layer create substantial communication overhead, with uplink especially burdensome for wireless devices.The uplink burden is amplified by asymmetric wireless data rates and limited on-device energy.
  • Low-rank structure: LoRA rank-r updates induce adjacent-epoch activation residuals with effective rank at most 2r and gradient residuals with effective rank at most 4r.The residuals are small because they arise from subtle parameter changes, and Fig. 2 reports that low-rank approximations capture most residual energy.
  • SplitLite: SplitLite transmits quantized truncated-SVD factors for rank-2r activation and rank-4r gradient residuals instead of full residual tensors.The method reduces both transmitted element count and bits per element through truncated SVD and unbiased stochastic quantization.
  • Results: Up to 93.5% activation uplink and 83.7% total communication reductions are reported on GLUE without sacrificing model performance.The framework also includes convergence analysis that accounts for SVD truncation and stochastic quantization errors.
  • Related work: Existing split fine-tuning methods overlook repeated activation uploads and gradient downloads, while prior compression methods do not exploit cross-epoch redundancy.This motivates residual-based compression for split federated LoRA fine-tuning.

IV. DESIGN OF SPLITLITE

SplitLite exploits LoRA-induced low-effective-rank activation and gradient residuals between adjacent epochs, using rank-2r and rank-4r representations for compression. The design is supported as an effective-rank property under first-order linearization rather than an exact algebraic property of nonlinear Transformers.

  • Residual Construction: For each sample, activation and gradient residuals are formed relative to adjacent-epoch or cached tensors at the cut layer.The framework analyzes residuals for the same data sample and uses synchronized activation and gradient caches in the SFL procedure.
  • Activation Residual: First-order linearization propagates the rank-constrained LoRA parameter variation to a dominant activation residual of approximately rank 2r.Nonlinear Transformer operations may add higher-order components, so the rank characterization is effective rather than exact.
  • Gradient Residual: The dominant gradient variation is expected to concentrate in a subspace of approximately 4r because client-side activation shifts and server-side LoRA updates jointly induce it.This provides the rationale for using rank-4r gradient representations.
  • Empirical Validation: Top-k energy ratios empirically support effective ranks of 2r for activation residuals and 4r for gradient residuals.The observed concentration motivates rank-2r and rank-4r truncated SVD compression.

C. Algorithm: Low-Rank Bidirectional Residual Compression

The bidirectional compression algorithm maintains synchronized sample-wise caches and communicates only low-rank residual factors after the initial full-tensor exchange. Truncated SVD and stochastic quantization jointly reduce transmitted elements and bits per element while preserving the reconstructed tensors used for training.

  • Cache-Based Residuals: Clients and the server maintain synchronized sample-wise activation and gradient caches, transmitting full tensors initially and residuals thereafter.Each subsequent residual is computed against the most recently reconstructed tensor for the same sample.
  • Low-Rank Factorization: Activation and gradient residuals are compressed with rank-2r and rank-4r truncated SVD, respectively.The algorithm sets κy = 2r and κg = 4r for the two communication directions.
  • Low-Rank Factorization: Absorbing singular values into the left singular-vector factors avoids transmitting the singular-value matrix separately.This further reduces the factor payload after truncated SVD.
  • Communication Reduction: 6r(S+E) transmitted elements replace 2SE elements for bidirectional communication, with the relative cost given as 3r.For S = 128, E = 3072, and r = 8, the method requires approximately 19.5% of uncompressed bidirectional communication.
  • Quantization and Reconstruction: Stochastic uniform quantization lowers the bit width of retained low-rank factors without degrading model performance.The quantized factors and quantization parameters are transmitted, reconstructed, and added to cached values at the receiver.
  • Communication Analysis: Low-rank factorization changes per-direction communication complexity from O(SE) to O(κd(S + E)).The resulting gain becomes more pronounced as sequence length or hidden dimension increases, while quantization further reduces bits per element.

D. Implementation Considerations

SplitLite adds truncated randomized SVD and cache storage requirements, but supports batched, pipelined execution and cache offloading. Despite the extra compression computation, it achieves the lowest reported end-to-end fine-tuning latency among the evaluated methods.

  • Computation Costs: Truncated randomized SVD computes only the leading 2r or 4r singular triplets at cost O(SEκd).SVD and quantization can execute independently across samples and overlap with communication.
  • Computation Costs: SplitLite achieves the lowest end-to-end fine-tuning latency across the three evaluated GLUE tasks.For Llama-3.2-3B, latency is reduced by 65.5%–73.6% relative to SplitLoRA.
  • Storage Costs: Per-sample reconstructed activation caches require at most 2niSE entries on client i.For Llama-3.2-3B, this is about 1 GB per client on MRPC and 100 GB on MNLI.
  • Storage Costs: Cache storage can be offloaded to CPU memory or SSD and stored in low precision with synchronized rounding.Cache loading and updates can overlap with forward and backward computation or occur during training idle periods.

V. CONVERGENCE ANALYSIS

The analysis establishes convergence guarantees for SplitLite under residual compression, truncated SVD, and stochastic quantization. It characterizes compression, drift, and quantization effects while identifying rank, bitwidth, and synchronization trade-offs.

  • Assumptions: SplitLite’s convergence analysis assumes Lipschitz regularity, bounded stochastic-gradient variance, and unbiased bounded quantization.The quantizer’s error variance is bounded by cq, under the stated assumptions.
  • Cache stability: The activation-cache error remains recursively stable under rank-2r truncated SVD and stochastic quantization.Lemma 1 establishes this stability under Assumption 3.
  • Cache stability: The gradient-cache error likewise remains recursively stable using rank-4r truncated SVD.Lemma 2 mirrors the activation-cache argument with rank 4r.
  • Main convergence result: Theorem 1 gives a convergence bound under 0 < η ≤ 1/(2β), combining optimization gap, client drift, stochastic-gradient noise, and compression error.The minimum objective value is denoted by f⋆, and the proof uses the stated smoothness and perturbation bounds.
  • Trade-offs: Increasing r reduces SVD-tail error but increases quantization error and communication, while higher bitwidth lowers cq and tightens cache stability.Smaller client-side aggregation intervals reduce drift-related terms but require more frequent LoRA synchronization.

A. Experimental Setup

Experiments evaluate SplitLite on three GLUE tasks using four causal LLMs under Non-IID client partitions. The study compares it with four communication or split-learning baselines under fixed system settings.

  • Datasets and models: The evaluation covers MNLI, MRPC, and RTE using Non-IID Dirichlet partitions with α = 0.5.All split federated learning experiments use N = 6 clients.
  • Datasets and models: The tested models are Qwen-2.5-1.5B, Qwen-2.5-3B, Gemma-3-1B, and Llama-3.2-3B.These are pre-trained causal language models spanning different scales.
  • Baselines: SplitLite is benchmarked against SplitLoRA, AdaptSFL, AQ-SGD, and Rand-Top-k.The baselines respectively represent uncompressed split LoRA, adaptive split learning, temporal quantization, and randomized sparsification.
  • System settings: For LoRA-based methods, the rank is 16 and homogeneous models split at layer 3, while SplitLite uses residual ranks 32 and 64 with 4-bit and 8-bit quantization.AdaptSFL instead uses heterogeneous local depths.

1) Fine-Tuning Model Performance:

SplitLite maintains or improves fine-tuning accuracy while substantially reducing communication and end-to-end latency. Training curves also indicate stable MRPC convergence and a favorable communication–accuracy trade-off.

  • Accuracy: Accuracy is comparable to or higher than SplitLoRA across nearly all model–task combinations.On MNLI, Gemma-3-1B improves from 82.7% to 83.8%, while Qwen-2.5-3B improves from 86.9% to 87.4%.
  • Communication: 93.5% is the maximum activation-uplink reduction, while 83.7% is the maximum total-communication reduction versus SplitLoRA.SplitLite uses 6.5%–6.9% of SplitLoRA’s activation uplink and 16.3%–17.4% of its total communication.
  • Convergence: MRPC training curves show stable convergence across Gemma-3-1B, Qwen-2.5-3B, and Llama-3.2-3B.Comparisons with AQ-SGD and Rand-Top-k support SplitLite’s communication–accuracy trade-off.
  • Latency: 74% is the maximum training-time reduction over SplitLoRA across the evaluated settings.On MNLI, Qwen-2.5-3B decreases from 9.40 to 2.73 hours, and Llama-3.2-3B from 6.90 to 2.38 hours.

2) Communication Compression:

SplitLite has the lowest reported communication overhead across evaluated models and tasks, reducing both activation uplink and total traffic relative to SplitLoRA. Its savings are especially relevant for wireless edge devices because uplink rates are typically lower than downlink rates.

  • SplitLite consistently achieves the lowest communication overhead across evaluated models and tasks.Communication costs are reported from the second epoch onward using steady-state accounting.
  • 6.5%–6.9% activation uplink and 16.3%–17.4% total communication remain relative to SplitLoRA.These correspond to maximum reductions of 93.5% and 83.7%, respectively.
  • On MNLI with Gemma-3-1B, SplitLite retains 6.9% uplink and 17.4% total communication while improving accuracy from 82.7% to 83.8%.
  • SplitLite outperforms AQ-SGD and Rand-Top-k in both uplink and total communication cost.AQ-SGD retains 25.0% uplink and 37.5% total cost, while Rand-Top-k retains 50.5%–52.5% uplink and 40.2%–41.2% total cost.

3) Ablation Study:

SplitLite exploits low-rank activation and gradient residuals to reduce cut-layer communication, with experiments showing substantial savings without fine-tuning performance degradation. The study also examines homogeneous and heterogeneous cut-layer configurations, while extending the approach to model-parallel cloud training remains future work.

  • Ablation Study: Heterogeneous cut-layer configurations follow AdaptSFL’s client-specific split depths, unlike SplitLoRA’s shared cut layer.The comparison assesses the impact of cut-layer heterogeneity and notes that it introduces additional aggregation.
  • Ablation Study: SplitLite transmits quantized truncated-SVD factors for rank-2r activation and rank-4r gradient residuals across adjacent epochs.The framework exploits LoRA-induced low effective-rank structure without auxiliary learning.
  • Ablation Study: 93.5% maximum activation uplink and 83.7% maximum total communication reductions are reported without degrading fine-tuning performance.
  • Ablation Study: SplitLite’s convergence analysis incorporates both SVD truncation and quantization errors.
  • Ablation Study: Model-parallel cloud training is identified as a potential extension left for future work.
Loading 2608.23018v1…