PaperlayerView exact revision on arXiv

Source-linked AI summary

RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks

Xingran Chen, Rohit Bhagat, Ghadir Ayache, Rawad Bitar, Yanmin Gong, Salim El Rouayheb

arXiv:2609.00078v1cs.LGcs.AI

TL;DR

Distributed LoRA methods face communication and aggregation challenges because centralized and gossip approaches coordinate multiple model copies. RW-LoRA instead moves a single model token through the network for sequential local updates, achieving competitive performance with substantially lower communication and computation than gossip-based LoRA.

  • Problem

    Existing distributed LoRA methods rely on centralized aggregation or repeated synchronization of multiple model copies, incurring communication overhead and aggregation errors.

  • Method

    RW-LoRA traverses the network with a single token containing LoRA factors, updating it sequentially using each visited node’s local objective.

  • Results

    RW-LoRA consistently matches or exceeds gossip-based LoRA performance across evaluated tasks while substantially reducing communication and computation overhead.

  • Takeaways & Limitations

    Sequential token-based learning provides a decentralized alternative that avoids repeated neighbor-wise synchronization and aggregation of multiple model updates.

Abstract

from arXiv · show

Parameter-efficient fine-tuning methods such as LoRA have become a standard approach for adapting large foundation models. Adopting fine-tuning to distributed settings faces several challenges. Most existing distributed LoRA methods rely on centralized aggregation, and gossip-based decentralized LoRA requires repeated synchronization among multiple model copies. Both methods incur significant communication overhead and introduce errors due to simultaneous aggregation of multiple model updates. In this paper, we take a different perspective and propose a random-walk-based LoRA fine-tuning scheme. Instead of maintaining multiple model replicas, a single model token traverses the network and is updated sequentially using local fine-tuning objectives. This design eliminates the need for global synchronization, substantially reduces communication and computation costs, and avoids aggregation errors. We provide rigorous convergence guarantees for non-convex objectives under standard assumptions. Through empirical results on multiple NLP tasks and graph topologies, we show that the proposed method achieves competitive task performance with substantially less communication and computation than gossip-based LoRA.

I. INTRODUCTION

Distributed LoRA fine-tuning faces communication, memory, synchronization, and aggregation challenges. RW-LoRA addresses them by passing one model token through the network for sequential local updates instead of synchronizing multiple model copies.

  • Motivation: Foundation models often require task-specific adaptation, but full fine-tuning is expensive, communication-heavy, and prone to overfitting at large parameter scales.PEFT methods such as LoRA address this challenge by restricting adaptation to fewer trainable parameters.
  • Motivation: Federated LoRA relies on centralized aggregation, while gossip-based LoRA repeatedly exchanges updates among neighboring nodes, creating coordination and communication overhead.Centralized approaches also introduce memory overhead and a single point of failure.
  • Motivation: Averaging LoRA factors introduces bilinear mismatch, whereas averaging reconstructed updates and refactoring them with SVD adds computation and potential truncation errors.Both issues arise when distributed methods aggregate multiple model updates.
  • RW-LoRA approach: RW-LoRA passes a single token containing the current low-rank factors across the graph, updating it sequentially with each visited node’s local objective.The method avoids global synchronization and simultaneous aggregation of multiple updates.
  • System model: The framework models communication as a finite graph and node visitation as a random walk governed by transition probabilities and, under assumptions, a stationary distribution.Nodes hold local i.i.d. datasets and communicate only with neighbors.
  • Scope: The paper leaves extension of RW-LoRA to non-i.i.d. data for future work.The stated system model assumes local i.i.d. datasets.

III. PRELIMINARIES ON LORA

LoRA adapts foundation models by optimizing low-rank factors instead of full weight updates, reducing trainable parameters and resource requirements. Distributed LoRA methods include federated aggregation, gossip-based synchronization, and emerging random-walk approaches.

  • Low-Rank Adaptation: LoRA represents each update as ΔW = BA, where the factor rank r is much smaller than the layer dimensions.The base weight matrix remains fixed while A and B are trained.
  • Low-Rank Adaptation: Optimizing A and B reduces trainable parameters from d1d2 to r(d1+d2), lowering computation and memory requirements.The paper refers to r as the LoRA rank.
  • Distributed LoRA: Federated LoRA methods primarily differ in how they aggregate or personalize adapters across clients.Representative methods include FLoRA, FedALT, FedLoRA, FRLoRA, FedSA-LoRA, FedEx-LoRA, and FedMomentum.
  • Decentralized Learning: Federated methods rely on centralized aggregation, while gossip-based LoRA avoids a central node but requires repeated neighbor-wise synchronization.The paper positions random-walk LoRA as an alternative to these coordination patterns.
  • Random-Walk Learning: Random-walk learning uses token algorithms driven by Markov-chain sampling and has also been studied for privacy and node-failure robustness.The literature includes incremental optimization, stochastic-gradient, and mirror-descent directions.

IV. THE RW-LORA ALGORITHM

RW-LoRA fine-tunes LoRA factors through a single model token that moves across a graph and receives local updates. The algorithm initializes the factors, repeatedly trains at the current node, transfers the token, and returns the final adapted weights.

  • Algorithm Overview: RW-LoRA is presented as an algorithm for random-walk-based LoRA fine-tuning.Algorithm 1 summarizes the procedure.
  • Inputs and Initialization: The method carries LoRA rank r, iteration count T, and pretrained weight matrix W0 as algorithm inputs.These inputs define the factorization and duration of the token process.
  • Inputs and Initialization: The algorithm initializes A randomly, sets B to zero, and starts the random walk at node v0.The initialization uses Gaussian entries for A.
  • Token Update: At each node, RW-LoRA forms W(t) = W0 + B(t)A(t), updates the factors, samples the next node, and transfers the token.The returned model is W(T) = W0 + B(T)A(T).
  • Objective: The optimization objective uses each node’s local fine-tuning objective fv over LoRA factors A and B.The local objective is defined for the matrix-valued model weight W, with π as the target node distribution.
  • Token Update: The random walk evolves through transition matrix P, and the token applies the current node’s local objective to update both LoRA matrices.The transition matrix is constructed so π is stationary.

V. THEORETICAL RESULTS

The theoretical analysis establishes convergence of RW-LoRA to a factor-wise stationary point for non-convex objectives under assumptions controlling smoothness, gradients, variance, norms, and random-walk mixing. Its rate depends on the Markov-chain mixing time and differs from standard vector-valued analyses because LoRA updates couple two matrix factors.

  • Convergence Setting: RW-LoRA is analyzed for non-convex objectives under Assumptions 1–5.The assumptions are stated after the theorem and cover matrix norms, smoothness, gradients, variance, and bounded factors.
  • Stationarity: Theorem 1 measures stationarity with respect to the trainable LoRA factors A and B rather than the full update matrix ΔW.This matches the variables optimized by LoRA fine-tuning.
  • Rate Interpretation: The theorem’s notation uses eO, which hides logarithmic factors in relevant problem parameters.For example, eO(1/T) may include terms such as O(log T/T).
  • Rate Dependence: The convergence bound depends on the mixing time τ of the underlying Markov chain.The theorem’s conditions include τ ≥ τmix(πminϵ) and T ≥ τ^2.
  • Rate Dependence: LoRA’s 1/T coefficient scales linearly with τ, whereas standard vector-valued random-walk analyses use √τmix dependence.The difference is attributed to coupling introduced by ΔW = BA and separate updates to A and B.
  • Assumptions: The proof requires both bounded-gradient and bounded-variance assumptions because A and B are separately updated matrix-valued variables.Dropping either assumption would prevent completing the convergence proof.

VI. EXPERIMENTS

Experiments compare RW-LoRA with a gossip-based decentralized LoRA baseline. The reported results indicate lower communication and computational overhead while avoiding bilinear mismatch, using RoBERTa-base across tasks and graph nodes.

  • Comparison: RW-LoRA significantly reduces communication and computational overhead compared with gossip-based decentralized LoRA.The comparison also reports avoidance of the bilinear mismatch issue.
  • Experimental Setup: The experiments use RoBERTa-base with 125M parameters as the backbone model for all tasks.The random-walk fine-tuning process runs over 30 nodes.
  • Experimental Setup: The default configuration uses LoRA rank r = 16, ADAMW with learning rate 10^-3, K = 10 local steps, and batch size b = 32.An ablation varies r across {4, 8, 16, 32}.

2) Datasets:

The experiments evaluate RW-LoRA across five GLUE datasets, two graph topologies, communication efficiency, and LoRA-rank sensitivity. RW-LoRA maintains comparable task performance to gossip-based LoRA while reducing communication and computation overhead.

  • Datasets: RW-LoRA is evaluated on five GLUE datasets covering sentence-pair and sentiment classification.The datasets are MRPC, QQP, QNLI, MNLI, and SST-2.
  • Experimental setup: The decentralized framework uses 30-node ring and complete communication graphs with i.i.d. data partitions.The target distribution is uniform and transitions use the Metropolis–Hastings rule.
  • Task performance: RW-LoRA matches or exceeds gossip-based LoRA across all evaluated tasks while using substantially lower communication and computation overhead.Comparable final and best accuracies are reported for MRPC, QNLI, MNLI, QQP, and SST2.
  • Communication and computation efficiency: The communication cost per round is 0.295M for RW-LoRA on both graphs, compared with 17.7M on a ring and 256.65M on a complete graph for gossip-based LoRA.Gossip cost scales with network size, degree, and model size, whereas random-walk communication is O(model_size).
  • Communication and computation efficiency: RW-LoRA converges with substantially lower communication costs than gossip-based LoRA on QNLI, whose curves represent similar trends across the five datasets.Figure 1 uses total edge activations for communication overhead and accuracy or loss on the vertical axis.
  • Ablation study: Performance remains largely unchanged for LoRA ranks r ∈{4, 8, 16, 32}, indicating relative insensitivity to rank choice.The rank ablation is conducted on a complete graph.

APPENDIX A PROOF OF THEOREM 1

The proof of Theorem 1 is organized into three stages: establishing supporting lemmas, decomposing the critical objective term, and applying a telescoping argument.

  • Proof structure: The proof first establishes several useful lemmas.
  • Proof structure: It then decomposes f(W (t+1)) = f(B(t+1)A(t+1)) into analyzable terms.
  • Proof structure: Finally, a telescoping argument is applied to obtain the desired result.

A. Useful lemmas

The useful-lemma section establishes smoothness, bounded gradients and parameter changes, random-walk distribution properties, and relationships between factorized and full-model objectives.

  • Gradient bounds: Expected squared norms of gradients with respect to A, B, and W are bounded.
  • Update bounds: Expected squared changes in A, B, and W between consecutive iterations are upper bounded.
  • Smoothness: The local objective f_v is smooth with respect to A when B is fixed and with respect to B when A is fixed.
  • Smoothness: The global objective f is L-smooth, with corresponding factor-wise smoothness also established.
  • Random-walk distribution: The random-walk analysis relates the time-t node distribution π_t to the target distribution π through total variation and π_min.
  • Factorized model: The factor gradients of the local objective are derived from W = BA using the chain rule.

B. Decomposition of E f(B(t+1)A(t+1))

The decomposition section bounds the change in the factorized objective by using smoothness, intermediate terms, update rules, and random-walk expectations before combining the resulting inequalities.

  • Objective decrease: The analysis applies the Descent Lemma because the objective f is smooth.
  • Expectation: The expectation in the decomposition is taken over target sampling π and the random-walk position v_t.
  • Combining bounds: The proof combines the component inequalities and applies Cauchy–Schwarz and Young’s inequalities to obtain the final bound.
  • Bounding factor updates: The bounds use Assumption 2 together with inequalities controlling the A- and B-factor contributions.
  • Objective decomposition: f(B(t+1)A(t+1)) is decomposed through the intermediate terms f(B(t)A(t+1)) and f(B(t)A(t)).

C. Telescoping

The analysis bounds the relevant quantities using smoothness, inequalities, repeated substitution, and telescoping over a time interval. It then obtains an explicit expression for Γ.

  • Telescoping over τ ≤ t < T + τ produces the stated result.
  • The resulting constant is Γ = 2(2K + 1)(a2 + b2)Lτc2 + 4c2KL(a4 + b4) + 2K2c2(a2 + b2).
  • The proof invokes the Descent Lemma after using the L-smoothness of f.
  • Cauchy–Schwarz and previously established lemmas provide intermediate bounds for the analysis.
  • Repeated application of the preceding inequality yields a cumulative bound.
Loading 2609.00078v1…