PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

Lei Wang, Jieming Bian, Letian Zhang, Jie Xu

arXiv:2609.00632v1cs.LGcs.AI

TL;DR

Federated LoRA must accommodate clients with different resource budgets and non-IID data, yet existing rank-heterogeneous methods usually assign unified same-rank models. FedRoRA separates shared adaptation directions from personalized rank-wise magnitudes, using server-side SVD projection and rank-adaptive selection to create client-specific initializations. Across NLU and NLG benchmarks, it consistently outperforms state-of-the-art rank-heterogeneous FL-LoRA methods.

  • Problem

    Existing rank-heterogeneous federated LoRA methods bridge rank mismatches but generally provide unified same-rank models, while personalized approaches assume homogeneous ranks or add substantial local overhead.

  • Method

    FedRoRA decomposes LoRA updates into shared normalized directions and learnable client-specific rank-wise magnitudes, then uses server-side SVD projection and rank-adaptive Top-k selection for personalized initialization.

  • Results

    FedRoRA consistently outperforms state-of-the-art rank-heterogeneous federated LoRA methods across diverse non-IID NLU and NLG settings.

  • Takeaways & Limitations

    FedRoRA enables fine-grained personalization among clients with different rank budgets while retaining shared adaptation directions.

  • Takeaways & Limitations

    Scaling exact SVD to models above 70B parameters or federations with thousands of clients may require randomized or approximate SVD techniques.

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

1 Introduction

Federated LoRA enables privacy-preserving, resource-efficient adaptation, but rank heterogeneity and non-IID data expose limitations in unified same-rank models. FedRoRA addresses this by separating shared adaptation directions from personalized rank-wise magnitudes and returning client-specific initializations.

  • Federated Learning supports collaborative adaptation without sharing raw local data, while LoRA reduces fine-tuning overhead by optimizing few low-rank parameters.
  • Resource disparities constrain the LoRA ranks different clients can train, motivating rank-heterogeneous aggregation methods.
  • Existing rank-heterogeneous methods bridge cross-rank dimension mismatches but provide one unified model to clients sharing a rank.
  • Non-IID client data can require distinct adaptation directions and magnitudes, making unified same-rank initialization insufficient.
  • FedRoRA decomposes each update as ∆Wi = ˜BiSi ˜Ai, separating normalized directions from client-specific rank-wise magnitudes.
  • Its server extracts a shared subspace with SVD and redistributes personalized triplets through projection and rank-adaptive Top-k selection.

2 Related Works

Federated LoRA aggregation methods address rank mismatches through padding, stacking, SVD projection, or PLoRA construction, but generally retain shared same-rank initializations. FedRoRA instead combines shared directions with personalized magnitudes without increasing the local footprint.

  • FedIT and SLoRA applied FedAvg directly to LoRA factors, but later work identified a mismatch because averaged products generally differ from products of averages.
  • HETLoRA uses zero-padding, FLoRA stacks local modules, and Fed-PLoRA uses Select-N-Fold PLoRA to support arbitrary rank aggregation.
  • FlexLoRA uses SVD projection, but parameter-wise operations rely on fixed dimensions and cannot be trivially extended to varying matrix shapes.
  • These approaches enable cross-rank communication yet typically converge to a single global direction for each rank group.
  • FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes, preserving fine-grained personalization under rank heterogeneity without increasing local footprint.

3 Problem Formulation

The formulation considers clients with different rank budgets collaboratively fine-tuning a shared language model. Its objective is to learn personalized low-rank updates that minimize weighted empirical risk while respecting each client’s assigned rank.

  • 3.1 Rank-Heterogeneous LoRA Adaptation: N clients collaboratively fine-tune a shared pretrained model with weights W0 ∈ R^dout×din.
  • 3.1 Rank-Heterogeneous LoRA Adaptation: Each client represents its update as ∆Wi = BiAi, with Bi ∈ R^dout×ri and Ai ∈ R^ri×din.
  • 3.1 Rank-Heterogeneous LoRA Adaptation: Client rank ri varies across the federation according to computational and memory budgets.
  • 3.1 Rank-Heterogeneous LoRA Adaptation: For each client, the forward pass evaluates the shared model plus its local low-rank update on local input x.
  • 3.2 Objective: Rank-Adaptive Personalization: The optimization seeks a set of client-specific updates {∆Wi} that minimizes total empirical risk.
  • 3.2 Objective: Rank-Adaptive Personalization: Each local loss Li is weighted by pi, the client’s data-size ratio, in the federated objective.
  • 3.2 Objective: Rank-Adaptive Personalization: Every update must satisfy its assigned rank budget ri, while capturing shared structural knowledge and local task-specific features.

4 Method: FedRoRA

FedRoRA addresses the failure of unified same-rank initializations under non-IID data by separating shared adaptation directions from personalized rank-wise magnitudes. It aggregates a shared subspace with SVD, then returns client-specific directions and magnitudes for rank-constrained local training.

  • Motivation: Non-IID clients can require distinct adaptation directions, so unified same-rank initializations may poorly align with client-specific objectives.Low cosine similarity between local LoRA updates and underperformance versus independent local training motivate personalization.
  • Motivation: Scaling-only adaptation with transferred fixed directions achieves performance close to full local training, supporting shared directions with personalized magnitudes.This motivates separating adaptation structure from client-specific rank-wise scaling.
  • Local Decoupled Parameterization: FedRoRA replaces ∆Wi = BiAi with ∆Wi = ˜BiSi ˜Ai, where normalized factors encode directions and Si is a learnable diagonal matrix encoding rank-wise magnitudes.The three components are jointly updated locally, with directional factors re-normalized after each iteration.
  • Global Personalized Aggregation: The server reconstructs client updates, computes a dataset-weighted aggregate, extracts a rank-rmax shared subspace with truncated SVD, and projects clients onto global directions.Client-specific coefficients represent alignment-dependent magnitudes along the global rank-one directions.
  • Global Personalized Aggregation: FedRoRA selects the most relevant global directions within each client’s rank budget and returns personalized triplets for the next training round.Clients with the same rank may receive different basis directions and rank-wise magnitudes without increasing the local footprint.

5 Experiments

Experiments evaluate FedRoRA against rank-heterogeneous federated LoRA baselines on NLU and NLG benchmarks, plus ablations and robustness analyses. FedRoRA achieves the strongest reported performance and remains advantageous across heterogeneous settings.

  • 5.1 Natural Language Understanding: 91.44% average accuracy on NLU surpasses FlexLoRA’s 88.70% by +2.74 points, with larger gains on MNLI (+2.98) and QQP (+3.98).FedRoRA leads every evaluated NLU task under the rank-heterogeneous, non-IID setup.
  • 5.2 Natural Language Generation: 71.95 average ROUGE-1 on NLG surpasses FedPLoRA’s 70.77 by +1.18, with strongest gains on Struct2Text (+1.64) and Reasoning (+1.38).FedRoRA achieves the highest ROUGE-1 score across all four generation tasks.
  • 5.3 Ablation Study: The diagonal scale S re-weights shared global directions for each client according to its local data.The ablation isolates S as FedRoRA’s personalization mechanism on GLUE with RoBERTa-Large and α = 0.5.
  • 5.3 Ablation Study: Personalized top-k selection gives each client a subspace tailored to its local update rather than identical top-r_i directions for same-rank clients.The study compares client-specific selection with Fixed-Top and Random-Selection variants.
  • 5.4 Analysis and Scalability: FedRoRA maintains its performance advantage across varying label-skew levels, although method gaps narrow at α = 1.0.This robustness analysis is reported in Figure 5a.
  • 5.4 Analysis and Scalability: FedRoRA maintains a consistent advantage in both low-rank-heavy and high-rank-heavy rank distributions.Both configurations use ranks {8, 16, 32, 64} and N = 20 while changing per-rank client counts.

6 Conclusion

The paper proposes FedRoRA for federated LLM fine-tuning under resource and data heterogeneity. It separates shared adaptation directions from personalized rank-wise magnitudes and reports stronger performance than existing rank-heterogeneous FL-LoRA methods, while identifying broader scalability as future work.

  • FedRoRA targets federated LLM fine-tuning under both resource and data heterogeneity.
  • FedRoRA decomposes each local update into shared adaptation directions and client-specific rank-wise magnitudes using personalized triplets.
  • FedRoRA outperforms state-of-the-art rank-heterogeneous FL-LoRA methods across NLU and NLG benchmarks under diverse non-IID settings.
  • Future work will examine scalability to more extreme task heterogeneity and other PEFT architectures.

Limitations

FedRoRA’s limitations concern server-side scalability and the absence of explicit cryptographic privacy protections. Exact SVD may become costly for very large models or federations, while integration with Differential Privacy remains future work.

  • Server-Side Computational Overhead: Exact SVD and per-client projection coefficients add server-side computation, although the overhead is negligible relative to local training for RoBERTa-Large and LLaMA-2-7B.The paper identifies randomized or approximated SVD as a possible future solution for larger deployments.
  • Server-Side Computational Overhead: Scaling exact SVD to 70B+ models or federations with thousands of clients may require randomized or approximated techniques.
  • Privacy Guarantees: FedRoRA exchanges low-rank updates rather than raw data but does not explicitly integrate cryptographic protections such as Differential Privacy.The paper calls for studying interactions between SVD projection, magnitude scaling, and such protections.

B Detailed Motivation Experimental Setup

The motivation experiments test whether shared directions with personalized scaling can support non-IID clients, using controlled client distributions, rank groups, and SVD-based transfer and aggregation procedures.

  • LoRA Configuration: FedRoRA’s local parameterization normalizes directional factors while the diagonal scale encodes rank-wise magnitudes.
  • Client Data Distributions: The experiments use RoBERTa-Large on four GLUE datasets with two clients per task assigned strongly non-IID label distributions.Binary-task label-0 ratios are (0.8, 0.2) and (0.2, 0.8); MNLI distributions are (0.2, 0.6, 0.2) and (0.2, 0.2, 0.6).
  • Motivation Experiments: Exp1 measures average cosine similarity between independently converged clients’ full LoRA updates across layers and target modules.
  • Motivation Experiments: Exp2 compares unified SVD-based rank-truncated initialization against local training in a federation spanning rank and label-distribution groups.The setup uses four clients, ranks 8 and 16, 10 rounds, and two local epochs.
  • Motivation Experiments: Exp3 transfers SVD-derived orthonormal directions between clients and trains only the diagonal scale, also reporting zero-shot and reverse-direction results.

C.1 Robustness to Non-IID Degree (α)

FedRoRA remains strongest across Dirichlet concentration levels, with larger advantages under stronger label skew, and retains a clear advantage with deeper local optimization and larger federations.

  • Robustness to Data Heterogeneity: FedRoRA achieves the highest average accuracy across α ∈ {0.3, 0.5, 0.7, 1.0}, with margins widening as α decreases.
  • Deeper Local Optimization: FedRoRA maintains a clear advantage when local epochs increase to E = 4.The reported analysis attributes this robustness to server-side subspace recomputation absorbing larger client drift.
  • Scalability to Client Number: FedRoRA preserves its advantage for N ∈ {12, 40}, with the gap over baselines widening at N = 40.

D Effect of Rank Distribution

FedRoRA consistently outperforms baselines across imbalanced and alternative rank configurations and retains leading performance under partial participation.

  • Effect of Rank Distribution: FedRoRA consistently outperforms all baselines across imbalanced and alternative rank-set configurations.
  • Effect of Rank Distribution: The projection-based selection prioritizes task-relevant directions regardless of how the rank budget is allocated.
  • Partial Participation: FedRoRA retains leading performance when only 5 of 20 clients participate per communication round.This setting uses random client selection while keeping the remaining NLU settings unchanged.

F Task Heterogeneity Setting

The evaluation examines FedRoRA across task heterogeneity, backbone changes, client and rank-distribution variations, partial participation, and implementation overhead. It also reports server-side computational and communication costs under the tested hardware configuration.

  • Task Heterogeneity Setting: Task heterogeneity is evaluated with 16 clients divided into four IID groups corresponding to MNLI, QNLI, SST-2, and QQP.This setting tests personalization when client divergence arises from different NLP tasks rather than label distributions within one task.
  • Backbone Robustness: FedRoRA’s NLG evaluation is repeated with Qwen3-8B while retaining the original FLAN grouping, rank choices, LoRA targets, optimization settings, rounds, and ROUGE-1 metric.The configuration uses 16 clients, ranks {8, 16, 32, 64}, q_proj and v_proj adapters, two local epochs, and ten rounds.
  • Efficiency and Communication: FedRoRA adds server-side truncated SVD and client-specific projection computations, while clients train LoRA factors with diagonal scales adding only r_i scalars per layer.The SVD costs O(d_out d_in r_max) per layer, projection costs O(N r_max d_out d_in), and projection is described as parallelizable.
  • Efficiency and Communication: Communication is r_i(d_out + d_in + 1) parameters per layer, with only r_i additional scalars attributable to the diagonal scale relative to standard rank-heterogeneous FL-LoRA.Per-round wall-clock times are reported under Intel Xeon Platinum 8570 CPUs and NVIDIA B200 GPUs.
Loading 2609.00632v1…