PaperlayerView exact revision on arXiv

Source-linked AI summary

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Mengjun Yi, Huaian Gu, Yinghao Ai, Furao Shen, Jian Zhao

arXiv:2609.37033v1cs.AI

TL;DR

Heterogeneous clients make a single global adapter insufficient for personalized federated fine-tuning, while existing personalized designs often use the same low-rank structure for shared and private adaptation. FedLAFP assigns LoRA to shared aggregation and RandLoRA to private adaptation. Across four visual benchmarks, it achieves the best accuracy on every dataset, averaging 86.93% and improving over FedRandLoRA by 1.30 percentage points.

  • Problem

    Heterogeneous clients challenge a single global LoRA, while personalized federated methods often use the same adapter family for shared and private roles despite their distinct requirements.

    Source: Introduction · p. 1

  • Method

    FedLAFP combines a globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch and client- and layer-specific fusion.

    Source: Introduction · p. 2

  • Results

    FedLAFP achieves the best accuracy on all four visual benchmarks, averaging 86.93% and improving over FedRandLoRA by 1.30 percentage points.

    Source: Visual benchmarks · p. 6

  • Takeaways & Limitations

    The controlled analysis supports assigning LoRA to shared aggregation and RandLoRA to client-specific residual recovery.

    Source: Conclusion · p. 7

    The controlled role-assignment study uses a synthetic linear-regression setting with isotropic Gaussian inputs and additive Gaussian noise.

    Source: Controlled setup · p. 3

Abstract

from arXiv · show

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of $86.93\%$ and exceeding the best baseline average by $1.30$ percentage points.

Introduction

Federated fine-tuning can avoid sharing raw data and communicating a full foundation model, but heterogeneous clients challenge a single global adapter. FedLAFP assigns different adapter families to shared aggregation and private personalization.

Introduction

  • Heterogeneous clients may lose client-specific information under a single global LoRA, while local-only adapters forgo cross-client knowledge sharing.Existing personalized methods often use structurally identical LoRA modules for shared and private branches, overlooking their distinct roles.Source: Introduction · p. 1
  • The shared component is designed to capture knowledge that generalizes across clients, while the private component models each client's local-distribution characteristics.The shared branch is repeatedly communicated and aggregated; the private branch remains local and is not averaged across clients.Source: Introduction · p. 1
  • FedLAFP pairs a globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch and client- and layer-specific fusion.Only shared LoRA parameters are communicated; private RandLoRA parameters remain local.Source: Introduction · p. 2
  • FedLAFP achieves the best accuracy on every one of four visual recognition benchmarks, outperforming local-only adaptation and a broad range of federated LoRA baselines.The benchmarks cover texture, fine-grained pet, scene, and action recognition.Source: Introduction · p. 2

Related Work

Related work covers parameter-efficient adaptation, personalized federated learning, and federated LoRA designs. FedLAFP differs by assigning LoRA to shared aggregation and RandLoRA to private adaptation.

Related Work

  • LoRA represents updates to frozen weights with two trainable low-rank factors, reducing communication overhead and allowing updates to be merged into the backbone without additional inference latency.Other variants adapt rank budgets, decompose weight magnitude and direction, or learn combinations of fixed random low-rank matrices.Source: PEFT and LoRA Variants · p. 2

Federated Learning and Personalization

  • Personalized federated methods either train a shared initialization or representation for later client adaptation, or retain private components while aggregating shared ones.The cited methods were developed for conventional task-specific models and do not examine assigning different PEFT parameterizations to shared and private roles.Source: Federated Learning and Personalization · p. 2

Federated LoRA Fine-Tuning

  • Federated LoRA methods explore global aggregation, selective sharing, subspace separation, or explicit shared–private branches.Unlike methods that use one low-rank adapter or the same adapter family in both branches, FedLAFP assigns LoRA to shared aggregation and RandLoRA to private adaptation.Source: Federated LoRA Fine-Tuning · pp. 2–3

Motivation

A controlled linear study compares LoRA and RandLoRA for client-specific recovery and cross-client aggregation. The results support using RandLoRA privately and LoRA for shared aggregation.

Motivation

  • The study uses a federated linear-regression problem with a frozen base model and trains both adapters on the same client samples to isolate parameterization effects.The synthetic construction provides known shared and private components for evaluating recovery.Source: Controlled setup · p. 3
  • Personalization error measures recovered client-specific residuals against ground truth, while aggregation error measures the average update against the ground-truth shared update.Mean pairwise cosine similarity quantifies directional consistency; lower aggregation error and higher similarity indicate more aggregation-friendly updates.Source: Controlled setup · p. 3
  • RandLoRA better captures client-specific variation, consistently achieving lower personalization error as heterogeneity increases.The study attributes this to its ability to represent a broader range of update directions and recover client-specific residuals more accurately.Source: Q1: Which adapter better captures… · p. 3

Q2: Which adapter better supports aggregation?

  • LoRA yields lower aggregation error and more aligned client updates, preserving shared directions more effectively as heterogeneity rises.Increasing heterogeneity makes aggregation more difficult for both adapters.Source: Q2: Which adapter better supports… · p. 3
  • The controlled study concludes that RandLoRA is better suited to expressive private adaptation, while LoRA is better suited to compact shared aggregation.This complementary role assignment motivates FedLAFP's asymmetric design.Source: Q2: Which adapter better supports… · p. 3

Method

FedLAFP assigns distinct roles to shared and private adaptation: a shared LoRA branch is aggregated across clients, while private RandLoRA parameters and client-specific mixing logits remain local. The branches are fused within adapted layers and trained through alternating local updates and server aggregation.

Method

  • FedLAFP attaches shared LoRA and private RandLoRA branches to adapted layers of a frozen backbone and combines them with client- and layer-specific weights.The formulation applies the branches to the query and value projections in every Transformer attention block in the experiments; it can also be applied to other linear layers.Source: Method · p. 3, Role-Aware Heterogeneous Adaptation · p. 4

Problem Formulation

  • Only shared LoRA parameters are communicated and aggregated; client-specific RandLoRA parameters and mixing logits persist locally and are optimized jointly with the shared branch.The personalized objective is sample-weighted across clients.Source: Problem Formulation · pp. 3–4

Role-Aware Heterogeneous Adaptation

  • The shared update uses LoRA as the compact branch communicated to the server, while the private RandLoRA update sums fixed random low-rank bases scaled by locally learned coefficients.RandLoRA is full-rank-capable, its bases are deterministically generated from a shared public seed, and its private scaling vectors remain local.Source: Shared low-rank aggregation branch · p. 4, Private full-rank-capable… · p. 4
  • Client- and layer-specific trainable logits produce input-independent softmax weights that fuse the shared and private adapter outputs within each target layer.The logits start at zero, giving the branches equal initial weights, and are optimized locally with the task loss.Source: Client-specific layer-wise fusion · pp. 4–5

Federated Training and Aggregation

  • Each round, selected clients receive shared LoRA parameters, retain their private state, and jointly update shared parameters, private scaling variables, and mixing logits before aggregation.The default server step averages the shared LoRA factors separately using sample-weighted FedAvg; inference uses the final shared parameters with each client's retained private parameters and logits.Source: Federated Training and Aggregation · p. 5, Personalized inference · p. 5

Experiments

Across four visual recognition benchmarks, FedLAFP achieves the highest average accuracy among the compared methods. The experiments also support assigning LoRA to shared aggregation and RandLoRA to private adaptation, with client–layer fusion performing best among the tested fusion strategies.

Experimental Setup

  • The visual evaluation covers texture, pet, scene, and action recognition using few-shot federated data partitioned among 12 clients.The ViT-B/16 backbone is frozen, and adaptation targets query and value projections; visual results average three runs.Source: Datasets and backbone · p. 5, Implementation details · p. 5

Performance Comparisons

  • 86.93% average accuracy for FedLAFP exceeds the best remaining federated LoRA baseline’s 85.34% across four visual benchmarks.FedLAFP also beats FedLoRA and FedRandLoRA by 1.71 and 1.30 percentage points, respectively.Source: Table 1 · p. 6, Visual benchmarks · p. 6

Method MRPC SST-2 Average

  • FedLAFP exceeds the stronger FedALT baseline by 0.49 percentage points on MRPC and 0.35 on SST-2.The comparison uses RoBERTa-Large on two GLUE language tasks.Source: Table 2 · p. 6, Language benchmarks · p. 6

Ablation Study

Scheme DTD Pets SUN UCF

  • Client–layer fusion achieves the highest accuracy on all four benchmarks without an auxiliary gating unit.It learns input-independent weights per client and adapted layer using two trainable scalar logits per adapted layer for each client.Source: Impact of Fusion Weights · pp. 6–7, Table 3 · p. 6

Sensitivity Analysis

  • Private RandLoRA rank affects accuracy more than shared LoRA rank, and DTD responds more strongly to rank changes than UCF101.Figure 5 varies one hyperparameter at a time against the default configuration.Source: Sensitivity Analysis · p. 7, Figure 5 · p. 7
  • Accuracy generally decreases as client count increases, while raising local epochs from 1 to 3 improves accuracy and raising them to 5 adds little.With fixed training data, more clients divide the data into smaller local partitions.Source: Sensitivity Analysis · p. 7, Figure 5 · p. 7
  • FedLAFP’s gain over FedLoRA is 1.49, 1.70, 0.88, and 0.21 percentage points at Dirichlet α values of 0.01, 0.1, 1, and 100, respectively.The largest gain occurs at α = 0.1; α = 100 approximates an IID allocation.Source: Sensitivity Analysis · p. 7, Figure 6 · p. 7

Conclusion

FedLAFP assigns compact LoRA to shared aggregation and full-rank-capable RandLoRA to private adaptation, with client- and layer-specific fusion. Across four visual benchmarks, it achieved the best accuracy on every dataset and averaged 86.93%, exceeding the strongest baseline by 1.30 percentage points.

Conclusion

  • 86.93% average accuracy across four visual benchmarks, with the best accuracy on every dataset and a 1.30 percentage-point lead over the strongest baseline.Ablation studies and sensitivity analyses further confirmed the asymmetric design's value under data heterogeneity.Source: Conclusion · p. 7
  • FedLAFP pairs compact LoRA for shared aggregation with client-private, full-rank-capable RandLoRA and client- and layer-specific fusion.Only shared LoRA parameters are communicated; private RandLoRA parameters and fusion logits remain local.Source: Conclusion · p. 7
  • Controlled analysis found that LoRA produces more aggregation-stable updates, whereas RandLoRA better recovers client-specific residuals.Source: Conclusion · p. 7
  • Future work will extend FedLAFP to larger models, study realistic client resource heterogeneity, add privacy-preserving aggregation, and explore automatic shared/private capacity allocation.Source: Conclusion · p. 7

Related papers

Loading 2609.37033v1…