PaperlayerView exact revision on arXiv

Source-linked AI summary

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning

Mengjun Yi, Langxing Yang, Suhan Guo, Furao Shen, Jian Zhao

arXiv:2609.36986v1cs.AIcs.DC

TL;DR

Federated LoRA must resolve mismatches in how low-rank updates are aggregated and which clients collaborate. CF-LoRA decouples factor aggregation and clusters clients by learned B factors; it reports the best average performance across language and vision tasks while communicating one factor per round.

  • Problem

    Federated LoRA must determine how to aggregate low-rank adaptations and which clients should collaborate under heterogeneous data.

    Source: Introduction · p. 1

  • Method

    CF-LoRA learns a shared A while retaining personalized B_i factors, clusters clients by B_i similarity, and aggregates B within clusters under frozen A.

    Source: Introduction · p. 2, Stage I: Global A Aggregation & Local… · p. 4

  • Results

    CF-LoRA achieves the best average performance across language and vision tasks and communicates only one LoRA factor per optimization round.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    Decoupled aggregation and B-based client clustering jointly address structural aggregation and statistical collaboration mismatches.

    Source: Introduction · p. 2, Conclusion · p. 7

    CF-LoRA uses a predefined cluster count and a one-time partition, which may not fit intrinsic client groupings or adapt to evolving distributions and new clients or data.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Federated LoRA fine-tuning enables parameter-efficient adaptation of pre-trained models without sharing private data, but suffers from two fundamental mismatches under heterogeneous client data: a structural aggregation mismatch caused by independently averaging LoRA factors, and a statistical collaboration mismatch caused by enforcing a single global adapter across divergent clients. To address these issues, we propose CF-LoRA, a clustered federated LoRA fine-tuning framework that combines decoupled factor aggregation with adaptation-aware client clustering. CF-LoRA first learns a globally shared $A$ factor while retaining personalized $B_i$ factors, then identifies clients with similar adaptation patterns based on the cosine similarity of their learned $B_i$ factors, and finally performs intra-cluster $B$-factor aggregation with a frozen $A$ factor. By decoupling LoRA factor aggregation, CF-LoRA preserves the low-rank structure and mitigates the structural aggregation mismatch, while adaptation-aware clustering promotes collaboration among clients with similar adaptation patterns and reduces negative transfer caused by statistical heterogeneity. Experiments on four language tasks and four vision datasets with RoBERTa and ViT show that CF-LoRA achieves the highest average accuracy in both modalities while communicating only one LoRA factor per optimization round.

Introduction

Federated LoRA must address both how low-rank updates are aggregated and which clients collaborate. CF-LoRA combines decoupled factor aggregation with B-based client clustering, and reports strong performance across language and vision benchmarks with one factor communicated per round.

Introduction

  • Factor-wise averaging can introduce cross-client products absent from the ideal update, while aggregating full products destroys LoRA’s compact low-rank parameterization.These alternatives create a structural aggregation mismatch.Source: Introduction · p. 1
  • Under non-IID data, global sharing can cause negative transfer, whereas fully local adaptation discards transferable knowledge among related clients.This is the statistical collaboration mismatch.Source: Introduction · pp. 1–2
  • Existing approaches address only part of the coupled challenge: they handle aggregation consistency or heterogeneity-aware collaboration, but not both with aggregation-consistent LoRA updates.CF-LoRA instead globally learns A with personalized B_i factors, clusters clients by B_i cosine similarity, then aggregates B within clusters under frozen A.Source: Introduction · p. 2
  • CF-LoRA reports the best average performance across language and vision tasks while communicating only one LoRA factor per optimization round.The framework combines decoupled factor aggregation with adaptation-aware client collaboration.Source: Introduction · p. 2

Related Work

Prior federated learning work addresses non-IID data through global optimization, personalization, or clustered collaboration, while full-model cluster communication is expensive for large pre-trained models. Federated PEFT reduces adaptation costs, and CF-LoRA combines adaptation-aware clustering with decoupled factor aggregation.

Federated Learning under Data Heterogeneity

  • Clustered federated learning provides an intermediate collaboration granularity for related clients, but classical methods communicate multiple full models, making direct use with large pre-trained models expensive.Earlier approaches include model assignment and data-subspace similarity, while global and fully local approaches have distinct transfer tradeoffs.Source: Federated Learning under Data… · p. 2

PEFT and Federated LoRA Fine-Tuning

  • Federated LoRA methods have addressed aggregation consistency or heterogeneity-aware collaboration; CF-LoRA couples B-based client clustering with decoupled factor aggregation to address both.LoRA freezes the pretrained backbone and uses two trainable low-rank factors; prior approaches include freezing a factor, residual correction, personalized B, factor alignment, cluster experts, and adaptive combinations.Source: PEFT and Federated LoRA Fine-Tuning · pp. 2–3

Method

CF-LoRA separates the roles of LoRA factors: it shares A while retaining local B factors, then groups clients by B similarity for cluster-level fine-tuning. This design draws on the observed stability of A and heterogeneity-sensitive divergence of B.

Preliminaries

  • LoRA adds a low-rank update to a pre-trained model, whose backbone remains frozen, while clients may hold heterogeneous private datasets.The setup considers m clients with local data distributions that may differ.Source: Preliminaries · p. 3

Motivating Observation

  • Across clients, A factors remain comparatively similar while B-factor distances grow with data heterogeneity, motivating shared A and B-based client representations.The motivating study uses SST-2, RoBERTa-base, 12 clients, and varying Dirichlet concentration α.Source: Motivating Observation · p. 3, Figure 2 · p. 3

Overview of CF-LoRA

  • CF-LoRA proceeds through shared-A/local-B training, clustering by learned B similarity, and intra-cluster B aggregation with A frozen.Each optimization stage aggregates only one low-rank factor, avoiding the product of independently averaged factors.Source: Overview of CF-LoRA · p. 3, Figure 3 · p. 4

Stage I: Global A Aggregation & Local B Personalization.

  • Stage I averages clients’ A matrices globally while keeping B local, producing a shared A and personalized B matrices for later clustering.This shared-A, personalized-B parameterization avoids multiplying independently averaged LoRA factors.Source: Stage I: Global A Aggregation & Local… · pp. 3–4

Stage II: Clustering based on B-Similarity Matrix.

  • Stage II flattens each learned B matrix, computes pairwise cosine similarities, and clusters clients expected to benefit from collaboration.The resulting B representations encode client-specific adaptations associated with local data distributions.Source: Stage II: Clustering based on… · p. 4

Stage III: Intra-Cluster Federated Fine-Tuning.

  • Stage III freezes the shared A and aggregates B matrices within each cluster, making the aggregate update exactly the average of clients’ induced updates.Before this training, each cluster’s shared B initialization is formed by averaging its members’ final Stage I matrices.Source: Stage III: Intra-Cluster Federated… · pp. 4–5

Experimental Setup

The experiments evaluate federated fine-tuning on five GLUE sets under a Dirichlet non-IID partition and compare CF-LoRA with federated and clustered baselines. RoBERTa-base is trained with fixed client, round, and LoRA settings, with accuracy averaged across client test sets by size.

Datasets and Data Partitioning.

  • Five GLUE sets cover inference, paraphrase identification, and sentiment classification; training and validation data are partitioned across clients with Dirichlet α = 1.The same class-wise proportions create non-IID local training and test sets unless otherwise specified.Source: Datasets and Data Partitioning · p. 5

Baseline Methods.

  • Baselines include independent factor averaging, single-factor or residual-corrected aggregation, personalized B, factor alignment, and PACFL-based clustering.PACFL + FedIT clusters clients by principal angles between local data subspaces before FL-LoRA fine-tuning.Source: Baseline Methods · p. 5

Implementation Details.

  • Experiments use RoBERTa-base with 12 clients and 50 rounds; CF-LoRA allocates 4 rounds to Stage I and 46 to clustered training, using rank-4 LoRA on Q/V.Top-1 accuracy is weighted by client test-set size; Table 1 reports five-set accuracy over three runs, and Table 2 compares per-round communication payloads.Source: Implementation Details · p. 5, Table 1 · p. 6, Table 2 · p. 6

Overall Performance

CF-LoRA combines decoupled factor aggregation with adaptation-aware clustering, achieving the strongest reported GLUE accuracy while transmitting only one LoRA factor per optimization round.

Better NLU Performance.

  • CF-LoRA achieves the highest accuracy on all five GLUE sets, raising average accuracy from 89.00% for FL-LoRA to 90.51% and exceeding FedRot-LoRA by 1.26 percentage points on average.Table 1 reports the overall GLUE comparison.Source: Better NLU Performance · p. 6
  • Unlike methods that address only aggregation consistency or client heterogeneity, CF-LoRA combines decoupled factor aggregation with adaptation-aware clustering for consistent improvements across tasks.PACFL + FedIT, for example, improves over FL-LoRA on QQP and SST-2 but underperforms it on both MNLI sets.Source: Better NLU Performance · p. 6

Communication Efficiency.

  • Figure 4 plots mean accuracy across five GLUE sets at different Dirichlet heterogeneity levels, with smaller α indicating stronger heterogeneity.Values above the bars report the corresponding mean accuracies.Source: Figure 4 · p. 6
  • CF-LoRA transmits only one factor per direction in each round, reducing the per-round LoRA-parameter payload by 50% versus methods that transmit A and B.It uploads and downloads A in Stage I and B in Stage III; A and B have equal size in these experiments.Source: Communication Efficiency · p. 6

Robustness Analysis

CF-LoRA is evaluated across data heterogeneity, federation size, and client-grouping strategies, with results favoring its approach across these settings.

Effect of Data Heterogeneity.

  • At α = 0.1, CF-LoRA reaches 93.94% average accuracy versus 78.37% for FL-LoRA, its largest reported margin across the tested heterogeneity levels.CF-LoRA consistently outperforms FL-LoRA across the tested Dirichlet heterogeneity levels.Source: Effect of Data Heterogeneity · p. 6
  • Under extreme heterogeneity, similar-adaptation grouping lets compatible updates reinforce one another, whereas global aggregation mixes updates from clients with different label distributions.The authors suggest that tasks with only two or three labels can leave some clients with samples from only one or a few classes.Source: Effect of Data Heterogeneity · pp. 6–7

Effect of Federation Size.

  • As the federation grows from 12 to 50 clients, CF-LoRA’s average accuracy drops 1.28 percentage points, less than FL-LoRA’s 1.75-point drop.CF-LoRA maintains a clear performance advantage as client count increases.Source: Effect of Federation Size · p. 7, Table 3 · p. 7

Effect of Client Grouping.

  • Using the same B-based client representations, hierarchical clustering reaches 90.51% average accuracy, exceeding K-means, spectral clustering, and random grouping by 1.43, 1.96, and 2.52 percentage points, respectively.The reported improvements over alternative grouping strategies indicate that the clustering choice matters within this setup.Source: Effect of Client Grouping · p. 7, Table 4 · p. 7

Evaluation on Vision Tasks with ViT-B/16

The vision evaluation applies CF-LoRA with ViT-B/16 to four image-classification datasets and reports the best performance across all four.

Evaluation on Vision Tasks with ViT-B/16

  • The evaluation uses ViT-B/16 as the pretrained backbone for federated fine-tuning on four image-classification benchmarks.The datasets are Flowers102, DTD, UCF101, and Caltech101.Source: Evaluation on Vision Tasks with ViT-B/16 · p. 7
  • CF-LoRA achieves the best performance across all four vision datasets, demonstrating effectiveness on diverse visual downstream tasks and generalizing decoupled factor aggregation and B-based clustered collaboration across model architectures and modalities.Table 5 reports classification accuracy (%) using ViT-B/16 as the pre-trained backbone.Source: Evaluation on Vision Tasks with ViT-B/16 · p. 7, Table 5 · p. 7
  • Together with the language results, the vision findings report generalization of decoupled factor aggregation and B-based clustered collaboration across RoBERTa and ViT architectures.The paper describes the two modalities as language understanding and image classification.Source: Evaluation on Vision Tasks with ViT-B/16 · p. 7

Limitations

CF-LoRA uses a fixed cluster count and clusters clients only once after Stage I, limiting adaptation of group assignments as the federation changes.

Limitations

  • CF-LoRA prespecifies K and clusters once after Stage I, so its partition cannot adapt when client distributions evolve, new clients join, or new data become available.Adaptive clustering without a prespecified cluster count is proposed as future work, with adaptivity, stability, and computation trade-offs left to evaluate.Source: Limitations · p. 7

Conclusion

CF-LoRA achieves the best average performance on language and vision tasks while remaining robust across heterogeneity levels and federation sizes. It addresses aggregation and collaboration mismatches through decoupled factor aggregation and B-based client clustering, communicating only one LoRA factor per optimization round.

Conclusion

  • CF-LoRA achieves the best average performance on language and vision tasks and remains robust across heterogeneity levels and federation sizes.Experiments report these outcomes across the evaluated settings.Source: Conclusion · p. 7
  • Decoupled factor aggregation addresses the structural aggregation mismatch, while B-based client clustering mitigates the statistical collaboration mismatch.Source: Conclusion · p. 7
  • CF-LoRA communicates only one LoRA factor per optimization round.Source: Conclusion · p. 7
Loading 2609.36986v1…