FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

• Non-IID client data can produce suboptimal global models, while LLM parameter sizes make federated computation and communication expensive.These challenges motivate personalized training with parameter-efficient adaptation.Source: §1 · pp. 1–2
• FDLoRA lets clients contribute labeled private data without directly sharing sensitive logs, supporting collaboration across ISPs while preserving privacy.The framework is motivated by private network-log analysis with limited labeled data.Source: §1 · pp. 1–2
• FDLoRA combines personalized and global LoRA modules, freezing the LLM and adaptively fusing the modules to improve each client’s local task.The personalized module remains private, while the global module is aggregated across clients.Source: §1 · p. 2
• Experiments in log analysis and medical diagnosis show superior performance and stability, while reducing computation and communication costs.The paper compares FDLoRA with six baselines in two practical scenarios.Source: §1 · p. 2

2.1 Problem Definition of FL & PFL

• PFL addresses client drift and weak personalization by separating shared general knowledge from client-specific knowledge across heterogeneous clients.A single global model may fit the average client poorly when local distributions differ substantially.Source: §2.1 · p. 3
• PFL seeks high accuracy across heterogeneous data distributions without requiring prior knowledge of those distributions.Its decomposition assigns shared parameters to general knowledge and personalized parameters to client-specific knowledge.Source: §2.1 · p. 3

2.2 Low-rank Adaptation

• LoRA adapts a frozen pretrained LLM by learning low-rank matrices A and B, where r << min(d,k), instead of changing the original parameter matrix.The low-rank update uses substantially fewer trainable parameters.Source: §2.2 · p. 3
• LoRA reduces storage requirements and supports efficient task switching without introducing inference latency.These benefits follow from freezing the original LLM parameters while learning rank-decomposition matrices.Source: §2.2 · p. 3

3.1 Problem Formulation

• FDLoRA represents basic, domain-general, and personalized knowledge as distinct components for multi-client LLM customization.Basic knowledge remains in the pretrained LLM, domain-general knowledge is shared, and personalized knowledge reflects each client’s non-IID distribution.Source: §3.1 · p. 3
• FDLoRA uses a shared LoRA module θ_s for domain-general knowledge and a client-specific LoRA module θ_p^(i) for personalized knowledge.The basic pretrained knowledge Θ remains unchanged.Source: §3.1 · p. 3
• A function G dynamically fuses basic, domain-general, and personalized knowledge while the objective minimizes the summed loss across clients.The fusion parameters are learnable for each client.Source: §3.1 · p. 3

3.2 Framework Overview

• Each client contains an LLM, personalized LoRA, global LoRA, and AdaFusion module; only global LoRA parameters are uploaded, aggregated, and broadcast.The personalized module learns from local data, while AdaFusion combines the personalized and global modules after federated updates.Source: §3.2 · p. 4

3.4 Federate Learning

• Averaged client gradients update the global LoRA module, which is then redistributed to clients at each outer step.Communication occurs once every K inner optimization steps, and clients perform T × K inner steps across the stage.Source: §3.4 · pp. 4–5
• Clients independently update personalized LoRA parameters for K > 1 inner steps, making cross-client communication occur only after multiple local updates.K = 3 is the default setting in the experiments, reducing communication frequency.Source: §3.4 · p. 4
• The personalized LoRA preserves local knowledge, whereas the global LoRA accumulates knowledge from all clients and is aligned with local knowledge through SFT.Increasing K gives more attention to local knowledge during this alignment.Source: §3.4 · p. 5

3.5 Adaptive Fusion

• AdaFusion combines corresponding parameters from the two LoRA modules element-wise and requires equal rank r for structural alignment.The combined module is formed from weighted factors of the personalized and global LoRAs.Source: §3.5 · p. 5
• Gradient-free black-box optimization selects fusion weights by minimizing few-shot cross-entropy loss with L1 regularization, avoiding a memory-intensive client hypernetwork.The optimized coefficients are w1 and w2, and the regularization penalizes extreme values.Source: §3.5 · p. 5

4 EVALUATION

• The evaluation covers system log analysis and medical diagnosis, using three LogHub datasets and a bio-medicine dataset.The study formulates seven research questions covering baseline performance, communication, fusion, and computation trade-offs.Source: §4.1 · p. 6, §4 · p. 6

4.1 Experiment Setup

• The datasets exhibit heterogeneous sample sizes and input-length distributions across clients, with Scenario-2 datasets containing 500 to 2,000 samples.Scenario-1 uses three datasets with sample sizes ranging from 10,000 samples each, while Scenario-2 uses five datasets.Source: §4.1 · p. 6
• Client data is partitioned under a Dirichlet non-IID setting, with α controlling class imbalance and a default value of α = 0.5.Increasing α reduces class imbalance within each client’s dataset.Source: §4.1 · p. 6
• FDLoRA is compared with six baselines: Local, FedAVG, FedKD, FedAMP, FedRep, and FedRoD.The baselines include parameter averaging, knowledge distillation, attentive message passing, shared representations with client heads, and robust decoupling.Source: §4.1 · p. 6
• The experiments fine-tune LLaMA2-7B with LoRA, leaving about 0.03B trainable parameters, approximately 5% of the total parameters.LoRA reduces computation and storage costs while retaining performance comparable to full fine-tuning.Source: §4.1 · pp. 6–7, Figure 4 · p. 7
• Log anomaly detection is evaluated with F1 score, while medical diagnosis uses exact-match accuracy against ground-truth answers.The paper refers to both measures uniformly as accuracy in subsequent descriptions.Source: §4.1 · p. 7

4.2 RQ1: Performance on different scenarios

• FDLoRA achieves the highest mean accuracy across both scenarios and varying α values, consistently outperforming other methods under non-IID data.As α increases and class imbalance decreases, all methods improve, but FDLoRA remains ahead.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 RQ2: Performance with varies communication round

• Increasing communication rounds consistently improves average accuracy across different client counts, indicating robust gains across deployment scales.The evaluation varies T from 0 to 30 with N=3, 5, and 7 clients and K=3 local updates.Source: §4.3 · p. 7, Figure 5 · p. 8

4.4 RQ3: Performance with varies communication frequency

• More frequent communication generally improves average accuracy, with K=1 outperforming less frequent InnerOpt updates such as K=5.The improvement rate varies by scenario and update frequency, so communication must be balanced against overhead.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5 RQ4: Performance with varies synchronous update frequency

• Synchronous updates improve accuracy steadily, whereas H=3 and H=5 fluctuate, H=10 and H=T improve more consistently, and H=∞ decreases accuracy.The results indicate that synchronization is important, but its frequency should balance alignment benefits against update costs.Source: §4.5 · p. 9, Figure 7 · p. 9

4.6 RQ5: Effectiveness of combining personalized LoRA and global LoRA

• Combining personalized and global LoRA modules yields the highest scores in both scenarios, exceeding either standalone module.The personalized module captures client-specific knowledge, while the global module captures general patterns across clients.Source: §4.6 · p. 9, Table 4 · p. 9

4.7 RQ6: Trade-off between communication cost and computational cost

• Data parallelism achieves optimal Scenario-1 performance, whereas larger batch sizes or accumulation factors increase training time and can reduce average accuracy.The result indicates that increasing parallel resources does not uniformly improve convergence.Source: §4.7 · p. 10, Table 5 · p. 10
• In Scenario-2, larger batch sizes outperform smaller ones, but increasing the accumulation factor degrades performance.The findings show that preferred compute strategies differ across scenarios.Source: §4.7 · p. 10, Table 5 · p. 10
• FDLoRA balances communication and computation costs while maintaining competitive performance across log anomaly detection and medical question answering.The comparison spans diverse scenarios rather than a single task.Source: §4.7 · p. 10

4.8 RQ7: Advantages of AdaFusion method

• AdaFusion consistently outperforms Random, Average, and Sum across all α values in Scenario-1 by adapting to varying data heterogeneity.The compared fusion methods use fixed or randomly assigned weights, whereas AdaFusion dynamically adjusts fusion weights.Source: §4.8 · p. 10
• AdaFusion fluctuates in Scenario-2 and is inferior to Sum at α=1.0, showing that fusion efficacy depends on the FL environment and data distribution.It remains competitive at α={0.1, 0.5}.Source: §4.8 · p. 10

Personalized federated learning

• PFL addresses non-IID data by personalizing global models or directly learning personalized models for clients.The two broad approaches include separate global-personalization stages and single-stage personalized-model learning.Source: §5 · pp. 10–11
• Data-based PFL reduces client drift but can alter local distributions and lose information about client behavior diversity.Model-based approaches instead use regularization, meta-learning, or transfer learning to connect global and local solutions.Source: §5 · p. 11
• Similarity-based methods personalize FL by modeling pairwise or group-level client relationships, including attention, clustering, and capability-aware subnetworks.Examples include FedAMP, HeteroFL, and FedGroup.Source: §5 · p. 11

Parameter-efficient fine-tuning

• PEFT reduces adaptation cost by updating small parameter sets through adapters, soft prompts, or reparameterized low-rank representations such as LoRA [19].These methods avoid full fine-tuning of increasingly large LLMs.Source: §5 · pp. 11–12

Compared with the previous works

• FDLoRA extends reparameterized PEFT by combining personalized and global LoRA modules through fusion for federated clients with scarce labeled data.The framework is designed to minimize fine-tuned parameters while incorporating knowledge across clients.Source: §5 · p. 12

6 CONCLUSION

• FDLoRA uses dual LoRA modules to capture personalized and global knowledge while reducing the number of trainable parameters.The design leverages collaboration across clients without requiring full-model training.Source: §6 · p. 12
• FDLoRA achieves state-of-the-art performance in log analysis and medical diagnosis, with stability and robustness across non-IID degree, client count, and communication settings.The conclusion reports robustness with respect to the number of communication rounds and their frequency.Source: §6 · p. 12
• FDLoRA outperforms increased-batch-size strategies in communication cost, training time, and compute and data usage.The comparison concerns resource efficiency rather than only predictive performance.Source: §6 · p. 12
• Future work will evaluate FDLoRA in additional industrial scenarios, including natural language understanding, sentiment analysis, and recommendation systems.The authors identify broader real-world applicability as an open evaluation direction.Source: §6 · p. 12

A1 CONVERSATION TEMPLATE

• The conversation templates combine scenario-specific prompts with input segments to guide LLM responses for log analysis and medical diagnosis.The templates provide structured context and constraints for targeted generation.Source: A1 CONVERSATION TEMPLATE · p. 14, Figure A1 · p. 14
• The log-analysis template asks the model to determine whether supplied logs contain an anomaly and output only yes or no.The prompt includes the question, log sequence, and response field.Source: A1 CONVERSATION TEMPLATE · p. 14, A3 DATA PREPOSSESSING · p. 14

A2 DATASET EXAMPLE

• Training examples for both scenarios are stored in JSON format.The appendix presents the dataset format as part of the training-data preparation.Source: A2 DATASET EXAMPLE · p. 14, Figure A2 · p. 14

A3 DATA PREPOSSESSING

• Raw log preprocessing removes prefixes and noise, then uses automated parsing such as Drain to extract event and parameter information.The process separates consistent log-event content from variable parameters.Source: A3 DATA PREPOSSESSING · p. 14
• Parsed log content is grouped into fixed-size sliding windows, with a default size of 50, to form anomaly-detection sequences.The appendix states that the content part is selected to improve detection performance.Source: A3 DATA PREPOSSESSING · p. 15

A4 EXPERIMENT ENVIRONMENT DETAILS

• The experimental hardware uses three homogeneous devices containing six V100 GPUs in total.The environment details are reported in Table A1.Source: A4 EXPERIMENT ENVIRONMENT DETAILS · p. 15, Table A1 · p. 15
• The evaluated LLaMA2-7B configuration has 2.0T tokens, a 32,000-word vocabulary, 4K sequence length, and 32 layers.These are the listed model specifications.Source: A4 EXPERIMENT ENVIRONMENT DETAILS · p. 15