PaperlayerView exact revision on arXiv

Source-linked AI summary

When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair

Wenqiang Luo, Jacky Wai Keung, Boyang Yang, He Ye, Claire Le Goues, Tegawende F. Bissyande, Haoye Tian, Bach Le

arXiv:2412.01072v1cs.SE

TL;DR

LLM-based program repair depends on high-quality code data, while proprietary repositories are difficult to share because of privacy concerns. This paper evaluates federated fine-tuning on program repair and finds gains comparable to centralized fine-tuning, with little impact from tested code heterogeneity. Its findings also show that algorithm performance varies across LLMs and that personalization remains challenging.

  • Problem

    LLM-based program repair relies on high-quality code repositories, but prior research has not sufficiently addressed federated learning for privacy-preserving code-related tasks.

    Source: §1 · p. 2, §3.2 · p. 10

  • Method

    The study evaluates federated fine-tuning for program repair across code LLMs, using private industrial data and an augmented repair benchmark.

    Source: §1 · p. 4, §8 · p. 34

  • Results

    Federated fine-tuning raises performance by up to 16.57% on Top@10 and 18.44% on Pass@10, reaches comparable centralized performance, and tested heterogeneity has no significant impact.

    Source: §1 · p. 4

  • Takeaways & Limitations

    FedAvg shows stability and practicality, whereas personalized methods such as pFedMe face challenges adapting to LLMs and program repair.

    Source: §8 · p. 35

    The tested code features and single-company online-programming dataset may not represent repositories in other industries.

    Source: §7 · p. 33

Abstract

from arXiv · show

Software systems have been evolving rapidly and inevitably introducing bugs at an increasing rate, leading to significant losses in resources consumed by software maintenance. Recently, large language models (LLMs) have demonstrated remarkable potential in enhancing software development and maintenance practices, particularly in automated program repair (APR) with improved accuracy and efficiency of bug fixing. However, LLM-based APR heavily relies on high-quality code repositories. A larger portion of existing code repositories are for private use and proprietary assets from various industries, reflecting more diversity and nuances in the data since real-world industries often have more extensive software development practices, which cannot be covered by merely public datasets. Therefore, utilizing private datasets shows significant potential in enhancing software development and maintenance. However, obtaining such data from various industries is hindered by data privacy concerns, as companies are reluctant to share their codebases. To address the gap, we investigate the use of federated learning as a privacy-preserving approach that enables private entities to fine-tune LLMs on proprietary and decentralized data, facilitating the collaboration between clients to fully utilize their data to help enhance software development and maintenance. Our evaluation reveals that federated fine-tuning can effectively enhance program repair capabilities. Notably, the impact of heterogeneous code on LLM fine-tuning is negligible, indicating that real-world industries can benefit from collaborative development regardless of diverse data distributions. Furthermore, each type of federated algorithm exhibits unique strengths across different LLMs, suggesting that fine-tuning for program repair can be enhanced by tailoring the optimization process to specific characteristics of different LLMs.

1 INTRODUCTION

LLM-based program repair benefits from fine-tuning, but public data alone may not capture proprietary industrial code, and privacy constrains direct collaboration. This study evaluates federated fine-tuning, finding gains comparable to centralized fine-tuning, little impact from tested code heterogeneity, and algorithm strengths that vary across LLMs.

1 INTRODUCTION

  • Prior work has explored LLM fine-tuning for APR through approaches including NMT, multi-objective training, additional code-review information, data augmentation, and parameter-efficient techniques.Fine-tuning leverages pretrained LLMs for downstream repair tasks.Source: §1 · p. 2
  • The study addresses the gap between public-code fine-tuning and proprietary industrial repositories, whose diversity is difficult to use collaboratively because of privacy concerns.The introduction also notes that academic models may underperform on unseen private industrial data.Source: §1 · pp. 2–3
  • Federated fine-tuning achieves maximal increases of 16.57% on Top@10 and 18.44% on Pass@10, with performance comparable to centralized fine-tuning.CodeQWen-7B and CodeLlama-13B fix 7 and 2 unique bugs, respectively, the most among the evaluated LLMs.Source: §1 · p. 4
  • Heterogeneous code has no significant impact compared with ideal homogeneous data, while tested distributions yield maximal increases of 18.41% on Top@10 and 21.46% on Pass@10.The study constructs heterogeneous distributions using coding style, code complexity, and code embedding.Source: §1 · p. 4
  • FedAvg performs best overall across the evaluated federated algorithms, while personalized learning remains challenging for LLM fine-tuning.Different federated optimization approaches show varying strengths and weaknesses across LLMs.Source: §1 · pp. 4–5

2 BACKGROUND

The background outlines parameter-efficient federated fine-tuning, which keeps data local while clients train and aggregate adapters. It also frames code heterogeneity primarily as feature skew and motivates studying its effect on global models.

2.1 Parameter-Efficient Federated Fine-tuning

  • Clients fine-tune locally, upload lightweight adapters for server aggregation, then receive the global adapter for the next round; datasets remain local.The final-round global model is used for downstream tasks.Source: §2.1 · pp. 5–7
  • QLoRA quantizes pretrained weights to 4-bit NF4, freezes them, and trains low-rank adapters initialized consistently across clients.The adapter is represented as ΔW = BA, with rank r as a hyperparameter.Source: §2.1 · p. 6
  • Each client minimizes its local loss over adapter parameters using cross-entropy to compare predicted output and actual code token distributions.The text describes optimizers such as SGD for updating the low-rank adapter parameters.Source: §2.1 · p. 6

2.2 Data Heterogeneity in Federated Learning

  • Non-IID client data can pull local models toward different objectives, leaving the aggregated global model away from the optimum.The background identifies label, feature, and quantity skew as common forms of heterogeneity.Source: §2.2 · pp. 7–8
  • The study focuses on feature skew in code, not label skew or quantity skew, because program repair generates patches without fixed explicit labels.Code features may vary with languages, frameworks, and development practices; quantity skew often coexists with other skews.Source: §2.2 · p. 8

3 RELATED WORK

Prior work applies federated learning to software engineering, privacy-preserving LLM training, and fine-tuned program repair. The paper positions its study at their intersection: decentralized repair with private industrial data and heterogeneous code.

3.1 Federated Learning for Software Engineering

  • Software-engineering federated-learning studies cover code clone detection, defect prediction, debugging, commit classification, and code summarization, but leave gaps in LLMs, heterogeneous data, generation tasks, and private industrial data.The paper identifies reliance on simulated public data as a limit on prior findings' generalizability and practical relevance.Source: §3.1 · p. 9

3.2 Federated Learning with LLMs for Privacy Preservation

  • Prior LLM federated-learning work examines training components, distillation, privacy noise, recommendation, legal tasks, attacks, and prompt exchange, while code-related privacy-preserving applications remain insufficiently addressed.The cited prompt-tuning approaches exchange prompts or select important prompt layers to reduce parameter-transfer demands.Source: §3.2 · pp. 9–10

3.3 Fine-tuning LLMs for Program Repair

  • Earlier APR work also explores domain-specific code representations, tool integration, and multilingual repair through continual learning and replay.Examples include static analysis for bug detection and completion engines to address syntactic and semantic validity challenges.Source: §3.3 · p. 10
  • Fine-tuned LLMs have improved APR across benchmarks, but prior repair work mainly uses centralized settings, leaving decentralized program repair for further investigation.The study simulates distributed devices and datasets to examine federated fine-tuning for repair and other code-related tasks.Source: §3.3 · pp. 10–11

4 STUDY DESIGN

The study compares federated fine-tuning across research questions, private-data training and repair evaluation, six code LLMs, repair metrics, and a specified compute and training setup.

4.1 Research Questions

  • The study asks whether federated learning improves repair while protecting privacy, how code heterogeneity affects repair, and how federated optimization algorithms perform across LLMs.The heterogeneity investigation constructs Non-IID scenarios; algorithm coverage includes client-side, server-side, and personalized optimization.Source: §4.1 · p. 11

4.2 Fine-tuning Dataset and Evaluation Benchmark

  • TutorCode supplies 1,239 proprietary buggy C++ programs from 427 programmers, while EvalRepair-Java evaluates 163 samples with test cases averaging 583 per problem.EvalRepair-Java builds on HumanEval-Java and adds EvalPlus tests; the expanded test suites target patch-overfitting risk.Source: §4.2 · pp. 12–13

4.3 Evaluation Models

  • Six code LLMs sized 7B–15B were selected, with model choice guided by code focus, architecture and size diversity, instruction following, prevalence, and leaderboard performance.The selected set includes CodeLlama, DeepSeekCoder, WizardCoder, Mistral, and CodeQwen variants.Source: §4.3 · p. 13

4.4 Evaluation Metrics

  • Repair is measured with Top@1, Top@5, Top@10, Pass@5, and Pass@10; Top@k counts problems solved by at least one of k samples, while Pass@k is an unbiased estimator.The metrics are intended to assess repair performance at limited attempts and across generated samples.Source: §4.4 · pp. 14–15

4.5 Experimental Setup

  • Experiments use four RTX 4090 GPUs, paged AdamW at learning rate 1e-4, batch size one, up to 30 local epochs, and 10 global rounds.QLoRA rank and dropout are 32 and 0.05, with early stopping if validation loss does not significantly decrease after 10 epochs.Source: §4.5 · p. 15

5 EXPERIMENTS & RESULTS

This section evaluates federated fine-tuning for program repair against other fine-tuning approaches and across varying levels of code and data heterogeneity. It also examines federated algorithms across different LLMs.

5.1 RQ1: Effectiveness of Federated Fine-tuning

  • Federated fine-tuning improves repair over original models, beats local fine-tuning across all LLMs, and matches or surpasses centralized fine-tuning in several cases.It achieves the best Top@k and Pass@k scores for four models, while centralized fine-tuning is best for DeepSeekCoder-7B and CodeQWen-7B; federated fine-tuning is second-best for both.Source: Table 2 · p. 16, §5.1 · pp. 16–17
  • Centralized and local fine-tuning are inconsistent: centralized fine-tuning can underperform the original model, while local fine-tuning performs worst among the baselines.For example, centralized fine-tuning on Mistral-7B reduces Pass@1 by 5.62% versus the original model; the authors suggest dataset size may influence fine-tuning performance.Source: §5.1 · pp. 17–18
  • Across federated-tuned models, CodeQWen-7B fixes 7 unique bugs and CodeLlama-13B fixes 2, with CodeQWen-7B complementing other models’ repair coverage.CodeQWen-7B overlaps with CodeLlama-13B and other models while also addressing bugs beyond CodeLlama-13B’s scope.Source: Figure 2 · p. 18, §5.1 · pp. 18–19

5.2 RQ2: Impact of data heterogeneity

  • RQ2 constructs feature-skewed Non-IID code using coding style, modified-hunk code complexity, and CodeBERT NL-PL embeddings, then allocates clusters to clients with Dirichlet distributions.The study uses 40 coding-style clusters and 45 embedding clusters; the Dirichlet concentration parameter controls heterogeneity, with 0.1 and 0.01 for mild and medium scenarios and a single cluster per client for the extreme scenario.Source: §5.2 · pp. 19–22
  • Under medium heterogeneity, DeepSeekCoder-7B’s coding-style results improve over the original by 22.09% Top@10 and 25.62% Pass@10, while CodeQWen-7B is the exception to the finding that all LLMs improve.CodeLlama-7B has the smallest reported improvement under medium embedding heterogeneity: 3.68% Top@10 and 5.54% Pass@10.Source: Table 4 · p. 25, Table 5 · p. 25, Table 6 · p. 26
  • Under mild code-complexity heterogeneity, DeepSeekCoder-7B reaches 76.07% Top@10 and 75.42% Pass@10; under mild coding-style heterogeneity, its gains over the original are 14.73% Top@10 and 16.84% Pass@10.Under mild complexity heterogeneity, DeepSeekCoder-7B improves by 22.27% Top@10 and 24.80% Pass@10; under mild embedding heterogeneity, Mistral-7B improves by 14.72% Top@10 and 13.79% Pass@10.Source: Table 5 · p. 25, Table 4 · p. 25, §5.2 · pp. 25–26
  • Most LLMs outperform centralized fine-tuning under Non-IID distributions, except DeepSeekCoder-7B and CodeQWen-7B; Mistral-7B gains 14.11% on Top@10 and 13.92% on Pass@10 under medium Non-IID.The gains are compared with the central approach.Source: §5.2 · pp. 26–27
  • Federated fine-tuning significantly improves repair over original models across data distributions, including extreme Non-IID, with reported comparisons at p-value<0.05.For extremely heterogeneous coding style, the reported p-value is 2.56𝑒−6 and effect size is 0.34.Source: §5.2 · p. 27
  • Differences between Non-IID and IID are mostly not significant, and even significant cases have negligible effects, indicating heterogeneous code has little overall impact on repair performance.The significant cases have Cliff’s delta values of -0.11 and 0.06; overall results across the three code features have p-values≥0.05 and trivial effect sizes.Source: §5.2 · pp. 27–28
  • Model responses to heterogeneity differ: WizardCoder-15B is particularly susceptible, while CodeQWen-7B remains stable across distributions and code features.WizardCoder-15B’s Pass@10 decreases by up to 11.52% under extremely heterogeneous coding style; CodeQWen-7B’s Top@10 and Pass@10 remain around 90%.Source: §5.2 · pp. 28–29

5.3 RQ3: Impact of federated algorithms

  • FedAvg ranks first overall with 117 Borda points, ahead of FedSWA, FedOPT, FedProx, and pFedMe, despite rarely achieving the best result for individual LLMs.It is second-best on most metrics across LLMs, supporting its stability as a general-purpose aggregation algorithm.Source: §5.3 · pp. 30–31, Figure 9 · p. 31
  • FedSWA ranks second overall with 113 Borda points and leads among non-FedAvg algorithms on CodeLlama-7B and WizardCoder-15B, while FedOPT and FedProx excel on particular LLMs.FedOPT leads on CodeLlama-13B and CodeQWen-7B; FedProx achieves the highest Top@10 and Pass@10 of 66.26% and 65.37% on Mistral-7B.Source: §5.3 · p. 32
  • pFedMe ranks last with 50 Borda points and substantially underperforms alternatives, including on DeepSeekCoder-7B relative to FedAvg.On DeepSeekCoder-7B, Top@10 and Pass@10 decrease by 28.84% and 26.31% compared to FedAvg, indicating that personalization remains challenging for LLM fine-tuning.Source: §5.3 · p. 32

6 DISCUSSION

The discussion links instability in local and centralized fine-tuning to dataset-size concerns, while noting that the relevant boundary remains unclear. It also reports robustness across code features, model-specific federated-algorithm strengths, and weak personalization under adapter fine-tuning.

6 DISCUSSION

  • Local fine-tuning performed worst, while federated fine-tuning sometimes surpassed centralized fine-tuning; dataset-size effects may matter, but their effective boundary remains unclear.The authors note that both centralized training on the full dataset and local training on only a fraction may yield suboptimal performance.Source: §6 · pp. 32–33
  • Federated fine-tuning combines local privacy benefits with centralized data utilization and achieves stable, consistent performance across most scenarios.Source: §6 · p. 33
  • Different coding styles, code complexities, and code embeddings had little overall impact, suggesting companies with varied repositories can benefit across data distributions.Source: §6 · p. 33
  • FedSWA performed consistently well on CodeLlama-13B, CodeLlama-7B, and WizardCoder-15B, prompting further study of how LLM architecture affects algorithm performance.Source: §6 · p. 33
  • Personalization performed poorly for program-repair fine-tuning; unlike conventional federated personalization, this study fine-tuned adapters rather than the original model parameters.The authors call for personalization strategies tailored to LLM fine-tuning for program repair and other code-related tasks.Source: §6 · p. 33

7 THREATS TO VALIDITY

The validity of the findings is bounded by how code heterogeneity is represented and by evaluation choices involving Pass@k and test-suite strength. The study describes mitigations for these threats, but notes that broader code domains warrant further study.

7 THREATS TO VALIDITY

  • The single-company dataset and three code features may not capture heterogeneity in domains such as game development, AI, or scientific research.The authors recommend incorporating repositories from more diverse domains to further investigate heterogeneous collaboration.Source: §7 · p. 33
  • Larger Pass@k values require more generated solutions per problem and model, increasing inference cost.The authors selected k values of 5 and 10 from a developer perspective.Source: §7 · p. 34
  • Weak test suites can let incorrect patches pass, so evaluation used EvalRepair with expanded EvalPlus test cases to screen for potentially faulty patches.Manual verification across large problem sets is resource-intensive.Source: §7 · p. 34

8 CONCLUSION

The study evaluates federated fine-tuning for program repair across six code-related LLMs using private industrial training data and an augmented benchmark. It reports competitive repair performance, little impact from code heterogeneity, and different strengths across federated algorithms and LLMs.

8 CONCLUSION

  • Federated fine-tuning showed competitive bug-fixing capabilities and sometimes outperformed centralized fine-tuning.The evaluation used six code-related LLMs, 1,239 programming problems for fine-tuning, and an augmented benchmark with 163 problems and 583 test cases per problem.Source: §8 · p. 34
  • Data heterogeneity had minimal impact on fine-tuning performance across feature-skewed Non-IID distributions based on coding style, code complexity, and code embedding.The study concludes that enterprises with diverse code repositories can still benefit from collaborative model training without compromising performance.Source: §8 · p. 34
  • FedAvg showed stability and practicality, whereas pFedMe faced challenges adapting to LLMs and program repair.The authors call for personalization strategies tailored to LLM fine-tuning.Source: §8 · p. 35
  • The study identifies federated learning's potential for collaborative software development and maintenance while preserving data privacy.Source: §8 · p. 35

Related papers

Loading 2412.01072v1…