PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, Siheng Chen

arXiv:2402.06954v1cs.LGcs.CLcs.DCcs.MA

TL;DR

High-quality public data may soon be exhausted, motivating collaborative training on decentralized private data without direct data sharing. The paper presents OpenFedLLM, a framework for federated instruction tuning and value alignment across diverse algorithms and datasets. Experiments report that FL algorithms outperform local training, with financial-domain models surpassing GPT-4 in a benchmark.

  • Problem

    High-quality public data may be exhausted before 2026, while valuable private data remains distributed and cannot be publicly shared.

  • Method

    The paper builds OpenFedLLM for privacy-preserving federated instruction tuning and value alignment on decentralized private data.

  • Results

    FL algorithms consistently outperform individual training across settings, with financial-domain FL models outperforming GPT-4 by an evident gap.

  • Takeaways & Limitations

    The findings provide motivation for organizations, especially those with limited data, to participate in FL for training better LLMs.

  • Takeaways & Limitations

    FedLLM still faces unresolved challenges including heterogeneous preferences, stealthy harmful data, and decentralized private-data management.

Abstract

from arXiv · show

Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields. While more data contributes to better performance, a disconcerting reality is that high-quality public data will be exhausted in a few years. In this paper, we offer a potential next step for contemporary LLMs: collaborative and privacy-preserving LLM training on the underutilized distributed private data via federated learning (FL), where multiple data owners collaboratively train a shared model without transmitting raw data. To achieve this, we build a concise, integrated, and research-friendly framework/codebase, named OpenFedLLM. It covers federated instruction tuning for enhancing instruction-following capability, federated value alignment for aligning with human values, and 7 representative FL algorithms. Besides, OpenFedLLM supports training on diverse domains, where we cover 8 training datasets; and provides comprehensive evaluations, where we cover 30+ evaluation metrics. Through extensive experiments, we observe that all FL algorithms outperform local training on training LLMs, demonstrating a clear performance improvement across a variety of settings. Notably, in a financial benchmark, Llama2-7B fine-tuned by applying any FL algorithm can outperform GPT-4 by a significant margin while the model obtained through individual training cannot, demonstrating strong motivation for clients to participate in FL. The code is available at https://github.com/rui-ye/OpenFedLLM.

1 Introduction

OpenFedLLM investigates privacy-preserving collaborative fine-tuning of LLMs on decentralized private data through federated learning. It integrates instruction tuning, value alignment, diverse FL methods, and broad empirical evaluation, finding consistent benefits over individual training.

  • Motivation: High-quality public data may be exhausted before 2026, while valuable private data remains distributed and underutilized because of privacy or physical constraints.This motivates collaborative training without direct data sharing.
  • Framework: FedIT trains LLMs on instruction-response pairs with supervised fine-tuning, whereas FedVA uses preference pairs with direct preference optimization to align outputs with human values.The two stages target instruction following and value alignment, respectively.
  • Future directions: Future work is needed for heterogeneous preferences, logically correct yet harmful attackers, decentralized private-data management, and FL algorithms tailored to LLM training.The paper notes that no FL algorithm guarantees the best performance in every scenario.
  • Approach: OpenFedLLM explores the complete pipeline for fine-tuning contemporary LLMs on decentralized private data via federated learning.FL coordinates global model downloading, local training, local uploading, and global updating without transmitting raw data.
  • Framework: OpenFedLLM integrates federated instruction tuning, federated value alignment, 7 FL baselines, 8 training datasets, and 30+ evaluation metrics.The framework also decouples FL and LLM training and supports execution on one consumer GPU through efficiency techniques.
  • Empirical findings: All tested FL algorithms outperform individual training across settings, including ≥12% improvement on MT-Bench on a general dataset.The study also reports that FL on a financial dataset can outperform GPT-4 with an evident gap.

2 Related Work

Prior work established federated learning for collaborative training but provided limited evidence for contemporary LLM instruction tuning, value alignment, and diverse FL algorithms. OpenFedLLM addresses this gap through broader coverage of training stages, algorithms, datasets, and evaluations.

  • Large Language Models: Contemporary LLMs commonly use pre-training, instruction tuning, and value alignment, with these stages mostly conducted on public or AI-generated data.High-quality public data has been estimated to exhaust before 2026.
  • Large Language Models: Each party may have limited data, while the union of many parties’ private data could form a large database for training a powerful LLM.The paper therefore focuses on collaborative training with parameter-efficient fine-tuning rather than pre-training.
  • Federated Learning: Federated learning lets multiple clients train a shared global model without transmitting raw data, using broadcasting, local training, uploading, and aggregation steps.These steps support privacy-preserving collaboration under a central server.
  • Federated Learning: FedAvg may achieve only moderate performance under data heterogeneity, motivating FL algorithms that improve local-model consistency or aggregated-model performance.Examples discussed include FedProx and SCAFFOLD.
  • Federated Learning and Large Language Models: Earlier FL-LLM frameworks often had limited datasets, omitted value alignment, or supported only FedAvg, leaving their LLM-training performance insufficiently explored.The cited prior studies differed in empirical breadth and covered fewer contemporary LLM training stages.
  • Federated Learning and Large Language Models: OpenFedLLM broadens prior work by covering instruction tuning, value alignment, 7 representative FL algorithms, 8 training datasets, and over 30 evaluation metrics.Table 1 frames these as comparison dimensions among OpenFedLLM and other FL frameworks.

3 OpenFedLLM Framework

OpenFedLLM organizes federated LLM training into a conventional FL process with federated instruction tuning and value alignment. It uses parameter-efficient updates so clients communicate only a small set of learnable parameters.

  • Framework overview: OpenFedLLM combines federated instruction tuning, federated value alignment, representative FL algorithms, diverse datasets, and extensive evaluation support.The framework is designed to let users focus on either FL or LLM training without extensive background in the other field.
  • Framework overview: Each communication round broadcasts the global model, performs local client updates, uploads local models, and aggregates them at the server.This process follows conventional FL protocols and remains compatible with secure aggregation and differential privacy.
  • Federated instruction tuning: Federated instruction tuning trains clients on instruction-response pairs, applying supervised loss to the response so the global model learns to follow diverse instructions.The approach addresses the difficulty of obtaining enough high-quality and complex instruction-tuning samples on a single client.
  • Federated value alignment: Federated value alignment uses instruction, preferred-response, and dispreferred-response triples to favor preferred outputs and discourage dispreferred ones.DPO avoids the reward-model stage required by RLHF and controls deviation from a fixed reference policy through a hyperparameter.
  • Parameter-efficient fine-tuning: LoRA freezes the base LLM and represents weight updates as low-rank matrices, reducing learnable parameters to less than 1% of the base model.Only the small set of trainable parameters is updated and communicated, improving computational and communication efficiency without additional inference latency.

4 Experiments

This section defines the experimental scope of OpenFedLLM, covering representative federated-learning baselines, diverse datasets, parameter-efficient training, and broad evaluation. General-domain results show that federated methods outperform local training, with FedYogi and SCAFFOLD performing especially well.

  • Basic Setups: The experiments cover general, code, math, finance, and medical domains, plus instruction tuning and value alignment scenarios.The adopted datasets represent diverse domains and applied settings, with client partitions constructed from same-source or heterogeneous datasets.
  • Basic Setups: LoRA represents each weight update as W+∆W = W+AB, using low-rank matrices where r ≪ min(d, m).The base model remains largely frozen while only low-rank adaptation parameters are trained and communicated.
  • Basic Setups: Only 0.06% of total model parameters are trainable and communicated per round, while the base model parameters remain frozen.This setup targets computational and communication efficiency in federated LLM training.
  • General-Domain Evaluation: General-domain experiments evaluate Alpaca-GPT4-trained models on knowledge, reasoning, coding, counterfactual, and open-ended instruction-following benchmarks.The evaluation includes MMLU, BBH, DROP, HumanEval, CRASS, Vicuna-Bench, and MT-Bench.
  • General-Domain Results: FL methods consistently outperform local training on open-ended benchmarks and significantly outperform it on most close-ended benchmarks.On BBH, all methods perform comparably; FedYogi and SCAFFOLD perform better overall in the general domain.

4.3 Federated Instruction Tuning on Financial Dataset

The paper evaluates federated instruction tuning across financial, medical, and code settings, using domain-specific datasets and multiple baselines. Across these settings, federated methods generally outperform local training, while the strongest algorithm varies by domain and metric.

  • 4.3 Federated Instruction Tuning on Financial Dataset: Financial experiments use 50 clients, sample 5 clients per round, and train on 10k financial sentiment samples.Evaluation uses FPB, FIQA-SA, TFNS, and NWGI, measuring accuracy and F1 score.
  • 4.3 Federated Instruction Tuning on Financial Dataset: 11.5% relative: FedAvg outperforms local training on average across four financial evaluation datasets.The reported ordering is FL methods > GPT-4 > GPT-3.5 > local training.
  • 4.4 Federated Instruction Tuning on Medical Datasets: FedAvg consistently outperforms local training in the medical domain, while no FL algorithm is best on every metric.FedAdam achieves the best result on 4 of 9 metrics and is relatively better for this scenario.
  • 4.5 Federated Instruction Tuning on Code Datasets: Code experiments compare 8 baselines across 11 metrics covering seven benchmarks and Python, Java, and JavaScript.FedAvg performs better or comparably to local training, while FedAdagrad leads on 6 of 11 metrics.

4.6 Federated Instruction Tuning on Diverse Domains

The paper studies federated instruction tuning across heterogeneous domains and federated value alignment for helpfulness and harmlessness. Collaboration usually improves over local training, but domain expertise and alignment objectives create meaningful trade-offs.

  • 4.6 Federated Instruction Tuning on Diverse Domains: The heterogeneous-domain experiment assigns separate general, math, code, and finance datasets to four clients.Each client holds 5k samples, and FedAvg is compared with individual client training using domain-specific benchmarks.
  • 4.6 Federated Instruction Tuning on Diverse Domains: FedAvg achieves the highest overall rank across general, math, code, and finance evaluations, indicating strong cross-domain collaboration.The comparison includes MT-Bench, GSM8K, HumanEval, and FPB.
  • 4.6 Federated Instruction Tuning on Diverse Domains: 0.805 F1: FedAvg on finance, versus 0.834 for the client trained entirely on financial data.The result shows that federated collaboration may underperform a domain expert on a specific domain.
  • 4.7 Federated Value Alignment for Helpfulness: UltraFeedback value-alignment experiments use preferred and dispreferred responses across five clients and five evaluation metrics.The setup compares three FL algorithms with local training and the instruction-tuned base model.
  • 4.7 Federated Value Alignment for Helpfulness: All FL algorithms outperform local training across the five helpfulness metrics, with FedAvg best on the four open-ended benchmarks.All methods improve overall performance relative to the base model, except local training on MMLU.
  • 4.8 Federated Value Alignment for Harmlessness: HH-RLHF value-alignment experiments evaluate harmlessness and helpfulness using 161k samples distributed across five clients.Metrics include HHH harmlessness, AdvBench rejection rate, and MT-Bench helpfulness.
  • 4.8 Federated Value Alignment for Harmlessness: FedAvg and FedAvgM consistently outperform local training across five metrics, while FedAvgM performs best overall for harmlessness and helpfulness.FedProx exceeds local training on helpfulness but not harmlessness, indicating differing alignment outcomes.

5 Discussions and Future Directions

The discussion identifies data management, personalization, robustness, privacy, efficiency, and deployment setting as unresolved challenges for federated LLM training. It also outlines practical constraints and future directions, including heterogeneous client values, stealthy attacks, and cross-device resource limits.

  • 5.1 Data Management in FedLLM: Federated data management lacks a comprehensive dataset overview, complicating global thresholding or ranking while client data quality varies substantially.Each client holds only a fraction of the data, requiring more individualized selection criteria than centralized systems.
  • 5.2 Heterogeneous Preference in FedVA: FedVA must reconcile heterogeneous cultural, ethical, and contextual preferences; clustering clients with similar values is proposed as one possible approach.Shared models must integrate varying values while preserving commonly shared preferences such as helpfulness.
  • 5.3 Personalized Federated Learning for LLMs: Conventional FL may underperform local training in a client’s expert domain, motivating personalized models that balance collaboration with individual tasks.The paper distinguishes personalization for domain-specific tasks from personalization for client-specific values.
  • 5.4 Robustness and Security in FedLLM: Existing robustness methods may not transfer reliably to FedLLM, where only a small parameter subset is fine-tuned and tasks involve language rather than image classification.The paper also highlights logically correct but harmful data as a stealthy attacker vector that can compromise model responsibility.
  • 5.5 Privacy Preservation in FedLLM: LLM memorization creates a privacy–effectiveness trade-off, with differential privacy, reduced private-data exposure, and non-private data proposed as possible mitigations.Differential privacy adds controlled noise to gradients or updates, while limiting private data can increase under-fitting.
  • 5.6 Efficiency in FedLLM: FedLLM efficiency is constrained by model scale and multi-round communication, although 8-bit quantization and LoRA make training executable on one consumer GPU.More efficient techniques remain necessary as model sizes grow; QLoRA is cited as a potential direction.
  • 5.7 Applicability to Cross-Silo and Cross-Device FL: Cross-silo FedLLM is feasible with substantial hardware, whereas cross-device training faces limited memory, processing power, and unreliable device availability.Knowledge distillation, pruning, and parameter-efficient training are suggested for smaller-device deployments.

6 Conclusion

The paper establishes a pipeline and framework for training LLMs on distributed private data through federated learning. Its experiments show benefits over local training, including a financial setting where federated Llama2-7B outperformed GPT-4.

  • 6 Conclusion: OpenFedLLM provides an integrated pipeline covering federated instruction tuning, federated value alignment, 7 FL baselines, 8 datasets, and 30+ evaluation metrics.The framework is designed for comprehensive empirical exploration of federated LLM training.
  • 6 Conclusion: Federated learning improves performance over individual local training, including a financial experiment where Llama2-7B significantly outperformed GPT-4.The reported result motivates further study of collaborative training on distributed private data.

A.1 Summary of Hyper-Parameters

The appendix summarizes experiment hyper-parameters by domain and explains the notation used for client sampling and LoRA rank.

  • A.1 Summary of Hyper-Parameters: Table 10 lists dataset names, total clients, clients sampled per round, LoRA rank, and selected FL-algorithm hyper-parameters.In the Client column, x / y denotes sampling x clients from y total clients per round.

A.2 Prompt Template

The appendix provides the prompt templates used for federated instruction tuning and federated value alignment.

  • A.2 Prompt Template: Federated instruction tuning follows the Alpaca template, while federated value alignment follows the Vicuna template to better support chatting.The two stages therefore use distinct prompt formats aligned with their training objectives.

A.3 Case Study

The case studies compare locally trained models with federated models on reasoning, medical, financial, and harmful questions. Across these examples, federated models answer several domain questions correctly and respond more responsibly to harmful requests.

  • Instruction tuning: Federated models trained with FedAvg and SCAFFOLD correctly answer an MT-Bench reasoning question that local training gets wrong.The ground-truth answer is $12000.
  • Value alignment: FedAvg recognizes a harmful HH-RLHF question and rejects a detailed response, while local training provides detailed harmful content.This case study compares local training with FedAvg on the same harmful question from AdvBench.
  • Instruction tuning: FedAvg and SCAFFOLD correctly answer a PubMedQA medical question, whereas local training fails.The ground-truth answer is yes.
  • Instruction tuning: FedAvg and SCAFFOLD correctly answer an FPB financial question, whereas local training fails.The ground-truth answer is positive.
Loading 2402.06954v1…