PaperlayerView exact revision on arXiv

Source-linked AI summary

FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, Jingren Zhou

arXiv:2309.00363v1cs.LG

TL;DR

Federated fine-tuning can adapt LLMs to private, domain-specific data, but existing FL frameworks provide inadequate support for its resource, data, and protection requirements. FS-LLM addresses this gap with benchmarking, parameter-efficient algorithms, flexible model-access interfaces, and resource-efficient training; experiments expose efficiency and federated-tuning challenges that motivate further research.

  • Problem

    Existing FL frameworks lack comprehensive LLM fine-tuning support for resource costs, benchmark preparation, and distinct information-protection settings.

  • Method

    FS-LLM provides an end-to-end benchmark pipeline, federated PEFT algorithms and interfaces, and accelerating and resource-efficient training operators.

  • Results

    Experiments benchmark federated LLM fine-tuning and identify substantial variation in transmission and computation time across PEFT algorithms and hardware settings.

  • Takeaways & Limitations

    FS-LLM supports federated LLM fine-tuning across varied FL scenarios and provides empirical insights into challenges and future research directions.

  • Takeaways & Limitations

    Federated fine-tuning remains challenged by client resource heterogeneity, including idle computation time and bandwidth-utilization constraints.

Abstract

from arXiv · show

LLMs have demonstrated great capabilities in various NLP tasks. Different entities can further improve the performance of those LLMs on their specific downstream tasks by fine-tuning LLMs. When several entities have similar interested tasks, but their data cannot be shared because of privacy concerns regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities. However, fine-tuning LLMs in federated learning settings still lacks adequate support from existing FL frameworks because it has to deal with optimizing the consumption of significant communication and computational resources, data preparation for different tasks, and distinct information protection demands. This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution, which consists of the following components: (1) we build an end-to-end benchmarking pipeline, automizing the processes of dataset preprocessing, federated fine-tuning execution, and performance evaluation on federated LLM fine-tuning; (2) we provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios with low communication and computation costs, even without accessing the full model; (3) we adopt several accelerating and resource-efficient operators for fine-tuning LLMs with limited resources and the flexible pluggable sub-routines for interdisciplinary study. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings, which also yields valuable insights into federated fine-tuning LLMs for the research community. To facilitate further research and adoption, we release FS-LLM at https://github.com/alibaba/FederatedScope/tree/llm.

1 INTRODUCTION

Federated fine-tuning of LLMs must reconcile domain adaptation and privacy with substantial resource demands and incomplete framework support. FS-LLM addresses these gaps through benchmarking, federated fine-tuning algorithms, and resource-efficient training components.

  • Domain-specific fine-tuning can improve LLM knowledge, task ability, fairness, and reliability, but domain data may be private and computation-intensive.
  • Existing FL packages lack comprehensive LLM fine-tuning implementations and standardized benchmarks covering performance, communication cost, and computation overhead.
  • FS-LLM combines diverse federated datasets and evaluation tasks into a benchmarking pipeline for comparing federated fine-tuning algorithms.
  • FS-LLM provides federated fine-tuning algorithms, flexible interfaces for full-model and restricted-access settings, and efficiency-oriented training support.
  • Its architecture comprises LLM-BENCHMARKS, LLM-ALGZOO, and LLM-TRAINER, with an example using PEFT, half-precision training, offloading, and pFedMe.

2 OVERVIEW

FS-LLM extends FederatedScope with three coordinated modules for federated LLM fine-tuning. These modules cover benchmarking, algorithm implementations, and configurable resource-efficient training.

  • FS-LLM builds on FederatedScope and adds LLM-BENCHMARKS, LLM-ALGZOO, and LLM-TRAINER to address federated LLM fine-tuning gaps.
  • LLM-BENCHMARKS: LLM-BENCHMARKS prepares domain datasets as federated versions, pairs them with evaluation tasks, and supports centralized-to-federated splitting.
  • LLM-ALGZOO: LLM-ALGZOO integrates PEFT algorithms such as LoRA, prefix-tuning, P-tuning, and prompt tuning for constrained FL clients.
  • LLM-TRAINER: LLM-TRAINER uses hook-like training schemes with accelerating and resource-efficient operators, including ZeRO, data parallelism, and quantization.

3 LLM-BENCHMARKS: A COMPLETE PIPELINE FOR BENCHMARKING

LLM-BENCHMARKS provides an end-to-end framework for evaluating federated LLM fine-tuning across heterogeneous datasets, downstream tasks, and efficiency costs. It combines curated federated data, distribution-shifted evaluation, reproducible environments, and cost metrics.

  • Dataset construction: Its datasets represent diverse domains and client heterogeneity, including code generation, question-response tasks, and chain-of-thought mathematics.
  • Dataset construction: Users can construct federated datasets from centralized data with uniform, Dirichlet, or metadata-based splitters and tunable heterogeneity.
  • LLM-BENCHMARKS addresses the absence of ready-to-use tools and consensus for evaluating federated LLM fine-tuning across accuracy and efficiency.
  • Evaluation: The module pairs federated fine-tuning datasets with corresponding evaluation tasks to assess domain-specific and generic LLM capabilities.
  • Evaluation: Evaluation includes distribution shifts between fine-tuning and evaluation data, containerized runtimes, and a smaller HELM-MINI option for brief evaluation.
  • Efficiency measurement: Cost metrics cover GPU usage, computation time, FLOPs, and message size alongside evaluation scores for a broader efficiency assessment.

4 LLM-ALGZOO: A COLLECTION OF FINE-TUNING ALGORITHMS

LLM-ALGZOO collects federated fine-tuning algorithms designed for constrained communication, computation, and model-access settings. Unified interfaces support parameter-efficient methods, privacy-preserving emulator training, and future algorithm extensions.

  • LLM-ALGZOO implements PEFT algorithms for full-model-access settings and an additional method for clients that cannot access the complete LLM.
  • PEFT algorithms: Full-parameter fine-tuning can require 28GB per-round message transfer and at least 112GB of GPU memory for LLaMA-7B.
  • PEFT algorithms: PEFT methods train limited-parameter adapters while freezing other parameters, reducing communication and computation demands for resource-limited clients.
  • Restricted model access: FedOT sends clients a lossy compressed emulator with untrainable parameters, enabling adapter fine-tuning without sharing the complete model.
  • Unified interfaces: The event-driven interface uses message-handler pairs so users can customize and extend federated fine-tuning algorithms.
  • Unified interfaces: Unified interfaces coordinate model preprocessing, broadcasting, parameter aggregation, and redistribution across accessible and inaccessible-model settings.

5 LLM-TRAINER: TRAINING OPERATORS AND PARADIGM

LLM-TRAINER equips FS-LLM with generalized, mode-specific, and parallelization operators to reduce resource demands and support extensible federated fine-tuning workflows. It also adapts federated personalization and hyperparameter-optimization methods for LLM settings.

  • Training operators: LLM-TRAINER provides accelerating and resource-efficient operators targeting CPU/GPU memory consumption, multi-GPU parallelism, and communication cost.The operators can be combined to improve compatibility and efficiency during federated fine-tuning.
  • Mode-generic operators: Mixed-precision training, gradient accumulation, and PyTorch data parallelism reduce GPU resource use and accelerate local fine-tuning.These generalized operators are implemented through hook-like functions and support multi-GPU parallelism.
  • Mode-specific operators: In simulated mode, round-robin switching lets clients take turns using a frozen full model, reducing memory consumption from multiple independently instantiated models.
  • Parallelization operators: DeepSpeed adds multi-GPU memory optimization and CPU offloading, while non-Rank-0 subprocesses disable logging, WandB, and file writing to avoid conflicts.In distributed and clustered modes, subprocesses communicate independently and synchronize during local fine-tuning.
  • Extensible training paradigm: LLM-TRAINER supports fine-grained hook-like extensions across dataset preparation, parameter updates, and validation evaluation.
  • FL extensions: FS-LLM adapts pFL implementations to avoid maintaining global and local models simultaneously when client resources are limited.It also extends federated LLM fine-tuning with model-free, model-based, multi-fidelity, and FedHPO methods.

6 EXPERIMENTS

The experiments evaluate federated LLM fine-tuning across effectiveness, efficiency, full-model access, personalized learning, and hyperparameter optimization. Results show benefits from collaborative PEFT, while exposing communication, computation, compression, compatibility, and evaluation challenges.

  • Effectiveness: The benchmark evaluates PEFT algorithms across code generation, generic language capability, and mathematical reasoning tasks using three federated fine-tuning datasets.The experiments use Fed-CodeAlpaca with HumanEval, Fed-Dolly with HELM, and Fed-GSM8K-3 with its corresponding evaluation task.
  • Effectiveness: Federated fine-tuning with PEFT algorithms significantly outperforms local fine-tuning and produces results competitive with global training.This enables collaborative training without directly sharing private data.
  • Effectiveness: LoRA outperforms P-tuning and prompt tuning by a large margin across local, federated, and global scenarios.LoRA augments each LLM layer with two low-rank matrices, whereas the prompt methods insert learnable tokens into the input.
  • Effectiveness: Federated fine-tuning improves OPT-2.7B over local training only modestly, while LLaMA-7B has an obvious advantage across evaluated scenarios.OPT-2.7B also fails some HELM subtasks because of exceeded input length, limiting its application scope.
  • Efficiency: PEFT algorithms have negligible effects on model GPU memory, but message sizes cause transmission times ranging from about 0.01 to 40 seconds per round.Computation time varies by GPU, with almost a twofold difference, creating client-side idle time and potentially reducing the benefit of asynchronous communication.
  • Open challenges: Key open challenges include exploiting heterogeneous-client idle time, maintaining performance under compression, resolving efficient-operator and pFL compatibility, and avoiding exhaustive FedHPO searches.These directions follow from observed communication-computation trade-offs, access conflicts, and non-smooth hyperparameter behavior.
  • Fine-tuning without full-model access: FedOT benefits privacy-preserving fine-tuning without full-model access, but its larger adapter sacrifices communication efficiency for performance.At a 50% dropping rate, performance degrades sharply as the model loses pretraining knowledge and struggles with chain-of-thought and code generation.
  • Personalized FL and FedHPO: Efficient training operators can impair personalized FL, while hyperparameter evaluation scores are highly sensitive and validation loss may not reflect generalization.Half precision causes more precision loss for pFedMe than FedAvg, and acceleration operators restrict the valid hyperparameter space.

7 DISCUSSIONS AND FUTURE DIRECTIONS

The paper reports limitations and outlines research directions for making federated LLM fine-tuning more efficient, private, compatible, and applicable across deployment settings.

  • Limitations: The experiments are limited by batch size 1 and fixed prompts, although larger batches and alternative prompts may affect evaluation results.The authors use a fixed prompt for fair comparison and note that larger batch sizes might perform better.
  • Future directions: Even with PEFT, federated LLM fine-tuning remains too computationally costly for many resource-limited clients.Reducing computation could lower participation barriers for data holders.
  • Future directions: FedOT exposes a trade-off between model compression and performance when clients cannot access the full model.Higher compression improves privacy but degrades performance, motivating methods that preserve both generalization and model privacy.
  • Future directions: Compatibility between efficient training operators and personalized federated learning algorithms remains unclear, especially under heterogeneous data and constrained resources.Improving this compatibility could support personalized models in such settings.
  • Future directions: Validation loss and LLM generalization performance are inconsistent, complicating low-cost hyperparameter optimization for federated fine-tuning.The authors identify low-fidelity FedHPO as a direction for finding hyperparameters associated with better generalization.
  • Future directions: Cross-device federated LLM fine-tuning remains an urgent problem because clients are more numerous and heterogeneous, with more limited resources and diverse network conditions.The paper identifies extending beyond cross-silo settings as a research priority.

8 CONCLUSIONS

The paper introduces FS-LLM to address gaps between universal federated learning frameworks and federated LLM fine-tuning, validates its utility experimentally, and identifies future research directions.

  • Conclusion: FS-LLM provides an open-source, extensible package supporting federated LLM fine-tuning under various federated learning scenarios.The package is released to promote further research.
  • Conclusion: Its experiments demonstrate the package’s utility and generate insights into federated LLM fine-tuning and promising future research problems.The release is intended to advance work across the federated learning and LLM communities.

A.1 FINE-TUNING DATASET DESCRIPTION

FS-LLM curates fine-tuning datasets from diverse domains with varied client heterogeneity, summarized through dataset statistics and information in Table 6.

  • Dataset scope: The curated datasets cover code, natural language, dialogues, and mathematics.They are derived from existing and widely used fine-tuning datasets.
  • Dataset scope: The datasets exhibit different degrees of heterogeneity across clients, creating varied challenges and opportunities for federated LLM fine-tuning.Table 6 summarizes the datasets’ statistics and information.

A.1.1 FEDERATED FINE-TUNING DATASET

FS-LLM provides federated versions of code, NLP, and mathematical datasets by partitioning source corpora according to language, task category, or uniform splits.

  • Dataset construction: Users can directly use curated federated fine-tuning datasets constructed from source-corpus metadata or distributions.The datasets are prepared specifically for federated LLM fine-tuning.
  • NLP datasets: Fed-Dolly assigns each client data from one of eight NLP task categories, including classification, question answering, and summarization.The source corpus is Databricks-dolly-15k.
  • Mathematics dataset: Fed-GSM8K-3 uniformly splits the GSM8K training problems into three subsets and assigns each subset to one client.GSM8K contains 7.5K training problems and 1K test problems.
  • Code datasets: Fed-CodeSearchNet offers an alternative code dataset by splitting functions and documentation into six language-based client subsets.It covers Go, Java, JavaScript, PHP, Python, and Ruby using data from open-source GitHub projects.

A.1.2 CENTRALIZED FINE-TUNING DATASET

The paper assembles centralized corpora and evaluation tasks for federated LLM fine-tuning, then uses them to compare parameter-efficient methods and assess language, code, and reasoning capabilities. It also introduces HELM-MINI as a reduced-cost alternative to the full HELM evaluation.

  • Datasets: Centralized corpora can be partitioned with configurable splitters according to heterogeneity, balance, or other criteria for federated fine-tuning.The datasets are collected without prior partitioning, allowing users to customize federated versions.
  • Benchmark design: FS-LLM pairs fine-tuning corpora with domain-relevant evaluation tasks and provides a pipeline for benchmarking federated fine-tuning algorithms.The benchmark covers multiple domains and supports comparisons across federated learning scenarios.
  • Evaluation tasks: HumanEval measures code-generation correctness with Pass@k, while GSM8K-test measures mathematical problem-solving accuracy under 8-shot-CoT prompting.The reported GSM8K-test evaluation uses 1k test problems, and the practical HumanEval setting uses m = 5 with Pass@1.
  • Evaluation tasks: HELM evaluates generic language capability across 16 subtasks, whereas HELM-MINI retains four subtasks to reduce evaluation time while approximating full-HELM scores.The four HELM-MINI subtasks are MMLU, NaturalQA (open-book), OpenbookQA, and IMDB.
  • Results: The experiments compare LoRA, P-tuning, and prompt tuning across federated datasets and report improved question-answering accuracy and fluency after federated fine-tuning.The comparison includes Fed-CodeAlpaca, Fed-Dolly, and Fed-GSM8K-3, with separate evaluation tables for each method and dataset.
Loading 2309.00363v1…