Source-linked AI summary
FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, Jing Gao
TL;DR
Private, distributed domain data make centralized LLM fine-tuning unsuitable, while federated clients may lack the computation, communication capacity, or model access needed for full-model training. FedBiOT compresses the LLM into an emulator and adapter, uses bi-level optimization to account for mismatched server and client data, and lets clients fine-tune only the adapter. On LLaMA-2, it improves accuracy over Offsite-tuning and FedOT across math, code-generation, and question-answering tasks while reducing resource consumption, although its communication can exceed those baselines and remains far below full-LLM transmission.
Problem
Private domain data are distributed across owners, while federated clients face limited computation, communication capacity, and access to full LLMs for collaborative fine-tuning.
Method
FedBiOT compresses the LLM into an emulator and adapter and formulates bi-level optimization so clients fine-tune the adapter while the server aligns the emulator despite different data distributions.
Results
FedBiOT improves accuracy over Offsite-tuning and FedOT across math problem-solving, code generation, and question answering while reducing computation overhead.
Takeaways & Limitations
Clients can collaboratively fine-tune a compressed LLM without full-model fine-tuning while keeping their private data local.
Takeaways & Limitations
FedBiOT may require more communication than baselines because it transmits LoRA parameters for both the adapter and emulator, though this remains trivial relative to transmitting the 28 GB full LLM.
Abstract
from arXiv · showhide
Large language models (LLMs) show amazing performance on many domain-specific tasks after fine-tuning with some appropriate data. However, many domain-specific data are privately distributed across multiple owners. Thus, this dilemma raises the interest in how to perform LLM fine-tuning in federated learning (FL). However, confronted with limited computation and communication capacities, FL clients struggle to fine-tune an LLM effectively. To this end, we introduce FedBiOT, a resource-efficient LLM fine-tuning approach to FL. Specifically, our method involves the server generating a compressed LLM and aligning its performance with the full model. Subsequently, the clients fine-tune a lightweight yet important part of the compressed model, referred to as an adapter. Notice that as the server has no access to the private data owned by the clients, the data used for alignment by the server has a different distribution from the one used for fine-tuning by clients. We formulate the problem into a bi-level optimization problem to minimize the negative effect of data discrepancy and derive the updating rules for the server and clients. We conduct extensive experiments on LLaMA-2, empirically showing that the adapter has exceptional performance when reintegrated into the global LLM. The results also indicate that the proposed FedBiOT significantly reduces resource consumption compared to existing benchmarks, all while achieving comparable performance levels.
1 Introduction
Domain-specific LLM fine-tuning requires private data distributed across owners, while conventional federated approaches face full-model access and resource constraints. FedBiOT addresses these challenges by compressing the model, separating emulator and adapter components, and optimizing them through a bi-level procedure.
- 1 Introduction: Federated learning keeps each owner’s task-specific data local while allowing a server to aggregate local fine-tuning updates.This setting addresses privacy concerns that arise when users upload data to centralized fine-tuning services.
- 1 Introduction: The target setting includes LLMs whose full architecture and parameters are unavailable to clients, making collaborative fine-tuning difficult under standard federated procedures.Closed-source models may provide stronger performance, but black-box fine-tuning can conflict with users’ privacy requirements.
- 1 Introduction: Clients fine-tune only an adapter in a compressed model, reducing computation and communication costs compared with loading and updating the complete model.The adapter affects only the last few layers, while the compressed model has fewer layers.
- 1 Introduction: FedBiOT improves accuracy over existing baselines across math problem-solving, code generation, and question answering while avoiding full-model fine-tuning and reducing communication and computation overhead.The experiments use LLaMA-2 and compare against Offsite-tuning and FedOT.
- 1 Introduction: The method partitions the compressed model into an emulator that simulates the original LLM and an adapter that learns domain-specific linguistic patterns from client data.The server distills the emulator while clients update the adapter, with the two processes coupled through bi-level optimization.
2 Preliminary
Federated learning formulates a weighted optimization problem over clients’ local datasets and repeatedly exchanges model updates with a server. Existing PEFT and offsite-tuning approaches reduce trainable parameters or expose only compressed layers, but clients may still struggle to load large LLMs and model privacy remains a concern.
- 2.1 Traditional FL Formulation: Federated learning minimizes a weighted aggregate of client losses by distributing model parameters, performing local updates, and aggregating the resulting models.Client weights are proportional to local dataset sizes.
- 2.1 Traditional FL Formulation: A client loss evaluates the model output on local input examples against ground-truth next-token targets.The ground truth is included in the input sequence to identify the tokens that require prediction.
- 2.1 Traditional FL Formulation: FedPEFT reduces trainable parameters by adding a PEFT module while keeping the original LLM parameters unchanged.This reduces update size but does not remove the need for clients to load the LLM.
- 2.1 Traditional FL Formulation: Loading a full-precision LLaMA-2-7B requires at least 28 GB of memory, limiting the feasibility of client-side LLM fine-tuning.The computational burden persists even when only a small parameter-efficient module is trained.
- 2.2 Related Work: Offsite-tuning and FedOT protect model privacy by exposing compressed layers, but they preserve many layers and do not account for differences between server alignment data and client fine-tuning data.FedBiOT addresses the latter discrepancy through bi-level optimization and exposes fewer original-model parameters.
3 FedBiOT
FedBiOT enables clients with limited resources to fine-tune an LLM by separating a compressed model into an emulator and an adapter. Its bi-level objectives align the emulator with the full model while clients locally tune the adapter, which can later be reintegrated into the full model.
- Compressed model components: FedBiOT separates a compressed model into an emulator that replicates the uncompressed LLM and an adapter that learns domain-specific patterns from clients.The adapter is reintegrated with either the compressed or non-compressed emulator after training.
- Resource-efficient client updates: Clients fine-tune only the adapter, reducing computation and communication costs compared with directly fine-tuning a full layer.Direct adapter fine-tuning can involve millions of parameters; LoRA reduces a LLaMA-2 adapter to 0.13 million trainable parameters, or 0.06% of the original layer.
- Compressed model preparation: The compressed model is constructed by selecting bottom layers as the adapter and applying uniform layer dropout to the remaining layers to form the emulator.The adapter contains the trainable parameters, while the emulator is a submodel of the remaining non-compressed component.
- Pre-alignment: Before federated training, the server pre-aligns the emulator with the non-compressed model using a public dataset and knowledge distillation.The alignment targets both intermediate representations and final output distributions.
- Bi-level optimization: The bi-level optimization improves the emulator-adapter combination on local data while constraining adapter changes to support emulator-distillation convergence.The lower-level objective combines an activation-distance term with a KL-divergence term, and the adapter regularizer keeps local updates near the synchronized adapter.
- Framework scope: FedBiOT is a general emulator-adapter framework, with uniform layer dropout used as one practical emulator design rather than a required choice.The framework permits other emulator constructions that simulate the non-compressed part of the LLM.
4 Experiments
Experiments evaluate FedBiOT on math problem-solving, code generation, and question-answering with LLaMA-2, comparing adapter settings, baselines, dropout rates, and resource costs. Across these evaluations, FedBiOT generally improves performance while reducing computation overhead, although it can require more communication than baselines.
- Math problem-solving and code generation: FedBiOT with a four-layer adapter performs better than the two-layer adapter under AdapFu for i.i.d. data, whereas the two-layer adapter performs better for non-i.i.d. code-generation data.The non-i.i.d. result is attributed to clients’ locally biased objectives and the importance of the emulator.
- Code generation: Up to 6% Pass@1 and 10% Pass@10 improvements over other baselines are achieved across programming languages at a dropout rate of 0.5.The strongest dominance is reported for Java in Table 4.
- Question Answering: FedBiOT outperforms offsite-tuning and FedOT in most datasets, with the advantage becoming more pronounced as dropout increases.At a 0.5 dropout rate, FedBiOT is approximately 10% better than baselines on Natural Questions (closed-book), compared with a 2% improvement at 0.2.
- Computation and communication overhead: FedBiOT costs less computation than offsite-tuning and FedOT because its adapter is near the output layer, shortening backward-propagation paths.The computation metric is measured in per-token FLOP/token.
- Computation and communication overhead: FedBiOT may require more communication than baselines because clients receive LoRA parameters for both the adapter and emulator, though this remains small relative to transmitting the full 28GB LLM.Baselines transmit only aggregated adapter LoRA parameters.
5 Conclusion
FedBiOT avoids full-model fine-tuning by compressing the LLM into an emulator and adapter trained through bi-level optimization. Experiments with LLaMA-2 show improved accuracy over Offsite-tuning and FedOT across math, code-generation, and question-answering tasks.
- FedBiOT achieves significant accuracy improvements over Offsite-tuning and FedOT across math problem-solving, code generation, and question answering.
- The method compresses the LLM into an emulator and adapter, with the emulator partially simulating the original model and the adapter learning domain-specific patterns.
- Bi-level optimization coordinates emulator simulation of the original LLM with adapter-based domain adaptation while avoiding full-model fine-tuning.
A Testing Dataset and Evaluation
The evaluation uses GSM-8K for mathematical reasoning, HumanEvalX for code generation, and HELM for question answering. These benchmarks assess correctness through task-specific answer, unit-test, and benchmark metrics.
- The study uses three datasets to assess fine-tuning performance and describes how each evaluates a given LLM.
- GSM-8K evaluates mathematical problem solving by checking whether generated answers match ground-truth solutions.
- HumanEvalX evaluates code autofill across C++, Go, Java, and Python using compilation and unit tests, with multiple generated code versions per prompt.
- HELM evaluates question answering across eight datasets using exact match, quasi-exact match, and F1 metrics.