Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent

Motivation and contributions

• LLM deployment and training on edge devices are hindered by high storage, processing, memory, and computational requirements, while FL communication remains a primary bottleneck.Source: §1 · pp. 1–2

FedCyBGD

• FedCyBGD cyclically trains responsible blocks on edge devices, periodically synchronizes their updates, and uses a scaled-down surrogate for unresponsible blocks.The surrogate model supplies the frozen unresponsible blocks without requiring additional updates or uploads.Source: §1 · p. 2, Figure 2 · p. 2

Empirical evaluation

• Experiments across public LLMs, datasets, and NLP tasks evaluate communication, memory, and computation costs while reporting competitive performance.Source: §1 · p. 2

Motivation and contributions

• FedCyBGD reduces communication, computation, and memory costs while enabling full-parameter LLM tuning in federated learning.The framework cyclically updates LLM blocks on clients and is presented as the first approach enabling this on resource-limited edge devices.Source: §1 · pp. 1–3

Compression and efficiency

• Its compression scheme lets clients download a parameter-scaled proxy model while preserving updates to the responsible block, reducing download communication cost.Source: §1 · pp. 2–3

Empirical evaluation

• The reported evaluation reduces resource usage to 13.8% and improves training performance by up to 58.3% while preserving model and data privacy.Source: §1 · p. 3

Parameter-efficient fine-tuning

• PEFT methods reduce memory by restricting parameter updates to a much lower-dimensional subspace, but this could limit downstream performance.Examples include LoRA, adapters, prompt tuning, and prefix tuning.Source: §2 · p. 3
• Federated PEFT methods often sacrifice performance to reduce communication and computation overhead, leaving full-parameter tuning without an acceptable resource cost.Source: §2 · p. 3

Compression for FL LLM communication

• Existing FL compression methods include pruning, quantization, layer dropping, knowledge distillation, gradient compression, and low-rank adapter approximations.Source: §2 · p. 3
• Knowledge-distillation approaches often rely on a scenario-constrained server auxiliary dataset, whereas FedCyBGD avoids downloading the entire pretrained model by sending a smaller model containing the necessary block.Source: §2 · p. 3

FedCyBGD training procedure

• FedCyBGD assigns clients responsible blocks, sends a compressed global model, and updates blocks through the training procedure described in Algorithm 1.Source: §2 · p. 4
• Frozen blocks are further reduced through model pruning and layer dropping to lower download cost while supporting updates to the responsible block.Source: §2 · p. 4

3.1 Observation

• Cyclical block updates outperform traditional multi-client aggregation and full-parameter federated tuning while using fewer computational resources.The paper attributes this potential advantage to reduced aggregation conflicts and more granular updates that may alleviate catastrophic forgetting.Source: §3.1 · p. 4

3.2 Problem Definition

• The FL objective is to update all LLM parameters collaboratively without exchanging client data, but full-parameter training imposes prohibitive communication, computation, and memory costs.Training a single responsible block for each client is presented as a more acceptable approximation for resource-limited devices.Source: §3.2 · p. 4

3.3 FedCyBGD

• FedCyBGD alternates clients and blocks: one client updates its responsible block while other blocks remain fixed at their latest values.This cycle update adapts block coordinate descent to federated learning, where clients take turns participating in training.Source: §3.3 · p. 5
• Because each client trains and uploads only one block, FedCyBGD reduces memory, computation, and communication costs during federated tuning.The approach is designed to enable full-parameter tuning without requiring every client to update the whole model at once.Source: §3.3 · p. 5
• To avoid full-model downloads, FedCyBGD compresses frozen components while preserving responsible blocks and uses them as approximate gradient-direction emulators.Hybrid pruning applies low-granularity pruning to previously updated blocks and layer dropping to blocks not yet updated.Source: §3.3 · pp. 5–6
• FedCyBGD is designed as an alternative to PEFT: it performs full parameter updates through cycle block optimization while remaining compatible with PEFT inside updated blocks.Unlike PEFT’s low-rank approximation, the framework focuses on updating the full model across cycles.Source: §3.3 · p. 6
• FedCyBGD reduces LLaMA-2 7B fine-tuning costs by 25% for client backward computation, to about 17.3 GB memory, and to 3.5 GB communication.These figures correspond to approximately 30% of traditional full-tuning memory and roughly 75% lower communication cost.Source: §3.3 · p. 6

4.1 Experimental Setup

• The experiments use five LLMs across GLUE and four additional tasks in federated settings with 32 or 64 resource-limited devices.Clients update in a randomly chosen cyclic order during each cycle, and baselines include full tuning, PEFT, and optimization methods.Source: §4.1 · pp. 6–7

4.2 Evaluation of FedCyBGD

• FedCyBGD reduces memory consumption by approximately 72.7% versus existing full-parameter tuning and communication costs by about 44.1% versus the best-performing PEFT method.These reductions are reported for the evaluation of full-parameter tuning in federated learning.Source: §4.2 · p. 7
• FedCyBGD cuts LLaMA2 fine-tuning backward time nearly in half compared with Fed-full and FedLora, while forward times remain similar to Fed-full.The timing comparison separates forward, backward, and update phases per epoch.Source: §4.2 · p. 7, Table 2 · p. 7
• Reduced to 13.8% resource usage while retaining competitive full-parameter performance, with up to 58.3% higher question-answering performance than FedAdaptor.The evaluation spans multiple LLMs and datasets, including Alpaca-GPT4, OASST1, Bloom-1.3B, and SQuAD.Source: §4.2 · p. 7, Table 3 · p. 8, Table 4 · p. 8
• On LLaMA2 and ChatGLM tasks, FedCyBGD consistently outperforms baselines, while remaining competitive with full tuning on Bloom-1.3B.The reported comparisons cover Alpaca-GPT4 and OASST1, with additional question-answering results on SQuAD.Source: §4.2 · p. 7, Table 4 · p. 8, Table 5 · p. 8

• FedCyBGD enables full-parameter LLM pre-training or fine-tuning in federated learning on a single consumer-level GPU.The framework is designed to reduce computation, memory, and communication costs for resource-limited edge clients.Source: §5 · pp. 8–9
• Accuracy improves by up to 58.3%, while efficiency costs are reduced to 13.8% across evaluations on various LLMs and NLP tasks.Source: §5 · p. 9

• FedCyBGD lacks theoretical guarantees for convergence of its cycle block update.Prior cycle-update analysis may provide a basis for studying the block-update paradigm.Source: §6 · p. 9
• The evaluation focuses primarily on NLP tasks, leaving applications to visual and other tasks for future investigation.Source: §6 · p. 9

7.1 Experimental Environment

• Experiments use NVIDIA RTX 3090 and H20 GPUs for all evaluations.Source: §7.1 · p. 13

7.2 Experimental Setup

• The evaluation covers GPT2-small, BLOOM, RoBERTa-base, ChatGLM3-6B, and LLaMA2-7B on GLUE, Alpaca-GPT4, OASST1, and SQuAD tasks.The tasks include sentiment analysis, question answering, textual entailment, instruction following, and conversational evaluation.Source: §7.2 · p. 13
• Federated scenarios use 32 or 64 resource-limited devices with a parameter server and a randomly selected cyclic model-update order each training cycle.Source: §7.2 · p. 13
• Comparisons include zero-shot, full tuning, Adaptor, LoRA, BADAM, ADAM, and PowerSGD, with ADAM as the default backbone optimizer unless otherwise specified.Source: §7.2 · p. 13

7.3 Additional results

• Training details are tabulated for FedCyBGD and baselines across models and tasks, while separate tables report federated client participation-round time and memory consumption under specified model, dataset, and batch-size conditions.Table 8 reports time per federated client participation round for fine-tuning LLaMA2-7B on Alpaca-GPT4 with batch size 2; Table 10 reports memory consumption on MRPC with RoBERTa across batch sizes.Source: Table 7 · p. 13, Table 8 · p. 13, Table 10 · p. 14
• Additional analyses examine FedCyBGD performance under block-allocation strategies, convergence from beginning to ending, and varying client numbers.The client-number analysis uses LLaMA-2 7B on Alpaca-GPT4 and reports one communication round.Source: Figure 3 · p. 14, Figure 4 · p. 14, Table 9 · p. 14