PaperlayerView exact revision on arXiv

Source-linked AI summary

FwdLLM: Efficient FedLLM using Forward Gradient

Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, Shangguang Wang

arXiv:2308.13894v2cs.AIcs.LG

TL;DR

FedLLM still struggles with mobile memory, convergence, and device limitations despite addressing model-size networking challenges. FwdLLM replaces backpropagation with perturbed inference and combines PEFT, adaptive pacing, and discriminative sampling. It outperforms contemporary FedLLM methods in convergence time and scalability, including up to 14.6× lower memory footprint and billion-parameter LLaMA fine-tuning on COTS smartphones.

  • Problem

    FedLLM faces practical mobile barriers from backpropagation, including high memory use, limited NPU compatibility, and limited device scalability.

    Source: §1 · pp. 1–2

  • Method

    FwdLLM uses BP-free perturbed inference with PEFT, adaptive perturbation pacing, and discriminative perturbation sampling for federated LLM fine-tuning.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Results

    FwdLLM outperforms contemporary FedLLM methods in convergence time and scalability, reducing memory footprint by up to 14.6× and enabling LLaMA fine-tuning on COTS smartphones within 10 minutes.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    FwdLLM provides a practical pathway for federated fine-tuning of billion-parameter LLMs on mobile devices.

    Source: §1 · p. 3, §7 · p. 13

    Prior BP-free methods were mainly validated on models 1–3 orders of magnitude smaller than the standard LLMs targeted by FwdLLM, and perturbation requirements are difficult to determine beforehand.

    Source: §1 · p. 2, §3.1 · p. 5

Abstract

from arXiv · show

Large Language Models (LLMs) are transforming the landscape of mobile intelligence. Federated Learning (FL), a method to preserve user data privacy, is often employed in fine-tuning LLMs to downstream mobile tasks, an approach known as FedLLM. Though recent efforts have addressed the network issue induced by the vast model size, they have not practically mitigated vital challenges concerning integration with mobile devices, such as significant memory consumption and sluggish model convergence. In response to these challenges, this work introduces FwdLLM, an innovative FL protocol designed to enhance the FedLLM efficiency. The key idea of FwdLLM to employ backpropagation (BP)-free training methods, requiring devices only to execute ``perturbed inferences''. Consequently, FwdLLM delivers way better memory efficiency and time efficiency (expedited by mobile NPUs and an expanded array of participant devices). FwdLLM centers around three key designs: (1) it combines BP-free training with parameter-efficient training methods, an essential way to scale the approach to the LLM era; (2) it systematically and adaptively allocates computational loads across devices, striking a careful balance between convergence speed and accuracy; (3) it discriminatively samples perturbed predictions that are more valuable to model convergence. Comprehensive experiments with five LLMs and three NLP tasks illustrate FwdLLM's significant advantages over conventional methods, including up to three orders of magnitude faster convergence and a 14.6x reduction in memory footprint. Uniquely, FwdLLM paves the way for federated learning of billion-parameter LLMs such as LLaMA on COTS mobile devices -- a feat previously unattained.

1 INTRODUCTION

FedLLM brings private LLM fine-tuning to mobile devices, but backpropagation creates memory, accelerator, and scalability barriers. FwdLLM addresses these barriers with BP-free perturbed inference, parameter-efficient fine-tuning, adaptive pacing, and discriminative perturbation sampling.

1 INTRODUCTION

  • FedLLM remains impractical because mobile training faces huge memory demands, limited NPU compatibility, and poor scalability across available devices.RoBERTa-large requires 3.9 GB for activations, mobile NPUs generally lack training support, and deployed FL may sample only about 1% of training-ready devices.Source: §1 · p. 1
  • FwdLLM replaces backpropagation with perturbed inferences that estimate an unbiased gradient while requiring only inference computations on devices.Devices perturb model weights, compare prediction deviations with the unmodified model, and use the results to construct a BP-free gradient.Source: §1 · p. 2
  • Parameter-efficient fine-tuning makes BP-free training scale with trainable rather than total parameters, aligning it with LoRA and Adapter methods for LLM fine-tuning.The authors identify the orchestration of BP-free training and PEFT as significant for FedLLM, especially because larger LLMs can require fewer trainable PEFT parameters.Source: §1 · p. 2
  • FwdLLM adaptively paces perturbation computation using gradient variance and prioritizes devices, data sizes, and perturbation counts to improve convergence efficiency.The aggregator stops perturbation inference when observed variance falls below a threshold, rather than relying on a fixed user-defined setting.Source: §1 · p. 2
  • FwdLLM filters low-value perturbations by comparing them with previous-round gradient directions, retaining perturbations more likely to contribute to convergence.Perturbations with small cosine similarity to the previous BP-free gradients are bypassed rather than computed on devices.Source: §1 · p. 2
  • Up to 217.3× shorter training time and up to 14.6× lower memory footprint position FwdLLM as a substantially more efficient alternative for mobile FedLLM.Against full-model fine-tuning, training time falls from 10.9–97.9 hours to 0.2–0.8 hours; against PEFT baselines, it achieves 2.0×–93.4× speedup and 11.5× memory reduction.Source: §1 · pp. 2–3
  • FwdLLM combines BP-free training with quantization to fine-tune the 7B-parameter LLaMA on a COTS smartphone in 10 minutes.The evaluation covers five transformer-based models and four NLP tasks, with on-device profiling on a Google Pixel 7 Pro and Jetson TX2.Source: §1 · pp. 2–3

2 BACKGROUND AND MOTIVATIONS

FedLLM applies federated learning to privacy-sensitive LLM fine-tuning, but mobile deployment remains constrained by memory demands, limited NPU training support, and poor device scalability. Preliminary experiments show that these limitations are closely associated with backpropagation-based on-device training.

2.1 Federated Fine-tuning of LLMs

  • Recent FedLLM work mainly reduces network traffic with PEFT, but excessive memory use and lengthy convergence remain unresolved on mobile devices.FedLLM combines privacy-preserving federated learning with fine-tuning of pretrained LLMs for downstream tasks.Source: §2.1 · p. 3, Table 3 · p. 4, §2.2 · p. 4

2.2 Preliminary Experiments of FedLLM

  • 7.7GB for RoBERTa-large and 2.5GB for ALBERT-base exceed practical mobile memory budgets, while inference uses less than 1GB.Typical mobile devices possess 4GB–12GB DRAM, and only a fraction may be available for training.Source: §2.2 · p. 3
  • 21.2%-35.2% memory savings from adapter and bitfit methods remain inadequate because forward-pass activations dominate memory usage.Reducing trainable parameters does not eliminate the activations required for backward-gradient computation.Source: §2.2 · p. 3, Table 1 · p. 3
  • Nearly 30× NPU speedup over CPUs on recent Snapdragon hardware highlights an acceleration opportunity that backpropagation-based training cannot use.Mobile NPUs are designed for inference and lack operators required for backpropagation-specific training.Source: §2.2 · p. 4, Figure 2 · p. 4, Table 2 · p. 4
  • Mobile NPUs significantly reduce inference latency but provide no support for DNN training because they lack backpropagation-specific operators.The limitation applies to the measured Google Pixel 7 Pro setting; a Snapdragon Hexagon approach also faces precision and portability challenges.Source: §2.2 · p. 4, Table 2 · p. 4
  • 1.04× acceleration from 100 devices shows that backpropagation-based FL saturates after only tens of participants, even with adapters.This limits convergence gains despite millions of potentially available smartphones and IoT devices.Source: §2.2 · p. 4, Figure 3 · p. 4

3 FWDLLM DESIGN

FwdLLM replaces device-side backpropagation with perturbed inferences, combining parameter-efficient training, adaptive perturbation pacing, and discriminative sampling for scalable FedLLM.

3.1 Overview

  • FwdLLM replaces backpropagation with perturbed inferences, reducing device memory and computation pressure while enabling mobile accelerator execution.Clients compare predictions from perturbed and unmodified models to estimate forward gradients, which are validated and aggregated by the server.Source: §3.1 · p. 5
  • FwdLLM’s design remains subject to BP-free training challenges because larger models require more perturbations and random perturbations can slow convergence.The required perturbation count is difficult to determine beforehand, motivating the paper’s PEFT, pacing, and sampling techniques.Source: §3.1 · p. 5, Figure 6 · p. 6, Figure 7 · p. 6

3.2 Parameter-Efficient BP-Free FedLLM

  • FwdLLM combines forward-gradient training with PEFT because BP-free complexity depends primarily on trainable parameters rather than total model size.An offline profiler compares forward and backpropagation gradients to select PEFT methods that save parameters while limiting performance degradation.Source: §3.2 · pp. 5–6

3.3 Var.-controlled Perturbation Pacing

  • FwdLLM adapts global perturbation size using gradient variance, increasing computational effort as convergence makes forward-gradient estimates less stable.The method prioritizes additional devices before additional perturbations per device, and uses a variance threshold to decide when aggregation can proceed.Source: §3.3 · pp. 6–7

3.4 Discriminative Perturbation Sampling

  • Discriminative sampling filters perturbations with low similarity to prior-round gradients, avoiding computations that contribute little to convergence.In DistilBERT on AGNEWS, over 60% of sampled perturbations had cosine similarity below 0.03 and contributed less than 29.6% of final forward gradients.Source: §3.4 · pp. 7–8, Figure 9 · p. 7

4 IMPLEMENTATION AND SETUPS

The implementation evaluates FwdLLM across five language models and four NLP datasets using mobile and edge-device measurements, with time-to-accuracy and runtime costs as primary metrics.

Implementation

  • The prototype implements numerical forward gradients with functorch and supports AdapterHub adapters, self-implemented LoRA and BitFit, and quantized LLaMA training through AutoGPTQ.The implementation is built atop FedNLP.Source: §4 · p. 8

Models and Hardware

  • Experiments cover five models, including quantized LLaMA-7B, with model-specific PEFT choices and federated training on mobile-oriented hardware.The evaluated models include four BERT-like encoders and INT4 LLaMA-7B; Pixel 7 Pro provides the primary mobile platform, while Jetson TX2 is a secondary edge platform.Source: §4 · p. 8

Datasets

  • The study uses AGNEWS, YAHOO, YELP-Polarity, and SQuAD-v1.1 under uniform and non-IID client partitions.Client counts vary by dataset, with skewed label distributions used for non-IID settings.Source: §4 · p. 8

Metrics and Baselines

  • Evaluation reports time-to-accuracy together with on-device memory, network, and energy costs, using fixed target accuracies for three classification datasets.Baselines include Full-FT, Adapter, BitFit, and FedAdapter, while FwdLLM uses the same default federated settings for comparison.Source: §4 · p. 8

5 EVALUATION

Across sub-billion and billion-parameter models, FwdLLM accelerates federated convergence while reducing device resource costs. Its adaptive and discriminative perturbation designs further improve convergence efficiency.

5.1 Convergence Performance on Sub-billion-sized Models

  • 132.7× faster than Full-FT and 9.6× faster on average than PEFT and FedAdapter baselines on sub-billion models.FwdLLM benefits from forward-only computation and can exploit mobile NPUs, GPUs, and CPUs.Source: §5.1 · p. 9
  • 12.2–3143.7× faster than Full-FT and up to 39.1× faster than strong baselines across heterogeneous hardware.With 1 to 500 clients, convergence time falls from 563.65 to 9.96 minutes; under non-iid data, accuracy drops by up to 3.8% while convergence remains up to 337.5% faster.Source: §5.1 · p. 9, Table 6 · p. 10, Figure 11 · p. 11

5.2 Convergence Performance on Billion-Sized Model (LLaMA)

  • FwdLLM enables federated LLaMA-7B fine-tuning on COTS mobile devices with INT4 quantization, reaching target accuracy in 0.19 hours on mobile CPU.This is comparable to centralized training on one NVIDIA A100, while NPU acceleration is estimated to be faster; LLaMA mobile-NPU speed is emulated rather than directly measured.Source: §5.2 · pp. 9–10, Table 7 · p. 11
  • INT8 and INT4 quantization reduce LLaMA training-time memory from 15.6GB to 7.9GB and 4.0GB, with accuracy degradation below 1.2%.FwdLLM reaches 83% F1 on SQuAD after 312 instruction-tuning rounds, only 1.9% below centralized BP-based training.Source: §5.2 · p. 11

5.3 System Cost

  • FwdLLM reduces memory usage by up to 93% versus Full-FT and 91.3% versus PEFT benchmarks, requiring 169MB per round.Its forward-only design stores trainable parameters and perturbation weights rather than backpropagation activations.Source: §5.3 · pp. 11–12, Figure 15 · p. 11
  • FwdLLM saves 96.7% energy and 98.6% network cost versus Full-FT, while using 2.6× less energy than PEFT baselines on average.It also uses 2.1× less network traffic than Adapter, excluding BitFit’s exceptionally small bias updates.Source: §5.3 · p. 12

5.4 Significance of key designs

  • Vanilla Forward-FL reduces convergence time by 14.13×–22.45×, while discriminative sampling and variance-controlled pacing add 1.63×–2.88× and 1.78×–2.02× improvements.These designs increase useful client and perturbation contributions, eliminate ineffective perturbations early, and identify better perturbations later.Source: §5.4 · p. 12, Figure 16 · p. 12
  • Adaptive Global-PS achieves 85% accuracy 2.99× faster than fixed Global-PS=50, while a 20% sampling ratio gives a 2.33× runtime improvement over no sampling.A 1% sampling ratio is 1.42× slower than the optimal 20% ratio, whereas 10–40% provides comparable runtime benefits.Source: §5.4 · p. 12, Figure 17 · p. 12, Figure 18 · p. 12

6 RELATED WORK

Prior FedLLM work uses privacy-preserving federated fine-tuning and PEFT, but these approaches leave major device-side memory and speed constraints. Backpropagation-free methods offer an alternative, while FwdLLM targets scalable mobile LLM training.

FedLLM

  • FedLLM applies federated learning to privacy-sensitive LLM fine-tuning, while PEFT methods such as Adapter and LoRA mainly mitigate network overhead.The paper states that PEFT-based FedLLM does not significantly optimize on-device memory or training speed.Source: §6 · pp. 12–13

Backpropagation-free training

  • Prior forward-gradient work used toy models, while concurrent BAFFLE and FedZeN lack FwdLLM’s techniques for systematic and efficient backpropagation-free federated training.FedKSeed reduces network cost but requires lengthy gradient decoding and excessive local training steps.Source: §6 · p. 13

FL Optimizations

  • FwdLLM abandons backpropagation to address the larger gap between LLM demands and mobile resource constraints than existing FL optimizations provide.Its claimed advantages span convergence time, scalability, and device-side overhead rather than communication efficiency alone.Source: §6 · p. 13

7 CONCLUSIONS

FwdLLM introduces a backpropagation-free federated learning framework for LLMs that targets memory-intensive training and improves scalability on mobile devices. Its adaptive perturbation generation, discriminative sampling, and reported advantages in convergence time and scalability define the framework’s central contributions.

  • FwdLLM is a pioneering FedLLM framework that trains LLMs without backpropagation, avoiding memory-intensive training and enabling scalable mobile-device training.Source: §7 · p. 13
  • An adaptive perturbation generator determines the number of perturbations on the fly to produce forward gradients more efficiently and precisely.Source: §7 · p. 13
  • A discriminative sampler selectively screens generated perturbations, while comparative analyses report better convergence time and scalability than contemporary FedLLM methods.Source: §7 · p. 13
Loading 2308.13894v2…