PaperlayerView exact revision on arXiv

Source-linked AI summary

FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, Holger R. Roth

arXiv:2310.01467v1cs.CLcs.AI

TL;DR

Federated PLM fine-tuning can use sensitive, device-resident data, but clients may lack model access and resources for local training. FedBPT optimizes prompts with gradient-free methods and reports comparable performance with sharply lower communication and memory costs.

  • Problem

    Federated PLM adaptation must work with sensitive device-resident data while addressing clients’ limited parameter access, compute, storage, and communication capacity.

    Source: §1 · p. 1

  • Method

    FedBPT uses gradient-free optimization to train prompts, exchanging prompts rather than model parameters and requiring only forward propagation for local training.

    Source: §1 · p. 2

  • Results

    Across multiple datasets and state-of-the-art PLMs, FedBPT achieves comparable accuracy to gradient-based baselines while cutting communication cost by more than 500k× and memory footprint by more than 3×.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Evaluations support adapting black-box PLMs in federated learning with comparable accuracy and markedly lower communication and memory overhead.

    Source: §6 · p. 10

Abstract

from arXiv · show

Pre-trained language models (PLM) have revolutionized the NLP landscape, achieving stellar performances across diverse tasks. These models, while benefiting from vast training data, often require fine-tuning on specific data to cater to distinct downstream tasks. However, this data adaptation process has inherent security and privacy concerns, primarily when leveraging user-generated, device-residing data. Federated learning (FL) provides a solution, allowing collaborative model fine-tuning without centralized data collection. However, applying FL to finetune PLMs is hampered by challenges, including restricted model parameter access, high computational requirements, and communication overheads. This paper introduces Federated Black-box Prompt Tuning (FedBPT), a framework designed to address these challenges. FedBPT does not require the clients to access the model parameters. By focusing on training optimal prompts and utilizing gradient-free optimization methods, FedBPT reduces the number of exchanged variables, boosts communication efficiency, and minimizes computational and storage costs. Experiments highlight the framework's ability to drastically cut communication and memory costs while maintaining competitive performance. Ultimately, FedBPT presents a promising solution for efficient, privacy-preserving fine-tuning of PLM in the age of large language models.

1 INTRODUCTION

FedBPT addresses barriers to federated adaptation of PLMs by optimizing prompts without requiring clients to access model parameters. Its evaluations report comparable performance with substantially lower communication and memory costs.

1 INTRODUCTION

  • Federated PLM fine-tuning uses device-resident user data, but collecting and storing that sensitive data raises security and privacy concerns.Federated learning enables collaboration over decentralized data while preserving data privacy.Source: §1 · p. 1
  • Real-world FL fine-tuning of PLMs is constrained by client parameter access, device compute and storage costs, and communication overhead.Clients may use inference APIs without access to model parameters, while transmitting parameters or gradients can involve millions or billions of variables.Source: §1 · pp. 1–2
  • FedBPT trains prompts with gradient-free local optimization and exchanges prompts rather than model parameters, reducing local training to forward propagation.The paper says this reduces communicated variables from millions or billions to hundreds and lowers computational and storage costs.Source: §1 · p. 2
  • Communication cost falls by more than 500k× and memory footprint by more than 3×, while FedBPT achieves comparable results to baselines requiring model access and back-propagation.The evaluation covers multiple datasets and state-of-the-art PLMs.Source: §1 · pp. 2–3

2 RELATED WORKS

Prior work motivates federated PLM adaptation for privacy-sensitive, decentralized data, while identifying resource constraints and non-IID data as challenges. Prompt-based learning offers an alternative to full fine-tuning, but manual prompts can lag fine-tuned models.

2.1 FEDERATED LEARNING

  • Non-IID device data can reduce FL accuracy, while PLM adaptation still faces communication, compute, storage, and labeled-data constraints on individual devices.Prior empirical work cited in the section suggests PLMs can lessen non-IID effects, with vanilla FedAvg sometimes comparable to centralized training.Source: §2.1 · p. 3

2.2 PROMPT-BASED LEARNING

  • Manual prompts can underperform fine-tuned models, whereas continuous prompt tuning optimizes embedding-space vectors while leaving PLM parameters untouched.P-tuning also learns prompts in embedding space but requires an additional LSTM to predict token embeddings.Source: §2.2 · p. 3

3 PRELIMINARY: BLACK-BOX PROMPT TUNING

Prompt tuning optimizes a continuous prompt for task predictions, and black-box tuning searches for it without back-propagation. BBT reduces the search dimension and uses CMA-ES to update a distribution over candidate solutions.

3 PRELIMINARY: BLACK-BOX PROMPT TUNING

  • Prompt tuning chooses a continuous prompt vector to improve PLM predictions on input texts against their labels.Its objective minimizes the loss over prompts in a chosen search space.Source: §3 · pp. 3–4
  • BBT avoids back-propagation by optimizing a lower-dimensional variable z and projecting it into the original prompt space with a random matrix A.The search dimension d is much smaller than the prompt dimension D.Source: §3 · p. 4
  • CMA-ES samples candidate solutions from a multivariate normal search distribution and updates its mean, covariance, and step size using successful lower-loss steps.The sampled population size is denoted by λ.Source: §3 · p. 4

4 METHOD

FedBPT adapts black-box prompt tuning to federated learning by exchanging prompts rather than model parameters and using inference-only client optimization. Its design addresses both prompt overfitting across clients and the difficulty of aggregating CMA-ES search distributions.

4 METHOD

  • FedBPT lets clients collaboratively tune prompts without accessing PLM parameters, using inference instead of back-propagation and exchanging prompts rather than model parameters.The server is agnostic to the LLM architecture.Source: §4 · p. 4

4.1 PROBLEM FORMULATION

  • Given a shared projection matrix and private client datasets, clients collaborate to optimize z against their local losses.The resulting prompt is formed from the optimized z and the projection matrix.Source: §4.1 · p. 4, §4.2 · p. 5

4.2 OVERVIEW OF FEDBPT

  • Each round, the server sends global CMA-ES parameters, clients optimize locally and return parameters with loss values, and the server updates the global search state.After training, the global mean z is projected to compute the final prompt.Source: §4.2 · pp. 4–5

4.3 SERVER-LEVEL CMA-ES ALGORITHM

  • Unlike FedAvg’s direct averaging of distribution statistics, FedBPT uses server-level CMA-ES informed by uploaded local search results to update the global prompt distribution.Directly averaging CMA-ES statistics or mixing better and worse local search results may not yield an optimal global distribution.Source: §4.3 · p. 5, Figure 2 · p. 5

4.4 LOCAL BLACK-BOX PROMPT TUNING AGAINST OVERFITTING

  • To reduce local prompt overfitting, FedBPT regularizes training with perturbed inputs, including randomly sampled vocabulary sentences matching the original sentence length.The stated intuition is that even an optimal prompt should not produce confident predictions for a perturbed input.Source: §4.4 · p. 6, Figure 4 · p. 6

5 EXPERIMENTS

Experiments across three datasets and two backbones compare FedBPT with gradient-based and gradient-free baselines. FedBPT is more accurate than gradient-free baselines and comparable to gradient-based methods, while ablations examine random placement and local population size.

5.1 EXPERIMENTAL SETUP

  • Experiments use SST-2, Yelp, and AG’s News with RoBERTa and 7-billion-parameter Llama 2.The setup uses ten participating clients, 40 training samples per class, and both IID and non-IID splits, with α = 1.0 by default for non-IID data.Source: §5.1 · p. 7
  • Baselines include FedAvg, FedPrompt, and FedP-tuning for gradient-based methods, plus manual prompts, in-context learning, and FedAvg-BBT for gradient-free methods.FedAvg-BBT combines BBT and FedAvg to assess the designed server-level prompt tuning.Source: §5.1 · p. 7

5.2 EXPERIMENTAL RESULTS

  • 0.92% higher accuracy than FedPrompt and 0.69% lower than FedP-tuning on non-IID SST-2 with RoBERTa, with over 100× and 30,000× fewer trainable parameters, respectively.FedBPT reduces per-device, per-round communication versus FedP-tuning from 120MB to 4KB; on SST-2 it also cuts memory costs by more than 3× versus gradient-based methods.Source: §5.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
  • 2.3%, 4.57%, and 1.08% higher accuracies than FedAvg-BBT on non-IID SST-2, AG’s News, and Yelp, respectively.FedBPT also has higher accuracy than gradient-free baselines across all datasets under both IID and non-IID settings.Source: §5.2 · p. 8
  • More than 12%, 11%, and 13% higher accuracy than manual prompts on SST-2, AG’s News, and Yelp, respectively, under non-IID Llama 2 settings.Compared with FedP-tuning, FedBPT reduces trainable parameters from 235M to 500 and per-device, per-round communication from nearly 2GB to 4KB.Source: Figure 5 · p. 8, Table 3 · p. 8, §5.2 · p. 9
  • Across RoBERTa and Llama 2, FedBPT achieves much higher accuracy than gradient-free baselines and comparable accuracy to gradient-based methods.Its trainable-parameter count does not increase with model scale because a projection matrix maps the embedding space to a lower-dimensional space for CMA-ES learning.Source: §5.2 · p. 9

5.3 ABLATION STUDIES

  • Random placement improves global accuracy over vanilla BBT local training with no random replacement (r_p = 0) on non-IID RoBERTa experiments.The study tests r_p from 0% to 80% on SST-2 and AG’s News.Source: §5.3 · p. 9, Table 4 · p. 9
  • FedBPT accuracy is not sensitive to local population size λ_k across tested values from 5 to 20 on RoBERTa.The authors note that λ_k can be set relatively small to reduce computational cost.Source: §5.3 · p. 9, Figure 6 · p. 9

6 CONCLUSION

FedBPT adapts black-box PLMs through gradient-free optimization, without client access to model parameters. Evaluations report accuracy matching gradient-based methods with markedly lower communication and memory overhead.

  • FedBPT adapts black-box PLMs with gradient-free optimization, requiring only forward propagation for local training and lowering device and service-provider compute and storage demands.Evaluations on several datasets with state-of-the-art PLMs report accuracy matching gradient-based methods alongside markedly lower communication and memory overhead.Source: §6 · p. 10

A DERIVATION OF CORRECTED SEARCH STEP ON THE SERVER

FedBPT uses CMA-ES at the server to search for the global prompt distribution and search-step length, estimating a corrected global step because the server-side step is intractable. The correction addresses the variance and model-divergence risk of directly using clients’ local step lengths.

A DERIVATION OF CORRECTED SEARCH STEP ON THE SERVER

  • The server uses CMA-ES to search for the global prompt distribution and search-step length, but its global search step is intractable and must be estimated.The estimate is based on the distribution of the change in the global prompt between rounds.Source: A DERIVATION OF CORRECTED SEARCH STEP… · pp. 14–15
  • The server updates the global prompt from the weighted sum of the best client prompts, then derives an intermediate evolution-path value before updating the step length and covariance.The passage identifies this as the Rank-µ-Update and says it uses the µ best client prompts.Source: A DERIVATION OF CORRECTED SEARCH STEP… · p. 14
  • Each client’s local prompt update is represented using sampled points and their weights across local iterations, with the local search-step length specified for each iteration.The passages define the iteration count, sampled-point indexing, and corresponding weights used in the weighted sum.Source: A DERIVATION OF CORRECTED SEARCH STEP… · p. 14
  • The estimation setup assumes local covariance matrices change little during a round and uses equal CMA-ES recombination weights.The paper also sets equal client population sizes and relates the local rank to half the local population size.Source: A DERIVATION OF CORRECTED SEARCH STEP… · pp. 14–15
  • Directly using client search-step lengths as the global step increases the variance of prompt updates, preventing correct evolution-path adaptation and potentially causing local-model divergence.This also prevents CMA-ES from finding an optimal global-prompt distribution and next-round search-step length.Source: A DERIVATION OF CORRECTED SEARCH STEP… · p. 15

B ALGORITHM

The algorithm initializes a shared projection matrix and global CMA-ES parameters, then performs repeated rounds with client-side updates. Client updates include sampling binary masks and tokens shaped like the client input.

B ALGORITHM

  • Training initializes projection matrix A and global CMA-ES parameters {z0, σ0, C0} before iterating over communication rounds.Source: B ALGORITHM · p. 16
  • A client update starts from the current global prompt, search step, and covariance, then iterates locally.The initialization is shown as t,1 ← zt, σt, Ct.Source: B ALGORITHM · p. 16
  • During a client update, the algorithm samples binary masks and token sets with the same shape as the client input.The mask sampling uses a specified rate of zero elements.Source: B ALGORITHM · p. 16

C MANUAL PROMPT TEMPLATES

Manual prompt templates organize dataset examples for model-specific prediction formats. RoBERTa uses masked-language-model prompts, while Llama 2 uses prompts for generating labels.

C MANUAL PROMPT TEMPLATES

  • The templates include dataset-specific examples for SST-2, AG’s News, and Yelp, using ⟨S⟩ to mark the sentence.Source: C MANUAL PROMPT TEMPLATES · p. 17, Table 5 · p. 17
  • In the template notation, ⟨MASK⟩ is RoBERTa’s mask token and [OUTPUT] is Llama 2’s output placeholder.Source: Table 5 · p. 17
  • RoBERTa prompts organize examples as masked-language-model inputs, whereas Llama 2 prompts organize them for causal-language-model label generation.Source: C MANUAL PROMPT TEMPLATES · p. 17
Loading 2310.01467v1…