PaperlayerView exact revision on arXiv

Source-linked AI summary

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Terence Jie Chua, Wenhan Yu, Jun Zhao, Kwok-Yan Lam

arXiv:2310.17491v2cs.LGcs.NI

TL;DR

Fine-tuning foundation models is difficult when data remain distributed across resource-constrained devices and model owners cannot share complete models. The paper combines fixed emulators, trainable adapters, PEFT, federated learning, and adaptive deep reinforcement learning in FedPEAT; experimentally, adaptive FedPEAT outperforms Fed-FT in communication and computation efficiency while SABPPO improves the adaptive-control baselines.

Abstract

from arXiv · show

The emergence of foundation models, including language and vision models, has reshaped AI's landscape, offering capabilities across various applications. Deploying and fine-tuning these large models, like GPT-3 and BERT, presents challenges, especially in the current foundation model era. We introduce Emulator-Assisted Tuning (EAT) combined with Parameter-Efficient Fine-Tuning (PEFT) to form Parameter-Efficient Emulator-Assisted Tuning (PEAT). Further, we expand this into federated learning as Federated PEAT (FedPEAT). FedPEAT uses adapters, emulators, and PEFT for federated model tuning, enhancing model privacy and memory efficiency. Adapters adjust pre-trained models, while emulators give a compact representation of original models, addressing both privacy and efficiency. Adaptable to various neural networks, our approach also uses deep reinforcement learning for hyper-parameter optimization. We tested FedPEAT in a unique scenario with a server participating in collaborative federated tuning, showcasing its potential in tackling foundation model challenges.

Overview of FedPEAT with adaptive control

FedPEAT combines flexible emulator-assisted tuning with PEFT and federated learning to address privacy, ownership, and resource constraints. Adaptive control uses deep reinforcement learning to coordinate compression, tuning, communication, and participation decisions, including server-device collaborative tuning.

Proposed EAT structure

  • EAT generalizes emulator and adapter configurations across neural network architectures and task complexities while targeting privacy and memory-efficient fine-tuning.Adapters and emulators can vary in layers, nodes, and transformer attention arrangements.Source: Proposed EAT structure · p. 2

Expansion to PEAT architecture

FedPEAT framework

  • FedPEAT extends PEAT to federated learning, avoiding full-model transmission while improving memory and computational efficiency during collaborative downstream tuning.Source: FedPEAT framework · p. 3

FedPEAT adaptive control mechanism

Server-Device collaborative tuning

Parameter-Efficient Emulator-Assisted Tuning (PEAT) Sub-units

PEAT uses fixed, compressed emulators to represent foundation models and small trainable adapters to encode downstream-task information. PEFT further reduces the parameters and memory involved in federated fine-tuning.

Emulator

  • Emulators compress foundation-model knowledge into smaller fixed-parameter networks that reduce device storage and computation while protecting model ownership.Compression may use pruning, layer dropping, or knowledge distillation.Source: Emulator · p. 4

Adapter

  • Adapters are small trainable modules attached to foundation models or emulators, encoding downstream-task information while reducing transmission costs.Source: Adapter · p. 4

PEFT integration

  • PEFT methods such as LoRA and adapters reduce model size and memory requirements while retaining comparable performance, and can be applied to federated adapter modules.Source: PEFT integration · p. 4

Federated Parameter-Efficient Emulator-Assisted Tuning (FedPEAT) Framework

FedPEAT separates a foundation model into trainable adapter parameters and fixed emulator or model components, allowing heterogeneous devices to tune locally without receiving the full model. Its iterative process combines adaptive device and emulator selection with local tuning, adapter aggregation, and repeated global redistribution.

Emulators and Adapters

  • FedPEAT uses an adapter Aφn for downstream-task tuning and an emulator Eθn as a fixed, compressed representation that guides adapter updates.The emulator is produced after removing adapter layers and can be tailored to each device.Source: Emulators and Adapters · p. 4
  • The framework permits distinct device emulators and fixed untrainable weights, accommodating heterogeneous storage and computational constraints.Source: Emulators and Adapters · p. 4

Tuning Process

  • The server model combines untrainable foundation-model components with collaboratively trainable adapter parameters, while the proposed FedPEAT framework formalizes this federated fine-tuning workflow.Source: Tuning Process · pp. 5–6

Emulators and Adapters

  • EAT generalizes emulator-adapter arrangements beyond prior offsite tuning, including flexible layer ordering and collaborative multi-device tuning.The framework also extends to settings involving server-side data and participation in fine-tuning.Source: Tuning Process · p. 5

Tuning Process

  • At each iteration, adaptive control selects users, emulator sizes, bandwidth, and transmission power before devices receive the global adapter and any changed emulator.Source: Tuning Process · p. 6
  • Selected devices fine-tune adapters locally with their datasets, upload adapter parameters, and receive an aggregated global adapter for subsequent iterations.The process continues until convergence or a specified criterion.Source: Tuning Process · p. 6

FedPEAT Adaptive Control Mechanism

FedPEAT adaptive control formulates federated fine-tuning as a constrained, multi-variable optimization problem and uses deep reinforcement learning to coordinate device, communication, emulator, and PEFT decisions. SABPPO branches action selection to avoid the combinatorial output growth of conventional discrete-action control.

Adaptive Control Scenario

  • The adaptive-control problem jointly optimizes emulator compression, device selection, bandwidth, and downlink power under memory, exchange-count, bandwidth, and power constraints.The objective minimizes cumulative perplexity, maximum device training time, and emulator exchanges across federated tuning iterations.Source: Adaptive Control Scenario · p. 7

Deep reinforcement learning approach

State

  • The state includes channel gain, device memory, transmission-rate information, emulator-exchange history, and previously selected control variables.Branch outputs are appended successively so later action branches receive the updated state representation.Source: State · p. 7

Action

  • The agent action space contains four coordinated decisions: UE selection, bandwidth selection, power selection, and emulator-compression selection.These decisions are produced through successive actor branches that concatenate each branch’s output with the state.Source: Action · p. 8, Single-Agent Action Branching Proximal… · p. 9

Reward

  • The reward follows the optimization objective and applies large penalties when emulator memory or exchange counts violate their allowable limits.These penalties enforce the memory and exchange constraints during adaptive control.Source: Reward · p. 8

Reinforcement Learning Algorithm

  • PPO stabilizes policy updates with clipped ratios, while its actor-critic training updates actor gradients and critic parameters from sampled trajectories.The clipping mechanism is intended to limit bias and keep policy changes within a trusted range.Source: Reinforcement Learning Algorithm · pp. 8–9

Single-Agent Action Branching Proximal Policy Optimization (SABPPO)

  • SABPPO reduces action-output growth from combinatorial to linear as action dimensionality increases by distributing controllers across coordinated actor branches.The architecture uses a shared decision module, one critic, and branches for user selection, bandwidth, power, and emulator compression.Source: Single-Agent Action Branching Proximal… · p. 9
  • The SABPPO architecture uses one shared critic and branched actor controllers whose outputs are passed sequentially through user, bandwidth, power, and emulator-compression decisions.Figure 4 depicts the actor-critic interaction with the environment and model-update process.Source: Figure 4 · p. 8, Single-Agent Action Branching Proximal… · p. 9

Numerical Experiments

FedPEAT with adaptive control is evaluated against federated full-model fine-tuning and alternative reinforcement-learning controllers under a server-participating federated setup. The experiments emphasize communication, computation, convergence, emulator exchanges, and perplexity.

Experiment configuration

  • The experiment uses OPT-1.3B with 1,208 million parameters, 1 server, 10 devices, 100 federated rounds per episode, and 5 selected devices per round.The emulator uses layer-drop compression, while the server holds 30% of the training data.Source: Experiment configuration · p. 10
  • Training uses 6G settings with bandwidth selected from 7–20 GHz, server power constrained to 0.0–15.0 W, ADAM optimization, and evaluation every 5,000 steps.The algorithms are trained for 5,000,000 steps.Source: Experiment configuration · p. 10

Experiment results

  • 4.60× longer delay is observed for Fed-FT than for FedPEAT with adaptive control in one computation-and-communication round.FedPEAT incurs 2.10× average emulator exchanges per 10 iterations despite this delay advantage.Source: Experiment results · p. 11
  • SABPPO reaches reward -168 after 5,000,000 steps, ahead of HAPPO at -225 and iterRL at -214.Its log(delay) is -4.02, compared with -1.33 for HAPPO and -3.27 for iterRL.Source: Experiment results · p. 11
  • SABPPO records 2.10× emulator exchanges and perplexity 15.03, versus 4.91× and 17.86 for HAPPO and 3.95× and 16.60 for iterRL.Source: Experiment results · p. 11
  • SABPPO training delay is 0.308 seconds, compared with 0.534 seconds for HAPPO and 0.549 seconds for iterRL.The reported comparison indicates lower training delay for SABPPO than both baselines.Source: Experiment results · p. 11

Discussion and Conclusion

The paper concludes that FedPEAT combines adapters, emulators, PEFT, EAT, and federated learning to address privacy, ownership, memory, and computation constraints. Its adaptive-control version improves efficiency and supports resource-aware federated tuning, while retaining a perplexity trade-off.

Discussion and Conclusion

  • EAT, PEAT, and FedPEAT extend offsite tuning from emulator-assisted and parameter-efficient tuning into the federated-learning domain.Source: Discussion and Conclusion · p. 11
  • Adapters provide trainable task-specific parameters, while emulators provide compressed fixed-parameter representations without transmitting complete models to edge devices.The framework is presented as improving model privacy, memory efficiency, and computational efficiency.Source: Discussion and Conclusion · p. 11
  • FedPEAT reduces communication and computation overhead relative to Fed-FT while exhibiting a slightly higher perplexity score.The paper attributes efficiency to reducing the foundation model’s memory footprint and the number of parameters tuned.Source: Discussion and Conclusion · p. 11
  • SABPPO trains faster than HAPPO and iterRL because it uses one actor and one critic instead of multiple actors and critics.The paper frames FedPEAT with adaptive control as balancing model performance, privacy, and resource efficiency.Source: Discussion and Conclusion · p. 11

Legends

The legends identify the conceptual intersection of federated learning, PEFT, and EAT, along with the two adaptive-control algorithms used in the framework.

Legends

  • Figure 1 depicts the intersection of federated learning, PEFT, and EAT that defines FedPEAT.Source: Legends · p. 14
  • Algorithm 1 is the FedPEAT procedure with its adaptive control mechanism.Source: Legends · p. 14
  • Algorithm 2 presents the SABPPO adaptive control algorithm.Source: Legends · p. 14
Loading 2310.17491v2…