PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devices

Dingzhu Wen, Ki-Jun Jeon, Kaibin Huang

arXiv:2109.15258v3cs.LGcs.IT

TL;DR

Federated learning must reduce the communication and computation burden placed on resource-constrained devices. FedDrop addresses this by assigning independently generated subnets with device-adapted dropout rates, reducing C2 overhead while improving performance under overfitting and relative to uniform dropout.

  • Problem

    Resource-constrained devices struggle with the communication and computation burdens of updating deep-neural-network models in federated learning.

  • Method

    FedDrop generates multiple dropout-based subnets at the server and assigns them to devices with dropout rates adapted to communication and computation states.

  • Results

    FedDrop reduces C2 overhead and outperforms uniform dropout, with dropout benefiting learning under overfitting.

  • Takeaways & Limitations

    FedDrop enables federated learning on resource-constrained edge devices while combining resource reduction with improved handling of model overfitting.

Abstract

from arXiv · show

Federated learning (FL) is a popular framework for training an AI model using distributed mobile data in a wireless network. It features data parallelism by distributing the learning task to multiple edge devices while attempting to preserve their local-data privacy. One main challenge confronting practical FL is that resource constrained devices struggle with the computation intensive task of updating of a deep-neural network model. To tackle the challenge, in this paper, a federated dropout (FedDrop) scheme is proposed building on the classic dropout scheme for random model pruning. Specifically, in each iteration of the FL algorithm, several subnets are independently generated from the global model at the server using dropout but with heterogeneous dropout rates (i.e., parameter-pruning probabilities),each of which is adapted to the state of an assigned channel. The subnets are downloaded to associated devices for updating. Thereby, FedDrop reduces both the communication overhead and devices' computation loads compared with the conventional FL while outperforming the latter in the case of overfitting and also the FL scheme with uniform dropout (i.e., identical subnets).

I. INTRODUCTION

FedDrop targets federated learning’s communication-and-computation bottlenecks on resource-constrained devices by assigning adapted dropout subnets to devices. Unlike uniform dropout, it updates the whole model through subnet aggregation and can improve learning under overfitting.

  • I. INTRODUCTION: Existing communication techniques reduce transmission burdens but generally leave devices responsible for updating full models, while partitioning methods may not apply to DNNs.The cited discussion contrasts communication methods with computation-focused approaches and notes that DNN loss functions are generally not linearly decomposable.
  • I. INTRODUCTION: FedDrop addresses both communication and computation bottlenecks in federated learning for resource-constrained devices.The approach is inspired by dropout and is designed to reduce C2 overhead while coping with model overfitting.
  • I. INTRODUCTION: FedDrop assigns randomly generated subnets to devices, which train and transmit subnets instead of full models to alleviate C2 bottlenecks.The server later aggregates updated subnets to update the global model.
  • I. INTRODUCTION: Device dropout rates adapt to communication and computation states, reducing device C2 loads by (1−p_k)^2 for device k.The scheme uses heterogeneous dropout rates under a per-round latency constraint.
  • I. INTRODUCTION: FedDrop outperforms uniform dropout because it updates the whole model and trains different subnets, increasing model diversity.For Cifar-10, the introduction reports a 2.5% accuracy increase under dropout rate 0.6.

II. SYSTEM MODEL

The system is a single-cell wireless federated-learning network with one coordinating server and K edge devices. Each round uses orthogonal bandwidth and server-side channel-state information to support subnet assignment and synchronized training.

  • II. SYSTEM MODEL: A single server with an access point coordinates K edge devices connected through wireless links to complete a FedDrop task.The devices cooperatively train the model in a single-cell system.
  • II. SYSTEM MODEL: Each device uses orthogonal bandwidth for downloading and uploading during every training iteration.The channels are assumed frequency non-selective.
  • II. SYSTEM MODEL: The server coordinates training and has channel-state information for all wireless links.This information supports the system’s resource-aware operation.
  • II. SYSTEM MODEL: FedDrop adds server-generated subnet assignment to the conventional broadcast, local-update, and aggregation sequence.Several dropout-generated subnets are created and each is assigned to one device for updating.

2) Dropout:

Dropout generates reduced DNN subnets by randomly deactivating neurons, with dropout and presence rates controlling deactivation and retention. Progressive pruning is used when deterministic subnet size is needed for predictable training latency.

  • 2) Dropout:: Dropout generates a subnet by deactivating each neuron and its connections with probability p_k, called the dropout rate.The corresponding presence rate is 1−p_k, and dropout is applied to fully connected layers in the described DNN setting.
  • 2) Dropout:: The scaling factor 1/(1−p_k) preserves the expected neuron output, making the subnet’s expected outputs equal to those of the original DNN.This expectation-preserving property is part of the random dropout formulation.
  • 2) Dropout:: Random dropout produces uncertain subnet size and therefore uncertain C2 overhead and learning latency.The paper addresses this issue with progressive random parametric pruning.
  • 2) Dropout:: Progressive pruning repeatedly selects neurons uniformly at random and deactivates exactly p_k of the neurons in each layer.This produces a subnet with deterministic size rather than a random number of active neurons.

3) Per-Round Latency:

Per-round FedDrop latency combines subnet communication and local computation, then is determined by the slowest device because updates are synchronized. Communication and computation costs depend on subnet size and device resources.

  • 3) Per-Round Latency:: Each device’s subnet-update latency consists of communication latency for downloading and uploading, plus computation latency for local calculation.The communication component depends on subnet parameters, bandwidth, quantization, and spectrum efficiency; computation depends on operations, samples, and processor speed.
  • 3) Per-Round Latency:: The communication latency depends on subnet parameter count, quantization bits, assigned bandwidth, and downlink and uplink spectrum efficiencies.The subnet parameter count is denoted M_k and the quantization level is Q.
  • 3) Per-Round Latency:: The computation latency depends on per-sample processor operations, the number of local training samples, and device computation speed.These quantities are denoted C_k, D_k, and f_k, respectively.
  • 3) Per-Round Latency:: Synchronized training makes the round latency depend on the slowest device’s total latency.The server cannot finish the round before every device uploads its update.

III. FEDERATED DROPOUT

FedDrop generates subnets of different sizes for devices and adapts their sizes to device communication-and-computation capacities. Each round assigns, downloads, updates, and uploads subnets before the server updates the global model.

  • III. FEDERATED DROPOUT: FedDrop generates subnets of different sizes and adapts them to devices' communication-and-computation capacities.The server uses different dropout rates to create device-specific subnets.
  • III. FEDERATED DROPOUT: Each round consists of subnet generation, downloading, local updating, uploading, and global model updating.
  • III. FEDERATED DROPOUT: Assigning subnets rather than full models alleviates devices' communication-and-computation bottlenecks.
  • III. FEDERATED DROPOUT: Unlike conventional FL, FedDrop lets devices adaptively select a subnet for updating based on their communication-and-computation capacities.

B. Analysis and Optimization of FedDrop

FedDrop assigns devices different dropout rates and analyzes their communication-and-computation overhead under a latency target. The design uses channel conditions, spectrum efficiencies, and bandwidth allocations to optimize these rates.

  • B. Analysis and Optimization of FedDrop: FedDrop assigns devices different subnets generated with different dropout rates in each training round.
  • B. Analysis and Optimization of FedDrop: The adaptive FedDrop rate is designed from a per-round latency target, spectrum efficiencies, and allocated bandwidth.

1) Communication overhead analysis:

Communication and computation overhead decrease when dropout reduces the fully connected portions of a subnet. The analysis models both parameter counts and update operations using the same dropout-dependent scaling.

  • 1) Communication overhead analysis:: A dropout rate p_k reduces fully connected-layer parameters to (1 − p_k)^2 M_full while retaining convolutional parameters.M_conv and M_full denote the original convolutional- and fully-connected-layer parameter counts.
  • 1) Communication overhead analysis:: The subnet's fully connected-layer computation is approximated as (1 − p_k)^2 C_full, while convolutional computation remains represented separately.C_conv and C_full are the original computation overheads for convolutional and fully connected layers.
  • 1) Communication overhead analysis:: For adjacent fully connected layers, dropout scales multiplications, additions, gradients, and related operations according to the retained neuron counts.

3) Optimization of FedDrop rates:

FedDrop rates are optimized from per-round communication-and-computation latency. The resulting device dropout rate decreases monotonically as communication rate and computation speed increase.

  • 3) Optimization of FedDrop rates:: The minimum FedDrop rate is derived by incorporating communication and computation overhead into the per-round latency.
  • 3) Optimization of FedDrop rates:: The device dropout rate is a monotone decreasing function of its communication rate and computation speed.

IV. EXPERIMENTS AND PERFORMANCE ANALYSIS

Experiments evaluate FedDrop with wireless-resource and CNN settings, including CIFAR-10 and MNIST models, against conventional and uniform-dropout baselines.

  • The evaluation uses a single-cell network with one access point, K = 10 devices, 1 MHz bandwidth per device, and computation capacities from 0.1 to 1.0 GHz.
  • Testing accuracy is evaluated against dropout rate, while C2 overhead comparisons use test accuracy over communication rounds under varying latency constraints.
  • The experiments train CNNCifar on CIFAR-10 and CNNMnist on MNIST, representing complex and simple feature settings respectively.

1) Learning Performance:

FedDrop generally outperforms uniform dropout, with dropout helping most under overfitting; however, dropout can reduce performance when a simple model underfits.

  • 1) Learning Performance:: For CNNCifar, testing accuracy first increases and then decreases as dropout rate rises, reflecting a trade-off between model diversity and expression ability.
  • 1) Learning Performance:: FedDrop outperforms uniform dropout because it updates the whole model and trains different subnets in each round, increasing model diversity.
  • 1) Learning Performance:: For CNNMnist, testing accuracy decreases with dropout rate, but performance degradation remains slight when p ≤0.6.
  • 1) Learning Performance:: Under equal C2 resources, FedDrop always outperforms uniform dropout, while both dropout schemes can lose performance relative to FL without dropout when overfitting is absent.
  • 1) Learning Performance:: Longer per-round latency improves both dropout schemes, and the performance gap between them becomes larger at lower latency.

2) C2 Overhead Comparison:

The paper concludes that FedDrop alleviates resource constraints while retaining benefits against model overfitting, and identifies latency–accuracy and resource-management tradeoffs for future study.

  • FedDrop reduces communication-and-computation overhead for resource-constrained edge devices while coping with model overfitting.
  • A future direction is quantifying the tradeoff between training latency and testing accuracy by adjusting the dropout rate.
  • Another future direction is jointly designing radio resource management and load balancing for FedDrop systems.
Loading 2109.15258v3…