Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation

Yang Chen; Xiaoyan Sun; Yaochu Jin

Source-linked AI summary

Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation

Yang Chen, Xiaoyan Sun, Yaochu Jin

arXiv:1903.07424v1Mar 18, 2019cs.LGcs.AIcs.DCstat.ML

TL;DR

Federated learning must preserve client privacy while reducing communication over limited-bandwidth links and handling heterogeneous participation. The paper combines asynchronous layer-wise model updates with temporally weighted aggregation, and experiments report better accuracy and lower communication costs than canonical federated learning.

Problem
Federated learning preserves privacy by keeping data on clients, but client-server communication is costly and client data may be unbalanced, non-IID, massively distributed, or unreliable.
Method
The paper updates shallow and deep DNN parameters at different frequencies and uses temporally weighted aggregation to integrate local models, emphasizing more recent models.
Results
The proposed asynchronous federated learning with temporally weighted aggregation outperforms canonical federated learning in learning performance and communication costs on MNIST and human action recognition datasets.
Takeaways & Limitations
About 75 communication rounds achieved 90% accuracy with TEFL, compared with about 750 rounds for FedAVG, nearly reducing communication cost by 90%.
Takeaways & Limitations
The study assumes all local models use the same neural-network architecture and share hyperparameters such as the SGD learning rate.

Abstract

from arXiv · show

Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server communication since the end devices typically have very limited communication bandwidth. This paper presents an enhanced federated learning technique by proposing a synchronous learning strategy on the clients and a temporally weighted aggregation of the local models on the server. In the asynchronous learning strategy, different layers of the deep neural networks are categorized into shallow and deeps layers and the parameters of the deep layers are updated less frequently than those of the shallow layers. Furthermore, a temporally weighted aggregation strategy is introduced on the server to make use of the previously trained local models, thereby enhancing the accuracy and convergence of the central model. The proposed algorithm is empirically on two datasets with different deep neural networks. Our results demonstrate that the proposed asynchronous federated deep learning outperforms the baseline algorithm both in terms of communication cost and model accuracy.

I. INTRODUCTION

Federated learning trains models locally and exchanges only parameters, preserving client data privacy while facing heterogeneous, distributed, and unreliable participation. This work targets communication and learning-performance challenges through asynchronous parameter updates and temporally weighted aggregation.

Federated learning trains local models on clients, exchanges their parameters with a server, and aggregates them into a central model without uploading client data.
Client data may be unbalanced and non-IID, clients can be massively distributed, and many participating clients may be offline or connected unreliably.
The proposed asynchronous strategy updates shallow and deep DNN parameters at different frequencies to reduce communication, while temporally weighted aggregation gives higher weight to recent models.

A. Asynchronous Model Update

The asynchronous update strategy communicates shallow-layer parameters more frequently than deep-layer parameters, exploiting their different feature roles to reduce transmitted model data. Parameter-exchange schedules determine when deep-layer parameters participate in communication rounds.

The method updates shallow-layer parameters more frequently than deep-layer parameters because shallow layers learn general features while deep layers learn dataset-specific features.
The DNN parameters are divided into general-feature ωg and specific-feature ωs, with Sg ≪ Ss, and ωg exchanged more frequently than ωs.
Because Ss is usually very large in DNNs, exchanging deep-layer parameters less often can significantly reduce communication cost.
Figure 2 depicts parameter exchange across five clients and a server, with points marking client participation in particular communication rounds.
In the illustrated six-round loop, deep-layer parameters are exchanged only in rounds t−1 and t, whereas conventional aggregation exchanges both parameter groups every round.

B. Temporally Weighted Aggregation

The proposed aggregation accounts for the timeliness of local models, giving more recent updates greater influence than older ones. This contrasts with conventional aggregation, where previously trained local models do not participate in the current update.

Conventional aggregation: Conventional aggregation weights participating local models by client training-data size, while previously trained models are excluded from the current update.The conventional strategy illustrates current local models as participating and previously trained models as excluded.
Temporally weighted aggregation: Because client training data change across rounds, the proposed method assigns higher aggregation weight to the most recently updated local models.The method explicitly incorporates model timeliness into aggregation.
Temporally weighted aggregation: The temporally weighted aggregation uses t, timestampk, and e to represent the current round, newest-model update round, and time effect, respectively.In the accompanying illustration, deeper brown color indicates greater timeliness and higher aggregation weight.

C. Framework

TWAFL organizes training into loops and uses a flag to determine whether all layers or only shallow layers are updated and communicated. The server coordinates client updates, timestamps, and aggregation across communication rounds.

Server framework: TWAFL divides training into loops and periodically sets a flag during the final 1/freq rounds of each loop.The flag controls whether deep-layer parameters are included in updates and communication.
Server framework: The server randomly selects participating clients according to C, invokes ClientUpdate in parallel, updates timestamps, and aggregates the returned models.These operations form the main per-round server process.
Client framework: The flag specifies whether clients update and communicate all layers or only shallow layers.This choice implements the asynchronous model-update component.
Aggregation: The aggregation weighting includes parameter a, which is set to e or e/2 to examine different time-weighting effects experimentally.The parameter is introduced relative to the stated aggregation weighting function.
Client framework: ClientUpdate splits local data into batches, downloads either all or shallow-layer parameters, and performs local SGD for E epochs.B denotes local mini-batch size and E denotes the local epoch setting.

A. Experimental Design

The experiments evaluate TWAFL with CNN and LSTM models and compare it with FedAVG and ablated variants. The design also specifies server and client algorithm components and key parameter settings.

Experimental models: Experiments use CNNs for MNIST handwritten-digit recognition and LSTMs for human-activity recognition to examine learning performance and communication cost.The study evaluates the proposed method with two deep neural network architectures and corresponding tasks.
Server implementation: The server component initializes the central model and client timestamps before executing repeated communication rounds.The server algorithm is identified as Algorithm 2.
Server implementation: Each round selects m = max(C ∗ K, 1) clients and invokes client updates for the selected subset.The selected subset is denoted St.
Compared algorithms: FedAVG is the baseline, while TWFL removes asynchronous model updates and AFL removes temporally weighted aggregation from the comparison.The passage names FedAVG, TWAFL, TWFL, and AFL as the compared algorithms.
Parameter settings: freq controls how often deep-layer parameters are exchanged, while a adjusts the time effect in aggregation.For example, freq = 5/15 exchanges deep-layer parameters only during the last five of fifteen loop rounds.
Client implementation: The client component uses flag to choose all-layer or shallow-layer updates, performs local SGD, and returns the corresponding parameters to the server.The client procedure is specified through batch construction, local epochs, and conditional parameter return.

B. Settings on Datasets

The client datasets are designed to reflect federated learning challenges, including non-IID, unbalanced, and massively decentralized data. Their generation is controlled by four parameters.

Dataset design: The experimental datasets are designed to reflect non-IID, unbalanced, and massively decentralized federated-learning data.The paper states that these challenges motivate the dataset design.
Dataset generation: Algorithm 4 controls client-dataset generation using Labels, Nc, Smin, and Smax.Nc controls the number of classes per local dataset, while Smin and Smax specify the sample-size range.

1) Handwritten Digit Recognition Using CNN:

The experiments use non-IID, unbalanced client partitions for MNIST and HAR, evaluating CNN and LSTM models with predefined local datasets.

MNIST dataset: MNIST contains ten digit classes represented as 28×28-pixel grayscale images and is partitioned into client datasets using label-based shards.The partitioning uses ten shards, with K = 20 and randomly selected Nc values of 2 or 3.
Local dataset generation: Algorithm 4 generates non-IID and unbalanced local datasets from class labels and partition parameters.Its inputs are Labels, NC, Smin, and Smax, and its output is a client partition Pk.
Pre-generated datasets: Five predefined MNIST client datasets, 1@MNIST through 5@MNIST, are visualized with 3-D column charts.The corresponding charts are presented in Fig. 6.
CNN architecture: The MNIST CNN uses two 5 × 5 convolution layers, max pooling, a 512-unit fully connected layer, and a softmax output.The convolution layers have 32 and 64 channels, respectively, and the fully connected layer uses ReLU activation.
HAR dataset: HAR consists of image sequences labeled with six activities and is similarly distributed across local clients.For HAR, K = 20, Nc is randomly selected from {2, 3}, and local dataset sizes range from 250 to 500 samples.

C. Results and analysis

Parameter studies examine communication-frequency, temporal-weighting, and scalability effects using CNN experiments, with performance measured by accuracy and communication rounds.

Experimental design: The experiments vary freq, a, K, and m to study parameter effects and compare algorithms using CNN models on 1@MNIST.A second experiment set evaluates communication cost and learning accuracy with LSTM models on HAR.
Scope of analysis: The experiments provide basic practical understanding of parameter settings rather than a detailed sensitivity analysis.Other parameters are fixed at default values when one parameter is investigated.
Metrics: Best central-model accuracy within 200 rounds and rounds required to reach 95.0% accuracy are the two evaluation metrics.Ten independent runs report averages and standard deviations; equal computing rounds imply equal communication cost.
Analysis on freq: Lower deep-layer exchange frequency reduces communication cost, but an excessively low freq deteriorates central-model accuracy.The frequency analysis uses freq values of 3/15, 5/15, and 7/15 with a = e/2, K = 20, and m = 2.
Analysis on a: When a = e, recent local models receive greater aggregation weight, whereas a = e/2 gives previously updated local models greater influence.The experiments report e/2 as better for the CNN on 1@MNIST; a = 1 reduces the method to AFL with equal round weights.
Scalability: Larger active-client participation m improves recognition accuracy, while TWAFL outperforms FedAVG at active client fraction C = 0.1.FedAVG is slightly better when the total client count is smaller and C is higher, suggesting a stronger TWAFL advantage as client numbers increase.

2) Comparison on Accuracy and Communication Cost:

Comparisons on MNIST and HAR evaluate temporal weighting and asynchronous updates against FedAVG and related variants. The proposed methods generally reduce communication and improve convergence, with task-dependent trade-offs.

Temporal weighting: TEFL reaches 95.0% accuracy in about 30 rounds on 1@MNIST and 2@MNIST, compared with about 75 rounds for FedAVG.This corresponds to a reported 40% reduction in communication cost.
Temporal weighting: TEFL reaches 90% accuracy on HAR in about 75 rounds, whereas FedAVG requires about 750 rounds, yielding a nearly 90% communication-cost reduction.TEFL generally achieves higher HAR accuracy except on 5@HAR and converges faster there during the early stage.
Overall comparison: Both TWAFL and TEFL outperform FedAVG on most cases in total rounds, best accuracy, and total communication cost.These comparisons cover ten test cases generated from MNIST and HAR tasks.
Algorithm comparison: TEFL achieves the best performance on most tasks for total rounds and best accuracy, while TWAFL performs slightly better on MNIST communication cost.TEFL performs better than TWAFL on HAR datasets.
Asynchronous updating: The asynchronous model-update strategy reduces communication cost per round, but AFL performs worst among the four compared algorithms.The results indicate that asynchronous updating benefits from temporal weighting rather than being used alone.

V. CONCLUSIONS AND FUTURE WORK

The paper reports that its asynchronous federated learning with temporally weighted aggregation improves learning performance and communication costs relative to canonical federated learning. The study assumes identical neural-network architectures and shared hyperparameters, while identifying evolving local models as future work.

Conclusions: The proposed asynchronous federated learning with temporally weighted aggregation outperforms canonical federated learning in learning performance and communication costs.The comparison uses empirical studies on MNIST and human action recognition datasets.
Assumptions: The study assumes that all local models use the same neural-network architecture and share hyperparameters such as the SGD learning rate.
Performance and Cost Evaluation: The performance experiments use accuracy thresholds and round limits, while total communication cost is reported relative to TWAFL as 1.The criteria include 95% within 200 rounds and 90% within 1000 rounds.
Future Work: Future research will develop federated learning algorithms that allow clients to evolve their local models to further improve learning performance and reduce communication costs.