PaperlayerView exact revision on arXiv

Source-linked AI summary

Client-Edge-Cloud Hierarchical Federated Learning

Lumin Liu, Jun Zhang, S. H. Song, Khaled B. Letaief

arXiv:1905.06641v2cs.NIcs.LG

TL;DR

The paper addresses the trade-off between cloud-based FL’s broad data access and edge-based FL’s efficient client communication. It proposes hierarchical FL with HierFAVG, providing convergence analysis and experiments showing reduced training time and end-device energy consumption compared with cloud-based FL.

  • Problem

    Cloud-based FL offers broad data access but incurs high communication overhead and latency, whereas edge-based FL communicates efficiently but accesses fewer clients.

  • Method

    The paper introduces a client-edge-cloud hierarchical FL architecture and HierFAVG, which performs partial aggregation at multiple edge servers before cloud aggregation.

  • Results

    Experiments show that hierarchical FL can simultaneously reduce model training time and end-device energy consumption compared with traditional cloud-based FL.

  • Takeaways & Limitations

    Convergence analysis and experiments support qualitative guidelines for selecting HierFAVG parameters across different data-distribution scenarios.

Abstract

from arXiv · show

Federated Learning is a collaborative machine learning framework to train a deep learning model without accessing clients' private data. Previous works assume one central parameter server either at the cloud or at the edge. The cloud server can access more data but with excessive communication overhead and long latency, while the edge server enjoys more efficient communications with the clients. To combine their advantages, we propose a client-edge-cloud hierarchical Federated Learning system, supported with a HierFAVG algorithm that allows multiple edge servers to perform partial model aggregation. In this way, the model can be trained faster and better communication-computation trade-offs can be achieved. Convergence analysis is provided for HierFAVG and the effects of key parameters are also investigated, which lead to qualitative design guidelines. Empirical experiments verify the analysis and demonstrate the benefits of this hierarchical architecture in different data distribution scenarios. Particularly, it is shown that by introducing the intermediate edge servers, the model training time and the energy consumption of the end devices can be simultaneously reduced compared to cloud-based Federated Learning.

I. INTRODUCTION

Federated learning keeps client data local while coordinating model updates, but cloud and edge deployments expose different communication, latency, and data-access trade-offs. The paper therefore proposes a client-edge-cloud hierarchy that combines cloud-scale data access with efficient edge updates.

  • Federated learning trains models from client-held data through local updates and server aggregation without directly accessing clients’ private data.
  • Cloud-based FL can access millions of clients and massive datasets, but cloud communication is slow and unpredictable.
  • Edge-based FL reduces communication latency and enables a better computation-communication trade-off, but each edge server accesses fewer clients.
  • The proposed client-edge-cloud hierarchy combines cloud access to massive training samples with quick edge-server updates for local clients.
  • Hierarchical FL is introduced as a three-layer system alongside the HierFAVG optimization algorithm.

A. Learning Problem

The paper formulates supervised federated learning as empirical-loss minimization when training data are distributed across clients and inaccessible to the parameter server. Model parameters are updated using gradient descent and weighted local losses.

  • The learning problem uses supervised data pairs, with model parameters w and per-sample loss functions measuring prediction error.
  • The global empirical loss F(w) is minimized over the training dataset.
  • The loss function may be convex, as in logistic regression, or non-convex, as in neural networks.
  • Gradient descent updates model parameters using a step size η and the gradient at update step k.
  • Because client datasets cannot be accessed directly by the server, the global loss is computed as a weighted average of local loss functions.

B. Traditional Two-Layer FL

Traditional two-layer federated learning uses periodic federated averaging between one central parameter server and distributed clients. This reduces communication relative to per-step aggregation by allowing multiple local gradient updates between exchanges.

  • FAVG operates with one central parameter server and N clients in the traditional two-layer FL system.
  • Clients perform κ local gradient-descent steps before communicating and aggregating their models.
  • The FAVG process repeats until the model reaches the desired accuracy or communication, time, or other resource limits are reached.

C. Client-Edge-Cloud Hierarchical FL

HierFAVG extends federated averaging to a three-layer client-edge-cloud architecture. Edge servers periodically aggregate models from local clients, while the cloud periodically aggregates the edge models, separating local and cloud communication intervals.

  • The hierarchical system contains one cloud server, L edge servers with disjoint client sets, and N clients with distributed datasets.
  • Algorithm 1 initializes all clients with w0, runs local updates in parallel, and synchronizes clients after edge or cloud aggregation.
  • Each edge server aggregates models from the clients assigned to it.
  • After every κ1 local updates, each edge performs client-model aggregation.
  • After every κ2 edge aggregations, the cloud aggregates all edge-server models, so cloud communication occurs every κ1κ2 local updates.

III. CONVERGENCE ANALYSIS OF HIERFAVG

The paper establishes convergence results for HierFAVG and uses the analysis to identify how key algorithm parameters affect its behavior.

  • HierFAVG's convergence is proved for both convex and non-convex loss functions.
  • Figure 3 compares the aggregation procedures of conventional FAVG and HierFAVG.
  • The convergence analysis reveals key properties of HierFAVG and the effects of its key parameters.

A. Definitions

This section defines the intervals, functions, divergences, and assumptions used to analyze hierarchical federated learning under non-IID client data.

  • The K local training iterations are divided into B cloud intervals of length κ1κ2 and Bκ2 edge intervals of length κ1.Local aggregation occurs at the end of each edge interval, while global aggregation occurs at the end of each cloud interval.
  • The analysis defines edge loss, weighted distributed weights, and a virtually centralized gradient descent sequence.These quantities support comparison between distributed and centralized training behavior.
  • FAVG convergence is bounded under smoothness, convexity, fixed-step-size, and deviation-related conditions.The stated conditions include η ≤ 1/β and additional requirements involving ρ, M, ε, and ϕ.
  • Non-IID client data distinguish federated learning from datacenter distributed learning and can influence model performance through unbalanced local distributions.
  • Gradient divergence measures differences between client and edge gradients and between edge and global gradients.The two levels are represented by δ for client-edge divergence and ∆ for edge-cloud divergence.

B. Convergence

The convergence analysis bounds HierFAVG’s weight deviation and establishes convergence for convex and non-convex losses. It also shows how aggregation intervals and data heterogeneity affect deviation and convergence behavior.

  • Convergence analysis: HierFAVG convergence is proved for both convex and non-convex loss functions using bounds on distributed-weight deviation.The proof compares the actual distributed weights with a virtually centralized gradient-descent sequence.
  • Special cases: When κ2 = 1, HierFAVG reduces to FAVG; when κ1 = κ2 = 1, it reduces to centralized gradient descent.The latter case gives zero distributed-weight deviation.
  • Data heterogeneity: When client and edge data are IID, the distributed-weight deviation bound becomes zero.This makes the distributed-weight iteration identical to the centralized-weight iteration.
  • Data heterogeneity: For non-IID data, edge-cloud divergence grows exponentially with κ1 and κ2, while client-edge divergence grows exponentially with κ1 and quadratically with κ2.Smaller weight deviation is associated with faster convergence in the analysis.
  • Design guidelines: With fixed κ1κ2, choosing a smaller κ1 and larger κ2 produces a smaller deviation bound.When edge data are IID, increasing κ2 does not substantially increase the bound, suggesting reduced cloud communication.
  • Stepsizes: With fixed stepsizes, the weighted average gradient norm converges to a nonzero value; diminishing stepsizes are needed for convergence toward zero.Fixed-step convergence is limited by gradient variance from non-IID data.

IV. EXPERIMENTS

Simulations verify the convergence-analysis observations and illustrate the advantages of hierarchical federated learning. The reported model-accuracy advantage over edge-based federated learning is obvious.

  • Simulation results: Simulations verify the convergence-analysis observations and show an obvious model-accuracy advantage over edge-based federated learning.The experiments therefore focus subsequent comparisons on cloud-based federated learning.

A. Settings

The experiments use a hierarchical FL system with specified datasets, non-IID distributions, and metrics for training time and local energy consumption.

  • The system includes 50 clients, 5 edge servers, and one cloud server, with equal client assignments and training-data amounts per edge.
  • Experiments use MNIST and CIFAR-10 image classification tasks with mini-batch SGD using batch size 20.MNIST uses a CNN with 21840 trainable parameters; CIFAR-10 uses an initial learning rate of 0.1 and exponential decay of 0.992 per epoch.
  • The experiments omit momentum to remain consistent with the convergence analysis.The authors report that momentum can speed training and improve final accuracy, while hierarchical FL benefits persist with or without it.
  • The study examines simple NIID, edge-IID, and edge-NIID client-data distributions.Edge-IID assigns different classes across clients at each edge, whereas edge-NIID assigns five total classes across each edge.
  • Training time Tα and local energy consumption Eα measure the resources needed to reach test accuracy level α.

B. Results

Results evaluate HierFAVG parameters through MNIST accuracy curves and compare hierarchical with cloud-based FL using training time and local energy consumption.

  • With cloud communication fixed at κ1κ2=60, decreasing κ1 reaches the desired accuracy in fewer training epochs under both edge-IID and edge-NIID distributions.This corresponds to more frequent communication with the edge and fewer local computations on devices.
  • Under edge-IID data, accuracy curves with κ1=60 nearly coincide across different κ2 values, so less frequent cloud communication does not slow training.The result indicates that high-cost cloud communication may be reduced with little performance loss in this scenario.
  • Under edge-NIID data with κ1=60, increasing κ2 slows training.
  • Across MNIST and CIFAR-10, training time decreases monotonically as communication with the edge becomes more frequent.The comparison is between cloud-based FL and hierarchical FL under fixed κ1κ2.
  • Local energy consumption first decreases and then increases as κ2 grows, requiring a balance between computation and communication energy.Moderate edge-client communication reduces local computation, whereas very frequent communication adds transmission energy.

V. CONCLUSIONS

The paper proposes a client-edge-cloud hierarchical FL architecture with HierFAVG, analyzes its convergence, and reports simultaneous reductions in training time and end-device energy versus cloud-based FL.

  • The proposed client-edge-cloud architecture is supported by the HierFAVG collaborative training algorithm.
  • Convergence analysis of HierFAVG produces qualitative design guidelines for selecting key algorithm parameters.
  • Experiments show that hierarchical FL can simultaneously reduce model training time and end-device energy consumption compared with traditional cloud-based FL.The study also identifies trade-offs in choosing key HierFAVG parameters and calls for further characterization and optimization.
Loading 1905.06641v2…