Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks

Stefano Savazzi; Monica Nicoli; Vittorio Rampa

Source-linked AI summary

Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks

Stefano Savazzi, Monica Nicoli, Vittorio Rampa

arXiv:1912.13163v1Dec 27, 2019eess.SPcs.DCcs.LG

TL;DR

The paper addresses the dependence of federated learning on centralized aggregation in decentralized IoT networks. It proposes server-less consensus-based FL using local computation and neighbor exchanges, and validates the approach in an Industrial IoT scenario. The results show that CFA-GE achieves centralized federated-optimization performance, while requiring more bandwidth and communication effort than CFA.

Problem
Centralized FL relies on server aggregation, whereas decentralized IoT settings require learning through device cooperation without exporting local training data.
Method
The paper develops consensus-based, server-less FL algorithms in which devices iteratively exchange local model updates and gradients over D2D links.
Results
CFA-GE achieves the performance of server-side federated optimization in an Industrial IoT passive body-detection scenario.
Takeaways & Limitations
Consensus-based FL provides a framework for model optimization over massive IoT networks with decentralized connectivity and computing.
Takeaways & Limitations
The study uses a simple neural network for constrained IoT devices, while deeper networks may require more efficient bandwidth use through techniques such as quantization or compression.

Abstract

from arXiv · show

Federated learning (FL) is emerging as a new paradigm to train machine learning models in distributed systems. Rather than sharing, and disclosing, the training dataset with the server, the model parameters (e.g. neural networks weights and biases) are optimized collectively by large populations of interconnected devices, acting as local learners. FL can be applied to power-constrained IoT devices with slow and sporadic connections. In addition, it does not need data to be exported to third parties, preserving privacy. Despite these benefits, a main limit of existing approaches is the centralized optimization which relies on a server for aggregation and fusion of local parameters; this has the drawback of a single point of failure and scaling issues for increasing network size. The paper proposes a fully distributed (or server-less) learning approach: the proposed FL algorithms leverage the cooperation of devices that perform data operations inside the network by iterating local computations and mutual interactions via consensus-based methods. The approach lays the groundwork for integration of FL within 5G and beyond networks characterized by decentralized connectivity and computing, with intelligence distributed over the end-devices. The proposed methodology is verified by experimental datasets collected inside an industrial IoT environment.

I. INTRODUCTION

Federated learning trains shared neural-network models from data retained on distributed devices, but conventional server coordination conflicts with decentralized IoT settings. The paper introduces consensus-based, server-less FL that uses device cooperation and evaluates its scalability and convergence in large, intermittently connected networks.

Motivation: Federated learning trains a shared neural-network model from locally held data without sending training data to the server, providing privacy advantages.Devices act as local learners and derive model parameters from their available examples.
Motivation: Decentralized IoT networks use direct, infrastructure-less device cooperation to support local computing under intermittent connectivity and communication constraints.Such networks are relevant to mission-critical applications requiring fast feedback from edge nodes.
Proposed direction: The paper targets FL without a central server by applying consensus-based methods to mutual in-network cooperation among devices.Devices perform local training and exchange model updates with neighbors over decentralized links.
Contributions: Consensus extends federated averaging through local-update exchange, while a new algorithm exchanges model updates and gradients to improve convergence and reduce communication rounds.The latter approach trades more intensive D2D-link and local-computing use for faster convergence.
Validation: The algorithms are validated on large networks with intermittent or varying connectivity, including an experimental Industrial IoT setup and assessments of complexity, convergence, overhead, and execution time.The paper focuses on neural-network optimization across distributed devices and considers non-IID local datasets.

III. A CONSENSUS-BASED APPROACH TO IN-NETWORK FL

The proposed in-network FL methods replace centralized aggregation with local cooperation over a directed D2D graph. Devices combine neighbor models through consensus and perform local model updates, with convergence defined by all local models approaching the global solution.

Network model: Devices learn global model parameters through local cooperation and in-network processing over a directed graph representing D2D interactions.Each device has a neighbor set determined by the graph's directed edges.
Consensus-based Federated Averaging: Consensus-based Federated Averaging receives neighbor models, combines them with consensus weights, performs a local model update, and sends the updated model onward.The procedure repeats these receive, consensus, local-SGD, and transmit operations each round.
Local optimization: Each device trains a local neural-network model on its own examples using a generally non-convex local cost function.The local dataset consists of input-output examples retained by that device.
Convergence objectives: The methods exploit adaptive diffusion and consensus so every local model converges toward the global model while limiting convergence time and communication rounds.The design targets cooperation across potentially large populations of federated devices.

A. Consensus based Federated Averaging (CFA)

Consensus-based Federated Averaging (CFA) extends federated averaging by replacing server aggregation with iterative model-update exchange among neighboring devices.

A. Consensus based Federated Averaging (CFA): CFA lets devices exchange local model updates and aggregate them through consensus rather than relying on centralized server fusion.Each device sends its model update once per communication round and receives neighboring weights.
A. Consensus based Federated Averaging (CFA): All devices host similarly initialized models with the same architecture before consensus-based training begins.Initialization occurs at t = 0, after which communication rounds proceed for t > 0.
A. Consensus based Federated Averaging (CFA): Each device first forms an aggregated model from neighbors’ updates, then performs a local gradient update using SGD.The aggregated model is denoted ψt,k, and local training uses mini-batches smaller than the local dataset.
A. Consensus based Federated Averaging (CFA): The consensus step-size ϵt controls consensus stability, while mixing weights αk,i determine how model innovations are combined.Setting ϵt = 1 makes the aggregation similar to sum-weight gossip, but CFA combines model innovations explicitly.

B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE)

CFA-GE extends CFA by exchanging both model updates and gradients, allowing devices to incorporate gradient information computed from neighbors’ data during iterative training.

B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE): CFA-GE jointly exchanges local model updates and gradients through a four-stage procedure for each training epoch.The procedure begins with consensus-based model aggregation and includes a negotiation stage before gradient-based updating.
B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE): Neighbor gradients let each device exploit additional data information, which the paper reports makes learning much faster than CFA.Gradients are computed over a single local batch, and the selected batch may change across communication rounds.
B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE): CFA-GE combines the local gradient with gradients computed by neighbors at the aggregated model, then performs further SGD updates on local mini-batches.The negotiation stage aggregates the model, applies one gradient-descent round using received gradients, and then runs local SGD rounds.
B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE): Sending multiple gradients is an alternative not considered because of bandwidth limitations.The implementation therefore uses one batch per neighbor for gradient exchange.
B. Consensus based Federated Averaging with Gradients Exchange (CFA-GE): Gradient mixing weights βk,i are critical for convergence because neighbor gradients are computed from single batches rather than local mini-batches.The paper identifies βk,i > 1 as a reasonable choice for neighbor gradients.

C. Two-stage negotiation and implementation aspects

The two-stage CFA-GE implementation removes synchronous negotiation and predicts neighbor gradients from past models, reducing waiting while retaining asynchronous updates.

C. Two-stage negotiation and implementation aspects: CFA-GE normally requires synchronous gradient exchange, greater bandwidth use, and more communication rounds than CFA.Devices exchange models during negotiation and then forward gradients over D2D links.
C. Two-stage negotiation and implementation aspects: The proposed implementation uses two stages so devices can update without waiting for replies from neighbors.This simplification is intended to improve convergence time and latency.
C. Two-stage negotiation and implementation aspects: Because synchronous model sharing is omitted, device k predicts neighbor gradients using gradients evaluated on outdated neighbor models.The predictions use past models such as ψt−1,i, ψt−2,i, and later history.
C. Two-stage negotiation and implementation aspects: A multivariate exponentially weighted moving average incorporates gradient history through the hyper-parameter ϱ.With ϱ = 1 only the last available model is used; smaller ϱ values add memory from earlier models and are reported beneficial on real data.
C. Two-stage negotiation and implementation aspects: The simplified algorithm applies one gradient-descent round per received gradient to support asynchronous model updates.A final gradient update on local data follows the received-gradient updates.

D. Communication overhead and complexity analysis

The decentralized algorithms shift aggregation from the server to devices, adding in-network computation and D2D traffic while keeping model-update exchange smaller than dataset transfer under limited neighborhood size.

D. Communication overhead and complexity analysis: Decentralized CFA and CFA-GE reduce server-side load at the cost of additional in-network operations and D2D communication overhead.CFA exchanges local model updates at most once per round, whereas CFA-GE additionally exchanges gradients.
D. Communication overhead and complexity analysis: CFA communication overhead scales with O(d1d2Q |Nk|), where |Nk| is the number of device k’s neighbors plus one.The paper states this remains considerably below training-dataset size when participating neighbors are limited.
D. Communication overhead and complexity analysis: Two active neighbors are sufficient in the reported examples to achieve convergence, while quantization can further reduce transmission payload.The paper notes that active-neighbor counts are typically kept small to avoid traffic issues.
D. Communication overhead and complexity analysis: CFA-GE adds up to |N̄k| neighbor-gradient computations and |N̄k| additional gradient-descent rounds relative to CFA.Both algorithms perform model aggregation and local SGD on devices, unlike federated averaging.
D. Communication overhead and complexity analysis: Communication with a server can be traded against in-network consensus by alternating federated-averaging rounds with CFA or CFA-GE rounds.This supports settings where server connectivity is intermittent, sporadic, unreliable, or costly.

IV. CONSENSUS-BASED FL: AN INTRODUCTORY EXAMPLE

This example evaluates consensus-based federated learning in a four-device multihop network, including IID and uneven non-IID MNIST data. CFA and CFA-GE are compared with federated, isolated, and centralized baselines using validation loss.

Network and setup: The four devices communicate over a multihop topology without central coordination while collaboratively learning a global neural-network model.The neighbor sets form a four-node chain, and all devices begin from the same random initialization.
Data and evaluation: Each device receives 400 IID MNIST images in the uniform-data experiment, partitioning 1,600 training images equally across devices.MNIST inputs are 28 × 28 grayscale images, with ten output classes.
Data and evaluation: Validation loss is evaluated on the full 60,000-image MNIST validation dataset as local model updates approach the global model.The loss is tracked for all four device models over consecutive epochs.
Algorithms and comparisons: CFA-GE uses a two-stage negotiation scheme from epoch t = 3, following four-stage negotiations during initial epochs t = 0, 1, 2.The comparison includes CFA, CFA-GE, federated averaging, isolated training, and centralized non-federated learning.
Non-IID evaluation: Under uneven non-IID data, device 1 has 80 images from six classes while device 3 has 720 images, and convergence is more penalized when this allocation is reversed.CFA on device 3 can still converge, while CFA-GE further reduces loss compared with CFA in Fig. 3.

V. VALIDATION IN AN EXPERIMENTAL IIOT SCENARIO

The paper validates in-network federated learning using real industrial radar data from human-robot collaboration. It evaluates convergence, latency, scalability, connectivity, communication overhead, and computational complexity across CNN and 2NN models.

Industrial scenario: The industrial use case monitors human operators near robotic manipulators with autonomous IIoT devices using 122 GHz FMCW radars.The application targets worker protection in a fenceless human-robot collaboration workspace.
Industrial scenario: Radar data cannot be shipped to a server because worker-safety latency constraints prohibit using the available direct link for data distribution.The setup therefore uses device-to-device connectivity as a replacement for centralized communication infrastructure.
Data collection and processing: Each radar beat signal is transformed into a 512-point FFT spectrum and averaged over 10 consecutive frames before model processing.The resulting FFT samples serve as model inputs.
Data collection and processing: The non-IID dataset contains eight global classes, with most devices holding examples from only a random subset while the aggregate network covers every class.Local datasets represent 1% of the full training database, and local gradient mini-batches contain five samples.
Scalability and evaluation: The evaluation compares CNN and 2NN architectures across networks growing from K = 30 to K = 80 devices and connectivity degrees from 2 to 10 neighbors.CFA and CFA-GE are compared with federated averaging and centralized non-federated learning using validation loss and communication rounds.

B. Gradient exchange optimization for NN

The study evaluates CFA-GE across device counts, network connectivities, models, and gradient-exchange learning rates, showing faster convergence with denser connectivity but greater sensitivity to rate selection.

Experimental setup: CFA-GE is evaluated for K = 30÷80 devices and k-regular networks with 2, 4, 6, or 10 neighbors.Validation loss is averaged over all devices and compared with CFA, FA, and centralized machine learning.
CNN results: For CNN models, CFA-GE is comparable with FA and converges after 50−60 communication rounds, while two neighbors suffice to approach FA performance.Increasing connectivity improves performance and makes convergence closer to centralized learning.
2NN results: For 2NN models, CFA-GE remains comparable with FA mostly after 70 rounds and converges toward centralized ML after 110 rounds.Validation loss is higher than for CNN because the 2NN model has more parameters to train.
Convergence and connectivity: Increasing the network degree accelerates convergence and reduces the communication rounds needed to reach the target validation loss L(val)_t,k ≤0.5.For CNN layers, two neighbors require 9–21 rounds, while six and ten neighbors reduce the requirement to 18 and 14 rounds, respectively.
Learning-rate optimization: For degree-2 networks, learning rates µtβk,i = 0.1 ÷ 0.2 do not affect performance, whereas denser networks benefit from smaller rates µtβk,i ≤0.1.The analysis uses K = 80 devices and degrees 2, 6, and 10.

C. Communication and computational cost assessment

The cost assessment compares communication and computation for FA, CFA, and CFA-GE, finding that CFA-GE achieves centralized-learning performance with higher per-round costs that scale with neighborhood size.

Overall cost: CFA-GE achieves the performance of centralized ML without federation, but requires more intensive D2D communication and local computation.Both costs scale with the number of neighbors |N¯k|.
Communication overhead: CFA-GE communication overhead scales linearly with |N_k|, reaching 2.98·|N_k| Kbyte/round/device for CNN and 33.36·|N_k| Kbyte/round/device for 2NN.The additional exchange reflects cooperation with neighboring devices.
Computational cost: CFA-GE costs more per round than CFA and FA and scales almost linearly with the number of neighbors, while CFA remains comparable with FA when |N¯k| < 10.CFA-GE requires an additional gradient, MEWMA update, and SGD round per neighbor.
Cost-performance trade-off: CFA-GE needs only |N¯k| = 2 neighbors for convergence and reduces communication rounds by almost one order of magnitude compared with CFA.More than two cooperating neighbors provide only marginal improvements and are not recommended in the considered setup.
Deployment implications: CFA-GE is proposed for cases where limiting communication rounds is critical, whereas CFA suits non-critical tasks and bandwidth-limited D2D systems.The execution-time measurements use a 1.5 GHz quad-core ARM Cortex-A72 device with 4 GB RAM.

APPENDIX

The appendix extends consensus-based federated optimization to momentum methods by incorporating exchanged gradients and velocity terms into local updates.

Momentum formulation: Momentum uses both current and past gradients in the local model update, with past contributions weighted by an exponentially decaying function.The momentum or velocity νt,k is stored by device k, and ϱ ∈[0, 1) controls the decay.
CFA and CFA-GE: Momentum can be combined with CFA by replacing its local update, because CFA does not permit gradient sharing.CFA-GE requires additional adaptations because neighboring devices exchange local gradients.
CFA-GE adaptation: In CFA-GE, received gradients are used for the local momentum update before the model update is formed.The method replaces the standard update with eψt,k = ψt,k + νt,k and modifies the gradient-exchange step.
Nesterov adaptation: Nesterov momentum requires devices to exchange locally updated model parameters and gradients instead of the original terms in equation (17).The exchanged model is ψt,k+ϱνt−1,k after applying the velocity term.

Appendix C: Description of Python scripts and datasets

The appendix documents the IIoT radar database and Python scripts for running CNN and 2NN experiments with CFA and CFA-GE under configurable device, neighbor, learning-rate, and epoch settings.

Dataset: The database contains FMCW THz-radar range measurements collected inside a Human-Robot workspace and is importable through Python.The database is stored in the folder dati−radar−05 −07 −2019.
Dataset files: Data_test_2.mat and Data_train_2.mat each contain 16000 512-point FFT range measurements used for testing and training, respectively.The corresponding labels are stored in label_test_2.mat and lable_train_2.mat.
Dataset labels: label_test_2.mat contains 16000 × 1 true labels whose classes are represented by integers from 0 to 7.The labels encode human-worker distance categories relative to the radar.
Python scripts: The scripts support CNN and 2NN models with CFA or CFA-GE methods and expose parameters for learning rates, mixing, device count, neighbors, epochs, and MEWMA.CFA-GE scripts additionally configure separate gradient-exchange rates and the MEWMA parameter.
Example configurations: Example configurations use K=40 devices for CNN and K=80 devices for 2NN, with N=2 neighbors and MEWMA parameter 0.99.The examples set separate hidden- and output-layer learning rates for each model.

Python package description

The Python package provides constructors and controls for initializing consensus processes, applying federated weights, and dynamically enabling or disabling consensus. Its scripts also expose the CNN architecture for modification of filters, strides, and layers.

Package and process initialization: The package provides constructors for CFA and CFA-GE processes, with CFA-GE additionally accepting the parameter ro.The package is available through the listed TestPyPI project.
Package and process initialization: A CFA process can be initialized on device 2 with 2 neighbors in a network of 80 devices.
Federated-weight operations: Federated weights are applied or updated through the getFederatedWeight function.
Consensus control: Consensus can be dynamically enabled or disabled through the disable_consensus function.The function accepts True or False to control the consensus state.
CNN model configuration: The CNN model is defined in the federated_sample_XXX_YYY.py scripts using a one-dimensional convolutional layer followed by a fully connected layer.The shown computation includes max pooling, reshaping, matrix multiplication, and softmax prediction.
CNN model configuration: The CNN architecture can be changed by modifying filter and stride sizes or adding further layers in the specified script lines.