PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning Assisted Federated Learning Algorithm for Data Management of IIoT

Peiying Zhang, Chao Wang, Chunxiao Jiang, Zhu Han

arXiv:2202.03575v1cs.LG

TL;DR

The paper addresses efficient and privacy-conscious management of heterogeneous IIoT data in wireless networks. It combines federated learning with DDPG-based deep reinforcement learning to select equipment nodes for aggregation, achieving more than 98% training accuracy on MNIST and Fashion MNIST and 85% test accuracy on CIFAR-10.

  • Problem

    IIoT generates large volumes of heterogeneous, private data that require efficient management and training while limiting exposure and communication costs.

  • Method

    The paper proposes a federated-learning framework using DDPG-based deep reinforcement learning to select high-quality IIoT equipment nodes for model aggregation.

  • Results

    MNIST and Fashion MNIST achieve more than 98% training accuracy, while CIFAR-10 achieves 85% test accuracy.

  • Takeaways & Limitations

    The authors conclude that DRL-assisted federated learning can effectively manage and train IIoT equipment data while improving aggregation efficiency and reducing communication costs.

Abstract

from arXiv · show

The continuous expanded scale of the industrial Internet of Things (IIoT) leads to IIoT equipments generating massive amounts of user data every moment. According to the different requirement of end users, these data usually have high heterogeneity and privacy, while most of users are reluctant to expose them to the public view. How to manage these time series data in an efficient and safe way in the field of IIoT is still an open issue, such that it has attracted extensive attention from academia and industry. As a new machine learning (ML) paradigm, federated learning (FL) has great advantages in training heterogeneous and private data. This paper studies the FL technology applications to manage IIoT equipment data in wireless network environments. In order to increase the model aggregation rate and reduce communication costs, we apply deep reinforcement learning (DRL) to IIoT equipment selection process, specifically to select those IIoT equipment nodes with accurate models. Therefore, we propose a FL algorithm assisted by DRL, which can take into account the privacy and efficiency of data training of IIoT equipment. By analyzing the data characteristics of IIoT equipments, we use MNIST, fashion MNIST and CIFAR-10 data sets to represent the data generated by IIoT. During the experiment, we employ the deep neural network (DNN) model to train the data, and experimental results show that the accuracy can reach more than 97\%, which corroborates the effectiveness of the proposed algorithm.

I. INTRODUCTION

IIoT produces large volumes of heterogeneous, private data that are difficult to manage efficiently in wireless networks. The paper proposes combining federated learning with deep reinforcement learning to select high-quality equipment models while reducing communication costs.

  • IIoT equipment generates massive data streams, creating challenges for storage, spectrum, computing-resource allocation, and time-series data management.
  • IIoT data contains private information and remains vulnerable across heterogeneous networks, equipment, and malware threats.
  • The proposed framework applies federated learning to train and manage heterogeneous, private IIoT data without directly exposing original data to the central server.
  • Deep reinforcement learning, specifically DDPG, selects IIoT nodes with high-quality local models to increase aggregation rate and reduce communication costs.
  • Experiments use MNIST, Fashion MNIST, and CIFAR-10, with IID and non-IID settings, to evaluate the proposed approach.

B. Related Work of Industrial Internet of Things Based on Federal Learning

The paper situates its work within federated learning for privacy-preserving IIoT data management and distinguishes its approach by using deep reinforcement learning to select high-quality participating equipment nodes. It also reviews federated-learning workflows, loss functions, non-IID data handling, and related applications.

  • Research positioning: The proposed work uses DRL to select high-quality IIoT equipment nodes for FL, aiming to improve model aggregation and reduce communication.The authors identify this node-selection strategy as the main difference from prior work.
  • Federated-learning background: FL enables multiple IIoT equipment nodes to train models locally while preserving the security and privacy of data communication.Local models are uploaded for central optimization rather than raw data being directly exchanged.
  • Federated-learning background: The standard FL process alternates local training, centralized model aggregation, global-model redistribution, and repeated rounds until convergence or a maximum iteration count.The workflow is described as a repeated exchange between equipment nodes, an access point, and the FL unit.
  • Learning objectives: The paper formulates local and global training objectives using equipment-specific loss functions and a weighted average of local losses.The reviewed formulation defines sample-level losses and combines equipment training objectives globally.
  • Data and device setting: The study considers private, large-scale IIoT data, non-IID distributions, and fixed device participation across communication rounds.Devices are randomly grouped, and a DDPG-based DRL algorithm selects participating nodes to improve communication quality.
  • Review scope: The related-work review covers applications combining FL and DRL as well as loss functions used by common machine-learning models.These topics are summarized in Tables I and II.

IV. IMPLEMENTATION OF FEDERATED LEARNING ALGORITHM ASSISTED BY DRL

This section presents the implementation of the DRL-assisted FL algorithm, covering its framework, algorithmic steps, and DRL-based IIoT equipment selection process.

  • Implementation overview: The DRL-assisted FL framework is identified as the core technology for training IIoT equipment data.The section then develops the algorithm implementation and equipment-selection process based on this framework.
  • Implementation overview: The implementation is organized around a DRL-assisted FL framework followed by detailed algorithm steps and DRL-based equipment selection.The stated sequence moves from framework design to algorithm realization and then node selection.
  • Equipment selection: The equipment-selection stage applies DRL to determine which IIoT nodes participate in federated learning.The passage presents this selection process as the final implementation component.

A. Framework

The proposed framework integrates DRL with FL to coordinate parameter exchange while keeping IIoT data training local. It uses distributed deep learning to improve edge training efficiency and help limit network congestion.

  • Framework motivation: DRL is integrated with FL to use cooperation among IIoT equipment and nodes for exchanging learning parameters and training local models.Local training avoids directly transmitting large amounts of data from resource-constrained edge equipment.
  • Framework motivation: The framework is designed for settings where edge equipment has limited cache and computing performance and direct data transmission may cause congestion or data leakage.These constraints motivate local data training and the DRL-assisted federated design.
  • Framework design: The framework adopts distributed deep learning because it offers fast training and edge efficiency for IIoT data processing.The authors connect faster edge training with avoiding network congestion to some extent.

B. Implementation Steps

The framework proceeds through device scheduling, local model training, parameter aggregation, and repeated global-model updates. It includes random, cyclic, and proportional-fairness scheduling strategies for selecting IIoT devices and managing wireless communication.

  • The DRL-assisted FL framework has initialization, training, and aggregation stages repeated until convergence or a maximum iteration count.The server selects participating devices, sends the global model, receives local models, and updates the next global model.
  • Model aggregation: Selected IIoT devices compute local gradients, upload their models or gradients, and the server aggregates them for the next global update.
  • Scheduling strategies: Random scheduling selects N IIoT devices per communication round and establishes a dedicated sub-channel for parameter transmission.
  • Scheduling strategies: Cyclic scheduling partitions devices into G groups and selects one group per communication round for cyclic model updates.
  • Scheduling strategies: Proportional fairness selects N devices from C candidates using instantaneous and time-average SNRs.The SNR formulation accounts for link loss, small-scale fading, Gaussian noise variance, and device position.

C. IIoT Equipment Node Selection Based on DRL

The paper formulates IIoT device selection as a cost-sensitive Markov decision process and uses DDPG to choose devices while balancing training time, communication cost, and learning quality.

  • Cost formulation: The selection cost combines local training time, communication cost, and a learning-quality indicator based on training loss.The formulation accounts for the local training and communication costs of selected devices and the loss used to quantify model quality.
  • IIoT device selection is modeled as a combinatorial optimization problem represented by an MDP with state, action, transition-probability, and cost components.
  • DDPG selection: The DRL agent interacts with the environment and uses DDPG to select an optimal IIoT equipment node solution.
  • DDPG selection: The reward evaluates an action using model-training computation requirements, available device resources, and wireless-channel transmission rate.
  • DDPG selection: DDPG uses actor and critic DNNs, replay-memory experiences, and a target network to update its policy and value estimates.Stored experiences include the current state, action, next state, and received reward.

V. EXPERIMENTAL SETUP AND RESULT ANALYSIS

The study evaluates the DRL-assisted FL algorithm on MNIST, Fashion MNIST, and CIFAR-10 under IID and non-IID settings, focusing on training accuracy. The results indicate advantages for training IIoT equipment data.

  • The evaluation uses MNIST, Fashion MNIST, and CIFAR-10 datasets with both IID and non-IID distributions.
  • The experiments mainly measure algorithm accuracy when training data representing IIoT equipment data.
  • The reported results show advantages for the DRL-assisted FL algorithm in training IIoT equipment data.

A. Experimental Setup

The experiments use small agent datasets, neural networks matched to each dataset, and IID or non-IID data partitions across IIoT equipment. CIFAR-10 is distributed equally across 100 devices.

  • Dataset models: MNIST is trained with a two-hidden-layer MLP containing 200 units per layer and ReLU activations.
  • Dataset models: Fashion MNIST is trained with a CNN containing two 5 × 5 convolution layers, a 512-unit ReLU fully connected layer, and a softmax output layer.
  • Data partitioning: IID data are shuffled and divided across 100 equipment, with each equipment receiving 600 samples.
  • Data partitioning: Non-IID data are divided into 200 pieces from 300 class-based pieces, with each client allocated two pieces.

B. Performance Evaluation

The evaluation examines how client participation, training settings, and datasets affect federated-learning performance. Across MNIST, Fashion MNIST, and CIFAR-10, the DRL-assisted approach reaches high accuracy and shows convergent training behavior.

  • Client participation: Client proportion C is evaluated on MNIST with E = 1 and Fashion MNIST with E = 5, using communication rounds as the unit.The study compares different numbers of participating clients to assess training efficiency.
  • Accuracy and convergence: 99% MNIST accuracy is reached after about 1,000 communication rounds and then stabilizes.The MNIST experiments use IID data with C = 0.1.
  • Accuracy and convergence: 97% Fashion MNIST training accuracy is reached after about 2,000 communication rounds and stabilizes.The result is reported for the Fashion MNIST training experiments.
  • Accuracy and convergence: Loss decreases and converges with communication rounds for MNIST under both IID and non-IID settings, stabilizing at a low level after about 1,000 rounds.Fashion MNIST loss also converges similarly as communication rounds increase.
  • Cross-dataset results: CIFAR-10 reaches 85% test accuracy, while MNIST and Fashion MNIST exceed 98% training accuracy in the reported experiments.The datasets represent IIoT equipment data and are trained using MLP and CNN models.
  • Training settings: For a 98% target accuracy, increasing E and reducing B generally lowers the communication rounds required, with a clearer effect for IID data.The overall optimization effect is less obvious for non-IID data.

VI. CONCLUSION

The conclusion addresses IIoT data-management and training challenges caused by explosive user-data growth. It proposes DRL-assisted federated learning for node selection and reports effective training of representative datasets with high accuracy.

  • Motivation: IIoT faces management and training problems from the explosive growth of user data.The conclusion frames this as the paper’s central problem.
  • Approach: The proposed method uses DRL based on DDPG to select IIoT equipment nodes within a federated-learning algorithm.MNIST, Fashion MNIST, and CIFAR-10 represent IIoT equipment data in the experiments.
  • Conclusion: The reported results show effective training of the three datasets and high accuracy for managing IIoT equipment data.The conclusion characterizes the results as evidence of the algorithm’s effectiveness.

BIOGRAPHIES

The biographies identify the paper’s authors and describe their academic positions, research interests, and publication or service backgrounds.

  • Authors: Peiying Zhang is an associate professor at China University of Petroleum (East China) with research and publication experience in networking and IoT-related areas.His listed interests include virtual network embedding, network AI, deep reinforcement learning, future network architecture, IoT, and wireless communication.
  • Authors: Chao Wang is a master’s student in computer science and technology at China University of Petroleum (East China).The biography notes publications and participation in international conferences.
  • Authors: Chunxiao Jiang is an associate professor at Tsinghua University and has served as an editor or guest editor for several IEEE venues.His research includes game theory, optimization, statistics, communication, networking, and resource allocation.
  • Authors: Zhu Han’s biography begins with his electrical-engineering education at Tsinghua University and the University of Maryland, followed by research and academic appointments.The supplied biography passage provides his educational and early professional background.
Loading 2202.03575v1…