PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT

Zonghang Li, Yihong He, Hongfang Yu, Jiawen Kang, Xiaoping Li, Zenglin Xu, Dusit Niyato

arXiv:2202.01512v1cs.LGcs.AIcs.DC

TL;DR

IIoT federated learning faces skewed, changing local data distributions, while existing approaches may overlook factory clustering and security requirements. FEDGS uses GBP-CS to form homogeneous super nodes, and experiments report higher accuracy and fewer training rounds on average.

  • Problem

    IIoT devices have heterogeneous, changing local data distributions, while prior approaches may overlook streaming data and factory clustering and raise data-security concerns.

    Source: §I · pp. 1–2

  • Method

    FEDGS uses GBP-CS to select device subsets and form homogeneous FL super nodes within a hierarchical cloud-edge-end framework.

    Source: B. Gradient-based Binary Permutation… · p. 7, §VIII · p. 12

  • Results

    FEDGS reaches 86.0% accuracy, 3.5% higher than averaged accuracy, and reaches 82% in 147 rounds, reducing rounds by 59% on average.

    Source: B. Results and Discussion · p. 12

  • Takeaways & Limitations

    GBP-CS is 66× faster than GA, with 15 milliseconds execution time and a distribution-divergence loss of only 0.001.

    Source: B. Results and Discussion · p. 10

Abstract

from arXiv · show

Nowadays, the industrial Internet of Things (IIoT) has played an integral role in Industry 4.0 and produced massive amounts of data for industrial intelligence. These data locate on decentralized devices in modern factories. To protect the confidentiality of industrial data, federated learning (FL) was introduced to collaboratively train shared machine learning models. However, the local data collected by different devices skew in class distribution and degrade industrial FL performance. This challenge has been widely studied at the mobile edge, but they ignored the rapidly changing streaming data and clustering nature of factory devices, and more seriously, they may threaten data security. In this paper, we propose FedGS, which is a hierarchical cloud-edge-end FL framework for 5G empowered industries, to improve industrial FL performance on non-i.i.d. data. Taking advantage of naturally clustered factory devices, FedGS uses a gradient-based binary permutation algorithm (GBP-CS) to select a subset of devices within each factory and build homogeneous super nodes participating in FL training. Then, we propose a compound-step synchronization protocol to coordinate the training process within and among these super nodes, which shows great robustness against data heterogeneity. The proposed methods are time-efficient and can adapt to dynamic environments, without exposing confidential industrial data in risky manipulation. We prove that FedGS has better convergence performance than FedAvg and give a relaxed condition under which FedGS is more communication-efficient. Extensive experiments show that FedGS improves accuracy by 3.5% and reduces training rounds by 59% on average, confirming its superior effectiveness and efficiency on non-i.i.d. data.

I. INTRODUCTION

IIoT federated learning must address skewed, changing device data while preserving industrial confidentiality. The paper proposes device selection and grouped synchronization, and reports improved accuracy and fewer training rounds alongside a review of related approaches.

I. INTRODUCTION

  • Industrial devices have skewed class distributions that can degrade FL, and changing sensor data makes the non-i.i.d. challenge more difficult.Prior mobile-edge work is limited for IIoT's security requirements, streaming data, and natural factory clustering; existing IIoT approaches also omit changing distributions and geographic clustering,,.Source: §I · pp. 1–2
  • GBP-CS selects device subsets to form homogeneous FL super nodes, while FEDGS combines synchronization within and among groups to address data heterogeneity.The paper analyzes convergence and optimality gap, reporting better convergence bounds and a relaxed condition for greater time efficiency than FedAvg.Source: §I · pp. 2–3
  • 3.5% accuracy improvement and 59% fewer training rounds on average versus 10 advanced approaches.The experiments compare FEDGS with 10 approaches and report these average improvements.Source: §I · p. 3

II. RELATED WORK

  • Data-sharing methods distribute shared data [9]–, while augmentation methods rebalance classes,,; the paper notes privacy and credibility risks.The cited concerns include possible private-data leakage, synthetic data that can conceal clients' original data, and misleading claims about its volume or quality.Source: §II · p. 3
  • Hyperparameter studies recommend one local iteration, but resource constraints encourage larger iteration counts, and learning-rate decay is needed for convergence in that setting.Careful tuning of local iterations, learning rates, and decay rates incurs high time costs.Source: §II · p. 3
  • Client-side adaptation methods add storage and computation costs –,, while adaptive server optimizers require careful initial learning-rate tuning.The paper also reports drastic accuracy oscillation for adaptive federated optimizers in its experiments.Source: §II · pp. 3–4

III. SYSTEM MODEL

FedGS organizes devices by factory in a 5G cloud-edge-end architecture, then forms selected homogeneous super nodes and synchronizes locally before periodic cloud aggregation.

IV. FEDGS: FRAMEWORK AND WORKFLOW

  • Each base station selects L devices to form a super node; selected devices train on one mini-batch, synchronize locally, and repeat these steps T times before server synchronization.The top server aggregates base-station models and synchronizes the updated model back to the base stations.Source: Figure 2 · p. 4, §IV · p. 5

III. SYSTEM MODEL

  • Factories form device groups in a 5G network, with devices sending model data to nearby base stations that communicate with the cloud center.The setting includes fixed and mobile industrial devices and real-time local data streams with heterogeneous class distributions.Source: §III · p. 4
  • The system targets heterogeneous streaming data, which degrades the traditional workflow that trains sampled devices for several local epochs before aggregation.The paper identifies this workflow as insufficiently attentive to industrial data's streaming nature.Source: §III · p. 4

IV. FEDGS: FRAMEWORK AND WORKFLOW

  • One-step synchronization within groups is combined with less-frequent multi-step synchronization across base stations to address heterogeneity while retaining communication efficiency.The group protocol uses one local update before each synchronization; the top server aggregates after every T iterations.Source: §IV · p. 5

V. CLIENT SELECTION VIA GBP-CS

The section formulates client selection as constrained 0-1 optimization and introduces GBP-CS as a fast approximate solver.

V. CLIENT SELECTION VIA GBP-CS

  • GBP-CS is proposed to solve the client-selection problem in an acceptable short time with a desirable solution.The optimization is formulated as a 0-1 integer program with vector weight constraints.Source: §V · p. 5

A. Problem Modeling

Problem modeling casts device selection within each factory as constrained binary optimization, while random pre-sampling and stated hardware assumptions shape the formulation.

A. Problem Modeling

  • The model selects L devices from each factory group to form a super node, framing client selection as a constrained 0-1 optimization.The objective is to minimize distribution divergence, with a constraint on the number selected.Source: A. Problem Modeling · p. 6
  • Randomly pre-sampling Lrnd devices before choosing the remaining Lsel devices preserves randomness so each device has the same selection probability.The remaining candidates are selected from the devices not included in the random pre-sample.Source: A. Problem Modeling · p. 6
  • The formulation considers data size and data quality but assumes similar device hardware capabilities; ESync is noted as compatible when system heterogeneity matters..Source: A. Problem Modeling · p. 6
  • The constrained selection variant is at least NP-complete, motivating a fast sub-optimal solution rather than polynomial-time optimal search.The proof reduces Problem A to Problem B in polynomial complexity.Source: A. Problem Modeling · p. 6

B. Gradient-based Binary Permutation Client Selection

GBP-CS iteratively swaps binary selection variables to reduce distribution mismatch while preserving the selection-count constraint.

B. Gradient-based Binary Permutation Client Selection

  • GBP-CS swaps selected and unselected binary variables with opposite key gradients, maintaining the number selected while iteratively minimizing the objective.The unselected variable with the smallest gradient is paired with the selected variable with the largest gradient.Source: B. Gradient-based Binary Permutation… · pp. 6–7
  • The permutation iterations stop when the objective distance no longer decreases, after which the selected-device set is constructed.The method returns the selected devices for the super node.Source: B. Gradient-based Binary Permutation… · p. 7
  • GBP-CS has complexity O(F^3 + αF^2 + α^2Fτ + L) and is reported to produce a near-optimal solution with execution efficiency comparable to random sampling.Here α = Km − Lrnd and τ is the number of GBP-CS iterations.Source: B. Gradient-based Binary Permutation… · p. 7

VI. PERFORMANCE ANALYSIS

The section analyzes FEDGS’s optimality gap and convergence rate, compares them qualitatively with FedAvg under non-i.i.d. data, and considers when FEDGS is time-efficient.

VI. PERFORMANCE ANALYSIS

  • The analysis considers FEDGS’s optimality gap and convergence rate under non-i.i.d. data.It also qualitatively compares these properties with FedAvg.Source: §VI · p. 7
  • FEDGS is qualitatively compared with FedAvg in the presence of non-i.i.d. data.Source: §VI · p. 7
  • The section gives a condition under which FEDGS is time-efficient.Source: §VI · p. 7

A. Convergence Analysis

The analysis links class-distribution divergence to gradient divergence and argues that FEDGS has tighter convergence bounds and a smaller optimality gap than FedAvg under the stated assumptions.

A. Convergence Analysis

  • The analysis assumes local functions are μ-strongly convex, β-smooth, and ρ-Lipschitz.Source: A. Convergence Analysis · p. 7
  • Smaller class-distribution divergence corresponds to a smaller upper bound on gradient divergence, which captures the impact of distribution mismatch.Source: A. Convergence Analysis · p. 7
  • Proposition 3 gives a convergence upper bound and an optimality-gap bound for FEDGS.Source: A. Convergence Analysis · p. 7
  • FEDGS is theoretically equivalent to FedAvg with M super nodes using mini-batch SGD of batch size nL for T local iterations per round.Source: A. Convergence Analysis · p. 7
  • FEDGS has a smaller gradient-divergence upper bound than FedAvg, yielding smaller convergence bounds and optimality gap and potentially improving convergence speed and accuracy.Source: A. Convergence Analysis · p. 7

B. Time-Efficiency Condition

The section models FEDGS and FedAvg round time and states a condition under which FEDGS is faster, assuming equal uplink and downlink bandwidths and equal SNRs.

B. Time-Efficiency Condition

A. Experiment Setup

Experiments model federated OCR across factories using FEMNIST, compare client-selection and FL baselines, and test GBP-CS initialization choices.

A. Experiment Setup

  • The OCR experiment uses FEMNIST’s 805,263 images across 3,550 non-i.i.d. devices and selects 10 of 35 cameras in each of 10 factories per iteration.Training uses a four-layer CNN, mini-batch SGD, learning rate 0.01, batch size 32, and 50 iterations per round unless otherwise specified.Source: A. Experiment Setup · p. 8
  • GBP-CS selects eight devices after two are randomly pre-sampled, and the experiment tests random, zero, and Moore-Penrose inverse initialization.Source: A. Experiment Setup · p. 9
  • The evaluation compares GBP-CS with random, Monte Carlo, brute-force, Bayesian, and genetic client samplers, plus FedAvg and nine advanced FL approaches.Source: A. Experiment Setup · p. 9

B. Results and Discussion

Across sampler, hyperparameter, and baseline comparisons, FEDGS shows strong accuracy and training efficiency on non-i.i.d. data. GBP-CS finds low-divergence device selections quickly, while FEDGS outperforms the compared federated approaches.

GBP-CS initialization

  • MPInv and Zero initialization reach divergence values of 0.030 and 0.029, close to brute force's 0.028; Random reaches 0.044.MPInv is faster than Zero because it requires no additional warm-up, and is GBP-CS's default initializer.Source: B. Results and Discussion · p. 9

GBP-CS and other samplers

  • GBP-CS and GA are the best samplers, with distribution divergence ranges of 0.029 ∼0.042 and 0.028 ∼0.041, respectively.Random sampling yields 0.072 ∼0.105, while brute force yields 0.026 ∼0.038.Source: B. Results and Discussion · pp. 9–10
  • GBP-CS runs 66× faster than GA in 15 milliseconds, with a distribution-divergence loss of only 0.001.GBP-CS converges to divergence 0.029, close to the 0.028 optimum, in the shortest time.Source: B. Results and Discussion · p. 10

Hyperparameter effects

  • Accuracy improves with moderately larger T and with more groups M or selected devices L, while batch size n has little effect.The study uses n = 32, T = 50, and L = 10 by default; M = 10 is determined by the real-world environment.Source: B. Results and Discussion · p. 10

Comparisons with FedProx and IDA

  • FedProx reaches 82.0% versus FedAvg's 82.1%, while IDA-series approaches reach 80.5% ∼81.0%; FEDGS reaches 86.0%.The IDA discussion also notes server memory costs from caching parameters uploaded by all devices.Source: B. Results and Discussion · pp. 10, 12

Comparisons with CGAU, FedMMD, and FedFusion

  • FEDGS is 2.7% above 83.3% FineTunning+1×CGAU and 3% above FedMMD's 83.0%; FedFusion variants reach 82.0%, 81.7%, or fall 1.4%.FEDGS also has lower test loss and faster convergence than FineTunning+1×CGAU.Source: B. Results and Discussion · p. 12

Comparison with FedOpt

  • FedAvgM, FedAdagrad, FedAdam, and FedYogi improve baseline accuracy by 1.7% ∼2.9%, while FEDGS accuracy is still 1% higher.These adaptive methods are sensitive to initial learning rates, which the experiments tune.Source: B. Results and Discussion · p. 12

FEDGS vs. other federated approaches

  • FEDGS achieves 86.0% accuracy, 3.5% above averaged accuracy, and reaches 82% in 147 rounds, reducing training rounds by 59% on average.Table II compares test accuracy, test loss, and convergence speed against ten federated approaches.Source: Table II · p. 10, B. Results and Discussion · p. 12

VIII. CONCLUSION

FEDGS combines device selection and hierarchical synchronization to address heterogeneous industrial data. Theoretical analysis and experiments support its convergence and performance advantages over FedAvg and other approaches.

VIII. CONCLUSION

  • FEDGS uses GBP-CS to select devices within factories and form homogeneous FL super nodes, then coordinates training with compound-step synchronization.One-step synchronization within super nodes suppresses heterogeneity's negative impact; multi-step synchronization among super nodes reduces communication frequency.Source: §VIII · p. 12
  • FEDGS uses factory-device geographic clustering and adapts at runtime to rapidly changing streaming data without exposing confidential data in high-risk manipulation.Source: §VIII · p. 12
  • Theoretical analysis finds better convergence rate and optimality gap than FedAvg, with greater time efficiency under a relaxed hyperparameter condition; experiments report state-of-the-art performance on non-i.i.d. data.Source: §VIII · p. 13

Related papers

Loading 2202.01512v1…