Source-linked AI summary
Computational Intelligence and Deep Learning for Next-Generation Edge-Enabled Industrial IoT
Shunpu Tang, Lunyuan Chen, Ke HeJunjuan Xia, Lisheng Fan, Arumugam Nallanathan
TL;DR
Resource-limited IoT devices can struggle to train deep models and upload updates within latency constraints. ME-FEEL combines multi-exit training with greedy exit selection and bandwidth allocation. In Fashion-MNIST simulations, it outperforms conventional FEEL in resource-limited settings, while distillation improves the first exit’s accuracy by 6% over ME-FEEL without KD.
Problem
In resource-constrained IoT networks, devices may be unable to train a deep model or upload weights within the latency limit, so their updates are discarded.
Source: §I · p. 1Method
ME-FEEL lets devices train to selected exits and uses greedy exit adjustment with bandwidth allocation to meet resource constraints.
Source: §20 · p. 6, §VI · p. 10Results
In Fashion-MNIST results, ME-FEEL with the first exit gains 6% accuracy over ME-FEEL without KD.
Source: C. Simulation results and discussion · p. 8Takeaways & Limitations
Choosing early exits reduces training time to 4ms and can reduce model size below 5MB for the first three exits.
Source: C. Simulation results and discussion · p. 8
Abstract
from arXiv · showhide
In this paper, we investigate how to deploy computational intelligence and deep learning (DL) in edge-enabled industrial IoT networks. In this system, the IoT devices can collaboratively train a shared model without compromising data privacy. However, due to limited resources in the industrial IoT networks, including computational power, bandwidth, and channel state, it is challenging for many devices to accomplish local training and upload weights to the edge server in time. To address this issue, we propose a novel multi-exit-based federated edge learning (ME-FEEL) framework, where the deep model can be divided into several sub-models with different depths and output prediction from the exit in the corresponding sub-model. In this way, the devices with insufficient computational power can choose the earlier exits and avoid training the complete model, which can help reduce computational latency and enable devices to participate into aggregation as much as possible within a latency threshold. Moreover, we propose a greedy approach-based exit selection and bandwidth allocation algorithm to maximize the total number of exits in each communication round. Simulation experiments are conducted on the classical Fashion-MNIST dataset under a non-independent and identically distributed (non-IID) setting, and it shows that the proposed strategy outperforms the conventional FL. In particular, the proposed ME-FEEL can achieve an accuracy gain up to 32.7% in the industrial IoT networks with the severely limited resources.
I. INTRODUCTION
The introduction motivates federated edge learning for resource-constrained IoT devices that must train and upload within latency limits. It presents ME-FEEL and greedy exit and bandwidth selection as responses to those constraints, and reports better performance than conventional strategies in simulation.
I. INTRODUCTION
- Resource-limited devices may fail to train a deep model or upload its weights within the latency constraint, causing their updates to be discarded.Devices differ in computational power and channel state, while communication bandwidth is limited.Source: §I · pp. 1–2
- ME-FEEL lets devices with limited computational power train part of a deep model, and layer-wise model averaging aggregates models of different sizes.Source: §I · p. 2
- The greedy joint exit-selection and bandwidth-allocation approach aims to maximize the number of exits that upload successfully in each FL round.The objective accounts for upload time, which can be critical in bandwidth-limited IoT networks.Source: §I · p. 2
- Simulations report that ME-FEEL outperforms conventional strategies and can be flexibly deployed in edge-enabled industrial IoT networks.The evaluation uses a popular model and practical dataset.Source: §I · p. 2
II. RELATED WORKS
Related work covers AI for optimizing edge systems, AI deployed on edge devices, and federated-learning resource management. The paper then introduces the federated edge learning setup, where devices train locally and upload updates for server aggregation.
II. RELATED WORKS
- Prior edge-intelligence work addresses both AI for edge optimization and AI-on-edge deployment, including inference partitioning, adaptive inference, and cooperative computing.The cited examples include CoEdge, which reduced energy consumption by up to 25%.Source: §II · p. 2
- FL research in edge IoT has addressed client selection, uplink quantization, bandwidth allocation, device scheduling, and battery-aware resource use.Source: §II · p. 2
A. Federated edge learning system
- In FEEL, K IoT devices hold local datasets and train a shared model with an edge server to minimize a global objective.Devices train locally and upload updates for server aggregation; practical rounds may select only a subset of devices.Source: A. Federated edge learning system · p. 3
B. The proposed multi-exit approach
ME-FEEL uses exits at different depths to let devices train only the model portion their resources permit. Devices select exits based on their current state, and the local objective combines the selected exits' losses.
B. The proposed multi-exit approach
- The multi-exit architecture places prediction exits at different depths, allowing a device to train a sub-model rather than the full deep model.Exits share weights, with few additional parameters; deeper exits consume more resources and incur greater latency.Source: B. The proposed multi-exit approach · p. 3
- Each device chooses up to M exits and trains the continuous model portion through its selected exit according to its computational power and channel state.Source: B. The proposed multi-exit approach · p. 3
- The local training loss sums the prediction losses from all exits selected by a device.Source: 1) Local training · p. 4
1) Local training:
Local training combines prediction loss with knowledge distillation, using more capable exits as teachers for earlier exits. The teacher target is formed from the ensemble of exits.
1) Local training:
- Knowledge distillation addresses the weaker fitting ability of early exits by transferring knowledge from later exits that often perform better.The paper models the difference between teacher and student outputs using KL divergence.Source: 1) Local training · p. 4
- The teacher output is the ensemble of all exits, which the paper associates with stronger generalization and reduced overfitting risk.Source: 1) Local training · p. 4
- The local objective jointly uses prediction loss and KD loss, with mean-square error for regression and cross-entropy for classification prediction loss.Source: 1) Local training · pp. 4–5
2) Model aggregation:
ME-FEEL aggregation must accommodate devices uploading models with different numbers of exits. The section describes layer-wise aggregation and a joint exit-selection and bandwidth-allocation strategy.
2) Model aggregation:
- Layer-wise ME-FedAvg averages each shared layer across uploaded models that contain it, accommodating updates of different sizes.The edge server searches the uploaded updates layer by layer, using the initial global model as a reference.Source: 2) Model aggregation · p. 5
- For each global-model layer, the update is formed from devices that have that layer.The devices with the l-th layer form the subset C_l used for the layer-wise update.Source: 2) Model aggregation · p. 5
IV. EXIT SELECTION AND BANDWIDTH ALLOCATION
- The framework combines models with different sizes and uses joint exit selection and bandwidth allocation to adapt participation to network conditions.The procedure is summarized in Algorithm 2, with the joint strategy identified as a later step.Source: 2) Model aggregation · p. 5, §IV · p. 5
A. Latency constrained model
The latency-constrained model accounts for local training and wireless transmission within a shared time limit. Its transmission component depends on channel bandwidth and the modeled data rate.
A. Latency constrained model
- Each device must upload its weights within latency threshold γ_th, under total communication bandwidth B_A.The upload duration includes local-training and transmission latency.Source: A. Latency constrained model · p. 5
- Local-training latency depends on the number of exits and the size of the device's local dataset.The passage characterizes this latency using the device's computational coefficient, batch size, and a mapping from exit count to one-batch training time.Source: A. Latency constrained model · p. 5
- Transmission latency is modeled from the wireless data rate, which is determined using allocated bandwidth and channel parameters.The passages identify transmit power, channel parameter, and noise variance among the rate-model inputs, and relate exit count to transmitted bits.Source: A. Latency constrained model · p. 5
B. Problem formulation
The formulation seeks to maximize uploaded exits subject to a latency threshold, while the proposed heuristic addresses the bandwidth-constrained optimization problem.
B. Problem formulation
- The objective is to maximize the number of exits whose updates upload without violating latency threshold γ_t.The formulation also identifies K_s as the set of devices that upload updates successfully.Source: B. Problem formulation · p. 6
- The formulation uses B_A for total communication bandwidth.Source: B. Problem formulation · p. 6
C. Optimization strategy
- The optimization problem is a variant of knapsack and is described as NP-complete, motivating a greedy heuristic.The passage notes that pseudo-polynomial-time solving is possible but proposes the heuristic to facilitate solving the problem.Source: C. Optimization strategy · p. 6
- For exit index m, the algorithm determines the minimum required bandwidth B_k(m).The passages introduce a bandwidth requirement for uploading m exits and identify B_k(m) as the minimum required bandwidth.Source: C. Optimization strategy · p. 6
- Each device checks candidate exits from the latest exit toward the first, using B_k(m) to assess feasibility within the latency threshold.The algorithm uses the threshold γ_th in this check.Source: C. Optimization strategy · p. 6
- A positive B_k(m) indicates local training can finish within γ_th; a non-positive value indicates it cannot.Source: C. Optimization strategy · p. 6
20 Output {Bk, Mk}
The greedy scheduler first checks device exit choices, then adjusts exits until aggregate bandwidth fits the constraint.
20 Output {Bk, Mk}
- Each device checks exits from latest to earliest and selects a preliminary exit using its required bandwidth.The check also tests whether local training fits within the latency threshold.Source: §20 · p. 6
- If total bandwidth exceeds the constraint, the server reduces the exit of the device with the smallest M_k-to-B_k(m) ratio.Source: §20 · p. 6
- The server repeats greedy exit adjustments until the bandwidth constraint is satisfied.Source: §20 · p. 6
V. SIMULATION AND DISCUSSION
The section introduces simulations of ME-FEEL in an edge-IoT network and specifies an image-classification task using Fashion-MNIST with a non-IID training partition.
V. SIMULATION AND DISCUSSION
- The simulations evaluate ME-FEEL, first describing the simulation setting and implementation details before presenting results and discussion.Source: §V · p. 6
A. Simulation Settings
- The simulated IoT network contains one edge server at the base station and 100 edge devices, each with its own dataset and able to participate in federated learning.Source: A. Simulation Settings · p. 6
1) Task and Dataset:
- The image-classification task uses Fashion-MNIST with 10 classes, 5,000 training samples, and 1,000 test samples, divided among 100 devices under a non-IID setting.Training data are sorted by class and divided into 200 groups of 300 samples, with two groups assigned to each device.Source: 1) Task and Dataset · pp. 6–7
- The modified ResNet-18 architecture, named ME-ResNet, has seven exits.Source: Table I · p. 7
2) Multi-exit FL:
The simulations test ME-FEEL under constrained network conditions and report accuracy advantages over conventional FEEL methods. Early exits reduce training time and model size, while self-knowledge distillation improves the performance of earlier exits.
3) Network environment / B. Competitive methods
- The evaluation uses LTE with 40 MHz available bandwidth and compares FEEL-ideal, constrained FEEL, and FEEL-UB, which allocates bandwidth evenly.FEEL-ideal has no constraints; constrained FEEL uses the bandwidth method from, which allocates first to clients requiring less bandwidth.Source: 3) Network environment · p. 7, B. Competitive methods · p. 7
C. Simulation results and discussion
- At the first exit, ME-ResNet and ME-ResNet-KD train in 4 ms, versus about 14 ms for original ResNet; KD adds at most 1 ms across exits 2–7.At exit 4, both multi-exit models reduce training time to half the original ResNet’s; at the last exit, ME-ResNet-KD takes 16.9 ms and ME-ResNet 14.70 ms.Source: Figure 5 · p. 7, C. Simulation results and discussion · p. 8, Figure 7 · p. 9
- At the first three exits, ME-ResNet models are each under 5 MB, about ten times smaller than original ResNet; the last-exit sizes are 44.7 MB and 44.6 MB, respectively.Fig. 6 compares model size by exit index.Source: C. Simulation results and discussion · p. 8, Figure 6 · p. 8
- ME-FEEL outperforms FEEL and FEEL-UB in accuracy and convergence rate, with more stable training; early exits also outperform both single-exit methods in the experiments.Later exits transfer knowledge to earlier ones through KD, which the passage says helps improve performance; Fig. 7 plots test accuracy over communication rounds.Source: C. Simulation results and discussion · p. 8, Figure 7 · p. 9
- On Fashion-MNIST, ME-FEEL without KD reaches 84% maximum accuracy, versus 80.8% for FEEL and 87% for unconstrained FEEL-ideal.The comparison uses non-IID data; the other resource-limited methods use 40 MHz bandwidth and a 15 s latency threshold.Source: C. Simulation results and discussion · p. 8, Table II · p. 9
- Up to 32.7% accuracy gain over FEEL-UB is reported as the latency threshold tightens on Fashion-MNIST with 40 MHz bandwidth.The reported gain over FEEL is 2.68% at a 12 s threshold; Fig. 8 varies the threshold from 12 s to 15 s.Source: C. Simulation results and discussion · p. 9, Figure 8 · p. 10
- At 35 MHz, ME-FEEL’s accuracy gain reaches 20.75%, and its degradation with reduced bandwidth is smaller than the other methods’.At 50 MHz, its gains over FEEL and FEEL-UB are 2.6% and 10.5%, respectively; Fig. 9 compares bandwidth from 35 MHz to 50 MHz.Source: C. Simulation results and discussion · pp. 9–10, Figure 9 · p. 10
VI. CONCLUSIONS
ME-FEEL combines early exits, knowledge distillation, and greedy exit-and-bandwidth allocation for resource-limited federated edge learning. Fashion-MNIST simulations under non-IID data showed it outperformed conventional FEEL, while unlabeled data, streaming data, and distributed inference remain challenges.
VI. CONCLUSIONS
- ME-FEEL adds multiple exits so devices can train only part of a model under latency constraints, with knowledge distillation addressing performance degradation from early exits.A greedy joint exit-selection and bandwidth-allocation algorithm aims to maximize the expected number of exits per communication round.Source: §VI · p. 10
- Simulations on Fashion-MNIST with non-IID data showed ME-FEEL outperformed conventional FEEL in resource-limited IoT networks.Source: §VI · p. 10
- Unlabeled samples, streaming data, and more efficient distributed inference remain challenges for deployment in edge-enabled industrial IoT networks.The paper identifies reducing inference latency and energy consumption as a goal for such protocols.Source: §VI · p. 10