Source-linked AI summary
Evaluating Federated Learning for Intrusion Detection in Internet of Things: Review and Challenges
Enrique Mármol Campos, Pablo Fernández Saura, Aurora González-Vidal, José L. Hernández-Ramos, Jorge Bernal Bernabe, Gianmarco Baldini, Antonio Skarmeta
TL;DR
IoT intrusion-detection research needs alternatives to centralized data sharing, while FL-enabled IDS evidence remains limited and often uses unrealistic settings. The paper evaluates multiclass FL with three ToN IoT data partitions and aggregation methods, then derives deployment challenges and future directions from the results and literature.
Problem
FL-enabled IDS for IoT lacks comprehensive evidence under non-iid data, realistic distributions, and multiclass attack detection, despite centralized approaches raising privacy concerns.
Method
The study partitions the ToN IoT dataset by destination IP address, attack-type balance, and a hybrid Shannon-entropy criterion, then evaluates multiclass FL with aggregation methods using IBMFL.
Results
The evaluation demonstrates that non-iid and highly skewed data distributions affect FL performance and the effectiveness of security attack detection.
Takeaways & Limitations
FL-enabled IoT IDS development must account for data distributions and address deployment challenges identified through evaluation and literature analysis.
Takeaways & Limitations
The study uses a simulated testbed, while real IoT deployment must address constrained devices’ memory, computing power, and energy consumption.
Abstract
from arXiv · showhide
The application of Machine Learning (ML) techniques to the well-known intrusion detection systems (IDS) is key to cope with increasingly sophisticated cybersecurity attacks through an effective and efficient detection process. In the context of the Internet of Things (IoT), most ML-enabled IDS approaches use centralized approaches where IoT devices share their data with data centers for further analysis. To mitigate privacy concerns associated with centralized approaches, in recent years the use of Federated Learning (FL) has attracted a significant interest in different sectors, including healthcare and transport systems. However, the development of FL-enabled IDS for IoT is in its infancy, and still requires research efforts from various areas, in order to identify the main challenges for the deployment in real-world scenarios. In this direction, our work evaluates a FL-enabled IDS approach based on a multiclass classifier considering different data distributions for the detection of different attacks in an IoT scenario. In particular, we use three different settings that are obtained by partitioning the recent ToN\_IoT dataset according to IoT devices' IP address and types of attack. Furthermore, we evaluate the impact of different aggregation functions according to such setting by using the recent IBMFL framework as FL implementation. Additionally, we identify a set of challenges and future directions based on the existing literature and the analysis of our evaluation results.
1. Introduction
IoT’s expanding attack surface has driven ML-based intrusion detection, but centralized approaches create privacy concerns and FL-enabled IDS research remains immature. This work evaluates multiclass FL under varied client data distributions and aggregation settings while identifying deployment challenges.
- Motivation: IoT’s expanding attack surface and increasingly sophisticated attacks motivate ML-enhanced intrusion detection systems.The paper highlights Mirai botnet variants as an example of threats to large-scale IoT-enabled systems.
- Motivation: Most existing IoT IDS approaches centralize device data at cloud or server data centers, raising privacy concerns for users and organizations.Traffic can reveal users’ daily habits and expose company network activity to third parties.
- Research gap: Existing FL-enabled IoT IDS studies often use unrealistic data distributions, unsuitable datasets, or binary classification rather than distinguishing attack types.These limitations motivate evaluation under more realistic non-iid settings.
- Approach: The study partitions ToN IoT data by destination IP address, attack-type balance, and a hybrid Shannon-entropy criterion to create three client configurations.The configurations are designed to examine how data distribution affects federated intrusion detection.
- Approach: The evaluation quantitatively examines non-iid data across aggregation methods and training rounds using IBMFL, with multiclass classification for specific attack types.The contributions include comparing aggregation behavior and distinguishing benign traffic from specific attacks.
- Scope: The paper synthesizes evaluation findings and prior literature to define challenges and future trends for FL-enabled IDS deployment in IoT.The stated contribution is a set of challenges and future directions for subsequent development.
2. FL-enabled IDS for IoT scenarios
FL-enabled IDS addresses centralized IoT learning’s privacy and responsiveness concerns by keeping data local while aggregating model updates. Its deployment still involves communication, computation, privacy, and security challenges.
- IDS background: Traditional IDS commonly uses signature or anomaly detection, with anomaly systems identifying deviations in network-traffic behavior.Signature systems rely on pre-established patterns and cannot detect new attacks.
- Centralized limitations: Centralized IoT IDS sends local traffic to cloud or powerful servers, creating privacy risks and potentially delaying detection in dynamic environments.Device traffic may reveal users’ habits or sensitive organizational activity.
- Federated learning: FL keeps device data local while clients exchange model updates through a coordinator that aggregates them into a global model.This preserves collaborative learning without directly sharing device data.
- Federated learning: Each FL training round selects clients, sends global parameters, performs local SGD training, and aggregates returned parameters into the next global model.Client selection can account for IoT computation and communication resources.
- Aggregation: FedAvg is widely used for aggregation, while FedProx and Fed+ provide alternative algorithms; this evaluation uses Fed+ among its alternatives.The cited passage identifies Fed+ as the recent alternative used in the evaluation.
- Open challenges: FL-enabled IoT IDS remains immature, with communication and computing requirements alongside potential security and privacy attacks.These challenges constrain practical deployment beyond the collaborative learning design.
3. Related Work
Existing FL-enabled IoT IDS research spans attacks, datasets, models, aggregation methods, and frameworks, but the literature remains incomplete. Many studies lack IoT-specific data, implementation details, or realistic client distributions.
- Review scope: The review classifies existing FL-enabled IoT IDS proposals by attacks, datasets, ML or DL algorithms, aggregation methods, and implementation frameworks.These criteria organize the literature summarized in Table 1.
- Datasets: Some proposals use generic network datasets rather than traffic from IoT devices, limiting their direct representation of IoT scenarios.Examples include KDDCup99 and NSL-KDD-based federated IDS approaches.
- Datasets and models: IoT-specific studies use datasets such as N-BaIoT, but some omit implementation and evaluation details including client counts and data distributions.Industrial studies also use CNN, GRU, or combined CNN-GRU models with protocol-specific datasets.
- Evidence gaps: The literature is still in its infancy, with many works omitting frameworks, training rounds, client counts, or client-level data distributions.Some studies instead assume equal samples for each attack, unlike non-iid and highly skewed settings.
4. Methodology
The methodology evaluates federated intrusion detection for IoT using suitable IoT datasets, three client-data distributions, multiclass classification, and alternative aggregation functions. It focuses on how non-IID and imbalanced data affect performance while comparing FedAvg with Fed+.
- Dataset selection: The study reviews IoT intrusion datasets and selects CIC-ToN-IoT because it supports distribution by IP address or device and includes diverse attacks and IoT/IIoT data.ToN IoT also offers a relatively favorable benign-to-attack ratio and includes sensor and telemetry data.
- ToN IoT partitioning: CIC-ToN-IoT flows are extracted into numeric features, separated by destination IP address, and used to construct three client-data scenarios.Each federated client is represented by one IP address, while Shannon entropy measures local class imbalance.
- Data-distribution scenarios: The basic scenario preserves each IoT device’s traffic, producing highly unbalanced clients that may contain only benign traffic or very few attack samples.The balanced scenario gives all ten parties equal class representation, whereas the mixed scenario locally balances selected parties while retaining their own samples.
- Multiclass classification: A multinomial logistic regression model classifies traffic as benign or a specific attack using softmax probabilities over unordered classes.The model is chosen for implementation simplicity, training efficiency, and coefficient-based feature-importance interpretation; cross-entropy is its loss function.
- Aggregation functions: The evaluation compares FedAvg, which averages client weights, with Fed+, which relaxes single-model convergence to address heterogeneous and skewed data distributions.The study varies aggregation methods and training rounds, and reports that Fed+ mitigates FedAvg convergence issues in non-IID settings.
5. Evaluation results
The evaluation examines FL-enabled IDS performance across non-iid, balanced, and mixed data distributions using FedAvg and Fed+. Fed+ generally handles imbalanced distributions better, while accuracy alone can obscure poor class-specific detection.
- The evaluation measures accuracy, F1-score, recall, precision, and false positive rate across federated scenarios.Accuracy, F1-score, and FPR are defined from TP, TN, FP, and FN; multiclass metrics can use micro, macro, or weighted averaging.
- Basic scenario: In the basic scenario, single-IP clients have highly skewed non-iid data, making high accuracy potentially misleading when legitimate traffic dominates.Parties 3, 4, 7, and 9 show declining accuracy after around 200 rounds.
- Basic scenario: Beyond accuracy, Fed+ metrics remain stable, but some parties achieve low recall, precision, and F1-score alongside FPR=1 despite high accuracy.Parties 2 and 8 report 0 for precision, recall, and F1-score, and 1 for FPR.
- Implications: Directly applying FL to highly skewed non-iid data can produce undesirable results, motivating client or instance selection and aggregation methods beyond FedAvg.The balanced and mixed evaluations demonstrate the importance of making client data more balanced in classes and samples.
- Balanced and mixed scenarios: Balanced and mixed scenarios show improving Fed+ metrics, with recall, F1-score, and precision increasing while FPR decreases until convergence.In the balanced setting, Fed+ improves accuracy and stabilizes after the initial training rounds; FedAvg can produce metric drops for some parties.
- Scenario comparison: Fed+ provides higher average accuracy than FedAvg across the federated scenarios and handles unbalanced client datasets better.In the basic scenario, Fed+ remains about 0.8725 versus 0.8718 for the distributed method, while FedAvg slowly declines from 0.8725.
6. Challenges and research directions
Deploying FL-enabled IDS on real IoT devices introduces resource constraints beyond the simulated testbed evaluation. Edge intermediaries can support computation, but sharing traffic with them may preserve privacy concerns.
- The study evaluates data-distribution effects using a simulated testbed, whereas real IoT deployment introduces additional challenges.
- Constrained IoT devices may not satisfy the memory, computing-power, and energy requirements of ML approaches, especially deep learning.
- Intermediate edge nodes can act as FL clients for local training or determine global-model aggregation frequency.This approach shifts computation away from end devices but introduces additional entities into the data path.
- Sharing network traffic with intermediate nodes to identify attacks can still pose privacy concerns.
6.2. Limitations of existing IDS-IoT datasets for FL
Existing IoT IDS datasets often lack the device- or IP-level structure required for FL and commonly contain imbalanced traffic with limited attack coverage. Broader IoT technologies and attack-relevant attributes remain needed.
- Some existing FL-enabled IDS studies use general network datasets that do not represent IoT technologies and devices.
- Some IoT datasets cannot support FL because they lack data associated with different IP addresses or devices that can serve as parties.
- Many IoT IDS datasets have substantial benign-versus-attack imbalance and cover only a limited set of attacks.
- Future IoT IDS datasets should broaden technology coverage, include protocols such as CoAP, and consider attributes such as energy consumption.
6.3. Aggregator as bottleneck
The FL coordinator can become a performance and privacy bottleneck as well as a single point of failure. Blockchain-based alternatives and communication optimizations are proposed, but broader evaluation remains necessary.
- A centralized FL coordinator may create performance and privacy bottlenecks and constitute a single point of failure.
- Blockchain can replace the centralized coordinator with distributed nodes using smart contracts and make model updates accountable.
- Current approaches lack comprehensive evaluation of training frequency and scenarios involving large numbers of devices.
- FL requires substantial bandwidth for exchanging global model updates, especially over constrained IoT networks and devices.Gradient compression techniques such as quantization and sparsification address the amount of exchanged gradient data.
- The number of rounds needed for convergence varies with the scenario, dataset, data distribution, and ML algorithm.In the reported evaluation, metrics remain stable after 50 rounds in the balanced and mixed scenarios.
6.5. Client selection
Client selection affects federated IDS performance, but IoT heterogeneity and changing availability require more adaptive selection strategies.
- Static client selection can improve the ML algorithm’s performance in the evaluated setting.
- Selection strategies can consider device status, battery level, computing and communication capacity, or model accuracy.
- Mobility, connectivity loss, and heterogeneous local-training times can make clients unavailable or slow overall federated training.
6.6. Dynamic IoT devices’ behavior throughout their lifecycle
IoT device behavior can change during its lifecycle, complicating benign-traffic modeling and distinguishing legitimate updates from attacks. Existing FL-enabled IDS approaches do not address these changes with datasets that reflect them.
- Software updates can change device behavior, requiring new learning to represent the changed behavior as benign IDS traffic.
- Network management must distinguish intentional behavioral changes from malicious actions affecting devices.
- Behavioral changes in one device can affect interacting devices and require additional training rounds in constrained settings.
- FL-enabled IDS approaches based on existing datasets do not reflect behavioral changes across IoT device lifecycles.
- FL remains vulnerable to data poisoning and model update poisoning attacks during the learning process.
6.8. Privacy concerns
Federated learning can still expose client information and enable manipulation of model updates, which is especially sensitive for IoT intrusion detection traffic.
- A malicious server can infer information from client model updates and alter them to fool the global model.
- IoT network traffic used for IDS can reveal everyday user habits, increasing the sensitivity of potential information leakage.
- Privacy-preserving mechanisms are needed for FL-enabled IDS because FL does not eliminate privacy risks.
7. Conclusions
The paper reviews FL-enabled IDS for IoT and evaluates non-iid data, aggregation functions, and data balancing effects. It identifies deployment challenges and proposes real-world and personalized-FL evaluation as future work.
- 7. Conclusions: The evaluation shows that non-iid and highly skewed data distributions affect FL performance and attack-detection effectiveness.
- 7. Conclusions: Shannon-entropy-based instance selection can improve overall accuracy, producing results similar to balanced data across parties.
- 7. Conclusions: The study evaluates FedAvg and Fed+ aggregation functions with the ToN IoT dataset.
- 7. Conclusions: The paper identifies main challenges for deploying FL-enabled IDS in IoT over the coming years.
- 7. Conclusions: Future work will deploy FL-enabled IDS in real IoT environments with constrained devices and networks.
- 7. Conclusions: Future work will analyze personalized FL, in which each node uses the most appropriate learning model for attack detection.