PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Anomaly Detection for Time-series Data in Industrial IoT: A Communication-Efficient On-device Federated Learning Approach

Yi Liu, Sahil Garg, Jiangtian Nie, Yang Zhang, Zehui Xiong, Jiawen Kang, M. Shamim Hossain

arXiv:2007.09712v1cs.LGcs.DCstat.ML

TL;DR

Industrial IoT anomaly detection must remain accurate and timely despite private, fragmented edge data and federated-learning communication costs. The paper proposes an on-device FL framework using AMCNN-LSTM detection and Top-k gradient compression, and reports accurate, timely detection with 50% lower communication overhead than an uncompressed FL framework. Its stated system setting remains constrained by missing labels and communication overhead.

  • Problem

    IIoT anomaly detection must handle production-impacting device failures while protecting private sensing time-series data distributed across local data islands.

  • Method

    The paper combines on-device federated learning, an attention mechanism-based CNN-LSTM anomaly detector, and Top-k gradient compression.

  • Results

    The framework accurately and timely detects anomalies on four real-world datasets while reducing communication overhead by 50% versus federated learning without gradient compression.

  • Takeaways & Limitations

    The proposed model achieves performance comparable to advanced centralized models while retaining local data access for privacy-preserving anomaly detection.

  • Takeaways & Limitations

    The stated DAD model-learning setting is limited by missing labels and communication overhead.

Abstract

from arXiv · show

Since edge device failures (i.e., anomalies) seriously affect the production of industrial products in Industrial IoT (IIoT), accurately and timely detecting anomalies is becoming increasingly important. Furthermore, data collected by the edge device may contain the user's private data, which is challenging the current detection approaches as user privacy is calling for the public concern in recent years. With this focus, this paper proposes a new communication-efficient on-device federated learning (FL)-based deep anomaly detection framework for sensing time-series data in IIoT. Specifically, we first introduce a FL framework to enable decentralized edge devices to collaboratively train an anomaly detection model, which can improve its generalization ability. Second, we propose an Attention Mechanism-based Convolutional Neural Network-Long Short Term Memory (AMCNN-LSTM) model to accurately detect anomalies. The AMCNN-LSTM model uses attention mechanism-based CNN units to capture important fine-grained features, thereby preventing memory loss and gradient dispersion problems. Furthermore, this model retains the advantages of LSTM unit in predicting time series data. Third, to adapt the proposed framework to the timeliness of industrial anomaly detection, we propose a gradient compression mechanism based on Top-\textit{k} selection to improve communication efficiency. Extensive experiment studies on four real-world datasets demonstrate that the proposed framework can accurately and timely detect anomalies and also reduce the communication overhead by 50\% compared to the federated learning framework that does not use a gradient compression scheme.

I. INTRODUCTION

IIoT anomaly detection must address abnormal device behavior, fragmented private time-series data, and federated-learning communication costs. The paper proposes an on-device FL framework combining AMCNN-LSTM anomaly detection with Top-k gradient compression.

  • Abnormal IIoT device behavior creates security risks and can hinder smart-manufacturing applications.
  • Fragmented time-series data and privacy concerns limit collaborative anomaly detection because devices retain sensitive data locally.
  • On-device federated learning enables edge devices to train a shared anomaly-detection model without sharing raw training data, but communication overhead remains overlooked.
  • The proposed framework combines collaborative FL, an attention-based CNN-LSTM model, and Top-k gradient compression for accurate, timely, and communication-efficient detection.
  • Experiments use four real-world datasets to evaluate anomaly detection with low communication overhead.

B. Communication-Efficient Federated Learning

Federated learning keeps training data on edge devices while coordinating model updates through a cloud aggregator. The section describes its privacy benefits and the communication bottleneck created by exchanging many gradients.

  • Exchanging many gradients between edge devices and the cloud aggregator can impose excessive communication overhead.
  • The paper addresses this bottleneck with Top-k gradient compression to reduce the number of exchanged gradients.
  • Anomaly detection classifies observations that differ substantially from normal regions, using probability values to judge whether a vector is anomalous.
  • Federated learning lets devices contribute to a global model while keeping training data locally, reducing direct raw-data privacy risks.
  • The FL procedure includes initialization, local training, aggregation of uploaded gradients, and global-model updates using FedAVG.

C. Gradient Compression

Gradient compression reduces federated-learning communication by sparsifying exchanged weight updates, while preserving the underlying neural-network computation.

  • 99.9% of D-SGD gradient exchange is described as redundant, motivating gradient compression for large-scale federated learning.The paper identifies gradient quantization and gradient sparsification as general approaches to reduce communication bandwidth.
  • A fully connected layer computes b = f(W ∗a + v), where W is the weight matrix and b is the output.The passage defines a, v, W, f, and b for the basic neural-network operation.
  • Gradient compression converts the corresponding weight matrix into a sparse matrix to reduce exchanged gradient information.The compressed matrix retains selected gradient positions and reduces communication overhead through sparsification.

IV. SYSTEM MODEL

The system model combines edge devices, a cloud aggregator, anomaly detection, and gradient compression in an on-device federated-learning workflow for IIoT.

  • A. System Model Limitations: The framework involves N distributed edge devices and a cloud aggregator, with missing labels and communication overhead identified as limitations.These limitations motivate an unsupervised, communication-efficient federated anomaly-detection framework.
  • A. System Model Limitations: Missing training labels can create class imbalance and reduce anomaly-detection accuracy, while excessive communication may prevent model convergence.The missing-label proportion is denoted p, with 0 < p < 1.
  • System Workflow: The workflow uses local IIoT sensing time-series data, trains AMCNN-LSTM models locally, uploads sparse gradients, aggregates them, and redistributes the global model cyclically.The cycle continues until the global model reaches optimal convergence, after which devices perform anomaly detection.
  • System Components: The cloud aggregator initializes and aggregates the global model, while edge devices train local models on sensing time-series datasets and upload gradients.The framework separately deploys anomaly detection and gradient compression mechanisms on edge devices.
  • Design Goals: The framework targets accurate unsupervised detection, improved communication efficiency, and performance comparable to traditional approaches.Its stated components are federated collaborative training, AMCNN-LSTM anomaly detection, and gradient compression.

V. A COMMUNICATION-EFFICIENT ON-DEVICE DEEP ANOMALY DETECTION FRAMEWORK

The AMCNN-LSTM framework combines attention-based CNN feature extraction with LSTM time-series prediction and anomaly scoring for unsupervised IIoT detection.

  • A. Attention Mechanism-based CNN-LSTM Model: The model processes normalized sensing time-series data through an input layer, attention-based CNN unit, LSTM unit, and output layer.CNN extracts fine-grained features, attention emphasizes important features, and LSTM predicts future time-series values.
  • CNN Unit: The CNN stacks one-dimensional convolutional, batch-normalization, and nonlinear layers to extract hierarchical fine-grained features.Pooling performs sampling aggregation, and the module outputs m feature sequences of length n.
  • Attention Mechanism-based CNN Unit: The attention output is multiplied element-wise with CNN features, and the resulting feature layer is passed to the LSTM block.W(i, c) combines WCNN(i, c) and Wattention(i, c) through element-wise multiplication.
  • Attention Mechanism: Attention expands the receptive field and suppresses unimportant features, enabling more contextual and importance-sensitive sequence representations.The mechanism addresses difficulty distinguishing the importance of time-series features.
  • LSTM Unit: LSTM uses forget, input, and output gates to selectively remove or add information to the cell state during time-series prediction.The hidden state and cell state are updated through the gated recurrent structure.
  • Anomaly Detection: Anomaly scores are based on reconstruction errors modeled with a Normal distribution, and points exceeding threshold ς are classified as anomalous.The threshold is defined using the maximum Fθ score involving precision P, recall R, and parameter θ.

B. Gradient Compression Mechanism

The gradient compression mechanism accumulates small local gradients, transmits only sufficiently large updates, and aggregates sparse gradients while addressing convergence and information-loss risks.

  • B. Gradient Compression Mechanism: Only 0.1% of gradients with the largest absolute values remain useful when gradient sparsity reaches 99.9%.This observation supports transmitting a sparse subset rather than the complete gradient matrix.
  • Gradient Selection: Top-k-style compression selects gradients above a threshold, reducing the byte size and number of gradients exchanged between edge devices and the cloud.The mechanism targets communication efficiency in distributed machine learning.
  • Gradient Accumulation: Local gradient accumulation retains smaller gradients in a buffer until their accumulated value reaches a threshold, preventing information loss from immediate filtering.The cloud returns smaller gradients to the device rather than discarding them outright.
  • Optimization: The optimization objective uses D-SGD over local device losses, with model weights updated using learning rate η and minibatch data.F(ω) denotes the global loss, while f(x, ω) denotes a local-device loss.
  • Convergence Control: 99% sparsification can affect convergence, so momentum correction and local gradient clipping are used to mitigate convergence and gradient-explosion effects.Momentum correction accelerates convergence by directing accumulated gradients toward larger-magnitude gradients.
  • Implementation Phases: The three-phase procedure consists of local training with gradient accumulation, threshold-based compression, and cloud aggregation of sparse gradients.The cloud sends the resulting global model back to edge devices after aggregation.

VI. EXPERIMENTS

Experiments evaluate the framework on four real-world sensor time-series datasets, with the study using a gradient-compression mechanism on edge nodes and reporting dataset details.

  • Datasets: The framework is evaluated on power demand, space shuttle, ECG, and engine datasets collected from different types of sensors.The datasets contain both normal and anomalous subsequences.
  • Gradient compression: The experiment section includes an algorithm for gradient compression on edge node k.Its inputs include the node gradient, local mini-batch size, local dataset, learning rate, loss function, and SGD optimization.
  • Gradient compression: The edge node samples data from its local dataset and can apply gradient clipping before communicating gradients.The procedure sends gradients to the cloud aggregator and accumulates local gradients until they reach a threshold.
  • Datasets: Table I provides details for the four real-world datasets used in the evaluation.The supplied table passage identifies the table but does not include its individual dataset values.

A. Evaluation Setup

The evaluation uses accuracy-oriented framework experiments and RMSE to assess AMCNN-LSTM prediction performance, with figures covering threshold effects and model comparisons.

  • Experimental configuration: The hyperparameter study uses a simple CNN with two convolutional layers followed by one fully connected layer on MNIST and CIFAR-10.Pixels are normalized into [0,1], with N = 10 edge devices, learning rate η = 0.001, and training epoch E = 1000.
  • Evaluation metrics: RMSE is adopted to indicate the performance of the AMCNN-LSTM model.The metric compares observed sensing time-series data with predicted sensing time-series data.
  • Reported comparisons: Figure 4 reports the proposed framework’s accuracy with different ρ values on MNIST and CIFAR-10.Figure 5 compares detection accuracy for AMCNN-LSTM, CNN-LSTM, LSTM, GRU, SAEs, and SVM across four datasets.

B. Hyperparameters Selection of the Proposed Framework

The study selects the gradient-compression threshold by examining how different ρ values affect accuracy and the resulting trade-off between compression settings and performance.

  • Threshold evaluation: For MNIST, accuracy is 97.25% at ρ = 0.3 and 99.08% at ρ = 100.The paper reports that increasing the gradient size by about 300 times improves accuracy by only 1.83%.
  • Threshold selection: The authors observe that larger ρ improves framework performance but creates a trade-off between the gradient threshold and accuracy.They choose ρ = 0.3 as the best threshold to obtain a good trade-off.

C. Performance of the Proposed Framework

The proposed AMCNN-LSTM is compared with five alternative anomaly-detection models across four datasets, while communication efficiency is compared for federated learning with and without GCM.

  • Prediction error: The model comparison also evaluates RMSE across power demand, space shuttle, ECG, and engine datasets.Figure 6 is the reported comparison of RMSE for the six listed models.
  • Model comparison: The comparison includes CNN-LSTM, LSTM, GRU, SAEs, and SVM alongside AMCNN-LSTM across power demand, space shuttle, ECG, and engine datasets.AMCNN-LSTM is FL-based, whereas the other listed methods are centralized.
  • Detection accuracy: The proposed model achieves the highest detection accuracy on all four datasets.On power demand, AMCNN-LSTM reaches 96.85%, which is 7.87% higher than SVM.
  • Prediction error: On the ECG dataset, AMCNN-LSTM’s RMSE is 63.9% lower than SVM’s.The paper attributes this performance to attention-based CNN units capturing fine-grained features and LSTM units retaining time-series prediction advantages.
  • Communication efficiency: Figure 7 compares communication efficiency between FL with GCM and FL without GCM using different models.The study fixes communication overhead per round and compares running time as the efficiency measure.

D. Communication Efficiency of the Proposed Framework

The framework evaluates communication efficiency while preserving privacy and anomaly-detection performance. It combines federated learning with gradient compression, with experiments reporting substantial compression and competitive model performance.

  • Communication-efficiency evaluation: The evaluation compares FL with gradient compression against traditional FL without compression using the same anomaly-detection models and fixed per-round communication overhead.Running time is used to compare communication efficiency under equal communication overhead per round.
  • Privacy: FL keeps user datasets locally, preventing third parties from accessing raw data during collaborative model training.This supports anomaly detection without sharing users’ raw datasets.
  • Performance: The proposed model achieves performance comparable to advanced centralized CNN-LSTM, LSTM, GRU, and SVM models.The paper presents this as a compromise between privacy protection and model performance.
  • Gradient compression: 300× gradient compression is achieved with no accuracy loss using a Top-k selection mechanism.The mechanism is designed to improve communication efficiency for timely industrial anomaly detection.
  • Limitation: The framework remains vulnerable to malicious attacks by participating devices, motivating future privacy-enhanced FL research.The authors also identify more robust anomaly-detection models as future work.
Loading 2007.09712v1…