Source-linked AI summary
A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions
Peng Yan, Ahmed Abdulkadir, Paul-Philipp Luley, Matthias Rosenthal, Gerrit A. Schatte, Benjamin F. Grewe, Thilo Stadelmann
TL;DR
Industrial anomaly detection must handle changing processes and tasks without repeatedly acquiring large labeled datasets or retraining models from scratch. This survey reviews deep transfer-learning settings, methods, and industrial applications, finding that practice relies mainly on simple parameter transfer while identifying challenges and directions for broader adoption.
Problem
Changing industrial processes and tasks make it impractical to obtain large labeled datasets and retrain deep-learning models from scratch for every new setup.
Method
The survey provides an application-oriented review that defines transfer-learning settings, categorizes deep transfer-learning approaches, and examines industrial time series anomaly-detection applications.
Results
Industrial anomaly-detection studies employ a limited variety of deep transfer-learning methods, with almost all applications using straightforward parameter transfer and fine-tuning.
Takeaways & Limitations
The survey identifies substantial potential for more sophisticated transfer schemes and diverse DNN models while emphasizing feasibility, reliability, explainability, and real-time data streams.
Takeaways & Limitations
Deep transfer learning may be inappropriate when source and target data are highly dissimilar, tasks are poorly aligned, or domain shift is high.
Abstract
from arXiv · showhide
Automating the monitoring of industrial processes has the potential to enhance efficiency and optimize quality by promptly detecting abnormal events and thus facilitating timely interventions. Deep learning, with its capacity to discern non-trivial patterns within large datasets, plays a pivotal role in this process. Standard deep learning methods are suitable to solve a specific task given a specific type of data. During training, deep learning demands large volumes of labeled data. However, due to the dynamic nature of the industrial processes and environment, it is impractical to acquire large-scale labeled data for standard deep learning training for every slightly different case anew. Deep transfer learning offers a solution to this problem. By leveraging knowledge from related tasks and accounting for variations in data distributions, the transfer learning framework solves new tasks with little or even no additional labeled data. The approach bypasses the need to retrain a model from scratch for every new setup and dramatically reduces the labeled data requirement. This survey first provides an in-depth review of deep transfer learning, examining the problem settings of transfer learning and classifying the prevailing deep transfer learning methods. Moreover, we delve into applications of deep transfer learning in the context of a broad spectrum of time series anomaly detection tasks prevalent in primary industrial domains, e.g., manufacturing process monitoring, predictive maintenance, energy management, and infrastructure facility monitoring. We discuss the challenges and limitations of deep transfer learning in industrial contexts and conclude the survey with practical directions and actionable suggestions to address the need to leverage diverse time series data for anomaly detection in an increasingly dynamic production environment.
1 Introduction
Industrial monitoring must adapt to changing processes, tasks, and environments, while large-scale labeled data for retraining each variation is impractical. This survey reviews deep transfer learning methods, applications, challenges, and future directions for industrial time series anomaly detection.
- Motivation: Industrial process changes can alter monitoring tasks and require model adjustments, while obtaining large labeled datasets for each variation is impractical.Examples include detecting blunt tools or insufficient cooling lubricant from vibration measurements under different operating conditions.
- Motivation: Transfer learning reuses knowledge from related tasks or domains to improve new anomaly-detection tasks without retraining models from scratch.It accounts for changes in data distributions and tasks while reducing the data and computation required for new settings.
- Contributions: The paper categorizes transfer-learning problem settings and deep transfer-learning approaches, surveys industrial applications, and discusses challenges and actionable future directions.The review covers taxonomy, anomaly-detection tasks, selected applications, limitations, and implementation guidance.
- Scope: The survey focuses on the intersection of transfer learning, time series analysis, and anomaly detection across manufacturing, maintenance, energy, and infrastructure applications.Its scope is organized through a generic taxonomy and an application-oriented review.
- Survey methodology: The survey uses an application-oriented literature-selection process spanning Google Scholar, Scopus, Elsevier, and IEEE databases.Studies were preselected by title and screened for fit with the narrow topic using abstracts and content.
2.1 Overview of the field
In industrial time series anomaly detection, transfer learning adapts monitoring models to related unseen settings with limited training data. The survey frames this field through inductive and transductive transfer and illustrates representative industrial scenarios.
- 2.1 Overview of the field: Transfer learning adapts autonomous process monitoring to related but previously unseen settings where training data are limited.It avoids training a deep learning model from scratch and reduces the data and computation needed for new tasks or domains.
- Transfer-learning settings: The survey distinguishes inductive transfer, where tasks differ, from transductive transfer, where tasks match but domains differ.These settings organize how industrial anomaly-detection knowledge is transferred across tasks and data conditions.
- Examples: Figure 4 illustrates inductive transfer from labeled screw data for collective anomalies to labeled wrench data for point anomalies using parameter transfer.Its transductive example adapts data from B2 to a model trained on B1 for contextual anomaly detection.
2.2 Formal description of deep transfer learning
The formal framework represents domains through feature spaces and marginal distributions, and tasks through label spaces and predictive functions. Transfer learning reuses knowledge between source and target domains under inductive or transductive conditions.
- Formal definitions: A domain consists of a feature space and marginal data distribution, while a task consists of a label space and predictive function.For anomaly detection, the task labels are typically “normal” and “abnormal.”
- Formal definitions: Transfer learning uses a source domain and task to improve learning in a target domain and task, potentially drawing on multiple related source domains.This contrasts with traditional machine learning, which trains each task separately from scratch.
- Approach taxonomy: Table 1 organizes deep transfer-learning approaches into instance, parameter, mapping, and domain-adversarial transfer.The classification reflects how knowledge is shared across domains and target tasks.
- Problem settings: Inductive transfer applies when source and target tasks differ, with labeled target-domain data used to induce the target predictive distribution.The survey’s example transfers from collective-anomaly recognition to point-anomaly recognition.
- Problem settings: Transductive transfer applies when source and target tasks are the same but their domains differ; domain adaptation is a subcategory with shared feature spaces and different marginal distributions.This setting includes shifts in feature spaces or data distributions between source and target domains.
2.3 Deep transfer learning approaches
Deep transfer-learning approaches implement knowledge transfer within neural-network pipelines through data, parameters, feature representations, or adversarial objectives. The surveyed methods range from instance reuse and fine-tuning to feature alignment and domain-invariant representation learning.
- Overview: Deep transfer learning embeds transfer-learning principles in neural networks so models and learned representations can be applied to new datasets.DNNs learn feature representations through back-propagation from large, high-dimensional data.
- Instance transfer: Instance transfer transforms or reuses source data with limited target samples to create target-compatible training instances.Examples include reconstructing pseudo-labeled data and removing source or target samples that harm target performance.
- Parameter transfer: Parameter transfer adapts parts of a pretrained model and its parameters or hyperparameters to target data and tasks.The approach assumes related domains can produce similar feature representations; adaptive fine-tuning can also route instances through the pretrained model selectively.
- Mapping transfer: Mapping transfer learns related target-domain feature representations by alignment, mapping, or encoding to reduce source–target distribution discrepancies.Distribution differences can be measured with criteria such as Wasserstein distance or Kullback-Leibler divergence.
- Domain-adversarial transfer: Domain-adversarial transfer uses adversarial training to learn feature representations that are indiscriminative between source and target domains.The approach primarily addresses domain-adaptation problems, although GAN training can be unstable in real-world applications.
2.4 Related learning paradigms
The survey distinguishes transfer learning from related paradigms that share knowledge across tasks, domains, models, or representations. These approaches differ in how they use source and target data and in whether they preserve prior capabilities.
- Multi-task learning trains one model on multiple tasks to learn shared representations that may generalize to new data and tasks.
- Continuous learning sequentially incorporates new tasks while preserving performance on previous tasks, accumulating knowledge over time.
- Few-shot learning targets new tasks with limited labeled samples, whereas transfer learning typically reuses a model and continues training on target data.
- Domain generalization learns from one or more source domains to detect unseen target-domain data without using target-domain data during transfer.
- Meta-learning trains across tasks rather than ordinary datasets, while knowledge distillation transfers behavior from a larger model to a smaller one.
- Self-supervised learning predicts aspects of input data without external supervision, producing representations usable for downstream transfer tasks.
3 Time series anomaly detection in industry
Industrial time series anomaly detection addresses complex, shifting sensor data using deep models that reconstruct or forecast normal behavior. Detection effectiveness depends on anomaly type, temporal granularity, and the selected modeling approach.
- Industrial anomaly detection supports automated monitoring, maintenance scheduling, and control of process efficiency, quality, and performance.
- Challenges: Increasing system complexity requires multivariate sensor analysis, while changing conditions create data and domain shifts that complicate detection.
- Detection methods: Reconstruction-based methods compare actual sequences with decoder-generated reconstructions, commonly using reconstruction error to identify abnormalities.
- Detection methods: Forecasting-based methods predict subsequent values or sequences and detect anomalies from prediction deviations; graph models can also score sensor-specific deviations.
- Detection methods: Hybrid approaches jointly combine reconstruction and forecasting or prediction, while supervised classifiers and contrastive methods represent other alternatives.
- Limitations: Although these approaches can address each anomaly type, effectiveness varies with the granularity at which time series data are observed and analyzed.
4 Industrial applications
Deep transfer learning has been applied across industrial anomaly-detection domains, but the surveyed literature is dominated by straightforward parameter-transfer designs. Data scarcity and domain shift remain common concerns, while reported applications show reuse of models can reduce training demands.
- Overview: The survey covers industrial applications including manufacturing monitoring, predictive maintenance, energy management, and infrastructure monitoring, where public time-series datasets remain limited.
- Overview: Parameter transfer is the most frequent approach because fine-tuning pretrained models is straightforward and usually requires no fundamental architectural modification.
- Overview: Most surveyed applications use inductive transfer learning with labeled source and target data, while instance and adversarial transfer do not appear.
- Manufacturing and maintenance: Manufacturing studies use mapping and parameter transfer for control-system monitoring, remaining-useful-life prediction, and virtual-to-physical fault diagnosis.
- Manufacturing and maintenance: Transfer learning can reduce training time and required data, improve model reusability, and help predictive-maintenance systems adapt across similar tasks and environments.
- Cross-application findings: Data scarcity and domain shift are the two common problems across industrial applications, with mapping and parameter transfer most often used for the latter.
- Cross-application findings: Across the surveyed literature, industry predominantly uses simple, established transfer-learning design patterns despite the broader range of available approaches.
5 Discussion
The discussion emphasizes that industrial deep transfer learning must handle shifting domains, scarce or unreliable labels, imbalance, and missing information. It outlines preprocessing, augmentation, hybrid architectures, foundation models, and federated learning as directions for improving deployment.
- 5.2 Challenges: Domain shift arises from changing products, sensors, distributions, feature-label relationships, or label distributions, making transfer between industrial settings difficult.The survey identifies covariate, concept, and label shift as distinct forms of mismatch.
- 5.2 Challenges: Industrial monitoring often lacks sufficient reliable labels because annotation is expensive, so self-supervised relabeling and human-in-the-loop processes are used.Unreliable labels can still affect transfer-learning training.
- 5.2 Challenges: 0?
- 5.2 Challenges: Missing data and unrecorded process information can remove important features, including geometry and environmental conditions relevant to injection molding.
- 5.2 Challenges: Reducing the source-target gap requires understanding target-data characteristics and selecting suitable strategies, including domain generalization, contrastive learning, and adversarial examples.The survey states that domain shift remains unsolved.
- 5.3 Directions for anomaly detection solution design: Preprocessing remains open: feature selection and transformations reduce complexity but may discard trends, while direct deep-learning processing preserves raw time-series information.Common transformations include mapping one-dimensional signals to images or the frequency domain.
- 5.3 Directions for anomaly detection solution design: Synthetic simulation data, imbalance-handling methods, hybrid CNN-RNN/LSTM architectures, foundation models, and federated learning address data, modeling, and deployment constraints.Foundation models may provide transfer capability without explicit transfer learning, while federated learning supports privacy-preserving collaboration on resource-constrained devices.
- 5.4 Directions for deep transfer learning implementation: Deep transfer learning may be unsuitable with irrelevant source data, highly specific target tasks, large domain shifts, or abundant target labels.In such cases, training a task-specific model from scratch may be more effective.
6 Conclusions
The survey synthesizes deep transfer learning methods and industrial time series anomaly-detection applications, then identifies practical priorities for deployment and future research. It finds that current industrial use is dominated by simple parameter transfer, while broader methods and model integration remain promising directions.
- Survey scope and synthesis: The survey defines transfer-learning settings, categorizes deep transfer learning approaches, and reviews representative industrial anomaly-detection applications for practitioners.Its application coverage includes fielded examples across industrial settings and provides a guide to the possibilities of time series anomaly detection.
- Survey findings: Current industrial time series anomaly detection uses a limited variety of deep transfer learning methods, mainly straightforward parameter transfer with fine-tuning.The reviewed applications predominantly employ parameter transfer, while advanced DNN building blocks such as Transformers were not used in the surveyed papers.
- Future research directions: Future work should test more sophisticated transfer schemes and diverse DNN models, including mapping transfer, adversarial transfer, Transformers, and diffusion models.The survey notes that large models can be effective for certain data and tasks, but computational efficiency and interpretability also require consideration.
- Future research directions: Real-time anomaly detection should combine transfer learning with data-centric practices that improve label quality through re-labeling, re-weighting, and data augmentation.The surveyed data-centric methods still require human-in-the-loop support, while greater automation remains a future research goal.
- Future research directions: Generative models can augment limited target data with synthetic time series and generated anomalies, supporting robustness and alleviating class imbalance.The survey specifically identifies GANs and diffusion models as useful for data augmentation and anomaly generation.
- Future research directions: Robust industrial anomaly-detection solutions should integrate transfer learning with continuous learning, meta-learning, and federated learning.The survey states that relying solely on transfer learning is insufficient for developing robust industrial AI solutions.