PaperlayerView exact revision on arXiv

Source-linked AI summary

IPM-FM: A Foundation Model with Consensus Feature Selection for Industrial Process Monitoring

Liang Cao, Weide Liu, Yan Qin, Jun Cheng, Weisi Lin, Bhushan Gopaluni

arXiv:2609.08375v1cs.LGcs.AI

TL;DR

Industrial process monitoring remains label-inefficient under a one-task-one-model practice and faces spurious channels, safety-critical uncertainty requirements, and mismatched process and laboratory sampling. IPM-FM addresses these challenges through shared self-supervised representations, consensus feature selection, recursive lag features, and calibrated uncertainty, achieving 2.99 RMSE with 97% coverage of its 95% predictive interval while outperforming reported baselines.

  • Problem

    Industrial process monitoring commonly uses separate models trained from scratch for each task, while spurious channels, safety-critical decisions, and mismatched process and laboratory sampling create domain-specific challenges.

  • Method

    IPM-FM decouples representation learning from task supervision using a shared self-supervised Informer backbone with consensus feature selection, recursive lag features, and calibrated MC-dropout uncertainty.

  • Results

    2.99 RMSE and 0.50 R2 were achieved with 97% empirical coverage of the 95% predictive interval, while RMSE improved over classical and from-scratch sequence baselines by 8.3% and 14.6%, respectively.

  • Takeaways & Limitations

    The unified pretraining–adaptation pipeline provides a route to reuse representation learning across soft sensing, fault detection, and prognostic tasks maintained separately in industrial plants.

Abstract

from arXiv · show

Industrial process monitoring is fundamental to the safety and economic performance of modern process plants. Current practice remains a one-task-one-model paradigm that is label-inefficient and prone to degradation under operating drift. Foundation models have reshaped language, vision, and generic time-series forecasting, but it has not been adapted to industrial process monitoring. This setting poses domain-specific challenges, including safety-critical decisions and asymmetric sampling between process variables and laboratory measurements. We propose the industrial process monitoring foundation model (IPM-FM). It first learns general-purpose representations from unlabeled industrial process data through self-supervised pretraining, then adapts to specific monitoring tasks using a small amount of task-labeled data, and finally produces calibrated predictions through an uncertainty-aware prediction head. IPM-FM integrates a self-supervised Informer backbone with a multi-criteria consensus feature selector, a recursive lag-feature regression head, and a calibrated Monte Carlo dropout uncertainty module. On a seven-year hydrotreater dataset for diesel flash-point soft sensing, IPM-FM attains an RMSE of 2.99, $R^2$ of 0.50, and 97\% coverage of its 95\% predictive interval, outperforming the strongest classical and from-scratch sequence baselines by 8.3\% and 14.6\% in RMSE respectively, supporting the viability of a unified pretraining--adaptation framework for industrial process monitoring.

I. INTRODUCTION

Industrial process monitoring remains fragmented into task-specific models despite scarce labels, limited transferability, operating drift, safety-critical decisions, irrelevant channels, and mismatched laboratory sampling. IPM-FM addresses these challenges with a unified foundation-model framework combining self-supervised representations, consensus feature selection, recursive laboratory lags, and calibrated uncertainty.

  • Motivation: Industrial monitoring commonly assigns each task a dedicated model trained from scratch on its own labeled history.
  • Motivation: Three limitations are label inefficiency, weak transferability across tasks or plants, and degradation under operating drift.
  • Research gap: Existing foundation models cover language, vision, generic forecasting, and representation learning, but not physically coupled industrial process-monitoring signals.
  • Domain challenges: Industrial adaptation must handle many irrelevant sensor channels, safety-critical decisions requiring calibrated uncertainty, and severe sampling mismatch between process variables and laboratory measurements.
  • Proposed framework: IPM-FM unifies industrial monitoring tasks through a shared self-supervised backbone that decouples representation learning from task supervision.
  • Proposed framework: The framework combines multi-criteria tree-based, spectral, and causal feature evidence with recursive laboratory lags and calibrated MC-dropout predictive intervals.

II. IPM-FM FRAMEWORK

IPM-FM uses a two-stage architecture: self-supervised pretraining on unlabeled process data followed by lightweight task adaptation. Its shared Informer-derived backbone models long sequences with ProbSparse attention and distillation before downstream modules produce task-specific monitoring outputs.

  • Overall architecture: The two-stage pipeline pretrains a shared backbone on unlabeled multivariate process data, then adds task-specific adaptation modules.
  • Overall architecture: Switching monitoring tasks replaces only the lightweight task head while preserving the pretrained backbone, enabling cross-task reuse.
  • Problem setup: The pretraining set contains unlabeled segments, while downstream adaptation uses aligned input-target pairs with far fewer labeled examples.
  • Backbone architecture: The shared Informer-derived encoder transforms multivariate process segments into latent representations through stacked ProbSparse self-attention and distillation layers.
  • Backbone architecture: ProbSparse attention restricts interactions to dominant query-key pairs, reducing memory and floating-point operations for long industrial sequences.

2) Distillation Layer:

Each encoder stage ends with a distillation layer that halves sequence length while preserving the most informative activations. The pretraining objectives are label-free, supporting reusable representations across monitoring tasks.

  • 2) Distillation Layer:: Each encoder-stage distillation layer halves the sequence length while preserving the most informative activations.
  • 2) Distillation Layer:: The pretraining objectives require no task labels and are intended to produce representations reusable across monitoring tasks.

1) Masked Segment Reconstruction:

IPM-FM pretrains on masked process segments and operating-regime relationships to learn representations without downstream labels. Its reconstruction and contrastive objectives jointly use local context and regime structure.

  • 1) Masked Segment Reconstruction:: Masked Segment Reconstruction predicts randomly masked non-overlapping sub-segments from surrounding context.The reconstruction loss is evaluated over the masked positions.
  • 2) Operating-Regime Contrastive Learning:: Operating-regime contrastive learning pulls together segments from the same regime and pushes apart segments from different regimes.Regimes are formed through k-means clustering of daily aggregated process statistics without downstream labels.
  • 2) Operating-Regime Contrastive Learning:: The contrastive objective uses pooled segment representations, same-regime positive samples, and a temperature hyperparameter.The total pretraining loss is a weighted sum, with λcont selected on a held-out validation split.

IV. DOWNSTREAM TASK ADAPTATION

Downstream adaptation combines task-specific feature selection with spectral and tree-based relevance signals. The resulting modules target the high-dimensional, task-dependent sensor inputs of industrial monitoring.

  • IV. DOWNSTREAM TASK ADAPTATION: IPM-FM connects its pretrained backbone to task-specific modules during downstream adaptation.These modules include feature selection, recursive lag augmentation, and an uncertainty-aware prediction head.
  • IV. DOWNSTREAM TASK ADAPTATION: The consensus selector addresses sensor redundancy by fusing three complementary feature-selection families.Industrial plants may contain dozens to hundreds of channels, many irrelevant or redundant for a specific target.
  • IV. DOWNSTREAM TASK ADAPTATION: Tree-based ensembles rank channels using impurity decrease, while spectral similarity captures dynamic correspondence with the monitoring target.The spectral criterion is designed to remain invariant to time delays and phase shifts.

3) Causal Discovery:

The causal-discovery component contributes a third feature-selection criterion to the consensus framework. It combines complementary causal algorithms and a voting rule intended to suppress spurious selections.

  • 3) Causal Discovery:: Causal discovery uses PC, FCI, and DirectLiNGAM to evaluate feature relevance under complementary assumptions.The methods address conditional independence, latent confounding, and non-Gaussianity, respectively.
  • 3) Causal Discovery:: Each method score is min-max normalized before entering the weighted consensus score.The consensus framework combines tree-based, spectral, and causal criteria.
  • 3) Causal Discovery:: A feature is selected only when it satisfies the consensus selection rule.The implementation uses τcons = 0.30, τm = 0.30, Vmin = 2, and category weights of 0.40, 0.30, and 0.30.
  • 3) Causal Discovery:: Under approximate independence across method categories, consensus voting multiplicatively suppresses spurious-feature selection relative to a single criterion.The bound assumes category-specific false-positive rates and requires at least two supporting categories.

B. Recursive Lag-Feature Mechanism

The downstream head uses recent laboratory measurements to handle sparse quality labels and can continue operating when new measurements are unavailable. MC dropout with temperature scaling produces calibrated predictive uncertainty for regression and classification monitoring.

  • B. Recursive Lag-Feature Mechanism: Recursive lag features retrieve the Nlag most recent laboratory measurements before each query timestamp and augment the encoder embedding.Each lag includes a standardized value, elapsed time, and a presence mask governed by the maximum-staleness budget.
  • B. Recursive Lag-Feature Mechanism: When fresh laboratory measurements are unavailable, prior model predictions become surrogate lag inputs for continuous operation.Real measurements define closed-loop inference, while model-generated surrogates define open-loop inference.
  • C. Calibrated Uncertainty Head: MC dropout keeps dropout active during inference and uses M stochastic forward passes to obtain an empirical predictive distribution.The same dropout machinery extends to classification by applying it to softmax outputs.
  • C. Calibrated Uncertainty Head: Temperature scaling rescales predictive standard deviation to calibrate 95% Gaussian predictive intervals.The optimal γ is fitted by minimizing Gaussian negative log-likelihood on a held-out calibration split.
  • B. Recursive Lag-Feature Mechanism: For soft sensing, the downstream loss combines prediction and regularization, while classification replaces squared error with cross-entropy.The shared backbone features support both task formulations.

A. Primary Downstream Task: Quality-Variable Soft Sensing

The paper evaluates IPM-FM on diesel flash-point soft sensing using a seven-year hydrotreater dataset with sparse laboratory measurements and online process variables. IPM-FM achieves the strongest reported performance while supporting calibrated intervals and recursive open-loop prediction.

  • Dataset and setup: The benchmark uses 6,157 flash-point samples collected over seven years, aligned with 24 online process variables sampled every 10 minutes.Flash point is measured offline every 4–8 hours, creating asymmetric sampling between laboratory and process data.
  • Performance: IPM-FM achieves the lowest RMSE, MAE, and MAPE and the highest R2 among the evaluated classical and sequence baselines.It is compared with classical learners and LSTM, Transformer, and Informer sequence models.
  • Performance: 8.3% RMSE reduction versus Partial Least Squares and 14.6% versus the strongest from-scratch sequence baseline demonstrate the reported performance advantage.The comparison values are based on PLS with RMSE 3.26 and LSTM without feature selection or pretraining with RMSE 3.50.
  • Uncertainty: Figure 4 compares IPM-FM predictions with laboratory measurements and displays a γ-calibrated 95% predictive interval.The interval is accompanied by per-sample residuals.
  • Recursive inference: Open-loop recursive inference tracks the laboratory trajectory with only a small RMSE penalty relative to closed-loop inference.Closed-loop inference uses actual laboratory values, whereas open-loop inference feeds model predictions into the lag builder.

D. Ablation Study

The ablation study identifies consensus feature selection as the largest contributor to performance, followed by recursive lag features and self-supervised pretraining. Criterion-level results show that tree-based and spectral evidence are strongest on this dataset, while causal evidence adds interpretability despite dataset-dependent RMSE effects.

  • Component ablations: Removing consensus feature selection increases RMSE from 2.99 to 3.72 and decreases R2 by 0.26, the largest component degradation.This result identifies multi-criteria channel selection as the primary driver in the ablation.
  • Component ablations: Removing recursive lag features increases RMSE from 2.99 to 3.22, showing the value of coupling predictions to prior laboratory values.Process-variable information alone does not replace the autoregressive lag mechanism in this experiment.
  • Component ablations: Removing self-supervised pretraining increases RMSE to 3.11, supporting transferable representations from masked reconstruction and operating-regime contrastive objectives.The result concerns downstream regression on the evaluated soft-sensing task.
  • Consensus criteria: Removing tree-based or spectral evidence worsens RMSE to 3.08 or 3.09, whereas removing causal evidence slightly improves RMSE to 2.94.The paper interprets tree-based and spectral evidence as stronger selection signals for this dataset, with causal evidence partly redundant for this task.
  • Interpretation: The authors retain causal selection because it adds process-knowledge consistency and interpretability despite dataset-dependent marginal RMSE effects.The overall results attribute additional benefits to self-supervised pretraining and MC-dropout calibration for representation quality and uncertainty reliability.
Loading 2609.08375v1…