Source-linked AI summary
Text-Informed Training Data Filtering for Enhanced Virtual Flow Meter Performance
Kristoffer Hjelmen Nesland, Bjarne Grimstad, Antorweep Chakravorty, Dan Sui
TL;DR
Industrial anomaly filtering is limited when sensor readings need operational context that is held in unstructured comments, while manual curation across data streams can be infeasible. The paper uses an LLM to combine comments and sensor data in a training-data filter; multimodal filtering improves on classical filters and reduces MAPE relative to sensor-only filtering. The results are based on a synthetic benchmark, and real-world performance may differ.
Problem
Industrial anomalies may depend on context beyond sensor readings, while manual curation across large, disparate data streams can be infeasible.
Source: §1 · p. 2, §1.1 · p. 3Method
An LLM jointly reasons over sensor measurements and operator comments to produce binary weights for filtering training samples.
Source: §1.2 · p. 3, §4.1 · p. 6Results
5.4% versus 7.1% MAPE: multimodal LLM filtering improves on sensor-only LLM filtering, and it outperforms Tukey and Mahalanobis on MSE, MAPE, and W10.
Source: §6.1 · p. 12Takeaways & Limitations
Operator comments are an underused data-centric resource that can be folded into model development alongside sensor data.
Source: §7 · p. 16Results use a synthetic benchmark of 13 wells, and performance on real-world data may deviate.
Source: §7 · p. 16
Abstract
from arXiv · showhide
In sectors such as energy, aquaculture and chemical processing, the effectiveness of both mechanistic and data-driven models is frequently limited by poor-quality training data, which obscures underlying physical laws and degrades predictive accuracy. Such quality issues commonly manifest as anomalies in the dataset, ranging from sensor faults to process upsets and incorrect operator inputs. While traditional rule-based filtering can remove a subset of the anomalies, it is unable to leverage contextual information stored as unstructured text, such as engineer comments and operational logs that explain under what conditions the data was collected. We propose a text-informed data filtering method that leverages large language models (LLMs) to integrate unstructured operational data with tabular sensor data. The method consists of two steps: (1) contextual anomaly detection by reasoning over both data modalities, and (2) generation of a binary mask that is applied to filter the training dataset for model development. To investigate the potential benefit, a steady-state soft-sensor regression model, namely a virtual flow meter that estimates well flow rates, is used as the downstream evaluation target. Experiments on a synthetic oil and gas dataset show improved model performance compared to traditional rule-based filtering. Ablation studies show that textual operator comments can provide filtering-relevant signals beyond sensor data alone, with the multimodal approach reducing downstream MAPE from 7.1% to 5.4% relative to sensor-only filtering. This work demonstrates that integrating unstructured operational knowledge with sensor data through LLMs can improve data quality for industrial machine learning applications, offering a scalable alternative to manual data curation.
1 Introduction
Industrial data quality problems are difficult to address manually, and sensor readings alone may miss anomalies that depend on operational context. The paper proposes using an LLM to combine sensor measurements and operator comments when filtering training data.
1 Introduction
- Industrial data quality issues and anomalies are significant, while manual quality control can be infeasible at scale and across disparate data streams.The paper notes that unresolved data quality issues affect downstream modelling tasks.Source: §1 · p. 2
1.1 Problem Definition
- Industrial anomalies can be contextual: readings within historical ranges may still be unreliable during an instrument fault, making sensor-only statistical detection insufficient.The problem is framed as assigning each point a binary weight: retain it for training or exclude it as anomalous.Source: §1.1 · p. 3
1.2 Contributions
- The paper proposes an LLM method that jointly reasons over tabular sensor measurements and unstructured operator comments to produce a binary data mask.It evaluates the method on a synthetic oil and gas dataset and reports that comments carry filtering-relevant signals not captured by sensor-only methods.Source: §1.2 · p. 3
2 Related Work
Prior work applies language models to tabular and multimodal classification, but the cited methods do not address filtering data to improve downstream industrial model performance. The paper evaluates filtering through a soft-sensor regression task.
2.1 Language Models for Tabular Data and Multimodal Data Fusion
- Prior LLM work covers few-shot learning, tabular classification, tabular anomaly detection, and multimodal in-context anomaly detection, but not industrial training-data filtering for downstream predictive performance.The distinction is between classifying anomalies as an end in itself and filtering data to improve later model training.Source: §2.1 · pp. 3–4
2.2 Soft-Sensors and Virtual Flow Metering
- Soft-sensor training-data filtering has received little attention, so the paper uses a soft-sensor model to measure filtering strategies’ downstream impact.Soft sensors estimate difficult-to-measure variables from process measurements; the evaluation target is a virtual flow meter.Source: §2.2 · p. 4
3 Benchmark Dataset
The benchmark uses ManyWells simulations to create well-test data with controlled anomalies and operator comments. It covers 13 well configurations, with text coverage and specificity designed to reflect a motivating offshore asset.
3.1 Dataset Implementation
- The synthetic benchmark uses ManyWells steady-state multiphase simulations, with each simulated well-test point containing process measurements and flow rates.Reservoir pressure and flow composition are held constant per well, so temporal effects such as reservoir depletion are not modelled.Source: §3.1 · p. 4
- Controlled perturbations create anomalies, with ground-truth weights marking perturbed points for exclusion; the data are divided into reference, training, and nominal evaluation sets.Anomalies include physical-parameter changes and post-simulation sensor-reading changes. The benchmark has 13 well configurations, 80 reference points, 311 training points, and 500 evaluation points; training anomaly rates range from roughly 30% to 60%.Source: Data Generation · pp. 4–5, Dataset Sizes · p. 6
3.2 Dataset Details
- Operator comments vary in specificity and language, and anomalies may go unreported while nominal points may carry routine notes.Anomalous points include comments describing faults in 55.4% of cases; 92.9% of nominal points have no comment.Source: Textual Comments · p. 5, Dataset Sizes · p. 6
- The sensor vector contains four features: bottomhole pressure, wellhead pressure, liquid flow rate, and wellbore length.The benchmark uses a small subset of the measurements available from ManyWells.Source: Sensor Features · p. 5
4 Method
The method assigns binary retention weights to paired sensor and text data, then uses those weights to fit a downstream regression model. It compares LLM filtering with classical sensor-based methods and tests modality and prompt augmentations.
4.1 Problem Formulation
- A filtering method assigns each data point a binary weight that determines whether it is retained for training, and those weights enter the downstream regression objective.The intended filter excludes anomalous points while retaining normal well behaviour.Source: §4.1 · p. 6
4.2 LLM-Based Multimodal Anomaly Detection
- The LLM jointly uses sensor measurements and textual comments to make one anomaly-classification decision per point, returning structured JSON decisions with brief reasoning.The prompt includes a task description, feature dictionary, reference datasets with known weights, and a target dataset with unknown weights; the LLM uses in-context learning without gradient-based training.Source: §4.2 · p. 7
4.3 Baseline Methods
- Classical baselines apply Tukey’s fences and robust Mahalanobis detection to liquid flow rate and pressure differential; uniform filtering and simulator-ground-truth Oracle provide lower and upper reference bounds.Tukey flags points outside 1.5 IQR fences on either axis; robust Mahalanobis uses an MCD estimator and a chi-square threshold.Source: §4.3 · p. 8
4.4 Ablation and Prompt Augmentation
- The ablations compare text-only and sensor-only inputs, while prompt augmentations add Tukey flags or Mahalanobis distances as precomputed statistics.The sensor-only view removes comments; the text-only view removes sensor measurements.Source: §4.4 · p. 8
5 Experimental Setup
The setup evaluates filtering methods by their effect on downstream soft-sensor performance, defining three research questions, evaluation metrics, and a per-well regression model.
5.1 Research Questions and Metrics
- The experiments ask whether multimodal LLM filtering beats classical filters, whether comments add signals beyond sensors, and whether classical scores improve multimodal filtering.Source: §5.1 · pp. 8–9
- Performance is measured by MAPE, MSE, and W10; a method counts as outperforming another only if it is strictly better on two metrics.Source: §5.1 · p. 9
5.2 LLM Inference
- Claude Sonnet 4.6 returns a binary weight and one-sentence rationale for each point; each LLM method runs three times per well to assess filtering consistency.Source: §5.2 · p. 9
5.3 Soft-Sensor Model
- The downstream soft sensor is a per-well mechanistic model that predicts liquid flow rate from pressure differential and vertical wellbore length.Source: §5.3 · p. 10
- The per-well model parameters are fitted by minimizing weighted mean squared error.Source: §5.3 · p. 10
- The model is intentionally simple so retained training data and model fit remain human-interpretable in two dimensions, as illustrated by Fig. 3.Its varying predictor per well is pressure differential; the effective density is found by grid search and the flow coefficient is solved in closed form for each candidate density.Source: §5.3 · p. 10, Figure 3 · p. 11
6 Results and Discussion
Multimodal LLM filtering outperforms classical filters on downstream soft-sensor metrics, and adding classical anomaly scores has method-dependent effects. Agreement analyses and examples show that LLM decisions vary across modalities and weigh comments alongside sensor readings.
6.1 Downstream Soft-Sensor Performance
- LLM (multimodal) records MSE 53, MAPE 5.4 %, and W10 87 %, outperforming Tukey and Mahalanobis on all three metrics.The confidence intervals do not overlap those of either classical method, and the same ordering appears across the full error distribution.Source: Table 3 · p. 12, §6.1 · p. 12
- Adding operator comments lowers MAPE from 7.1 % to 5.4 %, lowers MSE from 88 to 53, and raises W10 from 84 % to 87 % versus sensor-only filtering.The reported confidence intervals do not overlap for any of the three metrics.Source: §6.1 · p. 12
- LLM + Mahalanobis performs best among non-Oracle methods at MSE 47, MAPE 5.1 %, and W10 89 %, while Tukey-score prompting has mixed effects.Mahalanobis improves on multimodal-only filtering; Tukey-score prompting worsens MSE and MAPE but improves W10.Source: §6.1 · p. 12
6.2 Cross-Method Agreement and Consistency of LLM Filtering
- Across repeated runs, LLM (text only) matches its own decisions 100 % of the time, compared with 92.7 % for LLM (sensor only).Adding Mahalanobis distances raises consistency to 97.6 % from 96.6 % for multimodal LLM filtering; LLM + Tukey reaches 96.4 %.Source: §6.2 · p. 14, Table 4 · p. 14
- LLM (sensor only) and LLM (text only) match on 74.1 % of decisions, while multimodal LLM matches each unimodal method on 84.9 % and 86.5 %.The authors interpret this pattern as joint reasoning over both inputs rather than deferring to one modality.Source: §6.2 · p. 14
6.3 Explainability of LLM Filtering
- For well 745, the six illustrated decisions show that neither text nor distance from the sensor-data cluster alone determines classification.Severe emulsion comments can lead to exclusion despite cluster proximity, while distant points without anomalous comments can be retained.Source: §6.3 · p. 15, Figure 5 · p. 15
7 Conclusion
The multimodal LLM filter improved downstream regression over sensor-only filtering, while operator comments offer an industrial data resource; the conclusion also flags scope and deployment constraints.
7 Conclusion
- By 1.7 percentage points, the multimodal method outperformed the sensor-only method on downstream regression, and textual data added signal beyond sensor-only assessment.Source: §7 · p. 16
- Operator comments are an underused data-centric resource already recorded in industrial workflows that can be folded into model development without current manual curation.Source: §7 · p. 16
- Evidence is bounded by a synthetic 13-well benchmark and one model, Claude Sonnet 4.6; real-world performance and results from other models may differ.The magnitude of the text benefit depends on comment informativeness calibrated to a real asset; latency, nondeterminism, and proprietary-model reliance may limit some deployments.Source: §7 · p. 16
7.1 Future Work
- Future work calls for real-world validation, time-aware handling of how long text remains valid, and analysis of sensor-versus-text importance.Real-world data would also help assess sensitivity to reference-dataset size and composition.Source: §7.1 · p. 16