PaperlayerView exact revision on arXiv

Source-linked AI summary

FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes

Christodoulos Constantinides, Dhaval Patel, Shuxin Lin, Claudio Guerrero, Sunil Dagajirao Patil, Jayant Kalagnanam

arXiv:2506.03278v1cs.CL

TL;DR

FailureSensorIQ addresses the limited availability of comprehensive benchmarks for industrial LLM reasoning about sensors, failure modes, and asset relationships. It builds an ISO- and expert-curated MCQA benchmark and evaluates reasoning, perturbation robustness, expertise, retrieval, and feature selection. Models approach expert-level performance in some settings but remain fragile to perturbations, distractors, and knowledge gaps, while the benchmark’s static focus limits temporal sensor-failure analysis.

  • Problem

    Comprehensive benchmarks for LLM reasoning over industrial challenges such as predictive maintenance, sensor fault detection, and asset management remain limited.

    Source: §1 · pp. 1–2

  • Method

    The paper creates FailureSensorIQ from ISO documents and expert-curated templates, then evaluates LLMs through perturbation analysis, expert studies, asset-level knowledge analysis, retrieval agents, and feature selection.

    Source: §1.2 · p. 3, §2.2 · p. 4, §3.1 · p. 5

  • Results

    Models show substantial performance degradation under perturbations and distractors, while expert knowledge improves performance and retrieval augmentation does not reliably improve model results.

    Source: §1.2 · p. 3, §4 · pp. 8–9

  • Takeaways & Limitations

    FailureSensorIQ provides an industrial reasoning benchmark and LLMFeatureSelect pipeline for studying sensor–failure relationships and modeling decisions across real-world assets.

    Source: §1.2 · p. 3

    The benchmark focuses on static knowledge and does not model temporal sensor-failure dynamics; uneven online knowledge availability also affects benchmarking consistency.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.

1 Introduction

FailureSensorIQ addresses the lack of comprehensive industrial benchmarks by testing LLM reasoning about failure modes, sensors, and asset relationships. It also examines whether models can support predictive-maintenance decisions through reasoning, expert evaluation, perturbation analysis, and feature selection.

1 Introduction

  • FailureSensorIQ targets a gap in industrial QA benchmarks covering predictive maintenance, sensor fault detection, and asset management.Existing efforts address isolated maintenance tasks, but comprehensive benchmarks for these industrial challenges remain limited.Source: §1 · pp. 1–2

1.1 Fundamentals of Condition-Based Maintenance for Industrial Assets

  • Condition-Based Maintenance uses sensor data to monitor asset health, while FM2Sensor and Sensor2FM map failure modes to relevant sensors and sensors to detectable failures.These mappings can support anomaly detection, sensor planning, root-cause analysis, and proactive maintenance tasks.Source: §1 · p. 2, §1.1 · pp. 2–3

1.2 Key Contributions and Insights

  • FailureSensorIQ contains 8,296 questions across 10 industrial assets, including 2,667 single-correct and 5,629 multi-correct questions.Questions use FM2Sensor and Sensor2FM formats and selection-versus-elimination reasoning based on ISO documents and expert-curated templates.Source: §1.2 · p. 3
  • Top-performing models average 53.5% on single-correct questions, while perturbations reduce performance by 5–20%, uncertainty requires about three selected options for 90% coverage, and distractors cap accuracy at 12%.These findings motivate the benchmark’s Perturbation–Uncertainty–Complexity analysis.Source: §1.2 · p. 3
  • LLM reasoning strategies improve performance, with medium-sized models around 70 billion parameters gaining substantially from effective prompting.The evaluated strategies include Chain-of-Thought, Role-Playing Chain-of-Thought, and Self-Plan.Source: §1.2 · p. 3
  • Experts achieved a maximum accuracy of 66.19% and mean accuracy of 60.20%, while external knowledge volume showed only a modest correlation with asset-level model performance.A ReAct agent did not deliver the expected performance improvements, indicating unresolved reasoning and retrieval challenges.Source: §1.2 · p. 3
  • Exact-match performance remains below 21% when models must select two correct answers without being told the answer count.This result highlights the difficulty of avoiding plausible distractors in multi-correct MCQA.Source: §1.2 · p. 3
  • LLMFeatureSelect is an sklearn and LLM-based feature-selection pipeline evaluated on three real-world failure-prediction datasets.The tool reasons about relationships among assets, failure modes, and sensors, producing promising but improvable recommendations.Source: §1.2 · p. 3

2 Methodology

The methodology defines single- and multi-correct industrial MCQA questions and generates them from asset, task, relevance, and answer-type choices. Domain-expert-curated templates and ISO-based mappings produce a structured benchmark with varied option counts and answer distributions.

2 Methodology

  • The dataset contains Single Correct questions with one answer and Multi Correct questions requiring two correct answers from five options.Both formats are generated through the automated question-generation pipeline.Source: §2 · p. 4

2.1 Dataset Structure

  • Each question is represented as (Q, C, O, A), containing the question, option identifier, option contents, and ground-truth answer.The option set O contains k candidate options, while A includes the correct answer’s identifier and content.Source: §2.1 · p. 4

2.2 Question Generation Pipeline

  • The generation pipeline selects an industrial asset, identifies FM2Sensor or Sensor2FM, determines relevance, and specifies whether the answer is single or multiple.These attributes become question metadata and are defined without using an LLM to generate the dataset.Source: §2.2 · p. 4
  • Domain-expert-curated templates and ISO documents generate structured closed-ended questions with answer options and corresponding correct answers.Approximately 10 templates are prepared per relevance and task type, producing 2,667 SC-MCQA questions.Source: §2.2 · p. 4
  • The SC-MCQA corpus contains 752 A, 729 B, 491 C, 408 D, and 208 E selected answers, with 1,525 questions containing five options.The resulting imbalance motivates qualification tests for option-choice bias and response patterns.Source: §2.2 · p. 5

3 Perturbation-Uncertainty-Complexity Analysis

The study evaluates industrial LLM reasoning under perturbation, uncertainty, increased choice complexity, and asset-specific knowledge variation. Across these settings, performance is fragile to perturbations and distractors, uncertainty varies with model scale, and asset accuracy shows a mild association with public documentation volume.

3.1 Perturbed/Complex Dataset Preparation

  • The evaluation combines original and perturbed datasets, zero-shot structured-output prompting, and metrics covering accuracy, consistency, selectivity, coverage, and uncertainty-adjusted accuracy.ComplexPert changes question content and formatting, while OptionsPert adds distractor choices and randomizes the options.Source: §3.1 · p. 5, §3.2 · p. 5

3.3 Real Knowledge Capacity Evaluation using Perturbation Analysis

  • 60.4% on original questions fell to 44.2% under ComplexPert for o1, while llama-4-scout dropped from 54.0% to 24.1%, showing substantial perturbation sensitivity.The reported declines were statistically significant by a Wilcoxon signed-rank test with alpha = 0.1.Source: §3.3 · p. 5
  • ACC@Consist exceeded random guessing for every model but remained below 50%, indicating partial knowledge retention under perturbed conditions.For k = 4 options, the random-guessing baseline is 6.25%.Source: §3.3 · p. 6, Figure 4 · p. 6
  • All tested models had negative Performance Drop Ratios under ComplexPert, indicating broadly reduced performance after knowledge-invariant perturbations.The authors describe this as vulnerability to content-level paraphrasing and format-level changes.Source: §3.3 · pp. 6–7, Table 2 · p. 6

3.4 Uncertainty Quantification Analysis

  • Mistral-large was better calibrated than smaller models, which often selected larger answer sets with lower accuracy despite high coverage.The analysis reports a coverage guarantee only at a higher set size and identifies calibration as a challenge for smaller models.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 Question Complexity Analysis

  • o1 accuracy dropped from 60% to 5.74% when distractor options were introduced, highlighting vulnerability to increased choice complexity and ambiguity.OptionsPert reduces random guessing and enables systematic evaluation of robustness under increased ambiguity; more than 10 options are common in some real-world scenarios.Source: §3.5 · p. 7

3.6 Knowledge Gap Study for Industrial Assets

  • Steam Turbine and Electric Motor achieved the lowest reported o1 accuracy at 43.59%, while asset accuracy showed a mild positive correlation with Arxiv documentation volume.The authors also note persistent variation associated with terminology ambiguity, data formatting, and documentation consistency.Source: §3.6 · pp. 7–8, Table 5 · p. 8, Figure 6 · p. 8

4 Experimental Results: Reasoning, Agent and Experts

Reasoning-oriented prompting improves llama-model performance, whereas external retrieval is unreliable; expert evaluation shows that domain expertise materially improves industrial QA accuracy.

Reasoning-Based Prompting

  • 56.90% average accuracy for 4-Maverick-17B-128E exceeded 3.1-8b’s 40.27%, while llama-3-70b rose from 41.69% to 51.18% with CoT.CoT prompting enabled smaller models to match or outperform larger models using direct prompting, and reasoning-oriented prompts consistently enhanced performance across scales.Source: §4 · p. 8
  • The benchmark’s reasoning-model comparison is organized in Table 6, which reports performance across different reasoning models for llama models.The supplied table caption identifies the comparison scope, while the accompanying passages report the associated reasoning-prompting findings.Source: Table 6 · p. 8, §4 · p. 8

AI Agent with External Knowledgebase

  • External knowledge retrieval did not reliably improve llama-model accuracy, with llama-4-maverick dropping from 55.83% to 48.44%.The ReAct agent issued 2–3 interleaved Wikipedia and arXiv queries per question, but only one of five llama models showed marginal improvement.Source: §4 · p. 8

Human Evaluation

  • Domain experts outperformed the practitioner by over 25 percentage points on average and by up to 39.45 points in Quiz 1.Expert agreement was moderate overall, with Cohen’s Kappa κ = 0.462, indicating nuanced reasoning demands.Source: §4 · p. 9, Table 7 · p. 9

5 LLMFeatureSelect: scikit-learn Transformer

LLMFeatureSelector uses prompts to recommend industrial sensor features and evaluates them against next-timestep targets across three real-world datasets.

LLMFeatureSelect

  • LLMFeatureSelector recommends relevant features from a problem statement, sensor names, and target variable using a prompt-based method.The approach is implemented as a scikit-learn pipeline and tested on three open-source, real-world datasets.Source: §5 · p. 9

Evaluation

  • The evaluation computes correlations between recommended features and the next-timestep target variable.Absolute correlations between the top-five recommended sensors and the target variable are reported in Table 8.Source: §5 · p. 9, Table 8 · p. 9

6 Limitations and Future Work

FailureSensorIQ currently evaluates static knowledge rather than temporal sensor-failure dynamics, with benchmark consistency also affected by uneven online knowledge availability across assets.

Limitations and Future Work

  • FailureSensorIQ does not model temporal sensor-failure dynamics, limiting evaluation of how LLMs integrate dynamic signals into feature selection.Future work will extend the benchmark with temporal reasoning tasks.Source: §6 · p. 9
  • Uneven online knowledge availability across assets causes performance variation and affects benchmarking consistency.The authors identify reasoning-aware and agentic systems tailored to industrial diagnostics as an opportunity for future innovation.Source: §6 · p. 9

A Dataset Benefits for AI and Industrial Research

The dataset supports industrial reasoning and knowledge-integration research through structured MCQA generation, asset mappings, perturbation pipelines, and configurable experimental infrastructure.

A Dataset Benefits for AI and Industrial Research

  • The dataset enables MCQA reasoning analysis, ReAct retrieval experiments, model-scale transfer studies, knowledge-invariant testing, domain embeddings, synthetic data generation, and IoT validation.These applications span reasoning, knowledge integration, and domain-specific industrial use cases.Source: A Dataset Benefits for AI and… · p. 14

Algorithm 1: Generate Multiple-Choice Questions

B.1 Question Templates

  • Templates cover FM2Sensor and Sensor2FM tasks in selection and elimination forms for identifying relevant or irrelevant sensors and failure modes.The templates support analysis of sensor–failure-mode interactions for condition-based monitoring and reliability analysis.Source: §B.1 · p. 15, Table 9 · p. 15

B.2 Raw Data

  • The raw data contains two knowledge-graph mappings for each of 10 industrial assets.Figure 8 provides a knowledge-graph view of these mappings.Source: §B.2 · p. 16, Figure 8 · p. 16

B.3 Perturbed Pipeline

  • The perturbation pipeline applies option encoding, permutation, formatting changes, question-type changes, paraphrase, and element swapping to create easy or hard perturbed questions.Transformations are applied sequentially, with harder outputs produced after more extensive modifications.Source: §B.3 · pp. 16–17, Figure 9 · p. 16

C Experimental Configuration

  • Experiments use IBM watsonx and Microsoft Azure, with unavailable models run on an internal cluster containing 8 × Nvidia A100 GPUs and 16 × AMD EPYC 7763 CPUs.The cluster also includes 128 GB RAM and a 1 TB SSD.Source: C Experimental Configuration · p. 17

D Result Trace Analysis Demonstration

A trace analysis compares how three LLMs reason about sensor selection for a fuel-filter blockage, showing that deeper causal reasoning does not guarantee the correct answer. The supplied materials also identify reasoning as central to benchmark performance and introduce prompting strategies for induced reasoning.

D Result Trace Analysis Demonstration

E.1 Leaderboard

  • The benchmark evaluates 24 LLMs on SC-MCQA and ComplexPert, with leaderboard results summarized across multiple performance dimensions.Source: §E.1 · p. 20, Table 10 · p. 20

E.2 Trigger Statement for Induced Reasoning

  • The study uses trigger statements for Chain-of-Thought prompting and a Plan-Solve mechanism to induce reasoning in LLMs.Source: §E.2 · p. 20

E.3 Probing External Knowledge Source

The paper probes external document volumes to characterize asset-specific knowledge exposure, then tests whether external retrieval improves industrial MCQA. The results indicate uneven knowledge coverage, limited benefit from fine-tuning on general QA data, and performance degradation with ReAct retrieval.

E.3 Probing External Knowledge Source

  • The study estimates document volumes through keyword queries across Wikipedia, arXiv, CrossRef, and Google, including a CrossRef-to-arXiv research-intensity ratio.Steam turbine has a ratio of 91.7, whereas electric generator has 0.10.Source: §E.3 · pp. 20, 23
  • External knowledge coverage is uneven: CrossRef has over 1.8M entries for industrial gas turbine, while some assets have strong research coverage but limited public-web visibility.Source: §E.3 · p. 23
  • Asset representations diverge across sources, with fan exceeding 253K Wikipedia articles but only 20K arXiv entries, while electric motor has 75K arXiv entries and 33.5K Wikipedia articles.Source: Figure 14 · p. 22, Figure 15 · p. 22, Figure 16 · p. 22
  • Public visibility can differ from scholarly attention: pump and fan have 1.4B and 7.08B Google hits despite comparatively limited academic representation.Source: §E.3 · p. 23

E.3.1 Implications for Model Exposure and Bias

  • Asset-specific exposure may shape model reliability: higher document volumes may influence pretraining, while missing modalities can create knowledge blind spots or biases.The paper links these variations to evaluating knowledge-dependent technical QA.Source: §E.3.1 · pp. 23–24

E.4 Fine-tuning on HotpotQA

  • Fine-tuning Flan-T5-XL on HotpotQA questions and answers reduced accuracy from 36% to 29%, which the authors attribute to catastrophic forgetting and insufficient industrial-asset information.Source: §E.4 · p. 24

F Does External Knowledge Help?

  • ReAct retrieval does not consistently improve MCQA: LLaMA-3-70B falls from 41.7% baseline accuracy to 37.6%, while most evaluated models decline.The authors report more invalid or malformed responses under ReAct for several models.Source: F Does External Knowledge Help? · p. 24
  • Successful performance appears to require integrating latent knowledge and fine-grained distractor reasoning, which external retrieval alone does not capture.The findings position the benchmark as a test of internal reasoning rather than surface factual recall.Source: F Does External Knowledge Help? · p. 24

G Human Evaluation in Dataset Development

The paper describes expert-centered human evaluation practices and details an uncertainty-aware accuracy procedure. Human evaluation supports dataset validation, while the uncertainty protocol uses asset-disjoint calibration and test sets to produce variable-size prediction sets.

G Human Evaluation in Dataset Development

  • Human evaluation supports answer validation, question assessment, distractor checking, and human baselines in domain-specific benchmarks.The paper’s evaluators are IBM industrial product experts who assessed dataset correctness and difficulty.Source: G Human Evaluation in Dataset… · p. 24
  • The paper contrasts evaluator roles across benchmarks, including answering questions, verifying correctness, assessing coherence, and reviewing distractors.Source: G Human Evaluation in Dataset… · p. 25
  • Across related benchmarks, human evaluation commonly covers 10–20% of samples, but expert involvement remains resource-intensive and limits dataset scalability and update frequency.Source: G Human Evaluation in Dataset… · p. 25

H Uncertainty Quantification Details

  • Uncertainty-aware evaluation splits asset types without overlap between calibration and test sets, then computes conformal scores to select options exceeding a confidence threshold.The test prediction can contain a variable number of options.Source: H Uncertainty Quantification Details · p. 25
  • The protocol uses complete-question base prompting, direct option selection from model logits, and an error tolerance of α = 0.1.The stated confidence target is at least 0.9 coverage of the true label.Source: H Uncertainty Quantification Details · pp. 25–26
  • UAcc is scaled by 100 and can exceed 100 when model certainty is high and prediction sets are small.Source: H Uncertainty Quantification Details · p. 26

I Experimental Results on MC-MCQA

The MC-MCQA-Direct evaluation tests whether models can jointly select exactly two correct answers, while the paper also situates FailureSensorIQ alongside three predictive-maintenance and forecasting datasets. Exact-match performance remains low despite moderate partial-match scores.

I Experimental Results on MC-MCQA

J.1 Electrical Transformer Predictive Maintenance

  • The electrical-transformer dataset contains 48 IoT time-series variables recorded every 15 minutes from June 2019 to April 2020 for predicting magnetic oil gauge faults.Source: §J.1 · p. 26

J.2 Air Compressor Predictive Maintenance

  • The air-compressor dataset contains 19 sensor variables and five failure modes, with top-five sensor recommendations and correlations provided for each included mode.Motor failure is omitted because the dataset contains no motor-failure instances.Source: §J.2 · p. 26

J.3 Wind Mill Power Production Forecasting

  • The windmill dataset predicts wind power over the next 15 days using 20 sensor variables.Source: §J.3 · p. 26

J.4 Findings

LLMFeatureSelect shows promising but uneven alignment with empirical sensor importance across industrial failure and forecasting tasks. A separate expert quiz evaluated generated outputs using domain expertise without external reference materials.

J.4 Findings

  • In 3 of 6 test cases, the most highly correlated signal appeared among the LLM’s top-5 suggestions.This indicates promising alignment between model predictions and empirical sensor importance in the evaluated tasks.Source: §J.4 · p. 26
  • LLMFeatureSelect prompts the model to rank variables by relevance for predicting motor failure in air compressors.The input includes sensor descriptions and requests ordered selected variables with reasoning.Source: §J.4 · p. 27
  • Production-forecasting features reached correlations up to 94.40%, showing that the LLM surfaced high-value variables for forecasting and failure prediction.Source: §J.4 · p. 27
  • Recommendations were weaker for Bearings and Valve failures, where low-correlation features were selected, while the most correlated Magnetic Oil Gauge feature ranked third.The findings suggest semantic or contextual cues may influence rankings alongside raw correlation, while highlighting current limitations.Source: §J.4 · p. 28

K Human Evaluation via Expert Quiz

  • The human evaluation used an expert quiz simulating asset-management decisions, with participants relying only on their domain expertise and the presented quiz context.Participants had no access to ISO standards, reference documents, source datasets, or external information.Source: K Human Evaluation via Expert Quiz · p. 28

Related papers

Loading 2506.03278v1…