PaperlayerView exact revision on arXiv

Source-linked AI summary

DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

Devin Yasith De Silva, Dhaval Patel, Christodoulos Constantinides, Shuxin Lin, Nianjun Zhou, Paul J Adams, Sal Rosato, Nicolas Constantinides, Deborah L. McGuinness, Jayant Kalagnanam

arXiv:2605.08614v1cs.AI

TL;DR

Industrial maintenance still lacks systematic evidence on whether LLMs can recommend actions after symbolic rules trigger, a downstream task requiring asset-specific expertise. DiagnosticIQ benchmarks this capability across expert-validated rule-action questions and variants, finding frontier models achieve strong baseline accuracy but remain brittle under structural perturbations.

  • Problem

    Systematic evaluation is missing for LLM decision support that recommends maintenance actions after symbolic rules trigger on known industrial assets.

  • Method

    DiagnosticIQ converts expert-authored rules and actions into an expert-validated MCQA benchmark with targeted variants probing reasoning, generalization, and robustness.

  • Results

    73.6% Macro is achieved by the leading model, while frontier models cluster within one point and lose 13.8 points on DiagnosticIQ Pro.

  • Takeaways & Limitations

    LLMs handle template-style fault detection but treat structured rules as templates rather than conditions, limiting readiness for high-stakes operational deployment.

  • Takeaways & Limitations

    The benchmark covers English-language rules under zero-shot evaluation, excluding multilingual rules, agentic settings, and the full industrial asset taxonomy.

Abstract

from arXiv · show

Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce \ours{}, a benchmark of 6{,}690 expert-validated multiple-choice questions from 118 rule-action pairs across 16 asset types. We contribute (i) a symbolic-to-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding-based distractor sampling, (ii) five variants probing distinct failure modes (Pro, Pert, Verbose, Aug, Rationale), and (iii) a benchmark of 29 LLMs and 4 embedding baselines. A human evaluation (9 practitioners, mean 45.0\%) confirms \ours{} requires specialist knowledge beyond operational experience. Three findings stand out. The frontier has closed: the top three LLMs lie within one Macro point, with Bradley-Terry Elo placing claude-opus-4-6 30 points above the next model. Yet \ours{}\,Pro exposes brittleness, with every model losing 13--60\% relative accuracy under distractor expansion. \ours{}\,Aug exposes pattern-matching: under condition inversion, frontier models still select the original answer 49--63\% of the time. The deployment bottleneck is not capability but calibration: frontier models handle template-style fault detection but break under structural perturbation.

1 Introduction

DiagnosticIQ targets the unresolved rule-to-action bottleneck in industrial maintenance by evaluating whether LLMs can recommend actions from triggered symbolic conditions on known assets. It introduces an expert-grounded MCQA benchmark and targeted variants, revealing that current frontier models remain unsaturated and lose substantial performance under distractor expansion.

  • Scope of this work: DiagnosticIQ addresses the action bottleneck by taking a triggered symbolic condition on a known asset as input and recommending an appropriate maintenance action.The work does not study LLMs as time-series reasoners, but complements research on rule discovery, predictive maintenance, and signal monitoring.
  • Scope of this work: The benchmark is grounded in real-world rules and expert-validated actions, with a primary MCQA dataset and variants testing capabilities such as reasoning, generalization, and robustness.The suite is designed for industrial maintenance action recommendation rather than anomaly detection alone.
  • Contributions: A deterministic symbolic-to-MCQA pipeline normalizes expert-authored rules, including temporal predicates, into Disjunctive Normal Form and samples diverse plausible distractors using embeddings.The pipeline expands rule-to-action coverage while constructing challenging multiple-choice alternatives.
  • Evaluation: The evaluation benchmarks 29 LLMs across 9 families and 4 embedding baselines on DiagnosticIQ and five targeted variants: Pro, Aug, Verbose, Pert, and Rationale.Analyses cover question complexity, asset type, robustness, fine-tuning, rationale validation, and operational impact.
  • Summary of Empirical Insights: 73.6% Macro is achieved by the leading model, which loses 13.8 points on DiagnosticIQ Pro, while claude-opus-4-6, gpt-5.4, and gemini-3.1 cluster within one point.DiagnosticIQ remains unsaturated for current frontier models; model selection also reflected compute budget, availability, and family-coverage priorities.

2 Related Work

Prior benchmarks cover domain-grounded QA, distractor robustness, compositional instruction following, and maintenance detection, but none standardize downstream rule-to-action recommendation. DiagnosticIQ fills this gap for industrial maintenance by probing logical sensitivity, rationale evaluation, and cross-asset transfer across 16 asset types.

  • MCQA robustness and cross-asset transfer: DiagnosticIQ extends robustness and rationale-based evaluation to industrial maintenance, using its templated structure to probe logical sensitivity and cross-asset transfer.This work builds on prior studies of MCQA bias, benchmark robustness, contamination detection, evaluation methodology, and rationale evaluation in healthcare [17] [43] [48] [37].
  • Industrial-asset Specific (IS): DiagnosticIQ extends industrial maintenance benchmarks beyond aviation, wind-turbine logs, prognostics, and sensor-failure association by evaluating rule-to-action transfer across 16 asset types.Related domain-grounded QA benchmarks span telecommunications, climate, finance, healthcare, IT operations, and power plants.
  • Condition Understanding (CU): DiagnosticIQ targets compositional maintenance-rule understanding that ComplexBench does not cover because it lacks sensor predicates, asset-specific terminology, and temporal duration constraints.The benchmark treats deployed rules as encoded domain knowledge relevant to production maintenance systems.
  • Action Recommendation (AR): DiagnosticIQ standardizes the downstream rule-to-action decision—choosing a technician’s maintenance response when a rule fires—which prior maintenance benchmarks have not addressed.Existing work primarily targets fault recognition, log classification, sensor-failure association, rule discovery, and predictive maintenance.

3 Symbolic Conditions to MCQA

DiagnosticIQ deterministically converts expert-authored industrial rules into grounded multiple-choice questions by structuring conditions, expanding them into valid trigger scenarios, and sampling operationally adjacent distractors. The pipeline supports complementary selection and elimination questions across 118 rules spanning 16 asset types.

  • 3.1 Rule Normalization: The pipeline parses expert rules into condition trees, normalizes them to DNF, and composes grounded questions with asset metadata, conditions, prompts, options, and single-correct answers.Question conditions preserve triggering durations where applicable, making temporal sensitivity testable.
  • 3.3 Dataset Scope: The benchmark covers 118 rules across 16 asset types, with disjunction counts bounding the branching complexity and number of derivable trigger scenarios.Asset descriptions contextualize questions for configuration differences; removing them reduces accuracy by 8.6–10.2% across three models.
  • 3.2 Question Construction: Selection questions identify the most relevant supported action, while elimination questions identify an unsupported action, probing complementary condition-action reasoning skills.Selection uses distractors from dissimilar rules, while elimination reverses the polarity and tests recognition of unsupported mappings.
  • 3.2 Question Construction: Embedding-based Rule-Rule Similarity retrieves semantically dissimilar rules whose observations provide operationally adjacent, non-correct distractors for question construction.The method embeds condition-tree representations with all-mpnet-base-v2 [30] and uses cosine similarity for retrieval.
  • 3.1 Rule Normalization: DNF expansion turns each rule into multiple logically valid conjunctive trigger scenarios, rather than a single example, matching production monitoring execution.For example, (c1 ∨ c2) ∧ c3 yields two scenarios: (c1 ∧ c3) and (c2 ∧ c3).

4 DiagnosticIQ

DiagnosticIQ is built from 118 expert-curated rules into 6,690 questions, preserving operationally motivated asset and action-set variation. Five variants probe difficulty, formatting sensitivity, reasoning traces, linguistic paraphrase, and logical-content robustness.

  • Dataset construction: DiagnosticIQ contains 6,690 questions generated from 118 expert-curated rules, with selection questions comprising 77.4% and elimination questions 22.6%.The pipeline uses Nsel_topk = 25, Neli_topk = 25, NQT = 10, α = 10, and β = 10.
  • Dataset composition: AHU scenarios dominate DiagnosticIQ at 58%, followed by Chiller and Boiler at 7% and 6%, preserving operationally motivated coverage imbalance.The skew reflects AHU prevalence at facility sites and their share of energy-savings opportunities.
  • Dataset construction: Elimination questions retain rule-specific action-set sizes, producing variable option counts rather than truncating incorrect options.Selection questions use fixed option sets, whereas elimination distractors come from each source rule’s action set.
  • Benchmark variants: DiagnosticIQ Pro expands questions from 4 to 10 options, while Pert changes ordering and wording without changing semantics to test distractor discrimination and formatting sensitivity.The Pro design follows FailureSensorIQ [8] and MMLU-Pro [39], while PertEval [17] motivates Pert’s construction.
  • Benchmark variants: Rationale collects model reasoning traces for expert ratings and fine-tuning, while Verbose rewrites symbolic conditions in natural language to isolate the symbolic-to-linguistic gap.Rationale evaluates knowledge accuracy and operational relevance; its rationales also provide supervision for fine-tuning.
  • Benchmark variants: DiagnosticIQ Aug inverts temporal comparison operators and adds “None of the above” to isolate logical-content sensitivity from rule-shape recognition.The operators invert as <↔>, ≤↔≥, with manual validity verification.

5 Experimental Results

DiagnosticIQ reveals a narrowing frontier but persistent brittleness: leading LLMs cluster on the standard benchmark yet degrade sharply under distractor expansion, condition inversion, paraphrasing, and prompt perturbation. These failures reflect pattern matching and dependence on symbolic structure, while operational impact and adaptation outcomes can diverge from aggregate accuracy.

  • Overall benchmark results: Three latest-generation models fall within one Macro point on DiagnosticIQ, with claude-opus-4-6 ranked 30 Bradley-Terry Elo points above gemini-3.1-pro-preview.The Elo rating breaks the benchmark tie despite overlapping 95% Wilson confidence intervals.
  • DiagnosticIQ Pro: 13.78 Macro points separate claude-opus-4-6 on DiagnosticIQ from DiagnosticIQ Pro (73.59% →59.81%), and only 5 of 29 models exceed 50% on Pro.Relative accuracy losses span 19% for frontier models to over 60% for smaller models, showing that distractor expansion exposes transferable brittleness.
  • Embedding baselines: 52.73% Macro on DiagnosticIQ and 38.89% on DiagnosticIQ Pro make all-mpnet-base-v2 the strongest embedding baseline, surpassing 14 of 29 LLMs on Pro.Half the leaderboard cannot beat unsupervised cosine similarity after distractor expansion, including reasoning and large open-weight models.
  • Operational impact: Accuracy does not determine operational value: gemini-3.1-pro-pre leads savings at USD 3.13–4.82M, while accuracy leader claude-opus-4-6 ranks second.The savings ranking diverges because consistently answering high-impact rules matters more than small aggregate accuracy differences.
  • DiagnosticIQ Aug: 48.8–63.4% of models retain the original answer after temporal-operator inversion, while only 15.5–16.6% select “None of the above.”claude-opus-4-6 shows the strongest logical inertia at 63.4%, despite leading DiagnosticIQ.
  • DiagnosticIQ Verbose: Paraphrasing symbolic conditions reduces Macro accuracy by 4.2 points on average across 17 models, with individual declines ranging from −1.8 to −9.6.The result indicates that models use symbolic structure as reasoning scaffolding, which paraphrasing erodes.
  • DiagnosticIQ Pert: No model exceeds 51.5% consistency across original and perturbed phrasings, while the best frontier model reaches approximately 48% on DiagnosticIQ Pert.gpt-5.4 has the largest significant PDR decline (−0.05**), whereas claude-opus-4-6 and gemini-3.1-pro-pre show negligible PDRs (±0.01).
  • Cross-asset generalization: On 11 held-out SME-authored rules, one-shot DiagnosticIQ exemplars improve MAP@1 from 41.7% to 58.3%.Fine-tuning on AHU rules also improves three 8B-scale base models on Other assets, although the best method depends on the base model’s reasoning structure.

6 Conclusion and Limitations … A.9.1 Logical inversion via condition flipping.

DiagnosticIQ exposes a gap between template-style fault detection and condition-based reasoning: leading models reach 73.6% Macro but fall to 59.8% on Pro and about 48% consistency on Pert, while condition inversion causes systematic selection of obsolete actions. The benchmark’s broader analyses show that asset context affects accuracy, estimated savings can diverge from accuracy rankings, and the released pipeline generalizes beyond its primary rule corpus.

  • 6 Conclusion and Limitations: 73.6% Macro for the leading model falls to 59.8% on Pro and approximately 48% consistency on Pert, establishing a brittleness arc under distractor expansion and perturbation.Paraphrasing and operator inversion indicate that current LLMs treat structured rules as templates rather than conditions to evaluate.
  • A.1 Asset Type Descriptions; A.2 Rule Provenance and Curation: The benchmark comprises 6,690 expert-validated MCQs from 118 production rules spanning 16 equipment types, with rules originating in a seven-year, 11-update operational corpus targeting 5–15% energy savings and 30% fewer maintenance hours.The rules were authored and refined by domain experts, including reliability engineers, system administrators, and a rules-logic committee.
  • A. Technical appendices and supplementary material; A.6 DiagnosticIQ Variants; A.6.1 DiagnosticIQ Pro; A.6.2 DiagnosticIQ Pert; A.6.3 DiagnosticIQ Verbose; A.6.4 DiagnosticIQ-Aug: DiagnosticIQ’s variants probe distinct failure modes: Pro expands options from 4 to 10, Pert changes surface form while preserving semantics, Verbose paraphrases symbolic conditions, and Aug inverts temporal operators while adding “None of the above.”The appendix supplies construction details, ablations, per-asset analyses, selection-versus-elimination gaps, universal-failure questions, Elo ratings, and human-evaluation materials.
  • A.7 Asset Description in the Prompt: Removing the expert-written asset description reduces Macro Accuracy by 8.6–10.2 points across three models, including an 8.77-point loss for claude-opus-4-6.This shows that asset context is load-bearing rather than presentational decoration.
  • A.8 Real-World Application: Estimated Energy Savings: USD 3.13–4.82M in estimated savings for the leading model demonstrates operational value, while gemini-3.1-pro-pre achieves the highest savings despite ranking second on Micro accuracy at 75.92% versus 77.76% for claude-opus-4-6.Savings sum expert-authored per-rule ranges only for correct answers, assigning incorrect recommendations zero savings; the authors therefore release annotations and propose impact-weighted evaluation.
  • A.9.1 Logical inversion via condition flipping.: 63% of Claude-Opus-4-6 responses and 50% and 49% of Mistral-Medium and Mistral-Small responses select the obsolete action after condition inversion, although “None of the above” is the only valid answer.The old action dominates by a 3× to 4× margin, revealing systematic pattern matching rather than evaluation of inverted conditions.

A.9.2 Rationale evaluation by domain experts. · A.10 What Makes a Question Hard? Difficulty Decomposition

Expert review finds mistral-large rationales generally acceptable on unperturbed inputs but uneven across assets, while condition inversion exposes failures in logical sensitivity. Difficulty analysis shows arithmetic and comparison operations are the strongest structural predictors of LLM errors, motivating targeted procedural or tool-based support.

  • A.9.2 Rationale evaluation by domain experts.: Experts generally rated mistral-large rationales favorably on unperturbed inputs, supporting their potential to augment maintenance tasks.Five domain experts rated 27 representative rationales for knowledge accuracy and operational relevance on 0–10 Likert scales.
  • A.9.2 Rationale evaluation by domain experts.: Rationale quality varies by asset: PDU, Pump, and Boiler show high inter-rater disagreement, while Plate & Frame HX assets receive consistently lower scores.Experts attributed disagreement partly to differing expectations about explanation granularity.
  • A.9.2 Rationale evaluation by domain experts.: Models can generate expert-acceptable reasoning when rule structure matches the underlying logic, yet condition flipping reveals failures to detect logical inversion.Holding rule structure constant while changing logical content isolates logical sensitivity from surface pattern recognition.
  • A.10 What Makes a Question Hard? Difficulty Decomposition: The difficulty model extracts logic complexity, special-function count, normalized question length, and number of options from all 6,690 DiagnosticIQ core questions.The four-option count is included as a confounding control, and correctness is modeled for claude-opus-4-6 using logistic regression with Wald z-tests.
  • A.10 What Makes a Question Hard? Difficulty Decomposition: Aggregate accuracy can conceal structural failures because models may perform well on simple rules but struggle on math-heavy ones.The analysis frames structural decomposition as useful for identifying which DiagnosticIQ subsets are hardest.
  • A.10 What Makes a Question Hard? Difficulty Decomposition: 47% lower odds of correctness per added arithmetic or comparison operation make special-function count the strongest failure predictor for claude-opus-4-6.The logistic regression uses DiagnosticIQ core (n = 6,690) and the model’s 73.59% baseline accuracy; three of four features are highly significant (p < 0.001).
  • A.10 What Makes a Question Hard? Difficulty Decomposition: Fine-tuning on arithmetic-heavy maintenance rules or adding explicit numerical-evaluation tools are proposed interventions for the special-function deficit.Per-question feature annotations are provided in the dataset release.

A.11 Per-Asset and Per-Question-Type Analysis · A.12 DiagnosticIQ Verbose: Per-Model Symbolic-to-Linguistic Gap

DiagnosticIQ performance varies sharply by asset and question type: structurally complex assets and selection questions are especially vulnerable to distractor expansion. Linguistic paraphrasing causes a smaller average decline, though frontier models were not evaluated on the Verbose variant.

  • A.11 Per-Asset and Per-Question-Type Analysis: Structurally complex assets suffer the largest Pro losses, while PDU remains unchanged at 60.0% because its rules are short and its failure mode is consistent across distractor counts.Cooling towers span chiller and ambient state, whereas pumps depend on flow and pressure across multiple sensors; PDU is an exception.
  • A.11 Per-Asset and Per-Question-Type Analysis: 13.8 points is the average Pro decline across assets, with Pump, HXU, and Cooling Tower losing 27.4, 25.0, and 24.5 points, respectively.UPS, Plate & Frame, Boiler, and HXU exceed 85% on core DiagnosticIQ, whereas CRAC, Meter, Cooling Tower, and PDU remain below 65%.
  • A.11 Per-Asset and Per-Question-Type Analysis: Selection is harder than elimination: mistral-large scores 63.9% versus 71.0% on DiagnosticIQ, while Pro reduces selection by 35.0 points but elimination by only 8.4.The pattern replicates across model families, indicating that ranking the relevant action among adjacent alternatives is harder than identifying an unsupported action.
  • A.12 DiagnosticIQ Verbose: Per-Model Symbolic-to-Linguistic Gap: The Verbose evaluation covered 15 prior-generation models and reported Macro accuracy alongside each model’s change from DiagnosticIQ core.Negative changes indicate accuracy loss under linguistic paraphrasing.
  • A.12 DiagnosticIQ Verbose: Per-Model Symbolic-to-Linguistic Gap: Across 15 prior-generation models, DiagnosticIQ Verbose lowers mean accuracy by 4.3 points, or 4.7 excluding llama-3-1-8b’s anomalous +1.69.The largest absolute drop occurs for llama-3-3-70b.
  • A.12 DiagnosticIQ Verbose: Per-Model Symbolic-to-Linguistic Gap: claude-opus-4-6, gpt-5.4, and gemini-3.1-pro-pre were not evaluated on DiagnosticIQ Verbose because of compute constraints, leaving frontier-model performance as future work.The reported Verbose results therefore concern earlier-generation models rather than the latest frontier systems.

A.13 Ablation: Condition Tree vs. Conjunctive Scenarios … B Extended Empirical Analyses

Extended analyses show that preserving full condition trees yields only modest gains over conjunctive scenarios, while one-shot prompting best supports generalization to unseen rules. Error severity rises with larger answer sets, and the appendix documents reproducibility settings, fine-tuning protocols, ethical safeguards, and additional per-question analyses.

  • A.13 Ablation: Condition Tree vs. Conjunctive Scenarios: Condition-tree prompts improve Macro Accuracy by 1.60, 1.20, and 0.94 points for claude-opus-4-6, mistral-medium, and mistral-small, respectively, but conjunctive scenarios remain the deployment-aligned choice.Production rules execute one fired conjunction at a time, while condition-tree expansion can create combinatorially many scenarios; both variants are released.
  • A.14 Few-Shot Generalization to Unseen Rules: On 11 SME-authored unseen rules, mistral-large reaches 58.33% MAP@1 with one in-context example versus 41.67% zero-shot, while additional examples plateau at 50.00%.The result suggests one example anchors the rule-to-action format, but the small sample limits firm conclusions.
  • A.15 Set-Size Error Severity Analysis: Semantic-rank analysis measures whether wrong answers are near the correct option in embedding space, with rank 1 denoting the most semantically similar incorrect choice.The appendix also defines this metric for interpreting the nature of model errors.
  • A.15 Set-Size Error Severity Analysis: Incorrect predictions consistently involve larger average answer sets than overall questions across DiagnosticIQ and DiagnosticIQ Pro, indicating systematic error severity linked to option-set size.DiagnosticIQ Pro’s harder 10-option variant widens the Set.Wrong–Set.All gap relative to the core dataset.
  • A.16.2 LLM Inference Setup: LLM inference uses temperature = 0 and maximum tokens = 4096, with all other parameters left at their defaults.These fixed settings support reproducibility of the inference runs.
  • A.16.3 Fine-Tuning Experiments Setup: Fine-tuning uses QLoRA-based SFT and GRPO on 4× Nvidia A100 80GB GPUs, with explicitly specified training schedules, learning rates, batch sizes, and model-specific output formats.Missing answers are scored as incorrect; think fields are omitted for SFT.
  • A.18 Ethics Statement: The study reports compliance with ethical and legal standards for human subjects, privacy, fairness, and data security, with data planned for release on Hugging Face and/or Kaggle.The human evaluation involved nine anonymous adult engineering practitioners, labeled R1–R9, and no consequential decisions were based on their responses.
  • B Extended Empirical Analyses: Extended empirical analyses aggregate per-question evaluation results across all 29 LLMs on DiagnosticIQ simpleV.This appendix-level analysis provides the basis for four additional analyses.

B.1 Universal-Failure Questions · B.2 Wilson 95% Confidence Intervals on the Main Leaderboard

Universal failures affect 2.8% of positive-type questions, with CRAC disproportionately represented and associated with multi-step diagnostic reasoning that current LLMs uniformly handle poorly. Wilson intervals show overlapping top-three Macro accuracies but non-overlapping example Micro intervals, motivating separate reporting of both metrics and Bradley-Terry analysis.

  • B.1 Universal-Failure Questions: 2.8% of 5,180 simpleV positive-type questions are universal failures, while 3.4% are universal passes.The remaining questions occupy a roughly U-shaped mid-range, with more questions where 20+ models succeed than where 5 or fewer succeed.
  • B.1 Universal-Failure Questions: 11.4% is CRAC’s universal-failure rate, compared with 0% for UPS, HXU, and Plate & Frame and elevated rates for PDU, Air Compressor, and Lighting/Fan.CRAC contributes 32 of 144 universal failures despite representing only 280 of 5,180 questions; PDU, Air Compressor, and Lighting/Fan have rates of 6.7%, 5.5%, and 5.0%.
  • B.1 Universal-Failure Questions: Current LLMs are uniformly weak on CRAC questions requiring nested temporal conditions, compound causal chains, and genuine multi-step diagnostic reasoning.These patterns involve tracking multiple stateful variables and diagnosing upstream causes rather than matching single patterns.
  • B.1 Universal-Failure Questions: Universal-failure questions concentrate on a small set of rules, whose full list is provided as supplementary training data for industrial diagnostic reasoning.The supplementary file is universal_failures_full.csv.
  • B.2 Wilson 95% Confidence Intervals on the Main Leaderboard: Wilson score intervals are computed from existing per-question correctness data, using direct Wilson intervals for Micro accuracy and delta-method propagation across per-asset Wilson intervals for Macro accuracy [5].No model re-evaluation is required, and Wilson intervals provide better coverage near 0 and 1 than Wald intervals.
  • B.2 Wilson 95% Confidence Intervals on the Main Leaderboard: 73.59 ± 2.21, 73.27 ± 2.01, and 72.74 ± 1.94 are overlapping 95% Macro CIs for claude-opus-4-6, gpt-5.4, and gemini-3.1-pro-pre, respectively.The Macro ranking among the top three is not statistically distinguishable from accuracy alone; Table 17 reports the per-cell confidence intervals.
  • B.2 Wilson 95% Confidence Intervals on the Main Leaderboard: Bradley-Terry Elo places claude-opus-4-6 30 Elo points above gemini-3.1-pro-pre using 633,252 pooled pairwise comparisons, supplementing accuracy-based interval comparisons.The passage identifies this as the motivation for relying on Bradley-Terry Elo with bootstrap confidence intervals.
  • B.2 Wilson 95% Confidence Intervals on the Main Leaderboard: 0.80–1.20% Micro CI half-widths are roughly half the 1.94–2.56% Macro half-widths across all 30 models.Macro averages 16 per-asset proportions, whereas Micro pools approximately 5,000 questions; for example, claude-opus-4-6 has 77.76 ± 1.00 Micro accuracy versus gemini-3.1-pro-pre at 75.92 ± 1.02.

B.3 Bradley-Terry Elo Ratings … C.9 LLM Population on the Same 40 Questions

Bradley-Terry Elo confirms a tightly clustered frontier while distinguishing claude-opus-4-6 as the statistically higher-rated model. Human evaluation shows that DiagnosticIQ remains difficult for practitioners, with LLM performance spanning and generally exceeding human accuracy.

  • B.3 Bradley-Terry Elo Ratings: 633,252 pairwise outcomes from 29 models support Bradley-Terry ratings that account for question-difficulty heterogeneity beyond equal-weighted Macro accuracy.Ratings use wins and losses while treating ties as uninformative, with 95% confidence intervals estimated from 500 question-level bootstrap resamples.
  • B.3 Bradley-Terry Elo Ratings: Elo 1727 places claude-opus-4-6 30–35 points above Gemini-3.1-pro-preview and gpt-5.4, whose confidence intervals overlap.The top three are tightly clustered, while claude-opus-4-6 retains a statistically distinguishable advantage; Elo and Macro rankings closely agree.
  • C Human Evaluation: Benchmark Difficulty Validation; C.1 Motivation; C.2 Study Design; C.3 Respondent Profiles; C.4 Per-Respondent Results: 45.0% mean human accuracy across 360 responses shows that practitioners with operational AHU and Chiller experience did not outperform the overall panel.The directly experienced subgroup scored 41.9% overall, including 43.8% on AHU and 41.7% on Chiller questions, near the 45.0% human mean.
  • C.2 Study Design; C.5 Per-Asset Accuracy: 42.8% AHU accuracy and 40.7% Chiller accuracy align with the overall human mean, despite those asset classes dominating the 40-question subset.The evaluation sampled positive-type questions across 10 asset categories, with 23 of 40 questions from AHU and Chiller; all respondents completed the study without external resources.
  • C.6 Per-Question Difficulty Distribution; C.8 Universal-Failure Questions: 12 of 40 questions had zero correct human responses, whereas 19 had at least six correct, revealing strongly bimodal question difficulty.Only one question was answered correctly by all nine respondents, and the universal failures were labeled Entry-Level by rule authors.
  • C.7 Inter-Rater Agreement: Fleiss’ κ = 0.40 and 55.4% mean pairwise agreement indicate fair but limited inter-rater consistency among the nine practitioners.The agreement pattern is consistent with diverse and limited specialist exposure rather than calibrated specialist-level performance.
  • C.8 Universal-Failure Questions; C.9 LLM Population on the Same 40 Questions: 56.0% mean LLM accuracy on the same 40 questions exceeds the 45.0% human mean, with scores ranging from 25.0% to 80.0%.Seven of 29 LLMs exceed the best human’s 67.5%, while none fall below the worst human’s 25.0%; among universal failures, LLM accuracy is bimodal at 86% versus 31%.

C.10 Limitations … D.2 Excess at the High-Extreme Versus a Binomial Null

DiagnosticIQ requires specialist fault-pattern knowledge beyond working exposure, while its evaluation is limited by a small, non-specialist practitioner panel. Across models, rule-level accuracy is strongly concentrated at extremes beyond binomial expectations, indicating rule-shape recognition rather than stable per-instance condition evaluation.

  • C.10 Limitations: The human evaluation is limited by a nine-person panel of working practitioners without required certified HVAC fault-detection credentials, unlike the rule authors’ certified 10–15-year specialists.A larger panel with more diverse asset specializations and credentialed specialists is left for future work.
  • C.11 Implications for the Benchmark: Practitioners with self-reported AHU and Chiller experience scored 43.8% and 41.7%, respectively, indicating that working familiarity does not transfer reliably to fault-pattern recognition.The benchmark therefore requires specialist knowledge beyond operational exposure.
  • C.11 Implications for the Benchmark: The benchmark is difficult but not impossible: 19 of 40 questions were answered correctly by at least six humans, and the best respondent reached 67.5%.These results distinguish specialist from practitioner-level knowledge rather than requiring knowledge no human possesses.
  • C.11 Implications for the Benchmark: Frontier LLMs solve a substantial fraction of practitioner failures, but bimodal performance suggests they capture HVAC fault-detection vocabulary more reliably than inferential diagnostic reasoning.The comparison is against the working-practitioner panel, not certified fault-detection specialists; the top frontier model answered 10 of 12 universally failed questions correctly.
  • D.2 Excess at the High-Extreme Versus a Binomial Null: The observed fraction of rules with at least 90% accuracy exceeds a uniform-difficulty binomial null by a median factor of 10.2× across 30 models.Figure 22 shows a positive observed-null gap for every model, with the gap increasing as overall accuracy decreases.
  • D.2 Excess at the High-Extreme Versus a Binomial Null: The excess rises from 3.2× for claude-opus-4-6 to 45× for microsoft-phi-4, showing weaker models produce disproportionately lopsided rule-level successes.Intermediate factors are 8.1× for o1 and 20.7× for llama-3-1-405b.
  • D.2 Excess at the High-Extreme Versus a Binomial Null: This concentration of perfect-on-rule outcomes cannot be explained by stable independent competence, even at low accuracy, and directly supports rule-shape recognition as a failure mode.The result indicates models do not evaluate conditions independently on each question within a rule.

D.3 Effective Sample Size Under Within-Rule Clustering … D.6 Reporting Recommendations

Within-rule clustering reduces DiagnosticIQ’s effective information to roughly 127–133 independent observations, widening uncertainty and preventing statistical separation of the frontier models. Correct answers concentrate in a shared minority of rules, motivating rule-clustered reporting and rule-level comparisons.

  • D.3 Effective Sample Size Under Within-Rule Clustering: ICC values of 0.872–0.914 imply a roughly 40× design effect and an effective sample size of only 127–133 across the 30 models.The benchmark’s apparent statistical mass is concentrated in rule structure rather than independent question variants.
  • D.4 Implications for Model Comparison: Design-effect correction expands the 95% confidence-interval half-width from ±1.2% to ±7.5%, approximately sixfold.This correction materially changes significance testing and model-comparison interpretation.
  • D.4 Implications for Model Comparison: The three frontier models—claude-opus-4-6, gpt-5.4, and gemini-3.1—are not statistically separable under rule-level analysis.They cluster within one Macro point in the main results, while rule-level comparisons across the closest top-model pairs do not reach p < 0.05.
  • D.4 Implications for Model Comparison: Question-level testing understates p-values by approximately 3× relative to correctly clustered rule-level testing.The comparison treats per-rule accuracies as the unit of analysis rather than treating every question as independent.
  • D.5 Concentration of Correct Answers Across Rules: The top 20% of rules account for 49–56% of correct answers and the top 50% for 79–86% across all 30 models.Strong and weak models share nearly identical concentration curves, differing mainly in absolute accuracy rather than which rules dominate.
  • D.6 Reporting Recommendations: The paper therefore reports rule-clustered Wald 95% confidence intervals, uses rule-level paired tests, and supplements accuracy with rule-recognition diagnostics.The diagnostics are the bimodality coefficient and excess-over-binomial-null factor.
  • D.6 Reporting Recommendations: DiagnosticIQ resolves rule-shape coverage at an effective resolution of approximately 118 independent rules rather than the nominal question count.This limitation changes how benchmark results should be read but does not invalidate the evaluation instrument.
Loading 2605.08614v1…