Source-linked AI summary
IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs
Songlin Bai, Xintong Wang, Linlin Yu, Bin Chen, Zhiang Xu, Yuyang Sheng, Changtong Zan, Xiaofeng Zhu, Yizhe Zhang, Jiru Li, Mingze Guo, Ling Zou, Yalong Li, Chengfu Huo, Liang Ding
TL;DR
Industrial procurement QA lacks benchmarks that jointly test standards compliance, multilingual consistency, and safety contradictions. IndustryBench addresses this gap with a standards-grounded, safety-aware benchmark, finding persistent terminology weaknesses and lower safety-adjusted scores for 12 of 13 models in thinking mode.
Problem
Industrial procurement QA lacks benchmarks combining external verification, procurement-specific diagnostics, fixed-item multilingual comparison, and source-grounded safety scoring.
Method
IndustryBench evaluates 2,049 standards-grounded procurement items across seven capabilities, ten industries, three difficulty tiers, and four aligned languages with safety-aware scoring.
Results
Current models remain far from saturation: the best system reaches a Final (SV) score of 2.083 on the 0–3 scale, while Standards & Terminology remains the persistent weakness.
Takeaways & Limitations
Industrial LLM evaluation should diagnose standards-grounded correctness and safety-violation risk separately rather than rely on aggregate accuracy alone.
Takeaways & Limitations
IndustryBench covers Chinese GB/T standards and domestic product records, so it does not represent international standards, regional regimes, or procurement practices outside this source domain.
Abstract
from arXiv · showhide
In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark for industrial procurement QA in Chinese, grounded in Chinese national standards (GB/T) and structured industrial product records, organized by seven capability dimensions, ten industry categories, and panel-derived difficulty tiers, with item-aligned English, Russian, and Vietnamese renderings. Our construction pipeline rejects 70.3% of LLM-generated candidates at a search-based external-verification stage, calibrating how unreliable industrial QA remains after LLM-only filtering. Our evaluation decouples raw correctness, scored by a Qwen3-Max judge validated at $κ_w = 0.798$ against a domain expert, from a separate safety-violation (SV) check against source texts. Across 17 models in Chinese and an 8-model intersection over four languages, we find: (i) the best system reaches only 2.083 on the 0--3 rubric, leaving substantial headroom; (ii) Standards & Terminology is the most persistent capability weakness and survives item-aligned translation; (iii) extended reasoning lowers safety-adjusted scores for 12 of 13 models, primarily by introducing unsupported safety-critical details into longer final answers; and (iv) safety-violation rates reshuffle the leaderboard -- GPT-5.4 climbs from rank 6 to rank 3 after SV adjustment, while Kimi-k2.5-1T-A32B drops seven positions. Industrial LLM evaluation therefore requires source-grounded, safety-aware diagnosis rather than aggregate accuracy. We release IndustryBench with all prompts, scoring scripts, and dataset documentation.
1 Introduction
IndustryBench evaluates industrial procurement QA as a source-grounded, safety-aware task in which correctness requires standards traceability, not merely fluent or partially correct answers. Its benchmark design and evaluation protocol diagnose persistent standards weaknesses, remaining performance headroom, and safety risks in current LLMs.
- Motivation: Industrial procurement answers must match operating conditions, satisfy required thresholds, and avoid violating safety clauses.Fluency, relevance, or partial correctness does not make an answer acceptable when it contradicts a source-backed constraint.
- Benchmark: The benchmark combines externally verified construction, procurement-specific diagnostic labels, multilingual comparison under fixed item identity, and safety-aware source-constraint scoring.This design addresses the need to penalize unsafe contradictions even when responses are otherwise plausible.
- Benchmark: IndustryBench contains 2,049 items grounded in Chinese national standards (GB/T) or domestic industrial product records.Items are annotated across seven capability dimensions, ten industry categories, and three panel-derived difficulty tiers.
- Evaluation: A validated Qwen3-Max judge scores raw correctness on a 0–3 rubric, with κw = 0.798 against a domain expert, while a separate SV check tests source compliance.Models are evaluated zero-shot and closed-book, receiving only the question.
- Findings: The best model reaches 2.083 on the 0–3 Final (SV) scale, Standards & Terminology remains the most persistent weakness, and 12 of 13 models score lower with extended reasoning.The reported findings also show divergence between raw accuracy and safety-adjusted reliability, including reasoning-mode safety degradation.
2 Related Work
Prior benchmarks cover general knowledge, engineering, commercial products, factuality, and safety, but none combines authoritative industrial procurement sources with external verification and safety-aware diagnosis. IndustryBench is designed to fill this methodological and domain-specific gap.
- General and domain-specific benchmarks: General benchmarks measure broad knowledge and reasoning, while Chinese counterparts extend this coverage to Chinese-language evaluation.Examples include MMLU, MMLU-Pro, HELM, C-Eval, and CMMLU.
- Engineering and industrial benchmarks: Engineering benchmarks target problem solving, architecture, engineering, construction, strength-of-materials reasoning, and industrial operations agents.Examples include EngiBench, AECBench, SoM-1K, and AssetOpsBench.
- E-commerce and commercial product evaluation: Commercial-product benchmarks evaluate e-commerce workflows, knowledge-graph factuality, consumer product concepts, and related industrial product understanding.Examples include EcomBench, ECKGBench, ChineseEcomQA, and SuperCLUE-Industry1.
- Factuality and safety evaluation: Factuality and safety methods motivate IndustryBench’s evaluation framework, but industrial procurement requires a more specific safety notion tied to standards-grounded violations.TruthfulQA, FActScore, and SafetyBench provide methodological precedents for misconception resistance, evidence grounding, and general safety-risk evaluation.
- Benchmark gap: No public benchmark combines national standards and structured product records, externally verified QA pairs, diagnostic labels, panel-derived difficulty, and safety-aware scoring in industrial procurement.IndustryBench is designed to make model weaknesses visible at a more granular level.
3 Benchmark Construction
IndustryBench is constructed from GB/T standards and structured industrial product records through a conservative, five-stage pipeline combining automated filtering, search-based verification, claim-level refinement, and human oversight. The released 2,049-item benchmark is labeled for diagnostic slice analysis and covers standards- and product-grounded procurement scenarios.
- Source construction: 13,000 GB/T documents provide the normative layer, while approximately 630,000 industrial product records provide concrete procurement instances and operating constraints.The two source families jointly connect standards-level technical knowledge with product-level specifications such as materials, dimensions, model identifiers, and rated power.
- Five-stage pipeline: Approximately 230,000 candidate QA pairs pass through five successive stages, progressing from deduplication and quality screening to external verification and claim-level answer refinement.Candidates are anchored in standards excerpts or product-record content rather than generated free-form.
- Five-stage pipeline: Stage 5 yields approximately 9,600 verified items by checking claim-level support and refining imprecise answers or removing items whose evidence is conflicting, insufficient, or unsupported.The stage specifically scrutinizes numerical values, standard identifiers, material grades, technical specifications, and safety constraints.
- Five-stage pipeline: 70.3% of Stage 3 survivors are rejected during Stage 4 search-based fact verification, which retains 20,457 items for subsequent deep verification.Stage 4 uses three structured Google Search2 queries per candidate and up to 15 retrieved results to corroborate factual claims.
- Release and diagnostics: The released benchmark contains 2,049 questions, with 21.15% derived from GB/T standards and 78.85% from structured industrial product records.Each question also receives single-label annotations for panel-derived difficulty, capability dimension, and industry category, enabling slice-level diagnosis.
4 Evaluation Methodology
IndustryBench separates raw correctness from source-grounded safety-violation detection, using graded scoring to distinguish partial industrial QA correctness from unsafe contradictions. Qwen3-Max is validated against domain experts and cross-judge checks, while additional analyses find no evidence that large self-preference bias drives rankings.
- Safety-violation adjustment: Safety violations are checked separately against the original GB/T excerpt or product record, and flagged responses receive an adjusted score of 0 regardless of raw correctness.Unflagged responses retain their raw score; the final SV score is the mean adjusted item score, with Δ defined relative to the raw mean.
- Raw correctness scoring: Raw correctness uses a four-level score, ri ∈{0, 1, 2, 3}, because industrial answers can be broadly correct yet incomplete or missing essential constraints.The judge sees the question, reference answer, and model response, but not the underlying source text.
- Safety-violation validation: 200 stratified GLM-5-744B-A40B responses validate the automated SV detector against a domain expert, with 27 detected violations, 24 confirmed, and no missed violations.All disagreements are false positives, indicating conservative detection in the validation sample.
- Judge validation: Qwen3-Max aligns most closely with the domain expert at κw = 0.798, with 84.3% exact match and 96.0% of items within one score point.Only 8 of 198 items differ by two or more points; the three-judge median reaches κw = 0.818 but requires three model calls.
- Judge self-preference checks: Self-preference checks argue against a large Qwen-family scoring shift, while acknowledging that smaller family-specific effects cannot be ruled out.Qwen3-Max assigns fewer perfect scores than the expert (61.6% vs. 72.2%) and a lower mean (2.20 vs. 2.34) on the 198-response GLM-5 calibration sample; capability leadership is distributed across vendors.
5 Experiments
Across 17 models, IndustryBench shows substantial room for improvement, with safety adjustment materially changing performance rankings. Extended reasoning generally worsens safety-adjusted results, while Standards & Terminology remains the most persistent capability weakness.
- Overall results: 2.083 is the best Final (SV) score, while the full model range spans 1.394–2.083 on the 0–3 scale.Gemini 3.1 Pro also achieves a 54.2% perfect rate and 69.8% pass rate.
- Overall results: 0.012 points separate the top three Final (SV) scores: Gemini 3.1 Pro (2.083), Qwen3.6-Plus (2.073), and GPT-5.4 (2.071).A paired item-level bootstrap does not reliably distinguish the top four models at the 95% level.
- Safety adjustment: 2.174 is Kimi-k2.5-1T-A32B’s highest raw mean among open-source models, yet it falls to rank 10 after safety adjustment; GPT-5.4 reaches the top three with Delta −0.060.These cases demonstrate that raw correctness alone is insufficient for industrial evaluation.
- Reasoning-mode analysis: −0.150 to −0.323 is the average Delta change from non-thinking to thinking mode, more than doubling the safety penalty despite comparable or slightly higher raw means.Models often add technically detailed but source-contradicting safety-critical content after producing a substantively correct answer.
- Capability analysis: 1.462 is the lowest SV-adjusted aggregate mean for Standards & Terminology, which is also every model’s lowest-scoring capability in the 17-model matrix.This finding is supported by 610 items, or 29.8% of the benchmark; source coverage and task difficulty are both plausible explanations.
- Industry analysis: 1.662 and 1.675 are the weakest industry-level aggregate means, for Energy & Storage and Textile & Leather, compared with 1.982 for Electronics & Sensors.The paper cautions that these gaps may combine vertical difficulty with documentation and dataset-composition effects.
6 Discussion
IndustryBench is a source-grounded diagnostic benchmark: it exposes persistent standards weaknesses, multilingual and reasoning-related reliability risks, and the need to evaluate safety separately from raw accuracy. Its methodology supports scalable comparison when judging is human-calibrated, but benchmark scores are not deployment certification.
- Diagnostic interpretation: Standards & Terminology remains consistently weak across models, showing that industrial competence cannot be reduced to a single aggregate score.Higher-scoring capability slices vary more, while low-support dimensions require caution.
- Benchmark construction: 70.3% of LLM-filtered candidates were rejected during search-based verification, demonstrating that industrial QA generation requires independent evidence checks and human review.Plausible questions and answers often fail when treated as claims requiring external evidence.
- Multilingual evaluation: Standards & Terminology weakness persists across item-aligned multilingual versions, showing that translation can change model behavior through different linguistic and terminological surfaces.The multilingual design preserves item identity while changing language realization rather than constructing independent monolingual benchmarks.
- Reasoning-mode effects: 12 of 13 models scored lower with extended reasoning enabled, mainly because longer answers deepened safety-violation penalties.Longer final answers create more opportunities for unsupported safety-critical details, over-specified thresholds, or conflicting procedural claims.
- Safety-aware evaluation: Raw accuracy and safety-violation risk are distinct signals: similar raw scores can incur different SV penalties, so accuracy-only leaderboards are inadequate for industrial deployment.Deployment requires consistency with mandatory limits, operating requirements, and safety procedures.
- Evaluation methodology: κw = 0.798 against a domain expert supports validated LLM-as-judge evaluation for scalable comparison, but residual disagreement means high IndustryBench scores are not deployment certification.Future judge ablations remain an important limitation.
7 Limitations
IndustryBench’s coverage, labels, safety checks, and evaluation protocol impose limits on how broadly its results should be interpreted. Standards and product evidence can become outdated, and benchmark performance does not certify safety, compliance, or legal suitability in deployment.
- Scope and representativeness: IndustryBench covers Chinese GB/T standards and domestic industrial e-commerce records, not international standards, regional regulations, or procurement practices outside its source domain.Its English, Russian, and Vietnamese versions are language-aligned renderings rather than independently sampled monolingual benchmarks.
- Labels, judges, and sparse cells: Difficulty labels reflect model-panel performance ranks, while capability and industry labels are diagnostic benchmark categories rather than intrinsic human-rated difficulty.Labels are produced through three-model labeling with human adjudication for disagreement cases.
- Labels, judges, and sparse cells: The SV detector was validated on a stratified GLM-5 response sample but may miss violation styles across model families or cases requiring broader process context.Fault Diagnosis and Engineering Calculation have low support in Appendix B, so their means are indicative rather than definitive.
- Evaluation protocol and uncertainty: One standardized evaluation pass per model quantifies item-sampling uncertainty, not repeated-run or decoding-level variability, so small adjacent rank differences should not be over-interpreted.Paired item-level bootstrap intervals quantify uncertainty from the finite 2,049-item benchmark sample.
- Freshness, deployment, and comparability: Standards, product records, and web evidence may drift over time, requiring periodic refresh; high benchmark scores do not certify live-procurement safety, compliance, or legal suitability.Deployment still requires process controls, human oversight, and jurisdiction-specific review.
8 Conclusion
IndustryBench is a 2,049-item, standards-grounded benchmark for industrial product trading knowledge, built from Chinese standards and domestic product records. It combines external verification, multilingual aligned versions, and validated Qwen3-Max judging.
- Benchmark contribution: IndustryBench contains 2,049 items for evaluating LLMs on industrial product trading knowledge.The benchmark is standards-grounded and uses Chinese national standards (GB/T) alongside domestic industrial product records.
- Benchmark contribution: A five-stage construction pipeline filters benchmark items through external verification.The passage identifies external verification as part of the pipeline but does not provide stage-level rejection statistics.
- Benchmark contribution: English, Russian, and Vietnamese versions align the benchmark across languages, while a Qwen3-Max judge achieves κw = 0.798 against a domain expert.The passage also notes documented construction details and a benchmark release.
† Author Contributions
Liang Ding led the project, with Songlin Bai, Xintong Wang, Linlin Yu, Bin Chen, and Liang Ding identified as core contributors. Ten additional contributors are also acknowledged.
- Liang Ding is identified as the project leader.
- Songlin Bai, Xintong Wang, Linlin Yu, Bin Chen, and Liang Ding are identified as core contributors.
- Zhiang Xu, Yuyang Sheng, Changtong Zan, Xiaofeng Zhu, Yizhe Zhang, Jiru Li, Mingze Guo, Ling Zou, Yalong Li, and Chengfu Huo are listed as contributors.
Ethics Statement
IndustryBench limits released materials to benchmark artifacts and source-grounding fields, excluding full standards, raw product pages, private communications, and personal data. Human annotators received fair market-rate compensation.
- Data release: The release includes QA pairs, labels, prompts, evaluation code, and source-grounding fields needed for verification.These materials support benchmark use without redistributing underlying source documents or raw listings.
- Data release: It does not redistribute full GB/T documents, raw product pages, private communications, or personal data.The benchmark is constructed from national standards and public product listings, but those underlying materials are not fully released.
- Annotator compensation: Human annotators conducting label reviews and translation quality checks were compensated at fair market rates.
Reproducibility Statement
The study documents its pipeline and evaluation specifications in detail and commits to releasing the dataset, scripts, and prompt templates upon publication.
- Each pipeline stage records model versions, prompt templates, hyperparameters, and data counts, while the full judge prompt appears in Appendix D.Documented hyperparameters include similarity thresholds and scoring cutoffs.
- The dataset documentation indexes documentation fields to sections, and known study limitations are listed in §7.
- The dataset, evaluation scripts, and all prompt templates will be released upon publication.
Broader Impact Statement … H Capability Dimension Scores (Full)
IndustryBench documents a safety-oriented benchmark pipeline, multilingual quality controls, reproducible judging, and SV-adjusted capability results. Its supplementary analyses also state usage safeguards and caution against overinterpreting sparse dimensions or imbalanced categories.
- Broader Impact Statement: The benchmark is intended solely for evaluation, because using it for training or fine-tuning could undermine its validity.IndustryBench aims to make industrial-procurement knowledge gaps visible and measurable while improving deployment safety and reliability.
- A Stage 4 Search Query Generation Prompt: Stage 4 generates 3 structured search queries per QA pair, retrieves the top 5 results per query, and uses a second Qwen3-Max pass for binary factuality judgments.The queries target terminology, identifiers, model numbers, standards, materials, and processes, with coded identifiers quoted in q2.
- B Benchmark Data Distributions; B.1 Difficulty Distribution; B.2 Capability Dimension Distribution: The released benchmark contains 2,049 items split into approximately equal difficulty terciles, while capability labels preserve a natural long-tail and sparse dimensions require caution.Fault Diagnosis and Engineering Calculation are the two smallest capability cells; difficulty is assigned from panel-averaged model scores.
- B.3 Industry Category Distribution: Industry frequencies reflect source coverage and release sampling rather than deliberate balancing, across 10 categories inferred from question content.Sparse industry categories therefore require cautious interpretation of per-category statistics.
- C Multilingual Translation Details: English, Russian, and Vietnamese versions retain standards, model identifiers, numbers, units, formulas, and technical terminology, with a 1–5 faithfulness review and human editing below 5.Human review rates were English 49 items (2.4%), Russian 29 items (1.4%), and Vietnamese 20 items (1.0%).
- D Judge Prompt: Qwen3-Max is the primary reproducible judge, scoring answer consistency against references on a 0–3 rubric used across Chinese and translated benchmarks.The rubric distinguishes substantive consistency with aligned reasoning, differing reasoning, partial correctness, and complete incorrectness.
- E Safety Violation Review Prompt: Safety violations are checked separately by Qwen3-Max against authoritative knowledge text and references, covering unsuitable protection grades or materials, omitted critical procedures, and mandatory-clause violations.A violation is true when the response exhibits one of these industrial safety risks, and its justification must identify the violated regulation.
- F Pairwise Judge Agreement; G Human–Judge Score Distributions; H Capability Dimension Scores (Full): Pairwise agreement remained reliable across tested models, with Qwen3-Max versus Claude Opus 4.6 consistently achieving the highest κ_w and tightest |∆| ≤1 rates.On 198 calibration triples, Qwen3-Max was stricter than the domain expert: mean 2.20 vs. 2.34, perfect scores 61.6% vs. 72.2%, and score-2 results 11.1% vs. 3.0%. Table 17 reports SV-adjusted 0–3 capability means for all 17 models, with sparse dimensions requiring caution.
I Bootstrap Confidence Intervals for Final (SV) · J Industry Category Scores (Full)
Paired item-level bootstrap analysis quantifies uncertainty in SV-adjusted model scores and shows that the top four rankings are not reliably separated at the 95% level. Industry-category scores are reported for all 17 models, while unequal category support means observed differences combine difficulty and sampling noise.
- I Bootstrap Confidence Intervals for Final (SV): B=10,000 paired resamples of 2,049 Chinese-benchmark items recompute Final (SV) jointly across models, preserving per-item correlation.Paired resampling makes score differences more informative than independent per-model intervals.
- I Bootstrap Confidence Intervals for Final (SV): 95% CI half-widths measure sensitivity to item resampling, not run-to-run, decoding, prompt, or judge-sampling variance.Paired-bootstrap difference tests separately assess close rank differences among the top nine models.
- I Bootstrap Confidence Intervals for Final (SV): Ranks 1–6 form one conservative per-model CI-overlap cluster despite a 0.089 rank-1-to-rank-6 gap exceeding either model’s CI half-width.Paired score-difference intervals directly test whether observed gaps remain separated from zero.
- I Bootstrap Confidence Intervals for Final (SV): The top four models are not reliably distinguished by paired-bootstrap tests at the 95% level.Beyond this frontier group, some gaps to ranks 7–9 remain separated while several adjacent upper-middle comparisons do not.
- I Bootstrap Confidence Intervals for Final (SV): The two lowest-ranked models are separated from the top fifteen under per-model item-level intervals.Table 18 reports global ranks and groups rows by model family.
- J Industry Category Scores (Full): Table 20 aggregates SV-adjusted Final scores by industry category for all 17 models.The matrix uses abbreviations including Mach., Chem., Elec., Electr., Metal., Sec., Pack., and Text.
- J Industry Category Scores (Full): Unequal category support means industry differences reflect both vertical difficulty and sampling noise in sparse cells.The category aggregation mirrors the preceding industry-category score table.
K Dataset Documentation
IndustryBench documents a 2,049-item, four-language benchmark for standards-grounded industrial procurement QA, with structured labels and public-release plans. Its intended use is model evaluation and diagnosis, while users must account for its GB/T-centric scope and multilingual reporting limits.
- Composition: The dataset contains 2,049 Chinese open-ended question–answer pairs with aligned English, Russian, and Vietnamese versions, labeled by 7 capability dimensions, 10 industry categories, and 3 difficulty levels.Each instance includes a question, reference answer, and three categorical labels; the dataset excludes personally identifiable information, offensive content, and privacy-restricted data.
- Collection process: Questions and reference answers are generated from GB/T excerpts and structured industrial product records, then processed through a five-stage quality pipeline including deduplication, LLM screening, search-based fact verification, and deep verification.The documented stages are source-grounded generation, semantic deduplication, LLM-based quality screening, search-based verification against independent web sources, and deep verification with answer refinement.
- Uses and limitations: IndustryBench is intended for horizontal LLM comparison, diagnostic localization of domain-specific weaknesses, and assessment of fine-tuning or retrieval-augmented systems.The benchmark is grounded in Chinese National Standards, so ISO, DIN, and ANSI systems are not yet represented; it should not be used as training data.
- Distribution and maintenance: The dataset, evaluation scripts, and prompt templates will be released publicly upon publication under a permissive open-source license, without export controls or access restrictions.The authoring team will maintain the dataset through periodic multilingual, industry, and standards updates, version tracking, and public feedback and error reports.
- Limitations: The four-language results cover only the 8-model intersection with valid outputs in all languages, so reported cross-lingual performance should not be inferred for other models.Other documented limitations include the GB/T-centric scope, model-derived difficulty, residual judge variance, and sparse cells.