Source-linked AI summary
IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products
Haonan Qi, Jin Cao, Yongqi Zhang, Xintong Wang, Weidong Tang, Bin Chen, Chengfu Huo, Haojun Pan, Hengyu You, Jing Li, Yingde Wang, Liang Ding
TL;DR
Industrial product specifications are scattered across heterogeneous images, leaving MLLM reliability for multi-image attribute extraction underexplored. IndustryBench-MIPU benchmarks structured extraction across 4,559 products and 27,652 images, finding that models remain precise but recover at most half of product-level specifications.
Problem
Whether MLLMs can reliably integrate dense industrial specifications scattered across heterogeneous product images remains underexplored.
Method
IndustryBench-MIPU evaluates schema-constrained property-value extraction from industrial product images across single-image and product-level multi-image settings.
Results
At most 49.9% of product-level attributes are recovered, despite 86–94% precision, with multi-image extraction costing 15–34 percentage points of recall.
Takeaways & Limitations
Cross-image evidence integration, rather than per-image recognition, is the central completeness bottleneck for current MLLMs on industrial product understanding.
Takeaways & Limitations
The Chinese-only product profiles may limit generalizability to other languages, and prompt choice can shift F1 by up to 2.7 percentage points.
Abstract
from arXiv · showhide
Industrial products such as valves and circuit breakers are defined by dense technical specifications that govern procurement, compatibility, and safety across supply chains. These specifications are scattered across multiple heterogeneous product images, including specification tables, nameplates, and technical drawings, yet whether Multimodal Large Language Models (MLLMs) can reliably recover them remains underexplored. To fill this gap, we introduce IndustryBench-MIPU, the first large-scale benchmark for multi-image industrial product understanding, built around structured attribute extraction -- recovering property-value pairs from product images. This task jointly probes text recognition on specification tables and nameplates, visual reasoning over technical drawings, domain knowledge to decode industrial terminology, and cross-image evidence integration to assemble scattered specifications. Concretely, the benchmark comprises 4,559 products across 27,652 images with 103,703 annotations spanning 18 industrial categories, constructed through multi-model consensus and three-tier quality assurance. Evaluating nine MLLMs under both single-image and product-level multi-image settings reveals a stark completeness gap: models achieve high precision (86--94%) but the best recovers only 49.9% of product-level attributes; moving from single-image to multi-image extraction costs 15--34 percentage points of recall. Multi-image completeness, not single-image accuracy, is the core bottleneck. Dataset and code are publicly available.
1 Introduction
IndustryBench-MIPU addresses the lack of benchmarks jointly evaluating industrial specification understanding across images by formalizing structured attribute extraction in single-image and product-level multi-image settings. The benchmark targets text recognition, visual reasoning, domain knowledge interpretation, and cross-image evidence integration across 18 industrial categories with over 100K product-level annotations.
- Industrial specifications govern purchasing, compatibility, and operational safety, motivating MLLM-based extraction for procurement, inventory cataloging, and product matching.
- Four challenges define multi-image industrial understanding: text recognition, technical-drawing reasoning, domain knowledge interpretation, and cross-image evidence integration.Examples include interpreting “DN20” as nominal diameter and “304” as a steel grade.
- Existing benchmarks cover broad visual reasoning, text recognition, or limited attribute extraction, but do not jointly evaluate industrial multimodal specification comprehension.Prior industrial benchmarks focus on defect detection, assembly recognition, or text-only domain question answering.
- IndustryBench-MIPU is introduced as the first multi-image benchmark for industrial product understanding, using structured property-value extraction to engage all four challenges.A complete product record requires reading specification tables and interpreting technical information across product images.
- 18 industrial categories and over 100K product-level annotations comprise the benchmark, which uses multi-model consensus and three-tier quality assurance.The task formalization defines single-image and product-level multi-image settings to probe localized extraction and cross-image integration.
2 Related Work
Prior benchmarks separately evaluate multimodal reasoning, attribute-value extraction, or industrial perception and knowledge. IndustryBench-MIPU addresses their untested intersection: structured technical attribute extraction from multi-image industrial product profiles.
- Multimodal LLM Benchmarks: MLLM benchmarks measure per-question multimodal correctness, text recognition, and cross-image reasoning across diverse tasks.Examples include MMMU, MMMU-Pro, MMBench, OCRBench, MMIU, and MMRB.
- Attribute Value Extraction: Attribute-value extraction has progressed from product-title sequence labeling to multimodal methods integrating visual features and generative MLLMs.Recent work expands multimodal extraction through approaches including contrastive learning.
- Industrial AI Benchmarks: Industrial benchmarks evaluate surface anomalies, manufacturing tasks, or industrial knowledge, but not understanding product specifications.MVTec AD and FORGE focus on visual perception, while IndustryBench probes text-only industrial domain knowledge.
- Research Gap: Existing benchmarks address visual reasoning, attribute extraction, and industrial knowledge in isolation, leaving structured technical attribute extraction from multi-image industrial profiles untested.IndustryBench-MIPU is positioned at the intersection of these three capabilities.
3 Task Definition
The task formalizes industrial product attribute extraction from product metadata, schemas, and images into property-value pairs. It evaluates models under single-image and multi-image settings, with the latter requiring cross-image information integration.
- Input: The input profile includes product metadata, hierarchical categories, associated images, and a product-specific schema defining permissible property names.The profile is represented as x = (t, e, c1, cl, {I(1), . . . , I(n)}, Sx).
- Output: The output is a set of property-value pairs whose property names belong to the schema and whose values are extracted from visual and textual input.The formal output is Y = {(pj, vj)}.
- Evaluation Settings: Single-image extraction recovers all property-value pairs visible in one image using product metadata and the attribute schema.This setting isolates text recognition, visual reasoning, and domain knowledge application.
- Evaluation Settings: Multi-image extraction produces a unified product-level attribute set by locating relevant information, filtering irrelevant content, and integrating attributes across images.It uses the complete product profile with all associated images.
4 IndustryBench-MIPU
IndustryBench-MIPU is a Chinese industrial-product benchmark that measures structured attribute extraction at image and product levels across diverse, schema-complex products. It builds high-coverage annotations through five-model consensus, multi-stage extraction, and sequential automated and human quality assurance.
- Annotation Construction: Five MLLMs independently annotate each product, and their outputs are unioned to create a recall ceiling above any individual annotator.The union is chosen because valid attributes found by even one model should contribute to the target record rather than being limited by majority agreement.
- Dataset: 4,559 products, 27,652 valid images, and 103,703 product-level annotations span 18 top-level industrial categories and 3,564 unique property names.The benchmark also supports image-level evaluation with 182,527 annotations, and all profiles, schemas, and annotations are in Chinese.
- Annotation Pipeline: 31% of candidate images are filtered out before extraction to remove facilities, marketing banners, and unrelated products that could cause false annotations.The pipeline first identifies the product entity, then selects informative images, and finally extracts schema-conditioned property-value pairs with rationales.
- Annotation Construction: Product-level records merge attributes across all images while retaining multiple valid values for a property and deduplicating semantically equivalent entries.This aggregation reflects products with multiple models or supported specification values rather than forcing one resolution.
- Quality Assurance: 96.7% verification pass rate on a random 10% product sample confirms the effectiveness of the preceding automated quality-assurance tiers.The three tiers audit visual grounding and semantic validity, replace covered fields with verified gold attributes, and then perform expert spot checks.
5 Evaluation Protocol
The evaluation matches predicted property-value pairs within image or product scope using exact property names and cascaded value matching, then reports precision, recall, and F1 at both levels. A bottleneck analysis examines how growing image and attribute counts affect evidence, outputs, recall, and F1.
- Matching Strategy: Property names require exact string matches, while values undergo normalization, thresholded character-level matching, and semantic-equivalence judging for remaining candidates.Normalization includes Unicode NFKC, case folding, punctuation and space removal, unit normalization, and numeric canonicalization; character-level F1 must exceed 0.6 for acceptance before semantic judging.
- Metrics: Precision, recall, and F1 are reported at both image and product levels.Precision measures matched predictions, while recall measures benchmark pairs matched by at least one prediction across the evaluation scope.
- Bottleneck Analysis: As input images and benchmark attributes increase, gold evidence grows faster than model outputs, causing recall and F1 to decline.Figure 3 presents this bottleneck analysis for Gemini 3.1 Pro using average gold and predicted property-value pairs per product alongside recall and F1.
- Additional Analysis: Error-type breakdowns and additional property-name and value-matching details are provided in Appendices B.1 and E.These appendices supplement the evaluation protocol with error analysis and matching specifications.
6 Experiments
Across nine MLLMs, multi-image extraction is limited primarily by recall: precision remains high, but models recover only a fraction of product-level attributes. The gap widens with cross-image complexity, specification density, visual reasoning, and multi-value enumeration, while larger models help substantially.
- Main results: 49.9% is the best multi-image recall, despite precision of 86–94% across models; Gemini 3.1 Pro leads with F1 = 65.1%.Qwen 3.5-397B-A17B follows at 62.7% F1, while recall ranges from 11.7% to 49.9%.
- Main results: 15–34 percentage points is the typical recall loss when moving from single-image to multi-image extraction, while precision holds or rises.Qwen 3.5-35B drops from 63.3% to 11.7% recall, whereas Qwen 3.5 Plus has the best single-image F1 at 81.3% but ranks fourth multi-image.
- Complexity analysis: 60.3% to 38.7% is Gemini’s recall decline as inputs grow from 1–3 to more than 15 images, with F1 falling from 73.0% to 55.1%.Products in the largest image bucket average 60.1 gold attributes, but Gemini outputs only 25.6 predictions, indicating under-proportional extraction.
- Complexity analysis: 45.1% is Gemini’s recall for products with more than 40 gold attributes, while precision remains stable and extraction stops before available evidence is exhausted.For these products, the model outputs 33.0 predictions for 65.9 gold attributes.
- Semantic subtypes: 36.6% is recall for visual-reasoning attributes, below direct standardized fields at 45.9%, domain knowledge at 44.1%, and multi-value compounds at 42.8%.The 9.3-point spread indicates greater difficulty for technical interpretation and dense-list aggregation.
- Scaling analysis: 42 points is the multi-image F1 spread within Qwen 3.5, from 20.6% for 35B-A3B to 62.7% for 397B-A17B, versus 7.3 points single-image.The sharpest multi-image gain occurs between 10B and 17B active parameters, reaching 12.6 points versus 5.5 in single-image.
- Ablations: 52.6% recall is achieved with the simpler prompt versus 45.4%, raising F1 from 59.9% to 62.6% but lowering precision from 88.1% to 77.3%.The simpler prompt elicits 47% more predictions, increasing output from 50K to 74K by removing schema constraints.
- Case study: 100% precision but only 45.0% recall in a seven-image, 69-attribute microscope case illustrates premature stopping in dense multi-value specifications.The model recovers over 91% on compact evidence but only 35% on the objective-spec sheet, and reports just the first 4–5 matrix values.
7 Conclusion
IndustryBench-MIPU is introduced as the first multi-image benchmark for structured attribute extraction from industrial-product images. Its results reveal a consistent completeness gap: MLLMs achieve high precision but recover at most half of product-level specifications, with deficits concentrated in cross-image evidence integration.
- Benchmark contribution: 4,559 products, 27,652 images, and 103,703 annotations span 18 industrial categories.These data constitute the IndustryBench-MIPU benchmark.
- Benchmark contribution: IndustryBench-MIPU is the first multi-image benchmark evaluating MLLM understanding of industrial products through structured attribute extraction.The benchmark evaluates recovery of structured attributes from industrial-product images.
- Key finding: At most half of product-level specifications are recovered despite high precision, revealing a consistent completeness gap.The deficit concentrates at the intersection of cross-image evidence integration.
† Author Contributions
Liang Ding led the project, with core contributions from eight researchers and additional contributions from four researchers.
- Liang Ding served as project leader.
- Haonan Qi, Jin Cao, Yongqi Zhang, Xintong Wang1, Weidong Tang, Bin Chen, Chengfu Huo, and Liang Ding1 were core contributors.
- Haojun Pan, Hengyu You, Jing Li, and Yingde Wang were contributors.
Limitations
The benchmark’s generalizability is limited by its Chinese-only product profiles, while annotation may inherit MLLM biases and evaluation results may depend on prompt choice.
- Chinese-only product profiles may limit generalizability to other languages.
- LLM-in-the-loop annotation may inherit systematic biases from MLLM-based candidate generation.The study mitigates this risk through multi-model consensus and human verification.
- Prompt choice can shift F1 by up to 2.7 percentage points, making model rankings partially prompt-sensitive.The main results use a single prompt per evaluation setting.
Ethics Statement … E Evaluation Protocol Details
IndustryBench-MIPU’s evaluation protocol combines schema-constrained extraction, multi-image evidence requirements, and cascaded matching, while additional analyses show that completeness—not value accuracy—is the dominant challenge. Results vary by category and semantic subtype, and explicit reasoning substantially improves single-image extraction but does not resolve multi-image evidence localization.
- Ethics Statement: The benchmark contains commercial product images and specifications rather than personal or conversational data, and is intended for multimodal product-understanding research.Images may include brand names, logos, and supplier-provided markings.
- A Formal Metric Definitions: The evaluation matches predicted and gold property-value pairs greedily by property name and soft-matching value, preferring exact values when multiple candidates exist.Let G and P denote gold and predicted CPV pairs, with matched subsets MG and MP.
- B.1 Error Patterns: Missing attributes dominate non-omission errors, while value mismatch rates remain comparatively small across models.Extra property errors occur when the property is absent from the benchmark; value mismatches retain the property name but predict a nonmatching value.
- B.2 Category-Level Variation: Precision remains 80–96% across categories, while recall ranges from 56.1% for Rubber & Plastics to 35.3% for Textiles & Leather.The category-level precision–recall asymmetry indicates that difficulty varies primarily in completeness.
- B.3 Per-Model Semantic Subtype Results: Visual reasoning has the lowest recall across semantic subtypes, and model recall spreads are widest for multi-value (14.7 pp) and domain knowledge (12.6 pp).The recall ordering is visual reasoning < domain knowledge ≈ multi-value < direct standardized, while visual reasoning has the narrowest spread at 5.4 pp.
- C Evaluation Prompt Templates: The multi-image prompt requires schema-listed property-value pairs supported by image evidence across product images, while the single-image prompt restricts evidence to one image.Inputs include product identity, categories, schema, and images; title and category information cannot directly support attribute values.
- D Thinking Mode in Single-Image Extraction: Thinking improves single-image recall by 9.2 pp for Qwen 3.5 Plus, whereas disabling thinking in multi-image extraction produces only a +1.1 pp recall gain.The contrast suggests that multi-image performance is constrained by evidence localization rather than per-image attribute enumeration.
- E Evaluation Protocol Details: Property names use exact matching after Unicode NFKC normalization, with over 99% of correct predictions matching on the first pass; residual value comparisons use an LLM semantic judge.The judge allows formatting, ordering, abbreviation, and unit-surface variation when meaning is unchanged, limiting its impact to a small residual fraction.
F Annotation Overlap and Self-Evaluation Analysis
The benchmark’s annotations show substantial cross-model consensus while retaining QA-validated long-tail attributes proposed by only one annotator. Independent filtering and gold-standard cross-checks limit potential self-evaluation bias.
- Consensus distribution: 53.2% of annotations were independently proposed by at least 3 of 5 annotator models, while 71.5% were proposed by at least 2.28.5% came from a single model but survived three-tier QA, indicating valid long-tail attributes proposed by only one annotator.
- Why self-evaluation bias is limited: The benchmark uses the union of five annotator models’ outputs, diluting any evaluated model’s own contributions through the other four annotators.This design choice mitigates concern that annotator models are advantaged during evaluation.
- Why self-evaluation bias is limited: 23.9% of candidates were removed by independent Tier 1 QA using Qwen 3.6 Plus, which was not among the nine evaluated models.Tier 1 included removal of model-specific hallucinations, while Tier 2 replaced generated values with verified ground truth for covered standard attributes.