Source-linked AI summary
FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios
Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang, Guanzhi Deng, Alex Xue, Juan Du, Tianshu Yu, Garth Tarr, Linqi Song, Qiuzhuang Sun, Dacheng Tao
TL;DR
Existing manufacturing MLLM evaluations do not adequately capture fine-grained domain semantics and real-world reasoning demands. FORGE addresses this gap with a multimodal dataset and three manufacturing tasks, finding that domain knowledge—not visual grounding—is the key bottleneck and that the annotations can support substantial fine-tuning gains.
Problem
Existing manufacturing evaluations rarely assess fine-grained domain semantics or the reasoning, understanding, and decision-making required in rigorous manufacturing settings.
Method
FORGE builds a dataset of annotated 2D images and 3D point clouds and evaluates MLLMs on WORKVERI, SURFINSP, and ASSYVERI.
Results
MLLMs perform differently across tasks, with SURFINSP weakest, while domain knowledge rather than visual grounding is identified as the principal bottleneck.
Takeaways & Limitations
FORGE’s structured annotations provide an actionable resource for evaluating and developing manufacturing MLLMs beyond basic perception.
Abstract
from arXiv · showhide
The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.
1 Introduction
FORGE addresses the mismatch between manufacturing’s multimodal, fine-grained decision demands and existing MLLM evaluations. It introduces a multimodal benchmark, three manufacturing tasks, and an actionable training resource while identifying domain knowledge and morphology as bottlenecks.
- Findings: Benchmark results show that task-domain knowledge and morphology are major bottlenecks for MLLMs in manufacturing scenarios.The introduction highlights performance gaps in microscopic surface analysis and manufacturing-task reasoning.
- Training resource: +25.6% performance boost is reported for the actionable supervised fine-tuning resource.The benchmark’s structured annotations are also positioned as training data beyond evaluation.
- Motivation: Existing evaluations rarely test MLLMs’ fine-grained manufacturing semantics or the reasoning, understanding, and decision-making required in real-world settings.The paper identifies data scarcity, limited fine-grained annotations, and absent comprehensive evaluation frameworks as central challenges.
- Benchmark: FORGE combines aligned real-world 2D images and 3D point clouds with annotations spanning diverse workpiece model numbers.The dataset includes examples such as nuts ranging from M10 to M20.
- Benchmark: FORGE evaluates Workpiece Verification, Structural Surface Inspection, and Assembly Verification across material sorting, quality inspection, and assembly recognition.These tasks target fine-grained visual discrimination and complex logical reasoning, including assembly compatibility.
2 Related Work
Prior manufacturing benchmarks cover selected perception, documentation, cognition, design, and reliability tasks, but they do not adequately test fine-grained semantics under standardized manufacturing constraints. FORGE responds with multimodal annotations and tasks that require precise variant matching and reasoning.
- Existing benchmarks: Existing MLLM evaluations address visual anomaly detection, engineering documentation, manufacturing cognition, design synthesis, and failure diagnosis.These efforts span perceptual, technical, cognitive, and reliability-engineering settings.
- Research gap: Current methods rarely validate adherence to highly structured manufacturing settings requiring stringent precision.Such settings include mixed-model assembly lines, flexible manufacturing systems, product family design, and tool management.
- Research gap: Manufacturing applications require distinguishing specific workpiece variants, including model numbers, rather than only classifying general categories.The paper links this requirement to precise matching of fine-grained attributes.
- FORGE: FORGE uses fine-grained multimodal annotations to address limitations caused by scarce multimodal data and insufficient semantic detail.Its design is intended to support more rigorous manufacturing evaluation than existing frameworks.
3 FORGE
FORGE constructs a multimodal manufacturing dataset and organizes it into three tasks spanning workpiece sorting, surface inspection, and assembly recognition. Its protocol converts these scenarios into grounded multiple-choice evaluations of fine-grained perception and reasoning.
- Dataset: The dataset contains high-fidelity point clouds covering 14 workpiece categories and 90 distinct models, plus approximately 3,000 images from four manufacturing scenarios.The point-cloud subset supports WORKVERI, SURFINSP, and ASSYVERI.
- Dataset: Data processing combines automated contour and coordinate extraction with manual refinement for images and task-specific synthesis strategies for point clouds.WORKVERI and ASSYVERI point-cloud samples are synthesized by stitching 4–5 point clouds with random orientations.
- Input representation: FORGE uses multi-view projection to represent 3D point clouds because general MLLMs typically lack native 3D encoders.The paper reports that text-based coordinate serialization is highly ineffective for raw 3D data.
- WORKVERI: WORKVERI identifies workpieces that do not belong to a specified batch across image and point-cloud scenarios.Its scenarios include pneumatic connectors, cup head screws, and nuts.
- SURFINSP: SURFINSP detects manufacturing defects and classifies their types, such as cracks and dents, from point clouds or images.The evaluation covers 14 distinct manufacturing components with 3D point clouds.
- ASSYVERI: ASSYVERI tests whether MLLMs can reason over assembly relationships and compatibility constraints to identify specification violations.It covers four scenarios, including metal and plastic expansion screws and CNC fixtures.
4 Experiments
FORGE evaluates 18 MLLMs across three manufacturing tasks using multimodal inputs and exact-match accuracy. Results show strong variation across tasks and settings, with fine-grained reasoning and domain knowledge remaining major challenges.
- Evaluation setup: 18 MLLMs are evaluated across WORKVERI, SURFINSP, and ASSYVERI using image and point-cloud modalities.The benchmark includes open- and closed-source model families and multiple evaluation settings.
- Main findings: SURFINSP yields the lowest performance, indicating weaker microscopic surface analysis than macroscopic part recognition.The contrast appears in the results for WORKVERI, ASSYVERI, and SURFINSP.
- Main findings: Simple reference conditioning does not consistently improve WORKVERI and ASSYVERI, whereas in-context demonstrations universally improve over zero-shot image baselines.Several reference-conditioning cases instead show performance degradation, while complete reasoning demonstrations provide gains.
- Main findings: Three-view performance declines after adding reference conditioning and in-context demonstrations, suggesting the examples do not effectively guide multi-angular spatial reasoning.The point-cloud bottleneck analysis provides additional evidence of limitations in explicit spatial perception.
- Main findings: Model-number-level tasks consistently underperform workpiece-level tasks across both open-source and closed-source models.The gap reflects limited capture of fine-grained manufacturing specificity despite stronger general workpiece understanding.
- Error analysis: Qualitative errors reveal latent reasoning abilities for identifying materials and assessing service status despite limited classification precision.One ASSYVERI error shows models may infer material properties and use them in reasoning even when irrelevant.
B. Failure on model number
The bottleneck analyses distinguish visual grounding, domain-specific part identification, and 3D input representation. They indicate that fine-grained manufacturing knowledge is a central limitation, while rendered visual grounding is comparatively strong.
- Bottleneck analysis: The benchmark decomposes performance into component grounding, reference-test comparison, and domain-specific verdict reasoning.Five representative models are evaluated with complementary bottleneck analyses.
- Visual grounding: Gemini-3-Flash reaches 98.9% average single-image grounding accuracy, and four of five models exceed 97.6% on L→C.Cross-image matching is harder but remains above chance for the top four models, so localization is not the primary bottleneck.
- Domain knowledge: Fine-grained part identification remains a domain-knowledge bottleneck even when assembly specifications explicitly list required parts and functions.The missing-part setup separates counting whether something is absent from identifying which component is absent.
- 3D representation: Raw coordinate text produces near-20% random-baseline performance on SURFINSP, while WORKVERI reaches 53.6% zero-shot for Gemini-3-Flash.Visual rendering through multi-view projection is therefore more effective than direct coordinate serialization for general-purpose MLLMs.
- Domain adaptation: Domain-specific SFT improves WORKVERI three-view accuracy by 90.8% and ASSYVERI image accuracy by 27.1% on held-out scenarios.The WORKVERI result brings the 3B model to 54.4%, matching Qwen3-VL-235B; ASSYVERI surpasses all reference models except Gemini-3-Flash and GPT-5.2.
5 Conclusion
FORGE is presented as a fine-grained multimodal benchmark and training resource for manufacturing MLLMs. Its evaluation identifies domain knowledge and morphology understanding as key gaps, while fine-tuning improves compact-model performance on held-out scenarios.
- Conclusion: FORGE combines real-world 2D images and 3D point clouds across three manufacturing tasks.The benchmark targets workpiece verification, structural surface inspection, and assembly verification.
- Conclusion: Current MLLMs handle macroscopic part recognition better than fine-grained reasoning and microscopic surface analysis.The conclusion summarizes substantial limitations in physical-detail understanding across the evaluated tasks.
- Conclusion: Visual grounding is not the primary limiting factor; insufficient manufacturing domain knowledge and morphology understanding are identified as key gaps.This conclusion is supported by the bottleneck analysis and the broader evaluation findings.
- Conclusion: Domain-specific fine-tuning enables a compact 3B-parameter model to approach frontier-scale performance on held-out scenarios.The paper frames FORGE as both an evaluation baseline and a practical starting point for addressing the domain-knowledge gap.
A.2 Extended Main Results
Across the three manufacturing tasks, current MLLMs show substantial performance gaps, especially on demanding surface inspection. Detailed analyses indicate that specialized manufacturing knowledge and, in some settings, 3D spatial understanding remain major challenges.
- Extended Results: Current MLLMs remain far from acceptable overall, with most open-source models performing near random and advanced models struggling beyond 50% accuracy on SURFINSP.Closed-source models such as Gemini-3-Flash and GPT-5 achieve state-of-the-art results in WORKVERI and ASSYVERI, but specialized surface inspection remains difficult.
- Extended Results: In the 3V setting, model-number-level and workpiece-level WORKVERI and ASSYVERI expose a significant bottleneck in 3D spatial understanding.Ref-Cond and ICD both degrade relative to the zero-shot baseline under this setting.
- Extended Results: SURFINSP performance generally improves with more references, but DEFORMATION and DENT follow different trajectories across evaluation settings.These anomalies require different domain knowledge despite the task’s relatively low task-comprehension requirement.
- Task Scope: FORGE evaluates material sorting, quality inspection, and assembly recognition through WORKVERI, SURFINSP, and ASSYVERI.The framework covers task design, data construction, evaluation settings, and data collection for these manufacturing scenarios.
A.3.1 Raw Data Collection.
FORGE collects authentic manufacturing workpiece data through separate 3D point-cloud and image acquisition processes, then curates both modalities into task-ready subsets and evaluation settings.
- Raw Data Collection: The collection platform used a precision rotary table and custom fixture, with separate phases for point-cloud acquisition and image collection.The raw data came from authentic manufacturing workpieces.
- Acquisition Process: Point-cloud acquisition used a handheld 3D scanner configured to 0.02 mm precision, with markers and developer spray supporting metallic-workpiece scanning.Markers established a local coordinate system, while spray mitigated surface reflectivity.
- Acquisition Process: Image acquisition randomly varied workpiece models and orientations while maintaining consistent shooting angles and distances.Images used a 50-megapixel sensor with a Leica Vario-Summilux optical lens system.
- Raw Data Collection: The raw dataset contains both geometric and visual representations, organized into Point Cloud and Image subsets.These subsets provide the raw material for WORKVERI, SURFINSP, and ASSYVERI before task-specific curation.
- Raw Data Collection: The Point Cloud Subset covers 14 component categories and 90 model numbers, while the Image Subset contains approximately 3000 images from four manufacturing scenarios.Point clouds support all three tasks, whereas images primarily support WORKVERI and ASSYVERI.
- Data Preparation: Preprocessing extracted 2D artifact contours and coordinates, then manually mapped them to manufacturing model information.Separate modality-specific pipelines were required because raw images and point clouds could not be used directly.
- Data Preparation: WORKVERI and ASSYVERI batch samples were synthesized by stitching 4–5 point clouds with randomized orientations and positions, while SURFINSP used synthetic defect generation.Labels were automatically generated from artifact models and workpiece information during assembly.
- Evaluation Settings: Evaluation compares Zero-Shot, Reference-Conditioned, and In-Context Demonstration settings with progressively richer examples.Ref-Cond adds three correct normal cases, while ICD adds one similar example to the test case.
A.3.2 Task Description of WORKVERI.
WORKVERI evaluates whether MLLMs can identify and verify manufacturing workpieces against specified requirements across representative point-cloud and image scenarios.
- Task Definition: WORKVERI targets material identification and verification, fundamental steps in sorting, selecting, classifying, assembling, matching, and delivering inventory materials.The task uses manufacturing workpiece point clouds or images to identify items that fail specified requirements.
- Task Definition: WORKVERI operates on manufacturing workpiece point clouds or image data and identifies those that do not satisfy specified requirements.The task therefore tests verification against requirements rather than only generic object recognition.
- Application Scenarios: The task includes point-cloud scenarios for Nuts and Cup Head Screws and an image scenario for Pneumatic Connectors.These represent three manufacturing application scenarios.
A.3.3 Task Description of SURFINSP.
SURFINSP evaluates MLLMs’ ability to detect and classify manufacturing defects from workpiece point clouds across 14 component categories.
- Task Definition: SURFINSP requires determining whether a workpiece is defective and identifying the defect type when present.The evaluated defect types are Crack, Cut, Deformation, and Dent.
- Task Scope: The task covers 14 manufacturing component categories using workpiece point-cloud data.It is framed as a quality-inspection problem concerned with compliance with design specifications, process standards, and customer requirements.
A.3.4 Task Description of ASSYVERI.
ASSYVERI evaluates whether MLLMs understand assembly relationships and identify workpieces that violate compatibility or model requirements. It covers image- and point-cloud-based scenarios spanning expansion screws, CNC fixtures, and screw–washer–nut compatibility.
- ASSYVERI assesses assembly relationships, structural hierarchies, and compatibility constraints among manufacturing workpieces or materials.
- Given compatibility rules, specifications, or model requirements, MLLMs identify workpieces that do not meet assembly requirements from images or point clouds.
- The image scenarios cover metal expansion screws, plastic expansion screws, and CNC fixtures.
- The point-cloud scenario evaluates compatibility relationships among metal screws, washers, and nuts.
- ASSYVERI forms one of three core tasks in a multimodal benchmark targeting typical manufacturing contexts including assembly recognition.
B Training Details and Extended Results
The training study uses supervised fine-tuning and evaluates generalization with scenario-held-out splits. Extended analyses compare the finetuned compact model with frontier and ablation models and examine task-level transfer.
- Training Configurations: Qwen2.5-VL-3B is trained with full-parameter supervised fine-tuning, independently for each task.Training uses three epochs, bfloat16 precision, an effective batch size of 32, and a learning rate of 1 × 10^-5.
- Scenario-Based Splitting: Scenario-based splitting assigns entirely different product categories to training and evaluation rather than using random train/test splits.WORKVERI trains on CHS and evaluates on NUTS, while image ASSYVERI trains on MES and PES and evaluates on CNC.
- Full Comparison with Frontier Models: The extended comparison evaluates the finetuned Qwen2.5-VL-3B against five ablation models on the same held-out scenarios.
- Task-Level Analysis: WORKVERI transfers across scenarios because training teaches a general specification-mismatch concept despite geometrically similar components.
Model Type WORKVERI 3V (NUTS SCENARIO) ASSYVERI Img (CNC SCENARIO)
On held-out scenarios, the finetuned 3B model nearly matches a much larger model on WORKVERI and outperforms several reference models on ASSYVERI. Category-level results show stronger gains for normal cases and more uniform improvements across ASSYVERI error types.
- 53.8% vs. 54.4% zero-shot accuracy: the finetuned Qwen2.5-VL-3B nearly matches Qwen3-VL-235B on WORKVERI.
- The finetuned model surpasses Seed 1.6, Qwen3-VL-235B, and Mistral 3 8B on ASSYVERI.
- 627 training examples produce measurable improvement on a completely novel product category, indicating transfer of FORGE annotations.
- WORKVERI 3V (NUTS SCENARIO): 74.8% vs. 24.3%: WORKVERI Normal-case accuracy rises substantially after fine-tuning, while Different Model Number accuracy falls from 33.8% to 23.8%.Normal cases comprise 59% of the evaluation set, producing a large overall improvement despite reduced sensitivity to subtle mismatches.
- ASSYVERI Img (CNC SCENARIO): 27.0% vs. 20.5%: ASSYVERI Different Model Number accuracy increases on the CNC scenario, while Extra Part accuracy rises from 37.0% to 44.4%.
C Extended Error Cases and Bottleneck Analysis
The extended section provides complete transcripts for representative error cases and detailed experimental configurations. These materials support tracing model reasoning from inputs to incorrect predictions and reproducing the reported bottleneck analyses.
- C.1 Extended Error Cases: Complete prompt–response pairs document two representative error cases discussed in the main paper.They allow readers to trace each model’s reasoning chain from input to incorrect prediction.
- C.2 Experimental Configurations: Detailed experimental settings and configurations are provided for all three bottleneck-related experiments.
C.1 Error Cases for Main Experiment
The error cases expose recurring failures in assembly verification: models can identify component types yet misjudge specifications, materials, or correspondence to the required assembly. Additional examples show confusion when reasoning from missing parts, coordinates, or fine-grained connector and fastener attributes.
- CNC SCENARIO: The CNC error case correctly identifies all four fixture components but selects the Hex Nut (B) instead of the oversized Step Block (D) as anomalous.The response anchors on the nut’s apparent size relative to the T-slot bolt, despite the actual anomaly being the step block.
- MES SCENARIO: The MES error case hallucinates a plastic/nylon material for a metal Flat Washer (E), then predicts E instead of the correct answer A.The erroneous material inference drives selection of the wrong anomalous part.
- Bottleneck Analysis B: In the missing-part analysis, excluding coordinates forces models to rely on logical deduction, producing hesitation and repeated self-correction about assembly components.The analysis is intended to separate domain-specific reasoning from explicit grounding capabilities.
- Bottleneck Analysis C: Point-cloud coordinate inputs prompt repeated range calculations about relative part sizes, indicating that models process geometric data through extended numerical deliberation.One response compares z-span ratios across Parts A–D to infer relative dimensions.
- PCS SCENARIO: The PCS example illustrates specification confusion: the model classifies all four connectors as straight unions even though the query requires a reducing union.A straight union has equal-sized ends, whereas the required connector is 8mm to 6mm.
- MES SCENARIO: The MES expansion-bolt example distinguishes a standard five-component assembly while flagging a bolt with a different head shape as incorrect.The referenced assembly contains a Bolt, Nut, Flat Washer, Spring Washer, and Sleeve.
Examples of SWN SCENARIO
The SWN examples show MLLM responses inconsistently interpreting washer types, labels, and assembly correctness across three-view inputs and reference examples.
- One response identifies C as faulty because it appears to be a solid flat washer instead of the required spring washer.
- The standard SWN assembly contains an M18 bolt, nut, flat washer, and spring washer.
- Other responses conclude that all four required components are present, illustrating disagreement over whether the image contains an assembly error.
- Reference-based reasoning maps C to the spring washer and D to the flat washer, but the test image creates label and shape ambiguities.
E.5 Examples of Bottleneck Analysis B
The missing-part bottleneck example asks models to compare a three-view assembly against an explicit four-component specification, but responses mainly infer completeness from label counts.
- The task specifies four required M18 components: bolt, nut, flat washer, and spring washer.
- The rendered assembly uses labels A–D, and several responses map these four labels one-to-one onto the required components.
- Responses conclude that no part is missing because the image contains four labeled parts corresponding to the four specified components.
E.6 Examples of Bottleneck Analysis C
The raw point-cloud bottleneck example evaluates whether a model can identify one geometrically deviant part directly from serialized 3D coordinates.
- The task provides four parts and asks for the only part whose point-cloud geometry deviates from the specified model.
- Part C has the largest z-span, unusual aspect ratios, and extreme coordinate distributions compared with Parts A, B, and D.
- Part C is substantially larger across all dimensions, with spans approximately 3.7, 2.4, and 3.3 times larger for x, y, and z respectively.
- The combined dimensional and aspect-ratio deviations classify Part C as the defective part under the industrial quality-control framing.