PaperlayerView exact revision on arXiv

Source-linked AI summary

FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang, Guanzhi Deng, Alex Xue, Juan Du, Tianshu Yu, Garth Tarr, Linqi Song, Qiuzhuang Sun, Dacheng Tao

arXiv:2604.07413v2cs.CVcs.AIcs.LG

TL;DR

Manufacturing MLLM evaluations lack representative benchmarks and fine-grained multimodal data for standardized settings. FORGE pairs annotated images and point clouds with three-task evaluation, finding that domain knowledge, rather than visual grounding, is the main bottleneck.

  • Problem

    Existing manufacturing evaluations lack representative systematic benchmarks and multimodal data with fine-grained semantic annotations for standardized settings.

    Source: §1 · p. 3, §2 · p. 4

  • Method

    FORGE combines real-world images and point clouds with fine-grained annotations, then evaluates MLLMs on workpiece verification, surface inspection, and assembly verification.

    Source: §3.1 · p. 4, §3.2 · p. 5

  • Results

    MLLMs perform worse on model-number-level than workpiece-level tasks, and domain knowledge—not visual grounding—is identified as the main bottleneck.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    FORGE provides a fine-grained manufacturing dataset and benchmark for systematic MLLM evaluation and development.

    Source: §1 · p. 3

Abstract

from arXiv · show

The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.

1 Introduction

FORGE addresses gaps in manufacturing MLLM evaluation with a fine-grained multimodal dataset and three real-world cognitive tasks. Its benchmarking identifies morphology understanding and domain knowledge as deployment bottlenecks, while fine-tuning results suggest the data can support domain adaptation.

Introduction

  • Traditional vision systems mainly extract information for downstream manufacturing execution systems, whereas MLLMs may integrate heterogeneous inputs with reasoning to support higher-level planning and autonomous decisions.The paper frames whether MLLMs can understand, explain, and execute manufacturing decisions as its motivating question.Source: §1 · p. 2
  • Existing manufacturing evaluations rarely test fine-grained domain semantics and lack sufficient data scale, semantic detail, and comprehensive assessment of MLLM reasoning and decision-making.Many datasets use simulated or CAD-based data and treat workpieces as generic visual subjects rather than encoding details such as model numbers.Source: §1 · pp. 2–3
  • FORGE benchmarking reveals significant performance gaps in microscopic surface analysis and manufacturing-task reasoning, with morphology understanding and domain knowledge as major bottlenecks.The paper reports these findings from evaluation of state-of-the-art MLLMs under different evaluation settings.Source: §1 · p. 3
  • Supervised fine-tuning of a compact open-weight model on FORGE training data substantially improves accuracy on held-out manufacturing scenarios unseen during training.The authors present this as preliminary evidence that structured annotations can help close the domain-knowledge gap.Source: §1 · p. 3
  • FORGE provides a large-scale dataset integrating aligned 2D images and 3D point clouds with rich, fine-grained annotations across workpiece model numbers.The dataset is intended to support systematic MLLM evaluation and development for manufacturing perception and reasoning.Source: §1 · p. 3
  • The benchmark evaluates Workpiece Verification, Structural Surface Inspection, and Assembly Verification, which require fine-grained visual discrimination and complex reasoning such as checking assembly compatibility.The introduction also describes these applications as material sorting, quality inspection, and assembly recognition.Source: §1 · p. 3

2 Related Work

Prior MLLM evaluations cover visual anomaly detection and several manufacturing-related cognitive domains, but structured manufacturing settings require finer-grained precision than these frameworks typically assess. FORGE addresses this gap with multimodal data annotated for workpiece attributes and three tasks designed to reflect real-world manufacturing complexity.

2 Related Work

  • MMAD [20] evaluates fine-grained visual anomaly detection on datasets such as MVTec-AD, but manufacturing also requires domain-specific knowledge and reasoning.Other work covers manufacturing cognition and safety regulations [51], technical blueprints and standards [5], design synthesis and trade-offs [15], and reliability engineering and failure diagnosis [3].Source: §2 · pp. 3–4
  • Existing frameworks address manufacturing-related cognitive domains, including safety, blueprint standards, design trade-offs, and failure diagnosis.These areas are represented by MME-Industry [51], DesignQA [5], EngDesign [15], and FailureSensorIQ [3], respectively.Source: §2 · pp. 3–4
  • Structured manufacturing settings require distinguishing specific workpiece variants by fine-grained attributes such as model numbers, a demand constrained by scarce multimodal data and limited annotations.FORGE introduces a multimodal dataset with fine-grained workpiece attributes and three tasks designed to assess MLLMs against real-world manufacturing complexity.Source: §2 · p. 4

3 FORGE

FORGE combines real-world images and 3D point clouds with fine-grained manufacturing annotations to evaluate MLLMs on workpiece verification, surface inspection, and assembly verification. Its task designs target both coarse mismatches and subtle model-number errors, as well as defect recognition and assembly compatibility.

3.1 Dataset Curation

  • FORGE contains approximately 12,000 samples across tasks, including point clouds covering 14 workpiece categories and 90 models, plus about 3,000 images from four manufacturing scenarios.The data capture fine-grained geometric and visual details and include normal and abnormal samples.Source: §3.1 · pp. 4–5
  • For general MLLMs without native 3D encoders, FORGE renders point clouds as three-view images; image labels are manually refined, while task-specific point-cloud samples and defects are synthesized.Image labels begin with automated contour and coordinate extraction; WORKVERI and ASSYVERI use stitched point clouds, while SURFINSP defects are simulated using morphology-based algorithms and non-rigid deformation.Source: §3.1 · pp. 4–5

3.2 Task Description

  • The benchmark covers three manufacturing tasks and distinguishes coarse workpiece mismatches or missing components from fine-grained model-number errors in WORKVERI and ASSYVERI.It defines three WORKVERI scenarios, four ASSYVERI scenarios, and 14 SURFINSP workpieces.Source: §3.2 · p. 5
  • WORKVERI identifies items outside a specified batch, while SURFINSP detects defects and classifies their types across 14 components using 3D point clouds.WORKVERI spans image-based pneumatic connectors and point-cloud cup head screws and nuts; SURFINSP includes defect types such as cracks and dents.Source: §3.2 · p. 5
  • ASSYVERI tests whether MLLMs can reason over assembly rules to identify incompatible workpieces across four image- and point-cloud scenarios.The scenarios include metal and plastic expansion screws, CNC fixtures, and compatibility among metal screws, washers, and nuts; the overall design assesses fine-grained logical reasoning and judgment in manufacturing scenarios.Source: §3.2 · p. 5

4 Experiments

Experiments evaluate 18 MLLMs across manufacturing tasks and reveal gaps in surface-morphology understanding, domain knowledge, spatial reasoning, and fine-grained model-number recognition. They also examine evaluation settings and qualitative error cases.

4.1 Evaluated Models

  • The benchmark evaluates 18 representative MLLMs from open- and closed-source families across three manufacturing tasks.The experiments also report error cases, analyze perception versus domain-knowledge limitations, and examine small-model fine-tuning for cross-scenario generalization.Source: §4 · p. 5, §4.1 · p. 7

4.2 Evaluation Settings and Metrics

  • Tasks use multiple-choice exact-match accuracy, with zero-shot, reference-conditioned, and demonstration settings; SURFINSP defaults to six representative workpieces and demonstrations to two examples.WORKVERI and ASSYVERI distinguish coarse workpiece discrepancies from fine-grained model-number differences.Source: §4.2 · p. 7

4.3 Research Results and Analysis

  • MLLMs lead on workpiece and assembly understanding but perform worst on surface inspection, exposing gaps in microscopic morphology and physical-detail perception.Kimi-K2.5 and Gemini-3-Flash achieve leading performance in WORKVERI and ASSYVERI, while SURFINSP yields the lowest performance among the three tasks.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • In image-based WORKVERI and ASSYVERI, in-context demonstrations universally improve on zero-shot, while reference conditioning does not consistently help and can degrade performance.This pattern suggests demonstrations bridge gaps in task logic and reasoning that simple reference images do not; the authors report visual grounding is not the bottleneck.Source: §4.3 · p. 8
  • For three-view inputs, zero-shot performs best; reference conditioning and demonstrations can induce spatial confusion, especially harming workpiece-level recognition.The authors attribute the decline to ineffective contextual guidance for multi-angle spatial representations and note point-cloud results also indicate limited explicit spatial perception.Source: §4.3 · p. 8
  • Across open- and closed-source models, workpiece-level recognition outperforms model-number-level recognition, highlighting persistent difficulty with fine-grained domain specificity.The disparity appears in both model families, despite the importance of fine-grained analysis in manufacturing scenarios.Source: Figure 4 · p. 8, §4.3 · pp. 8–9

4.4 Qualitative error cases analysis

  • Error analysis shows MLLMs may infer materials from visual texture and use those inferred properties in reasoning, even when they misidentify the material or it is irrelevant.The example suggests latent potential to recognize workpiece materials and incorporate physical properties; the paper also notes reasoning about service status.Source: §4.4 · p. 9

B. Failure on model number

The analyses identify domain-specific manufacturing knowledge—not visual localization—as a key bottleneck, while showing that FORGE annotations can support transferable gains from fine-tuning. The section also reports that models can recognize nuanced wear and that visual projection is more effective than raw coordinate text for generic MLLMs.

B. Failure on model number

  • Recognizing nuanced wear and chipping in assembly components suggests MLLMs may provide perceptual feedback useful for graded workpiece-degradation assessment and predictive maintenance.The model’s reasoning noted heavy wear or chipping on the strap clamp and wear on block D.Source: B. Failure on model number · p. 9

4.5 Bottleneck Analysis

  • 98.9% average single-image grounding for Gemini-3-Flash and 84.3% cross-image accuracy for Gemini indicate that visual localization is not the primary source of full-benchmark failures.GPT-5.2 reached 80.5% on cross-image comparison, while four of five models exceeded 97.6% on L→C; cross-image comparison remained harder but was above chance for the top four models on letter matching.Source: §4.5 · p. 10, Table 4 · p. 10
  • 74.9–90.7% overall image accuracy with structured assembly descriptions exceeded the 23.3% random baseline, but flat-washer accuracy remained 23.3–60.0% on images and 8.3–74.5% on three-view.Models detected that a washer was absent but struggled to identify which washer; Seed 1.6 also showed a bias toward predicting a missing part, scoring 43.3% on normal cases overall and as low as 10% on MES SCENARIO.Source: §4.5 · p. 10, Table 5 · p. 10
  • Near-20% random-baseline performance on SURFINSP and Gemini-3-Flash’s 53.6% zero-shot WORKVERI accuracy show that raw point-cloud text poorly replaces visual projection for generic MLLMs.The SURFINSP models tended to predict the dominant class rather than discriminate defect types; coordinate distributions offered some signal for coarse-grained shape comparison.Source: §4.5 · pp. 10–11, Table 6 · p. 10

4.6 From Benchmark to Training Resource

  • 90.8% relative improvement on held-out WORKVERI three-view and 27.1% on ASSYVERI image followed fine-tuning, with the 3B model matching Qwen3-VL-235B’s 54.4% on WORKVERI.Training and evaluation used different product scenarios; on ASSYVERI, the fine-tuned model surpassed all reference models except Gemini-3-Flash and GPT-5.2.Source: §4.6 · p. 11

5 Conclusion … A.1.2 MLLMs in Manufacturing Scenario.

FORGE evaluates 18 MLLMs across three manufacturing tasks and finds gaps in fine-grained reasoning and microscopic surface analysis, with domain knowledge and morphology understanding—not visual grounding—as key bottlenecks. The related-work sections trace traditional CV approaches and emerging manufacturing MLLMs, highlighting limitations in reasoning and real-world factory coverage.

5 Conclusion

  • Across three manufacturing tasks, 18 MLLMs recognize macroscopic parts but struggle with fine-grained reasoning and microscopic surface analysis; domain knowledge and morphology, not visual grounding, are key gaps.Fine-tuning a compact 3B-parameter model on FORGE annotations approaches frontier-scale performance on held-out scenarios.Source: §5 · p. 11, Figure 6 · p. 11

A.1.1 CV models in Manufacturing Scenario.

  • Manufacturing computer vision spans surface defect detection, geometric measurement, and robotic guidance.The literature has evolved from traditional image processing to deep learning-based approaches.Source: §A.1.1 · p. 17
  • Surface anomaly detection has shifted from supervised defect classification to unsupervised methods that identify deviations from normal data.Early CNN-based approaches used large annotated datasets [16] [33]; later methods include reconstruction-based AutoEncoders and GANs and feature-embedding approaches [35].Source: §A.1.1 · p. 17
  • 3D-aware anomaly detection combines appearance and depth or analyzes point clouds directly to address geometric information and unstructured 3D data.G2SF combines 2D appearance and 3D depth with geometry-guided score fusion, while PointSGRADE uses graph representations and sparse learning.Source: §A.1.1 · p. 17
  • CV supports geometric metrology and robotic manipulation through 6D pose estimation for component assembly [46] and dimensional measurement for quality control.These models provide spatial coordinates and geometric parameters for precise execution of predefined mechanical tasks.Source: §A.1.1 · p. 17
  • Traditional CV systems map pixels to fixed classes and struggle with semantic reasoning, complex instructions, causal understanding, and unseen manufacturing scenarios.Despite perceptual precision, their closed-set operation limits autonomous control and adaptation, motivating MLLMs with cognitive reasoning capabilities.Source: §A.1.1 · p. 17

A.1.2 MLLMs in Manufacturing Scenario.

  • Manufacturing MLLM approaches include code-based logical verification, checklist-based zero- or few-shot detection, and difficulty-aware alignment, but remain tied to existing datasets and predefined tasks.LogiCode generates executable Python for logical constraints but relies on manual annotations [54]; LogicQA generates checklists from standard samples [23]; EMIT uses response resampling and advantage reweighting [13].Source: §A.1.2 · p. 17

A.2 Extended Main Results · A.3 Task description

Across FORGE’s manufacturing tasks, MLLMs remain far from acceptable performance, with domain-specific manufacturing knowledge a major challenge and even advanced models struggling on SURFINSP. Extended results further show that reference-based methods can worsen 3V workpiece verification, while SURFINSP’s response to more references varies by anomaly type.

A.2 Extended Main Results

  • Most open-source models perform near the random baseline, and the results identify insufficient ability to internalize and reason about complex manufacturing standards as a central challenge.Even stronger general-purpose capabilities leave specialized manufacturing scenarios difficult for MLLMs.Source: §A.2 · pp. 17–18
  • Gemini-3-Flash and GPT-5 achieve state-of-the-art results in WORKVERI and ASSYVERI, yet even the most advanced models struggle to exceed 50% accuracy in SURFINSP.The reported results are consistent with the main text and show a substantial performance gap across tasks.Source: §A.2 · pp. 17–18, Table 7 · p. 18
  • In 3V WORKVERI and ASSYVERI, model-number- and workpiece-level tasks both expose a bottleneck in 3D spatial understanding.Table 7 breaks down performance by workpieces and model numbers; the passage identifies this pattern across both levels.Source: §A.2 · p. 18, Table 7 · p. 18
  • Under 3V, Ref-Cond and ICD degrade performance versus zero-shot, with a steeper drop for workpiece-level tasks because workpiece similarity is lower and visual features vary more under spatial and angular shifts.The greater variability makes it harder to align spatial semantics when integrating reference examples.Source: §A.2 · p. 18
  • In SURFINSP, performance generally improves as references increase, but DEFORMATION and DENT are exceptions, indicating distinct performance trajectories across anomaly types.The passage attributes these differences to varying domain-knowledge requirements that lead MLLMs to interpret the task differently.Source: §A.2 · p. 18

A.3 Task description

  • FORGE designs three manufacturing tasks to evaluate MLLM capabilities in material sorting, quality inspection, and assembly recognition, with task design, test and evaluation data, settings, and data collection described.Source: §A.3 · p. 18

A.3.1 Raw Data Collection.

FORGE’s raw data combines real manufacturing images and 3D point clouds, which are curated and transformed into task-specific evaluation inputs. The section also defines three evaluation settings and two classes of manufacturing errors.

Data acquisition

  • Point clouds were scanned at 0.02 mm precision and processed into reconstructed models, while images were captured with a 50-megapixel sensor and Leica lens at consistent angles and distances.Marker recognition established a local coordinate system, and developer spray mitigated metallic reflectivity; image scenarios used randomly selected workpieces in random orientations.Source: §A.3.1 · pp. 18–19

Raw data subsets

  • The point-cloud subset covers 14 component categories and 90 model numbers, while the image subset contains approximately 3,000 images from four manufacturing scenarios.Point clouds support WORKVERI, SURFINSP, and ASSYVERI; images primarily support WORKVERI and ASSYVERI under real-world conditions, with normal and abnormal samples in each scenario.Source: §A.3.1 · p. 19

Data Processing

  • Image preprocessing extracted artifact contours and coordinates, then manual calibration mapped them to manufacturing model information for ground-truth labels.Point-cloud preprocessing used task-specific strategies rather than feeding raw files directly to the evaluation models.Source: §A.3.1 · p. 19
  • Approximately 30,000 point-cloud samples were constructed across all three tasks, with synthetic defects, constrained defect-point proportions, and 20 random rotations per sample; MLLMs received 3V orthogonal projections.WORKVERI and ASSYVERI samples stitch 4–5 normal artifacts; SURFINSP simulates cracks, deformation, dents, and cuts, with defect points constrained to 5%–15%.Source: §A.3.1 · p. 20

Eval setting

  • Zero-Shot uses only the test image and query; Ref-Cond adds three correct normal references, while ICD adds a similar labeled example, and errors distinguish workpiece mismatches from model-number mismatches.Different-workpiece errors include component absence and other workpiece-level anomalies; different-model-number errors concern variants within the correct workpiece category.Source: §A.3.1 · p. 20

A.3.2 Task Description of WORKVERI.

WORKVERI evaluates whether MLLMs can identify manufacturing workpieces that fail explicit specifications or model requirements using 3D point clouds or images.

A.3.2 Task Description of WORKVERI.

  • WORKVERI tests whether MLLMs can identify workpieces that do not meet explicit specifications or model requirements from 3D point clouds or images.Material identification and verification are described as fundamental steps in manufacturing material sorting.Source: §A.3.2 · pp. 20–21
  • The task covers three scenarios: point-cloud evaluation of nuts and cup head screws, and image-based evaluation of pneumatic connectors.Source: §A.3.2 · p. 21

A.3.3 Task Description of SURFINSP.

SURFINSP evaluates whether MLLMs can identify manufacturing defects from workpiece point clouds, determine defect types, and support quality inspection against manufacturing requirements.

A.3.3 Task Description of SURFINSP.

  • SURFINSP evaluates MLLMs’ ability to identify manufacturing defects in workpiece point cloud data.Quality inspection aims to ensure workpieces, components, and production processes comply with design specifications, process standards, and customer requirements.Source: §A.3.3 · pp. 21, 23
  • The task asks whether a workpiece contains defects and, if so, which type of defect it has.The considered defect types are Crack, Cut, Deformation, and Dent.Source: §A.3.3 · p. 23
  • The evaluation covers 14 categories of manufacturing components.Point cloud examples for different mechanical components are shown in Figures 11–24.Source: §A.3.3 · p. 23

A.3.4 Task Description of ASSYVERI.

ASSYVERI evaluates whether MLLMs can identify workpieces that violate specified assembly requirements by reasoning over images or point clouds. It covers four manufacturing scenarios involving expansion screws, CNC fixtures, and screw–washer–nut compatibility.

A.3.4 Task Description of ASSYVERI.

  • ASSYVERI asks MLLMs to identify workpieces that fail specified assembly requirements using compatibility rules, specifications, or model requirements.It targets reasoning about assembly relationships, structural hierarchies, and compatibility constraints.Source: §A.3.4 · p. 23
  • Four scenarios cover image-based metal and plastic expansion screws and CNC fixtures, plus point-cloud compatibility among metal screws, washers, and nuts.ASSYVERI is one of three core tasks in a multimodal benchmark designed to assess perception and logical judgment under complex manufacturing rules.Source: §A.3.4 · p. 23

B Training Details and Extended Results · Model Type WORKVERI 3V (NUTS SCENARIO) ASSYVERI Img (CNC SCENARIO)

Scenario-held-out fine-tuning improves performance on novel manufacturing categories, with stronger transfer on WORKVERI than the more visually and semantically varied ASSYVERI task. The results also reveal a WORKVERI calibration trade-off and more uniform ASSYVERI gains across error types.

B Training Details and Extended Results

  • Each task-specific Qwen2.5-VL-3B model uses full-parameter supervised fine-tuning, with 8,337 CHS examples for WORKVERI 3V and 627 MES/PES examples for ASSYVERI image.Training uses bfloat16, 3 epochs, effective batch size 32, learning rate 1 × 10−5, a frozen ViT, and standardized MCQ responses.Source: B Training Details and Extended Results · p. 23
  • Scenario-based evaluation trains on one product category and tests on another: CHS to NUTS for WORKVERI 3V, and MES/PES to CNC for ASSYVERI image.This protocol tests transfer of manufacturing reasoning rather than assembly-specific visual patterns.Source: B Training Details and Extended Results · p. 23

Model Type WORKVERI 3V (NUTS SCENARIO) ASSYVERI Img (CNC SCENARIO)

  • WORKVERI transfers well because it tests recognition of specification mismatches across assemblies, while ASSYVERI is harder due to diverse error modes, visual complexity, and unseen parts and layouts.Even with 627 training examples, ASSYVERI fine-tuning measurably improves performance on the novel CNC product category.Source: B Training Details and Extended Results · p. 23, Model Type WORKVERI 3V (NUTS SCENARIO)… · p. 24
  • On held-out scenarios, fine-tuned Qwen2.5-VL-3B nearly matches Qwen3-VL-235B on WORKVERI (53.8% vs. 54.4%) and surpasses Seed 1.6, Qwen3-VL-235B, and Mistral 3 8B on ASSYVERI.Table 12 compares the fine-tuned model with five ablation models on the same held-out scenarios.Source: B Training Details and Extended Results · p. 23, Table 12 · p. 24
  • On WORKVERI 3V, Normal-case accuracy rises from 24.3% to 74.8%, while Different Model Number accuracy falls from 33.8% to 23.8%.The model becomes more conservative; Normal cases are 59% of the evaluation set (115/195), yielding a large overall improvement, while the mismatch-sensitivity trade-off may warrant loss weighting or balanced sampling.Source: Model Type WORKVERI 3V (NUTS SCENARIO)… · p. 24
  • On ASSYVERI image, Different Model Number accuracy increases from 20.5% to 27.0% and Extra Part accuracy from 37.0% to 44.4%.These more uniform gains suggest training transfers across error types to the novel CNC product category.Source: Model Type WORKVERI 3V (NUTS SCENARIO)… · p. 24

C Extended Error Cases and Bottleneck Analysis

Section C supplements the main paper’s qualitative error and bottleneck analyses with complete transcripts, detailed experimental configurations, and additional discussion of the findings.

C.1

C.2

C.1 Error Cases for Main Experiment … D Image examples

The appendix documents two ASSYVERI failure cases, specifies the bottleneck-analysis protocols, discusses reasoning and point-cloud limitations, and illustrates benchmark image diversity. Together, these materials show how models can misjudge part specifications even when identifying components correctly.

C.1 Error Cases for Main Experiment

  • The CNC case wrongly flags undersized-looking nut B instead of oversized step block D, while the MES case selects washer E after inferring nylon material, despite correct answer A.The two examples are zero-shot ASSYVERI errors from Gemini-2.5-Flash; their transcripts expose the models’ mistaken size and material judgments.Source: §C.1 · pp. 24, 25, 27, 28

C.2.1 Settings and Configurations.

  • Bottleneck analyses use five representative open- and closed-source models at temperature 0.0; grounding tests localization, while missing-part tests provide assembly specifications.Grounding includes coordinate-to-letter and letter-to-coordinate tasks on 500 sampled cases, plus cross-image correspondence on 513 sampled pairs; missing-part tests cover image and three-view cases with a “No part is missing” option.Source: §C.2.1 · pp. 28–29
  • Raw point-cloud text tests whether serialized coordinates can serve as an alternative 3D interface for general-purpose MLLMs.Part coordinates are randomly sampled to 500 points and scaled by 10^2; WORKVERI, SURFINSP, and ASSYVERI are evaluated in Zero-Shot, Ref-Cond, and ICD modes.Source: §C.2.1 · p. 29

C.2.2 Additional Discussion of Bottleneck Analyses.

  • Without coordinate cues, missing-part responses reveal fragile assembly comprehension, with models hesitating and repeatedly correcting their logical assumptions.The analysis contrasts recognizing individual components with robustly understanding their relational assembly logic.Source: §C.2.2 · pp. 29–30
  • Point-cloud responses use coordinate ranges to estimate relative part sizes, but struggle with microscopic morphology, helping explain weaker performance beyond WORKVERI.The analysis also identifies coordinate-based size reasoning as a potential capability for extracting insights from point-cloud inputs.Source: §C.2.2 · p. 30

D Image examples

  • Image examples span WORKVERI, SURFINSP, and ASSYVERI scenarios, showing normal and representative anomalous cases; the appendix also includes Set-of-Mark grounding images.The examples include WORKVERI Figures 8–10, SURFINSP Figures 11–24, and ASSYVERI Figures 25–28.Source: D Image examples · p. 31, Figure 8 · p. 31, Figure 11 · p. 33

D.1 The Grounding Examples of Image Data (Bottleneck Analysis A) … Examples of SWN SCENARIO

This appendix section presents labeled image-grounding examples and prompt–response formats for manufacturing benchmark tasks, including identifying specification mismatches, classifying surface defects, and checking assemblies against references. The examples show the visual context provided to models and the variation in their diagnoses.

D.1 The Grounding Examples of Image Data (Bottleneck Analysis A)

  • Labeled assembly images from MES, PES, CNC, and PCS scenarios support single-image grounding (C→L, L→C) and cross-image correspondence (L→L, C→C) tasks.Each component is overlaid with a distinct letter label.Source: §D.1 · p. 42, Figure 29 · p. 42, Figure 30 · p. 43

E Data examples

  • The appendix provides complete system prompts, user prompts, and raw model responses for all three benchmark tasks and three bottleneck analyses.These examples expose the evaluation pipeline’s full input–output format.Source: E Data examples · p. 46

E.1 Examples of WORKVERI

  • WORKVERI examples cover image and three-view inputs, asking models to identify the assembly part with the wrong model number or specification.The task framing includes mismatch against the stated specification or the majority of parts.Source: Examples of PCS SCENARIO · p. 46

Examples of PCS SCENARIO

  • PCS examples use reference images of 8mm-to-6mm pneumatic connectors, candidate coordinates, and a letter-choice output to identify mismatched parts.A reference-conditioned raw response identifies candidate C from its differing body geometry.Source: Examples of PCS SCENARIO · pp. 51–52

Examples of CHS SCENARIO

  • The CHS zero-shot example frames three-view bolt inspection as comparing test parts with standard references and returning the letter of any faulty component.Parts are labeled and rendered in distinct colors; the prompt names wrong model numbers, dimensions, or workpiece type as possible issues.Source: Examples of CHS SCENARIO · p. 52
  • In the CHS example, the response identifies part D as mismatched because its top-view head shape differs from parts A, B, and C.The prompt asks the model to compare labeled parts across merged front, side, and top views.Source: Examples of CHS SCENARIO · p. 53

E.2 Examples of SURFINSP

  • SURFINSP examples describe workpieces such as corner brackets and rivet nuts, and frame surface inspection as classification into five defect or good-condition categories.The categories are crack, cut, deformation, dent, and good.Source: §E.2 · pp. 53–54

Examples of SURFINSP Cup Head Screw

  • For a cup-head screw shown with three normal reference images, the example response classifies the part as abnormal with defect type B, cut.The prompt defines at most one abnormality per part and offers crack, cut, deformation, and dent as defect choices.Source: Examples of SURFINSP Cup Head Screw · p. 55

E.3 Examples of ASSYVERI

  • ASSYVERI examples cover image and point-cloud inputs for detecting extra, unnecessary, or incorrectly specified parts in assemblies.The MES examples compare test assemblies with reference images and ask models to select a mismatched component.Source: §E.3 · p. 55, Examples of MES SCENARIO · p. 56

Examples of MES SCENARIO

  • The MES examples identify different mismatches across responses: zero-shot diagnoses select D or E, while the reference-conditioned response selects C as the wrong bolt.The responses variously describe an incorrect bolt type or an extra small washer in the metal expansion bolt assembly.Source: Examples of MES SCENARIO · pp. 59, 61, 62

Examples of PES SCENARIO

  • The PES example prompt uses reference images and a query to identify the test-image part with the wrong model number.Source: Examples of PES SCENARIO · p. 63
  • PES responses identify F as an extra screw, E as a differently shaped anchor, or F as a mismatched anchor, depending on the response.The ICD response counts four screws where three are expected; the zero-shot response selects the differently patterned anchor; the Ref-Cond response selects an anchor with a different profile.Source: Examples of PES SCENARIO · pp. 65, 67, 68

Examples of CNC SCENARIO

  • CNC responses identify part B, the nut, as mismatched because it is smaller or a different type than the nuts in the reference assemblies.The responses describe the test nut as a possible T-nut or small hex nut, unlike the larger standard or flange nuts shown in the references.Source: Examples of CNC SCENARIO · pp. 70–71

Examples of SWN SCENARIO

  • The SWN excerpt provides an industrial assembly quality-inspection system prompt but no example response or identified mismatch.The supplied passage contains only the system prompt.Source: Examples of SWN SCENARIO · p. 72
  • The initial response flags C because it interprets the pink washer as a solid flat washer rather than the required split spring washer.The standard set comprises a bolt, nut, flat washer, and spring washer; the response identifies D as the flat washer and C as the expected spring washer.Source: Examples of SWN SCENARIO · pp. 73–74
  • A zero-shot response first recognizes the four components as the required set, but then speculates that spring washer D is extra and answers D.It describes C as a flat washer and D as a split spring washer, both matching the stated standard, before assuming the spring washer is redundant.Source: Examples of SWN SCENARIO · pp. 75–77
  • In the reference-conditioned example, the response maps A to the nut and B to the bolt, then flags A when the test image shows A as the bolt.It identifies C as the spring washer and D as the flat washer, and concludes that all four parts are present but the A/B assignments conflict with the references.Source: Examples of SWN SCENARIO · pp. 78–81

E.4 Examples of Bottleneck Analysis A

This section illustrates cross-image visual grounding with Set-of-Mark images, showing coordinate-to-coordinate and letter-to-letter examples from Mistral-3-8B. The examples match identified parts across images using their visual features.

Cross-image visual grounding analysis

  • The analysis tests whether a model can match a part identified in one Set-of-Mark image to its visual counterpart in another, using C→C and L→L cases.The examples are from Mistral-3-8B.Source: §E.4 · p. 81

Grounding example 1

  • In the first example, the response matches the cylindrical part with a hexagonal section near the top to label C in Image 2.The response gives C as the answer.Source: §E.4 · p. 82

Grounding example 2

  • In the second example, the response matches Part D in Image 1 to Part C in Image 2 based on their threaded and smooth sections.The response gives C as the answer.Source: §E.4 · p. 82

E.5 Examples of Bottleneck Analysis B

The example presents a missing-part task with an explicit four-component assembly specification. Qwen3-VL-235B repeatedly reconsiders the part mapping, then answers that no component is missing.

E.5 Examples of Bottleneck Analysis B

  • The task provides a four-part M18 Metal Expansion Bolt specification and asks the model to identify any missing component from a three-view rendering.The prompt lists a bolt, nut, flat washer, and spring washer as required parts.Source: §E.5 · pp. 82–83
  • The response vacillates over whether the four labels establish that all required parts are present, before settling on a one-to-one mapping and no omission.It initially entertains alternative mappings, then identifies A as the flat washer, B as the bolt, C as the nut, and D as the spring washer.Source: §E.5 · p. 84
  • Qwen3-VL-235B concludes that no component is missing, answering A after mapping the four labels to the bolt, nut, flat washer, and spring washer.The response treats the four labeled parts as corresponding one-to-one with the four required components.Source: §E.5 · p. 86

E.6 Examples of Bottleneck Analysis C

In a raw point-cloud example, Qwen3-235B receives text-serialized coordinates without visual rendering and identifies Part C as the deviation based on its dimensional differences.

E.6 Examples of Bottleneck Analysis C

  • The example tests Qwen3-235B on ASSYVERI using integer-scaled 3D coordinates serialized as text, without visual rendering.The prompt asks the model to identify the single deviating part among four parts.Source: §E.6 · pp. 86–87
  • The most striking difference is in z: Part C’s span is ~6200, 3.26 larger than Part A’s (1900) and 4.77 larger than Part B’s.Source: §E.6 · p. 88
  • The response flags Part C’s z values up to 49900 and y values down to -6400 as unusual relative to the other parts.It also describes Part C’s y:z ratio as 1.13:1, versus 1.53:1 for A, 2.00:1 for B, and 1.23:1 for D.Source: §E.6 · p. 88
  • The response identifies Part C as defective because its disproportionate scale and abnormal y:z ratio indicate a fundamentally different shape.The example is a model response, not an independent verification of the part’s defect status.Source: §E.6 · p. 88

Related papers

Loading 2604.07413v2…