PaperlayerView exact revision on arXiv

Source-linked AI summary

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing

arXiv:2605.30794v2cs.CVcs.AI

TL;DR

MLLMs remain brittle on dense mechanical drawings, and existing benchmarks do not systematically evaluate comprehensive, engineering-grade understanding. The paper introduces MechVQA and trains the domain-specialized MechVL model, which outperforms the strongest closed-source baseline by 7.57 percentage points on the MechVQA total score.

  • Problem

    Existing MLLMs and benchmarks lack reliable, unified evaluation of dense mechanical drawings requiring structured perception, multi-view consistency, and engineering-grade reasoning.

  • Method

    The paper introduces MechVQA and trains MechVL through supervised instruction tuning followed by DAPO-based reinforcement learning with taxonomy-aligned rewards.

  • Results

    MechVL outperforms the strongest closed-source baseline by 7.57 percentage points on the MechVQA total score, with especially consistent gains on reasoning-intensive and standards-sensitive subtasks.

  • Takeaways & Limitations

    MechVQA provides a comprehensive testbed, while MechVL establishes a domain-specialized baseline for mechanical drawing understanding.

  • Takeaways & Limitations

    MechVL is intended as decision support rather than a replacement for certified professional judgment, requiring human oversight and verification in engineering workflows.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks. However, they remain brittle on mechanical engineering drawings, where high annotation density and weak domain knowledge, compounded by unreliable spatial relation reasoning under strict projection rules and geometric constraints, make decisive cues easy to miss and frequently lead to wrong answers. To bridge this gap, we introduce the first comprehensive mechanical drawing understanding dataset, MechVQA, created through a semi-automated construction and quality-control pipeline. MechVQA contains 3.3k high-density pictures with 21K question-answer pairs, spanning 10 different fine-grained tasks across three capability levels: Recognition, Reasoning, and Judging, providing a testbed to evaluate and improve MLLM understanding on real-world mechanical drawings. On top of MechVQA, we then develop the MechVL model through a multi-stage training paradigm, building a strong domain-specialized baseline. Extensive experimental results demonstrate that MechVL outperforms the strongest closed-source baseline by 7.57 percentage points on the MechVQA total score, significantly enhancing mechanical drawing understanding ability and providing a reusable foundation for deploying MLLMs in mechanical design and inspection scenarios.

1. Introduction

The paper addresses the brittleness of general-purpose MLLMs on dense mechanical drawings by introducing the MechVQA benchmark and the domain-specialized MechVL model. MechVQA covers comprehensive drawing understanding, while MechVL uses multi-stage post-training to improve reliability and performance.

  • Motivation: General-purpose MLLMs remain brittle because dense annotations, weak domain priors, and unreliable projection-constrained spatial reasoning cause decisive cues to be missed.These failures can produce structurally inconsistent interpretations of mechanical drawings.
  • Motivation: Existing multimodal benchmarks lack dedicated, systematic coverage of comprehensive mechanical drawing understanding.Adjacent engineering benchmarks typically focus on narrower tasks such as requirements QA, symbol recognition, or floor-plan literacy.
  • MechVQA: 3.3K drawings and 21K question-answer pairs comprise MechVQA, spanning Recognition, Reasoning, and Judging across 10 fine-grained subtasks.The benchmark uses real part and complex assembly drawings collected from publicly available textbooks, professional handbooks, and design platforms.
  • MechVL: MechVL is a domain-specialized baseline trained through multi-stage post-training with supervised instruction tuning and DAPO-based reinforcement learning.Its taxonomy-aligned rewards target strict output formatting, factual accuracy, and response quality.
  • Results: 6% improvement over closed-source baselines is achieved by MechVL on the MechVQA total score.Benchmarking and ablation results also show gains in cross-view reasoning, constraint-sensitive inference, and standards-aware judgment.

2. Related Work

Related work spans general-purpose multimodal language models, emerging engineering and CAD-oriented VQA benchmarks, and post-training methods that combine supervised fine-tuning with reinforcement learning. These studies motivate domain-specific mechanical drawing understanding and structured reasoning.

  • MLLMs and Visual Question Answering: General-purpose MLLMs excel at image–text retrieval and VQA, while visual instruction tuning improves instruction following in models such as LLaVA, MiniGPT-4, and Gemini 1.5.However, they often lack domain-specific drafting knowledge, including projection conventions and tolerances.
  • Engineering and CAD VQA: Traditional VQA benchmarks emphasize everyday scenes, leaving engineering and CAD domains underrepresented.Recent efforts include CReFT-CAD for three-view reasoning and PHT-CAD for parametric primitive analysis.
  • Engineering and CAD VQA: Mechanical reasoning benchmarks such as MechBench probe physical laws through schematic puzzles but lack text-rich engineering drawing contexts.This contrasts with newer engineering and CAD efforts addressing three-view reasoning and parametric primitive analysis.
  • Instruction tuning with SFT and RL: A prevailing post-training paradigm first uses supervised fine-tuning for instruction following and then reinforcement learning to enhance performance.DeepSeek-R1 demonstrates that GRPO can boost reasoning by estimating advantages from group-normalized rewards without a memory-intensive value critic.

3. MechVQA Dataset

MechVQA is constructed from expert-curated mechanical drawings through metadata extraction, taxonomy-driven question generation, multi-stage quality control, and expert auditing. The resulting 20,778-question benchmark spans three capability axes and ten subtasks, with leakage-mitigated train, validation, and test splits.

  • Data curation: 3,281 high-quality drawing images remain after domain experts remove low-quality, incomplete, and poorly scanned drawings, followed by OCR and MLLM-based metadata extraction.The corpus draws from public textbooks, handbooks, and design platforms and includes orthographic, isometric, part, and assembly drawings.
  • Question taxonomy: Three capability axes—Recognition, Reasoning, and Judging—organize ten subtasks spanning explicit extraction, multi-step mechanical inference, and engineering-rule verification.The subtasks are IC, DA, TT, IL; SU, GC, AR, PM; and AD, CJ, respectively.
  • Question construction: Multi-stage quality control filters candidate questions for format, scope, alignment, factual grounding, and unique answers, then uses multi-model majority voting and stratified expert audits.Pairs without clear majority agreement are discarded, while retained samples receive difficulty labels based on drawing and question complexity.
  • Dataset composition and splitting: 20,778 question-answer pairs form MechVQA, which is split into train, validation, and test sets at an 8:1:1 ratio using drawing-group similarity clustering and stratification.The protocol reduces leakage from duplicated drawings and near-duplicate variants, although contamination from public sources cannot be ruled out.

4. MechVL: A Domain-Specialized Baseline

MechVL is initialized from Qwen3-VL-Instruct-4B and trained through full-parameter supervised fine-tuning followed by DAPO-based self-play reinforcement learning. Its composite rewards target correctness, schema validity, and professional explanation quality for dense mechanical drawings.

  • Supervised fine-tuning: MechVL initializes from Qwen3-VL-Instruct-4B and applies full-parameter SFT to the LLM module while freezing the vision encoder and projection layers.SFT uses the MechVQA training split and trains responses grounded in drawing cues and the required response schema.
  • DAPO reinforcement learning: DAPO improves training stability and efficiency through decoupled clipping, Dynamic Sampling, and Overlong Reward Shaping, without an explicit KL penalty.Dynamic Sampling retains groups containing both positively and negatively rewarded samples, while overlong shaping reduces truncation-related reward noise.
  • Reward design: The composite reward balances verifiable accuracy, strict format compliance, and explanation quality for mechanical drawing VQA.Accuracy uses semantic-equivalence judging, format requires exactly one <think>...</think> rationale and one <answer>...</answer> span, and quality scores logic, professionalism, and conciseness.
  • Two-stage self-play RL: Two-stage self-play RL first trains on the full MechVQA training split and then oversamples underperforming subtasks using the same objective and reward.The resulting reward design addresses missed annotations, cross-view inconsistencies, and numerically plausible answers that violate drawing constraints.

5. Experiments

Experiments evaluate MechVL and broad open- and closed-source MLLM baselines on MechVQA using accuracy, showing that MechVL-4B-RL achieves superior overall, difficulty-balanced, and capability-wise performance. Ablations further identify targeted RL, DAPO, and the full reward design as important contributors.

  • Main results: 84.85 overall score: MechVL-4B-RL surpasses GLM-4.6V at 78.91 by +5.94 and Gemini-3-Pro-Preview at 77.28 by +7.57.Relative to MechVL-4B-SFT at 76.36, RL adds +8.49, indicating improved reliability on dense drawings and constraint-sensitive tasks.
  • Difficulty stratification: 94% easy, 79% medium, and 75% hard accuracy: MechVL-4B-RL delivers the strongest and most balanced performance across difficulty levels.Against SFT, RL raises medium accuracy from 70% to 79% and hard accuracy from 53% to 75%, while easy accuracy changes from 92% to 94%.
  • Capability-wise comparison: 89.70 Recognition, 77.04 Reasoning, and 82.81 Judging: MechVL-4B-RL improves over GLM-4.6V by +5.68, +6.54, and +11.00, respectively.Relative to Gemini-3-Pro-Preview, the gains are +8.14, +19.62, and +2.29.
  • SFT vs. SFT+RL: 83.26 Avg. score: targeted RL improves Reasoning from 70.75 to 77.04 and Recognition from 86.26 to 89.70 beyond full-data RL.Full-data DAPO raises Total from 76.36 to 81.95 and Avg. from 70.39 to 79.12 relative to SFT-only training.
  • RL algorithm ablation: 81.95 Total and 79.12 Avg. scores: DAPO outperforms GRPO at 80.47 / 74.80 and GSPO at 78.77 / 73.73.DAPO leads all capability-wise averages, with Reasoning at 70.75 versus 64.49 for GRPO and 61.29 for GSPO.
  • Reward design ablation: 84.85 Total and 83.26 Avg.: the Full reward performs best, whereas removing quality yields 83.44 Total and token-level F1 yields 80.33.The full reward combines semantic correctness, schema compliance, and explanation quality; it maintains controlled response lengths around 1.2K–1.25K tokens.

6. Conclusion

The paper studies mechanical drawing understanding with MLLMs under orthodox drafting conventions and introduces MechVQA and MechVL as contributions toward this task.

  • MechVQA is a benchmark of real part and assembly drawings with ten subtasks across Recognition, Reasoning, and Judging.
  • Mechanical drawing interpretation requires dense visual reading, projection-consistent spatial understanding, multi-step geometric reasoning, and standards-aware judgment.
  • MechVL is established as a domain-specialized baseline for mechanical drawing understanding.

Impact Statement

The work advances AI understanding of engineering drawings through the MechVQA benchmark and MechVL model, enabling systematic evaluation and development for complex mechanical diagrams. It highlights productivity, error-reduction, and accessibility benefits while requiring human oversight because misuse can cause design flaws or safety-critical failures.

  • Research contribution: MechVQA and MechVL advance AI understanding and reasoning for complex engineering drawings, enabling systematic model evaluation and development.MechVL is enhanced through supervised fine-tuning and DAPO-based reinforcement learning.
  • Potential benefits: Automating standardized-symbol and multi-view-projection interpretation can streamline design reviews and inspection workflows.The stated benefit is improved industrial productivity in engineering workflows.
  • Potential benefits: MechVL can assist engineers in crosschecking dimension chains and geometric tolerances, helping mitigate oversights in high-density technical documents.This benefit concerns error reduction during interpretation of complex technical documents.
  • Risks and safeguards: Because blind trust in automated interpretations could cause design flaws or safety-critical failures, MechVL should support rather than replace certified professional judgment.The work recommends rigorous human oversight and verification, especially because mechanical design can dictate structural integrity.

A. Limitations … B.2. Effect of Expert Verification

The paper identifies limitations in MechVQA’s source coverage, 2D focus, visual-processing dependence, contamination control, validation, and release constraints. It also shows that expert verification materially improves mechanically meaningful metadata, especially view information and technical requirements, while part-category labels remain comparatively stable.

  • A. Limitations: MechVQA may not capture the variability of factory drawings, legacy blueprints, or company-specific drafting conventions because it uses public educational and professional sources.The source materials include textbooks, handbooks, and design platforms rather than proprietary industrial archives.
  • A. Limitations: The benchmark focuses on drawing-grounded understanding of 2D mechanical drawings rather than full 3D CAD reconstruction or engineering-file generation.It includes multi-view reasoning and isometric-view questions but does not target STEP or IGES generation.
  • A. Limitations: Performance may degrade on drawings with extreme annotation clutter, poor scan quality, or visually ambiguous regions because construction relies partly on OCR, metadata extraction, and expert verification.The pipeline applies multi-stage validation, semantic voting, and expert audit, but visual-processing dependence remains a limitation.
  • A. Limitations: Absolute benchmark contamination cannot be ruled out despite drawing-level splits and similarity-aware allocation, because the drawings come from public sources and models train on broad web-scale corpora.The stated mitigation specifically targets benchmark-internal leakage.
  • A. Limitations: Human-expert upper bounds and formal interannotator agreement statistics are not yet reported, although annotation follows a written handbook and structured workflow.Future releases are intended to quantify benchmark difficulty, annotation consistency, and the gap between models and domain experts.
  • A. Limitations: Release of MechVQA will follow the redistribution permissions of its underlying public materials, with annotations and related metadata released when original drawings cannot be redistributed.Permitted supplementary releases may include split information, prompts, and source references.
  • B.1. Example of Mechanical Drawings and Metadata: The MechVQA details section illustrates mechanical drawings and their metadata through Figure 5, Table 4, and Figure 6.Figure 5 presents an example drawing, Table 4 summarizes the metadata schema, and Figure 6 gives a concrete extracted-metadata example.
  • B.2. Effect of Expert Verification: Expert verification substantially changes view counting, view-type classification, and technical-requirement metadata, while part-category labels change in less than 1% of audited cases.The comparison uses model-extracted metadata before human checking versus final expert-corrected metadata on audited typical and newstandard groups, excluding bookkeeping-style schema migration fields.

B.3. Task Taxonomy and Difficulty Level Definitions · B.4. Generation Prompts for MechVQA · B.5. Validation and Fixing Prompt for MechVQA

The appendix defines MechVQA’s task taxonomy and difficulty levels, specifies grounded procedures for generating recognition, location, and calculation questions, and establishes validation and answer-merging constraints. These procedures emphasize explicit visual evidence, answerability, and strict language consistency.

  • B.3. Task Taxonomy and Difficulty Level Definitions: MechVQA defines Recognition as extracting explicit textual, symbolic, dimensional, and spatial information from engineering drawings.Its taxonomy and difficulty-level definitions are presented in Figures 7 and 8.
  • B.3. Task Taxonomy and Difficulty Level Definitions: The Simple difficulty level covers direct recognition and information extraction without reasoning, computation, or domain knowledge.Typical cases involve reading one explicit item, clear dimensions or symbols, naming, localization, or counting.
  • B.4. Generation Prompts for MechVQA: Question-generation prompts require analysis of drawing structure and generation of professional questions strictly grounded in graphical content.They prohibit hallucinated features, disregard surrounding textual descriptions, identify referenced sub-figures, and prioritize quality and answerability.
  • B.4. Generation Prompts for MechVQA: Dimension prompts identify up to five explicitly annotated locations, recording each dimension’s value, feature, and view while favoring diverse dimension types.If fewer than five valid annotations exist, the prompts use only those that can be reliably identified.
  • B.4. Generation Prompts for MechVQA: Annotation and location prompts cover existing engineering annotations, views, parts, annotation positions, and cross-view correspondences with explicit target locations.Questions are generated only for categories and elements present in the drawing, and should remain clear, correct, and answerable.
  • B.4. Generation Prompts for MechVQA: Calculation prompts target unlabeled dimensions computable solely by addition or subtraction from clearly annotated dimensions.They exclude trigonometry and Pythagorean reasoning and require explicit start and end points, view, and calculation basis.
  • B.5. Validation and Fixing Prompt for MechVQA: Validation checks whether each question is consistent with and answerable from the actual drawing, while answer merging produces one refined response from 2–3 candidates.The final answer must use only the question’s specified Chinese or English language, translating non-essential foreign words and technical terms appropriately.

B.6. Dataset Split … C.2. Reinforcement Learning

The dataset uses drawing-level splits with distributionally consistent coverage, while training proceeds from supervised fine-tuning to reinforcement learning with shared infrastructure and algorithm-specific configurations. Across RL algorithms, training produces longer responses alongside higher validation accuracy.

  • B.6. Dataset Split: 8:1:1 drawing-level splits keep all QA pairs from each drawing in one partition, with t-SNE showing similar train, validation, and test coverage.QA-pair counts therefore follow drawing allocation rather than independent question sampling.
  • C. Training Details: Training used eight NVIDIA H800 GPUs, with supervised fine-tuning followed by reinforcement learning.The pipeline was organized as multiple successive training stages.
  • C.1. Supervised Fine-Tuning: SFT used cross-entropy loss on high-quality question–answer pairs to provide a stable initialization for later RL optimization.All model parameters were fine-tuned for three epochs with AdamW, learning rate 1.0 × 10−5, weight decay 0.01, and global batch size 64.
  • C.1. Supervised Fine-Tuning: SFT preprocessing preserved image aspect ratio, constrained the longer edge to 1,024 pixels, capped total pixels at 262,144, and limited sequences to 4,096 tokens.Inputs concatenated a system prompt, the question with cropped drawing context, and the ground-truth answer.
  • C.1. Supervised Fine-Tuning: SFT optimization used a cosine schedule with warmup ratio 0.1 and DeepSpeed ZeRO-3 for memory efficiency.These settings accompanied full-parameter fine-tuning with AdamW.
  • C.2. Reinforcement Learning: RL compared GRPO, GSPO, and DAPO after SFT under a unified data-processing pipeline, optimizer configuration, and distributed infrastructure.All RL experiments used AdamW with base learning rate 1.0 × 10−6, global batch size 128, and a linear schedule, with FSDP CPU offloading.
  • C.2. Reinforcement Learning: GRPO sampled 10 outputs per input and used β = 0.01, ϵ = 0.2, and one training epoch unless otherwise specified.GSPO disabled KL penalties, averaged at sequence level, and used clipping between 3×10−4 and 4×10−4; DAPO used [0.20, 0.28] with online filtering.
  • C.2. Reinforcement Learning: As training progresses under different RL algorithms, the model produces longer responses and achieves higher validation accuracy.Figure 17 summarizes these validation-accuracy and response-length trends.

C.3. Reward-Weight Ablation

The reward-weight ablation fixes the format reward weight at 0.10 and finds that weighting accuracy, quality, and format as 0.60/0.30/0.10 yields the best overall voted score.

  • C.3. Reward-Weight Ablation: 0.60/0.30/0.10 accuracy/quality/format weighting achieves the best overall voted score, with format reward weight fixed at 0.10.The ablation evaluates reward weights in targeted RL.

C.4. Automatic Evaluation Protocol

The protocol extracts each model’s final answer, evaluates it independently with three LLM judges using only textual inputs, and aggregates validated scores conservatively. Scores are clipped, rounded, and combined by majority frequency, with explicit fallbacks for malformed outputs, missing answers, and failed judge calls.

  • Answer extraction: Final answers are extracted from <answer>, after </think>, or the full response when those tags are absent.The extraction uses the content inside the model’s <answer> tag when present; otherwise, it uses text after </think> or the full response.
  • Judge-based evaluation: Three LLM judges—GPT-OSS-120B, DeepSeek-V3.2, and Kimi-k2—evaluate each answer independently at temperature 0.1.Judges receive only the question, ground-truth answer, and model answer, excluding the original image and target model identity.
  • Score aggregation: Judge outputs are parsed from the first JSON object, clipped to [0, 1], rounded to one decimal place, and aggregated by selecting the most frequent score.If JSON parsing fails, the evaluator conservatively parses explicit 0/1 strings when possible; otherwise it assigns 0.
  • Failure handling: Answers with no model response receive 0, while failed judge calls trigger the protocol’s fallback handling.The passage specifies a score of 0 for no-answer outputs and describes fallback handling for judge-call failures.
Loading 2605.30794v2…