PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Assembly Instruction Generation from CAD Models Using Grounded Large Language Models: A Human-in-the-Loop Framework

Aaron Dsouza, Mohammed Azeez Khan, Ashutosh Mishra, Arshaan Khan, Neha K. Nair, Amar Kumar Behera

arXiv:2610.11896v1cs.LG

TL;DR

Assembly documentation is still commonly authored by interpreting CAD by hand, while related research has treated CAD interpretation, sequencing, instruction writing, and oversight separately. This paper maps STEP assemblies to a typed ProductGraph and combines deterministic ordering, grounded step language, checks, visuals, and human approval. On a six-part reference assembly, the prototype preserves the reported order and carries quantity and torque into an exported page; broader validation remains open.

  • Problem

    Assembly manuals are commonly authored by interpreting CAD by hand, and research on CAD interpretation, sequencing, instruction writing, and oversight has largely proceeded separately.

    Source: §1 · pp. 1–2

  • Method

    The framework maps STEP assembly data to a ProductGraph, orders steps deterministically, generates language from selected graph context, and uses checks and human review before export.

    Source: §1 · p. 2, §4.5 · p. 9

  • Results

    On a six-part reference assembly, the prototype preserves the reported order and carries quantity and torque into an exported page.

    Source: §10 · p. 15

  • Takeaways & Limitations

    The case study supports the feasibility of an architecture separating engineering state, deterministic reasoning, language realization, verification, and human release.

    Source: §7 · p. 12, §10 · p. 15

    The interpreter does not yet extract kinematic mates from STEP, and geometric reasoning remains an open research problem.

    Source: §8.1 · p. 13, §8 · p. 13

Abstract

from arXiv · show

Assembly documentation is a downstream manufacturing artifact that is still usually authored by interpreting CAD models by hand. Structured product data and large language models are both available, yet studies of CAD interpretation, assembly sequence planning, instruction writing, and human oversight have largely proceeded separately. This paper formulates CAD-grounded assembly instruction generation: the production of natural-language assembly procedures constrained by structured engineering information extracted from CAD models. The proposed framework maps a STEP assembly to a typed ProductGraph intermediate representation, derives a precedence order by deterministic topological sorting, realizes each step as language conditioned only on selected graph context, attaches per-step visual documentation, and applies rule-based and model-assisted checks. PDF export remains disabled until a human reviewer resolves every quality flag. The case study establishes endto-end feasibility on a built-in six-part reference assembly: the pipeline preserves a reported assembly order and carries quantity, material, and torque into an exported manual page. Generalization and geometric validation remain open empirical questions. The contribution is an architecture that separates engineering state, deterministic reasoning, grounded language realization, verification, and human release.

1 Introduction

Assembly manuals remain costly to author and can drift from revised CAD, while related work has treated CAD interpretation, sequencing, instruction writing, and oversight as neighboring problems. This paper proposes a STEP-to-documentation architecture that combines structured engineering data, deterministic ordering, grounded language generation, verification, and human approval.

1 Introduction

  • Assembly documentation is costly to author, can be inconsistent across product families, and can detach from revised CAD models.Digital-thread work argues downstream activities should reuse engineering models rather than reinterpret drawings.Source: §1 · p. 1
  • Prior research separately addresses CAD-based sequencing, language-model instruction generation, CAD property extraction and geometry synthesis, and human oversight.Source: §1 · pp. 1–2
  • The research question is how CAD-extracted structured information can ground assembly instructions while deterministic engineering reasoning and human accountability remain in the loop.Unconstrained generation can state unsupported part names, quantities, torques, or safety conditions.Source: §1 · p. 2
  • The framework maps STEP information to a typed ProductGraph, derives an order deterministically, realizes steps from graph context, and adds visuals, checks, and human release.Its verification architecture combines deterministic rules, model-assisted review, and mandatory human approval before export.Source: §1 · p. 2
  • The contributions include a hybrid deterministic-sequence and grounded-language architecture, plus an end-to-end proof of concept on a six-part reference assembly.Topological sorting is the ordering mechanism, and Llama 3.3 70B is the case-study language model.Source: §1 · p. 2

2 Related Work

Related work covers product representations, assembly planning, instruction generation, CAD workflows, grounded generation, and human oversight. The paper identifies an integrative gap: these functions have not been joined into one CAD-grounded documentation workflow.

2.1 Model-Based Product and Assembly Representation

  • Product and assembly representations support downstream reuse, while earlier work modeled liaison, precedence, and alternative plans; this paper's ProductGraph carries engineering and documentation state into generation.STEP supports product-structure exchange.Source: §2.1 · pp. 2–3

2.2 CAD-Based Assembly Sequence Planning

  • Prior sequence-planning work addresses geometric, mechanical, and precedence constraints, whereas this framework computes one topological order from stored constraint records.Its case-study order is unique for the reference assembly's stored constraints; geometric interference and path planning remain complementary validators.Source: §2.2 · p. 3

2.3 LLM-Based Assembly Instruction Generation

  • Assembly-instruction studies use process knowledge, predetermined-time analyses, designs, retrieval, or demonstrations, while this workflow starts from a STEP-derived ProductGraph and keeps ordering outside the language model.Source: §2.3 · p. 3

2.4 LLMs for CAD and Engineering Workflows

  • CAD-related language-model work extracts structured properties or synthesizes geometry from prompts, whereas this framework consumes an existing STEP assembly and emits documentation.Source: §2.4 · p. 4

2.5 Grounded and Verifiable Generation

  • The generation calls use a ProductGraph step slice, with rules requiring exact part names, quantities, and torque and prohibiting unflagged safety advice.The prototype inspects the exported page against that contract.Source: §2.5 · p. 4

2.6 Human-in-the-Loop Manufacturing AI

  • Human-oversight research emphasizes verification because industrial model output is not self-certifying; here, reviewers can edit, reorder, resolve flags, and add flags before PDF export.Source: §2.6 · p. 4

2.7 Research Gap

  • The identified gap is integration: this workflow connects STEP-derived structure, deterministic ordering, grounded generation, visuals, automated checks, and a human release gate.Table 1 compares the functions reported by related studies; its dashes mean a function was not reported, not that it is impossible.Source: §2.7 · pp. 4–5, Table 1 · p. 5

3 Problem Formulation

The formulation represents a CAD-derived assembly and the documentation artifact as structured objects, then factors the pipeline into interpretable stages. It specifies grounding and ordering failure conditions that the prototype records.

3 Problem Formulation

  • The input assembly S is represented by parts, constraint records, assembly metadata, and retained geometric information.In the prototype, geometry consists of axis-aligned bounding boxes and assembly-tree parent–child links; kinematic mates remain an open input.Source: §3 · p. 5
  • The documentation artifact D contains a bill of materials, assembly order, instructions, per-step visuals, and quality and safety annotations.Source: §3 · p. 5
  • The pipeline composes CAD interpretation, precedence ordering, graph-slice language realization, visual attachment, quality checks, and human review.The human review stage can change text, order, and flags; export is defined only after required flags are resolved.Source: §3 · p. 5
  • A directed cycle prevents any order from satisfying every constraint, while a sentence asserting facts absent from its step slice or flags violates the grounding contract.The prototype logs cycles as inconsistencies and grounds sentences against their step slices.Source: §3 · p. 6

4 Proposed Framework

The framework separates engineering state, deterministic ordering, grounded language realization, and human-controlled release around a shared ProductGraph. It interprets CAD data, derives an assembly order from stored constraints, and records ambiguity or inconsistency for review.

4.1 Overall Architecture

  • The architecture keeps product state in a shared ProductGraph while ordering, language realization, verification, and release remain distinct stages.Diagram and automated-review failures are logged and flagged; failure to interpret CAD input or produce an ordering aborts the run.Source: §4.1 · p. 6, Figure 1 · p. 6

4.2 ProductGraph Intermediate Representation

  • The typed ProductGraph records part and constraint fields alongside each step’s generated sentence, diagram path, ambiguity bit, and quality flags.It serves as an engineering-state contract, grounding boundary, traceability layer, and modular interface between CAD interpretation and language generation.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 CAD Interpretation

  • When the geometric stack is available, the interpreter walks the STEP shape tree; the reported case instead uses a built-in six-part reference graph when that stack is unavailable.The interpreter records product labels, bounding boxes, parent–child containment, and repeated names in the bill of materials; reading kinematic mates from STEP remains open.Source: §4.3 · p. 7

4.4 Assembly Sequence Inference

  • Kahn’s topological sort produces a total order from stored constraint edges; nodes simultaneously eligible for selection are marked ambiguous for review.The implemented edge mapping orients parts for mate, align, thread-mate, and coincident links, while ordering-only edges are withheld from the writing context.Source: §4.4 · p. 8
  • A directed cycle means the stored constraints cannot all be satisfied by a total order, so the breadth-first fallback is diagnostic rather than a repaired sequence.The inconsistency is recorded, and deterministic ordering remains inspectable without asking the language model to choose which part comes first.Source: §4.4 · p. 8

4.5 Grounded Instruction Generation

Each step is realized from a constrained ProductGraph slice, then illustrated and checked by automated rules and model-assisted review. Human review can edit the documentation, and required flags must be resolved before PDF export.

4.5 Grounded Instruction Generation

  • Each step receives one model call with its step index, participating part fields, mating constraints, and stored safety warnings, under rules requiring a one-sentence imperative and forbidding unsupported facts.The rules require exact torque and quantity when present, prohibit unflagged safety statements, and preserve part names as given.Source: §4.5 · p. 9
  • The reference fastener instruction carries quantity 4 and torque 2.5 Nm into the exported page, while excluding tools, tolerances, and warnings absent from its context.Responses are cached by a SHA-256 fingerprint of the step context for reproducibility of stored model output on repeated runs.Source: §4.5 · p. 9
  • The prototype instantiates language realization with Llama 3.3 70B through the Groq API, while allowing another model that accepts the same slice and prohibitions.The case study uses this single model instantiation.Source: §4.5 · p. 9

4.6 Visual Documentation

  • Per-step visuals use a Blender render when available or a Pillow schematic fallback, which highlights the active part and can show warning or torque cues.The case-study manual page uses the Pillow schematic to communicate step identity and the active part.Source: §4.6 · p. 9

4.7 Automated QA

  • Deterministic QA applies lexical part-name patterns to flag missing fastener torque and append electrical-isolation, sharp-edge, or pressure warnings.A trigger substring in a part name activates its associated warning.Source: §4.7 · p. 9
  • Model-assisted QA reviews each sentence for missing safety information, ambiguous language, missing part references, and inconsistent quantities; both QA tiers feed the human gate.The model receives the sentence, part names, and existing warnings and returns a structured JSON review.Source: §4.7 · p. 9

4.8 Human-in-the-Loop Verification

  • Reviewers may edit sentences, resolve or add flags, and reorder steps; required flags must be resolved before PDF export is enabled.The edited order replaces the inferred order at export, and the bill of materials remains available as a table.Source: §4.8 · p. 9, Figure 4 · p. 10

5 Implementation

The proof-of-concept is implemented as a sequential ProductGraph pipeline with a web review client. Its service stores reviewable jobs and exports a multi-page PDF.

5 Implementation

  • The implementation is a sequential pipeline around the ProductGraph with a web review client; its component choices instantiate the framework rather than define the research claims.Table 3 maps framework stages to proof-of-concept components.Source: §5 · p. 10, Table 3 · p. 10
  • Each ProductGraph is stored as JSON so the review client can reload a job and present editable steps, diagram thumbnails, flags, and the bill of materials.The client polls until the pipeline finishes.Source: §5 · p. 10
  • The export endpoint fills an HTML template and prints a multi-page PDF with the bill of materials, steps, diagrams, safety notes, and document metadata.The service is packaged with Docker Compose.Source: §5 · p. 10

6 Proof-of-Concept Case Study

The proof of concept runs the framework on a six-part reference assembly, producing an ordered, grounded manual with quality review and human-gated export.

6 Proof-of-Concept Case Study

  • The case study uses a built-in reference assembly with six parts, eight constraints, and one torque-specified fastener constraint.Parsing uses the reference assembly, and diagrams use the Pillow fallback.Source: §6 · p. 10

6.1 End-to-End Pipeline Execution

  • The exported page carries step instructions, motor warnings, a 2.5 Nm fastener callout, and base-plate quantity and material.The run used six parts, eight constraints, and the Pillow schematic fallback.Source: §6 · p. 10, Figure 5 · p. 11, §6.1 · p. 11

6.2 Assembly Sequence Inference

  • Kahn’s algorithm returns the unique stored-constraint order: Base Plate → Motor Bracket → DC Motor → M4 Hex Bolt (×4) → Shaft Coupler → Cover Plate.The precedence graph is acyclic and has one ready node at each step.Source: §6.2 · p. 11

6.3 Instruction Grounding

  • Inspected instructions preserve reference part names and introduce no foreign components; the base plate also carries its recorded material and quantity.The motor bracket, DC motor, and four M4 hex bolts are named in their respective steps.Source: §6.3 · p. 11

6.4 Quantity and Torque Preservation

  • The fastener instruction preserves quantity 4 and torque 2.5 Nm, while the base-plate instruction states quantity one and its stored material.The fastener’s torque is also shown as a separate callout on the exported page.Source: §6.1 · p. 11, §6.4 · p. 12

6.5 QA Behavior

  • The electrical-name rule adds a warning to both the DC motor and motor bracket, although the bracket match is lexical rather than a verified electrical role.The model-assisted pass added no further flags on this run; the human gate remains in the release path.Source: §6.5 · p. 12

6.6 Human Review Workflow

  • Reviewers can edit instructions, reorder steps, resolve flags, and add human-authored flags; export stays disabled until required flags are resolved.Figure 5 is an export produced after that condition was met.Source: §6.6 · p. 12

6.7 Pipeline Latency

  • End-to-end execution took on the order of half a minute on one laptop, with schematic rendering and twelve model calls dominating the time.Parsing and ordering were negligible for this graph; reported ranges reflect run-to-run variation, not a confidence interval.Source: §6.7 · p. 12, Table 4 · p. 12

7 Discussion

The discussion presents the ProductGraph as an inspectable boundary between engineering information and language generation, with deterministic ordering and human release. It frames the case study’s scope and identifies broader applications and unresolved empirical questions.

Discussion

  • The case study demonstrates a small electromechanical assembly moving from structured product record to reviewable PDF without the model inventing the order.Product identity, quantity, material, and torque are stored in the ProductGraph; order follows explicit edges and release depends on a person.Source: §7 · p. 12
  • The ProductGraph supports validation, stable interfaces, deterministic processing, sentence-to-field traceability, and independent replacement of CAD interpretation or language generation.Its current geometric payload remains limited to bounding boxes and an assembly tree.Source: §7 · pp. 12–13
  • Deterministic ordering keeps precedence decisions outside the language model; when constraints form a cycle, a fallback listing is logged as not satisfying them.The model may phrase an instruction, but stored edges determine whether it can appear at a given step.Source: §7 · p. 13
  • Prompt rules constrain language realization but do not guarantee compliance, so human verification remains the control for residual risk before release.The paper notes that assembly documentation can specify torque and isolation, making release an accountability decision.Source: §7 · p. 13
  • Unseen CAD, geometric validation, and the electrical status of lexical warnings remain open empirical questions.The discussion also notes that the prototype’s geometric payload is limited to bounding boxes and an assembly tree.Source: §7 · p. 13
  • The same separation of authorized facts, deterministic transformation, grounded realization, and human release is proposed for inspection plans, torque sheets, and service procedures.This transfer is stated as an architectural applicability, not an evaluated result for those documents.Source: §7 · p. 13

8 Limitations and Open Research Problems

The case study’s open research agenda identifies limits of the present pipeline and extensions needed for a broader empirical test. These span richer CAD inputs and geometry, hierarchy, multimodal generation, verification, feedback, benchmarking, and collision checks.

8.1 Mating and kinematic constraint extraction

  • Extracting STEP kinematic mates would populate constraint relations C that the current interpreter leaves supplied with assembly structure.The interpreter currently records labels, bounding boxes, and containment.Source: §8.1 · p. 13

8.2 Geometry-aware spatial reasoning

  • Grounding spatial phrases requires insertion direction, contact, and stable pose in addition to bounding boxes.Bounding boxes currently locate schematics but do not provide these geometric relations.Source: §8.2 · p. 14

8.3 Hierarchical subassembly preservation

  • The pipeline flattens parts despite stored parent identifiers, so a motor subassembly should become a section rather than siblings of the housing.Hierarchical ordering and documentation remain open.Source: §8.3 · p. 14

8.4 Multimodal instruction generation

  • The writer receives JSON; conditioning a vision-language model on both the graph slice and its corresponding step image is a proposed extension.Source: §8.4 · p. 14

8.5 Safety-aware engineering verification

  • Name-pattern rules can flag incidental substrings, so safety verification would need to consult standards, connectivity, tool access, and process limits.The current quality checks direct human attention rather than establish safety.Source: §8.5 · p. 14

8.6 Human feedback and continuous improvement

  • Review edits are not retained for training or retrieval, and learning from them could reinforce reviewer mistakes.Any learning step must retain the ProductGraph as authoritative and follow the language-generation grounding contract.Source: §8.6 · p. 14

8.7 Benchmark datasets

  • A transferable benchmark needs published STEP assemblies, authorized precedence, bills of materials, engineering fields, and expert instructions alongside atomic grounding errors.Source: §8.7 · p. 14

8.8 Simulation and collision-aware validation

  • Collision-aware path-planning or interference methods could validate an order before language realization, distinguishing acyclic constraints from collision-aware feasibility.Source: §8.8 · p. 14

9 Future Directions

The research agenda locates proposed extensions at different pipeline stages, from richer interpretation before ordering to evaluation and model selection after review.

9 Future Directions

  • Richer interpretation would populate constraints and geometry before ordering, while hierarchy changes the shape of the order and manual.Multimodal realization changes language realization without changing the source of facts; simulation can assess orders before sentences are trusted.Source: §9 · p. 14
  • A target-language parameter belongs inside language realization because it changes wording, not the facts held in the graph.Source: §9 · p. 14
  • Learned proposals for missing constraint types belong upstream of ordering and remain hypotheses until accepted by a person or geometric check.Source: §9 · p. 15

10 Conclusion

The paper presents a ProductGraph-centered architecture that separates engineering state, deterministic reasoning, language realization, verification, and human release. Its six-part case study preserves the reported order and carries engineering details into an exported page, while broader validation remains open.

10 Conclusion

  • The ProductGraph serves as the contract linking CAD interpretation, ordering, graph-conditioned language realization, and reviewer approval before export.Source: §10 · p. 15
  • On a six-part reference assembly, the prototype preserves the reported order and carries quantity and torque into an exported page in well under a minute.Source: §10 · p. 15
  • The conclusion identifies mating extraction, geometric reasoning, hierarchy, multimodal grounding, safety verification, learning from edits, public benchmarks, and collision checks as open problems.Source: §10 · p. 15
Loading 2610.11896v1…