PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Autonomous Generation of Human Assembly Plans

Faustin Arion von Arx, Millicent Schlafly, Mark D. Fuge

arXiv:2610.09781v1cs.ROcs.GR

TL;DR

Assembly planning remains largely manual, and this paper asks how to automate it from component geometries while reducing assembly time. It combines mesh-based physics simulation and DfA costs with multimodal models; DfA-aware sequencing reduces proxy time, and tool selection reaches 88.6% accuracy over 158 parts.

  • Problem

    Assembly planning is still largely manual, motivating the question of whether MLLMs and robotics can generate plans from component geometries and whether DfA optimization reduces assembly time.

    Source: §1 · p. 1

  • Method

    The pipeline uses physics-based disassembly with a DfA cost function for sequencing, while multimodal models support tool labelling, manual generation, and design feedback.

    Source: §6 · p. 15

  • Results

    DfA-aware sequencing reduces robot-arm proxy assembly time, while tool selection reaches 88.6% accuracy over 158 parts.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    The open-source tool is intended for closed-loop product updates by human engineers or AI agents.

    Source: §1 · p. 1

    Sequence-planning runtime grows faster than linearly with assembly size, largely because stability checks simulate all remaining parts in the CPU-only simulator.

    Source: §5 · p. 14

Abstract

from arXiv · show

Turning a CAD design into an assembly plan is still largely done by hand, requiring engineers to reason about geometric feasibility, tool access, stability, and the ergonomics of human assembly. In this work, we encode long-established design for assembly (DfA) principles into a contained, end-to-end approach for generating assembly plans. Our approach takes only a mesh assembly and produces either a step-by-step assembly manual or a structured failure report, requiring no joint metadata, fastener annotations, or additional information. Four major components of a manufacturing plan are addressed autonomously: an assembly tool list, the assembly sequence and subassemblies, an assembly manual, and design feedback for improving assemblability. For determining the sequence plan, we systematically disassemble the object in a physics simulator and apply a cost function that encodes DfA principles. Manual generation, tool labelling, and assembly feedback rely primarily on multimodal large language models. Compared with a baseline that always removes the outermost part first from Tian et al., DfA-aware sequence planning reduces simulated assembly time, measured with a robot-arm assembly-time proxy, by 35% on 136 assemblies of 5 to 30 parts. The correct tool is selected for 88.6% of assembly steps. A vision-language model judge compares the generated manuals against ablated variants, identifying which page elements carry the information a reader needs. The presented approach and open-source code are available for use by engineers or AI agents looking to rapidly accelerate the creation of manufacturing plans for a given product design.

1 Introduction

The paper targets manual assembly planning by building an autonomous pipeline from mesh geometry alone. It combines physics-based sequencing with multimodal models to produce manufacturing-plan components and design feedback.

1 Introduction

  • The approach aims to reduce product-development and assembly time by quantifying assemblability with assumptions about gravity and geometric collisions, while encoding DfA principles in objective terms and MLLMs.The stated intended use includes a closed loop in which engineers or AI agents update product designs.Source: §1 · p. 1
  • The pipeline accepts only part meshes and produces an assembly tool list, sequence and subassemblies, manual, and design feedback, using physics simulation for sequencing and MLLMs for other stages.The input requires no joint metadata or fastener annotations.Source: §1 · p. 1, Figure 1 · p. 2
  • Tool selection achieves 88.6% accuracy over 158 parts, each added in its own assembly step.The method addresses parts with a radially symmetric contact region, such as those requiring a screwdriver or hex key.Source: §1 · p. 2
  • DfA principles incorporated through the objective function reduced estimated assembly time by 35% across 136 assemblies of 5 to 30 parts, compared with sequentially selecting the outermost part [Tian et al. 2024].Source: §1 · p. 3
  • 37% estimated time reduction over the same baseline is reported when two workers assemble detected subassemblies in parallel; splitting alone does not reduce time and costs time with one worker.The parallel result is on par with the linear sequence.Source: §1 · p. 3

2 Related Work

Related work spans sequencing, subassembly identification, DfA, tool access, instruction generation, and multimodal CAD reasoning. The paper positions its approach around mesh-only input and integration of these problems into a pipeline.

2.1 Automatic Assembly Sequencing

  • Physics-based sequencing checks collision-free removals and stability without supplied liaison information, while explicit search supports a reweightable DfA objective where sequence-time training data are unavailable.Prior sequencing work distinguishes feasibility from optimization, and learned policies trade search for training data and a fixed training-time objective.Source: §2.1 · p. 3

2.2 Subassembly Identification

  • The proposed subassembly method reuses blocking information from sequence search to derive an approximate directional blocking matrix at negligible additional cost.Prior approaches include graph partitioning with separate geometric tests and geometric blocking representations such as NDBG and DIG.Source: §2.2 · p. 4

2.3 Design for Assembly and Assembly Time Estimation

  • DfA scores quantify assembly effort using principles empirically validated as predictors of human assembly time, while predetermined motion-time systems require a described motion sequence rather than geometry.The paper's cost terms use DfA principles evaluable from quantities already computed during sequence search, in the absence of labelled assembly-time data.Source: §2.3 · p. 4

2.4 Tool, Fastener, and Accessibility Reasoning

  • Prior tool-access methods assume identified fasteners and tools, whereas this pipeline recovers candidate tool axes from mesh geometry, asks a VLM to choose, and validates accessibility.The pipeline's axis candidates come from convex decomposition; related feature and joint prediction methods rely on B-rep input or supervision.Source: §2.4 · p. 4

2.5 Generating Assembly Instructions

  • Generating assembly instructions has received less attention than parsing existing manuals, and the closest cited CAD-generation system requires assembly-relevant non-geometric information.The adopted page layout follows findings that images and text together convey procedures more effectively than either alone.Source: §2.5 · p. 5

2.6 Multimodal Models for CAD, Design, and Evaluation

  • VLM reasoning over CAD geometry has documented weaknesses in component recognition, rotation, perspective-taking, and occlusion, while tight binary judging is associated with higher human-rater agreement.The pipeline is also framed as a verifier for generated CAD assemblies that are not guaranteed to be assemblable.Source: §2.6 · p. 5

3 Methods

The methods section describes a pipeline that transforms part meshes into assembly sequences, manuals, or design feedback, including preprocessing, part and tool inference, physics-based sequencing, subassembly identification, and VLM generation. It also covers sequencing evaluation scope and constraints, the part-count limit, and multimodal-language-model evaluation of manuals.

3.1 Preprocessing

  • Preprocessing repairs non-watertight meshes, flags excessive part overlaps for feedback, and checks stability by iteratively joining parts that move relative to the assembly.Small interpenetrations are left for the simulator to absorb; the stability check tests three resting poses and allows whole-assembly rotation or falling while detecting parts that move relative to the bulk.Source: §3.1 · p. 5, §3.3.1 · p. 6

3.2.1 Naming parts; 3.2.2 Tool Application.

  • VLMs name parts and infer tool requirements; tool application axes and contact directions are recovered from mesh geometry, while fasteners need no special annotations.For tools with approximately radially symmetric contact regions, convex decomposition supplies candidate axes for VLM selection; fasteners are treated as ordinary single-part steps, and missing fasteners are not inferred.Source: §3.2.2 · p. 6, §3.2.1 · p. 6

3.3 Sequencing

  • The planner searches removal sequences using path, tool-access, and stability checks, then returns the lowest-cost complete sequence found by a beam search.The checks also record parts that must be held; the beam width is 4 and candidate checks are capped at 6,000 per plan.Source: §3.3.2 · p. 6, §3.3.1 · p. 6

3.3.3 Cost Function; 3.3.4 Weight Optimisation.

  • Sequence cost combines DfA principles to approximate human assembly time, with four of five weights tuned against a robot-arm proxy using TPE.The proxy includes part extraction-path length, base travel between steps, and assembly reorientation; it does not capture factors such as precision under tight fits, part weight, or handlability.Source: §3.3.4 · p. 7

3.3.5 Subassembly Detection

  • Subassembly detection reuses disassembly blocking information to search connected part subsets that can translate along an axis-aligned direction without collision.The method builds an approximate blocking matrix, enumerates candidates, and validates top-ranked candidates in simulation; the directional restriction misses subsets requiring oblique or sequential translations.Source: §3.3.5 · pp. 7–9

3.4 Assembly Manual Generation

  • Each manual page combines a rendered action and motion path with VLM-generated text and conditional tool or reorientation callouts.The view is selected from four angles by pixel-wise difference; validation found that removing either text or rendered path lowered conveyed checklist items, while briefing the judge on the layout improved results.Source: §3.4 · p. 9

3.5 Feedback Generation

  • For unbuildable inputs, the VLM explains detected failures; for buildable designs, it suggests actionable changes based on common assembly pitfalls.Failure reports can cover infeasible sequences, unstable subassemblies, inaccessible tools, and excessive overlaps; design feedback is generated per step and may return no suggestion when no specific change is warranted.Source: §3.5.1 · p. 9, §3.5 · p. 9, §3.5.2 · p. 9

3.6 Experimental Evaluation

  • Evaluation uses IKEA assemblies for tool application and the ASAP dataset for timing and runtime, with manual evaluation on ten selected test assemblies plus one qualifying IKEA assembly.The IKEA tool evaluation uses four assemblies and 158 parts; the ASAP dataset contains 2,146 assemblies, and six assemblies are excluded from timing evaluation because they train the sequence-cost weights.Source: §3.6 · p. 10
  • Of 200 test assemblies with up to 30 parts, 136 were planned completely by every method; 47 were excluded because no stable initial pose existed with at most two parts held.Methods used beam width k=4, 6,000 candidate checks, and a 12-hour limit per plan; tool checking was disabled because the assemblies contain almost no fasteners.Source: §3.6 · p. 10
  • The 30-part limit reflects sparse data and growing pose or sequence failures, while runtime grows faster than linearly, averaging 2,560 s per assembly at 21–25 parts.The test split has at most ten assemblies per part count; the authors report that relative gains do not change with assembly size within the evaluated range.Source: §3.6 · p. 10
  • Manuals are evaluated with multimodal language models as zero-shot classifiers or VLM-as-judge graders [Gu et al. 2025].The authors cite studies [Chen et al. 2024; Han et al. 2025; Lee et al. 2024] supporting VLM reliability, particularly for comparative judgments.Source: §3.6 · p. 10

4 Results

The evaluation tests tool application, sequence planning, assembly manuals, and design feedback component by component. DfA-aware sequencing reduces predicted assembly time relative to an outermost-part-first baseline, while the manual and feedback stages are assessed through a VLM judge and representative cases.

4.1.1 Tool Decision.

  • The VLM selects the correct tool for 88.6% of 158 parts, recognizes a needed tool in 94.6% of tool-required parts, and has 97.6% screwdriver and 88.7% hex-key accuracy.The false-negative rate is 5.4%; most errors occur on visually ambiguous parts.Source: §4.1.1 · p. 10, Figure 2 · p. 11

4.1.2 Contact Decision.

  • On a catalogue of 10 screwing tools, the VLM selects both the correct application axis and direction for 5 tools and is internally consistent across repeated runs.The per-tool breakdown and error analysis are in Appendix A.1.Source: §4.1.2 · p. 10

4.2.1 Sequence and subassembly validation.

  • 35% lower predicted assembly time (geometric mean ×0.65, 95% CI [0.61, 0.70]) than the outermost-part-first baseline across 136 assemblies; the optimised planner is faster on 120.Table 2 reports predicted time relative to baseline; Figure 3 breaks the proxy into motion, travel, and reorientation components.Source: §4.2.1 · p. 11, Table 2 · p. 11, Figure 3 · p. 12
  • Hand-set DfA weights reduce predicted time by 24% (×0.76) and hold the fewest parts, 0.16 per step; optimised weights prioritize proxy time while keeping steps feasible.The optimised weights’ advantage over reference weights partly reflects fitting the same assembly-time proxy used for evaluation.Source: §4.2.1 · p. 12
  • Subassembly plans are found for 96 of 136 assemblies; they make one-worker plans 15% slower, while two-worker makespan is ×0.63 versus baseline and statistically similar to the linear plan.Twelve proposed plans fail simulator re-checks and revert to the linear sequence.Source: §4.2.1 · p. 12

4.3 Assembly Manual

  • The judge marks 77 of 127 checklist items as conveyed by the full manual; removing text or the motion path lowers this to 44 or 38, while skipping angle selection leaves the count unchanged.Providing the judge a layout pre-brief raises the count to 104 items.Source: §4.3 · p. 13

4.4 Design for Assembly Feedback

  • Feedback examples identify spring–shaft interference and unstable pivot plates, proposing targeted geometry fixes; suggestions lack downstream validation and paired defect-correction data.The authors describe reviewer-like feedback as necessary but not sufficient evidence for a corrective signal.Source: §4.4 · p. 13

5 Limitations

The pipeline’s limitations arise from simulation cost, restricted tool reasoning, rigid-body physics, and stepwise sequencing. These constraints bound scaling and exclude some tool uses and assembly types.

5 Limitations

  • Sequence-planning runtime grows faster than linearly with assembly size, chiefly because stability checks simulate every remaining part; larger assemblies need cheaper per-candidate simulation.Path finding stays roughly constant because the rest of the assembly can be treated as one fixed body. A faster geometric prefilter could skip some simulations, but must not reject candidates the simulator might accept.Source: §5 · p. 14
  • In the instability example, the pivot plates fall before the slider rod is fitted, illustrating how a small lip on the clevis lever could support them while preserving their pivoting motion.The suggested lip addresses the unstable intermediate assembly state, not merely the completed assembly.Source: Figure 6 · p. 14
  • Tool application is limited to radially symmetric tools, and its candidate-generation heuristic can omit the correct axis; straight-path checks also miss swing clearance needed by tools such as hammers.The contact-axis heuristic is tailored primarily to screwing tools; accounting for tool-use volumes would require per-tool annotation.Source: §5 · p. 14
  • Rigid-body simulation excludes assemblies held together by friction or compliance, including snap-fits that require elastic deflection for insertion.The authors suggest detecting snap-fit features and modeling flagged contacts as compliant, or allowing one-way insertion.Source: §5 · pp. 14–15
  • Stepwise sequencing does not handle assemblies that strictly require several parts to be removed simultaneously; validating candidate simultaneous removals is intractable with the current simulator.The existing subassembly algorithm could be adapted to propose simultaneous removals, but the paper leaves this for future work.Source: §5 · p. 15

6 Conclusions

The paper presents a mesh-only pipeline that generates assembly plans or structured failure reports, combining physics-based sequencing with MLLM-based tool labelling, manuals, and design feedback. It reports lower proxy assembly time and connects the tool to automated design workflows.

6 Conclusions

  • DfA-aware sequencing reduces robot-arm proxy assembly time by 35% on 136 assemblies of 5 to 30 parts; two-worker subassemblies yield a 37% reduction, on par with the linear sequence.Subassembly splitting does not shorten the sequence itself, but allows two workers to build subassemblies in parallel.Source: §6 · p. 15
  • For generated CAD, the pipeline can act as a physical verifier and feedback channel, returning a confirming manual or structured failure report for use in a corrective design loop.The proposed loop places an explicit physical verifier between generation steps.Source: §6 · p. 15
  • The authors describe the pipeline as a lightweight end-to-end tool that returns a feasible assembly plan and instruction manual from finished CAD without manual annotation work.They position it as addressing a design-for-production gap for both human-designed and generated assemblies.Source: §6 · p. 15

7 Ethics and Privacy Statement

Generated manuals should be reviewed by a competent person before use, particularly for load-bearing or otherwise dangerous assemblies.

7 Ethics and Privacy Statement

  • The tool-decision stage fails to identify a required tool on 5.4% of parts that need one, so generated manuals should be reviewed before use, especially for dangerous assemblies.The paper specifically flags load-bearing assemblies as a safety concern if the approach is used uncritically.Source: §7 · p. 15

A.1 Contact-Point and Axis Selection

Across ten screwing tools, the VLM consistently repeated its selection but chose both the correct axis and direction for only half. Errors arose both from incorrect VLM choices and from candidate generation that omitted the correct axis.

A.1 Contact-Point and Axis Selection

  • The VLM selected both the correct application axis and direction on 5 of 10 screwing tools, despite each tool’s ten repeated runs collapsing to a single answer cluster.Each distinct answer was validated against the hand-labelled ground truth once.Source: §A.1 · p. 17
  • Errors split roughly equally between wrong VLM choices despite a correct candidate set and convex-decomposition failures that left no correct axis to choose.Decomposition failures occurred on wrenches, while VLM errors arose on tools with many candidate axes, such as electric drills.Source: §A.1 · p. 17

A.2 Directional Blocking Matrix

The directional blocking matrix is accumulated during disassembly-path search, and its partially observed entries suffice to enumerate candidate subassemblies.

A.2 Directional Blocking Matrix

  • The directional blocking matrix is collected as a by-product of disassembly-path search; in the example, 272 of 792 entries are decided.Source: §A.2 · p. 17
  • Despite being partial, the matrix suffices to enumerate candidate subassemblies without additional simulation.Source: §A.2 · p. 17

A.3 Beam Search Approach

Beam-search quality improves as width increases, while the width trades computation for lower-cost sequences; the evaluation compares its results with exhaustive optima on small assemblies.

A.3 Beam Search Approach

  • The evaluation compares beam-search sequences with exhaustive optima on 26 randomly sampled test-split assemblies, limiting assemblies to 8 parts because larger exhaustive searches grow exponentially more costly.Source: §A.3 · p. 17
  • A larger k trades additional simulation for lower-cost sequences, and k must scale with part count to maintain similar relative performance.Source: §A.3 · p. 17
  • Global average cost falls from 115.5% at k=1 to 104.3% at k=8, relative to the global optimum.Source: §A.3 · p. 17, Figure 9 · p. 18
  • In 15 of 26 assemblies, the global optimum is recovered at k≤n_i, where n_i is the assembly's part count.Source: Figure 9 · p. 18

A.4 Sequence-Planning Runtime

Runtime rises with assembly size in the measured sequence-planning runs, primarily because stability checks simulate all remaining parts together under gravity.

A.4 Sequence-Planning Runtime

  • Of 295 assemblies, 119 lacked a stable initial pose and 29 searches found no complete sequence; these runs were excluded from runtime statistics.The completed runs provide 8 sequences per part count through 17 parts and 4.78 per count on average above 17.Source: §A.4 · pp. 18–19, Figure 10 · p. 18
  • Mean time per part rises from 39.7 s at 5 parts to 107.4 s at 21 to 26 parts.A plan averages 199 s at 5 parts and 2,560 s across 23 completed runs with 21 to 25 parts.Source: Figure 10 · p. 18, §A.4 · p. 19
  • Most of the runtime increase comes from stability checks, which simulate all remaining parts together under gravity in three resting poses.Source: §A.4 · p. 19
  • Path finding stays roughly constant across assembly sizes because the remaining parts are treated as one fixed body, leaving two bodies to simulate at a time.Source: §A.4 · p. 19

A.5 Sequence Planner Pseudocode

The planner first searches for a low-cost feasible linear sequence, then uses its accumulated directional blocking information to detect hierarchical subassemblies.

A.5 Sequence Planner Pseudocode

  • Phase 1 performs parallelized beam search over part removals, retaining the k lowest-cost candidates from each frontier and returning the lowest-cost complete sequence found.Source: §A.5 · pp. 19–20
  • Feasibility checks cover path, tool, and stability, and their observations update the directional blocking matrix as a by-product of path planning.Source: §A.5 · p. 20
  • Phase 2 uses the linear sequence and blocking matrix, together with the simulator, to recursively split segments and return a hierarchical decomposition.A candidate subassembly is recorded when at least one direction has no external blocking.Source: §A.5 · p. 20

A.6 Design Feedback Prompts

Separate VLM prompts provide selective, geometry-specific feedback for buildable steps and plain-language explanations for assemblies that cannot be built. The appendix also shows a six-step generated manual page combining action visuals, instructions, and conditional callouts.

A.6 Design Feedback Prompts

  • Both feedback prompts are passed to the VLM with the structured record of the step or problem and its renders.The two feedback types use separate prompts.Source: §A.6 · p. 20
  • For buildable assemblies, the VLM reviews one step at a time and returns only high-signal, non-trivial feedback when an actionable issue stands out.It can leave the feedback list empty and categorizes applicable points as observations, DFA issues, or recommendations.Source: §A.6 · p. 20
  • The buildable-step prompt considers access and clearance, alignment features, ergonomic strain, ambiguous orientation, and lack of self-retention.It asks the model to consider these pitfalls only when they are actually present.Source: §A.6 · p. 20
  • For unbuildable assemblies, a separate prompt asks for a plain-language explanation of what goes wrong and one simple shape change to fix it.It prioritizes what a person holding the parts would notice, names the part to change, and permits a second option only when clearly worthwhile.Source: §A.6 · p. 20
  • The unbuildable-step prompt was condensed from a longer guideline list after the authors found it gave better results in their runs.It is written for designers who know the product but are not assembly specialists, and favors everyday words over specialist terminology.Source: §A.6 · pp. 20–21
  • Feedback should suggest shape changes, not a different step order, and preserve any pivoting or sliding function required of a part.The prompt also limits claims to what the image and supplied facts show, allowing a general fix when the failure cause is unclear.Source: §A.6 · p. 21

A.7 Example Manual Page

  • Figure 11 shows a generated manual page covering six assembly steps, with each panel pairing an action render and motion marker with a text instruction.Conditional callouts identify the added part and any required reorientation relative to the previous step.Source: §A.7 · p. 21, Figure 11 · p. 22
Loading 2610.09781v1…