PaperlayerView exact revision on arXiv

Source-linked AI summary

Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models

Alexander Htet Kyaw, Richa Gupta, Dhruv Shah, Anoop Sinha, Kory Mathewson, Stefanie Pender, Sachin Chitta, Yotto Koga, Faez Ahmed, Lawrence Sass, Randall Davis

arXiv:2511.02162v4cs.ROcs.AIcs.HC

TL;DR

Multi-component physical objects remain difficult to create from text because generated meshes are usually monolithic and lack component-level structure. The paper combines 3D generative AI, VLM-based geometry and function reasoning, and robotic assembly with predefined components and conversational feedback. Users preferred VLM-generated panel assignments 90.6% of the time, versus 59.4% for rule-based and 2.5% for random assignments.

  • Problem

    Text-to-3D systems generally produce monolithic meshes, making text-driven robotic assembly of objects with multiple component types challenging.

    Source: §1 · pp. 1–2

  • Method

    The pipeline uses zero-shot multimodal VLM reasoning to decompose generated meshes into structural and panel components, then supports conversational assignment refinement and robotic assembly.

    Source: §1 · p. 2, §3 · p. 3, §5 · p. 7

  • Results

    90.6% of users agreed with VLM-generated panel assignments, compared with 59.4% for rule-based and 2.5% for random assignment.

    Source: §4 · p. 6

  • Takeaways & Limitations

    VLMs can decompose AI-generated meshes into predefined components using object functionality and geometry without task-specific training, while feedback preserves human control.

    Source: §5 · p. 7

    The implementation is restricted to two predefined component types and evaluation uses common objects with simple prompts.

    Source: §5 · p. 7

Abstract

from arXiv · show

Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.

1 Introduction

The paper targets text-driven robotic assembly of multi-component objects, where monolithic generated meshes lack component-level structure. It uses VLM reasoning and conversational feedback to assign predefined components while preserving human control.

Motivation

  • Multi-component fabrication remains difficult because generated meshes are typically monolithic and component assignments depend on object and part geometry and functionality.Panel placement can also vary with user preferences.Source: §1 · pp. 1–2

Contributions

  • VLM-based multimodal reasoning assigns panel components and decomposes AI-generated meshes into multi-component 3D models.The assignment considers both object functionality and geometry.Source: §1 · p. 2
  • Conversational VLM feedback lets users adjust component assignments without task-specific training.The workflow supports human control over AI-driven robotic assembly.Source: §1 · p. 2
  • The system connects natural-language prompts, 3D generative AI, VLM reasoning, and robotic assembly using predefined components.This creates physical multi-component objects from text prompts.Source: §1 · p. 2

2 Related Work

Prior work has advanced text-to-3D fabrication, part-aware generation, component segmentation, and language-guided assembly. However, these strands do not fully address geometry- and function-aware decomposition for robotic assembly.

3D Generative AI for Prompt-Based Design and Fabrication

  • Text-to-3D systems generate geometries from natural language and support physical fabrication, mainly through 3D printing pipelines.Examples include DreamFusion [19], Get3D [7], Latte3D [23], Sketch2Prototype, and Style2Fab [6].Source: §2 · p. 2

Part Aware Generation of 3D Objects

  • Part-aware generators synthesize structured shapes or part layouts, but primarily target coherent geometry and part-level editing.Examples include PartGen [3], StructureNet [16], and ShapeAssembly [9].Source: §2 · p. 2

Component Segmentation of 3D Objects

  • Supervised, unsupervised, and generative segmentation methods improve geometric decomposition and connection modeling for multi-part reconstruction.These include hierarchical networks, point-based segmentation, semantic graphs, Neural Parts [18], diffusion auto-decoders [17], and joint-aware techniques [13].Source: §2 · p. 2

Vision-Language Models for Prompt Based Robotic Assembly

  • VLMs have supported language grounding, affordance scoring, compositional tasks, and assembly-sequence generation, but prior work does not establish this paper’s component-assignment pipeline.Examples include CLIPort [20], SayCan [1], StructDiffusion [14], and Zhu et al. [28].Source: §2 · p. 3

3 Methods

The pipeline converts a text prompt into a generated mesh, discretizes it into structural elements, uses VLMs to select functional panel regions and map them to reachable faces, then assembles the result robotically. Users can further refine assignments through conversational feedback.

Text-to-Mesh Generation

  • The system begins with a user prompt and generates a mesh using Autodesk’s 3D generative AI model.The resulting mesh is the input to subsequent component decomposition.Source: §3 · p. 3

Mesh Discretization

  • Structural components form the load-bearing frame, while panels provide functional surfaces and are assigned selectively to avoid unnecessary weight and component use.The mesh is voxelized at a grid resolution based on structural-component dimensions.Source: §3 · p. 3

VLM for Function Aware Part Selection

  • The function-aware VLM uses the object description, an axonometric mesh image, and the panel component type to identify the minimal functional parts needing panels.Gemini 2.5 Pro performs this zero-shot multimodal selection.Source: §3 · p. 3

VLM for Geometry Aware Part Selection

  • A geometry-aware VLM maps selected parts to labeled mesh faces after coplanar-face merging and omission of inward or downward faces that may be unreachable.The output is a minimal set of face labels for panel placement.Source: §3 · p. 4

VLM for Human in the Loop Conversational User Alignment

  • Conversational alignment uses the object description, labeled mesh rendering, and user feedback to refine or override initial assignments.This incorporates user preferences into component selection.Source: §3 · p. 4

Robotic Assembly

  • The UR20 robot executes the assembly from coordinate and component-type lists, placing structural and panel parts in bottom-to-top order while preserving connectivity.Component types determine the source from which the gripper retrieves each part.Source: §3 · pp. 4–5

4 Experiments and Results

The experiment compares VLM-based panel assignments with rule-based and random alternatives through a non-exclusive user study and robotic demonstrations. Users preferred VLM assignments, while conversational feedback and end-to-end assembly supported adaptable physical realization.

Experiment Setup

  • Thirty-two participants evaluated panel assignments for five objects across random, rule-based, and VLM approaches, selecting any number of acceptable alternatives or none.This design allowed multiple valid assignments based on user preferences rather than imposing a single ground truth.Source: §4 · pp. 5–6

Results

  • 90.6% of users agreed with VLM-generated panel assignments, versus 59.4% for rule-based and 2.5% for random assignment.The rule-based approach matched VLM performance on predominantly horizontal objects but failed on more complex objects such as chairs, lamps, and trash cans.Source: Figure 5 · p. 5, §4 · p. 6
  • VLM assignments were selected significantly more often than rule-based assignments (χ2 = 38.11) and random assignments (χ2 = 137.11), with all differences surviving Bonferroni correction.The comparison used pairwise McNemar tests under non-exclusive selection.Source: §4 · p. 6, Table 2 · p. 6

Human in the Loop Feedback

  • Participants proposed refinements such as paneling only a chair seat, lampshade, or shelf’s bottom two tiers, showing that acceptable assignments can vary by preference.The feedback demonstrates that VLM outputs can serve as starting points for user-directed divergence.Source: §4 · p. 6

Robotic Assembly

  • The robot executed feasible grasp configurations and assembly sequences for VLM-generated and user-refined outputs without placing inward- or downward-facing panels.Preprocessing omitted potentially unreachable faces from the input image.Source: §4 · p. 6, Figure 6 · p. 7

5 Discussion and Limitations

The discussion attributes the VLM approach’s success to zero-shot multimodal reasoning over object function and geometry, while positioning conversational edits as a bridge between automation and human agency. The system remains constrained by its predefined component library and evaluation scope.

Discussion

  • VLMs decompose AI-generated meshes into predefined components using zero-shot multimodal reasoning grounded in object functionality and geometry.The paper attributes this capability to broad prior knowledge, geometric reasoning, and multimodal pretraining.Source: §5 · p. 7

Limitations

  • The current implementation is restricted to two predefined component types, limiting the diversity of assembly elements despite enabling controlled evaluation and physical assembly.Future extensions include functional components such as hinges and handles and material-specific components such as wood, plastic, or metal.Source: §5 · p. 7, §A.1 · p. 10, Figure 7 · p. 10
  • Evaluation is limited to common objects and simple prompts, leaving complex prompts and unconventional objects for future study.Such cases may require multi-turn human–AI interaction with larger edit distances.Source: §5 · p. 7, Figure 8 · p. 11, Figure 9 · p. 11

Discussion

  • Natural-language feedback lets users refine VLM assignments, accommodating varied preferences and creative liberties in robotic assembly.The pipeline connects generative AI, VLM reasoning, and robotic assembly while preserving human control.Source: §5 · p. 7

B Robotic Assembly Implementation Details

The robotic implementation represents assemblies as typed components with target poses and source locations, then executes pick-and-place operations sequentially. The demonstrations use Autodesk’s internal robotics platform integrated with Fusion 360.

Implementation

Algorithm

  • The assembly algorithm takes component coordinates, component types, and source locations, then places every component at its target using pick-and-place operations.Structural and panel components are distinguished by type and assigned different source locations.Source: B Robotic Assembly Implementation… · p. 14
  • For each component, the robot selects its type-specific source, computes pickup and placement poses, and iterates through the coordinate sequence.The algorithm initializes the robot at a home pose and opens the gripper before processing components.Source: B Robotic Assembly Implementation… · p. 14

C Additional Details on Experiment and Results

The supplementary experiment details describe an IRB-exempt, anonymous survey in which participants evaluated panel placements across AI-generated objects. Instructions permitted multiple or no selections and invited descriptions of alternative assignments.

Study Governance

Survey Instructions

Alternative Assignments

C.1 Derivation of Test Statistics: Pairwise McNemar Tests

The study uses pairwise McNemar tests for non-exclusive method selections, comparing discordant participant choices between methods. All reported comparisons remain statistically significant after Bonferroni correction, with VLM-generated placements preferred over both baselines.

Test statistic

  • McNemar’s test evaluates the null hypothesis that the two discordant cells are equal, b = c.Only discordant cells from the 2 × 2 contingency table are considered.Source: §C.1 · p. 16

Statistical significance

  • All pairwise p-values remain below the Bonferroni-adjusted threshold α = 0.05/3 ≈ 0.0167, and continuity correction does not change any conclusion.The reported analysis includes both uncorrected and continuity-corrected statistics.Source: §C.1 · p. 16

Results

  • VLM-generated placements were preferred over both rule-based and random assignments in non-exclusive participant selections.The analysis compares methods using discordant response pairs rather than forced-choice selections.Source: §C.1 · p. 16

Interpretation

  • A larger χ2 statistic indicates a stronger imbalance between the discordant cells and therefore a stronger preference for one method.The statistic reflects how often participants selected one method but not the other.Source: §C.1 · p. 16

Scope

  • The paper’s introduction lists three concrete contributions that are implemented in the Methods section and validated in Experiments.This supports consistency between the stated contributions and the evaluated system.Source: §1 · p. 17

2. Limitations

The paper identifies limitations in its component library, evaluation scope, and reproducibility resources. It also reports that the work is a system-design and empirical-evaluation study rather than a theoretical-results paper.

Scope limitations

  • The evaluation is limited to a fixed library, simple prompts, and a narrow set of functional components.The paper identifies broader component types and multi-turn interaction as future extensions.Source: §2 · p. 17

Component coverage

  • The paper uses only two component types, restricting the current implementation to structural and panel components.The authors propose adding functional components such as hinges and handles, as well as material-specific components.Source: §2 · p. 17

Reproducibility and scope

  • The paper describes experimental setup and component-assignment logic in detail, but its limitations section constrains the generality of the evaluated setting.The supported evaluation scope remains tied to the reported prompts, objects, and component library.Source: §2 · p. 17, §4 · p. 17

Theoretical scope

  • The paper presents no formal theoretical results or mathematical theorems, focusing instead on system design and empirical evaluation.Consequently, theorem-level assumptions and proofs are not part of the contribution.Source: §3 · p. 17

5. Open access to data and code

The paper does not provide fully open access to all code and data needed for direct reproduction. Key proprietary tools remain closed-source, while prompts and the robotic assembly algorithm are released in the appendix.

Access constraints

  • Autodesk’s internal robotics platform and Project Bernini remain closed-source or proprietary, preventing fully open release of the complete implementation.The paper uses these tools as part of its generative and robotic pipeline.Source: §5 · p. 17

Released materials

  • The appendix provides open access to the VLM prompts and robotic assembly algorithm.The paper also describes alternative tools that can substitute for the proprietary framework components.Source: §5 · p. 17

Reproducibility

  • The paper offers detailed methodology and substitute-tool guidance despite not releasing the complete proprietary codebase.This supports partial methodological reproducibility rather than full open-source reproduction.Source: §5 · p. 17

6. Experimental setting/details

The paper reports a detailed empirical setup spanning VLM prompting, user evaluation, statistical testing, robotic assembly, ethics, and broader-impact considerations. The core VLM component uses zero-shot multimodal reasoning to assign functional components to generated meshes.

Statistical evaluation

  • Pairwise McNemar tests and Bonferroni-corrected p-values were reported for the non-exclusive selection setup.The statistical calculation is described in the appendix and uses discordant participant responses.Source: §7 · p. 18

Computational and robotic setup

  • The VLM experiments used Google’s Gemini 2.5 Pro in a zero-shot setting, while robotic assembly used a Universal Robots UR20 industrial arm.The experiments were run from a standard desktop interfacing with Project Bernini and the robotic system.Source: §8 · p. 18, §16 · p. 19

Broader impacts

  • The discussion identifies accessible physical making and modular construction as potential societal implications while calling for further study of complex prompts and broader impacts.These implications are presented alongside future-impact considerations rather than as established outcomes.Source: §10 · p. 18

User study

  • The user study involved 25 participants evaluating component assignments across five objects under non-exclusive selection.Participants could evaluate multiple methods for the same object.Source: §14 · p. 19

Ethics

  • The study received IRB exemption, obtained informed consent, and collected no sensitive personal data.The reported user study was classified as minimal-risk human-subjects research.Source: §14 · p. 19

VLM method

  • The VLM performs zero-shot multimodal reasoning to assign functional components to AI-generated 3D meshes.The model uses object prompts and geometry-related inputs as part of the methodology.Source: §16 · p. 19

Related papers

Loading 2511.02162v4…