PaperlayerView exact revision on arXiv

Source-linked AI summary

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset

Andrew Goldberg, Kavish Kondap, Tianshuang Qiu, Zehan Ma, Letian Fu, Justin Kerr, Huang Huang, Kaiyuan Chen, Kuan Fang, Ken Goldberg

arXiv:2409.17126v1cs.ROcs.AIcs.LG

TL;DR

GDfRA asks whether a system can turn a natural-language prompt and available components into a recognizable structure that a robot can assemble. Blox-Net combines VLM design, simulation-based refinement, and physical robot assembly; its designs achieved 63.5% Top-1 recognizability and repeated successful assembly in reported trials.

  • Problem

    GDfRA requires designing structures from a prompt and available components that both resemble the requested object and can be assembled by a robot.

    Source: §III · p. 3

  • Method

    Blox-Net uses a VLM to generate designs, simulation and perturbation analysis to refine constructability, and a physical robot to assemble the selected design.

    Source: §I · p. 2, §IV · p. 3

  • Results

    GPT-4o evaluation gave Blox-Net designs a Top-1 accuracy of 63.5% with N = 5 labels, and five designs were assembled 10 consecutive times by a robot.

    Source: §VI · p. 6, §VII · p. 6

  • Takeaways & Limitations

    The reported experiments demonstrate a pipeline that connects prompted designs to robot-executable assemblies, including repeated physical assembly trials.

    Source: §VII · p. 6

    Blox-Net is limited to non-deformable cuboid and cylinder blocks and a suction gripper, and reset sometimes requires human intervention.

    Source: §VII · p. 6

Abstract

from arXiv · show

Generative AI systems have shown impressive capabilities in creating text, code, and images. Inspired by the rich history of research in industrial ''Design for Assembly'', we introduce a novel problem: Generative Design-for-Robot-Assembly (GDfRA). The task is to generate an assembly based on a natural language prompt (e.g., ''giraffe'') and an image of available physical components, such as 3D-printed blocks. The output is an assembly, a spatial arrangement of these components, and instructions for a robot to build this assembly. The output must 1) resemble the requested object and 2) be reliably assembled by a 6 DoF robot arm with a suction gripper. We then present Blox-Net, a GDfRA system that combines generative vision language models with well-established methods in computer vision, simulation, perturbation analysis, motion planning, and physical robot experimentation to solve a class of GDfRA problems with minimal human supervision. Blox-Net achieved a Top-1 accuracy of 63.5% in the ''recognizability'' of its designed assemblies (eg, resembling giraffe as judged by a VLM). These designs, after automated perturbation redesign, were reliably assembled by a robot, achieving near-perfect success across 10 consecutive assembly iterations with human intervention only during reset prior to assembly. Surprisingly, this entire design process from textual word (''giraffe'') to reliable physical assembly is performed with zero human intervention.

I. INTRODUCTION

Existing DfRA systems retain human designers, and robot-assembly reliability is difficult to model accurately in simulation. Blox-Net combines generative design, simulation-based refinement, and physical robot evaluation for a class of GDfRA problems.

I. INTRODUCTION

  • DfA’s history includes interchangeable parts for human assembly lines, while later DfRA considered robotic-system error tolerances and joint product-system feasibility.CAD/CAM tools supported human visualization and evaluation using finite-element methods and perturbation analysis.Source: §I · p. 1
  • All existing DfRA systems require human designers, while uncertainty in perception, control, and physics complicates reliable robot assembly and is not fully captured by simulation.Simulations can struggle with minute deformations and collisions during grasping, motivating physical assembly trials for evaluation.Source: §I · p. 1
  • Blox-Net uses iterative VLM prompting to design an assembly approximating a requested object, then applies simulation and perturbation analysis to revise it for constructability.A camera-equipped physical robot uses computer vision, motion planning, and control to construct the assembly and evaluate reliability.Source: §I · p. 2
  • Blox-Net formulates GDfRA and combines GPT-4o prompting, physics simulation, and motion planning to generate assemblies from available 3D-printed blocks.Experiments assess whether the assemblies resemble requested objects, remain stable during construction, and can be reliably assembled by a six-axis robot arm.Source: §I · p. 2

II. RELATED WORK

Prior DfRA approaches developed tools for evaluating designs and assembly tolerances, but still require extensive human input. Blox-Net addresses both design and execution of robot assembly.

A. Design for Robot Assembly

  • DfRA extended DfA principles to account for robotic systems’ capabilities and limitations, with CAD/CAM, finite-element methods, and perturbation analysis supporting design evaluation.Earlier foundations include Boothroyd and Dewhurst’s DfA work and Hitachi’s Assemblability Evaluation Method.Source: A. Design for Robot Assembly · p. 2
  • Existing DfRA tools support visualization and analysis of tolerances, stresses, and forces, but require extensive human input and face uncertainty in perception, control, and physics.Simulation struggles to capture some grasping-related deformations and collisions, motivating iterative real-world testing and redesign.Source: A. Design for Robot Assembly · p. 2
  • Unlike approaches that focus on assembly sequences for fixed designs, Blox-Net addresses both design and execution to create physically feasible designs for robotic assembly.The related work describes LLM uses in task planning, robot-code generation, engineering-document understanding, and layout or CAD generation.Source: A. Design for Robot Assembly · p. 2

B. Text-to-Shape Generation

Prior work generates or reconstructs semantic 3D shapes in several representations, including from text. Blox-Net instead plans structures from available physical blocks and reports the time needed to generate and select candidates.

B. Text-to-Shape Generation

  • Semantic 3D shape generation has been studied using voxel maps, meshes, point clouds, signed distance functions, and implicit representations.Other approaches reconstruct shapes from images or generate semantically meaningful shapes from natural-language instructions.Source: B. Text-to-Shape Generation · p. 2
  • A set of 10 Blox-Net designs takes 81 seconds to generate, and selecting the best design takes an additional 60 seconds.The figure describes diverse candidate designs and structural designs adhering to input constraints.Source: Figure 3 · p. 3
  • Blox-Net generates 3D shapes by prompting an LLM and planning block assembly from the available physical building blocks, rather than generating unconstrained meshes.Prior methods include text-conditioned semantic shape generation and reconstruction from one or more images.Source: B. Text-to-Shape Generation · p. 3

C. Robot Task Planning with Foundation Models

Blox-Net uses a VLM to plan semantically meaningful, physically feasible block poses, then assembles the selected design with a robot. The problem requires both resemblance to a text prompt and robot constructibility.

C. Robot Task Planning with Foundation Models

  • Blox-Net prompts a VLM to determine block poses for semantically meaningful and physically feasible structures, then uses motion planning and force-feedback control for assembly.This differs from foundation-model approaches focused on motion or waypoint planning.Source: C. Robot Task Planning with Foundation… · p. 3

III. GDFRA PROBLEM

  • GDfRA takes a word or phrase and an image of available components as input, and seeks a structure that is both recognizable and constructible by a robot.Source: §III · p. 3

IV. METHOD

  • Blox-Net assumes cuboid or cylinder components lie in stable poses within a reachable planar area.Source: §IV · p. 3
  • A VLM generates candidate designs and selects one; a physics simulator iteratively refines it under controlled perturbations, and a camera-equipped robot assembles it using 3D-printed blocks.A tilt plate actuates to reset the blocks to a tray.Source: §IV · p. 3

A. Phase I: VLM Design and Selection

Phase I uses a VLM to turn an object description and available blocks into constrained assembly plans, then selects among simulated candidates for visual fidelity and stability.

A. Phase I: VLM Design and Selection

  • Blox-Net generates structure designs from the target description and available blocks, specifying block locations and orientations subject to the components’ physical constraints.Source: A. Phase I: VLM Design and Selection · p. 3
  • The VLM first elaborates the target prompt, then receives the description, block set, and a request to explain each block’s role.Source: A. Phase I: VLM Design and Selection · p. 3
  • Block rotations are specified by rearranging dimensions, while placement uses (x, y) coordinates and blocks are dropped in sequence.The system requests rotation, position, and color; restricting coordinates to x and y simplifies the action space and avoids blocks being placed inside one another.Source: A. Phase I: VLM Design and Selection · pp. 3–4
  • For planned x- or y-axis rotations, the robot places a block in a 90-degree bracket and regr asps it from another face to rotate it 90 degrees.Source: Figure 4 · p. 4
  • Each block is simulated as it is dropped onto the structure; instability feedback prompts VLM corrections until stability or a two-iteration limit.Feedback identifies the moved block and direction and includes two orthographic views highlighting the unstable block.Source: A. Phase I: VLM Design and Selection · p. 4
  • The pipeline generates 10 candidates, rates their visual resemblance, and chooses among top-rated stable designs through repeated head-to-head comparisons.Source: A. Phase I: VLM Design and Selection · p. 4

B. Phase II: Perturbation-Based Redesign

Phase II adjusts block positions through simulation-based perturbation checks to address collisions and instability while preserving the design’s core characteristics.

B. Phase II: Perturbation-Based Redesign

C. Phase III: Robot Assembly and Evaluation

Phase III perceives the blocks, reorients them when needed, and uses force feedback to assemble the planned design; an automatic tilt-plate reset prepares subsequent trials.

C. Phase III: Robot Assembly and Evaluation

V. EXPERIMENTS

Experiments assess semantic recognizability across 200 designs and physical constructability on a representative subset, with additional robot tests of perturbation redesign.

V. EXPERIMENTS

  • The evaluation measures prompt alignment and real-robot constructability; it tests recognizability on 200 designs, constructability on 11, and perturbation redesign on 5.Source: §V · pp. 4–5
  • Table I reports GPT-4o recognizability metrics averaged across all 200 objects, while Fig. 5 pairs robot-assembled designs with simulation renderings.Source: Table I · p. 5, Figure 5 · p. 5

A. Semantic Recognizability

  • For recognizability, GPT-4o ranks the correct object label against randomized distractors; the study reports Top-1 accuracy and average rankings for N=5, 10, 15, and 20.Source: A. Semantic Recognizability · p. 5

B. Constructability

The experiments measure robot assembly across repeated trials and isolate the effect of perturbation redesign under fully autonomous assembly.

B. Constructability

  • Across six diverse designs, each was tested for 10 robot trials, recording correct placement and full-structure completion.The experiments also track the average share of blocks needing intervention during reset.Source: B. Constructability · p. 5
  • Human intervention occurred only during resetting; assembly itself was fully autonomous, with specified recovery repositioning after certain failures.Reset interventions are counted per block moved; blocks repositioned after failures are returned to stable poses.Source: B. Constructability · p. 5

C. Perturbation Redesign Ablation

  • Five objects were each assembled 10 times with and without perturbation redesign, starting from singulated blocks in stable poses to isolate redesign’s effect.The ablation reports placement correctness, end-state correctness, and full-completion rate.Source: C. Perturbation Redesign Ablation · p. 5

D. Implementation Details

The paper reports recognizability and robot-assembly outcomes, examines perturbation redesign, and describes the simulator, hardware, and design-generation failures.

D. Implementation Details

  • Blox-Net uses GPT-4o, PyBullet, a UR5e arm, a suction gripper, a stereo camera, and 3D-printed cuboidal and cylindrical blocks.Simulation uses specified density, friction, and gravity settings; instability is measured after 500 steps at 240 Hz.Source: D. Implementation Details · p. 5
  • Perturbation redesign samples points on concentric circles around blocks and limits each block to 10 perturbations.Mask filtering fits a minimum-area bounding rectangle and minimum bounding circle, discarding masks below the stated area threshold.Source: D. Implementation Details · p. 5

VI. RESULTS

  • 63.5% Top-1 accuracy at N = 5; five of six designs achieved perfect completion, all exceeded 98% placement success, and redesign improved full-completion success 4x on average.The robot-assembly table covers six designs over 10 trials; the redesign table covers five objects over 10 iterations.Source: Table II · p. 6, Table III · p. 6, §VI · p. 6
  • Generation failures include unavailable blocks, incorrectly oriented blocks, and designs that do not account for gravity.The examples are a rook chess piece, a bicycle handlebar, and the letter H, respectively.Source: Figure 6 · p. 6
  • Without perturbation redesign, small placement inaccuracies can cause collisions, fallen blocks, and structural collapses.The ablation reports worse performance across its metrics when redesign is omitted.Source: Figure 7 · p. 6, Table III · p. 6

VII. LIMITATIONS AND CONCLUSION

Blox-Net is limited by its block geometries and suction-gripper setup, while the conclusion reports repeated robot assembly of five designs.

VII. LIMITATIONS AND CONCLUSION

  • The system is limited to non-deformable cuboid and cylinder blocks, restricting geometric diversity and its ability to represent complex shapes.The paper notes that many assemblies are not clearly recognizable, likely in part because of these block limitations.Source: §VII · p. 6
  • The system uses only suction grasping, does not account for gripper width or slanted surfaces, and sometimes needs human intervention during reset.The paper says these constraints reduce assembly efficacy.Source: §VII · p. 6
  • Five Blox-Net designs using 3 to 10 blocks were assembled 10 consecutive times by the robot without human intervention.The conclusion describes these designs as scoring high in recognizability.Source: §VII · p. 6
Loading 2409.17126v1…