PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Demand Robotic Assembly via Differentiable Geometric Part Repair

Millicent Schlafly, Fabio Schaub, Diogo Costa Pais, Luca Lelli, Janne Dvorak, Claire Colmont, Sven Marti, Mark D. Fuge

arXiv:2610.09777v1cs.RO

TL;DR

Robotic assembly of bespoke designs requires months of expert work to reconcile part geometry with physical and hardware constraints. This paper uses a generative agent and a differentiable GAT surrogate to repair geometry, modifying 52/60 unseen inputs and satisfying all cell constraints for 22/60, versus Blox-Net’s 5/60 and 4/60 outcomes. Two UR5e robots physically assembled ten structures, though some builds required human intervention.

  • Problem

    Reconciling bespoke part geometries with robotic constraints such as reachability, collisions, and fastening clearances can require months of expert tuning, with additional site-specific constraints across facilities.

    Source: §I · p. 1

  • Method

    A generative agent proposes assemblies, and gradients from a graph-attention feasibility surrogate optimize connected-part geometry for assembly constraints.

    Source: §I · p. 1, 1) Graph construction · p. 3, 2) Architecture · p. 3

  • Results

    The pipeline modified 52/60 unseen inputs and satisfied all cell constraints for 22/60, versus Blox-Net’s 5/60 target-check and 4/60 end-to-end successes.

    Source: §I · p. 1

  • Takeaways & Limitations

    Two UR5e robots completed all ten physical builds with minor human interventions, demonstrating feasibility for bespoke assemblies in this setup.

    Source: §IV · p. 5

    Unmodeled gripper collisions and motion-planning constraints account for most of the gap between 86.7% geometric-constraint success and 36.7% full cell-build success.

    Source: §V · p. 6

Abstract

from arXiv · show

Transitioning from a digital design to a robotic assembly process currently requires months of expert manual tuning to reconcile part geometries with robotic constraints. This paper presents an end-to-end, autonomous pipeline for the design and physical construction of bespoke wooden assemblies. A generative AI agent translates user prompts into initial 3D geometries, balancing the visual fidelity of the design with select physical constraints. The assemblability of the design is further improved by a gradient-based repair stage that backpropagates through a graph attention network surrogate to adjust component geometries. In addition to correcting for disjointed and overlapping components, we demonstrate hardware-specific corrections, differentiably optimizing the geometry of components to enable robot screwdriving for 86.7% of 60 novel natural language inputs, significantly outperforming prior work by a factor of ten. For ten of the structures, we physically demonstrate assemblability with two UR5e robots. This work marks a meaningful step toward on-demand robotic manufacturing, enabling the rapid production of customized, low-volume goods.

I. INTRODUCTION

Robotic assembly still depends on months of expert tuning to reconcile bespoke part geometries with site-specific hardware constraints. The paper proposes differentiable geometry repair within an autonomous pipeline and reports improved assembly outcomes on unseen designs.

I. INTRODUCTION

  • Months of expert tuning are needed to reconcile complex part geometries with reachability, collision avoidance, and end-effector clearances across heterogeneous robotic cells.This makes low-volume customized production slow and expensive.Source: §I · p. 1
  • Screwing and non-uniform components create hardware bottlenecks, including inadequate gripper or screwdriver clearance and parts too thick for available screws, that require geometry changes.Pose adjustments alone do not address these constraints.Source: §I · p. 1
  • A surrogate trained on labeled simulation data maps part geometry to assembly-success probabilities, whose gradients locally optimize designs; the framework can extend to other robotic setups when suitable simulation is available.The approach builds on prior stochastic perturbation or post-hoc filtering methods.Source: §I · p. 1
  • The pipeline modified 52/60 unseen inputs and satisfied all cell constraints for 22/60, outperforming Blox-Net’s 5/60 target-check and 4/60 end-to-end successes.The evaluation used 60 natural-language prompts.Source: §I · p. 1

1) Automating Robot Manufacturing With Fixed Components:

Prior work spans language-driven assembly planning, trial-and-error design repair, and graph-network surrogates. This paper positions continuous, hardware-specific geometry modification alongside those approaches.

1) Automating Robot Manufacturing With Fixed Components:

  • Language-driven systems generate assembly instructions or physically stable structures, while this work modifies part geometry for complex assembly mechanisms.The cited systems include Blox-Net, Kyaw et al., BrickGPT, and Prompt-to-Product [6].Source: 1) Automating Robot Manufacturing With… · p. 2

2) Design for Robot Assembly Through Human Alterations or Trial-and-Error:

  • Existing design methods use stochastic perturbation or post-hoc filtering and often restrict parts to modular units, whereas this approach treats component geometry as continuous.Specialized simulators for interlocking mechanics can limit application to bespoke, nonmodular geometries.Source: 2) Design for Robot Assembly Through… · p. 2

3) Surrogate Modeling With Graph Neural Networks:

  • Graph neural networks have served as surrogates for physical and combinatorial systems, and this work uses an attention-based GNN to model part-geometry assemblability.Prior examples include learning dynamics on particle graphs and meshes, engineering design transfer, and constraint satisfaction.Source: 3) Surrogate Modeling With Graph… · p. 2

III. METHODS

The method generates candidate wooden assemblies, checks them against simulation and cell constraints, and applies differentiable geometric optimization before robotic construction. Its training data come from a simulated replica of the assembly setup.

III. METHODS

  • A generative agent creates eight candidate designs, filters them for stability and motion-planning feasibility, and selects the best prompt match before GAT-based geometry optimization and possible position perturbation.The pipeline adapts Blox-Net’s generation and perturbation steps while adding initial motion-planning checks.Source: §III · p. 2

A. Design Constraints Imposed by Physical Setup

  • Assembly requires graspable parts, screwdriver access, screw-compatible thickness, sufficient joint overlap, and feasible motion plans in a cell with two UR5e arms and a robot screwdriver.The gripper width is 35 mm; fastening requires part thickness below 20 mm and intersection-plane dimensions above 10 mm.Source: A. Design Constraints Imposed by… · pp. 2–3

B. Dataset Creation and Assessing Assemblability

  • The simulator labels randomized two-block assemblies by whether parts reach their desired locations and which objects collide, but data collection omits motion planning and the robot gripper.Block sizes range from 7 to 200 mm, with relative positions offset by −0.5 to 0.5 relative to parent-block dimensions.Source: B. Dataset Creation and Assessing… · p. 3

C. Feasibility Surrogate

The feasibility surrogate makes geometric assembly constraints differentiable, representing connected block pairs as graphs and predicting geometry and feasibility through separate network branches.

C. Feasibility Surrogate

  • A learned GAT surrogate provides a unified differentiable model for geometric constraints that are analytically tractable for rectangular blocks but may extend to complex geometries and hardware metrics.The model enables gradient backpropagation through constraints that may be computationally intractable analytically.Source: C. Feasibility Surrogate · p. 3

1) Graph construction:

  • Feasibility is predicted per connected block pair using a two-node graph with directed edges, block dimensions and anchor/candidate roles as node features, and relative geometry as edge features.The generative agent supplies block connectivity during inference.Source: 1) Graph construction · p. 3

2) Architecture:

  • A late-fork GAT uses shared layers before separate regression and classification branches for overlap and thickness, feasibility, and failure modes.The architecture uses two shared GATv2 layers; the active node supplies the readout while the anchor provides context.Source: 2) Architecture · pp. 3–4, Figure 2 · p. 4

3) Training objective:

Training combines geometric regression with feasibility and mode classification, then evaluates the retained model on a test set. The best run reports high feasibility F1 and low geometric prediction errors.

3) Training objective:

  • The training objective combines regression on two geometric outputs with binary cross-entropies for feasibility and assembly mode.The total loss is Lreg + α Lfeas + β Lmode, with α = β = 0.1.Source: 3) Training objective · p. 4
  • The mode head is auxiliary: it shares the classification trunk during training and is discarded at inference.Its two-component mode logit uses weighted binary cross-entropy; per-mode weights are clipped at 50 and targets use label smoothing εls = 0.05.Source: 3) Training objective · p. 4

4) Training:

  • Training uses 174,000 graphs with an 80/10/10 split, Adam at learning rate 2 × 10^-3, and batch size 2048.Learning rate decays on validation plateaus; training runs up to 350 epochs with early stopping at patience 60, retaining the best checkpoint.Source: 4) Training · p. 4
  • 0.965 test feasibility-classification F1; threshold-derived F1 is 0.924, with 4.4 mm overlap MAE and 1.7 mm thickness MAE.The corresponding thresholds are 10 and 20 mm; prediction-to-target standard-deviation ratios are 0.89 for overlap and 0.93 for thickness.Source: 5) Performance · p. 4

5) Performance:

A four-stage repair process combines analytical corrections, gradient-based block edits, and constrained fastener placement. In simulation it improves screwability over Blox-Net, and ten prompted objects are physically built, though some builds require intervention.

D. Gradient-based assembly repair

  • Repair resolves interpenetrations and snaps contacts analytically, then optimizes child-block size and position bottom-up against committed parents.Adam updates combine GAT constraint hinges with a drift regularizer; projections bound position and size changes, and periodic snaps preserve contact alignment before constrained fastener placement.Source: D. Gradient-based assembly repair · p. 4

IV. EXPERIMENTAL RESULTS

  • 10/10 physical builds completed with minor interventions, while 7/10 were intervention-free; 47/50 individual block placements needed no intervention.All ten had full simulated feasibility; initial component locations were manually estimated, and interventions included screw-feeder jams and block resets due to vibration.Source: §IV · pp. 4–5, Figure 3 · p. 5
  • 86.7% screw-failure success, significantly higher than Blox-Net; Blox-Net more often passed gap/concurrence and tipping checks, insignificantly.Across 60 prompts, our approach also had significantly fewer screwable and sufficient-overlap failures than Blox-Net; Table I reports pass percentages and counts for simulated checks.Source: §IV · pp. 5–6, Table I · p. 5
  • Removing the differentiable geometry step reduces assembly performance; the ablation identifies geometric optimization as key and reports about 90 s for that step.The full pipeline averaged 54 min on the stated laptop; removing generative-AI parallelization also meaningfully reduced assembly success, especially for tipping.Source: §IV · p. 6

V. LIMITATIONS

The approach optimizes part-level geometric constraints, while multi-body kinematics remain outside the surrogate’s optimization. Its local, continuous edits also depend on a sufficiently assemblable starting design.

V. LIMITATIONS

  • The GAT surrogate optimizes part-level constraints such as screwdriver clearance, thickness, and face overlap, leaving multi-body kinematics to separate checks.The stated kinematic constraints include gripper collisions, swept-volume clearance, and global reachability.Source: §V · p. 6
  • Unmodeled kinematics, especially insertion gripper collisions, account for most of the gap between 86.7% geometric-constraint success and 36.7% full cell-build success.The castle’s central tower could not be placed because the gripper collided with surrounding components; adding more assembly criteria may make accurate assemblable-space classification harder.Source: §V · p. 6
  • Local continuous optimization is reasonable only when the starting design is close enough to assemblable; it cannot add or remove parts or reorder assembly sequences.Large necessary changes can evade the optimizer, and some modified designs lose recognizability.Source: §V · p. 6

VI. CONCLUSION

The paper uses a differentiable feasibility model to modify geometries for robotic assembly, including hardware-specific criteria such as screwdriving. It presents this as a step toward autonomous on-demand manufacturing while identifying residual failures and broader potential applications.

VI. CONCLUSION

  • The learned feasibility model differentiably modifies geometries, generalizes to dozens of unseen structures, and outperforms Blox-Net when incorporating hardware-specific criteria such as robot screwdriving.The system presents autonomous design generation, modification, and assembly as a route toward on-demand manufacturing.Source: §VI · p. 6
  • Figure 4 shows residual failures involving gripper collisions, screw-length mismatch, and designs that lose visual recognizability after repair.These examples span hardware interference, component geometry, and visual fidelity.Source: Figure 4 · p. 7
  • The approach could extend to motion-planning feasibility or differentiable stress and compliance objectives, and the authors envision applications beyond wooden assemblies.They cite novel aircraft or appliance components as possible applications.Source: §VI · p. 7

APPENDIX I SYSTEM ARCHITECTURE AND MOTION PLANNING

The system combines robotic motion planning and perception with a learned surrogate training objective. Its execution uses three assembly primitives, while the surrogate objective combines regression and classification losses and emphasizes samples near the feasibility boundary.

APPENDIX I SYSTEM ARCHITECTURE AND MOTION PLANNING

  • The execution stack uses ROS 2 and MoveIt 2, with RRTConnect for free-space motion, collision-checked Cartesian paths for alignment, and FoundationPose initialized from SAM2 segmentations for object tracking.Assembly operations use PICKANDPLACE, PICKANDHOLD, and SCREWPARTS primitives; designs follow a 15-rule prompt covering assembly order, stability, gripper clearance, and screw allocation.Source: Appendix I · p. 8

APPENDIX II GAT SURROGATE TRAINING OBJECTIVE

  • The GAT surrogate combines regression and feasibility and mode classification losses, while standardizing overlap and thickness targets.The loss is Ltot = Lreg + αLfeas + βLmode, with α = β = 0.1.Source: APPENDIX II GAT SURROGATE TRAINING… · p. 8
  • Regression uses a boundary-weighted Huber loss, and classification uses positive-class-weighted binary cross-entropies with target-label smoothing.The Huber loss uses δ = 1; the classification settings are wmax = 50 and εls = 0.05, with samples near the feasibility boundary weighted heavily.Source: APPENDIX II GAT SURROGATE TRAINING… · p. 8

APPENDIX III GRADIENT-BASED ASSEMBLY REPAIR

The repair procedure adjusts candidate geometry from its initial parameters using Adam on a composite loss. It tightens feasibility thresholds over time, constrains changes to a trust region, and periodically snaps parts to parent contact faces.

APPENDIX III GRADIENT-BASED ASSEMBLY REPAIR

  • Candidate geometry is adjusted bottom-up from initial size and position parameters using Adam on a composite repair loss.The parent part is frozen while the child part is optimized against the GAT model.Source: APPENDIX III GRADIENT-BASED ASSEMBLY… · p. 8
  • The repair loss combines feasibility penalties with squared drift penalties on size and position, using λs = 5 and λp = 1.The hinge-loss scaling is K = 10^6.Source: APPENDIX III GRADIENT-BASED ASSEMBLY… · p. 8
  • Over 1,000 steps, overlap and thickness thresholds tighten from twice the slack to nominal limits of 10 mm and 20 mm, respectively.Trust-region projections cap position drift at 20 mm, tangential growth at 30%, and contact-axis shrinkage at 90%.Source: APPENDIX III GRADIENT-BASED ASSEMBLY… · p. 8
  • The algorithm periodically snaps the child to the parent contact face and stops when overlap and thickness meet thresholds and predicted feasibility exceeds 0.9.It probes overlap first and freezes position gradients when the initial overlap is already above threshold.Source: APPENDIX III GRADIENT-BASED ASSEMBLY… · p. 8

APPENDIX IV MLP + KD-TREE BASELINE FORMULATION

The MLP baseline represents part-pair geometry with a 12-dimensional feature vector and optimizes an assembly-good objective. Its optimization also penalizes distance from valid training configurations and uses multiple random restarts.

APPENDIX IV MLP + KD-TREE BASELINE FORMULATION

  • The MLP baseline maps 12-dimensional relative-position, block-dimension, and absolute-height features through three hidden layers of width 128.The layers use ReLU activations and dropout of 0.35.Source: APPENDIX IV MLP + KD-TREE BASELINE… · p. 8
  • The optimization adds a KD-tree distance penalty to valid training configurations and runs 10 random-restart Adam trajectories with soft parameter clipping.The distance penalty uses λkd = 0.2 and threshold τ = 1.5.Source: APPENDIX IV MLP + KD-TREE BASELINE… · p. 8

Related papers

Loading 2610.09777v1…