PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to build covering structures with continuous adjustments

Gabriel Vallat, Maryam Kamgarpour, Stefana Parascho

arXiv:2609.08669v1cs.ROcs.LG

TL;DR

Robotic construction plans can be disrupted by fabrication tolerances and unexpected changes, while learning construction policies also requires coupled discrete block choices and continuous placements. This paper introduces HSAC, a graph-based reinforcement learning method for plan-free construction with mixed actions. It outperforms HPPO asymptotically, handles larger discrete action spaces, and transfers to a physical two-robot arch build.

  • Problem

    Fixed high-precision construction plans are vulnerable to fabrication tolerances and unexpected changes, and mixed-action learning must couple block selection with continuous placement in costly stability-check environments.

    Source: §I · p. 1

  • Method

    The paper models construction as policy learning over graph-structured states and mixed discrete–continuous actions, extending SAC as HSAC.

    Source: §I · p. 1, §V · p. 6

  • Results

    HSAC outperforms HPPO asymptotically, remains robust across hyperparameters and up to ten discrete block types, and transfers to a physical two-robot arch build.

    Source: §V · p. 7

  • Takeaways & Limitations

    Two robots built an arch from 3D-printed blocks in closed-loop execution, demonstrating transfer of policies trained in simulation to real hardware.

    Source: §V · p. 7

Abstract

from arXiv · show

Robotic construction offers the potential to use materials more efficiently and create complex geometries, but current methods rely on rigid, high-precision plans that cannot accommodate the tolerances, inaccuracies, and unexpected changes inherent in physical fabrication. In this work, we introduce a reinforcement learning approach that forgoes predefined plans entirely, instead generating construction sequences adaptively as the structure is built. Our method operates on graph-structured state representations and a mixed (parameterized) action space, requiring both discrete block selection and continuous placement parameters. Because the stability simulation of a structure is computationally heavy, we develop an efficient exploration strategy by incorporating unilateral edges into graph neural networks, extending soft actor-critic (SAC) to this hybrid setting. We evaluate our algorithm, HSAC, against the prior method hybrid-PPO (HPPO), demonstrating significantly higher asymptotic performance and good sample efficiency. We also demonstrate HSAC's robustness to hyperparameter choices and its exploration capability, handling up to 10 discrete actions without performance degradation. Finally, we validate our approach on a physical two-robot setup, successfully building a spanning arch with 3D-printed blocks in closed-loop execution, confirming that policies trained in simulation transfer to real hardware.

I. INTRODUCTION

Robotic construction can use materials efficiently and produce complex geometries, but fixed high-precision plans are vulnerable to fabrication tolerances and unexpected changes. The paper proposes adaptive, plan-free policy generation and evaluates HSAC for mixed-action construction, including transfer to physical robots.

  • Rigid, high-precision plans can mismatch fabrication tolerances and unexpected changes, potentially causing catastrophic failure or costly, time-consuming revisions.The paper therefore forgoes an initial plan and generates one incrementally.Source: §I · p. 1
  • The method replaces a predefined construction plan with a policy that generates actions incrementally as construction proceeds.The policy maps states to actions and accommodates continuous adjustments during fabrication.Source: §I · p. 1
  • Mixed-action construction requires coupled choices of block type and continuous placement, a challenge existing HPPO and parametrized-DDPG methods struggle with in costly stability-check environments.The discrete and continuous decisions depend on one another.Source: §I · p. 1
  • HSAC extends SAC to mixed action spaces, while graph representations leverage geometric invariants to improve generalization in the construction task.The representation uses translation, permutation, and vertical-rotation invariants.Source: §I · pp. 1–2
  • HSAC achieved sample-efficient learning in mixed action spaces with costly simulation and transferred simulation-trained policies to a physical two-robot arch build.The demonstration used 3D-printed blocks.Source: §I · p. 2

II. DYNAMICAL SYSTEM DEFINITION

The construction task is represented as a sequential controlled process, with a graph state for the current structure and actions that combine block selection with placement parameters.

II. DYNAMICAL SYSTEM DEFINITION

  • Construction is modeled as a discrete-time controlled process with state, action, transition, reward, initial state, and discount factor, using deterministic transitions while allowing extension to stochastic dynamics.The task begins from a fixed block and aims to build a spanning structure.Source: §II · p. 2

A. State

  • The state is a labeled heterogeneous graph whose block-relative positions create invariance to rotation, translation, and permutation, while block identity need not encode its exact shape.This representation lets the agent learn more efficiently and can accommodate different shapes when their physical behavior can be simulated.Source: A. State · p. 2

B. Actions

  • Actions first select a block type, then specify placement through 2D translation and angle, represented continuously by four parameters.The translation is measured between the supported block’s face A and the new block’s face B; the angle is represented by its cosine and sine.Source: B. Actions · pp. 2–3

C. Initial state

The system starts with a fixed block and a target area, then evaluates placements through stability and coverage. The section motivates HSAC by the difficulty of optimizing graph states, mixed actions, and delayed collapse risk.

C. Initial state

  • The initial state contains a fixed base block and a rectangular area to cover, with its geometry shaping whether the agent is rewarded for taller or wider arches.A smaller initial-block angle tends toward taller arches, while a longer target area rewards wider arches.Source: C. Initial state · p. 3

D. Reward and transition function

  • Each transition tests structural stability: unstable placements end in failure, stable placements earn reward proportional to covered area, and a block corner below ground marks success.The stability simulation follows placement and release of the previously held block.Source: D. Reward and transition function · p. 3

III. HYBRID SAC ALGORITHM

  • Hybrid SAC addresses the challenge of graph-based states and coupled discrete–continuous actions, while SAC’s state-dependent exploration contrasts with mixed-action methods whose exploration is the same across states.The section also describes using GNNs to process graph states and remove common Q-function over-parameterization from hybrid actions.Source: §III · p. 3, D. Reward and transition function · p. 3

1) HSAC:

HSAC trains separate discrete and continuous policy components with SAC-like objectives, using replay-buffer sampling and updates to value and Q-functions. Separate entropy weights let the discrete policy remain exploratory when a shared coefficient would make it nearly deterministic.

1) HSAC:

  • Training alternates between sampling transitions with a stochastic mixed-action policy into replay buffer D and optimizing the policy, a value function, and two Q-functions with double Q-learning.The mixed action comprises a discrete choice a and continuous action u.Source: 1) HSAC · p. 3
  • The discrete and continuous policy components are optimized separately with objectives similar to SAC, and their costs are combined into the total policy cost.The continuous action is reparameterized as u = f_u(s,a,ε), with Gaussian noise ε as input.Source: 1) HSAC · pp. 3–4
  • Separate policy costs and entropy weights keep the discrete policy exploratory; a shared coefficient instead drove discrete entropy near zero and learning into local minima.The shared-coefficient agents used stochastic continuous actions to reach the target total entropy, while the split method controls each component's entropy independently.Source: 1) HSAC · p. 4

2) Graph neural networks:

The graph neural network maps heterogeneous construction states to action-level features for mixed discrete and continuous decisions. Unidirectional edges structurally separate action-specific Q-value estimates.

2) Graph neural networks:

  • The GNN processes heterogeneous state graphs into action-level features, which are used to compute the policy and value functions.The architecture uses transformer convolutional networks; block-node features are discarded, while the output of f_y(s) has size 2×n_y in the illustrated case.Source: 2) Graph neural networks · p. 4, Figure 4 · p. 4
  • The value function aggregates action-node features by their minimum, maximum, and mean before passing them to a fully connected network.The aggregation produces a 3n_φ-dimensional vector and the network has one output.Source: 2) Graph neural networks · p. 4
  • The policy is computed by processing each action-node feature as a separate input to three fully connected networks.The continuous policy is represented as a Gaussian distribution with mean and covariance.Source: 2) Graph neural networks · p. 4
  • A DQN-style network predicts values for discrete actions, but extending it to mixed actions can be over-parameterized and fail to ensure equal Q-values for equivalent parameters.The unidirectional-edge design addresses this equality constraint.Source: 2) Graph neural networks · pp. 4–5
  • Unidirectional edges keep continuous parameters for different discrete actions from interacting, enforcing equality between the corresponding Q-value components.The method is presented as a structural inductive bias for the mixed-action Q-function.Source: 2) Graph neural networks · p. 5, Figure 5 · p. 5

A. Simulation

The simulation implementation combines parallel contact detection, Rigid Block Analysis stability checks, and graph-based GPU training.

A. Simulation

  • Warp parallelizes contact detection across multiple simulation threads to accelerate training.The environment implementation and algorithm code are available on GitHub.Source: A. Simulation · p. 5
  • Gurobi is used as the optimization solver for Rigid Block Analysis stability checks.Source: A. Simulation · p. 5
  • PyTorch Geometric stores state-action pairs and supports batching and GPU-based graph neural network training.Source: A. Simulation · p. 5

B. Simulation Results

The simulation results compare HSAC with HPPO and probe HSAC's sensitivity to training choices and environment settings. HSAC achieves better converged policies, with a per-step optimization-time overhead.

B. Simulation Results

  • The experiments include a learning-curve comparison, four hyperparameter versions, and tests varying friction and the number of discrete actions.The stated base setup uses around 24 million parameters and 64 update steps per environment step.Source: B. Simulation Results · p. 5, 1) HSAC vs HPPO · p. 5

1) HSAC vs HPPO:

  • A single HSAC step takes on average five times longer than an HPPO step, though adjusting the update-to-simulation ratio can mitigate the overhead.The ratio can be tuned to the available simulation budget.Source: 1) HSAC vs HPPO · p. 5
  • HSAC achieves better policies when converged, while HPPO gets stuck in local minima and never reaches maximum return.The reward curves are averaged over three runs.Source: Figure 6 · p. 5, 1) HSAC vs HPPO · p. 5
  • Fig. 7 reports total reward over time rather than samples because the duration of an individual step can vary widely.Results are averaged over three runs.Source: Figure 7 · p. 5

2) Hyperparameters:

The experiments test sensitivity to training choices and environmental changes, the policy’s response to forced actions, and closed-loop construction on physical robots. HSAC handles expanded action spaces, adapts to forced changes, and keeps the physical arch closer to its target plane than the open-loop baseline.

2) Hyperparameters:

  • A larger model did not significantly improve results, a smaller training ratio slowed convergence, and deterministic rollout performed slightly worse.The compared models used around 143 million parameters and 64 update steps per environment step, versus 24 million parameters and 8 update steps; deterministic rollout could not use environment parallelization.Source: 2) Hyperparameters · p. 5

3) Change in environment:

  • N = 10 discrete actions remained tractable, and higher friction produced better results, indicating efficient exploration without narrowing the search space.Higher friction increases the number of stable states.Source: Figure 8 · p. 6, 3) Change in environment · p. 6

4) Advantage of a policy:

  • After forced actions at steps 2 and 4, the policy compensated over time and produced a nearly identical arch.The authors describe such adaptations as often needed when simulation omits temporary obstacles or human intervention.Source: 4) Advantage of a policy · p. 6, Figure 9 · p. 6

C. Prototype

  • The closed-loop arch stayed in the target plane, while the open-loop structure deviated more.The comparison used a simulated structure as the open-loop plan; the closed-loop result was better aligned with the ground block.Source: Figure 10 · p. 6, C. Prototype · p. 6

V. CONCLUSION

The conclusion presents plan-free construction as policy learning over graph-structured states and mixed discrete–continuous actions, with HSAC’s graph-network design supporting stable learning. Experiments report stronger performance than prior methods, robustness across settings, and a physical two-robot arch built in closed loop.

V. CONCLUSION

  • The framework models construction as policy learning over graph-structured states, selecting both a block type and its precise placement.The policy maps states to mixed continuous–discrete actions.Source: §V · p. 6
  • HSAC uses graph neural networks with unidirectional edges to separate Q-values across discrete actions, a design the authors say was essential for stable learning.The algorithm extends SAC to mixed action spaces and uses separate entropy coefficients for discrete and continuous components.Source: §V · p. 6
  • After each block placement, a camera captures block positions and the policy computes the next target location during two-robot construction.One robot places the block while the other supports it; the placing robot then withdraws.Source: Figure 11 · p. 7
  • HSAC consistently outperformed HPPO asymptotically, handled up to ten discrete block types without degradation, and adapted to changes in friction.Two ABB GoFa arms also built an arch from 3D-printed blocks in closed-loop execution, demonstrating transfer from simulation to hardware.Source: §V · p. 7

APPENDIX

The appendix describes a graph state that represents existing blocks, candidate actions, and the coverage objective, with labeled relations connecting these elements. It also identifies the base setup tables and notes selected model and action-space changes for experiments.

A. State Construction

  • Existing-block nodes encode block type, Izz inertia, and whether a block is robot-held or fixed; candidate-action nodes act as indexed placeholders for policy outputs.Existing-block node labels omit position, while the candidate set can be reduced when parts are available only in finite quantity.Source: A. State Construction • · p. 7
  • Labeled graph edges encode block-to-block transformations, attach the latest block to each candidate action, and connect blocks to objective corners.Objective-corner edges define the coverage polygon; constrained Delaunay triangulation connects its corners.Source: A. State Construction • · p. 7

B. Hyperparameters

  • Tables I and II list the environment and HSAC base setups; the larger model used 12 GNN layers and 128 hidden channels, while action types varied by experiment.The larger-model setup also used batch size 128 and two update steps; action types were spaced from γ = 5° to 20° for larger action sets.Source: B. Hyperparameters · pp. 7–8, Table I · p. 8, Table II · p. 8

Related papers

Loading 2609.08669v1…