PaperlayerView exact revision on arXiv

Source-linked AI summary

StackGen3D: Stable 3D Block Stack Generation from Multiview Silhouettes

Xiaopeng Zhang, Luzhe Sun, Yiqing Xu, Matthew R. Walter

arXiv:2610.31803v1cs.CVcs.RO

TL;DR

Generating stable 3D block structures from silhouettes is underdetermined, and curating diverse stable examples is costly. StackGen3D combines a PI-GNN-based data-curation pipeline with conditional flow matching and geometric inference-time refinement. Best-of-10 selection raises post-hoc physical stability while retaining high silhouette fidelity, though geometric refinement does not guarantee stability.

  • Problem

    Generating structures from silhouettes is underdetermined, while building diverse stable training data requires costly physical-feasibility evaluation.

    Source: §I · p. 1

  • Method

    StackGen3D uses a PI-GNN to curate stable structures, trains conditional flow matching on them, and applies AABB-based selection or refinement at inference.

    Source: §I · p. 1, B. Conditional Flow Matching and… · p. 2

  • Results

    Best-of-10 selection raises post-hoc physical stability from 45.3% to 72.4%, while a single sample achieves 0.967 silhouette IoU.

    Source: §I · p. 1

  • Takeaways & Limitations

    The preliminary results provide evidence that scalable stable-data generation can support learned generative models for underdetermined creative structures.

    Source: §I · p. 1

    AABB-based selection and refinement do not guarantee physical stability; simulation is used only afterward for evaluation.

    Source: B. Conditional Flow Matching and… · p. 2

Abstract

from arXiv · show

Generating physically stable 3D structures from visual specifications requires resolving geometric ambiguity while accounting for physical interactions among objects. We present StackGen3D, a framework for generating stable block assemblies conditioned on a given block inventory and three orthographic silhouettes. Our approach consists of two stages. First, we develop a physics-informed graph neural network (PI-GNN) that models support relations between blocks and serves as a fast stability predictor for curating a diverse dataset of stable structures. Second, we train a conditional flow-matching model on this dataset to capture the multimodal distribution of block configurations consistent with the target silhouettes. At inference time, lightweight geometric selection and refinement reduce inter-block penetration without physics simulation. Preliminary experiments show that the PI-GNN generalizes from simple stacks to complex structures substantially better than a parameter-matched multilayer perceptron. For structure generation, Best-of-10 selection improves physical stability from 45.3% to 72.4% while achieving a silhouette IoU of 0.970. These results highlight the potential of combining relational physical reasoning, scalable data curation, and conditional generative modeling for creative 3D assembly. Project website: [https://ripl.github.io/StackGen3D-web/](https://ripl.github.io/StackGen3D-web/)

I. INTRODUCTION

StackGen3D generates physically stable block structures from a fixed inventory and three orthographic silhouettes, addressing both geometric ambiguity and the cost of building diverse stable training data. It combines a PI-GNN stability predictor with conditional flow matching and lightweight inference-time refinement.

I. INTRODUCTION

  • The task is underdetermined: many block-pose configurations can match the silhouettes, but only some avoid interpenetration and remain stable.Curating diverse stable structures is difficult because evaluating physical feasibility with repeated simulation is expensive to scale.Source: §I · p. 1
  • The approach uses a fast PI-GNN stability surrogate to curate diverse stable structures, then trains conditional flow matching on the resulting dataset.At inference, AABB-based Best-of-N selection or Remove-then-Rerun refinement reduces residual inter-block collisions without physics simulation.Source: §I · p. 1
  • 92.66% vs. 46.86% accuracy: a PI-GNN trained on simple structures transfers substantially better to complex structures than a parameter-matched MLP.The authors interpret this as evidence that support-relation reasoning generalizes across structural complexity.Source: §I · p. 1
  • Best-of-10 selection raises post-hoc physical stability from 45.3% to 72.4% while preserving silhouette fidelity, with a single-sample silhouette IoU of 0.967.The authors describe these as preliminary results.Source: §I · p. 1

II. STACKGEN3D

StackGen3D first constructs a large dataset of physically stable structures, then learns a conditional generative model from it.

II. STACKGEN3D

  • StackGen3D is organized as two stages: construct a dataset of physically stable structures, then learn a conditional generative model from that dataset.The learned model is conditioned on the task specification described in the paper's introduction.Source: §II · p. 1
  • The first stage builds a large-scale dataset of physically stable structures for model training.Source: §II · p. 1
  • The second stage learns a conditional generative model from the constructed dataset.Source: §II · p. 1

A. Relational Stability and Dataset Curation

The data-curation pipeline represents stability through block support relations and combines simple-stack and complex-structure examples to train a predictor that filters candidate assemblies. Stable structures are paired with multiview silhouettes for training.

A. Relational Stability and Dataset Curation

  • A PI-GNN, models block-level stability and displacement over a directed support graph whose nodes encode geometry, pose, mass, and supported mass, and whose edges encode contact and relative displacement.Message passing propagates local support interactions through the structure.Source: A. Relational Stability and Dataset… · p. 1
  • The curation pipeline trains Gsimple on COM-heuristic simple stacks, then combines those examples with complex candidates labeled using MuJoCo to train Gcurated for efficient candidate filtering.Complex candidates are formed by randomly removing blocks from dense Filler3D structures.Source: A. Relational Stability and Dataset… · p. 1
  • The stable structures are paired with multiview silhouettes to form the training data.Source: A. Relational Stability and Dataset… · p. 2

B. Conditional Flow Matching and Geometric Inference

Conditional flow matching models multiple stable pose configurations compatible with the silhouettes, and inference integrates a learned velocity field before applying geometric overlap-based refinement. These geometric procedures reduce penetration but do not guarantee physical stability.

B. Conditional Flow Matching and Geometric Inference

  • Conditional flow matching represents the one-to-many mapping by interpolating Gaussian noise and simulator-verified stable block poses, with each pose encoded as a position and quaternion.The interpolation is defined over time t from x0 ∼ N(0, I) to x1.Source: B. Conditional Flow Matching and… · p. 2
  • At inference, an ODE solver integrates the Transformer-based velocity field from t = 0 to 1, conditioned on the block inventory and silhouettes.Source: B. Conditional Flow Matching and… · p. 2
  • Best-of-N selects the sample with the smallest maximum pairwise AABB penetration, while Remove-then-Rerun removes a block with the largest overlap and resamples until a threshold is met or all blocks are removed.Neither procedure guarantees physical stability; simulation is used only afterward for evaluation.Source: B. Conditional Flow Matching and… · p. 2

A. PI-GNN Stability Prediction

The PI-GNN transfers from simple to complex structures substantially better than a parameter-matched MLP, while direct training on complex structures remains beneficial for identifying unstable blocks.

A. PI-GNN Stability Prediction

  • 92.66% vs. 46.86% accuracy: Gsimple transfers to complex structures substantially better than the parameter-matched MLPsimple.The models contain 261,576 and 261,383 parameters, respectively.Source: A. PI-GNN Stability Prediction · p. 2
  • On the complex test set, Gsimple approaches Gcomplex at 92.66% vs. 97.58%, though the balanced-accuracy gap favors direct complex-domain training for identifying unstable blocks.Fig. 2 qualitatively compares block-level predictions with MuJoCo labels on complex structures.Source: A. PI-GNN Stability Prediction · p. 2, Figure 2 · p. 2
  • Gcurated achieves 96.32% accuracy, 92.87% balanced accuracy, and 93.90% macro-F1 on its combination validation set, as a reference rather than a controlled baseline.Source: A. PI-GNN Stability Prediction · p. 2

B. Structure Generation without Online Simulation

The conditional flow model closely matches target silhouettes, while lightweight AABB-based selection improves stability without online physics simulation; post-hoc evaluation shows remaining physical limitations.

B. Structure Generation without Online Simulation

Loading 2610.31803v1…