PaperlayerView exact revision on arXiv

Source-linked AI summary

ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning

Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti, Eduardo Montijano, Mac Schwager

arXiv:2610.00487v1cs.ROcs.LG

TL;DR

Construction planning must handle physical feasibility at each step, while existing approaches have limits in conditioning, temporary-support reasoning, and single-hypothesis rollout. ScaffoldM3C combines multimodal candidate generation, auxiliary scaffolds, and SMC; it is smaller and faster than competing baselines while achieving comparable quality and higher stability.

  • Problem

    Construction requires selecting components, placements, and step order while keeping every intermediate state physically realizable, and existing approaches are limited in representation, conditioning, or assembly process.

    Source: §I · p. 2

  • Method

    ScaffoldM3C proposes probabilistic next-block candidates, uses scaffolds for intermediate support, and applies Sequential Monte Carlo to maintain multiple assembly sequences.

    Source: §I · p. 2, B. Assembly Distributions · p. 3, §IX · p. 11

  • Results

    ScaffoldM3C uses 4× fewer parameters and runs 5× to 20× faster than competing baselines, with comparable visual quality, superior physical stability, and a 100% collision-free rate versus BrickGPT-Scaffold’s 90.19%.

    Source: B. Structure Quality and Feasibility · p. 7, §IX · p. 11

  • Takeaways & Limitations

    The authors report physically feasible build plans in simulation and real-world robotic assembly without computationally expensive sequence rollbacks.

    Source: §IX · p. 11

    Scaffold-stabilized placement order can obstruct workspace clearance and complicate robotic motion planning.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large language models for text-based generative construction. However, these approaches do not allow for Multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds. Therefore, we formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task conditioning modalities. Concurrently, we explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block token. We present Scaffold Multimodal Monte Carlo (ScaffoldM3C), a multimodal, lightweight, auto-regressive model for stable block-based construction, that proposes a set of next-step candidate blocks. Leveraging these candidates, we utilize Sequential Monte Carlo (SMC) to maintain a population of possible assembly sequences, allowing us to consider multiple, potentially different, assembly directions simultaneously. We train our multimodal architecture by extending the StableText2Brick dataset to contain image conditioning prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4x smaller than competing baselines, yielding a 5x to 20x speedup during inference, while achieving comparable construction quality to state-of-the-art methods and higher overall stability. We demonstrate the effectiveness of our approach through simulations and real-world robot assembly demonstrations.

I. INTRODUCTION

The introduction frames construction as long-horizon planning under physical-feasibility and sequence-choice constraints, and motivates ScaffoldM3C as a multimodal, scaffold-aware, multi-hypothesis alternative.

I. INTRODUCTION

  • Construction planning must choose components, placements, and step order so every intermediate state remains physically realizable and unobstructed.Temporary support may be needed, and different choices or sequence orders can produce the same final structure.Source: §I · p. 2

II. RELATED WORK

  • Prior approaches constrain representations, modalities, or assembly processes, while validity checks, rollbacks, or rigid inference constraints can be costly.Related work also identifies limits in geometric fidelity, placement uncertainty, and multimodal conditioning.Source: §I · p. 2, §II · p. 2

I. INTRODUCTION

  • ScaffoldM3C uses 4× fewer parameters and runs 5× to 20× faster than competing baselines while delivering comparable or superior overall structural stability.It is trained from scratch without relying on foundation models.Source: §I · p. 2

A. Stable Assembly Sequences

  • A placement is stable when it contacts the ground or its centroid projects inside the convex hull of supporting contact faces; scaffolds can restore missing support.The figure contrasts a single-support placement outside the hull with a scaffolded placement that enlarges the hull.Source: A. Stable Assembly Sequences · p. 3, Figure 2 · p. 3

B. Assembly Distributions

The framework represents each construction step as a distribution over feasible standard and scaffold-block proposals, then uses Sequential Monte Carlo to explore multiple sequences. Candidate repair and resampling support feasible search without exhaustive branching.

B. Assembly Distributions

  • At each step, the model distributes probability across feasible standard-block and scaffold-block candidates, allowing construction to branch across prior actions.Candidate selection probabilities make later sequences conditionally dependent on earlier choices.Source: B. Assembly Distributions · p. 3

C. Sequential Monte Carlo Inference

  • Sequential Monte Carlo maintains parallel particle trajectories, samples candidate placements, updates sequence weights, and resamples active trajectories.The final completed sequence is selected by length, with cumulative particle weight breaking ties.Source: C. Sequential Monte Carlo Inference · pp. 3–4
  • Proposed candidates that violate non-overlap constraints are repaired using a 2D constrained minimum translation vector, with larger corrections penalized and unresolved collisions scored zero.This repair avoids costly full stability checks during online expansion.Source: C. Sequential Monte Carlo Inference · p. 3

IV. MULTIMODAL TASK-SPECIFIC ARCHITECTURE 

ScaffoldM3C maps partial build sequences and text or image prompts to candidate block proposals. Its architecture combines separate sequence and conditioning encoders with attention and a multi-head proposal objective that encourages diverse candidates.

IV. MULTIMODAL TASK-SPECIFIC ARCHITECTURE

  • The autoregressive transformer maps a partial build sequence and a text or image prompt to next-block proposals that drive SMC search for feasible build sequences.Inference selects the longest valid trajectory at termination.Source: §IV · p. 4
  • Block positions use learned high-frequency Fourier features, while block classes use categorical embeddings and prompts are encoded by a quantized Gemma 3 model.The model combines learned absolute and rotary relative positional embeddings; using both yields better structural quality than either alone.Source: §IV · p. 4
  • Interwoven causal and cross-attention layers combine prior-placement context with conditioning, while windowed attention links local features to candidate locations.Three cascaded heads predict candidate positions, shape distributions, and placement likelihoods.Source: §IV · p. 4
  • Training treats each candidate as a mixture-of-experts proposal and combines task, routing, utilization, router-stability, and candidate-diversity losses.The diversity term penalizes identical position-class predictions.Source: §IV · p. 4

V. MULTIMODAL STABILITY-AWARE DATASET GENERATION

The dataset extension adds scaffold-stabilized sequences and image conditions to StableText2Brick. Training pairs combine text, image, or both while sharing the same stability-aware target sequence.

V. MULTIMODAL STABILITY-AWARE DATASET GENERATION

  • The original StableText2Brick sequences do not meet the paper’s stability constraints at every intermediate placement, so each sequence is replayed and augmented with the minimum required scaffold blocks.The update preserves target-block placement order and structure geometry while stabilizing intermediate and final assemblies.Source: §V · pp. 4–5
  • Image conditions include synthetic renders of completed simulated builds and realistic or abstract images generated from text-based guidance prompts with Qwen.Synthetic views preserve exact block geometry; the other domains use style-specific prompts.Source: Figure 4 · p. 5, §V · p. 5
  • Each build sequence yields 13 condition–sequence pairs: five text-only, three image-only, and five multimodal, all sharing the same stability-aware target sequence.Multimodal pairs combine one caption with an image condition sampled from the three image domains.Source: §V · p. 5

VI. EXPERIMENTS

The experiments compare ScaffoldM3C with BrickGPT variants and compare SMC with Top-K sampling. They evaluate text and image prompts in simulation and include robotic assembly demonstrations.

VI. EXPERIMENTS

  • The benchmark compares ScaffoldM3C with BrickGPT variants using different post-processing and scaffold-aware settings.One BrickGPT variant disables post-processing; another disables rollback but retains step-wise rejection, and a scaffold-aware variant uses rejection sampling.Source: §VI · pp. 5–6
  • The model takes a partial build sequence and conditioning prompt, predicts candidate block positions, classes, and likelihoods, then selects a block using Top-K or SMC.Its architecture uses interwoven transformer layers, windowed attention over local candidate features, and three cascading prediction heads.Source: Figure 3 · p. 5
  • The study compares SMC with Top-K sampling as inference heuristics for generating build sequences.Additional analyses vary SMC population size, candidate and class temperatures, and inspect candidate-block belief states as sequences grow.Source: §VI · p. 6
  • The simulation evaluation uses 479 text prompts and 288 image prompts, with incompatible conditioning modalities marked in Table I.The study also includes physical construction demonstrations with an xArm Lite robotic manipulator.Source: §VI · p. 6

A. Evaluation Metrics

The evaluation groups metrics into stability, feasibility, and semantic alignment, distinguishing overall trends from per-sample averages. It measures stability during construction and after completion, collisions and joint feasibility, and prompt alignment and geometric similarity.

Stability Metrics

  • Intermediate stability, sinter, averages each new block’s stability relative only to previously placed blocks; sfinal instead assumes all blocks are placed.The latter definition is inspired by BrickGPT [1].Source: A. Evaluation Metrics · p. 6
  • Together, sinter and sfinal estimate out-of-order placement, while soverall is the percentage of test structures that are fully sequentially stable.The fully stable indicator specifies whether a structure is fully stable.Source: A. Evaluation Metrics · p. 6

Feasibility Metrics

  • Feasibility metrics include average per-block voxel collisions, ecollision, collision-free rate, erate, and overall feasibility, efeasible, requiring both full stability and collision-free construction.The collision measure counts voxels occupied by two or more blocks.Source: A. Evaluation Metrics · p. 6

Semantic Alignment Metrics

  • Semantic alignment uses asim to compare rendered structures with prompts via CLIP for text and DINOv2 for images, and agt to compare predicted and ground-truth renderings via DINOv2.The evaluation also distinguishes overall trends from per-sample statistical averages.Source: A. Evaluation Metrics · p. 6
  • Generalization is assessed with cmin, the minimum Chamfer distance to any training structure, where higher values indicate greater novelty, and csem, distance to the ground-truth structure most aligned with its instruction.csem is used to assess prompt-generation flexibility.Source: A. Evaluation Metrics · p. 6

B. Structure Quality and Feasibility

The section reports higher overall stability and collision-free construction, alongside effective scaffold placement and visually comparable outputs. It also notes that completed structures can be more stable than intermediate states.

Visual Quality

  • SMC outputs align with text or image prompts, look visually comparable to BrickGPT variants, and produce fewer artifacts than Top-K sampling.The baseline cannot interpret image prompts.Source: B. Structure Quality and Feasibility · p. 6

Feasibility

Block counts

  • 127 total blocks and 29 scaffold blocks, versus BrickGPT-Scaffold's approximately 195 total and 42 scaffold blocks, match the expected structural scale and scaffold distribution.These block counts are compared against the ground truth.Source: Table I · p. 7, B. Structure Quality and Feasibility · p. 7

Sequential Monte Carlo

  • SMC maintains diverse, feasible sequence candidates, allowing exploration of alternative assembly plans and recovery from early suboptimal placements.The reported limitation of Top-K is that it conditions candidates on a single past build sequence and can become trapped in local minima.Source: B. Structure Quality and Feasibility · p. 7

Stability

  • ScaffoldM3C distributes scaffold blocks more effectively across structures than BrickGPT-Scaffold, resulting in higher overall stability.Standard BrickGPT variants cannot place scaffold blocks.Source: B. Structure Quality and Feasibility · p. 7
  • 15.24% higher overall stability than BrickGPT-Scaffold; SMC also gains 6% relative to Top-K, while BrickGPT-Scaffold's per-block score reflects 1.45× more scaffold blocks.The reported relative gain from Top-K to SMC is attributed to parallel hypothesis tracking correcting early placement errors.Source: B. Structure Quality and Feasibility · p. 7
  • Completed structures can be more stable than intermediate states, with occasional out-of-order placements such as placing a block before its supports.The authors reserve lightweight online stability checks during SMC reweighting for future work.Source: B. Structure Quality and Feasibility · pp. 7–8

C. Model Semantic Alignment and Generalization

ScaffoldM3C shows strong semantic alignment and generalization alongside faster inference, while population size and sampling temperatures shape build quality. Robot experiments demonstrate scaffold-supported overhangs in complex structures.

C. Model Semantic Alignment and Generalization

  • SMC matches BrickGPT in structural scale, semantic alignment, and minimum Chamfer distance (0.81), while its minimum semantic Chamfer distance (4.20) is comparable to BrickGPT’s (4.57).The reported visual ground-truth similarity is 0.84 for SMC, versus 0.82 for BrickGPT baselines and 0.81 for Top-K.Source: C. Model Semantic Alignment and… · p. 8

D. Inference Speed

  • SMC is at least 5× faster than standard BrickGPT and up to 36× faster than BrickGPT-Scaffold.Top-K is 3× faster than SMC and 20× to 111× faster than BrickGPT, though it generally produces coarser plans.Source: D. Inference Speed · p. 8

E. Robot Hardware Experiments

  • Scaffold-supported overhangs were demonstrated in three robot-built structures using 45 blocks for the boat, 86 for the car, and 89 for the bookshelf.The experiments used a 6-DOF XFactory arm and nine different blocks.Source: Figure 6 · p. 8, E. Robot Hardware Experiments · p. 8

A. Sequential Monte Carlo Belief State

  • Larger SMC populations generally improve stability and structural quality, with diminishing returns beyond 160 sequences; the best quality-speed tradeoff occurs between 20 and 40.After six steps with four sequences, the framework was already considering distinct build sequences.Source: Figure 7 · p. 9, A. Sequential Monte Carlo Belief State · p. 9

B. Candidate Versus Class Likelihood

  • Stability is highly sensitive to class-distribution temperature, while increasing candidate-distribution temperature generally improves stability and increases standard and scaffold block counts.Higher class temperatures can favor implausible block types and disrupt assembly; broader candidate exploration considers lower-probability placement locations.Source: Figure 8 · p. 9, B. Candidate Versus Class Likelihood · p. 10

VIII. LIMITATIONS

The paper identifies ambiguity in generated prompts and reference images, execution-order obstacles, and limits in sequence selection. It also describes failure modes and concludes with claims about the framework’s efficiency, stability, and validation.

VIII. LIMITATIONS

  • Ambiguous generated prompts and reference images can reduce block-sampling confidence; scaffold placement order may obstruct workspace clearance, and selecting the longest completed sequence can favor larger structures without explicitly evaluating stability.The authors propose jointly optimizing structural and scaffold placement order for robotic accessibility and adding explicit stability validation for closed-loop selection.Source: §VIII · p. 10

A. Failure Modes

  • Block misalignment can degrade candidate confidence and downstream plans, while prompt confusion, early termination, and missing blocks or scaffolding are also reported failure modes.The authors identify block misalignment as the most prevalent issue and note that some failures can produce physically stable but visually or semantically poor structures.Source: A. Failure Modes · pp. 10–11, Figure 10 · p. 10

IX. CONCLUSION

  • The conclusion reports 4× fewer parameters and a 5× to 20× inference speedup, with baseline-level visual quality, superior physical stability, and feasible simulated and real-world robotic assembly.The framework is reported to achieve these outcomes without computationally expensive sequence rollbacks.Source: §IX · p. 11

APPENDIX A LOSS FUNCTION TERMS

The loss function combines individual prediction objectives with candidate-balance terms. Its task losses address positions, classes, and candidate likelihoods, while auxiliary terms encourage balanced and diverse candidate predictions.

Candidate balance loss

  • Candidate-balance terms penalize unequal candidate use and training instability in candidate and class selection, while a cosine-similarity term encourages class diversity.The diversity penalty applies to spatially close, non-top-ranked candidate pairs.Source: Appendix A · p. 12

Task loss

  • The task loss combines sequence mean-square error for block positions with sequence cross-entropy for block classes.The position loss compares the highest-likelihood candidate position with the ground-truth position; the class loss compares its class logits with the ground-truth class.Source: Appendix A · p. 12
  • Pointwise KL loss trains a distribution over candidate predictions, approximating target likelihoods from negative distances to the same ground-truth block because training data lack multiple ground-truth candidates.The passage defines the approximate likelihood weights using the position and class losses for each candidate.Source: Appendix A · p. 12

APPENDIX B TRAINING AND HYPERPARAMETERS

Training uses a 225M-parameter model and specifies distinct inference settings for SMC and Top-K sampling. The configuration predicts 20 candidates per forward pass and uses temperatures chosen to balance exploration and exploitation for SMC.

Model configuration

  • The model has 225M trainable parameters and predicts 20 candidate blocks during each forward pass.Its total hidden dimension is 1024, split across block position, block class, and sequence positional embeddings.Source: Appendix B · p. 13

Inference settings

  • SMC inference uses a population of 20 feasible build sequences, with candidate and class likelihood temperatures of 2 and 1, respectively.The authors selected these temperatures to balance exploration and exploitation.Source: Appendix B · p. 13
  • Top-K sampling considers 80 block-position and type combinations per forward pass and sets both likelihood temperatures to 0.1.The low temperatures are specified to prevent excessive uncertainty in candidate selection.Source: Appendix B · p. 13

Related papers

Loading 2610.00487v1…