PaperlayerView exact revision on arXiv

Source-linked AI summary

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

arXiv:2211.04604v2cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robots must build physically valid, semantically meaningful structures from partial observations of previously unseen objects and high-level language instructions. StructDiffusion combines language-conditioned diffusion, object-centric transformers, and a learned discriminator to generate and filter object placements, achieving higher success rates than strong baselines.

  • Problem

    Robots need to rearrange previously unseen objects into structures satisfying language-driven spatial, geometric, and physical constraints.

    Source: §I · p. 1

  • Method

    StructDiffusion combines an object-centric language-conditioned diffusion model with multimodal transformers and a learned discriminator to generate and filter 6-DoF object poses.

    Source: §I · p. 2

  • Results

    StructDiffusion achieves 16.3% higher success than an existing single-task model and 13.5% higher success than a strong multi-task baseline.

    Source: §I · p. 2

  • Takeaways & Limitations

    The method creates multiple physically valid structures from high-level language instructions while generalizing to previously unseen objects.

    Source: §I · p. 2, §VII · p. 9

    The assumption that objects remain rigidly attached to the gripper after grasping does not generally hold in the real world and causes occasional failures.

    Source: C. Rearrangement with Pick and Place · p. 9

Abstract

from arXiv · show

Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen. In this work, we focus on the problem of building physically-valid structures without step-by-step instructions. We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures given partial-view point clouds and high-level language goals, such as "set the table". Our method can perform multiple challenging language-conditioned multi-step 3D planning tasks using one model. StructDiffusion even improves the success rate of assembling physically-valid structures out of unseen objects by on average 16% over an existing multi-modal transformer model trained on specific structures. We show experiments on held-out objects in both simulation and on real-world rearrangement tasks. Importantly, we show how integrating both a diffusion model and a collision-discriminator model allows for improved generalization over other methods when rearranging previously-unseen objects. For videos and additional results, see our website: https://structdiffusion.github.io/.

I. INTRODUCTION · II. RELATED WORK

StructDiffusion addresses language-guided construction of physically valid structures from partial-view point clouds of previously unseen objects. It combines object-centric language conditioning, diffusion-based pose refinement, and physical-validity discrimination, while extending prior manipulation and planning work beyond direct regression and known-object assumptions.

I. INTRODUCTION

  • StructDiffusion predicts rearrangements for unseen objects from partial-view point clouds while jointly reasoning about geometry, language semantics, and physics.The task requires solutions satisfying multiple simultaneous constraints rather than only following a command.Source: §I · p. 1
  • Directly regressing a single action or pose can fail when constraints conflict, whereas diffusion models represent and refine distributions over possible solutions [1] [3] [5] [8].The introduction highlights tensions such as placing objects tightly while avoiding collisions.Source: §I · pp. 1–2
  • The method combines an object-centric language-conditioned diffusion model with a learned discriminator that rejects samples violating physical constraints.A multimodal transformer predicts 6-DoF goal poses from word tokens and point-cloud object encodings; diffusion iteratively refines them before discriminator-based selection.Source: §I · p. 2
  • Because goals are predicted relative to observed partial-view object point clouds, StructDiffusion does not require object models or prior object foreknowledge.Unknown object instance segmentation first separates the scene into objects, following prior work [13] [14].Source: §I · p. 2
  • 16.3% and 13.5% higher success rates are achieved than an existing single-task model and a strong multi-task baseline, respectively.StructDiffusion is presented as the first method to build multiple stacking and 3D-reasoning structures across unseen objects from high-level language instructions.Source: §I · p. 2

II. RELATED WORK

  • Language-conditioned manipulation has achieved strong pick-and-place and articulated-object results, but prior skills had not demonstrated sequencing into the complex structures studied here [3].LLM-based approaches such as SayCan [21] sequence skills but use language-based world representations and require prompt engineering [13] [23] [21].Source: §II · p. 2
  • Object-centric multimodal methods such as VIMA [1] segment or detect objects and encode them separately, but direct regression does not specifically target physically realistic structures.The paper reports that predicting placement poses or actions directly produces more failures and lower-quality structures.Source: §II · p. 2
  • Prior unknown-object planning assumes rigid-object frameworks, explicit predicates, or learned relations, while StructDiffusion implicitly classifies whole object relationships for sequential manipulation [24] [29].Existing diffusion robotics methods [31] [8] [12] [32] typically require known object models or fixed visual viewpoints; StructDiffusion instead generates placement poses directly from point clouds and flexible language goals.Source: §II · p. 2

III. PRELIMINARIES · A. Diffusion Models

The preliminaries introduce diffusion models and transformers as core components, then describe diffusion as noise-based generative modeling with a learned reverse process and simplified noise-prediction training.

III. PRELIMINARIES

  • Diffusion models and transformers provide the core neural components of the approach.Source: §III · pp. 2–3
  • The object-centric diffusion model combines diffusion with a multimodal transformer to reason iteratively over 3D object embeddings and language task specifications.It predicts goal poses for objects.Source: Figure 2 · p. 3

A. Diffusion Models

  • Denoising diffusion models are generative models that begin with data samples drawn from the data distribution.Source: A. Diffusion Models · p. 3
  • The forward process is a Markov chain that gradually adds Gaussian noise, creating latent variables x_1 through x_T.Source: A. Diffusion Models · p. 3
  • A fixed variance schedule uses small per-step variance while making total accumulated noise large, enabling reverse sampling from Gaussian noise through a learned Gaussian posterior.Source: A. Diffusion Models · p. 3
  • The work fixes the covariance to an untrained time-dependent constant, reparameterizes the mean with noise, and trains using a simplified variational objective.The objective derives from minimizing the variational lower bound on the negative log-likelihood E[−log p_θ(x_0)].Source: A. Diffusion Models · p. 3

B. Transformers · IV. StructDiffusion FOR OBJECT REARRANGEMENT

The paper uses Transformer encoders to model variable-length object and language inputs, then introduces StructDiffusion for language-guided rearrangement from partial-view point clouds. StructDiffusion jointly generates object goal poses with diffusion and filters invalid configurations using a learned discriminator.

B. Transformers

  • Transformer encoders use scaled dot-product attention to model interactions among sequence elements and retain the original architecture’s encoder layers.Each input is projected into queries, keys, and values, whose compatibility determines weighted aggregation.Source: B. Transformers · p. 3
  • Attention masking enables each Transformer encoder layer to process sequences with different lengths.Each layer combines an attention module with a positionwise fully connected feed-forward network.Source: B. Transformers · p. 3

IV. StructDiffusion FOR OBJECT REARRANGEMENT

  • StructDiffusion rearranges rigid objects from a single partial-view, segmented point cloud and a language specification by predicting goal poses for the objects.Initial object poses are estimated from point-cloud centroids, and the robot executes pick-and-place actions to reach predicted goals.Source: §IV · p. 3
  • The pick-and-place setup can be extended with sampling-based TAMP solutions to incorporate motion primitives such as pushing and regrasping.These additional primitives can be used to reach the goal poses predicted by the system.Source: §IV · p. 3
  • The framework combines a diffusion-model generator with a learned discriminator that filters invalid goal-pose samples.The discriminator evaluates the imagined scene after objects are transformed to their predicted goals.Source: §IV · pp. 3–4
  • Its object-centric Transformer maintains a separate attention stream for each object to learn geometric interactions and ground language concepts in spatio-semantic relations.Examples include relations involving attributes such as large, circle, and top.Source: §IV · p. 3
  • Diffusion jointly predicts all object goal poses from randomly placed objects, enabling interaction-aware reasoning that outperforms direct goal-pose prediction from multimodal inputs.The formulation is presented as semantic rearrangement through a reverse diffusion process.Source: Figure 3 · p. 4

A. Encoders · B. Conditional Pose Diffusion Model

StructDiffusion encodes point clouds, language, diffusion states, and sequence roles into latent tokens for transformer processing. Its conditional pose diffusion model jointly generates 6-DoF object poses with object-centric reasoning over global constraints, local interactions, and diverse language-consistent structures.

A. Encoders

  • The object encoder combines centered point-cloud features from PCT with an MLP encoding the original mean position for semantic, geometric, and spatial reasoning.The two encodings are concatenated into each object’s latent representation.Source: A. Encoders · p. 4
  • Language instructions are encoded token-by-token with learned embeddings, establishing fine-grained correspondences between linguistic components and generated-structure constraints.The paper also investigates whole-sentence encoding with a sentence transformer in Appendix A.Source: A. Encoders · p. 4
  • Diffusion encodings represent iteratively optimized object poses and time-dependent reverse-diffusion information through learned MLP and time embeddings.These encodings feed pose states and the diffusion timestep back into the model.Source: A. Encoders · p. 4
  • Learned position and type embeddings distinguish word and object positions while differentiating object point clouds from word tokens.Object point clouds use υ_i = 1 and word tokens use υ_i = 0.Source: A. Encoders · p. 4

B. Conditional Pose Diffusion Model

  • StructDiffusion conditions reverse diffusion on object point clouds and language, using a transformer to predict per-object noise while jointly generating all goal poses from ξ_T ∼ N(0, I).The model represents each target as a 6-DoF pose in SE(3), directly predicting translation and constructing rotation from two vectors via a Gram–Schmidt-like process.Source: B. Conditional Pose Diffusion Model · pp. 4–5
  • The object-centric transformer models higher-order interactions among individual objects, capturing global constraints and local interactions while rearranging variable numbers of objects through attention masks.Unlike diffusion models that directly generate goal images [12], this representation explicitly models scene objects.Source: B. Conditional Pose Diffusion Model · p. 4
  • Diffusion sampling from multiscale Gaussian noise produces diverse placements and object orientations, addressing ambiguity in language goals and object-dependent structural proportions.Different arrangements of differently sized objects can impose different geometric constraints.Source: B. Conditional Pose Diffusion Model · p. 4

C. Discriminators

StructDiffusion uses learned discriminators to rank diffusion-generated structures for realism, reasoning over transformed point clouds and language-conditioned structural validity. Point-level collision discrimination was more effective than latent object-centric discrimination, while classifier guidance provided no significant benefit.

Discriminator inference

  • The discriminator scores imagined scenes after rigidly transforming object point clouds to goal poses, then ranks diffusion-generated samples by predicted realism scores s ∈[0, 1].This filtering step supplements the generator during inference.Source: C. Discriminators · p. 5

Discriminator representation

  • The discriminator reasons directly over interactions among transformed point-cloud objects and adds one-hot point features to preserve object identity.This point-level representation differs from the generator’s latent object-centric representation.Source: C. Discriminators · p. 5

Discriminator comparisons

  • Point-level collision discrimination had more discrimination power than latent object-centric discrimination, while integrating classifier guidance into diffusion produced no significant benefits.These findings came from preliminary experiments; further details are provided in Appendix B.Source: C. Discriminators · p. 5

Discriminator types

  • The method explores collision and structure discriminators, with the latter language-conditioned to classify validity under structure-specific constraints.The collision discriminator predicts pairwise collisions from partial point clouds, whereas the structure discriminator evaluates whole multi-object structures.Source: C. Discriminators · p. 5

Structure discriminator

  • The structure discriminator performs better when predicting only local constraints, using normalized scene point clouds and omitting language specifying global placement constraints.This design focuses validity prediction on local structural requirements rather than where the structure should be placed on the table.Source: C. Discriminators · p. 5

D. Planning and Inference · E. Training Details

StructDiffusion samples multi-object structures by diffusing noisy goal poses, transforming object point clouds in batches, and ranking candidates with discriminators. Its diffusion and discriminator models are trained separately using balanced structure data, sampled collision configurations, perturbed negatives, and temporally varied point clouds.

D. Planning and Inference

  • The framework combines diffusion with discriminator-based ranking so generated structures satisfy both physical and semantic validity.This follows the inference procedure in which discriminator scores determine the returned ranking.Source: D. Planning and Inference · p. 5
  • The inference pipeline initializes noisy goal poses, performs diffusion, and transforms multiple objects in parallel on a GPU.Batch processing supports multiple structure samples efficiently.Source: D. Planning and Inference · p. 5
  • After diffusion, collision and structure discriminators score batched combined-object point clouds, and ranked samples are returned.The selected samples represent physically and semantically valid multi-object structures for downstream manipulation planning.Source: D. Planning and Inference · p. 5

E. Training Details

  • The diffusion model and discriminators are trained separately using language instructions, segmented object point clouds, and rearrangement-action sequences from [4].Goal poses are extracted from the action sequences, and one balanced model is trained across all structure classes.Source: E. Training Details · p. 5
  • Training uses batch size 128 and requires about 12 hours on an RTX3090 GPU; collision discrimination uses 100,000 randomly sampled pairwise object configurations.These settings apply to the diffusion model and collision discriminator, respectively.Source: E. Training Details · p. 5
  • Structure-discriminator negatives perturb randomly selected subsets of ground-truth target poses, producing varied numbers of objects out of place.Point clouds from different rearrangement time steps augment imagined scenes because occlusions vary across the sequence.Source: E. Training Details · p. 5

V. SIMULATION EXPERIMENTS … C. Comparison with Other Generative Models

StructDiffusion is evaluated in PyBullet on language-guided rearrangement scenes containing held-out objects, where success requires both physical validity and satisfaction of semantic constraints. It significantly outperforms alternative generative models, especially for precise placements, object contacts, and partial or occluded observations.

V. SIMULATION EXPERIMENTS

  • The evaluation compares StructDiffusion with existing methods and strong baselines for language-guided multi-object rearrangement.The study is designed to assess both comparative performance and the generator-discriminator design.Source: §V · p. 5, C. Comparison with Other Generative… · p. 6

A. Baselines

  • The compared baselines include StructFormer, a CVAE, and optimization with a learned structure discriminator using the cross-entropy method.StructFormer predicts poses autoregressively with a separate model per structure class; the CVAE predicts all poses in one forward pass using one model for four structure classes.Source: A. Baselines · pp. 5–6

B. Experimental Setup

  • Experiments use PyBullet with NViSII-rendered point clouds and novel object models from known and unknown categories, including randomly sampled language goals, distractors, and occlusions.Testing objects come from Google Scanned Objects [40], ReplicaCAD, and the YCB Object Set [42], with none appearing in training data.Source: B. Experimental Setup · p. 6, Figure 4 · p. 6
  • Success rate isolates pose prediction by placing objects 3cm above predicted targets and checking physics validity plus all semantic constraints, with approximately 5 constraints per structure type.Failures include physics-related placement problems, collisions, intersections, or violation of the language-defined scene goal.Source: B. Experimental Setup · p. 6

V. SIMULATION EXPERIMENTS

  • The simulator evaluates four structure classes on held-out objects, and success requires every object to be physically valid while the final scene satisfies the language goal.The comparison is summarized in Fig. 5 for unseen objects in the physics simulator.Source: Figure 5 · p. 6

C. Comparison with Other Generative Models

  • StructDiffusion significantly outperformed all baselines across every structure type, with the largest gains on structures requiring precise placements and contact modeling.The generator-discriminator design was necessary because diffusion alone still produced invalid samples, particularly for line structures.Source: C. Comparison with Other Generative… · p. 6, Figure 5 · p. 6
  • Diffusion benefited more from the collision discriminator than StructFormer or CVAE because it models uncertainty at multiple scales and generates diverse placement hypotheses from partial or heavily occluded point clouds.StructFormer produces similar samples because its autoregressive transformer does not explicitly model uncertainty, while CVAE stochasticity comes from a single latent variable.Source: C. Comparison with Other Generative… · p. 7

D. Comparison with Other Iterative Methods · VI. REAL WORLD EXPERIMENTS · A. Perception and Hardware

StructDiffusion outperformed two optimization-based iterative baselines, while unresolved cases involved highly related object placements where discriminator guidance could produce local minima. Real-world deployment used a 7-DoF JACO arm, RGB-D perception, grasp sampling, and motion planning to assemble structures from incomplete point clouds.

VI. REAL WORLD EXPERIMENTS

  • Real-world experiments tested structure assembly on a real robot.Source: §VI · p. 7

A. Perception and Hardware

  • The system deployed a 7-DoF JACO arm with an Asus Xtion RGB-D camera and extracted segmented object point clouds using table detection and Euclidean clustering.Source: A. Perception and Hardware · p. 7
  • Antipodal grasps were pairwise-ranked for most objects, while Contact-GraspNet handled table-setting objects; RRT-Connect planned motion and objects were released 3cm above targets.Source: A. Perception and Hardware · pp. 7–8
  • Real-world structures were predicted from incomplete single-camera point clouds across four broad language-instruction classes.Fig. 8 shows examples for a wide range of real-world objects and instructions.Source: Figure 8 · p. 8

B. Predictions for Real-World Objects · C. Rearrangement with Pick and Place · VII. CONCLUSIONS

StructDiffusion generates physically valid, language-conditioned structures for previously unseen real-world objects and supports robot rearrangement through grasp and motion planning. Its diffusion-based diverse sampling and discriminator filtering improve performance, while real-world execution remains limited by grasping, sensing, and rigid-grasp assumptions.

B. Predictions for Real-World Objects

  • The model generates semantically diverse and physically valid structures for real-world objects despite training exclusively on simulation data.It supports multiple structure variations, language-conditioned structure classes, specified orientations, and correct alignment of nonsymmetrical objects.Source: B. Predictions for Real-World Objects · p. 8

C. Rearrangement with Pick and Place

  • Robot rearrangement combines StructDiffusion with grasp and motion planning, searching ranked predicted structures until all object motion plans are valid before execution.For each candidate, the method samples grasp poses and computes pre-grasp, standoff, and placement waypoints.Source: C. Rearrangement with Pick and Place · p. 8
  • Valid grasp and motion plans are found most of the time for circle, tower, and line structures, whereas table settings remain difficult because grasp regions have small tolerance and sampled grasps are limited.No valid inverse-kinematic solutions occur in some cases.Source: C. Rearrangement with Pick and Place · p. 8, Table I · p. 8
  • Noisy partial point clouds and self-occlusions cause some invalid structure predictions during real-world rearrangement.These issues particularly affect large objects.Source: C. Rearrangement with Pick and Place · p. 8
  • Assuming objects remain rigidly attached to the gripper after grasping causes occasional failures because real-world slippage violates that assumption.The authors suggest predicting postgrasp displacement with learned models such as.Source: C. Rearrangement with Pick and Place · p. 9

VII. CONCLUSIONS

  • StructDiffusion creates structures for previously unseen objects across a range of language instructions using point cloud images, multimodal transformers, and diffusion models.This is the paper’s central contribution.Source: §VII · p. 9
  • StructDiffusion’s diffusion sampling and discriminator filtering significantly improve performance over strong baselines, including the previous state of the art and a conditional variational autoencoder.End-to-end policies underperform because they cannot refine nearly correct placement poses.Source: §VII · p. 9
  • The approach does not address optimal planning, but future work could combine it with task and motion planning for unknown objects and pretrained language models for more natural commands.The authors also aim to extend the method to more structures and complex language instructions.Source: §VII · p. 9

APPENDIX … 4) Using the Large Language Model to Parse Natural Language:

The appendix extends StructDiffusion from individually encoded word tokens to natural-language instructions using pretrained sentence embeddings and language-model prompting. These approaches support varied abstraction levels, unseen sentences, and only a small performance decrease relative to the original input representation.

A. Towards Natural Language:

  • The framework targets natural language without predefined vocabularies, including instructions that specify different abstraction levels such as structure type, size, or location.The appendix motivates combining the model with a pretrained language model to handle natural instructions such as “make a line” and “make a short line at the bottom of the table.”Source: A. Towards Natural Language · p. 12

1) Language Data:

  • Training uses 3,345 procedurally generated sentences formed from 669 token combinations and manually authored templates, then embeds them with all-MiniLM-L6-v2 [52].The generated sentences cover combinations of shape, size, position, and rotation tokens, including some grammatical errors.Source: 1) Language Data · p. 12

2) Model and Training:

  • The diffusion model replaces individual token embeddings with a 384-dimensional sentence embedding projected to the point-cloud embedding dimension.During training, sampled token combinations map to predefined sentence embeddings, allowing ambiguous instructions to correspond to multiple possible structures.Source: 2) Model and Training · p. 12

4) Using the Large Language Model to Parse Natural Language:

  • Prompting techniques [23] [21] provide another route by translating natural-language arrangement descriptions into the original model’s domain-specific word tokens.The appendix presents a GPT-3 prompt that maps instructions to available shape, size, and location values, returning NONE for unspecified elements.Source: 4) Using the Large Language Model to… · p. 12
  • The prompt maps examples such as a small circle, a line at the top, a bottom table setting, and teacups surrounding a teapot to structured tokens.The examples output shape, size, and location fields, using NONE when an attribute is unspecified.Source: 4) Using the Large Language Model to… · p. 12

3) Inference:

  • Performance decreases by within 5% across structure types when using fully specified sentence embeddings instead of the original word-token representation.Table A1 compares simulation success rates for the two language-input types.Source: Table A1 · p. 12, 3) Inference · p. 12
  • The sentence-embedding model generalizes to test sentences absent from rearrangement training, while generated structures preserve specified properties and vary unspecified ones.Figure A1 shows sampled goals from models trained on procedurally generated sentences at different abstraction levels.Source: Figure A1 · p. 12, 3) Inference · p. 12

B. Classifier-Based Collision Guidance

Classifier-based guidance integrates a noisy collision discriminator directly into diffusion sampling. This improves performance for towers and table settings but provides limited gains for other structures.

B. Classifier-Based Collision Guidance

  • Classifier-based guidance improves tower and table-setting performance, while yielding limited gains for other structures.The method integrates a noisy collision discriminator into the diffusion model’s sampling procedure; results are reported in Table A4.Source: B. Classifier-Based Collision Guidance · p. 13

C. Failure Mode Analysis · D. Real World Results

StructDiffusion reduces intersection-related simulation errors, while real-world experiments demonstrate rearrangement of unseen objects despite failures mainly caused by grasping, planning, and execution. Failure rates vary by structure type, motivating environment-aware reasoning and improved task-and-motion integration.

C. Failure Mode Analysis

  • StructDiffusion’s collision discriminator and diffusion model drastically reduce simulation errors caused by object intersections.Compared with CVAE and optimization using a CVAE prior, learned discriminator optimization reduces intersection errors but produces more semantically incorrect structures.Source: C. Failure Mode Analysis · p. 13
  • Circle structures fail more often from abnormal velocity because large circles can place objects outside the table, whereas towers are less affected because objects are packed more closely.Reasoning about environmental properties such as table size is identified as a necessary next step.Source: C. Failure Mode Analysis · p. 13

D. Real World Results

  • Most real-world failures were simple motion-planning or grasping issues rather than failures of StructDiffusion itself.The reported remedy is better integration with task-and-motion planning algorithms and reactive replanning.Source: D. Real World Results · p. 13, Figure A2 · p. 14
  • Additional real-world success cases show StructDiffusion rearranging previously unseen objects that were absent from training data.These examples support generalization to objects not encountered during training.Source: D. Real World Results · p. 13, Figure A3 · p. 15
  • Collision classifier guidance shows no clear advantage in real-world task success rates.The table evaluates noisy classifier guidance across each provided task.Source: Table A4 · p. 13
  • Real-world failure examples indicate that only a minority of failures come from StructDiffusion, while grasping, planning, and execution failures dominate.This motivates integrating the method with task-and-motion planning or reactive replanning.Source: D. Real World Results · p. 13, Figure A2 · p. 14
  • 11 of 20 real-world experiments succeeded on unseen objects without real-world training data; most failures came from grasping and motion planning.StructDiffusion was not trained on any real-world training data.Source: D. Real World Results · p. 13, Figure A3 · p. 15
Loading 2211.04604v2…