PaperlayerView exact revision on arXiv

Source-linked AI summary

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Weiyu Liu, Chris Paxton, Tucker Hermans, Dieter Fox

arXiv:2110.10189v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robots must infer object configurations from language and jointly handle geometric, semantic, and physical constraints, while prior work had not demonstrated arranging complex structures such as circles or table settings. StructFormer uses transformer-based object selection and autoregressive pose generation to convert a partial-view point cloud and structured language instruction into rearrangement actions. The full pipeline achieved 58/156 (37%) overall; with groundtruth object selections, StructFormer consistently outperformed Binary on all four structures.

  • Problem

    Robots must infer object configurations from language and jointly handle geometric, semantic, and physical constraints, while prior work had not demonstrated arranging complex structures such as circles or table settings.

    Source: §I · p. 1

  • Method

    StructFormer uses transformer-based object selection and autoregressive pose generation to convert a partial-view point cloud and structured language instruction into rearrangement actions.

    Source: §IV · p. 3, C. Language Conditioned Pose Generator · p. 4

  • Results

    The full pipeline achieved 58/156 (37%) overall; with groundtruth object selections, StructFormer consistently outperformed Binary on all four structures.

    Source: B. Evaluating Full System in Simulation · p. 6

  • Takeaways & Limitations

    StructFormer’s architecture is reported to benefit multi-object semantic rearrangement, including reasoning over varying numbers of objects without object models.

    Source: §I · p. 2, §VII · p. 6

    The method uses structured rather than natural language, does not address placement in clutter or optimal action ordering, and always builds structures in a predefined order.

    Source: §VII · p. 6

Abstract

from arXiv · show

Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without significant programming. While previous works have examined recognizing pairwise semantic relations and sequential manipulation to change these simple relations none have shown the ability to arrange objects into complex structures such as circles or table settings. To address this problem we propose a novel transformer-based neural network, StructFormer, which takes as input a partial-view point cloud of the current object arrangement and a structured language command encoding the desired object configuration. We show through rigorous experiments that StructFormer enables a physical robot to rearrange novel objects into semantically meaningful structures with multi-object relational constraints inferred from the language command.

I. INTRODUCTION

Semantic rearrangement asks robots to arrange novel objects into structures specified by high-level language, despite the need to reason jointly about geometry, semantics, physical interactions, and distractors. StructFormer addresses this with transformer-based multi-object reasoning, and the paper evaluates it on generated structures, novel objects, and a physical robot.

I. INTRODUCTION

  • Semantic rearrangement targets high-level instructions such as putting mugs in a row, building a circle, or setting a table.The introduction motivates the task with applications including setting tables, organizing books, and loading dishwashers.Source: §I · p. 1
  • Robots must satisfy geometric and semantic constraints while accounting for physical interactions, distractors, and language instructions that must be translated into robot-interpretable goals.These requirements arise when rearranging previously unseen objects in unstructured environments.Source: §I · p. 1
  • StructFormer explicitly reasons about multi-object semantic relations from structured language, using transformer encoders and an autoregressive decoder to predict objects to move and their destinations.The model handles varying numbers of objects without access to object models.Source: §I · p. 2
  • The authors procedurally generate circles, lines, towers, and table settings from models of 335 real-world objects, then test novel objects and arrangements in simulation and on a physical robot.They report higher manipulation success than a model reasoning only about pairwise object relations.Source: §I · p. 2

II. RELATED WORK

Prior work includes pairwise spatial-relation reasoning, object-centric visual reasoning, and 3D structure synthesis; this paper distinguishes active object rearrangement guided by language. The transformer preliminary describes attention and encoder-decoder components used for sequential modeling.

II. RELATED WORK

  • Prior spatial-relation work primarily models relations between pairs, while this work models relations among multiple objects and arranges them from language instructions.Other approaches chain pairwise goals or use spatial relations as a planning abstraction.Source: §II · p. 2
  • Visual reasoning benchmarks study compositional structures among scene elements, whereas StructFormer actively manipulates objects toward language-specified arrangements.Related 3D synthesis methods use holistic generation or structure-aware substructures; StructFormer conditions object arrangements on high-level language instead of object-part priors.Source: §II · p. 2

III. TRANSFORMER PRELIMINARY

  • Scaled dot-product attention maps an input sequence to an output sequence of the same length, weighting values according to query-key compatibility.The attention function computes over a set of queries simultaneously with matrix multiplication.Source: §III · p. 2
  • Transformer encoders combine attention with position-wise feed-forward layers, while decoder layers add encoder attention and can use causal masks for autoregressive prediction.The causal mask prevents predictions from relying on future inputs.Source: §III · p. 2

IV. StructFormer FOR OBJECT REARRANGEMENT

The task is to transform a partial-view, segmented point cloud and structured language instruction into physically valid poses for the objects selected for rearrangement. StructFormer uses object selection and pose generation networks to produce a sequence of rearrangement actions.

IV. StructFormer FOR OBJECT REARRANGEMENT

  • The system takes object point clouds and a language instruction, then outputs a sequence of rearrangement actions intended to achieve the instructed configuration.The figure presents the object-selection and pose-generation components.Source: Figure 2 · p. 3
  • Given a single scene view and structured instruction, the task is to move a subset of objects so the goal scene satisfies the instruction’s spatial and semantic constraints and remains physically valid.The robot receives a partial-view point cloud with segment labels identifying objects.Source: §IV · pp. 2–3
  • StructFormer combines an object selection network with a pose generator, both using transformers to reason jointly about object point clouds and language.The selection encoder predicts which objects to move; the pose decoder autoregressively predicts their pose offsets.Source: §IV · p. 3

A. Object and Sentence Encoders

StructFormer encodes object point clouds and language tokens into a shared sequence for transformer reasoning. Its selection network identifies objects to move, and its pose generator predicts ordered pose offsets, including placements relative to a virtual structure frame.

A. Object and Sentence Encoders

  • Object and word tokens receive learned latent representations, with position and type embeddings distinguishing their sequence positions and whether each token represents an object or a word.The object encoder is based on the permutation-invariant point cloud transformer model.Source: A. Object and Sentence Encoders · p. 3

B. Object Selection Network

  • The object selection network uses a transformer encoder over scene objects and instruction words to predict a binary move-or-not indicator for each object.Its relational inference can identify objects described by properties such as being smaller than a pan.Source: B. Object Selection Network · p. 3

C. Language Conditioned Pose Generator

  • The pose generator encodes the scene and instruction, then autoregressively predicts pose offsets conditioned on global context and previously predicted offsets.A causal mask and shifted inputs prevent the decoder from using future object poses; dropout provides stochasticity during training and inference.Source: C. Language Conditioned Pose Generator · pp. 3–4
  • Each 6-DoF pose offset contains translation and rotation; object positions are represented in a virtual structure frame, whose pose is predicted in the world frame.The rotation representation uses two predicted vectors and a Gram–Schmidt-like construction; the authors report it has no discontinuities and facilitates learning accurate placement poses.Source: C. Language Conditioned Pose Generator · p. 4
  • A predefined query-object order, such as clockwise for a circle, and a virtual frame empirically help create precise spatial structures.Source: C. Language Conditioned Pose Generator · p. 4

D. Inference and Training

At inference, StructFormer selects objects to rearrange, then samples and autoregressively predicts target poses for the structure frame and objects. Training uses binary cross-entropy for object selection and an L2 pose-placement loss for the generator.

D. Inference and Training

  • At inference, the system selects query objects, samples B rearrangements, then autoregressively predicts the structure-frame and object poses conditioned on earlier predictions.The selected objects come from the object-selection network; pose samples come from the generator.Source: D. Inference and Training · p. 4
  • The object-selection network is trained with binary cross-entropy, while the pose generator minimizes L2 distance between ground-truth and predicted placement poses.Source: D. Inference and Training · p. 4

V. DATA GENERATION

The dataset pairs language instructions with rearrangement sequences generated through procedural scene construction. Its pipeline selects query objects, places them in one of four structures, then reverses randomized actions to create rearrangement sequences.

V. DATA GENERATION

  • The dataset contains more than 100,000 language-paired rearrangement sequences, procedurally generated in PyBullet and rendered with NVISII using 335 household objects.The objects span 35 classes.Source: §V · p. 4
  • The pipeline samples referring expressions for query objects, then adds optional anchors and distractors; table settings omit referring-expression generation.Expressions can specify object features or relate query objects to an anchor by semantic or geometric properties.Source: §V · p. 4
  • Query objects are arranged into circles, lines, towers, or table settings using geometric parameters that are discretized into language descriptions.Parameters include radius, size, position, and rotation.Source: §V · p. 4
  • Objects are moved from goal arrangements to random collision-free poses, and reversing those actions produces sequences from random poses to the language-specified goals.NVISII renders color, depth, and instance-segmentation images for the sequences.Source: §V · p. 4

VI. EXPERIMENTS

The experiments evaluate individual model components, the full planning system, and physical-robot execution with real-world objects and sensing. The pose-generator table reports smaller translational and rotational errors for four structures.

VI. EXPERIMENTS

  • The evaluation proceeds from component testing to full-system planning performance and physical-robot execution with real-world objects and sensing.Source: §VI · pp. 4–5
  • The pose generator produces smaller translational and rotational errors of predicted object poses for four structures.Source: Table II · p. 5
  • Figure 4 visualizes predicted circular arrangements from the pose generator and baselines across different object counts and geometries.The caption describes the model's structures as precise.Source: Figure 4 · p. 5

A. Model Component Testing

StructFormer's pose generator outperforms the tested baselines on placement precision across four structures. Component tests compare global-context and virtual-structure-frame variants with a pairwise-relation baseline.

A. Model Component Testing

1) Language Conditioned Pose Generator:

  • Binary predicts placements iteratively from language, the current object, and the previously moved object as a spatial anchor.No Encoder omits the transformer encoder's global context, while No Struct predicts offsets directly in the world frame.Source: 1) Language Conditioned Pose Generator · p. 5
  • StructFormer outperforms all baselines in precise object placement across four structures, and visualizations show qualitative improvements in predicted arrangements.Source: 1) Language Conditioned Pose Generator · p. 5
  • No Struct produces larger errors and less consistent alignments, while the virtual structure frame anchors object placements.The authors hypothesize that separating structure and object placement also helps handle language ambiguities such as locating a circle on a table.Source: 1) Language Conditioned Pose Generator · p. 5
  • The No Encoder and Binary baselines inadequately produce circles because they cannot condition a placement on future objects, including their number and dimensions.Source: 1) Language Conditioned Pose Generator · p. 5

2) Object Selection Network:

The object selection network retrieves objects described by direct or relational references, including continuous-valued properties, while maintaining high accuracy as the number of target objects increases.

2) Object Selection Network:

  • The network retrieves objects from both direct and relational references and handles continuous-valued properties such as height and volume.The property-type chart organizes F1 scores by referring-expression property, while the other chart groups scores by target-object count.Source: 2) Object Selection Network · pp. 5–6, Figure 5 · p. 6
  • F1 scores are organized by referring-expression property type and by the number of objects identified.The top chart groups by associated property type; the bottom orders results by target count.Source: Figure 5 · p. 6
  • The network maintains high accuracy as the number of target objects increases.The figure’s lower chart orders performance by the number of objects identified in the referring expression.Source: 2) Object Selection Network · p. 6, Figure 5 · p. 6

B. Evaluating Full System in Simulation

In simulation, StructFormer’s pose generator outperformed the Binary baseline on both complex and pairwise structures. The full pipeline succeeded in 37% of tested scenes, with tower construction especially challenging.

B. Evaluating Full System in Simulation

  • The simulation used 138 novel object models from 23 known classes, preserved physical interactions, and used ground-truth instance segmentation.The evaluation omitted low-level robot control and simulated placement by dropping moved objects from 3 cm above the predicted target z value.Source: B. Evaluating Full System in Simulation · p. 6
  • 37% of scenes succeeded end to end, while all query objects were identified in 61% and at least 70% of specified objects in 83%.The whole pipeline built circles, lines, and towers in 40/61 (66%), 40/61 (65%), and 10/34 (29%) of scenes, respectively; table settings were excluded.Source: B. Evaluating Full System in Simulation · p. 6
  • Tower construction was especially challenging because some tested objects, including apples and teapots, could not inherently be stacked.Another reported failure mode was incompatibility between structure parameters and objects, such as large pans assigned to a small circle.Source: B. Evaluating Full System in Simulation · p. 6
  • 58/112 (52%) more complex-relation structures and 14/95 (15%) pairwise-relation structures were built than with Binary.The comparison used ground-truth object selections; Fig. 7 compares success rates for semantically correct, physically valid structures.Source: Figure 7 · p. 6, B. Evaluating Full System in Simulation · p. 6

C. Physical Robot Experiments

The system was deployed on a Franka Panda robot with an arm-mounted RGB-D camera. The paper identifies perception and planning constraints and concludes that direct natural-language operation and non-predefined rearrangement order remain open.

C. Physical Robot Experiments

  • The physical robot setup used a Franka Panda with an arm-mounted RGB-D camera, grasps from, and RRT-connect motion planning.Successful rearrangements are visualized in Fig. 1.Source: C. Physical Robot Experiments · p. 6
  • Occluded objects can be placed intersecting other objects, and motion planning sometimes fails when objects are unreachable.The system does not use sophisticated planning; candidate goal scenes could be combined with vision-based planners in.Source: C. Physical Robot Experiments · p. 6

VII. CONCLUSIONS

  • The conclusion presents a transformer that generates plans as sequences from scene point clouds and language commands, but direct natural-language operation remains open.The method uses structured language to specify structure parameters, builds in a predefined order, and does not address placement in clutter or optimal action ordering.Source: §VII · p. 6

Related papers

Loading 2110.10189v1…