PaperlayerView exact revision on arXiv

Source-linked AI summary

Relational Learning for Skill Preconditions

Mohit Sharma, Oliver Kroemer

arXiv:2012.01693v1cs.ROcs.CVcs.LG

TL;DR

Robots need skill-precondition models that generalize across variable object sets and learn from few examples. This paper learns continuous pairwise object relations from simulated interactions and uses them in a separate real-world precondition model. The authors report direct transfer of the simulation-trained representations without sim2real adaptation.

  • Problem

    Skill-precondition models need to generalize across variable numbers and types of objects, while learning from few real-world examples is important because collecting large datasets for every skill is impractical.

    Source: §1 · p. 1

  • Method

    The approach learns continuous pairwise relation embeddings from simulated object interactions and supplies them to a separate real-world precondition model.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    The authors report that simulation-trained relation representations can be used directly for real-world manipulation precondition learning without specific sim2real adaptation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Interaction-grounded continuous relations provide a shared representation for precondition models without requiring a fixed set of object relations.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

To determine if a skill can be executed in any given environment, a robot needs to learn the preconditions for the skill. As robots begin to operate in dynamic and unstructured environments, precondition models will need to generalize to variable number of objects with different shapes and sizes. In this work, we focus on learning precondition models for manipulation skills in unconstrained environments. Our work is motivated by the intuition that many complex manipulation tasks, with multiple objects, can be simplified by focusing on less complex pairwise object relations. We propose an object-relation model that learns continuous representations for these pairwise object relations. Our object-relation model is trained completely in simulation, and once learned, is used by a separate precondition model to predict skill preconditions for real world tasks. We evaluate our precondition model on $3$ different manipulation tasks: sweeping, cutting, and unstacking. We show that our approach leads to significant improvements in predicting preconditions for all 3 tasks, across objects of different shapes and sizes.

1 Introduction

The paper targets skill-precondition models that can handle variable object sets and learn from few examples. It proposes learning continuous pairwise relations from simulated interactions for use in real-world precondition learning.

1 Introduction

  • Precondition models must generalize across changing numbers and types of objects, while learning from few real-world examples.The paper gives food cutting across apples and carrots as an example.Source: §1 · p. 1
  • The authors propose learning continuous object-relation representations from simulated interactions rather than assuming a fixed relation set.Interaction effects such as contacts and pose changes provide labels for a contrastive loss.Source: §1 · p. 1
  • Simulation-trained representations are used for real-world manipulation precondition learning without specific sim2real adaptation.Source: §1 · p. 2

2 Related Works

Prior work learns discrete object relations or continuous relations over predefined categories. The paper distinguishes its approach by grounding continuous representations in object interactions rather than fixed categories or binary scene-similarity labels.

2 Related Works

  • Discrete relation categories can fail to distinguish scenes that share a label such as “above” but have different semantics.Source: §2 · p. 2
  • Prior continuous-relation approaches use predefined relation sets and binary similarity labels, which can group scenes with different object distances alike.Those approaches use scenes built from known object templates and predefined arrangements.Source: §2 · p. 2
  • The proposed alternative learns continuous relations from simulated pairwise interactions, incorporating geometry, contact, and interaction dynamics rather than position features alone.Earlier position-feature work is described as limited for tasks depending on object size, orientation, geometry, and contact distributions.Source: §2 · p. 2

3 Approach

The approach learns continuous pairwise relation embeddings from simulated object interactions, then uses those embeddings as input to a separate real-world precondition model.

3 Approach

  • Scenes are decomposed into objects and continuous pairwise relations, represented by a voxel-input function trained from simulated interaction effects with a contrastive formulation.The relation model is not fine-tuned when its embeddings are used by the separate real-world precondition model.Source: §3 · p. 3

3.1 Generating Pairwise Interactions In Simulation

  • Pairwise interactions are generated by holding an anchor object static and perturbing a second object in simulation, with fixed-magnitude or distance-adaptive actions.The referrant object is sampled within 0.5 meter of the anchor and rotated around the z-axis within (−π/6, π/6) radians.Source: §3.1 · p. 3

3.2 Learning Object Relations

  • The relation model uses triplet losses to group scenes with similar position and orientation effects, alongside prediction losses for displacement, orientation, and contact distributions.Source: §3.2 · pp. 4–5

3.3 Learning Precondition Models

  • For precondition learning, the model computes embeddings for all ordered object pairs and supplies them as graph-edge information to relational-network or graph-neural-network classifiers.The described GNN uses bidirectional edges and classifies pooled node and edge embeddings.Source: §3.3 · p. 5

4 Experiments

The experiments assess precondition learning across sweeping, food cutting, and block unstacking, including changes in object count, shape, and size. Relational models outperform the compared baselines across the reported tasks, with performance affected by training coverage and model design.

4 Experiments

  • The experiments evaluate F1 and weighted F1 across three manipulation tasks, using learning and non-learning baselines and leaving the learned relation model unfine-tuned during precondition learning.The reported tasks are sweeping, food cutting, and block unstacking; real2sim is included as a baseline where simulator task support allows.Source: §4 · pp. 5–6

4.1 Sweeping Objects in Line

  • 0.96 and 0.95 weighted F1 scores were achieved by the two relation models for sweeping, outperforming the baselines across the reported test splits.Learning baselines fell to weighted F1 scores of 0.83 and 0.66 on splits involving six objects or different shapes and sizes.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Food Cutting

  • 0.960 and 0.940 weighted F1 scores were reported for the relational precondition models in food cutting, which outperformed all baselines across the train-test splits.With one distractor in training, the maximum weighted F1 was 0.804; training with up to two distractors improved tests with three or four distractors.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Block Unstacking

  • GNN-based models outperformed the other methods across block-unstacking scenarios, while a 3D CNN baseline weakened on objects with different sizes.The 3D CNN baseline had weighted F1 of approximately 0.80 on the first two splits; the passage attributes its weaker transfer to overfitting to training objects.Source: §4.3 · pp. 7–8
  • The real2sim baseline had weighted F1 of approximately 0.70 on the first two unstacking splits, while the relational network performed much worse than the corresponding GNN models.The passage attributes real2sim errors to noisy voxel representations of close or contacting blocks, and suggests the one-layer relational network has less capacity than the two-layer GNNs.Source: §4.3 · p. 8

5 Ablation Study

The ablations compare simpler voxel inputs and contact-based losses with learned object relations, reporting their effects on weighted F1.

5 Ablation Study

  • Mean 3D position alone yields a weighted F1-score of 0.72, while adding object bounds yields a weighted F1-score of 0.816.The learned-relation GNN still outperforms these inputs.Source: §5 · p. 8
  • Contact-based losses increase the block-unstacking weighted F1-score from 0.80 to 0.824.The authors relate this task to inferring contact locations for stability prediction.Source: §5 · p. 8
  • Learned embeddings perform well despite sensory noise and limited data, while object bounds are sensitive to outliers.The authors identify outlier sensitivity as one reason bounds underperform learned relations.Source: §5 · p. 8

6 Conclusion

The conclusion presents continuous, interaction-grounded object relations as a compositional basis for learning manipulation preconditions.

6 Conclusion

  • The approach learns continuous representations of object relations grounded in robot interactions, without assuming a fixed set of discrete relations.Scenes are decomposed into objects and their relations.Source: §6 · p. 8
  • Simulation provides large sets of ground-truth data, including contact distributions, for learning rich relation representations.Source: §6 · p. 8
  • The learned approach can be used directly for sample-efficient precondition learning.Source: §6 · p. 8

Appendices

The appendix describes how simulated object-pair scenes, perturbations, and a virtual robot are configured.

Appendices

B Experimental Setup

The appendix specifies real-world scene capture, voxel input formats, object-relation inputs, and simulator-based baselines.

B Experimental Setup

C Input Format

  • Model and baseline inputs have size (C, 100, 100, 100), where C is the number of channels.Source: C Input Format · p. 11

C.1 Visual Baseline

  • Visual baselines use an occupancy-mask channel or add object-identity labels, with task-specific labels for cutting and unstacking.For cutting, knife and food labels are used when present and other objects receive a distractor label; for unstacking, the target block is distinguished from the rest.Source: §C.1 · p. 11

C.2 Object Relation Model

  • The relation model takes three voxel-mask channels and centers the anchor object at (0, 0, 0) when processing real-world object pairs.Source: §C.2 · p. 11

D.1 DiscreteRel

  • DiscreteRel divides space around each anchor into 26 regions and assigns the referent the region with the most intersecting voxels.Source: §D.1 · p. 12

D.2 Real2Sim

  • The real2sim baseline imports voxel representations into V-REP, while the sim2real baseline is limited to sweeping and unstacking because V-REP cannot support cutting.For unstacking, stability is checked by allowing blocks to interact and measuring movement; thresholds of 0.0015m and 0.002m gave similar best results.Source: §D.2 · pp. 12–13

E Architecture Details

The section specifies visual baselines and relational precondition models, then describes the object-relation model’s network architecture. The RN aggregates pairwise relations, while the GNN updates node and edge representations.

E.1 Baselines

  • The ResNet and VGG-style visual baselines process 3D inputs and map their convolutional features through fully connected layers to a scalar output.The ResNet baseline omits BatchNorm and uses the same stride for all three input channels in its first convolution; the VGG-style model has six convolution layers.Source: §E.1 · p. 13

E.2 Precondition Learning Model

  • The RN concatenates embeddings for both directions of each object pair, processes them with fθ, and passes their aggregate through gφ.Both functions use two fully connected layers with ReLU between layers.Source: §E.2 · p. 13
  • The GNN has separate edge and node models: nodes sum connected edge embeddings, combine that sum with the original node input, and update representations across two layers.The final node and edge embeddings are summed and concatenated before fully connected layers produce the output.Source: §E.2 · p. 13

E.3 Object Relation Model

  • The 3D ResNet object-relation model outputs a 256-dimensional embedding, then predicts position changes, orientation changes, and mean contact position alongside the action vector.Its final prediction layer has size 9, comprising three values for each predicted quantity.Source: §E.3 · p. 14

F Training Details

Training uses simulation-generated interaction data to learn object-relation embeddings, then trains precondition models and visual baselines separately. The object-relation model combines contrastive and supervised losses, with specified action-sampling and comparison rules.

F.1 Object Relation Model

  • The object-relation model combines position- and orientation-based contrastive losses with supervised prediction, orientation, and contact losses.The supervised-loss weights are λpos = 1, λorient = 10, and λcontact = 1; the contrastive orientation weight is 2.Source: §F.1 · p. 14
  • Contrastive training compares all scene triplets in a batch, separates normal from adaptive action comparisons, and excludes normal actions whose magnitudes differ by more than 0.04 m.For position effects, scenes are similar when action-effect differences stay below the specified threshold for all actions.Source: §F.1 · pp. 14–15

F.2 Precondition Learning Models

  • The precondition models use Adam with a 3e-4 initial learning rate and batch size 8, while baseline results use the best settings from learning-rate and batch-size searches.Baselines test learning rates from 1e-3 to 1e-4 and batch sizes 4, 8, and 16.Source: §F.2 · p. 15

G Task Setup

The task setup includes food cutting with varied foods and obstacles, and block unstacking with varied block-stack configurations. In an additional cutting test with more distractors than seen during training, the GNN performs best, while labeled food and knife inputs help visual baselines.

G Task Setup

  • Food-cutting scenes use varied food items and obstacles, label food and knife voxels separately, and contain up to six objects.A custom 3D-printed tool holder carries an embedded knife grasped by the robot.Source: G Task Setup · p. 16
  • The block-unstacking dataset contains scenes with three to seven blocks, with 10–20 configurations per block set and manual labels for blocks crucial to stability.An additional GNN connects only blocks within 0.1 m.Source: G Task Setup · p. 16

H.1 Cutting Food

  • With three test distractors after training on at most two, the GNN achieves weighted F1-score 0.94, while the best baseline reaches 0.902.The authors attribute the baselines’ strong cutting performance to separate food and knife labels, which let them focus on the two main objects despite distractors.Source: §H.1 · p. 16

H.2 Block Unstacking

Block unstacking results show that generalization from larger training scenes to smaller test scenes is not automatic, even for structured models. Examples also illustrate why discrete relations can fail to distinguish scenes with different stability outcomes.

H.2 Block Unstacking

  • Training on six or seven blocks and testing on four yields a maximum GNN weighted F1-score of 0.825, below the Real2Sim baseline’s 0.889 but above visual baselines’ maximum of 0.784.The authors caution that training on more objects does not guarantee generalization to fewer; they attribute Real2Sim’s result partly to the limited variety of four-block test stacks and fewer perception errors.Source: §H.2 · p. 17
  • In example scenes differing only in top-block position, the relational precondition model predicts both labels correctly while the visual baseline predicts false for both.The authors present this as an example of complex reasoning using learned object-relational embeddings.Source: §H.2 · p. 17
  • With only three objects, scenes can have similar discrete relations but different stability labels; the authors report that ambiguity increases as scene object counts grow.In one pair, removing the black block leaves one stack stable but causes the orange block in the other to fall.Source: §H.2 · p. 17
Loading 2012.01693v1…