PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

Richard Li, Allan Jabri, Trevor Darrell, Pulkit Agrawal

arXiv:1912.11032v1cs.ROcs.AIcs.LG

TL;DR

Sparse-reward manipulation tasks require substantial data, and curricula can be difficult to exploit when task changes alter the agent’s inputs. This paper combines an attention-based graph policy with an increasing-block curriculum; it learns six-block stacking efficiently and generalizes without further training to taller towers and pyramids.

  • Problem

    Sparse rewards simplify reward definition, but long action sequences make exploration and credit assignment difficult, often causing failure or massive data requirements; adding objects can also shift inputs and impede transfer across curriculum tasks.

    Source: §I · pp. 1–2

  • Method

    The paper trains an attention-based graph neural network policy with a curriculum that increases the number of blocks as the agent masters simpler tasks.

    Source: §I · p. 2

  • Results

    75% success at stacking 6 blocks in 30 million timesteps, versus 32% for the demonstration-and-reset method [4] after over 2.3 billion timesteps; the policy also stacks taller towers and pyramids without further training.

    Source: §I · p. 2, §VI · p. 5

  • Takeaways & Limitations

    The learned policy generalizes zero-shot to taller towers and previously unseen configurations such as pyramids.

    Source: §I · p. 2

Abstract

from arXiv · show

Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring knowledge from simpler to complex tasks. We show that graph-based relational architectures overcome this limitation and enable learning of complex tasks when provided with a simple curriculum of tasks with increasing numbers of objects. We demonstrate the utility of our framework on a simulated block stacking task. Starting from scratch, our agent learns to stack six blocks into a tower. Despite using step-wise sparse rewards, our method is orders of magnitude more data-efficient and outperforms the existing state-of-the-art method that utilizes human demonstrations. Furthermore, the learned policy exhibits zero-shot generalization, successfully stacking blocks into taller towers and previously unseen configurations such as pyramids, without any further training.

I. INTRODUCTION

Sparse-reward manipulation demands substantial data, and curricula can fail when agents cannot transfer knowledge across tasks with changing inputs. The paper uses an attention-based graph neural network with an increasing-block curriculum to learn multi-object stacking and generalize to new configurations.

I. INTRODUCTION

  • Sparse rewards are easier to define than dense rewards, but long action sequences make exploration and credit assignment difficult, often causing failure or massive data requirements.Manipulation rewards must be manually constructed, and intuitively reasonable dense rewards can produce undesirable behavior.Source: §I · p. 1
  • Adding a block changes the agent’s input distribution, and without suitable inductive biases it can treat the new distribution as a new learning problem.The agent then cannot efficiently leverage knowledge from earlier tasks.Source: §I · p. 2
  • The agent learns to stack six or more blocks from scratch without expert demonstrations, with orders-of-magnitude greater data efficiency than the demonstration-based state-of-the-art method [4].The learned policy also stacks taller towers and pyramids without further training.Source: §I · p. 2
  • An attention-based graph neural network supplies an inductive bias for transferring knowledge across tasks with different numbers of objects.The curriculum increases the number of blocks after the agent masters a task with fewer blocks.Source: §I · p. 2
  • The policy generalizes without further training to towers taller than those seen in training and configurations such as pyramids.The authors state that their approach makes no task-specific assumption and is applicable to a wide range of multi-object manipulation tasks.Source: §I · p. 2

II. RELATED WORK

Prior work has used relational representations, curricula, and engineered solutions for robotic block stacking, but the paper combines graph-based policies with a simple curriculum for complex manipulation.

Relational Inductive Bias

  • Relational representations can support variable numbers of objects and explicitly model relationships, while prior GNN policy work studied transfer across agents with different morphologies.The paper frames its contribution as combining GNNs with a curriculum for long-horizon manipulation tasks.Source: §II · p. 2

Curriculum Learning

  • Earlier GNN manipulation studies relied on many expert demonstrations or focused on video games, while curriculum research had not produced automatic curricula of the considered complexity.The authors do not propose a new curriculum-selection algorithm; they demonstrate that graph-based representations can use a curriculum to learn complex tasks.Source: §II · pp. 2–3

Block Stacking

  • Prior block-stacking approaches relied on demonstrations, reward engineering, carefully designed curricula, dense rewards, or substantial human-defined knowledge or simulation assumptions.The paper presents its approach as making minimal assumptions about task structure or the environment.Source: §II · p. 3

III. EXPERIMENTAL SETUP

The experimental setup uses a simulated Fetch arm to manipulate variable numbers of blocks toward goal positions, with step-wise sparse rewards. The paper also gives the standard off-policy Q-learning formulation used as background.

III. EXPERIMENTAL SETUP

  • The simulated task uses a 7-DoF Fetch arm to manipulate 1–9 blocks, with 4D actions and observations containing gripper, block, and goal features.Each block is represented by a 15-D feature vector; episodes last at most 50*N steps.Source: §III · p. 3
  • The robot receives step-wise sparse reward when it places a block within δ of its goal, with an added penalty encouraging the gripper to move away from a fully stacked tower.The threshold is δ = 5 cm, and the penalty applies when the hand is more than 2δ from the tower.Source: §III · p. 3

A. Reinforcement Learning

  • The background RL formulation seeks a policy maximizing expected discounted cumulative reward and describes off-policy Q-learning as learning a Q-function through Bellman-error minimization.The Q-function is approximated by a neural network with a slowly updated target network.Source: A. Reinforcement Learning · p. 3

B. Goal-Conditioned RL

Goal-conditioned policies take a goal state as input to support learning across tasks. The GNN section describes message passing that updates node features using attention-weighted messages.

B. Goal-Conditioned RL

  • For multiple tasks, a goal-conditioned policy takes both the state and sampled goal state as input.The goal is sampled from a goal distribution.Source: B. Goal-Conditioned RL · p. 3

C. Graph Neural Networks (GNN)

  • A GNN message-passing module processes a variable-size vertex set, with each vertex receiving messages from other vertices and updating its feature representation.After multiple rounds, attention weights messages by relevance and node features can be pooled into a fixed-sized graph embedding.Source: C. Graph Neural Networks (GNN) · pp. 3–4
  • In attention-based message passing, vertices generate queries, keys, and messages, and attention weights determine the weighted incoming-message update.The passage describes multiple rounds of message passing.Source: C. Graph Neural Networks (GNN) · p. 4

V. METHOD

The method combines relational graph-based policy representations with a curriculum that increases task complexity to learn long-horizon, sparse-reward manipulation. It uses goal-conditioned SAC with hindsight experience replay and compares ReNN against an MLP across three curricula.

V. METHOD

  • The agent uses Soft Actor-Critic (SAC) with relational inductive biases to learn long-horizon, sparse-reward tasks from a curriculum of increasing complexity.The authors state that their contribution is the relational bias for curriculum learning, not a new curriculum-selection heuristic; the approach is applicable to other policy-learning methods.Source: §V · p. 4
  • Both the actor and critic use the graph neural network architecture, called ReNN, with three rounds of message passing and 64D linear layers.The architecture uses separate weights per message-passing round, with residual connections and layer normalization between rounds.Source: §V · p. 4
  • The MLP baseline uses four fully connected layers of 256D, while the ReNN uses graph-based representations for both actor and critic.The comparison tests whether the relational architecture supports learning across tasks with different numbers of objects.Source: §V · p. 4
  • The direct curriculum trains on six blocks from scratch, while the uniform curriculum samples between one and six blocks each episode.The sequential curriculum progresses from single-block placement to stacking two through six blocks, increasing the target after the preceding task is mastered.Source: §V · p. 4
  • Only ReNN trained with the sequential curriculum succeeds at stacking six blocks; both architectures fail when trained directly on six blocks.The figure reports mean blocks stacked per timestep and standard deviation across multiple workers and/or seeds.Source: Figure 2 · p. 4

A. Testing Details

The testing details define evaluations for single towers, multiple towers, and pyramids. Policies trained to stack a single tower are tested on these configurations without further training.

A. Testing Details

  • The single-tower test samples a table point as the base and sets block goals by translating upward along the z-axis.The testing setup includes taller towers than those used during training.Source: A. Testing Details · p. 4, Figure 3 · p. 5
  • The multiple-tower test samples two or three table bases and randomly assigns blocks to towers of approximately equal height.The policies are evaluated without further training.Source: A. Testing Details · p. 4, Figure 3 · p. 5
  • The pyramid test uses a uniformly sampled table point as a corner, and Figure 3 reports zero-shot results for pyramid configurations.Across the tested configurations, the figure reports that the robot can generalize to many tasks without further training.Source: A. Testing Details · p. 5, Figure 3 · p. 5

VI. RESULTS

The method achieves a 75% success rate stacking six blocks in 30 million timesteps, compared with 32% for a demonstration-based method after over 2.3 billion timesteps. ReNN with sequential training succeeds where direct training and standard MLP or uniform-curriculum variants fail.

VI. RESULTS

  • 75% success rate at stacking 6 blocks in 30 million timesteps, versus 32% after over 2.3 billion timesteps for the demonstration-based method [4].Table I compares accuracy after environment steps; the authors report their method as more sample efficient and higher-performing.Source: Table I · p. 5, §VI · p. 5
  • ReNN with sequential training succeeds at stacking six blocks, while direct training with either ReNN or MLP and uniform curriculum training are insufficient.The authors conclude that both ReNN and sequential training are critical for success.Source: §VI · p. 5
  • Most performance dips after increasing task complexity are overcome with little additional experience, but the 9M-step transition from one to two blocks is a notable exception.This transition is the first time the agent observes multiple objects.Source: §VI · p. 5

A. Zero-shot Generalization

The learned policy generalizes without fine-tuning to taller towers and unseen block configurations, though performance drops on harder tasks. The authors report generalization to pyramids and emergent manipulation strategies, while noting room to improve zero-shot performance.

A. Zero-shot Generalization

  • A policy trained on a single tower of N blocks stacks N + 1 blocks without training, but performance drops significantly for N + k when k > 1.The authors suggest that stabilizing taller towers becomes progressively harder and may require substantial strategy refinement.Source: A. Zero-shot Generalization · p. 5
  • The policy stacks multiple towers of N + k blocks, with performance dropping for k > 2 and better generalization for k ≥ 2 than on single towers.The authors attribute the single-tower difficulty to stabilizing taller stacks.Source: A. Zero-shot Generalization · pp. 5–6
  • Without pyramid training, the system generalizes to larger pyramid configurations; Single Tower 4 performs better on Pyramid 5 and Pyramid 6 than Single Tower 6.The authors suggest that training on taller towers may overfit the policy to vertical stacking, which is less useful for pyramid placement.Source: A. Zero-shot Generalization · p. 6
  • The agent automatically learns behaviors including pushing blocks to enable grasping and grasping two blocks before placing them one by one.The authors say these strategies emerge from optimizing a sparse reward function.Source: A. Zero-shot Generalization · p. 6
  • The authors report this as the first zero-shot generalization result for block stacking using RL, while acknowledging substantial room to improve generalization and stacking performance.Figure 4 analyzes internal attention patterns during six-block tower stacking.Source: A. Zero-shot Generalization · p. 6, Figure 4 · p. 6

B. Analyzing the learned representations

Attention patterns during six-block stacking suggest that ReNN focuses on blocks relevant to the current placement subproblem. The discussion notes remaining compute and task-scope constraints, as well as limitations of the manually designed curriculum and state-observation setting.

B. Analyzing the learned representations

  • Attention is highest on the block being placed, while attention to blocks already in the stack decreases from top to bottom.The authors suggest this pattern indicates decomposition of the task into simpler subproblems and may explain curriculum learning and zero-shot generalization.Source: B. Analyzing the learned representations · p. 6

VII. DISCUSSION

  • Despite higher sample efficiency than the existing state of the art, comparable real-world environment-step complexity would require 35 workers and several days.The estimate assumes each action takes 0.25 seconds; the authors also call for scaling to more complex geometries and more granular manipulation.Source: §VII · p. 6
  • The curriculum is manually designed around the premise that manipulating fewer objects is easier than manipulating more.The authors identify more complicated and effective curricula as a possible direction.Source: §VII · p. 6
  • Relational-architecture computation is quadratic in the number of entities, and the reported system uses state observations rather than visual or other sensory observations.The authors identify computational efficiency and scaling to sensory observations as future needs.Source: §VII · p. 7

APPENDIX

The appendix introduces additional experimental details, visualizes successful goal configurations, and documents several failure modes. It also identifies the experiment and ReNN architecture hyperparameter tables.

APPENDIX

  • The appendix adds experimental details and results beyond the main text.The appendix states that it provides additional details regarding the experimental setup and results.Source: APPENDIX · p. 9

A. Hyperparameter Tables

C. Failure Modes

  • The reported failure modes include end-effector oscillation, insufficient recovery time after a topple, and blocks falling off during or after stacking.Oscillation can occur near the tower base, where picking up a block risks toppling the tower; recovery can also exceed the episode's maximum length.Source: C. Failure Modes · p. 9

B. Successful Configurations for Goal Types

  • Figure A.1 visualizes Pyramid goals from 4 to 7 blocks and Multiple Towers goals from 4 to 9 blocks.Source: Figure A.1 · p. 9

D. Message Passing Rounds Ablation

The ablation compares one and three graph message-passing rounds. One round converges faster early in the curriculum but performs worse when transferring from five to six blocks; larger block counts may require more rounds to propagate stability information.

D. Message Passing Rounds Ablation

Loading 1912.11032v1…