Source-linked AI summary
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, Andy Zeng
TL;DR
Manipulating deformable cables, fabrics, and bags is difficult because their dynamics, configurations, sensing, and goal states are complex. The paper introduces DeformableRavens and goal-conditioned Transporter Networks for vision-based sequential manipulation, reporting success in simulation and physical experiments while identifying bag-task failure modes.
Problem
Deformable manipulation remains difficult because deformables have complex dynamics and high-dimensional configurations, while their goals are not easily specified by compact poses.
Method
The paper introduces a 12-task DeformableRavens benchmark and goal-conditioned Transporter Network architectures for vision-based multi-step manipulation of cables, fabrics, and bags.
Results
7/10 physical test episodes succeeded with different test-time goal images, requiring an average of 7 pick-and-place actions per episode.
Takeaways & Limitations
The proposed vision-based architectures manipulate deformable structures across 1D, 2D, and 3D tasks and are competitive with or superior to ground-truth state baselines.
Takeaways & Limitations
Bag opening can be hard to recover from after visual occlusion, and lifted items may fail to remain enclosed because of the discrete-time planar action parameterization.
Abstract
from arXiv · showhide
Rearranging and manipulating deformable objects such as cables, fabrics, and bags is a long-standing challenge in robotic manipulation. The complex dynamics and high-dimensional configuration spaces of deformables, compared to rigid objects, make manipulation difficult not only for multi-step planning, but even for goal specification. Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag". In this work, we develop a suite of simulated benchmarks with 1D, 2D, and 3D deformable structures, including tasks that involve image-based goal-conditioning and multi-step deformable manipulation. We propose embedding goal-conditioning into Transporter Networks, a recently proposed model architecture for learning robotic manipulation that rearranges deep features to infer displacements that can represent pick and place actions. In simulation and in physical experiments, we demonstrate that goal-conditioned Transporter Networks enable agents to manipulate deformable structures into flexibly specified configurations without test-time visual anchors for target locations. We also significantly extend prior results using Transporter Networks for manipulating deformable objects by testing on tasks with 2D and 3D deformables. Supplementary material is available at https://berkeleyautomation.github.io/bags/.
I. INTRODUCTION
The paper addresses difficult vision-based manipulation of deformable cables, fabrics, and bags by introducing a broad benchmark and goal-conditioned Transporter Networks. Its experiments extend deformable manipulation from primarily 1D tasks to 12 tasks spanning 1D, 2D, and 3D structures.
- Motivation: Deformable manipulation is challenging because deformables have more complex configuration spaces, dynamics, and sensing than rigid objects.
- Motivation: Prior work largely focused on 1D ropes or 2D fabrics, leaving generalizable vision-based manipulation of 3D structures such as inserting objects into bags less explored.Deformable goals may also require complex relative spatial relations and are not easily represented by compact poses.
- Scope: The benchmark includes 1D cables, 2D fabrics, and 3D bags, extending prior Transporter Network results beyond one 1D deformable task.The paper reports results on 12 tasks, including fabric and bag manipulation.
- Contributions: The proposed goal-conditioned Transporter Networks learn vision-based multi-step manipulation of deformable structures.Goal-conditioned tasks use target images, while other tasks are specified without goal images.
B. Data-Driven Robot Manipulation
The paper frames deformable manipulation as learning closed-loop pick-and-place policies from demonstrations and visual observations. Its formulation supports optional goal images and planar motion primitives for sequential manipulation.
- Data-Driven Robot Manipulation: Learned data-driven methods for robot manipulation include imitation learning, reinforcement learning, and model-based approaches applied to deformable tasks.
- Data-Driven Robot Manipulation: The proposed image-based architecture uses fully convolutional networks to produce per-pixel scores, with each pixel corresponding to an action.
- Problem Formulation: The problem is formulated as learning a policy that sequences pick-and-place actions from visual observations, optionally conditioned on a target goal observation.The policy is trained from stochastic expert demonstrations.
- Problem Formulation: Each action consists of pick and place end-effector poses, with closed-loop visual feedback supporting multi-step rearrangement.The pick and place poses specify grasping and releasing, respectively.
- Problem Formulation: Tabletop actions use SE(2) poses and a fixed-height motion primitive that approaches, grasps, lifts, places, and releases objects.
B. Background: Transporter Networks
Transporter Networks attend to a local image region, then predict a placing displacement by cross-correlating dense visual features. Goal-conditioned variants preserve spatial structure while incorporating a desired goal image.
- Transporter Networks: Transporter Networks first predict a per-pixel picking action, then estimate the placing displacement that maximizes Qplace.The picking location is selected from Qpick, while dense feature embeddings are cross-correlated over possible placing poses.
- Transporter Networks: The architecture uses three FCNs: fpick predicts picking values, while Φkey and Φquery produce features for placing prediction.Φquery processes a crop centered on the selected pick, and Φkey processes the full observation.
- Transporter Networks: Spatially consistent top-down orthographic observations preserve the scene’s 3D structure under visuo-spatial transforms.This property also supports rotation and translation data augmentation.
- Goal conditioning: For goal-conditioned tasks, a fixed goal observation og is provided alongside the current observation ot at the same pixel resolution.The goal image specifies a desired object configuration rather than only a target pose.
- Goal conditioning: Transporter-Goal-Stack concatenates current and goal images, whereas Transporter-Goal-Split separately processes the goal through Φgoal before feature combination.The split architecture combines goal features with query and key features using the Hadamard product.
- Goal conditioning: Preserving goal-image spatial structure encourages visual correspondences between current and desired configurations, which better captures deformable-object changes.These correspondences anchor pick-and-place actions on dense image information.
B. Training Details for Goal Conditioned Models
Goal-conditioned models train from demonstration transitions paired with the episode’s final observation as a goal. Current and goal images receive matched geometric augmentation to preserve spatial consistency.
- Training procedure: Training samples for goal-conditioned Transporter Networks contain a current observation, action, and the corresponding demonstration episode’s final observation.The procedure samples (ok, ak) as in standard Transporter training, then adds og from the episode after its last action.
- Model variants: Transporter-Goal-Split processes the goal image with a separate Φgoal module, while Transporter-Goal-Stack stacks current and goal images channel-wise.Both architectures use goal observations to condition action prediction.
- Task goals: DeformableRavens includes visible-zone tasks and goal-image tasks, with success criteria defined by target coverage or desired rearrangement configuration.The table describes green workspace zones for some tasks and separate goal images for others.
- Data augmentation: Current and goal observations receive the same random rotation and translation during augmentation to maintain consistency.This paired transformation is enabled by spatially consistent top-down orthographic images.
- Evaluation setup: The benchmark evaluates cables, fabrics, and bags in PyBullet through an OpenAI Gym interface.The evaluation suite is DeformableRavens, a simulated collection of deformable-manipulation tasks.
A. Deformable Objects (Soft Bodies) in PyBullet
The paper adds deformable soft-body support to PyBullet using finite-element-based simulation with mass-springs and self-collisions. It instantiates cables, fabrics, and bags in a standardized robot-workspace setup for benchmark manipulation tasks.
- Soft-body simulation: The PyBullet extension uses finite-element-method soft-body simulation with mass-springs, vertex self-collisions, and unified contact and friction constraints.This simulator support addresses the difficulty of building deformable-manipulation benchmarks.
- Deformable representations: Cables are modeled as bead sequences, while fabrics and bags are soft bodies represented by mesh vertices.Bags additionally use rigid beads around the opening to improve drawstring-style simulation stability.
- Bag construction: The physical bags resemble sack and drawstring designs; bags with handles or greater stiffness are outside the current work.Bag geometry is created from a sphere mesh with vertices removed above a chosen height.
- Standardized setup: Each benchmark task uses a UR5 arm, a 0.5×1m tabletop, and three calibrated RGB-D cameras producing 320×160 top-down observations.Each pixel represents a 3.125×3.125mm vertical column of 3D space, with RGB and channel-wise depth values.
- Manipulation tasks: Bag-items tasks are staged to open the bag, insert items, and lift the bag, with grasping approximated by locking the nearest contacted deformable vertex to the end effector.Action heights vary according to the task stage and gripped item.
- Training and baselines: The experiments use scripted stochastic demonstrators and compare Transporter variants with ground-truth-state MLP baselines.The listed models include Transporter-Goal-Split, Transporter-Goal-Stack, GT-State MLP, and a two-step ground-truth baseline.
VII. SIMULATION RESULTS
Simulation results show that Transporter policies can perform strongly on several deformable-object tasks, while bag manipulation remains vulnerable to recovery and enclosure failures.
- Transporter achieves orders of magnitude better sample efficiency than ground-truth models on the evaluated non-goal-conditioned tasks.
- With enough demonstrations, Transporter reaches at least 90.1% on cable-shape and at least 86.5% on fabric-flat.
- At 1000 demonstrations, Transporter reaches 51.7% on bag-items-1 and 46.7% on bag-items-2, exceeding demonstrator performances of 41.7% and 32.5%.
- Goal-conditioned Transporters substantially outperform ground-truth state baselines across all dataset sizes for cable-line-notarget, cable-shape-notarget, and fabric-flat-notarget.
- Bag policies may cover the opening or leave inserted items insufficiently enclosed, causing difficult recovery or item loss when the bag is lifted.
B. Experiment Details and Protocol
The physical evaluation tests goal-conditioned cable manipulation on a Franka robot using hand-mounted vision, bounded episodes, and cable-mask IoU success. The experiments report both successes and failure modes arising from grasping heuristics and inefficient action sequences.
- 30 human demonstrations were collected, with 24 used for training and 6 for validation.
- The Franka uses a mounted Azure Kinect camera and returns to a top-down home pose after each action to obtain updated workspace images.
- Success requires cable-mask IoU above 0.25 within a 10-action episode, a threshold chosen to remain forgiving of physical imprecision.
- 7/10 test episodes succeeded, reaching the threshold after an average of 7 pick-and-place actions.
- One failure involved repeated grasping failures on a self-overlapping cable, while two others exceeded the action limit after repetitive back-and-forth motions.
- The paper presents goal-conditioned Transporter architectures for vision-based rearrangement of deformables spanning cables, fabrics, and bags.
- Bag tasks remain challenging because occluded openings hinder recovery, lifted items may fall out, and physical experiments are limited to cables.
This appendix is structured as follows:
The appendix organizes task, demonstrator, architecture, and experiment details, while documenting the benchmark’s object representations and later revisions.
- Appendix II describes the tasks in more detail, Appendix III defines demonstrator policies, and Appendix IV details Transporter Network architectures.
- Appendix V provides experiment-methodology details, and Appendix VI presents additional results, videos, and open-source code.
- Earlier paper versions used only PyBullet simulation because physical robotics experiments were infeasible during the COVID-19 pandemic.
- A later revision added physical cable experiments and fixed a goal-image processing bug in both goal-conditioned Transporter variants.
- DeformableRavens contains 12 tasks, with cables represented by 24 or 32 beads, fabrics by 100 vertices, and bags by 353 vertices plus 32 opening beads.
- Ground-truth bag states use opening-bead poses and a displacement pose, yielding a 99D representation rather than full bag vertex information.
- Bag tasks vary pull height and end-effector velocity across stages including opening, item insertion, and post-insertion transport.
B. DeformableRavens Task Details
DeformableRavens spans cable, fabric, bag, and rigid-block tasks with visible or image-specified goals and task-specific success criteria. The benchmark includes multi-stage bag manipulation and documents a fabric-simulation limitation.
- Cable tasks include rings, free-end shapes, and lines, with target zones either visible or specified by separate goal images.
- Fabric-cover requires folding fabric over a cube, while fabric-flat tasks move fabric toward a target using an 85% coverage threshold.
- Fabric-flat-notarget specifies the desired fabric configuration with a separate image and uses current-plus-target fabric representations for ground-truth input.
- Bag-alone-open measures whether the opening’s bead-ring convex-hull area exceeds a threshold.
- Bag-items-1 requires opening a bag, inserting a cube, and transporting the bag so the cube and part of the opening reach the target zone.
- Bag-items-2 transports two differently shaped or colored items and uses a 108-dimensional ground-truth state.
- The authors abandoned attempts to perturb fabric configurations into folds because tuning self-collisions among fabric vertices was difficult.
- Bag-color-goal requires placing an item in the correctly colored bag, even when the same-colored bag shares the target bag’s location.
C. Additional Task: Block-Notarget
Block-notarget adapts a precise L-shaped-block insertion task to image-based goal conditioning, while the appendix describes scripted demonstrations and their data-generation constraints.
- Block-notarget requires picking, rotating, and placing a red L-shaped block into a receptacle specified only in a separate goal image.
- All tasks use scripted demonstrators, with detailed implementations referred to in the open-source code.
- Cable and fabric demonstrators choose corrective pick-and-place actions using bead-to-target distances or corner-pulling procedures.
- Ring and bag-opening demonstrators match beads to target positions by enumerating valid bipartite assignments and minimizing distance.
- Demonstrator data reports mean and median episode lengths for 1000 episodes per task, with unsuccessful bag episodes filtered out.
- After the convex-hull area exceeds a threshold, demonstrations either terminate or continue to item insertion for bag tasks.
- Because behavior-cloned policies are bounded by the data-generating policy, unsuccessful episodes are ignored and additional runs generate 1000 successful bag demonstrations.
APPENDIX IV ADDITIONAL TRANSPORTER NETWORK DETAILS
The appendix details Transporter-based forward passes, including separate goal processing and feature combination, alongside implementation and baseline-reference materials.
- Transport operation: The implementation crops query-network logits around the picking point after feature processing, enabling later features to use a larger receptive field.Crops are transformed across SE(2) rotations before cross-convolution with the key-derived kernel.
- Transport-Goal-Split: Transport-Goal-Split processes current and goal images through separate key, query, and goal FCNs before combining their dense features.The transport module includes Φkey, Φquery, and Φgoal; the picking attention module fpick occurs beforehand.
- Implementation settings: The appendix identifies a 64-pixel crop size and typically one rotation for the transport-module forward pass.These settings are listed as pseudo-code arguments used in the experiments.
- Feature combination: Goal-conditioned features are combined with key and query features using elementwise multiplication.The pseudo-code computes g_logits × k_logits and g_logits × q_logits through Hadamard products.
- Model references: The reported model materials include Transporter variants and ground-truth baseline models for comparison.Table IV summarizes relevant hyperparameters for these model families, while Table V reports block-task baseline comparisons.
APPENDIX V ADDITIONAL EXPERIMENT DETAILS
The appendix documents evaluation procedures across demonstration budgets and learning checkpoints, then reports strong sample-efficiency results for Transporter variants on block and deformable-object tasks.
- Evaluation protocol: Experiments train models for 20K iterations, save 10 snapshots every 2K iterations, and evaluate each snapshot over 20 held-out episodes.Three separate training runs provide repeated evaluation data for each model type and demonstration count, with an exception for one computationally expensive task.
- Block-task results: Two goal-conditioned Transporter variants are roughly two orders of magnitude more sample efficient than ground-truth models on block-notarget.Their 10-demo success rates are 88.3% and 90.0%, comparable to ground-truth models’ 1000-demo rates of 90.0% and 91.7%.
- Evaluation protocol: Models are compared across 1, 10, 100, and 1000 demonstrations using learning curves and consistent experimental settings.Figures 9–12 correspond to these demonstration counts, while Figure 8 reports block-notarget results across the same budgets.