PaperlayerView exact revision on arXiv

Source-linked AI summary

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Ayzaan Wahid, Vikas Sindhwani, Johnny Lee

arXiv:2010.14406v3cs.RO

TL;DR

Transporter Networks address the challenge of improving vision-based manipulation sample efficiency without object-centric assumptions. They preserve spatial structure to infer spatial displacements and achieve stronger generalization and sample efficiency across manipulation tasks than benchmarked alternatives.

  • Problem

    End-to-end pixel-to-action models require copious data, while object-centric representations impose collection burdens and struggle with unseen, occluded, deformable, or piled objects.

  • Method

    Transporter Networks preserve spatial structure by attending to local visual regions and predicting their spatial displacements without assuming objectness.

  • Results

    Transporter Networks achieve greater than 90% success on most of 10 tabletop tasks with 100 expert demonstrations and show orders-of-magnitude greater sample efficiency than end-to-end alternatives.

  • Takeaways & Limitations

    The approach generalizes across rotations and translations, new objects and configurations, sequential visual-feedback tasks, and some one-shot settings.

  • Takeaways & Limitations

    The approach assumes calibrated camera-to-robot correspondence and remains sensitive to camera-robot calibration.

Abstract

from arXiv · show

Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input - which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world. Experiment videos and code are available at https://transporternets.github.io

1 Introduction

The paper introduces Transporter Networks, which preserve spatial structure to infer manipulation-relevant displacements without object-centric assumptions. Across diverse tabletop tasks, they improve sample efficiency, generalization, and sequential manipulation capabilities.

  • Motivation: End-to-end pixel-to-action models often require substantial data, while object-centric representations struggle with unseen, occluded, deformable, or densely piled objects.These limitations motivate incorporating structure without imposing object-specific assumptions.
  • Approach: Transporter Networks attend to a local visual region and predict its target spatial displacement through deep feature template matching.The predicted displacement parameterizes robot manipulation actions.
  • Evaluation scope: The architecture applies across tasks including fixture insertion, Towers of Hanoi, kit assembly with unseen objects, palletizing, block stacking, rope manipulation, and closed-loop pile pushing.These examples span rigid, deformable, sequential, and cluttered manipulation settings.
  • Capabilities: Transporter Networks model multi-modal spatial action distributions and generalize across rotations and translations using partial RGB-D demonstrations.They require no prior object knowledge and can achieve one-shot learning on some tasks.
  • Results: 10 unique tabletop tasks show greater than 90% success on most tasks with objects in new configurations using 100 expert demonstrations.The paper also extends the approach to 6DoF tasks and introduces the Ravens simulated benchmark.

2 Related Work

Prior manipulation systems rely on object-specific perception or scripted control, limiting scalability beyond structured settings. Learned object representations improve generalization but remain challenged by unseen, occluded, deformable, or numerous objects.

  • Object-centric Representations: Object detectors and pose estimators often require object-specific training data, making them difficult to scale to applications with unseen objects.Keypoints and dense descriptors offer category-level generalization but do not fully resolve difficult scene structures.
  • Pick-and-Place: Classic pick-and-place systems use known-object pose estimation, scripted planning, and motion control, but are difficult to deploy in unstructured environments.This limitation motivates learned general pick-and-place policies that can handle unseen objects.

3 Method

Transporter Networks formulate manipulation as predicting spatial displacements through picking and pick-conditioned placing, while preserving spatial structure and avoiding object-centric assumptions. The method uses equivariant visual representations, feature transport, and multi-stage inference to handle planar rotations and 6DoF tasks.

  • Problem formulation: Manipulation is represented as a sequence of two-pose motion primitives, with T_pick and T_place defined in SE(2) or SE(3).The framework also applies to pushing by interpreting the two poses as the start and end of the push.
  • Action distributions: The method recovers distributions of successful picks and corresponding place poses rather than assuming a single canonical action.Successful picks can be multi-modal, including cases with repeated objects or symmetric shapes.
  • Architecture: Transporter Networks decompose actions into picking and pick-conditioned placing, with the primary contribution being feature-based transporting for the latter.The picking model predicts T_pick, while the placing model predicts T_place conditioned on that pick.
  • Picking: Fully convolutional networks model picking distributions using translational equivariance between visual-object translations and corresponding picking poses.This spatial structure supports complex multi-modal picking distributions anchored on visual features.
  • Pick-conditioned placing: For placing, a local feature crop centered at T_pick is transformed and cross-correlated across candidate poses to estimate the best displacement.The operation overlays deep features rather than RGB-D pixels, and its action-value function is invariant to T_pick under the desired transform.
  • Rotations and 6DoF: Planar rotations are handled by discretizing SO(2) and running the fully convolutional network across rotation bins, while SE(3) uses multi-stage inference.The multi-stage approach addresses the three SE(2) degrees of freedom before extending transport to the remaining rigid-body dimensions.

4 Results

The experiments compare Transporter Networks with end-to-end and ground-truth-state baselines across simulated manipulation tasks, evaluating sample efficiency, generalization, sequential feedback, and 6DoF control. The method generally learns more efficiently, handles unseen objects and recovery behaviors, and is validated on real robots.

  • Simulation setup: The Ravens benchmark compares behavior-cloned methods on 10 tabletop tasks with randomized object configurations and expert demonstrations.Baselines include Form2Fit, ConvMLP, and MLPs given ground-truth object state; all use identical random SE(2) augmentation.
  • Sample efficiency: Transporter Networks achieve orders of magnitude greater sample efficiency than image-based alternatives and outperform ground-truth-state multilayer perceptrons.Most baselines overfit demonstrations but generalize poorly with only 1,000 demonstrations.
  • Unseen-object generalization: On unseen-object kit assembly, Transporter Networks show a large performance gap over Form2Fit, while Form2Fit reaches 96.3% on simpler primitive geometries.The authors attribute the difference to Transporter Networks matching higher-resolution geometric information with the same amount of data.
  • Closed-loop feedback: Transporter Networks use current visual input to infer task context, enabling recovery after fallen stacks or swept objects leaving the target zone.These stateless models condition action-value predictions on contextual visual cues and can perform sequential multi-step behaviors.
  • Demonstration stochasticity: Transporter Networks work with both deterministic and stochastic demonstrations, whereas most baselines struggle when successful actions are randomly sampled.The comparison uses a simplified translation-only block-insertion task.
  • 6DoF pick-and-place: The SE(3) formulation achieves better sample efficiency on unseen 6DoF fixture configurations than image-based baselines trained on the easier 3DoF task.The hybrid discrete-continuous SE(3) model does not achieve one-shot generalization in this setting.
  • Real-robot validation: Real UR5e experiments validate kit assembly and sweeping using different camera and end-effector configurations.Kit assembly uses suction with a Photoneo PhoXi camera; sweeping uses a brush with an Azure Kinect.

5 Conclusion

The conclusion presents Transporter Networks as a spatial-displacement architecture for visual manipulation that avoids objectness assumptions and improves sample efficiency across varied tasks. It also identifies calibration sensitivity and incomplete integration of force-based actions as current limitations.

  • Conclusion: Transporter Networks infer spatial displacements from visual input to parameterize robot actions without assuming objectness and while exploiting spatial symmetries.The reported scope includes block stacking, unseen-object kit assembly, and closed-loop pile pushing.
  • Conclusion: The method is reported to be orders of magnitude more sample efficient than end-to-end alternatives across diverse vision-based manipulation tasks.The conclusion covers stacking, unseen-object assembly, and closed-loop manipulation of small-object piles.
  • Limitations: Current limitations are sensitivity to camera–robot calibration and unclear integration of torque or force actions with spatial action spaces.The authors identify real-time high-rate control and tool-use tasks as future extensions.

6 Appendix

The appendix documents the Ravens simulation framework, benchmark tasks, evaluation conventions, and task-specific setups. It covers randomized demonstrations and configurations, sequential and deformable manipulation, and the unseen-object kit-assembly split.

  • Framework: Ravens provides a consistent simulation environment with RGB-D observations, camera parameters, end-effector actions, and a ground-truth-state baseline exception.Simulation and real-world models are trained only on data from their respective settings.
  • Demonstrations and randomization: A scripted oracle generates demonstrations by sampling successful picking and placing actions, while objects and target zones are randomized during training and testing.These distributions expose task multimodality and sequential permutations through demonstrations.
  • Benchmark tasks: The Ravens-10 benchmark contains 10 tabletop tasks, including insertion, placement, sequencing, stacking, packing, kit assembly, rope manipulation, and sweeping.The tasks use two-pose SE(2) end-effector motion primitives, which can still produce 3D behaviors.
  • Sequential manipulation: Towers of Hanoi requires seven sequential pick-and-place actions while inferring the next disk from contextual visual cues despite occlusions.The disks must be moved between pegs without placing a larger disk over a smaller one.
  • Unseen objects: Kit assembly samples five objects from 20 total, with 14 used for training and 6 held out for testing to assess generalization to new shapes.Success requires placing objects onto corresponding silhouettes on a board.
  • Deformable and cluttered scenes: Rope manipulation and pile sweeping require sequencing actions with closed-loop visual feedback on deformable or numerous small objects.The rope must be untangled and aligned, while piles are pushed into a target zone.

6.4 Additional Tasks

Additional-task experiments examine simplified translational insertion, 6DoF insertion, evaluation metrics, and augmentation assumptions. They define pose- and zone-based rewards and use augmentation that assumes table-plane position is task-irrelevant.

  • Additional environments: The additional-task suite includes a simplified 2DoF block-insertion environment and a 6DoF variant requiring SE(3) placing.The simplified task varies only fixture translation, whereas the 6DoF task also varies height, roll, and pitch.
  • Evaluation metrics: Simulation performance is evaluated using either pose accuracy or the fraction of object volume inside a target zone.Pose tasks use translation and rotation thresholds; zone tasks discretize 3D bounding boxes into 2cm^3 voxels.
  • Zone metric: For palletizing and packing, achieving reward 1 requires tightly fitting all objects, making high rewards substantially harder than initial partial coverage.The reported difficulty increases sharply near complete target-zone coverage.
  • Data augmentation: All methods receive identical random SE(2) augmentation, which assumes absolute table-plane object position does not matter for the tasks.The augmentation transforms observations or ground-truth poses into a sampled augmentation frame.

6.7 Additional Baseline Details

The baselines include image-based and ground-truth-state policies with differing input representations, action spaces, and regression strategies. Comparisons suggest Transporter Networks outperform image-based baselines and often use less data than state-based MLP policies, while benchmark differences remain relevant.

  • Form2Fit uses dense visual descriptors to associate picking actions with placing actions, with a matched 43-layer ResNet implementation for comparison.
  • ConvMLP processes top-down RGB-D views with convolutional layers, spatial soft(arg)max, and mixture-density regression over two SE(2) end-effector poses.
  • GT-State MLP learns from object poses and 3D bounding boxes, approximating perfect external object-pose estimation rather than direct visual input.
  • The authors attribute faster end-to-end learning to a rearrangement prior and spatial structure that exploits data symmetries, while noting possible future GNN and translation-equivariant baselines.
  • Transporter Networks outperformed both image-based baselines in all tested scenarios, while Form2Fit generally exceeded ConvMLP except on sweeping piles.
  • Transporter Networks often required less data than ground-truth-state MLP policies, although those baselines used continuous actions and multi-modal regression rather than discrete classification.
  • On sweeping piles, constraining the model to two translation-only poses outperformed allowing independent rotation choices, suggesting limits of the rigid-transformation bias for complex pile dynamics.

6.9 SE(3) Transporter Networks: Architecture Discussion and 6DoF Task Analysis

The SE(3) extension retains the SE(2) transport pipeline and regresses roll, pitch, and height separately. It generalizes above 90% in tested settings with 1,000 examples, while performance depends on degrees of freedom and object geometry.

  • Architecture Discussion: The presented SE(3) architecture uses discrete convolution for image-plane dimensions and continuous regression for the remaining out-of-plane degrees of freedom.
  • Architecture Discussion: An additional model handles roll, pitch, and z after SE(2) processing, using three independent regressions conditioned on the preceding transport stages.
  • Architecture Discussion: Separate classification and regression networks were found beneficial compared with a single pick-conditioned placing network combining both losses.
  • 6DoF Task Analysis: Above-90% generalization was achieved on the tested challenging 6DoF task with 1,000 training examples, although long thin objects may be more difficult.
  • 6DoF Task Analysis: With 1,000 examples, roll and pitch MAEs were 3–4 degrees, while yaw MAE was about 10 degrees and most often contributed to unsuccessful test deployments.
  • Generalization Analysis: Transporter Networks extrapolated to new block-insertion configurations from one demonstration, using dense feature transport to exploit translational equivariance.
  • Generalization Analysis: Form2Fit also extrapolated from one example, but additional stochastic demonstrations made descriptor matching harder, while GT-State MLP showed incorrect correlations and limited extrapolation.
  • Ablations: The ablations compare task success across 1, 10, 100, and 1,000 demonstration episodes, including no-transport and per-pixel cross-entropy variants.

6.12 Training Convergence

Training generally converges quickly relative to image-based baselines, while qualitative predictions expose both symmetry-aware behavior and an unseen-object failure mode. Real-world evaluation covers kit assembly and sweeping piles, with kit assembly reaching 98.9% success.

  • Transporter Networks generally converged within a few thousand iterations, corresponding to roughly one or two hours of wall-clock time for rope manipulation.
  • Prediction visualizations show modes changing with object locations, shapes, and conditioned picks, reflecting learned rotation and translation equivariance.
  • Unseen E-shaped objects produced 180-degree-flipped false-positive placement modes because local geometry made flipped shapes appear similar.
  • The real-world evaluation used kit assembly of mouthwash bottles and sweeping piles of small Go pieces.
  • 98.9% kit-assembly success was achieved from 8,141 human-demonstrated pick-and-place actions, with picking and disassembly success both above 99%.
  • Training data came from 13 teleoperators with varying skill, producing suboptimal, biased, and noisy demonstrations that increased action variance.

Hardware Setup

The hardware system combines UR5 workstations, suction or brush end-effectors, depth cameras, Linux PCs, and vacuum equipment. Calibration, teleoperation, autonomous execution, and cloud logging support data collection and evaluation.

  • The experiments used two UR5 workstations, each with an industrial Linux PC, vacuum system, end-effector, and statically mounted depth camera.
  • Kit assembly used a Photoneo PhoXi Model M depth camera, while the sweeping setup used observations from a lower-cost camera source.
  • Camera calibration used QR-code panels for intrinsics and wrist-mounted QR tags across random robot poses for extrinsics.
  • The Linux system forwarded robot and camera data to remote teleoperators, queued robot commands, and logged data to cloud storage in real time.
  • The data-collection interface lets operators select pickup regions, adjust placement and rotation, and use overlaid depth around the pickup point.
  • Autonomous execution begins when a user enters a unique task code and presses the play button.

Data Collection

The system supports remote human teleoperation through a 2D interface that controls robot configuration, motion, and parameterized actions. The authors collected demonstrations for kit assembly and sweeping tasks using this shared pipeline.

  • The Unity-based 2D interface supports robot workcell configuration, suction-end-effector control, parameterized trajectories, calibration, and model-compatible robot control.
  • Operators inspect workstation depth data, select pickup points, move attached point-cloud regions, and choose 3DOF placement locations through mouse interactions.
  • Transporter Networks use the same data pipeline and motion primitives as human teleoperation, replacing UI interaction with model inference.
  • 8,141 pick-and-place actions from 13 operators trained the kit assembly model, alongside 2,300 validation actions and 6,759 pushing actions for sweeping.

Success Detection

An image classifier detects whether kits are full, empty, or partial after pick-and-place attempts. This provides an automatic task-level success signal for autonomous operation.

  • A pretrained ResNet50 performs 3-way classification of full, empty, and partial kits.
  • 96.9% mean validation accuracy was achieved on 128 samples from a unique camera angle.
  • Classifier inference supplies a success metric after each attempt, allowing autonomous alternation between kitting and emptying tasks.
Loading 2010.14406v3…