PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPort: What and Where Pathways for Robotic Manipulation

Mohit Shridhar, Lucas Manuelli, Dieter Fox

arXiv:2109.12098v1cs.ROcs.CLcs.CVcs.LG

TL;DR

Robots need to connect abstract language concepts with precise manipulation, but existing manipulation and vision-language methods do not jointly provide broad semantic generalization and fine spatial reasoning. CLIPort combines CLIP’s semantic pathway with Transporter’s spatial pathway in a language-conditioned imitation-learning agent. It solves varied tabletop tasks and supports multi-task models, while remaining limited beyond two-step tabletop manipulation.

  • Problem

    Existing manipulation methods often fail to generalize to new goals, while vision-language representations lack the spatial understanding needed for fine-grained manipulation.

  • Method

    CLIPort uses a two-stream language-conditioned imitation-learning architecture that combines CLIP’s semantic understanding with Transporter’s spatial precision for pick-and-place policies.

  • Results

    CLIPort learns multi-task policies for 10 simulated tasks and 9 real-world tasks, with the simulated multi-task model outperforming single-task models in 41/72 = 57% of evaluations.

  • Takeaways & Limitations

    Data-driven policies with action abstractions and spatio-semantic priors can learn new manipulation skills without task-specific top-down engineering.

  • Takeaways & Limitations

    CLIPort remains limited to tabletop, two-step manipulation and does not handle dexterous 6-DOF manipulation, complex partially observable scenes, continuous multi-finger control, or task completion prediction.

Abstract

from arXiv · show

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

1 Introduction

CLIPORT addresses the challenge of grounding abstract language concepts in precise tabletop manipulation by combining semantic and spatial reasoning. It supports broad language-conditioned tasks, including multi-task models evaluated in simulation and on a real robot.

  • Motivation: End-to-end manipulation methods often require new demonstrations or goal images when goals or object attributes change, limiting scalable language-based interaction.These methods also lack semantics underlying the tasks.
  • Evaluation: CLIPORT evaluates 10 simulated tasks with thousands of unique instances per task and learns one multi-task model that performs better or comparably to single-task models.The experiments use the Ravens framework.
  • Evaluation: The approach is also validated with multi-task models on 9 real-world tasks using a Franka Panda manipulator.The figure caption reports training with just 179 image-action pairs.
  • Approach: CLIPORT combines CLIP’s semantic understanding with Transporter’s spatial precision in a language-conditioned imitation-learning framework.The approach uses semantic and spatial pathways for fine-grained manipulation.
  • Scope: The framework targets a broad range of language-conditioned tabletop manipulation tasks without explicit object poses, instance segmentations, memory, symbolic states, or syntactic structures.The paper frames this as combining fine-grained manipulation with multi-goal and multi-task generalization.
  • Contributions: The paper contributes an extended benchmark of language-grounding manipulation tasks and a two-stream architecture for conditioning precise policies with language goals.The contributions include empirical results across manipulation tasks and real-robot experiments.

2 Related Work

Prior work separately advances vision-based manipulation, semantic vision-language representations, and language grounding, but these lines of work leave important gaps in spatial precision, generalization, or action integration.

  • Vision-based Manipulation: Object detectors, segmentors, and pose estimators struggle with deformable objects, granular media, and unseen objects without object-specific training data.Dense descriptors and keypoints avoid some explicit representations but struggle with sequential actions and variable object counts.
  • Semantic Models: Vision-language methods commonly rely on bounding boxes or instance segmentations, limiting their applicability to piles of coffee beans or squares on a chessboard.Contrastive approaches learn continuous representations without top-down object detection.
  • Language Grounding for Robotics: Language-grounding systems often separate perception and action, leaving them without the spatial precision needed for tasks such as folding cloths.An end-to-end continuous-control system required several hours of teleoperation data for one simulated desk setting.
  • Two-Stream Architectures: Existing two-stream robotics pipelines require goal images or are restricted to one-step grasps with single-category goals, whereas CLIPORT supports composable language commands for sequential tasks.The comparison identifies the interface and task scope that CLIPORT targets.

3 CLIPORT

CLIPORT extends Transporter into a language-conditioned two-stream imitation-learning agent that combines semantic and spatial pathways for pick-and-place manipulation. Its policy predicts pixelwise pick and place affordances from visual observations and language instructions, trained from expert demonstrations.

  • CLIPORT Principles: CLIPORT uses a two-step manipulation primitive in which each action specifies start and final end-effector poses.The actions are represented as pick and place poses in SE(2).
  • Problem Formulation: The policy receives an RGB-D observation and English instruction, then outputs pick and place actions for tabletop tasks.Instructions may specify sequential steps or a single whole-task goal.
  • Two-Stream Architecture: The final dense features support pick and place affordance predictions across the three fully convolutional networks fpick, Φquery, and Φkey.The figure specifies addition for pick fusion and 1 × 1 convolution for place fusion.
  • Pick-and-Place Policy: Transporter’s pick module predicts where to pick, while its place module predicts where to place conditioned on the selected pick.Both modules are translationally equivariant fully convolutional networks.
  • Pick Prediction: The pick network produces dense pixelwise action values, and camera calibration maps the selected pixel to a 3D picking location.The network is supervised to imitate expert picks for the specified language instruction.
  • Place Prediction: The place module cross-correlates query features from a crop centered at the pick with key features from the full input to select a placement pose.Placement rotations are handled by stacking 36 discrete crop-angle rotations; the implementation uses c = 64 and d = 3.
  • Two-Stream Architecture: The semantic pathway uses frozen CLIP RGB features and tiled language encodings, while the spatial pathway processes RGB-D features and fuses with semantic features.Language conditioning is applied at the bottleneck and subsequent decoder layers, with skip connections from the CLIP encoder.
  • Training: CLIPORT is trained end-to-end by imitation learning from expert input-action pairs with one-hot pixel encodings for picks and discretized placement rotations.Multi-task training samples a task and then an input-action pair from that task.

4 Results

Experiments evaluate CLIPORT on language-conditioned manipulation tasks in simulation and hardware, including multi-task learning and generalization to unseen attributes. The results show that combining semantic and spatial pathways supports precise manipulation, few-shot learning, cross-task transfer, and real-world performance.

  • Evaluation setup: The evaluation uses 10 simulated language-conditioned tasks with seen and unseen attribute splits, scoring task success from 0 to 100.Task instances vary poses, colors, sizes, and object categories; scores assign partial credit for partially completed tasks.
  • Evaluation setup: Figure 3 summarizes average scores across seen and unseen splits for all tasks in Table 1.Table 1 reports mean success scores from 100 evaluation instances across 1, 10, 100, or 1000 demonstrations.
  • Two-stream performance: Only CLIPORT (single) exceeds 90% on seen tasks, while Transporter-only saturates at 50% and CLIP-only at 76%.The comparison indicates that both language-conditioned semantic understanding and spatial precision contribute to fine-grained manipulation.
  • Multi-task performance: A single CLIPORT (multi) model trained on all 10 tasks outperforms single-task models in 41/72 evaluations, although it performs worse on longer-horizon tasks such as align-rope.The multi-task model uses the same amount of data per task as single-task models while learning from more diverse data overall.
  • Generalization: Explicit cross-task attribute transfer raises put-blocks-in-bowls-unseen-colors performance from 45.8 to 75.7 at n = 1000.Unseen-attribute performance remains lower overall, but CLIPORT (single) performs substantially better than the language-free Transporter-only baseline.
  • Real-robot experiments: Hardware experiments with a Franka Panda show effective few-shot learning on 9 real-world tasks using 179 samples, with simple block manipulation reaching approximately 70%.The authors estimate that 50 to 100 demonstrations are needed for more robust real-world performance and report sensitivity to training-data biases.

5 Conclusion

CLIPORT is an end-to-end framework for language-conditioned fine-grained manipulation, but its tabletop action abstraction does not yet extend to dexterous 6-DOF control or complex partially observable scenes.

  • CLIPORT is an end-to-end framework for language-conditioned fine-grained manipulation.
  • Coupling action abstraction with spatio-semantic priors can help end-to-end methods learn new skills without task-specific top-down engineering.
  • Extending CLIPORT beyond its two-step primitive to dexterous 6-DOF manipulation remains a challenge.
  • The current system cannot handle complex partially observable scenes, continuous control for multi-fingered hands, or task-completion prediction.

A Task Details

The benchmark extends Ravens with language-conditioned tasks spanning semantic grounding, precise placement, sequencing, deformable objects, and unseen attributes across randomized scenes.

  • The extended Ravens benchmark contains 10 language-conditioned tasks, with eight tasks offering seen and unseen variants.Experts generate demonstrations using privileged simulator state and prespecified heuristics.
  • Align-rope requires manipulating a deformable rope between two square corners under four possible endpoint alignments.The rope and square poses are randomized for each instance.
  • Packing unseen shapes tests semantic understanding by placing one specified shape among four distractors, without requiring precise placement.Training uses seen shapes, while evaluation uses unseen shapes.
  • Assembling kits sequentially requires placing five specified shapes into matching holes at instruction-prescribed timesteps, including unseen shapes, colors, and spatial relations.
  • Color-grounding tasks place specified blocks into matching bowls, allowing multiple valid solutions and testing unseen color transfer.
  • Other tasks test tight packing, sequential object placement, category grouping, pyramid construction, and sweeping toward a specified zone.These tasks vary in semantic demands, ordering constraints, object attributes, and placement precision.

B Evaluation Workflow and Validation Results

Evaluation selects checkpoints by task execution performance rather than validation loss, using fixed evaluation instances and reported results across demonstration budgets and task settings.

  • Table 4 reports mean task success scores across 1, 10, 100, or 1000 demonstrations for single-task and multi-task models on seen and unseen splits.
  • Validation and test sets each contain 100 evaluation instances within a four-phase train, checkpoint-selection, and test workflow.
  • Task execution performance is used to select checkpoints because validation loss can penalize alternative actions that still achieve a task goal.This matters for multimodal tasks such as placing any yellow block into a red bowl.
  • Validation evaluates checkpoints from 1K–200K iterations for single-task models and 1K–600K iterations for multi-task models.
  • The evaluation section includes average validation scores across seen and unseen splits for all benchmark tasks.

C Two Stream Architecture Details

The implementation details the two-stream architecture and the real-robot data-collection and grasp-rotation procedures used to realize CLIPORT’s manipulation policy.

  • Architecture: CLIPORT’s architecture uses ReLU activations, identity blocks without Batch Normalization, repeated depth channels, and a pretrained CLIP checkpoint.
  • Real-Robot Setup: Real-robot experiments use a Franka Panda with a parallel gripper and a Kinect-2 RGB-D camera calibrated to the robot base frame.
  • Demonstrations: Demonstrations are collected through a 2D interactive tool where users select bounding boxes and discrete grasp rotations from top-down RGB views.
  • Pick Rotations: For parallel-gripper grasping, CLIPORT separates pick prediction into a pixel locator and a rotation predictor operating on a 64 × 64 crop.

E Data Augmentation

CLIPORT augments training samples with random SE(2) transformations to improve spatial generalization while retaining valid expert pick and place actions.

  • Random SE(2) transformations are applied to RGB-D training inputs, with transformed expert T_pick and T_place actions retained when in frame.Augmentations with either action out of frame are discarded.

F Ablations and Baselines

Ablations compare one-stream and two-stream architectures, showing the importance of combining semantic and spatial information, pretrained multimodal features, and skip connections.

  • Table 5 evaluates baselines and ablations on stacking block pyramids and packing objects, which differ in their demands for spatial and semantic reasoning.Scores are reported as mean percentages from 100 evaluation runs.
  • One-stream baselines: The Transporter-only baseline supplies the spatial stream, while CLIP-only supplies the semantic stream with RGB and language input.These one-stream models isolate the two pathways used by CLIPORT.
  • One-stream baselines: Language-conditioned Transporter performs very poorly because high-level language features corrupt the low-level spatial features needed for precise pick-and-place actions.
  • One-stream baselines: The image-goal Transporter baseline receives a goal image from the next timestep for sequential tasks and from the final timestep for non-sequential tasks.
  • Two-stream variants: Skip connections are particularly important for good performance in the two-stream CLIP-Transporter variant.The authors hypothesize that multiple semantic levels, from patterns and shapes to objects and concepts, help condition the decoder.
  • Two-stream variants: CLIPORT’s two-stream architecture uses CLIP ResNet50 and a CLIP sentence encoder, whereas the RN50-BERT variant substitutes ImageNet ResNet50 and DistilBERT.CLIP provides multimodal vision-language alignment without restricting representations to detection or segmentation pipelines.
  • Two-stream variants: Untrained semantic-stream features provide some conditioning benefit, but perform substantially worse than pretrained multimodal features.

G Performance on Demo-Conditioned Tasks

On demo-conditioned tasks without language instructions, the two-stream architecture outperforms Transporter in most evaluations, especially when training data are scarce.

  • Table 6 reports mean validation task-success scores across 100 evaluation instances for 1, 10, 100, or 1000 demonstration episodes.
  • 75% of evaluations favor the two-stream architecture over Transporter, with especially large gains on assembling-kits and manipulating-rope tasks.The comparison covers 30 of 40 evaluations and is strongest in low-data regimes with 100 demonstrations or fewer.
  • The two-stream method is reported to outperform Transporter particularly in low-data settings with 100 demonstrations or less.
  • The authors hypothesize that CLIP-ResNet supplies a strong visual prior for learning generalizable policies.
  • The evaluated affordance examples span simulated and real-world tasks, including instructions involving cherries, colored blocks, chess pieces, ropes, and shoes.

I Limitations and Risks

CLIPORT’s limitations include dependence on balanced demonstrations and calibration, reduced dexterity, restricted generalization to novel objects and relationships, and risks from pretrained models.

  • Balanced datasets: CLIPORT relies heavily on balanced training data covering expected skills and invariances, otherwise it can learn dataset-specific biases.The authors found some such biases only during real-world execution and iteratively refined datasets.
  • Calibration and control: Execution is sensitive to hand-eye calibration because 2D pixel actions with yaw rotations must be converted into end-effector poses using calibrated camera extrinsics.
  • Dexterous manipulation: Extending CLIPORT from SE(2) to 6-DOF or N-DOF dexterous control is non-trivial and may make 3D equivariant processing expensive and slow.The SE(2) action space supports data efficiency and high-level affordance reasoning but limits dexterity.
  • Novel objects: With one training example, CLIPORT correctly grasps 2/3 unseen pliers but fails on an instance significantly outside the training distribution.
  • Object relationships: CLIPORT struggles with complex multi-object relationships and cannot count objects because it maintains no history or belief across timesteps.These constraints limit instructions to ‘any’ or ‘all’ quantifiers.
  • Language grounding: CLIPORT’s verb-noun understanding is tightly grounded in the demonstrations and tasks seen during training rather than generic usage across contexts.
  • Task completion: CLIPORT relies on an expert to indicate task completion, so real-world execution continues until a user stops it.A success classifier is suggested as a future remedy.
  • Pretrained-model risks: Using CLIP representations for physical actions exposes the system to pretrained-model biases, harmful associations, and adversarial attacks.
Loading 2109.12098v1…