Source-linked AI summary
Open-World Object Manipulation using Pre-trained Vision-Language Models
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, Karol Hausman
TL;DR
Robots need to manipulate objects named in human instructions even when those categories are absent from their first-hand experience. MOO interfaces a robot policy with a pre-trained vision-language model that extracts object-identifying information from language and images. Real-robot evaluations report zero-shot generalization to novel objects and environments, multiple input modalities, and integrated open-world navigation and manipulation.
Problem
The paper asks whether robots can use semantic knowledge from static datasets to complete manipulation tasks involving object categories absent from their first-hand experience.
Method
MOO uses a frozen pre-trained vision-language model to extract object-identifying information from commands and images, conditioning a language-conditioned policy on that information and the current observation.
Results
Across 1,472 real-world evaluations, MOO generalizes to novel objects and environments, supports multiple input modalities, and combines with open-vocabulary navigation for mobile manipulation.
Takeaways & Limitations
The approach supports open-world object manipulation beyond verbal descriptions and can be integrated with open-vocabulary semantic navigation.
Takeaways & Limitations
MOO may struggle with overlapping or occluded objects, motion diversity outside training demonstrations, and instructions that do not fit its current templates.
Abstract
from arXiv · showhide
For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evaluation videos can be found at https://robot-moo.github.io/
1 Introduction
The paper addresses how robots can manipulate object categories absent from their first-hand experience by combining robot learning with semantic knowledge from pre-trained vision-language models. MOO uses a frozen VLM for coarse object identification while an end-to-end policy performs manipulation, enabling open-world generalization.
- Motivation: Robots must connect broad human vocabulary to sensory observations and actions, including object categories absent from their interaction history.Static internet-scale datasets provide semantic coverage beyond a robot’s first-hand experience.
- Motivation: Existing approaches face a trade-off between brittle pipelines that can generalize across object categories and less brittle policies that cannot ground unseen semantic concepts.Pipeline success depends on precise localization, while vanilla pre-training lacks a mechanism for grounding novel semantic object categories.
- Approach: MOO conditions a language-conditioned policy on object locations extracted by a frozen vision-language model, alongside the original observation and instruction.The policy is trained with the real VLM detector in the loop rather than relying on the VLM as a complete state-estimation pipeline.
- Approach: The policy learns from demonstrations covering 106 physically diverse objects, while the pre-trained model supplies semantic information for commands involving novel categories.This composition produces a language-conditioned policy for open-world object manipulation.
- Results: Across 1,472 real-robot evaluations, MOO is significantly more successful than recent robot-learning methods and supports multiple intent modalities plus open-world mobile manipulation.The demonstrated modalities include language, pointing, generic object images, and a GUI; CoW-MOO combines manipulation with open-vocabulary navigation.
2 Related Work
Related work has used pre-trained visual or language representations and studied several forms of robotic generalization. This paper instead emphasizes grounding entirely new semantic object categories through vision-language models in broader manipulation settings.
- Leveraging Pre-Trained Models in Robotic Learning: Prior robotic-learning work commonly builds policies on frozen or fine-tuned visual representations, pre-trained language embeddings, or other off-the-shelf perceptual models.These approaches extend pre-trained representations into robot-learning pipelines.
- Leveraging Pre-Trained Models in Robotic Learning: MOO differs by using vision-language models that ground language in visual observations, enabling semantic-category generalization unavailable from vision-only or language-only models individually.The paper positions this grounding as the basis for generalizing to novel semantic object categories.
- Generalization in Robotic Learning: Earlier robotic generalization studies often target novel word combinations, descriptions of familiar objects, or visual and physical attributes of unseen objects.These settings differ from entirely new words referring to object categories absent from the demonstration dataset.
- Generalization in Robotic Learning: The paper’s experimental setting requires visual, physical, and semantic object generalization, including grounding descriptions of previously unseen object categories.This distinguishes the target capability from generalizing only object appearance, physical properties, or language composition.
- Vision-Language Models for Robotic Manipulation: CLIPort and PerAct use CLIP as a vision-language backbone; MOO aims to extend this direction to more general manipulation settings.The stated extensions include removing dependence on depth cameras or camera calibration.
- Open-World Object Detection in Computer Vision: Traditional object detectors use fixed category maps, whereas open-world detection replaces fixed one-hot categories with shared representations intended to cover broader vocabularies.The passage frames open-world detection as a response to detectors’ limited trained category maps.
3 Manipulation of Open-World Objects (MOO)
MOO interfaces an instruction-conditioned robot policy with a VLM that localizes described objects, encoding their locations as masks for manipulation. Its training and evaluation setup targets generalization from 106 diverse training objects to unseen objects and categories.
- MOO overview: MOO uses a VLM to localize objects named in an instruction, then conditions the robot policy on the instruction and object-localization information.The method separates semantic object grounding from policy execution in a two-stage pipeline.
- Problem setup: Demonstrations contain observation-action sequences with structured instructions composed of a task template and object descriptions.The setup includes object descriptions such as “yellow banana” and “cup” within templates like “move X near Y.”
- Problem setup: The policy is trained on demonstrations with objects from S_robot but is evaluated on new instructions using familiar templates and novel object descriptions.The objective is to manipulate objects absent from the robot’s demonstration set.
- Representing object information: MOO represents each detected object by the center pixel of its VLM-predicted bounding box, using distinct mask values when multiple objects are specified.A single object uses a one-channel mask; for two objects, the first and second receive values 1.0 and 0.5.
- Representing object information: The object-centric pixel representation works across visible object sizes and can convert bounding boxes or segmentation masks into a common policy input.The paper also evaluates alternative object representations through ablations.
- Open-world generalization: Novel descriptions can be handled when represented in the VLM’s static training data, leaving the robot policy to interpret mask locations and manipulate the object’s physical shape.This avoids requiring the policy itself to ground every novel semantic description.
- Model architecture: The architecture appends the object mask channel-wise to current images and image history, while the task verb is encoded separately for the RT-1 policy.The image stream uses EfficientNet, FiLM conditioning, Token Learner, and a Transformer.
- Practical implementation: MOO uses Owl-ViT with descriptive natural-language prompts and extracts object information only in the first episode frame for practical real-time control.The policy must track or relocate the object in later current images if it moves.
4 Experiments
Experiments evaluate MOO’s object generalization, robustness, input-modality flexibility, scaling behavior, and integration with open-vocabulary navigation. Across these settings, MOO outperforms prior methods and extends manipulation to novel objects, environments, and task specifications.
- The experiments test generalization across skills and unseen objects, robustness to distractors, backgrounds, and environments, non-linguistic object specification, training-data and model scaling, and navigation integration.
- Generalization to Novel Objects: MOO improves substantially over baselines on pick tasks for both seen and unseen objects, with improvements of approximately 50% in each case.
- Robustness Beyond New Objects: MOO generalizes beyond new objects, showing greater robustness than VIMA-like and RT-1 across challenging textures, distractors, and environments.
- Input Modality Experiments: MOO supports masks generated from text, VLM-derived captions, target images, pointing, and GUI input, including scenes with duplicate objects and clutter.
- MOO Ablations: Unseen-object generalization is sensitive to training-object diversity, while reducing object diversity does not affect seen-object performance.
- MOO Ablations: Reducing model size from 111M parameters lowers seen performance from 98% to 54% and 39%, and unseen performance from 79% to 50% and 13%.Models larger than 111M parameters were not used because robot latency became unacceptable.
- Open-World Navigation and Manipulation: CoW-MOO combines open-vocabulary navigation with MOO manipulation, enabling a robot to find and manipulate objects it has never previously interacted with.
5 Conclusion and Limitations
The paper reports that MOO supports broad open-world manipulation capabilities, but identifies important boundaries involving visual ambiguity, motion diversity, instruction form, and spatial descriptions.
- Across 1,472 real-world evaluations, MOO generalizes to novel objects and instructions, challenging textures and environments, multiple input modalities, and open-vocabulary navigation.
- MOO may struggle with overlapping or occluded objects because its object-mask representation can become visually ambiguous.
- Policy generalization remains limited by training-motion diversity, especially for novel objects with shapes or sizes unlike demonstration objects.
- Instructions are expected to follow templates separating target objects and verbs, and complex spatial relations are not currently handled.
Imitation Learning and RT-1
MOO builds on language-conditioned imitation learning, where a policy maps task instructions and current observations to robot actions. Its implementation builds on the RT-1 robotics transformer and behavioral cloning.
- Language-conditioned imitation learning learns a policy π(a | ℓ, o) that selects robot actions from the task instruction and current observation.
- MOO builds on RT-1, a robotics transformer-based model, and uses behavioral cloning to train the language-conditioned policy.
Vision-Language Models
Vision-language models detect objects from natural-language queries by aligning text and image representations in a shared embedding space. This supports open-vocabulary object identification beyond fixed category labels.
- Vision-language models detect image objects using natural-language queries rather than a fixed category vocabulary.
- They tokenize and embed text, extract image features with a separate network, and combine both representations in a shared embedding space.
Datasets
The dataset combines teleoperated demonstrations with evaluation objects spanning seen objects, unseen objects in seen categories, and unseen categories. The training data emphasizes object diversity for picking.
- Training data: 13,239 teleoperated demonstration episodes focus on increasing object diversity for the picking task.Expert users collected the demonstrations with Oculus Virtual Reality controllers.
- Evaluation objects: The listed evaluation objects include household items such as cans, fruit, toys, utensils, plates, and other manipulation targets.
- Evaluation objects: The evaluation set contains 49 objects seen during training, 22 unseen objects from seen categories, and 25 objects from unseen categories.
Experiments
Experiments evaluate manipulation success across five skills and test robustness to difficult scenes, training-data reductions, alternative mask sources, and localization noise. The results indicate that data scale and diversity matter, while imperfect masks can still support successful manipulation.
- Skills: The evaluation measures successful commands for pick, move near, knock, place upright, and place into skills.For pick, success requires grasping the specified object and lifting it at least 6 inches from the tabletop.
- Robustness evaluation: Robustness tests cover visually challenging scenes with different furniture and backgrounds, including additional open-world objects absent from training.
- Mask robustness: MOO can recover from misspecified masks that are not centered on the target object for seen and novel objects.Table 2 reports 20 trials with artificially added localization noise.
- Input modalities: MOO is tested with object masks generated from alternative processes, including generative-VLM text captions and image queries, using a frozen policy.
- Mask robustness: Mask-noise performance decreases from 5/5 successes with centroid masks to 4/5 when remaining on-object and 3/5 when off-object or more than 5 cm away.
- Training-data ablation: Both training-data diversity and training-data scale are important for MOO performance.The study varies the number of unique training objects and total training episodes.
Prompts used
The experiments use natural-language prompts to OWL-ViT for object detection, while model scaling and qualitative visualizations examine performance and detections on unseen objects. Prompt wording maps object names to descriptive visual phrases.
- Prompt construction: OWL-ViT prompts are prefixed with “An image of a” before detecting objects.
- Model scaling: Pick success improves continuously on both seen and unseen objects as model parameter count increases with dataset size fixed.
- Model scaling: Larger model capacity is expected to yield further gains, but real-time inference constraints prevent scaling beyond the evaluated models.
- Prompt construction: Object names are mapped to descriptive phrases such as “white can of soda,” “round orange fruit,” and “small thin flat wooden popsicle stick.”
- Qualitative results: Visualizations show detections and grasps for objects not seen during training, with detection colors matching the corresponding text.Images are ordered from left to right across time.