PaperlayerView exact revision on arXiv

Source-linked AI summary

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

arXiv:2401.12202v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Robotics systems generalize poorly to unseen, unstructured homes, even as vision and robot skills advance. OK-Robot combines open-knowledge models with navigation and grasping primitives for pick-and-drop, achieving 58.5% success across ten homes, with a higher rate under more favorable conditions.

  • Problem

    Robotics systems remain brittle in unseen scenarios, while errors in object retrieval, navigation targets, and grasping can compound.

    Source: §I · p. 2

  • Method

    OK-Robot combines open-knowledge vision-language models with navigation, grasping, and dropping components to perform pick-and-drop without additional training.

    Source: §I · p. 2, §II · p. 3

  • Results

    58.5% success on full pick-and-drops across ten homes rose to 82.4% with improved queries, decluttering, and exclusion of adversarial objects.

    Source: §I · p. 2, A. Results of home experiments · p. 6

  • Takeaways & Limitations

    The authors find that careful component integration and heuristics counteracting physical limitations matter for real-world success.

    Source: §I · p. 2

    The authors identify unresolved opportunities to improve methods operating in completely novel home environments.

    Source: §V · p. 9

Abstract

from arXiv · show

Remarkable progress has been made in recent years in the fields of vision, language, and robotics. We now have vision models capable of recognizing objects based on language queries, navigation systems that can effectively control mobile systems, and grasping models that can handle a wide range of objects. Despite these advancements, general-purpose applications of robotics still lag behind, even though they rely on these fundamental capabilities of recognition, navigation, and grasping. In this paper, we adopt a systems-first approach to develop a new Open Knowledge-based robotics framework called OK-Robot. By combining Vision-Language Models (VLMs) for object detection, navigation primitives for movement, and grasping primitives for object manipulation, OK-Robot offers a integrated solution for pick-and-drop operations without requiring any training. To evaluate its performance, we run OK-Robot in 10 real-world home environments. The results demonstrate that OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered environments, OK-Robot's performance increases to 82%. However, the most important insight gained from OK-Robot is the critical role of nuanced details when combining Open Knowledge systems like VLMs with robotic modules. Videos of our experiments and code are available on our website: https://ok-robot.github.io

I. INTRODUCTION

Robotics systems remain brittle in unseen, unstructured homes despite progress in vision and robot skills, and errors across components can compound. OK-Robot integrates open-knowledge models and robotics primitives, achieving 58.5% zero-shot success across ten homes, with higher success under more favorable conditions.

I. INTRODUCTION

  • Brittle systems often fail in unseen scenarios, especially in unstructured homes with little robotic data, despite progress in data-driven robotics.Vision models and robot skills have advanced, yet the winning solution in the NeurIPS 2023 OVMM challenge achieved 33%.Source: §I · p. 2
  • OK-Robot integrates vision-language models with navigation and grasping primitives for pick-and-drop, using learned models trained on large public datasets.A home scan seeds semantic memory, which is queried with an object-language embedding before navigation and picking.Source: §I · p. 2
  • 58.5% success across ten real-world homes came from zero-shot pick-and-drop; success reached 82.4% with improved queries, decluttering, and exclusion of adversarial objects.The reported 82.4% condition excludes objects described as too large, translucent, or slippery.Source: §I · p. 2
  • Pre-trained vision-language models such as CLIP and OWL-ViT can identify arbitrary real-world objects and support zero-shot navigation to them.The paper reports this as an observation from its experiments.Source: §I · p. 2
  • Pre-trained grasping models can be applied to mobile manipulation in homes without additional training or fine-tuning.The paper describes special-purpose robot models trained on large datasets as usable out of the box.Source: §I · p. 2
  • A no-training state machine combines the pre-trained models, while heuristics that counteract physical limitations can improve real-world success.The authors emphasize that how components are combined matters.Source: §I · p. 2
  • The authors identify substantial room to improve VLMs, robot models, and robot morphology for open-knowledge manipulation agents.They state that improvements in these areas will directly increase performance.Source: §I · p. 2

II. TECHNICAL COMPONENTS AND METHOD

OK-Robot combines open-vocabulary object navigation, RGB-D grasping, and a dropping heuristic on a mobile robot. Its navigation module builds a scanned semantic map, retrieves likely object locations from language queries, and plans a safe, reachable approach.

II. TECHNICAL COMPONENTS AND METHOD

  • The method combines open-vocabulary object navigation, open-vocabulary RGB-D grasping, and a dropping heuristic on a Hello Robot: Stretch to handle pick-and-drop queries.The query specifies an object to pick, an optional source place, and a destination place.Source: §II · p. 3

A. Open-home, open-vocabulary object navigation

  • A manual iPhone scan supplies posed RGB-D images; OWL-ViT detections and SAM masks are back-projected into a 5 cm VoxelMap with CLIP semantic vectors.The map is static after the initial scan and cannot adapt during robot operation.Source: A. Open-home, open-vocabulary object… · p. 3
  • Navigation balances reachability, gripper clearance, and obstacle avoidance, selecting a position using three scores and planning over a 10 cm grid with A*.Occupied and unexplored cells are non-navigable, with a 20 cm buffer around obstacles.Source: A. Open-home, open-vocabulary object… · pp. 3–4, Figure 2 · p. 4

B. Open-vocabulary grasping in the real world

OK-Robot uses a pretrained grasp generator to propose candidates, then language-conditioned masking and a scoring heuristic select one for execution. A staged approach and closed-loop gripper motion complete the pickup.

B. Open-vocabulary grasping in the real world

  • AnyGrasp generates collision-free parallel-jaw grasps from an RGB image and point cloud, with each proposal carrying a grasp point, dimensions, and an uncalibrated confidence score.Source: B. Open-vocabulary grasping in the… · p. 4
  • LangSam segments the queried object and filters AnyGrasp proposals to grasp points inside its mask, as Fig. 3 illustrates.The selected grasp is chosen from the filtered set.Source: B. Open-vocabulary grasping in the… · p. 4, Figure 3 · p. 5
  • The heuristic subtracts θ^4/10 from grasp score, favoring horizontal grasps that are more robust to hand-eye calibration errors across homes.Horizontal grasps are preferred because vertical grasps require better hand-eye calibration.Source: B. Open-vocabulary grasping in the… · p. 4
  • Execution approaches the grasp point in progressively smaller motions, closes the gripper in a closed-loop manner, then lifts and retracts the arm.Slower motion near the object helps avoid knocking over light objects; the wrist rotates to tuck the object over the robot body during navigation.Source: B. Open-vocabulary grasping in the… · p. 4

C. Dropping heuristic

The dropping heuristic estimates a release height from a segmented target point cloud, while deployment combines the navigation, pick, and drop primitives into an automatic sequence. The paper evaluates this system in novel homes and examines its operating time and failure modes.

C. Dropping heuristic

  • For concave drop targets, the heuristic sets release height 0.2 above the maximum point in a central segmented region, but does not explicitly reason about clutter.It aligns and normalizes the segmented point cloud before estimating the drop height.Source: C. Dropping heuristic · p. 5

D. Deployment in homes

  • A new home is scanned in under a minute, processed into VoxelMap in under five minutes, and ready for its first pick-and-drop task within 10 minutes on average.Source: D. Deployment in homes · p. 5
  • After the user specifies the object and destination, a predefined state machine automatically runs navigation, grasping, goal navigation, and dropping without error detection or correction.Source: D. Deployment in homes · p. 5

III. EXPERIMENTS

  • The evaluation ran 171 pick-and-drop trials across 10 homes, filtering navigation failures before sequential trials and examining alternate primitives, home challenges, and component failures.Figure 4 aggregates outcomes across three cleaning phases and breaks failures down by component and long-tail mode.Source: D. Deployment in homes · p. 5, §III · p. 6, Figure 4 · p. 6

A. Results of home experiments

In novel homes, OK-Robot completes zero-shot pick-and-drops; module comparisons and cleanup analyses show how performance varies across components and environmental conditions.

A. Results of home experiments

B. Ablations over system components

C. Impact of clutter, object ambiguity, and affordance

  • As cleanup removes ambiguous objects and clutter, navigation accuracy rises as total error falls from 15% to 12% to 4%, while manipulation error falls from 25% to 16% to 13%.Dropping-primitive failure rate stays roughly constant across the three cleanup phases.Source: C. Impact of clutter, object… · p. 7

D. Understanding the performance of OK-Robot

Failures arise across semantic retrieval, grasp generation and execution, and robot hardware. The section identifies query ambiguity and manipulation constraints as recurring sources of failure.

D. Understanding the performance of OK-Robot

  • The coarse breakdown identifies manipulation as the leading failure module, while a closer analysis reveals a long tail of causes.The section’s failure analysis includes semantic-memory retrieval, difficult grasp poses, and hardware difficulties among the leading causes.Source: D. Understanding the performance of… · p. 7
  • The three leading failure causes were retrieving the wrong navigation object from semantic memory (9.3%), obtaining a difficult manipulation pose (8.0%), and robot hardware difficulties (7.5%).Some queries may fail while semantically similar, slightly modified wording of the same query succeeds.Source: D. Understanding the performance of… · p. 7
  • Ambiguous queries can retrieve the wrong object, especially when scenes contain visually or semantically similar objects.An interactive confirmation step is suggested as a way to avoid such retrieval issues.Source: D. Understanding the performance of… · p. 7
  • Grasp poses can be unreliable or unrealistic because they are generated from one RGB-D image by a model not designed for the Stretch gripper.Some proposed poses are infeasible under joint limits or too far from the robot body.Source: D. Understanding the performance of… · p. 7
  • Using the same grasp trajectory in every case can lead to collisions with minor obstacles, even when the grasp pose is good.The module also struggles to grasp flat objects with its two-fingered gripper; the robot has a 1 kg payload limit and reach constraints.Source: D. Understanding the performance of… · p. 8

A. Vision-Language models for robotic navigation

Prior work has explored open-vocabulary navigation, pretrained grasping, and language-enabled robot systems, with differing limits in object scope and deployment setting. The paper also identifies opportunities to improve its approach.

A. Vision-Language models for robotic navigation

  • Open-vocabulary navigation has progressed from pretrained vision-language representations to systems that localize objects and navigate in real environments, though some work remains simulated or controlled.Earlier work often covers relatively few objects, while Objaverse demonstrates navigation to thousands of object types.Source: A. Vision-Language models for robotic… · p. 8

B. Pretrained robot manipulation models

  • Pretrained grasping models trained on large datasets predict grasps across object surfaces, while newer general-purpose manipulation models extend beyond grasping.Some language-conditioned models are limited to a small set of scenes and objects; HACMan focuses on pushing and prodding.Source: B. Pretrained robot manipulation models · pp. 8–9

C. Open vocabulary robot systems

  • Open-vocabulary mobile manipulation differs from language systems limited to a fixed set of goals; prior systems include modular pick-and-place and GPT-based planning approaches.SayPlan handles large environments and multi-step tasks but does not address implementing pick and place.Source: C. Open vocabulary robot systems · p. 9

V. LIMITATIONS, OPEN PROBLEMS AND REQUESTS FOR RESEARCH

  • Despite success in completely novel home environments, the method still has several areas for improvement.Source: §V · p. 9

A. Live semantic memory and obstacle maps

The discussion identifies improvements spanning world representation, grasp planning, user interaction, failure recovery, and robot hardware. These issues constrain continuous use and overall task success.

A. Live semantic memory and obstacle maps

  • Current semantic memories and obstacle maps are static, while homes change over time.The authors propose dynamic representations as a direction for continuous pick-and-drop in novel homes.Source: A. Live semantic memory and obstacle… · p. 9

B. Grasp plans instead of proposals

  • The grasping module proposes generic grasps without accounting for the robot’s body or dynamics, and open-loop trajectories can collide with obstacles.The authors suggest generating grasp plans rather than poses alone.Source: B. Grasp plans instead of proposals · p. 9

C. Improving interactivity between robot and user

D. Detecting and recovering from failure

  • Independent module errors accumulate multiplicatively: even components each succeeding at or above 80% can yield an overall success rate below 60%.The authors suggest error detection and retries as ways to recover from more single-stage errors.Source: D. Detecting and recovering from failure · p. 9

E. Robustifying robot hardware

  • The Stretch platform’s hardware constrains reachable locations, payload, and grasp precision.The authors note that more robust hardware may enable reaching high and low places and lifting heavier objects; improved odometry may support finer grasps.Source: E. Robustifying robot hardware · p. 9

APPENDIX B SCANNET200 TEXT QUERIES

The appendix lists ScanNet200 text labels used by OWL-ViT to detect objects in a home environment.

APPENDIX B SCANNET200 TEXT QUERIES

  • The listed labels include household objects and fixtures such as shower head, spray, inhaler, guitar case, and toilet paper dispenser.Source: Appendix B · pp. 13–14
  • Additional labels include towel roll, stove, mailbox, scissors, tape, laptop, toaster, and cabinet.Source: Appendix B · p. 13
  • The vocabulary also includes refrigerator, mug, bed, couch, plant, pillow, lamp, and sink.Source: Appendix B · p. 13

APPENDIX C SAMPLE OBJECTS FROM OUR TRIALS

The experiments sampled objects from home environments that could plausibly be handled by the Hello Robot Stretch gripper.

APPENDIX C SAMPLE OBJECTS FROM OUR TRIALS

  • Objects were sampled from home environments based on whether the Hello Robot Stretch gripper could plausibly manipulate them.The samples included objects with varied shapes and visual features.Source: Appendix C · p. 14
  • Figures 9 and 10 show a subsample of the objects encountered in the experiments.Source: Appendix C · p. 14
  • The sampled objects came from the home environments used in the experiments.Source: Appendix C · p. 14

APPENDIX D SAMPLE HOME ENVIRONMENTS FROM OUR TRIALS

The appendix shows evaluated home environments and two additional homes where OK-Robot was reproduced.

APPENDIX D SAMPLE HOME ENVIRONMENTS FROM OUR TRIALS

  • Figure 11 shows a subset of the home environments used to evaluate the method.Source: Appendix D · p. 14
  • OK-Robot was reproduced in two homes outside New York: one in Pittsburgh, Pennsylvania, and one in Fremont, California.Source: Appendix D · p. 14
  • The authors checked that OK-Robot functioned in those homes by attempting pick-and-drop on several objects.Source: Appendix D · p. 14

APPENDIX E LIST OF HOME EXPERIMENTS

Appendix E lists home experiments and trial outcomes, with task categories and success rates reported across ten trials.

APPENDIX E LIST OF HOME EXPERIMENTS

  • The appendix includes examples of objects successfully picked and dropped and objects the robot failed to pick up.Source: Figure 9 · p. 15, Figure 10 · p. 16
  • Table I lists home tasks by category and reports success rates out of 10 trials.Source: Table I · p. 19
  • Individual records classify attempts as success, navigation failure, manipulation failure, or placing failure.Source: Appendix E · p. 22
  • Home 5 records include none, low, and high cleanup conditions; pink shampoo is a navigation failure at low cleanup and a success at high cleanup.Source: Appendix E · p. 22
Loading 2401.12202v2…