PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge

Sriram Yenamandra, Arun Ramachandran, Mukul Khanna, Karmesh Yadav, Jay Vakil, Andrew Melnik, Michael Büttner, Leon Harz, Lyon Brown, Gora Chand Nandi, Arjun PS, Gaurav Kumar Yadav, Rahul Kala, Robert Haschke, Yang Luo, Jinxin Zhu, Yansen Han, Bingyi Lu, Xuan Gu, Qinyuan Liu, Yaping Zhao, Qiting Ye, Chenxiao Dou, Yansong Chua, Volodymyr Kuzma, Vladyslav Humennyy, Ruslan Partsey, Jonathan Francis, Devendra Singh Chaplot, Gunjan Chhablani, Alexander Clegg, Theophile Gervet, Vidhi Jain, Ram Ramrakhya, Andrew Szot, Austin Wang, Tsung-Yen Yang, Aaron Edsinger, Charlie Kemp, Binit Shah, Zsolt Kira, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, Chris Paxton

arXiv:2407.06939v1cs.ROcs.CV

TL;DR

Open-vocabulary mobile manipulation asks robots to find objects in novel environments and place them on receptacles, a challenge for home assistants handling diverse objects. The paper presents a competition spanning simulation and physical robots, and reports that successful approaches emphasized error recovery and perception-informed decision-making.

  • Problem

    Home assistants must perceive and interact with diverse objects across environments, while many objects are poorly represented in training data.

    Source: §1 · pp. 1–2

  • Method

    The paper proposed an open-vocabulary mobile manipulation competition with simulation and physical-robot evaluation, supported by a modular control stack and baselines.

    Source: §1 · p. 2

  • Results

    Successful approaches emphasized error detection and recovery, with teams adding retries and checks around task skills and perception remaining a challenge in real and simulated scenarios.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    The competition points to substantial remaining work in perception, error recovery, and scaling robot learning to real-world environments.

    Source: §6 · p. 9

    Real-world evaluation relied on one apartment and robot, constraining evaluation throughput by their availability.

    Source: §5 · p. 9

Abstract

from arXiv · show

In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings.

1 Introduction

The paper frames open-vocabulary mobile manipulation as finding an object in a novel environment and placing it on a receptacle, then introduces a competition to evaluate this capability in simulation and on physical robots. Its lessons emphasize perception limits, error recovery, and the challenges of applying robot learning in open environments.

1 Introduction

  • The benchmark’s core task is to move an object from a start location to a goal location, requiring perception, planning, and policy learning in unseen environments.The paper identifies these as jointly necessary for the task.Source: §1 · p. 2
  • The competition evaluated open-vocabulary mobile manipulation in simulation and on physical robots, targeting navigation and manipulation of previously unseen objects in novel environments.This contrasts with prior work that assumed known classes.Source: §1 · p. 2
  • The paper provided a modular, algorithm-agnostic control stack with both reinforcement-learning and heuristic baselines for testing component advances.The stack supports work on perception, grasping, and navigation without reimplementing the full control infrastructure.Source: §1 · p. 2
  • 61 teams and 79 submissions took part, while open-vocabulary perception remained a bottleneck in challenging real and simulated scenarios.Only three teams were selected for real-world evaluation.Source: §1 · p. 2
  • Current perception models were not sufficient on their own, and teams engineered around open-vocabulary vision models to improve performance.The paper also stresses continual planning and reasoning about next actions because perception is unreliable at larger distances.Source: §1 · p. 2
  • Teams reported that reinforcement learning did not seem to reduce engineering effort, and tuning reward functions was difficult compared with using a classical robotics stack.Source: §1 · p. 3

2 Competition Setup

The challenge combined simulation qualification with evaluation on physical robots, using shared metrics and a containerized evaluation pipeline. Its setup included held-out simulation splits, real-world trials, and modular HomeRobot baselines and components.

2 Competition Setup

  • Entries were evaluated in simulation and on physical robots using overall success, four subtask success measures, and action count, with winners selected by overall success.The real-world challenge followed the simulation qualifier.Source: §2 · p. 3

2.1 Simulation Challenge

  • Habitat supplied cluttered simulated environments, and the challenge used 60 HSSD scenes with held-out scenes and object categories in test splits.Train, validation, and minival episodes were released for local training and evaluation; the test splits targeted generalization to unseen objects and environments.Source: §2.1 · pp. 3–4

2.2 Real Robot Challenge

  • The top three teams advanced to real-world evaluation on a Hello Robot Stretch RE2, with three evaluations per team in a controlled apartment setting.Docker containers enabled use of the same environment and code from simulation; the experiments totaled about 10 hours.Source: §2.2 · p. 5

2.3 Containerized Testing

  • The containerized pipeline used task, environment, and participant images; environment and participant containers exchanged sensor observations and actions through gRPC during evaluation.The task image provided a shared base, while the environment image used private dataset paths and the participant image contained the submitted approach.Source: §2.3 · p. 6

2.4 HomeRobot Library

  • The HomeRobot library provided heuristic and reinforcement-learning baselines plus modular components for perception, navigation, grasping, and manipulation.Participants could choose which parts to address, including using discrete grasp planning without access to a physical robot.Source: §2.4 · p. 6

3 Results

The simulation challenge produced a large gap between the top team and the next best entry, while the real-world phase evaluated the top three teams on a physical robot.

3.1 Simulation

  • 10.8% overall success rate put UniTeam 7.6 percentage points ahead of the next best team on the simulation test-challenge leaderboard.Table 1 reports simulation results for the top six teams and the original HomeRobot baselines.Source: §3.1 · p. 6, Table 1 · p. 7

3.2 Real World

  • Real-world trials halted after three collisions, with partial success awarded according to the task step reached.The evaluation policy explored, navigated, picked, navigated again, and placed.Source: §3.2 · p. 7

4 Analysis and Discussion

The analysis finds that perception remains a major obstacle and that stronger task-level error recovery distinguishes successful approaches. It also notes challenges in manipulation, simulation-to-real transfer, and learning-based solutions.

Perception

  • Open-vocabulary perception remained a major bottleneck: teams used varied detection and segmentation models, but many still struggled to recognize previously unseen objects.UniTeam also reported goal receptacles predicted with high confidence despite being incorrect.Source: §4 · pp. 7–8

Task Model

  • Unlike the baseline’s one-pass sequence of four skills, UniTeam’s state machine checks success and retries or returns to earlier steps.The figure shows four states: navigating to the object, picking, navigating to a receptacle, and placing.Source: §4 · p. 8, Figure 5 · p. 8
  • Choosing when and how to retry failed skills remains an open research problem, with vision-language models noted as a possible route to automatic error-recovery behavior.The paper contrasts this prospect with manual engineering.Source: §4 · p. 8

Manipulation

  • Manipulation challenged all teams: placement reward engineering and motion-planning collision avoidance were difficult, although basic skills transferred reasonably well to the real world.Source: §4 · p. 8

Simulation vs Real World

  • Survey respondents lacked access to a real-world Stretch robot, which the authors say suggests simulation is useful for testing and development.Source: §4 · p. 8

The Role of Learning in Successful Solutions

  • The winning team used no learning, while two of the top three fine-tuned; teams reported that making learning work took more time than improving model-based solutions.Source: §4 · p. 8

5 Running the Challenge: Lessons Learned

The challenge lessons focus on participant communication, the limits of centralized real-world evaluation, and the value of containerized testing.

Enhancing Communication for Future Challenges

  • Participants reported unclear or late-breaking information about deadlines and submission limits, motivating recommendations for faster, more reliable communication.The authors recommend platforms such as Discord rather than a mailing list.Source: §5 · pp. 8–9

Distributed Evaluation

  • Relying on one real-world evaluation environment bottlenecked scaling because access to the robot and apartment was limited.The authors suggest distributed evaluation to increase diversity and resilience.Source: §5 · p. 9

Containerized Testing

  • Docker allowed identical policies to be tested in simulation and real-world settings across countries and labs without dependency issues.The authors recommend continuing containerized evaluation.Source: §5 · p. 9

6 Conclusions

The paper concludes that improving perception, error recovery, and the scaling of robot learning remains important. Its UniTeam example illustrates a largely heuristic approach augmented with detection, navigation, exploration, and manipulation changes.

A Team UniTeam

  • UniTeam’s mostly heuristic approach used Detic and added improvements to navigation, exploration, and manipulation, with little learning.Source: A Team UniTeam · p. 13

A.1 Overview

  • The agent repeatedly detects objects and receptacles, then explores, navigates, picks, and places; failed picking returns it to exploration.Detection uses Detic masks together with RGB-D observations.Source: §A.1 · p. 13

A.2.1 Detection

  • Object-specific lower thresholds improved object detection, while higher receptacle thresholds reduced end-receptacle misclassification.Depth-based height filtering also removed floor-level detections that could otherwise be treated as objects or receptacles.Source: §A.2.1 · p. 13

A.2.2 Exploration

  • To avoid oscillating around an unattainable goal, the navigation decision mechanism can choose between the goal point and exploring the frontier.The goal map stores likely end receptacles and selects the reachable receptacle with the highest matching probability.Source: §A.2.2 · p. 14

A.2.3 Navigation

  • Navigation incorporates detected collisions into the obstacle map and aims to position the agent near the receptacle center for placement.Source: §A.2.3 · p. 14

A.2.4 Picking

  • After a failed initial pick, the agent rotates to widen its view; it advances only after verifying pickup and otherwise returns to navigation and exploration.Source: §A.2.4 · p. 14

A.2.5 Placing

  • Placement adjustments include moving in small steps, keeping away from receptacle edges, using a nearby flat surface when the goal is undetected, and dropping large objects from higher up.Source: §A.2.5 · p. 15

A.3 Panorama Update

  • The later panorama approach uses regular 360° image capture, OWL-ViT v2 detection, and MobileSAM segmentation to cover unexplored areas more completely.The agent begins with a panorama and takes another after entering an unexplored map region.Source: §A.3 · p. 15

A.4 Major Issues

UniTeam’s failures centered on object and receptacle detection, while manipulation and simulation conditions introduced additional obstacles. These included arm collisions, objects outside the camera view, navigation getting stuck, incorrect receptacle labels, and objects that never settled.

A.4 Major Issues

  • Incorrect object or receptacle detection and classification were the most prominent issue, and incorrect end receptacles could receive high confidence scores.Detection precision varied by object or receptacle.Source: §A.4 · p. 15
  • Arm collisions during picking or placement, objects outside the camera frame, oversized receptacles, and getting stuck also caused failures.The robot could hit miscellaneous objects or the receptacle itself while extending its arm.Source: §A.4 · p. 15
  • Simulation labeling and success conditions could also block completion: a sink was not labeled as one, and pens could roll without coming to rest.The sink-label error caused placement attempts to fail; objects had to come to rest to count as successful.Source: §A.4 · p. 15

B Team Rulai

Team Rulai divided open-vocabulary mobile manipulation into detection, navigation, and manipulation, combining domain-adaptive segmentation with policy-based mobile manipulation. Its approach adapted the heuristic baseline and targeted each stage of the task.

B Team Rulai

  • Rulai’s approach combined domain-adaptive semantic segmentation with policy-based mobile manipulation, organized around detection, navigation, and manipulation.The manipulation stage covered picking and placing.Source: B Team Rulai · p. 16
  • For detection, Rulai fine-tuned VLDet using a mixed dataset of simulator images and real images from common datasets such as LVIS [Gupta et al. 2019] and VLDet [Lin et al. 2023].Source: B Team Rulai · p. 16
  • For navigation, Rulai built on the heuristic baseline with autonomous escape, target-region selection, and accelerated exploration strategies.Source: B Team Rulai · p. 16
  • For manipulation, the agent explored open areas to find a reachable location with enough space, then selected a placement point.Source: B Team Rulai · p. 16

B.1 Improvements over the Baseline

Rulai improved detection, navigation, and manipulation over the baseline. Its changes included indoor-data fine-tuning and model selection, navigation recovery and exploration, and searching for a suitable placement position and point.

B.1 Improvements over the Baseline

  • Rulai improved detection, navigation, and manipulation to achieve its second-place performance.Source: §B.1 · p. 16

B.1.1 Detection

  • Rulai fine-tuned its baseline detector on indoor and simulator images, then selected a model balancing performance and response time after benchmarking eight vision-language detectors.The benchmark used images extracted from Habitat-Sim and the competition’s runtime requirement.Source: §B.1.1 · p. 16

B.1.2 Navigation

  • Rulai added self-rescue to avoid getting stuck, rapid exploration using prior-state encoding, and target navigation-area selection.The rapid-exploration method aimed to reduce the overall number of operations.Source: §B.1.2 · p. 16

B.1.3 Manipulation

  • After reaching the receiver, Rulai rotated to search for a suitable placement area; if none appeared in view, it navigated to another receiver.Source: §B.1.3 · p. 16
  • Once it found an area, Rulai adjusted orientation and moved closer, then matched the area mask to the object footprint to select a placement point.Source: §B.1.3 · p. 16

B.2 Major Issues

The reported issues span detection, navigation, and placement, while Team KuzHum’s changes focused on perception, reinforcement-learning skills, and retries. Its segmentation pipeline combined open-vocabulary detections with task-specific masks, and placement fine-tuning encouraged keeping the object visible and approaching the receptacle.

B.2 Major Issues

  • Detection errors included word-object mismatch and high false positives from general-dataset training on low-fidelity simulator images; mixed-data training may help while preserving real-world perception.The team identified further work on the reality gap as worth exploring.Source: §B.2 · p. 17
  • Navigation was constrained by unknown layouts, no distance sensors, and imprecise target locations, complicating path planning and target finding.Source: §B.2 · p. 17
  • Placement could fail when the navigation endpoint left the arm unable to reach, or when objects landed in positions where they slid or shook.The team noted that placement position and point selection matter, including choosing a middle or open area to reduce slipping risk.Source: §B.2 · p. 17

C.1 Improvements over the Baseline

  • KuzHum enabled YOLO object tracking for more robust segmentation and fine-tuned its placement module with a modified reward function.Source: C Team KuzHum · p. 17
  • KuzHum’s changes focused on detection, improved reinforcement-learning skills, and retrying high-level skills; its baseline analysis found that replacing ground-truth segmentation with Detic sharply reduced skill success.Because perception influenced every skill and episode completion, the team focused on improving it.Source: §C.1 · p. 17, §C.1.1 · p. 17

C.1.1 Detection

  • KuzHum combined fine-tuned YOLOv8 for known classes with MobileSAM segmentation and Detic detection and segmentation for unseen classes.YOLOv8 bounding boxes prompted MobileSAM; masks were combined by overlaying task-specific objects, preserving Detic’s open-vocabulary masks alongside YOLO-SAM’s furniture and small-object distinctions.Source: §C.1.1 · p. 17, Figure 7 · p. 18

C.1.2 RL Checkpoint Finetuning

  • For the Place phase, KuzHum adjusted the training reward to encourage keeping the object in camera view and moving closer to the goal receptacle.These changes targeted the low success rate of the place skill, which directly affected overall task success.Source: §C.1.2 · pp. 17–18

C.1.3 Task Model

The task-model changes address premature episode endings after unsuccessful Gaze attempts by using pickup sensing to guide recovery. Fine-tuning the RL placement policy remained difficult, with reward exploitation and unresolved camera blockage limiting progress.

C.1.3 Task Model

  • Repeated unsuccessful Gaze attempts could end exploration even when the task had not failed, so pickup sensing enabled conditional recovery behavior.The agent could attempt Gaze up to 10 times, then revert to Navigate to object.Source: §C.1.3 · p. 18
  • The revised heuristic made the agent more robust to faulty goal-object detections and consequent Gaze calls.After 10 failed Gaze attempts, it returned to Navigate to object.Source: §C.1.3 · p. 18

C.2 Major Issues

  • RL place-policy fine-tuning was difficult: prolonged training could lower success as the agent exploited reward without successful placement.The policy sometimes dropped the object at the start; camera view blockage by the manipulator also remained unresolved.Source: §C.2 · p. 18

D Simulation Results on Test-Standard Split

On the test-standard split, UniTeam remained the top team, while only six teams exceeded 1.0% on the leaderboard. The section also describes the real-world test environments and the topics covered by the participant survey.

E Full Set of Test Environments

  • Ten environments were used for real-world baseline testing, but time constraints limited competition testing to three, including environments with unseen objects.The environments combined seen and unseen objects, including objects absent from test or training data.Source: E Full Set of Test Environments · p. 18, Figure 8 · p. 19

D Simulation Results on Test-Standard Split

F Survey Questions

  • The participant survey asked about teams’ methods, navigation and manipulation approaches, foundation models, detection changes, real-world testing, and challenge feedback.Questions also covered team affiliations, associated reports, surprises, and participation in a follow-up challenge.Source: F Survey Questions · pp. 20–21

Related papers

Loading 2407.06939v1…