Source-linked AI summary
HomeRobot: Open-Vocabulary Mobile Manipulation
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, Chris Paxton
TL;DR
Household robots must manipulate arbitrary objects in unseen homes, a challenge requiring integrated perception, language, navigation, and manipulation. HomeRobot addresses this with a reproducible OVMM benchmark spanning simulation and real-world execution, and evaluates heuristic and reinforcement-learning baselines that achieve 20% real-world success. The authors identify perception and real-world placement as areas needing improvement.
Problem
OVMM asks robots to find and move arbitrary objects in arbitrary homes, requiring integrated perception, language understanding, navigation, and manipulation.
Method
HomeRobot combines multi-room simulation, a real-world Hello Robot Stretch stack, identical APIs, and heuristic and reinforcement-learning baseline policies.
Results
20% real-world success was achieved across 20 episodes, with reinforcement learning completing one more episode than the heuristic baseline.
Takeaways & Limitations
The benchmark enables reproducible integrated mobile-manipulation evaluation across simulation and physical environments.
Takeaways & Limitations
DETIC misclassification caused many failures, and the real-world reinforcement-learning place policy needs further improvement.
Abstract
from arXiv · showhide
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. In addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments to grasp novel objects and place them on target receptacles. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic (model-based) baselines and show evidence of sim-to-real transfer. Our baselines achieve a 20% success rate in the real world; our experiments identify ways future research work improve performance. See videos on our website: https://ovmm.github.io/.
1 Introduction
The paper frames Open-Vocabulary Mobile Manipulation as the integrated challenge of finding and moving arbitrary objects in homes, then introduces reproducible simulation and real-world infrastructure to evaluate it.
- OVMM requires robots to find and move arbitrary objects across human environments, integrating perception, language understanding, navigation, and manipulation.
- Prior work generally simplified mobile manipulation through discrete actions, limited object sets, or small single-room environments, while environments and robots remained unstandardized.
- The proposed benchmark combines challenging multi-room simulation with a reproducible real-world benchmark spanning seen and unseen object categories.
- HomeRobot provides identical APIs across simulation and physical settings, plus a Hello Robot Stretch stack supporting OVMM and related robotics sub-tasks.
- The baselines compare heuristic planning with reinforcement learning, revealing efficiency advantages for visible objects, exploration advantages for heuristics, and a substantial segmentation-related performance drop.
2 Related Work
Related work includes robotics datasets, benchmarking challenges, reproducibility efforts, and real-world manipulation benchmarks; HomeRobot combines a broad stack with simulation and end-to-end evaluation.
- Prior robotics benchmarks often evaluate isolated components because robots and environments vary, limiting fair comparison and reproducibility.
- Procedurally generated scenes such as ProcThor offer an unbounded number of environments, whereas HomeRobot emphasizes reproducible multi-purpose benchmarking with real-robot deployment.
- HomeRobot extends standardized robotics efforts with a full stack, multiple sub-module implementations, and end-to-end benchmarking across common robots, objects, and environments.
- The benchmark couples varied, long-horizon simulation with a corresponding real-world stack, while prior real-world benchmarks commonly remain local, tabletop, or environment-specific.
3 Open-Vocabulary Mobile Manipulation
OVMM requires a robot to move a named household object from a known starting receptacle to a valid goal receptacle in an unknown home. The benchmark evaluates this task across simulated and real environments with varied objects, scenes, and staged success criteria.
- Task definition: OVMM instructs a robot to move a named object from a start receptacle to any valid goal receptacle in an unknown single-floor home.Success requires finding the start receptacle, picking up the object, finding the goal receptacle, and placing the object.
- Benchmark settings: The benchmark evaluates both simulation and real-robot settings to support reproducibility, training, fast iteration, and experiments with real data.Simulation enables varied, long-horizon experimentation, while the real-world stack supports tests of learned and heuristic baselines.
- Generalization: Evaluation uses seen and unseen object categories and instances, with held-out validation and test scenes that do not appear during training.Training uses seen instances of seen categories, while validation and test include unseen instances from seen or unseen categories.
- Simulation dataset: The simulation dataset combines 2,535 objects across 129 categories with 21 receptacle categories, using HSSD scenes and procedurally generated object arrangements.Receptacles are processed for stable, accessible placement, and collision proxies support physically accurate arrangements.
- Real-world benchmark: Real-world evaluation uses a controlled three-room apartment, eight object categories, 20 trials per baseline, and seven receptacle classes.The object categories include five seen and three unseen categories, with test instances absent from simulation training.
HomeRobot Library
HomeRobot is an open-source stack that exposes shared simulation and real-world capabilities for mobile manipulation. Its OVMMAgent composes perception, navigation, grasping, and placement skills, alongside heuristic and reinforcement-learning baselines.
- System design: HomeRobot provides a mobile-manipulation stack with shared state and action spaces across simulation and physical robots.The library supports high-level actions and low-level continuous joint control, with modular perception and action components.
- Baselines and tools: The library includes heuristic and reinforcement-learning baselines plus example projects spanning object-goal navigation, skill learning, continual learning, and image instance navigation.These tools are intended to support systematic exploration of OVMM and related capabilities.
- Baseline agent: OVMMAgent is a state machine that executes FindObj, Gaze, Pick, FindRec, and Place in sequence.Pick uses a robot-library grasping policy in the real world, while the other skills are implemented with the described approaches.
- Baseline implementations: The heuristic baseline uses DETIC open-vocabulary masks, motion planning, and simple manipulation rules, whereas RL policies use depth, segmentation, and proprioceptive inputs trained with DDPPO.The RL policies do not directly use RGB during simulation training but rely on DETIC-predicted segmentation at test time.
5 Results
The baselines are evaluated first in held-out simulation scenes and then in a held-out real apartment. RL generally performs better for navigation and placement, but perception errors and constrained environments remain major obstacles.
- Evaluation: Table 3 reports partial and overall success rates across skill and perception combinations, with overall success requiring completion of the final Place phase.Partial success averages the four staged success rates, whose values depend on preceding stages.
- Simulation: In simulation, RL outperformed heuristic methods for navigation and placement, but DETIC perception reduced performance for all policies.With DETIC, heuristic FindObj surpassed RL, partly because its semantic map handled noisy predictions and small-object misclassification.
- Real World: 20% real-world success was achieved, with RL completing one more episode than the heuristic baseline in 20 total episodes.The difference primarily came from RL’s pick and place skills, including improved grasp alignment and placement precision.
- Limitations: The baselines were promising but insufficient for OVMM because DETIC misclassification caused failures in both simulation and the real world.RL navigation was on par with or better than heuristic navigation, while real-world RL placement still needed improvement.
6 Conclusions and Future Work
The paper introduces a combined simulation and real-world OVMM benchmark with heuristic and reinforcement-learning baselines. It reports promising but limited real-world performance and identifies infrastructure, reproducibility, and future task-complexity needs.
- 6 Conclusions and Future Work: The benchmark combines simulation and real-world evaluation for Open-Vocabulary Mobile Manipulation, using heuristic and DDPPO-based reinforcement-learning baselines.The paper frames this infrastructure as a way to enable progress on integrated mobile manipulation.
- 6 Conclusions and Future Work: Future work includes more complex natural-language and multi-step commands, plus end-to-end baselines instead of modular policies.
B Limitations
The current benchmark leaves physical grasping outside simulation and does not cover full natural-language queries. The authors also did not evaluate many motion planners or task-and-motion planning with replanning.
- B Limitations: Physical grasping is not simulated, so the benchmark supplies a separate grasping policy for real-world experiments.
- B Limitations: Full natural-language queries are currently out of scope.
- B Limitations: The study evaluates neither many motion planners nor task-and-motion planning with replanning, despite the task's long-horizon character.
C.1 Simulation Success Metrics
Simulation and real-world episodes are scored stage by stage across finding, picking, finding the receptacle, and placing. Overall success requires completing all four stages within the allotted time.
- C.1 Simulation Success Metrics: Simulation FindObj requires reaching within 0.1m of an object viewpoint and observing an object instance in at least 0.1% of camera pixels.
- C.1 Simulation Success Metrics: Simulation Pick uses a visible object and an end-effector within 0.8m, then magically snaps the object to the gripper.
- C.1 Simulation Success Metrics: Simulation Place requires stable contact for 50 contiguous steps, velocities below 5e−3 m/s and 5e−2 rad/s, and no collision during placement.
- C.1 Simulation Success Metrics: An episode succeeds only when FindObj, Pick, FindRec, and Place all succeed within 1250 simulation steps.
- C.1 Simulation Success Metrics: A stricter simulated grasp condition replaces snap-based picking by requiring arm motion near the object without collisions, with sideways grasps used when top-down access is unavailable.
- C.1 Simulation Success Metrics: Real-world success also requires four stages, but episodes are limited to 300 timesteps because the evaluated scene is smaller than simulated apartments.
D.1 Object Categories Appearing in the Scene Dataset
The scene dataset combines varied object categories, receptacles, clutter, navigable viewpoints, and improved rendering to create challenging open-vocabulary manipulation environments. Navigation supports discrete or continuous actions while heuristic control remains constrained by its motion-planning assumptions.
- D.1 Object Categories Appearing in the Scene Dataset: The dataset includes objects from multiple categories, with train, validation, and test splits separating seen and unseen categories for open-vocabulary manipulation.
- D.1 Object Categories Appearing in the Scene Dataset: Episodes use navigable geometry and precomputed viewpoints to place objects near reachable receptacles and train navigation, placement, gaze, and grasping skills.
- D.1 Object Categories Appearing in the Scene Dataset: Receptacle instances vary substantially, while procedural target and distractor placement creates clutter, occlusion, approach-direction, and stable-placement challenges.
- D.1 Object Categories Appearing in the Scene Dataset: Improved HBAO and PBR rendering raises visual quality while reducing ObjectNav training speed by only 3%, from around 340 to around 330 FPS.
- D.1 Object Categories Appearing in the Scene Dataset: Navigation agents can use fixed-step discrete actions or continuous local-waypoint predictions, with navmesh checks preventing invalid simulated motion.
- D.1 Object Categories Appearing in the Scene Dataset: The Fast Marching Method planner models the agent as a cylinder, limiting orientation-aware navigation compared with sampling-based planners such as RRT-connect.
- D.1 Object Categories Appearing in the Scene Dataset: Pretrained grasping methods showed intermittent sensor-noise failures after adaptation to Stretch's gripper, leaving better grasp-policy training for future work.
E.6 Navigation Limitations
The navigation baseline combines heuristic and RL skills, but its planner and sensing assumptions limit robustness in open-vocabulary environments.
- Navigation limitations: The navigation system needed extensive tuning to approach objects closely enough for grasping while avoiding collisions across many object classes.The earlier system targeted only six large object classes; broader coverage makes reliable tuning difficult.
- Planner limitations: The Fast Marching planner ignores orientation and uses 5cm world discretization, limiting its suitability for manipulation-aware navigation.The authors suggest sampling-based or optimization-based planners as alternatives.
- Sensing and modeling: The policy depends on accurate Hector SLAM poses, does not handle dynamic obstacles, and approximates the robot location as a cylinder.The cylinder model also overlooks the Stretch’s offset center of rotation.
- RL design: RL navigation uses discrete actions because they explored better and were easier to train than alternatives in the experiments.The navigation skills are FindObject and FindReceptacle.
- Inputs: The policies use depth, semantic segmentation, SLAM-relative pose, camera pose, and joint states, while RGB is available but unused during training.
F.3 Training Setup
The training setup decomposes OVMM into navigation, gaze, and placement skills trained with task-specific observations, actions, rewards, and starting states.
- Skill decomposition: The baseline trains navigation, gaze, and placement skills with distinct success conditions and reward structures for reaching, centering, grasping, and releasing objects.The skills include FindObject, FindReceptacle, GazeAtObject, and PlaceObject.
- Navigation skills: FindObject and FindReceptacle receive goal embeddings and segmentation masks, start at least 3m from targets, and predict waypoints plus a stop action.Navigation success requires stopping near a goal viewpoint while facing the target.
- Navigation rewards: The navigation reward combines geodesic progress, angular alignment near the goal, and collision penalties.The formulation uses α, β, and γ to weight these terms.
- Gaze skill: GazeAtObject refines base position and camera tilt until the object is centered, visible, and within arm’s length before grasping.It uses depth, segmentation, joint states, gripper state, and a CLIP target embedding, with success requiring a centered object and base distance below 0.8m.
- Placement skill: PlaceObject starts near a goal receptacle, controls the base and arm, and succeeds when the released object remains on the receptacle for 50 timesteps.Its reward includes contact, sustained placement, and failed-release terms.
- Perception dependency: The baseline depends on open-vocabulary detections, although the underlying 2D detector does not use prior time steps to inform detection.
G Additional Analysis
Additional analyses show that RL and heuristic policies behave differently across skills, while DETIC perception and real-world system constraints remain important sources of inefficiency.
- Skill efficiency: RL skills explored less efficiently than the heuristic planner, while successful RL episodes used fewer steps; RL could also give up when objects were not nearby.
- Perception effects: Gaze and Place policies trained with ground-truth perception took significantly longer to terminate when evaluated with DETIC perception.
- Skill invocation: RL FindObject terminated more often than heuristic FindObject, while DETIC reduced episode termination frequency relative to ground-truth perception.
- Seen versus unseen objects: Performance showed no large difference between seen-category/unseen-instance and unseen-category/unseen-instance objects because the baselines relied on DETIC for generalization.
- Detection analysis: DETIC often failed to detect objects consistently in egocentric views, especially for the Gaze skill’s novel-object centering behavior.
- Real-world infrastructure: The real-world system separates desktop visualization and high-level computation from robot-side control, SLAM, and time-synchronized observations.The setup uses ROS networking, a dedicated router, and updated robot images to align poses and depth measurements.
- Visualization: RVIZ visualizes DETIC images, the planner’s top-down map, and registered point clouds against the SLAM map.
H.4 Using The Stretch: Navigation vs. Position Mode
Stretch separates navigation and manipulation because its camera and arm geometry prevent simultaneous forward viewing and object manipulation.
- Hardware constraint: Stretch cannot look in its travel direction and manipulate objects simultaneously because its arm points to the right of its motion direction.This geometry lowers cost and preserves a human-scale form factor.
- Operating modes: Navigation mode faces the camera forward and uses reactive low-level control, while manipulation mode faces the arm and disables base rotation.
- Mode transition: Heuristic policies switch into manipulation mode after approaching a goal and use limited lateral base motion as an extra manipulation degree of freedom.
- Practical consequence: These hardware and control changes make the low-cost robot more capable and easier to use across tasks.