HomeRobot: Open-Vocabulary Mobile Manipulation

1 Introduction

• OVMM asks robots to find and move arbitrary objects in arbitrary homes, including previously unseen objects that must reach specified receptacles.The benchmark evaluates full-stack integrated mobile-manipulation systems in human-centric environments with open object sets.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Prior mobile-manipulation work simplifies action spaces, object sets, or environments, while unstandardized robots and settings hinder comparison and cross-lab reproduction.Vision-language developments enable generalization beyond specific categories [9] [10] [11] [12] [13], often through CLIP [14], strengthening the motivation for a common platform and benchmarks.Source: §1 · pp. 1–2
• 200 human-authored interactive 3D scenes [19] in AI Habitat [20] [21] underpin the simulation benchmark paired with controlled-apartment real-world tests.Both settings mix seen and unseen object categories; real-world tests use the affordable, compliant Hello Robot Stretch [22]. The authors propose the first reproducible real-world mobile-manipulation benchmark, with planned yearly evaluation beginning with the NeurIPS 2023 competition.Source: §1 · p. 2
• HomeRobot provides identical simulation and real-world APIs plus a multipurpose Stretch robotics stack, supporting reproducible experiments beyond OVMM.The stack demonstrates sim-to-real capabilities and supports manipulation learning [24], continuous learning, navigation [26], and object-goal navigation [2].Source: §1 · p. 2, Table 1 · p. 3
• RL navigates more efficiently when objects are visible, whereas heuristic planning explores better over long horizons; replacing ground-truth segmentation with DETIC [27] substantially reduces performance.Both policies use DETIC [27] segmentation; the heuristic uses a motion planner demonstrated for real-world object search [2], while RL uses depth and predicted segmentation.Source: §1 · p. 3

Challenges

• HomeRobot targets a real-world benchmark with standard hardware sustainable for several years, addressing hardware and task changes that complicate comparisons.Participant-built platforms hinder algorithm comparisons in challenges. Even when organizers provide hardware, changing evaluation tasks makes progress difficult to track.Source: §2 · p. 3

Reproducibility of robotics research

• Open robot designs and low-cost platforms pursue standardization, but varying environments encourage isolated component evaluation rather than assessment within larger systems.HomeRobot seeks to move beyond standardized object sets [50–52] toward a common set of robots, objects, and environments.Source: §2 · pp. 3–4
• HomeRobot combines a full modular robotics stack with challenging simulated tasks and corresponding real-world infrastructure to support end-to-end benchmarking and reproducibility.Multiple sub-module implementations allow components to be evaluated within the complete system. The authors distinguish HomeRobot as the only benchmark providing a broadly capable code-sharing robotics stack; PyRobot lacks a strong simulation benchmark.Source: §2 · p. 4, Figure 2 · p. 4

Real World Benchmarks

• RoboTHOR [33] benchmarks navigation using shared scenes and objects, while HaPy supports real-world LoCoBot navigation testing without a generally useful robotics stack.HaPy lacks the broader visualizations, debugging, motion planners, and tooling discussed for a general robotics stack.Source: §2 · p. 4
• TOTO and NIST Task Board are limited to tabletop manipulation, while RB2 ranks manipulation algorithms locally.TOTO supplies training data and runs experiments for users, but training and testing occur in the same environments. NIST Task Board evaluates fine-grained manipulation skills.Source: §2 · p. 4

3 Open-Vocabulary Mobile Manipulation

• OVMM asks a robot in an unknown single-floor home to move a named household object from a specified starting receptacle to any valid goal receptacle.Objects are small and manipulable; receptacles are furniture surfaces supporting object placement. The starting receptacle is always specified to guide the search.Source: §3 · p. 4
• Evaluation uses unseen object instances from both seen and unseen categories in held-out scenes, while every receptacle category is seen during training.Training uses seen instances of seen object categories; validation and testing exclude those instances. Receptacles may reappear in held-out scenes, but at previously unseen locations.Source: §3 · p. 5, §3.1 · p. 6
• Four successive stages determine success: finding the object on its starting receptacle, picking it up, finding the goal receptacle, and placing it there.Overall success requires all stages; partial success awards 1 point per successive completed stage, normalized by the number of stages, as a tie-breaker. Multiple objects can satisfy a query.Source: §3 · pp. 4–5

3.1 Simulation Dataset

• 60 HSSD [19] scenes support simulation experiments, divided into 38 training, 12 validation, and 10 test scenes.The dataset annotates 2,535 objects across 129 categories and identifies 21 receptacle categories. Simulation supports reproducibility, training, and fast iteration.Source: §3 · p. 4, §3.1 · p. 5
• Simulation preparation refines stable placement surfaces and collision meshes, while episode generation requires navigable, reachable object locations and goal receptacles.Automatic surface labeling is manually refined to remove invalid or inaccessible receptacles; collision proxies are corrected where needed for physically accurate placement. Episodes require navigation from the object location to a navigable, reachable goal.Source: §3.1 · pp. 5–6

3.2 Real World Benchmark

• 20 experiments per baseline evaluate two baselines in a controlled three-room apartment, using eight object categories and seven receptacle classes.All test object instances are absent from simulation training; five categories are seen and three unseen. Trials document object and robot starting positions for reproducibility and use the Hello Robot Stretch [22] with DexWrist.Source: §3 · p. 5, §3.2 · p. 6

HomeRobot Library

• The open-source HomeRobot library supports navigation and manipulation on Hello Robot’s Stretch [22], splitting robot control from workstation-based perception and planning.The robot runs low-level control and localization; an NVIDIA GPU-powered workstation runs high-level perception and planning. They communicate through an off-the-shelf router.Source: HomeRobot Library · p. 6
• Unified simulation–real-world state and action spaces support transferability, while modular perception and action components expose high-level representations and behaviors.Control options range from premade grasping policies to continuous joint control. Components support semantic maps, segmented point clouds, goal-directed navigation, and object pickup.Source: HomeRobot Library · p. 7

4.1 Baseline Agent Implementation

• The heuristic baseline combines motion planning [2] and simple manipulation rules with DETIC masks for open-vocabulary objects.It uses off-the-shelf learned models and heuristics, with the start receptacle, object, and goal receptacle supplied for each episode.Source: §4.1 · p. 7
• Example projects cover object-goal navigation [1] [2], skill learning [24], continual learning, and image instance navigation [26].These implementations let researchers test capabilities from recently released papers.Source: §4.1 · p. 7
• OVMMAgent implements a state machine that executes FindObj, Gaze, Pick, FindRec, and Place in that order.FindObj/FindRec locate the object and goal receptacle; Gaze approaches and adjusts the view for grasping, while Place deposits the object on the goal receptacle. Pick uses a robot-library grasping policy in reality and a high-level action in Habitat, which does not simulate gripper interactions; learned grasping skills are supported.Source: §4.1 · p. 7
• The reinforcement-learning baseline trains four skills in modified Habitat [21] using DDPPO, with depth, ground-truth semantic segmentation, and proprioceptive inputs.Policies do not directly use RGB; at test time, they instead rely on predicted segmentation from Detic [27].Source: §4.1 · p. 7

Simulation

• Simulation evaluated held-out scenes with unseen instances of seen and unseen classes, comparing simulator-provided ground-truth segmentation against DETIC [27] open-vocabulary detection from RGB images.Results track FindObj, Pick, FindRec, and Place, with each skill’s success dependent on preceding skills. Place success equals overall episode success, while the partial-success metric averages the four skill success rates.Source: §5 · p. 7, Table 3 · p. 8
• RL outperformed heuristic navigation and placement in simulation, but every policy performed worse with DETIC than with ground-truth segmentation.Heuristics degraded less; heuristic FindObj surpassed RL with DETIC, attributed to noise-tolerant 2D semantic mapping that helps handle small objects prone to misclassification. Learned gaze improved picking except when combined with heuristic navigation and DETIC.Source: Table 3 · p. 8, §5 · p. 8

Real World

• 20% real-world success gave RL one more completed episode than the heuristic baseline across 20 episodes targeting seen and unseen object classes.RL gaze improved alignment for grasping, while RL placement kept objects closer to the receptacle surface.Source: §5 · p. 8

Overall Findings

• Perception errors, constrained multi-room navigation, and placement remained major limitations, with DETIC [27] misclassification causing failures in simulation and reality.RL navigation matched or exceeded heuristics in both settings, but real-world RL placement still needed improvement. The authors suggest tuning web-scale pretrained vision-language models to the agents.Source: §5 · p. 8, Table 3 · p. 8

6 Conclusions and Future Work

• The heuristic baseline using motion planning [2] and the DDPPO-trained reinforcement-learning baseline show promising results in simulation and the real world.Both real-world baselines use the same grasping procedure, but initial robot positions relative to objects or obstacles can cause failures.Source: Table 4 · p. 8, §6 · p. 8
• Future work targets more complex natural language, multi-step commands, and end-to-end baselines instead of modular policies.These extensions would increase the complexity of the benchmark’s problem space.Source: §6 · p. 8

A Extended Related Work

• HomeRobot provides a simulator for generating large-scale, consistent training data rather than restricting users to a fixed dataset, and evaluates models in unseen environments.Most interaction-dataset works discussed instead use the same environment for training and testing.Source: A Extended Related Work · p. 17
• HomeRobot extends reproducible comparison into the real world, whereas simulation benchmarks retain a reality gap and sometimes impose simplified discrete action spaces [20] [6].Some previous work even imposes that discrete structure on real-world operation [2].Source: A Extended Related Work · p. 17
• Hardware diversity complicates robotics benchmarking: simulation may not reflect real-world progress [2], while competitions using team-specific hardware hinder fair comparisons and cross-lab transfer.Differences in hardware, morphology, and resources also create barriers to participation.Source: A Extended Related Work · p. 17
• Gervet et al. [2] find planning-based exploration competitive with the best learning-based methods and substantially stronger than end-to-end reinforcement learning on real tasks.Their approach combines heuristic/model-based planning and reinforcement learning across varied real-world environments.Source: A Extended Related Work · p. 17
• Kurenkov et al. and SayPlan [95] search existing scene graphs, leaving iterative graph construction in unseen environments unresolved.Despite active work on iterative exploration and hierarchical representation construction [96], strongly established methods are not yet available.Source: A Extended Related Work · p. 17

B Limitations

• Grasping is not physically simulated in the current benchmark, so a separate policy handles grasping in the real world.Source: B Limitations · p. 17
• Full natural-language queries are currently outside the benchmark’s scope.Source: B Limitations · p. 17
• Although grasping is well studied, simulations used to train useful real-world grasping systems require special consideration.Source: B Limitations · p. 17
• Task-and-motion planning with replanning was not performed, although it would be ideal for long-horizon tasks.Source: B Limitations · p. 18
• The evaluation did not cover many motion planners.Source: B Limitations · p. 18

C Metrics

• The metrics section formalizes the partial success metrics introduced informally in Section 3.Source: C Metrics · p. 18

C.1 Simulation Success Metrics

• 0.1m viewpoint proximity and at least 0.1% relevant-pixel visibility define simulated FindObj and FindRec; FindRec additionally requires successful Pick.FindObj targets an object on start_receptacle and counts pixels belonging to an object instance. FindRec targets a goal_receptacle and counts pixels belonging to the object containing a valid receptacle.Source: §C.1 · p. 18
• 0.8m end-effector proximity to a target object, a visible object instance, gripper activation, and successful FindObj establish simulated Pick success.The object is magically snapped to the gripper rather than physically grasped.Source: §C.1 · p. 18
• 50 contiguous steps of stable goal-receptacle contact after release, successful FindRec, and collision-free placement are required for simulated Place success.Linear and angular velocities must remain below 5e−3 m/s and 5e−2 rad/s, respectively.Source: §C.1 · p. 18
• 1250 steps is the simulation episode limit, within which all four stages must succeed.The stages are FindObj, Pick, FindRec, and Place.Source: §C.1 · p. 18
• 79% reaching success versus 47% collision-free reaching success characterizes the tested grasping baseline when starting from an object viewpoint.The proposed stricter simulated condition replaces magical snapping with arm approach near the object without collisions with the scene or other objects. The baseline uses top-down grasps with sideways fallback; most grasping failures involve scene collisions.Source: §C.1 · p. 18, Figure 5 · p. 19

C.2 Real World Success Metrics

• Real-world Pick requires successful FindObj and actually picking up the object; Place requires successful FindRec and stable placement on a goal_receptacle.The object must be picked up from start_receptacle and settle stably on goal_receptacle.Source: §C.2 · p. 18
• In the real-world scene, which is smaller than the simulated apartments, agents are allowed 300 timesteps per episode, and episode success requires success in all four stages.FindObj requires reaching within 1m of the target object on start_receptacle, with the object visible in the camera’s RGB image. FindRec requires successful Pick and reaching within 1m of a goal_receptacle visible in the camera’s RGB image.Source: §C.2 · p. 18

D.1 Object Categories Appearing in the Scene Dataset / D.3 Diversity in Receptacle Instances

• The dataset spans household object categories, separates training-seen from unseen categories, and includes substantial variation within receptacle categories.Example training categories include cushion, cup, and pan; vase, plate, and plant saucer appear in validation and test sets. Table instances illustrate within-category receptacle diversity.Source: §D.1 · p. 19, Figure 6 · p. 20, Figure 7 · p. 20

D.2 Episode Generation Details / D.4 Scene Clutter Complexity

• Episode generation samples split-specific objects onto reachable receptacles in the largest indoor navigable area, using physics to ensure stable placement.Object counts range from 1.5–2 times receptacle area in m2; navigable viewpoints within 1.5 meters that show receptacles train navigation and initialize placement and gaze/grasping. Procedural target-and-distractor clutter requires approach-direction reasoning, stable placement, and open-vocabulary detection under occlusion.Source: Figure 8 · p. 21, Figure 9 · p. 21, §D.2 · p. 22

D.5 Improved scene visuals

• 3% lower training FPS, from around 340 to around 330, accompanies the improved HBAO- and PBR-based scene rendering.The renderer adds Horizon-based Ambient Occlusion and expands Physically-Based Rendering support, improving scene visuals for HSSD [19].Source: §D.5 · pp. 23–24, Figure 13 · p. 23, Figure 14 · p. 24

D.6 Action Space Implementation / E.2 Low-Level Control for Navigation

• 0.25 m forward steps and fixed rotations define discrete actions, whereas continuous actions predict local waypoints checked against the navmesh.The hardware controller turns toward the goal, translates, then aligns orientation, using trapezoidal velocity profiles and overshoot checks. Fast Marching Method planning [2] assumes a cylindrical agent, restricting navigation, while DETIC [27] semantic mapping requires predefined classes.Source: §D.6 · p. 24, §E.2 · p. 25

E HomeRobot Implementation Details / E.1 Pose Information

• HomeRobot separates shared components, Habitat simulation, and robot hardware into three software repositories.Agents execute heuristic and reinforcement-learned policies; Environments supply observations and apply actions in real or simulated settings. Hector SLAM provides Hello Robot Stretch [22] poses for constructing 2D semantic navigation maps.Source: E HomeRobot Implementation Details · pp. 24–25, §E.1 · p. 25

E.3 Heuristic Grasping Policy

• The top-down heuristic grasps object regions with free space on both sides, achieving performance comparable to tested open-source models.It voxelizes the object's top 10% at 0.5 cm resolution, scores occupied interiors and empty finger regions, smooths scores to reject outliers, and thresholds candidates. Pretrained ContactGraspnet, 6-DoF Graspnet, and Graspnet 1-Billion suffered more intermittent sensor-noise failures despite gripper-offset adaptation.Source: §E.3 · p. 26

E.4 Heuristic Placement Policy

• Heuristic placement requires a barely visible destination receptacle and projects its DETIC [27] segmentation into a point cloud to select a placement location.It samples 50 points and selects the center of the largest slab, scoring surrounding X/Y points within a 3 cm height threshold. The robot faces the location, advances if farther than 38.5 cm, re-estimates placement, positions the gripper a few centimeters above it, and releases.Source: §E.4 · pp. 26–27

E.5 Navigation Planning / 3. Otherwise, plan to reach the closest frontier

• Navigation backprojects Detic [27] detections into semantic maps and uses Fast Marching Method to replan at every step.Planning prioritizes an object co-occurring with its start_receptacle, then a start_receptacle outside regions within 1 m of past locations, otherwise the closest frontier by geodesic distance. Excluding previously approached regions prevents revisiting receptacle instances.Source: §E.5 · p. 27, §3 · p. 28

E.6 Navigation Limitations

• Adapting navigation from Gervet et al. [2] to grasp diverse objects required extensive tuning to balance close approaches against collision avoidance.The original system targeted large objects from only six classes; supporting many more classes makes this balance difficult.Source: §E.6 · p. 28
• 5cm world discretization and orientation-blind Fast Marching Method planning limit navigation, motivating alternative sampling-based or optimization-based planners.Suggested alternatives include RRT-Connect, planners from Task and Motion Planning [8], and optimization-based open-vocabulary navigation planning [12].Source: §E.6 · p. 28
• Navigation depends on Hector SLAM pose accuracy, cannot handle dynamic obstacles, and ignores Stretch’s offset rotation center within its cylindrical robot model.Sampling-based planners are suggested as a possible improvement.Source: §E.6 · p. 28

F Reinforcement Learning Baseline

• Four reinforcement-learning policies implement FindObject, FindReceptacle, GazeAtObject, and PlaceObject.FindObject and FindReceptacle collectively provide the navigation skills.Source: F Reinforcement Learning Baseline · p. 28, §F.1.1 · p. 29

F.1.1 Navigation Skills

• FindObject and FindReceptacle use discrete actions, which experiments found better for exploration and easier to train.These navigation skills use the discrete action space described in Section D.6.Source: §F.1.1 · p. 29

F.1.2 Manipulation Skills

• Manipulation skills use continuous actions for fine-grained control, with simulated movement limits reflecting real-world low-level controller constraints.Per step, simulation allows 10-25 cm forward motion or 5-30 degrees turning; head tilt/pan and gripper yaw/roll/pitch changes are capped at 0.02-0.1 radians, and arm extension/lift changes at 2-10cm. Training teleports the base and arm to target locations.Source: §F.1.2 · p. 29

F.2 Observation Space

• Policies observe head-camera depth, semantic segmentation, relative robot pose, camera pose, and joint states, but do not use RGB during training.Robot pose is relative to the starting pose and comes from SLAM in real-world operation; joint states include the gripper.Source: §F.2 · p. 29

F.3 Training Setup

• −0.005 per step penalizes delay across all skills, while 160x120 training images replace Stretch’s original 640x480 resolution to accelerate training.Source: §F.3 · p. 29

F.3.1 Navigation Skills

• FindObject and FindReceptacle share navigation training, using goal specifications and segmentation masks to approach candidate objects or receptacles.Agents start at least 3 meters away and predict translation/rotation waypoints plus stop actions; FindObject receives a CLIP [14] embedding and start-receptacle category, while FindReceptacle receives the goal-receptacle category. Training uses ground-truth goal masks with object-level dropout; real-robot masks come from DETIC [27].Source: §F.3.1 · pp. 29–30
• 0.5 meters from a goal viewpoint and heading error no greater than 15 degrees define the navigation stop-success condition.Navigation rewards combine geodesic-distance reduction, heading improvement within 3 meters, and a collision-indicator term.Source: §F.3.1 · p. 30

F.3.2 GazeAtObject

• GazeAtObject refines base position and camera tilt until a visible candidate object is centered and within grasping reach.Training starts at precomputed viewpoints with the object assumed visible; actions include base translation/rotation, camera tilt, and a discrete grasp command. Success requires the center pixel to identify a candidate object and the base to be within 0.8 meters; rewards encourage distance reduction and looking toward the object.Source: §F.3.2 · p. 30

F.3.3 PlaceObject

• 50 timesteps of remaining on the receptacle after release defines PlaceObject success.Starting near a goal receptacle, the robot moves its arm to place the held object on a free surface location and opens its gripper.Source: §F.3.3 · p. 31

F.4 ConceptFusion

• ConceptFusion [15] explores multi-image 3D mapping as an alternative to the baselines’ open-vocabulary detector [27], which does not use prior timesteps.The tested implementation combines SAM [110] segmentation with CLIP [14] mask features. Simulation supplies ground-truth depth and poses, removing the need for GradSLAM [111].Source: §F.4 · p. 31
• Up to 5 seconds per frame was required for ConceptFusion map construction, alongside substantial computation and memory demands.ConceptGraphs [112] addresses some encountered computational challenges, but its exploration is left for future work.Source: §F.4 · p. 31

G Additional Analysis

• With DETIC perception, RL exploration is less efficient than the heuristic planner; real-world RL policies sometimes explore insufficiently to find a goal.Reported per-skill step counts consider only cases where the skill terminates.Source: Table 5 · p. 32, §G.1 · p. 32

G.1 Number of steps taken in each stage by different baselines

• Gaze and Place policies trained with ground-truth perception take significantly longer to terminate when using DETIC perception.RL FindObj terminates more often than heuristic FindObj, while episodes terminate less frequently with DETIC than with ground-truth perception.Source: Table 5 · p. 32, §G.1 · p. 33, Table 6 · p. 33

G.2 Performance on Seen vs. Unseen Object Categories

• Performance differs little between unseen instances from seen versus unseen categories, with the baselines relying on DETIC rather than learning semantic perception.SC,UI denotes unseen instances of categories present in training; UC,UI denotes unseen instances of categories absent from training.Source: Table 7 · p. 33, §G.2 · p. 33

G.2.1 Example DETIC predictions

• DETIC [27] inconsistently detects target objects in egocentric observations taken where the Gaze skill is expected to begin.Gaze moves the base and tilts the camera to approach and center the target. The evaluated DETIC vocabulary contains fixed receptacle categories and the target object name.Source: §G.2.1 · p. 33, Figure 19 · p. 34, Figure 20 · p. 35

H.1 Hardware Choice

• Supporting affordable research, Stretch with DexWrist balances manipulation, navigation, and cost while remaining human-safe and approximately human-sized.The hardware comparison includes the non-mobile Franka Emika Panda as a price reference because it is common in research laboratories.Source: Table 8 · p. 34, H Hardware Setup · p. 34, §H.1 · p. 34

H Hardware Setup

• The hardware appendix provides real-world robotics-stack setup guidance, visualization tools, and development practices to make mobile-robot research easier.It expands on hardware choices and practical setup considerations.Source: H Hardware Setup · p. 34

H.2 Robot Setup

• HomeRobot uses a workstation for compute-intensive processing while the GPU-less Stretch runs low-level control and SLAM.A dedicated router connects the systems, and navigation pauses after most actions until fresh images arrive. All systems use a robot-published ROS clock to prevent mismatched SLAM poses and depth measurements.Source: §H.2 · pp. 35–36, Figure 21 · p. 36

H.3 Visualizing The Robot

• HomeRobot integrates existing ROS debugging tools and uses RVIZ to visualize perception, navigation, and registered sensor data.Outputs include DETIC-processed images, a planner-generated top-down map, and head-camera point clouds registered against Hector SLAM’s 2D lidar map. ROS offers online resources, official Hello Robot support, and an open-source community.Source: §H.3 · p. 36

• Stretch’s right-facing arm prevents simultaneous forward viewing and manipulation; this design supports lower cost and fits the human profile [22].Source: §H.4 · pp. 36–37
• Navigation mode points the camera forward and uses reactive low-level control to rotate in place, roll backward, and track goals sent from the desktop.Source: §H.4 · p. 37
• Manipulation mode points the camera toward the arm, disables reactive navigation and base rotation, and treats the base as an additional lateral manipulation degree of freedom.Source: §H.4 · p. 37
• Heuristic policies switch to manipulation mode near grasping or placement goals, using lateral base movements to act as if Stretch had a full 6dof manipulator.These changes make the low-cost robot more capable and easier to use across tasks [12] [24].Source: §H.4 · p. 37