HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigation

• Prior ObjectNav benchmarks typically use 6–21 fixed object categories, while real-world robots must also generalize to open-set goals specified through language.The introduction motivates open-vocabulary ObjectNav as navigation to objects seen or unseen during training.Source: §I · p. 1
• HM3D-OVON evaluates generalization to seen categories, synonymous categories, and categories neither seen nor semantically similar during training.The benchmark uses the HM3DSem scene dataset.Source: §I · p. 1
• DAgRL+OD improves end-to-end generalization on VAL SEEN SYNONYMS and VAL UNSEEN by 9.6–18.8% in success rate.DAgRL combines DAgger with frontier-exploration trajectories; DAgRL+OD adds an object detector and a navigation module for bee-lining to detected objects.Source: §I · p. 2
• DAgRL’s success rate drops 11.9–23.0% on VAL SEEN SYNONYMS and VAL UNSEEN relative to VAL SEEN, while VLFM achieves 32.4–35.2% success across all three splits.The introduction also reports that DAgRL is more robust to simulated real-world noise than VLFM.Source: §I · p. 2
• Transformers, DAgger, and frontier-exploration trajectories each outperform their respective alternatives in the reported policy analyses.The trajectory finding contradicts SPOC’s reported advantage for shortest-path trajectories; the authors also analyze agent failure modes.Source: §I · p. 2

• Earlier benchmarks can restrict goals to a small fixed category set: HM3D ObjectNav has 6 categories and MP3D ObjectNav has 21.The section contrasts these limitations with HM3D-OVON’s broader categories and real-world scene scans rather than synthetic scenes.Source: §II · p. 2, Table I · p. 2
• Modular ObjectNav systems combine exploration, recognition, mapping, and path planning, while end-to-end methods map sensor observations directly to actions.Modular approaches may use heuristic exploration, semantic maps, object detectors, and fast-marching path planning.Source: §II · p. 2
• The paper compares transformers with RNNs and investigates imitation-learning algorithms and trajectory types for end-to-end open-vocabulary ObjectNav.The related-work section notes that prior end-to-end methods often use RNNs, and that imitation learning has used demonstrations or shortest-path planners.Source: §II · p. 3

A. ObjectNav task definition

• An episode supplies RGB, depth, odometry, and a goal category; success requires stopping within 1 m of a goal within 500 steps.The simulated agent follows Stretch specifications, with a 1.41 m height, 17 cm base radius, and an RGB-D camera at 1.31 m.Source: A. ObjectNav task definition · p. 3

B. The HM3D-OVON dataset

• HM3D-OVON uses HM3DSem annotations to provide 379 goal categories across 181 photorealistic scans, with goals visible in at least 5% of the camera view from a vantage point within 1 m.Training and evaluation use 145 and 36 scenes respectively, with no scene or goal-instance overlap.Source: B. The HM3D-OVON dataset · p. 3
• Evaluation splits share scenes but use mutually exclusive goal categories, with VAL SEEN using categories encountered during training and the other splits using unseen categories.The table caption specifies that scenes and object instances are unseen during training in all evaluation splits.Source: B. The HM3D-OVON dataset · p. 3, Table II · p. 3
• VAL SEEN SYNONYMS contains unseen categories similar to training categories, whereas VAL UNSEEN contains categories semantically divergent from them.SentenceBERT similarity to training categories determines the assignment to these two splits.Source: B. The HM3D-OVON dataset · p. 3

C. Episode generation

• Episodes sample a goal category and start position with a same-floor goal 1–30 m away; the protocol yields 50k episodes per training scene and 3k per validation scene.The benchmark uses 145 training scenes and 36 validation scenes.Source: C. Episode generation · p. 3

IV. HM3D-OVON BASELINES

• The OVON policy concatenates frozen SigLIP visual and goal embeddings with a previous-action embedding, then uses a four-layer transformer over up to 100 observations to predict actions.RL adds a critic head; RNN comparisons replace the transformer with a four-layer LSTM of similar parameter count.Source: §IV · pp. 3–4, Figure 2 · p. 4
• Demonstration training compares frontier exploration followed by goal-directed bee-lining with shortest-path trajectories, generating 7.25 million trajectories of each type for behavioral cloning.Frontier exploration continues until the goal is within 3.5 m, then a shortest-path planner guides the agent to it.Source: §IV · p. 4
• DAgger labels the policy’s own online trajectories with expert actions, motivating a fast frontier-and-bee-line expert that suggests actions in an average of 1 ms per time step.The authors contrast this with prior frontier-based trajectory generation taking 250 ms per time step.Source: §IV · pp. 4–5
• Baselines include PPO with shaped distance-to-goal and success rewards, BC- or DAgger-pretrained policies fine-tuned with sparse-reward RL, and modular VLFM and DAgRL+OD approaches.VLFM uses an open-vocabulary detector and point-goal policy; DAgRL+OD combines a trained DAgRL explorer with an object detector and point-goal bee-lining policy.Source: §IV · p. 5
• BC and DAgger train for 150M steps; BCRL and DAgRL receive another 150M PPO steps, while RL trains for 300M steps for comparison.Training uses 16 environments per GPU across eight TITAN Xp GPUs.Source: §IV · p. 5

V. RESULTS

• The evaluation asks how RL, imitation learning, and modular learning differ in performance, and how trajectory-generation strategies affect imitation-learning policies.Source: §V · p. 5
• The authors ask how transformer and RNN state encoders affect performance.Source: §V · p. 5
• The evaluation also asks how robust the baselines are to noise simulating real-world conditions.Source: §V · p. 5

A. Benchmarking learning methods

• DAgRL, which combines DAgger with frontier-exploration trajectories and sparse-reward RL fine-tuning, outperforms end-to-end baselines without a navigation module for bee-lining.The authors attribute its performance to learned exploration and RL fine-tuning that improves frontier selection.Source: A. Benchmarking learning methods · p. 5
• DAgRL+OD outperforms VLFM on all three evaluation splits, by 1.9–6.6% on SR and 0.3–4.1% on SPL.The table reports performance across HM3D-OVON’s three evaluation splits.Source: Table III · p. 6, A. Benchmarking learning methods · p. 6
• VLFM beats DAgRL on VAL SEEN SYNONYMS and VAL UNSEEN by 3.0–16.9% on SR and 2.9–11.7% on SPL, but trails it on VAL SEEN.The VAL SEEN differences are 6.1% on SR and 2.6% on SPL in DAgRL’s favor.Source: A. Benchmarking learning methods · p. 6

B. Comparison of trajectories used for imitation learning

• The frontier-exploration advantage contradicts SPOC, whose exploration trajectories use a different room-by-room strategy.The authors suggest that strategy may consume more steps and generalize less well than frontier exploration, which focuses on spatial layout.Source: B. Comparison of trajectories used for… · p. 6
• Frontier exploration improves SR by 4.6–6.0% and SPL by 1.0–2.2% over shortest paths in three Table IV rows.For BCRL, both trajectory types remain below about 20% SR because poor pretrained BC policies led to suboptimal sparse RL fine-tuning.Source: Table IV · p. 6, B. Comparison of trajectories used for… · p. 6

C. Transformer vs. RNN

• Transformer policies improve SR by up to 6.3% and SPL by up to 4.5% over RNN-based counterparts in three comparisons.The behavioral-cloning baselines are exceptions and show no significant improvement.Source: C. Transformer vs. RNN · p. 6
• The reported transformer advantage is not significant for behavioral-cloning baselines.The authors suggest that BC’s static teacher trajectories provide less diverse training data than DAgger and RL, which allow exploration of new trajectories as the policy evolves.Source: C. Transformer vs. RNN · p. 6
• The authors suggest limited BC training-data diversity may explain why transformers did not significantly outperform RNNs in those baselines.They note that transformer performance scales more favorably with training-data volume.Source: C. Transformer vs. RNN · p. 6

D. Robustness to noise

• DAgRL+OD drops by up to 0.3% in SR and 3.0% in SPL, while VLFM drops by up to 6.1% in SR and 5.4% in SPL.The authors attribute VLFM’s sensitivity to inaccurate top-down maps and DAgRL+OD’s robustness to its reliance on sequences of RGB observations rather than precise localization.Source: Table VI · p. 7, D. Robustness to noise · p. 7
• Test-time noise is added to policies that were not trained with these noise types.The evaluation uses Gaussian robot position and rotation actuation noise, plus uniformly sampled odometry noise of ±7 mm and ±2°.Source: D. Robustness to noise · p. 7
• For DAgRL+OD, noise causes a larger SPL drop than SR drop because corrective actions lengthen paths.The authors link this to deviations from the desired path, while attributing its limited SR drop to not relying on precise localization.Source: D. Robustness to noise · p. 7

E. Failure analysis

• Timeouts rise from 19.5% to 37.3% across VAL SEEN to VAL UNSEEN, primarily as failures to notice the goal increase from 2.6% to 16.8%.Failures from inefficient exploration remain between 16.8% and 20.5%.Source: E. Failure analysis · p. 7
• Moving away from the goal and stopping elsewhere rises from 2.2% to 17.8% as goal categories become less similar to training categories.The authors interpret the trends as difficulty generalizing and calling STOP when encountering semantically different goals.Source: E. Failure analysis · p. 7
• Wrong-object stops and failures to see the goal each account for 16.8%–20.5% of failures across all three splits.The paper identifies reducing false positives and improving exploration within the time limit as future priorities.Source: E. Failure analysis · pp. 7–8

• HM3D-OVON provides 379 goal categories and over 15k annotated household-object instances across 181 photo-realistic virtual scans.Source: §VI · p. 8
• Unlike prior datasets with predefined test-time object categories, HM3D-OVON supports an open set of goals specified through free-form language.Source: §VI · p. 8
• Experiments show that HM3D-OVON can train an open-vocabulary ObjectNav agent with higher performance and better robustness to localization and actuation noise than the state of the art.Source: §VI · p. 8