PaperlayerView exact revision on arXiv

Source-linked AI summary

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, Vanja Videnovic, Feras Dayoub, Anton van den Hengel

arXiv:2602.02220v2cs.CVcs.RO

TL;DR

LGN benchmarks often lack reliable instance descriptions and broad coverage of language-specified goals across semantic levels. The paper introduces HieraNav and LangMap for human-verified hierarchical evaluation, plus PlaNaVid as an RGB-only baseline; PlaNaVid achieves top-tier success rates, while small targets remain challenging.

  • Problem

    Existing LGN benchmarks focus largely on category-level goals or use VLM-generated instance descriptions with semantic errors and ambiguities, limiting reliable evaluation.

  • Method

    The paper introduces HieraNav, LangMap’s human-verified contrastive annotations across four goal levels, and PlaNaVid’s memory-based RGB-only planning baseline.

  • Results

    PlaNaVid achieves top-tier success rates without depth, 3D scene representations, or object masks; its multi-goal success rate is 42.6%.

  • Takeaways & Limitations

    LangMap supports systematic open-vocabulary evaluation across hierarchical goals, while memory and richer context benefit navigation performance.

  • Takeaways & Limitations

    All methods degrade on small targets with mean IoU below 3.3%, making low-visibility localization a continuing challenge.

Abstract

from arXiv · show

Language-conditioned goal navigation (LGN) requires agents to locate user-specified targets without step-by-step guidance. However, existing benchmarks largely focus on category-level goals or rely on instance descriptions generated by vision-language models (VLMs), which often contain ambiguities and semantic errors, limiting systematic and reliable evaluation. We introduce HieraNav, an open-vocabulary LGN task with goals specified at four hierarchical semantic levels: scene, room, region, and instance. To this end, we present Language as a Map (LangMap), to our knowledge the first real-world 3D indoor navigation benchmark with human-verified semantic annotations to support tasks across all four goal levels. LangMap provides region labels and discriminative region and instance descriptions covering 414 object categories, produced through a rigorous contrastive annotation protocol comparing same-scene regions and instances, and contains over 18K tasks. Each target is paired with concise and detailed descriptions, enabling evaluation across instruction styles. Quantitative and qualitative analyses validate our annotation quality; notably, our instance descriptions outperform GOAT-Bench annotations by 23 percentage points in text-to-view matching. We further introduce PlaNaVid, a strong RGB-only baseline that combines Bounded Diverse Memory (BDM) with high-level planning to prime a reactive policy for multi-goal navigation. PlaNaVid achieves top-tier success rates without depth, 3D scene representations, or object masks. Further analysis shows that memory and richer context boost performance, while long-tailed categories, small objects, distant targets, and multi-goal completion remain open challenges. The benchmark is available at https://bo-miao.github.io/LangMap

1 Introduction

Existing LGN benchmarks underrepresent context-dependent, multi-level goals and often use unreliable VLM-generated instance descriptions. HieraNav, LangMap, and PlaNaVid address these gaps through hierarchical evaluation, human-verified annotations, and RGB-only navigation.

  • Motivation: Category-level navigation misses context-dependent goals requiring semantic and spatial disambiguation.Prior benchmarks largely target any instance of an object category, which is insufficient for instructions such as finding a phone on a particular bed.
  • Motivation: 39.8% of inspected GOAT-Bench instance descriptions contain semantic errors or ambiguities.The reported errors comprise 9.9% semantic errors and 29.9% ambiguities in the inspected scenes.
  • Contributions: HieraNav unifies open-vocabulary goals across scene, room, region, and instance levels.The task supports natural-language specifications that require interpretation, spatiotemporal reasoning, and navigation to the target.
  • Contributions: LangMap provides human-verified contrastive descriptions for 414 object categories and over 18K tasks.Each target has concise and detailed descriptions, while 349 filtered categories are used to construct the navigation tasks.
  • Contributions: PlaNaVid combines Bounded Diverse Memory with high-level planning for RGB-only multi-goal navigation without depth, 3D representations, or object masks.The baseline primes a reactive policy by selecting an initial waypoint and heading for each goal.

2 Related Work

Prior LGN work covers categories, rooms, images, or VLM-generated instances, but leaves multi-level language goals and reliable instance annotations insufficiently addressed. HieraNav targets both gaps with human-verified, multi-granularity evaluation.

  • Existing formulations: Goal navigation benchmarks specify targets through points, object categories, reference images, or generated instance descriptions.These formulations support different degrees of semantic specification but do not establish a unified multi-level language setting.
  • Open gaps: Mixed-level multi-goal episodes spanning scene, room, region, and instance goals remain largely overlooked.Strong RGB-only multi-goal baselines without depth, explicit 3D representations, and object detection are also underexplored.
  • Benchmark scope: Prior LGN benchmarks commonly use small closed-set category vocabularies, while synthetic environments introduce realism and sim-to-real concerns.Early benchmarks cover 6–21 common categories; synthetic alternatives expand scene diversity but may lack real-world realism.
  • Open gaps: VLM-generated instance descriptions can be ambiguous or inaccurate because they miss distinctive cues and have limited 3D spatial reasoning.This creates a benchmark gap involving both annotation reliability and coverage across semantic levels.

3 Task and Benchmark

HieraNav evaluates language-conditioned navigation across four semantic goal levels in real-world HM3D environments, with both single-goal tasks and mixed-level multi-goal episodes. LangMap supplies broad human-verified annotations through contrastive region and instance labeling.

  • 3.1 HieraNav: HieraNav evaluates scene-, room-, region-, and instance-level goals in unseen environments using single-goal tasks and multi-goal episodes.Multi-goal episodes require reaching a sequence of instructed goals in order, while single-goal tasks contain one language-specified goal.
  • 3.1 HieraNav: The agent receives RGB, relative odometry, optional depth, and discrete movement or stopping actions under a 1 m success threshold.The action space includes 0.25 m forward moves, 30° turns, and STOP, with a 500-step limit.
  • 3.2 LangMap: LangMap covers all 36 HM3D-Sem validation scenes with human-verified annotations and tasks across all four semantic levels.The benchmark is designed for broader coverage, greater task diversity, higher annotation quality, and larger scale.
  • 3.2 LangMap: LangMap contains 414 object categories, with 349 used for navigation tasks, and provides human-verified discriminative descriptions.It includes more small-object categories and retains small but visible targets for realistic evaluation.
  • 3.2 LangMap: LangMap provides mixed-level tasks whose shortest geodesic paths are mostly 5–15 m across semantic levels.This distribution is intended to avoid path-length bias while supporting rigorous language-driven embodied-navigation evaluation.
  • 3.3 Contrastive Annotation: Contrastive annotation compares same-category regions and instances within each scene to produce concise and detailed descriptions that distinguish targets.Region descriptions add visual attributes, objects, and spatial context; instance descriptions use intrinsic attributes and discriminative region context.

4 Method

PlaNaVid separates memory-guided high-level planning from reactive RGB navigation for multi-goal tasks. Its bounded memory preserves temporally distributed and semantically diverse context for waypoint selection within limited planner context windows.

  • PlaNaVid framework: PlaNaVid uses memory-guided planning followed by reactive policy navigation, with online memory updates supporting subsequent goals.The planner selects a starting waypoint and heading from retrieved context, while the reactive policy maps RGB observations to actions.
  • Bounded Diverse Memory: Bounded Diverse Memory stores at most Nmax RGB snapshots with near-uniform temporal coverage and semantic diversity.Its components are global-uniform update and semantic-diverse retrieval.
  • Global-Uniform Update: When memory exceeds Nmax, gap-aligned pruning removes the entry with the minimum local temporal gap to reduce temporal redundancy.This maintains near-uniform coverage under a fixed capacity.
  • Global-Uniform Update: 92.9% memory compression is achieved with Nmax = 50 for a typical T ≈700-step episode, yielding ϵmax ≈7 and average error ≈3.5.These bounds correspond to near-uniform spacing ∆≈14 and indicate small temporal coverage error under the fixed budget.
  • Semantic-Diverse Retrieval: Semantic-diverse retrieval anchors the latest state and greedily selects K−1 additional states that maximize global semantic diversity.In this work, K=10; selected states are reached by replaying compressed trajectories without querying the simulator pathfinder.

5 Experiments

Experiments evaluate sequential reliability alongside per-goal navigation metrics and compare recent LGN methods, PlaNaVid components, instruction styles, category frequency, path length, and target visibility. PlaNaVid achieves the strongest reported success rates, while long-tail categories, distant targets, small objects, and complete multi-goal sequences remain challenging.

  • 5.1 Metrics: SeqSR@k measures the fraction of episodes whose first k tasks are completed successfully in order, complementing SR and SPL for multi-goal reliability.Each episode contains G=5 tasks, with k ≤ G.
  • 5.3 Main Results: 42.6% multi-goal SR and 31.4% single-goal SR are PlaNaVid’s highest reported success rates using only RGB.It exceeds Uni-NaVid and MTU3D in multi-goal SR by 8.5 and 1.2 points, respectively.
  • 5.4 Ablation Study: 14.3% SeqSR@2 accompanies PlaNaVid’s best ablated configuration, combining global-uniform updates with semantic-diverse retrieval.The combination reduces memory from 656 to 50 frames, a 13× reduction.
  • 5.4 Ablation Study: Detailed descriptions improve region- and instance-level navigation, whereas concise descriptions can be harder to ground in cluttered scenes.The richer cues matter most where fine-grained target disambiguation is required.
  • 5.4 Ablation Study: Performance consistently drops on long-tail categories and decreases as target distance increases.Head categories cover 77% of tasks, while long-tail categories cover 23%.
  • 5.4 Ablation Study: All methods degrade on small targets with mean IoU < 3.3%, highlighting low-visibility localization as a persistent challenge.This analysis identifies target visibility as a separate limitation from navigation distance and category frequency.

6 Conclusion

HieraNav and LangMap establish a multi-level, open-vocabulary benchmark for language-driven indoor navigation, while PlaNaVid provides a strong RGB-only baseline. Evaluations highlight both the value of memory and richer context and persistent challenges for future work.

  • HieraNav spans scene, room, region, and instance goal levels for open-vocabulary navigation.
  • LangMap combines human-verified annotations with navigation tasks across all four goal levels.
  • PlaNaVid uses Bounded Diverse Memory to prime reactive navigation without depth, 3D representations, or object masks.
  • Systematic evaluations identify long-tailed categories, small objects, distant targets, and multi-goal completion as key challenges.

A Dataset Documentation and Access

LangMap stores structured annotations and task metadata in organized JSON files. This design supports multiple goal-level episode types and equivalent instructions with varied wording.

  • Each annotation file contains goals, region annotations, and episode groups for object, room, region, instance, and sequence levels.
  • The goals field records categories, identifiers, 3D positions, success viewpoints, region associations, and concise and detailed instance descriptions.
  • Structured task metadata lets users generate differently worded instructions while preserving the same target set.
  • Default instructions vary by semantic level, from category-only goals to room-, region-, and instance-specific descriptions.

B.1 Representative Object-View and Region-View Capture

The benchmark constructs representative object and region views before annotation and task generation. Contrastive descriptions and filtering procedures are used to improve target distinctiveness and data quality.

  • Object instances are sampled from viewpoints every 10° between 0.5 m and 3 m, selecting the view with highest visible coverage.
  • Region views use a pseudo-center based on the midpoint of the bounding box enclosing objects in the region.
  • Annotators compare same-scene candidates to write concise and detailed descriptions containing distinctive visual and spatial cues.
  • Categories with abstract, unstable, or low-quality navigation targets, along with low-quality object views, are excluded from relevant tasks.

B.4 Annotation Protocol and Quality Control

LangMap’s annotation protocol uses trained annotators, contrastive interfaces, and cross-checking to produce descriptions that distinguish targets from same-scene alternatives. Concise and detailed forms provide different levels of visual and spatial context.

  • Trained annotators write natural, visually grounded descriptions intended to uniquely identify regions and object instances within each scene.
  • Interfaces display targets alongside same-category distractors, including region panoramas, contained object views, and instance views.
  • Annotators compare similar instances within scenes and may use region descriptions or 3D scene files to resolve difficult cases.
  • Concise descriptions usually fit within seven words, while detailed descriptions add attributes, nearby objects, and relative positions.
  • Reviewers cross-check descriptions for correctness, visual grounding, and discriminability using an interactive annotation interface.

C Qualitative Comparison of Annotations in GOAT-Bench and LangMap

Figure 9’s interactive viewer supports side-by-side inspection of GOAT-Bench and LangMap instance annotations across scenes, objects, same-category crops, and region panoramas. Figure 10 further compares description quality by highlighting semantic errors and ambiguity across regions.

  • The viewer lets users select scenes and object instances for direct annotation comparison.It displays the target object, same-category object crops, and corresponding region panoramas side by side.
  • Figure 10 highlights semantic errors and descriptions that ambiguously match multiple objects.

D Additional Details of Bounded Diverse Memory

Bounded Diverse Memory maintains and retrieves a compact, semantically diverse set of RGB states for high-level waypoint and heading selection. Additional examples illustrate how LangMap descriptions distinguish same-scene instances more precisely than GOAT-Bench descriptions.

  • Bounded Diverse Memory: Bounded Diverse Memory updates a bounded RGB-state set by pruning temporally redundant entries and retrieves memories maximizing global semantic diversity.Retrieved memories are passed to the high-level planner, while waypoint-reaching replays the corresponding compressed trajectory without simulator path queries.
  • Description Examples: GOAT-Bench descriptions can be ambiguous across same-scene targets, with several kitchen-cabinet descriptions explicitly matching multiple instances.Examples report matches between descriptions ① and ④–⑤, ② and ④–⑤, and ③ and ⑤.
  • Description Examples: Concise LangMap descriptions identify targets with specific spatial, visual, and contextual attributes, such as cabinets relative to appliances, mirrors, fixtures, and furniture.The examples include multiple bathroom cabinets distinguished by color, sink configuration, nearby objects, and room context.
  • Description Examples: LangMap descriptions provide finer distinctions among similar objects, including cabinets above different landmarks, beside particular appliances, and with distinct sink or countertop properties.Bathroom examples distinguish black and light-colored single-sink cabinets, a white double-sink cabinet, and cabinets associated with different prints, mirrors, showers, toilets, or towels.
  • Description Quality: Figure 10 marks semantic errors in blue and ambiguous descriptions matching multiple objects in green, contrasting GOAT-Bench with LangMap’s human-verified descriptions.The figure characterizes LangMap descriptions as correct, fine-grained, and uniquely referring to the target instance.
  • Bounded Diverse Memory: The Bounded Diverse Memory pseudocode adds states until the storage budget is exceeded, then merges a selected adjacent trajectory pair before removing one memory entry.Retrieval anchors the latest state, iteratively adds selected memories until the retrieval budget is met, and returns them sorted by state index.
Loading 2602.02220v2…