PaperlayerView exact revision on arXiv

Source-linked AI summary

StopWatch: Goal Grounding for Object-goal Navigation

Xinhao Liu, Yunsheng Tian, Yuheng Qiu, Pieter Abbeel, Xi Chen, Guanya Shi, Chen Feng

arXiv:2610.23841v1cs.CVcs.RO

TL;DR

Fine-tuned VLM navigation policies still generalize poorly to unseen goals, and the capability they lack is unclear. StopWatch makes goal grounding explicit through a two-model pipeline and corrects benchmark annotations. It reports state-of-the-art results across benchmarked tasks and finds that, with ground-truth grounding, its policy nearly matches its demonstration expert on clean benchmarks.

  • Problem

    Fine-tuned VLM navigation policies generalize poorly to unseen goal categories, and which capability they lack remains unclear.

    Source: §1 · p. 2

  • Method

    StopWatch supervises explicit grounding from scene annotations and uses a perception model to predict grounding text for a separately trained policy.

    Source: §1 · p. 2

  • Results

    The system reaches state-of-the-art results on all benchmarked object-goal navigation tasks, and the policy nearly matches its demonstration expert with ground-truth grounding on clean benchmarks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors identify goal grounding as a missing piece and release corrected evaluations and two open-vocabulary benchmarks to support more faithful comparisons.

    Source: §5 · p. 9

    Grounding supervision relies on semantic and geometric annotations from 3D scene collections, and evaluation is limited to simulation in static indoor object-goal navigation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Pretrained VLMs bring rich world knowledge and semantic priors to object-goal navigation, and fine-tuning a VLM into a navigation policy has become a common recipe. Despite this progress, success rates remain low, and it is unclear which capability these fine-tuned policies lack. Our diagnosis identifies goal grounding as the dominant failure. These policies explore well, yet often fail to notice that the goal is already in view or to estimate how far away it is. To address this failure, we ground the goal explicitly. Explicit grounding substantially improves success and reaches state-of-the-art results on six benchmarks across three scene collections. We also correct annotation defects in these benchmarks, so that future methods are evaluated more faithfully. Our analysis on the clean benchmarks shows that, with ground-truth grounding, our policy nearly matches the expert that generated its demonstrations, leaving the headroom in perception rather than policy learning. Our work points to goal grounding as a missing piece of object-goal navigation and offers a simple and effective solution to the task.

1 INTRODUCTION

The paper diagnoses goal grounding—knowing whether the goal is in view and how far away—as a key failure in fine-tuned VLM navigation policies. It makes grounding explicit and reports stronger benchmark performance alongside corrected evaluations.

1 INTRODUCTION

  • Policies often reach the goal but fail to stop because they cannot tell whether it is in view or how far away it is.With ground-truth semantics and distance, the same policy reaches 89% success, versus 52% end to end.Source: §1 · p. 2
  • The system makes grounding explicit before each action, using scene annotations to supervise grounding text for every demonstration frame at no extra labeling cost.Unlike free-form reasoning traces, its grounding is derived from semantic and geometric scene labels.Source: §1 · p. 2
  • 8.8 points above the strongest baseline on HM3D-OVON val unseen with corrected annotations; the system reaches state-of-the-art results on six benchmarks.The authors attribute the remaining headroom on clean benchmarks primarily to perception, not policy learning.Source: §1 · p. 2
  • The authors also correct benchmark annotation defects and show that, on clean benchmarks, the policy is already strong while perception accounts for most remaining headroom.Their contributions include more faithful evaluation and a policy that nearly matches its demonstration expert when given ground-truth grounding.Source: §1 · p. 2

2 RELATED WORK

The paper situates its approach among object-goal navigation, modular systems, fine-tuned VLM policies, and intermediate reasoning methods. Its distinction is to supervise explicit grounding cues identified by a diagnosis of policy failures.

Embodied navigation tasks

  • Object-goal navigation asks an agent to find an instance of a named category, while open-vocabulary variants include test categories unseen during training.The task also requires efficient search, long-horizon memory, and recognizing the goal when it comes into view.Source: §2 · p. 2

Modular navigation systems

  • Modular navigation makes perception and planning explicit, whereas this approach exposes only the two cues diagnosed as missing and retains a learned policy for search and stopping.Unlike planners acting by fixed rules on an interface, its policy sees every frame and learns from demonstrations.Source: §2 · p. 3

VLMs for navigation

  • Recent work commonly fine-tunes a single pretrained VLM end to end on expert demonstrations; this paper instead supervises perception explicitly with structured text.The cited work notes that end-to-end training can harm pretrained semantic priors and language understanding.Source: §2 · p. 3

Intermediate reasoning

  • Prior navigation reasoning text is typically designed as plans, subtasks, or instruction progress, while this paper uses scene annotations to provide the diagnosed missing cues.Those cues are object semantics in view and distance to the goal, available from every training frame.Source: §2 · p. 3

3 STOPWATCH

StopWatch diagnoses failures to stop at visible goals as a grounding problem, then builds a pipeline that predicts explicit grounding for a learned navigation policy. It also repairs evaluation defects that can reject valid stops or score incorrect targets.

3.1 PROBLEM SETUP

  • The task requires finding an instance of a goal category and stopping near it within a step budget; test goals may be unseen during training.Episodes trace an end-to-end RGB policy and the same policy with ground-truth grounding.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 FAILURE ANALYSIS OF EXISTING METHODS

  • End-to-end policies come within 0.25 m of a goal in 76% of novel-goal episodes but succeed in only 52%, often walking past without stopping.Stopping is harder to learn because demonstrations provide only one STOP action per episode and only for training goal categories.Source: §3.2 · p. 4
  • With ground-truth grounding, success rises from 52% to 89%, and failures after reaching the goal fall from 24% to 5%.Wrong stops disappear entirely, leaving timeouts as the remaining failures.Source: §3.2 · p. 4

3.3 DESIGN CHOICES

  • The grounding text names object categories in view and distance to the goal, compiled from scene labels for every demonstration frame at no extra labeling cost.The category list also supervises objects that are never training goals.Source: §3.3 · pp. 4–5
  • The pipeline predicts grounding text from recent frames and the goal word, then gives it alongside observations to a separately trained policy.The policy is trained on annotated groundings, while the perception model learns grounding prediction; more accurate grounding can improve navigation without retraining the policy.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4 UNRELIABLE EVALUATION

  • Evaluation defects include inconsistent names for the same object type and false-positive or false-negative annotations; corrected evaluation raises scores for all three reported systems.The correction merges 2,408 raw labels into 540 classes and audits affected episodes; on HM3D-OVON val unseen it raises LightNav-0 by 13 points, the system by 16, and ground-truth-grounded policy by 19.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Across six benchmarks, the system leads under every reported protocol, with its advantage lying mainly in stopping at the goal. Analysis finds perception errors account for much of the remaining gap, while ground-truth grounding lets the policy nearly match its expert.

4.1 SETUP

  • Evaluation covers open- and closed-vocabulary tasks in HM3D, MP3D, and HSSD, with official and audited clean protocols; models train on demonstrations from all three collections.The baselines include end-to-end VLM policies, a benchmark baseline, and a zero-shot modular method.Source: §4.1 · p. 6

4.2 COMPARISON WITH BASELINES

  • The system leads on all six benchmarks under every reported protocol, with its HM3D-OVON clean lead over LightNav-0 widening from 6.4 to 8.8 points.On clean HM3D-OVON val unseen, it succeeds after reaching within 0.25 m in 90% of cases, versus 80% for LightNav-0.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 ANALYSIS

  • Disjoint grounding matters most: separate perception and policy models achieve 69.6% success on val unseen, versus 50.7% when one model writes grounding and acts.The semantic field alone reaches 68.7%, while adding distance improves results further; both fields together perform best.Source: Table 2 · p. 8, §4.3 · p. 8
  • With ground-truth grounding, the policy nearly matches its expert in reaching the goal across all three benchmarks, and the remaining gap mostly follows arrival.On clean HM3D-OVON val unseen, correcting distance adds 9.0 points to 78.6%; also correcting categories adds 13.0 points to 91.6%, leaving a 4.5-point expert gap.Source: §4.3 · p. 8
  • On HM3D-OVON val unseen, the expert succeeds in 96% of episodes under clean scoring but 72% under the official protocol, with the difference due to goal-acceptance criteria.Under the clean protocol, nearly all expert failures result from running out of frontiers or steps before finding the goal.Source: §4.3 · p. 8

4.4 PERCEPTION ERRORS

  • Wrong visibility and distance account for 69% of HM3D-OVON val unseen failures: visibility errors cause 43%, while distance errors cause 26%.Across all three benchmarks, category success rises with how often perception reports the goal in view; small or thin objects are reported least often.Source: §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

The discussion identifies goal grounding as a key limit on VLM navigation and reports that corrected evaluations and new benchmarks broaden and improve assessment. On clean benchmarks, the policy nearly matches its demonstration-generating expert with ground-truth grounding, while perception remains the main headroom.

5 DISCUSSION AND CONCLUSION

  • Official evaluations reject valid stops, increasingly so for stronger agents, motivating corrected evaluations for fairer comparisons.The work also releases MP3D-OV and HSSD-OV to extend open-vocabulary evaluation beyond HM3D.Source: §5 · p. 9
  • Grounding supervision relies on semantic and geometric annotations, limiting training data to annotated scene collections.The evaluation is limited to simulated object-goal navigation in static indoor scenes; learning from real videos is proposed as a route to larger data sources.Source: §5 · p. 9
  • The paper traces the gap in generalization to unseen goals to goal grounding, which it identifies as a capability limiting VLM navigation policies.The authors expect object-goal navigation to advance with goal grounding.Source: §5 · pp. 9–10
  • With ground-truth grounding, the policy nearly matches its demonstration-generating expert, leaving most remaining headroom in perception.The paper presents a two-model pipeline as one way to supply grounding; single-model and agentic alternatives could also do so.Source: §5 · p. 10

A COMPARISON ON PERCEPTION MODELS

The perception comparison finds that better grounding can raise success with the same policy, although false positives can offset higher goal recall and small goals remain difficult to perceive. The system also leads baselines in path-efficiency metrics across the clean open-vocabulary benchmarks.

A COMPARISON ON PERCEPTION MODELS

  • GPT-6 Astra outperforms the task-trained perception model, raising success with the same policy, but runs about nine times slower even on Ultrafast.The task-trained model also far outperforms its untrained Qwen3-VL-4B starting model, while all tested perception models remain below ground-truth grounding.Source: A COMPARISON ON PERCEPTION MODELS · p. 14, Table 3 · p. 14
  • Higher goal recall helps only when it does not bring more false positives: Luna and Qwen3-VL-4B report more goals but have lower success.Their lower object F1 indicates they also report objects absent from the annotations; Astra almost never misses a visible goal after the agent reaches it.Source: A COMPARISON ON PERCEPTION MODELS · p. 14, Figure 9 · p. 14
  • Small goals reduce success most with the task-trained perception model, while Astra succeeds almost equally across goal sizes.With ground-truth grounding, success depends far less on size, indicating small goals are harder to perceive than to reach.Source: A COMPARISON ON PERCEPTION MODELS · p. 14, Figure 9 · p. 14
  • The evaluation sets use held-out scenes; clean subsets exclude episodes with empty destination sets or no finite start-to-goal path on the protocol navmesh.Sixteen baseline successes fall on excluded episodes: six for LightNav-0 on HM3D ObjectNav and ten for Nav-R2 on HSSD ObjectNav.Source: Table 5 · p. 15

B PATH EFFICIENCY

  • The system leads every baseline on all three path-efficiency metrics across all three clean open-vocabulary benchmarks.Table 4 presents the path-efficiency comparison.Source: B PATH EFFICIENCY · p. 15, Table 4 · p. 15
  • SPL uses shortest-path distance as its reference, although that assumes the agent already knows the goal location during a search task.SPLFE instead uses the path length of the frontier-exploration expert on the same episode as its reference.Source: B PATH EFFICIENCY · p. 15

C BENCHMARKS AND SIMULATION

The benchmark and simulation setup uses held-out validation scenes and clean evaluation subsets, with additional open-vocabulary goals generated under reachability constraints. The system navigates with a Stretch embodiment and a four-action set.

C BENCHMARKS AND SIMULATION

  • Evaluation uses held-out scenes, with clean subsets excluding episodes whose destination set is empty or lacks a finite path from the start on the protocol navmesh.The raw evaluation sets include official validation episodes for HM3D-OVON and closed-vocabulary tasks, and publisher validation scenes for MP3D-OV and HSSD-OV.Source: C BENCHMARKS AND SIMULATION · p. 15
  • Generated goals require annotated instances and valid starts in multiple validation scenes, with reachable targets 1–30 m away geodesically.Starts are sampled from navigable locations with uniform yaw angles.Source: C BENCHMARKS AND SIMULATION · p. 15
  • The system uses the Stretch embodiment with Forward, Left, Right, and Stop actions; exhausting the action budget without stopping counts as failure.Table 6 summarizes simulation settings, and Table 7 lists the grounding text read by the policy before its action prompt.Source: C BENCHMARKS AND SIMULATION · p. 15, Table 6 · p. 16, Table 7 · p. 16

D QUALITATIVE COMPARISONS

Qualitative episodes show both systems sometimes passing near goals, but differ in when they stop; an erroneous book detection also leads the proposed system far from the goal.

D QUALITATIVE COMPARISONS

  • On the HM3D-OVON rug episode, LightNav-0 passes within 0.01 m but stops 1.3 m away at step 136; the proposed policy stops 0.04 m away at step 77.Source: D QUALITATIVE COMPARISONS · p. 16
  • On MP3D-OV carpet, both systems pass within 0.25 m early; the proposed system returns and stops at step 149, while LightNav-0 stops 3.9 m away at step 328.Source: D QUALITATIVE COMPARISONS · p. 16
  • On the HSSD-OV cabinet episode, LightNav-0 passes within 0.02 m but stops 9.2 m away at step 275; the proposed policy stops 0.11 m away at step 172.Source: D QUALITATIVE COMPARISONS · p. 16
  • In the HM3D-OVON book episode, a false in-view report at step 450 leads the proposed policy to stop 13.2 m away; LightNav-0 stops 0.06 m away at step 36.Source: D QUALITATIVE COMPARISONS · p. 16

E CORRECTED EVALUATION

The clean evaluation protocol combines annotation-based episode exclusions with the paper’s success criterion. Audited scene/category pairs with repeated annotation errors are excluded, as are broad goal names that do not specify an object type on MP3D-OV and HSSD-OV.

E CORRECTED EVALUATION

  • Success requires stopping within 0.25 m geodesic distance of an accepted instance viewpoint, measured on the Stretch navigation mesh.An instance viewpoint is one where the goal fills more than 5% of the image, summed over three camera pitches; official success uses benchmark goal viewpoints.Source: E CORRECTED EVALUATION · p. 16
  • The clean protocol excludes episodes with goals in flagged scene/category pairs and broad goal names that do not specify an object type on MP3D-OV and HSSD-OV.Pairs are flagged after repeated evidence of a visible goal missing from annotations or an annotated object that is not a goal.Source: E CORRECTED EVALUATION · p. 16

F GROUNDING TEXT AND EXPERT DEMONSTRATIONS

Grounding text provides the policy with a bounded inventory of visible object categories and a distance bin when the goal is visible. The policy’s frontier-exploration expert seeks viewpoints adjacent to accepted goal instances and stops near its destination after facing the object.

F GROUNDING TEXT AND EXPERT DEMONSTRATIONS

  • The grounding inventory contains at most 16 categories ordered by image area, with goal visibility defined by inventory membership and distance supplied only when the goal is visible.Category thresholds come from training-instance geometry, and the goal is retained whenever it passes its threshold.Source: F GROUNDING TEXT AND EXPERT… · p. 16
  • The frontier-exploration expert targets viewpoints adjacent to accepted goal instances and stops within 0.15 m of its destination after facing the object.Its distance label is the geodesic distance to the nearest destination.Source: F GROUNDING TEXT AND EXPERT… · pp. 16–17
  • Table 8 reports training hyperparameters and defines frame budgets as global caps per update; its training-time totals include all job attempts, setup, and checkpointing.The policy run resumed twice after preemption.Source: Table 8 · p. 17
  • The policy corpus contains 3,138,329 demonstrations and 252,526,115 frames, while the perception corpus contains 3,140,902 demonstrations and 250,013,894 frames from the same 145 HM3D, 61 MP3D, and 123 HSSD training scenes.These totals precede training-index filtering.Source: F GROUNDING TEXT AND EXPERT… · p. 17

G TRAINING DETAILS

The perception and policy models are trained separately from Qwen3-VL initializations, with grounding prediction and expert-action learning using different inputs.

G TRAINING DETAILS

  • The models are trained separately: perception predicts grounding text from RGB and expert-action history, while the policy learns expert actions from RGB, ground-truth grounding text, and prior actions.Table 8 summarizes training from Qwen3-VL initializations (Bai et al., 2025).Source: G TRAINING DETAILS · p. 17

H FAILURE ANALYSIS PROTOCOL

The failure analysis compares reaching and stopping outcomes across LightNav-0, the proposed system, and a policy given ground-truth grounding, then assigns causes to failures on clean HM3D-OVON episodes. Its protocol also specifies the reach criterion and the conditions used to evaluate grounding.

H FAILURE ANALYSIS PROTOCOL

  • The Figure 2 comparison uses the same Qwen3-VL-2B policy trained for 20,000 updates, with one run receiving ground-truth grounding text and both evaluated on 3,000 HM3D-OVON val unseen episodes.Inputs are 320 × 160, and history spans up to 500 frames; the grounding distance-bin edges are 0.15, 0.25, 0.5, 1, 2, and 4 m.Source: H FAILURE ANALYSIS PROTOCOL · p. 17, Table 9 · p. 19
  • A reached episode has a trajectory position within 0.25 m geodesic distance of an accepted instance viewpoint; outcomes distinguish wrong stops from timeouts and partition failures by reach.A timeout is 500 actions without stopping, and failure after reaching includes both wrong stops and timeouts in the reached branch.Source: H FAILURE ANALYSIS PROTOCOL · p. 17
  • With similar reach-vicinity rates, LightNav-0 stops in the wrong place after reaching nearly three times as often as the proposed system: 10.3% versus 3.6%.LightNav-0 and the proposed system reach the vicinity in 75.7% and 77.4% of episodes, respectively; with ground-truth grounding, the policy reaches in 93.9% and fails after reaching in 2.3%.Source: H FAILURE ANALYSIS PROTOCOL · pp. 17, 19, Figure 11 · p. 19
  • The grounding ladder compares predicted grounding, a condition substituting ground-truth distance when both inventories contain the goal, and full ground-truth inventory and distance against the demonstration-generating expert.Clean-subset counts exclude episodes without usable goal viewpoints.Source: H FAILURE ANALYSIS PROTOCOL · p. 19, Table 10 · p. 19
  • Figure 8 assigns each of 703 failures on 2,309 clean episodes to one cause, using the final pre-action frame at a stop or the interval from first reach through a reached timeout.The six category counts, in order, are 241, 183, 62, 90, 94, and 33; the category-level recall analysis retains categories with at least ten clean episodes.Source: H FAILURE ANALYSIS PROTOCOL · p. 20

Related papers

Loading 2610.23841v1…