PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-LN Bench: Towards Long-horizon Goal-oriented Navigation with Active Dialogs

Wensi Huang, Shaohao Zhu, Meng Wei, Siqi Zhang, Jinming Xu, Xihui Liu, Hanqing Wang, Tai Wang, Feng Zhao, Jiangmiao Pang

arXiv:2512.22342v5cs.RO

TL;DR

Embodied navigation often assumes clear instructions, while real-world goals can be underspecified. The paper introduces IIGN and VL-LN Bench, including large-scale dialog-augmented training data and oracle-based evaluation. Experiments report that dialog improves performance, while models remain behind humans and face exploration and instance-grounding challenges.

  • Problem

    Many navigation tasks use unambiguous instructions, while underspecified goals require interaction to clarify intent and identify a specific instance.

    Source: §1 · p. 1

  • Method

    The paper introduces IIGN, where agents query an oracle during instance-goal navigation, and VL-LN Bench with automated collection of dialog-augmented trajectories and oracle-based evaluation.

    Source: §1 · pp. 1–2

  • Results

    Experiments find that active dialog improves IGN and IIGN performance, but models still lag far behind human performance and struggle with long-horizon exploration and instance grounding.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    VL-LN Bench provides a practical testbed for dialog-enabled embodied navigation, with ablations supporting the collected data and scalable oracle assistance comparable to human support.

    Source: §5 · p. 11

    IIGN assumes an oracle with human-like prior knowledge of target attributes and location, environment structure, and the agent’s current state.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In most existing embodied navigation tasks, instructions are well-defined and unambiguous, such as instruction following and object searching. Under this idealized setting, agents are required solely to produce effective navigation outputs conditioned on vision and language (VL) inputs. Real-world instructions, however, are often underspecified and require interaction to resolve ambiguity and infer user intent. To bridge this gap, we propose Interactive Instance Goal Navigation (IIGN), which extends Instance Goal Navigation (IGN) by allowing agents to freely consult an oracle in natural language while searching for a specific instance. IIGN requires agents to produce both Language and Navigation (LN) outputs, enabling interaction while moving in the environment. To support this task, we introduce VL-LN Bench, a benchmark with an automated data collection pipeline and over 41k collected long-horizon dialog-augmented trajectories for training, alongside an automatic evaluation protocol paired with a dedicated oracle for answering agent queries. Experiments reveal two core bottlenecks of IIGN: long-horizon exploration and fine-grained grounding of textual information to the correct instance among same-category distractors. Although active dialog partially alleviates these challenges, current models still lag far behind human performance. Further ablations validate the value of the data generated by our pipeline and show that the proposed oracle provides scalable assistance comparable to human support, proving VL-LN Bench as a practical testbed for dialog-enabled embodied navigation.

1 Introduction

IIGN makes instance-goal navigation interactive: agents can ask an oracle to resolve underspecified goals while navigating. VL-LN Bench supplies dialog-enabled training and evaluation, and experiments find that dialog helps but does not close the gap to human performance.

Task motivation

  • IIGN extends IGN by requiring agents to consult an oracle when a category-only instruction cannot identify the intended instance.The task combines long-horizon exploration with target disambiguation.Source: §1 · p. 1, Figure 1 · p. 2

Research gap

  • Prior interactive navigation work lacks large-scale training data for learning when to ask targeted questions and how to use dialog information during navigation.Earlier efforts either use small room-level settings, focus on instance descriptions, or collect human-human dialog without testing proactive questioning.Source: §1 · p. 1

Contributions

  • VL-LN Bench combines an automated collection pipeline, about 41K dialog-augmented trajectories, and an oracle supporting attribute, route, and disambiguation questions.Attribute and disambiguation questions help identify the target among similar candidates, while route questions support long-horizon exploration.Source: §1 · p. 2

Experiments

  • Active dialog consistently improves IGN and IIGN performance, but current models still lag far behind humans.The experiments attribute dialog gains to more efficient exploration and discriminative cues for instance disambiguation.Source: §1 · p. 3

2 Related Works

Prior work addresses object-category navigation, instance-goal navigation, and interactive navigation, but existing interactive benchmarks offer limited support for house-scale exploration and large-scale policy training. VL-LN Bench combines a dialog-augmented collection pipeline with an oracle for instance-goal navigation.

Goal-oriented navigation

  • ObjectNav accepts any instance of a category, whereas IGN targets a specific instance using a detailed description; IIGN adds proactive questions and navigation guidance.The paper motivates IIGN by the ambiguity of descriptions among visually similar instances and the need to recover from wrong turns.Source: §2 · p. 3

VL-LN Bench

  • VL-LN Bench provides large-scale training data, an open-source generation pipeline, and evaluation for long-horizon interactive instance-goal navigation.Its oracle supports attribute, route, and disambiguation queries.Source: §2 · pp. 3–4

Interactive navigation benchmarks

  • DialFRED is limited to small single-room scenes, CoIN focuses on attribute questions, and CVDN’s human-human dialogs offer insufficient scale for training active exploration and conversation.The paper positions these limits against realistic house-scale navigation among visually similar instances.Source: §2 · p. 3

3 VL-LN Benchmark

VL-LN Bench defines an instance-goal task in which agents can query an oracle while navigating, and provides a pipeline for constructing episodes and dialog-augmented training trajectories. Its dataset spans MP3D scenes, and its evaluation includes a measure of dialog utility.

3.1 Task Definition

  • IIGN asks a dialog-enabled agent to find a specific instance from an ambiguous category instruction, using movement or oracle queries under a limited interaction budget.The oracle is assumed to know target attributes and location, environment structure, and the agent’s current state.Source: §3.1 · p. 4

3.2 Interaction-Supportive Design

  • Attribute questions narrow candidates using target cues, disambiguation questions verify a grounded candidate, and route questions seek orientation-, path-, or step-level guidance.The collection focuses on these three recurring query types because they provide core information for IIGN.Source: §3.2 · p. 4

3.3 Dataset Generation

  • The dataset uses a three-stage automatic pipeline, but validation and test use only its first two stages; training additionally uses the third stage to generate trajectories from episode annotations.Instance annotations are organized in per-scene instance dictionaries, while region annotations are unified into a house-level index; a spatial-relation graph connects instance nodes within 1 m. Episodes combine an initial pose, instruction, and target-instance viewpoints; partial category-only instructions support IIGN, full descriptions support IGN, and success means stopping within 0.25 m of a viewpoint.Source: §3.3 · p. 5, §3.3.1 · p. 5, §3.3.2 · p. 6

3.3.3 Training Trajectories Collection

  • Training trajectories use frontier-based exploration that balances local exploration with geodesic distance to the target, with attribute, route, and disambiguation queries triggered during navigation.The agent uses a ground-truth object detector; the disambiguation trigger requires a matching target-category instance near the image center and within 3 m.Source: §3.3.3 · p. 6

3.3.4 Trajectories Filtering

  • Trajectory filtering removes collisions and excessive turning, then retains only trajectories whose terminal image places the target in its central region.More than 13 consecutive 30° turns are removed; the target must appear within the inner 3/4 × 3/4 area after cropping a 1/8 margin from each border.Source: §3.3.4 · p. 6

3.4 Data Statistics

  • The dataset covers 112 categories, 20,476 instances, and 90 MP3D scenes; it includes 41,891 IIGN trajectories and 500 test episodes.The reported IIGN training set contains 246,433 episodes, and the filtered IIGN trajectories number 11,661.Source: §3.4 · p. 6

3.5 Evaluation Metric

  • Mean Success Progress averages success-rate gains over the no-dialog baseline across dialog budgets, capturing dialog utility while favoring gains achieved with fewer turns.The paper sets the maximum dialog allowance to five turns.Source: §3.5 · p. 7

4 Experiments

Experiments benchmark IGN and IIGN, analyze their challenges and the effects of dialog, and test the benchmark through ablations and real-world deployment. The results highlight fine-grained instance grounding as a bottleneck, while dialog improves performance and automated data supports multiple navigation tasks.

4.1 Experimental Setup

  • Experiments compare IGN with a full target description against dialog-enabled IIGN, using FBE, VLFM, AIUTA, and VLLN baselines.Source: §4.1 · p. 7

4.2 Main Results

  • VLLN with dialog achieves the best overall performance on both tasks, improving SR by 3.2–5.8% and SPL by 1.71–3.86% over its non-dialog counterpart.Source: Table 2 · p. 7, §4.2 · p. 9
  • Ambiguous Detection is the largest error component (27.8%–34.6% on IGN/IIGN), exposing difficulty grounding fine-grained attributes to the intended same-category instance.Source: §4.2 · p. 8
  • Route guidance reduces blind exploration, while attribute and disambiguation answers provide cues to rule out candidates and steer the agent toward the target.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3 Ablation Studies

  • On ObjectNav, frontier-exploration training reaches 72.4% SR and 81.1% OS versus 60.5% SR and 69.3% OS with shortest-path data; on IGN, it gains 6.4% SR and 6.8% OS.Frontier-trained models have higher exploration error but lower wrong detection, while shortest-path training is more prone to premature stopping.Source: §4.3 · pp. 9–10, Table 3 · p. 10
  • Co-training causes only minor performance changes across ObjectNav, IGN, and IIGN; interaction learned from IIGN also raises IGN SR from 27.0% to 30.2%.The pipeline collects trajectories for all three tasks, supporting its use as a unified data engine.Source: Table 4 · p. 10, §4.3 · p. 10
  • Human navigators achieve over 90% SR, far above VLLN’s 25%–25.4%; replacing the GPT-based NPC with a human yields comparable performance.Human navigators using the GPT-based NPC also perform close to human-human evaluation.Source: §4.3 · p. 11

4.4 Real-world Experiments

  • In real-world chair searches, the robot proactively asks a disambiguation question when unsure, and human responses lead it to different target chairs.Source: Figure 5 · p. 11, §4.4 · p. 11

5 Conclusion

The paper introduces VL-LN Bench for studying IIGN, with automated data collection, 41k dialog-augmented trajectories, and oracle-based evaluation. Experiments identify challenges for IGN and IIGN, while showing that dialog improves performance and the oracle provides scalable assistance comparable to human support.

5 Conclusion

  • VL-LN Bench combines an automatic data collection pipeline, 41k long-horizon dialog-augmented trajectories, and an evaluation protocol with a reliable oracle.The benchmark is presented as a practical testbed for dialog-enabled embodied navigation.Source: §5 · p. 11
  • Experiments show that active dialog consistently improves performance on both IGN and IIGN, while the tasks remain challenging.Ablations validate the collected data and show that the oracle provides scalable assistance comparable to human support.Source: §5 · p. 11

A.1 NPC Implementation

The appendix organizes IIGN dialogs around target identification and navigation support, with type-specific oracle responses. It also describes how episodes, initial poses, target viewpoints, and full instructions are generated.

A.1 NPC Implementation

  • The dialog taxonomy distinguishes target-identification questions from navigation-support questions, with attribute, disambiguation, and route subtypes.Attribute subtypes cover intrinsic attributes and spatial context; route subtypes cover orientation, path, and step guidance.Source: §A.1 · p. 15, Figure 6 · p. 15
  • The NPC classifies queries as Attribute, Route, Disambiguation, or Other, then uses type-specific response logic.Attribute replies use target metadata, while disambiguation replies are “yes” only when the target is centered and within 3 m.Source: §A.1 · p. 15
  • Route replies vary by granularity: relative turning angle, a natural-language description of the first 4 m of the shortest path, or the next ShortestPathFollower action.Path descriptions use waypoints at major turns or room transitions, anchored to nearby salient objects.Source: §A.1 · p. 15
  • For other queries, the NPC answers when privileged target, task-progress, or agent-state information supports a response; otherwise it returns a default refusal.The default response is “Sorry, I cannot answer your question now.”Source: §A.1 · pp. 15–16

A.2 Episode Generation

  • Episodes comprise an initial agent pose, an instruction, and target viewpoints; pose sampling in 18 scenes plus VLN-CE poses yielded 3,785 initial poses.Candidate poses were accepted only when on the navmesh, collision-free, and within scene bounds; target viewpoints are navigable points within instance boxes expanded by 0.6 m.Source: §A.2 · p. 17
  • Full instructions identify a specific instance using discriminative attributes and spatial relations, whereas partial instructions require only the target category.Full instructions are generated from house-level dictionaries and a spatial-relation graph, with GPT-4o producing the natural-language wording.Source: §A.2 · p. 17, Table 7 · p. 17

A.3 Training Trajectories collection

The paper describes frontier-based trajectory collection, VLLN training and prompting, and real-world robot execution. Together, these sections specify how the benchmark data is collected, how the model predicts movement and dialog, and how its outputs are deployed.

A.3 Training Trajectories collection

  • The FBE collector uses a ground-truth detector and selects the frontier nearest the previous one with 90% probability, or the target-nearest frontier with 10% probability.When the target is observed, it navigates directly to it; Figure 7 summarizes trajectory, dialog, question-type, and target-object distributions.Source: §A.3 · p. 17, Figure 7 · p. 18

A.4 Training Implementation

  • The collected training data includes 20,833 ObjectNav, 17,224 IGN, and 11,661 IIGN trajectories; ablated VLLN variants train on one task’s data only.VLLN is trained jointly on all three datasets unless otherwise specified.Source: §A.4 · p. 19
  • VLLN prompts combine historical and current egocentric images with a look-down image, retaining the latter at 640 × 480 for waypoint localization.The supervision predicts a discrete movement control before localizing a floor goal with image coordinates, or generates a natural-language oracle query; dialog history is also included for dialog-enabled samples.Source: §A.4 · p. 19

A.5 Real-world Implementation

  • In real-world deployment, a predicted floor-image goal is projected through depth into a 3D point for a PID-based local planner, while questions go to a human oracle.The robot resumes navigation after receiving the oracle’s reply.Source: §A.5 · p. 20

B.1 Gap between Objectnav and IGN

The error analysis identifies same-category instance confusion as a major source of the ObjectNav–IGN performance gap. It also describes human failures from ambiguous references and incomplete visual information, alongside exploration failures in Human-GPT trials.

B.1 Gap between Objectnav and IGN

  • The failure taxonomy distinguishes detection, exploration, and stopping errors, with detection errors including missed, wrong-category, and ambiguous-instance cases.Stop failure means stopping near the target but outside the stopping threshold; exploration failure means never observing the target.Source: §B.1 · pp. 20–21
  • Nearly 30% of evaluation episodes involve Ambiguous Detection, the largest ObjectNav–IGN performance gap, attributed mainly to insufficient attribute–image alignment.Agents stop at a same-category distractor rather than grounding the complete instance description to the matching object.Source: §B.1 · p. 21

B.2 Human Failure Analysis

  • Human failures include referential ambiguity between same-category instances and partial observability that prompts commitment without disambiguation.In Human-GPT trials, exploration also fails when the target is difficult to locate or the scene is large enough to exceed the step limit.Source: §B.2 · p. 22, Figure 10 · p. 22

B.3 Question-Type Ablation

The ablation evaluates each question type, pairs of types, and all three while holding other settings fixed. Each type contributes to the improvement, with route questions alone accounting for less than half of the full gain.

B.3 Question-Type Ablation

  • Each question type, each pair, and all three types are evaluated with all other settings fixed.Source: §B.3 · p. 23
  • SR rises by 1.2 points with attribute questions, 2.2 with disambiguation, and 2.4 with route questions; Attr.+Route yields the largest two-type gain (+3.4).Route questions provide the largest single-type gain, but account for only 41.38% of the full improvement.Source: §B.3 · p. 23, Table 8 · p. 23

B.4 Oracle-Backbone Ablation

The oracle-backbone ablation tests whether evaluation results depend on GPT-4o. SR varies little across the tested alternatives, supporting robustness to oracle choice.

B.4 Oracle-Backbone Ablation

  • SR varies by only 0.6 points (24.8–25.4) across three alternative oracle backbones, indicating robustness to oracle choice.Source: §B.4 · p. 23, Table 9 · p. 23
  • The ablation replaces GPT-4o with three alternatives while keeping all other settings fixed.Source: §B.4 · p. 23
  • The reported results indicate VL-LN Bench can be reproduced with open-source or lower-cost oracle models.Source: §B.4 · p. 23

B.5 Dialog-Turn Budget Ablation

The dialog-turn ablation examines how dialog budget affects performance and query use. Performance generally improves with more turns, while the agent averages fewer than two questions even at larger budgets.

B.5 Dialog-Turn Budget Ablation

  • SR and SPL generally improve as the dialog-turn budget increases.The largest improvement occurs from 0 to 1 turn, likely because the first question often helps eliminate distractors and narrow candidate instances.Source: §B.5 · p. 23, Table 10 · p. 23
  • With larger dialog budgets, VLLN still asks fewer than two questions on average (1.71–1.98).Source: §B.5 · p. 23
  • The average number of dialog turns in the training distribution is around 1.9, a trend the authors suggest may shape query efficiency and dialog frequency.Source: §B.5 · p. 23

C Limitation and Future Work

The discussion identifies weak image–attribute alignment and limitations in dialog-policy learning and oracle assistance. It proposes training and oracle directions to address these issues.

C.1 Ambiguity Detection

  • Weak image–attribute alignment is a limitation; hard negatives such as same-category objects differing in color, material, or placement are proposed to strengthen ambiguity detection.Source: §C.1 · p. 24

C.2 Learning Better Dialog Policies

  • Future agents could learn question-asking policies through reinforcement learning, with rewards for ambiguity reduction, efficient exploration, and task completion with minimal turns.Source: §C.2 · p. 24

C.3 Toward a More Helpful Oracle

  • The oracle is reliable enough for evaluation but not fully aligned with human assistance; Human–GPT reaches 91% SR versus 93% for Human–Human while using 9.72 versus 2.04 turns on average.The authors attribute the gap to queries outside the oracle’s knowledge or response schema and users seeking additional confirmation; proposed improvements include uncertainty-aware responses, candidate comparison, proactive clarification, and adaptive route guidance.Source: §C.3 · p. 24

Related papers

Loading 2512.22342v5…