PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework

Delun Li, Xueyao Wei, Rui Chen, Yifei Wang, Junzhi Yu, Wenjie Song, Xingyu Chen

arXiv:2610.17089v1cs.ROcs.AI

TL;DR

Embodied navigation often assumes traversability and object states remain fixed, although reaching a goal may require interactions that change connectivity or reveal targets. The paper introduces a benchmark and graph-based agent for these tasks; the agent outperforms adapted baselines across task types.

  • Problem

    Many embodied navigation tasks assume traversability and object states remain fixed, despite goals that may require changing connectivity or revealing hidden targets.

    Source: §1 · p. 2

  • Method

    The paper introduces a 3K-episode benchmark and a three-stage agent that uses a persistent interaction graph for perception, task-relevant action selection, and verified state updates.

    Source: §1 · p. 2

  • Results

    MultINav-Agent consistently outperforms adapted baselines across channel, container, and mixed tasks; overall SR is 41.63% versus 10.53% for the adapted baseline.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The benchmark and framework evaluate navigation when channel and container interactions change connectivity, target visibility, and their sequential dependencies.

    Source: §1 · p. 2

Abstract

from arXiv · show

Existing embodied navigation tasks commonly assume that traversability and object states remain fixed during an episode. Yet reaching a goal may require opening a door to access a room and then opening a container to reveal a hidden target. We formulate interactive navigation beyond reachability, unifying channel-induced changes in spatial connectivity, container-induced changes in target visibility, and their sequential dependencies. We introduce MultINav-Bench, a 3K-episode ObjectGoal benchmark in MolmoSpaces spanning channel, container, and mixed tasks, with interaction-necessity annotations covering required and unnecessary episodes. Its data generation pipeline automates task construction across multiple task types. The evaluation protocol separately measures navigation success, interaction effectiveness, and cost. We further propose MultINav-Agent, a three-stage agentic framework built on a persistent, state-conditioned interaction graph. Semantic interactive perception grounds entities and interaction states; a vision-language model selects task-relevant actions based on expected changes in connectivity and target observability; and post-interaction verification updates the graph to guide subsequent navigation, exploration, or interaction. Experiments on MultINav-Bench show that MultINav-Agent consistently outperforms adapted baselines across channel, container, and mixed tasks.

1 Introduction

MultINav addresses navigation settings where interaction changes connectivity or target visibility, sometimes in sequence. The paper introduces a benchmark and an agent that reasons about those changes, with reported gains over adapted baselines across task types.

Problem formulation

  • MultINav unifies channel-induced connectivity changes, container-induced target-visibility changes, and their sequential dependencies.A door may enable room entry before opening a container reveals the target, making interaction relevance and timing part of navigation decisions.Source: §1 · p. 2

Contributions

  • MultINav-Bench is a 3K-episode MolmoSpaces ObjectGoal benchmark spanning channel, container, and mixed tasks, with required and unnecessary interaction annotations.Its pipeline automates task construction, and its evaluation separates navigation success, interaction effectiveness, and cost.Source: §1 · p. 2
  • MultINav-Agent consistently outperforms adapted baselines across channel, container, and mixed tasks.Its persistent, state-conditioned interaction graph connects perception, action selection, and verified state updates.Source: §1 · p. 2

2 Related Work

Prior navigation work commonly evaluates fixed episode states or interaction effects such as free-space recovery and path cost. This paper distinguishes interactions by whether they restore connectivity or expose a target, and evaluates both under an ObjectGoal endpoint.

Embodied Navigation and Interactive Benchmarks

  • Prior navigation tasks generally assume fixed episode states, while interactive benchmarks often retain a navigation endpoint as interactions change free space or path cost.Interactive Gibson, Interactive Visual Navigation, and CaMP alter free space or path cost; InterNav evaluates door-mediated connectivity in PointGoal.Source: §2 · p. 2

Benchmark comparison

  • MultINav-Bench retains ObjectGoal while evaluating channel reconfiguration, target-container exposure, their composition, and required/unnecessary controls.Source: §2 · p. 3

Interactive Navigation and Decision

  • Many interaction methods focus on obstacle clearance, path cost, or control feasibility; MultINav-Agent instead selects candidates by expected graph effect and verifies that effect before replanning.This distinguishes restoring connectivity from exposing a target, with required/unnecessary and distractor controls assessing task-level accessibility.Source: §2 · p. 3

Semantic Navigation and Structured Scene Representations

  • Scene graphs and language-guided planners support search and feasible-action planning, but action-induced connectivity or visibility changes are seldom treated as transitions to predict and verify.MultINav-Agent specializes interactive graphs to verified channel effects on connectivity and container effects on target visibility and accessibility.Source: §2 · p. 3

3 MultINav Benchmark

MultINav-Bench evaluates ObjectGoal navigation when interaction changes reachability, target visibility, or both. Its benchmark construction, episode taxonomy, and separate outcome and process metrics support diagnostic evaluation.

3 MultINav Benchmark

  • MultINav-Bench freezes 3,000 domain-balanced light episodes in MolmoSpaces, with 1,000 each for channel, container, and mixed tasks across 639 houses.Episodes include interaction-necessity labels, distractor objects, and oracle demonstration paths.Source: §3 · p. 4

3.1 Task and Interaction Taxonomy

  • Channel tasks use doors to change reachability, container tasks expose occluded targets, and mixed tasks require sequentially composing both operations.Of the 3,000 episodes, 2,654 are labeled required and 346 unnecessary, enabling measurement of unnecessary interaction attempts.Source: §3.1 · pp. 4–5

3.2 Data Generation and Collection

  • Candidates are synthesized using counterfactual reachability and visibility checks, then validated through articulation readback, target-identity checks, and oracle rollouts.The full collection mode also records synchronized visual streams, base trajectories, end-effector targets, and articulation states for expert demonstrations.Source: §3.2 · p. 5

3.3 Evaluation Metrics

  • Five complementary metrics separate terminal goal achievement from intermediate interaction efficiency, including SR, SPL, ISR, IP, and Total Cost.ISR measures verified necessary-effect progress, IP measures target-class physical effects, and Total Cost accounts for navigation and interaction expenditure.Source: §3.3 · p. 5

4 MultINav-Agent Framework

MultINav-Agent uses a persistent, state-conditioned interaction graph to connect perception, action choice, and post-interaction verification. Its updates guide later navigation, exploration, or interaction.

4 MultINav-Agent Framework

  • MultINav-Agent maintains a persistent, state-conditioned interaction graph whose verified changes guide subsequent navigation, exploration, or interaction.Source: §4 · p. 5

4.1 Semantic Interactive Perception

  • Semantic interactive perception tracks rooms, entities, objects, and the robot through connectivity, containment, and visibility relations, retaining interaction states across observations.Relation validity depends on observed state: a closed door blocks its room connection, while a verified opening permits traversal.Source: §4.1 · pp. 5–6

4.2 Effect-Guided Action Selection

  • The agent generates navigation, exploration, and interaction candidates, filters interactions for feasibility and evidence reliability, then ranks eligible actions by expected task contribution.The VLM uses the goal, relevant graph, candidate effects, route costs, and decision history; navigation and exploration remain options when interaction offers little progress.Source: §4.2 · p. 6

4.3 Interaction Verification and Graph Updates

  • After an interaction, fresh visual observations and execution feedback assess the achieved state and graph effect; uncertain evidence is marked uncertain and failed attempts remain in decision history.Updated connectivity or containment and visibility relations regenerate candidates, supporting sequential dependencies such as opening a door before a container.Source: §4.3 · p. 7

5 Experiments

Experiments compare interactive navigation methods on MultINav-Bench, examine performance across task types and agent components, and illustrate sequential interactions in simulated and real-world settings. MultINav-Agent outperforms the adapted MoMa-LLM baseline on aggregate interactive metrics, while ablations expose trade-offs among its components.

5.1 Experimental Setup

  • The 3,000 episodes span 639 houses, balanced across Channel, Container, and Mixed splits; the primary interactive comparison uses restricted ground-truth perception for both methods.The interface supplies identities, categories, and spatial observations for visible objects, while agents must still explore and choose interactions.Source: §5.1 · p. 7

5.2 Main Results

  • 41.63% SR versus 10.53% (+31.10 percentage points) for the MoMa-LLM adaptation: MultINav-Agent completes 1,249 of 3,000 episodes.SPL rises from 0.0848 to 0.2903, ISR from 23.69% to 49.38%, and IP from 27.59% to 46.79%; normalized Cost falls from 0.9309 to 0.7743.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Ablation Study

  • On a fixed 200-episode Mixed subset, Full has the highest SR and IP, while flat memory raises ISR and greedy selection improves SPL and Cost.Flat Object Memory lowers IP from 40.4% to 26.7%; Greedy Selection has SPL 0.134 versus 0.107 and Cost 0.933 versus 0.961. These results expose trade-offs rather than uniform gains.Source: Table 4 · p. 8, §5.3 · p. 10

5.2 Main Results

  • SR gains over MoMa-LLM are 36.20, 34.70, and 22.40 percentage points on Channel, Container, and Mixed tasks, respectively.Mixed ISR is 45.07%, but only 24.70% of episodes complete all necessary effects; partial interaction progress does not ensure task completion.Source: §5.2 · p. 8
  • Mixed-task normalized Cost remains near ceiling for both methods—0.9493 for MultINav-Agent and 0.9951 for MoMa-LLM—partly reflecting navigation failures.Cost is one for navigation failures; the paper cautions that aggregate SPL and Cost do not establish shorter paths or cheaper execution conditional on success.Source: §5.2 · p. 9

5.3 Ablation Study

  • Perception-only Update lowers SR from 23.0% to 5.5% (17.5 percentage points), while IP remains 36.6%.Among tested variants, this is the strongest evidence for retaining execution-informed graph updates when coordinating channel and container interactions.Source: §5.3 · p. 10

5.4 Case Studies

  • A simulated episode shows two doors opened before the refrigerator, with verified graph updates guiding replanning as the target is revealed.The figure distinguishes navigation and exploration in blue from interactions in orange.Source: Figure 4 · p. 9, §5.4 · p. 10

5.5 Real-World Demonstration

  • In a Go2 demonstration, human-assisted door and refrigerator opening lets the robot explore newly accessible space and inspect the refrigerator for an apple.The rollout links door-induced connectivity change with container opening that exposes the target.Source: Figure 5 · p. 9, §5.5 · p. 10

6 Conclusion

The paper formulates MultINav, introduces a 3,000-episode benchmark and an agent with verified interaction-graph updates, and reports higher success than the adapted baseline plus a real-world Go2 demonstration.

6 Conclusion

  • 41.63% SR versus 10.53% for the adapted baseline is reported for MultINav-Agent with its persistent interaction graph and verified state updates.The paper also reports transfer to a real-world Go2 demonstration.Source: §6 · p. 10
  • MultINav unifies channel and container interactions within ObjectGoal navigation, and MultINav-Bench contains 3,000 episodes.The benchmark and agent are introduced together.Source: §6 · p. 10
  • The paper identifies broader interaction types, learned perception, and end-to-end policy training as future directions.Source: §6 · p. 10

A Experimental Platform and Reproducibility

The appendix reports the simulation and demonstration hardware, software stack, and evaluation configuration, and notes that its distribution details expand the main paper’s light ObjectGoal split.

A Experimental Platform and Reproducibility

  • Simulation used NVIDIA A100 GPUs with 80 GB memory, while the real-world demonstration used one NVIDIA GeForce RTX 4090 with 48 GB memory.The simulation stack used Python 3.11.15, MuJoCo 3.4.0, and PyTorch 2.7.1.Source: A Experimental Platform and… · p. 15
  • The MolmoSpaces ObjectGoal evaluation required both goal distance and head-camera visibility for terminal success.Compared methods shared RGB-D observations, restricted records for currently visible objects, and the interaction executor; operation cost used λ = 0.3, µ = 1, and B = 30, with B a normalization constant rather than a rollout limit.Source: A Experimental Platform and… · p. 15

B MultINav-Bench Distribution Details

  • The appendix expands the main paper’s light ObjectGoal split, whose 3,000 episodes span 1,000 channel, 1,000 container, and 1,000 mixed tasks across 639 houses.Its distribution counts come from the frozen balanced manifest and quality reports; the passage says primary main-paper claims do not depend on the appendix.Source: B MultINav-Bench Distribution Details · p. 15

B.1 Domain, Recipe, and Necessity

These appendix details describe how necessity labels are assigned, summarize path, target-category, and house distributions, and identify the interaction hardware represented in the benchmark.

B.1 Domain, Recipe, and Necessity

  • Container and mixed episodes are labeled required based on the construction-time closed state and validated reference plan; unnecessary episodes use only the channel distractor-door recipe.The necessity label describes intended interaction dependency, while terminal success is assessed separately from completion of annotated effects.Source: §B.1 · p. 15

B.2 Path Length and Interaction Count

  • Mixed episodes are longer on average because they chain channel and container interactions; channel counts below one reflect unnecessary distractor episodes with zero required interactions.Source: §B.2 · p. 15

B.3 Target-Category Skew

  • Egg is the most frequent listed target category (439; 14.6%), while plate, soapbar, and spraybottle appear only a few times.Main tables use split macro averages so no single category dominates the headline comparison by construction, though residual imbalance remains.Source: §B.3 · p. 16
  • Container hardware is dominated by hinged doors and sliding drawers, with channel and mixed episodes also using hinged doors.Source: §B.3 · p. 16

B.4 Per-House Multiplicity

  • The benchmark spans 639 houses, and train/evaluation separation is enforced at the house level.Per-domain counts are 253 channel, 569 container, and 369 mixed houses; the frozen set permits multiple episodes per house, up to 16 in one house.Source: §B.4 · p. 16

B.5 Ablation Subset Construction

The appendix fixes a shared mixed-task subset for ablations, specifies a runtime scoring-eligibility check, and describes domain-specific collection validation for channel, container, and mixed episodes.

B.5 Ablation Subset Construction

  • The ablation subset contains 200 mixed episodes from 165 houses, with every episode requiring both channel and container interactions.All methods and three ablated variants use the same fixed subset, sampled without replacement without using policy outcomes.Source: §B.5 · p. 16

B.6 Scoring Eligibility and Quality Gates

  • A runtime gate excludes traces from primary aggregation when frozen oracle terminal waypoints are inconsistent with the live selected-instance pose.The offline audit reported 3,000/3,000 valid episodes, zero duplicate case IDs, and zero missing comparable ground-truth path lengths; runtime eligibility is a separate online check.Source: §B.6 · p. 16

B.7 Domain-Specific Collection Gates

  • Channel collection validates closed- and open-state route feasibility, requiring routes to become feasible after the annotated door effect for required episodes.Unnecessary episodes retain a feasible route with closed distractor doors; a route-shortening pilot is diagnostic rather than a balanced benchmark stratum.Source: §B.7 · p. 16
  • Container collection verifies target-compartment identity, an interaction pose, the opening effect, and a terminal pose; mixed collection validates the full ordered door-container rollout.For mixed episodes, the post-door state alone is not terminal, and omitting either annotated effect must break the validated chain; failures across domains trigger rejection.Source: §B.7 · p. 16

C Task Specifications and Collection Richness

The collection pipeline separates task specification from record richness, while the persistent interaction graph tracks how entity states affect traversal and visibility and verifies updates after actions.

C Task Specifications and Collection Richness

  • Only the ObjectGoal-light cell is frozen as MultINav-Bench; other specification and record-richness cells reuse the same interaction entities, roles, effects, and oracle transitions.Table 7 summarizes three task specifications supported by the shared collector.Source: C Task Specifications and Collection… · p. 17, Table 7 · p. 17
  • Light records specify scene, initial pose, goal, initial articulation states, necessity label, and construction-quality evidence, enabling episode reconstruction and scoring without a policy rollout.Full trajectory records add synchronized views, base actions, end-effector poses, and policy-rendered video, but are not required for scoring or assumed to train the evaluated policy.Source: C Task Specifications and Collection… · p. 17
  • PointGoal candidates require an open-state ground-truth path, and closed-versus-open path comparisons label channel interactions required, beneficial, or unnecessary.Beneficial strata can be denser than in the ObjectGoal split because the sampler filters by path-delta.Source: C Task Specifications and Collection… · p. 17
  • InstructionGoal generation layers rule-, LLM-, VLM-, or graph-grounded text over an oracle plan, recording entity IDs, plan steps, disclosure, and optional evidence frames.Instructions are auditable annotations grounded in the interaction graph.Source: C Task Specifications and Collection… · p. 17, Figure 6 · p. 17

D Interaction Graph and State Updates

  • The interaction graph links rooms, interactive entities, and candidate objects, with door state governing room connectivity and container state governing target visibility.Uncertain doors are conservatively blocked, and uncertain containers do not establish visibility.Source: D Interaction Graph and State Updates · p. 18
  • The graph persists entity state, confidence, and observation time across views; post-action verification writes the requested state only with sufficient matching evidence and otherwise retains uncertainty or the prior state.Verified effects reweight door-mediated edges or add newly exposed objects and their containment and visibility relations.Source: D Interaction Graph and State Updates · p. 18

E Metric Formulas

The metric protocol separates terminal navigation success, progress on required effects, target-class interaction precision, and operation cost, with explicit aggregation and scoring conventions.

E Metric Formulas

  • Success requires verified target evidence and the benchmark distance criterion, while interaction completion is neither necessary nor sufficient for terminal success.An alternative observed route or visibility change can satisfy the terminal goal without completing every annotated reference effect.Source: E Metric Formulas · p. 18
  • SPL compares successful planar base-path length with a feasible oracle reference that permits necessary interactions; interaction time and attempt cost are handled separately.Invalid reference lengths make aggregate SPL unavailable rather than changing its denominator.Source: E Metric Formulas · p. 19
  • ISR averages verified required interaction effects over required episodes, excludes unnecessary episodes, and can award partial credit even when a mixed episode fails its full chain or navigation goal.Each required ID earns credit at most once per episode.Source: E Metric Formulas · p. 19
  • Target-Class IP measures attempts producing verified effects on annotated interaction classes, while unnecessary episodes use a separate indicator and non-target-class attempts remain in the denominator.A qualifying effect can earn credit despite a failed macro acknowledgement; effect-free repeated requests do not.Source: E Metric Formulas · p. 19
  • Under the default protocol, λ = 0.3, µ = 1, and B = 30 are frozen before evaluation; B is a shared weighted operation-cost ceiling, not a rollout stopping rule.Failures and successes over budget both score one.Source: E Metric Formulas · p. 19
  • Overall values average scoring-eligible completed episodes rather than averaging split means; IP averages within-episode ratios, and ISR uses required episodes only.The main table reports 3,000 episodes per method and 2,654 required episodes for ISR.Source: E Metric Formulas · pp. 19–20
  • 60.09% ISR on required Channel episodes and 46.70% ISR on Container episodes; Mixed IP is 47.16% despite 24.10% navigation SR.Mixed Cost is 0.9493, with 759/1,000 Mixed episodes failing navigation and failures assigned cost one.Source: E Metric Formulas · p. 20
  • On the fixed 200-episode Mixed ablation subset, Full has the highest SR and IP, while Greedy Selection has the highest SPL and lowest Cost.Full is reported separately from the main benchmark aggregate; Perception-only Update yields 5.5% SR.Source: E Metric Formulas · p. 20

F Additional Implementation Notes

The implementation uses visible-entity observations and shared interaction execution, with a VLM selecting inspectable actions from graph context and current candidates; aggregate comparisons do not isolate graph updates as the cause of performance differences.

F Additional Implementation Notes

  • The primary benchmark comparison gives both interactive methods the same restricted-GT interface for visible entities, while withholding hidden entities, joint metadata, and oracle plans.The simulator interaction executor is shared, and the terminal evaluator independently checks goal distance and head-camera visibility.Source: F Additional Implementation Notes · p. 20
  • Aggregate benchmark comparisons characterize end-to-end behavior but do not by themselves isolate graph updates as the cause of performance differences.Full rollouts retain episode identifiers and success criteria, allowing trajectory records from different policies to join the frozen evaluator manifest without changing benchmark membership.Source: F Additional Implementation Notes · pp. 20–21

G.1 Model Configuration and Serving

  • MultINav-Agent uses Qwen3.6-35B-A3B-FP8 for high-level decisions, served through an OpenAI-compatible chatcompletions API.Visual input combines a global egocentric view with a target crop for fine-grained actionable-region localization.Source: §G.1 · p. 21

G.2 Subgoal Categorical Reason Codes and Subgraph Context

  • The VLM receives room and anchor entities, candidate door and container states with relational effects, and standardized reason codes such as UNLOCK_ROUTE and REVEAL_TARGET_CONTAINER.The structured protocol makes candidate choices inspectable, but selected actions still require execution and outcome verification.Source: §G.2 · p. 21

G.3 High-Level Candidate Selection Prompt

  • At each high-level decision, the prompt asks the model to rank current executable candidates while respecting route prerequisites, distinguishing observations from priors, and avoiding repeated failures.The adapter validates returned IDs against the current candidate list before execution.Source: §G.3 · p. 22

Related papers

Loading 2610.17089v1…