PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

Yiyuan Yao, Xinpeng Ding, Minglei Chen, Fei Gao, Lihuo He, Jie Yuan, Aijun Li, Nannan Wang, Xinbo Gao

arXiv:2610.34028v1cs.CV

TL;DR

VLN action explanations must account for evidence spread across historical and current observations and associated with different instruction stages, a challenge existing methods overlook. The paper proposes stage-aware temporal and spatial attribution, reporting benchmark gains over DEX-AR and demonstrating failure diagnosis and Unitree G1 navigation applicability.

  • Problem

    VLN action evidence may span historical and current observations and different instruction stages, while existing explanations can be insensitive to stage transitions.

    Source: §I · pp. 1–2

  • Method

    The framework estimates instruction-stage associations from action-conditioned temporal attributions to refine current explanations with relevant history, then filters spatial responses using instruction–patch associations.

    Source: §I · p. 2

  • Results

    On R2R and RxR, Precision improves by 34.58% and 35.37% and F1-Score by 25.72% and 23.50%, respectively, over DEX-AR, while maintaining competitive decision faithfulness.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework is demonstrated as a tool for diagnosing VLN failures and as applicable to real-world navigation on a Unitree G1 robot.

    Source: §I · p. 2, E. Attribution as a Diagnostic Tool · p. 8, §V · p. 8

Abstract

from arXiv · show

As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important for reliable deployment. Existing explainability methods overlook a key challenge in Vision-Language Navigation (VLN): evidence for the current action may be distributed across historical and current observations and associated with different instruction stages. We identify instruction-stage insensitivity, where existing methods produce similar explanations despite changes across different navigation stages. To address this issue, we propose a stage-aware attribution framework for VLN. We first construct action-conditioned temporal instruction--visual attributions across observations. These attributions are projected into instruction space to estimate stage associations, which guide the aggregation of relevant historical evidence for refining the current explanation. An instruction-guided spatial filter further suppresses isolated responses while preserving compact, instruction-relevant regions. Experiments across multiple VLN policies and benchmarks show improved attribution quality over existing methods. We further demonstrate that our method acts as a diagnostic tool for analyzing failure cases, and validate its applicability to real-world navigation on a Unitree G1 robot.

I. INTRODUCTION

VLN explanations must link visual evidence across navigation history to the instruction stage relevant to each action. The paper introduces stage-aware temporal and spatial attribution and reports gains over DEX-AR on two benchmarks.

I. INTRODUCTION

  • Existing explanations can remain similar across instruction stages, while action evidence may span historical and current observations and mix evidence from different subgoals.Discarding history can remove useful evidence; retaining it without stage awareness can mix explanations across subgoals.Source: §I · pp. 1–2
  • The framework builds action-conditioned temporal instruction–visual attributions, estimates stage relevance, and uses stage-aligned history to refine current-frame explanations.Instruction-guided spatial filtering reduces isolated responses while preserving compact, relevant evidence.Source: §I · p. 2
  • On R2R and RxR, Precision improves by 34.58% and 35.37% and F1-Score by 25.72% and 23.50%, respectively, over DEX-AR, while decision faithfulness remains competitive.The authors also report stage-consistent localization, failure diagnosis, and real-world applicability on a Unitree G1.Source: §I · p. 2
  • The paper identifies and quantifies instruction-stage insensitivity as explanations changing insufficiently when the relevant navigation stage progresses.The motivating context is that visual evidence for a navigation action and its relationship to the instruction remain underexplored.Source: §I · pp. 1–2

II. RELATED WORKS

Prior explainability work largely targets visual or generated-text outputs, while VLN requires explaining actions with evidence distributed across observations and instruction stages. The paper positions its framework as a stage-aware way to explain that evidence.

II. RELATED WORKS

  • Existing visual explainability methods target recognition, generated tokens, or perturbed action inputs; the paper instead focuses on how visual evidence across observations relates to instruction stages and supports an action.Perturbation-based ISS requires repeated policy evaluations, which the cited discussion describes as costly to extend to VLN with changing viewpoints and accumulated history.Source: §II · p. 2

A. Vision-Language Navigation Preliminaries

  • In VLN, evidence for a current action may span historical and current observations and correspond to different parts of the instruction.The framework constructs temporal attributions, estimates stage relevance, refines current attribution with stage-aligned historical support, and filters spatial responses.Source: A. Vision-Language Navigation… · pp. 2–3, Figure 2 · p. 3
  • The paper distinguishes its explanation scope from the policy’s internal computation: it describes evidence relationships rather than the policy’s computation.Source: A. Vision-Language Navigation… · p. 3

B. Temporal Instruction–Visual Attribution

The method builds action-conditioned instruction–patch attributions for each observation, then retains their temporal order and instruction dimension to represent evidence across history and the current frame.

B. Temporal Instruction–Visual Attribution

  • For each observation, the method constructs an instruction–patch attribution conditioned on the selected action.The attribution links instruction tokens to visual patches through the multimodal self-attention matrix.Source: B. Temporal Instruction–Visual… · p. 3
  • Attention is combined with its gradient, with positive contributions aggregated using normalized sensitivity weights, to measure action relevance.The weights are computed from maximum absolute gradients within instruction–visual attention submatrices.Source: B. Temporal Instruction–Visual… · p. 3
  • The observation-wise matrices are collected in temporal order, preserving instruction associations for identifying evidence stages before aggregation.The current observation’s matrix is M_T; preceding matrices represent historical observations.Source: B. Temporal Instruction–Visual… · p. 3

C. Instruction-Stage Attribution

Instruction-stage attribution projects each observation’s action-conditioned evidence into instruction space and estimates the stage associated with that evidence.

C. Instruction-Stage Attribution

  • Each observation’s attribution is projected into instruction space by marginalizing over visual patches, producing a distribution over instruction tokens.A normalized positional expectation summarizes this distribution.Source: C. Instruction-Stage Attribution · p. 3
  • The resulting scalar is a soft descriptor of the instruction stage associated with an observation’s evidence, not a measure of task completion.The current-stage reference uses μ_T while retaining token-level support in p_T.Source: C. Instruction-Stage Attribution · p. 3

D. Stage-Guided Temporal Refinement

Stage-guided temporal refinement selects historical evidence aligned with the current instruction stage and uses it to reweight the current attribution while retaining newly observed evidence.

D. Stage-Guided Temporal Refinement

  • Historical observations are weighted by their correspondence to the current instruction stage, with closer stages receiving larger weights.The sensitivity to stage differences is controlled by τ_H > 0.Source: D. Stage-Guided Temporal Refinement · p. 4
  • Stage-aligned historical support is aggregated and used to reweight the current-frame attribution.The minimum retained response δ prevents weak historical support from suppressing newly observed evidence.Source: D. Stage-Guided Temporal Refinement · p. 4
  • The refinement reduces cross-stage interference along the instruction dimension while preserving current spatial localization and historical maps for tracing earlier evidence.Historical maps remain available to identify earlier evidence supporting the same action.Source: D. Stage-Guided Temporal Refinement · p. 4

E. Instruction-Guided Spatial Filter

The spatial filter regulates suppression using instruction–patch associations, aiming to remove isolated responses without discarding compact instruction-relevant evidence.

E. Instruction-Guided Spatial Filter

  • Rank-Gaussian filtering can suppress isolated responses but may also remove compact task-relevant regions.The proposed filter therefore regulates suppression using instruction–patch associations.Source: E. Instruction-Guided Spatial Filter · p. 4
  • Patch preservation scores use the strongest instruction association, whose bidirectional geometric mean favors associations concentrated from both patch and token perspectives.The patch-level attribution map retains the original instruction–patch associations.Source: E. Instruction-Guided Spatial Filter · p. 4
  • Strongly grounded patches resist spatial suppression, while weakly associated responses receive stronger suppression, preserving compact instruction-relevant evidence.The suppression proposed by the rank-Gaussian filter is modulated by the preservation score.Source: E. Instruction-Guided Spatial Filter · p. 4

IV. EXPERIMENTS

Experiments evaluate explanation quality on R2R and RxR in the VLN-CE setting, using decision faithfulness and instruction-aware semantic alignment. They compare methods on NaVILA and visualize explanations across navigation stages.

A. Experimental Setup

  • Evaluation uses R2R and RxR in Habitat’s VLN-CE setting, with NaVILA, NaVid, and StereoNav policies.Explanations are computed from the current view and valid model-selected historical observations; baselines use the same selected action and policy inputs.Source: A. Experimental Setup · p. 4
  • Decision faithfulness measures the action’s corrected negative-log-likelihood increase after salient regions are masked; larger values indicate more faithful explanations.The metric subtracts the average effect of random masks to account for perturbations caused by masking itself.Source: A. Experimental Setup · p. 4
  • Semantic alignment evaluates whether highlighted regions match instruction-relevant visual concepts, with Precision indicating alignment strength.The evaluation uses semantic scene annotations and instruction-conditioned category relevance to form a target.Source: A. Experimental Setup · pp. 4–5
  • The qualitative comparison visualizes early-history, recent-history, and current observations with attribution maps at Steps 1, 7, and 21.Highlighted instruction tokens are induced by action-conditioned instruction attribution at each decision step.Source: Figure 3 · p. 5

B. Comparison with State-of-the-Art Methods

Across R2R and RxR, the method improves semantic alignment and overall F1-Score while maintaining decision faithfulness. Comparisons also report gains across VLN policies and show more stage-consistent explanations.

B. Comparison with State-of-the-Art Methods

  • Precision rises from 0.3591 to 0.4370 on R2R and from 0.2862 to 0.4241 on RxR over the strongest baselines, while the method achieves the best overall F1-Score on both.It also records the highest ∆NLLcorr on R2R (0.3198) and the highest faithfulness score on RxR (0.2846).Source: B. Comparison with State-of-the-Art… · pp. 5–6
  • Stage-guided temporal refinement reduces cross-stage interference and focuses attribution on evidence relevant to the current decision.In Fig. 4, red boxes mark irrelevant regions.Source: Figure 4 · p. 6
  • The spatial-filtering comparison shows the base rank filter and uni- and bidirectional variants, with green and red boxes marking relevant and irrelevant regions.The caption identifies the variants but does not state a quantitative winner.Source: Figure 5 · p. 6
  • Across navigation stages, explanations shift toward evidence associated with the active sub-instruction and retain relevant evidence across historical and current observations.The prose contrasts this with diffuse DEX-AR activations and broad regions highlighted by GradCAM and TAM.Source: B. Comparison with State-of-the-Art… · p. 6
  • Across NaVid, NaVILA, and StereoNav on both R2R and RxR, the method consistently outperforms DEX-AR, with Precision gains reaching 34.58% on R2R and 42.03% on RxR.The reported gains across policies and datasets indicate generalization beyond a specific VLN architecture.Source: B. Comparison with State-of-the-Art… · p. 6
  • The dataset-level analysis reports stronger sensitivity to instruction-stage transitions and more concentrated visual explanations than the baseline TAM [13].The figure caption describes token-level instruction attribution, frame-level similarity, and a dataset-level baseline comparison.Source: Figure 6 · p. 6

C. Ablation Study

The ablations examine temporal refinement and spatial filtering, while stage-sensitivity measures compare instruction and visual attributions across navigation stages.

C. Ablation Study

  • Temporal refinement alone raises Precision from 0.4180 to 0.4328 and F1-Score from 0.3252 to 0.3386, with cleaner current-frame attributions after cross-stage responses are suppressed.The refinement enhances evidence relevant to the current decision while reducing interference from other instruction stages.Source: C. Ablation Study · p. 6
  • Spatial filtering alone raises ∆NLL from 0.2661 to 0.3082 and F1-Score from 0.3252 to 0.3553; instruction–patch associations help preserve compact relevant evidence.The bidirectional variant further reduces irrelevant activations while retaining the dining-table and wall-picture regions.Source: C. Ablation Study · p. 6
  • Instruction-attribution JSD is 0.33 vs. 0.02 and cross-stage visual similarity is 57.4% vs. 96.8%, while top-10% attribution mass is 43.4% vs. 13.5% for the method versus baseline.The authors interpret these comparisons as clearer attribution changes across stages and more concentrated visual explanations.Source: C. Ablation Study · p. 7

D. Real-World Evaluation

The framework is deployed on a Unitree G1 humanoid robot in two representative navigation trajectories.

D. Real-World Evaluation

  • Across two real-world trajectories, action-conditioned attributions highlight visual evidence relevant to the current instruction at different decision steps.The figure pairs egocentric attribution maps with third-person execution views.Source: D. Real-World Evaluation · p. 7

E. Attribution as a Diagnostic Tool

The attribution examples expose distinct sources of erroneous VLN decisions, while the conclusion reports improved semantic alignment on R2R and RxR with strong decision faithfulness and applicability to real-world navigation on a Unitree G1 robot.

E. Attribution as a Diagnostic Tool

  • Three representative failures involve premature progression to a later subgoal, neglect of an action phrase despite grounding its landmark, and incomplete grounding of the current step.The examples associate each error with attribution patterns such as attention to a later hallway clause, emphasis on “stove” over “passing by,” or carryover from the preceding “kitchen” clause.Source: E. Attribution as a Diagnostic Tool · p. 7
  • The attribution examples expose stage mismatches, action-phrase neglect, and incomplete grounding as patterns underlying erroneous VLN decisions.The diagnostic cases compare visual and instruction-token attributions with the model’s selected action and instruction requirement.Source: E. Attribution as a Diagnostic Tool · pp. 7–8

V. CONCLUSION

  • Experiments on R2R and RxR show improved semantic alignment while maintaining strong decision faithfulness, with further application to failure diagnosis and real-world navigation on a Unitree G1 robot.The conclusion identifies extension to longer histories and continuous low-level control as future work.Source: §V · p. 8

Related papers

Loading 2610.34028v1…