AVERT-VLN: Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation

I. INTRODUCTION

• Unseen layouts and visual conditions can cause execution to drift from instructions, while continued autonomous actions may compound errors.The proposed deployment strategy requests lightweight correction when needed rather than relying on continuous supervision.Source: §I · p. 1
• AVERT-VLN links off-track detection, human-assisted recovery, and failure-driven policy improvement through a shared vision-language Monitor.The Monitor assesses instruction–execution consistency separately from the navigation model’s action-generation process.Source: §I · pp. 1–2
• LOSTNAV DATASET supplies 20K counterfactual risk trajectories with rule-based deviation labels for Monitor training.The Monitor is fine-tuned first on 40K normal trajectories, then jointly on normal and risk trajectories to recognize semantic deviations.Source: §I · p. 2
• The framework pairs Asynchronous Sidecar Monitoring for abstention and recovery with Trajectory-Anchored Preference Learning for decision-level preferences under shared contexts.Preference supervision targets only the decisions selected for correction.Source: §I · p. 2
• Experiments on R2R-CE and RxR-CE val-unseen show human-assisted recovery gains across three navigation architectures.Autonomous ablations separately examine failure-derived preference learning without online recovery.Source: §I · p. 2

A. Execution Assessment and Interactive Assistance

• AVERT-VLN’s Monitor assesses instruction–execution consistency outside the navigation model’s native action-generation process and supports recovery across architectures.Related approaches include advisor queries, confidence-based or learned assistance requests, progress estimation, VLM state reasoning, and deviation-triggered rollback.Source: A. Execution Assessment and… · p. 2

B. Learning from Navigation Failures

• Trajectory-Anchored Preference Learning uses Monitor judgments to locate deviation-associated decision anchors and corrective feedback to form shared-context preference pairs, optimizing only targeted outputs.Related work mines error trajectories, reanchors corrections to valid states, extracts supervision from valid prefixes and divergence points, or uses rollback and privileged interventions.Source: B. Learning from Navigation Failures · p. 2

A. Overview

• AVERT-VLN keeps the navigation model’s native action-generation process while using Monitor verdicts for recovery and deviation-associated failures for decision-level preference supervision.The framework connects the monitoring and recovery loop to later policy improvement.Source: A. Overview · p. 2

B. Counterfactual Risk-Trajectory Construction

• A navigation deviation is an observable instruction mismatch in target selection or navigation behavior; geometric distance alone can miss valid alternative routes and nearby incorrect targets.LOST labels are assigned only for verifiable instruction conflicts, with diagnostics grounded in the instruction, visual history, and current observation.Source: B. Counterfactual Risk-Trajectory… · pp. 2–3
• Interventions vary behavior mode (explore or exploit) and target granularity (region or object), yielding four controlled intervention types.Before rollout, candidates must pass structural checks and physical and semantic checks, including correspondence, alignment, visibility, navigability, and consistency with the intended intervention.Source: B. Counterfactual Risk-Trajectory… · p. 3, 2) Candidate Validation · p. 3

3) Simulator Rollout and Annotation:

• Validated simulator interventions produce Dlost with 20K counterfactual risk trajectories and rendered observations.Rule-based criteria assign deviation labels, and a model generates the corresponding diagnostic descriptions.Source: 3) Simulator Rollout and Annotation · p. 3
• Scene metadata and privileged navigation states support data construction and verification but are not provided as Monitor inputs..Source: 3) Simulator Rollout and Annotation · p. 3

C. Monitor Training

• Monitor diagnostics summarize instruction progress supported by visual history and assess the current observation against that progress.The Monitor is initialized from Qwen3.5-4B and conditioned on the instruction, up to eight ordered history frames, and the current observation.Source: C. Monitor Training · p. 3

1) Stage 1: Normal-Trajectory Reasoning:

• The Monitor is first fine-tuned on 40K normal trajectories to model instruction progress and diagnose the current state during normal execution.Source: 1) Stage 1: Normal-Trajectory Reasoning · p. 3

2) Stage 2: Joint Pass/Lost Fine-Tuning:

• Joint fine-tuning adds semantic-deviation diagnostics and LOST verdicts for risk examples while retaining normal diagnostic supervision.Normal and risk executions use a common input–output format.Source: 2) Stage 2: Joint Pass/Lost Fine-Tuning · p. 3

D. Asynchronous Sidecar Monitoring

• The Monitor assesses navigation asynchronously from the instruction, visual history, and current observation, without changing the navigation model’s native action-generation process.It generates a diagnostic response from which the controller obtains a PASS or LOST verdict.Source: D. Asynchronous Sidecar Monitoring · p. 4
• A committed LOST verdict halts unexecuted actions and invalidates pending monitoring requests; human guidance supplies a pixel goal or local turn, after which monitoring resumes on a new observation.Previously executed actions are not reverted; recovery starts from the current execution state.Source: D. Asynchronous Sidecar Monitoring · p. 4

E. Trajectory-Anchored Preference Learning

• Monitor judgments locate decision anchors, and corrective feedback supplies a task-consistent preferred decision paired with the original output under the same context.The context comprises the instruction, visual history, and current observation.Source: E. Trajectory-Anchored Preference… · p. 4

1) Anchored Preference Construction:

• For a LOST trajectory, the anchor is the nearest preceding PASS boundary, where feedback defines a preferred decision and the original output is the alternative.Pixel-goal corrections target visual grounding; local-turn corrections target direction selection.Source: 1) Anchored Preference Construction · p. 4

• The corrected stage selects a candidate-specific token mask, and the trainable and frozen reference policies score candidates under the same context and mask.The policies are initialized from the same DualVLN System 2 checkpoint.Source: 2) Decision-Focused Optimization · p. 4
• The reference-adjusted preference margin compares the preferred and original outputs under the trainable and reference policies.Source: 2) Decision-Focused Optimization · p. 4
• The method optimizes a decision-focused objective based on DPO, with β scaling the margin and λ weighting the preferred-output likelihood term.σ is the sigmoid function; β > 0 and λ ≥ 0.Source: 2) Decision-Focused Optimization · p. 4

A. Experimental Setup

• The evaluation covers deviation recognition, preference learning’s effect on autonomous navigation, and online human-assisted recovery across navigation architectures.Source: A. Experimental Setup · p. 4

1) Navigation Benchmarks and Metrics:

• Navigation is evaluated on R2R-CE and RxR-CE val-unseen splits using NE, OS, SR, SPL, and nDTW; the table also records observation requirements.NE is measured in meters; the other navigation scores are percentages.Source: 1) Navigation Benchmarks and Metrics · pp. 4–5, Table I · p. 5

2) LOSTAWARE BENCHMARK:

• LOSTAWARE pairs failed Follower states with Guide states from the same scene, instruction, and goal, retaining 1,095 matched pairs for evaluation.The 2,190 samples include instruction, visual history, and current observation; recall and F1 use LOST as the positive class.Source: 2) LOSTAWARE BENCHMARK · p. 5

1) Monitor Training:

• Stage 1 fine-tunes the Monitor on normal trajectories for one epoch, then Stage 2 trains for one epoch on normal and lost trajectories.The Monitor is initialized from Qwen3.5-4B and full-parameter fine-tuned on eight NVIDIA H20 GPUs.Source: 1) Monitor Training · p. 5

2) Navigation Training:

• System 2 of DualVLN is trained for two epochs with β = 0.5 and preferred-output weight λ = 0.1.Training uses the Trajectory-Anchored Preference Learning objective in Eq. (5).Source: 2) Navigation Training · p. 5

3) Online Monitoring:

• The asynchronous monitoring interface batches up to four requests with a 0.05 s coalescing timeout; a committed LOST verdict triggers human-assisted recovery.Recovery uses a pixel goal or a local turning command.Source: 3) Online Monitoring · p. 5

1) Navigation Performance:

• The † configurations add monitoring and human-assisted recovery to three existing checkpoints without additional navigation-model training; the final ‡ configuration applies preference learning and then fine-tunes System 1.The final configuration uses DualVLN’s dual-system architecture and the same monitoring and recovery evaluation interface.Source: 1) Navigation Performance · pp. 5–6
• 76.2% SR on R2R-CE and 66.3% SR on RxR-CE; versus DualVLN with the same interface, SR rises 0.7 points on both and SPL rises 1.1 and 5.0 points, respectively.The SPL gains are particularly pronounced on RxR-CE, alongside a modest increase in task completion.Source: 1) Navigation Performance · p. 6
• Across three navigation architectures, monitoring and human-assisted recovery raise SR by 6.8–14.5 points on R2R-CE and 2.1–8.3 points on RxR-CE, while reducing NE on both.For InternVLA-N1 on RxR-CE, nDTW decreases from 62.6 to 57.3; the passage notes that recovery movements may contribute to the lower trajectory fidelity.Source: 1) Navigation Performance · p. 6

2) Lost-State Recognition and Diagnostic Reasoning:

• Monitor-4B achieves 63.38% recall and 59.52% F1 for LOST detection, the highest among 18 compared models, with precision of 56.10%.Its recall and F1 exceed Qwen3.5-27B, the strongest F1 baseline, by 20.37 and 9.73 points, respectively.Source: 2) Lost-State Recognition and… · p. 6, Figure 4 · p. 6
• Monitor-4B scores highest across six diagnostic dimensions; its largest reported gaps are 21.45 points in history consistency and 11.17 in recovery usefulness.Its respective scores are 74.93 and 61.44, evaluated on a 0–100 scale.Source: 2) Lost-State Recognition and… · p. 6, Figure 4 · p. 6

D. Component Ablation

• 60.21% SR is achieved by combining DPO with online monitoring and human-assisted recovery on RxR-CE val-unseen; removing recovery or preference learning lowers SR by 5.18 or 2.43 points.System 2 of DualVLN is the component-ablation baseline.Source: D. Component Ablation · p. 6
• With assistance disabled, DPO exceeds additional SFT by 1.62 percentage points in SR; in two selected episodes, +DPO reaches destinations that Original fails to finish.The qualitative comparison covers autonomous navigation in RxR-CE val-unseen episodes.Source: D. Component Ablation · p. 7, Figure 5 · p. 7

V. CONCLUSION

• AVERT-VLN couples online human-assisted recovery with failure-derived decision-level preference supervision, and the conclusion reports improvements in task success and subsequent navigation without online assistance.The reported conclusions draw on experiments across R2R-CE and RxR-CE and multiple navigation architectures.Source: §V · p. 7